Version 60 (modified by hfinkel, 4 years ago)

--

bgclang (LLVM/clang on the BG/Q)

For usage information, and information specific to using bgclang on ALCF's BG/Q machines (Vesta, Mira and Cetus), please visit:  http://www.alcf.anl.gov/user-guides/bgclang-compiler

Other BG/Q systems

If your system administrators have not been kind enough to install bgclang on your system, you can either direct them to this page, or install the distribution yourself. RPMs are provided (see below), and these are *relocatable* RPMs, meaning that they can be installed by a non-root user in any directory.

Please note that, if you wish to use dynamic linking (which you must do when certain features, like address sanitizer, are enabled), you must install bgclang in a directory that is mounted from the compute nodes (read-only is sufficient).

Mailing List

If you're using bgclang, please subscribe to the mailing list:  http://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss.

bgclang downloads (for installing on your own)

For those managing their own installs, note that the MPI wrappers are installed in the PREFIX/mpi/{bgclang,bgclang.legacy}/bin directories. The non-MPI compiler wrappers are located in the PREFIX/wbin directory.

RPMs, etc.

r200401-20140129

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-binutils-r200401-20140129-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r200401-20140129-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r200401-20140129-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r200401-20140129-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libomp-r200401-20140129-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r200401-20140129-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

See instructions below regarding how to install these (but also install the libomp package).

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-binutils-r200401-20140129-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r200401-20140129-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r200401-20140129-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r200401-20140129-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libomp-r200401-20140129-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r200401-20140129-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

r193439-20131025

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-binutils-r193439-20131025-2-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r193439-20131025-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r193439-20131025-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r193439-20131025-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libomp-r193439-20131025-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r193439-20131025-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

See instructions below regarding how to install these.

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-binutils-r193439-20131025-2-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r193439-20131025-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r193439-20131025-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r193439-20131025-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libomp-r193439-20131025-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r193439-20131025-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

r192411-20131010

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-binutils-r192411-20131010-2-1.ppc64.rpm (updated)

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r192411-20131010-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r192411-20131010-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r192411-20131010-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libomp-r192411-20131010-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r192411-20131010-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

See instructions below regarding how to install these.

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-binutils-r192411-20131010-2-1.src.rpm (updated)

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r192411-20131010-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r192411-20131010-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r192411-20131010-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libomp-r192411-20131010-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r192411-20131010-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

r192297-20131009

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-binutils-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libomp-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

See instructions below regarding how to install these.

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-binutils-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libomp-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

r192001-20131004

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-binutils-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libomp-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

See instructions below regarding how to install these.

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-binutils-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libomp-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

r190771-20130914

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r190771-20130914-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r190771-20130914-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r190771-20130914-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libomp-r190771-20130914-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r190771-20130914-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

See instructions below regarding how to install these.

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r190771-20130914-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r190771-20130914-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r190771-20130914-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libomp-r190771-20130914-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r190771-20130914-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

r189357-20130827 (v3)

In order to make this process easier, I've converted the various build scripts and patches into relocatable RPMs. Relocatable RPMs can be installed under an arbitrary prefix directory. Download these RPMs:

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r189357-20130827-3-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r189357-20130827-3-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r189357-20130827-3-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r189357-20130827-3-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

A non-root (regular) user can install these RPMs (because they are relocatable), but in addition to specifying the installation prefix (with the --prefix argument), an alternate RPM database directory needs to be specified (in a directory to which you actually have write permission). For example, to install bgclang into the /tmp/bgclang directory using /tmp/rpm as the RPM database directory, run:

    rpm -Uhv --dbpath /tmp/rpm --prefix /tmp/bgclang \
        bgclang-r189357-20130827-3-1.ppc64.rpm \
        bgclang-sleef-r189357-20130827-3-1.ppc64.rpm \
        bgclang-libcxx-r189357-20130827-3-1.ppc64.rpm \
        bgclang-compiler-rt-r189357-20130827-3-1.ppc64.rpm

If the installation fails with an error like:

error: Failed dependencies:
	/bin/sh is needed by bgclang-r189357-20130827-3-1.ppc64

Then first install the vpkg-bin-sh-1-1.ppc64.rpm package (which is a virtual package which exists only to satisfy this /bin/sh dependency):

    rpm -Uhv --dbpath /tmp/rpm --prefix /tmp/bgclang vpkg-bin-sh-1-1.ppc64.rpm

After the install is complete, the prefix directory should look like this:

    $ ls -l /tmp/bgclang | awk '{print $1, $9, $10, $11}'
    lrwxrwxrwx bin -> current/bin
    lrwxrwxrwx compiler-rt -> current/compiler-rt
    lrwxrwxrwx current -> r189357-20130827
    lrwxrwxrwx docs -> current/docs
    lrwxrwxrwx include -> current/include
    lrwxrwxrwx lib -> current/lib
    lrwxrwxrwx libc++ -> current/libc++
    lrwxrwxrwx libstdc++fixup -> current/libstdc++fixup
    lrwxrwxrwx mpi -> current/mpi
    drwxr-xr-x r189357-20130827  
    lrwxrwxrwx scan-build -> current/scan-build
    lrwxrwxrwx scan-view -> current/scan-view
    lrwxrwxrwx share -> current/share
    lrwxrwxrwx sleef -> current/sleef
    lrwxrwxrwx wbin -> current/wbin

These symlinks are only created by the post-install scripts in the RPMs if they don't already exist. As a result, upon subsequent upgrades, you'll need to manually update the 'current' symlink as desired.

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r189357-20130827-3-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r189357-20130827-3-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r189357-20130827-3-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r189357-20130827-3-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

To rebuild these, first build the bgclang RPM, and install it. Then the others can be built, specifying the same --dbpath command-line argument provided rpm when installing the built bgclang RPM (if any).

r189357-20130827 (v2)

r189357-20130827

r188569-20130816 (v3)

r188569-20130816 (v2)

r188569-20130816

r188410-20130814

r186563-20130718

r185769-20130706 (v3)

r185769-20130706 (v2)

r185769-20130706

r185415-20130701 (v3)

r185415-20130701 (v2)

r185415-20130701

r184137-20130617

r181589-20130510

r180241-20130423

r157929-20120603

Significant improvements include:

  • Support for type-safety attributes (clang feature developed by Dmitri Gribenko for type checking calls to MPI functions).
  • LLVM now generates pre-increment load/store instructions (only r+imm form supported so far, r+r form, necessary for QPX pre-increment, is still under development).

r155766-20120426

Note: the date tag on the clang patch regressed compared to the previous version because upstream commits rendered later local changes irrelevant.

r154948-20120422

Building

Note: This information applies only to building from the outdated tar.gz archives. Please use the source RPMs if you wish to build bgclang yourself (or install the binary RPMs if you don't).

Follow the normal directions for checking out llvm and clang from the llvm.org repositories (using the revision number specified in the archive name). Then apply the provided patches. Run configure (I recommend building from a directory different from the source directory). See the build scripts for how to compile everything.

For a better C++11 environment, you'll want to use libc++ as the C++ standard library implementation. The attached build script patches and builds libc++ so that it can correctly interoperate (be linked with) the libstdc++ which the MPI/PAMI implementation requires. The bgclang++11 (and corresponding MPI wrappers) setup this environment.

You'll need to build compiler-rt to use the address sanitizer feature. Do not checkout compiler-rt into the llvm directory (as per the upstream instructions), it will not be correctly cross compiled for the compute nodes. Instead, checkout compiler-rt into its own top-level directory and use the provided build script.

The install now includes a SIMD math library based on SLEEF. To make use of the patches in the archive, you'll need the source code from:  http://shibatch.sourceforge.net/ (the current patches are against SLEEF version 2.80).

Repository Information

I have yet to figure out how to best mirror my local repositories. Here's why:

  • LLVM uses subversion, and I've been using git-svn to manage local BG/Q changes.
  • git-svn uses rebasing to apply local changes on top of upstream changes; this amounts to history rewriting (dangerous but effective). I cannot simply push from the git-svn-managed repository to the public one because doing so would confuse copies cloned from the public version.
  • LLVM uses separate repositories for LLVM and clang, it is not clear how to best combine them into one public repository.

Thus, for the time being, we'll just need to use patches (now contained in the source RPMs below). As time allows, I'll move to a different setup for version control.

Attachments