Version 44 (modified by hfinkel, 4 years ago) (diff)


bgclang (LLVM/clang on the BG/Q)

ANL Vesta, Mira and Cetus

If you have access to ANL's Vesta, Mira and Cetus systems, this BG/Q-enhanced LLVM/Clang is installed for you. You can use the softenv keys:

    +mpiwrapper-bgclang            bgclang wrappers and toolchain
    +mpiwrapper-bgclang.legacy     bgclang.legacy wrappers and toolchain

to have the corresponding MPI wrappers added to your path.

Other BG/Q systems

If your system administrators have not been kind enough to install bgclang on your system, you can either direct them to this page, or install the distribution yourself. RPMs are provided (see below), and these are *relocatable* RPMs, meaning that they can be installed by a non-root user in any directory.

Please note that, if you wish to use dynamic linking (which you must do when certain features, like address sanitizer, are enabled), you must install bgclang in a directory that is mounted from the compute nodes (read-only is sufficient).

MPI and other wrappers

On an ALCF system (or any other system with a similar setup), the MPI wrapper scripts and other related programs can be easily added to your PATH (see the description of the ALCF softenv keys above). These wrappers are:

  mpicc - The MPI C99 compiler
  mpic++ and mpicxx - The MPI C++03 compiler
  mpic++11 and mpicxx11 - The MPI C++11 compiler

For those managing their own installs, note that these are located in the PREFIX/mpi/{bgclang,bgclang.legacy}/bin directories. To use bgclang without using the MPI wrappers:

  bgclang - The C99 compiler
  bgclang++ - The C++03 compiler
  bgclang++11 - The C++11 compiler

For those managing their own installs, note that these are located in the PREFIX/wbin directory.

If you're using bgclang, please subscribe to the mailing list:

Repository Information

I have yet to figure out how to best mirror my local repositories. Here's why:

  • LLVM uses subversion, and I've been using git-svn to manage local BG/Q changes.
  • git-svn uses rebasing to apply local changes on top of upstream changes; this amounts to history rewriting (dangerous but effective). I cannot simply push from the git-svn-managed repository to the public one because doing so would confuse copies cloned from the public version.
  • LLVM uses separate repositories for LLVM and clang, it is not clear how to best combine them into one public repository.

Thus, for the time being, we'll just need to use patches. Maybe I should push into separate (new) branches every time I commit? I'm certainly open to suggestions.

RPMs, etc.

r192411-20131010 (updated)

See instructions below regarding how to install these.

The corresponding source RPMs are here: (updated)


See instructions below regarding how to install these.

The corresponding source RPMs are here:


See instructions below regarding how to install these.

The corresponding source RPMs are here:


See instructions below regarding how to install these.

The corresponding source RPMs are here:

r189357-20130827 (v3)

In order to make this process easier, I've converted the various build scripts and patches into relocatable RPMs. Relocatable RPMs can be installed under an arbitrary prefix directory. Download these RPMs:

A non-root (regular) user can install these RPMs (because they are relocatable), but in addition to specifying the installation prefix (with the --prefix argument), an alternate RPM database directory needs to be specified (in a directory to which you actually have write permission). For example, to install bgclang into the /tmp/bgclang directory using /tmp/rpm as the RPM database directory, run:

    rpm -Uhv --dbpath /tmp/rpm --prefix /tmp/bgclang \
        bgclang-r189357-20130827-3-1.ppc64.rpm \
        bgclang-sleef-r189357-20130827-3-1.ppc64.rpm \
        bgclang-libcxx-r189357-20130827-3-1.ppc64.rpm \

If the installation fails with an error like:

error: Failed dependencies:
	/bin/sh is needed by bgclang-r189357-20130827-3-1.ppc64

Then first install the vpkg-bin-sh-1-1.ppc64.rpm package (which is a virtual package which exists only to satisfy this /bin/sh dependency):

    rpm -Uhv --dbpath /tmp/rpm --prefix /tmp/bgclang vpkg-bin-sh-1-1.ppc64.rpm

After the install is complete, the prefix directory should look like this:

    $ ls -l /tmp/bgclang | awk '{print $1, $9, $10, $11}'
    lrwxrwxrwx bin -> current/bin
    lrwxrwxrwx compiler-rt -> current/compiler-rt
    lrwxrwxrwx current -> r189357-20130827
    lrwxrwxrwx docs -> current/docs
    lrwxrwxrwx include -> current/include
    lrwxrwxrwx lib -> current/lib
    lrwxrwxrwx libc++ -> current/libc++
    lrwxrwxrwx libstdc++fixup -> current/libstdc++fixup
    lrwxrwxrwx mpi -> current/mpi
    drwxr-xr-x r189357-20130827  
    lrwxrwxrwx scan-build -> current/scan-build
    lrwxrwxrwx scan-view -> current/scan-view
    lrwxrwxrwx share -> current/share
    lrwxrwxrwx sleef -> current/sleef
    lrwxrwxrwx wbin -> current/wbin

These symlinks are only created by the post-install scripts in the RPMs if they don't already exist. As a result, upon subsequent upgrades, you'll need to manually update the 'current' symlink as desired.

The corresponding source RPMs are here:

To rebuild these, first build the bgclang RPM, and install it. Then the others can be built, specifying the same --dbpath command-line argument provided rpm when installing the built bgclang RPM (if any).

r189357-20130827 (v2)


r188569-20130816 (v3)

r188569-20130816 (v2)




r185769-20130706 (v3)

r185769-20130706 (v2)


r185415-20130701 (v3)

r185415-20130701 (v2)






Significant improvements include:

  • Support for type-safety attributes (clang feature developed by Dmitri Gribenko for type checking calls to MPI functions).
  • LLVM now generates pre-increment load/store instructions (only r+imm form supported so far, r+r form, necessary for QPX pre-increment, is still under development).


Note: the date tag on the clang patch regressed compared to the previous version because upstream commits rendered later local changes irrelevant.



Follow the normal directions for checking out llvm and clang from the repositories (using the revision number specified in the archive name). Then apply the provided patches. Run configure (I recommend building from a directory different from the source directory). See the build scripts for how to compile everything.

For a better C++11 environment, you'll want to use libc++ as the C++ standard library implementation. The attached build script patches and builds libc++ so that it can correctly interoperate (be linked with) the libstdc++ which the MPI/PAMI implementation requires. The bgclang++11 (and corresponding MPI wrappers) setup this environment.

You'll need to build compiler-rt to use the address sanitizer feature. Do not checkout compiler-rt into the llvm directory (as per the upstream instructions), it will not be correctly cross compiled for the compute nodes. Instead, checkout compiler-rt into its own top-level directory and use the provided build script.

The install now includes a SIMD math library based on SLEEF. To make use of the patches in the archive, you'll need the source code from: (the current patches are against SLEEF version 2.80).