Version 40 (modified by hfinkel, 5 years ago) (diff)


Welcome to using LLVM/clang on the BG/Q

ANL Vesta, Mira and Cetus

If you have access to ANL's Vesta, Mira and Cetus systems, this BG/Q-enhanced LLVM/Clang is installed for you. You can use the softenv keys:

    +mpiwrapper-bgclang            bgclang wrappers and toolchain
    +mpiwrapper-bgclang.legacy     bgclang.legacy wrappers and toolchain

to have the corresponding MPI wrappers added to your path.

For the benefit of those without access, a copy of the bgclang wrapper script is available in the latest archive. For use on other systems, you'll need to change some of the paths, but this will let you see what optimization settings are used by default. You should symlink this to bgclang++ and bgclang++11 to create the C++ wrappers.

Similarly, the archive contains the MPI wrapper scripts (modified from the corresponding gcc wrapper scripts).

If you're using these patches, please subscribe to the mailing list:

Repository Information

I have yet to figure out how to best mirror my local repositories. Here's why:

  • LLVM uses subversion, and I've been using git-svn to manage local BG/Q changes.
  • git-svn uses rebasing to apply local changes on top of upstream changes; this amounts to history rewriting (dangerous but effective). I cannot simply push from the git-svn-managed repository to the public one because doing so would confuse copies cloned from the public version.
  • LLVM uses separate repositories for LLVM and clang, it is not clear how to best combine them into one public repository.

Thus, for the time being, we'll just need to use patches. Maybe I should push into separate (new) branches every time I commit? I'm certainly open to suggestions.

RPMs, etc.


See instructions below regarding how to install these.

The corresponding source RPMs are here:


See instructions below regarding how to install these.

The corresponding source RPMs are here:

r189357-20130827 (v3)

In order to make this process easier, I've converted the various build scripts and patches into relocatable RPMs. Relocatable RPMs can be installed under an arbitrary prefix directory. Download these RPMs:

A non-root (regular) user can install these RPMs (because they are relocatable), but in addition to specifying the installation prefix (with the --prefix argument), an alternate RPM database directory needs to be specified (in a directory to which you actually have write permission). For example, to install bgclang into the /tmp/bgclang directory using /tmp/rpm as the RPM database directory, run:

    rpm -Uhv --dbpath /tmp/rpm --prefix /tmp/bgclang \
        bgclang-r189357-20130827-3-1.ppc64.rpm \
        bgclang-sleef-r189357-20130827-3-1.ppc64.rpm \
        bgclang-libcxx-r189357-20130827-3-1.ppc64.rpm \

If the installation fails with an error like:

error: Failed dependencies:
	/bin/sh is needed by bgclang-r189357-20130827-3-1.ppc64

Then first install the vpkg-bin-sh-1-1.ppc64.rpm package (which is a virtual package which exists only to satisfy this /bin/sh dependency):

    rpm -Uhv --dbpath /tmp/rpm --prefix /tmp/bgclang vpkg-bin-sh-1-1.ppc64.rpm

After the install is complete, the prefix directory should look like this:

    $ ls -l /tmp/bgclang | awk '{print $1, $9, $10, $11}'
    lrwxrwxrwx bin -> current/bin
    lrwxrwxrwx compiler-rt -> current/compiler-rt
    lrwxrwxrwx current -> r189357-20130827
    lrwxrwxrwx docs -> current/docs
    lrwxrwxrwx include -> current/include
    lrwxrwxrwx lib -> current/lib
    lrwxrwxrwx libc++ -> current/libc++
    lrwxrwxrwx libstdc++fixup -> current/libstdc++fixup
    lrwxrwxrwx mpi -> current/mpi
    drwxr-xr-x r189357-20130827  
    lrwxrwxrwx scan-build -> current/scan-build
    lrwxrwxrwx scan-view -> current/scan-view
    lrwxrwxrwx share -> current/share
    lrwxrwxrwx sleef -> current/sleef
    lrwxrwxrwx wbin -> current/wbin

These symlinks are only created by the post-install scripts in the RPMs if they don't already exist. As a result, upon subsequent upgrades, you'll need to manually update the 'current' symlink as desired.

The corresponding source RPMs are here:

To rebuild these, first build the bgclang RPM, and install it. Then the others can be built, specifying the same --dbpath command-line argument provided rpm when installing the built bgclang RPM (if any).

r189357-20130827 (v2)


r188569-20130816 (v3)

r188569-20130816 (v2)




r185769-20130706 (v3)

r185769-20130706 (v2)


r185415-20130701 (v3)

r185415-20130701 (v2)






Significant improvements include:

  • Support for type-safety attributes (clang feature developed by Dmitri Gribenko for type checking calls to MPI functions).
  • LLVM now generates pre-increment load/store instructions (only r+imm form supported so far, r+r form, necessary for QPX pre-increment, is still under development).


Note: the date tag on the clang patch regressed compared to the previous version because upstream commits rendered later local changes irrelevant.



Follow the normal directions for checking out llvm and clang from the repositories (using the revision number specified in the archive name). Then apply the provided patches. Run configure (I recommend building from a directory different from the source directory). See the build scripts for how to compile everything.

For a better C++11 environment, you'll want to use libc++ as the C++ standard library implementation. The attached build script patches and builds libc++ so that it can correctly interoperate (be linked with) the libstdc++ which the MPI/PAMI implementation requires. The bgclang++11 (and corresponding MPI wrappers) setup this environment.

You'll need to build compiler-rt to use the address sanitizer feature. Do not checkout compiler-rt into the llvm directory (as per the upstream instructions), it will not be correctly cross compiled for the compute nodes. Instead, checkout compiler-rt into its own top-level directory and use the provided build script.

The install now includes a SIMD math library based on SLEEF. To make use of the patches in the archive, you'll need the source code from: (the current patches are against SLEEF version 2.80).