Version 48 (modified by hfinkel, 4 years ago)

--

bgclang (LLVM/clang on the BG/Q)

ANL Vesta, Mira and Cetus

If you have access to ANL's Vesta, Mira and Cetus systems, this BG/Q-enhanced LLVM/Clang is installed for you. You can use the softenv keys:

    +mpiwrapper-bgclang            bgclang wrappers and toolchain
    +mpiwrapper-bgclang.legacy     bgclang.legacy wrappers and toolchain

to have the corresponding MPI wrappers added to your path.

Other BG/Q systems

If your system administrators have not been kind enough to install bgclang on your system, you can either direct them to this page, or install the distribution yourself. RPMs are provided (see below), and these are *relocatable* RPMs, meaning that they can be installed by a non-root user in any directory.

Please note that, if you wish to use dynamic linking (which you must do when certain features, like address sanitizer, are enabled), you must install bgclang in a directory that is mounted from the compute nodes (read-only is sufficient).

MPI and other wrappers

On an ALCF system (or any other system with a similar setup), the MPI wrapper scripts and other related programs can be easily added to your PATH (see the description of the ALCF softenv keys above). These wrappers are:

  mpicc - The MPI C99 compiler
  mpic++ and mpicxx - The MPI C++03 compiler
  mpic++11 and mpicxx11 - The MPI C++11 compiler

For those managing their own installs, note that these are located in the PREFIX/mpi/{bgclang,bgclang.legacy}/bin directories. To use bgclang without using the MPI wrappers:

  bgclang - The C99 compiler
  bgclang++ - The C++03 compiler
  bgclang++11 - The C++11 compiler

For those managing their own installs, note that these are located in the PREFIX/wbin directory.

Mailing list

If you're using bgclang, please subscribe to the mailing list:  http://lists.alcf.anl.gov/mailman/listinfo/llvm-bgq-discuss

General usage

bgclang command-line argument handling is designed to be similar to gcc's command-line argument handling, and where possible, bgclang tries to support the same flags.

Like bgxlc and powerpc64-bgq-linux-gcc, bgclang defaults to static linking. If you pass the -dynamic flags (or the -shared flag), then dynamic linking will be used instead. As with gcc, when compiling objects intended to become part of a shared library or dynamically-linked executable, you should probably also pass the -fPIC flag. In general, use of dynamically-linked executables and shared libraries is discouraged on the BG/Q.

OpenMP

bgclang fully supports the OpenMP 3.1 specification, and features from OpenMP 4 are currently being added. To enable OpenMP support, pass the -fopenmp flag when both compiling and linking.

Note that bgclang's OpenMP runtime library (derived from Intel's open-source implementation) is different from that used by powerpc64-bgq-linux-gcc and bgxlc, and linking the OpenMP runtime library from either of those two compilers with an application compiled with bgclang -fopenmp will likely result in runtime failures.

Fast-math optimizations

bgclang supports a number of fast-math optimizations, enabled by passing -ffast-math, which increase performance but violate the relevant IEEE specification on floating-point computation. -ffast-math is conceptually similar to IBM's -qnostrict compiler flag.

Vector (QPX) intrinsics and math functions

bgclang supports the same QPX vector intrinsics (vec_add, etc.) as IBM's compiler, and it understands the vector4double type. No special flags or header files are required to enable this support.

bgclang also comes with a vector math library (derived from Naoki Shibata's SLEEF library). To use this library, include the qpxmath.h header. The bgclang wrapper scripts automatically handle linking to the vector math library, so no special linking flags are required.

#include <qpxmath.h>

the following functions are available (the functions with the _u1 suffix have no more than 1 ulp error):

vector4double xldexp(vector4double x, const int *q);
void xilogb(vector4double d, int *l);

vector4double xsin(vector4double d);
vector4double xcos(vector4double d);
void xsincos(vector4double d, vector4double *ds, vector4double *dc);
vector4double xtan(vector4double d);
vector4double xasin(vector4double s);
vector4double xacos(vector4double s);
vector4double xatan(vector4double s);
vector4double xatan2(vector4double y, vector4double x);
vector4double xlog(vector4double d);
vector4double xexp(vector4double d);
vector4double xpow(vector4double x, vector4double y);

vector4double xsinh(vector4double d);
vector4double xcosh(vector4double d);
vector4double xtanh(vector4double d);
vector4double xasinh(vector4double s);
vector4double xacosh(vector4double s);
vector4double xatanh(vector4double s);

vector4double xcbrt(vector4double d);

vector4double xexp2(vector4double a);
vector4double xexp10(vector4double a);
vector4double xexpm1(vector4double a);
vector4double xlog10(vector4double a);
vector4double xlog1p(vector4double a);

vector4double xsin_u1(vector4double d);
vector4double xcos_u1(vector4double d);
void xsincos_u1(vector4double d, vector4double *ds, vector4double *dc);
vector4double xtan_u1(vector4double d);
vector4double xasin_u1(vector4double s);
vector4double xacos_u1(vector4double s);
vector4double xatan_u1(vector4double s);
vector4double xatan2_u1(vector4double y, vector4double x);
vector4double xlog_u1(vector4double d);
vector4double xcbrt_u1(vector4double d);

plus single precision versions (which are named like the double-precision variants but have an 'f' as a suffix like this):

...
vector4double xsinf(vector4double d);
vector4double xcosf(vector4double d);
...
vector4double xsinf_u1(vector4double d);
vector4double xcosf_u1(vector4double d);
...

In addition, you can use IBM's SIMD MASS library by including the appropriate header and linking with -lmass_simd. Compared to IBM's SIMD MASS library, bgclang's vector math functions tend to be slower but more accurate. The maximum errors from the qpxmath functions are provided here: qpxmath_max_error.txt Download.

For convenience, if you define QPXMATH_MASS_SIMD_FUNCTIONS before including the qpxmath.h header, aliases will also be defined for libmass_simd function names (sind4, etc.). Note, however, that libmass_simd provides some functions not provided by bgclang's vector math library. Also, bgclang's vector math library provides vectorized ldexp and ilogb functions (which libmass_simd does not provide).

Autovectorization

bgclang's autovectorization support is enabled by default with the optimization flag -O3. There are two types of autovectorization used by bgclang: Loop autovectorization (which can be disabled using -fno-vectorize) and SLP autovectorization (which can be disabled using -fno-slp-vectorize) for the autovectorization of non-loop code.

bgclang can currently transform calls to the following standard library (libm) math functions into calls to its vector math library as part of the autovectorization process: acos, acosh, asin, asinh, atan, atan2, atanh, cbrt, cos, cosh, exp, exp10, exp2, expm1, log, log10, log1p, pow, sin, sinh, tan, tanh, along with the single-precision versions. Also sqrt (and division), but only with -ffast-math. For sin, cos, tan, asin, acos, atan, atan2, log faster (but slightly less accurate) variants are used with -ffast-math.

Memory bounds checking (address sanitizer)

bgclang supports a memory bounds-checking feature called address sanitizer. This feature, enabled by passing -fsanitize=address, instruments the compiled code, and will produce an error on both out-of-bounds stack and heap access.

When using address sanitizer, dynamic linking must be used (and bgclang will default to using dynamic linking when -fsanitize=address is passed).

LTO is a powerful feature of bgclang and its associated toolchain which enables the compiler to perform additional global optimizations, such as function inlining, as part of the final linking process. This can be expensive in terms of compile time, but can yield significant runtime performance gains.

To use LTO you must pass the -flto flag to bgclang, both when compiling and also when linking. In addition, because the object file produced by bgclang when using LTO is in a custom format, special tools are necessary in order to:

  • Combine such object files into static archives: use bgclang-ar (or equivalently powerpc64-bgq-linux-clang-ar) instead of ar (or powerpc64-bgq-linux-ar). Failure to do so will result in errors when attempting to use the static archives during linking; specifically this error (which is misleading in this context):
  error adding symbols: Archive has no index; run ranlib to add one
  • Inspect the symbols defined in such object files: use bgclang-nm (or equivalently powerpc64-bgq-linux-clang-nm) instead of nm (or powerpc64-bgq-linux-nm).

bgclang's LTO capability is currently experimental. The are known issues with how debugging data is handled, and you might run into problems using -flto and -g together. We're currently working on fixing these issues.

FAQ

Why do I receive linking errors complaining about multiple definitions of inline functions?

The source code you're compiling probably assumes the GNU semantics for the inline keyword, and not those defined by the C99 standard. Compile your code with the -fgnu89-inline flag to force bgclang to use the non-standard GNU semantics.

Linking code compiled with bgclang++ together with code compiled with bgclang++11 does not work, why?

Code compiled using bgclang++ uses the same libstdc++ standard template library (STL) implementation as the system-default GNU powerpc64-bgq-linux-g++ compiler. This provides compatibility with C++ libraries, including some system libraries, compiled with the GNU toolchain. This STL implementation, however, cannot provide a conforming C++11 programming environment, and so bgclang++11 uses an up-to-date STL implementation derived from LLVM's libc++. Unfortunately, this STL implementation is incompatible with libstdc++, and so linking errors will result for functions that use STL objects as part of their signatures (i.e. parameter or return types).

I'd like to use bgclang's OpenMP support and also link against code that uses IBM's OpenMP implementation (such as IBM's SMP ESSL library). Can I do that?

No, unfortunately both bgclang's OpenMP library and IBM's OpenMP library define functions of the same name, and using both at the same time is not generally possible. That having been said, if you're willing to play games with how your application is linked, it might be possible, and you should ask for advise on the mailing list.

Is there a corresponding Fortran compiler available?

No, not yet. This is also being worked on.

Repository Information

I have yet to figure out how to best mirror my local repositories. Here's why:

  • LLVM uses subversion, and I've been using git-svn to manage local BG/Q changes.
  • git-svn uses rebasing to apply local changes on top of upstream changes; this amounts to history rewriting (dangerous but effective). I cannot simply push from the git-svn-managed repository to the public one because doing so would confuse copies cloned from the public version.
  • LLVM uses separate repositories for LLVM and clang, it is not clear how to best combine them into one public repository.

Thus, for the time being, we'll just need to use patches (now contained in the source RPMs below). As time allows, I'll move to a different setup for version control.

RPMs, etc.

r192411-20131010

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-binutils-r192411-20131010-2-1.ppc64.rpm (updated)

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r192411-20131010-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r192411-20131010-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r192411-20131010-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libomp-r192411-20131010-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r192411-20131010-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

See instructions below regarding how to install these.

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-binutils-r192411-20131010-2-1.src.rpm (updated)

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r192411-20131010-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r192411-20131010-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r192411-20131010-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libomp-r192411-20131010-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r192411-20131010-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

r192297-20131009

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-binutils-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libomp-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r192297-20131009-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

See instructions below regarding how to install these.

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-binutils-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libomp-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r192297-20131009-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

r192001-20131004

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-binutils-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libomp-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r192001-20131004-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

See instructions below regarding how to install these.

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-binutils-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libomp-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r192001-20131004-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

r190771-20130914

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r190771-20130914-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r190771-20130914-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r190771-20130914-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libomp-r190771-20130914-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r190771-20130914-1-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

See instructions below regarding how to install these.

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r190771-20130914-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r190771-20130914-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r190771-20130914-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libomp-r190771-20130914-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r190771-20130914-1-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

r189357-20130827 (v3)

In order to make this process easier, I've converted the various build scripts and patches into relocatable RPMs. Relocatable RPMs can be installed under an arbitrary prefix directory. Download these RPMs:

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-r189357-20130827-3-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-sleef-r189357-20130827-3-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-libcxx-r189357-20130827-3-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/bgclang-compiler-rt-r189357-20130827-3-1.ppc64.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/RPMS/ppc64/vpkg-bin-sh-1-1.ppc64.rpm

A non-root (regular) user can install these RPMs (because they are relocatable), but in addition to specifying the installation prefix (with the --prefix argument), an alternate RPM database directory needs to be specified (in a directory to which you actually have write permission). For example, to install bgclang into the /tmp/bgclang directory using /tmp/rpm as the RPM database directory, run:

    rpm -Uhv --dbpath /tmp/rpm --prefix /tmp/bgclang \
        bgclang-r189357-20130827-3-1.ppc64.rpm \
        bgclang-sleef-r189357-20130827-3-1.ppc64.rpm \
        bgclang-libcxx-r189357-20130827-3-1.ppc64.rpm \
        bgclang-compiler-rt-r189357-20130827-3-1.ppc64.rpm

If the installation fails with an error like:

error: Failed dependencies:
	/bin/sh is needed by bgclang-r189357-20130827-3-1.ppc64

Then first install the vpkg-bin-sh-1-1.ppc64.rpm package (which is a virtual package which exists only to satisfy this /bin/sh dependency):

    rpm -Uhv --dbpath /tmp/rpm --prefix /tmp/bgclang vpkg-bin-sh-1-1.ppc64.rpm

After the install is complete, the prefix directory should look like this:

    $ ls -l /tmp/bgclang | awk '{print $1, $9, $10, $11}'
    lrwxrwxrwx bin -> current/bin
    lrwxrwxrwx compiler-rt -> current/compiler-rt
    lrwxrwxrwx current -> r189357-20130827
    lrwxrwxrwx docs -> current/docs
    lrwxrwxrwx include -> current/include
    lrwxrwxrwx lib -> current/lib
    lrwxrwxrwx libc++ -> current/libc++
    lrwxrwxrwx libstdc++fixup -> current/libstdc++fixup
    lrwxrwxrwx mpi -> current/mpi
    drwxr-xr-x r189357-20130827  
    lrwxrwxrwx scan-build -> current/scan-build
    lrwxrwxrwx scan-view -> current/scan-view
    lrwxrwxrwx share -> current/share
    lrwxrwxrwx sleef -> current/sleef
    lrwxrwxrwx wbin -> current/wbin

These symlinks are only created by the post-install scripts in the RPMs if they don't already exist. As a result, upon subsequent upgrades, you'll need to manually update the 'current' symlink as desired.

The corresponding source RPMs are here:

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-r189357-20130827-3-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-sleef-r189357-20130827-3-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-libcxx-r189357-20130827-3-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/bgclang-compiler-rt-r189357-20130827-3-1.src.rpm

 http://www.alcf.anl.gov/~hfinkel/bgclang/SRPMS/vpkg-bin-sh-1-1.src.rpm

To rebuild these, first build the bgclang RPM, and install it. Then the others can be built, specifying the same --dbpath command-line argument provided rpm when installing the built bgclang RPM (if any).

r189357-20130827 (v2)

r189357-20130827

r188569-20130816 (v3)

r188569-20130816 (v2)

r188569-20130816

r188410-20130814

r186563-20130718

r185769-20130706 (v3)

r185769-20130706 (v2)

r185769-20130706

r185415-20130701 (v3)

r185415-20130701 (v2)

r185415-20130701

r184137-20130617

r181589-20130510

r180241-20130423

r157929-20120603

Significant improvements include:

  • Support for type-safety attributes (clang feature developed by Dmitri Gribenko for type checking calls to MPI functions).
  • LLVM now generates pre-increment load/store instructions (only r+imm form supported so far, r+r form, necessary for QPX pre-increment, is still under development).

r155766-20120426

Note: the date tag on the clang patch regressed compared to the previous version because upstream commits rendered later local changes irrelevant.

r154948-20120422

Building

Follow the normal directions for checking out llvm and clang from the llvm.org repositories (using the revision number specified in the archive name). Then apply the provided patches. Run configure (I recommend building from a directory different from the source directory). See the build scripts for how to compile everything.

For a better C++11 environment, you'll want to use libc++ as the C++ standard library implementation. The attached build script patches and builds libc++ so that it can correctly interoperate (be linked with) the libstdc++ which the MPI/PAMI implementation requires. The bgclang++11 (and corresponding MPI wrappers) setup this environment.

You'll need to build compiler-rt to use the address sanitizer feature. Do not checkout compiler-rt into the llvm directory (as per the upstream instructions), it will not be correctly cross compiled for the compute nodes. Instead, checkout compiler-rt into its own top-level directory and use the provided build script.

The install now includes a SIMD math library based on SLEEF. To make use of the patches in the archive, you'll need the source code from:  http://shibatch.sourceforge.net/ (the current patches are against SLEEF version 2.80).

Attachments