Installing CODES on CCI's AiMOS Power9 Supercomputer

Compatibility Note

Effectively executing on a supercomputer generally requires a balance of speed and compatibility. To be able to effectively utilize the system's compute node interconnect, spectrum-mpi should generally be used as the MPI implementation for compilation. However, at the time of writing, there seems to be trouble with the xlC_r compiler when compiling/linking CODES.

At the moment, I have been unable to load spectrum-mpi as a wrapper for GCC and so we must use mpich as the MPI implementation instead. This will generally give poorer performance than spectrum-mpi on AiMOS but should be acceptable - especially if no multi-node-communication jobs are required.

First let's get CCI account configured to be able to pull from GitHub

In your ~/.bashrc file add the following lines

export http_proxy=http://proxy:8888
export https_proxy=$http_proxy

To install CODES, you need to install ROSS.

To install ROSS in a way that is compatible with CODES on AiMOS we need to load the following three modules

module load cmake
module load gcc/8.2.0/1
module load mpich/3.3/1

Because CCI doesn't automatically symlink the ccmake3 binary to ccmake, let's add an alias. (this can be put into your .bashrc file)

alias ccmake=ccmake3

To install ROSS, we have to address a problem when using MPICH on AiMOS. Right now it uses cmake's find_package() function to look for MPI and its libraries. For some reason this stalls on AiMOS. We'll use the aimos-compatibility branch of ROSS to install.

git clone https://github.com/ROSS-org/ROSS.git
cd ROSS
git checkout aimos-compatibility
cd ..
mkdir build_ross
cd build_ross

cmake -DCMAKE_INSTALL_PREFIX:path=`pwd` -DCMAKE_C_COMPILER=$(which mpicc) -DCMAKE_CXX_COMPILER=$(which mpicxx) ../ROSS
export ROSS_PREFIX=$(pwd)

make install

To install CODES: (replace the CODES repo address with your own if you are using a different fork of CODES)

git clone http://github.com/codes-org/codes.git
cd codes
./prepare.sh
mkdir build
cd build
../configure --disable-shared --enable-static --prefix=`pwd` PKG_CONFIG_PATH=$ROSS_PREFIX/lib/pkgconfig CC=mpicc CXX=mpicxx CFLAGS=-g CXXFLAGS=-g

make install

As your configure arguments get longer and more complicated, it might help to have a bash script to organize it. Bringing in DUMPI and SWM compatibility, for example, make this long and easy to mess up. Below is an example bash script to call inside your CODES build directory.

#! /bin/bash

PREFIX=
ROSS_PKG_CFG=

DUMPI_BUILD_PATH=
SWM_PATH=
SWM_PKG_CFG=

ARGOBOTS_PKG_CFG=

OTHER_ARGS="--disable-shared--enable-static"

CODES_CFLAGS="-g"
CODES_CXXFLAGS="-g"

PKG_CONFIG_PATH="$ROSS_PKG_CFG"
WITH_ARGS=""

if [[ -n "$DUMPI_BUILD_PATH" ]]; then
    WITH_ARGS+="--with-dumpi=$DUMPI_BUILD_PATH "
fi

if [[ -n "$SWM_PATH" ]]; then
    PKG_CONFIG_PATH+=":$SWM_PKG_CFG"
    if [[ -z "$ARGOBOTS_PKG_CFG" ]]; then
        echo "Argobots pkg config path required if using SWM"
        exit 1
    fi
    PKG_CONFIG_PATH+=":$ARGOBOTS_PKG_CFG"
    WITH_ARGS+="--with-online=$SWM_PATH "
fi

#Execute from within codes build directory inside of codes source
../configure --prefix=$PREFIX $WITH_ARGS $OTHER_ARGS PKG_CONFIG_PATH=$PKG_CONFIG_PATH CC=mpicc CXX=mpicxx CFLAGS=$CODES_CFLAGS CXXFLAGS=$CODES_CXXFLAGS