[TOC]

ParaStation MPI is an implementation of the Message-Passing Interface (MPI) Standard. Being an MPICH-derivate it bases on the MPICH-4.1.1 implementation.

As shown in the figure above, ParaStation MPI relies on the low-level communication layer pscom for the implementation of the PSP Device on the ADI-Layer of the MPICH architecture. ParaStation MPI provides full MPI-4 compliance including support for the MPI Sessions model. Furthermore, the Process Management Interface for Exascale (PMIx) is supported as interface to the process manager.

• C compiler with C99 support
• ParaStation Communication library (pscom)

• ParaStation Management (psmgmt) process manager if you do not want to use the Hydra process manager that comes with MPICH
• C++ compiler for applications written in C++
• Fortran compiler for applications written in Fortran
• PMIx library if you want to use PMIx as interface to the process manager (use of PMIx and Hydra is mutual exclusive)
• hcoll library for hcoll support
• hwloc library for hwloc support in MPICH/hydra
• CUDA runtime library for CUDA awareness

Download the source code from GitHub:

$ git clone https://github.com/ParaStation/psmpi.git psmpi

Run Autotools:

$ cd /path/to/psmpi
$ ./autogen.sh

We encourage you not to build ParaStation MPI directly within the main source folder:

$ mkdir build && cd build

ParaStation MPI relies on Autotools as the build system requiring a configure step. The build system already provides so-called "confsets" that pass the necessary configuration arguments for a particular installation type to the underlying MPICH build system. It is strongly recommended to rely on these confsets for a proper installation! Currently, the following confsets are provided:

default    : Like 'gcc'
none       : Do not configure mpich. Prepare only for tar, rpm and srpm build

gcc        : Use Gnu compiler (gcc)
intel      : Use Intel compiler (icx)
icx        : Like 'intel'
pgi        : Portland group compiler (pgcc)
nvhpc      : Nvidia HPC compiler (nvc)
llvm       : llvm-based toolchains (e.g., AOCC)

devel      : With error checking and debug info (gcc)
debug      : Like 'devel' but without any optimizations

user       : No predefined options
ch3        : original mpich ch3 device (no parastation)
ch4        : original mpich ch4 device (no parastation)

The following example configures ParaStation MPI for using the gcc compiler:

$ ../configure --prefix=/path/to/installation/dir --with-confset=default

We recommend developers to use the devel confset for more error checking and debug infos at compile time.

For a successful build, you have to provide a path to your pscom installation via the --with-pcom[=PATH] configure option (defaults to /opt/parastation) or alternatively by setting the LIBRARY_PATH and CPATH environment variables:

$ export LIBRARY_PATH=/path/to/pscom/installation/lib[64]:${LIBRARY_PATH}
$ export CPATH=/path/to/pscom/installation/include:${CPATH}

Now, ParaStation MPI can be built and installed in accordance with the configuration arguments:

$ make -j8 && make install

To use ParaStation MPI for building and running MPI applications, it is advisable to adjust the environment properly. This can be done, e.g., by using the following bash script:

#!/bin/bash

if [ $# -eq 0 ]; then
    echo "ERROR: Please provide the path to ParaStation MPI. Abort!"
    exit -1
fi

PARASTATION_MPI=`realpath ${1}`

export PATH="${PARASTATION_MPI}/bin${PATH:+:}${PATH}"
export CPATH="${PARASTATION_MPI}/include${CPATH:+:}${CPATH}"
export LD_LIBRARY_PATH="${PARASTATION_MPI}/lib${LD_LIBRARY_PATH:+:}${LD_LIBRARY_PATH}"
export LIBRARY_PATH="${PARASTATION_MPI}/lib${LIBRARY_PATH:+:}${LIBRARY_PATH}"

Instead of relying on the pscom as a shared-library, ParaStation MPI can be optionally compiled as a single shared-object by directly using the pscom sources. For doing so, the pscom source files are required:

$ git clone https://github.com/ParaStation/pscom.git pscom

After downloading the psmpi sources, the same configuration parameters apply as discussed above (see Configure). Additionally, you will need to add the --with-pscom-allin flag, e.g.:

$ ../configure --prefix=/path/to/installation/dir --with-confset=default --with-pscom-allin=/path/to/pscom/sources

By default, the pscom4ucp as well as the pscom4psm plugins are included firmly into ParaStation MPI if --with-pscom-allin is set and the related low-level drivers are found. For specifying the plugins to be built-in explicitly, the --with-pscom-builtin[=list] option can be used.

MPICH has a test suite that can also be used and is even extended by ParaStation MPI.

$ make test

...will run the complete test suite comprising tests in the following subfolders in mpich2/test/mpi/:

parastation
cuda
attr
ckpoint
coll
comm
cxx
datatype
errhan
errors
f08
f77
f90
ft
group
impls
info
io
init
mpi_t
perf
pt2pt
rma
spawn
threads
topo
part
session

However, you can also have only tests of a certain subdirectory to be executed by specifying a TESTDIR, for example:

$ make test TESTDIR=pt2pt

In addition, a TESTSET can be specified to further restrict the number of tests. The tests that belong to a certain test set are stated as a list in a file with the same name within the above listed subdirectories. Currently, only ps-test-minimal (minimal list of tests) and testlist (containing all tests) are valid test sets. So, for example, the following invocation runs all tests belonging to ps-test-minimal within all subdirectories:

$ make test TESTSET=ps-test-minimal

And if, for example, the test set ps-test-minimal should only be executed for tests within the subfolder pt2pt, then the following invocation is the means of choice:

$ make test TESTSET=ps-test-minimal TESTDIR=pt2pt

In the case that several directories and/or several test sets should be used for a run, one can do this by using comma-separated lists and the variables TESTDIRS and TESTSETS. (Please note the additional "S" at the end of the variable names!) However, in this case no explicit check for validity of the passed test sets or folder names is performed. Example:

$ make test TESTDIRS=pt2pt,coll,info

Furthermore, there is a "meta" set of tests that are specifically intended for large numbers of processes and/or large messages and data sets. This additional (and somehow orthogonal) set can be chosen via largetest as the make target:

$ make largetest

...which can also be used in combination with the TESTSET/TESTSETS and the TESTDIR/TESTDIRS options.

When running the tests, some settings like the mpiexec command to be used are pre-configured and depend on the configuration chosen for psmpi during its top-level configure call. However, there is still the possibility to overwrite some of such configurations by explicitly setting the following environment variables:

• MPIEXEC: choose the path and the executable to be used for process start
• MPIEXECARG: additional arguments to be passed to the mpiexec command

For example:

$ make test MPIEXEC=/opt/parastation/bin/mpiexec MPIEXECARG="-e MPIEXEC_UNIVERSE_SIZE=128"

Furthermore, there is a small set of such overriding configurations that can also be chosen via the TESTCONF option. The following configurations are available:

• hydra: Force the use of MPICH's Hydra mpiexec
• psmgmt: Force the use of psmgmt's mpiexec in /opt/parastation/bin
• verbose: Switch on verbose mode for runtests script
• psmgmt/memcheck: Call psmgmt's mpiexec with --memcheck
• psmgmt/valgrind: Call psmgmt's mpiexec with --valgrind
• psmgmt/pmix: Call psmgmt's mpiexec with --pmix

If multiple of such configurations are to be selected at the same time, then the TESTCONFS (mind the additional S at the end!) can be used. Example:

$ make test TESTCONFS=hydra,verbose

To ensure that all commits conform to the coding style, the pre-commit hook should be activated. Therefore, you have to link this from the top-level source directory:

$ ln -s ../../scripts/hooks/pre-commit .git/hooks/pre-commit