******************************************************************************
				  GPI-2
			  http://www.gpi-site.com
		  
		  	      Version: 1.0.2
    	          	    Copyright (C) 2013
			     Fraunhofer ITWM
			     
******************************************************************************

1. INTRODUCTION
===============

GPI-2 is the second generation of GPI (www.gpi-site.com). GPI-2
implements the GASPI specification (www.gaspi.de), an API
specification which originates from the ideas and concepts of GPI.

GPI-2 is an API for asynchronous communication. It provides a
flexible, scalable and fault tolerant interface for parallel
applications.


2. INSTALLATION
===============

The easiest way to install GPI-2 is by using the install.sh
script. The default settings install GPI-2 under /opt/GPI2/. This
location can be easily modified by passing the location with the -p
option to the install script.

For example,
    ./install.sh -p /prog/GPI2

installs GPI-2 under /prog/GPI2 instead of under the default location.

GPI-2 requires the libibverbs from the OFED stack. You can pass the
path of your OFED installation to the install script using the option
(-o) for that, in case the install script is not able to find it:

     ./install.sh -o <full_path_to_ofed>
     

Requirements:
-------------

The current version of GPI-2 has the following requirements.

Software:
- libibverbs (Verbs library from OFED).
- ssh server running on compute nodes (requiring no password).
- gawk (GNU Awk)

Hardware:
- Infiniband or RoCE devices.


3. BUILDING GPI-2
=================

You can build GPI2 on your own. There are the following make targets:

all	  - Build everything
gpi	  - Build the GPI-2 library (including debug version)
mic	  - Build the GPI-2 library for the MIC
tests	  - Build provided tests
docs	  - Generate documentation (requires doxygen)
clean	  - Clean-up

4. BUILDING GPI-2 APPLICATIONS
==============================

GPI-2 provides two libraries: libGPI2.a and libGPI2-dbg.a.

The libGPI2.a aims at high-performance and is to be used in production
whereas the libGPI2-dbg.a provides a debug version, with extra
parameter checking and debug messages and is to be used to debug and
during development.


5. RUNNING GPI-2 APPLICATIONS
=============================

The gaspi_run utility is used to start and run GPI-2
applications. A machine file with the hostnames of nodes where the
application will run, must be provided.

For example, to start 1 process per node (on 4 nodes), the machine
file looks like:

node01
node02
node03
node04

Similarly, to start 2 processes per node (on 4 nodes):

node01
node01
node02
node02
node03
node03
node04
node04

The gaspi_run utility is invoked as follows:

	gaspi_run -m <machinefile> [OPTIONS] <path GASPI program>

IMPORTANT: The path to the program must exist on all nodes where the
program should be started.
	
The gaspi_run utility has the following further options [OPTIONS]:

  -b <binary file> Use a different binary for first node (master).
     	     	   The master (first entry in the machine file) is
		   started with a different application than the rest
		   of the nodes (workers).
     	     	   
  -N               Enable NUMA for processes on same node. With this
  		   option it is only possible to start the same number
		   of processes as NUMA nodes present on the system.
		   The processes running on same node will be set with
		   affinity to the proper NUMA node.
  
  -n <procs>       Start as many <procs> from machine file.
     		   This option is used to start less processes than
		   those listed in the machine file.
		   
  -d               Run with GDB (debugger) on master node. With this
  		   option, GDB is started in the master node, to allow
		   debugging the application.
  
  -h               Show help.


5. THE GASPI_LOGGER
===================

The gaspi_logger utility is used to view the output from all nodes
except the master node (rank 0). The gaspi_logger is started, on
another session, on the master node. The output of the application,
when called with gaspi_printf, will be redirected to the
gaspi_logger. Other I/O routines (e.g. printf) will not.



6. GPI-2 WITH CO-PROCESSOR (INTEL XEON PHI)
==========================================

GPI-2 can be used with the Intel Xeon Phi (MIC) where the MIC is used
as one node (running possibly more than one GPI-2 process).

To use GPI-2 on the MIC, you need to compile it using the Intel
compiler with the -mmic option. The Makefile includes a build target
mic for that end but this build target is not used by the installation
script. After successful compilation, GPI-2 for the MIC can be found
under lib64/mic. If you are having problems with the compilation, make
sure the OFED_PATH in src/Makefile is setup correctly.

Assuming that the MIC(s) is set up properly, GPI-2 requires:

- ssh connectivity (requiring no password)

- grouping of local host and MIC(s) by using the tool micctrl

  micctrl --initdefaults
  micctrl --addbridge=br0 --type=internal --ip=172.31.1.254
  micctrl --network=static --bridge=br0 --ip=172.31.1.1

 where for instance, mic0: 172.31.1.1, mic1: 172.31.1.2 etc.

The MIC must be visible to the whole cluster network. For instance,
if your cluster network is 192.168.1.0 on eth0 and you want to map
mic0 to 192.168.1.100 do:

iptables -t nat -A PREROUTING -d 192.168.1.100 -i eth0 -j DNAT --to-destination 172.31.1.1
iptables -t nat -A POSTROUTING -s 172.31.1.1 -o eth0 -j SNAT --to-source 192.168.1.100

After setup, the MIC hostname can simply be added to the machinefile
and be used as another host.


7. TROUBLESHOOTING
==================

If you're having troubles when building GPI-2, make sure you have the
OFED stack correctly installed and running. You can change the OFED
path in the Makefile in the source directory (src) to the correct path
in your system.

8. UP COMING FEATURES
=====================

GPI-2 is on-going work and more features are still to come. Here are
some that are in our roadmap:

- fully asynchronous collectives
- include the support for GPU memory (available but not yet integrated)
- threads integration
- support to add spare nodes (fault tolerance)
- better debugging possibilities


9. MORE INFORMATION
===================

For more information, check the GPI-2 website: www.gpi-site.com