Bug (what bug..?) reporting: Email: asarje@lbl.gov

1. LICENSING
2. SUPPORTED PLATFORMS 1. Operating System Platforms 2. System Hardware (Compute Environment) 3. System Hardware (Processor Architectures)
3. SOFTWARE PRE-REQUISITES
4. HipGISAXS DIRECTORY LAYOUT
5. TO BUILD THE APPLICATION AND LIBRARY
6. TO USE THE HIPGISAXS LIBRARY IN YOUR OWN APPLICATION
7. TO RUN THE APPLICATION 1. Interactively on Dirac (NERSC) 2. Submit batch job for multiple GPU nodes on Dirac (NERSC) 3. Interactively on a generic Linux machine equipped with GPU
8. INPUTS
9. APPENDIX 1. SPECIAL BUILD INSTRUCTIONS FOR SPECIFIC SYSTEMS
   1. On Carver/Dirac at NERSC
   2. On Hopper/Edison at NERSC
   3. On Titan at OLCF
   4. On Stampede at TACC
   5. On Mira at ALCF
   6. On a generic Linux system
   7. For certain users, on ALS GPU servers

The HipGISAXS software is only available to be downloaded and used by employees of academic research institutions, not-for-profit research laboratories, or governmental research facilities. Please read the accompanying LICENSE file before downloading the software. By downloading the software, you are agreeing to be bound by the terms of this Non-Commercial End User License Agreement. This licensing information is also provided as a PDF file along with the documentation.

HipGISAXS has been successfully tested on the following platforms.

1. GNU/Linux x86_64: Ubuntu, Red Hat Linux, SUSE Linux, Cray Linux Environment (XK7, XE6, XC30, XC40).
2. Darwin x86_64: Mac OS X (El Capitan, Yosemite).
3. You could try HipGISAXS on any UNIX based OS: generally it should work.
4. Windows: Will probably be supported in future.

1. Generic x86 laptop/desktop/server.
2. Clusters/Supercomputers based on x86 processors.
3. Generic x86 laptop/desktop/server equipped with Nvidia GPUs.
4. Clusters/Supercomputers equipped with Nvidia GPUs as accelerators on each node.

1. Intel/AMD processors (64-bit).
2. Nvidia GPUs with compute capability >= 2.0.
3. Intel MIC architecture: support under development.

This software uses several third-party libraries, and they need to be installed and available in order to compile and run HipGISAXS. For ease, if possible use the installations already available on your system, if any. Alternatively, download and install them yourself. The following are the dependencies of this software:

1. GNU C/C++ compilers, version >= 4.7, OR Intel C/C++ compilers, version >= 15 (with GNU compatibility >= 4.7).
   To compile with GPU support enabled, use GNU compilers.
2. Scons, version >= 2.0. Scons can be obtained from: http://scons.org NOTE: Scons is based on Python, so you need to have Python installed too.
3. Boost C++ Libraries. Boost can be obtained from: http://www.boost.org
4. Tiff Library (libtiff).
   Tiff library can be obtained from: http://www.libtiff.org

1. Nvidia CUDA toolkit version >= 5.5.
   CUDA can be obtained from: http://developer.nvidia.com/cuda/cuda-downloads
   NOTE: CUDA required to enable GPU support. NOTE: CUDA is NOT required if compiling for CPU-only version.
2. OpenMPI compiled with GNU or Intel compilers which ever you are using to compile HipGISAXS. Version > 1.4.4.
   OpenMPI can be obtained from: http://www.open-mpi.org/software Alternative implementations of MPI may also be used, such as MPICH and MVAPICH. NOTE: MPI library is required to enable support for multi-node systems. NOTE: MPI library is NOT required if compiling for single node/server/desktop/laptop.
3. Parallel HDF5 library.
   HDF5 can be obtained from: http://www.hdfgroup.org/downloads
   NOTE: HDF5 library is NOT required if you do not plan to use input files in this format. NOTE: HDF5 depends on the zlib and szip libraries.
   zlib can be obtained from: http://www.zlib.net
   szip can be obtained from: http://www.hdfgroup.org/doc\_resource/SZIP

HipGISAXS
|- README.md  : Duh!
|- LICENSE    : This contains all the licensing information for HipGISAXS.
|- SConstruct : Scons file for installation.
|- SConscript : Scons file for installation.
|- bin        : This contains HipGISAXS binaries generated by compilation.
|- build      : A few Makefiles for various systems are provided in this.
|- data       : This provides some sample input shape definition files.
|- doc        : This will contain detailed documentation of HipGISAXS.
|- extras     : This contains some extra stuff such as syntax highlighting config for Vim.
|- include    : All the source headers are within this directory. It contains subdirectories.
|- inputs     : This contains some sample input files in HiG format.
|- lib        : The HipGISAXS library, libhipgisaxs.a, is generated here.
|- man        : This has the man pages.
|- obj        : All the object files generated during build are stored here.
|- samples    : This contains some compilable samples on how to use the HipGISAXS library.
|- src        : This is the main source code directory. It contains many subdirectories.

To build the HipGISAXS application binary and static library, use scons. Make sure you are passing paths to all the dependencies through --extrapath= option of scons:

$ scons --extrapath=<path1>,<path2>,<etc>

To enable GPU support, use --with-cuda option:

$ scons --extrapath=<path1>,<path2>,<etc> --with-gpu

To enable MPI support, use --with-mpi option:

$ scons --extrapath=<path1>,<path2>,<etc> --with-gpu --with-mpi

or,

$ scons --extrapath=<path1>,<path2>,<etc> --with-mpi

The generated binary, hipgisaxs, will be located in the bin directory. The generated static library, libhipgisaxs.a, will be located in the lib directory.

... and you are done. Go enjoy HipGISAXS!

NOTE: See Appendix at the end of this file for more detailed and customized building information.

Please refer to the examples provided in the samples directory. It contains simple code which uses the HipGISAXS library. The corresponsing Makefile is also provided as a reference.

1. Make sure all the paths (data, output) and other variables are correctly set in the input HiG file.
2. Execute the binary with the input file as an argument. For example,

   $ ./bin/hipgisaxs inputs/01-cylinder.hig
   

The HipGISAXS binary takes as input a file in HiG format. Please refer to detailed HipGISAXS documentation for details of the HiG format. A few sample input files are located in the directory "inputs", with extensions ".hig". Update the input file as needed. The main components in the input file to update are the following:

1. Shape name defines the input filename containing triangulated shape data. It should point to the correct location of the file (relative path):

       ...
       shape = {
    	   ...
    	   name = "data/flexrod.obj"
       } ...
   

   This file needs to be either in HDF5 format or the OBJ format.
   Some sample shape files are provided in the directory "data", with extensions ".hd5".

2. Output locatation needs to be defined as pathprefix and runname in the computation object:

      ...
      computation = {
    	  ...
    	  pathprefix = ".",
    	  runname = "myflexrod",
    	  ...
      } ...
   

   The pathprefix is a relative path to a directory.
   The runname is appended with a timestamp, and a directory by this resulting name is created in the directory specified by pathprefix.
   All generated output files are stored in this generated directory.

3. Resolution alters the final image resolution, and also affects the run time:

      ...
      computation = {
    	  ...
    	  resolution = [ 1000 500 ],
    	  ...
      } ...
   

   Obviously, simulations with lower resolutions finish faster.
   Modify the provided input file templates as needed.

1. Some makefiles for different systems are included in the directory "build". Replace the current "Makefile" with the one for Carver/Dirac (and name it as "Makefile"): $ cp build/Makefile.dirac Makefile
2. Unload the default PGI modules: $ module unload pgi openmpi
3. Load the required modules: $ module load openmpi-gnu gcc/4.5.2 cuda $ module load szip zlib $ module load hdf5-parallel/1.8.3-gnu
4. The Boost module available on Carver does NOT contain the GIL numeric extension. Please install and use your own copy.
5. Edit Makefile to specify the correct paths ("base directories" section) to the various libraries. Example: $ cat Makefile ...

   base directories

   BOOST_DIR = /global/homes/a/asarje/local/boost_1_49_0 MPI_DIR = CUDA_DIR = /usr/common/usg/cuda/5.0 HDF5_DIR = /global/homes/a/asarje/local/hdf5-1.8.8-gnu/parallel TIFF_DIR = /global/homes/a/asarje/local/tiff-4.0.2 Z_DIR = $(ZLIB_DIR) # an environment variable set by loading zlib module SZ_DIR = $(SZIP_DIR) # an environment variable set by loading szip module ...
6. Build the code from within the code directory: $ make clean $ make [or, make library] This will generate the binary in directory "bin". When building the library, it will be generated into the "lib" directory. All the intermediate generated object files are in the directory "obj". They can be removed if wanted.

Instructions coming soon. Meanwhile go try yourself!

Instructions coming soon. Meanwhile go try yourself!

Instructions coming soon. Meanwhile go try yourself!

Instructions coming soon. Meanwhile go try yourself!

1. Make sure all above prerequisites are available.
2. Make sure all system environment variables are set accordingly to include the prerequisites.
3. Set all paths correctly in the sample Makefile, and edit as needed (as above).
4. Build the library from within the code directory: $ make clean $ make library

1. Replace the current "Makefile" with the one for saxs-waxs-gpu (and name it as "Makefile"): $ cp build/Makefile.saxs1 Makefile OR $ cp build/Makefile.saxs2 Makefile OR $ cp build/Makefile.andromeda Makefile
2. All "base directories" in the provided Makefile are set to the correct locations. If you want to use your own installation of any of the required softwares, please edit its corresponding entry in the Makefile. Example: $ cat Makefile ...

   base directories

   BOOST_DIR = /usr/local/boost_1_45_0 MPI_DIR = /usr/local CUDA_DIR = /usr/local/cuda HDF5_DIR = /home/asarje/local/hdf5-1.8.8-gnu/parallel Z_DIR = /root/zlib-1.2.7 SZ_DIR = /root/szip-2.1 TIFF_LIB_DIR = /usr/local ...
3. Build the code from within the code's main directory: $ make clean $ make [or, make library] This will generate the binary in directory "bin". When building the library, it will be generated into the "lib" directory. All the generated object files are in the directory "obj".