========================================================================
                       Chombo
           Applied Numerical Algorithms Group
    National Energy Research Scientific Computing Center
          Lawrence Berkeley National Laboratory
                 chombo@anag.lbl.gov
              http://seesar.lbl.gov/anag/chombo
========================================================================
Disclaimer/Copyright
========================================================================

See "Copyright.txt" in this directory.

========================================================================
  Welcome to the Chombo 3.2 development package.
========================================================================

This README file contains an extremely terse description
of the Chombo software distribution.  We describe

(1)  What Chombo is.
(2)  User feedback mechanisms.
(3)  Where to find further documentation.
(4)  Software requirements for Chombo installation.
(5)  Installation and porting procedures.
(6)  Examples.

More complete documentation is contained within the distribution,
either in the doc subdirectory of this directory, or in the doxygen
documentation which may be generated by running "make doxygen" in the
Chombo/lib directory, or online, at http://Chombo.lbl.gov.

========================================================================
What is Chombo?
========================================================================

The Chombo package provides a set of tools for implementing finite
difference and finite-volume methods for the solution of partial
differential equations on block-structured adaptively refined
logically rectangular (i.e. Cartesian) grids. Both elliptic and
time-dependent modules are included.  Support for parallel platforms
and standardized self-describing file formats are included.

Chombo's design is uniquely flexible and accessible.  Any collaborators
will be able to develop parallel applications to solve the partial
differential equations in which they are interested with far shorter
development times than would be possible without the infrastructure.
Very careful design and documentation allows entry to the software at
many levels, enabling the use of Chombo to investigate deep
technical issues of adaptive mesh refinement algorithms or to simply
adapt the example applications to solve scientific problems of interest.

========================================================================
Feedback and questions
========================================================================

User feedback should be directed to <chombo@anag.lbl.gov>.
We are happy to field all questions and comments.

Also, there is a Chombo users e-mail list at
<chombousers@anag.lbl.gov>, which has been an excellent resource for
sharing ideas and questions with other Chombo users, many of whom are
quite expert users of Chombo. (The Chombo developers do follow the
chombousers mailing list as well.)

========================================================================
Documentation
========================================================================

The Chombo Design documents are included in the Chombo source
distribution in the Chombo/doc/ directory, and may also be found on
the Chombo website (http://Chombo.lbl.gov). These documents include
descriptions of both algorithms and the APIs of the Chombo
package. There are also detailed installation and requirements
descriptions.  

If the doxygen utility is installed on your system, the Chombo html
reference manual may be generated by running "make doxygen" in the
Chombo/lib directory and will then installed at:  

     Chombo/lib/doc/doxygen/html/index.html

The HTML files are produced directly from the source code using
the doxygen software, and document every class and function within the
public interface of the Chombo distribution. A link to the
doxygen-generated documentation may also be found online at the Chombo
website. 

========================================================================
Tested platforms and compilers.
========================================================================

GNU C++ (g++) 3.4 or higher -- stable releases only.

Intel C++ (icc/icpc) for Linux, Versions 10,11.

IBM C++ (xlC) version 5 and 6.  Recently tested on IBM SP2 with AIX 5.2.

SGI/MIPSpro C++ (CC) version 7.  Not recently tested in
serial on a IRIX64 platform.

Chombo has built at various times in the past (but not recently)
on compilers from: Compaq, KCC, Sun, Fujitsu.

Chombo has been compatible with all Fortran 77 compilers that
we have tried.  These include g77, gfortran, Intel ifc/ifort, IBM xlf*,
PGI pgf77/pgf90, Sun f77, DEC/Compaq f77/90/95, Cray T3E f90, Fujitsu f90.

Chombo works with all the MPI implementations we have tried.

We strive to make the Chombo code compliant with the ISO/IEC 14882 C++
Standard (1998).  Some C++ compilers are still not fully compliant
and may not be able to build Chombo.  In particular, building templates
is known to be a problem in some compilers (e.g. pgCC).

We would like to hear about successful builds with other compiler/platforms.
Please send email to <chombo@anag.lbl.gov>.

========================================================================
Installation
========================================================================

Installation is described in more detail in  doc/chomboDesign.pdf.
Chombo uses GNU make as its build engine.  Chombo can be built with
several options which are specified by setting make variables on the
command line if the defaults are not desirable.  The resulting library
names reflect the option values, so libraries with different sets of
options may be built and used without conflicting with each other.

The main options are: choice of C++ and Fortran compilers, number of
dimensions (1,2,3,4,5, or 6 ), floating point precision (float or
double), with optimization or debugging, serial or parallel, with or
without performance profiling, with HDF5 files or not, with memory
tracking or not, with 32 or 64bit pointers on systems that support
both. 

Many packages address these options with a configure procedure. The
problem is that the options then tend to be mutually exclusive for the
installation.  Our approach requires some additional vigilance from the
installer (since configure is not going to auto-magically figure
everything out for you), but has the advantage of allowing multiple
versions (e.g. 2d and 3d, serial and MPI, etc) to be used
simultaneously. 

----------------------------------------------------------------
Prerequisites:

   1./
If you want to use the Chombo facilities for writing data files,
you will need the HDF5 package installed on your machine.  See:

   http://www.hdfgroup.org/HDF5

Chombo works with versions 1.4.1 through 1.8.x.  To run with HDF5 1.8 you
need to compile with the 1.6 compatibility flags.


We would suggest that HDF5 be installed with the configure options
"--enable-production".

   2./
If you are going to run Chombo in parallel, you will need a
functioning MPI-1.2 compliant C-binding and a parallel version of HDF5
built with that MPI.

Use the "--enable-parallel" configure option for HDF5.  Install the
parallel libraries into a separate directory from the serial libs.

Chombo should build with any standard-compliant implementation of MPI.

The Argonne Lab version of MPI, called MPICH, is freely available from:

    http://www-unix.mcs.anl.gov/mpi/mpich/

Chombo has been successfully built with MPICH and the IBM, SGI
and Cray implementations of MPI.

   3./
You will need the GNU version of make to build Chombo.
Any non-ancient version should work.

You can check to see if you have GNU make by running the command:

      make --version

If this doesn't say it is GNU make, try

      gmake --version

If this works, you should _always_ use 'gmake' on
Chombo code.  Nothing else needs to be changed.

If this doesn't say it is GNU make, you need help from a local expert
to find GNU make on your system or install it.  It can be downloaded
from:
    http://directory.fsf.org/wiki/Make


   4./
You will need perl to build Chombo.  Any version 5.x should work.
Available from:

    http://www.perl.org/get.html

----------------------------------------------------------------
To build Chombo:

   0./ Download and untar the source.  This will give you a directory 
named "Chombo".

   1./ Customizing for the local computer

Customizing the Chombo make system is done by setting make variables.  You
should not have to change any of the Chombo makefiles except one:
"Chombo/lib/mk/Make.defs.local".  The command 'make setup' will create this
file from a template that contains instructions for setting the make
variables that Chombo uses.  These variables specify what configuration to
build by default, what compiler and compiler options to use, where to find
the HDF library and include files, etc.

The Chombo makefiles already know how to set the compiler options for a
variety of compilers, so you may not need to customize these.  The default
values for the configuration variables are in the file:
"Chombo/lib/mk/Make.def.defaults", and you may wish to reset these to
different values in "Make.defs.local".  It is likely that you will have to
set the HDF variables, since they vary from system to system.

You can see some examples of setting the make variables in the file

    Chombo/lib/mk/local/Make.defs.ANAG, 

which is what we use here on our local network.  This subdirectory also
contains several other files for specific supercomputers that we use
(eg. The NERSC computers "hopper" and "carver", the OLCF computer
"jaguar", etc.).  You can use these files by copying or symlinking
them to "Chombo/lib/mk/Make.defs.local". 

If you must customize the makefile variables differently for multiple
computers that share the same copy of the Chombo files, you can do this
by creating a file in "Chombo/lib/mk/local" for each different computer.
The filename must be "Make.defs.<name>" where "<name>" is the same as
the output from the "uname -n" command (or the "hostname" command if uname
does not work).  The Chombo make system will load this file only when it
is running on the computer with that name.  

Every time you modify "Make.defs.local", the main makefile in the "Chombo/lib"
directory will check the makefile variables to see that they have valid values.
A frequent mistake is to use invalid values for the HDF* variables, especially
the HDFMPI* variables.  If you do not have MPI installed, you should not set
the MPICXX and HDFMPI* variables.

Most of the major Chombo make variables can be set on the 'make' command
line, so once you learn which variables you want to change, it is easy
to build multiple configurations of the Chombo code without modifying the
makefiles.

More documentation for using the Chombo make system can be found in:
  1) the file Chombo/lib/mk/README
  2) the "Requirements" and "Installation" sections of
     "Chombo Software Package for AMR Applications Design Document"

   2./  Compiling the Chombo code into libraries and executables

Enter the Chombo/lib directory.  Type: "make all"

This will check the variables that are set in "Make.defs.local" and
complain if they are invalid.  This check will be run every time
"Make.defs.local" is changed.  If the variables are ok, it will build the
default version of the Chombo libraries and test programs.

The top level GNUMakefile has a description of the other targets.

When make completes, it will have built a set of archive libraries
in Chombo/lib/.  The library names are different for each version of
Chombo that you can build.  The pattern is:

  lib<libname><N>d.<OS>.<c++compiler>.<f77compiler><.DEBUG><.PROF><.MPI><.64>

where:
  <libname> corresponds to the subdirectories of Chombo/lib/src
  <N> is the value of the DIM variable, (the number of dimensions) (1,2,3,4,5,6)
  <OS> is the operating system name (computed automatically)
  <c++compiler> is the name of the C++ compiler, taken from the CXX variable.
  <f77compiler> is the name of the Fortran compiler, taken from the FC variable.
  <.DEBUG> is present if the DEBUG variable is TRUE and absent otherwise
  <.PROF> is present if the PROFILE variable is TRUE and absent otherwise
  <.MPI> is present if the MPI variable is TRUE and absent otherwise
  <.64> is present if the USE_64 variable is present and absent otherwise

Note: the compiler variables (CXX and FC) may include a path and
options, but only the command name is stored in the library filename.

The Chombo makefiles support parallel build using the GNU make '-j' option.
The libraries and test and example programs can be built in parallel but
must be built separately using the commands:
   make -j 6 all
   cd ../example ; make -j 6 example

The number of processes (6 above) should be set to whatever makes the
build go the quickest.  Using too many processes will slow things down.
The optimal value is usually around 2-3 times the number of physical 
CPUs in the computer.

There are various kinds of errors that might occur in making Chombo.
Read the documentation for details of how to diagnose and correct
these errors. 


   3./  Running the Chombo test programs

Type: "make run"

This will try to run all the Chombo tests.  Several results are possible.
1) none of the programs run
2) all of the programs run and complete successfully with no errors
3) one or more programs do not run or abort or run with errors

If 1) happens, it might be because there is a dynamic library
directory missing from the LD_LIBRARY_PATH environment variable.  This
is usually indicated by an error message that says something about being
unable to load (or find) a file (or library) named lib<something>.so
(possibly with numbers appended).

The HDF5 library is a likely culprit.  The path set in the HDFLIBFLAGS
and HDFMPILIBFLAGS make variables probably should be in the
LD_LIBRARY_PATH environment variable too.

The canonical output from a Chombo test program looks like:

   Running broadcastTest for configuration 2d.Linux.g++.g77 ...
      broadcastTest: passed all tests
   ... broadcastTest finished with status 0

If the status value is not 0, the test failed.  A failed test may print
other messages.  If many tests fail, there's probably something wrong
with the compiler options that were used.  Try changing the CXXFLAGS
and FFLAGS variables in Chombo/lib/mk/Make.defs.local.

If a test fails with a segmentation violation or a stack overflow, it
may mean you need to increase the stack size available to your
processes.  How to do this varies depending on the operating system
and shell, but the command is usually something like 'limit stacksize
25600' or 'ulimit -s 25600'. 

To recompile Chombo after changing compiler options, you must first type
"make realclean" then "make all".  Alternatively, if you don't want to
destroy the existing copy of the libraries, you can use the XTRACONFIG
make variable to produce a set of Chombo libraries with different filenames.
For example:

      make all XTRACONFIG=.O3 CXXFLAGS=-O3 FFLAGS=-O3

will compile Chombo with -O3 and name all the library files with
the pattern: lib<name><config>.O3.a.

If you continue to get errors while building or testing, send email to
<chombo@anag.lbl.gov> with the output from running the commands 'make vars'
and 'make all'.  If the problem happens while running the test programs,
also include the output from 'make run'.  Also include a copy of
Chombo/lib/mk/Make.defs.local.  It is also helpful to list the versions of
the compilers and operating system used.  (e.g., "g++ -dumpversion" and
"g77 -v" and "uname -a").




========================================================================
Info for Porting Chombo
========================================================================

If you are endeavoring to make our code run in a new environment
you should know the following from Chombo:

1. Chombo neither throws nor catches any exceptions.  Exception handling
can safely be turned off. Chombo does makes use of STL, and most vendors
have implemented their STL with exceptions, hence most compilers will
complain about their own STL if you turn exceptions off.

2. Chombo is not currently MT-safe.  There are only a very small number
of routines that are known MT-unsafe, but the code has not been scrubbed.
This work will depend on demand.

3. HDF5 currently does not supply an MT-safe version, although the
next major release does make this promise.