This text explains how to make agk/gs2. (2008/05/05 RN)

* Changelog
2010/06/07 [RN]: Add USE_LOCAL_RAN switch to choose what random number generator
		 to use.
2009/12/18 [RN]: Removed USE_NETCDF=new,old
2009/04/06 [RN]: SYSTEM environment variable is replaced by GK_SYSTEM
2009/01/26 [TT]: Removed agk_ from layouts_* files due to C_INDEX port to gs2
	   	 Removed tungsten from supported system as it has retired
2008/07/01 [RN]: Add new switches, USE_LOCAL_SPFUNC, USE_NAGLIB to choose
	   	 what special functions to use.

* Compile

You must set two environment variables to compile agk/gs2: GK_SYSTEM and MAKEFLAGS.
Since system-specific settings are written in Makefiles/Makefile.$(GK_SYSTEM),
you should specify what system you are on. Officially supported common systems
are franklin, hopper, carver (NERSC), ranger (TACC), jaguar, jaguarpf (ORNL),
and Kraken (NICS).

And, to load Makefile.$(GK_SYSTEM) in Makefiles/ directory, you must tell "make" 
command that you need to include some files in Makefiles/ directory. This can 
be done by command line option

 # make -IMakefiles

or, alternatively, use MAKEFLAGS environment variable to give "-I" option.

Those environment variables can be set in your .cshrc or .bashrc. Add the 
following,

(tcsh)
 # setenv MAKEFLAGS -IMakefiles
 # setenv GK_SYSTEM 'your system'

(bash)
 # export MAKEFLAGS=-IMakefiles
 # export GK_SYSTEM='your system'

For utils and geo directories in gs2, do as follows,

 # make -I.. -I../Makefiles

After setting those environment variable, you just type "make" to get an
executable. The default target is automatically chosen depending on 
where you are making. Other common targets are,
test_make, clean, distclean, tar, and depend.

We have prepared some compile-time switches. To invoke them, add SWITCH=value
to your make command. For example, if you want to invoke DEBUG flags,
 
 # make DEBUG=on

Following is the full list of switches:
 DEBUG (defined [undefined]): turns on debug mode
 TEST (defined [undefined]): turns on test mode (not working yet)
 PROF (gprof,ipm [undefined]): turns on profiling mode
			       can set gprof and ipm at the same time
			       ipm is working on limited system. if not
			       supported this option is simply ignored
 OPT (defined,aggressive [defined]): optimization
 STATIC (defined, [undefined]): prevents linking with shared libraries
 DBLE (defined [defined]): promotes precisions of real and complex
 USE_MPI (defined [defined]): turns on distributed memory parallelization
			      with MPI
 USE_SHMEM (defined [undefined]): turns on SHMEM parallel communications 
                                  on SGI (not working yet)
 USE_FFT (fftw [fftw]): FFT library to use
 USE_NETCDF (defined [defined]): uses NETCDF library or no
 USE_HDF5 (defined [undefined]): uses HDF5 library or no
 USE_MDSPLUS (defined [undefind]): uses MDSplus for geo (gs2 only)
 USE_C_INDEX (defined [undefined]): uses function pointer
 USE_POSIX (defined [undefined]): uses posix routines for argument processing
 USE_LOCAL_RAN (mt [undefined]): uses local random number generator
 USE_LOCAL_SPFUNC (defined [undefined]): use local special functions
	 (this overrides compiler's intrinsic special functions and
	 NAG library's)
 USE_NAGLIB (spfunc [undefined]): uses NAG library
 	    	    		  Currently, only used for special functions
				  (NOTE: NAG_PREC=dble (or sngl) must be 
				  chosen appropriately for your NAG library)

Values that each switches takes are shown in the bracket, where
"defined" ("undefined") means whether the switch is defined (or not),
but not having that value. (Technically speaking, ifdef statement is 
used in the makefile.) Thus, note that DEBUG=off means DEBUG is defined
because this flag is just checked if it is defined or not. Be careful!

In the square bracket, default values in the main makefile (Makefile)
are given. Some of those default values are overwritten depending on
hosts, compilers, and also environment variables. This implies that,
for some systems where "module" is provided, those default values are 
set properly depending on what modules are loaded.

* More details in CPPFLAGS (by Tomo Tatsuno 04/16/08)

Here, the description of full CPPFLAGS available are given. However,
CPPFLAGS are basically set by above make swithces and users need not
be aware of the exact CPPFLAGS.

In the values column, the top line is the value chosen for the undefined
macro (-D{variable} is NOT invoked).  The second line corresponds to the
DEFAULT value when the variable is invoked but the value is not specified
(-D{variable} only).  The rest of choices will become available only when
the value is specified by -D{variable}=_value_.  Note that no space should
be placed within each definition ('-D{value} = _value_' do not work as one
might expect).

variable	values		description

FCOMPILER	_XL_		Fortran compiler choices
		_GFORTRAN_      (replaces CL and FILE in the previous version)
		_G95_           (must be Fortran 95 Standard compliant)
		_NAG_
		_INTEL_
		_PGI_
		_PATHSCALE_
		_LAHEY_ 
		_ABSOFT_ (*1)
		_ALPHA_ (*1)
		_COMPAQ_ (*1)
		_CRAY_ (*1)
		_SUN_ (*1)

FFT		_NONE_		We can choose to not compile fftw.
		_FFTW_		At the moment only fftw is the available 
		_FFTW3_		fft option.

MPI		_NONE_		No mpi subroutines compiled.
				Use of -DMPI makes mpi work.

NETCDF		_NONE_		No netcdf subroutines compiled.
				Use of -DNETCDF makes netcdf work.

HDF		_NONE_		No hdf5 subroutines compiled.
				Use of -DHDF makes hdf work.

NO_SIZEOF	_NONE_		This is for memory accounting.
				In most compilers, non-standard sizeof function
				works fine as C function.
				For a compiler which does not support sizeof
				function, one can use hard-wired fortran
				implementation of size_of function instead
				by specifying -DNO_SIZEOF.

USE_C_INDEX	_NONE_		When invoked, function pointers in layouts_
				indices.c is used.  A little bit of performance
				upgrade.  For layouts_indices.c, there is
				another CFLAG '-DNO_UNDERSCORE' for the
				resolution of Fortran-C linking problem.
				The CFLAG '-DUNDERSCORE=2' can also be used
				when 2 underscores are needed.

RANDOM		_NONE_		Uses compiler's intrinsic random 
		_RANMT_		Uses Mersenne Twister 19937 random

SPFUNC 		_NONE_	 	Uses compiler's intrinsic special functions
		_LOCAL_      	Uses local special functions
		_NAG_		Uses NAG library's special functions
		 		(NOTE: for compiles not having intrinsic 
				special functions like XL fortran, _LOCAL_ or
				_NAG_ should be chosen. Current default for XL
				is _LOCAL_)

USE_L2E_MAP	_NONE_		Affects collision.fpp.  When invoked, l2e_map
(agk only)			becomes available, which describes direct map
				from lorentz to ediffuse layouts.
				Requires additional memory, but might enhance
				performance --- not much, though.

USE_LE_LAYOUT	_NONE_		Affects collision.fpp.  Under development.
(agk only)			Mutually exclusive with USE_L2E_MAP.

*1) Currently, not supported

The standard .F90 extension is not used since some file systems do not
distinguish the case difference of the file names, eg. Windows or Mac.
In order to make your emacs recognize .fpp extension as a f90 source code,
you may add the following line to your .emacs file:

  (setq auto-mode-alist (append '(("\\.fpp$" . f90-mode)) auto-mode-alist))

* Porting

Users of a new system needs to prepare Makefile.$(GK_SYSTEM) in Makefiles/
directory. GK_SYSTEM can be whatever you like, but we recommend to use hostname.
In this file, most simply, you just needs to set your compiler
and to include your compiler's setting as written in Makefile.simple.
If you want to get parallelized version, override FC by mpif90 etc after
loading Makefile.$(COMPILER). You may also want to set libraries places.
Define FFT_INC, FFT_LIB, NETCDF_INC, NETCDF_LIB etc in this file.
Write other your host's specific setting if necessary.

Makefile.local is another recommended file to write your local settings.
This file is always included if exists.

* Dependency

The dependency list is given in Makefile.depend, which is auto-generated
by the depend target using the perl script "fortdep". The script is
distributed for developers convenience. Users do not need to reconstruct
it.

* TODO

1. auto detection of the system for common system (like test_os script)
2. can test the Sun f90 on ranger