===================
MAGMA README FILE
===================

* Further documentation is provided in docs/html/index.html.

----------------------------------------------------------------------------
* Configuring MAGMA

    Modify the make.inc file to indicate your C/C++ compiler,
    Fortran compiler, and where CUDA, CPU BLAS, and LAPACK are installed on your
    system. Examples are given in make.inc.acml, make.inc.atlas, make.inc.macos,
    make.inc.mkl-*, make.inc.openblas showing how to link to ACML, ATLAS, MacOS
    veclib, MKL, and OpenBLAS, respectively.

----------------------------------------------------------------------------
* Library paths

    All the make.inc files assume $CUDADIR is set in your environment.
    For bash (sh), put in ~/.bashrc (with your system's path):
        export CUDADIR=/usr/loca/cuda
    For csh/tcsh, put in ~/.cshrc:
        setenv CUDADIR /usr/local/cuda
    MAGMA is tested with CUDA >= 5.5. Some functionality requires a newer version.

    The MKL make.inc files assume $MKLROOT is set in your environment.
    For bash (sh), put in ~/.bashrc (with your system's path):
        source /opt/intel/composerxe/mkl/bin/mklvars.sh intel64
    For csh/tcsh, put in ~/.cshrc:
        source /opt/intel/composerxe/mkl/bin/mklvars.csh intel64
    MAGMA is tested with MKL 11.2.3 (2015), both LP64 and ILP64;
    other versions may work.

    The ACML make.inc file assumes $ACMLDIR is set in your environment.
    For bash (sh), put in ~/.bashrc (with your system's path):
        export ACMLDIR=/opt/acml-5.3.1
    For csh/tcsh, put in ~/.cshrc:
        setenv ACMLDIR  /opt/acml-5.3.1
    MAGMA is tested with ACML 5.3.1; other versions may work.
    See comments in make.inc.acml regarding ACML 4;
    a couple testers fail to compile with ACML 4.

    The ATLAS make.inc file assumes $ATLASDIR and $LAPACKDIR are set in your environment.
    If not installed, install LAPACK from http://www.netlib.org/lapack/
    For bash (sh), put in ~/.bashrc (with your system's path):
        export ATLASDIR=/opt/atlas
        export LAPACKDIR=/opt/LAPACK
    For csh/tcsh, put in ~/.cshrc:
        setenv ATLASDIR  /opt/atlas
        setenv LAPACKDIR /opt/LAPACK

    The OpenBLAS make.inc file assumes $OPENBLASDIR is set in your environment.
    For bash (sh), put in ~/.bashrc (with your system's path):
        export OPENBLASDIR=/opt/openblas
    For csh/tcsh, put in ~/.cshrc:
        setenv OPENBLASDIR /opt/openblas

----------------------------------------------------------------------------
* Linking to BLAS

    Depending on the Fortran compiler used for your BLAS and LAPACK libraries,
    the linking convention is one of:
        * Add underscore, so gemm() in Fortran becomes gemm_() in C.
        * Uppercase,      so gemm() in Fortran becomes GEMM()  in C.
        * No change,      so gemm() in Fortran stays   gemm()  in C.
    Set -DADD_, -DUPCASE, or -DNOCHANGE, respectively, in all FLAGS in your
    make.inc file to select the appropriate one. Use nm to examine your BLAS
    library:

        acml-5.3.1/gfortran64_mp/lib> nm libacml_mp.a | grep -i 'T.*dgemm'
        0000000000000000 T dgemm
        00000000000004e0 T dgemm_

    Which shows that either -DADD_ (dgemm_) or -DNOCHANGE (dgemm) should work.
    The default in all make.inc files is -DADD_.

----------------------------------------------------------------------------
* Compile-time options

    Several compiler defines, below, affect how MAGMA is compiled and
    might have a large performance impact. These are set in make.inc files
    using the -D compiler flag, e.g., -DMAGMA_WITH_MKL in CFLAGS.

    MAGMA_WITH_MKL
        If linked with MKL, allows MAGMA to get MKL's version and
        set MKL's number of threads.

    MAGMA_WITH_ACML
        If linked with ACML 5 or later, allows MAGMA to get ACML's version.
        ACML's number of threads are set via OpenMP.

    MAGMA_NOAFFINITY
        Disables thread affinity, available in glibc 2.6 and later.

    BATCH_DISABLE_CHECKING
        For batched routines it disables the info_array that contains errors.
        In other term, for example, for Cholesky factorization if you are sure
        your matrix is SPD and want better performance you can compile with this flag.

    BATCH_DISABLE_CLEANUP
        For batched routines it disables the cleanup code.
        For example the {sy|he}rk called with "lower" will write data on
        the upper triangular portion of the matrix.

    BATCHED_DISABLE_PARCPU
        In the testing directory, it disable the parallel implementation of the
        batched computation on CPU. Can be used to compare a naive versus a parallelized
        CPU batched computation.

----------------------------------------------------------------------------
* Building without Fortran

    MAGMA can be built without Fortran by commenting out FORT in the make.inc file.
    However, some testers will not be able to check their results.

----------------------------------------------------------------------------
* Building Shared Libraries

    By default now, all make.inc files add the -fPIC option to CFLAGS, FFLAGS,
    F90FLAGS, and NVCCFLAGS, required for building a shared library. Note in
    NVCCFLAGS that -fPIC is passed via the -Xcompiler option. Running:
        make
    or
        make lib
        make test
        make sparse
    will create shared libraries:
        lib/libmagma.so
        lib/libmagma_sparse.so
    and static libraries:
        lib/libmagma.a
        lib/libmagma_sparse.a
    and testing drivers in 'testing' and 'sparse-iter/testing'.

    (The current exception is for ATLAS, in make.inc.atlas, which in our
    install is a static library, thus requiring MAGMA to be a static library.)

----------------------------------------------------------------------------
* Building Static Libraries

    Alternatively, comment out FPIC in the make.inc file to compile only a static
    library. Then, running:
        make
    or
        make lib
        make test
        make sparse
    will create static libraries:
        lib/libmagma.a
        lib/libmagma_sparse.a
    and testing drivers in 'testing' and 'sparse-iter/testing'.

----------------------------------------------------------------------------
* Installation

    To install libraries and include files in a given prefix, run:

        make install prefix=/usr/local/magma

    The default prefix is /usr/local/magma. You can also set prefix in make.inc.

----------------------------------------------------------------------------
* Environment variables

    These variables control MAGMA, BLAS, and LAPACK runtime behavior.

    $MAGMA_NUM_GPUS
        For multi-GPU functions, set $MAGMA_NUM_GPUS to the number of GPUs to use.

    $OMP_NUM_THREADS
    $MKL_NUM_THREADS
    $VECLIB_MAXIMUM_THREADS
        For multi-core BLAS libraries, set $OMP_NUM_THREADS or $MKL_NUM_THREADS or
        $VECLIB_MAXIMUM_THREADS to the number of CPU threads, depending on your
        BLAS library.

----------------------------------------------------------------------------
* Example

    A short standalone EXAMPLE is provided in directory 'example'. This is
    intended to show the minimum needed to start using MAGMA, without all the
    extra Makefiles, headers, and libraries used in testing. You must edit
    example/Makefile to reflect your make.inc, or use pkg-config, as described in
    example/README.txt.

----------------------------------------------------------------------------
* Testing

    To TEST MAGMA, go to directory 'testing'. Drivers testing different routines
    are provided. These drivers are also useful as examples on how to use MAGMA,
    as well as to benchmark the performance.

    The testers print "ok" or "failed" for whether the error passes the tolerance.
    In some cases, the tolerance may be too strict, so a test may "fail" even
    though it is only slightly above the tolerance. Error values around 1e-15 for
    double and double-complex, and 1e-7 for single and single-complex, are
    generally acceptable. Values larger than 1e-4 are very suspicious.

----------------------------------------------------------------------------
* Tuning

    You can modify the blocking factors for the algorithms of
    interest in file 'control/get_nb.cpp'. The default values are tuned for
    general Fermi (2.x) and Kepler (3.x) GPUs.

    Performance results are included in results/vA.B.C/cudaX.Y-zzz/*.txt
    for MAGMA version A.B.C, CUDA version X.Y, and GPU zzz.

----------------------------------------------------------------------------
* Experimental MAGMA

    Experimental dynamic scheduling code is in directory 'exp'. After making,
    type 'export MKL_NUM_THREADS=1'. Next go to directory 'testing'.
    Provided are drivers testing multi-core Cholesky, QR, and LU; and
    single GPU all, available CPU cores QR. Typing a given driver name will
    result in clear instructions on what to provide on the command line.

    Note that on some systems the experimental code must be linked against
    sequential MKL. If you notice excessively poor performance, change
    make.inc to link against sequential MKL.

----------------------------------------------------------------------------
* Forum

    For more INFORMATION, please refer to the MAGMA homepage and user forum:

        http://icl.cs.utk.edu/magma/
        http://icl.cs.utk.edu/magma/forum/

    The MAGMA project supports the package in the sense that reports of
    errors or poor performance will gain immediate attention from the
    developers. Such reports, descriptions of interesting applications,
    and other comments should be posted on the MAGMA user forum.