===================================================================
Introduction
===================================================================

mpileaks is a library to detect application leaks of MPI objects. Using
the MPI profiling interface, it keeps track of all MPI calls that
"allocate" and "free" objects.  Any unfreed objects are identified and
reported either during MPI_Finalize or whenever MPI_Pcontrol is called.

mpileaks displays the stack trace where the leak occurred. The trace
includes file name and line number of the allocation call. Since many
MPI tasks often have the same leak, mpileaks computes and reports
the total number of objects leaked across all processes for a given
source code location.  To acquire stack traces, mpileaks internally
uses Todd Gamblin's Callpath class.

https://github.com/scalability-llnl/callpath

To obtain stack traces, applications must be compiled with the
'-g' option. However, it is not necessary to link to mpileaks to
use it.  One may select to use mpileaks at run time with existing
executables (see using mpileaks below). 

The following objects are tracked:
   x ==> supported
   - ==> still todo

   x Requests
     x pt2pt
     x persistent pt2pt
     x nonblocking file IO
     x generalized requests
   x Datatypes
   x Groups
   x Communicators
     x dup / split / create
     x intercomm create / merge
     x topology
     x dynamic processes
   x User-defined reduction ops
   x Files
   x Windows
   x Info objects
   x Attribute keyvals
   x Errhandlers (need example)
   x Memory
   - Ports
   - Published names


===================================================================
Building and using mpileaks
===================================================================

A basic build and install:

  ./configure \
    --prefix=<installdir> \
    --with-adept-utils=<adeptutils_installdir> \
    --with-callpath=<callpath_installdir>
  make
  make install

Just before running, set:

  export LD_PRELOAD="<installdir>/lib/libmpileaks.so"

By default, mpileaks reports a single line from the stacktrace,
but one may increase the number of lines by setting the
$MPILEAKS_STACK_DEPTH environment variable.  For example, to
get two lines:

  export MPILEAKS_STACK_DEPTH=2

To get the entire stacktrace, set this value to -1.

Also, by default, mpileaks chops off the first 4 function calls
in the stack trace, which are internal to the mpileaks library.
One can adjust the number of functions removed from the
initial part of the trace by setting $MPILEAKS_STACK_START to
any non-negative integer (the default value is 4).  This feature
is useful for MPI implementations in which the Fortran API is
just a thin wrapper around the C API. For instance, if the
Fortran-to-C wrapper imposes one more level of function calls
between the application and the MPI library, one may increase
this number from 4 to 5 to shave off this single layer so that
the stack trace begins in application code.

As a convenience, mpileaks installs SLURM srun wrappers.
It creates an srun-mpileaks wrapper for C and C++ codes and
another srun-mpileaksf wrapper for Fortran applications.
These scripts run the application within an environment that
sets LD_PRELOAD, MPILEAKS_STACK_START, and MPILEAKS_STACK_DEPTH.
This way, the user simply must replace "srun" with the
corresponding script to enable mpileaks.

Different systems and different MPI libraries may require
different values for MPILEAKS_STACK_START.  There are configure
options to set different values in the srun wrappers.  For example,
to set MPILEAKS_STACK_START to 4 in srun-mpileaks and to 6 in
srun-mpileaksf, specify the following during configure:

  --with-stack-start-c=4
  --with-stack-start-fortran=6


===================================================================
Extending MPILEAKS to handle other types
===================================================================
   
mpileaks is really a framework to detect mismatched function calls. As
such it can be extended to detect other types of leaks. To use this
framework, one must: (a) register the handle type used to uniquely
identify the function calls of interest; and (b) use the provided
functions 'allocate' and 'free' to associate a function call as
allocating or freeing resources. We provide an example below that,
for any given program, identifies MPI open files that are not closed.  

--
#include "mpi.h"
#include "mpileaks.h"                 /* Handle2Callpath */ 

/*
 * Instantiate class with the specific handle-type that identifies
 * one function call from another (MPI_File in this case). 
 * Define 'is_handle_null' for your specific handle type. 
 */
 
static class MPI_File2Callpath : public Handle2Callpath<MPI_File>
{
public: 
  bool is_handle_null(MPI_File handle) {
    return (handle == MPI_FILE_NULL) ? 1 : 0; 
  }
} File2Callpath; 


/* 
 * Associate function calls as "allocate" or "free" functions. 
 */ 

int MPI_File_open(MPI_Comm comm, char *filename, int amode, 
                  MPI_Info info, MPI_File *fh)
{
  int rc = PMPI_File_open(comm, filename, amode, info, fh); 
  File2Callpath.allocate(*fh); 
  return rc; 
}


int MPI_File_close(MPI_File *fh) 
{
  MPI_File handle_copy = *fh; 
  
  int rc = PMPI_File_close(fh);  
  File2Callpath.free(handle_copy); 
  
  return rc; 
}