==============================================================================
Downloading roottest:

You can download read-only it via:

   git clone http://root.cern.ch/git/roottest

==============================================================================
Running roottest:

To run it just do:

   cd roottest
   gmake

The output will be something like:

   Running test in ./cint/array
   Tests in ./cint/array .............................................. OK
   Running test in ./cint/const
   Tests in ./cint/const .............................................. OK
   Running test in ./cint/controls
   Warning Loopbreak has some known skipped failures (in ./controls)
   Tests in ./cint/controls ........................................... OK
   ....
   Tests in . ......................................................... OK

The message "Warning xyz has some known skipped failures" is reminder of
a known failure

An unexpected failure will look like:

   Running test in ./cint/template
   2a3
   > Warning in <TClassTable::Add>: class TParameter<Long64_t> already in TClassTable
   gmake[2]: *** [longlong] Error 1
   Test in ./cint/template ........................................... FAIL

This says that the test named 'longlong' in the directory roottest/cint/template
has failed because the script issues the unexpected message
   > Warning in <TClassTable::Add>: class TParameter<Long64_t> already in TClassTable

For a given test the expected screen output is in testname.ref (in the above
example the file name is roottest/cint/template/longlong.ref).  The actual
output is located in roottest/cint/template/longlong.log.  See note on
references files for goadditional details.

There are also .log files containing the output of the compilation of (when
applicable) of auxiliary files.

To see what the are the command actually issued do:

   cd roottest; gmake V=t (or gmake VERBOSE=t) (or export V=t; gmake )

==============================================================================
Simple test addition:

In any existing directory create a file starting with the letter 'assert'
and with the extension .C or .cxx and fill out with the actual test, the
script needs to return 0 in case of success and any other value otherwise.

The files assert*.C will be interpreted while the files assert*.cxx will
be compiled via ACLiC.   The assert*.C can be either a named or unamed macro.
The assert*.cxx must be a named macro.

The Makefile will test both for the return value and for the output of
standard error.  The output of standard error will be compare to a
reference file; if you do not provide a reference file, the output will
be compared to an empty file.

To check both the output streams, instead of using the assert file names,
use the names:
    execMytest.C or execMytest.cxx or execMyTest.py

After running the test (successfully) once you will need to copy the log file over to
the ref file as the standard output is never empty.

==============================================================================
Adding new test

To add a new test
   cd roottest; addtest pathname Testname
For example:
   cd roottest; addtest root/io/newstl vectorOfvector 
add a new test name vectorOfvector in the directory roottest/root/io/newstl.
In particular this will update (or create) the makefile and generate 2 files:
runTestname.C and runTestname.ref.
To actually implement your test modify the file runTestname.C

If you want the runTestname.C to being compiled instead of interpreted, modify
in the makefile the line:
   Testname: testname.log
into
   Testname: testname.clog

Then the makefile will run
   root.exe -q -b -l runTestname.C > runTestname.log 2>&1
or
   root.exe -q -b -l runTestname.C+ > runTestname.clog 2>&1
And
   diff -bw runTestname.ref runTestname.log

Failure of test can be indicated by a change in the output or by an error return code.

Library can be build automatically using ACLiC. For example, if there is a Testname.C that needs to be compiled just add
   Testname: Testname_C.$(DllSuf)
To the local Makefile

There are other helpful macros in roottest/scripts/Rules.mk

==============================================================================
The organization.
   scripts : some helper scripts for the makefile
   root : ROOT specific test
   cint : CINT test

==============================================================================
Writing Makefile(s)

The default test rule, for example for the test 'sometest', depends on
a log file named using the testname plus the suffix .log, (eg 'sometest.log')
The file 'sometest.log' can be made automatically from from 'runsometest.C'
This target will attempts to create the log file (which should be the output of 
running the test), and compares its output with a prebuilt reference file.

The default rule using the make file variable $(TestDiff), to a diff and ignore 
all white spaces use $(TestDiffW):

sometest: sometest.log sometest.ref
	$(TestDiff)

To compile the script runsometest.C before executing replace
	sometest: sometest.log
with
	sometest: sometest.clog
   
To only compare the output of standard error to the reference file replace
	sometest: sometest.log
with
	sometest: sometest.elog

or replace

	sometest: sometest.clog
with
	sometest: sometest.eclog

   
==============================================================================
Disable a specific test:

Create a file named !DISABLE containing the name of the test to skip.
For example:

echo operatorLessThanTemplate > cint/parsing/\!DISABLE
make
...
Running test in ./cint/parsing
c:/root/roottest//scripts/Rules.mk:25: Test(s) operatorLessThanTemplate
disabled!

and it skips these tests (if not runnnig make clean).

==============================================================================
Prevent a subdirectory from being recursed into by the Makefile:

Create a file named SpecialDirs containing the name of the subdirectory to skip.
For example:

echo cmake-build > cint/reflex_standalone/SpecialDirs

and it skips this subdirectory

==============================================================================
Running the test

To clean the current directory and any sub-directory
	make clean
	
To run the tests in the current directory and any sub-directory
	make
	
To run the tests in current directory and any sub-directory in a fresh
enviroment.  In this case the output file will be deleted before running
the test
	make cleantest
	
==============================================================================
Reference files:
	
For a given test the expected screen output is stored in testname.ref;  For
the test longlong located in roottest/cint/template the file name is 
roottest/cint/template/longlong.ref).  The actual output is located in 
roottest/cint/template/longlong.log. 

In case where the output is different when run on a 32bit platform and
when run on 64bit platform, replace the file testname.ref with 2 files
with the respective extension ref32 and ref64.  For example the test
TExMap in roottest/root/collection 2 reference files: 
   TExMap.ref64 and TExMap.ref32.

==============================================================================
Summary files:

Summary will contains a version of the error output that is guaranteed
to be linear (but the order is not guaranteed) even when using
parallel make.  When using the summary target, a list of the failing
test is printed at the end of a (failing) make invocation.

The creation of summary files is enabled by default.  The default
prefix is 'make'.  To change the prefix use: 
   make SUMMARY=prefix
To disable the summary file generation use:
   make SUMMARY=

to make.  The result will be set of file in each directory summarizing
the failure in the directory and any subdirectory.  The summary files
name will follow the patterns:

   prefix.subdirectory.summary
or for individual tests:

   prefix.testname.summary

or (if the test name can not be determined)

   prefix.number.summary

For each individual test failure there will be a correponding summary
file that will contains how to reproduce the failure and what the
failure output was:

--- FAILING TEST: make -C ./cint/array/test execFail.C

Processing execFail.C...
This test in failing
'root.exe -b -l -q execFail.C' exited with error code: 1

The subdirectory.summary files will contains the aggregation of all
the summary files for that subdirectory and all its subdirectories.

NOTE: to collect all the summary files output for a complete
run of roottest, you will need to do something like:

  if [ `ls make.*.summary 2>/dev/null | wc -l` -gt 0 ] ; then \
     cat $(SUMMARY).*.summary > $(SUMMARY).summary; \
  fi

in the roottest top level directory.

At this point of time roughly 20% of the tests in roottest will not
produce a summary file upon failure.  If in a subdirectory none of the
failing test produce a summary file, the subdirectory summary file
will contain:

  --- FAILING TEST: make -C subdirectory_path test
  There is at least one failing test which does not create a summary file in the directory subdirectory.

==============================================================================
Performance testing:

Additional tests checking the performance of ROOT can be run by setting the
variable ROOTTEST_DEDICATED_NODE

When using the target 'perftrack':

   make perftrack

The duration of each time is calculated and accumulated into ROOT files (pt_*.root,
where the star is replace by a string encoding of the test name), gif files 
showing the evolution of the performance of each test over time and html index files
showing the graphs whether there is any anomalous performance.