This code uses the DBIngestor library (https://github.com/adrpar/DBIngestor) to ingest Friends-Of-Friends (FOF) catalogues into a database.

It assumes that these catalogues come in the binary format used by Stefan Gottlöber. Thus it is very specific, but can serve as an example for how to use the DBIngestor library and write your own binary reader.

The code is based on the HelloWord program delivered with the DBIngestor.

For any questions, please contact me at Kristin Riebe, kriebe@aip.de

The FOF catalogues were created by Stefan Gottloeber, who applied his FOF algorithm to find groups of connected particles for a given linking length to cosmological simulations. These FOF groups resemble something similar to dark matter halos, but no bounding criterion is applied. For more information on the basic method, see Davis et al. 1985 (http://adsabs.harvard.edu/abs/1985ApJ...292..371D) and for a description of the applied version, see Knebe et al. 2011 (http://adsabs.harvard.edu/abs/2011MNRAS.415.2293K).

The files were written with Fortran, option "unformatted", but direct access so each record is wrapped by a leading and trailing (4 byte) integer.

Each binary file is usually accompanied by an ascii-header file (...info), which contains all the information about boxsize, redshift, etc. FofIngest won't read this file, but expects you to provide the necessary information as command line arguments. See FofReader.cpp for options (or use the help flag) to get a list of options.

A data file structured as follows:

• one row per FOF group, containing following columns (4 byte int or 4 byte float each):

and two additional Fortran-specific integers.

• Read given datafile and ingest the data as following columns into a database table: (see Fof_SchemaMapper.cpp)

phkey = Peano-Hilbert key for the grid cell in which the particle is located,
is just filled with 0's, values can be updated via the database server using e.g. libhilbert (https://github.com/adrpar/libhilbert)

• If swap=1 is given, values will be byteswapped
• Accepts parameters startRow and maxRows for jumping to the specified row and reading only a certain (contiguous) chunk of rows.

see INSTALL

An example data file and create-table statement are given in the Example directory. Data can be ingested with a command line like this:

FofIngest/build/FofIngest.x -s mysql -D TestDB -T FOF -U myusername -P mypassword -H 127.0.0.1 -O 3306 -M 50 -a 0.5 -l 2 -b 1000 -n 100 -g 1024 -i 5 -m 20 -I 1.e6 -d fof_example.DAT

Replace myusername and mypassword with your own credentials for your own database.

The possible options are:
-s: type of database (e.g. mysql, unix_sqlsrv_odbc)
-D: database name
-T: table name
-H: host
-O: port
-d: data file
-M: snapshot number (snapnum)
-a: expansion factor
-l: level
-b: boxsize
-n: min. number of particles in FOF group; NOTE: code assumes that FOF-groups are sorted by np in decreasing order, will stop reading if number of particles falls below this minimum
-g: grid cells for ix,iy,iz, per dimension
-i: start at this row
-m: read at most this number of rows
-I: idfactor for fofId calculation

NOTE: One can also use -R 1. This would try to resume the connection, if something fails. But then be careful and check later on if all rows were ingested and contain meaningful values. Erroneous rows must be deleted manually. Missing rows can be inserted using startRow and maxRows. That's quite fast, since the reader will just jump to the correct position. Finding out which rows are missing takes the most time here.

The database table could have been created like this (see also Example/createfoftable-mysql.sql):

mysql> use TestDB;

create table FOF (
    fofId bigint not null,      
    snapnum smallint not null,  
    `level` smallint not null,      
    NInFile int not null,        
    x float not null,
    y float not null,
    z float not null,
    vx float not null,
    vy float not null,
    vz float not null,
    np int not null,                
    mass float not null,
    `size` float not null,              
    disp float not null,
    disp_v float not null,
    delta float not null, 
    spin float not null,
    angMom_x float not null,
    angMom_y float not null, 
    angMom_z float not null,
    angMom float not null,
    axis1 float not null,
    axis2 float not null,
    axis3 float not null,
    axis1_x float not null,
    axis1_y float not null,
    axis1_z float not null,
    axis2_x float not null,
    axis2_y float not null,
    axis2_z float not null,
    axis3_x float not null,
    axis3_y float not null,
    axis3_z float not null,
    ix int not null,
    iy int not null,
    iz int not null,
    phkey int not null
)

An example data file is also given in the Example directory.