This is a C reference implementation of the Journal, Audit, and Logging Protocol (JALoP) that uses HTTP as the transport layer. This is also known as "JALoP over HTTP" or simply, "JALoP v2.x"

It is important to note that -
  Branch 1.x.x.x implements JALoP over BEEP i.e. JALoP v1.x.
  Branch 2.x.x.x implements JALoP over HTTP i.e. JALoP v2.x.

The implementation is divided into a number of components:
  The JALoP Network Library (JNL)
  The JALoP Producer Library (JPL)
  JALoP Local Store
  JALoP Network Stores

This project makes use of the following libraries:
  lcov - http://ltp.sourceforge.net/coverage/lcov.php
  libaxl - http://www.aspl.es/xml
  libconfig - http://freshmeat.net/projects/libconfigduo
  libcurl - https://curl.se/libcurl/
  libuuid - http://e2fsprogs.sourceforge.net/
  libxml2 - http://xmlsoft.org/
  libxmlsec1 - http://www.aleksey.com/xmlsec/
  openssl - http://www.openssl.org/
  site_scons - http://scons.org/
  test-dept - http://code.google.com/p/test-dept/
  Berkeley DB - http://www.oracle.com/us/products/database/berkeley-db/overview/index.html
  libmicrohttpd - http://www.gnu.org/software/libmicrohttpd/

The following libraries are included in the 3rd-party directory:
  axl
  lcov
  libconfig
  site_scons

Building and Installing

This project uses the scons (www.scons.org) build tool.
Below are the platform-specific versions of python and scons that were used to test JALoP v2.x.

  RHEL 7.x: python3.6 (python 3.6.8), scons-3.6
  RHEL 8.x: python3.6 (python 3.6.8), scons-3.6

Later versions of Python should work as well, but were not tested.

Note, the following commands assume your system installation invokes scons-3.6
On RHEL7, it may be necessary to invoke "scons-3.6"

Building requires a C++ compiler with full C++17 capabilities.
For EL7, use of a devtoolset can provide a capable compiler:
 - Add the repository
  - For CentOS 7:
    - # yum install centos-release-scl
  - For RHEL 7:
    - # subscription-manager repos --enable rhel-7-server-devtools-rpms
 - Install GCC 7 toolset
  - # yum install devtoolset-7
 - Launch a new shell using Devtoolset 7
  - scl enable devtoolset-7 $SHELL

The version of libmicrohttpd available in the CentOS/RHEL7 packages is not sufficient for this project
It is necessary to acquire v0.9.59 (or newer) from http://www.gnu.org/software/libmicrohttpd/
or one of the mirrors - or clone the repo from: https://git.gnunet.org/libmicrohttpd.git
v0.9.59 is recommended because it is the version available in the CentOS/RHEL8 packages and has
been the most thoroughly tested

Once the source has been acquired, cd to that directory
There are a few additional dependencies to build libmicrohttpd "makeinfo", but these dependencies
are already available in the rhel/centos packages
$ sudo yum install texinfo gnutls gnutls-devel

At this point, you're ready to build the expected version of libmicrohttpd
$ git checkout v0.9.59
$ ./boostrap
$ ./configure --enable-https --with-gnutls
$ make
$ make install (This will likely require sudo, depending on the prefix given to ./configure)

By default, make install places the installed library in /usr/local/lib and the header in
/usr/local/include, which SCons will not understand to look for by default.
The JALoP SConstruct uses pkg-config to find dependencies. Additional
locations can be added to pkg-config's search path. Use the command:

$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig/

to inform pkg-config of this additional search location. This command will only affect
the current shell and must be re-run if a new shell is used for building. If a different
--prefix=PREFIX option is passed to the ./configure for libmicrohttpd, the path given to
PKG_CONFIG_PATH must be modified accordingly. If necessary, the installation locations
can be configured even more precisely. Use ./configure --help to see the available options.
e.g. --libdir, --includedir

At run-time, it's necessary to link against this newly built .so with the jal_sub_cpp and jal_subscribe
CLIs - or any program that creates a JalSubscriber object. There are a few ways to do this.
1. Add a file under /etc/ld.so.conf.d/ containing the directory in which your built library exists
then run sudo ldconfig to update the cache
2. Add the directory containing your built library to environment variable LD_LIBRARY_PATH in the shell
where jal_sub_cpp, jal_subscribe, or other v2 subscriber will be run
3. Modify the call to ./configure to use a --prefix=PREFIX that will place the library somewhere
it will be found by the usual means.
e.g. ./configure --prefix=/usr/ - oddly on CentOS7 this places the lib in /lib, not /usr/lib, but
it does appear to otherwise work as you would expect and links correctly without other modifications.

To build, simply run 'scons' within the top level directory:
$ scons

SCons generates debug and release directories.

To install the library, simply run the appropriate installation script for your
platform (e.g. rhel_x86_64):

$ install_<platform>.sh


Other Options (these steps do not need to be executed in order to use the JALoP libraries)
To build the library and execute the unit tests, run

$ scons --no-release tests

To generate code coverage reports, run

$ scons cov

code coverage reports will appear in the 'cov' directory in the root of the source tree.

To generate doxygen documentation, run
$ scons doc
The generated documents will appear in the directory doc/doxygen.out

To clean the build output you can simply run
$ scons -c

Scons accepts a '-j' argument for parallel builds, so

$ scons -j6 
will run 6 worker threads in parallel.


The build environment honors a number of variables, these are:
CFLAGS - a set of flags to use when compiling C source files. To enable flags
for both C and C++, use the CCFLAGS environment variable.
CCFLAGS - a set of flags to use when compiling C and C++ source files.
CXXFLAGS - a set of flags to use when compiling C++ source fils.
LDFLAGS - a set of flags to pass to the linker
CC The C compiler to use
CXX The C++ compiler to use
LD The linker to use
CPP The C preprocessor to use


All of the source code for the implementation is under the 'src' directory.
The src directory is divided into different sections based on functionality,
subdirs such as network_lib, local_store, etc.

A number of patches for external project are included. These patches modify
existing projects (e.g. log4cxx) to utilize JALoP as an output method. These
may be found under the 'patches' directory.

This project is built and tested in the following environments:
  RHEL 6 on x86_64, using gcc version 4.4.5
  RHEL 7 on x86_64, using gcc version 4.8.5

This project requires GNU awk to build and expects the 'awk' found on the path
to be GNU awk. If this is not the case, you will need to install GNU awk either
from source (http://www.gnu.org/s/gawk/), Sunfree (http://www.sunfreeware.com).
You may also need to modify the 'PATH' enivronment variable to get scons to see
the right version, i.e.:

$> PATH=/path/to/awk:$PATH scons tests

KNOWN ISSUES
-	If the publisher provides the same unique identifier for multiple different records, the subscriber may lose records.
-	Journal resume handling in live mode may temporarily result in outdated records being transferred.
-	Under high record volume, repeated interruption of the subscriber with Ctrl-C can cause the publisher to no longer accept connections from that client.
-	When adding log or audits to the local store that greatly exceed the size limit of 200MB, the local store can fail to output a console message that records have been rejected.
-	If a subscriber tries to connect without TLS, to a publisher using TLS, the subscriber will display a thread exited message, but will not close until it times out, or a ctrl-c is pressed.


NOTES
  Due to the method in which the JALoP implementation allocates memory for
  records and possible system limitations, large files should be sent as
  journals and passed by file descriptor (i.e. using the jalp_journal_fd()
  function).

  Whenever a process crashes, or returns the error to run DB_RECOVER, stop all
  processes with access to the DB in question, run db_recover, and then restart
  the processes.

  The Berkeley DB database can be tailored to the user's load requirements
  by adding a DB_CONFIG file, with configuration options, to the directory where
  the database is to be created.

  Example DB_CONFIG:

  set_lk_max_locks    400000
  set_lk_max_lockers  400000
  set_lg_regionmax    400000
  set_tx_max          400000
  set_cachesize 1 0 4
  mutex_set_max       400000
  mutex_set_increment 400000

  A sample DB_CONFIG file is provided within the test-input directory.

  Before running the local store or either of the network stores, the directory
  that is to contain the database should be created and the sample DB_CONFIG file
  should be placed within it.

  http://docs.oracle.com/cd/E17076_02/html/programmer_reference/env_db_config.html

  Berkeley DB tools must be used to periodically checkpoint and archive DB log files, to prevent the database files from filling the available disk space.

  http://docs.oracle.com/cd/E17275_01/html/api_reference/C/db_checkpoint.html
  http://docs.oracle.com/cd/E17275_01/html/api_reference/C/db_archive.html