LZ4_SG is a scatter-gather variant of the fast compression library LZ4 (by Cyan). LZ4_SG links consecutive blocks into a single LZ4F frame compatible with LZ4F (v.1.5.0, i.e., can be decoded by LZ4F_decompress). LZ4_SG avoids copying memory of incompressible blocks, which are stored literally in place.

LZ4_SG library is provided as open-source software using BSD license.

The benchmark program used is based on density's benchmark (by gpnuma Density), which was extended to support scatter-gather compressors (TODO: fork it and push updates). In order to evaluate SG, input buffer is first copied to 4KB buffers. 4KB Output buffers are also allocated to accomodate compressed data. Decompression uses compressed buffers as input and stores output in the 4KB buffers used as input for compression. The benchmark program was compiled with GCC 4.8.3 -O3 on centos with kernel 2.6.32, and ran on reference system Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz. Benchmark evaluates the compression of reference Silesia Corpus in single-thread mode.

dual_memcpy reference point corresponds to copying scattered input blocks to a contiguous buffer, applying compression to another contiguous buffer, and then copying results to scattered output blocks. LZ4' is the modified version of LZ4 that supports SG, but is applied to contiguous input and output buffers.

Multi-core benchmark results (8 concurrent runs in terminal):

Decompress buffers should be same size as original buffers containing plain-text. That is, buffer lengths used as source,target for compression can be safely used as target,source for decompression.

LZ4_SG_compressBound (used to verify sufficient memory for output before compression) is not tight with extremely small buffers lengths (e.g., 16 bytes).

See lib/lz4sg.h for usage details (basically, similar to vectored I/O iovec).