The project includes:

• an executable flatcc FlatBuffers schema compiler for C and a corresponding library libflatcc.a. The compiler generates C header files or a binary flatbuffers schema.
• a typeless runtime library libflatccrt.a for building and verifying flatbuffers from C. Generated builder headers depend on this library. It may also be useful for other language interfaces. The library maintains a stack state to make it easy to build buffers from a parser or similar.
• a small portable header only library for non-C11 compliant compilers, and small helpers for all compilers including endian handling and numeric printing and parsing.

The flatcc compiler is implemented as a standalone tool instead of extending Googles flatc compiler in order to have a pure portable C library implementation of the schema compiler that is designed to fail graciously on abusive input in long running processes. It is also believed a C version may help provide schema parsing to other language interfaces that find interfacing with C easier than C++. The FlatBuffers team at Googles FPL lab has been very helpful in providing feedback and answering many questions to help ensure the best possible compatibility. Notice the name flatcc (FlatBuffers C Compiler) vs Googles flatc.

The flatcc compiler has some extra features such as allowing for referencing structs defined later in the schema - which makes sense since it is already possible with other types. The binary schema format also does not preserve the order of structs. The option is controlled by a flag in config.h The generated source supports both bottom-up and top-down construction mixed freely.

flatcc does not support parsing JSON objects.

See also the Google FPL FlatBuffers homepage.

The project is still young but test cases cover most functionality and has been run on OS-X 10.11 with clang and Ubuntu 14.04 with gcc.

Buffer verification was introduced in v0.1.1 which verifies boundaries and alignment. Please note that verification does not ensure that it is safe to mutate buffers because an attacker may still overlap objects within the buffer.

Native code generation for FlatBuffer JSON printing and parsing added in 0.2.0. This is a very fast json processor that can parse or print a 700 byte json message at a rate of ca 1/2 million messages/second (ca. 600MB/s print and 400MB/s parse). See benchmarks below.

NOTE: In version 0.2.0 the buffer verifier result code has been inverted so non-zero is now indicates an error similar to other integer return codes in the runtime library. This makes it possible to return meaningful errors without loss of performance, e.g. that a buffer failed due to a string that was not zero terminated, or a vector was aligned to its element size.

Big endian platforms have not been tested at all. While care has been taken to handle endian encoding, there are bound to be some issues. The approach taken is to make it work on little endian - then it can always be made to work on big endian later given that output generated by an little endian platforms presumable will be correct regardless of bugs in endian encoding.

The portability layer for older compilers is considered important, but it is not well tested and it will not be prioritized - rather it will be updated based on user feedback from target specific testing.

flatcc is able encode and decode complex flatbuffer structures as is, and there are no big known issues apart from the above. The structure of the common header files are subject to change, notably all functions and macros prefixed with __. The general interface is subject to change based on feedback but it does appear functional as is. The typeless flatcc runtime library should hopefully not change much - it is mostly not user facing but other language exports might want to use it.

There are no plans to make frequent updates once the project becomes stable, but input from the community will always be welcome and included in releases where relevant, especially with respect to testing on different target platforms.

The priority has been to design an easy to use C builder interface that is reasonably fast, suitable for both servers and embedded devices, but with usability over absolute performance - still the small buffer output rate is measured in millons per second and read access 10-100 millon buffers per second from a rough estimate.

For 100MB buffers with 1000 monsters, dynamically extended monster names, monster vector, and inventory vector, the bandwidth reaches about 2.2GB/s and 45ms/buffer on 2.2GHz Haswell Core i7 CPU. This inlcudes reading back and validating all data. Reading only a few key fields increases bandwidth to 2.7GB/s and 37ms/op. For 10MB buffers bandwidth may be higher but eventually smaller buffers will be hit by call overhead and thus we down to 300MB/s at about 150ns/op encoding small buffers. These numbers are just a rough guideline - they obviously depend on hardware, compiler, and data encoded. Measurements are excluding an ininitial warmup step.

See also benchmark below.

The client C code can avoid almost any kind of allocation to build buffers as a builder stack provides an extensible arena before committing objects - for example appending strings or vectors piecemeal. The stack is mostly bypassed when a complete object can be constructed directly such as a vector from integer array on little endian platforms.

The reader interface should be pretty fast as is with less room for improvement performance wise. It is also much simpler than the builder.

Usability has also been prioritized over smallest possible generated source code and compile time. It shouldn't affect the compiled size by much.

The compiled binary output should be reasonably small for everything but the most restrictive microcontrollers. A 33K monster source test file (in addition to the generated headers and the builder library) results in a less than 50K optimized binary executable file including overhead for printf statements and other support logic, or a 30K object file excluding the builder library.

Read-only binaries are smaller but not necessarily much smaller considering they do less work: The compatibility test reads a pre-generated binary monsterdata_test.golden monster file and verifies that all content is as expected. This results in a 13K optimized binary executable or a 6K object file. The source for this check is 5K excluding header files. Readers do not need to link with a library.

In earlier releases it was attempted to read-only code self-sufficient in a few geenrated header files. It is still possible to not rely on a runtime library. Other generated files depend on the same headers and a runtime library. The builder also depends on the generated reader, but other generated files such as the verifier, json parser and json printer are self sufficient and only depend on the fixed library code.

The reader and builder rely on generated common reader and builder code. This common file makes it possible to change global namespace and redefine basic types (uoffset_t etc.). In the future this might move into library code and use macros for these abstractions and eventually have a set of predefined files for types beyond the standard 32-bit unsigned offset (uoffset_t). The runtime library is specific to on set of type definitions.

Reader code is reasonably straight forward and the generated code is more readable than the builder code because the generated functions headers are not buried in macros. Refer to monster_test.c and the generated files for detailed guidance on use. The monster schema used in this project is a slight adaptation to the original to test some additional edge cases.

For building flatbuffers a separate builder header file is generated per schema. It requires a flatbuffers_common_builder.h file also generated by the compiler and a small runtime library libflatccrt.a. It is because of this requirement that the reader and builder generated code is kept separate. Typical uses can be seen in the monster_test.c file. The builder allows of repeated pushing of content to a vector or a string while a containing table is being updated which simplifies parsing of external formats. It is also possible to build nested buffers in-line - at first this may sound excessive but it is useful when wrapping a union of buffers in a network interface and it ensures proper alignment of all buffer levels.

For verifying flatbuffers, a myschema_verifier.h is generated. It depends on the runtime library but is completely independent of the the generated reader, builder, and common files.

Low level note: the builder generates all vtables at the end of the buffer instead of ad-hoc in front of each table but otherwise does the same deduplication of vtables. This makes it possible to cluster vtables in hot cache or to make sure all vtables are available when partially transmitting a buffer. This behavior can be disabled by a runtime flag.

Because some use cases may inlclude very constrained embedded devices, the builder library can be customized with an allocator object and a buffer emitter object. The separate emitter ensures a buffer can be constructed without requiring a full buffer to be present in memory at once, if so desired.

The typeless builder library is documented in flatcc_builder.h and flatcc_emitter.h while the generated typed builder api for C is documented in doc/builder.md.

Refer to flatcc -h for details.

The compiler can either generate a single header file or headers for all included schema and a common file and with or without support for both reading (default) and writing (-w) flatbuffers. The simplest option is to use (-a) for all and include the myschema_builder.h file.

The (-a) or (-v) also generates a verifier file which only depends on the runtime library (and other verifiers from included schema).

Make sure flatcc under the include folder is visible in the C compilers include path when compiling flatbuffer builders. It is not needed for readers without -DFLATCC_PORTABLE defined.

The flatcc (-I) include path will assume all files with same base name (case insentive) are identical and only include the first. All generated files use the input basename and land in working directory or the path set by (-o).

Note that the binary schema output can be with or without namespace prefixes and the default differs from flatc which strips namespaces. The binary schema can also have a non-standard size field prefixed so multiple schema can be concatenated in a single file if so desired (see also the bfbs2json example).

Files can be generated to stout using (--stdout). C headers will be ordered and concatenated, but otherwise identical to the separate file output. Each include statement is guarded so this will not lead to missing include files. When including builder logic.

The generated code, especially with all combined with --stdout, may appear large, but only the parts actually used will take space up the the final executable or object file. Modern compilers inline and include only necessary parts of the statically linked builder library.

The generated C builder headers normally do require the builder library. But in reality it is possible to use them without for some purposes. The main reason is to use mystruct_assign/copy_from/to_pe for general endian handling of structs - eventually also with native big endian for other network protocols.

JSON printer and parser can be generated using the --json flag or --json-printer or json-parser if only one of them is required. There are some certain runtime library compile time flags that can optimize out printing symbolic enums, but these can also be disabled at runtime.

Here we provide a quick example of read-only access to Monster flatbuffer

• it is an adapted extract of the monster_test.c file.

First we compile the schema read-only with common (-c) support header and we add the recursion because monster_test.fbs includes other files.

flatcc -cr monster_test.fbs

we get:

flatbuffers_common_reader.h
include_test1_reader.h
include_test2_reader.h
monster_test_reader.h

Namespaces can be long so we use a macro to manage this.

#include "monster_test_reader.h"

#undef ns
#define ns(x) FLATBUFFERS_WRAP_NAMESPACE(MyGame_Example, x)

int verify_monster(void *buffer)
{
    ns(Monster_table_t) monster;
    /* This is a read-only reference to a flatbuffer encoded struct. */
    ns(Vec3_struct_t) vec;
    flatbuffers_string_t name;
    size_t offset;

    if (!(monster = ns(Monster_as_root(buffer)))) {
        printf("Monster not available\n");
        return -1;
    }
    if (ns(Monster_hp(monster)) != 80) {
        printf("Health points are not as expected\n");
        return -1;
    }
    if (!(vec = ns(Monster_pos(monster)))) {
        printf("Position is absent\n");
        return -1;
    }

    /* -3.2f is actually -3.20000005 and not -3.2 due to representation loss. */
    if (ns(Vec3_z(vec)) != -3.2f) {
        printf("Position failing on z coordinate\n");
        return -1;
    }

    /* Verify force_align relative to buffer start. */
    offset = (char *)vec - (char *)buffer;
    if (offset & 15) {
        printf("Force align of Vec3 struct not correct\n");
        return -1;
    }

    /*
     * If we retrieved the buffer using `flatcc_builder_finalize_aligned_buffer` or
     * `flatcc_builder_get_direct_buffer` the struct should also
     * be aligned without subtracting the buffer.
     */
    if (vec & 15) {
        printf("warning: buffer not aligned in memory\n");
    }

    /* ... */
    return 0;
}
/* main() {...} */

The above example is not very elegant. For a more complete example of read only buffer access see the bfbs2json example which converts a binary FlatBuffers schema into a JSON file.

Assuming our above file is monster_test.c the following are a few ways to compile the project:

cc monster_example.c -o monster_example

cc --std=c11 monster_test.c -o monster_test

cc -D FLATCC_PORTABLE -I include monster_test.c -o monster_test

The portability layer requires the include directive to find the support files - these are all header files.

We can also create a type specific namespace wrapper - which also works with the builder below:

#define Monster(x) FLATBUFFERS_WRAP_NAMESPACE(MyGame_Example_Monster, x)
...
assert(Monster(hp(monster)) == 80);

or for even easer read access if we assume monster is stored in monster.

#define MonsterGet(x) FLATBUFFERS_WRAP_NAMESPACE(MyGame_Example_Monster, x(monster))
...
assert(MonsterGet(hp) == 80);

all without loosing type safety.

Here we provide a very limited example of how to build a buffer - only a few fields are updated. Pleaser refer to monster_test.c and the doc directory for more information.

First we must generate the files:

flatcc -a monster_test.fbs

This produces:

flatbuffers_common_reader.h
flatbuffers_common_builder.h
flatbuffers_common_verifier.h
include_test1_reader.h
include_test1_builder.h
include_test1_verifier.h
include_test2_reader.h
include_test2_builder.h
include_test2_verifier.h
monster_test_reader.h
monster_test_builder.h
monster_test_verifier.h

Note: we wouldn't actually do the readonly generation shown earlier unless we only intend to read buffers - the builder generation always generates read acces too.

By including "monster_test_builder.h" all other files are included automatically. The C compiler needs the -I include directive to access flatbuffers/flatcc_builder.h and related.

The verifiers are not required and just created because we lazily chose the -a option.

The builder must be initialized first to set up the runtime environment we need for building buffers efficiently - the builder depends on an emitter object to construct the actual buffer - here we implicitly use the default. Once we have that, we can just consider the builder a handle and focus on the FlatBuffers generated API until we finalize the buffer (i.e. access the result). For non-trivial uses it is recommended to provide a custom emitter and for example emit pages over the network as soon as they complete rather than merging all pages into a single buffer using flatcc_builder_finalize_buffer, or the simplistic direct buffer access shown below. See also flatcc_builder.h and flatcc_emitter.h.

#include "monster_test_builder.h"

/* See `monster_test.c` for more advanced examples. */
void build_monster(flatcc_builder_t *B)
{
    ns(Vec3_t *vec);

    /* Here we use a table, but structs can also be roots. */
    ns(Monster_start_as_root(B));

    ns(Monster_hp_add(B, 80));
    /* The vec struct is zero-initalized. */
    vec = ns(Monster_pos_start(B));
    /* Native endian. */
    vec->x = 1, vec->y = 2, vec->z = -3.2f;
    /* _end call converts to protocol endian format - for LE it is a nop. */
    ns(Monster_pos_end(B));

    /* Name is required, or we get an assertion in debug builds. */
    ns(Monster_name_create_str(B, "MyMonster"));

    ns(Monster_end_as_root(B));
}

#include "test/support/hexdump.h"

int main(int argc, char *argv[])
{
    flatcc_builder_t builder;
    void *buffer;
    size_t size;

    flatcc_builder_init(&builder);

    build_monster(&builder);
    /* We could also use `flatcc_builder_finalize_buffer` and free the buffer later. */
    buffer = flatcc_builder_get_direct_buffer(&builder, &size);
    assert(buffer);
    verify_monster(buffer);

    /* Visualize what we got ... */
    hexdump("monster example", buffer, size, stdout);

    /*
     * Here we can call `flatcc_builder_reset(&builder) if
     * we wish to build more buffers before deallocating
     * internal memory with `flatcc_builder_clear`.
     */

    flatcc_builder_clear(&builder);
    return 0;
}

Compile the example project:

cc --std=c11 -I include monster_example.c lib/libflatccrt.a -o monster_example

Note that here the include directive is required even without the portability layer so that flatbuffers/flatcc_builder.h can be found.

A buffer can be verified to ensure it does not contain any ranges that point outside the the given buffer size, that all data structures are aligned according to the flatbuffer principles, that strings are zero terminated, and that required fields are present.

In the builder example above, we can apply a verifier to the output:

#include "monster_test_builder.h"
#include "monster_test_verifier.h"
int ret;
...
... finalize
if ((ret = ns(Monster_verify_as_root(buffer, size, "MONS")))) {
    printf("Monster buffer is invalid: %s\n",
    flatcc_verify_error_string(ret));
}

Flatbuffers can optionally leave out the identifier, here "MONS". Use a null pointer as identifier argument to ignore any existing identifiers and allow for missing identifiers.

Nested flatbbuffers are always verified with a null identifier, but it may be checked later when accessing the buffer.

The verifier does NOT verify that two datastructures are not overlapping. Sometimes this is indeed valid, such as a DAG (directed acyclic graph) where for example two string references refer to the same string in the buffer. In other cases an attacker may maliciously construct overlapping datastructures such that in-place updates may cause subsequent invalid buffers. Therefore an untrusted buffer should never be updated in-place without first rewriting it to a new buffer.

Note: prior to version 0.2.0, the verifier would fail on 0 and report success on non-zero value. As of 0.2.0, success is indicated by 0, and non-zero yields an error code that can be translated into a string.

The CMake build system has build option to enable assertions in the verifier. This will break debug builds and not usually what is desired, but it can be very useful when debugging why a buffer is invalid.

See also include/flatcc/flatcc_verifier.h.

By using the flatbuffer schema it is possible to generate schema specific JSON printers and parsers. This differs for better and worse from Googles flatc tool which takes a binary schema as input and processes json input and output. Here that parser and printer only rely on the flatcc runtime library, is faster (probably significantly so), but requires recompilition when new json formats are to be supported - this is not as bad as it sounds - it would for example not be difficult to create a Docker container to process a specific schema in a web server context.

The json printer and parser are assymtric in the sense that the printer is a self-contained object with its own print context object that operates on a binary flatbuffer, while the parser has only a lightweight context for error messages and really rely on the flatcc builder object for its context.

The parser always takes a text buffer as input and produces output according to how the builder object is initialized. The printer has different init functions: one for printing to a file pointer, including stdout, one for printing to a fixed size external buffer, and one for printing to a dynamically growing buffer. The dynamic buffer may be reused between prints via the reset function. See flatcc_json_parser.h for details.

The parser will accept unquoted names (not strings) and trailing commas, i.e. non-strict json and also allows for hex \x03 in strings. Strict mode must be enabled by a compile time flag. In addition the parser schema specific symbolic enum values that can optionally be unquoted where a numeric value is expected:

`color: Green`
`color: Color.Green`
`color: MyGame.Example.Color.Green`
`color: 2`

The symbolic values do not have to be quoted, but can be while numeric values cannot be quoted. If no namespace is provided, like color: Green, the symbol must match the receiving enum type. Any scalar value may receive a symbolic value either in a relative namespace like hp: Color.Green, or an absolute namespace like hp: MyGame.Example.Color.Green, but not hp: Green (since hp in the monster example schema) is not an enum type with a Green value). A namespace is relative to the namespace of the receiving object.

It is also possible to have multiple values:

`color: "Green Red"`
`color: Green Red`

These are originally intended for enums that have the bit flag attribute defined (which Color does have), but this is tricky to process, so therefore any symblic value can be listed in a sequence with or without namespace as appropriate. Because this further causes problems with signed symbols the exact definition is that all symbols are first coerced to the target type (or fail), then added to the target type if not the first this results in:

`color: Green Blue Red Blue`
`color: 19`

Because Green is 2, Red is 1, Blue is 8 and repeated.

It is not valid to specify an empty set like:

`color: ""`

because it might be understood as 0 or the default value, and it does not unquote very well.

The printer will by default print valid json without any spaces and everything quoted. Use the non-strict formatting option (see headers and test examples) to produce pretty printing. It is possibly to disable symbolic enum values using the noenum option.

Only enums will print symbolic values are there is no history of any parsed symbolic values at all. Furthermore, symbolic values are only printed if the stored value maps cleanly to one value, or in the case of bit-flags, cleanly to multiple values. For exmaple if parsing color: Green Red it will print as "color":"Red Green" by default, while color: Green Blue Red Blue will print as color:19.

Both printer and parser are limited to roughly 100 table nesting levels and an additional 100 nested struct depths. This can be changed by configuration flags but must fit in the runtime stack since the operation is recursive descent. Exceedning the limits will result in an error.

Numeric values are coerced to the receiving type. Integer types will fail if the assignment does not fit the target while floating point values may loose precision silently. Integer types never accepts floating point values. Strings only accept strings.

Nested flatbuffers may either by arrays of byte sized integers, or a table or a struct of the target type. See test cases for details.

The parser will by default fail on unknown fields, but these can also be skipped silently with a runtime option.

Unions are difficult to parse. A union is two json fields: a table as usual, and an enum to indicate the type which has the same name with a _type suffix and accepts a numeric or symbolic type code:

{
  name: "Container Monster", test_type: Monster,
  test: { name: "Contained Monster" }
}

Because other json processors may sort fields, it is possible to receive the type field after the test field. The parser does not store temporary datastructures. It constructs a flatbuffer directly. This is not possible when the type is late. This is handled by parsing the field as a skipped field on a first pass, followed by a typed back-tracking second pass once the type is known (only the table is parsed twice, but for nested unions this can still expand). Needless to say this slows down parsing. It is an error to provide only the table field or the type field alone, except if the type is NONE or 0 in which case the table is not allowed to be present.

For detailed usage, please refer to

`test/json_test/test_json_printer.c`
`test/json_test/test_json_parser.c`
`test/json_test/json_test.c`
`test/benchmark/benchflatccjson`

Be sure to have external/grisu3, include/portable and include/flatcc visible when compiling with default configuration. Do this by adding external and include to the include path, or copy grisu3 from external to include directory and just add inlcude.

Note that json parsing and printing is very fast reaching 500MB/s for printing and about 300 MB/s for parsing. Floating point parsing can signficantly skew these numbers. The integer and floating point parsing and printing are handled via support functions in the portable library. In addition the floating point external/grisu3 library is expected to be available, but can be disabled. Disabling grisu3 will revert to sprintf and strtod. Grisu3 will fall back to strtod and grisu3 in some rare special cases. Due to the reliance on strtod and because strtod cannot efficiently handle non-zero-terminated buffers, it is recommended to zero terminate buffers. Alternatively, grisu3 can be compiled with a flag that allows errors in conversion. These errors are very small and still correct, but may break some checksums. Allowing for these errors can significantly improve parsing speed and moves the benchmark from below half a million parses to above half a million parses per second on 700 byte json string, on a 2.2 GHz core-i7.

While unquoted strings may sound more efficient due to the compact size, it is actually slower to process. Furthermore, large flatbuffer generated JSON files may compress by a factor 8 using gzip or a factor 4 using LZ4 so this is probably the better place to optimize. For small buffers it may be more efficient to compress flatbuffer binaries, but for large files, json may actually compress significantly better due to the absence of pointers in the format.

SSE 4.2 has been experimentally added, but it the gains are limited because it works best when parsing space, and the space parsing is already fast without SSE 4.2 and because one might just leave out the spaces if in a hurry. For parsing strings, trivial use of SSE 4.2 string scanning doesn't work well becasuse all the escape codes below ASCII 32 must be detected rather than just searching for \ and ". That is not to say there are not gains, they just don't seem worthwhile.

The parser is heavily optimized for 64-bit because it implements an 8-byte wide trie directly in code. It might work well for 32-bit compilers too, but this hasn't been tested. The large trie does put some strain on compile time. Optimizing beyond -O2 leads to too large binaries which offsets any speed gains.

The FlatBuffers schema is intended to have a single root object and a single file identifier and extension - but this mostly makes sense for a JSON parser needing to make an automated choice (and flatcc does not read JSON objecs).

In praxis all objects (tables and structs) can be roots. This makes it ambigious as to what identifier an individual object should use as root. flatcc uses the file_identifier last set in the current schema file (ignoring included schema). The identifier is frozen by a define myobject_identifer and myobject_extension so it does not change with different included include schema. There is also a global file_identifier representing the last set schema identifier, and ditto extension. The binary schema code generator uses this to set the identifier and file extension of its output.

Each object created _as_root will automatically use the identifier from its myobject_identifier definition. When an object is read as_root this identifier is checked and will return null or assert in debug. The _as_root call is really a wrapper around first opening or closing a buffer then reading the first object as the given root type. This can be done as two steps and allows detailed control of identifier handling - see builder.md for more.

Attributes included in the schema are viewed in a global namespace and each include file adds to this namespace so a schema file can use included attributes without namespace prefixes.

Each included schema will also add types to a global scope until it sees a namespace declaration. An included schema does not inherit the namespace of an including file or an earlier included file, so all schema files starts in the global scope. An included file can, however, see other types previously defined in the global scope. Because include statements always appear first in a schema, this can only be earlier included files, not types from a containing schema.

The generated output for any included schema is indendent of how it was included, but it might not compile without the earlier included files being present and included first. By including the toplevel myschema.h or myschema_builder.h all these dependencies are handled correctly.

Note: libflatcc.a can only parse a single schema when the schema is given as a memory buffer, but can handle the above when given a filename. It is almost possible to simply concatenate schema together to obtain the same effect in memory, but for this to work each schema must be separated by a global scope reset such as an empty namespace;. This is currently not supported.

If a field is required such as Monster.name, the table end call will assert in debug mode and create incorrect tables in non-debug builds. The assertion may not be easy to decipher as it happens in library code and it will not tell which field is missing.

When reading the name, debug mode will again assert and non-debug builds will return a default value.

Writing the same field twice will also trigger an assertion in debug builds.

Buffers can be used for high speed communication by using the ability to create buffers with structs as root. In addition the default emitter supports flatcc_emitter_direct_buffer for small buffers so no extra copy step is required to get a linear buffer in memory. Preliminary measurements suggests there is a limit to how fast this can go (about 6-7 mill. buffers/sec) because the builder object must be reset between buffers which involves zeroing allocated buffers. Small tables with a simple vector achieve roughly half that speed. For really high speed a dedicated builder for structs would be needed. See also monster_test.c.

All types stored in a buffer has a type suffix such as Monster_table_t or Vec3_struct_t (and namespace prefix which we leave out here). These types are read-only pointers into endian encoded data. Enum types are just constants easily grasped from the generated code. Tables are dense so they are never accessed directly.

Structs have a dual purpose because they are also valid types in native format, yet the native reprsention has a slightly different purpose. Thus the convention is that a const pointer to a struct encoded in a flatbuffer has the type Vec3_struct_t where as a writeable pointer to a native struct has the type Vec3_t * or struct Vec3 *.

Union types are just any of a set of possible table types and an enum named as for example Any_union_type_t. There is a compound union type that can store both type and table reference such that create calls can represent unions as a single argument - see flatcc_builder.h and doc/builder.md. Union table fields return a pointer of type flatbuffers_generic_table_t which is defined as const void *.

All types have a _vec_t suffix which is a const pointer to the underlying type. For example Monster_table_t has the vector type Monster_vec_t. There is also a non-const variant with suffix _mutable_vec_t which is rarely used. However, it is possible to sort vectors in-place in a buffer, and for this to work, the vector must be cast to mutable first. A vector (or string) type points to the element with index 0 in the buffer, just after the length field, and it may be cast to a native type for direct access with attention to endian encoding. (Note that table_t types do point to the header field unlike vectors.) These types are all for the reader interface. Corresponding types with a _ref_t suffix such as _vec_ref_t are used during the construction of buffers.

Native scalar types are mapped from the FlatBuffers schema type names such as ubyte to uint8_t and so forth. These types also have vector types provided in the common namespace (default flatbuffers_) so a [ubyte] vector has type flatbuffers_uint8_vec_t which is defined as const uint8_t *.

The FlatBuffers boolean type is strictly 8 bits wide so we cannot use or emulate <stdbool.h> where sizeof(bool) is implementation dependent. Therefore flatbuffers_bool_t is defined as uint8_t and used to represent FlatBuffers boolean values and the constants of same type: flatbuffers_true = 1 and flatbuffers_false = 0. Even so, pstdbool.h is available in the include/portable directory if bool, true, and false are desired in user code and <stdbool.h> is unavailable.

flatbuffers_string_t is const char * but imply the returned pointer has a length prefix just before the pointer. flatbuffers_string_vec_t is a vector of strings. The flatbufers_string_t type guarantees that a length field is present using flatbuffers_string_len(s) and that the string is zero terminated. It also suggests that it is in utf-8 format according to the FlatBuffers specification, but not checks are done and the flatbuffers_create_string(B, s, n) call explicitly allows for storing embedded null characters and other binary data.

All vector types have operations defined as the typename with _vec_t replaced by _vec_at and _vec_len. For example flatbuffers_uint8_vec_at(inv, 1) or Monster_vec_len(inv). The length or _vec_len will be 0 if the vector is missing whereas _vec_at will assert in debug or behave undefined in release builds following out of bounds access. This also applies to related string operations.

To be consistent with a user define namespace ns, the common namespace can also be wrapped in a macro in exactly the same way:

#undef nsc
#define nsc(x) FLATBUFFERS_WRAP_NAMESPACE(flatbuffers, x)

nsc(uint8_vec_t) vec;

The common namespace can also be redefined using the flatcc --common-prefx myprefix argument to change flatbuffers_ to something else. In that case the macro above becomes:

#undef nsc
#define nsc MYPREFIX_WRAPNAMESPACE(myprefix, x)

One reason to change the common namespace is if the flatbuffer offset types are changed, for example to handle large 64-bit binary chunks. Note however that the builder library can currently only be compiled with one specific set of types.

The generated code supports the FLATBUFFERS_LITTLEENDIAN flag defined by the flatc compiler and it can be used to force endianness. Big Endian will define it as 0. Other endian may lead to unexpected results. In most cases FLATBUFFERS_LITTLEENDIAN will be defined but in some cases a decision is made in runtime where the flag cannot be defined. This is likely just as fast, but #if FLATBUFFERS_LITTLEENDIAN can lead to wrong results alone - use #if defined(FLATBUFFERS_LITTLEENDIAN) && FLATBUFFERS_ENDIAN to be sure the platform is recognized as little endian. The detection logic will set FLATBUFFERS_LITTLEENDIAN if at all possible and can be improved with the pendian.h file included by the -DFLATCC_PORTABLE.

The user can always define -DFLATBUFFERS_LITTLEENDIAN as a compile time option and then this will take precedence.

It is recommended to use flatbuffers_is_native_pe() instead of testing FLATBUFFERS_LITTLEENDIAN whenever it can be avoided to use the proprocessor for several reasons:

• if it isn't defined, the source won't compile at all
• combined with pendian.h it provides endian swapping even without preprocessor detection.
• it is normally a constant similar to FLATBUFFERS_LITTLEENDIAN
• it works even with undefined FLATBUFFERS_LITTLEENDIAN
• compiler should optimize out even runtime detection
• protocol might not always be little endian.
• it is defined as a macro and can be checked for existence.

pe means protocol endian. This suggests that flatcc output may be used for other protocols in the future, or for variations of flatbuffers. The main reason for this is the generated structs that can be very useful on other predefined network protols in big endian. Each struct has a mystruct_copy_from_pe method and similar to do these conversions. Internally flatcc optimizes struct conversions by testing flatbuffers_is_native_pe() in some heavier struct conversions.

In a few cases it may be relevant to test for FLATBUFFERS_LITTLEENDIAN but then code intended for general use should provide an alternitive for when the flag isn't defined.

Even with correct endian detection, the code may break on platforms where flatbuffers_is_native_pe() is false because the necessary system headers could not found. In this case -DFLATCC_PORTABLE should help.

FlatBuffers use 32-bit uoffset_t and 16-bit voffset_t. soffset_t always has the same size as uoffset_t. These types can be changed by preprocessor defines without regenerating code. However, it is important that libflatccrt.a is compiled with the same types as defined in flatcc_types.h.

uoffset_t currently always point forward like flatc. In retrospect it would probably have simplified buffer constrution if offsets pointed the opposite direction or were allowed to be signed. This is a major change and not likely to happen for reasons of effort and compatibility, but it is worth keeping in mind for a v2.0 of the format.

Vector header fields storing the lengthare defined as uoffset_t which is 32-bit wide by default. If uoffset_t is redefined this will therefore also affect vectors and strings.

The practical buffer size is limited to about half of the uoffset_t range because vtable references are signed which in effect means that buffers are limited to about 2GB by default.

The builder API often returns a reference or a pointer where null is considered an error or at least a missing object default. However, some operations do not have a meaningful object or value to return. These follow the convention of 0 for success and non-zero for failure. Also, if anything fails, it is not safe to proceed with building a buffer. However, to avoid overheads, there is no hand holding here. On the upside, failures only happen with incorrect use or allocation failure and since the allocator can be customized, it is possible to provide a central error state there or to guarantee no failure will happen depending on use case, assuming the API is otherwise used correctly. By not checking error codes, this logic also optimizes out for better performance.

The builder API does not support sorting due to the complexity of customizable emitters, but the reader API does support sorting so a buffer can be sorted at a later stage. This requires casting a vector to mutable and calling the sort method available for fields with keys.

The sort uses heap sort and can sort a vector in-place without using external memory or recursion. Due to the lack of external memory, the sort is not stable. The corresponding find operation returns the lowest index of any matching key, or flatbuffers_not_found.

When configured in config.h, the flatcc compiler allows multiple keyed fields unlike Googles flatc compiler. This works transparently by providing <table_name>_vec_sort_by_<field_name> and <table_name>_vec_find_by_<field_name> methods for all keyed fields. The first field maps to <table_name>_vec_sort and <table_name>_vec_find. Obviously the chosen find method must match the chosen sort method.

See also doc/builder.md and test/monster_test/monster_test.c.

The FlatBuffers format does not fully distinguish between default values and missing or null values but it is possible to force values to be written to the buffer. This is discussed further in the builder.md. For SQL data roundtrips this may be more important that having compact data.

The _is_present suffix on table access methods can be used to detect if value is present in a vtable, for example Monster_hp_present. Unions return true of the type field is present, even if it holds the value None.

The add methods have corresponding force_add methods for scalar and enum values to force storing the value even if it is default and thus making it detectable by is_present.

The portablity layer is not required when --std=c11 is defined on a clang compiler where little endian is avaiable and easily detected, or where <endian.h> is available and easily detected. flatbuffers_common_reader.h contains a minimal portability abstraction that also works for some platforms even without C11, e.g. OS-X clang. The portablity file can be included before other headers, or by setting the compile time directives:

cc -D FLATCC_PORTABLE -I include monster_test.c -o monster_test

Also see the top of this file on how to include the actual portability layer when needed.

If a specific platform has been tested, it would be good with feedback and possibly patches to the portability layer so this can be documented for other users.

To initialize and run the build:

scripts/build.sh

The bin and lib folders will be created with debug and release builds.

The build depends on CMake and the Ninja build tool to be installed along with Python. The Ninja build backend can be replaced by editing scripts/initbuild.sh and scripts/build.sh.

To install build tools on OS-X:

brew update
brew install cmake ninja

To install build tools on Ubuntu:

sudo apt-get update
sudo apt-get install cmake ninja-build

Run

scripts/test.sh

It will also initialize the build if needed. Several sub-tests have shell scripts that can be run in isolation. A large dataset test is part of the test, but a separate benchmark is not. See benchmark section.

The script must end with TEST PASSED, or it didn't.

Windows hasn't been tested but the portability layer does include abstractions to support the Windows platform, such as defining inline as __inline and providing <endian.h> abstractions. However, the Windows Visual Studio platform is fairly focused on C++ so it wouldn't be the primary target.

With clang being ported to Windows there shouldn't be much reason for any portablitiy issues using that tool chain - although clangs runtime libraries do depend on the platform affecting the degree to which C11 is supported.

The build scripts are shell scripts wrapping CMake, but CMake can also be used directly. The test scripts are more involved and expect a bash shell, but this is also available on Windows.

The file logic in the compiler may fail on Windows paths as this hasn't been tested, but at least it has been implemented with Windows in mind.

The compiler library libflatcc.a can compile schemas provided in a memory buffer or as a filename. When given as a buffer, the schema cannot contain include statements - these will cause a compile error.

When given a filename the behavior is similar to the commandline flatcc interface, but with more options - see flatcc.h and config/config.h.

The binary schema options are named bgen_... and can be used to choose namespace prefixes or not.

libflatcc.a supports functoins named flatcc_.... reflection... may also be available which are simple the C generated interface for the binary schema. The builder library is also included. These last two interfaces are only present because the library supports binary schema generation.

The standalone libflatcc_builder.a is much smaller and may in fact be linked directly as a single flatcc_builder.c file, optionally also with flatcc_emitter.c. The builder libraries primary funciton is to support the generated C builder wrappers, but it can do more things: for example creating vtables ahead of time and creating tables with a specific vtable. This may be used to create high performance applications. The library is also typeless as the wrappers provide actual types. Therefore dynamic languages can use it to construct flatbuffers on the fly, and the same applies to generic JSON, XML or similar parsers. Of course, some schema is eventually required to understand the generated output but even that might be generated on the fly by a parser or language interface.

Benchmarks are defined for raw C structs, Googles flatc generated C++ and the flatcc compilers C ouput.

These can be run with:

scripts/benchmark.sh

and requires a C++ compiler installed - the benchmark for flatcc alone can be run with:

test/benchmark/benchflatcc/run.sh

this only requires a system C compiler (cc) to be installed (and flatcc's build environment).

A summary for OS-X 2.2 GHz Haswell core-i7 is found below. Generated files for OS-X and Ubuntu are found in the benchmark folder.

The benchmarks use the same schema and dataset as Google FPL's FlatBuffers benchmark.

In summary, 1 million iterations runs at about 500-540MB/s at 620-700 ns/op encoding buffers and 29-34ns/op traversing buffers. flatc and flatcc are close enough in performance for it not to matter much. flatcc is a bit faster encoding but it is likely due to less memory allocation. Throughput and time per operatin are of course very specific to this test case.

Generated JSON parser/printer shown below, for flatcc only but for OS-X and Linux.

elapsed time: 0.055 (s)
iterations: 1000000
size: 312 (bytes)
bandwidth: 5665.517 (MB/s)
throughput in ops per sec: 18158707.100
throughput in 1M ops per sec: 18.159
time per op: 55.070 (ns)

elapsed time: 0.012 (s)
iterations: 1000000
size: 312 (bytes)
bandwidth: 25978.351 (MB/s)
throughput in ops per sec: 83263946.711
throughput in 1M ops per sec: 83.264
time per op: 12.010 (ns)

elapsed time: 0.702 (s)
iterations: 1000000
size: 344 (bytes)
bandwidth: 490.304 (MB/s)
throughput in ops per sec: 1425301.380
throughput in 1M ops per sec: 1.425
time per op: 701.606 (ns)

elapsed time: 0.029 (s)
iterations: 1000000
size: 344 (bytes)
bandwidth: 11917.134 (MB/s)
throughput in ops per sec: 34642832.398
throughput in 1M ops per sec: 34.643
time per op: 28.866 (ns)

elapsed time: 0.626 (s)
iterations: 1000000
size: 336 (bytes)
bandwidth: 536.678 (MB/s)
throughput in ops per sec: 1597255.277
throughput in 1M ops per sec: 1.597
time per op: 626.074 (ns)

elapsed time: 0.029 (s)
iterations: 1000000
size: 336 (bytes)
bandwidth: 11726.930 (MB/s)
throughput in ops per sec: 34901577.551
throughput in 1M ops per sec: 34.902
time per op: 28.652 (ns)

The benchmark uses Grisu3 floating point parsing and printing algorithm with exact fallback to strtod/sprintf when the algorithm fails to be exact. Better performance can be gained by enabling inexact Grisu3 and SSE 4.2 in build options, but likely not worthwhile in praxis.

(encode means printing from existing binary buffer to JSON)

elapsed time: 1.407 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 513.068 (MB/s)
throughput in ops per sec: 710619.931
throughput in 1M ops per sec: 0.711
time per op: 1.407 (us)

(decode/traverse means parsing json to flatbuffer binary and calculating checksum)

elapsed time: 2.218 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 325.448 (MB/s)
throughput in ops per sec: 450758.672
throughput in 1M ops per sec: 0.451
time per op: 2.218 (us)

Numbers for Linux included because parsing is significantly faster.

elapsed time: 1.210 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 596.609 (MB/s)
throughput in ops per sec: 826328.137
throughput in 1M ops per sec: 0.826
time per op: 1.210 (us)

elapsed time: 1.772 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 407.372 (MB/s)
throughput in ops per sec: 564227.736
throughput in 1M ops per sec: 0.564
time per op: 1.772 (us)