简体中文

rawbuf is a powerful tool kit used in object serialization and deserialization with full automation feature, all new design based on YAS.

The design of rawbuf:

rawbuf is inspired by flatbuffers and slothjson. flatbuffers is efficient enough but not so stupid & succinct, while slothjson is stupid & succinct enough but not so efficient. That is what rawbuf needs to do.
rawbuf uses flatbuffers-like protocol to describe data, and uses YAS for automation. So it is faster than slothjson and easier than flatbuffers.

Please refer to slothjson for more details.

• Efficient (4x faster than slothjson)
• Succinct interface for people (everything can be done with just a single line of code)
• Simple, powerful code generator with full automation (not need to implement serialize/deserialize interfaces manually)
• Support optional field (easy to serialize/deserialize field optionally)
• Flexible schema (support array, dict, nested object and nested array & dict)
• Succinct design (no tricky C++ template technology, easy to understand), reusable, extensible (easy to support new types)
• Cross-Platform (Windows & Linux & OS X)

Take C++ implement of rawbuf as an example. In the beginning, you need to add the following items to your project:

• rawbuf: refer to cpp/include/rawbuf.h and cpp/include/rawbuf.cpp, the library of rawbuf

That's all the dependency.

Then, write a schema named fxxx_gfw.json:

{
    "structs": 
    [
        {
            "type": "fxxx_gfw_t",
            "members": 
            [
                ["bool", "bool_val", "100"],
                ["int8_t", "int8_val"],
                ["int32_t", "int32_val"],
                ["uint64_t", "uint64_val"],
                ["double", "double_val", "101"],
                ["string", "str_val"],
                ["[int32_t]", "vec_val", "110"],
                ["{string}", "dict_val"]
            ]
        }
    ]
}

Run command line:

python cpp/generator/rawbuf.py -f cpp/src/fxxx_gfw.json

It will generate fxxx_gfw.h and fxxx_gfw.cpp, which you need to add to your project.
Then you can code like this:

rawbuf::fxxx_gfw_t obj_val;
// set the value of "obj_val"
......
// output as instance of "rb_buf_t"
rawbuf::rb_buf_t rb_val = rawbuf::rb_create_buf(rawbuf::rb_sizeof(obj_val));
bool rc = rawbuf::rb_encode(obj_val, rb_val);
// use value of "rb_val"
......
rawbuf::rb_dispose_buf(rb_val); // do not forget!
// output as file
std::string path = "fxxx_gfw_t.bin";
bool rc = rawbuf::rb_dump(obj_val, path);

If you don't want to serialize all fields, code like this:

obj_val.skip_dict_val(); // call "skip_xxx"

The same as deserialize:

rawbuf::rb_buf_t rb_val = rawbuf::rb_create_buf(rawbuf::rb_sizeof(obj_val));
// set the value of "rb_val"
......
// load from "rb_val"
rawbuf::fxxx_gfw_t obj_val;
bool rc = rawbuf::rb_decode(rb_val, 0, obj_val);
......
rawbuf::rb_dispose_buf(rb_val); // do not forget!

// load from file
std::string path = "fxxx_gfw_t.bin";
rawbuf::fxxx_gfw_t obj_val;
bool rc = rawbuf::rb_load(path, obj_val);

After deserialized, if you need to know whether a field is in binary buffer or not, code like this:

if (obj_val.rb_has_dict_val()) // call "rb_has_xxx()"
{
     ......
}

That's all about the usage, simple & stupid, isn't it ?

• C++
• C
• Go

I implement rawbuf using php & python, but not merge to master branch as the performance does not come up to expectation. Welcome contribution on other programming languages' implementation.

• php-alpha
• python-alpha
• php-beta
• python-beta

Note: the performance of beta is better than alpha.

rawbuf and slothjson share the same design, same schema. The difference between them is the protocol (text vs binary) and performance.

You can get all details from here and here

rawbuf is licensed under New BSD License, a very flexible license to use.

• chengzhuo (jobs, yao050421103@163.com)

• Yet Another Schema - YAS
• Object Serialization Artifact For Lazy Man - slothjson
• High-performance Distributed Storage - huststore