Grape - realtime pipeline processing engine.

Its main goal is to create pipeline engine to process data stored to elliptics.
You can create a signal-slot topology on top of elliptics cluster,
where processing nodes are linked to events and can emit one or more events on its own.
Elliptics cluster can host as many application topologies as you wish.

Code can be executed in separate process with specified rlimits
or in a cgroup jailed environment, and later we plan to add containers
(if there will be a strict demand).

Execution workers are those where elliptics runs its server-side scripts,
and recently we switched from home-made pool of processes to very feature-rish Cocaine engine,
which allows elliptics to run not only Python, but also JavaScript and Perl.
It supports dynamic management of the process pool, jailed setup,
extended statistics and quite flexible configuration. For example Cocaine can load your code
from remote storage (elliptics or mongodb) or from local filesystem,
it provides storage and logging abstractions to workers and so on.

Grape in turn runs inside those workers and currently only on top of elliptics.
It is possible to add different message passing layers to this simple model,
like your own transport (let's say you want to deliver messages using multicast)
and processing nodes (like build them on top of your own routing technology
and not using elliptics).

Pipeline looks something like this (in c++-like pseudocode):

init() {
    topology = new topology(config)
    nodes = new elliptics_node_t[num]

    topology->add_slot("start-event", node[0])
    topology->add_slot("event0", node[0])
    topology->add_slot("event1", node[1])
    topology->add_slot("event2", node[2])
    topology->add_slot("finish", node[3])
}
node0::process(event, data) {
     /* process data */
    emit("event0", new-data)

     /* process data */
    emit("event1", new-data)
}
and so on...

Grape was made for C/C++ code processing for now,
but will work with Python, Perl and JavaScript as soon as we create bindings.

Links:
Elliptics: http://www.reverbrain.com/elliptics/
Cocaine: https://github.com/cocaine/cocaine-core
Google group: https://groups.google.com/forum/?fromgroups=#!forum/reverbrain