light-weight implementation of distributed data processing ala apache spark.

sprk_ctx_t *context = sprk_ctx_new();
sprk_dataset_t *dataset = sprk_dataset_new(context, "pathlist");

sprk_dataset_mul(dataset, 2);
sprk_dataset_checkpoint(dataset);  // evaluate and cache dataset
double sum = sprk_dataset_sum(dataset); // only has to perform sum
double sum2 = sprk_dataset_sum(dataset); // only has to perform sum

sprk_dataset_destroy(dataset);
sprk_ctx_destroy(context);

sprk_ctx_t *context = sprk_ctx_new();

• get list of executor hostnames/ports
• create mapping of executor-id -> hostname:port
• maintain this list as clients join cluster and disconnect
• this context will manage all communication to executors
  • e.g. assign_block(block_t) -> executor-id

sprk_dataset_t *dataset = sprk_dataset_new(context, "pathlist");

• create a dataset-id
• parse pathlist into paths
• break paths into blocks (id, path, offset, length)

sprk_dataset_mul(dataset, 2);

• append mul(2) transformation to all blocks in dataset

sprk_dataset_checkpoint(dataset);

• causes a resolution of transformations in executors
• blocking
• broadcast (block-id, new-block-id, transformations)
  • only broadcast blocks which we know have been assigned
• executors are subscribed to block-ids they own
• executors accept the transformation work and send an ack
• context receives acks that work was received
• for block-ids that were not acked, then context.assign_block(new_block)
• context.assign_block() all blocks which have never been assigned

double sum = sprk_dataset_sum(dataset);

• causes sum compututations in executors
• blocking
• broadcast (dataset-id, sum())
• executors ack when they accept the work
• for block-ids that were not acked, then context.assign_block_with_computation(block, sum())
• subscribe to (block-id, sum(), result) messages
• aggregate as results come in
• when all blocks accounted for, return.