Hemp Weaves Data Into Text
--------------------------

Hemp is a C library implementing a fast, flexible and extensible framework
for parsing, representing, manipulating and rendering document-based
languages.  These include traditional template languages (TT2, TT3, etc), 
mini-markup languages (Pod, Markdown, Textile, etc), declarative document
langauges (HAML, SASS, etc.) and data languages (JSON, YAML, etc).  (Note
that this is speaking hypothetically - Hemp *can* support these kinds of 
languages, but doesn't necessarily do so yet).

Hemp exists to provide an implementation platform for TT3.  This is the 
long-awaited version 3 of the Perl Template Toolkit (TT).  Version 2 (TT2)
is widely used in the Perl community, and as one of the more "mature"
templating solutions, it has influenced (for better or worse) numerous other 
templating languages written in Perl and other programming languages.

A working prototype of TT3 exists in Perl (Template::TT3 - see 
http://github.com/abw/Template-TT3).  Hemp is a rough translation to C of the 
concepts implemented therein.  Hemp will also provide support for version 2 
of the Template Toolkit language (TT2) for the sake of backward compatibility.  

This is made possible by the fact that the Hemp document processing framework 
simultaneously supports any number of different document languages.  Hemp 
supports dynamic loading of different template, markup and data languages,
providing great flexibility and infinite extensibility.  Hemp's document
parsing tools make it easy to implement document processing languages, or 
compose, configure and extend features from existing languages to tailor
them for your specific needs.  The benefit of using Hemp is (or will be)
that you only need to write the code to parse a language into an internal
representation that Hemp can understand (and Hemp provides numerous tools
to help do this).  Once a document has been converted to Hemp's internal
document model it can then be inspected, manipulated and processed using
all of the standard Hemp tools and techniques.  This allows you to freely
intermix different languages in a project.  e.g. using JSON to define all 
your static data, Markdown for all your static content, and a mixture of 
older TT2 templates and newer TT3 templates to render dynamic HTML pages 
for your web applications.  Hemp can handle all these different languages,
all at the same time.  You can, for example, 'include' a TT2 header into a 
TT3 page template, while loading in some data from a JSON file.  Hemp 
understands these different formats, and ultimately, treats them equally.

Basic library wrappers are provided for using Hemp from Perl.  Writing 
wrappers for other languages (Python, Ruby, PHP, etc) should be a relatively 
simple task for anyone with experience in writing interfaces to C libraries 
in those languages (that's not me, so contributions are most welcome).

Hemp is still very much in development and should be considered a working
proof of concept at this stage.  Things are starting to fall into shape,
to the point where it is now possible to process some fairly basic templates.
However, nothing is guaranteed to work and you SHOULD NOT use Hemp for 
anything even remotely important.

Latest Updates: 

- Compiling cleanly on 64 bit OSX (Snow Leopard) and 32 bit Linux (Ubuntu
  10.04 LTS Lucid Lynx).  Most tests pass (tag, uri and 00_load fail at
  the time of writing, but that's ok - they're reminders of TODO items
  or things I've broken that need to be fixed).

- Most of the expression operators are implemented and I've started adding 
  some high-level commands like 'if' (see tests/scripts/command/if).  

- Dynamic loading of languages, codecs, etc., is also working so that Hemp 
  is now extensible at runtime.

- TT3 is the language I'm concentrating on.

- JSON language works. It's little more than a re-configuration of existing 
  template elements to define the basic syntax for JSON.  It's a nice little
  proof of concept that we can treat data languages more-or-less the same as
  template languages.  For example, there's little difference between the
  following, except for a bit of syntax:

  # JSON                # Perl/TT2/TT3         # TT2/3 ('>' is optional)
  { language: 'json' } 	{ language => 'JSON' } { language = 'json' }

  I think this is useful for 2 reasons (talking out loud here, feel free to
  skip).  Firstly, it allows us to load, parse and instantiate Hemp data from
  JSON files (or, theoretically, any other data language) with minimal effort.
  We already have support for data defined in TT2/3 style, and hemp allows 
  us to redefine operator tokens (e.g. ':' instead of '=>'), so we effectively
  get JSON support for free.  We could add a template directive to load data
  in from an external file, e.g.

    [% load /path/to/data.json %]  # load data definitions

  Or perhaps from the command line, we could load up some data and then 
  process a template.

    $ hemp data /path/to/data.json file /path/to/template.tt3

  We can also support configuration of Hemp itself from JSON files (or any
  other data format that hemp can be programmed to read).  e.g.

    $ hemp -f /path/to/hemp/config.json ...

  See modules/language/json.c for the implementation and tests/language/hemp
  for tests.

- I've also been working on Hemp's internal test language.  It's based on the
  mini-markup language that I've been using in TT3/3 for as long as I can
  remember.  It looks something like this:

  -- test This is the name of the test
  This is some template markup, [% msg %]
  -- expect
  This is some template markup, Hello World!

  The 'test' section defines a fragment of template markup which is processed
  and compared to the expect output in the 'expect' section.  It can also be
  used to test error reporting.

  -- test This test is expected to fail
  Hello [% 'worl
  -- error
  Error at line 1 of test... etc...

  This test language is typical of the kind of mini-languages that people
  often knock up to perform a specific task.  Implementing it in Hemp is
  a good exercise in eating your own dog food.


-- Andy Wardley (abw)
   December 2010