Spider Engine is the file-scanning core of UNIX Spider.  

It's intended to simple be the regex scan/file processing engine of 
various CLI and GUI Spiders that never got written.  Under Solaris/Linux 
that was supposed to by a Gnome Spider.  Under OSX, it is the current 
OSX Spider beta.

Building spider (the truncated version):
  ./configure && make [&& make install]

  Spider doesn't need to be installed. It will run fine from the builddir.
  If you like it will be installed to the tree specified by the --prefix
  arguement to ./configure (the default prefix is /usr/local).

The following dependencies will be automatically detected and the
optional libraries will be used if found.

Compiling Engine requires:

	- OpenSSL (hash functions and crypto)
	- libexpat (reading the SSN area/max group XML)
	- PCRE (regexes)

Optional:
	- libmagic (file type identification)
	- libzzip (ZIP archive handling)
	- libbz (bzip2 handling)
	- libz (gzip handling)

If you leave out libmagic, skiptypes goes from being a type fragment like 
"Office" or "JPEG" to being a file extension, just like Windows. (This is not
true, yet... without libmagic, skiptypes shouldn't work SMN 10/20/08)

Leaving out the various archive handlers causes Engine to process those as a
bit stream.  I.e., you won't see anything.

Engine will look for a spider.conf in one of three places:
	
	~/.spider/spider.conf
	$PWD/spider.conf
	@PREFIX@/spider.conf (Where prefix is the --prefix arg to ./configure)

It'll parse that config file then start running.  You won't get any outward
indication of what it's doing until it finishes unless you specify the -v
(verbose) flag or enable debug output with the --enable-debug ./configure
flag.  

Config items are named to correspond to values used by Windows Spider 2.x.  
An example config is provided.

Items that are bitmasked, like logattributes, are the base10 integer sum of
the config items in config.h:

logattributes 8135

means log path, hash, hits, score, regex, etc.

In use, we generally hand-edit (all this stuff was intended to be managed
by a GUI someday ...) spider.conf to set the start directory, then kick off 
engine.

If you want to add regexes, add them to the bottom of spider.conf:

regex \d{9}
regex \d{3}-\d{3}-\d{3}

...etc...

I'd intended to include a syntax for validators, but none is defined for 
custom regexes.  The reason for this is PCRE is going to require an 
end-of-regex callout: "(?C)" which allows Engine to grab the match and do
something with it.  I figured most users wouldn't get it, Engine could 
insert it, but I also needed a way to indicate which validator to use.

Be aware: just like its Windows counterpart, Engine will trample access 
times as it goes.  If you're using this for incident response, mount the 
filesystem read-only.

Future Directions:

- libpst integration to read Outlook mailboxes
- something that reads PDF files
- smarter custom-regex handling
- all manner of things to bring it in line with Windows Spider.