huku <huku@grhack.net>

pyrsistence is a WIP Python extension, developed in C, for managing External Memory Data Structures (usually referred to as EMDs). At its current version, pyrsistence supports external memory lists and dictionaries.

External Memory Data Structures are used to store data off-line (think of a graph stored on the disk for example), while allowing for on-line access on the data structure's elements (think of traversing the aforementioned graph as if it's in the main memory). SQL/no-SQL servers are notable examples of programs utilizing EMD related technologies, however pyrsistence is a clean, minimal implementation for home use, that doesn't require any server software (pyrsistence is, in fact, implemented purely using memory-mapped files, a mature facility offered by all modern operating systems).

This extension was specially designed for big data analysis problems that may arise during reverse engineering, however, it may be used for other purposes as well. Halvar Flake describes this class of problems in his DailyDave post entitled "Code analysis and scale".

For more information on EMDs have a look at the references.

The code should compile without errors or warnings on all supported platforms. Both Python 2.7.x and Python 3.x are supported.

Set PYTHON_PREFIX in Makefile.nmake to the directory where you have installed Python 2.7.x or 3.x and then run the following command to build a 32-bit version of pyrsistence.

Z:\pyrsistence>"C:\Program Files (x86)\Microsoft Visual Studio 10.0\vc\vcvarsall.bat"
Z:\pyrsistence>nmake /F Makefile.nmake

Alternatively, run the following to produce a 64-bit version of pyrsistence.

Z:\pyrsistence>"C:\Program Files (x86)\Microsoft Visual Studio 10.0\vc\vcvarsall.bat" amd64
Z:\pyrsistence>nmake /F Makefile.nmake

Depending on your Visual Studio version, you might need to replace the amd64 argument with x86_amd64.

To install pyrsistence issue the following command:

Z:\pyrsistence>nmake /F Makefile.nmake install

Set PYTHON_VERSION in Makefile to the version of Python you would like to use, or leave it blank to use system's default.

Then, run make and make install.

Set PYTHON_VERSION in Makefile to the version of Python you would like to use, or leave it blank to use system's default.

Then, run make and make install.

Below is a list of experimental, precompiled binaries for all supported platforms.

• pyrsistence-win-vs2012-python27.tgz (Microsoft Windows, Visual Studio 2012, Python 2.7)
• pyrsistence-macosx-llvm730-python27.tgz (MacOS X, LLVM 7.3.0, Python 2.7)
• pyrsistence-linux-gcc484-python27.tgz (Linux, GCC 4.8.4, Python 2.7)

For more information on pyrsistence, have a look at the wiki.

Special thanks to the following people for contributing to pyrsistence:

• argp, vats and anestisb for using, testing, reporting and fixing bugs.

• adc for Python 3.x support.

• Halvar's Code analysis and scale
• Algorithms and Data Structures for External Memory
• Basic External Memory Data Structures
• MMap: Fast Billion-Scale Graph Computation on a PC via Memory Mapping
• GraphChi: Large-Scale Graph Computation on Just a PC
• TurboGraph: A Fast Parallel Graph Engine Handling Billion-scale Graphs in a Single PC
• SSDAlloc: Hybrid SSD/RAM Memory Management Made Easy