DimmWitted is a high-performance execution engine for statistical analytics and is the core of DeepDive.To see the type of analytics that is enabled by DimmmWitted, you can find examples in our paper http://arxiv.org/abs/1403.7550.

We first describe how to install DimmWitted.

##Dependencies

DimmWitted is designed to be compiled on Linux or MacOS. Before downloading the system, you need one C++ compiler, and we have successfully compiled DimmWitted using

• clang++ (Apple LLVM version 6.0, clang-600.0.45.3)
• g++ 4.8.2 (On Linux)

Note that DimmWitted needs a compiler with C++0x support. To let DimmWitted know which compiler you are using during compilation, you can use options like

CXX=g++-4.8 ...

##Downloading and Compiling DimmWitted

Now you are ready to install DimmWitted. First, you can check out the most recent version of the code by

git clone https://github.com/HazyResearch/dimmwitted

By default, this will create a folder called dimmwitted, from now on, lets use the name DW_HOME to refer to this folder.

There are two ways that DimmWitted can be used. We write a logsitic regression application with DimmWitted that thats as input the same format as LIBLINEAR and output the same format. DimmWitted can also be used to write different other applications. We describe these two use cases as follows.

###Logistic Regression Application

To compile the logistic regression application, you can compile DimmWitted as

cd DW_HOME
make lr

This produces two binary files, dw-lr-train and dw-lr-test. To train a model, you can try to run

./dw-lr-train -s 0.01 -e 100 -r 0.0001 ./test/a6a  

where ./test/a6a is one example input from LIBLINEAR. -s specifies the stepsize, -e specifies the number of epoches to run, and -r specifies the regularization parameter (l2). This will produces a model file test/a6a.model.

To test this model on test data, you can run

./dw-lr-test ./test/a6a.t ./test/a6a.model ./test/a6a.output

where ./test/a6a.t is one example test input of LIBLINEAR, ./test/a6a.model is the trained model, and ./test/a6a.output is the output that is of the same format as LIBLINEAR. Running this command will give you

#elements=317325; #examples=21341; #n_features_test=124; #n_features_train=123
#n_features=124
Start testing...
| Running on 8 Cores...
[DimmWitted FUNC=1] TIME=0.000622 secs THROUGHPUT=  7.86 GB/sec.
Testing loss=0.33
| Running on 8 Cores...
[DimmWitted FUNC=0] TIME=0.000459 secs THROUGHPUT=  10.6 GB/sec.
Testing acc =0.847
Dumping the result to ./test/a6a.output...

where you can see the testing loss and testing accuracy. Running LIBLINER on the same dataset obtains similar accuracy.

You can find more data at http://ntucsu.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html Currently, this example application only supports binary logistic regression (with label +1 and -1). You can write other applications by yourself, which will be the topic as you keep reading this document.

###DimmWitted as Library

DimmWitted can be used to write different applications, and we include an example one in the repositary. To compile this example, you can type in

cd DW_HOME
make

This will generate a binary file with the name example, to validate whether the installation is successful, you can try to run this binary file by typing in

./example

This should output:

TIME=0.062495001 secs THROUGHPUT=12.208008 GB/sec.
1.2478489    loss=0.50147917
TIME=0.063390002 secs THROUGHPUT=12.035643 GB/sec.
1.2431277    loss=0.50159193
TIME=0.063591003 secs THROUGHPUT=11.9976 GB/sec.
1.2588885    loss=0.50123058
TIME=0.062996998 secs THROUGHPUT=12.110727 GB/sec.
1.2576657    loss=0.50125708
TIME=0.062763996 secs THROUGHPUT=12.155686 GB/sec.
SUM OF MODEL (Should be ~1.3-1.4): 1.2576657

This binary contains one example of using DimmWitted to train a logistic regression model on a synthetic data set. To check whether DimmWitted works properly on your machine for this application, you should see the last line to be similar to

SUM OF MODEL (Should be ~1.3-1.4): 1.2576657

Note that the number 1.26 might vary, but it should not be too far away from 1.3-1.4.

##Testing

DimmWitted contains a set of unit tests to better make sure it works properly on your machine. To run our suite of test cases, you first need to compile googletest by typing in

make test_dep

And then to run test case, you can type in

make runtest

Please allow couple minutes for the test to run, and ideally you should see

...
[----------] Global test environment tear-down
[==========] 12 tests from 2 test cases ran. (91775 ms total)
[  PASSED  ] 12 tests.

##What's Next?

Now you have successfully installed DimmWitted. The next step is to learn how to write an application inside DimmWitted. You can find examples in the examples folder and the walkthrough here.