This project aims to help novices make sophisticated use of sensors in interactive projects through the application of machine learning. It works on Mac OS X, Windows, and Linux.

Check out our paper and video for detailed descriptions.

The easiest way to use ESP is through the Processing Development Environment (PDE). See the Processing ESP mode for details.

Pre-requisites: you'll need git plus Xcode on Mac OS X, Visual Studio on Windows, and CMake on Linux. To install, first clone this repository, then run the setup script:

git clone --recursive https://github.com/damellis/ESP.git
cd ESP
./setup.sh

This will clone the relevant git submodules and create some symbolic links.

The main application is an openFrameworks-based GUI application named, unsurprisingly, ESP. Below are instructions to run on different platform (we support Mac OS X, Windows, and Linux).

Choose the ESP example you want to run by uncommenting the corresponding line at user.cpp. See below for a list of available examples.

Arduino Project Hub has a more comprehensive tutorial on how to use the software.

Use Xcode to open the project at Xcode/ESP/ESP.xcodeproj. Select either the "ESP Debug" or "ESP Release" scheme (not "openFrameworks").

We also support using CMake on OS X to compile the project:

# Compile openFramework by compiling an emptyExample
xcodebuild -configuration Release -target emptyExample \
  -project "third-party/openFrameworks/scripts/templates/osx/emptyExample.xcodeproj"

# Build ESP
mkdir build
cd build
cmake ..
make -j8

We use CMake on Linux to compile the project. The compilation is a bit more involved, but should be easy to follow:

# Install required package
sudo apt-get -y install doxygen
sudo apt-get -y install cmake
sudo apt-get -y install libblas-dev

# Then build openFrameworks
sudo third-party/openFrameworks/scripts/ci/linux/install.sh
sudo third-party/openFrameworks/scripts/ci/linux/build.sh

# Build and install GRT
cd third-party/grt/build
mkdir -p tmp && cd tmp
cmake .. -DBUILD_EXAMPLES=OFF
make
sudo make install
cd ../../../../

# Build ESP
mkdir build
cd build
cmake ..
make -j4

See this setup guide

Many of these examples expect an Arduino board to be connected to the computer and running an appropriate sketch. Some of the sketches are hosted in this repository as well (see Arduino folder). Some examples are:

• user_audio_beat.cpp: recognizes periodic sounds (e.g. dialtones, bells ringing, whistling) using an FFT and support vector machines algorithm. Works with your computer's built-in microphone.

• user_color_sensor.cpp: detects objects by color using a naive Bayes classifier. Works with either the Adafruit TCS34725 breakout (using the sketch in Arduino/ColorSensor) or the SparkFun ISL29125 breakout (using the sketch in Arduino/ColorSensor_SparkFun_ISL29125). See documentation for the sensors for hookup information.

• user_accelerometer_gesture.cpp: recognizes gestures using a dynamic time warping algorith. Works with either an ADXL335 accelerometer (using the Arduino/ADXL335 sketch) or the built-in accelerometer on an Arduino 101 (using the Arduino/Arduino101_Accelerometer sketch).

• user_accelerometer_poses.cpp: recognizes the orientations of an object using a naive Bayes classifier. Works with accelerometers as for the user_accelerometer_gesture.cpp example.

See the online documentation of the ESP API.

These should be automatically installed by the setup script:

• openFrameworks, a C++ toolkit for creative coding.

• GRT, Gesture Recognition Toolkit, a cross-platform, open-source, C++ machine learning library that has been specifically designed for real-time gesture recognition. Specifically our fork of the GRT repository.

• ofxGrt, an openFrameworks extension for the Gesture Recognition Toolkit (GRT). Specifically our fork of the ofxGrt repository.

• ofxDatGui, a framework for building GUIs in openFrameworks. Specifically our fork of the repository.

• ofxParagraph, for rendering multi-line text.

See LICENSE.txt for licensing information.