========================================================================
README for Kinect 3D Video Capture Project version 2.4
Copyright (c) 2010-2012 Oliver Kreylos
========================================================================

Requirements
============

This software requires the Vrui VR toolkit, version 2.6 build 001 or
newer. It also requires libusb version 1.0, and that Vrui was configured
with support for libusb prior to its installation. This software can
optionally create Kinect 3D video streaming plug-ins for Vrui's
collaboration infrastructure. The presence of Vrui's collaboration
infrastructure will be detected automatically during the build process.

Installation Guide
==================

It is recommended to download or move the source packages for Vrui and
the Kinect 3D Video Capture Project into a src directory underneath the
user's home directory. Otherwise, references to ~/src in the following
instructions need to be changed.

0. Install Vrui from ~/src/Vrui-<version>-<build> (see Vrui README file).

0.5. (Optional) Install Vrui's collaboration infrastructure from
     ~/src/CollaborationInfrastructure-<version> (see its README file).

1. Change into ~/src directory and unpack the Kinect 3D Video Capture
   Project tarball:
   > cd ~/src
   > tar xfz <download path>/Kinect-<version>.tar.gz
   - or -
   > tar xf <download path>/Kinect-<version>.tar

2. Change into the Kinect 3D Video Capture Project's base directory:
   > cd Kinect-<version>

3. If the Vrui version installed in step 0 was not 2.6, or Vrui's
   installation directory was changed from the default of ~/Vrui-2.6,
   adapt the makefile using a text editor. Change the value of
   VRUI_MAKEDIR close to the beginning of the file as follows:
   VRUI_MAKEDIR := <Vrui install dir>/share/make
   Where <Vrui install dir> is the installation directory chosen in
   step 0. Use $(HOME) to refer to the user's home directory instead
   of ~.

4. Build the Kinect 3D Video Capture Project:
   > make

5. Install the Kinect 3D Video Capture Project:
   > make install

Use
===

1. Run ./bin/KinectUtil list to list all Kinect devices connected to the
   host computer.

2. Run ./bin/RawKinectViewer <camera index> to intrinsically calibrate
   the <camera index>-th Kinect device on the host's USB bus (with the
   first device having index 0), using a semi-transparent checkerboard
   calibration target (see next section for details). This generates a
   binary IntrinsicParameters-<serial number>.dat file in
   <Vrui install dir>/etc/Kinect-<version>, where <serial number> is the
   unique factory-assigned serial number of the Kinect device as
   displayed in step 1.

Step 2 only has to be performed once for each Kinect device, unless the
resulting calibration proves unsatisfactory. Kinect devices typically
retain their intrinsic calibration over time.

3. Run ./bin/KinectViewer -c <camera index>, where <camera index> is the
   zero-based index of the Kinect device to use if there are multiple
   ones.

Intrinsically Calibrating a Single Kinect
=========================================

Before a Kinect can be used as a 3D camera, its two cameras need to be
calibrated. Calibration will calculate a 4x4 projection matrix to map
the depth camera's image stream into 3D space as a 3D point cloud or
triangulated surface, and a 4x4 projection matrix to map the color
camera's image stream onto that point cloud or surface. After proper
calibration, the virtual 3D objects captured by a Kinect will precisely
match the original real objects in shape and size.

The new intrinsic calibration method uses a semi-transparent
checkerboard calibration target. The idea is that the target looks like
a checkerboard both to the depth and color cameras. The user presents
the calibration target to the Kinect in a sequence of different poses,
and manually matches a 2D distorted grid to the observed calibration
target in the depth and color cameras' views using the RawKinectViewer
utility. After a sufficient number of poses have been captured,
RawKinectViewer calculates the two calibration matrices and stores them
in a uniquely identified file for subsequent retrieval whenever a Kinect
device is activated.

The easiest and most precise way to build a semi-transparent
checkerboard is to print a grid pattern onto a large sheet of paper,
glue the entire sheet of paper onto an IR-transparent glass plate, cut
precisely along all grid lines using a sharp knife and metal ruler, and
then peel off every other paper square and remove any glue residue from
the now transparent tiles. The grid should have an odd number of tiles
in both directions such that all four corner tiles can stay opaque. Mark
the center of the lower-left corner tile with a small dot (big enough to
be visible in the Kinect's color image stream). This mark will be used
to check for grid alignment during calibration.

The number of tiles, and the size of each tile, can be configured in
RawKinectViewer, but the default grid layout has 7x5 tiles of 3.5"x3.5"
each. This leads to an overall target size of 24.5"x17.5", which is just
the right size to cover the Kinect's full viewing range.

This is the calibration procedure for a Kinect:

0. Prepare calibration target

1. Start RawKinectViewer for the Kinect to be calibrated. If the
   calibration target does not have the default layout, specify the
   layout using the -gridSize and -tileSize command line options.

2. Create a "Draw Grids" tool by binding it to five (5) buttons
   (keyboard keys or mouse buttons). Here, we will use the keys "1"
   through "5". This will create two green grids, one in the depth and
   one in the color image stream. The grid can be deformed by grabbing
   any grid intersection using the mouse and the "1" key, or translated
   by grabbing the dot in the center of the grid, or rotated around
   the center by grabbing the dot slightly outside the right edge of
   the grid, again using the "1" key.

3. Place the calibration target in front of the Kinect at the next
   position and orientation. The target should be placed in a range of
   distances, starting from the near depth cutoff to the end of the
   desired tracking range, and in a variety of orientations from
   straight-on to about 45 degree angles. Calibration requires at least
   4 calibration poses, but ideally a larger number to improve
   calibration quality.

4. Collect an average depth frame by turning on the "Average Frames"
   button in the main menu. Wait until the depth image does not change
   anymore.

5. Align both the depth and color grids grids to the observed grids. The
   dot in the center of the lower-left corner tile must roughly coincide
   with the mark in the lower-left corner tile of the calibration target
   to ensure that the grid is not flipped or rotated. The orientation of
   the depth and color stream grids will be very similar.

6. Store the current calibration grids by pressing the "2" key.

7. Turn off the "Average Frames" button in the main menu so that the
   depth image stream updates in real time again.

8. Repeat from step 3 with the next calibration pose.

9. After all calibration poses have been captured, calculate the
   intrinsic calibration by pressing the "4" key. This will print some
   diagnostic information to the terminal, and create the calibration
   file specific to the Kinect device in the Kinect package's
   configuration directory. The files are named
   IntrinsicParameters-<serial number>.dat and contain the two
   calibration matrices in binary format. They cannot be hand-edited.

10. After calibration, use KinectViewer and a 3D measurement tool to
    ensure that the size and shape of the virtual calibration target as
    reconstructed by the Kinect matches the real calibration target. A
    good calibration will create a match to a few percent. Keep in mind
    that measurements inside KinectViewer will be reported in
    centimeters.

Recording 3D Movies
===================

KinectViewer now has a built-in recorder for 3D movies from one or more
Kinect devices. When running a live view (using one or more
-c <Kinect index> and/or -p <server host> <server port> command line
arguments), the live 3D streams can be saved to a set of files by
selecting the "Save Streams" main menu entry. KinectViewer will ask for
a file name prefix, and then save the depth and color streams of all
enabled Kinect devices to files <prefix>-<index>.depth and
<prefix>-<index>.color. It will also record synchronized audio from the
default capture device (selectable from the sound control panel) and
save it to <prefix>.wav.

Previously recorded 3D movies can be played back by running KinectViewer
with one or more -f <file name prefix>-<index> arguments, one per saved
3D stream, and an optional -s <sound file name> argument to play back
synchronized audio.

Merging 3D Facades from Multiple Kinects
========================================

To merge the 3D reconstructions of multiple Kinect cameras, first
intrinsically calibrate each one using RawKinectViewer, as described
above and shown in the video. Then extrinsically calibrate all Kinect
devices with respect to an arbitrarily chosen world coordinate system:

1. Place calibration target into field of view of all Kinect devices to
   be calibrated (use RawKinectViewer on each to check).

2. For each Kinect device:

   2a. Run RawKinectViewer and fit a grid to the calibration target's
       image in the depth stream (it is not necessary to fit to the
       color stream).

   2b. Save and unproject the grid to receive a list of 3D tie points on
       the console.

   2c. Copy the tie points into a file KinectPoints-<serial number>.csv.

3. Create a file TargetPoints.csv, containing the 3D interior corner
   positions of the calibration target, in some arbitrary right-handed
   world coordinate system using an arbitrary unit of measurement, in
   the same order as printed by RawKinectViewer (going from left to
   right, and then bottom to top). See example target file below.

4. For each Kinect device:

   4a. Run the Kinect device's tie points and the target points through
       AlignPoints:
       AlignPoints -OG KinectPoints-<serial number>.csv TargetPoints.csv

   4b. Observe the mismatches between purple camera points and green
       target points in AlignPoints' display. If the discrepancies are
       too large, or there is no fit at all, repeat step two for the
       Kinect device.

   4c. Paste the final best fitting transform displayed by AlignPoints
       (everything after the "Best transformation:" header) into a
       ProjectorTransform-<serial number>.txt file.

5. Run KinectViewer on all Kinect devices. This will show all individual
   3D video streams matching more or less seamlessly, depending on how
   much care was taken during intrinsic and extrinsic calibration. Under
   ideal circumstances, there should be no noticeable mismatches between
   individual streams.

Unlike intrinsic calibration, extrinsic calibration has to be repeated
any time any of the calibrated Kinect devices are moved, even by very
little. It is recommended to rigidly attach all Kinect devices to a
sturdy frame before attempting precise extrinsic calibration in a
production setting. If precision is paramount, it might even be
necessary to repeat calibration periodically if nothing changed, to
account for time-varying deformations inside the devices themselves.

An example TargetPoints.csv file, for a 5x4 calibration grid with a tile
size of 3" (a 5x4 grid has 4x3 interior corners):

0, 0, 0
3, 0, 0
6, 0, 0
9, 0, 0
0, 3, 0
3, 3, 0
6, 3, 0
9, 3, 0
0, 6, 0
3, 6, 0
6, 6, 0
9, 6, 0

The resulting world space will use whatever units were used to define
the target points; in the example above, world space will use inches.