#
# Gdev: First-Class GPU Resource Management in the Operating System
# 
# README
#
# Copyright (C) Shinpei Kato
#
# Nagoya University
# Parallel and Distributed Systems Lab (PDSL)
# http://pdsl.jp
#
# University of California, Santa Cruz
# Systems Research Lab (SRL)
# http://systems.soe.ucsc.edu
#
# All Rights Reserved.
#
#
# This project is supported by PathScale Inc and Nouveau Community:
# http://www.pathscale.com/
# http://nouveau.freedesktop.org/
#

Gdev contains open-source device drivers and runtime libraries to 
manage GPUs as first-class computing resources. Currently we support 
NVIDIA's Fermi GPUs, but Gdev is certainly applicable to many compute
devices and accelerators. Gdev provides two types of APIs. "Gdev API"
is a low-level primitive that allows programmers to control the detail
of GPU resource parameters, while Gdev also supports a high-level API,
such as CUDA. Gdev is available for GPGPU and graphics applications -
it is self-contained for GPGPU, but you may need additional packages,
such as OpenGL, LIBDRM, and DDX, for graphics applications.

Follow the instructions below to install Gdev. Some of the installation
stages may require you to install additional tools and packages.
$(TOPDIR) will represent your top working directory, henceforth.

1. Download

cd $(TOPDIR)
git clone git://0x04.net/envytools.git
git clone git://github.com/CS005/gdev.git

2. envytools

envytools is a rich set of open-source tools to compile or decompile
NVIDIA GPU program code, firmware code, macro code, and so on. It is
also used to generate C header files with GPU command definitions.
In addition, envytools document the NVIDIA GPU architecture details,
while are not disclosed to the public so far. If you are interested 
in GPU system software development, this is what you should read!
Please follow the instruction below to install envytools.

cd $(TOPDIR)/envytools
mkdir build
cd build
cmake .. # may require some packages on your distro
make
sudo make install # will install tools to /usr/local/{bin,lib}

3. Device Driver

Gdev disgregates from the device driver. You need to install a native
GPU device driver to use Gdev. Currently there exist two open-source 
device drivers available for NVIDIA GPUs: nouveau and pscnv.
Take either Step 3.1, 3.2, or 3.3 below, depending on your preference.

NOTE: In case that you are using Nouveau and you don't need first-class
resource management in the OS, jump to Step 5.2 (skip Step 3 and 4). 

3.1. When you use a very latest kernel (RECOMMENDED)

cd $(TOPDIR)
git clone --depth 1 git://anongit.freedesktop.org/nouveau/linux-2.6
cd linux-2.6
git remote add nouveau git://anongit.freedesktop.org/nouveau/linux-2.6
git remote update
git checkout -b nouveau-master nouveau/master
patch -p1 < $(TOPDIR)/gdev/mod/linux/patches/gdev-nouveau-X.X.patch 
make oldconfig
make 
sudo make modules_install install
sudo shutdown -r now # will reboot your machine

3.2. When you want to use the nouveau module alone

cd $(TOPDIR)/gdev/mod/linux/nouveau-X.X
make NOUVEAUROOTDIR=. # if make files, try Step 3.3.
sudo make install NOUVEAUROOTDIR=.
cd drivers/gpu/drm/nouveau
sudo sh gdev.sh
sudo shutdown -r now # will reboot your machine

If you are not sure if Nouveau is loaded successfully, do:
modprobe -r nouveau; modprobe nouveau modeset=1 noaccel=0

3.3. When you want to use pscnv

cd $(TOPDIR)/gdev/mod/linux/pscnv
./configure
make
sudo make modules_install
sudo shutdown -r now # will reboot your machine

4. Gdev Module

Gdev is double-edge, i.e., it provides a runtime-unified operating
system approach as well as a typical user-space runtime approach.
The following installation is required only for the former.

cd $(TOPDIR)/gdev/mod
mkdir build
cd build
../configure
make
sudo insmod gdev.ko
sudo sh gdev.sh

5. Gdev Library

Gdev's user-space library provides Gdev API. This API can be used
by either user programs directly or another high-level API library.
For instance, third party's CUDA library can use Gdev API.
If you took Step 4, this library is just a set of wrapper functions
that call the Gdev module functions via ioctl.
Take either Step 5.1 or 5.2 below, depending on your preference.

5.1. When you want to have the runtime in the OS

cd $(TOPDIR)/gdev/lib
mkdir build
cd build
../configure
make
sudo make install
export LD_LIBRARY_PATH="/usr/local/gdev/lib64:$LD_LIBRARY_PATH"
export PATH="/usr/local/gdev/bin:$PATH"

5.2. When you want to have the runtime in the user space

# In case that you use nouveau, you need to install libdrm:
#     cd $(TOPDIR)
#     git clone git://anongit.freedesktop.org/git/mesa/drm/
#     cd drm
#     ./autogen.sh
#     ./configure --enable-nouveau-experimental-api --prefix=/usr/
#     make
#     sudo make install
cd $(TOPDIR)/gdev/lib
mkdir build
cd build
../configure --target=user
EXTRA_CFLAG="-I /usr/include/libdrm" make
sudo make install
export LD_LIBRARY_PATH="/usr/local/gdev/lib64:$LD_LIBRARY_PATH"
export PATH="/usr/local/gdev/bin:$PATH"

6. CUDA Driver API

Gdev currently supports a limited set of CUDA Driver API. We plan to 
support a full set of CUDA Driver API in future work. If you need
CUDA Runtime API, you should use some compiler framework, such as
Ocelot, which can translate CUDA Drier API to Runtime API.

cd $(TOPDIR)/gdev/cuda
mkdir build
cd build
../configure
make
sudo make install

Gdev also supports CUDA in the operating system. You are required to
install "kcuda" module to use this functionality.

cd $(TOPDIR)/gdev/cuda
mkdir kbuild
cd kbuild
../configure --target=kcuda
make
sudo make install

7. CUDA Driver API test (user-space programs)

cd $(TOPDIR)/test/cuda/user/madd
make
./user_test 256 # a[256] + b[256] = c[256]

# In case that you use libdrm in the user space, you need to add the
# following options when compiling your host program:
# "-ldrm -ldrm_nouveau -L /usr/lib"

8. CUDA Driver API test (kernel-space programs)

cd $(TOPDIR)/test/cuda/kernel/memcpy
make
sudo insmod ./kernel_test.ko size=10000 # copy 0x10000 size

NOTE: Please be careful when doing this test as it runs a program
in module_init(). If you run a very long program as it is, you may
crash your system. If you want to run a very long program, you must
provide a proper module implementation, e.g., using kernel threads.

9. Reclock

NVIDIA's graphics cards are set very low clocks by default. To get
performance, you need to reclock your card at the maximum level.
How? Be root first, and then echo 3 to the following file:

echo 3 > /sys/class/drm/card0/device/performance_level

You can downclock your card by echoing 0 to the same file, i.e.,

echo 0 > /sys/class/drm/card0/device/performance_level

There are middleground levels 1 and 2, too. Note that Reclocking
is not completely supported by the open-source solution yet.
There are still some performance levels missing, and hence you may
not get as high performance as the blob. If you really need the same
level of performance as the blob, you can run some long-running
CUDA program with the blob, and do kexec -f your kernel before the
program is finished. Then the clock remains at the maximum level.

10. Benchmarks and Applications

Gdev currently supports CUDA Driver API, while many CUDA programs you
may find today are written using CUDA Runtime API. As mentioned above,
you can use Ocelot to translate CUDA Runtime API to CUDA Driver API,
but we also provide some benchmarks and applications that are written
using CUDA Driver API so that you can quickly start using Gdev.
Try downloading the following:
git@github.com:shinpei0208/gdev-app.git
git@github.com:shinpei0208/gdev-bench.git