================================================================================
== Occam Toolchain (BSD)
== Scott Moore (scott@thinkmoore.net)
================================================================================

--------------------------------------------------------------------------------
This package includes OCCAM software, available from
http://code.google.com/p/application-specialization/ .  The following terms
apply to OCCAM:

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
  list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice, this
  list of conditions and the following disclaimer in the documentation and/or
  other materials provided with the distribution.

* Neither the name of SRI International nor the names of its contributors may be
  used to endorse or promote products derived from this software without
  specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
--------------------------------------------------------------------------------

// Installation/Getting Started ////////////////////////////////////////////////

Instructions for installing the occam toolchain and building a bitcode version
of the BSD userland. The following instructions have been tested with FreeBSD
9.0 amd64, though they *should* work with later releases as well.

-- Prerequisites: --------------------------------------------------------------

1. FreeBSD

The ideal way to start is with a clean install of FreeBSD 9.0 with sources
installed. The simplest way to do this is to install from the Release 9.0 iso
image and and on the "Distribution Select" screen select just the following:
  [*] ports   Ports tree
  [*] src     System source code

If you are on an existing system that has either an old version of the source
tree or is missing source, you can follow the instructions in the FreeBSD
handbook section 25.6 to get the correct set. You should use the cvs tag
RELENG_9_0_0_RELEASE or later.

2. Necessary ports

Install the following ports using the BSD port tree: 
  bash subversion python27 protobuf
See the FreeBSD handbook section 5 for instructions. Quick and dirty:

   su -
   cd /usr/ports
   cd shells/bash && make -DBATCH install clean && \
     cd ../../devel/subversion && make -DBATCH install clean && \
     cd ../../devel/protobuf && make -DBATCH install clean

(Python27 is a prerequisite of subversion so that takes care of it)

I prefer bash so I switched my shell:
   chsh -s /usr/local/bin/bash
If you want to stick with csh, just replace bash-isms like export with the
appropriate equivalent.

I also install sudo and set up my account as a sudoer to make installing things
a little easier. You can do this, or replace commands including sudo below.

3. Checkout the BSD branch of SelfContainedPrevirt

   svn co svn+ssh://spartan.csl.sri.com/svn/private/occam/SelfContainedPrevirt/branches/bsd SelfContainedPrevirt
   export OCCAM_SRC=`pwd`/SelfContainedPrevirt

4. LLVM and Clang 3.1

Install llvm and clang version 3.1 (these instructions adapted from
http://llvm.org/docs/GettingStarted.html). First, decide where you want to
install LLVM. If you have root access, you might as well use /usr/local, but any
prefix is ok. I recommend adding this to your shell startup.

  export LLVM_HOME=/usr/local

Get LLVM and Clang release 3.1:

  svn co http://llvm.org/svn/llvm-project/llvm/branches/release_31 llvm
  cd llvm/tools
  svn co http://llvm.org/svn/llvm-project/cfe/branches/release_31 clang
  cd ../projects
  svn co http://llvm.org/svn/llvm-project/compiler-rt/branches/release_31 compiler-rt
  cd ../..

There is a bug in LLVM when linking objects compiled separately from the same
source file into a single module that we need to patch.

  patch -p1 < $OCCAM_SRC/patches/llvm-dwarf-debug.patch

Now finish the build and install:

  cd llvm
  mkdir build
  cd build
  ../configure --prefix=$LLVM_HOME --enable-assertions \
      --enable-targets=host-only --enable-optimized
  gmake
  sudo gmake install

N.B. FreeBSD 9.0 ships with clang 3.0 but does not include the llvm toolchain
itself. In addition to installing llvm, we'll also need to upgrade them to
version 3.1, to match the version of clang compiled by "make buildworld".

-- Installing SelfContainedPrevirt ---------------------------------------------

With prerequisites satisfied, we can install SelfContainedPrevirt. Decide where
you want to install the occam toolchain. I chose my home directory. I recommend
adding this export to your shell startup as well. Note that the python wrappers
will still be installed 

  export OCCAM_HOME=~/occam
  cd $OCCAM_SRC

Edit the Makefile and and uncomment the lines exporting these variables at the 
top of the file. Change the paths to your choices. Now make and install 
SelfContainedPrevirt. There is a dependency issue in the Makefiles I still need 
to fix, so you may need to invoke gmake twice to get it to build...

  gmake ; gmake
  sudo gmake install

If you make modifications to the python scripts and want to install the new ones
without rebuilding the C library, do "sudo gmake install-occam".

Create an alias to the occam toolchain wrapper. For bash, add the following to
.bashrc or .bash_profile:

  alias occam=$OCCAM_HOME/bin/occam

-- Mount the /proc filesystem --------------------------------------------------

Part of the parallel build driver in the toolchain reads from the /proc
filesystem. This isn't mounted by default on FreeBSD, so we'll need to mount it
before using the tool. You can mount it with the following command:

  sudo mount -t procfs proc /proc

Alternatively, you can add this filesystem to fstab.

-- buildworld ------------------------------------------------------------------

To build a lot of useful libraries and executables in a hurry, we'll first run
the bsd "buildworld" upgrade process using the occam toolchain. For details of
the buildworld process, see section 25.7 of the FreeBSD handbook.

First, we need to patch one of the buildworld makefiles to prevent it from
removing the occam toolchain from the path after it compiles fresh versions of
clang and the rest of the BSD toolchain. We'll also need to tell make to use
clang (which our toolchain will masquerade as) by changing /etc/src.conf. While
we're at it we'll turn off some parts of the build to reduce how much time it
takes.

  cd /usr/src
  sudo patch -p1 < $OCCAM_SRC/patches/Makefile.inc1-occambsd-paths.patch
  XXX patch is my paths... need to change based on occam install location
  sudo cp $OCCAM_SRC/patches/src.conf /etc/src.conf
  XXX edit occam_base to point to $OCCAM_HOME

Before starting the build, you can choose to record logs from the occam tool by
setting the following variables:

  export OCCAM_LOGFILE={absolute path to log location}
  export OCCAM_LOGLEVEL={INFO, WARNING, or ERROR}

Now kick off the build. You will want to use -jN for some N, as this process
takes a long time without the occam toolchain, and even longer with (hours).

  mkdir $OBJ_DIR (wherever you want to build everything)
  MAKEOBJDIRPREFIX=$OBJ_DIR ; export MAKEOBJDIRPREFIX
  occam make buildworld

This should go smoothly, but I occasionally see IOErrors related to reading the
proc file system. If this error occurs, it should be safe to restart the build:

  occam make -DNO_CLEAN buildworld

At this point, you have a few options. I followed (and have tested) option 1,
but the others should also work.

Option 1) Actually install your new "world"

  To be really safe, we'll build a new kernel with this source as well. You
  *may* be able to skip this step, along with mergemaster, but I haven't
  tried. For more information on mergemaster, see man mergemaster. "-iU" tries 
  to make this process as automatic as possible...
  
  !!! You'll need non-ssh access to the box for this.

     make buildkernel
     shutdown now                                  (drops into single user mode)
     make installkernel                            (don't use -j)
     mergemaster -p
     occam make installworld                       (don't use -j)
     mergermaster -iU
     shutdown -r now

  When the machine comes back up, you should have a full BSD system with bitcode
  versions of all of the core libraries and executables.

Option 2) Install your new "world" somewhere else, chroot into it, and set up a
development environment.

  Choose somewhere to install everything to $WORLD_DIR.
    
     occam make installworld DESTDIR=$WORLD_DIR
  
  !Note, you cannot (according to the handbook) use -j when using installworld
  chroot into $WORLD_DIR and reinstall LLVM, Clang, and SelfContainedPrevirt

Option 3) Extract all the libraries and store them somewhere else.

  Choose a folder to store all of your libraries, $LIB_DIR, then extract all the
  libraries you build to it.
  
     find /usr/obj/usr/src/lib -name "*.bc.a" -exec cp {} $LIB_DIR \;
  
  You can also extract manifests, partially built applications
  (pre-library linking), bitcode-compiled applications, and native applications
  compiled from the bitcode versions. The manifests may contain absolute paths
  to libraries that may need changing, however.

     find /usr/obj/usr/src/usr.* -name "*.manifest" -exec cp {} $DEST \;
     find /usr/obj/usr/src/usr.* -name "*_main.bc" -exec cp {} $DEST \;
     find /usr/obj/usr/src/usr.* -name "*.bc" -exec cp {} $DEST \;

  When trying to use the occam toolchain against these libraries, you'll need to
  add $LIB_DIR to $OCCAM_LIB_PATH (described more below). When previrtualizing
  or using "occam2 build" with a manifest, you'll need to add $LIB_DIR to the
  "search" entry in the manifest.

// Building applications and libraries /////////////////////////////////////////

Once you have your base system installed and ready to go, you should be ready to
compile and install other libraries or applications.

First, set these following environmental variables (works best in your startup
file):

   CC=clang
   CXX=clang++
   CPP=clang-cpp

You should now be able to compile most well-behaved projects with:

   occam ./configure <options>
   occam make
   occam make install

Occam has a few configuration options, set via environmental variables.

   OCCAM_LOGFILE        Absolute path of file to log to
   OCCAM_LOGLEVEL       Logging detail, may be one of INFO, WARNING, or ERROR
   OCCAM_LIB_PATH       Additional directories to include in the search path for 
   			finding bitcode versions of libraries. Not necessary if
			libraries are installed next to their native versions
			(e.g., /path/to/libs:/morelibs)
   OCCAM_PROTECT_PATH	Set/unset flag. If set, occam will remove all non-occam
   			items from the path and intercept calls to common 
			executables so that they can be logged at level INFO.
			Can be useful for debugging a build that isn't working
			properly.

To compile as much code as possible to bitcode, you should choose any available
configure options for building or using static libraries rather than shared ones.

-- Special notes for apache ----------------------------------------------------

For apache, you'll first need to compile and install libpcre, libapr, and 
libapr-util. The configurations I used for each of these are:

  libpcre: built from pcre-8.31.tar.gz
     occam ./configure --enable-shared=no

  libapr: built from apr-1.4.6.tar.gz
     occam ./configure --enable-shared=no

  libapr-util:
     occam ./configure --with-apr=/usr/local/apr --disable-util-dso --with-crypto

You may be able to drop --with-crypto, which could fix the configure issue for
apache that I describe below. For apache, I used the following configure:

    occam ./configure --prefix=$SOMEPREFIX --enable-modules=none   \
    	  --enable-mods-static=most --enable-static-support        \
	  --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr \
	  --with-pcre=/usr/local

You then need to patch build/config_vars.mk because something in the configure
script seems to be detecting crypto options wrong. I'll debug this, but the
workaround for now is to fix the following variables:

   CRYPT_LIBS = -lcrypt
   SSL_LIBS = -lssl -lcrypto -lcrypt -lpthread

To run apache using command line options (to make it easy to previrtualize),
you'll need to set up a basic ServerRoot directory somewhere with the following
configuration.

   $SERVERROOT
   |- conf
      |- mime.types       (an empty text file)
   |- logs		  (an empty directory)
   |- www
      |- index.html	  (something to serve...)
   |- httpd.conf	  (an empty text file)

The following set of arguments should then work to get apache serving files:

   -d $SERVERROOT
   -f httpd.conf
   -C 'Listen 8181'
   -C 'LogLevel debug'
   -C 'User apache' -C 'Group apache'
   -C 'ServerName localhost'
   -C 'ErrorLog /home/moore/root/error_log'
   -C 'DocumentRoot /home/moore/root/www' 
   -C 'AddType text/plain .html' 
   -k start

// Using manifest files ////////////////////////////////////////////////////////

Once you have a manifest file and a bundle, you can use it to build a final
bitcode version of an application or previrtualize it. To build from the
manifest, just execute:

   occam build $MANIFEST_FILE $NAME_FOR_EXECUTABLE

To previrtualize the application, add a "args": ["arg1", "arg2", ...] entry to
the manifest file and call the previrt tool:

   occam previrt --work-dir=$WORKDIR $MANIFEST_FILE

As before, you can set occam configuration variables to control logging.

// TODO items //////////////////////////////////////////////////////////////////

-- General -------------------------------------------------------------------

[ ] Dead code elimination on all of the repositories to clean up unused passes,
    targets, #ifdef 0-ed regions, etc, etc, etc

-- Toolchain -------------------------------------------------------------------

[ ] Save object files off when bundling. The bundling process uses llvm-ld which
    does not link ELF files into the resulting bitcode. We should save off the
    necessary ELF object files from the inputs or library to link in later at
    build or previrtualization time. (This causes the manifest build for at
    least clang during the buildworld process, and probably others)
[ ] Link in said object bundles during occam-build or occam-previrt

-- Previrt -------------------------------------------------------------------

[ ] Debug previrtualized version of httpd. The previrtualized version of httpd
    does not start correctly. It seems likely this is due to interactions with
    pthreads (possibly some incorrectly eliminated symbols?).
[ ] Figure out how to avoid needing to relink in libc, etc. to get the symbols
    we need for the C runtime. (and C++/pthreads?)
[ ] Add support for better call-graphs. Either improve Gregory's call graph,
    choose a better one that's already in llvm, or port over the DSA call-graph
    stuff in poolalloc.
[ ] Support for previrtualizing un-named functions
[ ] Support for previrtualizing internal function pointers accross module 
    boundaries for call backs.
[ ] To avoid creating multiple clones of the same specialized function, we check
    if it already exists by looking it up in the module by name. This check
    should probably be more robust. This is related to support for un-named
    functions because we need to make sure we're using unique specializations
    for the different functions.
[ ] For ease of understanding, give created globals for strings, etc., names so
    they don't show up as unnamed in the symbol table.
[ ] Extract statistics about the previrtualization process.