==========================================================================
Maven Cross-Compiler
==========================================================================
# Author  : Yunsup Lee, Christopher Batten, Rimas Avizienis, Chris Celio
# Date    : August 10, 2009
# Version : (under version control)

This is the maven C and C++ cross-compiler. It includes a toplevel
configure script and Makefile which drives the build process. The basic
build steps are as follows:

 % mkdir build
 % cd build
 % ../configure --enable-stow
 % make
 % make install

--------------------------------------------------------------------------
Combined Source Tree Build
--------------------------------------------------------------------------

To build a full cross-compiler from scratch we need the following
packages:

 - 'binutils'      : Assembler, linker, etc
 - 'newlib'        : Simple standard C library
 - 'sim'           : ISA simulators (usually part of gdb)
 - 'gmp'           : GNU Multiple Precision Arithmetic Library
 - 'mpfr'          : GNU Multiple Precision Floating-Point Library
 - 'gcc-core'      : Core of gcc including standard c compiler
 - 'gcc-g++'       : C++ compiler
 - 'gcc-testsuite' : testsuite for all of gcc

We will use a combined source tree build. This means that the various
packages are copied as subdirectories into the toplevel source directory
which comes with gcc. The gcc build system is setup such that it will
detect these subdirectories and take care of all the bootstrapping. This
greatly simplifies the cross-compilation process. Unfortunately, the
public releases of these packages do not always work together in such a
way as to make a combined source tree build possible. So instead we will
fetch the source for the binutils, newlib, and sim packages directly
from the sourceware CVS repository. gcc usually tests against the
sourceware CVS repository, and at least those three packages will be
consistent, which increases the changes that a combined source tree
build will work. We fetch gmp and mpfr from the public releases, and the
various gcc subpackages from the standard gnu site. See later in this
'README' for information on how to upgrade the upstream branch.

--------------------------------------------------------------------------
Maven Customizations
--------------------------------------------------------------------------

We have modified various configure and build scripts to enable a new
target alias called 'maven' which expands to 'mipsmaven-ucb-elf' through
a modified 'scripts/config.sub' and 'src/config.sub'. Currently this
produces big endian code. To change maven to use little endian by
default simply map 'maven' to 'mipsmavenel-ucb-elf' in the two
config.sub files.

If you want to eventually add another target and need to edit any of the
'configure.ac', 'configure.in', 'Makefile.am' files then you will need
to make sure you are using the right version of the autoconf tools.
Currently it seems that gcc wants you to use exactly autoconf-2.59 and
automake-1.9.6. The makefile dependency tracking doesn't seem to work to
automatically rerun the autoconf tools, so you will need to change into
the directory with the modified 'configure.ac', 'configure.in', or
'Makefile.am' and run the 'autoreconf' command. This will run
'autoconf', 'automake', 'autoheader', etc as necessary.

We have also made some changes in binutils, newlib, etc to support the
new target. The following are they key files which we have duplicated
for the maven target and thus this is where the majority of the changes
are located:

 - 'src/include/opcode/mips-maven.h'

 - 'src/opcodes/mips-maven-dis.c'
 - 'src/opcodes/mips-maven-opc.c'
 - 'src/opcodes/mips16-maven-opc.c'

 - 'src/gas/config/itbl-mips-maven.h'
 - 'src/gas/config/tc-mips-maven.h'
 - 'src/gas/config/tc-mips-maven.c'

 - 'src/gcc/config/mips/mips-maven.h'
 - 'src/gcc/config/mips/mips-maven.c'
 - 'src/gcc/config/mips/mips-maven.md'
 - 'src/gcc/config/mips/constraints-maven.md'
 - 'src/gcc/config/mips/predicates-maven.md'
 - 'src/gcc/config/mips/t-maven'

 - 'src/libgloss/maven/configure.ac'
 - 'src/libgloss/maven/Makefile.in'
 - 'src/libgloss/maven/crt1.S'
 - 'src/libgloss/maven/syscalls.c'
 - 'src/libgloss/maven/maven.ld'
 - 'src/libgloss/maven/machine/syscfg.h'
 - 'src/libgloss/maven/machine/*'

All maven specific code for newlib and libstdc++ (or at least as much as
possible) should be kept in the libgloss maven subdirectory.
Miscellaneous files like 'crt1.S' (C/C++ startup file), 'syscalls.c'
(implementation of narrow newlib OS interface), and 'maven.ld' (default
maven linker script) are kept at the top-level of the maven libgloss
maven subdirectory. The rest of the newlib and libstdc++ customizations
should be in the machine subdirectory. These customized headers can then
be referenced throughout the gcc build process (and by user programs) as
'<machine/header.h>'. One particularly important header is 'syscfg.h'
(ie. '<machine/syscfg.h>'). This header file should include all global
preprocessor defines which are specific to the maven setup. Examples
include maven syscall numbers, coprocessor 0 register numbers, thread
statck sizes, clock frequency, etc. This header should be manually kept
in sync with the 'syscfg.h' header in the simulator directories. All of
the headers in the 'libgloss/maven/machine' are installed into
'${sysroot}/include/machine'.

To add some maven specific code to newlib/libstdc++ we simply add a new
'#include <machine/header.h>' somewhere in 'xcc/src/newlib/libc/include'
and then we provide a default "place-holder" header in
'xcc/src/newlib/libc/include/machine'. We put our customized machine
header in 'xcc/src/libgloss/maven/machine' and then when libgloss is
installed it will overwrite the default header. We should use this
technique as much as possibly since it cleanly separates the maven
specific code into one place (namely libgloss).

The linker script (ie. 'maven.ld') is installed into '${sysroot}/lib'.
'mips-maven.h' has been modified to use the 'maven.ld' linker script by
default. You can still specify a different linker script on the gcc
command line with the '-T' option. This means that the actual linker in
binutils usees the basic mips elf emulation and doesn't really do
anything special for maven (and the default 'ld' linker script will not
work).

--------------------------------------------------------------------------
Testsuite
--------------------------------------------------------------------------

You can run all of the testsuites on an ISA simulator using the 'check'
and 'longcheck' targets like this:

 % make check
 % make longcheck

The basic 'check' target runs the tests located in the 'test'
subdirectory. Each test "module" in the test subdirectory runs a tool
and then tries to run the output on the simulator. For example, the
following does a clean build of just binutils/gas, runs the unit tests
which try and assemble some source files, then builds gcc, runs the unit
tests which try and compile some C and C++ files, and then finally
builds all the tools and tries and runs all the previously compiled
tests on the ISA simulator.

 % mkdir build
 % cd build
 % ../configure

 % make all-gas
 % make check-gas-compile

 % make all-gcc
 % make check-gcc-compile
 % make check-g++-compile

 % make
 % make check

These simple tests use small little wrappers around the actual programs.
The test wrappers are actually shell scripts which call each program
with the right arguments. For example, the 'maven-gcc' test wrapper will
correctly add the right include and library arguments to allow gcc to
compile and link programs. This makes it easy to quickly try a program
(without installing) during an edit-compile-debug loop like this:

 % make maven-gcc && ./maven-gcc ../test/gcc/test-gcc-simple.c

The 'longcheck' target will run the full testsuite which comes with gcc.
This includes tens of thousands of tests and can take quite a while to
run, which is why we have included the quicker basic 'check' target. The
'longcheck' target will also run the 'test-summary.rb' script in the
scripts subdirectory which summarizes the test results and all of the
'.sum' files are collected in a test-summary subdirectory for easy
archiving.

--------------------------------------------------------------------------
Upgrading Upstream Branch
--------------------------------------------------------------------------

The git repository is setup to make upgrading the cross-compiler to a
new version of gcc (or binutils, newlib, etc) relatively simple. The
process takes quite a while though because we need to download 500MB of
source files, and build/test the compiler four times. I used the
following steps to upgrade from gcc-4.4.0 to gcc-4.4.1.

First do a clean clone of the cross-compiler. We are going to completely
blow away the gcc source directory so we can fetch pristine upgraded
sources, but this means that any changes in your working directory will
be lost - so a clean clone is probably best.

 % XCCDIR=$HOME/tmp/misc/xcc
 % mkdir -p $XCCDIR
 % cd $XCCDIR
 % git clone $REMOTE_GITREPOS/projects/maven/sys/xcc.git .

The first thing we need to do is generate test results before we do the
upgrade so we can compare and make sure that the upgrade doesn't break
anything. If you already have recent test results you can skip this
step.

 % cd $XCCDIR
 % mkdir build-master-pre
 % cd build-master-pre
 % ../configure
 % make
 % make check
 % make longcheck
 % mv test-results-* $XCCDIR/test-results-master-pre

The 'xcc' repo contains a remote upstream branch which we will use to
mirror the original vendor sources. So now we need to create a local
branch (tracking the remote branch) and check it out.

 % cd $XCCDIR
 % git branch upstream origin/upstream
 % git checkout upstream

Now that we are on the upstream branch we should go ahead and generate
test results for this as well so we can again compare and make sure the
upgrade doesn't break anything. If you already have test results from
the upstream branch you can skip this step. Note that we are building
the raw gcc sources with none of the extra maven build scripts. We grab
the 'longcheck-summary.rb' ruby script out of the master branch to help
generate the test results.

 % cd $XCCDIR
 % mkdir build-upstream-pre
 % cd build-upstream-pre
 % ../src/configure --target=mipsisa32r2el --disable-multilib --with-newlib
 % make
 % make -k check RUNTESTFLAGS="--target_board=mips-sim"
 % mkdir test-results-upstream-pre
 % find . -name "*.sum" -exec cp {} test-results-upstream-pre \;
 % find . -name "*.log" -exec cp {} test-results-upstream-pre \;
 % git show master:scripts/longcheck-summary.rb > longcheck-summary.rb
 % chmod a+x longcheck-summary
 % ./longcheck-summary.rb . > test-results-upstream-pre/longcheck-summary.txt
 % mv test-results-upstream-pre $XCCDIR

Now that we have the "pre" test results we are ready to do the upgrade
and generate the "post" test results. The 'upgrade.sh' script in the
'scripts' subdirectory simplifies fetching and setting up the upgraded
combined source tree build. You will need to update the version numbers
for each package which are hard coded into that script. Notice that we
completely remove the 'src' directory before fetching the new versions.
If the new version has removed files then they will be correctly removed
in our new 'src' directory.

 % cd $XCCDIR
 % sed -i.bak 's/gcc_ver=4.4.0/gcc_ver=4.4.1/' scripts/upgrade.sh
 % rm -rf scripts/upgrade.sh.bak
 % rm -rf src
 % ./scripts/upgrade.sh src

Now that we have the upgraded source files we rebuild and rerun the
testsuite to make sure that the versions of binutils, newlib, gcc, mpfr,
and gmp are all correctly working together.

 % cd $XCCDIR
 % mkdir build-upstream-post
 % cd build-upstream-post
 % ../src/configure --target=mipsisa32r2el --disable-multilib --with-newlib
 % make
 % make -k check RUNTESTFLAGS="--target_board=mips-sim"
 % mkdir test-results-upstream-post
 % find . -name "*.sum" -exec cp {} test-results-upstream-post \;
 % find . -name "*.log" -exec cp {} test-results-upstream-post \;
 % git show master:scripts/longcheck-summary.rb > longcheck-summary.rb
 % chmod a+x longcheck-summary
 % ./longcheck-summary.rb . > test-results-upstream-post/longcheck-summary.txt
 % mv test-results-upstream-post $XCCDIR

Compare the test results in 'test-results-upstream-pre' with those in
'test-results-upstream-post' and make sure everything looks good.
Assuming there are no issues we can to commit the upgrade to the
upstream branch.

 % cd $XCCDIR
 % git add .
 % git commit -a -m "Upgrade to gcc-4.4.1"

By using both 'git add' and 'git commit -a' we make sure that we commit
all new files as well as files which have been removed in the upgrade.
Now merge the upstream branch into the master branch.

 % cd $XCCDIR
 % git checkout master
 % git merge upstream

Hopefully there are no conflicts but if they are fix them and commit the
merge. Now the final step is to build the maven cross-compiler and
compare the "post" results to the "pre" results.

 % cd $XCCDIR
 % mkdir build-master-post
 % cd build-master-post
 % ../configure
 % make
 % make check
 % make longcheck
 % mv test-results-* $XCCDIR/test-results-master-post

Compare the test results in 'test-results-master-pre' with those in
'test-results-master-post' and if everything looks good then push the
changes to the centralized git repos.

 % git push

You should now have four test results directories which you might want
to save somewhere to help in future upgrades.

 % cd $HOME/place/to/save/results
 % VER=`$XCCDIR/scripts/vcs-version.sh`
 % mkdir test-results-$VER
 % cd test-results-$VER
 % mv $XCCDIR/test-results-* .

--------------------------------------------------------------------------
Test Results
--------------------------------------------------------------------------

This lists the summary of unexpected failures generated by running the
longcheck testsuite.

                          bin gas  g++  gcc ld lc++

 upstream-0-g049c9b5     :  2   0    2   30 35  36  gcc-4.4.0
 upstream-1-gbdb290b     :  2   0    2   30 35  36  gcc-4.4.1

 master-0.0-008-gac0116a :  2  34    2   32 36  36  maven on gdb mips sim
 master-0.0-024-gab4824a :  1  34   19  239 36   -  maven-isa-run
 master-0.0-031-gaa28a35 :  1  34 1709 9115 36   -  maven.ld not working
 master-0.0-034-gceb52a2 :  1  34   19  237 36 287  fixed maven.ld
 master-0.0-060-g5ed3864 :  2   1   17  194 36 226  fix malloc issue
 master-0.0-063-gb2c3aef :  2   1    2  176 36  32  fix env, sbrk issues
 master-0.0-075-ga1dc508 :  2 343   14  161 39  38  unified gp/fp
 master-0.0-096-gb25f41e :  2 343    2  126 39  30  fix fp struct args
 master-0.0-133-gfe4bd8b :  2 343    2  120 39  31  remove hi/lo
 master-0.0-168-g89c485d :  4 504    2  126         before upgrade