$Id: README,v 1.17 2009/04/05 13:25:43 jt Exp $

Abstract:
================================================================================
The Cygwin rebase distribution contains four utilities, rebase, rebaseall,
peflags, and peflagsall.  The first utility is modeled after Microsoft's SDK
rebase.  However, instead of linking rebase against Microsoft's imagehlp
library, it is linked against Ralf Habacker's imagehelper library.

The following are the advantages of using Ralf's library:

    1. rebase can be a Cygwin application.  If rebase is linked against
       Microsoft's imagehlp library, then it must be a Mingw application.
       This is because imagehlp is dependent on Microsoft's C runtime and
       therefore cannot be used in an application that is dependent on
       another C runtime (i.e., Cygwin).

    2. With recent improvements, libimagehelper supports operations on
       64bit objects, and can itself be compiled as either a 32bit or
       64bit library.

    3. rebase can be used on Windows Me.  For some reason, Microsoft's
       imagehlp.dll does not function properly on this platform.

    4. Ralf's imagehelper library also provides FixImage() which fixes
       bad relocations in DLLs that can be caused by stripping.

I would like to thank Ralf Habacker for providing the imagehelper
library.  This library has enabled me to create a rebase utility that
is usable by all Cygwin user (regardless of platform) on all DLLs
(regardless of stripping).

I would like to thank Chuck Wilson for providing peflags and peflagsall.  This
enables Cygwin to take advantage of the Address Space Layout Randomization
facility on Windows Vista and later.


Background
================================================================================
The rebaseall utility is a convenient way for users that suffer from the
Cygwin rebase problem to rebase their entire system (i.e., all of their
DLLs).  The following is a list of known Cygwin applications that are
affected by the rebase problem:

    Apache
    Perl
    Python

The rebase problem is due to fork() failing when it is unable to load
DLLs in the child at the same address as in the parent.  This is caused
by DLLs which have conflicting base addresses.  An error message like
the following will be displayed when the problem is triggered:

    C:\cygwin\bin\python.exe: *** unable to remap C:\cygwin\bin\cygssl.dll to same address as parent(0xDF0000) != 0xE00000

Note that rebaseall is only a stop-gap measure.  Eventually the rebase
functionality will be added to Cygwin's setup.exe, so that rebasing will
happen automatically.

On Vista and newer, it is possible to employ the Address Space Layout
Randomization facility to help solve the image load address clashing
problem:

    Since Windows relies on relocations instead of position
    independent code, a DLL must be loaded at the same address
    in each process that uses it to allow the physical memory
    used by the DLL to be shared. To facilitate this behaviour,
    a global bitmap called _MiImageBitMap is used to represent
    the address space from 0x50000000 to 0x78000000. The bitmap
    is 0x2800 bits in length with each bit representing 64KB
    of memory. As each DLL is loaded, its position is recorded
    by setting the appropriate bits in the bitmap to mark the
    memory where the DLL is being mapped. When the same DLL
    is loaded in another process, its section object is reused
    and it is mapped at the same virtual addresses.
    http://taossa.com.nyud.net:8080/archive/bh08sotirovdowd.pdf

Thus, IF all cygwin dlls (except cygwin1.dll) -- or, at least the
ones for which you often get the *** unable to remap *** error --
are marked ASLR-compatible, then the Windows runtime loader can be
coerced into ensuring that the DLLs are loaded at the same memory
location for all concurrent processes. After using the peflagsall
utility to mark the dlls appropriately, you may need to reboot to
cause Windows to take notice.


Requirements:
================================================================================
The following packages or later are required to build and/or execute Cygwin
rebase/peflags, and/or rebaseall/peflagsall:

    binutils             2.20.51-2
    cygwin               1.7.9-1
    dash                 0.5.6.1-2
    gcc4                 4.3.4-4
    coreutils            8.10-1
    grep                 2.6.3-1
    gzip                 1.4-1
    sed                  4.2.1-1
    w32api               3.17-2

Install:
================================================================================
Cygwin rebase does not require any special installation procedures.

If the database is used, Cygwin and MSYS rebase will expect to find the
databases:
    /etc/rebase.db.i386
    /etc/rebase.db.x86_64
For MinGW, the database location is deduced as
    <location-of-rebase.exe>/../etc/rebase.db.i386
    <location-of-rebase.exe>/../etc/rebase.db.x86_64

rebaseall, by default, uses the database.


Usage:
================================================================================
Use the following procedure to rebase your entire system:

    1. shutdown all Cygwin processes and services
    2. start ash or dash (do not use bash or a terminal emulator like rxvt
       or mintty). The easiest way to do this is to use Windows Explorer
       and navigate to the top level of your cygwin installation, and
       double-click ash.exe or dash.exe in the bin/ directory.
    3. execute /bin/rebaseall (in the ash/dash window)

If you get any errors due to DLLs being in-use or read-only, then take the
appropriate action and rerun rebaseall.  Otherwise, you run the risk of fork()
failing.

Use a similar procedure to set the peflags for all cygwin executables and
dlls on your entire system:

    1. shutdown all Cygwin processes and services
    2. start ash or dash (do not use bash or a terminal emulator like rxvt
       or mintty). The easiest way to do this is to use Windows Explorer
       and navigate to the top level of your cygwin installation, and
       double-click ash.exe or dash.exe in the bin/ directory.
    3. execute /bin/peflagsall (in the ash window)

Both procedures may be combined into a single, 4-step procedure if desired.

Note that by default, peflagsall will set the tsaware flag on all executables
(ash.exe and peflags.exe themselves excluded), and will remove the dynamicbase
flag on all 'dll', 'so' and 'oct' files (cygwin1.dll and cyglsa64.dll excluded).

If you are used to using the -s option with rebaseall, to add additional
suffixes to the rebase list (such as .oct), be warned that peflags behaves
differently. If you specify custom suffixes, you must restate the default
ones, AND explicitly specify what action to perform; peflagsall disables
all default behavior when custom settings are used.  For instance:

    peflagsall -s oct -s dll -s so -s exe   -d 1 -t 1
               ^^^^^^ ^^^^^^^^^^^^^^^^^^^   ^^^^^^^^^
               custom   restate default      restate
               suffix      suffixes      default actions

Whereas rebaseall only has a single action, and new suffixes are added
to the existing default set of suffixes. See 'Command line syntax', in 
the next section.

Command line syntax
================================================================================

rebaseall
--------------------------------------------------------------------------------
The following is the rebaseall command line syntax:

    rebaseall [-b BaseAddress] [-o Offset] [-s DllSuffix] [-T FileList | -] [-4|-8] [-p] [-t] [-v]

where:

    -b => base address used by rebase (default: 0x70000000)
    -o => offset between each DLL rebased (default: 0x10000 MinGW/MSYS, 0x0 Cygwin)
    -p => skip test for running ash or dash only.  This option is supposed to
          be used by Cygwin's setup tool.  Only use it if you know what you're
	  doing!
    -s => specify DLL suffix, use multiple if necessary (default: dll, so, oct)
    -T => specify filelist (or stdin) to list additional files
    -4 => operate only on 32bit objects (ignore 64bit objects) (*)
    -8 => operate only on 64bit objects (ignore 32bit objects) (*)
    -t => change modification timestamp of successfully rebased files
    -v => verbose (default: off)

(*) -4 and -8 are mutually exclusive. -4 is the default if rebaseall is
    executed in a 32bit environment; -8 is the default if rebaseall is executed
    64bit environment.

rebaseall now uses the database functionality provided by rebase. See the
description below.


peflagsall
--------------------------------------------------------------------------------
The following is the peflagsall command line syntax:

    peflagsall [-p extra_args] [-d bool] [-t bool] [-s suffix] [-T FileList | -] [-vnh]

    When invoked with no arguments, peflagsall modifies every cygwin exe|dll|so|oct
    on the system: .exe files have their tsaware flag set, while .dll, .so and .oct
    files have their dynamicbase flag removed. However, if any of [-d|-t|-s] are
    specified then ONLY the actions so specified will occur.
       -p extra_args   pass extra_args to peflags.exe
       -d bool         set the dynamicbase flag to 'bool' on all specified files
       -t bool         set the tsaware flag to 'bool' on all specified files
       -s suffix       search for all files with the specified suffix(es)
                         default: 'exe|dll|so|oct'
       -T FileList     in addition to files located via the normal search and
                       suffix list, also operate on files listed in FileList
                       May be '-' which indicates stdin
       -n              do not modify any files, but display the peflags commands
       -k              keep all temporary files
       -v              verbose mode
       -h              show this help
    bool may be '0', '1', 'true', 'false', 'yes', or 'no'
    Note: peflagsall will NOT set the dynamicbase flag on executables, nor will
          it set the tsaware flag on dlls. If you must do this, use peflags itself

rebase
--------------------------------------------------------------------------------
The following is the rebase command line syntax:

    rebase [OPTIONS] [FILE]...
    Rebase PE files, usually DLLs, to a specified address or address range.
    
      -4, --32                Only rebase 32 bit DLLs. (*)
      -8, --64                Only rebase 64 bit DLLs. (*)
      -b, --base=BASEADDRESS  Specifies the base address at which to start rebasing.
      -s, --database          Utilize the rebase database to find unused memory
                              slots to rebase the files on the command line to.
                              (Implies -d).
                              If -b is given, too, the database gets recreated.
      -O, --oblivious         Do not change any files already in the database
                              and do not record any changes to the database.
                              (Implies -s).
      -i, --info              Rather then rebasing, just print the current base
                              address and size of the files.  With -s, use the
                              database.  The files are ordered by base address.
                              A '*' at the end of the line is printed if a
                              collisions with an adjacent file is detected.
    
      One of the options -b, -s, -O or -i is mandatory.  If no rebase database
      exists yet, -b is required together with -s.
    
      -d, --down              Treat the BaseAddress as upper ceiling and rebase
                              files top-down from there.  Without this option the
                              files are rebased from BaseAddress bottom-up.
                              With the -s option, this option is implicitly set.
      -n, --no-dynamicbase    Remove PE dynamicbase flag from rebased DLLs, if set.
      -o, --offset=OFFSET     Specify an additional offset between adjacent DLLs
                              when rebasing.  Default is no offset.
      -t, --touch             Use this option to make sure the file's modification
                              time is bumped if it has been successfully rebased.
			      Usually rebase does not change the file's time.
      -T, --filelist=FILE     Also rebase the files specified in FILE.  The format
                              of FILE is one DLL per line.
      -q, --quiet             Be quiet about non-critical issues.
      -v, --verbose           Print some debug output.
      -V, --version           Print version info and exit.
      -h, --help, --usage     This help.

(*) -4 and -8 are mutually exclusive. -4 is the default if rebase.exe itself
    was compiled as a 32bit application; -8 is the default if rebase.exe is
    a 64bit application.

There are actually two separate database files, for 32bit and 64bit objects.
On Cygwin and MSYS, these are:
    /etc/rebase.db.i386
    /etc/rebase.db.x86_64
For MinGW, the database location is deduced as
    <location-of-rebase.exe>/../etc/rebase.db.i386
    <location-of-rebase.exe>/../etc/rebase.db.x86_64

The (optional) database allows to easily rebase new DLLs so that they don't
collide with other DLLs which already have been rebased.  Because rebaseall uses
the database, rebaseall will now only rebase new DLLs, or those DLLs which
have been updated -- e.g. their current ImageBase does not match the value in
the database, or their size has changed (grown) such that they no longer 'fit'
in the previously-assigned location. (If an updated DLL still fits in its
previously-assigned location, it will be rebased back to that location).

Thus, the (optional) database allows to keep an installation in a stable
state, and only modifies the ImageBase of DLLs when absolutely necessary.
A complete rebase can be forced by (a) specifying -b to either rebase.exe
itself, or to rebaseall, or (b) deleting the existing database files and
re-running rebase/rebaseall.


peflags
--------------------------------------------------------------------------------
The following is the peflags command line syntax:

    Usage: peflags [OPTIONS] file(s)...
    Sets, clears, or displays various flags in PE files (that is,
    exes and dlls).  For each flag, if an argument is given, then
    the specified flag will be set or cleared; if no argument is
    given, then the current value of that flag will be displayed.
    
      -d, --dynamicbase  [BOOL]   Image base address may be relocated using
                                  address space layout randomization (ASLR)
      -f, --forceinteg   [BOOL]   Code integrity checks are enforced
      -n, --nxcompat     [BOOL]   Image is compatible with data execution
                                  prevention
      -i, --no-isolation [BOOL]   Image understands isolation but do not
                                  isolate the image
      -s, --no-seh       [BOOL]   Image does not use SEH. No SE handler may
                                  be called in this image
      -b, --no-bind      [BOOL]   Do not bind this image
      -W, --wdmdriver    [BOOL]   Driver uses the WDM model
      -t, --tsaware      [BOOL]   Image is Terminal Server aware
      -w, --wstrim       [BOOL]   Aggressively trim the working set.
      -l, --bigaddr      [BOOL]   The application can handle addresses
                                  larger than 2 GB
      -S, --sepdbg       [BOOL]   Debugging information was removed and
                                  stored separately in another file.
      -T, --filelist FILE         Indicate that FILE contains a list
                                  of PE files to process
      -v, --verbose               Display diagnostic information
      -V, --version               Display version information
      -h, --help                  Display this help
    
    BOOL: may be 1, true, or yes - indicates that the flag should be set
              if 0, false, or no - indicates that the flag should be cleared
              if not present, then display symbolicly the value of the flag
    Valid forms for short options: -d, -d0, -d1, -dfalse, etc
    Valid forms for long options :  --tsaware, --tsaware=true, etc
    To set a value, and display the results symbolic, repeat the option:
      --tsaware=true --tsaware -d0 -d

Source:
================================================================================
Cygwin rebase builds OOTB under Cygwin, MinGW, and MSYS. It also can be compiled
using the 64bit mingw64 compiler: x86_64-w64-mingw32-gcc. Cross compilation is
(somewhat) supported, but YMMV.


Build:
================================================================================
See build.sh in the source archive for my exact build recipe for configuring,
making, and packaging this distribution.


Test:
================================================================================
rebase does not contain any regression tests.

peflagsall may be invoked with the -n, -k, and -v options to allow inspection
of what it WOULD do, without actually doing it.

peflags will make no changes to any files unless it is given an flagname
argument *with* a boolean value for that flag (0 or 1). If no boolean argument
is provided, then peflags will merely display the current value of the flag in
the specified files.


Issues:
================================================================================
The following are the known Cygwin rebase issues:

1. rebase should be integrated with Cygwin's setup.exe.

2. rebase does not handle in-use DLLs.

3. rebase skips read-only DLLs.

Issues #1, #2, and #3 also apply to the peflags utility.


Homepage:
================================================================================
The primary rebase web site is:

    http://www.tishler.net/jason/software/rebase/


Download:
================================================================================
The primary rebase download site is:

    http://www.tishler.net/jason/software/rebase/

Access to the CVS development sources is available:

    cvs -d:pserver:anoncvs@cygwin.com:/cvs/cygwin-apps co rebase


Mailing Lists:
================================================================================
Please report problems, suggestions, etc. to <cygwin@cygwin.com>.


Maintainer:
================================================================================
    Jason Tishler <jason@tishler.net>