viewpoints -- Fast interactive linked plotting of large multivariate data sets.

Creon Levit <Creon.Levit@gmail.com> 

Preface:

Please let us know how viewpoints helps you. Did it enable you to solve a
problem?  To make a discovery?  To find an error?  To generate a figure for 
a paper?  The more we know that viewpoints is useful, the more time we'll 
spend working on it.  


Overview:

Each archive on this site contains the  viewpoints (vp) executable, 
documentation, a sample data file with examples, and for MS Windows, any 
dynamic link libraries (dlls) that may be required by the package.  The 
contents of these archives are listed below:

File                Comments
-----------------------------------------------------------------
README              This documentation file
vp.exe              viewpoints executable (MS windows only)
vp.ico              viewpoints icon (MS windows only)
vp                  viewpoints executable (linux only)
viewpoints.app      application bundle (Mac OS X only, see below)
vp_help_manual.htm  HTML help manual
sampledata.txt      sample data
libgsl.dll          DLL library for GSL (Windows only)
libgslcblas.dll     DLL for GSL CBLAS (Windows only)
OglExt.dll          DLL for OGL Extension Library (Windows only)


Installation:

Unpack these files into a target directory, then either click on the 'vp' 
icon or run the code from the command line as described below.

Data Organization and Format:

Viewpoints can read and write data from ASCII or binary files.  ASCII files 
consist of zero of more header lines, indicated by comment characters '!#&%', 
followed by a data block.  The data block is assumed to consist of a line of 
column labels followed by successive lines of data.  Words in the data block 
can be delimited by whitespace or some user-defined character.  Binary files
conists of a line of column label information followed by a block of binary
data in row- or column-major format.  Note that the line of column labels 
must NOT be preceded by a comment character unless they are specified by the
user to be part of the header.

Data can also be read directly from the input line using the --stdin command
line option described below.  This allows piping on systems that support it,
which allows the user to use a wide variety of third-party applications to
read or process different data formats such as FITS, CDF, etc.

FITS files: Viewpoints now has limited ability to read and write FITS files.  
When asked to read a FITS file, viewpoints will search for the first ASCII 
table extension, attempt to read it, and restore the existing data if 
unsuccessful.  When asked to write to a FITS file, it will create a new file, 
overwriting and destroying any original of the same name, and write the data 
to a single ASCII table extension in that file.  This FITS i/o capability is 
still under development and will be expanded in future revisions.

Configuration Files: *** DEPRECATED due to instability ***

// Viewpoints can save configuration information, such as axis and brush 
// settings, window positions, and the name of the last input or output data 
// file, to a configuration file for later reuse.  This is equivalent to saving
// a snapshot of the work session.  Note that this process does not save the 
// actual data or selection information itself!  If it did, this could lead to 
// unnecessary duplication of the data files.  For this reason, the user must 
// save any new or modified data they wish to associate with a configuration 
// before they save that configuration.

// Saved configurations can be loaded with or without reading their associated 
// input file.  When this happens, viewpoints will attempt to use the axis
// indices described in the configuration file (e.g., if the configuration
// used axes 1, 2, 6, 7, and 10, viewpoints will attempt to display the
// corresponding columns in the current data file).

// Configuration files are saved in XML format for ease of display, but these
// are not treated as conventional XML files.  In particular, due to limitations
// of the BOOST serialization library, the contents of these files are order
// dependant.  For this reason, any attempts to edit them should be performed
// with extreme caution.

Usage:

In its current form (Version 2.2.4, build 258) 'viewpoints' is run either by 
clicking on the 'vp' icon or from the command line in the directory in which 
it has been installed or to which a path is available.  When viewpoints is 
run from the icon, it will come up with an array of default data.  When it is
run from the command line, the user must specify the input data file 
explicitly.

Apple OS X: to run viewpoints from the command line, you will probably want 
to create a shell alias to the actual executable, which is located inside the 
application bundle.  For example, if you dragged the viewpoints application 
"viewpoints.app" to your /Applications folder, then you will probably want to 
add the following to your .cshrc file (assuming you are using csh or tcsh as 
your shell):

 alias vp  /Applications/viewpoints.app/Contents/MacOS/viewpoints

A symbolic link will not work.  Use a shell alias.

If you don't care about running it from the command line, ignore the above.  
You can just double-click on the viewpoints.app icon to start it up.


The invocation for the command line version is:

 vp [optional arguments] [optional inputfile]

 the optional arguments are:
    
  --format=ascii (default) [shortcut: -f a] 
    Read an ASCII input file that consists of a header block followed by a
    data block.  Lines in the header block are indicated by one the comment 
    characters '!', '#', or '%'.  In the absence of comment characters, the 
    header block is assumed to consist of a number of lines specified by the 
    '--skip_header' command below, with a default value of 0 lines.  The 
    first line of the data block contains the attribute names.  By default,
    this is not preceeded by a comment character and is delimited by
    whitespace, but this can be controlled from the main menu or by command
    line arguments (see --delimiter, below).  Successive lines in the data
    block contain the numeric attribute values for successive samples, 
    delimited by the delimiter character.  See the file "sampledata.txt" for 
    an example.

  --format=binary [shortcut: -f b]
    Read a binary input file.  For files that don't contain ASCII values, 
    the first record be a header that consists of a tab-delimited line of 
    ASCII attribute names ending with a newline (\n).  This will be followed 
    by a contiguous block that contains a table of binary floating point 
    values.  For files that contain ASCII values, the header will consist a
    succession of lines of ASCII text that describe the contents and provide
    ASCII lookup tables, if any, for each column of data.
    
  --format=fits [shortcut: -f f]
    Read the first ASCII table from a FITS file.  If no ASCII table is found,
    routine will assume file was empty.

  --skip_lines=<integer> [shortcut: -s <int>] (default 1)
    Specifies the number of lines that will be assigned to the header block 
    in the absence of comment characters.

  --npoints=<integer> [shortcut: -n <int>] (default min(all,3000000))
    Specifies the number of samples (records or rows in the data block of an 
    ASCII file) to read. End of file will terminate read.  Note the default:
    if you want to read more than 3 million samples, you must say so using 
    this argument.

  --rows=<integer> [shortcut: -r <int>] (default 2)
    Specifies the number of rows of scatterplots

  --cols=<integer> [shortcut: -c <int>] (default 2)
    Specifies the number of columns of scatterplots

  --input_file=<filespec> [shortcut: -i <filespec>]
    Filespec of the input file.  NOTE: if this parameter is not specified, 
    the code will assume that the final token in the command line is the 
    input filespec.

  --laptop_mode [shortcut -l]
    Shrink control panel to fit in a laptop screen
    
  --commented_labels [shortcut -L]
    Assume that column labels are contained in the last commented line 
    before the data block -- i.e., the last line of the header block.
    Default behavior is to read column labels from the first (uncommented) 
    line of the data block.

  --config_file=<filespec> [shortcut: -C <filespec>]
    Filespec of a saved configuration file.  NOTE: If this parameter is 
    specified, it will override any input filespec.

  --borderless [shortcut: -b]
    Attempt to maximize plot windows' usable area by removing window manager 
    decorations.  Note: this seems to cause problems with keyboard shortcuts 
    in plot windows under Mac OSX.

  --help [shortcut: -h]
    Print a short help message.

  --ordering={rowmajor,columnmajor}
    ordering for binary data, default=columnmajor

  --nvars [shortcut -v <int>]
    The number of variables (attributes) per sample is automatically 
    determined from the last header line in an ascii input file, or from the 
    first line of a column-major binary file.  This option is only for row 
    major binary data, in which case one must also specfiy --npoints above.

  --delimiter [shortcut -d=<char>]
    interpret char as field separator, default is whitespace.  Delimiter 
    characters can be escaped using the standard c-language convetions. This 
    delimiter is also used in the header to delimit variable (attribute) 
    names.
    e.g.  --delimiter=,  for comma delimited
    or    --delimter=\t  for tab delimited
    
  --missing_values [shortcut -M <number>]
    set the value of any unreadable, nonnumeric, empty, or missing values to 
    NUMBER, default=0.0.  NOTE: if you use the default delimiter (whitespace) 
    then lines with one or more missing values are skipped and so this 
    option has no effect. 

  --no_vbo [shortcut -B]
    don't use openGL vertex buffer objects.  Useful if you have an older 
    graphics card or if the graphics are inconsistent or very slow.  May be
    useful if you are attempting to look at huge datasets.

  --preserve_data=(T,F) [shortcut: -P <string>] (default TRUE)
    Preserve existing data for restoration if a read operation fails.  Turn
    this off to reduce memory usage for extremely karge data sets.

  --stdin [shortcut -I]
    read input data from stdin.  This allows piping on systems that support 
    it.
    e.g.   tail -n 100 bigfile | awk '{print $0 " " $3/$2}' | vp --stdin
  
  --trivial_columns=(T,F) [shortcut: -t <string>] (default TRUE)
    Removes columns with a single value

  --verbose [shortcut -O]
    print verbose output with additional diagnostics
    
  --version [shortcut -V]
    print version information and then exit.
    
  --expert [shortcut -x]
    enable expert mode, that bypasses confirmations and allows reads from
    stdin, etc.

  --help [shortcut -h]
    print out brief usage message and then exit.


When 'viewpoints' is invoked, it will read the input file, then display a 
control panel along with an 'r' x 'c' array of linked scatter plots.  These 
windows can be moved and resized in the conventional fashion.  If you delete 
some window by accident, you can restore every window using the 'reload 
plots' command. (If you delete every plot window, the program may crash).

You can also select a particular plot by giving it the mouse focus directly, 
or by clicking on its associated tab in the control panel window.  Within 
different windows, you can use the mouse to select portions of the data set.  
It is in this feature that the power of 'viewpoints' resides.  Rather than 
attempt to describe it in detail, we encourage you to experiment!


Main menu bar

 command                     action
 --------------------        ----------
 File|Open data file         Read data from an input file
 File|Append more data       Append additional samples to the existing data
 File|Merge another file     Merge additional attributes for these samples
 File|Write ASCII file       Write all the current data as an ASCII file
 File|Save all data          Write all data
 File|Save selected data     Write only the currently selected data
 File|Load Configuration     Load configuration information
 File|Save Configuration     Save configuration information
 File|Current File Name      Show name of current data file
 File|Clear all data         Replace data with a small default array
 File|Quit                   Quit
 View|Add Row                Add a row of plot windows
 View|Add Column             Add a column of plot windows
 View|Remove Row             Remove a row of plot windows
 View|Remove Column          Remove a column of plot windows
 View|Reload File            Reload the existing data file
 View|Restore Panels         Restore deleted plot windows
 View|Default Panels         Restore the default polt window configuration
 Tools|Edit Column Labels    Prototype of a column label editor
 Tools|Statistics            Show selection statistics
 Tools|Options               Set viewpoints options
 Help|Viewpoints Help        Opens a simple HELP window
 Help|About Viewpoints       Information about this version


Buttons and (keyboard shortcuts when a plot window has mouse focus):

 action                 key
 --------------------   ----------
 new selection          left-mouse
 move selection         left-mouse + shift
 invert selection       i
 display deselected     d
 clear selection        c
 search x-axis strings  F
 search y-axis strings  f
 kill selected points   x
 reset view             r
 quit                   q


Mouse Gestures (with the cursor in one of plot windows):

 action                 gesture
 --------------------   ----------
 select points          left-mouse
 translate              right-mouse  (opt-mouse in OSX)
 scale                  middle-mouse (ctl-mouse in OSX)
 scale both x and y     mouse-wheel
 scale histogram        middle-mouse + h (ctl-mouse + h in OSX)

    
Controls in the control panels:

 The control panel consists of a set of tabbed control panels for individual 
 windows, tabbed control panels for individual brushes, and a main control 
 panel for the entire array of panels.  For the most part, these controls 
 should be intuitive.  Some of these controls are described below:

 Tabbed control panels for individual windows:
 control                action
 --------------------   ----------
 lock X, Y, or Z        Lock axis so it won't change
 plot                   Attribute to be displayed in that axis
 scale                  Normalization scheme for that axis
 offset                 Offset data by +/-i points along this axis.  Note 
                        that this is a spinner rather than a slider.
 histog                 Show histograms along that axis.  
                        'Marginal/Selection/Conditional' corresponds to 
                        'All points/Selected points/Fraction selected'
 N bins                 number (log) of histogram bins for that axis
 bin ht                 height of histogram bins for that axis
 bkgrnd                 background color (try bkg=0.5, lum2=0.2)
 lumin                  luminosity for all points
 pntsize                default size of unselected points
 scale                  Scale point sizes along with axes
 rotate                 rotation angle in 3D about the y-axis.  NOTE: to take
                        advantage of this feature one must first select 
                        something for the Z-axis to display.
 spin                   continuous rotation about the y-axis.  NOTE: to start
                        this, you may have to give the rot slider a twitch.
 reset view             Reset rotation and other display params
 z-buffering            Use z-buffering (only for 3D plots)
 blending               Blending scheme for brushes
 don't clear            Don't clear selected points in this panel
 points                 Show data points
 unselected             Show unselected data points
 axes                   Show axes
 ticks                  Show tic marks
 grid                   Show grid
 identity               plot y vs x
 sum vs. diff plot      plot (x+y) vs. (x-y)
 rank(y|x)              Plot x vs the rank of y for a range about that x  (e.g., 
                        rank points within a sliding bin of x-values by their 
                        value in y.  The width of this bin is controlled the 
                        number of histogram bins, N bins.)
 fluc(y|x)              Plot x vs the deviation in y for a range about that x
                        (e.g., rank points within a sliding bin of x-values by 
                        their variation in y.)

 Tabbed control panels for individual brushes
 control                action
 --------------------   ----------
 size                   size of this brush, in pixels
 reset brush            reset this brush
 Alpha                  point opacity
 symbol                 symbol used by this brush
 lum1                   successive brightness increase for overplotting
 lum2                   successive brightness increase for overplotting
 Color chart            Color controls for this brush
 extend selection       combine successive selections with this brush
 clear selection        clear successive selection for this brush
 paint                  'dribble paint' while dragging the selection box
  
 Main control panel
 control                 action
 --------------------   ----------
 show nonselected       Show unselected points
 invert selection       Invert selected and nonselected points
 clear selcetion        Clear selection
 kill selection         Delete all selected points
 unselected color       Chose color of unselected points
 change axes            Change all (unlocked) axes
 link axes              Link similar axes
 defer redraw           Update selections on mouse-up only (for large data)


Normalization schemes (per axis)

 Data can be normalized in several different ways.  The normalization scheme 
 for any axis can be selected from the appropriate menu.

 type                   description
 --------------------   -----------
 none                   show all data, center of window at median
 minmax                 window spans maimimum to maximum.
 zeromax                show all positive data
 maxabs                 show all data, center the window at zero
 trim 10^-2             window spans the 1st-99th percentile
 trim 10^-3             window spans the 0.1th-99.9th percentile
 threesigma             center window at sample mean so that window spans 
                        +/- three sigma.
 log_10                 Logarithmic axis
 atan                   Simple sigmoid
 rank                   plot rank (order) of x instead of x -- i.e. 
                        nonuniform rescaling to force a uniform marginal
 partial rank           as 'rank' above, but overplot identical values
 gaussianize            plot inverse of the Gaussian cummulative 
                        distribution function of x -- i.e. nonuniform 
                        rescaling to force a uniform marginal
randomize               Randomize


Notes and warnings:

The current release of the 'viewpoints' package (version 2.2.4, build 258) is 
still a development version.  While every effort has been make to ensure that 
it will work -- or fail gracefully if it doesn't -- it still has many rough 
edges.  Some of these are described below.  These issues will all be 
addressed in future releases.

) If you drag a window over the main control panel, the main control panel
may need to be refreshed.  This can be accomplished by resizing it.

) The FITS file i/o capability is new, and is not yet guaranteed to work on 
every system.  It will undergo substantial improvement in future releases.

) The Edit Column Labels tool is still under development, and can produce
unexpected behavior.  It is not guaranteed to preserve axis or scaling 
information, and in the current release can only delete labels.  For ASCII
files, column labels can be renamed via by using an editor in a conventional 
fashion.  Small binary files can be saved to ASCII and edited.  For large 
binary files, it is possible, with difficulty, to rename labels by saving a 
small portion of the file as ASCII, editing this, reloading it, and appending 
the remaining portion of the original binary file.

) For a variety of reasons related to platform-independance and formatting
of multiple windows, there is, as yet, no 'print' command.  Images can be 
saved for use as screenshots in a conventional fashion using the relevant OS 
commands.

Please send any questions, bug reports, feature requests, and/or praise to: 

Creon Levit at   creon.levit@gmail.com