VHIST Users' Guide

The VHIST [1], [2] project defines a file format specification that allows to embed arbitrary binary data for the documentation of workflows together with structured meta-information and multiple facilities for validation. The format conforms to PDF and other open standards, is self-describing and particulary suited as an image or meta-image format in the context of multi-modality and functional imaging.

It includes a platform independent reference implementation which contains the essential features. VHIST can be used on top of existing workflows without the need to change major applications. Please refer to the VHIST white paper [1] for more information on the general concept and the specification, this User's Guide focuses on how to use the reference implementation.

Please see Disclaimer and Licensing for legal issues.

The reference implementation currently consists of a set of commandline tools and a user-friendly suite of GUI tools. It can be downloaded at the VHIST homepage [1].

The commandline tools (VHIST core):

• vhistadd - commandline tool for creating VHIST files.
• vhistxs - commandline tool for extracting embedded data from VHIST files. Minimum implementation.
• vhistxl - commandline tool for extracting embedded data from VHIST files, including facilities of individual sections, extraction of individual embedded files with validation.

The GUI tools:

• vhistzard - an application that helps configuring commandline options for vhistadd and creating VHIST files without using the commandline.

Platform independence was an important goal when conceiving VHIST and its Reference Implementation We believe that VHIST Core should work well on a wide range of systems for which a Python distribution [3] (equal or better to version 2.4) exists, especially on anything manufactured in this millenium.

We use Trolltech's Qt libraries [4] for the user friendly VHISTzard. Some distributions of VHIST ship with the appropiate libraries and should run out of the box.

MS Windows

A standard installer is available that can deploy a ready-to-run version of VHISTzard including all commandline tools. This setup does not require a Python distribution, as all commandline tools have been compiled to ready-to-use stand-alone ".exe" programs. Alternatively, you could install Python (using a proper "single-click" installer [3]) and use the "source" distribution of VHIST.

Mac OS X

a ready-to-run version of VHISTzard is available for this platform, it can be installed as a standard Mac application. Installing the source distribution is also possible, a suitable Python distribution is part of the operating system since Mac OS X 10.3 (for older systems it can be easily installed without compromising other applications that might rely on the older versions). VHIST should work just fine with the newer "Intel" Macs (tested with Tiger and Leopard) and for the older PPC platform.

Linux

Python should be ubiquitous on this platform. Use python -v to check whether your installed version is recent enough (it very likely is), anything equal or better version 2.4 should work. If the installed version is not recent enough, you should consider upgrading it. Please refer to the documentation of your distribution

If for some reason Python is not installed on your system, please refer to the documentation for your linux distribution on how to install Python.

Solaris

Python is shipped with Solaris 10 and available for previous versions of this operating system. VHIST should work on SPARC systems, as well as Solaris machines with x86 architecture.

The general idea behind VHIST is to provide a robust and simple means for documenting steps of a workflow by logging and optionally embedding all relevant information: which files were used, which files were written, what software package was used in which version and with what parameters.

An example from medical imaging is the process (workflow) to create an image volume suitable for scientific/diagnostics purposes. In this case, a typical workflow step could be the the application of an image filter (tool, this example works best with a commandline tool) to an image, present as an input file (infile). The commandline used in this workflow step, in addition to some filter option (say, Gauss filter scope), could be added to the VHIST file using Attributes. Assuming the filter application writes a new file containing the filtered image volume, we can add this as an output file (outfile). It is recommended practice to have vhistadd embed (sse Files: Embedded Vs. Reported) any logfile the filter application might write.

An attribute is a key-value pair. vhistadd provides commandline options for specifying either pre-defined or user-defined keys. We make this distinction to ensure that a basic set of attributes is universally available (e.g. description, or title). This is a prerequisite for comparing VHIST files from different sources. Attributes can be specified for the workflow step and even for individual in- and outfiles. The type and meaning of the value depends on the key is described in the documentation of the individual options.

VHIST files are stacks of sections which can be validated independently and usually refer to one workflow step. Sections are appended at the end of an existing file, so no previous data is changed. This is related to incremental writing of PDF [5] files. If you want to add on to a VHIST file containing information about previous workflow steps (this is recommended) you need to specify a VHIST root file (this can be any VHIST file).

The vhistadd implementation uses the terms reported and embedded, meaning similar but slightly different things.

A reported file is a file whose file properties are reported in the automatically generated XML summary [1] (which contains structured information on one workflow step and is suitable for automated processing) and in the corresponding "human-readable" PDF part of the VHIST document.

An embedded file's properties are also listed in the XML summary and the PDF listing. However, in this case the contents of the file are also completely contained in the VHIST document, either in compressed or uncompressed form.

Embedded files can be extracted by various means, e.g. with a PDF browser, VHISTzard, vhistxs, vhistxl. You can embed binary data in VHIST files.

THIS SOFTWARE IS PROVIDED AS-IS, WITHOUT ANY EXPRESSED OR IMPLIED WARRANTY. SPECIFICALLY, NEITHER THE MAX-PLANCK-INSTITUT FÜR NEUROLOGISCHE FORSCHUNG MIT KLAUS-JOACHIM-ZÜLCH LABORATORIEN DER MAX-PLANCK-GESELLSCHAFT UND DER MEDIZINISCHEN FAKULTÄT ZU KÖLN NOR THE AUTHORS WARRANT THAT THE FUNCTIONS CONTAINED IN THE SOFTWARE WILL MEET YOUR REQUIRMENTS, OR THAT THE OPERATION OF THE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT DEFECTS IN THE SOFTWARE WILL BE CORRECTED. TO THE EXTENT PERMITTED BY LAW, NEITHER THE MPI FÜR NEUROLOGISCHE FORSCHUNG NOR THE AUTHORS SHALL BE LIABLE FOR ANY DAMAGES ARISING OUT OF OR RELATING TO THE USE OF THE SOFTWARE, INCLUDING BUT NOT LIMITED TO INCIDENTAL, SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY LOST PROFITS, BUSINESS INTERRUPTION, LOSS OF PROGRAMS OR OTHER DATA ON YOUR INFORMATION HANDLING SYSTEM.

FOR NON-COMMERCIAL SCIENTIFIC RESEARCH USAGE ONLY, the VHIST spcification [1] and the VHIST Reference Implementation are available under the GNU General Public License (GPL Version 3) [6]. Any other usage requires written permission by the Max Planck Institute for Neurological Research Cologne.

You can write the arguments of a vhistadd call into a file and then specify this file instead of typing the whole commandline into the shell. These files are called Arg-Files.

Orginally, this feature was implemented due to limitations of the MS Windows platform, where only commandline arguments of fairly moderate size can be passed to programs (2048 characters on Windows 2000, 8191 on Windows XP [7], typical UNIX commandlines can be >100KB long). The corresponding error message does not make this obvious: "Window cannot access the specified device, path, or file. You may not have the appropriate permissions to access the item."

Since 2048 characters is not enough for all but the simplest use cases, a more powerful mechanism was needed. Arg-Files have several additional advantages to normal commandline calls: Arg-Files can easily be archived, copied around and passed to other uers. They can even be embedded into the VHIST file they generate, the lines in an Arg-File can be of arbitrary length and the syntax for Arg-Files is independent of platform, operating system of shell. It is very similar to the syntax of the bash [8] shell: single arguments are separated by one or several whitespace characters (space, tab or newline), strings which contain whitespaces must be enclosed in quotes (") and enviroment variables start with a $ sign, e.g. $PATH. Comments start with a hash symbol # and continue until the end of the line.

This table contains a brief summary of the commandline options for vhistadd.py.

This table contains the qualifiers that can be attachted to input or output files.

-h or --help
gives a brief description of all options.

-v or --version
will print out vhistadd's version information and exit.

-c <cmdfile> or
--cmdfile <cmdfile>
will read a file <cmdfile>.

An Arg-File is a file containing the commandline of one vhistadd call as descriped in Arg-Files.

It is possible to synchronize the current working directory used by vhistadd with the path of the selected Arg-File. If synchronizing is enabled, all paths (with the exception of readme- and firstpage files) specified in the commandline or any Arg-File are relative to the path of the first Arg-File.

-O <filename> [-O <filename>] [-O <filename>] ...

The name of the VHIST file which will be generated during this workflow step. If more than one file is specified, several VHIST files with identical content will be generated. At least one -O (capital letter "O") option must be specified.

By design, vhistadd prevents you form overwriting existing VHIST files except (a) when an output VHIST file is also specified as a VHIST root file or (b) you choose the Append to VHIST file option.

-A <filename>

If the file exists, the current workflow step will be appended to the file, preserving all previous content. If not, a new VHIST file of that name will be created.

-I <filename> (file must exist)
-J <filename> (ignore if file cannot be read)

The name of the VHIST file, which is used as the basis for the new VHIST file. The generated workflow step is appended to a copy of this VHIST document. The VHIST root file is not modified except if the root file is also specified as an output VHIST file. If this option is not specified, vhistadd starts with an empty document. The -I option will cause vhistadd to stop with an error message if the VHIST root file cannot be read, use -J if vhistadd should ignore a missing root file (we have introduced -A, see previous subsection, to achieve the same effect without having to specify an output VHIST file with -O).

-s <key> <value>

We distinguish between User-defined attributes (workflow step) and pre-defined properties or attributes of a workflow step. Use to set one or more of the following pre-defined attributes (it is an error to use other keywords with this option):

-U <key> <value>
to set a user-defined attribute.

We distinguish between user-defined and Pre-defined attributes (workflow step). -U "my key" "my value" would create the attribute my key and set it to my value.

-d <key> <value> or
-doc <key> value>

Valid keys are: producer, creator, author, keywords, title, subject

These properties are important when viewing VHIST-files using a PDF browser. We refer to [5] for a detailed description.

These properties can currentyl only be set for newly created VHIST files. When appending a workflowstep to an existing VHIST file, an attempt to set PDF-related properties is ignored.

-i <filename> or
--infile <filename>
specifies one or more input files.

The VHIST format distinguishes between the files present before the workflow step (infiles) and files which result from executing the specified tool (outfiles).

See Pre-defined flags (files) Pre-defined attributes (files), User-defined attributes (files) for further options.

-o <filename> or
--outfile <filename>
to specify one or more output files.

The VHIST format distinguishes between files present before the workflow step (infiles) and files which result from executing the specified tool (outfiles).

See Pre-defined flags (files) Pre-defined attributes (files), User-defined attributes (files) for further options.

-r <key>
sets one or more pre-defined flags.
It is an error to use other keywords then pre-defined flags with
this option.

A flag enabling or disabling a property of the previously specified in- or outfile. The flags can be negated by prepending the flag's name with no-, e.g. no-embed.

-a <key> <value>
sets one or more pre-defined attributes. It is an error to
use other keywords then pre-defined attributes with this option.

An attribute describing a property of the previously specified in- or outfile.

-u <key> <value>
sets a key value pair of your choice.

We distinguish between user-defined attributes and Pre-defined attributes (files)

sets the attribute my key to the value my value.

-v or
--verbose

Use this option to have vhistadd generate more verbose output to stdout.

-q or
--quiet

If this option is set, the output to stdout is reduced to errors only.

-p or
--pretend

If this option is set, vhistadd only parses the commandline and generates the workflow-step but does not generate a VHIST file. This option is useful for verifying a commandline.

-1 or
--firstpage

A file containing the content of the first page of a newly created VHIST document. The content of the file must be encoded in UTF-8 [12]. The text can contain Wiki-like markup which supports bold, italic and one coloured (blue) font attributes. By default, vhistadd will use res/title.txt.

This option is ignored if the workflowstep is appended to an existing VHIST file.

-r or
--readme

A file containing the readme, which is embedded at the beginning of a newly created VHIST document. The content of the file must be encoded in an ASCII compatible encoding, UTF-8 [12] is preferred. By default, vhistadd will use res/embedded_readme.txt. This option is ignored if the workflow step is appended to an existing VHIST file.

Please note that the following examples assume that you are using the "plain vanilla" Python version of vhistadd. The important part of the examples are the commandline parameters (everything after the initial vhistadd.py). We suggest you use bin/setup.py for setting up paths and links on unixoid platforms (Linux, Mac OS X).

The actual calling syntax of the vhistadd tool depends on your platform and configuration, e.g.

UNIX/Linux/Mac OS X, assuming vhistadd was installed in /usr/local/vhist: /usr/local/vhist/bin/vhistadd.py ...

UNIX/Linux/Mac OS X, with a suitable PATH variable, or ALIAS vhistadd.py ...

UNIX/Linux/Mac OS X, if the Python sources are (for some reason) not directly executable: python vhistadd.py ...

MS Windows without Python, assuming the VHIST executables were installed in Programs (no Python distribution is then necessary): C:\Program Files\VHIST\bin\vhistadd.exe

MS Windows with Python, assuming the VHIST executables were installed in Programs (you need to install a Python distribution first): cmd vhistadd.py

-h or --help
gives a brief description of all options.

-v or --version
will print out vhistxl's version information and exit.

-t or --list <vhistfile>
will only list all embedded files and not extract any data.

-v or --validate
will validate sections.

-x or --extract
will extract all files to disk (implies List Embedded Files).

-r <fileid> or --extract-file-<fileid>
will extract the file with the id <fileid>.

Use List Embedded Files to find the correct <fileid> of a particular file.

-p or --pretend
will only decompress and test MD5 checkums [9] of embedded files, but not
write anything to disk.

-d <dir> or --dir <dir>
will extract files to directory <dir>.

vhistzard also contains a tool which is deisgned to assist in the creation of VHIST files from existing VHIST Arg-Files. It allows you to execute vhistadd without using a commandline.

One part of vhistzard is a tool to view VHIST files nad to extract embedded files from them in a user-friendly manner. This tool provides a template-based preview ("themes") of the information contained insed the VHIST file (in HTML format) and allows for easy extraction of single files or complete workflow steps.