MetaTF: a Language for Specifying Trace Formats

Richard Jones, 30 March 2001

The files in this directory represent the current definition of Meta-TF, version 1.2.1. This has been developed from Meta-TF 1.0 after discussion at ISMM 2000 in Minneapolis, October 15-16, 2000. (Meta-TF 1.1 was a simplification of Meta-TF 1.0, designed to simplify tool construction).

Program traces

Dynamic storage allocation continues to play an important role in the performance and correctness of systems ranging from user productivity software to high-performance servers. While algorithms for dynamic storage allocation have been studied for decades, much of the literature is based on measuring the performance of benchmark programs unrepresentative of many important allocation-intensive workloads.

To overcome the difficulties of acquiring benchmarks and of generating program traces of realistic scale, Trishul Chilimbi, Ben Zorn and Richard Jones proposed in our ISMM 2000 paper, Designing a Trace Format for Heap Allocation Events that large traces from realistic benchmarks be collected and shared . This site provides a portable mechanism for sharing traces. We hope that others will take up these tools and share their traces with the community.

A specification language

MetaTF is proposed as a standard for publishing and exchanging representative allocation workloads. Although our interest lies in heap allocation events, MetaTF is a general-purpose specification language for program traces. A trace is specified by a DTD, a document type definition (but not to be confused with SGML or XML DTDs). The DTD defines sections of the trace and records within each section. Record formats allow trace events to be encoded in a number of ways: for example, defaults may be used, or an allocation address may be given as a delta from the previous address, or an event may refer to a record in another section (for example, that section may contain a list of fully qualified type names). A novel feature of MetaTF is that a trace may contain metadata to vary the encoding of the trace on the fly.

DTDs, traces and tools

We provide a full, albeit terse, specification of MetaTF and a tool for automatically constructing reader and writer routines from a particular MetaTF DTD. The description of MetaTF also includes a tutorial on the use of the tool. Currently MetaTFtool generates only Java. Offers to extend it to generate readers/writers in other languages (particularly C or C++) will be gratefully received.

• Changes from earlier versions.
• MetaTF 1.2 full description.
• MetaTFtool: a compiler for MetaTF1.2.
• A directory of DTDs: we hope that the community will contribute MetaTF DTDs and links to traces.
• A directory of traces.

MetaTFtool

MetaTFtool is a compiler for MetaTF. It takes a MetaTF 1.2 DTD and produces a set of Java classes representing a reader/writer for traces that conform to the DTD. MetaTFtool is described in the full report along with a brief tutorial. Provided here is

• A license for its use.
• A MetaTF reader/writer compiler, MetaTFtool, and a jar file containing its classes. For ease of cross-platform use, MetaTFtool is a perl program that invokes java with the appropriate arguments.
• Source code and documentation in javadoc format for MetaTFtool.
• A simple dtd and client program.
  To try this,
1. MetaTFtool -J Test dtd/test.dtd
   This will generate a reader/writer class plus a class for each section and record of the DTD in the directory Test.
2. javac -classpath MetaTF.jar Test/*.java
3. javac -classpath MetaTF.jar:. TestDriver.java
4. java -classpath MetaTF.jar:. TestDriver

Changes from MetaTF 1.2

The record field interpretation property is renamed encoding.

Changes from Meta-TF 1.0

The original definition of Meta-TF 1.0 can be found in Designing a Trace Format for Heap Allocation Events, Trishul Chilimbi, Richard Jones and Benjamin Zorn, ISMM 2000, Minneapolis, October 2000, ACM. Following discussion at ISMM~2000, Meta-TF 1.2 makes the following changes:

• The trace format is ASCII, one line per record. Although binary encodings may offer greater compression, the difference is slight and does not outweigh the benefit of being able to debug ASCII traces or to use them with other text-based tools. In the future, Meta-TF may also offer an XML format. This option is likely to become more attractive as more parsers and compressors for XML appear. In particular, we have been impressed by the compression offered by AT&T's xmill tool (which uses split-stream encoding on XML elements).
• Meta-TF 1.2 offers cached sections as well as plain sections. The intention of cached sections is that they may be used to look up attributes (such as types or call-stacks) of records in other sections. Meta-TF reader tools are expected to store cached records and make caches available to client programs. Sections may be interleaved. Other than the optional keyword, cache, the syntax for cached and other sections is identical; both share the same name- and number-space.
• Record fields are typed. The primitive types are byte, short, int, long, string or index. All values in a trace are either hexadecimal integers or strings. Integer literals in a dtd may be either decimal or hecadecimal; the latter are written as 0x....tag fields have an implicit type int.
• Record fields no longer have widths (since the format is ASCII). The effect of setting a field's width to 0 may be achieved by setting its interpretation to one with an implicitly zero width, i.e. default (recommended) or stride. Strings may only have interpretations none or default. A field of type index must have interpration sectionOffset, sectionStride or default 0.
• The syntax for defining fields is changed. For example, a field named foo, which has a default integer value of 4, is defined by int foo = default 3;
• Trace files are expected to contain conventional comments giving the provenance of the trace.
• The built-in record vfield is no longer provided.
• Metadata records may change either a given field of a single record type in the current section, or they may change all fields of a given name in all records of the current section.
• DTD names are case-insensitive in order to allow the code generators greater freedom in choice of capitalisation of class and member names.