|
Logfile Format
Over the years, several logfile formats have been developed for
postmortem performance visualization. The following list
has the logfile formats arranged in reverse chronological order of
development. The format has evolved from an
event-based logging mechanism used in
ALOG,
BLOG, and
CLOG to a
state-based logging mechanism used in
SLOG-1,
which has further evolved to the
drawable-based logging mechanism employed in
SLOG-2.
SLOG
-
A scalable logfile format. SLOG addresses the data-scalability
issue of the logfile for visualization. It is very easy to
generate a large trace file for a simple parallel program. Typical
graphical viewer like Jumpshot-2, which views CLOG file, requires
the whole logfile to be parsed and stored in the physical
memory. This requirement renders the performance of Jumphsot-2
slow or even useless for even moderately sized logfile. SLOG
allows the viewer to read only portion(s) of the logfile for
visualization. One of the main goals of SLOG is to
help users locate interesting portion(s) of the logfile for
analysis.
SLOG-2
-
A
drawable-based logging format. SLOG-2 is the reiteration
of the SLOG effort. Based on the experience while developing
SLOG-1,
SLOG-2 addresses data-scalability issue again by a complete
redesign of the logfile format. The new design allows the
logfile creation to be an one-pass mechanism through elimination
of pseudo-records. This one-pass mechanism not only scalably
reduces the time taken to generate the logfile but also makes
the file creation process much more robust. Except in
well-behaved MPI program (i.e. all communications finish in
relatively short time, few outstanding communications
at any one time during the run), SLOG-1 may require end-users
a vague understanding of SLOG-1 format to do some trial
and error to determine the right parameters for the SLOG-1 file
before file can be generated . Being able to eliminate
pseudo-records, SLOG-2 file can be generated easily without
any end-users' knowledge of the logfile format for almost
any communication patterns. Also, SLOG-2 usually deals with
very large trace files. Therefore, the one-pass logfile
creation mechanism is a very important goal for SLOG-2.
The new design employs the concept of Bounding Box,
often used in graphic design, to solve the data-scalability
problem. SLOG-2 categorizes the objects in the logfile along the
time-axis of the program into a binary tree of the Bounding
Boxes. (The hierachical structure of SLOG-2 is not limited to
binary tree, but binary tree possesses ideal structure for
optimal visual performance.) In addition, the tree structure
of SLOG-2 also allows the logfile to be
renormalized to provide a better
level-of-detail support at the logfile level.
The renormalized object is called
preview drawable
in the SLOG-2 context. These objects reduce the amount
of data processed by the visualization tool at low-resolution
display while still provide an overall high-level
graphical description of whole logfile. The enhanced SLOG-2
format allows tight coupling between the logfile format and the
zoomable graphical user interface offered by
Jumpshot-4.
SLOG-2 is an ongoing research project. An initial
implementation is available in
slog2sdk.
Currently, there are 3 different formats that can be converted
to SLOG-2 format: MPE's CLOG, MPICH2's RLOG and AIX/UTE trace
formats. Anyone who is interested in converting other native
trace formats to SLOG-2 needs to implement TRACE-API specified
in the
SLOG-2 draft document.
So the convertion tool, traceTOslog2, shipped with
slog2sdk can be used to generate SLOG-2 file from the
native trace format. The most updated TRACE-API definitions
can be found in slog2sdk distribution as well.
SLOG-1
-
A
state-based logging format. SLOG-1 is the first iteration
of the SLOG effort. The scalability of SLOG-1 comes
from separating all states in the logfile into continuous
frames of data along the time axis of the program. Each is
small enough to be processed efficiently by the display program,
Jumpshot-3.
States and arrows that cross the frame boundary are duplicated
in each frame that has the partial object. Duplicated states
or arrows are called pseudo-records.
They allow states and arrows to be shown leaving the earlier
frame and then entering into later frame seamlessly when
two adjacent frames of data are compared side by side.
However, creation of pseudo-records also demands the SLOG-1
generation to be a two-pass mechanism.
SLOG-1 and Jumpshot-3 represent a cornerstone in postmortem
performance visualization because they demonstrate the
possibility of visualization of a logfile in the
gigabyte range and still provide reasonable performance
when running the viewer on a desktop PC. To help user to
navigate such a large amount of information, simple
statistics of state activities are collected by SLOG-1 and
displayed by the Preview Window of Jumpshot-3 to enable users
to locate interesting behavior of the parallel program.
SLOG-1 is distributed with
MPE,
which is in turn distributed with
MPICH. SLOG-1
is located under directory mpich/mpe/slog-api.
There is an
API for using SLOG-1. It consists of input and output
APIs for SLOG-1. An example of how to use the API can be
found at mpe/slog-api/src/ts_incrEQendtime_resfnEQyes.c.
CLOG
-
A
event-based logging format. A descendant of ALOG and
BLOG, Its corresponding viewer is
Jumpshot-2.
A more complete description of CLOG can be found in the section
Producing Logfiles of the
jumpshot-2 paper.
The CLOG package is distributed as part of
MPE,
which is also distributed under
MPICH.
BLOG
-
An older
event-based logging format. A descendant of ALOG and
An event-based logging format. An intermediate logfile format
betweem ALOG and CLOG, it was used internally in
PETSc's ntools.
ALOG
-
The original
event-based logging format. A detailed description of ALOG
can be found
here.
Its corresponding viewer is either
Upshot or
Nupshot.
The ALOG package is distributed as part of
MPE,
which is also distributed under
MPICH.
Logging Mechanism
event-based logging
-
Basic logging entity has only one timestamp.
Every incident during the course of the
program is described by one event.
state-based logging
-
Basic logging entity consists of two
timestamps, beginning and ending timestamps.
A state is described by at least two events,
one for beginning event and one for ending
event. Some intermediate events may take place
between the beginning and ending timestamps.
-
drawable-based logging
-
Basic logging entity describes a complete
graphical object. In general, a drawable
is described by one or more state-like
entities, plus some events if necessary.
The concept of renormalization group transformation is often
used in statistical mechanics and quantum field theory.
In physics, renormalization group transformation refers to
coarse-graining transformation followed by a rescaling of
some system components or parameters. Since the creation
of a SLOG-2 tree can be viewed a coarse-graining process, it is
conceivable that rescaling can be done easily.
|