Both systems described here are log-based systems. That is, during the run of the parallel program, time-stamped data is collected and then interpreted offline. Although the immediacy of a real-time display is lost, more is gained in the ability to examine the sequence of events that occurred in particular run at leisure, in detail, and repeatedly.
The precise nature of the events to be logged is still very much a matter for debate. Different systems will have different ``critical'' system events, and of course different user programs will have different events altogether. Logging of all low-level events, such as all locking and unlocking operations, or all messages sent and received, while sometimes useful, may swamp the logging mechanism without providing real insight into parallel program behavior. Flexibility in specifying which events are to be logged is crucial.
Many researchers are taking up these challenges. One of the most advanced systems in this category is Paragraph [Heath and Etheridge 1991], a logfile display program developed at Oak Ridge National Laboratory. In general Paragraph provides more views of logfile information than the tools described here, although these systems provide more depth in the views they do provide.