Below are two snapshots of the tool jumpshot being used to view logs generated from the MPE logging
libraries when they are linked with csmtest. This provides logging of the pure MPI calls only.   Logging other
sections of the code will require adding some lines.

The y axis is processor rank in MPI_COMM_WORLD. In this example

0    = data atmosphere
1    = data ice
2   =  data land
3  =  data ocean
4  =  cpl5
x axis is time in seconds.

This is a zoomed in subsection of the full view.

The width of each bar is the time spent in that instance of the MPI call.

The color indicates which MPI call according to the key at the bottom of the page.
You have to zoom in really close to see the time spent in MPI_ISEND or MPI_BCAST.

The arrows show the origin and destination of messages.

The red lines are un-instrumented sections of the code.  Probably computation or file reading.
 

Jumpshot provides histograms of the various MPI calls by clicking on the state description.
Below is a histogram of  MPI_ISEND