MPE profiling of CCSM2

Purpose:  To show how cpl5 interacts with the coupled system (for eventual comparison with cpl6.)

Preliminary conclusion:  For this processor configuration, cpl5 is not a bottleneck.

Please send any questions or comments to

The two pictures below are jumpshot-2 plots of a CLOG file created
by placing MPE_Log_event calls in the CCSM2 source code.

MPE_Log_event calls work like a timer.  You make one call to start the timer (start the color bar) and
one call to stop the timer (stop the color bar).    A typical application might look like this:

call MPE_Log_event( start_compute)
......advect, do physics, etc.
call MPE_Log_event(end_compute)
call MPE_Log_event(start_comm)
call send_to_coupler
call MPE_Log_event(end_comm)
call MPE_Log_event(start_compute)

The various start_comm, start_compute variables are integers.
The result is a solid bar which changes color abruptly as the code enters and leaves a state.

Three states of each model are shown:  initialization, computation and communication.

Initialization for all 5 components is given the same color.

Computation is given a different color for each model:  atmosphere, land, river, ocean, ice and coupler.

IMPORTANT:  Communication is showing more than just the time for MPI calls.   It includes
wait time and whatever processing is done to form the message.  A key to what exactly is timed is below.

"Computation" includes everything but the communication routines listed below.

code used:  CCSM2 May 17th release with Patch 1 applied
case:  standard 5 day test run
machine:  blackforest
processor config:  default (68 nodes)
queue: csl_reg

Full 5 day run

Zoom in on 1 day

Key:  1 bar for each MPI process

Top 8 bars:  atmosphere
Next 3 bars:  land and river
Next 16 bars:  ice
Next 40 bars: ocean
bottom bar:  coupler

Communication routines timed (same color used for each side of communication).
c2o:   msg_ocns (cpl5),   recv_from_coupler (pop)
o2c:   msg_ocnr  (cpl5),   send_to_coupler (pop)
c2l:    msg_lnds (cpl5),  csm_recv (clm2)
l2c:    msg_lndr (cpl5),   csm_send (clm2)
c2i:    msg_ices (cpl5),   from_coupler (csim4)
i2c:    msg_icer (cpl5),   to_coupler (csim4)
c2a:   msg_atms (cpl5),   ccsmrcv (cam)
a2c:   msg_atmr (cpl5),   ccsmsnd (cam)

More info:


jumpshot viewer