Oceanography Section
National Center for Atmospheric Research
October, 2002
This manual is a companion reference to the CCSM2.1 POP User's Guide, in which it is intended that those who wish a more complete understanding of the CCSM2.1 POP code will find answers to their questions. This document is in its elementary stages; presently it contains only limited information, such as streamlined CCSM2.1 POP calling trees, various information on the technical aspects of the code, a small table which maps the names of some of the variables used in the CCSM2 POP Scientific Description manual to the variable names used in the actual POP Fortran 90 code, and a new section containing a collection of CCSM POP code requirements. Future releases of this manual will include POP code-interface information generated by the ProTeX utility; requests for other information are welcomed.
The CCSM2.1 POP is based upon POP Version 1.4.3 developed at Los Alamos National Laboratory. Over the course of CCSM2 POP model development, there have been many small changes to the original POP 1.4.3 code in order to allow the code to conform to CCSM requirements. Also, some corrections have been applied, diagnostics have been added, and more substantial modifications have been made to the code to improve the model physics. Wherever possible, a deliberate effort has been made to alter the base POP 1.4.3 code as little as possible, mainly by adding new modules instead of integrating the additions into the original POP code. However, many of the CCSM-related requirements resulted in modifications to numerous existing POP modules.
A simplified directory structure of the CCSM2.1 POP distribution is sketched below. Only those branches directly referenced in this guide appear in the diagram:
ccsm2.1 (main directory)
|
|
+----------- models (models subdirectory)
| |
| |
csm_share ocn (generic ocean and "share
| code" subdirectories)
|
pop (active-ocean component)
|
|
source ---------- input_templates ---------- mpi
| | |
(F90 source (resolution-dependent (MPI-dependent
code) input files) source code)
The CCSM2.1 POP is written in the Fortran 90 programming language. The single reference to a Fortran 95 feature, which was present in the CCSM2.0 POP release, has been eliminated from all releases from CCSM2.0.1 onward. (In CCSM2.0 POP, module types.F used the Fortran 95 feature which allows the initialization of a derived-type component within the derived- type declaration. Many Fortran 90 compilers allow this feature as an extension, but not all of the compilers which CCSM POP was expected to use allowed it, and therefore derived-type component initialization was eliminated.)
The CCSM POP code includes references to subroutines not contained within the CCSM2.1 models/ocn/pop directories. The following sections describe the external software and libraries referenced by the POP code.
The CCSM POP model "uses" (in the Fortran 90 sense) the CCSM shared-code modules that are included in the CCSM2.1 release (see the models/csm_share directory). Usually, developers refer to these as the "share code" modules. The share-code modules presently used in CCSM POP include:
The CCSM POP model has the capability to generate output history files in the netCDF format. (See NetCDF User's Guide for FORTRAN for details.)
At the time the netCDF capability was added to the POP model, the Fortran 90 netCDF interface did not exist; hence the CCSM POP module io_ncdf.F, which contains all of the netCDF-specific code references, contains multiple Fortran 90 interface blocks. All subroutines called by CCSM POP whose names begin with "nf_" are routines from the netCDF library. All POP subroutines that end with "ncdf" are POP subroutines that interface with the netCDF routines.
CCSM2.1 POP intra- and inter-model communications are conducted via MPI. The MPI library is automatically loaded in the CCSM2.1 build procedures.
The main calling tree for the CCSM POP model follows. The model flow at the highest level is quite simple, consisting of a single initialization phase, followed by the main time-stepping loop, and ending with a simple exit routine.
Detailed calling trees for initialize_pop, step, and output_driver follow the main calling tree. These calling trees have been manually "pruned" to eliminate references to the least significant routines, while retaining the more important ones. Furthermore, in cases where there is only one supported CCSM option, references to the other options have been eliminated.
The POP driver routine contains a main time-stepping loop, in which all of the model prognostic variables are advanced by one model timestep. The main work-horse subroutine is subroutine step, which is documented in a subsequent section.
Also in the main time-stepping loop, following the call to step, is the output_driver routine, which controls all model output, determining if and when output is created, and directing where that output should be sent. After time-stepping is completed, the exit_pop routine prints an exit message and exits the message-passing environment.
pop-+-initialize_pop
|
do while loop:
| |
| +-step
| |
| +-output_driver
| |
end do while loop
|
+-exit_pop
initialize_pop | +-init_constants | +-init_grid-----------+-horiz_grid_internal | | | +-read_horiz_grid | | | +-calc_tpoints | | | +-vert_grid_internal | | | +-read_vert_grid | | | +-topography_internal | | | +-read_topography | | | +-remove_points | | | +-smooth_topography | | | +-landmasks | | | +-area_masks | | | +-cf_area_avg | | | +-init_horizontal_mix-+-init_aniso-+-hmix_anis_spvar | | | +-init_smag | | | +-init_gm | +-init_advection | +-init_convection | +-init_baroclinic | +-init_ice | +-init_diagnostics | +-init_forcing--------+-init_ws | | | +-init_shf | | | +-init_sfwf | | | +-init_pt_interior | | | +-init_s_interior | | | +-init_ap | | | +-init_coupled | +-init_budget_diagnostics | +-init_communicate | +-init_io | +-init_time_manager1--+-reset_switches | | | +-prior_days | +-init_state----------+-pressure | +-init_vertical_mix---+-init_vmix_kpp | +-init_operators | +-init_solvers | +-init_pressure_grad | +-init_prognostic | +-init_restart | +-init_ts | +-init_time_manager2--+-get_tday | | | +-write_time_manager_options | +-init_tavg-----------+-read_contents_header | | | +-read_contents-----+-decode_grid | | | +-read_tavg_restart-+-tavg_global_qflux | | | | | +-open_parallel_file | | | | | +-read_array | | | | | +-tavg_exceptions | | | | | +-tavg_global | +-init_output | +-init_topostress-----+-topo_stress | | | +-smooth_topo2 | +-init_ms_balance-----+-init_balancing_regions | +-document_pop | +-pop_warnings
step-+-baroclinic_driver-+-adv_flux
| |
| +-vmix----------+-vmix_coeffs_kpp-+-cfl_vdiff
| | |
| | +-buoydiff-+-state
| | |
| | +-ri_iwmix
| | |
| | +-bldepth--+-state
| | | |
| | | +-sw_absorb_frac
| | | |
| | | +-wscale
| | |
| | +-blmix----+-wscale
| |
| +-tracer_update-+---advt----------+-tavg_free_sfc_resid_flx
| | | |
| | | +-advt_upwind3-+-hupw3
| | | |
| | | +-tavg_advt
| | | |
| | | +-cfl_advect
| | |
| | +-hdifft----------+-hdifft_gm----+-cfl_hdiff
| | | | |
| | | | +-state
| | | |
| | | +-tavg_hdifft
| | |
| | +-vdifft
| |
| +-boundary_2d
| |
| +-state
| |
| +-clinic--------+-advu
| | |
| | +-tavg_gradp
| | |
| | +-hdiffu-+-hdiffu_aniso
| | |
| | +-vdiffu
| |
| +-cfl_check
|
+-barotropic_driver-+-grad
| |
| +-div
| |
| +-pcg
| |
| +-grad
|
+-baroclinic_correct_adjust-+-convad
| |
| +-ice_formation
| |
| +-boundary_2d
| |
| +-boundary_2d
| |
| +-state
| |
| +-tavg_convad
|
+-diag_for_tracer_budgets
|
+-diag_init_sums
|
+-diag_global_preupdate
|
+-diag_global_afterupdate
|
+-diag_print
|
+-diag_transport
|
+-ice_flx_to_coupler--+-tavg_qflux_compute
|
+-set_surface_forcing-+-set_ws
| |
| +-set_coupled_forcing
| |
| +-set_shf
| |
| +-set_sfwf
| |
| +-set_combined_forcing
| |
| +-set_ap
|
+-state
|
+-tracer_budgets
|
+-time_manager--------+-reset_switches--reset_time_flag_all
| |
| +-set_switches
| |
| +-model_date-+-leap_adjust--+-is_leapyear
| | | |
| | | +-prior_days
| | |
| | +-ymd_hms------+-is_near
| | | |
| | | +-hms
| | |
| | +-ymd2eday
| | |
| | +-valid_ymd_hms
| |
| +-get_tday
| |
| +-set_time_flag_all---------+-time_to_do
| |
| +-set_time_flag
|
+-tavg_set_flag
|
+-tavg_2d_compute
|
+-dhdt
|
+-boundary_baroclinic
|
+-boundary_barotropic
output_driver-+-write_history
|
+-write_movie
|
+-write_restart
|
+-write_tavg_dump
Usually, the POP-code variable names are easily matched
with the corresponding names in the scientific equations
found in the
Scientific Description manual,
but there are some associations that may not be immediately
evident. For those variables, we offer the following table, in order
to make the task easier.
| Scientific equation variable name | POP code variable name | Module | Comment |
|---|---|---|---|
| Beddy | vconst_1 | hmix_aniso.F | spatially varying viscosity parameter |
| c2 | vconst_2 | hmix_aniso.F | spatially varying viscosity parameter |
| c3 | vconst_3 | hmix_aniso.F | spatially varying viscosity parameter |
| L-1M | vconst_4 | hmix_aniso.F | spatially varying viscosity parameter |
| N | vconst_5 | hmix_aniso.F | spatially varying viscosity parameter |
| Aeddy | vconst_6 | hmix_aniso.F | spatially varying viscosity parameter |
| CA | c_para | hmix_aniso.F | spatially varying viscosity parameter |
| CB | c_perp | hmix_aniso.F | spatially varying viscosity parameter |
| VA | u_para | LANL POP only | |
| VB | u_perp | LANL POP only | |
| A | visc_para | hmix_aniso.F | lsmag_aniso=.false. |
| B | visc_perp | hmix_aniso.F | lsmag_aniso=.false. |
This section of the Code Reference contains a collection of requirements for the CCSM POP code that have been developed on an as-needed basis. As such, these do not form a complete set of formal requirements for CCSM POP, but instead are a compilation of requirements and constraints as they apply to various new developments in the model.
All component models of the CCSM are required to "restart exactly," by which it is meant that a model, if restarted from model restart files, produces solutions identical to those that would have been produced if the model had instead been run continuously. On most systems with which we have experience, the CCSM POP satisfies this requirement.
At specified intervals, the CCSM POP model creates two restart files: an unformatted binary file which contains all 2- and 3-dimensional model fields required for restart, and a formatted ascii ("header") file which contains a Fortran namelist, including time-manager and precipitation-adjustment information.
On local NCAR machines, this strategy produces a mechanism to exactly restart the CCSM POP model, but on some machines, the information written to the ascii restart header file contains less precision than is available internally to the model. In this situation, CCSM POP model restarts are inexact.
Our strategy to correct this problem is constrained by several
factors:
This section contains information regarding various
CCSM developments made to the POP model. Initially, it
will contain information about only those developments
initiated after the CCSM2.0 release, but it is our goal
to ultimately describe all CCSM POP developements in this
section.
5 CCSM Model Developments
5.1 Restart header files
The following section describes the rationale behind
the changes to the CCSM POP code to avoid the
"exact restart problems"
encountered when running POP on certain machines.
Given the contraints listed in section 4.1, our solution to the exact restart problem is as follows:
if (luse_transfer .and. hdr_arch == runtime_arch
& .and. hdr_arch /= 'unknown' ) then
XXX = transfer (XXX_int, XXX)
endif
integer, parameter :: long_int_kind = selected_int_kind(10)
The CCSM POP read_restart routine will now automatically read the XXX_int form of each namelist variable "XXX", then use the "transfer" function to recreate the exact value for each XXX upon restart. If a user wishes to edit one of the XXX variables, he should do so, and then set the value of luse_transfer to F to disable the use of the transfer function on restart. Presently, the model will either use transfer on all of the XXX_int variables or none of them; the developers have decided that in the rare situation in which a user wishes to edit only some of the XXX variables but preserve the exact values of other XXX variables, the user must edit the restart.F module to suit this specialized case.