Usage Notes for MM90

MM90 is the Fortran-90 implementation of the Massively Parallel Mesoscale Model (MPMM) based on the PSU/NCAR MM5. MM90 provides a number of important improvements over the original MPMM which is based on Fortran-77. Whereas reconfiguration of nesting and physical parameterizations required recompilation with the MPMM code, the MM90 code is entirely run-time configurable. All data structures for model domains are allocated and decomposed at run-time, through settings of namelist variables. Because of the greatly enhanced flexibility for allocating nests, MM90 now also supports multiple nests at each nest level. Nested regions may overlap, but there is no special code yet to handle the overlap regions and overlapping of nests is not, as yet, recommended. Fortran-90's support for true recursion also makes for a much cleaner top-level implementation of the model code that handles the flow of control between nest levels.

The purpose of this document is to provide a brief usage guide to compiling the MM90 code and the R90 library. R90 is the version of RSL tailored to supporting Fortran-90 codes. It handles interprocessor communication, decomposition, and management of distributed execution. Running the MM90 code and in particular the configuration of nesting through namelist settings is discussed.

Platforms

Presently, MM90 and R90 are supported on IBM RS/6000 workstations communicating over MPI and on the IBM SP, using native MPL message passing. Porting to other platforms is limited only by the availability of Fortran-90 compilers and of MPI. It is our intention to port MM90 to the platforms that MPMM currently supports (T3D, Paragon, SGI, Fujitsu).

Language features

MM90 uses several important features of Fortran-90. Fortran-90 provides facilities for creating objects , collections of data structures and routines that act on them. The principal MM90 object is a domain, or model grid. Each grid in the nesting hierarchy is represented as an instance of a domain object. The domain object consists of a derived type (or structure, as the construct is known in C and other languages) to store state data from each model domain plus a number of routines for manipulating domains: allocating domains, initializing domains, and deallocating domains. Each two- and three-dimensional state-array for a given domain is represented as a field in the derived type. The fields themselves are specified as deferred shape arrays; that is, the size of each dimension is not known until the domain is allocated. Thus, different domains can be allocated different sizes and associated with their appropriate parent domains all at run time. Previously, model state data was allocated statically and stored in Fortran COMMON memory, requiring a complete recompile of the MPMM code to reconfigure nesting (the serial model also had this property). The file containing the definition of the MM90 domain object is domains_module.F.

Fortran-90 provides a syntactic nicety called array-syntax. However, this is avoided in MM90 because of demonstrable shortcomings in computational performance with respect to code that uses explicit loops and indices. One exception where array syntax is used in MM90 is in the initialization of an array or array to array copies. However, array syntax is employed only in sections of the code having to do with model or nest initialization that execute only once per model or nest instantiation.

Compiling

As with MPMM, the MM90 code relies on a runtime system library for the lower level tasks of decomposing data over processors, organizing computation on each processor, and implementing the intra- and inter-domain communication between processors. Compiling MM90 involves first compiling the R90 library in a neighboring directory, then compiling MM90 itself (R90 need only be compiled once on installation). The discussion that follows assumes a first time installation of the entire package fresh from the ANL anonymous ftp site.

Downloading MM90

Please contact the author (see below) for download information.

Unpacking the software

Choose a directory where you wish to install MM90 and R90. Typically, they are installed as neighboring directories. Execute the command:

   tar xvf model-xxxx.tar

This will unpack to MM90/mpmm.tar.Z and R90/rsltar.Z.

Building R90

Change directories to the R90 directory and uncompress and unpack from tar archive format the source and makefiles for the R90 library:

   zcat rsltar.Z | tar xvf -

The files of interest are makefile, makefile.mpl, and makefile.ibm. Makefile is the the top level makefile and has a number of configurable settings:

   # largest number of domains
   MAX_DOMAINS  = 10

   # largest number of processors
   MAX_PROC     = 128

These can be reset to reasonably larger values or smaller values without ill effect (some data structures in the RSL library itself will grow or shrink accordingly). If these are changed the library must be made clean (make clean) and recompiled.

The files makefile.mpl and makefile.ibm are invoked by the top level makefile in response to the commands make mpl or make ibm; they should not be invoked directly. There is usually no reason to modify these.

Making R90 produces a library file librsl.a and an include file rsl.inc (which is included by most routines of MM90). Compiling R90 with make mpl produces a version of the code for the IBM SP. Compiling R90 with make ibm produces a version of the code for networks of IBM workstations connected via MPI (MPI can also be used on the SP).

Building MM90

Change directories to the MM90 directory and uncompress and unpack from the tar archive format the source and makefiles for the MM90 code.

   zcat mpmm.tar.Z | tar xvf -

The files of interest are makefile, makefile.mpl, and makefile.ibm.

In the top level makefile, the setting of the make macro RSLLOC should indicate the location of the R90 library directory relative to the MM90 directory.

   RSLLOC =  ../R90

The files makefile.mpl and makefile.ibm are invoked by the top level makefile in response to make mpl or make ibm. New architecture-specific makefiles will appear as the MM90 model is ported to additional platforms. Compiling MM90 with make mpl produces a version of the code for the IBM SP. Compiling MM90 with make ibm produces a version of the code for networks of IBM workstations connected via MPI (MPI can also be used on the SP).

Settable parameters in makefile.mpl and makefile.ibm deal principally with optimization and libraries:

   ###################################################
   # User settable compile time constants.
   ###################################################

   ## compiler/preprocessor optimization flags
   DEBUG=   # -g
   # note -- qhsflt will change answers slightly
   OPTIM=   -O3 -qstrict -qarch=pwrx -qhsflt
   #
   # If you don't have mass lib, comment this out
   # for more info on -lmass, try the following URL or
   # contact IBM through your rep:
   #
   # http://www.austin.ibm.com/tech/MASS
   #
   LIB = $(RSLLIB)/librsl.a -lmass

The macro OPTIM is set to optimization flags to be passed to the compiler. The settings -O3 and -qstrict are the standard optimizations. The setting -qarch=pwrx enables the use of instructions Power2 processor instructions -- use this on the SP2 but not on the SP1. The -qhsflt option instructs the processor to do all internal arithmetic at full precision with truncation to model precision only on stores to memory. This improves performance on IBM processors when running at 32-bit precision but may change answers slightly (within the bounds of floating point rounding error). These compiler options are described in detail in the IBM Fortran Language Users' Manual. To enable compilation for debugging (thus disabling all optimization) set DEBUG to -g. The pound sign (#) in a setting comments out the rest of the line.

The macro LIB must include the R90 library and may also include other libraries: for example, accelerated math intrinsic subroutine libraries such as the IBM MASS library . One usually does not need to include message passing library files in this setting. Systems supporting MPI usually provide special versions of the link editor command, for example mpif77, that automatically link in these libraries. For MPL on the IBM SP the link editor command is mpxlf. Use of these special message-passing layer specific link editing commands is already set in the makefiles.

Running the model

This section provides a general description of the input, output, and run time settings for running the model. Machine specific information is given in subsequent sections.

Input and Output

The model reads in initial data for the top-most domain and, optionally, for the nested domains in the simulation. Nests are initialized from the parent domain if initial data is not provided -- this behavior is controlled by the setting of the namelist variable IOVERW.

Departing from the convention of the source MM5 model and from MPMM, MM90 reads data from named files rather than from default Fortran units. Initial data files are named ic.01, ic.02, .... The number denotes the number of the model domain in the namelist; 01 is always the mother domain.

Also input to the model are lateral boundary conditions which are read only by the mother domain. The name of this file is lbc.01.

Output from each domain of the model is written to the files out.01, out.02, and so on. The frequency of model output is expressed in simulation minutes and specified by the namelist variable TAPFRQ.

The Fortran namelist that is used to configure the model is read from standard Fortran unit 7, which on IBM and many other Unix workstations has the filename fort.7. Note for IBM workstations: MM90 namelists adhere to Fortran-77 style namelist syntax (as opposed the Fortran 90 syntax). Set the environment variable XLFRTEOPTS to namelist=old at the command line to allow the XLF Fortran runtime system to properly read the namelists. Under Unix C-shell ( csh ) the command is:

   setenv XLFRTEOPTS namelist=old

Namelist Settings

This section describes the MM90-specific namelist settings that are used to configure the parallel model at run-time. Most other MM90 namelist variable settings conform with those used by MPMM and are described in

http://www.mcs.anl.gov/~michalak/usage/usage.html.

The ones that differ are described here.

Refer to this sample namelist in the discussion that follows.

 &BASIC
 NPROC_LT = 2,
 NPROC_LN = 2,
 IICE = 1,
 NLEV = 23,
 &END

 &CPARAM
 indir = '.',
 outdir = '/tmp',
 &END

 &OPARAM
 LEVIDN  = 0,1,2,1,1,1,1,1,1,1,
 NUMNC   = 0,1,2,9,9,9,9,9,9,9,
 TAPFRQ = 180.,
 &END

 &LPARAM
 IFRAD     = 2,
 RADFRQ    = 30.,
 ICUSTB    = 1,
 IEXICE    = 0,
 IFDRY     = 0,
 IMVDIF    = 1,
 IBMOIST   = 0,
 ICOR3D    = 1,
 IFUPR     = 0,
 IBOUDY = 3, 2, 2, 2, 2, 2, 2, 2, 2, 2,
 IBLTYP = 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
 IDRY   = 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 IMOIST = 2, 2, 2, 1, 1, 1, 1, 1, 1, 1,
 ICUPA  = 3, 3, 3, 1, 1, 1, 1, 1, 1, 1,
 ISFFLX = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 ITGFLG = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 ISFPAR = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 ICLOUD = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 ICDCON = 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 IFSNOW = 1, 1, 1, 0, 0, 0, 0, 0, 0, 0,
 IMOIAV = 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 IVMIXM = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 HYDPRE = 1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,
 IEVAP  = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 ISHALLO= 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 NESTIX =  61,  61,  46,  46,  46,  46,  46,  46,  46,  46,
 NESTJX =  61,  61,  46,  46,  61,  61,  61,  61,  61,  61,
 NESTI  =   1,  30,  10,  10,   1,   1,   1,   1,   1,   1,
 NESTJ  =   1,  30,  10,  10,   1,   1,   1,   1,   1,   1,
 XSTNES =   0., 0., 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
 XENNES = 2880.,2880.,2880.,720.,720.,720.,720.,720.,720.,
 IOVERW =   1,   0,   0,   0,   0,   0,   0,   0,   0,   0,
 &END

 &PPARAM
 TIMAX  = 360.,
 ZZLND  = 0.1,
 ZZWTR  = 0.0001,
 ALBLND = 0.15,
 THINLD = 0.04,
 XMAVA  = 0.3,
 CONF   = 1.0,
 TISTEP = 303.,
 IFEED  = 4,
 IABSOR = 0,
 &END

 &FPARAM
 FDASTA=0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,
 FDAEND=0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,
 &END

Parallel Execution

The number of processors to decompose the model is specified by two numbers, NPROC_LT and NPROC_LN, the number of processors decomposing latitude and longitude respectively. These MM90 namelist settings replace the processor specification file, fort.50, that was used in MPMM. This is not needed for MM90. The sample namelist above is set to run on four processors.

Other Basic Model Settings

Certain settings that were compile time settings in the source MM5 and in MPMM are now set dynamically, at run time. The number of vertical levels (the same for all domains) is set by the namelist variable NLEV. Mixed phase ice physics is specified by setting the variable IICE to 1. Other settings not shown in the sample above are listed here with their default values:

 IEXMS = 1           ; explicit moisture
 IRDDIM = 1          ; dudhia radiation scheme

Nesting Options

The specification of nests and their hierarchy with respect to the mother domain is given using the namelist variables LEVIDN and NUMNC, which are lists of integers from 1 to an arbitrary maximum of nests (the maximum allowable number is maxnes=10 , set at compile time in the file MM90 source file domains_module.F. If this is increased, also increase the maximum allowable number of domains in R90). LEVIDN specifies the nesting level of the ith domain, starting with the mother domain at level 0. NUMNC specifies the parent of the ith domain by specifying the index of its entry in the list of domains. An entry of '9' in the list disables the domain so that no memory will be allocated.

----------------------------------------------------------------------
BUG -- using 9 as the invalid nest indicator in NUMNC limits the
maximum number of nests to 8. ---  jm 12/14/95
----------------------------------------------------------------------

For example, in the sample namelist above, three domains (a mother and two nests) are specified. Mother domain is the first entry in each list, followed by the the first nest at nest level 1 with the mother as its parent, then followed by the second nest at nest level 2 with the first nest as its parent.

Multiple domains may be specified at each nest level as shown in the following example which specifies a total of four domains: a mother domain, two nests at level 1 and a nest at level 2 with the first level-1 nest as its parent:

 LEVIDN  = 0,1,1,2,1,1,1,1,1,1,
 NUMNC   = 0,1,1,2,9,9,9,9,9,9,

The resulting hierarchy may be expressed:

    1.  mother domain (has no parent so 0 entered in NUMNC).
      2. first nest at level 1 with domain #1 (mother) as parent
        4. nest at level 2 with domain #2 as parent
      3. second nest at level 1 with domain #1 (mother) as parent

Except for the mother domain, which must always be domain number 1, nests may be specified in any order in the list.

The other namelist variables having to do with nesting: NESTIX, NESTJX (domain size), NESTI, NESTJ (nest position), XSTNES, XENNES (start and end times of nest) and IOVERW (initialize from file) are the same as in the source MM5 code and in MPMM. Note: MM90 only allocates memory for a domain while it is active (while the simulation time is between XSTNES and XENNES for a nest).

Input and Output directories

The namelist variables INDIR and OUTDIR specify the directories in which the model should find input files (ic and lbc) and write output, if different from the current working directory (default).

Running under MPL

MM90 compiled for MPL (using makefile.mpl) is run on the IBM SP under the IBM Parallel Operating Environment using the poe command. See your system documentation for additional information on running programs under POE.

Running under MPI

MM90 compiled for MPI is run on the IBM SP or on networks of IBM workstations using the mpirun command. See local system and MPI documentation for additional information.

Credits

John Michalakes, MCS Division, Argonne NL, michalak@mcs.anl.gov
Ian Foster, MCS Division, Argonne NL, foster@mcs.anl.gov
Ken Dritz, MCS Division, Argonne NL, dritz@mcs.anl.gov
Jace Mogill, MCS Division, Argonne NL, mogill@mcs.anl.gov

For more information, please contact:

John Michalakes ,
michalak@mcs.anl.gov, 708-252-6646
MCS Division , Argonne National Laboratory ,
Dec. 1995.

MPMM Home Page