Efficient High-Order Time-Integrators for Local High-Order Discretization Methods

Synopsis: This project is concerned with the development of high-order accurate and highly efficient time-integration methods for local high-order spatial discretizations of hyperbolic partial differential equations typically found in fluid flow problems. The focus is on the numerical solution of the shallow water, Euler, and compressible Navier-Stokes equations. For this class of differential equations, local high-order methods such as finite volumes, finite elements, and finite differences have shown to give excellent solution accuracy while scaling well on the current class of multi-core parallel computers. These efforts will be implemented and tested on the Non-hydrostatic Unified Model of the Atmosphere (NUMA)

Simulation of the rising thermal bubble hitting the top wall:

Papers, Conference Presentations, and Proceedings

• Francis X Giraldo, James F. Kelly, and Emil Constantinescu, "Implicit-explicit formulations of a three-dimensional nonhydrostatic unified model of the atmosphere (NUMA)." SIAM Journal on Scientific Computing (SISC), submitted, Preprint # ANL/MCS-P2083-0512, 2012.

   

Building NUMA3dCG with PETSc:

Configure & build PETSc with debug options and optimized (gFortran on Mac):

Downloading PETSc:

[w/ sshkey] git clone git@bitbucket.org:petsc/petsc.git petsc-dev
[w/o ssh key] git clone https://bitbucket.org/petsc/petsc petsc-dev

Building PETSc:

cd petsc-dev

Building NetCDF's Fortran API:

<move to the NUMA repository and get netcdf-fortran-xx download it, and read instructions or follow>
[e.g.] wget http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-fortran-4.2.tar.gz
tar -xzvf netcdf-fortran-4.2.tar.gz
cd netcdf-fortran-4.2
export FC=$PETSC_DIR/$PETSC_ARCH/bin/mpif90
export CC=$PETSC_DIR/$PETSC_ARCH/bin/mpicc
export LD_LIBRARY_PATH=$PETSC_DIR/$PETSC_ARCH/lib:$LD_LIBRARY_PATH
export CPPFLAGS="-DgFortran -I$PETSC_DIR/$PETSC_ARCH/include"
export LDFLAGS=-L$PETSC_DIR/$PETSC_ARCH/lib
./configure --prefix=$PETSC_DIR/$PETSC_ARCH

make
make check
make install

Building NUMA:

<move to numa3dCG_PETSc>

<define Metis and NetCDF paths e.g., in .bashrc>
export METIS_DIR=$PETSC_DIR/$PETSC_ARCH
export NETCDF_DIR=$PETSC_DIR/$PETSC_ARCH

make mcs_petsc_debug

<make sure /lib was created otherwise create it and re-run make>

Running NUMA:

<here's a config file used in preliminary testing for case 9 numa3dCG.in and for case 21 numa3dCG.in>

<here's an example run command: (from the run folder) >

$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -np 6 ../bin/numa3dCG -{ts,ksp,snes}_monitor -ts_adapt_type none -use-petsc true -snes_mf

[Top]

Configure & build PETSc with debug options (gFortran on Linux):

Downloading PETSc:

[w/ sshkey] git clone git@bitbucket.org:petsc/petsc.git petsc-dev
[w/o ssh key] git clone https://bitbucket.org/petsc/petsc petsc-dev

Building PETSc:

cd petsc-dev

Building NetCDF's Fortran API:

<move to the NUMA repository and get netcdf-fortran-xx download it, and read instructions or follow>
[e.g.] wget http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-fortran-4.2.tar.gz
tar -xzvf netcdf-fortran-4.2.tar.gz
cd netcdf-fortran-4.2
export FC=$PETSC_DIR/$PETSC_ARCH/bin/mpif90
export CC=$PETSC_DIR/$PETSC_ARCH/bin/mpicc
export LD_LIBRARY_PATH=$PETSC_DIR/$PETSC_ARCH/lib:$LD_LIBRARY_PATH
export CPPFLAGS="-DgFortran -I$PETSC_DIR/$PETSC_ARCH/include"
export LDFLAGS=-L$PETSC_DIR/$PETSC_ARCH/lib
./configure --prefix=$PETSC_DIR/$PETSC_ARCH

make
make check
make install

Building NUMA:

<move to numa3dCG_PETSc>

<define Metis and NetCDF paths e.g., in .bashrc>
export METIS_DIR=$PETSC_DIR/$PETSC_ARCH
export NETCDF_DIR=$PETSC_DIR/$PETSC_ARCH

make mcs_petsc_debug

<make sure /lib was created otherwise create it and re-run make>

Running NUMA:

<here's a config file used in preliminary testing for case 9 numa3dCG.in and for case 21 numa3dCG.in>

<here's an example run command: (from the run folder) >

$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -np 6 ../bin/numa3dCG -{ts,ksp,snes}_monitor -ts_adapt_type none -use-petsc true -snes_mf

Configure & build PETSc with debug options for the Fusion machine:

edit .soft to include:
+cmake
+mercurial
+git
@default

./configure --PETSC_ARCH=arch-linux2-c-debug --with-hdf5-include=/soft/hdf5/1.8.6-parallel/include --with-hdf5-lib="-L/soft/hdf5/1.8.6-parallel/lib -lhdf5_hl -lhdf5 -lgpfs" --with-mpi-dir=/soft/mvapich2/1.4.1-intel-11.1.064 --with-shared-libraries=0 --with-netcdf-include=/soft/netcdf/4.1.1-parallel/include --with-netcdf-lib="-L/soft/netcdf/4.1.1-parallel/lib -lnetcdff -lnetcdf -L/usr/kerberos/lib64 -lcurl -ldl -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lidn -lssl -lcrypto -lz" --download-metis=yes

make PETSC_DIR=$PWD PETSC_ARCH=arch-linux2-c-debug all test

<move to numa3dCG_PETSc>

make mcs_fusion_petsc_debug

<make sure /lib was created otherwise create it and re-run make>

<here's a config file used in preliminary testing numa3dCG.in>

Configure & build PETSc with debug options for the Beagle machine:

module swap PrgEnv-cray PrgEnv-pgi;
module swap PrgEnv-pgi PrgEnv-cray;
module load cmake

./configure --PETSC_ARCH=arch-cray-c-debug --download-metis=yes --with-cc=cc --with-clanguage=C++ --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-fc=ftn --with-fortranlib-autodetect=0 --with-x=0 --FFLAGS='-F -em' --LIBS='-L/opt/cray/cce/8.1.4/CC/x86-64/lib/x86-64/ -lcray-c++-rts -lcraystdc++ -lsupc++ -lgcc_eh' --with-shared-libraries=0

make PETSC_DIR=$PWD PETSC_ARCH=arch-cray-c-debug all test