HPC Successes Featured at SciDAC 2009 Electronic Visualization and Poster Night

June 16, 2009

Argonne showcased select simulations representing high performance computing (HPC) research results at the SciDAC 2009 Electronic Visualization and Poster Night held June 15 at the Sheraton San Diego Hotel & Marina.

The event offered a forum for those involved in Department of Energy (DOE) Office of Science efforts to share their successes visually with colleagues and DOE staff. Participants in DOE's Scientific Discovery through Advanced Computing (SciDAC) and Innovative and Novel Computational Impact on Theory and Experiment (INCITE) programs, as well as core-funded efforts, were encouraged to submit entries. As a benefit to both the research endeavors and program offices, all submissions were given to the program offices for future publicity.

Argonne featured the following eight projects. Two of the simulations won OASCRs at the conference--GCD Model of Type Ia Supernovae for Ignition at Multiple Points [video] and Turbulent Flow of Coolant in an Advanced Recycling Nuclear Reactor. [video](described below).

Searching for Baryon Acoustic Oscillations in Intergalactic Absorption
A research team from the University of California, San Diego used NERSC Seaborg to carry out the largest simulation of the IGM to date. The simulation follows the growth of density perturbations in both gas and dark matter components in a volume 1 billion light years on a side beginning shortly after the Big Bang and evolved to near the present age of the universe, where comparisons with observations are best made. The simulation calculates the gravitational clumping of intergalactic gas and dark matter modeled using a computational grid of 8 billion cells and 8 billion dark matter particles, making it the largest simulation of its kind. The principal scientific goal of this project was to see whether a phenomenon known as baryon acoustic oscillations (BAO) are detectable in absorption of light by the intergalactic medium (IGM). Detection of BAO in the large-scale distribution of galaxies and the IGM at later cosmic times would provide astronomers with a new tool to investigate cosmic dark energy. The visualizations were produced using vl3, Argonne-developed hardware-based volume rendering software.

GCD Model of Type Ia Supernovae for Ignition at Multiple Points
Type Ia supernovae are thought to be white dwarf stars in binary systems that explode due to a thermonuclear runaway. The movie shows a simulation of the gravitationally confined detonation (GCD) model of Type Ia supernovae for multiple ignition points. As the resulting bubbles of hot ash rise, they become Rayleigh-Taylor unstable, initiating a phase of buoyancy-driven turbulent nuclear combustion called the deflagration phase. When the hot ash breaks through the surface of the star, it spreads rapidly across the stellar surface, converges at the opposite point, and produces an inwardly directed jet-like flow that triggers a detonation. The simulation shows that more nuclear burning occurs in the case of multiple ignition points, producing more expansion of the star than in the case of a single ignition point. As a result, less radioactive nickel is produced during the detonation phase, and the explosion is therefore less luminous. The simulation shows that the GCD model can produce a range of peak luminosities.

This simulation won an OASCR at the SciDAC conference. It was computed on the ALCF IBM Blue Gene/P Intrepid and visualized on the ALCF Eureka using Argonne-developed volume rendering software. The visualization was done by Brad Gallagher of the University of Chicago ASC/Alliance Flash Center. The Flash group has a grant of 70 million hours on the Intrepid, from the U.S. Department of Energy's (DOE) Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, to conduct the first rigorous, systematic validation of four current models of the type Ia supernovae. [video]

Cardiac Rhythm Disorders
Catastrophic rhythm disturbances of the heart, including ventricular fibrillation, are a leading cause of death in the United States. Treatment and prevention of cardiac rhythm disorders remain difficult because the electrical signal that controls the heart's rhythm is determined by complex, multi-scale biological processes. Despite recent advances, the mechanism for ventricular fibrillation remains poorly understood. Large-scale computer simulations are testing hypotheses that represent a promising tool to help identify the underlying electrical mechanisms for dangerous arrhythmias and determine the effects of interventions, such as drugs, that may prevent or exacerbate these arrhythmias. This visualization shows front and top views of the simulated electrical wave propagation in a canine heart.

Kv1.2 Potassium Channel
Voltage-gated potassium channels are membrane proteins that open in response to changes in membrane potential and allow passive conduction of potassium ions across the cell membrane. Molecular dynamics (MD) simulations are performed to refine the atomic models of Kv1.2 in the open and closed states in an explicit lipid bilayer environment. The gating charge that is transferred across the membrane upon opening of the channel is calculated from extensive MD simulations of the final models of Kv1.2. The results are in reasonable agreement with experimental values obtained for the channel and quantify the coupling of individually charged residues of the protein to transmembrane potential.

GFDL Prototype Cloud-Resolving Model
Hurricane Emily formed on July 10, 2005 and dissipated on July 21, 2005 in the record-breaking 2005 Atlantic hurricane season. Although it was the strongest hurricane ever to form before August, the size of the storm was relatively small - so small that the Geophysical Fluid Dynamics Laboratories (GFDL) high-resolution 50 km and 25 km atmosphere models developed for simulating the tropical cyclone climatology and hurricane-climate change connections could not resolve the hurricane�s broad structure and failed to predict the intensification. The use of much higher-resolution models is therefore necessary to successfully predict hurricanes of this scale.

The animation for Hurricane Emily demonstrates the hindcasting capabilities of the GFDL's prototype global cloud resolving model at 14 km resolution. The hindcast is a key part of an extensive study to evaluate the prototype cloud resolving model's predictive capabilities with hurricanes with different spatial structures. Hurricanes' inner cores exhibit different spatial scales, from a few to several hundred kilometers. The capability to resolve these features dramatically impacts the ability to forecast the hurricane's track and intensity. The initial conditions for Emily were obtained by combining large-scale nudging using National Centers for Environmental Prediction analysis, with initial intensity and position defined by the International Best Track Archive for Climate Stewardship, using a newly developed 4D-vortex initialization technique.

Galaxy Cluster Merger
Since structure in the universe forms in a bottom-up fashion, with smaller structures merging to form larger ones, modeling the merging process in detail is crucial to an understanding of cosmology. At the current epoch, scientists observed clusters of galaxies undergoing mergers. Two major components of galaxy clusters, the hot intra-cluster gas and dark matter, behave very differently during the course of a merger. Using the N-body and hydrodynamics capabilities in the FLASH code, researchers simulated a suite of representative galaxy cluster mergers, including the dynamics of both the dark matter, which is collision-less, and the gas, which has the properties of a fluid. 3-D visualizations such as these clearly demonstrate the different behavior of these two components over time. The visualization was produced using hardware-based volume rendering software developed at Argonne for the FLASH project.

Large Eddy Simulation of Flow in a Nuclear Reactor
Researchers conducted a large eddy simulation of flow in a reactor subassembly with 37 wire-wrapped pins. This computation was based on 573,480 elements of order 7 (approximately 200 million gridpoints total) and run on 16,384 processors of the IBM Blue Gene/P at the Argonne Leadership Computing Facility by the Nek5000 code and visualized using VisIt. Simulation time was provided through an INCITE award.

Turbulent Flow of Coolant in an Advanced Recycling Nuclear Reactor
Members of Argonne's SHARP team won an OASCR at the SciDAC 2009 conference for the movie "Turbulent Flow of Coolant in an Advanced Recycling Nuclear Reactor." The SHARP project is a collaboration between the Nuclear Engineering and the Mathematics and Computer Science Divisions that is developing high-accuracy simulation tools for reactors. Both the visualization and the runs for the winning entry were done in the Argonne Leadership Computing Facility. The visualization was performed by Hank Childs (of LLNL, now LBL) on Eureka, one of the world's largest graphics processing units, which provides more than 111 teraflops and more than 3.2 terabytes of RAM. The runs were carried out on the IBM Blue Gene/P Intrepid, one of the world's fastest computers. The SHARP team's effort, led by Andrew Siegel of MCS, is supported by a grant of 7.5 million hours on the Intrepid, from the U.S. Department of Energy's Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, to conduct detailed numerical experiments of thermal striping in sodium-cooled fast reactors. The project's results will help scientists better understand the physics of jet mixing in reactor vessels, leading to more optimal designs for future facilities.