News & Announcements

September 26, 2011

"Enabling parallel access to multiresolution scientific datasets"

A popular format known as IDX is used to enable access to scientific datasets. IDX stores the data in a hierarchical order; the results can then be visualized interactively on systems ranging from desktops and laptop computers to portable devices such as iPhones. However, the IDX application programmer interface (API) is serial, thus limiting the use to relatively small datasets.

To address this limitation, researchers in Argonne National Laboratory’s Mathematics and Computer Science Division, together with colleagues from the University of Utah, Sandia National Laboratories, and the National Renewable Energy Laboratory, have developed PIDX—a parallel API for writing data in IDX format. Using PIDX, researchers can generate IDX datasets directly from large-scale scientific simulations. An added advantage is that PIDX enables real-time monitoring and visualization of the generated data.

The research team recently demonstrated the efficacy of PIDX in a real-world scientific simulation of turbulent combustion. On 8,192 cores of the Hopper2 supercomputer at the National Energy Research Scientific Computing center, PIDX achieved up to 18 GiB/s. Moreover, the researchers created an end-to-end pipeline using ViSUS, which allows remote visualization of the data. Using this pipeline, a scientist can monitor the progress of a simulation and steer or correct it as unforeseen events arise. This work was presented at IEEE Cluster 2011, held in Austin, Texas, Sept. 26–30.