Parallel I/O Benchmarking Consortium

Introduction

Parallel I/O performance remains a weak point in most parallel computing systems in use today. While research continues on parallel I/O solutions, results that apply to real systems and important application workloads are few and far between.

One way that the community can as a whole make progress in this direction is through standardizing how we test systems. By testing in the same manner, comparing performance results from different sources (both applications and systems) is more meaningful.

A second way that the community can progress towards more effective improvements is through identifying important workloads and providing test applications that benchmark performance for these workloads. There is no single workload that is "important" with respect to every system; however, by creating or gathering a collection of test codes and clearly describing what patterns these test codes exhibit, we can provide the community as a whole with a set of reusable benchmarks that can serve as a starting point for performance studies of new solutions.

Purpose

The purpose of the Parallel I/O Benchmarking Consortium is to provide a set of benchmarks for use in characterizing the performance of a parallel I/O system. A concise set will be used for obtaining an "at a glance" view of I/O system performance, and an extended set will be provided for discovering regions of poor performance within a large space of access characteristics. These tests will follow a consistent testing methodology, defined by the consortium, to help with comparing results across tests.

The test will not attempt to measure correctness of the underlying I/O system implementation, nor will it attempt to "score" the I/O system to aid in generation of a "Top 500 I/O systems" or the like.

We want to get everyone in the community involved. If you are interested, please join our mailing list and make comments on the document and/or test codes. If you have a test code that you would like to see become a part of the test suite, please let us know. It is only though the involvement of a wide range of groups that this can be a success.

Documents

The document describing the consortium's work is available in PDF format.

Test Suite

A small but growing collection of good parallel I/O benchmarks:

Project-hosted Tests:

Application I/O kernels:

I/O kernels capture the I/O workload of a given application without all the associated science. Tuning I/O kernels should have a direct impact on overall application performance.

An updated version of the FLASH-IO benchmark. Algorithmically identical to the one below. Contains updates for newer versions of Parallel-NetCDF and HDF5: flash-argonne-20060913
An adaptive-mesh refinement code from NERSC: Chombo I/O
I/O kernel for The MADspec astronomy code: MADbench2
An I/O kernel for the combustion code S3D: S3D-IO
The POP ocean model kernel: POP-IO
A fusion code: GTC-IO

Other Tests:

The FLASH-I/O kernel: official version http://flash.uchicago.edu/~jbgallag/io_bench/flash_io_bench.tar.gz
The LLNL Scalable I/O Project Downloads. LLNL hosts a collection of IO benchmarks used for acceptance testing on their clusters.
b_eff_io measures performance for quite a few access patterns.
Northwestern's Avery Ching has two good benchmarks: S3asim and HPIO

Other Resources:

Joachim Worringen's Perfbase (perfbase.tigris.org) can be helpful in tracking and analyzing benchmark results. He actively seeks out users and feedback, so give it a shot and let him know how well it works for you.

Mailing List

A mailing list has been created for discussion of parallel I/O benchmarking consortium topics. The list can be joined by mailing to [email protected] with the subject "subscribe pio-benchmark".