Taming Parallel I/O Complexity with Auto-Tuning

TitleTaming Parallel I/O Complexity with Auto-Tuning
Publication TypeConference Paper
Year of Publication2013
AuthorsBehzad, B, Byna, S, Koziol, Q, Luu, HVT, Prabhat, A, Huchette, J, Aydt, R, Snir, M
Conference NameSC2013
Other NumbersANL/MCS-P4091-0713

We present an auto-tuning system for optimizing I/O performance of HDF5 applications and demonstrate its value across platforms, applications, and scale. The system uses a genetic algorithm to search a large space of tunable parameters and to identify effective settings at all layers of the parallel I/O stack. The parameter settings are applied transparently by the auto-tuning system via intercepted HDF5 calls.

To validate our auto-tuning system, we applied it to three I/O benchmarks (VPIC, VORPAL and GCRM) that replicate the I/O activity of their respective applications. We tested the system with different weak-scaling configurations (128, 2048 and 4096 CPU cores) that generate 30 GB to 1 TB of data, and executed these configurations on diverse HPC platforms (Cray XE6, IBM BG/P, and Dell Cluster). In all cases, the auto-tuning framework identified tunable parameters that substantially improved write performance over default system settings. We consistently demonstrate I/O write speedups between 2x and 50x for test configurations.