Argonne National Laboratory

Improving I/O Forwarding Throughput with Data Compression

TitleImproving I/O Forwarding Throughput with Data Compression
Publication TypeConference Paper
Year of Publication2011
AuthorsWelton, B, Kimpe, D, Cope, J, Patrick, CM, Iskra, K, Ross, RB
Conference NameProc. 2011 IEEEE International Conference on Cluster Computing (CLUSTER)
Date Published09/2011
Conference LocationAustin, Texas
Other NumbersANL/MCS-P1889-0511

While network bandwidth is steadily increasing, it is doing so at a much slower rate than the corresponding increase in CPU performance. This trend has widened the gap between CPU and network speed. In this paper, we investigate improvements to I/O performance by exploiting this gap. We harness idle CPU resources to compress network data, reducing the amount of data transferred over the network and increasing effective network bandwidth. We created a set of data compression services within the I/O Forwarding Scalability Layer. These services transparently compress and decompress data as it is transferred over the network. We studied the effect of the data compression services on a variety of data sets and conducted experiments on a high-performance computing cluster.