Argonne National Laboratory

Characterizing the Performance of Big Memory on Blue Gene Linux

TitleCharacterizing the Performance of Big Memory on Blue Gene Linux
Publication TypeConference Paper
Year of Publication2009
AuthorsYoshii, K, Iskra, K, Broekema, PC, Naik, H, Beckman, PH
Conference NameProceedings of the 2009 International Conference on Parallel Processing Workshops
Date Published08/2009
PublisherIEEE Computer Society
Other NumbersANL/MCS-P1589-0309

Using Linux for high-performance applications on the compute nodes of IBM Blue Gene/P is challenging because of TLB misses and difficulties with programming the network DMA engine. We present a design and implementation of �big memory��an alternative, transparent memory
space for computational processes, which addresses these difficulties. The big memory uses extremely large memory pages available on PowerPC CPUs to create a TLB-missfree, flat memory area that can be used for application code and data and is easier to use for DMA operations. Singlenode benchmarks show that the performance gap narrows from more than a factor of 3 observed with a standard Linux kernel to just 0.03�0.2% with the big memory. We verify this at the scale of 1024 nodes using the NAS Parallel Benchmarks suite, finding the performance under Linux with the big memory support to fluctuate within 0.7% of the vendor microkernel. Originally intended exclusively for compute
node tasks, our new memory subsystem turns out to dramatically improve the performance of certain applications on the I/O nodes as well, as demonstrated by LOFAR.