Argonne National Laboratory

Cache-Efficient, Intranode, Large-Message MPI Communication with MPICH2-Nemesis

TitleCache-Efficient, Intranode, Large-Message MPI Communication with MPICH2-Nemesis
Publication TypeConference Paper
Year of Publication2009
AuthorsBuntinas, D, Goglin, B, Goodell, D, Mercier, G, Moreaud, S
Conference NameProceedings of the 2009 International Conference on Parallel Processing
Date Published04/2009
PublisherIEEE Computer Society
Other NumbersANL/MCS-P1620-0509

The emergence of multicore processors raises the need to efficiently transfer large amounts of data between local processes. MPICH2 is a highly portable MPI implementation whose large-message communication schemes suffer from high CPU utilization and cache pollution because of the use of a double-buffering strategy, common to many MPI implementations. We introduce two strategies offering a kernel- assisted, single-copy model with support for noncontiguous and asynchronous transfers. The first one uses the now widely available vmsplice Linux system call; the second one further improves performance thanks to a custom kernel module called KNEM. The latter also offers I/OAT copy offload, which is dynamically enabled depending
on both hardware cache characteristics and message size. These new solutions outperform the standard transfer method in the MPICH2 implementation when no cache is shared between the processing cores or when very large messages are being transferred. Collective communication operations show a dramatic improvement, and the IS NAS
parallel benchmark shows a 25% speedup and better cache efficiency.