Argonne National Laboratory

Minimizing MPI Resource Contention in Multithreaded Multicore Environments

TitleMinimizing MPI Resource Contention in Multithreaded Multicore Environments
Publication TypeConference Paper
Year of Publication2010
AuthorsGoodell, D, Balaji, P, Buntinas, D, Dozsa, G, Gropp, WD, Kumar, S, de Supinski, BR, Thakur, R
Conference NameProc. IEEE International Conf. on Cluster Computing
Date Published09/2010
Conference LocationHeraklion, Greece

With the ever-increasing numbers of cores per node in high-performance computing systems, a growing number of applications are using threads to exploit shared memory within a node and MPI across nodes. This hybrid programming model needs efficient support for multithreaded MPI communication. In this paper, we describe the optimization of one aspect of a multithreaded MPI implementation: concurrent accesses from multiple threads to various MPI objects, such as communicators, datatypes, and requests. The semantics of the creation, usage, and destruction of these objects implies, but does not strictly require, the use of reference counting to prevent memory leaks and premature object destruction. We demonstrate how a naive multithreaded implementation of MPI object management via reference counting incurs a significant performance penalty. We then detail two solutions that we have implemented in MPICH2 to mitigate this problem almost entirely, including one based on a novel garbage collection scheme. In our performance experiments, this new scheme improved the multithreaded messaging rate by as much as 31% over the naive reference counting method.