Argonne National Laboratory

Natively Supporting True One-Sided Communication in MPI on Multi-Core Systems with InfiniBand

TitleNatively Supporting True One-Sided Communication in MPI on Multi-Core Systems with InfiniBand
Publication TypeConference Paper
Year of Publication2009
AuthorsSanthanaraman, G, Balaji, P, Gopalakrishnan, K, Thakur, R, Gropp, WD, Panda, DK
Conference NameProceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid
Date Published03/2009
PublisherIEEE Computer Society
Other NumbersANL/MCS-P1598-0409

As high-end computing systems continue to grow in scale, the performance that applications can achieve on such large scale systems
depends heavily on their ability to avoid explicitly synchronized
communication with other processes in the system. Accordingly,
several modern and legacy parallel programming models (such as
MPI, UPC, Global Arrays) have provided many programming constructs that enable implicit communication using one-sided communication
operations. While MPI is the most widely used communication model for scientific computing, the usage of one-sided communication is restricted; this is mainly owing to the inefficiencies in current MPI implementations that internally rely on synchronization between processes even during one-sided communication, thus losing the potential of such constructs. In our previous work, we had utilized native one-sided communication primitives offered by high-speed networks such as InfiniBand (IB) to allow for true one-sided communication inMPI. In this paper, we extend this work to natively take advantage of one-sided atomic operations on cache-coherent multi-core/multi-processor architectures while still utilizing the benefits of networks such as IB. Specifically, we present a sophisticated hybrid design that uses locks that migrate between IB hardware atomics and multi-core CPU atomics to take advantage of both. We demonstrate the capability of our proposed design with a wide range of experiments illustrating its benefits in performance as well as its potential to avoid explicit synchronization.