Synchronization and Ordering Semantics in Hybrid MPI+GPU Programming

TitleSynchronization and Ordering Semantics in Hybrid MPI+GPU Programming
Publication TypeConference Paper
Year of Publication2013
AuthorsAji, AM, Balaji, P, Dinan, J, Feng, W, Thakur, R
Conference NameThe Third International Workshop on Accelerators and Hybrid Exascale System, IEEE International Parallel and Distributed Processing Symposium
Conference LocationBoston, MA
Other NumbersANL/MCS-P5051-1213

Despite the vast interest in accelerator-based sys- tems, programming large multinode GPUs is still a complex task, particularly with respect to optimal data movement across the host-GPU PCIe connection and then across the network. In order to address such issues, GPU-integrated MPI solu- tions have been developed that integrate GPU data movement into existing MPI implementations. Currently available GPU- integrated MPI frameworks differ in aspects related to the buffer synchronization and ordering semantics they provide to users. The noteworthy models are (1) unified virtual addressing (UVA)– based approach and (2) MPI attributes–based approach. In this paper, we compare these approaches, for both programmability and performance, and demonstrate that the UVA-based design is useful for isolated communication with no data dependencies or ordering requirements, while the attributes-based design might be more appropriate when multiple interdependent MPI and GPU operations are interleaved.