Synchronization and Ordering Semantics in Hybrid MPI+GPU Programming
|Title||Synchronization and Ordering Semantics in Hybrid MPI+GPU Programming|
|Publication Type||Conference Paper|
|Year of Publication||2013|
|Authors||Aji, AM, Balaji, P, Dinan, J, Feng, W, Thakur, R|
|Conference Name||The Third International Workshop on Accelerators and Hybrid Exascale System, IEEE International Parallel and Distributed Processing Symposium|
|Conference Location||Boston, MA|
Despite the vast interest in accelerator-based sys- tems, programming large multinode GPUs is still a complex task, particularly with respect to optimal data movement across the host-GPU PCIe connection and then across the network. In order to address such issues, GPU-integrated MPI solu- tions have been developed that integrate GPU data movement into existing MPI implementations. Currently available GPU- integrated MPI frameworks differ in aspects related to the buffer synchronization and ordering semantics they provide to users. The noteworthy models are (1) unified virtual addressing (UVA)– based approach and (2) MPI attributes–based approach. In this paper, we compare these approaches, for both programmability and performance, and demonstrate that the UVA-based design is useful for isolated communication with no data dependencies or ordering requirements, while the attributes-based design might be more appropriate when multiple interdependent MPI and GPU operations are interleaved.