Argonne National Laboratory

Advanced Flow-Control Mechanisms for the Sockets Direct Protocol Over InfiniBand

TitleAdvanced Flow-Control Mechanisms for the Sockets Direct Protocol Over InfiniBand
Publication TypeConference Paper
Year of Publication2007
AuthorsBalaji, P, Bhagvat, S, Panda, DK, Thakur, R, Gropp, WD
Conference Name2007 International Conference on Parallel Processing ICPP 2007
Conference LocationXiAn, China
Other NumbersANL/MCS-P1422-0507

The Sockets Direct Protocol (SDP) is an industry standard to allow existing TCP/IP sockets based applications to be executed on high-speed networks such as InfiniBand (IB). Like many other high-speed networks, IB requires the receiver process to inform the network interface card (NIC), before the data arrives, about buffers in which incoming data has to be placed. To ensure that the receiver process is ready to receive data, the sender process typically performs flow-control on the data transmission. Existing designs of SDP flow-control are naive and do not take advantage of several interesting features provided by IB. specifically, features such as RDMA are only used for performing zero-copy communication, although RDMA has more capabilities such as sender-side buffer management (where a sender process can manage SDP resources for the sender as well as the receiver). Similarly, IB also provides hardware flow-control capabilities that have not been studied in previous literature. In this paper, we utilize these capabilities to improve the SDP flow-control over IB using two designs: RDMA-based flow-control and NIC-assisted RDMA-based flow-control. We evaluate the designs using micro-benchmarks and real applications. Our evaluations reveal that these designs can improve the resource usage of SDP and consequently its performance by an order-of-magnitude in some cases. Moreover, we can achieve 10-20% improvement for various applications.