Argonne National Laboratory

High-Performance Serverless Data Transfer over Wide-Area Networks

TitleHigh-Performance Serverless Data Transfer over Wide-Area Networks
Publication TypeConference Paper
Year of Publication2015
AuthorsJung, E, Kettimuthu, R
Conference NameParallel and Distributed Processing Symposium Workshop (IPDPSW)
Date Published05/2015
Conference LocationHyderabad
Other NumbersANL/MCS-P5277-0115
AbstractWe propose a serverless data movement architecture in which we bypass data transfer nodes, the filesystem stack, and the host system stack and directly move the data from one disk array controller to another in order to obtain the highest end-to- end performance. Under the current data movement architecture, separate data transfer nodes arbitrate data transfer by input/out- put to parallel file systems over local-area networks and parallel file systems read/write actual data from/to disks through disk controllers. Our new architecture embeds parallel file system servers and data transfer processes into a disk controller to eliminate the data path between a data transfer process and a parallel file system. This prevents the network between those two entities from being a bottleneck in end-to-end data transfer. In addition, we propose a parallel data-layout-aware data transfer, where multiple embedded data transfer processes send segments of a file concurrently while considering data layout in disks to improve data transfer performance. Our experimental results show that our proposed architecture is feasible and outperforms the traditional architecture significantly.