Dataflow Coordination of Data-Parallel Tasks via MPI 3.0

TitleDataflow Coordination of Data-Parallel Tasks via MPI 3.0
Publication TypeConference Paper
Year of Publication2013
AuthorsWozniak, JM, Peterka, T, Armstrong, TG, Dinan, J, Lusk, EL, Wilde, M, Foster, IT
Conference NameEuroMPI'13
Date Published09/2013
Other NumbersANL/MCS-P4067-0413

Scientific applications are often complex collections of many large-scale tasks. Mature tools exist for describing task-parallel workflows consisting of serial tasks, and a variety of tools exist for programming a single data-parallel operation. However, few tools cover the intersection of these two models. In this work, we extend the load balancing library ADLB to support parallel tasks. We demonstrate how applications can easily be composed of parallel tasks using Swift dataflow scripts, which are compiled to ADLB programs with performance comparable to hand-coded equivalents. By combining this framework with data-parallel analysis libraries, we are able to dynamically execute many instances of a parallel data analysis application in support of a parameter exploration workload.