Argonne National Laboratory

Turbine: A Distributed-Memory Dataflow Engine for Extreme-Scale Many-Task Applications

TitleTurbine: A Distributed-Memory Dataflow Engine for Extreme-Scale Many-Task Applications
Publication TypeConference Paper
Year of Publication2012
AuthorsWozniak, JM, Armstrong, TG, Maheshwari, K, Lusk, EL, Katz, DS, Wilde, M, Foster, IT
Conference NameProceedings SWEET 2012
Date Published05/2012
Conference LocationScottsdale, AZ
Other NumbersANL/MCS-P2057-0312

Efficiently utilizing the rapidly increasing concurrency of multi-petaflop computing systems is a significant program-ming challenge. One approach is to structure applications with an upper-layer of many loosely-coupled coarse-grained tasks, each comprising a tightly coupled parallel function or program. \"Many-task\" programming models such as func-tional parallel dataflow may be used at the upper layer to generate massive numbers of tasks, each of which generates significant tighly-coupled parallelism at the lower level via multithreading, message passing, and/or partitioned global address spaces. At large scales, however, the management of task distribution, data dependencies, and inter-task data movement is a signifi cant performance challenge. In this work, we describe Turbine, a new highly scalable and dis-tributed many-task dataflow engine. Turbine executes a generalized many-task intermediate representation with au-tomated self-distribution, and is scalable to multi-petaflop infrastructures. We present here the architecture of Turbine and its performance on highly concurrent systems.