Scalable Data Flow Programming for Many-Task Applications

TitleScalable Data Flow Programming for Many-Task Applications
Publication TypeConference Paper
Year of Publication2012
AuthorsWozniak, JM, Armstrong, TG, Wilde, M, Katz, DS, Lusk, EL, Foster, IT
Other NumbersANL/MCS-P4007-1212

Many important application classes that are driving the requirements for extreme-scale systems branch and bound, stochastic programming, materials by design, uncertainty quantification can be productively expressed as many-task data flow programs. The data flow programming model of the Swift parallel scripting language can elegantly express, through implicit parallelism, the massive concurrency demanded by these applications while retaining the productivity benefits of a high-level language.

However, the centralized single-node evaluation model of the previously developed Swift implementation limits scalability. Overcoming this important limitation is difficult, as evidenced by the absence of any massively-scalable data flow languages in current use. The primary challenge is the efficient integration of highly distributed task load balancing with global access to shared data.