Compiler Optimization for Distributed Dynamic Data Flow Programs
|Title||Compiler Optimization for Distributed Dynamic Data Flow Programs|
|Publication Type||Conference Paper|
|Year of Publication||2013|
|Authors||Armstrong, TG, Wozniak, JM, Wilde, M, Foster, IT|
Distributed, dynamic data flow is an execution model well-suited for many large-scale parallel applications, particularly scientific simulations and analysis pipelines running on large, distributed memory clusters. In this paper, we describe compiler optimization techniques and an intermediate representation for distributed dynamic data flow programs. These techniques are applied to Swift/T, a high-level declarative language that allows flexible data flow composition of functions written in other programming languages such as C or Fortran. We show that compiler optimization can reduce communication overhead by 70-93% on distributed memory systems, making the high-level language competitive with hand-coded coordination logic for certain common application styles.