Argonne National Laboratory

Mega Python: Scalable Interlanguage Scripting for Scientific Computing

TitleMega Python: Scalable Interlanguage Scripting for Scientific Computing
Publication TypeConference Paper
Year of Publication2013
AuthorsWozniak, JM, Armstrong, TG, Krieder, SJ, Maheshwari, K, Wilde, M, Foster, IT
Conference NameSC 13
Conference LocationDenver, CO
Other NumbersANL/MCS-P5020-0913

Scripting languages such as Python and R have received wide adoption as tools for highly productive development of scientific software because of the power and expressiveness of the languages and available libraries. However, deploying scripting languages on large-scale production systems such as the IBM Blue Gene/Q or Cray XE6 is a challenge due to operating systems limitations, interoperability challenges, parallel filesystem overheads due to the small file system accesses common in scripted approaches, and other issues. In this work, we present a new approach based on Swift to integrate high-level languages such as Python, R, Tcl, and the shell with native code developed in C, C++, and Fortran, through the use of the library interfaces to the script systems. In this approach, Swift handles data management, movement, and marshaling among distributed-memory processes without direct user interaction with messaging libraries (such as MPI).