Argonne National Laboratory

Model-Driven Multisite Workflow Scheduling Based on Task-Resource Adaptation

TitleModel-Driven Multisite Workflow Scheduling Based on Task-Resource Adaptation
Publication TypeConference Proceedings
Year of Publication2013
AuthorsMaheshwari, K, Jung, E, Meng, J, Vishwanath, V, Kettimuthu, R
Conference NameIEEE Cluster 2013
Conference LocationIndianapolis, IN
Other NumbersANL/MCS-P4078-0613
AbstractWorkflows continue to play an important role in expressing and deploying scientific applications. In recent years, the number of scientific applications adoption to high-end computing has increased significantly. Moreover, a wide variety of computational sites have emerged with shared access to users such that a user often has access to multiple sites with a limited resource allocation at each site. Because of the scarcity and sparsity in the allocated resources, the user may not be able to complete an entire workflow at a single site. It is thus beneficial to run different tasks of a workflow on different sites. For such cases, judicious scheduling strategy is required in order to map tasks in the workflow to resources at multiple sites so that the workload is balanced among sites and the overhead is minimized in data transfer. The key challenges are that the execution time of a task varies across different sites and the data transfer rate varies based on the network capacity and load. In this paper, we propose a multi-site workflow scheduling technique that tackles the multi-site task distribution challenge by using data movement performance modeling. We applied this technique to schedule an earth observation science workflow over three sites. This approach, executed via the Swift parallel scripting paradigm, augments its default schedule and improves the time-to-schedule by up to 52%.