Argonne National Laboratory

Data-Aware Resource Scheduling for Multicloud Workflows: A Fine-Grained Simulation Approach

TitleData-Aware Resource Scheduling for Multicloud Workflows: A Fine-Grained Simulation Approach
Publication TypeConference Proceedings
Year of Publication2014
AuthorsTang, W, Jenkins, J, Meyer, F, Ross, RB, Kettimuthu, R, Winkler, L, Yang, X, Lehman, T, Desai, NL
Conference NameCloud Computing Technology and Science (CloudCom), 2014 IEEE 6th International Conference
Pagination887 - 892
PublisherIEEE Xplore
Conference LocationSingapore
Other NumbersANL/MCS-P5204-0914
AbstractCloud infrastructures have seen increasing popularity for addressing the growing computational needs of today’s scientific and engineering applications. However, resource management challenges exist in the elastic cloud environment, such as resource provisioning and task allocation, especially when data movement between multiple domains plays an important role. In this work, we study the impact of data-aware resource management and scheduling on scientific workflows in multicloud environments. We develop a workflow simulator based on a network simulation framework for fine-grained simulation for workflow computation and data movement. Using the workload traces from a production metagenomic data analysis service, we evaluate different resource scheduling mechanisms, including proposed data-aware scheduling policies under various resource and bandwidth configurations. The results of this work are expected to answer questions about how to provision computing resources for certain workloads efficiently and how to place tasks across multidomain clouds in order to reduce data movement costs for overall improved system performance.