Adapting Scientific Workflows on Networked Clouds Using Proactive Introspection

Published

Conference Paper

© 2015 IEEE. Recent advances in cloud technologies and on-demand network circuits have created an unprecedented opportunity to enable complex data-intensive scientific applications to run on dynamic, networked cloud infrastructure. However, there is a lack of tools for supporting high-level applications like scientific workflows on dynamically provisioned, virtualized, networked IaaS (NIaaS) systems. In this paper, we propose an end-to-end system consisting of application-aware and application-independent controllers that provision and adapt complex scientific workflows on NIaaS systems. The application-independent controller enhances the utility of NIaaS systems for higher-level applications by closing the gap between application abstractions and resource provisioning constructs. We also present our approach to predicting dynamic resource requirements for workflows using an application-aware controller that proactively evaluates alternative candidate resource allotments using workflow introspection. We show how these high-level resource requirements can be automatically transformed to low-level NIaaS operations to actuate infrastructure adaptation. The results of our evaluations show that we can make fairly accurate predictions, and the interplay of prediction and adaptation can balance performance and utilization for a representative data-intensive workflow.

Full Text

Duke Authors

Cited Authors

  • Mandal, A; Ruth, P; Baldin, I; Xin, Y; Castillo, C; Juve, G; Rynge, M; Deelman, E; Chase, J

Published Date

  • January 1, 2015

Published In

  • Proceedings 2015 Ieee/Acm 8th International Conference on Utility and Cloud Computing, Ucc 2015

Start / End Page

  • 162 - 173

International Standard Book Number 13 (ISBN-13)

  • 9780769556970

Digital Object Identifier (DOI)

  • 10.1109/UCC.2015.32

Citation Source

  • Scopus