WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows
Conference Paper
This paper introduces WIRE that manages resources for the DAG-based workflows on IaaS clouds. WIRE predicts and plans resources over the MAPE (Monitor-AnalyzePlan-Execute) loops to: 1) Estimate task performance with online data, 2) Conduct simulations to predict the upcoming loads based on online estimates and workflow DAGs, 3) Apply a resource-steering policy to size cloud instance pools for the maximal parallelism that is consistent with low cost. We implement WIRE on Pegasus WMS/HTCondor and evaluate its performance on the ExoGENI network cloud. The results show that WIRE attains low resource cost with the performance that is typically within a factor of two of optimal.
Full Text
Duke Authors
Cited Authors
- Xie, B; Cao, Q; Kunjir, M; Wan, L; Chase, J; Mandal, A; Rynge, M
Published Date
- January 1, 2021
Published In
Volume / Issue
- 2021-September /
Start / End Page
- 35 - 46
International Standard Serial Number (ISSN)
- 1552-5244
International Standard Book Number 13 (ISBN-13)
- 9781728196664
Digital Object Identifier (DOI)
- 10.1109/Cluster48925.2021.00025
Citation Source
- Scopus