WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows
Publication
, Conference
Xie, B; Cao, Q; Kunjir, M; Wan, L; Chase, J; Mandal, A; Rynge, M
Published in: Proceedings - IEEE International Conference on Cluster Computing, ICCC
January 1, 2021
This paper introduces WIRE that manages resources for the DAG-based workflows on IaaS clouds. WIRE predicts and plans resources over the MAPE (Monitor-AnalyzePlan-Execute) loops to: 1) Estimate task performance with online data, 2) Conduct simulations to predict the upcoming loads based on online estimates and workflow DAGs, 3) Apply a resource-steering policy to size cloud instance pools for the maximal parallelism that is consistent with low cost. We implement WIRE on Pegasus WMS/HTCondor and evaluate its performance on the ExoGENI network cloud. The results show that WIRE attains low resource cost with the performance that is typically within a factor of two of optimal.
Duke Scholars
Published In
Proceedings - IEEE International Conference on Cluster Computing, ICCC
DOI
ISSN
1552-5244
ISBN
9781728196664
Publication Date
January 1, 2021
Volume
2021-September
Start / End Page
35 / 46
Citation
APA
Chicago
ICMJE
MLA
NLM
Xie, B., Cao, Q., Kunjir, M., Wan, L., Chase, J., Mandal, A., & Rynge, M. (2021). WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows. In Proceedings - IEEE International Conference on Cluster Computing, ICCC (Vol. 2021-September, pp. 35–46). https://doi.org/10.1109/Cluster48925.2021.00025
Xie, B., Q. Cao, M. Kunjir, L. Wan, J. Chase, A. Mandal, and M. Rynge. “WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows.” In Proceedings - IEEE International Conference on Cluster Computing, ICCC, 2021-September:35–46, 2021. https://doi.org/10.1109/Cluster48925.2021.00025.
Xie B, Cao Q, Kunjir M, Wan L, Chase J, Mandal A, et al. WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows. In: Proceedings - IEEE International Conference on Cluster Computing, ICCC. 2021. p. 35–46.
Xie, B., et al. “WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows.” Proceedings - IEEE International Conference on Cluster Computing, ICCC, vol. 2021-September, 2021, pp. 35–46. Scopus, doi:10.1109/Cluster48925.2021.00025.
Xie B, Cao Q, Kunjir M, Wan L, Chase J, Mandal A, Rynge M. WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows. Proceedings - IEEE International Conference on Cluster Computing, ICCC. 2021. p. 35–46.
Published In
Proceedings - IEEE International Conference on Cluster Computing, ICCC
DOI
ISSN
1552-5244
ISBN
9781728196664
Publication Date
January 1, 2021
Volume
2021-September
Start / End Page
35 / 46