WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows

Conference Paper

This paper introduces WIRE that manages resources for the DAG-based workflows on IaaS clouds. WIRE predicts and plans resources over the MAPE (Monitor-AnalyzePlan-Execute) loops to: 1) Estimate task performance with online data, 2) Conduct simulations to predict the upcoming loads based on online estimates and workflow DAGs, 3) Apply a resource-steering policy to size cloud instance pools for the maximal parallelism that is consistent with low cost. We implement WIRE on Pegasus WMS/HTCondor and evaluate its performance on the ExoGENI network cloud. The results show that WIRE attains low resource cost with the performance that is typically within a factor of two of optimal.

Full Text

Duke Authors

Cited Authors

  • Xie, B; Cao, Q; Kunjir, M; Wan, L; Chase, J; Mandal, A; Rynge, M

Published Date

  • January 1, 2021

Published In

Volume / Issue

  • 2021-September /

Start / End Page

  • 35 - 46

International Standard Serial Number (ISSN)

  • 1552-5244

International Standard Book Number 13 (ISBN-13)

  • 9781728196664

Digital Object Identifier (DOI)

  • 10.1109/Cluster48925.2021.00025

Citation Source

  • Scopus