Cutting corners: Workbench automation for server benchmarking


Conference Paper

© 2008 USENIX Association. All rights reserved. A common approach to benchmarking a server is to measure its behavior under load from a workload generator. Often a set of such experiments is required—perhaps with different server configurations or workload parameters—to obtain a statistically sound result for a given benchmarking objective. This paper explores a framework and policies to conduct such benchmarking activities automatically and efficiently. The workbench automation framework is designed to be independent of the underlying benchmark harness, including the server implementation, configuration tools, and workload generator. Rather, we take those mechanisms as given and focus on automation policies within the framework. As a motivating example we focus on rating the peak load of an NFS file server for a given set of workload parameters, a common and costly activity in the storage server industry. Experimental results show how an automated workbench controller can plan and coordinate the benchmark runs to obtain a result with a target threshold of confidence and accuracy at lower cost than scripted approaches that are commonly practiced. In more complex benchmarking scenarios, the controller can consider various factors including accuracy vs. cost tradeoffs, availability of hardware resources, deadlines, and the results of previous experiments.

Duke Authors

Cited Authors

  • Shivam, P; Marupadi, V; Chase, J; Subramaniam, T; Babu, S

Published Date

  • January 1, 2008

Published In

  • Proceedings of the 2008 Usenix Annual Technical Conference, Usenix 2008

Start / End Page

  • 241 - 254

International Standard Book Number 13 (ISBN-13)

  • 9781931971591

Citation Source

  • Scopus