Multi-program benchmark definition

Conference Paper

Although definition of single-program benchmarks is relatively straight-forward-a benchmark is a program plus a specific input-definition of multi-program benchmarks is more complex. Each program may have a different runtime and they may have different interactions depending on how they align with each other. While prior work has focused on sampling multiprogram benchmarks, little attention has been paid to defining the benchmarks in their entirety. In this work, we propose a four-tuple that formally defines multi-program benchmarks in a well-defined way. We then examine how four different classes of benchmarks created by varying the elements of this tuple align with real-world use-cases. We evaluate the impact of these variations on real hardware, and see drastic variations in results between different benchmarks constructed from the same programs. Notable differences include significant speedups versus slowdowns (e.g., +57% vs -5% or +26% vs -18%), and large differences in magnitude even when the results are in the same direction (e.g., 67% versus 11%).

Full Text

Duke Authors

Cited Authors

  • Jacobvitz, AN; Hilton, AD; Sorin, DJ

Published Date

  • April 27, 2015

Published In

  • Ispass 2015 Ieee International Symposium on Performance Analysis of Systems and Software

Start / End Page

  • 72 - 82

International Standard Book Number 13 (ISBN-13)

  • 9781479919567

Digital Object Identifier (DOI)

  • 10.1109/ISPASS.2015.7095786

Citation Source

  • Scopus