Skip to main content

Multi-program benchmark definition

Publication ,  Conference
Jacobvitz, AN; Hilton, AD; Sorin, DJ
Published in: ISPASS 2015 - IEEE International Symposium on Performance Analysis of Systems and Software
April 27, 2015

Although definition of single-program benchmarks is relatively straight-forward-a benchmark is a program plus a specific input-definition of multi-program benchmarks is more complex. Each program may have a different runtime and they may have different interactions depending on how they align with each other. While prior work has focused on sampling multiprogram benchmarks, little attention has been paid to defining the benchmarks in their entirety. In this work, we propose a four-tuple that formally defines multi-program benchmarks in a well-defined way. We then examine how four different classes of benchmarks created by varying the elements of this tuple align with real-world use-cases. We evaluate the impact of these variations on real hardware, and see drastic variations in results between different benchmarks constructed from the same programs. Notable differences include significant speedups versus slowdowns (e.g., +57% vs -5% or +26% vs -18%), and large differences in magnitude even when the results are in the same direction (e.g., 67% versus 11%).

Duke Scholars

Published In

ISPASS 2015 - IEEE International Symposium on Performance Analysis of Systems and Software

DOI

ISBN

9781479919567

Publication Date

April 27, 2015

Start / End Page

72 / 82
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Jacobvitz, A. N., Hilton, A. D., & Sorin, D. J. (2015). Multi-program benchmark definition. In ISPASS 2015 - IEEE International Symposium on Performance Analysis of Systems and Software (pp. 72–82). https://doi.org/10.1109/ISPASS.2015.7095786
Jacobvitz, A. N., A. D. Hilton, and D. J. Sorin. “Multi-program benchmark definition.” In ISPASS 2015 - IEEE International Symposium on Performance Analysis of Systems and Software, 72–82, 2015. https://doi.org/10.1109/ISPASS.2015.7095786.
Jacobvitz AN, Hilton AD, Sorin DJ. Multi-program benchmark definition. In: ISPASS 2015 - IEEE International Symposium on Performance Analysis of Systems and Software. 2015. p. 72–82.
Jacobvitz, A. N., et al. “Multi-program benchmark definition.” ISPASS 2015 - IEEE International Symposium on Performance Analysis of Systems and Software, 2015, pp. 72–82. Scopus, doi:10.1109/ISPASS.2015.7095786.
Jacobvitz AN, Hilton AD, Sorin DJ. Multi-program benchmark definition. ISPASS 2015 - IEEE International Symposium on Performance Analysis of Systems and Software. 2015. p. 72–82.

Published In

ISPASS 2015 - IEEE International Symposium on Performance Analysis of Systems and Software

DOI

ISBN

9781479919567

Publication Date

April 27, 2015

Start / End Page

72 / 82