Evaluation criteria for human-automation performance metrics


Book Section

Previous research has identified broad metric classes for humanautomation performance in order to facilitate metric selection, as well as understanding and comparing research results. However, there is still a lack of a systematic method for selecting the most efficient set of metrics when designing evaluation experiments. This chapter identifies and presents a list of evaluation criteria that can help determine the quality of a metric in terms of experimental constraints, comprehensive understanding, construct validity, statistical efficiency, and measurement technique efficiency. Based on the evaluation criteria, a comprehensive list of potential metric costs and benefits is generated. The evaluation criteria, along with the list of metric costs and benefits, and the existing generic metric classes provide a foundation for the development of a cost-benefit analysis approach that can be used for metric selection. © 2009 Springer Science+Business Media, LLC.

Full Text

Duke Authors

Cited Authors

  • Donmez, B; Pina, PE; Cummings, ML

Published Date

  • December 1, 2009

Book Title

  • Performance Evaluation and Benchmarking of Intelligent Systems

Start / End Page

  • 21 - 40

International Standard Book Number 13 (ISBN-13)

  • 9781441904911

Digital Object Identifier (DOI)

  • 10.1007/978-1-4419-0492-8_2

Citation Source

  • Scopus