Skip to main content
Journal cover image

A review and analysis of research on the test-retest reliability of professional judgment

Publication ,  Journal Article
Ashton, RH
Published in: Journal of Behavioral Decision Making
January 1, 2000

This paper analyzes existing research on the test-retest reliability of human judgment, i.e. the extent to which a judge makes identical judgments when presented with identical stimuli on two occasions. Only research involving professional judges who make experimental judgments in a reasonable analog of their everyday experience is included. Studies of both internal consistency reliability and temporal stability reliability are analyzed (where the former refers to the inclusion of repeat stimuli in the same experimental session, and the latter refers to the repeating of the experimental task from a few days to several months later). It is found that (1) the test-retest reliability literature is concentrated in four substantive judgment areas (medicine/psychology, meteorology, human resources management, and business), (2) the literature is extremely variable in terms of research approach/ design, the determinants or correlates of test-retest reliability that have been studied, and the quality of the execution and analysis, and (3) mean test-retest reliability differs across both substantive judgment areas and the internal consistency versus temporal stability distinction. An inescapable conclusion from the analysis is that our knowledge of this fundamental property of human judgment is quite meager. Therefore, the paper concludes with suggestions about future research that would address test-retest reliability more systematically. Copyright © 2000 John Wiley & Sons, Ltd.

Duke Scholars

Altmetric Attention Stats
Dimensions Citation Stats

Published In

Journal of Behavioral Decision Making

DOI

ISSN

0894-3257

Publication Date

January 1, 2000

Volume

13

Issue

3

Start / End Page

277 / 294

Related Subject Headings

  • Social Psychology
  • 5204 Cognitive and computational psychology
  • 3507 Strategy, management and organisational behaviour
  • 3506 Marketing
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Ashton, R. H. (2000). A review and analysis of research on the test-retest reliability of professional judgment. Journal of Behavioral Decision Making, 13(3), 277–294. https://doi.org/10.1002/1099-0771(200007/09)13:3<277::AID-BDM350>3.0.CO;2-B
Ashton, R. H. “A review and analysis of research on the test-retest reliability of professional judgment.” Journal of Behavioral Decision Making 13, no. 3 (January 1, 2000): 277–94. https://doi.org/10.1002/1099-0771(200007/09)13:3<277::AID-BDM350>3.0.CO;2-B.
Ashton RH. A review and analysis of research on the test-retest reliability of professional judgment. Journal of Behavioral Decision Making. 2000 Jan 1;13(3):277–94.
Ashton, R. H. “A review and analysis of research on the test-retest reliability of professional judgment.” Journal of Behavioral Decision Making, vol. 13, no. 3, Jan. 2000, pp. 277–94. Scopus, doi:10.1002/1099-0771(200007/09)13:3<277::AID-BDM350>3.0.CO;2-B.
Ashton RH. A review and analysis of research on the test-retest reliability of professional judgment. Journal of Behavioral Decision Making. 2000 Jan 1;13(3):277–294.
Journal cover image

Published In

Journal of Behavioral Decision Making

DOI

ISSN

0894-3257

Publication Date

January 1, 2000

Volume

13

Issue

3

Start / End Page

277 / 294

Related Subject Headings

  • Social Psychology
  • 5204 Cognitive and computational psychology
  • 3507 Strategy, management and organisational behaviour
  • 3506 Marketing