Assessing observer agreement in studies involving replicated binary observations.

Journal Article

We propose a new approach to evaluating agreement between two observers making binary observations on a set of subjects. This approach compares the probability of disagreement (or discordance) between the observers to the probability of disagreement between replicated observations made by the same observer on the same subject. We consider two situations: (1) a symmetric assessment of agreement between two observers, and (2) an assessment of the agreement of a new observer with an imperfect "gold standard". We develop a nonparametric method for estimation of the new agreement coefficients when observers make replicated readings on each subject. The reliability of the estimation method is examined via a simulation study. Data from a study aimed at determining the validity of diagnosis of breast cancer based on mammograms is used to illustrate the new concepts and methods.

Full Text

Duke Authors

Cited Authors

  • Haber, M; Gao, J; Barnhart, HX

Published Date

  • 2007

Published In

Volume / Issue

  • 17 / 4

Start / End Page

  • 757 - 766

PubMed ID

  • 17613652

International Standard Serial Number (ISSN)

  • 1054-3406

Digital Object Identifier (DOI)

  • 10.1080/10543400701329547

Language

  • eng

Conference Location

  • England