A new permutation-based method for assessing agreement between two observers making replicated binary readings.

Journal Article (Journal Article)

A new coefficient for assessing agreement between two observers using a permutation-based method is introduced in this article. When observations are binary, this coefficient compares the observed disagreement (probability of discordance) between the observers with its expected value under the hypothesis of individual equivalence. This hypothesis states that for each subject, the conditional distributions of the readings of the two observers are identical, and therefore from a statistical viewpoint it does not matter which observer makes the reading on this subject. Let K and L denote the numbers of replicated observations that are available from observers X and Y, respectively, on a given subject. Then the expected disagreement under individual equivalence for a subject is based on the (K + L) choosing K possible assignments of X's and Y's to the K + L observations made on this subject. Simple methods for the estimation of the new coefficient and its standard error are derived. The new coefficient is compared with kappa and the coefficient of individual agreement, which is based on comparing the inter and intra observer disagreements. Simulation studies confirm the validity of the estimated coefficient and its standard error. Data from a study involving the evaluation of mammograms by 10 radiologists are used to illustrate this new approach to the evaluation of observer agreement.

Full Text

Duke Authors

Cited Authors

  • Pan, Y; Haber, M; Barnhart, HX

Published Date

  • April 15, 2011

Published In

Volume / Issue

  • 30 / 8

Start / End Page

  • 839 - 853

PubMed ID

  • 21432878

Electronic International Standard Serial Number (EISSN)

  • 1097-0258

Digital Object Identifier (DOI)

  • 10.1002/sim.4136


  • eng

Conference Location

  • England