![Journal cover image](https://secure.syndetics.com/index.aspx?isbn=/lc.gif&issn=0035-9254&client=dukeuniv)
A novel agreement statistic using data on uncertainty in ratings.
Publication
, Journal Article
Zee, J; Mariani, L; Barisoni, L; Mahajan, P; Gillespie, B
Published in: J R Stat Soc Ser C Appl Stat
November 2023
Many existing methods for estimating agreement correct for chance agreement by adjusting the observed proportion agreement by the probability of chance agreement based on different assumptions. These assumptions may not always be appropriate, as demonstrated by pathologists' ratings of kidney biopsy descriptors. We propose a novel agreement statistic that accounts for the empirical probability of chance agreement, estimated by collecting additional data on rater uncertainty for each rating. A standard error estimator for the proposed statistic is derived. Simulation studies show that in most cases, our proposed statistic is unbiased in estimating the probability of agreement after removing chance agreement.
Duke Scholars
Published In
J R Stat Soc Ser C Appl Stat
DOI
ISSN
0035-9254
Publication Date
November 2023
Volume
72
Issue
5
Start / End Page
1293 / 1309
Location
England
Related Subject Headings
- Statistics & Probability
- 4905 Statistics
- 0104 Statistics
Citation
APA
Chicago
ICMJE
MLA
NLM
Zee, J., Mariani, L., Barisoni, L., Mahajan, P., & Gillespie, B. (2023). A novel agreement statistic using data on uncertainty in ratings. J R Stat Soc Ser C Appl Stat, 72(5), 1293–1309. https://doi.org/10.1093/jrsssc/qlad063
Zee, Jarcy, Laura Mariani, Laura Barisoni, Parag Mahajan, and Brenda Gillespie. “A novel agreement statistic using data on uncertainty in ratings.” J R Stat Soc Ser C Appl Stat 72, no. 5 (November 2023): 1293–1309. https://doi.org/10.1093/jrsssc/qlad063.
Zee J, Mariani L, Barisoni L, Mahajan P, Gillespie B. A novel agreement statistic using data on uncertainty in ratings. J R Stat Soc Ser C Appl Stat. 2023 Nov;72(5):1293–309.
Zee, Jarcy, et al. “A novel agreement statistic using data on uncertainty in ratings.” J R Stat Soc Ser C Appl Stat, vol. 72, no. 5, Nov. 2023, pp. 1293–309. Pubmed, doi:10.1093/jrsssc/qlad063.
Zee J, Mariani L, Barisoni L, Mahajan P, Gillespie B. A novel agreement statistic using data on uncertainty in ratings. J R Stat Soc Ser C Appl Stat. 2023 Nov;72(5):1293–1309.
![Journal cover image](https://secure.syndetics.com/index.aspx?isbn=/lc.gif&issn=0035-9254&client=dukeuniv)
Published In
J R Stat Soc Ser C Appl Stat
DOI
ISSN
0035-9254
Publication Date
November 2023
Volume
72
Issue
5
Start / End Page
1293 / 1309
Location
England
Related Subject Headings
- Statistics & Probability
- 4905 Statistics
- 0104 Statistics