Skip to main content
construction release_alert
Scholars@Duke will be undergoing maintenance April 11-15. Some features may be unavailable during this time.
cancel
Journal cover image

Asymptotic distribution of ∆AUC, NRIs, and IDI based on theory of U-statistics.

Publication ,  Journal Article
Demler, OV; Pencina, MJ; Cook, NR; D'Agostino, RB
Published in: Stat Med
September 20, 2017

The change in area under the curve (∆AUC), the integrated discrimination improvement (IDI), and net reclassification index (NRI) are commonly used measures of risk prediction model performance. Some authors have reported good validity of associated methods of estimating their standard errors (SE) and construction of confidence intervals, whereas others have questioned their performance. To address these issues, we unite the ∆AUC, IDI, and three versions of the NRI under the umbrella of the U-statistics family. We rigorously show that the asymptotic behavior of ∆AUC, NRIs, and IDI fits the asymptotic distribution theory developed for U-statistics. We prove that the ∆AUC, NRIs, and IDI are asymptotically normal, unless they compare nested models under the null hypothesis. In the latter case, asymptotic normality and existing SE estimates cannot be applied to ∆AUC, NRIs, or IDI. In the former case, SE formulas proposed in the literature are equivalent to SE formulas obtained from U-statistics theory if we ignore adjustment for estimated parameters. We use Sukhatme-Randles-deWet condition to determine when adjustment for estimated parameters is necessary. We show that adjustment is not necessary for SEs of the ∆AUC and two versions of the NRI when added predictor variables are significant and normally distributed. The SEs of the IDI and three-category NRI should always be adjusted for estimated parameters. These results allow us to define when existing formulas for SE estimates can be used and when resampling methods such as the bootstrap should be used instead when comparing nested models. We also use the U-statistic theory to develop a new SE estimate of ∆AUC. Copyright © 2017 John Wiley & Sons, Ltd.

Duke Scholars

Altmetric Attention Stats
Dimensions Citation Stats

Published In

Stat Med

DOI

EISSN

1097-0258

Publication Date

September 20, 2017

Volume

36

Issue

21

Start / End Page

3334 / 3360

Location

England

Related Subject Headings

  • Statistics, Nonparametric
  • Statistics & Probability
  • Risk Assessment
  • Reproducibility of Results
  • ROC Curve
  • Models, Statistical
  • Logistic Models
  • Linear Models
  • Humans
  • Computer Simulation
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Demler, O. V., Pencina, M. J., Cook, N. R., & D’Agostino, R. B. (2017). Asymptotic distribution of ∆AUC, NRIs, and IDI based on theory of U-statistics. Stat Med, 36(21), 3334–3360. https://doi.org/10.1002/sim.7333
Demler, Olga V., Michael J. Pencina, Nancy R. Cook, and Ralph B. D’Agostino. “Asymptotic distribution of ∆AUC, NRIs, and IDI based on theory of U-statistics.Stat Med 36, no. 21 (September 20, 2017): 3334–60. https://doi.org/10.1002/sim.7333.
Demler OV, Pencina MJ, Cook NR, D’Agostino RB. Asymptotic distribution of ∆AUC, NRIs, and IDI based on theory of U-statistics. Stat Med. 2017 Sep 20;36(21):3334–60.
Demler, Olga V., et al. “Asymptotic distribution of ∆AUC, NRIs, and IDI based on theory of U-statistics.Stat Med, vol. 36, no. 21, Sept. 2017, pp. 3334–60. Pubmed, doi:10.1002/sim.7333.
Demler OV, Pencina MJ, Cook NR, D’Agostino RB. Asymptotic distribution of ∆AUC, NRIs, and IDI based on theory of U-statistics. Stat Med. 2017 Sep 20;36(21):3334–3360.
Journal cover image

Published In

Stat Med

DOI

EISSN

1097-0258

Publication Date

September 20, 2017

Volume

36

Issue

21

Start / End Page

3334 / 3360

Location

England

Related Subject Headings

  • Statistics, Nonparametric
  • Statistics & Probability
  • Risk Assessment
  • Reproducibility of Results
  • ROC Curve
  • Models, Statistical
  • Logistic Models
  • Linear Models
  • Humans
  • Computer Simulation