Skip to main content
Journal cover image

Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models.

Publication ,  Journal Article
Enserro, DM; Demler, OV; Pencina, MJ; D'Agostino, RB
Published in: Stat Med
September 10, 2019

When comparing performances of two risk prediction models, several metrics exist to quantify prognostic improvement, including the change in the area under the Receiver Operating Characteristic curve, the Integrated Discrimination Improvement, the Net Reclassification Index at event rate, the change in Standardized Net Benefit, the change in Brier score, and the change in scaled Brier score. We explore the behavior and interrelationships between these metrics under multivariate normality in nested and nonnested model comparisons. We demonstrate that, within the framework of linear discriminant analysis, all six statistics are functions of squared Mahalanobis distance, a robust metric that properly measures discrimination by quantifying the separation between the risk scores of events and nonevents. These relationships are important for overall interpretability and clinical usefulness. Through simulation, we demonstrate that the performance of the theoretical estimators under normality is comparable or superior to empirical estimation methods typically used by investigators. In particular, the theoretical estimators for the Net Reclassification Index and the change in Standardized Net Benefit exhibit less variability in their estimates as compared to their empirically estimated counterparts. Finally, we explore how these metrics behave with potentially nonnormal data by applying these methods in a practical example based on the sex-specific cardiovascular disease risk models from the Framingham Heart Study. Our findings aim to give greater insight into the behavior of these measures and the connections existing among them and to provide additional estimation methods with less variability for the Net Reclassification Index and the change in Standardized Net Benefit.

Duke Scholars

Published In

Stat Med

DOI

EISSN

1097-0258

Publication Date

September 10, 2019

Volume

38

Issue

20

Start / End Page

3817 / 3831

Location

England

Related Subject Headings

  • Statistics & Probability
  • Risk Assessment
  • Regression Analysis
  • Prognosis
  • Multivariate Analysis
  • Models, Statistical
  • Humans
  • Computer Simulation
  • 4905 Statistics
  • 4202 Epidemiology
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Enserro, D. M., Demler, O. V., Pencina, M. J., & D’Agostino, R. B. (2019). Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models. Stat Med, 38(20), 3817–3831. https://doi.org/10.1002/sim.8204
Enserro, Danielle M., Olga V. Demler, Michael J. Pencina, and Ralph B. D’Agostino. “Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models.Stat Med 38, no. 20 (September 10, 2019): 3817–31. https://doi.org/10.1002/sim.8204.
Enserro DM, Demler OV, Pencina MJ, D’Agostino RB. Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models. Stat Med. 2019 Sep 10;38(20):3817–31.
Enserro, Danielle M., et al. “Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models.Stat Med, vol. 38, no. 20, Sept. 2019, pp. 3817–31. Pubmed, doi:10.1002/sim.8204.
Enserro DM, Demler OV, Pencina MJ, D’Agostino RB. Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models. Stat Med. 2019 Sep 10;38(20):3817–3831.
Journal cover image

Published In

Stat Med

DOI

EISSN

1097-0258

Publication Date

September 10, 2019

Volume

38

Issue

20

Start / End Page

3817 / 3831

Location

England

Related Subject Headings

  • Statistics & Probability
  • Risk Assessment
  • Regression Analysis
  • Prognosis
  • Multivariate Analysis
  • Models, Statistical
  • Humans
  • Computer Simulation
  • 4905 Statistics
  • 4202 Epidemiology