What to expect from net reclassification improvement with three categories

Accepted

Journal Article

The net reclassification improvement (NRI) has become a popular measure of incremental usefulness of markers added to risk prediction models. However, the expected magnitude of the three-category NRI is not well understood, leading researchers to rely on statistical significance. In this paper, we describe a slight modification to the original definition of the NRI, which weighs each reclassification by the number of categories by which a given individual is reclassified. This modification resolves some recent criticisms of the three-category NRI and at the same time has a minimal impact on its magnitude. Then we show that using this modified definition, the event and nonevent NRIs have simple interpretations as sums of changes in sensitivities and specificities calculated at the risk thresholds. We exploit this relationship to arrive at closed-form solutions for the NRI under normality within the event and nonevent subgroups. We observe that the size of the intermediate risk category and the event rate have limited impact on the magnitude of the NRI. As expected, the NRI increases with the strength of the added marker, and this relationship appears fairly proportional for markers with non-weak net effect size (above 0.25). Furthermore, we conclude that using the NRI as a metric, it is harder to improve models that already perform well.

Full Text

Duke Authors

Cited Authors

  • Pencina, KM; Pencina, MJ; D'Agostino, RB

Published Date

  • 2014

Published In

Published By

Electronic International Standard Serial Number (EISSN)

  • 1097-0258

International Standard Serial Number (ISSN)

  • 0277-6715

Digital Object Identifier (DOI)

  • 10.1002/sim.6286

Language

  • eng

Citation Source

  • Scopus