Skip to main content
Journal cover image

Sensitivity and specificity can change in opposite directions when new predictive markers are added to risk models.

Publication ,  Journal Article
Van Calster, B; Steyerberg, EW; D'Agostino, RB; Pencina, MJ
Published in: Med Decis Making
May 2014

When comparing prediction models, it is essential to estimate the magnitude of change in performance rather than rely solely on statistical significance. In this paper we investigate measures that estimate change in classification performance, assuming 2-group classification based on a single risk threshold. We study the value of a new biomarker when added to a baseline risk prediction model. First, simulated data are used to investigate the change in sensitivity and specificity (ΔSe and ΔSp). Second, the influence of ΔSe and ΔSp on the net reclassification improvement (NRI; sum of ΔSe and ΔSp) and on decision-analytic measures (net benefit or relative utility) is studied. We assume normal distributions for the predictors and assume correctly specified models such that the extended model has a dominating receiver operating characteristic curve relative to the baseline model. Remarkably, we observe that even when a strong marker is added it is possible that either sensitivity (for thresholds below the event rate) or specificity (for thresholds above the event rate) decreases. In these cases, decision-analytic measures provide more modest support for improved classification than NRI, even though all measures confirm that adding the marker improved classification accuracy. Our results underscore the necessity of reporting ΔSe and ΔSp separately. When a single summary is desired, decision-analytic measures allow for a simple incorporation of the misclassification costs.

Duke Scholars

Altmetric Attention Stats
Dimensions Citation Stats

Published In

Med Decis Making

DOI

EISSN

1552-681X

Publication Date

May 2014

Volume

34

Issue

4

Start / End Page

513 / 522

Location

United States

Related Subject Headings

  • Risk
  • ROC Curve
  • Models, Theoretical
  • Humans
  • Health Policy & Services
  • Decision Making
  • Coronary Disease
  • Cholesterol, HDL
  • Biomarkers
  • 4206 Public health
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Van Calster, B., Steyerberg, E. W., D’Agostino, R. B., & Pencina, M. J. (2014). Sensitivity and specificity can change in opposite directions when new predictive markers are added to risk models. Med Decis Making, 34(4), 513–522. https://doi.org/10.1177/0272989X13513654
Van Calster, Ben, Ewout W. Steyerberg, Ralph B. D’Agostino, and Michael J. Pencina. “Sensitivity and specificity can change in opposite directions when new predictive markers are added to risk models.Med Decis Making 34, no. 4 (May 2014): 513–22. https://doi.org/10.1177/0272989X13513654.
Van Calster B, Steyerberg EW, D’Agostino RB, Pencina MJ. Sensitivity and specificity can change in opposite directions when new predictive markers are added to risk models. Med Decis Making. 2014 May;34(4):513–22.
Van Calster, Ben, et al. “Sensitivity and specificity can change in opposite directions when new predictive markers are added to risk models.Med Decis Making, vol. 34, no. 4, May 2014, pp. 513–22. Pubmed, doi:10.1177/0272989X13513654.
Van Calster B, Steyerberg EW, D’Agostino RB, Pencina MJ. Sensitivity and specificity can change in opposite directions when new predictive markers are added to risk models. Med Decis Making. 2014 May;34(4):513–522.
Journal cover image

Published In

Med Decis Making

DOI

EISSN

1552-681X

Publication Date

May 2014

Volume

34

Issue

4

Start / End Page

513 / 522

Location

United States

Related Subject Headings

  • Risk
  • ROC Curve
  • Models, Theoretical
  • Humans
  • Health Policy & Services
  • Decision Making
  • Coronary Disease
  • Cholesterol, HDL
  • Biomarkers
  • 4206 Public health