Testing the Relative Performance of Data Adaptive Prediction Algorithms: A Generalized Test of Conditional Risk Differences.


Journal Article

Comparing the relative fit of competing models can be used to address many different scientific questions. In classical statistics one can, if appropriate, use likelihood ratio tests and information based criterion, whereas clinical medicine has tended to rely on comparisons of fit metrics like C-statistics. However, for many data adaptive modelling procedures such approaches are not suitable. In these cases, statisticians have used cross-validation, which can make inference challenging. In this paper we propose a general approach that focuses on the "conditional" risk difference (conditional on the model fits being fixed) for the improvement in prediction risk. Specifically, we derive a Wald-type test statistic and associated confidence intervals for cross-validated test sets utilizing the independent validation within cross-validation in conjunction with a test for multiple comparisons. We show that this test maintains proper Type I Error under the null fit, and can be used as a general test of relative fit for any semi-parametric model alternative. We apply the test to a candidate gene study to test for the association of a set of genes in a genetic pathway.

Full Text

Duke Authors

Cited Authors

  • Goldstein, BA; Polley, EC; Briggs, FBS; van der Laan, MJ; Hubbard, A

Published Date

  • May 1, 2016

Published In

Volume / Issue

  • 12 / 1

Start / End Page

  • 117 - 129

PubMed ID

  • 26529567

Pubmed Central ID

  • 26529567

Electronic International Standard Serial Number (EISSN)

  • 1557-4679

Digital Object Identifier (DOI)

  • 10.1515/ijb-2015-0014


  • eng

Conference Location

  • Germany