Removing the influence of group variables in high-dimensional predictive modelling.

Journal Article (Journal Article)

In many application areas, predictive models are used to support or make important decisions. There is increasing awareness that these models may contain spurious or otherwise undesirable correlations. Such correlations may arise from a variety of sources, including batch effects, systematic measurement errors, or sampling bias. Without explicit adjustment, machine learning algorithms trained using these data can produce poor out-of-sample predictions which propagate these undesirable correlations. We propose a method to pre-process the training data, producing an adjusted dataset that is statistically independent of the nuisance variables with minimum information loss. We develop a conceptually simple approach for creating an adjusted dataset in high-dimensional settings based on a constrained form of matrix decomposition. The resulting dataset can then be used in any predictive algorithm with the guarantee that predictions will be statistically independent of the group variable. We develop a scalable algorithm for implementing the method, along with theory support in the form of independence guarantees and optimality. The method is illustrated on some simulation examples and applied to two case studies: removing machine-specific correlations from brain scan data, and removing race and ethnicity information from a dataset used to predict recidivism. That the motivation for removing undesirable correlations is quite different in the two applications illustrates the broad applicability of our approach.

Full Text

Duke Authors

Cited Authors

  • Aliverti, E; Lum, K; Johndrow, JE; Dunson, DB

Published Date

  • July 2021

Published In

Volume / Issue

  • 184 / 3

Start / End Page

  • 791 - 811

PubMed ID

  • 35755858

Pubmed Central ID

  • PMC9221581

Electronic International Standard Serial Number (EISSN)

  • 1467-985X

International Standard Serial Number (ISSN)

  • 0964-1998

Digital Object Identifier (DOI)

  • 10.1111/rssa.12613


  • eng