Comparing the performance of statistical methods that generalize effect estimates from randomized controlled trials to much larger target populations

Journal Article (Journal Article)

Policymakers use results from randomized controlled trials to inform decisions about whether to implement treatments in target populations. Various methods—including inverse probability weighting, outcome modeling, and Targeted Maximum Likelihood Estimation—that use baseline data available in both the trial and target population have been proposed to generalize the trial treatment effect estimate to the target population. Often the target population is significantly larger than the trial sample, which can cause estimation challenges. We conduct simulations to compare the performance of these methods in this setting. We vary the size of the target population, the proportion of the target population selected into the trial, and the complexity of the true selection and outcome models. All methods performed poorly when the trial size was only 2% of the target population size or the target population included only 1,000 units. When the target population or the proportion of units selected into the trial was larger, some methods, such as outcome modeling using Bayesian Additive Regression Trees, performed well. We caution against generalizing using these existing approaches when the target population is much larger than the trial sample and advocate future research strives to improve methods for generalizing to large target populations.

Full Text

Duke Authors

Cited Authors

  • Schmid, I; Rudolph, KE; Nguyen, TQ; Hong, H; Seamans, MJ; Ackerman, B; Stuart, EA

Published Date

  • January 1, 2022

Published In

Volume / Issue

  • 51 / 8

Start / End Page

  • 4326 - 4348

Electronic International Standard Serial Number (EISSN)

  • 1532-4141

International Standard Serial Number (ISSN)

  • 0361-0918

Digital Object Identifier (DOI)

  • 10.1080/03610918.2020.1741621

Citation Source

  • Scopus