Skip to main content
Journal cover image

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.

Publication ,  Journal Article
Cramer, EY; Ray, EL; Lopez, VK; Bracher, J; Brennen, A; Castro Rivadeneira, AJ; Gerding, A; Gneiting, T; House, KH; Huang, Y; Jayawardena, D ...
Published in: Proc Natl Acad Sci U S A
April 12, 2022

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.

Duke Scholars

Altmetric Attention Stats
Dimensions Citation Stats

Published In

Proc Natl Acad Sci U S A

DOI

EISSN

1091-6490

Publication Date

April 12, 2022

Volume

119

Issue

15

Start / End Page

e2113561119

Location

United States

Related Subject Headings

  • United States
  • Public Health
  • Probability
  • Pandemics
  • Humans
  • Forecasting
  • Data Accuracy
  • COVID-19
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Cramer, E. Y., Ray, E. L., Lopez, V. K., Bracher, J., Brennen, A., Castro Rivadeneira, A. J., … Reich, N. G. (2022). Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States. Proc Natl Acad Sci U S A, 119(15), e2113561119. https://doi.org/10.1073/pnas.2113561119
Cramer, Estee Y., Evan L. Ray, Velma K. Lopez, Johannes Bracher, Andrea Brennen, Alvaro J. Castro Rivadeneira, Aaron Gerding, et al. “Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.Proc Natl Acad Sci U S A 119, no. 15 (April 12, 2022): e2113561119. https://doi.org/10.1073/pnas.2113561119.
Cramer EY, Ray EL, Lopez VK, Bracher J, Brennen A, Castro Rivadeneira AJ, et al. Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States. Proc Natl Acad Sci U S A. 2022 Apr 12;119(15):e2113561119.
Cramer, Estee Y., et al. “Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.Proc Natl Acad Sci U S A, vol. 119, no. 15, Apr. 2022, p. e2113561119. Pubmed, doi:10.1073/pnas.2113561119.
Cramer EY, Ray EL, Lopez VK, Bracher J, Brennen A, Castro Rivadeneira AJ, Gerding A, Gneiting T, House KH, Huang Y, Jayawardena D, Kanji AH, Khandelwal A, Le K, Mühlemann A, Niemi J, Shah A, Stark A, Wang Y, Wattanachit N, Zorn MW, Gu Y, Jain S, Bannur N, Deva A, Kulkarni M, Merugu S, Raval A, Shingi S, Tiwari A, White J, Abernethy NF, Woody S, Dahan M, Fox S, Gaither K, Lachmann M, Meyers LA, Scott JG, Tec M, Srivastava A, George GE, Cegan JC, Dettwiller ID, England WP, Farthing MW, Hunter RH, Lafferty B, Linkov I, Mayo ML, Parno MD, Rowland MA, Trump BD, Zhang-James Y, Chen S, Faraone SV, Hess J, Morley CP, Salekin A, Wang D, Corsetti SM, Baer TM, Eisenberg MC, Falb K, Martin ET, McCauley E, Myers RL, Schwarz T, Sheldon D, Gibson GC, Yu R, Gao L, Ma Y, Wu D, Yan X, Jin X, Wang Y-X, Chen Y, Guo L, Zhao Y, Gu Q, Chen J, Wang L, Xu P, Zhang W, Zou D, Biegel H, Lega J, McConnell S, Nagraj VP, Guertin SL, Hulme-Lowe C, Turner SD, Shi Y, Ban X, Walraven R, Hong Q-J, Kong S, van de Walle A, Turtle JA, Ben-Nun M, Riley S, Riley P, Koyluoglu U, DesRoches D, Forli P, Hamory B, Kyriakides C, Leis H, Milliken J, Moloney M, Morgan J, Nirgudkar N, Ozcan G, Piwonka N, Ravi M, Schrader C, Shakhnovich E, Siegel D, Spatz R, Stiefeling C, Wilkinson B, Wong A, Cavany S, España G, Moore S, Oidtman R, Perkins A, Kraus D, Kraus A, Gao Z, Bian J, Cao W, Lavista Ferres J, Li C, Liu T-Y, Xie X, Zhang S, Zheng S, Vespignani A, Chinazzi M, Davis JT, Mu K, Pastore Y Piontti A, Xiong X, Zheng A, Baek J, Farias V, Georgescu A, Levi R, Sinha D, Wilde J, Perakis G, Bennouna MA, Nze-Ndong D, Singhvi D, Spantidakis I, Thayaparan L, Tsiourvas A, Sarker A, Jadbabaie A, Shah D, Della Penna N, Celi LA, Sundar S, Wolfinger R, Osthus D, Castro L, Fairchild G, Michaud I, Karlen D, Kinsey M, Mullany LC, Rainwater-Lovett K, Shin L, Tallaksen K, Wilson S, Lee EC, Dent J, Grantz KH, Hill AL, Kaminsky J, Kaminsky K, Keegan LT, Lauer SA, Lemaitre JC, Lessler J, Meredith HR, Perez-Saez J, Shah S, Smith CP, Truelove SA, Wills J, Marshall M, Gardner L, Nixon K, Burant JC, Gu Z, Kim M, Li X, Wang G, Yu S, Reiner RC, Barber R, Gakidou E, Hay SI, Lim S, Murray C, Pigott D, Gurung HL, Baccam P, Stage SA, Suchoski BT, Prakash BA, Adhikari B, Cui J, Rodríguez A, Tabassum A, Xie J, Keskinocak P, Asplund J, Baxter A, Oruc BE, Serban N, Arik SO, Dusenberry M, Epshteyn A, Kanal E, Le LT, Li C-L, Pfister T, Sava D, Sinha R, Tsai T, Yoder N, Yoon J, Zhang L, Abbott S, Bosse NI, Funk S, Hellewell J, Meakin SR, Sherratt K, Zhou M, Kalantari R, Yamana TK, Pei S, Shaman J, Li ML, Bertsimas D, Skali Lami O, Soni S, Tazi Bouardi H, Ayer T, Adee M, Chhatwal J, Dalgic OO, Ladd MA, Linas BP, Mueller P, Xiao J, Wang Q, Xie S, Zeng D, Green A, Bien J, Brooks L, Hu AJ, Jahja M, McDonald D, Narasimhan B, Politsch C, Rajanala S, Rumack A, Simon N, Tibshirani RJ, Tibshirani R, Ventura V, Wasserman L, O’Dea EB, Drake JM, Pagano R, Tran QT, Ho LST, Huynh H, Walker JW, Slayton RB, Johansson MA, Biggerstaff M, Reich NG. Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States. Proc Natl Acad Sci U S A. 2022 Apr 12;119(15):e2113561119.
Journal cover image

Published In

Proc Natl Acad Sci U S A

DOI

EISSN

1091-6490

Publication Date

April 12, 2022

Volume

119

Issue

15

Start / End Page

e2113561119

Location

United States

Related Subject Headings

  • United States
  • Public Health
  • Probability
  • Pandemics
  • Humans
  • Forecasting
  • Data Accuracy
  • COVID-19