Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.

Estee Y Cramer ORCID logo ; Evan L Ray ORCID logo ; Velma K Lopez ORCID logo ; Johannes Bracher ORCID logo ; Andrea Brennen ; Alvaro J Castro Rivadeneira ; Aaron Gerding ; Tilmann Gneiting ORCID logo ; Katie H House ; Yuxin Huang ; +285 more... Dasuni Jayawardena ; Abdul H Kanji ; Ayush Khandelwal ; Khoa Le ; Anja Mühlemann ; Jarad Niemi ORCID logo ; Apurv Shah ; Ariane Stark ; Yijin Wang ; Nutcha Wattanachit ; Martha W Zorn ; Youyang Gu ; Sansiddh Jain ; Nayana Bannur ; Ayush Deva ; Mihir Kulkarni ; Srujana Merugu ; Alpan Raval ; Siddhant Shingi ; Avtansh Tiwari ; Jerome White ORCID logo ; Neil F Abernethy ; Spencer Woody ORCID logo ; Maytal Dahan ; Spencer Fox ORCID logo ; Kelly Gaither ORCID logo ; Michael Lachmann ; Lauren Ancel Meyers ORCID logo ; James G Scott ; Mauricio Tec ORCID logo ; Ajitesh Srivastava ; Glover E George ORCID logo ; Jeffrey C Cegan ORCID logo ; Ian D Dettwiller ; William P England ; Matthew W Farthing ; Robert H Hunter ORCID logo ; Brandon Lafferty ORCID logo ; Igor Linkov ; Michael L Mayo ORCID logo ; Matthew D Parno ; Michael A Rowland ORCID logo ; Benjamin D Trump ; Yanli Zhang-James ; Samuel Chen ORCID logo ; Stephen V Faraone ; Jonathan Hess ; Christopher P Morley ; Asif Salekin ORCID logo ; Dongliang Wang ; Sabrina M Corsetti ORCID logo ; Thomas M Baer ; Marisa C Eisenberg ; Karl Falb ORCID logo ; Yitao Huang ORCID logo ; Emily T Martin ; Ella McCauley ; Robert L Myers ; Tom Schwarz ; Daniel Sheldon ORCID logo ; Graham Casey Gibson ; Rose Yu ; Liyao Gao ; Yian Ma ; Dongxia Wu ; Xifeng Yan ; Xiaoyong Jin ; Yu-Xiang Wang ; YangQuan Chen ; Lihong Guo ORCID logo ; Yanting Zhao ; Quanquan Gu ORCID logo ; Jinghui Chen ; Lingxiao Wang ; Pan Xu ORCID logo ; Weitong Zhang ; Difan Zou ; Hannah Biegel ; Joceline Lega ORCID logo ; Steve McConnell ORCID logo ; VP Nagraj ; Stephanie L Guertin ; Christopher Hulme-Lowe ; Stephen D Turner ORCID logo ; Yunfeng Shi ORCID logo ; Xuegang Ban ; Robert Walraven ORCID logo ; Qi-Jun Hong ; Stanley Kong ; Axel van de Walle ORCID logo ; James A Turtle ORCID logo ; Michal Ben-Nun ORCID logo ; Steven Riley ORCID logo ; Pete Riley ; Ugur Koyluoglu ORCID logo ; David DesRoches ; Pedro Forli ; Bruce Hamory ; Christina Kyriakides ; Helen Leis ; John Milliken ; Michael Moloney ; James Morgan ; Ninad Nirgudkar ; Gokce Ozcan ; Noah Piwonka ; Matt Ravi ; Chris Schrader ; Elizabeth Shakhnovich ; Daniel Siegel ; Ryan Spatz ; Chris Stiefeling ; Barrie Wilkinson ; Alexander Wong ; Sean Cavany ORCID logo ; Guido España ORCID logo ; Sean Moore ORCID logo ; Rachel Oidtman ORCID logo ; Alex Perkins ORCID logo ; David Kraus ORCID logo ; Andrea Kraus ; Zhifeng Gao ; Jiang Bian ; Wei Cao ORCID logo ; Juan Lavista Ferres ORCID logo ; Chaozhuo Li ; Tie-Yan Liu ; Xing Xie ; Shun Zhang ; Shun Zheng ; Alessandro Vespignani ORCID logo ; Matteo Chinazzi ; Jessica T Davis ; Kunpeng Mu ; Ana Pastore Y Piontti ; Xinyue Xiong ; Andrew Zheng ; Jackie Baek ; Vivek Farias ; Andreea Georgescu ; Retsef Levi ; Deeksha Sinha ORCID logo ; Joshua Wilde ; Georgia Perakis ORCID logo ; Mohammed Amine Bennouna ORCID logo ; David Nze-Ndong ; Divya Singhvi ; Ioannis Spantidakis ORCID logo ; Leann Thayaparan ; Asterios Tsiourvas ORCID logo ; Arnab Sarker ORCID logo ; Ali Jadbabaie ORCID logo ; Devavrat Shah ORCID logo ; Nicolas Della Penna ; Leo A Celi ORCID logo ; Saketh Sundar ; Russ Wolfinger ; Dave Osthus ORCID logo ; Lauren Castro ; Geoffrey Fairchild ORCID logo ; Isaac Michaud ; Dean Karlen ; Matt Kinsey ; Luke C Mullany ORCID logo ; Kaitlin Rainwater-Lovett ORCID logo ; Lauren Shin ; Katharine Tallaksen ; Shelby Wilson ; Elizabeth C Lee ORCID logo ; Juan Dent ORCID logo ; Kyra H Grantz ; Alison L Hill ORCID logo ; Joshua Kaminsky ; Kathryn Kaminsky ; Lindsay T Keegan ORCID logo ; Stephen A Lauer ; Joseph C Lemaitre ORCID logo ; Justin Lessler ; Hannah R Meredith ; Javier Perez-Saez ; Sam Shah ; Claire P Smith ; Shaun A Truelove ORCID logo ; Josh Wills ORCID logo ; Maximilian Marshall ; Lauren Gardner ; Kristen Nixon ; John C Burant ; Lily Wang ; Lei Gao ORCID logo ; Zhiling Gu ORCID logo ; Myungjin Kim ; Xinyi Li ; Guannan Wang ; Yueying Wang ; Shan Yu ORCID logo ; Robert C Reiner ; Ryan Barber ; Emmanuela Gakidou ; Simon I Hay ORCID logo ; Steve Lim ; Chris Murray ORCID logo ; David Pigott ; Heidi L Gurung ; Prasith Baccam ; Steven A Stage ORCID logo ; Bradley T Suchoski ; B Aditya Prakash ORCID logo ; Bijaya Adhikari ; Jiaming Cui ; Alexander Rodríguez ORCID logo ; Anika Tabassum ; Jiajia Xie ORCID logo ; Pinar Keskinocak ORCID logo ; John Asplund ; Arden Baxter ORCID logo ; Buse Eylul Oruc ORCID logo ; Nicoleta Serban ; Sercan O Arik ; Mike Dusenberry ; Arkady Epshteyn ; Elli Kanal ; Long T Le ; Chun-Liang Li ; Tomas Pfister ; Dario Sava ; Rajarishi Sinha ORCID logo ; Thomas Tsai ; Nate Yoder ORCID logo ; Jinsung Yoon ; Leyou Zhang ORCID logo ; Sam Abbott ; Nikos I Bosse ; Sebastian Funk ORCID logo ; Joel Hellewell ; Sophie R Meakin ORCID logo ; Katharine Sherratt ORCID logo ; Mingyuan Zhou ; Rahi Kalantari ; Teresa K Yamana ORCID logo ; Sen Pei ORCID logo ; Jeffrey Shaman ORCID logo ; Michael L Li ORCID logo ; Dimitris Bertsimas ORCID logo ; Omar Skali Lami ORCID logo ; Saksham Soni ORCID logo ; Hamza Tazi Bouardi ORCID logo ; Turgay Ayer ; Madeline Adee ; Jagpreet Chhatwal ; Ozden O Dalgic ; Mary A Ladd ; Benjamin P Linas ; Peter Mueller ; Jade Xiao ; Yuanjia Wang ORCID logo ; Qinxia Wang ; Shanghong Xie ; Donglin Zeng ; Alden Green ; Jacob Bien ; Logan Brooks ; Addison J Hu ; Maria Jahja ; Daniel McDonald ORCID logo ; Balasubramanian Narasimhan ; Collin Politsch ORCID logo ; Samyak Rajanala ORCID logo ; Aaron Rumack ORCID logo ; Noah Simon ; Ryan J Tibshirani ORCID logo ; Rob Tibshirani ; Valerie Ventura ; Larry Wasserman ; Eamon B O'Dea ; John M Drake ORCID logo ; Robert Pagano ; Quoc T Tran ; Lam Si Tung Ho ORCID logo ; Huong Huynh ; Jo W Walker ; Rachel B Slayton ORCID logo ; Michael A Johansson ORCID logo ; Matthew Biggerstaff ORCID logo ; Nicholas G Reich ORCID logo ; (2022) Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States. Proceedings of the National Academy of Sciences, 119 (15). e2113561119-. ISSN 0027-8424 DOI: 10.1073/pnas.2113561119
Copy

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.


picture_as_pdf
Cramer_etal_2021_Evaluation-of-individual-and-ensemble.pdf
subject
Published Version
Available under Creative Commons: Attribution 4.0

View Download

Atom BibTeX OpenURL ContextObject in Span Multiline CSV OpenURL ContextObject Dublin Core Dublin Core MPEG-21 DIDL Data Cite XML EndNote HTML Citation JSON MARC (ASCII) MARC (ISO 2709) METS MODS RDF+N3 RDF+N-Triples RDF+XML RIOXX2 XML Reference Manager Refer Simple Metadata ASCII Citation EP3 XML
Export

Downloads