Fielded Top Two Forecasting Systems in CDC Research Initiative
For the third year in a row, Carnegie Mellon University’s forecasts of national influenza activity have proven to be the most accurate among all forecasting systems evaluated by the U.S. Centers for Disease Control and Prevention.
CMU’s Delphi research group fielded two forecasting systems during the flu season that ended in May. The systems ranked 1 and 2 among the 28 systems submitted to the CDC’s forecasting research initiative by university and governmental research groups.
In contrast to the CDC’s longstanding flu surveillance network, which measures flu activity after it occurs, the forecasting effort attempts to look into the future, much like a weather forecast, so health officials can plan ahead.
"We’re gratified that our forecasting methods continue to perform as well as they do, but it’s important to remember that epidemiological forecasting remains in its infancy," said Roni Rosenfeld, Delphi leader and professor in the School of Computer Science’s Machine Learning Department and Language Technologies Institute. "The CDC’s flu forecasting initiative has proven invaluable to us, providing us with both the up-to-the-minute data and the feedback we need to constantly improve."
Many epidemiological forecasting systems are based on mechanistic models that consider how diseases spread and who is susceptible to them. The CMU team’s systems work differently. One version, called Delphi-Stat, is a non-mechanistic model that uses artificial intelligence - in particular, machine learning - to make predictions based on past patterns and on input from the CDC’s domestic influenza surveillance system. The other, called Delphi-Epicast, relies on the so-called "wisdom of the crowds," basing its forecasts on the judgments of a number of volunteers who submit their own weekly predictions.
During the 2016-17 flu season, Delphi-Stat did a bit better on short-term forecasts and Delphi-Epicast did a bit better on long-term forecasts, said Ryan Tibshirani, associate professor of statistics and machine learning and a member of the Delphi group.
In the combined "skill score" calculated by the CDC, Epicast received 0.451, while Stat received 0.438. In this measure of forecasting skill, perfect clairvoyance would earn a skill score of 1.00, whereas simply assuming what happens next is the average of what has already happened would correspond with a score of 0.237, he said.
The CDC also created an "ensemble" forecast by averaging all 28 submissions, Rosenfeld said. Usually, such ensembles outperform their components because the models tend to compensate for each other’s weaknesses. But even here, both CMU models did better than the ensemble’s skill score of 0.430.
Epidemiological modeling and forecasting is a highly interdisciplinary endeavor. The Delphi research group unites faculty and students from CMU’s Machine Learning, Statistics, Computer Science and Computational Biology departments. The group belongs to a University of Pittsburgh-based MIDAS National Center of Excellence , a National Institutes of Health-funded network of researchers developing computational models to guide responses to disease outbreaks.
Flu is useful for developing forecasting systems because incidence data is plentiful, if "noisy" - based largely on symptoms, rather than on tests for the flu viruses themselves - Rosenfeld said. The Delphi group also is developing forecasting systems for dengue fever, which sickens about 100 million people worldwide each year, killing thousands. The group plans to apply its forecasting tools to such diseases and conditions as HIV, drug resistance, Ebola, Zika and Chikungunya.
CMU’s flu forecasting is supported in part by the Machine Learning for Social Good fund established at the School of Computer Science by Uptake.
The forecasting effort is part of CMU AI , an initiative in the School of Computer Science to advance artificial intelligence research and education by leveraging the school’s strengths in computer vision, machine learning, robotics, natural language processing and human-computer interaction.