Mauricio Santillana1, D Wendong Zhang1, Benjamin M Althouse2, John W Ayers3. 1. School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts. 2. Santa Fe Institute, Santa Fe, New Mexico. 3. Graduate School of Public Health, San Diego State University, San Diego, California. Electronic address: ayers.john.w@gmail.com.
Abstract
BACKGROUND: Google Flu Trends (GFT) claimed to generate real-time, valid predictions of population influenza-like illness (ILI) using search queries, heralding acclaim and replication across public health. However, recent studies have questioned the validity of GFT. PURPOSE: To propose an alternative methodology that better realizes the potential of GFT, with collateral value for digital disease detection broadly. METHODS: Our alternative method automatically selects specific queries to monitor and autonomously updates the model each week as new information about CDC-reported ILI becomes available, as developed in 2013. Root mean squared errors (RMSEs) and Pearson correlations comparing predicted ILI (proportion of patient visits indicative of ILI) with subsequently observed ILI were used to judge model performance. RESULTS: During the height of the H1N1 pandemic (August 2 to December 22, 2009) and the 2012-2013 season (September 30, 2012, to April 12, 2013), GFT's predictions had RMSEs of 0.023 and 0.022 (i.e., hypothetically, if GFT predicted 0.061 ILI one week, it is expected to err by 0.023) and correlations of r=0.916 and 0.927. Our alternative method had RMSEs of 0.006 and 0.009, and correlations of r=0.961 and 0.919 for the same periods. Critically, during these important periods, the alternative method yielded more accurate ILI predictions every week, and was typically more accurate during other influenza seasons. CONCLUSIONS: GFT may be inaccurate, but improved methodologic underpinnings can yield accurate predictions. Applying similar methods elsewhere can improve digital disease detection, with broader transparency, improved accuracy, and real-world public health impacts.
BACKGROUND: Google Flu Trends (GFT) claimed to generate real-time, valid predictions of population influenza-like illness (ILI) using search queries, heralding acclaim and replication across public health. However, recent studies have questioned the validity of GFT. PURPOSE: To propose an alternative methodology that better realizes the potential of GFT, with collateral value for digital disease detection broadly. METHODS: Our alternative method automatically selects specific queries to monitor and autonomously updates the model each week as new information about CDC-reported ILI becomes available, as developed in 2013. Root mean squared errors (RMSEs) and Pearson correlations comparing predicted ILI (proportion of patient visits indicative of ILI) with subsequently observed ILI were used to judge model performance. RESULTS: During the height of the H1N1 pandemic (August 2 to December 22, 2009) and the 2012-2013 season (September 30, 2012, to April 12, 2013), GFT's predictions had RMSEs of 0.023 and 0.022 (i.e., hypothetically, if GFT predicted 0.061 ILI one week, it is expected to err by 0.023) and correlations of r=0.916 and 0.927. Our alternative method had RMSEs of 0.006 and 0.009, and correlations of r=0.961 and 0.919 for the same periods. Critically, during these important periods, the alternative method yielded more accurate ILI predictions every week, and was typically more accurate during other influenza seasons. CONCLUSIONS: GFT may be inaccurate, but improved methodologic underpinnings can yield accurate predictions. Applying similar methods elsewhere can improve digital disease detection, with broader transparency, improved accuracy, and real-world public health impacts.
Authors: Isaac Chun-Hai Fung; King-Wa Fu; Chung-Hong Chan; Benedict Shing Bun Chan; Chi-Ngai Cheung; Thomas Abraham; Zion Tsz Ho Tse Journal: Public Health Rep Date: 2016 May-Jun Impact factor: 2.792
Authors: John W Ayers; Benjamin M Althouse; Jon-Patrick Allem; Eric C Leas; Mark Dredze; Rebecca S Williams Journal: Am J Prev Med Date: 2016-02-11 Impact factor: 5.043
Authors: Jon-Patrick Allem; Patricia Escobedo; Kar-Hai Chu; Daniel W Soto; Tess Boley Cruz; Jennifer B Unger Journal: Tob Control Date: 2016-03-08 Impact factor: 7.552