Evaluation of Nowcasting for Detecting and Predicting Local Influenza Epidemics, Sweden, 2009-2014.

The growing availability of big data in healthcare and public health opens possibilities for infectious disease control in local settings. We prospectively evaluated a method for integrated local detection and prediction (nowcasting) of influenza epidemics over 5 years, using the total population in Östergötland County, Sweden. We used routine health information system data on influenza-diagnosis cases and syndromic telenursing data for July 2009-June 2014 to evaluate epidemic detection, peak-timing prediction, and peak-intensity prediction. Detection performance was satisfactory throughout the period, except for the 2011-12 influenza A(H3N2) season, which followed a season with influenza B and pandemic influenza A(H1N1)pdm09 virus activity. Peak-timing prediction performance was satisfactory for the 4 influenza seasons but not the pandemic. Peak-intensity levels were correctly categorized for the pandemic and 2 of 4 influenza seasons. We recommend using versions of this method modified with regard to local use context for further evaluations using standard methods.

Although the seasonal variations in influenza incidence among nations and global regions are well described (), the duration and intensity of influenza epidemics in local communities have been less well monitored and understood. The rapidly growing availability of big data from diagnostic and prediagnostic (syndromic) sources in healthcare and public health settings opens new possibilities for increasing the granularity in infectious disease control (,). However, development of outbreak models and efficient use of the information produced by prediction models in public health response decision-making remain challenging. This observation was recently highlighted by the Congress of the United States request that the Government Accountability Office gather information on validation of emerging infectious disease model predictions (https://energycommerce.house.gov/wp-content/uploads/2017/11/20171109GAO.pdf).

We previously reported the design of a nowcasting method (detection of influenza epidemics and short-term predictions) for local-level application in the northwestern region of the world (). In other fields, such as meteorology, nowcasting methods represent standard tools for warning the public against dangerous high-impact events (). The rationale for developing this novel method was to inform the planning of local responses and adjustments of healthcare capacities. Many such adjustments are planned and performed locally, at county and municipality levels. In Sweden, for instance, the hospital bed capacity is habitually overextended; on average, 103 patients occupy 100 regular hospital bed units (). It is therefore important that an influenza epidemic is noticed early at the local level to make time for implementation of adjustments (e.g., freeing hospital beds by removing from the waiting list those patients scheduled for elective interventions).

We performed a prospective 5-year evaluation of local influenza nowcasting by using routine health information system data. The evaluation period included 1 pandemic (2009) and 4 winter influenza seasons (Figure). The nowcasting method is based on mathematical modeling of epidemic curves generated from historic local data (). Nowcasting comprises 3 functions: detection of the local start of the epidemic, prediction of peak timing, and prediction of peak intensity.

Figure

Data used for evaluation of nowcasting for detection and prediction of local influenza epidemics, Östergötland County, Sweden, January 1, 2008, through June 30, 2014. A) Unadjusted daily numbers of influenza-diagnosis cases per 100,000 population. B) Unadjusted daily telenursing calls because of fever (child and adult) per 100,000 population.

Methods

We used an open cohort design based on the total population (n = 445,000) in Östergötland County, Sweden. We used prospective data from July 1, 2009, through June 30, 2014, from 2 sources in the countywide health information system: clinical influenza-diagnosis cases recorded by physicians and syndromic chief complaint data from a telenursing service (). The influenza-diagnosis case data were used for detection of the local start of the epidemic and prediction of its peak intensity; the syndromic data were used for prediction of the peak timing. Timeliness was used as a performance metric for detection of the local start of the epidemic and the peak-timing prediction; the correct identification of intensity category on a 5-grade scale was used for peak-intensity prediction. The study design was approved by the Regional Research Ethics Board in Linköping (no. 2012/104–31).

Definitions

We identified influenza cases by using codes from the International Classification of Diseases, 10th Revision, for influenza (J10.0, J10.1, J10.8, J11.0, J11.1, J11.8) () as recorded in the local electronic health data repository. We identified influenza-related telenursing call cases by using the chief complaint codes associated with influenza symptoms (dyspnea, fever [child and adult], cough [child and adult], sore throat, lethargy, syncope, dizziness, and headache [child and adult]).

The intensity level for the start of a local epidemic (i.e., the endpoint for the detection function) was set to 6.3 influenza-diagnosis cases/100,000 population recorded during a floating 7-day period in the countywide health information system (). A recent comparison of influenza intensity levels in Europe estimated a similar definition (6.4 influenza-diagnosis cases/wk/100,000 population) for the 2008–09 winter influenza season in Sweden (). Peak timing was defined as the date when the highest number of influenza-diagnosis cases were documented in the countywide electronic patient record. Peak intensity was defined as the number of influenza-diagnosis cases that had been documented on that date.

The detection threshold was adjusted to situations when extended simmering of influenza-related activity appears before an epidemic. Such simmering was associated with exceptionally mild winter influenza seasons and pandemics. Preepidemic simmering before winter influenza seasons was defined as occurring when the period between increased influenza incidence above baseline and the start of the epidemic is prolonged (). The upper threshold for the baseline was set to 3.2 influenza-diagnosis cases/100,000 population during a floating 7-day period (i.e., half of the defined start-of-epidemic intensity level). If the baseline threshold was surpassed for a period 3 times longer than the average period before previous epidemics without exceeding the start-of-epidemic level (i.e., 6.3 influenza-diagnosis cases/100,000 population during a floating 7-day period), preepidemic simmering was considered to have occurred. Preepidemic simmering in association with a pandemic was defined as that ensuing from the date of a World Health Organization pandemic alert. We determined all definitions and adjustments before using the method for detection and prediction.

Method Application

Technical details concerning the 3 functions of nowcasting are provided in Technical Appendix 1. The programming code for the analyses is provided in Technical Appendix 2.

To calibrate the detection component of the nowcasting method, we retrospectively determined weekday effects on recording of influenza-diagnosis cases and a baseline alarm threshold by using learning data. These data were collected from January 1, 2008, through June 30, 2009, including the 2 winter influenza seasons 2007–08 and 2008–09. To determine weekday effects, we used data from the entire learning data collection period. To determine the initial alerting threshold, we used only data from the winter influenza season in 2008–09. The 2007–08 winter influenza season could not be used for this purpose because collection of learning data had already started. Throughout the study period, the calibration data were updated after every winter influenza season (i.e., no updates after the 2009 pandemic outbreak). The detection algorithm was thus applied to the next epidemic by using the revised threshold determined in the updated learning dataset. We identified 2 exceptional situations: pandemic settings and winter influenza seasons that simmered before they started (). In these situations, the alarm threshold is doubled. Accordingly, following the World Health Organization pandemic alert (), the alarm threshold was doubled for the 2009 season. Before the 2010–11 winter influenza season, the threshold was reset to the regular level. No updates were performed because the set of learning data remained the same (i.e., it contained data from the 2008–09 winter influenza season). For the 2011–12 winter influenza season, we updated the threshold by using learning data from the 2008–09 and 2010–11 winter influenza seasons. For the 2012–13 winter influenza season, we updated the threshold by using learning data from the 2008–09, 2010–11, and 2011–12 winter influenza seasons. For the 2013–14 winter influenza season, we again updated the alerting threshold by using learning data from the previous winter influenza seasons (2008–09, 2010–11, 2011–12, and 2012–13). However, because this winter influenza season simmered before it started, the threshold was doubled before the detection method was applied. The weekday effects were assumed to be relatively constant over time in the local detection analyses and therefore were not updated after every winter influenza season.

We also used the set of learning data from the winter influenza seasons in 2007–08 and 2008–09 to initially calibrate the first component of the local prediction module. The dataset was used to decide the grouping of chief complaints with the largest correlation strength and longest lead time between influenza-diagnosis data and telenursing data (,). The best performing telenursing chief complaint was fever (child and adult), and the most favorable lead time was 14 days. When the peak timing had been determined, the second component of the local prediction module was applied to influenza-diagnosis data from the corresponding epidemics to find the peak intensity on the predicted peak day (). Regarding weekday effects on local prediction, the same calculation was applied and the same grouping of chief complaints and lead time were used throughout the study.

Metrics and Interpretations

For trustworthiness of the nowcasting method in local healthcare planning, we set the maximum acceptable timeliness error for detection and peak timing predictions to 1.5 weeks. Method performance was defined to be excellent if the absolute value of the timeliness error was <3 days, good if it was 4–7 days, acceptable if it was 8–11 days, and poor if it was >12 days. For peak intensity predictions, we used the epidemic threshold and intensity level categories (nonepidemic, low, medium, high, and very high) identified for Sweden in a study involving 28 European countries () (Table 1). If the predicted peak intensity fell into the same category as the actual peak intensity, the prediction was considered successful; otherwise, it was considered unsuccessful.

Table 1

Epidemic intensity categories used to interpret performance measurements in evaluation of nowcasting for detection and prediction of local influenza epidemics, Sweden, 2009–2014*

Intensity level	Threshold, cases/d/100,000 population
	2008–09	2009 pandemic	2010–11	2011–12	2012–13	2013–14
Nonepidemic	<0.9	<0.9	<0.9	<1.0	<1.2	<1.2
Low	0.9	0.9	0.9	1.0	1.2	1.2
Medium	2.4	2.5	2.5	2.5	2.8	2.9
High	5.5	5.4	5.4	5.2	5.6	5.5
Very high	7.9	7.5	7.5	7.1	7.7	7.4

*Based on (8).

Results

Local Detection

The detection component of the local nowcasting method showed good performance during the 2009 pandemic of influenza A(H1N1)pdm09 (pH1N1) virus (Table 2), alerting for the local influenza epidemic 5 days after it actually started. For the 2010–11 winter influenza season with influenza B and pH1N1 viruses circulating, the local detection performance was also good; the alarm was raised 5 days after the start of the local epidemic. During the 2011–12 winter influenza season, with influenza A(H3N2) virus activity only, detection performance was poor. During that season, the alarm was raised 15 days before the actual local epidemic started. During the 2012–13 and 2013–14 winter influenza seasons, with influenza A(H3N2), influenza B, and pH1N1 viruses circulating, the local detection performance was excellent; alarms were raised 3 days before the 2012–13 epidemic started and 3 days after the 2013–14 epidemic started.

Table 2

Performance of the detection algorithm displayed with alert thresholds updated by using data from previous winter influenza seasons in evaluation of nowcasting for detection and prediction of local influenza epidemics, Sweden, 2009–2014*

Influenza virus activity	Updated threshold, cases/d/100,000 population	Timeliness†	Interpretation
2009 pH1N1‡	0.424	−5	Good
2010–11 B and pH1N1	0.212	−5	Good
2011–12 A(H3N2)	0.207	15	Poor
2012–13 A(H3N2), B, and pH1N1	0.242	3	Excellent
2013–14 A(H3N2), B, and pH1N1‡	0.481	−3	Excellent

*pH1N1, pandemic influenza A(H1N1)pdm09 virus.
†Positive value means that the algorithm issued an alarm before the local epidemic had started; negative value means that the alarm was raised after the start of the epidemic.
‡The threshold was doubled because of a pandemic alert or observation of a period of simmering influenza activity.

Local Prediction

For the 2009 influenza pandemic, performance of the local peak-timing prediction was poor, but the peak-intensity level was correctly categorized (medium intensity epidemic) (Table 3). For the 2010–11 winter influenza season, with influenza B and pH1N1 viruses circulating, the local peak-timing prediction was excellent and the peak intensity was successfully predicted to the correct category (medium intensity). For the 2011–12 winter influenza seasons with influenza A(H3N2) virus circulating and the 2012–13 winter season with influenza A(H3N2), influenza B, and pH1N1 viruses circulating, the local peak-timing predictions were good. The local peak-intensity predictions were successful for the 2012–13 winter influenza season, correctly categorizing it to a very high intensity level and unsuccessful for the 2011–12 season, categorizing it as a medium intensity epidemic when it actually developed into a very high-intensity epidemic. For the 2013–14 winter influenza season, with influenza B and pH1N1 viruses circulating and a simmering start, the local peak-timing prediction was acceptable, but the local peak intensity was wrongly predicted to be at the nonepidemic level when the winter influenza season actually reached a medium intensity level.

Table 3

Performance of peak-timing and peak-intensity predictions from evaluation of nowcasting for detection and prediction of local influenza epidemics, Sweden, 2009–2014*

Influenza virus active	Time-of-peak predictions†					Peak intensity predictions
	Date when prediction made	Time to peak, d	Prediction error	Interpretation		Category (cases/d/100,000 population)		Interpretation
						Predicted	Factual
2009 pH1N1	2009 Sep 27	8	−28	Poor		Medium (3.3)	Medium (2.9)	Successful
2010–11 B and pH1N1	2011 Feb 11	10	0	Excellent		Medium (4.5)	Medium (4.9)	Successful
2011–12 A(H3N2)	2012 Feb 25	9	7	Good		Medium (4.5)	Very high (12.4)	Unsuccessful
2012–13 A(H3N2), B, and pH1N1	2013 Feb 22	10	−7	Good		Very high (10.1)	Very high (11.7)	Successful
2013–14 A(H3N2), B, and pH1N1	2014 Feb 17	8	−8	Acceptable		Nonepidemic (1.0)	Medium (3.4)	Unsuccessful

*pH1N1, pandemic influenza A(H1N1)pdm09 virus.
†Positive value means that the peak was predicted to be reached before the actual peak occurred; negative value means that the peak was predicted to be reached after the actual peak occurred.

Discussion

In this prospective 5-year evaluation of a method for local nowcasting of influenza epidemics that used routine health information system data, we identified aspects that were satisfactory and identified areas where improvements are needed. The detection function displayed satisfactory performance throughout the evaluation period, except for the 2011–12 winter influenza season, in which influenza A(H3N2) virus circulated after a season with influenza B and pH1N1 virus activity. Peak-timing prediction performance was satisfactory for the 4 winter influenza seasons but not for the 2009 pandemic. In addition, the method categorized the local peak-intensity levels correctly for the 2009 pandemic and for 2 of the winter influenza seasons, but it was unsuccessful at forecasting the very high peak intensity of the 2011–12 season and the medium peak intensity of the 2013–14 season, which was preceded by a simmering phase.

The results indicate that securing the availability of a new data source is only the first step toward using the data stream in routine surveillance. The syndromic data source used for the big data stream in this study was subjected to rigorous restructuring and maintenance. Nonetheless, not all parameters associated with the syndromic data stream were regularly updated. For the peak-timing predictions made by using telenursing data, we assumed that increases in telenursing activity precede influenza diagnoses by 14 days. Although this assumption is grounded (,), the interval may change over time and thereby influence influenza predictions. Using the constant interval estimate, we estimated the influenza diagnosis peaks for the 2011–12 and 2012–13 winter influenza seasons 1 week before and 1 week after the actual influenza-diagnosis peaks. In other words, the basic prediction method may have been more accurate at predicting the peak timing than the results of this study show. In the setting of our study, the performance was most likely decreased by the assumption that telenursing precedes influenza diagnosis by 14 days is applicable to all situations. Althouse et al. reported that methods underpinning the use of big data sources (e.g., search query logs) need regular upkeep to maintain their accuracy (). In future versions of nowcasting methods, regular updates and syndromic sources that are more stable than telenursing data (regarding time lag to influenza diagnosis data) may become available and can be used to improve the peak-timing predictions.

Influenza forecasting is methodologically challenging (,), and only a few prospective evaluations have transparently reported algorithms and study designs. In the first Centers for Disease Control and Prevention (CDC) challenge, a prospective study of state-of-the-art methods in which 4 aspects of influenza epidemics (start week, peak week, peak percentage, and duration) were forecasted by using routine data (), none of the evaluated methods showed satisfactory performance for all aspects. Similarly, in the second CDC challenge, in which 3 aspects of influenza epidemics (start week, peak week, and peak intensity) were forecasted by using 7 methods, none of the evaluated methods displayed satisfactory performance (). These challenge studies have substantially helped to widen the understanding of the difficulties of forecasting different aspects of influenza epidemics. In our study, the detection and peak-intensity prediction functions of the local nowcasting method underperformed during the 2011–12 winter influenza season. One reason for the observed underperformance may be that the data from preceding seasons used to generate local epidemic curves were insufficient for modeling the between-seasons drift in the immunity status of the population in relation to the circulating influenza strain. In other words, the present parameters used to compute epidemic curves were deficient when large drifts in population immunity with corresponding changes in virus dissemination patterns occurred. For instance, the epidemic phase of the influenza A(H3N2) season in 2011–12 may not have started with virus spread among the young persons in the community as it had during previous seasons (,). It has been suggested that including virologic information (i.e., influenza virus type and subtype) as model parameters may improve the predictive accuracy of mathematical models (). We contend also that, for local influenza detection and prediction, historical accounts of the circulating influenza virus types should be considered for inclusion in the statistical models and suggest adding information about the population age structure. However, such model extensions must also be paralleled by securing a continuous supply of the corresponding data in the local settings where the models are to be used.

This study has strengths and weaknesses that need to be considered when interpreting the results. The main strength of the study is that it prospectively evaluates an integrated influenza nowcasting method in a local community. On the basis of experiences from previous studies (,,,), we considered timeliness to be the most valid general evaluation metric for our purposes. To be able to accurately support adjustments of local healthcare capacity, we used daily data for the evaluations. In the CDC challenge studies (,), forecasts of the start and peak timing of an epidemic were based on weekly data and considered accurate if they occurred within 1 week of the actual timing of each component. Therefore, we consider the limits used for evaluating detection and the peak-timing predictions in this study to be at least as strict as those in the CDC challenge. Regarding the prediction of peak intensity, we considered a forecast to be accurate if it predicted the peak intensity to be the correct peak-intensity category as defined by Vega et al. (), who calculated the thresholds for each of these categories for every winter influenza season by applying the moving epidemic method () on 5–10 previously occurring seasons. Hence, we consider these categories to be reliable. A longer prospective evaluation period would have increased the possibility of drawing valid conclusions concerning the outcome of the evaluation, and it would be preferable to have corresponding local data from other cities or regions (). Evaluating our nowcasting method for epidemics from several other regions would enable conclusions to be drawn about the generalizability of the method.

We contend that methods for local nowcasting of influenza epidemics based on routine health information system data have potential for general dissemination and use. Future versions of the nowcasting model will be gradually extended with information on population age distribution and on current and previously circulating influenza types. Such extensions need to be paralleled by securing a routine supply of data to the added parameters in local health information systems. We recommend using versions of the nowcasting method modified with regard to their local use context for further evaluations with standard measures.

Technical Appendix 1. Method design overview of evaluation of nowcasting for detection and prediction of local influenza epidemics, Sweden, 2009–2014.

Technical Appendix 2. Programming code used in evaluation of nowcasting for detection and prediction of local influenza epidemics, Sweden, 2009–2014.