Forecasting respiratory infectious outbreaks using ED-based syndromic surveillance for febrile ED visits in a Metropolitan City.

BACKGROUND: Monitoring and detecting sudden outbreaks of respiratory infectious disease is important. Emergency Department (ED)-based syndromic surveillance systems have been introduced for early detection of infectious outbreaks. The aim of this study was to develop and validate a forecasting model of respiratory infectious disease outbreaks based on a nationwide ED syndromic surveillance using daily number of emergency department visits with fever.
METHODS: We measured the number of daily ED visits with body temperature ≥ 38.0 °C and daily number of patients diagnosed as respiratory illness by the ICD-10 codes from the National Emergency Department Information System (NEDIS) database of Seoul, Korea. We developed a forecast model according to the Autoregressive Integrated Moving Average (ARIMA) method using the NEDIS data from 2013 to 2014 and validated it using the data from 2015. We defined alarming criteria for extreme numbers of ED febrile visits that exceed the forecasted number. Finally, the predictive performance of the alarm generated by the forecast model was estimated.
RESULTS: From 2013 to 2015, data of 4,080,766 ED visits were collected. 303,469 (7.4%) were ED visits with fever, and 388,943 patients (9.5%) were diagnosed with respiratory infectious disease. The ARIMA (7.0.7) model was the most suitable model for predicting febrile ED visits the next day. The number of patients with respiratory infectious disease spiked concurrently with the alarms generated by the forecast model.
CONCLUSIONS: A forecast model using syndromic surveillance based on the number of ED visits was feasible for early detection of ED respiratory infectious disease outbreak.

Background

Infectious disease outbreak is major public health concern and impacts diverse populations. For example, an outbreak of Middle East Respiratory Syndrome (MERS) occurred in Seoul, Korea in 2015 and resulted in mortality and socioeconomic burden [[1], [2], [3], [4]]. Early detection and appropriate public health response is crucial to manage outbreaks and reduce the burden of infectious disease. Several public health surveillance systems have been developed for detection and recognition of infectious outbreaks [[5], [6], [7], [8]]. Monitoring and detecting sudden outbreak is important for outbreak of respiratory infectious disease because of high infectivity, fast propagation, and large burden on health care services.

Emergency Department (ED)-based syndromic surveillance systems have been introduced for faster and earlier detection of several disease categories. Using syndromic parameters derived from clinical data during ED stay instead of confirmed diagnosis or laboratory studies requiring time delay, ED based syndromic surveillance system have shown favorable accuracy and timeliness [[7], [8], [9], [10], [11], [12]].

During the MERS Coronavirus (CoV) outbreak in Korea, the ED was the initial contact point of medical services, and overcrowded EDs were the main place to propagate MERS CoV and delay early detection of outbreak. In Korea, clinical information during ED care is collected into a nationwide electronic database operated by the government called the National Emergency Department Information System (NEDIS). Information collected through the NEDIS network includes vital signs of patients at presentation to ED as well as their final diagnosis [[13], [14], [15]]. However, to detect the outbreak earlier, the development of a forecast model using a nationwide ED database is required.

The goal of this investigation was to develop and validate a forecast model to detect respiratory outbreaks using a real-time nationwide syndromic surveillance system for daily number of febrile ED visits using the NEDIS nationwide ED electronic database. We derived a forecast model for prediction of daily counts of ED visits due to fever and built an alarm system based on differences of predicted and expected values.

Methods

Study setting

This was a retrospective observational study to derive and validate an ED-based forecast model using syndromic surveillance from 2013 to 2015 in Seoul, Korea. Seoul is the capital of Korea, with a population of approximately 10 million and an area of 605.21 km2. In Korea, all EDs are designated and classified into 3 levels according to the level and capability of medical service they provide. In Seoul, there was 1 level I ED, approximately 30 level II EDs, and approximately 19 level III EDs included in 2015. The mean number of annual ED visits in Seoul between 2013 and 2015 was approximately 1,200,000. This study was reviewed and approved by an institutional review board (Seoul National University Hospital, IRB No.E-1801-007-911).

Database

The NEDIS database from 2013 to 2015 was used for the development and validation of an ED-based real-time syndromic surveillance model for febrile ED visits. The NEDIS database system was first established in 2004. All data from patients visiting an ED is collected from the medical records of each institution by a designated and trained coordinator across Korea. Information from each institution is uploaded through the internet to a central server maintained by the National Emergency Medical Center of Korea. Data variables in NEDIS include gender, age, type of insurance, means of transportation, level of consciousness at presentation, emergency operative procedures, time variables (time of ED visit, ED discharge, hospital admission, and hospital discharge), disposition status after the ED encounter, and final ED outcomes (information regarding discharge, transfer, and death). Initial vital signs, including body temperature after ED arrival is measured and collected. Transferring time of each variable in the NEDIS is different according to protocol of the NEDIS quality assessment. Chief complaint and vital signs are transmitted near real-time, while and ED final diagnosis and disposition variables may up to 14 days to transmit.

Development of ED-based syndromic surveillance for febrile ED visits

To develop a forecast model for respiratory illness outbreak, we used the ED-based syndromic surveillance database during 2013 and 2014. We developed a syndromic surveillance model using the NEDIS database and monitored the daily number of febrile ED visits as a syndromic indicator during the study period. We defined the febrile ED visit by patient records with initial ED presenting body temperature equal or higher than 38.0 °C in the NEDIS, and we collected the final diagnosis coded by the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10) of each case defined as a febrile ED visit. If ED final diagnosis of cases corresponded to the ICD-10 codes listed in Table 1 , we defined them as patients with respiratory infectious disease. If the daily proportion of patients with designated ICD-10 codes as respiratory infectious disease in syndromic surveillance exceeded the upper 95th percentile of day-of-week mean proportion of respiratory infectious disease calculated during 2013 and 2014, we defined the day as a respiratory infectious disease outbreak.

Table 1

List of ICD-10 codes designated as respiratory infectious disease in the forecast model.

ICD-10 code	Diagnosis
J00	Acute nasopharyngitis [common cold]
J02	Acute pharyngitis (includes sore throat)
J04	Acute laryngitis and tracheitis
J06	Acute upper respiratory infections of multiple and unspecified sites
J09	Avian influenza
J10	Influenza due to other identified influenza virus
J11	Influenza, virus not identified
J16	Pneumonia due to other infectious organisms, not elsewhere classified
J18	Pneumonia, organism unspecified

Derivation of an ED-based forecast model for respiratory infectious disease outbreak

To develop the forecast model, we conducted a 3-step process. The 1st step was building a forecast model for prediction of the number of febrile ED visits using the number of febrile ED visits of the past few days registered in the syndromic surveillance during 2013 and 2014. The 2nd step was configuring alarm criteria for the extreme number ED febrile visits that exceed the forecasted number using data from 2013 to 2014. The final step was to simulate the forecast model and measure the predictive performance of alarm generated using data from 2015.

To predict the number of febrile ED visits using the forecast model, we used the Autoregressive Integrated Moving Average (ARIMA) modeling method. ARIMA modeling has been used for forecasting time series variables, such as patient volume, in medical science [16,17]. We developed an ARIMA model optimized to forecast febrile ED visits using number of febrile ED visits during the past few days. We developed various ARIMA models and tested the auto-regression, difference, and moving average of each model. Correlograms were plotted and used for comparison between different ARIMA models. AIC and BIC were also measured and compared to select the most adequate ARIMA model with the highest level of fitting.

Alarm was defined according to 2 criteria. If the observed actual number of febrile ED visits exceeded the upper 95th percentile of the expected number of the forecast ARIMA model, we defined the situation as the first alarming criteria. If the observed number of febrile ED visits exceeded the upper 95th percentile of the average number of febrile ED visits according to day-of-week during 2013 and 2014, we defined it as the second criteria. If the daily number of febrile ED visits satisfied both alarm criteria, we activated the forecast alarm. We established 3 alarm durations: 1-day, 3-day, and 7-day alarms. For example, if 7-day alarm is activated, alarming lasts until the 7th day from the day of most recent alarm generation. We simulated and validated the forecast model using ED-based syndromic surveillance from the data of 2015. We also tested the correlation between the number of observed febrile ED visits and patients diagnosed with infectious respiratory disease as their final diagnosis in ED.

Statistical analysis

Correlation analysis was performed between the daily count of patients with fever and the daily count of patients with ED diagnosis with respiratory infectious disease and plotted simultaneously in a time-series plot. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated with 95% confidence intervals (CI) to measure predictive performance of our forecast model. Separate predictive values were calculated and presented according to length of alarm (1-day alarm, 3-day alarm, and 7-day alarm).

Results

From 2013 to 2015, data of 4,080,766 ED visits were collected in the NEDIS (Table 2 .). Among them, 303,469 (7.4%) had initial body temperature equal to or higher than 38.0 °C, and 388,943 patients (9.5%) were diagnosed with a designated respiratory infectious disease code (Table 2.) A total of 188,434 ED visits with fever in 2013 and 2014 were used to derive the forecast model. We measured the AIC and BIC values of three different forecasting ARIMA models. The AIC and BIC values of each model were as follows: AIC: 8805.3 and BIC: 8823.7 for ARIMA (1.0.1) model, AIC: 8647.5 and BIC: 8684.2 for ARIMA (3.1.3) model, AIC: 8140.6 and BIC: 8214.1 for ARIMA (7.0.7) model. The ARIMA (7.0.7) model was selected as the final model for predicting ED visits of febrile patients the next day.

Table 2

Number of ED visits, ED visits with fever, ED visits with final diagnosis of respiratory infection.

Year	Total ED visits	ED visits with fever		ED visits with final diagnosis of respiratory infection
	N	N	%	N	%
2013	1,238,478	74,003	6.0	109,070	8.8
2014	1,335,698	114,431	8.6	134,059	10.0
2015	1,506,590	115,035	7.6	145,814	9.7
Total	408,0766	303,469	7.4	388,943	9.5

The number of daily febrile ED visits and patients with final ED diagnosis of respiratory infectious disease was plotted together in Fig. 2. They showed moderate to strong correlation (Pearson correlation coefficient = 0.86, p < 0.01, Fig. 1 ). The expected number of febrile visits by the forecast ARIMA model was plotted simultaneously with the observed number of febrile ED visits as well as number of patients with infectious respiratory disease diagnosis (Fig. 2 ).

Fig. 2

Observed number and predicted number of daily febrile ED visits from forecast model.

Fig. 1

Correlation between ED visits with fever and ED diagnosis with respiratory infectious disease.

To validate the forecast model, we simulated our forecast model for ED visit data of 2015. Fig. 3, Fig. 4 show the results of the simulation of the forecast model using ED-based syndromic surveillance throughout the entire year of 2015 and the first quarter of 2015. The number of patients with respiratory infectious disease spiked concurrently with the alarms generated by the forecast model using ED-based syndromic surveillance (Fig. 3, Fig. 4).

Fig. 3

Simulation of alarming generated from forecast model to syndromic surveillance during 2015.

Fig. 4

Simulation of alarming generated from forecast model to syndromic surveillance during the first quarter of 2015.

A total of 29 days of 1-day alarms, 72 days of 3-day alarms, and 133 days of 7-day alarms were generated by the forecast model in 2015 (Table 3 ). According to length of alarm duration, sensitivity and specificity varied respectively.

Table 3

Predictive performance of syndromic surveillance model according to length of alarm.

Alarm duration	Total surveillance days	Total days with respiratory infection outbreak		Total days with alarm		Sensitivity	Specificity	PPV	NPV
	N	N	%	N	%	(95% CI)	(95% CI)	(95% CI)	(95% CI)
1-Day alarm	365	12	3.3	29	7.9	0.417(0.152–0.723)	0.932(0.901–0.956)	0.172(0.058–0.358)	0.979(0.958–0.992)
3-Day alarm	365	12	3.3	72	19.7	0.833(0.516–0.979)	0.824(0.781–0.863)	0.139(0.687–0.241)	0.993(0.976–0.999)
7-Day alarm	365	12	3.3	133	36.4	1.000(0.735–1.000)	0.657(0.605–0.707)	0.090(0.048–0.152)	1.000(0.984–1.000)

Discussion

In our study, we developed a forecast model for respiratory infectious disease outbreak using real time ED-based syndromic surveillance monitoring the daily number of febrile patients visiting EDs located in Seoul for 2 years. We retrospectively simulated and validated the derived forecast model using database information from the year 2015. Results of our study suggest that monitoring the number of daily febrile ED visits and alarming the outbreak could successfully predict the surge of patients diagnosed with respiratory infectious diseases in EDs.

A syndromic surveillance system based on the ED database might be beneficial in terms of efficiency and timeliness [[7], [8], [9], [10], [11], [12]]. In many countries, ED data are routinely collected automatically in electronic form with existing ED information systems [18]. Therefore, additional effort for data collection is not needed if a syndromic surveillance system is adopted based on an existing ED health record database. Patients visiting EDs usually have a shorter interval from symptom onset to hospital visit. Abnormal surges in certain disease categories were detected faster when the ED-based syndromic surveillance model was used instead of laboratory or other surveillance models.

In previous surveillance systems, final diagnoses were used as the key element of a surveillance system [7,11,19]. For respiratory infectious diseases, monitoring the number of patients diagnosed with respiratory infectious disease might be a more accurate indicator to predict respiratory infectious disease outbreak than monitoring the number of febrile patients at ED presentation. However final diagnosis at ED is difficult to collect in a timely manner, resulting in time delay of detection for outbreak. In the NEDIS system, time to complete collection of ED final diagnosis information takes up to 14 days from day of ED admission and differs according to each hospital. By contrast, body temperature is checked for all patients during initial triage after ED arrival, and body temperature informati0n is sent to the central NEDIS server directly with triage information without time delay. Body temperature can be used intuitionally for diagnosis or chief complaint without additional data cleaning and coding procedures.

After MERS-CoV outbreak in Korea, thermal scanning devices were installed for the mass screening of febrile patients entering EDs. Until now there has been no universal guideline for screening and alarming based on information obtained from thermal scanning in EDs. The volume of febrile ED patients could be estimated automatically by thermal scanning device, and application of thermal skin device to the forecast model using ED-based syndromic surveillance may be an efficient method of monitoring the outbreak of respiratory infectious disease.

Although they were not followed by respiratory infectious disease outbreak, a few alarms were generated during early summer and autumn period of 2015 (Fig. 3). We believe that the unexpected extreme number of febrile ED visits was caused by infectious disease categories other than respiratory infectious disease, such as GI tract and urinary tract infection. Further analysis is required to assess inclusions of other infectious diseases with fever that can be detected as a false positive case of respiratory infectious disease in our forecast model.

Various levels of sensitivity and specificity were measured according to different durations of syndromic surveillance alarms (Table 3). Although sensitivity increased according to increase of surveillance alarm duration, specificity decreased with the higher rate of false positive alarms. It is important not to miss the potential time point of respiratory infection outbreak by increasing sensitivity. However, to satisfy the operational purpose of syndromic surveillance, excessive generation of false positive alarms should be decreased to prevent alarm fatigue to laypersons and exhaustion of the public health system. To determine the optimal duration and frequency of surveillance alarms, the performance of predictive power and feasibility to implement into public health system should be considered.

There are a few limitations in our study. First, we could not use the surveillance system for the final diagnosis of respiratory infectious disease based on a culture study report that we could use as a gold standard to assess performance of our forecast model based on syndromic surveillance system. Second, we used 3 years of database values from a Seoul metropolitan city, which results in a limitation of generalizability. Prevalence of respiratory infectious disease and proportion of febrile illness among ED visits could affect the fitness of the ARIMA model or performance of prediction power. Third, we collected diagnosis and vital signs from the NEDIS database retrospectively. Therefore, the quality of the data depends on each institution. However, the National Emergency Medical Center of Korea conducted quality assessment process and certified each hospital for quality of the NEDIS database regularly.

Conclusion

We developed a forecast model for the outbreak of respiratory infectious disease using an ED-based syndromic surveillance in a metropolitan city. We monitored the number of febrile ED visits across the country daily based on a real-time syndromic surveillance system using the nationwide ED database network. The expected number of febrile visits by the ARIMA forecast model was identified similarly with observed number of febrile ED visits. The outbreaks of respiratory infectious disease were shown concurrently with the alarms generated by the forecast model using ED-based syndromic surveillance.

Funding acknowledgement

This study was supported by ‘the data science research support project 2016’ of Seoul National University Big Data Institute (SNU BDI 2016-002). And this study is also funded by the Korea Center for Disease Control, ‘Development of practical and advanced comprehensive surveillance system for assessment of health impacts of climate change’ project (Project No: 2013E2100100).