Automated detection of influenza-like illness using clinical surveillance markers at a Department of Veterans Affairs Medical Center.

BACKGROUND: Using demographic and clinical measures from emergency department evaluations, we developed an automated surveillance system for influenza-like illness (ILI).
METHODS: We selected a random sample of patients who were seen at the Durham, NC Veterans Affairs Medical Center between May 2002 and October 2009 with fever or a respiratory ICD-9 diagnosis code and divided this into subsets for system development and validation. Comprehensive chart reviews identified patients who met a standard case definition for ILI. Logistic regression models predicting ILI were fit in the development sample. We applied the parameter estimates from these models to the validation sample and evaluated their utility using receiver-operator characteristic analysis.
RESULTS: The models discriminated ILI very well in the validation sample; the C-statistics were >0.89.
CONCLUSIONS: Risk estimates based on statistical models can be incorporated into electronic medical records systems to assist clinicians and could be used in real-time surveillance for disease outbreaks.

Influenza-like illnesses (ILI) are responsible for significant morbidity and mortality, particularly in people over 65 years of age (1-7). It is estimated that seasonal influenza is responsible for 95,000 to 330,000 hospitalizations (2-4), as many as 70,000 deaths in the United States annually (2, 5-7), and up to 500,000 deaths worldwide each year 8. Other major causes of ILI such as rhinovirus, respiratory syncytial virus, adenovirus, and parainfluenza virus are responsible for a significant additional burden of hospitalization and death (1, 4, 6). The ILI also causes significant economic burden in the United States with estimates of annual direct medical costs of $10 billion and an overall yearly cost of $87 billion when accounting for lost wages and years of lost life (1, 9-11).

Timely and accurate surveillance for ILI is an important component of the public health efforts to respond to outbreaks, as evidenced by the rapid development of the novel H1N1 into a pandemic during 2009. Since the signs and symptoms of ILI are relatively non-specific, potential cases are often identified on the basis of acute respiratory symptoms, fever, and possibly other measures. Definitive identification of cases is made by either laboratory testing of patient samples for specific microorganisms or a manual chart review for additional symptoms (12-15), both of which are labor-intensive and time-consuming and may lead to delayed clinical and public health response.

In this analysis, we present an automated system for identifying patients with ILI using the Department of Veterans Affairs electronic clinical database and the NC Veterans Electronic Testbed (NCVET), an infectious disease specific database that was developed at the Durham VA Medical Center. Specifically, we developed and validated several statistical models predicting ILI based on electronic clinical information available at the time of presentation to the Durham VA emergency department or acute care clinic. Our aim was to use automated, electronically available data to better discriminate those with ILI from those with other unrelated conditions among the population who presented with either fever or respiratory symptoms.

Methods

Study population and data extraction

The sample for this study was drawn from the patients who presented to the Durham VA Medical Center (VAMC) emergency department or acute care clinic between May 2002 and October 2009 with either fever ≥38°C or an ICD-9 code for acute respiratory illness; these are the existing manual ILI screening criteria for the facility. At the Durham VAMC, the primary reasons for a clinical encounter are assigned ICD-9 codes by the treating physician at the time of the encounter. We used the ICD-9 codes for acute respiratory illness shown in Appendix Table 1 (16).

Table 1.

ICD-9 codes used to extract acute respiratory illness from the VA database

ICD-9 code	Description
32	Diphtheria
34	Streptococcal sore throat and Scarlet Fever
74	Specific diseases due to Coxsackie virus
460	Acute nasopharyngitis (common cold)
461	Acute sinusitis
462	Acute pharyngitis
463	Acute tonsillitis
464	Acute laryngitis and tracheitis
465	Acute upper respiratory infections of multiple or unspecified sites
466	Acute bronchitis and bronchiolitis
478	Other diseases of upper respiratory tract
480	Viral pneumonia
481	Pneumococcal pneumonia (Streptococcus pneumoniae pneumonia)
482	Other bacterial pneumonia
483	Pneumonia due to other specified organism
484	Pneumonia in infectious diseases classified elsewhere
485	Bronchopneumonia, organism unspecified
486	Pneumonia, organism unspecified
487	Influenza
488	Influenza due to identified avian influenza virus
490	Bronchitis, not specified as acute or chronic
494	Bronchiectasis
34.0	Streptococcal sore throat
79.0	Adenovirus
79.1	ECHO virus
79.2	Coxsackie virus
79.3	Rhinovirus
79.6	Respiratory syncytial virus (RSV)
79.98	Unspecified chlamydial infection
79.99	Unspecified viral infection
518.0	Pulmonary collapse
518.4	Acute edema of lung, unspecified
518.81	Acute respiratory failure
518.82	Other pulmonary insufficiency, not elsewhere classified
518.84	Acute and chronic respiratory failure
519.3	Other diseases of mediastinum, not elsewhere classified
784.1	Throat pain
786	Dyspnea and respiratory abnormalities
786.00	Respiratory abnormality, unspecified
786.05	Shortness of breath
786.06	Tachypnea
786.07	Wheezing
786.1	Stridor
786.2	Cough
786.3	Hemoptysis
786.4	Abnormal sputum
786.52	Painful respiration

All patient medical record information is entered in real-time and are maintained in the VA Computerized Patient Record System (CPRS). Our investigators performed a chart review of the CPRS data available for the patients at the time of their evaluation and recorded the information of interest to this study including chief complaint and assigned ICD-9 code, age, signs and symptoms (fever ≥38°C, respiratory ICD-9 code, cough, sore throat, shortness of breath, myalgias, chills/sweats, headache, fatigue), antibiotic and decongestant prescriptions, and chest radiograph orders. Chief complaint, ICD-9 code were coded as text, age was categorized in four levels (<50, 50–59, 60–69, ≥70) and all other variables were coded as present/absent. In addition, each patient was determined as having ILI or not using the North Carolina Division of Public Health case definition for ILI (17, 18), which is one respiratory symptom (cough, cyanosis, difficulty breathing, hemoptysis, hypoxia, pleural effusion, pleurisy, pneumonia, respiratory stridor, dyspnea, tachypnea) and one constitutional sign (achy, body aches, chills/shivers/rigors/shakes, diaphoresis/sweaty, dizziness, drowsy/sleepy/tired/exhausted/fatigue, fever (≥38°C)/febrile/FUO/temperature, hurts all over, joint pain, light headed, loss of appetite/poor/decreased/no appetite, malaise, muscle aches, myalgia, prostration, weariness, wooziness). See Appendix Table 2 for the complete case definition (17, 18).

Table 2.

Case definition for influenza-like illness (ILI)

Initiated 11/19/03

Revised 12/8/03, 2/24/04, 5/5/04, 10/31/04, 2/13/06, 2/24/06, 3/8/06

Source: NC Division of Public Health

Clinical description

Acute (≤14 days) onset of lower respiratory tract disease (from larynx to lungs)

Excludes certain chronic conditions

Specific signs and symptoms

Must have at least one respiratory AND one constitutional sign or symptom

May also have other symptoms

1. Respiratory: cough, cyanosis, difficulty breathing, hemoptysis, hypoxia, pleural effusion, pleurisy, pneumonia, respiratory stridor, shortness of breath/SOB/dyspnea, tachypnea/increased respiratory rate

2. Constitutional: achy, body aches, chills/shivers/rigors/shakes, diaphoresis/sweaty, dizziness, drowsy/sleepy/tired/exhausted/fatigue, fever/febrile/FUO (temperature ≥38°C), hurts all over, joint pain, light headed, loss of appetite/ poor/decreased/no appetite, malaise, muscle aches, myalgia, prostration, weariness, wooziness

In addition, any of these conditions by themselves is considered ILI: anthrax, plague, SARS, tularemia, influenza

Comments:

• Excluded conditions: CHF

• Diagnoses of particular concern:

– Category A agents: anthrax, plague, tularemia

– Other diagnoses of public health priority: SARS, influenza

– Chemical agents:

○ Vesicants/blister agents: sulfur mustard, lewisite, nitrogen mustard, mustard lewisite, phosgene-oxime

○ Pulmonary/choking agents: phosgene, chlorine, diphosgene, chloropicrin, oxide of nitrogen, sulfur dioxide

– Ricin (Castor bean oil extract)

– T-2 mycotoxins: Fusarium, Myrotecium, Trichoderma, verticimonosporium, Stachybotrys

Because a complete chart review of all patients presenting with fever or respiratory symptoms was impractical, we reviewed a total of 1,116 patients, 750 from the period May 2002 to June 2007 and 366 from July 2007 to October 2009. The absolute number of patients who presented with both fever and respiratory symptoms was much smaller than those with fever only, and this in turn was smaller than those with respiratory symptoms only. Patients were selected randomly and separately for the two time periods using stratified sampling to obtain approximately equal numbers of patients for each stratum; the sampling probabilities in each time period were 0.135 for those with fever only, 0.024 for those with respiratory symptoms only, and 0.700 for those with both fever and respiratory ICD-9 code. We used the larger sample from the earlier time period for model development, while data from the later period served as a validation set.

Statistical methods

Associations between ILI and the demographic and clinical variables were assessed with contingency tables and bivariable logistic models. Several multiple logistic regression models were fit to determine the set of clinical variables that best predict patients who met the North Carolina ILI criteria. A basic model included parameters for fever and respiratory ICD-9 code only. A full model included all of the clinical parameters available at ED evaluation. A more parsimonious model was developed using stepwise, backward elimination of variables from the full model and included only those variables that were significant at p≤0.05. A final model removed from the parsimonious model the prescription and radiograph variables, since these might not be immediately available at the evaluation of the patient. We used weighted logistic regression methods for survey data to account for the differential sampling probabilities and obtain variance estimates adjusted for the sampling design defined by strata of fever and acute respiratory complaints.

The relative performance of the development models was evaluated by comparing the receiver-operator characteristic (ROC) curves and C statistic (AUC or the area under the curve) for these models. The ROC analysis was also used to examine the validity of the development models as follows. The vector of parameter estimates from the development models was applied to the vector of variables for the patients in the validation sample to create a risk score for each individual. Then, the sensitivity, specificity, ROC curves, and AUC were calculated in the validation sample on a series of dichotomizations of ILI or not for each value in the range of risk scores. These results were compared for the development and validation samples.

All analyses were carried out with SAS 9.2 (Cary, NC). The research protocol for this study was approved by the Durham VAMC's Institutional Review Board in November 2006 (Research Protocol #0016).

Results

Comparison of ILI and covariates in the development and validation samples

After weighting the development sample, 3,423 (32.1%) of 10,667 patients evaluated in the ED met the NC case definition for ILI. The proportions of ILI among the fever and respiratory complaint subgroups were (1) 21.9% (266 of 1,211) for fever only, (2) 30.4% (2,739 of 9,005) for acute respiratory ICD-9 code, and 92.7% (418 of 451) for those with both. Similar proportions were found in the validation sample, where 1,677 (30.7%) of 5,464 patients had ILI by the NC definition and these were distributed as: (1) 22.4% (217 of 967) for fever only, (2) 29.4% (1,244 of 4,229) for acute respiratory ICD-9 code, and (3) 80.6% (216 of 268) for those with both. In both samples, those with both fever and respiratory symptoms met the case definition at much higher rates than those with either of these alone.

The distributions of the symptoms and medical management variables in the development and validation samples are shown in Table 1. The patients in these samples differed somewhat in age distribution and those in the validation set had higher rates of fever, chills/sweats, and fatigue. Patient management also differed between the two time periods, with higher rates of X-ray and fewer prescriptions of decongestants in the more recent validation sample.

Table 1.

Comparison of demographic and clinical characteristics of the VA patient populations from which the model development and validation samples were drawn

Parameter	Development (%)	Validation (%)	Fisher's exact p-value
Age (years)
< 50	23.5	22.1	0.007
50–59	33.9	24.0
60–69	18.1	31.7
≥ 70	34.4	22.2
Fever	15.6	22.6	0.003
Respiratory Sx	88.6	82.3	0.002
Cough	67.3	66.4	0.828
Sore throat	23.5	26.7	0.448
Dyspnea	26.5	31.7	0.247
Myalgia	13.6	18.0	0.210
Chills/sweats	19.0	27.6	0.031
Headache	13.1	9.7	0.278
Fatigue	9.5	16.7	0.019
Antibacterial	45.7	44.5	0.807
Decongestant	28.4	6.4	<0.0001
Chest X-ray	24.9	42.9	<0.0001

Predictive model performance

The parameter estimates from the four models are shown in Table 2 and the ROC curves in Fig. 1. Model 1 with only fever and respiratory symptoms as predictors performed less well relative to the other models, but the C statistic of 0.846 indicates that it fits the data well, nonetheless. The performance of models 2, 3, and 4 were essentially equivalent, as indicated by the coincident ROC curves and nearly identical C statistics.

Table 2.

Bivariate associations and parameter estimates from the development models predicting influenza-like illness.

	Estimate (Standard Error)
	Bivariate association	Model 1	Model 2	Model 3	Model 4
Intercept		−4.62 (0.19)	−13.31 (0.40)	−13.60 (0.39)	−13.17 (0.38)
Age (years)a
< 50	1		1
50–59	−0.49 (0.32)		−0.46 (0.11)
60–69	−0.48 (0.38)		−0.61 (0.13)
= 70	−0.72 (0.36)		−0.51 (0.13)
Fever	0.47 (0.17)	3.35 (0.18)	5.79 (0.28)	5.76 (0.28)	5.90 (0.28)
Resp ICD-9 code	0.58 (0.22)	3.79 (0.19)	5.51 (0.31)	5.51 (0.31)	5.83 (0.31)
Cough	1.98 (0.32)		3.68 (0.15)	3.61 (0.14)	3.91 (0.13)
Sore throat	1.88 (0.30)		2.39 (0.11)	2.39 (0.11)	2.32 (0.10)
Dyspnea	−0.85 (0.30)		0.01 (0.11)
Myalgia	2.71 (0.44)		4.99 (0.17)	4.99 (0.17)	4.94 (0.16)
Chills/sweats	2.38 (0.31)		4.02 (0.12)	4.02 (0.11)	4.12 (0.11)
Headache	1.59 (0.35)		3.60 (0.13)	3.60 (0.12)	3.83 (0.12)
Fatigue	2.27 (0.42)		4.42 (0.16)	4.35 (0.15)	4.53 (0.15)
Antibacterial	1.04 (0.25)		1.29 (0.09)	1.24 (0.09)
Decongestant	1.30 (0.28)		1.11 (0.10)	1.08 (0.10)
Chest X-ray	0.57 (0.27)		−0.03 (0.11)

a Age categories parameterized as indicator variables, with reference level <50.

Fig. 1

Comparison of ROC curves for the four development models. Variables included in the models are shown in Table 2. The ROC curves and values for the area under the ROC curve (AUC) were obtained from the multivariable logistic regression models.

Evaluation of the models in the validation sample

The performance of development models in the validation sample was excellent. The estimates from these models on the validation set gave C statistics of 0.757 for model 1 and 0.894 for the other three models. The ROC curves comparing model 4 in the development and validation samples shown in Fig. 2 illustrate the good concordance of the model in the two datasets. The fit for the other three models was similar.

Fig. 2

Comparison of the ROC curves for model 4 in the development and validation samples. The parameter estimates from the development model 4 were applied to the validation sample data to calculate the sensitivity, specificity, and area under the ROC curve (AUC).

Discussion

We have developed and validated several models that classify ILI in patients presenting to our emergency department. Including a set of symptoms and management practices that are commonly, but not exclusively, associated with ILI improved the model fit and discriminated ILI better than the current screening based on fever and respiratory symptoms alone. Accurate diagnosis of ILI is important for guiding appropriate clinical management of patients who have, and those who do not have, ILI. In addition, the novel H1N1 influenza pandemic that developed over the last year illustrated the need for rapid and accurate reporting of ILI for public health efforts.

This study required a chart review of the triage, chief complaint, and physician notes to acquire the analysis data. For this study, we did this manually, which is a time-consuming process. Since the clinical data are available electronically in real-time at VA Medical Centers, natural language processing methods could be used on free-text notes to extract the information used in these models, and the model results could be incorporated into the VA data systems to automate and immediately flag patients who are likely to have ILI. Numerous examples of natural language processing and disease classification systems have been developed for clinical purposes (19-22) and for syndromic surveillance (22-25).

While we have demonstrated the utility of predictive modeling for a straightforward illness, this method could be adapted to surveillance for other infections and clinical syndromes including natural infectious disease outbreaks, surgical site infections, and disease clusters caused by infrequently encountered agents that might be used in a bioterrorist attack (26, 27). Developing formal predictive models for bioterrorism agents would be more difficult than for natural outbreaks due to a lack of cases to use in developing models. As an alternative, a scoring system could be developed using the symptoms known to be associated with other agents, and these parameters could be weighted based on how common they are in affected individuals (27).

Although the models we constructed here were shown to be valid when applied to a temporally distinct sample of the same patient population, a caveat to this study is that the results are not necessarily extensible to other patient populations. The VA patients are older than patients seen at non-VA facilities and the elderly can present very differently than younger people. In addition, women are under-represented in the VA population, and veterans generally have more health problems than the general population.

Temporal differences in patient populations can also affect the validity of these models. Over the time period of this study, we noted two changes in the VA patient population. First, due to a shift toward relatively fewer WW II and more Viet Nam era veterans, the age distribution now includes fewer patients ≥70 years and more in the 60–69 year stratum. Second, our VA medical center engaged in a successful effort to encourage patients to use VA-associated regional outpatient clinics rather than the Durham VAMC emergency department. Together, these changes may account for the differences in presentation and management of patients: if the less sick patients now preferentially use the regional clinics, then the Durham facility patients could now be sicker, leading to more symptoms and more aggressive clinical evaluation and management. Given the potential for temporal changes in populations, predictive models like the ones used here should be re-evaluated periodically.

An important aspect of this study is the use of real-time clinical data available in the VA medical system. Including predictive models in the data system can allow for rapid and automated identification of syndromes that might be associated with disease outbreaks. As more hospitals deploy electronic clinical data systems, surveillance for important infections and diseases could be improved.