Emergency Medical Text Classifier: New system improves processing and classification of triage notes.

OBJECTIVE: Automated syndrome classification aims to aid near real-time syndromic surveillance to serve as an early warning system for disease outbreaks, using Emergency Department (ED) data. We present a system that improves the automatic classification of an ED record with triage note into one or more syndrome categories using the vector space model coupled with a 'learning' module that employs a pseudo-relevance feedback mechanism.
MATERIALS AND METHODS: Terms from standard syndrome definitions are used to construct an initial reference dictionary for generating the syndrome and triage note vectors. Based on cosine similarity between the vectors, each record is classified into a syndrome category. We then take terms from the top-ranked records that belong to the syndrome of interest as feedback. These terms are added to the reference dictionary and the process is repeated to determine the final classification. The system was tested on two different datasets for each of three syndromes: Gastro-Intestinal (GI), Respiratory (Resp) and Fever-Rash (FR). Performance was measured in terms of sensitivity (Se) and specificity (Sp).
RESULTS: The use of relevance feedback produced high values of sensitivity and specificity for all three syndromes in both test sets: GI: 90% and 71%, Resp: 97% and 73%, FR: 100% and 87%, respectively, in test set 1, and GI: 88% and 69%, Resp: 87% and 61%, FR: 97% and 71%, respectively, in test set 2.
CONCLUSIONS: The new system for pre-processing and syndromic classification of ED records with triage notes achieved improvements in Se and Sp. Our results also demonstrate that the system can be tuned to achieve different levels of performance based on user requirements.

Introduction and Background

The primary purpose of public health syndromic surveillance is to serve as an early warning system for disease outbreaks. In addition, syndromic surveillance systems provide situational awareness, allowing public health officials to monitor ongoing disease events, and identify related factors [1]. Syndromic surveillance can be framed as a classification problem: given syndrome definitions and information from the visit record, determine whether the record is likely to represent an instance of one, more than one, or none of the syndromes of interest. The costs of incorrect classification include unnecessary work for the public health official if a record is incorrectly classified as a syndrome instance requiring investigation or other action (a false positive), and not recognizing a potential outbreak in a timely manner if it is not classified (a false negative). Syndrome definitions include a text description (e.g., “acute infectious gastrointestinal illness of 7 days or less”) and a list of symptoms that are associated with the syndrome (e.g., diarrhea, fever). Therefore, a successful syndromic surveillance system must map symptoms as they are expressed in the visit record against those included in the syndrome definition, and determine if classification as an instance of the syndrome is warranted.

Early detection of outbreaks by syndromic surveillance systems depends on community-wide health-related data which are: 1) available in a timely manner and 2) quickly and accurately classified. Syndromes are typically created by local and state public health jurisdictions who may make use of bioterrorism syndromes defined by Centers for Disease Control and Prevention (CDC) (e.g., botulism-like, hemorrhagic illness) as well as other syndromes of interest (e.g., influenza-like illness, gastrointestinal) defined by the syndromic surveillance community through a consensus process [2,3]. Traditional surveillance systems use health-related data such as lab reports and final diagnoses to confirm the presence of diseases that meet case definitions such as influenza or MRSA [1,4]. In contrast, syndromic surveillance systems rely on symptom data recorded during Emergency Department (ED) patient visits. ED records contain timely clinical data which have been shown to act as an early warning system [5]. Although the diagnosis data and lab results from the ED record would provide the most accurate information for surveillance, these data elements are not available in a timely manner. In a study of diagnosis data available in a statewide public health surveillance system, the majority of ED diagnoses were not available for days to weeks after the ED visit [6]. For syndromic surveillance, "timeliness" is defined in terms of hours, and action by public health officials may be warranted before there is a definitive test result or diagnosis. Pre-diagnostic data used for surveillance include initial vital signs (e.g., measured temperature, heart rate) and chief complaint (CC) [7-9]. Researchers have also explored the addition of different portions of the electronic medical record to capture data in addition to chief complaint, such as discharge prescriptions, diagnostic test orders, structured clinical notes, and triage nurses' notes in narrative form [10-13].

Non-coded clinical data in unstructured (free) text form provide rich information for syndromic surveillance, but working with these data can be challenging. The triage note (TN) and chief complaint (CC) fields of an ED record capture the very initial interactions of clinicians with a patient. The CC describes the primary reason for the patient's ED visit in a few words or medical terms, while the TN, when present, contains a narrative with more detail about the history of the present illness and sometimes includes the nurse’s observations. Both fields may contain terms that represent syndrome-related symptoms, and can thus be used as evidence for the classification decision [14]. Because the CC is usually briefer and more focused, it is generally easier to automatically extract symptoms from the CC than from the TN. The TN is more wide ranging in the information it includes, as shown in Table 1. All three ED records have a CC of fever. When the triage note is added, each record meets a different syndrome definition, based on the highlighted keywords. Many syndromic surveillance systems use only the CC for syndromic classification, but the addition of data from clinical notes has been shown to increase the accuracy of syndromic surveillance [10]. In a pilot study, the addition of information from TNs led to improved sensitivity for three syndromes, Respiratory, Fever-Rash and Gastrointestinal, from 17% - 40% to over 81%, while specificity was maintained above 80% [11]. However, use of TNs can also increase the risk of false classification, since they can include syndrome-relevant information in non-standard terms, such as abbreviations, misspellings, negated forms, and other expressions characteristic of free-text notes [15].

Table 1

Examples of chief complaints and triage notes

Chief Complaint	Triage Note	Syndrome
Fever	diarhea then febrile x2d now vom	Gastrointestinal
Fever	Pt c/o wheezy cough, chest/throat sore, felt hot, w/shaking chills. Worse in last 48 hrs.	Respiratory
Fever	Mom reports child awoke w/fine red rash on trunk.. Lethargic, temp up to 103.2.Pt listless.	Fever-Rash

The development of syndromic surveillance systems using textual data is based on two major research areas: text mining of medical data to identify relevant concepts, and automated classification based on textual features. Meystre et al. [16] discuss some established techniques for information extraction from clinical unstructured text, such as incorporating supplemental information sources (e.g., the UMLS), mapping terms to a standard representation (e.g., ICD-9-CM codes), and cleaning and normalizing text using pre-processing software (e.g., Chief Complaint Processor (CCP), Emergency Medical Text Processor (EMT-P), NegEx, ConText) [3,7,17-27]. Although structured data would be easier to work with, fields such as the triage note are usually recorded in unstructured text. A system that is unable to extract concepts from these fields risks missing valuable information.

The North Carolina Disease Event Tracking and Epidemiologic Collection Tool (NC DETECT) is a state public health surveillance system which receives twice daily feeds of ED visit data. During the period covering the data used in this study, the number of hospitals providing data grew from 94 of 113 (83%) in 2006 to 110 of 112 (98%) by the beginning of 2009 [6,28]. All records contain the patient’s chief complaint and, if available, also include initial vital signs and triage notes. Once received, CC text is cleaned and normalized using Emergency Medical Text processor (EMT-P) [23] to correct misspellings, expand abbreviations, map synonyms to preferred terms, etc. Negated concepts are identified using NegEx [17]. After this pre-processing, NC DETECT classifies the records as positive or negative for each of several syndromes. The syndrome definitions are represented as Structured Query Language (SQL) queries; classification is thus operationalized as a query to the NC DETECT database of pre-processed ED visit records. Approximately 13,000 new ED records are processed each day.

This translational research project created a system that can be deployed for near-real time syndromic surveillance. The system employs techniques drawn from natural language processing, text mining, and automatic classification, to extract symptoms from triage notes. The goal was to further increase sensitivity above that found in the pilot study [11], while not sacrificing specificity, in order to reduce manual intervention to identify false positives and increase the overall reliability of the system. NegEx [24] has been shown to help reduce false positives by identifying negated terms and concepts in unstructured text including TNs [29], but we found that it is not possible to compile a comprehensive list of all such terms. With our new system we sought to identify new syndrome-relevant terms automatically from a training set to produce a more robust classification that could be used by surveillance systems.

Previous Research

At the time the study data were collected, only 27% of all visits in the NC DETECT database contain a TN; however that percentage has been growing and the increase is expected to continue as more EDs capture the TN electronically. Previously, researchers found that the sensitivity for acute respiratory surveillance improved from 13% without a TN to 35% with a TN [14] and that a disproportionate percentage (46%) of visits that are flagged as positive for one or more syndromic surveillance reports include a TN [11]. Although the additional terms found in the TN may improve the system's recognition of syndrome-positive records, they also increase the possibility of false positives.

NC DETECT currently uses a rule-based classification approach which is triggered by the presence of specific terms or concepts. An advantage of this approach is that the rule is a relatively straight-forward translation of the expert knowledge in the syndrome definition into the query. For example, the GI symptom "vomiting" is implemented as the inclusion of "vomiting" as a term in the SQL query. However, such term-by-term transfer is insufficient: synonyms and other ways of expressing a symptom must also be included; the query must also include "emesis", "threw up" and so on. This type of rule maintenance must be performed by hand [14,30,31].

Table 2 shows results of testing NC DETECT queries against a manually classified sample [14,32].These results illustrate the challenge of syndromic surveillance: previous methods tend to generate low sensitivity and high specificity. Increasing sensitivity improves case detection, but is usually accompanied by a decrease in specificity, resulting in an increased burden for public health staff who must review positive signals.

Table 2

Baseline NC DETECT Performance

	Gastrointestinal [ 32 ]	Respiratory [ 14 ]	Fever-Rash [ 32 ]
# records manually classified (weighted to total # records)	3353(2,418,168)	3699(956,015)	3640(2,418,168)
Sensitivity:	0.28	0.23	0.45
Specificity:	0.97	0.99	0.99

Classifiers can be developed by training machine learning algorithms, such as decision trees, support vector machines (SVM), and Bayesian classifiers (usually coupled with a weighting scheme), to identify patterns of relevant features that predict membership of a text file or patient case to a class [8,31,33]. One advantage of machine learning is that the system may learn patterns that are not obvious to a person. Features used by the classifier may include terms or sets of related terms (e.g., synonyms), and the features can be weighted since the presence of some features may provide stronger evidence than the presence of others. In machine learning, the classification model built during training is used to predict the correct classification for each incoming record. For syndromic surveillance, an accurate model will correctly identify records that are instances of a syndrome, based on the information available in the record, while minimizing false positives and false negatives. Selection of useful features to form the model is critical; preprocessing CC and TN improves the quality of features available for the model. Previous efforts on term expansion in medical information retrieval systems have used the Unified Medical Language System (UMLS) as a knowledge source to add only terms relevant to the context [34,35].

Relevance feedback is a strategy used to improve information retrieval performance by adding to and/or re-weighting the initial query [36-38]. The user identifies relevant documents returned by the initial query, which the system then uses as a source for additional query terms (e.g., synonyms) or as a basis for re-weighting existing query terms [39]. In machine learning, this technique may help identify new features or re-weight existing features by incorporating new information from the input stream or the incoming document set [39]. Fully automated systems use blind feedback, also known as pseudo-relevance feedback [34,40], which eliminates the need for human intervention. The system described here uses pseudo-relevance feedback to identify new features from classified ED records.

Study Objective

The objective of this study was to develop and test an automatic system for syndrome classification. Goals for the new system were to draw information from TNs as well as the CC, be easily adaptable to include new syndromes, and respond to changes in existing syndrome definitions and new ED data. Also, the new system should improve upon the performance of the baseline SQL queries currently in use by NC DETECT.

Methods

Our new system for processing and classifying ED records with triage notes was developed using syndrome definitions established by the CDC and expert consensus [2,3] as the basis for the initial query, making it comparable to the existing query or rule-based approach. The pseudo-relevance technique was based on the assumption that the top-ranked documents (i.e., ED records) returned by the initial query are relevant, and thus can serve as a source for additional terms. All terms were used as features in our vector space model for syndrome classification [41]. Our previous work [41] describes the basic vector space model, along with initial results obtained for GI syndrome. The version of the system described in this paper incorporates additional sources of features, including syndrome exclusion criteria and the UMLS.

We evaluated the new system in our informatics laboratory using data from the NC DETECT warehouse for ED visits from 2006-2008. This version was tested on 3 syndromes, Gastrointestinal Severe (GI), Respiratory (RESP), and Fever-Rash (FR), as defined by the NC DETECT Syndrome Definitions Workgroup.

System Components

Figure 1 illustrates the architecture of our new system, called Emergency Medical Text Classifier (EMT-C).

Figure 1

EMT-C Architecture

There are two inputs to the system. The first is a vector representing the Master Term List (MTL) for a syndrome; we describe its construction in the next section. The second is a set of pre-processed ED records which have been normalized using EMT-P modules [22,23]. These modules normalize terms commonly found in CC and TN (e.g., acronyms, abbreviations, truncations, misspellings and coordinate structures) to standardized terms from the UMLS. Note that although NC DETECT receives structured data from other fields of the ED record, EMT-C uses data only from the CC and TN.) A record's normalized terms are then translated to a binary vector, where 1 represents a term in the MTL also present in the record, and 0 represents a MTL term that is absent from the record. The record's vector is compared to the MTL vector using cosine similarity.

A mean similarity value is calculated from similarity values for a set of records, such as all records submitted to NC DETECT during a 24-hour period, and is used as the threshold for classifying the records in that set. Cosine similarity between a TN vector (v1) and syndrome vector (v2) is defined as:

Records with a similarity value greater or equal to the mean are classified as syndrome positive, and those whose similarity value is below the threshold are classified as syndrome negative. This design decision is based on the assumption that syndromic surveillance is based on identifying surges in the number of records fitting a syndrome definition. The threshold is set to a lower similarity value if most records in the batch are dissimilar to the MTL; conversely if most records are highly similar, a higher threshold is needed to identify a surge.

Creating the Master Term List

The Master Term List (MTL) represents the syndrome definition to which each ED record is compared. The initial MTL consisted only of terms in the existing NC DETECT baseline SQL query. This list is pre-processed using the same EMT-P modules used for ED records, and represented as a binary vector. This vector contains only 1s, as all MTL terms are present in the MTL vector.

Training Set

A training set of ED records was pre-processed with EMT-P modules, and then translated into vectors for comparison with the MTL vector as shown in Fig. 2. The training set consisted of 259,365 ED records that had been classified as positive by the baseline NC DETECT SQL queries for one or more of three syndromes (Gastrointestinal, Respiratory, Fever-Rash) over a period of three years (2006-08).

Figure 2

Creation of Master Term List (MTL) Vector

Based on the assumption that the n records with the highest similarity values are true positives for the syndrome, terms from these pre-processed records not already in the MTL are added to it. This forms the pseudo-relevance feedback loop. The augmented MTL is translated into a new vector, as are the records, and their vectors are compared again. The feedback loop could iterate multiple times; as new records from the training set rise into the top n they can be harvested for new terms, (although over-fitting would be a risk if more than a couple of iterations were run). For this experiment we iterated once and performance improved. Upon a second iteration, performance degraded, so in the production version of EMT-C, the system iterated only once. Any future revision of the MTL would be triggered by degradation in performance.

The pseudo-feedback approach allows EMT-C to automatically incorporate terms from actual records that are not in the syndrome definition, thus expanding the evidence it uses for classification. For example, after the initial classification for the Respiratory syndrome, high-ranking records contained the terms "crackle" and "albuterol", which were not in the initial MTL list. "Crackle" describes abnormal lung sounds, and "albuterol" is a medication used to treat respiratory conditions. These terms were then added to the MTL list.

Evaluation

We tested EMT-C on two sets of ED records from NC DETECT that were previously manually annotated for syndromic surveillance research [14,22]. Each record contained the TN, CC and vital signs from the ED visit. A set of 485 was used for initial pilot testing of EMT-C [22]. The system was modified based on these results before final testing on a set of 3,053 [14]. The TN terms from this set were weighted by their term frequency-inverse document frequency (tf-idf) values, a standard information retrieval method used to identify terms that provide good discrimination between relevant and non-relevant documents [42]. A subset of the highly weighted terms was used to construct the master term list for classifying the records in the final testing dataset.

Gold Standard Set:Records in both test sets [14,22] had been manually classified by three clinicians as part of the previous studies. The clinical experts used the entire ED record (CC, TN, final diagnosis, measured temperature, admitted v. discharged) to retrospectively determine whether the visit conformed to one of the NC DETECT syndrome definitions (Gastrointestinal Severe, Respiratory, Fever-Rash). Though the 3 classes are not mutually exclusive in the production environment, the samples we used for this study contained records that were positive for no more than one syndrome. The initial agreement between two of the clinicians was measured with the kappa statistic and for the two studies was 0.76 and 0.82 respectively [14,22]. A third clinical expert adjudicated the cases with disagreement, and these final judgments were treated as the gold standard classifications for this study. Table 3 lists the number of positive and negative records for each syndrome in both test sets. TNs in the EMT-C pilot dataset contain an average of 22 words (128 characters) per TN. TNs in the final testing dataset contain an average of 25 words (151 characters) per TN.

Table 3

Distribution of records in pilot and final test sets

Syndrome	Pilot study dataset N=485		Final test dataset N=3053
	Positive (%)	Negative (%)	Positive (%)	Negative (%)
Gastrointestinal	80 (16.5%)	405 (83.5%)	23 (0.8%)	3030 (99.2%)
Respiratory	87 (17.9%)	398 (82.1%)	171 (5.6%)	2882 (94.4%)
Fever-Rash	5 (1.0%)	480 (99.0%)	249 (8.2%)	2804 (91.8%)

Test Plan

EMT-C performance was measured in terms of sensitivity and specificity values calculated by comparing the system output with gold standard classification. Sensitivity is defined as the ratio of correctly classified syndrome positive records to gold standard syndrome positive records, while specificity is the ratio of correctly classified syndrome negative records to gold standard syndrome negative records.

Weighted versions of sensitivity and specificity were used for the final testing dataset to reflect the unbalanced stratified sampling used to select these records and make the results generalizable to the entire set of records. Several configurations of EMT-C were tested to assess the trade-off between sensitivity and specificity values. Configurations were based on 4 variations.

Augmentation of the MTL with terms from the training set. In one condition, the MTL contained only terms drawn from the syndrome SQL definition; in the other, it was augmented using terms extracted from the training set. These added terms were those with the highest tf-idf rankings.

Source of terms for record vector. In one condition, the vector was built using terms from both the TN and CC. In the other, terms were drawn only from the TN. In some records, the two fields contain the same terms: use of both fields does not add any additional information, nor does it add any "deceptive" terms. In other records, the two fields contain different terms. As mentioned earlier, this is a double-edged sword, providing stronger evidence of a true positive, or deceptive information leading to a false positive.

Use of exclusion terms. The NC DETECT syndromes include standalone exclusion criteria for each of the syndromes [7]. For example, the gastrointestinal syndrome excludes records with “Crohn” and “irritable bowel” since these chronic conditions are associated with symptoms that are identical to many of those in the GI syndrome, which could lead to false positive classification. The experiments were run with and without the use of exclusion terms.

Use of pseudo-relevance feedback. In one condition, the MTL was augmented through the pseudo-relevance feedback method described earlier; in the other, the feedback loop was omitted.

Results

Table 4 shows the maximum results of EMT-C processing on the final test set. Included in the table are the highest levels of Sensitivity (Se) and Specificity (Sp) produced for each syndrome, along with the settings for the variations (described in the previous section) for each value. Tables 5 and 6 provide the complete results for the pilot and final test sets, respectively.

Table 4

Maximum weighted Sensitivity and Specificity values, and the variation settings for the configurations that produced them, on Final Test Set (n = 3053 weighted to 1.34 million).

	Gastrointestinal	Respiratory	Fever-Rash
Maximum Sensitivity	0.971. MTL terms: syndrome2. Record terms: TN + CC3. Exclusion terms: either4. Relevance feedback: with	0.911. MTL terms: syndrome2. Record terms: TN + CC3. Exclusion terms: either4. Relevance feedback: with	>0.991. MTL terms: either2. Record terms: TN + CC3. Exclusion terms: either4. Relevance feedback: either
Maximum Specificity	0.941. MTL terms: syndrome2. Record terms: TN3. Exclusion terms: with4. Relevance feedback: without	0.911. MTL terms: either2. Record terms: TN3. Exclusion terms: either4. Relevance feedback: without	0.931. MTL terms: syndrome2. Record terms: TN3. Exclusion terms: either4. Relevance feedback: without

Table 5

Sensitivity and specificity values for all system configurations of EMT-C on Pilot Test Set (N = 485). Maximum values for sensitivity (Se) and specificity (Sp) are bolded. Configuration settings based on 4 system variations 1) Augmentation of Master Term List, 2) Source of terms for record vector, 3) Use of exclusion terms, 4) Use of pseudo-relevance feedback.

Master Term List Configuration	Gastrointestinal				Respiratory				Fever-Rash
	3. Without Exclusion Terms
	4. Without relevance feedback		4. With relevance feedback		4. Without relevance feedback		4. With relevance feedback		4. Without relevance feedback		4. With relevance feedback
	Se	Sp	Se	Sp	Se	Sp	Se	Sp	Se	Sp	Se	Sp
1. MTL terms:syndrome2. Record terms:TN+CC	0.89	0.86	0.90	0.71	0.87	0.86	0.97	0.73	1	0.87	1	.086
1. MTL terms:syndrome2. Record terms:TN	0.76	0.88	0.96	0.56	0.83	0.86	0.97	0.73	1	0.87	1	0.87
1. MTL terms:syndrome +training2. Record terms:TN+CC	0.88	0.86	0.94	0.57	0.86	0.86	0.97	0.73	1	0.58	1	0.57
1. MTL termssyndrome +training2. Record terms:TN	0.75	0.88	0.90	0.59	0.82	0.87	0.95	0.73	1	0.59	1	0.58
	3. With Exclusion Terms
1. MTL terms:syndrome2. Record terms:TN+CC	0.83	0.88	0.84	0.78	0.87	0.86	0.97	0.73	1	0.87	1	0.87
1. MTL terms:syndrome2. Record terms:TN	0.71	0.90	0.90	.063	0.83	0.86	0.97	0.73	1	0.87	1	0.87
1. MTL terms:syndrome +training2. Record terms:TN+CC	0.83	0.88	0.88	0.67	0.86	0.86	0.97	0.73	1	0.59	1	0.58
1. MTL termssyndrome +training2. Record terms:TN	0.71	0.90	0.85	0.66	0.82	0.87	0.95	0.73	1	0.60	1	0.58

Table 6

Weighted sensitivity and specificity for all system configurations of EMT-C on Final Test Set (N=3053 weighted to 1.34 million). Maximum values for sensitivity (Se) and specificity (Sp) are bolded. Configuration settings based on 4 system variations 1) Augmentation of Master Term List, 2) Source of terms for record vector, 3) Use of exclusion terms, 4) Use of pseudo-relevance feedback.

Master Term List Configuration	Gastrointestinal				Respiratory				Fever-Rash
	3. Without Exclusion Terms
	4. Without relevance feedback		4. With relevance feedback		4. Without relevance feedback		4. With relevance feedback		4. Without relevance feedback		4. With relevance feedback
	Se	Sp	Se	Sp	Se	Sp	Se	Sp	Se	Sp	Se	Sp
1. MTL terms:syndrome2. Record terms:TN+CC	0.83	0.92	0.97	0.76	0.65	0.89	0.91	0.62	>0.99	0.91	>0.99	0.69
1. MTL terms:syndrome2. Record terms:TN	0.77	0.93	0.86	0.79	0.54	0.91	0.87	0.64	0.96	0.93	0.98	0.79
1. MTL terms:syndrome +training2. Record terms:TN+CC	0.94	0.76	0.94	0.78	0.64	0.90	0.81	0.80	>0.99	0.91	>0.99	0.73
1. MTL termssyndrome +training2. Record terms:TN	0.92	0.80	0.86	0.81	0.54	0.91	0.76	0.80	0.96	0.93	0.98	0.76
	3. With Exclusion Terms
1. MTL terms:syndrome2. Record terms:TN+CC	0.77	0.93	0.91	0.79	0.65	0.89	0.91	0.62	>0.99	0.91	>0.99	0.69
1. MTL terms:syndrome2. Record terms:TN	0.72	.094	0.80	0.81	0.54	0.91	0.87	0.64	0.96	0.93	0.98	0.79
1. MTL terms:syndrome +training2. Record terms:TN+CC	0.89	0.80	0.89	0.81	0.64	0.90	0.81	0.80	>0.99	0.91	>0.99	0.73
1. MTL termssyndrome +training2. Record terms:TN	0.86	0.83	0.83	0.80	0.54	0.91	0.76	0.80	0.96	0.93	0.98	0.76

In all configurations of EMT-C in the final test set (Table 6), sensitivity was improved over the baseline NC DETECT queries (Table 2), while specificity decreased.

Augmenting the MTL with highly ranked (tf-idf) terms from the training set (variation 1) improved sensitivity for the GI syndrome in the absence of pseudo-relevance feedback, but decreased it for Respiratory syndrome in the same configurations. The effect of augmentation on specificity was mixed. For the GI syndrome, maximum values of both Se and Sp were produced without terms from the training set (see Table 4), but it decreased specificity in the absence of pseudo-relevance feedback. For the Respiratory syndrome, it had the opposite effect.

Sensitivity was higher in almost all configurations using pseudo-relevance feedback (variation 4) than in those without feedback for the GI and Respiratory syndromes. In contrast, specificity was higher in configurations without pseudo-relevance feedback. Maximum values of Se were produced using pseudo-relevance feedback for the GI and Respiratory syndromes; but maximum values of Sp were produced without feedback for all three syndromes (see Table 4).

Although drawing MTL terms from both TN and CC (variation 2) improved sensitivity in many configurations, in others, the opposite was true. This variation decreased specificity slightly in some configurations, and seemed to have no effect in others. Maximum Se values were produced using both TN and CC. In contrast, maximum Sp values were produced using TN alone (Table 4).

Using exclusion terms (variation 3) decreased sensitivity for the GI syndrome, and had little or no effect for the other two syndromes. There was slight improvement in specificity using exclusion terms for the GI syndrome, and again, little or no effect in the other two.

Maximum sensitivity values for GI and Respiratory were produced using pseudo-relevance feedback without augmentation from the training set. Sensitivity scores for the Fever-Rash syndrome were excellent for all configurations in the final test. This is likely due to the nature of the syndrome itself: its symptoms, and thus terms representing them, are not associated with other syndromes, and thus their presence or absence provides relatively unambiguous evidence for classification. In contrast, other symptoms such as headache and dizziness are associated with more than one syndrome, as well as with other non-syndromic conditions.

Maximum specificity values, on the other hand, occurred in configurations without pseudo-relevance feedback except for the GI syndrome when the MTL was augmented with additional terms. However, the increase was slight.

Discussion

The results indicate that EMT-C, a syndromic classification system based on the vector space model with pseudo-relevance feedback, is a promising approach to syndromic surveillance. In a laboratory setting, we achieved improvements in sensitivity with moderate decreases in specificity. Terms drawn from a training set of ED records improve performance over terms drawn only from the syndrome definitions. The unstructured text expressions found in ED CC and TN contain synonymous terms and phrases (often more colloquial in nature) for symptoms in the syndrome definitions. In addition, they may contain information related to syndromic symptoms that are not necessarily symptoms, but describe the patient's situation. The risk is that the additional terms may also be associated with false positives, thus, adding them increases the noise in the system. These findings reflect the difficulty of the surveillance task. Early warning of an outbreak is crucial to successful management, but the data that are available in near-real-time are often of mixed quality. The ED CC and TN are available in a timely fashion, and are a rich source of information, but are difficult to mine [43,44].

Our results also indicate that there may not be a single model or configuration that is ideal for all syndromes. This is apparent in comparing the Fever-Rash results with those of the other two syndromes, but there were also differences between Gastrointestinal and Respiratory. One contributing factor is the precision of symptom terms in identifying one and only one syndrome. Another factor to consider is the frequency of occurrence of a syndrome in the population (prevalence rate) [44], as well as the relative risk of false positives and false negatives in managing an outbreak. Public health officials weigh the relative risks involved in missing a case; their preferences should be reflected in system design. For a syndrome such as Fever-Rash, the risk of false negatives may be deemed greater than the burden of dealing with false positives, thus an EMT-C configuration that maximizes sensitivity may be preferred. The converse may be true for Respiratory; thus, maximizing specificity may be the goal [31,32]. These requirements highlight the advantage to a system architecture such as that used in EMT-C that supports multiple configurations.

Limitations

This study was carried out in an informatics laboratory setting using data from previous years. In the actual production environment, batched data are uploaded into NC DETECT twice daily. ED visits are only accepted for upload if they contain several key data elements, including CC, so some visits are added in subsequent batches. Also, additional data elements can be added after the initial upload. These production constraints may influence system performance.

The training set for this research consisted of ED records from 2006-2008. We do not know if the vocabulary used by triage nurses in CC and TN has changed over time. If it has, retraining would reveal any changes that were significant enough to affect syndromic classification, as well as improving system performance.

Future Work

Next steps include EMT-C testing in the production environment on current ED visit records. In addition to examining the effects of the constraints described above, we also plan to gather user input on how EMT-C performs in this environment. For example, while our approach has been to improve the ability to detect signals with the syndromic surveillance system, users may have differing needs for the sensitivity/specificity tradeoff.

Future work will also focus on leveraging the system architecture to improve system performance in several ways. One advantage of the EMT-C model is the relative ease with which it can be trained. The work presented here developed models for 3 syndromes: Gastro-intestinal, Respiratory, and Fever-Rash. We plan to expand coverage of EMT-C to include more syndromes for acute infectious diseases of interest (influenza, neurological). In principle, the EMT-C model should be applicable to other types of classification problems and to other types of unstructured text, although we have not done so at this time.

This version of EMT-C uses single terms as features, which is common in the literature (e.g., [39]). It may be that using unordered bigrams (pairs of words) as features would improve performance. This would allow the model to represent terms that commonly co-occur (e.g., “vomiting and diarrhea”) or are used as phrases in the CC and TN, such as “rash all over”.

EMT-C currently uses binary vectors to represent the MTL and the records. That is, a vector represents only the presence or absence of terms. The vector space model also supports weighted vectors, in which some terms are represented by values between 0 and 1 to indicate the strength of evidence they provide. The presence of a stronger term is then represented with a higher weight than that of a weaker term. Thus, a small number of strong terms or a larger number of weak terms are equivalent in terms of similarity value to the MTL vector. Weights can also be negative, indicating that the presence of a term is evidence against a syndrome classification.

We can also experiment with different ways of calculating the similarity threshold, such as averaging similarity values from a larger number of records, or setting a lower bound for the threshold. Training could also incorporate ED data produced later in the ED visit, such as diagnostic ICD-9-CM codes. Although not available for the real-time classification desired for syndromic surveillance, they may be helpful in building the MTL [4].

One purpose of this study was to investigate ways of utilizing the information provided in TNs to improve syndromic surveillance. Therefore, the records included in the test sets all included TNs. In reality, over 50% of ED records sent to NC DETECT do not include a TN at this time. We expect more EDs to include TNs in the future, following the trend we have observed over the past several years. However, as EMT-C is put into production, we will determine whether it should classify all ED records regardless of the presence or absence of the TN, or whether two processing streams should be established, i.e., EMT-C for records with the TN, and the existing NC DETECT SQL query for records without the TN.

Conclusion

We developed a new system for pre-processing and syndromic classification of ED records with triage notes that achieved improvements in sensitivity and specificity over previous results in the laboratory setting. Our work also shows that EMT-C can be tuned to achieve different levels of performance based on user requirements. After field testing, EMT-C could be incorporated into NC DETECT to augment or substitute for the SQL queries currently in use (described in section 1.1). EMT-C could also be incorporated into other surveillance systems that receive as inputs text fields containing useful information that is unstructured, uncoded, and/or not easily quantified.

Human Subjects Protections

This study was approved by the Public Health-Nursing Institutional Review Board in the Office of Human Research Ethics at the University of North Carolina at Chapel Hill.