Assessment of ambulance dispatch data for surveillance of influenza-like illness in Melbourne, Australia.

OBJECTIVES: Ambulance dispatch data are collated electronically in many jurisdictions and have a wide reach into the community. They may therefore be useful for syndromic surveillance and early recognition of emerging infectious diseases. This study assessed whether ambulance dispatch data are suitable for influenza surveillance. STUDY
DESIGN: Comparison of a time series of ambulance dispatch data from Melbourne, Australia for the years 1997-2005 with locum service and general practice (GP) sentinel surveillance data for influenza-like illness (ILI).
METHODS: All data were aggregated into 1-week periods, corresponding to the data collection period used in the GP sentinel surveillance system, which was used as the reference system. Rates of ambulance dispatches classified to respiratory or breathing problems per 1000 total dispatches were compared with rates of callouts for flu or influenza per 1000 locum calls, and rates of ILI per 1000 patients from the sentinel GPs. Signals from the ambulance data were generated using the log likelihood ratio CUSUM, a method of continuous monitoring suitable for surveillance.
RESULTS: The ambulance dispatch data displayed seasonal trends that were similar to those observed in locum service surveillance and GP sentinel systems, and identified the years with higher-than-expected seasonal ILI activity (1998 and 2003) and the epidemic year (1997). However, there was a high baseline rate of ambulance calls classified to respiratory or breathing problems (90-100 per 1000 calls) in months where there was minimal influenza activity.
CONCLUSION: Ambulance dispatch data have potential for syndromic surveillance, but because of the high background noise are not definitive and would need to be calibrated to suit particular local circumstances.

Introduction

Influenza viruses circulate in temperate climates from early autumn to late spring, and in tropical climates throughout the year with unpredictable peaks, usually occurring during the wet season and cooler months. Routine surveillance for influenza and influenza-like illness (ILI) can provide estimates of the relative severity of influenza seasons, and can also provide clinical specimens from which influenza viruses can be detected. This allows comparison of circulating influenza strains with vaccine strains in the current influenza season, and may provide potential vaccine strains for future seasons.

Routine ILI surveillance is conducted in most industrialized countries and consists of several elements. These include sentinel surveillance in general practices (GPs), monitoring ILI presentations to hospital emergency departments, monitoring hospital admissions and deaths due to influenza and pneumonia, and school and workplace absenteeism. In Victoria, Australia, a medical locum service, the Melbourne Medical Locum Service (MMLS), is also used to monitor ILI. Most countries also monitor laboratory confirmed influenza. It has been shown previously that all clinical data sources, specifically GP sentinel surveillance, locum service surveillance, and hospital admissions and emergency department presentations for laboratory confirmed or clinically diagnosed influenza, give a similar picture of the influenza season.

Most surveillance systems use a standardized case definition for ILI and describe circulating influenza activity by threshold level. ILI GP sentinel surveillance, with laboratory support, has been conducted in Melbourne, Australia since 1998 using an ILI case definition of fever (or feverishness), cough and fatigue (or malaise). Numeric thresholds, based on a comparison of sentinel ILI surveillance with hospital admissions for laboratory confirmed or clinically diagnosed influenza, have been developed that range from normal seasonal activity through to higher-than-expected seasonal activity and epidemic activity. The thresholds were developed retrospectively and have been applied prospectively.

Recent reports have suggested that ambulance dispatch data may be able to provide an alternative ILI surveillance system.11, 12 Ambulance dispatch data are collated electronically in many jurisdictions and have a wide reach into the community. The ambulance service is frequently the first point of call for the seriously ill person. In this study, it was hypothesized that ambulance dispatch data could provide an alternative ILI surveillance system in Melbourne, Australia. This was assessed by determining whether ambulance dispatch data displayed seasonal trends similar to those observed in existing ILI surveillance systems, and whether increased signalling from ambulance dispatch data corresponded with previously defined seasonal thresholds established for GP sentinel surveillance.

Methods

Data sources

Ambulance dispatch data were compared with locum service and GP sentinel surveillance data. The analyses were confined to data from Melbourne because the locum service is based there. Melbourne is the capital city of the Australian state of Victoria and has a population of approximately 3.8 million people. Data from ambulance dispatch, the locum service, and GP sentinel sites are all community based. Ambulance and locum service data are available for the whole year, while GP sentinel data are only collected from May to September; the typical influenza season in the southern hemisphere. Ambulance and GP sentinel data were available for the 9-year period 1997–2005, whereas the locum service data were only available for the 4-year period 2002–2005.

Ambulance dispatch data

All ambulance dispatch calls are classified by the dispatcher into one of 33 categories based on the Medical Priority Dispatch System. Classifications are broad and include abdominal pain, cardiac arrest, assault, headache and overdose. The dispatcher enters each call and its classification (based on specified query algorithms) immediately into the computerized dispatch system, which then recommends an appropriate resource based on acuity. Calls classified to respiratory or breathing problems (Card 6 in the Medical Patient Dispatch System) were evaluated, and these are referred to as ‘respiratory dispatch’ calls in this article.

GP sentinel surveillance data

GP sentinel surveillance, operational in Victoria during the influenza season (May–September), reports on the number of patients fulfilling the Australian nationally agreed ILI case definition of cough, fever and fatigue, in addition to the total number of patients seen each week. Respiratory specimens taken from a proportion of cases permit diagnosis of laboratory confirmed influenza or other respiratory viruses.2, 5 Weekly data are updated fortnightly on the Internet.

Locum service data

The MMLS is a deputizing, out-of-hours service for GPs that attends to patients in their homes within a 35-km radius of the Melbourne metropolitan area. Demographic and clinical information from patients seen by doctors from the MMLS are routinely entered into a database within 24 h of a consultation. A final diagnosis with free text including either the term ‘flu’ or ‘influenza’ is extracted from this database to conduct ILI surveillance.

Statistical analysis

All data were aggregated into 1-week periods, corresponding to the data collection period used by the paper-based, GP sentinel surveillance system, which was used as the reference system. This is the only system for which thresholds have been developed. Ambulance data were recorded as respiratory dispatches per 1000 total ambulance dispatches, GP sentinel rates were recorded as ILI cases per 1000 patients, and locum service data were recorded as ILI callouts per 1000 calls.

CUSUM (cumulative summation) was developed for industrial quality control in the 1950s, has much in common with exploratory data analysis, and is being used more frequently for surveillance. To generate signals for the ambulance data, the log likelihood ratio (LLR) form of the CUSUM was used, which is optimal, in a statistical sense, for detecting abrupt shifts from the baseline.

For a binomial process (i.e. monitoring weekly proportions), the LLR CUSUM for the ith week is:where:

C 0  =  h/2; h is a threshold where the process is set to signal (see later); and

W =  X  × log(OR) −  n  × log(1 −  Y  + {OR ×  Y})

X is the number of respiratory dispatches in the ith week

n is the total number of dispatches in the ith week

Y is the seasonally adjusted, baseline ILI dispatch rate for the ith week (i.e. the in-control ILI dispatch rate). This was estimated using data from 2 years of normal seasonal activity identified in the GP sentinel surveillance system (2004 and 2005). The Holt-Winters seasonal additive procedure was used to smooth weekly data, which was then averaged across the 2 years and within each month to obtain estimates of Y for each month, January–December (Table 1 ).

Table 1

Seasonally adjusted baseline respiratory dispatches per 1000 total ambulance dispatches

Month	Seasonally adjusted baseline influenza-like illness dispatch calls per 1000 total ambulance dispatches
January	91
February	95
March	95
April	95
May	109
June	114
July	114
August	115
September	111
October	104
November	103
December	100

OR is the odds ratio for a pre-specified increase in the ILI dispatch rate over baseline, which was considered to indicate higher-than-expected seasonal activity. This is like the minimum clinically important difference which must be pre-specified when designing a clinical trial. For this study, an OR of 1.22 was pre-specified, which corresponds to an increase of 20% over the seasonally adjusted, baseline ILI dispatch rate (Y).

The performance of the LLR CUSUM is based on the concept of run length because the familiar Type 1 and Type 2 error rates associated with hypothesis tests are difficult to interpret. With sequential monitoring, the Type 1 error rate is not constant but increases with the length of the monitoring period. The probability of eventually signalling an alarm is 1.0 for all sequential tests, so that the Type 1 error rate will eventually be 100% and the Type 2 error rate will eventually be 0%.

In the context of this study, the run length is the number of weeks until a signal; ideally, this should be large when there is normal seasonal activity (i.e. low false-alarm rate) and small when there is higher-than-expected seasonal activity (i.e. high power). Average run lengths for different values of h can be estimated using simulations, Markov chains or, in certain circumstances, by approximating formulae. Simulations were used for this study. Briefly, data sets of 10,000 weeks (under the assumptions of normal seasonal activity or 20% higher) were specified and iterated 10,000 times to obtain estimates of the average (median) run length to true and false alarms.

At present, there is no agreed convention to guide the selection of h (and the resultant average run lengths), in contrast to the somewhat arbitrary convention of P  = 0.05 which is commonly used in hypothesis testing. For this study, h was set at 10 because this gave an average run length to a true alarm of 4 weeks, and an average run length to a false alarm of approximately 1000 weeks. It was possible to select the relatively large value of h  = 10 and still obtain a short average run length to a true alarm because the number of ambulance dispatches per week was large (mean 4253; see Results section).

The starting value for the CUSUM was set at h/2 to reflect the uncertainty, before looking at the data, regarding whether or not it was a period of higher-than-expected activity. Similarly, after a signal, the CUSUM was reset to h/2 because there was no way of knowing whether the higher-than-expected proportions had returned to baseline values. This is different from the use of CUSUM in industry, where after a signal, the machine (say) is recalibrated and known to be ‘in control’ and therefore monitoring restarts at h  = 0.

The use of CUSUM in this paper corresponds with Phase 1 of calibration used for quality control in the industrial context, which is largely descriptive and involves setting the parameters of the systems such as estimating seasonally adjusted, baseline (in-control) values (Y) and confirming a suitable value for h.

Results

There were 165,000 ambulance dispatch calls in 1997, increasing to 260,000 in 2005. The mean number of dispatches per week was 3240 in 1997 and 4990 in 2005; the mean, per week, for all years combined was 4253. This varied by month, tending to be lowest in January and highest in August, although the pattern also varied from year to year. For all years combined, the mean number of weekly dispatches in January was 3936, and the corresponding figure for August was 4415.

Across all years and all months, the mean weekly number of respiratory dispatches per 1000 total dispatches was 103 (standard deviation 12.4, range 73–146). The smoothed, averaged baseline values varied from 91 in January to 115 in August (Table 1). For the period 2002–2005, ‘influenza’ as recorded in the locum service data and ‘respiratory or breathing problems’ as recorded in the ambulance dispatch data displayed similar seasonal patterns (Fig. 1 ) (2002–2005 was the only period for which locum service data were available).

Figure 1

Comparison of ambulance dispatch data and locum service data. ILI, influenza-like illness.

The baseline during non-influenza months was approximately 90 per 1000 ambulance dispatches; in contrast, for the locum service data, it was close to zero. Related to this, the plot for the ambulance dispatch data was less peaked for 2003 (a year known to have higher-than-expected seasonal activity) than the plot for the locum service data (Fig. 1). In short, the seasonal pattern was more blurred for the ambulance dispatch data than the locum service data. Use of subclassifications of respiratory or breathing problems, available in the Medical Priority Dispatch System, did not reduce the blurring (data not shown).

The CUSUM method was applied to the ambulance dispatch data for the period 1997–2005 to highlight the underlying patterns in the data. The CUSUM signals were compared with previously published analyses based on the GP sentinel surveillance data which reported an epidemic year in 1997 and higher-than-expected seasonal activity in 1998 and 2003 (Fig. 2 ).

Figure 2

Comparison of ambulance dispatch data and general practice sentinel surveillance data. The plus symbols indicate a signal from the log likelihood ratio CUSUM. ILI, influenza-like illness.

The GP sentinel surveillance data reached the threshold for higher-than-expected seasonal activity (15 cases per 1000 patients) on 24 June 1997, 25 June 1998 and 4 August 2003. The CUSUM alert either occurred earlier than this (1997 and 1998) or at the same time (2003) (Fig. 2 and Table 2 ).

Table 2

Comparison of dates of signals from ambulance dispatch data and date of reaching threshold for higher-than-expected seasonal activity in general practice (GP) sentinel surveillance system, 1997–2005a

1997		1998		2003
Ambulance	GP	Ambulance	GP	Ambulance	GP
17 February	24 June	27 April	25 June	4 August	4 August
7 April				11 August
14 April				18 August
16 June				25 August
30 June
7 July
14 July
21 July
4 August

2000, 2001, 2002, 2004 and 2005 were years of normal seasonal activity (see Figure 2).

Discussion

Ambulance dispatch data display seasonal trends similar to those observed in existing ILI surveillance systems, and signalling corresponds to previously defined seasonal thresholds established for GP sentinel surveillance.

The disadvantage of ambulance dispatch data is the high baseline rate: 90–100 respiratory dispatches per 1000 total dispatches in months when there is almost no influenza activity in the community. This contrasts with a background rate of less than two per 1000 for the locum service and GP data during the same period. For the ambulance data, seasonal trends are superimposed on this relatively high baseline, which means that any pattern is blurred in comparison with the more specific surveillance systems.

There are no studies on the sensitivity and specificity of the category of respiratory or breathing problems (Card 6 in the ambulance dispatch system) in identifying patients with ILI. The high baseline suggests that there are many false positives (i.e. poor specificity), although the seasonal trends (which correspond with those from other surveillance systems) suggest that there may only be a few false negatives (i.e. reasonable sensitivity). Arguably, the addition of specific indicator questions (e.g. concerning cough, fever and fatigue) to existing query algorithms would improve sensitivity and specificity for ILI, but this may not be a realistic commercial option for an ambulance dispatch system.

It was only possible to identify one other published study that reported respiratory dispatch calls as a proportion of all ambulance dispatches. That study, from New York, reported an even higher baseline rate in the non-influenza seasons than this study from Melbourne (approximately 200 respiratory dispatches per 1000 total dispatches). A Danish study monitored the total number of ambulance dispatches. When the total number of ambulance dispatches per week was analysed, using a Gaussian CUSUM (rather than a binomial CUSUM), the results were similar to those presented here. That is, for the time period analysed, an increase in ILI is the main reason for an increase in the total number of ambulance dispatches. However, it is possible that, in a particular year, there may be other reasons for an increase in total ambulance dispatches, such as a heat wave. In this case, respiratory dispatches as a proportion of total dispatches should perform better than the total number of dispatches.

The New York study used cyclical regression models with sine and cosine terms to account for seasonal variability, while the study from Denmark used time series methods. A recent comparison of statistical methods for influenza surveillance data found that these two methods, and the CUSUM method used in this study, had similar performance in generating sensitive, specific and timely alerts. Given the similarity of the methods, the present authors preferred the CUSUM method because it is simpler and therefore more suited to routine use. Once seasonally adjusted ILI rates (Y) and a threshold value (h) are agreed upon, the CUSUM method can be implemented using spreadsheet software.

Due to their wide reach into the community, ambulance dispatch data have been suggested as a means of real-time, syndromic surveillance for early detection of a covert bioterrorism attack or emerging respiratory infections such as severe acute respiratory syndrome or pandemic influenza. However, given the high baseline rates for ambulance data from Melbourne and elsewhere, users should be cautious about placing too much reliance on such data for syndromic surveillance.

The signalling characteristics of a particular CUSUM plot are determined by the value assigned to h. For this study, a relatively large value was chosen (h  = 10) to ensure that higher-than-expected seasonal activity (true alarm) was being identified rather than statistical noise (false alarm).

Using a value of 10, it was shown that the ambulance dispatch data signalled in the years known to have higher-than-expected seasonal activity (1998 and 2003) and the epidemic year (1997).10, 22 There were signals early in 1997 (17 February, 7 April and 14 April), well before the GP sentinel surveillance system reached the threshold for higher-than-expected seasonal activity on 25 June. It is difficult to know, in retrospect, whether these early signals were false positives or an early warning of an epidemic year, because sentinel surveillance and laboratory confirmation is only conducted in Melbourne from May to September (the influenza season in the southern hemisphere).

Smaller values of h would have generated more signals, possibly in years with normal seasonal activity. Larger values of h would have generated fewer signals, and may not have generated signals in the years with higher-than-expected seasonal activity. In other words, the sensitivity, specificity and timeliness of the system are dependent on h.

Investigators wishing to adapt this method for use in other locations will need to calibrate h against their data to arrive at a suitable value, depending on the local implications of false-negative and false-positive signals. The seasonally adjusted expected values (i.e. the in-control values or Y) may also need to be adapted for local circumstances.

A signal from a surveillance system requires further investigation to determine whether there is true higher-than-expected seasonal activity. For ILI surveillance, this is not a costly exercise and would typically involve contacting hospital emergency departments and sentinel general practices, and comparison with other surveillance data sources to determine whether the increase in respiratory dispatches in the ambulance data was reflected in the clinical setting. Given the low cost of investigating false alarms, values of h smaller than used in the present study may be more suitable for routine surveillance. That is, for routine surveillance, a more sensitive (fewer false negatives) but less specific (more false positives) system may be more appropriate.

Conclusions

Ambulance dispatch data display seasonal trends for ILI that are similar to those observed in locum service and GP sentinel surveillance systems, and therefore have potential for syndromic surveillance. However, ambulance dispatch data have high background noise compared with other surveillance data, and would need to be calibrated to suit particular local circumstances.