Perceived usefulness of syndromic surveillance in Ontario during the H1N1 pandemic.

BACKGROUND: Despite the growing popularity of syndromic surveillance, little is known about if or how these systems are accepted, utilized and valued by end users. This study seeks to describe the use of syndromic surveillance systems in Ontario and users' perceptions of the value of these systems within the context of other surveillance systems.
METHODS: Ontario's 36 public health units, the provincial ministry of health and federal public health agency completed a web survey to identify traditional and syndromic surveillance systems used routinely and during the pandemic and to describe system attributes and utility in monitoring pandemic activity and informing decision-making.
RESULTS: Syndromic surveillance systems are used by 20/38 (53%) organizations. For routine surveillance, laboratory, integrated Public Health Information System and school absenteeism data are the most frequently used sources. Laboratory data received the highest ratings for reliability, timeliness and accuracy ('very acceptable' by 92, 51 and 89%). Hospital/clinic screening data were rated as the most reliable and timely syndromic data source (50 and 43%) and ED visit data the most accurate (48%). During the pandemic, laboratory data were considered the most useful for monitoring the epidemiology and informing decision-making while ED screening and visit data were considered the most useful syndromic sources.
CONCLUSIONS: End user perceptions are valuable for identifying opportunities for improvement and guiding further investments in public health surveillance.

Introduction

With increasing availability of electronic data, syndromic surveillance, which uses pre-diagnostic data for surveillance purposes, has been growing in popularity.[1] In a 2008 US survey of state epidemiologists, 80% (33/41) of respondents reported having at least one syndromic surveillance system in operation, while another survey of selected health departments found 83% (43/52) of respondents conducted syndromic surveillance.[2,3] While similar estimates are not available for Canada, established syndromic surveillance systems in both Edmonton, Alberta and Kingston, Ontario have reported recent expansion throughout each province.[4,5] Outside of North America, syndromic surveillance has been utilized in the UK to monitor progression of the 2009 H1N1 pandemic and to identify areas with high activity to inform allocation of limited public health resources.[6,7] Despite growing popularity, there is a paucity of research on whether these systems are accepted by end users and utilized for their intended purpose of early outbreak detection. A 2010 US survey found that only 38% of states utilized emergency department (ED) chief complaint data and 28% utilized poison control data for enteric outbreak detection.[8]

Evaluations of syndromic surveillance have focused on determining the sensitivity and specificity of particular systems in detecting actual and simulated outbreaks.[9-13] Recent evaluations of seasonal influenza epidemics and the 2009 H1N1 pandemic show that data collected from various syndromic sources correlate well with traditional laboratory data and sentinel surveillance data for physician influenza-like-illness (ILI) consultations and can provide advanced warning of influenza activity.[9,11,14-16] Similarly, increases in ED visits and over-the-counter sales have been shown to correspond with annual seasonal outbreaks of norovirus and rotavirus.[17,18] The utility of syndromic surveillance data for early detection of smaller, more localized outbreaks is less clear. Although several studies have demonstrated the sensitivity of various systems to detect small outbreaks of Cryptosporidium and Legionnaires' disease,[19,20] other systems have failed to generate signals during known gastrointestinal outbreaks.[17,21,22] As a result, the value of syndromic surveillance for local outbreak detection is still debated.[23,24]

Understanding how syndromic surveillance systems are accepted, utilized and valued for public health surveillance, including how these systems are integrated with traditional surveillance systems and linked to public health action is necessary to inform investments to build surveillance capacity. The purpose of this study was to describe the use of syndromic surveillance systems in Ontario and users' perceptions of the value of these systems within the context of other public health surveillance systems.

Methods

Setting

During the 2009 H1N1 pandemic in Ontario, a comprehensive provincial surveillance plan was developed which included the use of traditional data from public health laboratories and Ontario's reportable infectious disease database, called the integrated Public Health Information System (iPHIS). In addition, several pre-existing and temporary syndromic surveillance systems were made operational (Table 1). Syndromic data included community sentinel practitioner reports of ILI, total all-cause ED visits or proportions of persons presenting with ILI or febrile respiratory illness, school absenteeism reports and data from calls to public health units (PHU) or Telehealth, a free nurse consultation telephone service provided by Ontario's Ministry of Health and Long-Term Care (MOHLTC). In Ontario, 36 local PHUs administer community public health programmes and services with provincial and federal guidance. Thus, decisions regarding the use of particular sources were made by individual organizations at the local, provincial (MOHLTC) and federal (Public Health Agency of Canada, PHAC) levels as appropriate.

Table 1

Description of sources of surveillance information included in the survey

Source of surveillance information	Considered syndromic	Description
Public health laboratory tests	No	Confirmatory reports of cases with infectious diseases from public health laboratory testing
iPHIS	No	Ontario's reportable disease database containing local and provincial information collected and entered by PHU personnel from active follow-up of individuals with a laboratory-confirmed reportable infectious disease
School absenteeism	Yes	All cause or illness-related school absenteeism rates or counts
Community sentinel ILI consults	Yes	Counts or rates of patients seeking care from community sentinel practitioners (physicians and nurse practitioners) with symptoms of ILI as a proportion of all patients seeking medical care
ED visits	Yes	Data based on patient visits to EDs such as total patients seeking care and counts by chief complaints categorized by syndrome
Telehealth	Yes	All cause or symptom-related call data from a free nurse consultation telephone service provided by the MOHLTC
Hospital/clinic screening data (includes ED screening)	Yes	Screening data (usually for ILI or febrile respiratory illness) from patients seeking care in hospital EDs or clinics regardless of chief complaint
Workplace absenteeism	Yes	All cause or illness-related absenteeism rates or counts in a workplace
Outpatient data	Yes	Information from health care providers in an outpatient setting
Pharmacy sales	Yes	Data from over-the-counter or prescription drug sales
Emergency medical service/911 call data	Yes	Data collected by emergency medical services personnel or from calls to 911

Participants

In July 2010, Ontario's 36 PHUs, the Public Health Protection and Prevention Branch and Emergency Management Branch of the MOHLTC and the Centre for Immunization and Respiratory Infectious Diseases at PHAC (total 38 organizations) were emailed an invitation to complete an online survey hosted by FluidSurveys® with a reminder email sent 4 weeks later.[25] Individual recipients of invitations were primarily epidemiologists and medical officers of health. However, participants were requested to reflect on perceptions and actions of the organization as a whole and thus, also to include the involvement of other public health professionals as appropriate. The survey was pilot tested with one PHU and active for the remaining participants from July to September 2010, approximately 8 months after the end of peak H1N1 activity in Ontario. Since the survey did not change substantively after piloting, responses from the pilot site were included in the final analyses.

Survey measures

The survey consisted of 24 multiple choice and short answer questions and aimed to identify the syndromic surveillance systems used routinely during non-pandemic periods and to identify those in operation during the pandemic. Organizations were asked whether they regularly receive notifications of suspected outbreaks or unusual events from selected non-syndromic and syndromic surveillance sources (Table 1), and their perceptions of these data sources in terms of reliability, timeliness and accuracy using an acceptability rating of ‘very acceptable’, ‘acceptable’ or ‘not very acceptable.’ To minimize variation in interpretation, definitions were provided for reliability (notification of an event every, or nearly every time an unusual event occurs), timeliness (notification received very soon after an unusual event) and accuracy (notification most probably indicates a true occurrence of an unusual event). Organizations were also asked to identify the data they used during the pandemic to monitor H1N1 epidemiology and to support decision-making related to their jurisdictional responses to the pandemic, and to rate the data's perceived usefulness.

To differentiate between access to syndromic data and use of a syndromic surveillance system, respondents were also asked whether they routinely and systematically collect pre-diagnostic data in (near) real time that is linked to a response. Organizations that collected and analysed syndromic data temporarily during the pandemic were not considered syndromic surveillance system users. Usefulness of syndromic data in monitoring pandemic activity and informing decision-making was also rated as ‘essential’, ‘useful’, ‘somewhat useful’ or ‘not useful.’

Data on the size of the population served by each PHU were obtained from the Initial Report on Public Health in Ontario.[26]

Data analysis

Responses were analysed using Excel 2007 and SAS 9.1. Missing and inconsistent responses were excluded from the analysis. For example, if an organization indicated that it did not receive notifications of suspected outbreaks or unusual events from ED visit data, but provided a response for the reliability of this source, the reliability rating was excluded. Confidence intervals for proportions were calculated using mid-P exact limits on OpenEpi 2.3.1.[27]

Ethics approval was obtained from the University of Toronto Office of Research Ethics.

Results

Responses were received from all invited organizations, resulting in a 100% response rate. Most contributors to survey responses were epidemiologists (29%), managers (24%) or medical officers of health (19%), while other professionals such as public health nurses, inspectors and directors also provided input.

Routine surveillance

Table 2 shows the percentage of organizations using each of the different data sources for routine surveillance. All organizations receive notification of reportable diseases from the public health laboratory and iPHIS (with the exception of PHAC). The most frequently used syndromic surveillance data sources are school absenteeism (97%) and community sentinel consultations for ILI (92%).

Table 2

Percentage of organizations receiving notification of suspected outbreaks or unusual events by source of surveillance information, Ontario, 2010

Source of surveillance information	Organizations receiving notification, n (%)	Total respondents, N
Public health laboratory tests	37 (100)	37
iPHIS	37 (97)	38
School absenteeism	37 (97)	38
Community sentinel ILI consults	35 (92)	38
ED visits	25 (66)	38
Telehealth	24 (63)	38
Hospital/clinic screening data	14 (38)	37
Workplace absenteeism	9 (24)	37
Outpatient data	8 (22)	37
Pharmacy sales	6 (16)	37
Emergency medical service/911 call data	2 (5)	37

Regarding perceptions of system qualities, traditional data sources (laboratory testing data and iPHIS) were rated most favourably overall with laboratory data receiving a ‘very acceptable’ level of reliability, timeliness and accuracy by 92, 51 and 89% of organizations (Table 3). Among syndromic data sources, hospital/clinic screening data and ED visits were generally considered more reliable, timely and accurate than school absenteeism and sentinel consultations for ILI, which were more frequently used. Overall, EMS/911 data, workplace absenteeism and pharmacy sales were reported as having the relatively lowest ‘very acceptable’ ratings across the three qualities, but were also used by fewer than 10 organizations (data not shown).

Table 3

Perceptions of data source reliability, timeliness and accuracy as ‘very acceptable’ by organizations who receive notification from that source (ordered by most acceptable reliability), Ontario, 2010

Source of surveillance informationa	Perceived as ‘very acceptable’
	Reliability		Timeliness		Accuracy		N
	%	95% CI	%	95% CI	%	95% CI
Public health laboratory tests	92	80, 98	51	35, 67	89	76, 96	37
iPHIS	84	69, 93	43	28, 59	62	46, 77	37
Hospital/clinic screening data	50	25, 75	43	20, 69	43	20, 69	14
ED visits	48	28, 68	35	18, 56	48	28, 68	23b
Community sentinel ILI consults	39	24, 57	18	8, 34	25	12, 42	33c
School absenteeism	22	11, 38	42	27, 58	22	11, 38	36d
Telehealth	29	12, 50	23	9, 43	27	12, 48	22e

aResponses for workplace absenteeism, outpatient data, pharmacy sales and emergency medical service/911 data are not displayed due to the low number of organizations using these data sources.

bTwo responses missing.

cTwo responses missing for reliability and timeliness, three responses missing for accuracy.

dOne response missing.

eTwo responses missing for timeliness and accuracy, three responses missing for reliability.

Overall, 53% (20/38) of organizations reported using at least one syndromic surveillance system at the time of the survey with 60% (12/20) using a customized system and 45% (9/20) using KFL&A Public Health's ED Syndromic Surveillance (EDSS) system[5] (not mutually exclusive). Among the 36 PHUs, the proportion of syndromic surveillance system users increased with increasing population size served by the PHU: 9/12 (75%) that serve a population >400 000 had syndromic surveillance systems compared with 8/13 (62%) that serve a population of 125 000–400 000 and only 1/11 (9%) serving a population of <125 000.

Surveillance during the 2009 H1N1 pandemic

As with routine surveillance, laboratory data were used by all organizations to monitor the epidemiology of H1N1 and most used it to inform decisions on responses to the pandemic (97%; Fig. 1). Data from iPHIS were used by 95% of organizations to monitor pandemic activity, but only by 79% for decision-making. School absenteeism was the most used syndromic data source for both monitoring (97%) and decision-making (92%).

Fig. 1

Data sources used by organizations to monitor the epidemiology of 2009 pandemic H1N1 and inform decision-making, Ontario. *Other includes EMS/paramedic activities, hospital admission, intensive care unit and ventilator use data, immunization visits, influenza assessment centre visits and provincial surveillance bulletins and teleconferences.

Examining usefulness of surveillance data, laboratory data were considered the most essential for monitoring activity and for decision-making (71% and 76%; Table 4). ED screening and ED visit data were the most useful syndromic data for both monitoring activity and making decisions. Both were rated as essential for monitoring activity by 52% of organizations, slightly lower than iPHIS data at 56%. For decision-making, ED visit and screening data were considered more essential than iPHIS. Among data sources used by >50% of respondents, Telehealth was considered the least essential for monitoring pandemic activity and sentinel ILI consultations least essential for informing decision-making.

Table 4

Level of usefulness of data sources in monitoring 2009 pandemic H1N1 epidemiology and informing decision-making and actions for organizations who reported use of these sources during the pandemic (ordered by most ‘essential’ in monitoring the epidemiology), Ontario

Source of surveillance informationa	Perceived as ‘essential’
	Monitoring H1N1 epidemiology		nb	Informing decision-making and actions		n
	%	95% CI		%	95% CI
Public health laboratory tests for influenza A	71	55, 84	38	76	60, 87	37
iPHIS	56	39, 71	36	63	45, 79	30
ED screening	52	34, 69	29	70	51, 85	27
ED visits	52	33, 71	25	68	48, 84	25
School absenteeism	41	26, 57	37	41	26, 58	34
Organization telephone information line	26	11, 47	23	36	19, 58	22
Community sentinel ILI consults	23	10, 40	31	29	12, 50	21
Calls from physicians or hospitals	21	8, 40	24	43	23, 64	21
Telehealth	0	0, 14	20	0	0, 39	6

aResponses for workplace absenteeism, outpatient data, pharmacy sales and emergency medical service/911 data are not displayed due to the low number of organizations using these data sources.

b n represents the number of organizations who reported using the data source for monitoring H1N1 epidemiology and/or informing decision-making.

Discussion

Main findings

This study describes the syndromic data sources currently in use in Ontario, and perceptions of their value within the context of other public health surveillance systems. Traditional surveillance data, specifically public health laboratory tests and Ontario's reportable disease database (iPHIS), were shown to be used by almost all respondents for surveillance both during non-epidemic periods and during the pandemic. Compared with syndromic data, both were considered more reliable, timely and accurate. Laboratory and iPHIS data were also viewed as essential for monitoring the epidemiology of H1N1 and for informing decision-making on responses, although ED visit and screening data were rated more essential than iPHIS for decision-making.

Given the diagnostic value of laboratory and iPHIS data (which is based on laboratory-confirmed infections), it is not surprising that both were perceived as more reliable and accurate than syndromic data, which is pre-diagnostic. It is surprising, however, that laboratory and iPHIS data were perceived as timelier than syndromic data. Although in theory, the advantage of syndromic data comes from its improved timeliness, in practice, delayed data acquisition or analysis may influence perceptions of timeliness. Additionally, if poor specificity and the preference to wait for diagnostic results delay responding to early alerts, the potential benefits of syndromic surveillance are lost, thereby influencing perceptions of timeliness, as well as usefulness for decision-making. With variability in methods and frequency of syndromic data acquisition (e.g. by email, fax or automatic and daily versus weekly), perceptions of timeliness may also not be a true reflection of automated systems which provide real-time data such as KFL&A Public Health's EDSS system.

Unfamiliarity with the syndromic data collected could have also affected perceptions of their usefulness and system qualities. All PHU personnel collect and enter information related to laboratory-confirmed cases into iPHIS and thus, are familiar with the data. In contrast, only 50% of PHUs routinely use syndromic surveillance systems. For PHUs who created temporary approaches to collecting local syndromic data during the pandemic, processes for obtaining the data in a reliable and timely manner may not have been optimized. Additionally, algorithms or thresholds for initiating a response may also not have been defined. Without baseline information or a standard approach to analysis, interpretation of this information would be difficult compared with laboratory and iPHIS data thereby affecting their perceived usefulness and qualities. Unfortunately, the small number of participants in this study limits our ability to confirm this hypothesis.

Among syndromic data sources, school absenteeism was the most frequently used source both routinely and during the pandemic for monitoring and decision-making, yet was viewed as less essential during the pandemic than ED screening and visit data on both measures. ED data may be viewed as more useful because it closely reflects health care utilization and burden of disease. In the context of the pandemic, monitoring ED utilization would be useful for informing the operation of influenza assessment centres and messaging on public health measures. However, these data may not be as useful in other situations such as foodborne outbreaks. Additionally, although hospital/clinic screening and school absenteeism data were considered the most timely syndromic data sources, the lower specificity of school absenteeism data likely contributes to its lower accuracy rating. School absenteeism data are often collected as all cause and even with efforts by some school boards to collect reasons for absence, the accuracy of such data remains uncertain. Reliability is also a concern as reporting by individual schools and school boards has the potential to be inconsistent, further questioning its value.

Data sources not frequently used in Ontario include out-patient data, workplace absenteeism, pharmacy sales and EMS/911 data. The low routine use of pharmacy sales is notable as over-the-counter drug sales can be particularly useful in situations that prompt individuals to self-medicate, as seen from analyses of two significant enteric outbreaks due to water contamination.[28] The challenges associated with store discounts, variations in consumer behaviour and classification of drugs into syndromes may limit their usefulness.[18] Further investigations are required to determine whether the low use of these syndromic data is due to data accessibility, or because the data are considered less useful.

Most available data sources were used in similar proportions for H1N1 monitoring and decision-making except for iPHIS, sentinel ILI consults and Telehealth. The high frequency of use of sentinel ILI consult data routinely and for monitoring H1N1 may be due to accessibility of local data through FluWatch, Canada's national surveillance system for monitoring influenza activity. However, dependence on sentinel practitioners reporting could limit its use for decision-making during the pandemic as delayed or missed reporting affects its usefulness, especially for PHUs with fewer practitioners registered in the system. The weekly reporting formats of FluWatch and Telehealth also contribute to their limited use when decisions are being made daily. Similarly, restricted laboratory testing and limited human resources to enable recording of case information into iPHIS by PHU staff during the pandemic may have contributed to the differences in its use for monitoring compared with decision-making.

What is already known on this topic?

Although similar studies describing the types of syndromic data used for surveillance in Canada have not been identified, a US survey of health departments in 46 states found ED visits were the most frequently used syndromic data source (84%) followed by outpatient clinic visits (49%) and over-the-counter medication sales (44%).[2] In contrast to this study, school absenteeism was used by only 35% of users. In general, US users also viewed syndromic surveillance as most useful for monitoring influenza and trends in cities or larger areas. This study found a greater number of syndromic system users among PHUs serving larger population sizes. However, greater resources and infrastructure likely explain this finding.

What does this study add?

With increasing interest in syndromic surveillance, yet the uncertainties in its added value, this study is an initial step in understanding its utility for surveillance during non-pandemic and pandemic periods,[29] and provides important comparisons of their perceived utility relative to other surveillance systems. Understanding and describing the current context of syndromic surveillance is an important first step in evaluating existing systems.[30] Data collected in this survey have provided valuable insights to help prioritize further evaluation of essential system characteristics using quantitative methods.[31]

Timeliness has typically been viewed as syndromic surveillance's greatest asset. However, the lower timeliness ratings observed in this study for syndromic surveillance systems call into question its value if it was also not viewed as more accurate or reliable than traditional systems. Greater understanding of how to analyse, interpret and respond to syndromic data may be required. Quantitative comparisons between syndromic and traditional surveillance systems of time intervals between behaviour onsets, data processing/capture and alert generation will further elucidate whether timelier syndromic data or greater familiarity with it would increase its usefulness in Ontario. Apart from timeliness, syndromic surveillance may accomplish tasks that traditional surveillance may not, such as detecting unusual or emerging diseases associated with common symptoms that may not alarm clinicians, and uncommon or new pathogens that remain undetected by laboratories.[12]

In addition, public health surveillance should be linked to action.[32] Users of syndromic surveillance value the ‘situational awareness’ or information about disease activity in the community that syndromic data can provide, and the reassurance the data provides to decision-makers.[33] Certainly, many respondents in this survey indicated that several syndromic surveillance data sources were essential for monitoring pandemic activity and informing decisions. However, implementing temporary syndromic surveillance systems during outbreaks such as the pandemic not only has potential impact on the timeliness of obtaining data, but also affects the interpretation of the data and subsequent public health response. These initial findings require further exploration to better understand the added benefits of syndromic surveillance for responding to public health events.

Limitations

Despite providing definitions of the terms ‘reliable’, ‘accurate’, ‘timely’ and ‘useful’, survey responses were self-reported and thus, what may be considered acceptable or useful is subject to different interpretations by different organizations or individuals. The high number of customized syndromic surveillance systems in use and lack of specific details about their operation also creates ambiguities in differentiating syndromic surveillance systems from syndromic data sources and the degree of active participation in surveillance. In assessing user perceptions of various data sources, despite knowing that responses would not be identifiable in reported results, it is also possible that providing more acceptable responses, or social desirability bias influenced responses such that organization-specific sources were viewed more favourably (or not). Finally, although emphasis was placed on providing responses to reflect the perceptions and actions of the organization as a whole, it is possible that not all individuals who use the data or conduct surveillance provided input. Subsequent key informant interviews will provide further details expanding on these responses, including specific details about system operations and the use of syndromic data in decision-making during H1N1.

Funding

This work was supported by the Institute of Population and Public Health and the Knowledge Synthesis and Exchange Branch of the Canadian Institutes for Health Research [H1N-104055].