Evaluation of sentinel surveillance system for monitoring hand, foot and mouth disease in Hong Kong.

OBJECTIVES: A sentinel surveillance system (SSS) was set up in Hong Kong to monitor hand, foot and mouth disease (HFMD) trends. This evaluation assessed the performance of the SSS from 2001 to 2009, and aimed to identify areas for improvement. STUDY
DESIGN: A retrospective review using structured guidelines for evaluating public health surveillance systems published by the US Centers for Disease Control and Prevention.
METHODS: The effectiveness of the SSS was evaluated using routine service statistics, laboratory surveillance data, and results of an acceptability survey conducted among the sentinel doctors. This information was used to assess various attributes of the SSS including simplicity, flexibility, data quality, usefulness, sensitivity, specificity, positive predictive value (PPV), representativeness, timeliness and acceptability.
RESULTS: The SSS was simple and flexible with high-quality data. It correlated well with the laboratory surveillance data (P < 0.001) and facilitated early detection of community epidemics. It helped to identify seasonal trends and high-risk groups. Specificity was high (83.4-88.5%), while sensitivity and PPV were borderline satisfactory (38.4-56.8%). The sentinel clinics were representative of the population distribution. The SSS was acceptable to the sentinel doctors, but 17.9-28.2% of them had delays in reporting.
CONCLUSIONS: The SSS is effective for monitoring HFMD trends in Hong Kong, and is useful for initiating preventive measures.

Introduction

Hand, foot and mouth disease (HFMD) is a common condition, especially in children <5 years of age. A seroprevalence study in Singapore showed that half of children aged ≥5 years had antibody to enterovirus (EV) 71, which is a commonly identified virus causing HFMD. Although most patients present with mild illness including fever, skin vesicles or rash, some children may suffer from severe complications such as meningitis, encephalitis, acute flaccid paralysis, myocariditis, pulmonary oedema or even death.2, 3, 4, 5, 6, 7 Since the 1970s, HFMD epidemics have been reported in Bulgaria, Hungary, the UK, the USA, Australia and India, with significant numbers of cases of aseptic meningitis, brain stem encephalitis and death.8, 9, 10, 11 From 1997 to 2000, areas in South East Asia including Malaysia, Taiwan, Singapore and Japan also experienced HFMD epidemics.1, 12, 13, 14, 15 The largest reported outbreak occurred in Taiwan in 1998, accounting for >100,000 cases of HFMD, 400 severe illnesses and 78 deaths. The potential public health impact of HFMD in Hong Kong has captured much attention. In 2008, another epidemic affected South East Asia, including Mainland China, Taiwan, Singapore and Hong Kong.16, 17, 18, 19 Apart from mortality and morbidity, outbreaks occurring in school settings caused significant social disturbance to children, their families and teachers. In large school outbreaks of HFMD or outbreaks associated with EV71 in Hong Kong and Singapore, the Health Authority may have to suspend classes or close schools in order to prevent disease transmission.18, 20

Unfortunately, there is currently no effective vaccination or chemoprophylaxis for the prevention of HFMD. When large community outbreaks occur, public health actions including increasing public awareness, vigilance of personal and environmental hygiene, and social distancing may be necessary to stop disease transmission. A sensitive surveillance system is therefore essential for early detection of community outbreaks in order to guide public health actions. In Hong Kong, HFMD was added to the sentinel surveillance system (SSS) in 1998. The main objectives were to monitor HFMD trends in Hong Kong and to enable early detection of community epidemics, which in turn helps to initiate timely preventive measures. The SSS includes over 100 sentinel clinics, including approximately 40 general practitioners (GPs) and 60 general outpatient clinics (GOPCs). GPs are doctors who practice in the private sector while GOPCs are public outpatient clinics. They are all distributed throughout the territory.

Each week, the sentinel doctors record the number of patients clinically diagnosed with HFMD, with age group breakdown, and the total number of patients seen. Patients with clinical features of HFMD are characterized by having fever, small discrete papulovesicular lesions in the oral cavity, and sparse, grayish vesicles surrounded by an erythematous base on the hands or feet. These data, in aggregated format, are sent to the Department of Health (DH) by facsimile or e-mail using a standardized surveillance record form. A weekly summary report is disseminated to all participating doctors. The authors evaluated the performance of the SSS in order to identify areas for improvement.

Methods

The SSS was assessed using a structured framework, ‘Updated Guidelines for Evaluating Public Health Surveillance Systems’ published by the US Centers for Disease Control and Prevention. The main quality attributes included usefulness, sensitivity, specificity, positive predictive value (PPV), data quality, timeliness, representativeness, acceptability, simplicity and flexibility. The SSS data from 2001 to 2009 were used to assess its performance. The data were analysed using Excel 2007 (Microsoft, Redmond, USA) and Statistical Package for the Social Sciences Version 14.0 (SPSS Inc., Chicago, IL, USA).

Usefulness

To assess the usefulness of the SSS, the HFMD consultation rate from the SSS was examined to determine whether it correlated with the laboratory surveillance data of the Public Health Laboratory Centre (PHLC) of the DH. The HFMD consultation rate was calculated by dividing the number of patients diagnosed with HFMD by the total number of patients seen. The positive viral culture rate was calculated by dividing the weekly number of specimens positive for enterovirus by the weekly total number of specimens performed. The enteroviruses included Coxsackie A and B, EV71, EV68, echovirus and some untyped enterovirus. Coxsackie A24 was excluded because it was associated with acute conjunctivitis instead of HFMD. The Pearson correlation coefficient of the weekly consultation rate with the positive viral culture rate was calculated. The HFMD consultation rate from the SSS was plotted on a monthly basis to document seasonal trends. HFMD cases were broken down by age group to determine the most susceptible group.

Sensitivity, specificity and PPV

Sensitivity, specificity and PPV were calculated by comparing the weekly HFMD consultation rate from the SSS with the weekly viral culture rate from the PHLC. A key step in these calculations was determining the epidemic and non-epidemic weeks by setting a threshold level. When the consultation rate or viral culture rate was above the threshold in a particular week, it was considered to be an epidemic week. Conversely, when the consultation rate or viral culture rate was below the threshold in a particular week, it was considered to be a non-epidemic week. Using the control chart method, the threshold level was calculated as two standard deviations (SD) above the ‘baseline level’. The ‘baseline level’ was the mean consultation rate or viral culture rate of the study period, after excluding extreme values above the 80th percentile. The respective epidemic and non-epidemic weeks for the consultation rate and viral culture rate were then determined, and sensitivity, specificity and PPV were calculated. Sensitivity analysis was conducted by varying the ‘baseline level’ and threshold level.

Data quality and timeliness

The surveillance record forms submitted by the sentinel doctors were assessed for data quality. The items ‘Total number of patients diagnosed with HFMD each day’ and ‘Total number of patients seen each day’ were checked for data completeness. The record forms were also examined for calculation error and illegible hand writing. Five weeks from each year between 2003 and 2009 were selected at random, and all record forms selected were reviewed individually for data quality. The delay in reporting (timeliness) was examined by calculating the percentage of sentinel doctors who submitted the report later than the prescribed time frame (every Tuesday).

Representativeness

The representativeness of the HFMD SSS was assessed by comparing the geographical distribution of the sentinel sites with the population distribution obtained from the Census and Statistics Department. The expected numbers of sentinel GPs/GOPCs in the 18 districts were compared with the actual numbers.

Acceptability, simplicity and flexibility

Acceptability was reflected by the willingness of the sentinel doctors to participate in the SSS by reporting the requested information to the DH each week. An acceptability survey was also conducted in June 2006. A questionnaire was sent to all sentinel GPs and GOPCs to collect their views on the SSS. The survey also examined the opinions of sentinel doctors regarding the simplicity of the SSS. The flexibility of the SSS was assessed in terms of its adaptability to change in data provider, organization structure, application of new technology, and advances in data analysis.

Results

The time series plots of the HFMD consultation rate and the positive viral culture rate of enterovirus are shown in Fig. 1, Fig. 2 . The consultation rates of both GPs and GOPCs were found to be significantly correlated with the viral culture rate from the PHLC, with Pearson correlation coefficients of 0.60 and 0.48, respectively (both P < 0.001). The correlation was higher for GPs than GOPCs.

Fig. 1

Time series plot of consultation rate for hand, foot and mouth disease (HFMD) for general practitioners (GPs) from the sentinel surveillence system (SSS) and viral detection rate, 2001–2009.

Fig. 2

Time series plot of consultation rate for hand, foot and mouth disease (HFMD) of general practice outpatient clinics (GOPCs) from the sentinel surveillence system (SSS) and viral detection rate, 2001–2009.

Fig. 1, Fig. 2 show a consistent summer peak of HFMD from May to July, except in 2003 and 2009. A severe acute respiratory syndrome epidemic and an influenza pandemic H1N1 occurred in 2003 and 2009, respectively, so the decrease in HFMD disease activity in 2003 and 2009 is likely to be associated with the territory-wide hygiene measures implemented at these times.23, 24 The seasonal epidemic was more consistent among GPs than GOPCs. Children aged 1–5 years were found to be most susceptible to HFMD, and 94.8% and 79.3% of cases occurred among children aged <12 years in the GP and GOPC groups, respectively (Fig. 3 ).

Fig. 3

Distribution of cases of hand, foot and mouth disease by age group, 2001–2009. GOPC, general outpatient clinic; GP, general practitioner.

Using the control chart method, the epidemic threshold was 1.41 per 1000 consultations for GPs and 0.12 per 1000 consultations for GOPCs (Fig. 1, Fig. 2). The epidemic threshold for the viral culture rate was 1.69%.

Sensitivity, specificity and PPV for GPs were 48.2, 88.5 and 56.8, respectively, while the corresponding values for GOPCs were 38.4, 83.4 and 42.2 (Table 1 ). Overall performance of GPs and GOPCs was fairly comparable. Both groups had high specificity, but sensitivity and PPV were borderline satisfactory.

Table 1

Sensitivity analysis for sensitivity, specificity and positive predictive value (PPV) for the hand, foot and mouth sentinel surveillence system: general practitioners and general outpatient clinics.

	Percentile			SD
	70	80	90	1	2	3
General practitioners
Sensitivity	51.3	48.2	44.3	59.3	48.2	41.7
Specificity	86.3	88.5	92.0	79.1	88.5	93.4
PPV	64.8	56.8	49.2	60.0	56.8	53.6
General outpatient clinics
Sensitivity	34.4	38.4	38.6	50.0	38.4	37.5
Specificity	84.4	83.4	90.5	71.6	83.4	90.4
PPV	52.0	42.2	41.5	48.0	42.2	41.5

SD, standard deviation.

Sensitivity analysis revealed that varying the percentile to exclude the extreme values and SDs would affect the sensitivity, specificity and PPV calculations (Table 1). For example, using 3 SD instead of 2 SD would result in lower sensitivity and PPV, but slightly higher specificity for both GPs and GOPCs. Using the present parameters (80th percentile and 2 SD) yielded fairly balanced results for both groups.

Data quality

In total, 3303 reporting forms were examined. The data required to calculate the consultation rate were largely complete. The completion rate for ‘Total no. of HFMD patients’ was 100% for both GPs and GOPCs, while completion rates for ‘Total no. of patients seen’ were 97.7% and 99.7% for GPs and GOPCs, respectively. Only a minority of providers (1.0% among GPs and 1.1% among GOPCs) had illegible hand writing and required clarification. A small percentage of the figures (1.3% among GPs and 1.2% among GOPCs) were calculated erroneously.

The geographical distributions of the sentinel GPs and GOPCs are shown in Table 2 . The actual distribution of sentinel GP and GOPC sites matched the expected sites fairly well, apart from an over-representation of GPs in Yau Tsim Mong District, an over-representation of GOPCs in the Island District, and an under-representation of GPs in the Shatin District.

Table 2

Geographic distribution of the actual and expected number of sentinel general practitioners (GPs) and general outpatient clinics (GOPCs) in Hong Kong.

District	Population	Current no. of GOPCs	Expected no. of GOPCs	Current no. of GPs	Expected no. of GPs
Hong Kong Island
Central & Western	250,064	3	2	3	1
Wan Chai	155,196	1	1	1	1
Eastern	587,690	5	4	2	4
Southern	275,162	3	2	2	2
Subtotal	1,335,469	12	12	8	8
Kowloon
Yau Tsim Mong	282,020	2	3	5	2
Shamshuipo	353,550	4	3	3	3
Kowloon City	381,352	4	4	2	3
Wong Tai Sin	444,630	4	4	2	4
Kwun Tong	562,427	5	5	5	5
Subtotal	2,023,979	19	19	17	17
New territories and islands
Kwai Tsing	477,092	6	5	3	3
Tsuen Wan	275,527	1	3	3	2
Tuen Mun	488,831	3	5	5	4
Yuen Long	449,070	4	4	1	3
North	298,657	4	3	3	2
Tai Po	310,879	2	3	3	2
Shatin	628,634	4	6	0	4
Sai Kung	327,689	3	3	3	2
Islands	86,667	6	1	2	1
Subtotal	3,343,046	33	33	23	23
Total	6,702,494	64	64	48	48

Timeliness

Only 80.5% and 91.0% of GPs and GOPCs, respectively, met the target of submitting the surveillance record form every Tuesday. The remaining sentinel doctors took a further 1–2 days after they received the reminder from the DH staff.

Acceptability

The overall participation rate was 83.3% and 93.3% among GPs and GOPCs, respectively. GOPCs had a consistently higher participation rate throughout the study period. In addition, the high degree of data completeness also reflected that the SSS was acceptable to the sentinel doctors.

The overall response rate of the acceptability survey was 87.3% (89/102), with a slightly higher rate among GOPCs (89.1%, 57/64) than GPs (81.6%, 31/38). On the whole, 65.9% of the participating doctors were satisfied or highly satisfied with the SSS (Table 3 ). Both GPs and GOPCs were satisfied with the case definition, surveillance record form and method of returning the surveillance record. Compared with GPs, GOPCs showed consistent lower satisfaction with all aspects, especially the timeliness and content of the feedback.

Table 3

Summary of results on the acceptability survey.

	Agree or strongly agree (%)
	GOPCs	GPs	All
Case definition easy to follow	92.3	96.4	93.8
Surveillance record tedious to complete	23.1	6.9	17.3
Method of returning the surveillance form is simple	80.7	90.3	84.1
Always receive feedback of surveillance information	42.1	90.3	69.1
The feedback is delivered in a timely manner	35.1	80.6	51.1
Satisfied with content of feedback	40.4	71.0	51.1
Satisfied overall	57.9	80.6	65.9

GPs, general practitioners; GOPCs, general outpatient clinics.

Simplicity and flexibility

The case definition was clinical without any need to take a specimen. Only aggregated data were required without patients’ personal information. Sentinel doctors were required to send the surveillance record form once every week by facsimile or e-mail. In the survey, 93.6% of the sentinel doctors agreed or strongly agreed that the case definition was simple. Only 17.7% regarded the notification form as tedious to complete. Approximately 88% of the doctors agreed or strongly agreed that the method for returning the form was simple. Overall, the system was regarded as relatively simple.

During the past few years, there have been significant changes related to the HFMD SSS. From 2001 to 2009, 15 GPs withdrew and 11 new GPs joined the SSS. Despite the management of GOPCs being transferred from the DH to the Hospital Authority in 2003, all GOPCs continued to serve as sentinel sites. The SSS incorporated new advancements in medical informatics, including the development of an electronic database in 2002.

Discussion

SSSs have been used to monitor various infectious diseases in the community in different parts of the world.25, 26, 27, 28 GPs are often the first point of entry of the patients into the medical system. The collection of simple data by GPs enables disease activity to be monitored at an early phase, compared with other data such as laboratory tests or hospital discharge diagnoses. In Taiwan, Japan and Malaysia, SSSs have been implemented to monitor HFMD in the community in order to allow the health authorities to issue early warning of seasonal activity, detect abnormal outbreaks and assess the impact of public health control measures.

Surveillance is an ongoing activity and programme evaluation is prudent for health authorities to demonstrate accountability and identify room for improvement. This is the first comprehensive evaluation of the HFMD SSS in Hong Kong since its establishment. The overall performance of this system was considered to be reasonably good. It has fulfilled its main objectives to monitor HFMD trends and to assist with the detection of community epidemics. Few evaluation reports from other countries were found in the literature for comparison. However, one published report from Japan reported sensitivity, specificity and PVP of the Japanese SSS of 59.7%, 98.9% and 40.1%, respectively. Compared with the Japanese counterpart, the Hong Kong HFMD SSS had lower sensitivity and specificity but a higher PPV.

In calculating sensitivity, specificity and PPV, the laboratory data were considered to be the gold standard to reflect the true viral activity in the community. The PHLC is the only public health laboratory in Hong Kong and receives hundreds of specimens from major public hospitals, GOPCs and GPs every week. From 2001 to 2009, an average of >37,000 specimens were tested for enterovirus each year by the PHLC. Given the representativeness and large scope of service, the laboratory data were considered to be a good reflection of viral activity in the community. There is no universally agreed method for setting the epidemic threshold level for HFMD surveillance. In this evaluation, the control chart method was used.30, 31 Such a method has been used both locally and internationally for public health surveillance, including SSS for influenza.32, 33 The control chart is a relatively simple and widely used method30, 31; one advantage of this method is that it gives persistent epidemic signals after the epidemic starts, continuing until the outbreak ends. The sensitivity analysis performed showed that this evaluation has obtained relatively balanced values for sensitivity, specificity and PPV.

The participation rate among GOPCs was consistently higher than that among GPs. However, in the acceptability survey, GOPCs were less satisfied with all aspects than GPs. This could be partly explained by voluntary participation by GPs as those GPs who are not satisfied with the SSS are likely to stop participating.

Regarding the study results, prompt actions have been taken to enhance the representativeness of the GP and GOPC sites by recruiting more sentinel partners in the under-represented areas. To ensure high data completeness and validity, and improve the timeliness of reporting, an online electronic system is being developed to replace the existing reporting format. Online reporting would also enhance the efficiency of data processing and dissemination of information. It is important to note that the DH should not depend solely on the SSS to monitor HFMD trends in Hong Kong. The SSS could be supplemented by other surveillance systems, such as hospital admission data and syndromic surveillance in childcare centres. Further intelligence is needed to integrate the findings detected from these various systems.

For accurate assessment of data quality, case record review or practice visits would be necessary to ensure that clinical presentations meet the case definition. However, this would require additional resources and could compromise the rapport between the sentinel doctors and the Health Authority, since review of case records or practice visits could be perceived as auditing their personal performance. In assessing timeliness, the time lapse between disease onset and date of reporting to the SSS should be assessed. This would help to assess the time lag between the occurrence of the outbreak and the time when public health actions were initiated. However, no such information was available from the existing data collection system. In assessing representativeness, the demographics of patients attending the sentinel clinics could be compared with the general population. However, as these data were not available, such analysis could not be undertaken.

The overall performance of the HFMD SSS in Hong Kong was considered to be reasonably good. The HFMD consultation rate was a good reflection of viral activities in the community, as evidenced by its significant correlation with the laboratory surveillance data. The SSS has provided useful information for understanding local epidemiology in terms of seasonal trends and high-risk groups. It allows early detection of increased disease activity in the community, and enabling early health warnings to be made to the public and health professionals. The results indicate that the SSS is simple, flexible, representative and acceptable by the participating doctors. However, its performance in terms of sensitivity and PPV needs further improvement.

Ethical approval

None sought.

Funding

None declared.

Competing interests

None declared.