Evaluating the effectiveness of an emergency preparedness training programme for public health staff in China.

BACKGROUND: The severe acute respiratory syndrome (SARS) crisis of 2003 provided a new urgency in China in terms of preparing public health staff to respond effectively to public health emergencies. Although the Chinese Government has already carried out a series of emergency education and training programmes to improve public health staff's capability of emergency preparedness, it remains unclear if these training programmes are effective and feasible. The purpose of this research was to evaluate an emergency preparedness training programme and to develop a participatory training approach for emergency response.
METHODS: Seventy-six public health staff completed the emergency preparedness training programme. The effectiveness of the training was evaluated by questionnaire before training, immediately after training and 12 months after training (follow-up). Additionally, semi-structured interviews were conducted throughout the training period.
RESULTS: The emergency preparedness training improved the knowledge levels and increased attitudinal and behavioural intention scores for emergency preparedness (P<0.01). The results at follow-up showed that the knowledge levels and attitudinal/behavioural intention scores of participants decreased slightly (P>0.05) compared with levels immediately after training (P<0.01). However, there was a significant increase compared with before training (P<0.01). Moreover, more than 80% of participants reported that the training process and resources were scientific and feasible.
CONCLUSIONS: The emergency preparedness training programme met its aims and objectives satisfactorily, and resulted in positive shifts in knowledge and attitudinal/behavioural intentions for public health staff. This suggests that this emergency training strategy was effective and feasible in improving the capability of emergency preparedness.

Introduction

Since the 9/11 disaster, anthrax bioterrorism, ‘mad cow’ disease, severe acute respiratory syndrome (SARS) and avian influenza outbreaks, public health emergencies have become an important threat to communities worldwide. Managing these emergencies and the threats they pose is part of the long-term government development plans in many countries, and expensive resources are being invested into preventing and responding to public health emergencies. In China, surge capacity is one of the most urgent problems regarding public health emergency response at the present time, along with the lack of equipment and the low efficiency of the public health emergency information system. Recognizing this, the Chinese Government carried out a series of emergency preparedness education and training programmes to improve the capability of public health staff to respond to emergencies nationwide. However, it remains unclear if these training programmes are feasible and effective in improving emergency preparedness.

Investigation has revealed that the emergency response of the public health sector was insufficient, especially the emergency preparedness of public health staff, such as not knowing the emergency response protocols and management procedures, and how to collect and analyse the relevant data during the SARS outbreak.2, 3 Therefore, in order to change the current situation and improve the capability of public health staff in China to respond to emergencies, a pilot study was developed and supported by the Ministry of Health of the People's Republic of China (MOH) and the World Health Organization (WHO). The study was carried out by Tongji Medical College in Hubei Province from 2004 to 2006. The training was completed in 2005, and the follow-up survey was conducted 12 months later.

Like any other successful health education programme, the emergency preparedness training programme should be subjected to a process of continuous monitoring, control, evaluation and, if needed, relevant modifications.4, 5, 6, 7, 8 The comprehensive evaluation of an emergency training preparedness programme should include its various aspects (contents, aims and objectives, training resources, methods, effects and impact), and it should also answer questions about the efficiency and impact of training on the participants.9, 10, 11 This study highlighted the procedures used in the evaluation of an emergency preparedness training programme, and focused on its most important aspects: training resources; training process; and effectiveness of training (before training, immediately after training and 12 months later).

Methods

Participants and trainers

Seventy-eight trainees from the Centers for Disease Control and Prevention (CDC) in 18 cities in Hubei participated in the emergency preparedness training programme in 2005. Two participants did not complete their training and were not part of the evaluation (n=76). Trainers were selected based on their expertise in the field of public health emergency response, related training programmes and their involvement in continuous consultations on health service programmes, both educational and promotional. Trainers came from the MOH, WHO, Chinese CDC, Health Department of Hubei Province, Fudan University, Wuhan University and Huazhong University of Science and Technology.

Training contents

The aims and objectives of training were designed carefully by educational and training experts with an intimate knowledge of the public health emergency response plan and the training programme, in consultation with public health personnel who did not participate in the training. The training programme was based on the US CDC's emergency preparedness core competencies for all public health workers.12, 13, 14 In brief, the training consisted of: (1) the definition of public health emergency; (2) the public health workers’ role during emergencies; (3) the responsibilities of local, province and government agencies during emergencies; (4) the role of the CDC during emergencies; (5) the CDC emergency response chain of command; (6) emergency communication strategies and use of special equipment; (7) emergency response protocols; and (8) management procedures, including the management of necessary supplies and equipment. The training contents and objectives were subjected to continuous monitoring and evaluation throughout the training period.

Training process

Various training methods were used, including case studies, workshops, tutorials, seminars, group discussions, role playing, drilling and fieldwork. Formal lecturing was the least used method. The training centre was equipped with modern audiovisual aids designed for training purposes. As well as the training logistics, other facilities and general services, such as transportation and accommodation, were provided free of charge to the participants.

Evaluation design

Individual basic information, knowledge levels, attitudes and behaviours regarding emergency preparedness were investigated by questionnaire, which was designed by experts in the field of training programmes and continuous consultation on emergency management. In order to assess the questionnaire, a pilot test was undertaken among other public health personnel who did not participate in the training, and modifications were made by experts based on the feedback.

Thirty questions assessed the participants’ knowledge of public health emergency competencies, which consisted of basic public health science knowledge, emergency management knowledge and emergency analytical/assessment skills (10 questions). If the correct answer was given to these questions, the participant received one point, whereas incorrect answers received no points. Eight items were designed to assess the staff's attitudinal and behavioural intentions related to the ‘eight core competencies for public health services’. Each of the eight items asked respondents to rate their attitudinal and behavioural intentions, as well as the frequency of their use of each of the competencies. Responses were rated on an ordinal scale (1=very low, 2=low, 3=middle, 4=high, 5=very high). Participants completed the first measurement (pre-test, baseline) on the first day of training. The post-training measurement (post-test) was conducted at the end of the last day of training. For the follow-up test, the participants were mailed a copy of the survey, with a self-addressed return envelope, 12 months after the training had been completed.

The training process and resources were subjected to continuous monitoring and evaluation by semi-structured interviews. The inclusion of the trainees in the evaluation process was extremely helpful in updating and modifying the programme. The items addressed in the semi-structured interviews were as follows: (1) the scientific methods offered; (2) the technical material presented; (3) the performance of the trainer; (4) the benefits derived by the participant; (5) the use of the audiovisual aids; (6) the strengths and weaknesses of the session; and (7) final critical comments and remarks. The forms were distributed at the end of each session to be completed anonymously by each participant. The forms were analysed immediately and the results were shown to the trainer who had conducted the session. If any defects were revealed, the necessary rectifications were made immediately. Evaluation of workshops and fieldwork was carried out in a similar fashion. Feedback of the results of the evaluation was given to the participants.

Data analysis

Most data were reported as scores. Frequency and confidence scores were derived for each domain by participants’ responses to the frequency questions and the self-efficacy questions. Repeated-measures analysis of variance was used to test differences between pre-test, post-test and follow-up test. The data from semi-structured interviews were categorized independently by three authors using the triangulation method, and the individual results of the analysis were compared and discussed until consensus was reached. All results were expressed as mean±standard deviation. Data were analysed by one-way analysis of variance using Statistical Package for the Social Sciences for Windows, Version 12.0 (SPSS. Inc., Chicago, IL, USA).

Results

Essential information

Seventy-six of the study participants completed the entire training programme and represented public health staff from the CDC of 18 cities (97.44% response). Most respondents were male (n=57, 75%) and over half (n=42, 55.26%) had earned a bachelor's or master's degree, of which one-sixth possessed Masters of Public Health degrees. Additionally, most participants (n=62, 81.58%) had more than 5 years of experience as public health staff. Some trainees (n=50, 65.79%) had participated in inter-related training approximately 12–24 months previously. The results of reliability assessment showed that test–retest reliability and the internal consistency of questionnaires was accredited to some extent (test–retest reliability of pre-training=0.83, Cronbach's alpha >0.61). The results of related analysis indicated that the construct validity of the questionnaire was of high quality (related coefficient fluctuated between 0.35 and 0.79, P<0.05).16, 17

Knowledge levels

The investigation revealed that knowledge levels of public health emergency preparedness were relatively low before training. After training, a significant increase in the mean knowledge scores was observed (pre-test: 19.79±2.41; post-test: 24.49±0.86; follow-up test: 24.24±1.58) (P<0.01). Basic public health science knowledge and emergency management knowledge scores decreased slightly (P>0.05), but the mean scores for emergency analytical/assessment skills were increased dramatically in the follow-up test compared with the post-test (P<0.01). Furthermore, there was a significant increase in overall knowledge scores between the follow-up test and the pre-test (P<0.01) (Fig. 1 ).

Fig. 1

The mean knowledge scores of participants before training (pre-test), immediately after training (post-test) and 12 months after training (follow-up) (n=76). Data are shown as mean±standard deviation. All comparisons were performed by one-way analysis of variance: (A) basic public health science knowledge; (B) emergency management knowledge; and (C) emergency analytic/assessment skills. *P<0.05 vs pre-test; ΔP<0.05 vs post-test.

Attitudinal and behavioural intentions

Descriptive statistics on attitudinal and behavioural intentions at pre-test, post-test and follow-up test are presented in Table 1 . As mentioned above, the responses ranged from high (5) to low (1). The results showed that participants reported a significant improvement in their attitudinal and behavioural intentions in all eight core competencies in the post-test compared with the pre-test. Twelve months later, there were slight decreases in participants’ attitudinal and behavioural intentions in some core competencies, but the mean score for emergency analytical/assessment skills was significantly increased compared with the post-test (4.35 vs 3.69), and mean scores for policy development/programme planning skills (2.94 vs 3.95) and financial planning and management skills (2.66 vs 3.47) were decreased compared with the post-test (P<0.05).

Table 1

Changes in attitudinal and behavioural intentions.

Competency	Competency level, mean (SD)
	Pre-test	Post-test	Follow-up test
Analytical/assessment skills	2.77 (0.81)	3.69 (0.61)*	4.35 (0.72)*Δ
Policy development/programme planning skills	2.77 (0.69)	3.95 (0.51)*	2.94 (0.55)Δ
Communication skills	2.68 (0.78)	3.95 (0.51)*	3.82 (0.61)*
Cultural competency skills	2.55 (0.96)	3.95 (0.69)*	3.56 (0.49)*
Community dimensions of practice skills	2.82 (0.73)	3.84 (0.59)*	3.99 (0.51)*
Basic public health sciences skills	2.68 (0.72)	4.11 (0.45)*	3.74 (0.69)*
Financial planning and management skills	2.32 (0.89)	3.47 (0.82)*	2.66 (0.74)Δ
Leadership and systems thinking skills	2.86 (0.99)	3.89 (0.55)*	3.82 (0.62)*

The ordinal scale ranged from 1 to 5 (1=very low, 2=low, 3=middle, 4=high, 5=very high).

SD, standard deviation; pre-test, mean scores before training; post-test, mean scores immediately after training; follow-up test, mean scores 12 months after training.

*P<0.05 vs pre-test; ΔP<0.05 vs post-test.

Training resources

The results of the semi-structured interviews showed that most participants (n=73, 96.05%) thought that the training methods were excellent/very good, and the training contents were clear and easy to understand. The remaining participants (n=3, 3.95%) indicated that the training methods needed to be improved/further developed. However, all of the participants recognized that the training was innovative.

Analysis showed that 80.26% (n=61) of participants were satisfied with the trainers’ performance, and 19.74% (n=15) of participants thought that the trainers’ performance needed to improve. However, no participants indicated that resource personnel were incompetent. Additionally, most participants (n=72, 94.74%) were very satisfied with the venue, training logistics and services, and only four participants (n=4, 5.26%) thought that logistics and services needed improvement.

Discussion

Continuous medical education and training is a process of updating knowledge, developing skills, bringing about attitudinal and behavioural changes, and improving the capability of participants to perform their tasks efficiently and effectively. Effective training methods are key to the success of an emergency training programme. A number of studies have shown that the training methods recommended by the present study educators were effective because different participants learn by different training methods, and methods of active training are especially helpful for adult learning.19, 20, 21 Formal lecturing was the least used training method because trainees do not participate actively in the learning process and the outcome is inferior to methods of active learning. The results of the evaluation suggested that up-to-date training of public health staff should focus on the development of effective training methods, and interactive training methods may help to increase the quality of training and improve retention of knowledge through immediate re-inforcement of learning.22, 23

Furthermore, comprehensive evaluation and feedback about the training programme were of vital importance for the participants and trainers as it helped participants to identify their limitations while monitoring their performance during the training period. Also, trainers tended to improve their performance as they were aware that it was being monitored and evaluated. Feedback of the results of evaluation of the training sessions to the trainers was found to be helpful in rectifying the weaknesses of sessions.

In addition, the mean scores of emergency analytical/assessment skills increased rather than decreased by 12-month follow-up. This is similar to results found by Qureshi et al. For this type of phenomenon, one must consider the experience of the public health staff at the end of 2005. Before the follow-up survey, the majority of trainees had participated in avian influenza emergency response activities, thus providing practice and increasing perceived relevance of the training. As such, this probably had a positive effect on the effectiveness of training. Nevertheless, the increased overall knowledge score and the positive change in attitudinal and behavioural parameters suggested that training programmes on emergency preparedness resulted in gaining knowledge and shifts in attitude and behaviour.

Limitations

This study had a few potential limitations. The analysis was limited to staff who were primarily engaged in disease monitoring and control, and epidemiological investigations in the CDC. In addition, evaluations were based on changes over time without the use of a horizontal comparison group. Thus, it was not possible to fully determine which changes were due to the emergency preparedness training programme and which were the result of other factors. These results, however, remained constant throughout, which provides support that these changes were due to the training programme.

Conclusion and recommendations

The effectiveness of any educational training programme depends on its continuous monitoring and evaluation, which should include appropriate and varied methods. Moreover, trainers and trainees should be actively subjected to the process of monitoring and evaluation, which was helpful in monitoring their overall performance. Immediate feedback with results analysis of the continuous monitoring and evaluation should be available to those involved so that necessary improvements can be made. The results of the evaluation suggested that the emergency training strategy was effective and feasible in improving the capability of public health staff to respond to an emergency.

Acknowledgements

The authors thank all of the participants and trainers for their hard work, and all of the coordinators for their support and help. In addition, the authors would like to thank C.K. Lee for his critical reading of the manuscript.

Ethical approval

Not required.

Funding

World Health Organization.

Competing interests

None declared.