Knowledge retention and usefulness of simulation exercises for disaster medicine - what do specialty trainees know and think?

INTRODUCTION: Disaster medicine education is an important but often neglected part of Emergency Medicine (EM) specialty trainees' curriculum. It is especially neglected in limited resource environments (1), which, owing to poor infrastructure generally, are more likely to be affected by disasters than better resourced environments. Disaster medicine cannot be taught solely in a classroom and various methods are required to teach practical concepts. This study aims to look at Emergency specialty trainees' perception of high-fidelity simulation and their needs with regards to Disaster Medicine Education.
METHODS: This was a prospective cross-sectional cohort study involving 27 EM specialty trainees from the University of the Witwatersrand, who, participated in a high-fidelity simulation and were given a questionnaire before and after the exercise. The questionnaire consisted of theory questions relating to disaster medicine as well as Emergency Specialty trainee's perception and needs towards disaster medicine education.
RESULTS: High fidelity simulation does not increase theoretical knowledge of Disaster Medicine but it does increase perceived confidence. EM specialty trainees seek yearly training, beginning in their first year and choose high fidelity simulation as their preferred method of training.
CONCLUSION: High fidelity simulation is crucial to increasing the confidence of EM specialty trainees during their training. More research is needed to develop core competencies and methods of evaluating training.

African relevance

Low-middle income countries are affected disproportionately with disasters

Disaster medicine education is an under researched field in developing nations

Emergency medicine specialty trainees and specialists are likely to be at the front line of a disaster response

Introduction

Disaster medicine is an important subject for doctors, particularly Emergency Medicine (EM) specialists who will often be the forefront of disaster incidents [2]. It is also useful to develop other core competencies required by the EM physician such as communication, collaboration and management skills [3]. Disaster medicine is an often-neglected area in the undergraduate and post graduate curricula of medical training [[4], [5], [6]]. It is difficult for educators to prioritise education on a “low probability” event and a challenge to find teaching methods that increase the above-mentioned competencies as well as other skills needed for disaster management. Disaster medicine is also a neglected area of research in resource limited environments.

To the author's knowledge there are no studies relating to disaster medicine education in resource constrained environments. These environments are often hit hardest during disaster and are at great risk for future disasters. Climate change, urbanisation and over population are factors which are likely to lead to an increase in severity and frequency of disasters and these may affect the developing world disproportionately [7].

It has been shown that medical professionals who have received training in disaster medicine are more willing to respond to a mass casualty incident [8]. Additionally, uncoordinated responses by individuals who have not received training can hamper the response to a disaster [9]⁠. In low income countries that already experience a low doctor to patient ratio it is imperative that all available medical staff are willing and trained to respond to a mass casualty or disaster incident. Regular and effective training is therefore imperative to ensure that when disaster strikes all available staff are prepared and willing to respond appropriately.

The various methods used for teaching disaster medicine are lectures, low fidelity simulations, so called “tabletop” exercises, computerised simulations, and high fidelity, in-field simulations. Lectures may be effective at imparting knowledge on theory but not necessarily practical skills that are useful in a disaster situation. There are studies proving that simulation-based learning is effective for teaching aspects of medical education [10], but few relating specifically to disaster medicine. It has been shown that although lectures are adequate at teaching theory, doctors feel more confident after simulations of disasters [11]. High fidelity simulations have been shown to improve a personal sense of preparedness [12]. Computerised simulations are an option but often the high development costs and limited technological reach make this approach impractical in poorly-resourced countries However, no studies have been done in a resource limited setting.

High fidelity simulation is a costly and time-consuming exercise [13]⁠. It therefore needs to be investigated as a method of disaster medicine training as to whether it improves confidence and EM specialty trainees' perceptions of their skills. It is very difficult to assess disaster medicine educational tools as no standardised method of evaluation exists [14]⁠. However, the Kirkpatrick model [15] is a well validated model that can be used to assess training methods. The Kirkpatrick model uses 4 levels to assess the efficacy of the intervention [15] ⁠. The first level is Reaction- how useful do the participants think the training exercise is. The second level is Learning which can be assessed using a pre and post test. The third level is Behavior and this component assesses whether the trainees use what they have learned in their daily work. The fourth level is Results and this evaluates whether the behavior assessed in level three results in actual improved outcomes. As disasters are rare in everyday practice, it is difficult to ascertain whether training has improved the relevant skills. The needs of EM specialty trainees should also be investigated with regards to disaster medicine education so that the efforts into disaster training can be effective.

Methods

Survey

A sample population of 27 EM specialty trainees from the University of Witwatersrand (Wits) took part in a disaster medicine simulation as part of their academic program. The specialty trainees were given pre-reading (selected articles chosen by the educators) as well as a questionnaire to complete pre simulation. The questionnaire was designed by the researchers using the Kirkpatrick model as a framework.

Part A of the questionnaire (see Appendix B data supplement) included basic demographic information, Likert scale questions regarding attitude and perceptions of a mass casualty incident and a theoretical component with questions pertaining to disaster medicine curriculum. This questionnaire was answered in Google Forms in the week prior to the simulation. This part of the questionnaire assessed level 2 (Learning) of the Kirkpatrick model. Part A was administered pre and post simulation. Part B of the questionnaire was only answered post high-fidelity simulation and contained specifics regarding the simulation and also the needs of the specialty trainees with regards to a disaster medicine curriculum. This part of the questionnaire assessed level 1 (reaction) of the Kirkpatrick model. In addition to using the Kirkpatrick model to develop the questionnaire, questions were included that assessed the specialty trainees perceived needs pertaining to disaster medicine education which would aid in building a disaster medicine curriculum.

The specialty trainees then participated in a high-fidelity simulation exercise in conjunction with the University of Johannesburg Department of Emergency Medical Care. A structural collapse scenario was simulated with actors playing the role of patients. The simulation costs were reduced by using a number of strategies. Firstly, emergency care students were used as volunteers for moulage. Secondly, through inter-facility collaboration with UJ, who provided rescue equipment and personal protective equipment, and WITS, who assisted with planning and provided catering. UJ also has a long standing relationship with City of Johannesburg Emergency Management Services (COJEMS), who own the site where the simulation took place.

The specialty trainees were assigned to roles (command and control, rescuer, triage, and treatment). These roles did however change during the exercise to reflect the dynamic environment of a disaster. The simulation lasted approximately 3 h. After the simulation, a debriefing was held. Six months after the exercise the specialty trainees were given Part A and B of the questionnaire. This questionnaire was answered on hard copy to ensure maximum response rate.

Data analysis

Statistical analyses were conducted in R software (version 4.00; www.R-project.org). Test for data normality were done using the Shapiro–Wilk test and examining Q-Q plots. All data in this study were non-normal and appropriate non-parametric analyses were conducted, including Wilcoxon matched pairs for data collected pre- and post-simulation, chi-squared contingency table analyses and Pearson's chi-squared analyses. Tests were two-tailed, and model significance set at 0.05.

Results

A total of 27 Emergency Medicine specialty trainees took part in this study, 26 of whom completed the questionnaire pre-simulation and post-simulation and one who participated in the pre-simulation only, indicating a response rate of 98%.

Theory

Specialty trainees scored a median of 22 (out of a total of 25) in both the pre- and post-simulation MCQ questions, which did not differ significantly from chance (Wilcoxon test, p = 0.898). Thus level 2 (learning) of the Kirkpatrick model was not significantly influenced by high fidelity simulation.

Improving skills

Specialty trainees were asked about their confidence in managing a structural collapse scenario in five areas. Pre simulation, the most common response was “not confident” for the role of command and control, “not confident” for rescuer, “slightly confident” for triage and “slightly confident” for safety. Post simulation, the most common response was “slightly confident” for command and control, “confident” for rescuer, “confident” for triage and “mostly confident” for safety. Therefore, the specialty trainees felt a subjective improvement in their confidence and thus a positive impression of the training exercise (Kirkpatrick level 1, Reaction). See Fig. 1.

Fig. 1

Usefulness to improving confidence of specialty trainees in taking part in a mass casualty incident.

Educational needs

Importance of training for specialty trainees

Significantly more specialty trainees agreed that the simulation training increased their knowledge (p < 0.001) and confidence (p < 0.001). Thus, the specialty trainees showed a positive reaction (Kirkpatrick level 1) to the high-fidelity simulation. Significantly more specialty trainees strongly agreed that simulation training was important in their curriculum (p < 0.001) and mentioned that insufficient time was spent in disaster training (p < 0.001).

Most (58%) of EM specialty trainees think that Disaster medicine training should begin in the first year, with 23% believing it should start in second year. The majority (68%) prefer yearly training, and 19% believe it should be done every 6 months. Only 12% believe it should be done every two years. See Fig. 2, Fig. 3.

Fig. 2

Responses to “which scenarios would you like to practice in a simulation?” (1 being most likely to experience and 5 being least).

Fig. 3

Ranked preferential teaching methods with 1 being most important and 5 being least.

Teaching perceptions of specialty trainees

For improving knowledge, significantly more specialty trainees maintained that pre-reading (p < 0.001), briefing (p < 0.001), simulation (p < 0.001) and debriefing (p < 0.001) were all very useful. A pre-test was regarded by most to be somewhat useful (p < 0.001). None of the specialty trainees mentioned that any training aspect was completely useless.

For improving confidence, significantly more specialty trainees maintained pre-reading (p < 0.001) was useful. Briefing (p < 0.001), simulation (p < 0.001) and debriefing (p < 0.001) were all considered very useful.

Specialty trainees were asked about their preference for combination of methods for simulation. Most preferred High-fidelity simulation and Exercises in simulation laboratory, although their preference did not differ significantly by chance (p = 0.76). See Table 1.

Table 1

Specialty trainees' preference for combination of methods for simulation.

Methods	Number of responses
High fidelity simulation and exercises in simulation laboratory	11
High fidelity simulation and lectures	10
High fidelity simulation and tabletop exercises	3
Lectures and exercises in the simulation laboratory	2

Discussion

Disaster medicine education is not well studied in resource limited environments [1] as most research on this topic is done in the United States and Europe [16]. Disaster medicine curricula lack robust research, standardised evaluation and competency driven goals [17]. There are also very few studies that evaluate high fidelity simulation for disaster medicine education.

The above study shows that high fidelity simulation, while a costly and resource intense effort, improves the self-reported knowledge and confidence of EM specialty trainees in a resource limited setting. It also shows that high fidelity simulation can be undertaken in low resource environments, with collaboration with other departments and use of existing training facilities which helps to keep within budget. However, a simulation does not increase theoretical knowledge of disaster medicine and thus didactic lectures/pre reading must be used to improve theoretical knowledge.

The perceived confidence of the specialty trainees improved post simulation. The simulation was very useful in improving their perceived skill sets in the different areas of disaster management (command and control, triage etc). This is important as even though the specialty trainees did not participate in every role in the simulation, they all felt an increase in skill across the various roles. A similar study of medical interns by Ngo et al. [18], studying high fidelity simulation produced comparable results.

All improvement of skills and knowledge by participants are self-reported and were not objectively measured by monitoring of performance or marking. This is a common finding with disaster medicine research [16,18] and may be an important measure of improvement for future studies on high fidelity simulations in disaster medicine education.

Studies have also used the Kirkpatrick model to assess disaster medicine education [16]. Using the Kirkpatrick model, we can see that the study evaluates level 1 and level 2. Reaction (level 1) was assessed and it was found to be perceived as very useful for the specialty trainees. Level two is learning and there was no difference in the theory component however the participants did perceive an improvement in skills. A more robust way to assess level 2 would be to record or observe the participants and develop a mark sheet or rubric for basic skill competencies. This study did not evaluate level 3 (Transfer), which is similar to other disaster medicine studies [16] and level 4 (Results) can only be assessed during a disaster event.

The specifics of the simulation were also examined in this study. Interestingly, the participants found that the briefing, simulation, and debriefings were all useful. This is important information for the development of a disaster curriculum for South African specialty trainees.

The study also shows that most specialty trainees feel that too little time is spent on disaster medicine education, which is in line with other studies [19,20]. Similar to studies in Europe, participants favoured road traffic accidents as the scenario for future simulations [21]. The EM specialty trainees feel that teaching should occur annually and most picked high-fidelity simulation as the most useful teaching method. This can assist those involved with EM specialty training to develop a curriculum that meets the needs of the specialty trainees.

The limitations of this study are that it includes a small number of specialty trainees from a single training centre. The second questionnaire was administered 6 months after the event and the duration between the exercise and the questionnaire might affect the answers. The specialty trainees were not evaluated during their performance and thus only report a subjective improvement. The simulation was only done on a single scenario. The high-fidelity nature of the simulation made it difficult for observers to evaluate the specialty trainees performance.

Conclusion

High fidelity simulation is a valuable tool for disaster medicine education in a resource limited setting. It is a neglected part of the curriculum of EM specialty trainees and standardised curricula need to be developed. This study has assessed the needs of EM specialty trainees and thus can aid developing future training for them.

Dissemination of results

Results from this study were shared with the specialty trainees who participated as well as the educators involved with developing the academic curriculum for the specialty trainees.

CRediT authorship contribution statement

Authors contributed as follow to the conception or design of the work; the acquisition, analysis, or interpretation of data for the work; and drafting the work or revising it critically for important intellectual content: LC contributed 70%, KS 20% and AG 10%. All authors approved the version to be published and agreed to be accountable for all aspects of the work.

Declaration of competing interest

The authors declared no conflicts of interest.