Is in situ simulation in emergency medicine safe? A scoping review.

OBJECTIVES: To provide an overview of the available evidence regarding the safety of in situ simulation (ISS) in the emergency department (ED).
DESIGN: Scoping review.
METHODS: Original articles published before March 2021 were included if they investigated the use of ISS in the field of emergency medicine. INFORMATION SOURCES: MEDLINE, EMBASE, Cochrane and Web of Science.
RESULTS: A total of 4077 records were identified by our search strategy and 2476 abstracts were screened. One hundred and thirty full articles were reviewed and 81 full articles were included. Only 33 studies (40%) assessed safety-related issues, among which 11 chose a safety-related primary outcome. Latent safety threats (LSTs) assessment was conducted in 24 studies (30%) and the cancellation rate was described in 9 studies (11%). The possible negative impact of ISS on real ED patients was assessed in two studies (2.5%), through a questionnaire and not through patient outcomes.
CONCLUSION: Most studies use ISS for systems-based or education-based applications. Patient safety during ISS is often evaluated in the context of identifying or mitigating LSTs and rarely on the potential impact and risks to patients simultaneously receiving care in the ED. Our scoping review identified knowledge gaps related to the safe conduct of ISS in the ED, which may warrant further investigation. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

First complete overview of safety of conducting in situ simulation (ISS) in emergency medicine.

The methodology includes a detailed search strategy, clear inclusion and exclusion criteria, duplicate screening and data extraction by independent reviewers.

The main limitation is the widespread design variations of the included studies and of the terminology and concepts related to ISS.

Isolating the impact of ISS from various potential confounding factors remains a major challenge.

Our study identified major knowledge gaps related to the safe conduct of ISS in emergency medicine, which require further investigation.

Introduction

Emergency medicine (EM) is a complex and challenging specialty requiring the mastery of numerous technical and non-technical skills. Most emergency departments (ED) are overcrowded, chaotic environments with reported rates of errors up to 10%.1 Regular simulation is needed to establish and maintain skills competency for ED professionals.2 3 However, recent literature suggests a bias towards simulations for rare and unexpected cases (cardiac arrest, difficult airway, disaster medicine and/or rare cases) rather than for more common, routine cases.4–6 The retention rate of acquired knowledge and skills can also be quite low.7 8

In situ simulation (ISS) offers an optimal training solution for these practical EM-related issues.9 ISS is defined as simulation taking place in the participant’s everyday work environment.10 11 The ED environment is very different from a simulation centre and this can greatly impact the learning process.12 The value of ISS is predicated on principles from the ‘situated learning theory’ which states that learning is closely linked to the context of the experience.12 ISS is a ‘point of care’ type of simulation training used to respond to specific local teaching and training needs and, as a result, may enhance patient safety by exposing latent safety threats (LSTs) and trialling mitigation strategies.10 13

As with any intervention (or any clinical research project),14 15 there are potential harms linked to ISS including inadvertent use of simulation equipment/medications for real patient care.16–18 There is a potential for tension between the present and future state of ED care related to the delivery of ISS. During an ISS session, patients in the ED may be at risk of negative impacts because of the misdirection of resources towards the ISS. The potential risks associated with ISS and the published no go criteria seem to have been reached empirically through expert consensus.16–18 The main objective of this scoping review is to provide an overview of the available evidence regarding the safety of ISS in the ED. Our secondary objective is to explore the benefits of ISS on all levels of Kirkpatrick’s pyramid in EM.19 20

Methods

Design

We followed published guidance for conducting a scoping review.21

The results of this scoping review are reported as per the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.22

Patient and public involvement

Patients or the public were not involved in the design or the conduct of this study.

Eligibility criteria

We used broad inclusion criteria to present a comprehensive overview of ISS in EM, without any language or date limitations. Original studies were included if they investigated the use of ISS in the ED. Specifically, this comprised studies pertaining to clinician education, system/process evaluation, patient outcomes and patient safety.

We included single-group pretest, non-randomised and randomised studies, parallel-group and cross-over designs, and studies of ‘adjuvant’ instruction in which simulation was added to other instructions common to all learners.

Protocols, commentaries, conference abstracts, posters and correspondences were excluded. This review excludes systematic reviews or meta-analyses and editorial-style reviews. However, those reviews (invited, narrative or systematic), editorials and letters to the editor were scrutinised and citation tracking was conducted to retrieve potentially relevant studies.

Search strategy and information sources

An experienced librarian designed a search strategy (online supplemental eAppendix 1) to search MEDLINE, EMBASE, Cochrane and Web of Science, using keywords specific to each database from inception to January 2020. The search was updated in March 2021.

Since there is no MESH keyword for ‘ISS’, a manual search was performed in our four targeted databases.

Selection process

All identified studies were exported to Endnote V.X9 (Clarivate Analytics, Philadelphia, Pennsylvania, USA) and duplicates were then removed. Four authors (JT, WL, GM and EJ) independently assessed the eligibility of each study. Title, abstract and full text were first screened and then cross-referenced between reviewers. Disagreements were resolved by consensus. Any unresolved disagreement was discussed with a third researcher with experience in the field of simulation studies (ER-D).

Given the nature of a scoping review, risk of bias and quality appraisals were not performed.21

Data collection

Four independent reviewers completed data extraction, ensuring a double reviewing for each article.

Outcomes

The main outcome, safety, is defined in this study as the absence of incidents, accidents or a state with the minimal acceptable level of risk. This definition of safety has been used in other simulation studies.23–25 We selected specific criteria such as wait time, patients leaving without being seen or accident reports, which are based on literature on safety metrics in EM.26 27 We explored two safety concepts: ‘ongoing’ safety of patients being managed while ISS was ongoing in the ED (measured by ISS cancellation rate, accident reports, ED median wait time, number of patients who left without being seen) and ‘future patient’ safety with more long-term benefits from ISS (measured by LST evaluation). We sought to explore the impact of ISS on patients’ safety throughout time. Indeed, future patients may benefit from ISS if its output translates into improved skills, clinical care and patient safety.

Secondary outcomes were assessed using the validated Kirkpatrick pyramid, which is a tool used to rank educational interventions on a scale of 1–5 (figure 1).19 Because of the specificity of ISS, we also distinguished studies that used ISS to teach and train from those that aimed to explore procedures and systems.

Figure 1

The validated Kirkpatrick pyramid used to rank educational interventions.

Data synthesis

Descriptive statistics and percentages were used to describe our results. No statistical analysis, quantitative meta-analysis or pooling was conducted due to the heterogeneous study designs and methodologies.

Results

Study selection

A total of 4077 records were identified by our initial search strategy. After removing 1601 duplicates, 2476 titles and abstracts were screened. Of those, 2366 did not meet our eligibility criteria. A second search was conducted in March 2021 after which seven original articles were added to our review. A total of 81 full articles were analysed (online supplemental eAppendix 2).

General characteristics of included studies

All reviewed articles were published in English between 2006 and 2021 in specialised journals (81%, n=66).

Most included studies used a prospective design (94%; n=76), 47 of which were observational (58%). Only 3 studies were randomised controlled trials and 28 were pre/post intervention studies (35%). For the most part, the included studies were single centre (80%; n=65) and quantitative (67%; n=54). A high majority included interprofessional participants, defined as >2 professions (84%; n=68).

The clinical topics explored in the studies included ED care (77%; n=62) and extrahospital care settings (such as HEMS, prehospital care, etc) (22%; n=18). Most studies focused on ISS training for advanced life support/basic life support (n=44, 54%). The remainder focused on a variety of clinical topics including trauma (n=35, 43%) and airway management (n=21, 26%). Three studies particularly focused on COVID-19 (4%). Very few studies used consistent scenarios and as a result more detailed thematic analysis was not performed. Ethical approval was obtained for 51 studies (63%). The full characteristics of the included studies are presented in table 1.

Table 1

General characteristic of studies (n=81)

Study characteristics	n (%)
Study design
Pre/post design	28 (35)
Comparative
≥2 groups	16 (20)
1 group	55 (68)
Prospective	76 (94)
Retrospective	4 (5)
Randomised controlled trial	3 (4)
Single centre	65 (80)
Quantitative	54 (67)
Qualitative	17 (21)
Mixed	9 (11)
Participants
Mean no of participants per study ±SD (min; max)	88.9±66 (4; 398)
Medical students	0 (0)
Residents	4 (4.9)
Emergency department teams	62 (76.5)
Nurses and nursing students	2 (2.5)
Physicians in practice	3 (3.7)
Paramedics	2 (2.5)
Interprofessional	68 (84)
Clinical topics
ALS, BLS	44 (54)
Trauma	35 (43)
Intubation	21 (26)
Peadiatrics	22 (27)
COVID-19	3 (4)
Goal of ISS
to train skills	34 (49)
to assess skills or a procedure	24 (35)
to train and assess	16 (18.5)
NA	7 (9)

ALS, advanced life support; BLS, basic life support; ISS, in situ simulation; NA, not available.

Is ISS safe for patients in EM?

We found that 33 studies (40%) evaluated safety issues, among which 12 (15%) chose a safety-related primary outcome (table 2).

Table 2

Safety

Satefy-related parameters	n (%)	References
Safety assessement	33 (40)	Amiel et al.28Abulebda et al.29Chan et al.31Fan et al.45Hamman et al.59Hamman et al.60Hargestam et al.61Hunt et al.44Kerner et al.62Kobayashi et al.52Kobayashi et al.63Mondrup et al.33O'Leary et al.64Patterson et al.10Patterson et al.65Petrosoniak et al.30Shrestha et al.34Sørensen et al.66Theilen et al.41Ullman et al.67Walsh et al.68Whitfill et al.69Zimmermann et al.70Bradley et al.71Paltved et al.32Bredmose et al.35Couto et al.36Geis et al.72Lakissian et al.73Shah et al.37Wong et al.74Petrosoniak et al.75Aljahany et al.76
Quantitative data such as number of departures without being seen, median waiting time	0
Accident reports	3 (4)	Chan et al.31Patterson et al.10Paltved et al.32
Cancellation rate	9 (11)	Chan et al.31Mondrup et al.33Patterson et al.10Shrestha et al.34Paltved et al.32Bredmose et al.35Couto et al.36Shah et al.37Auerbach et al.38
LST evaluation	24 (25)	Chan et al.31Hamman et al.59Hamman et al.60Hunt et al.44Kerner et al.62Kobayashi et al.52Kobayashi et al.63O'Leary et al.64Patterson et al.10Shrestha et al.34Theilen et al.41Ullman et al.67Walsh et al.68Whitfill et al.69Zimmermann et al.70Bradley et al.71Couto et al.36Geis et al.72Lakissian et al.73Shah et al.37Wong et al.74Petrosoniak et al.75Aljahany et al.76Zern et al.77Shrestha et al.78Gray et al.42
Others (survey, qualitative analysis)	5 (6.3)	Amiel et al28Abulebda et al29Petrosoniak et al30Wong et al74Sørensen et al79

LST, latent safety threat.

Ongoing safety

As for ongoing safety of patients during ISS, none of the included studies assessed the following: (1) The number of patients in ED who left without being seen during the period of the ISS training, or (2) The impact of ISS on patient wait times and (3) Safety-related quality parameters in the ED.

No studies included official institutional accident reports, but three studies included surveys on accident reports.28–30 No accident was reported during ISS.10 31 32 The cancellation rate of ISS was described in nine studies (11%).10 31–38 One study included analyses on feasibility (based on department status, such as patient waiting to be seen) in their secondary outcomes. The authors showed that 3 ISS were delayed, and none were cancelled but no other patient-related outcome was presented.39

The impact of ISS on real ED patients was assessed in two studies. In one multicentre clinical trial, ISS was used to prepare and facilitate the identification and mitigation of threats to study participation and patient safety.26 In this case, ISS was not really used to teach and directly improve skills but rather to prepare teams for the implementation of a clinical trial. As an ongoing research project can also jeopardise the safety of other patients by redirecting attention on the study and not on real patients, safety was analysed with parameters such as latent threats identification, mitigation and protocol errors and deviation. Patterson et al’s work on LST identification also addressed the impact of ISS training on patient care,10 which was assessed using a questionnaire. Four participants out of 118 reported that the simulation was disruptive or affected the participants in a negative way. These are the only studies exploring the impact of ISS on patients being managed in the ED.

Future safety of patients

The LST assessment was conducted in 24 studies (30%).

Kirkpatrick evaluation level

Most studies employed Kirkpatrick-related endpoints (n=75; 92%). Fourteen studies (17%) reported comparison with a preintervention assessment or a control group using knowledge as the outcome (KP1), mostly through a pre/post design. The assessment of KP2 level was provided for 10 studies (12%), and 27 (33%) using ISS as an intervention in EM to assess a level 3 outcome.

Twenty-two studies assessed a level 4 outcome. We considered LST identification as a level 4, because of the direct impact of this methodology on patient care. This classification was decided on consensus among the authors. For instance, in the study by Gray et al (38), the authors showed that the identification of various LSTs regarding massive blood transfusion led to improved patient outcomes (measured by a decrease in the delays between massive haemorrhage protocol activation and blood component administration). We identified three studies with a direct impact on patient outcomes (morbidity, mortality, length of stay). Using a pre–post study design, Steinemann et al demonstrated a significant improvement in mean teamwork scores of real patient care in addition to a 16% reduction in the mean resuscitation time in the ED.40 The other study that included a patient outcome analysis examined the effect of introducing a paediatric medical emergency team combined with regular ISS training on patient and system-level outcomes.41 Even though the decrease in paediatric intensive care unit mortality was nonsignificant, the authors found an association between ISS and a reduction of healthcare-related costs (level 5). Using a prospective observational design, Theilen et al showed that regular ISS training of ED healthcare professionals was associated with a reduction in costs and overall patient mortality. Another more recent study showed direct patient outcome improvement.42 The authors showed that a novel ISS-based quality improvement (QI) intervention for blood component administration in bleeding trauma patients led to a 21% mean reduction in time between massive haemorrhage protocol activation and blood component administration. The impact of changing the institution’s massive haemorrhage protocol after the identification of LSTs during ISS was explored in this study using a pre/post design.

No studies showed a negative impact when exploring the improvement of knowledge or skills. The efficacy of ISS was always superior when compared with any another training method or to a control group. We were unable to categorise 10 of our included studies (12.3%) within the 5 levels of KP, either because the studies assessed guideline adherence38 43 or because ISS was employed to assess a process or a system (table 3).44 45

Table 3

Outcomes

Kirkpatrick level	n (%)	References
KP 1: satisfaction, feelings and perceptions	14	Bischof et al.80Burke et al.81Couto et al.13Davison et al.82Gangadharan et al.83Gundrosen et al.84Hunt et al.44Mannenbach et al.85Meurling et al.86O'Leary et al.87Shrestha et al.34Sørensen et al.88Ullman et al.67Katznelson et al.89
KP 2: knowledge and skills	10	Walsh et al.68Bradley et al.71Thomas et al.90Lemke et al.91Kalidindi et al.92Ben-Ari et al.93Auerbach et al.94Abualenain et al.95Bischof et al.80Sørensen et al.88
KP 3: behaviours	27	Abu-Sultaneh et al.96Amiel et al.28Armstrong et al.97Auerbach et al.38Barker et al.98Bayouth et al.99Bredmose et al.35Campbell et al.100Coggins et al.101Farah et al.102Generoso et al.103Jörgens et al.104Jung et al.105Katznelson et al.106Lakissian et al.73Miller et al.107O'Leary et al.87Patterson et al.65Petrosoniak et al.30Pirie et al.108Qian 109et alSaqe-Rockoff et al.110Truta et al.111Wong et al.74Zern et al.77Zimmermann et al.70Walsh et al.68
KP4: patient outcome	22	Abulebda et al.29Aljahany et al.76Barni et al.112Chan et al.31Couto et al.36Geis et al.72Gray et al.42Hamman et al.59Hamman et al.60Hargestam et al.61Kerner RL et al.62Kobayashi et al.63Lavelle et al.113Paltved et al.32Patterson et al.10Shah et al.37Shrestha et al.34Steinemann et al.40Wang et al.114Whitfill et al.69Petrosoniak et al.75Theilen et al.41Walsh et al.68
KP5: costs	2	Theilen et al.41Petrosoniak et al.75
Other, NA	10	Hunt et al.44Kobayashi et al.52Mondrup et al.33Kessler et al.115Siegel et al.43Stevens et al.116Wieck et al.117Nadkarni et al.118Auerbach et al.119O'Leary et al.87

NA, not available.

Indeed, for 24 studies (29%), ISS was used to explore and assess a procedure, an organisation, or guideline adherence and not to teach or train. Seven studies (8.6%) assessed the feasibility of ISS itself without exploring learning or safety outcomes.

Discussion

This scoping review assessed the safety of ISS in EM.

ISS and patient safety

Conducting ISS within the chaotic and busy ED environment may raise concerns for the ongoing care of real patients. However, we found that few studies evaluated the impact of ISS on the safety of actual ED patients. Our scoping review focused on the available evidence regarding two safety concepts: ongoing ED patient safety during ISS, and the safety of future patients (long-term benefits of ISS). It is important for educators and ED professionals to be aware of the associated risks of ISS. Specific, rigorous guidelines are needed to help create a framework aiming to enhance ISS ‘ongoing’ safety. Surprisingly, we did not find any studies evaluating the impact of ISS on ongoing patient care, clinical adverse events, wait times or patients leaving without being seen. Some editorials warn against these risks,16–18 but we found no studies exploring nor demonstrating the potential consequences or risks of ISS in a busy ED for the ongoing care of patients. Most studies seemingly overvalue the ‘future’ safety of patients while undervaluing the ‘ongoing’ safety of ED patients during ISS. This seems crucial because ISS is increasingly used to teach and train EM professionals. As for any method aiming at improving the quality of care, the risks for patients should be controlled. In that sense, any type of clinical research may jeopardise patient safety. For example, the time it takes to consent and enrol a patient in a therapeutic trial could impact ED safety measures. The time-sensitive nature of EM research has been reported in research primers.14 Our scoping review shows the absence of scientific evidence from studies with rigorous methodology to confirm the absence of risks for patients. The Foundation for Healthcare Simulation Safety published a 10-item ‘pledge’ of ‘best practices’ for simulation programmes to reduce simulation-related hazards.17 46 However, even if those principles seem logical, they are not evidence based.

Educators and simulation teams should consider guidelines from existing literature on QI to ensure the sustainability of these interventions. In the case of ISS, the ongoing safety of patients could be referred to as a ‘balancing measure’. In the QI literature, balancing measures represent checkpoints that ensure there are no potential unintended consequences (risk to ongoing patient safety) resulting from an intervention (ISS).37 47–51

Patient-reported measures, which appear to be underutilised based on our study results, represent a useful lens to understand the value of ISS more comprehensively. For example, at one investigator’s institution (AP), a Code Orange simulation during which dedicated volunteers checked in with patients waiting and patients had positive perceptions regarding this ongoing training (unpublished data).

One would think that being closer to the patient and to the patient care facility would make it easier and more intuitive to assess ISS-related patient outcomes, but only two studies assessed direct patient outcomes.41 42 This raises the question of the relevance of the Kirkpatrick pyramid. Indeed, we deemed appropriate to classify the identification of LST as a level 4 on the KP scale, because it is directly linked to improving the quality of care. This decision was reached on consensus among the authors and is debatable because direct patient outcomes are not measured. However, we found data exploring the improvement of care following the identification of LSTs to validate our choice of classification.41 42 We believe even though this was not directly measured in every study, the identification of LSTs leads to improved patient outcomes by reducing medical errors.52 We were unable to classify a great number of studies using this tool. As other authors have previously suggested, it may be wise to adopt a new lexicon and new endpoints to assess the impact of ISS.53 54 Some studies did, however, explore level KP4 and even KP5.41 Their inspiring results should encourage researchers to pursue their efforts and lead projects aiming at exploring higher levels of the KP pyramid.

Study methodology limitations

As highlighted in previous studies, studies exploring the impact of simulation-based interventions are still needed.2 55–57 We observed that even though the included studies were quite recent, as ISS seems to have emerged in the last 20 years, the vast majority used observational prospective designs. With ISS, the proximity between the intervention and the possible patient-centred outcomes is inspiring and educators should seek collaboration with methodologists to elaborate research protocols leaning to prove the translational dimension of ISS. Simulation should be about improving patient outcomes, and the causative effect of isolating ISS should be a priority.

Strengths and limitations

Because there is no mesh word for ‘ISS’ we conducted a search on simulation and EM followed by more detailed reviews for ‘in situ’ specific articles, which made the screening process more complex. Furthermore, the terms used to describe ISS are highly heterogeneous. It is possible that this resulted in missed articles that otherwise should have been included. However, the corollary is this process required a close assessment of each article resulting in a more thorough review.

Also, several systematic reviews were excluded from our analysis. However, we reviewed and cross-referenced their reference list against our literature search. We found no discrepancies, supporting the accuracy of our approach.

Conclusion

Our scoping review identified major knowledge gaps related to the safe delivery of ISS in the ED, which require further investigation. Patient safety during ISS is often evaluated in the context of identifying or mitigating LSTs with little focus on the potential impact and risks to actual ED patients. For all these reasons, our research team sought to address this issue in a mix-method study, which will aim at demonstrating the safety and impact of ISS.58