A Novel Approach to Emergency Department Readiness for Airborne Precautions Using Simulation-Based Clinical Systems Testing.

STUDY
OBJECTIVES: Emergency department (ED) COVID-19 preparations required rethinking workflows and introducing the potential for errors. Simulation provides a nimble methodology integrating into situ training and systems testing to prepare staff, detect potential workflow latent safety threats and provide recommendations for mitigation.
METHODS: We developed 5 onsite rapid-cycle ED simulation cases using "tipping points" related to new protocols coupled with a structured observation tool. Staff observed simulations, recorded adherence to protocols, identified safety threats, discussed mitigation strategies, and participants completed an evaluation using a 5-point Likert scale. Latent safety threats were prioritized by risk and escalated to leadership.
RESULTS: Through 44 simulations, 76 staff identified 31 unique latent safety threats in the following categories: job aids 9 (29%), isolation measures 8 (26%), communication and personnel 6 (19%), and technology and equipment 8 (26%). Eleven high-priority safety threats were escalated to ED leadership. Sixty-five staff (86% of participants) completed a web-based evaluation reporting that simulations were worth the time (86% strongly agreed), an effective way to test the system (92% strongly agreed), and an acceptable way to improve (92% strongly agreed).
CONCLUSION: Our study demonstrated that simulation-based clinical systems test methods are adaptable for rapid preparedness evaluation and training. In combination with rapid-cycle deliberate practice, many latent safety threats were identified prior to clinical implementation. Our work highlights a novel application of simulation systems to increase system preparedness and reduce the potential for errors which may be applicable in diverse settings for designing, evaluating, and training staff in new protocols and procedures.

What is already known on this topic

Emergency department (ED) infection control practices had to change rapidly in response to the COVID-19 pandemic.

What question this study addressed

Can simulations be used for both training and threat identification?

What this study adds to our knowledge

Onsite simulations conducted in one of 2 pediatric EDs allowed for rapid identification of unanticipated safety threats and collection of mitigation solutions from ED clinicians.

How this is relevant to clinical practice

In a time of rapid practice change, onsite simulations can educate providers while providing an opportunity for improvement.

Introduction

Background

The emergence of the novel severe acute respiratory syndrome virus (SARS-CoV-2) and COVID-19 disease resulted in major challenges that have strained our global health systems and changed the way we manage care.1, 2, 3, 4, 5, 6, 7, 8, 9, 10 For example, as hospitals began to get overwhelmed early in the COVID-19 pandemic, the potential surge of pediatric and adult patients prompted our children’s hospital to devise an emergency response to limit infection spread among patients and protect staff.7, 8, 9, 10 In response, stakeholder leaders designed and implemented several new emergency department (ED) care guidelines.

However, rapid systems change is difficult and may introduce opportunities for error. Avoiding such errors can be challenging in a busy ED setting, especially in the face of rapidly changing federal recommendations. The goal of managing system change should be to mitigate harm, in part by identifying risks before they reach a patient or staff member. Often developed guidelines represent “work as imagined” and may not reflect work as actually performed. Trialing such guidelines prior to implementation may reveal potential errors or gaps, known as latent safety threats, which were not anticipated during design, but could nevertheless result in harm to patients and staff.13, 14, 15, 16

The onsite simulation could provide this opportunity for newly developed guidelines to be taught and tested. As such, simulation has emerged as an important educational tool for improving patient safety and quality of care.17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27 Rapid-Cycle Deliberate Practice is a simulation-based training approach that focuses on the rapid acquisition of procedural and teamwork skills. The advantage of this method is that steps in care are practiced multiple times in a single session.

Recently, simulation-based clinical systems tests have demonstrated the ability to identify and remediate potential latent safety threats.29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39 This simulation method changes the scenario/debriefing focus from training/education to probing the system function. Much of the literature on simulation-based clinical systems tests have focused on the trialing of a new space prior to opening or integrating new services into existing practice settings.33, 34, 35, 36, 37, 38, 39 These examples describe months of planning and implementation that were unavailable during the COVID-19 pandemic. In a recent publication, we demonstrated this method for COVID-19 testing in the operating room. To be effective requires creating a structured observation method focused on participants’ actions (the steps they should be taking) and focus debriefs with systems-level questions (What happened? What should happen?).38, 39, 40

Importance

Our approach of combining rapid-cycle deliberate education and systems testing methods allows frontline staff to quickly test and train in the use of evolving guidelines, thereby improving their adherence to new protocols and simultaneously addressing discovered latent safety threats with practical solutions. This highly responsive, nimble, and flexible model could be deployed beyond pandemic preparedness as a model to combine training and allow frontline staff to contribute to safety systems at the unit and institutional levels.

Goals of This Investigation

This study aimed to test a novel onsite simulation approach to train staff in new and evolving workflows, detect latent safety threats, and propose improvement solutions.

We had 2 main objectives: (1) to identify latent safety threats and recommendations for mitigation using simulation-based clinical systems test with rapid-cycle simulation concepts and (2) to evaluate this approach for feasibility and utility based on staff assessments. The primary outcomes were the number of latent safety threats identified, the mitigation solutions collected during debriefing, and the staff evaluations of the process collected through immediate posttraining surveys.

Materials and Methods

Study Design and Setting

This observational study took place in the EDs (1 suburban and 1 urban) of an academic tertiary care children’s hospital and was approved by the Children’s Mercy Hospital institutional review board as nonhuman subjects research and designed according to simulation-based Consolidated Standards of Reporting Trials research reporting guidelines. The exploratory mixed method approach combined training/education modes with probing the new workflows to explore potential latent safety threats. We adapted the methods from our recent work in the operating room and developed 5 rapid onsite cycle ED simulation cases related to new protocols coupled with a structured observation tool. The research team observed simulations logged adherence to protocols, identified latent safety threats, and participating staff were encouraged to provide input regarding potential solutions, and the case block was repeated with the suggested strategies. After each exercise, participants completed a web-based evaluation. High-priority latent safety threats were reported directly to the ED COVID-19 response team.

Selection of Participants

This study used a convenience sample of staff working in the ED. Staff was informed of the study during daily COVID-19 ED huddles. Because of very low ED patient volumes during the early days of the COVID-19 pandemic (∼25% of prepandemic daily patient volumes during the study period), onsite ED staff were invited and able to participate in these simulations. Of 78 unique staff members approached to participate, only 2 declined. Simulation sessions were scheduled around staff shift schedules and included days, nights, and weekends to provide enhanced opportunities for participation. Participation was voluntary, and participants consisted of staff from all ED disciplines, including physicians (only pediatric emergency attendings and fellows), nurses, ED technicians, respiratory therapists, care assistants, and pharmacists. Participating staff members assumed their actual clinical roles for each scenario.

Interventions

Simulation testing priorities

To set testing priorities, simulation staff met with key leadership from the emergency preparedness, infection control, critical care, anesthesia, resuscitation committee members, and ED divisions. Because of strict time limitations stemming from the urgent need to implement novel process changes during an ongoing pandemic, the team was able to design and implement the testing process in only 3 days. The priorities for scenario needs are outlined in Figures E1A and E1B (available online at http://www.annemergmed.com)

Scenario development

The scenarios were specifically designed to test any newly established guidelines regarding workflow, equipment needs, and care processes for patients encountered in both ED and ambulatory settings. Therefore, we convened a consensus panel to define the need for simulation-based testing. This panel consisted of 7 simulation staff members (this team included 4 certified simulation nurses, 1 simulation respiratory therapist, a simulation research director, and an ED attending/simulation provider). These members constituted the study team, responsible for simulation case design, facilitated debriefing observation tool development, technology set-up and operation, leading debriefing, video recording, and latent safety threats determination. Other members of this consensus panel included:

One infection control content expert (for consultation related the infection control measure related to the guidelines and Centers for Disease Control and Prevention recommendations)

Director of emergency response (case selection/design/testing priorities/reporting structure recommendations)

ED division chief (case selection/design/testing priorities/reporting)

ED nursing director (case selection/design/testing priorities/reporting)

Two ED nurse educators (case selection/design/testing priorities/reporting/and observers)

This group met to discuss testing priorities. These 5 cases were thought to be representative of the planned COVID-19 response process changes.

The team used a consensus approach through face-to-face discussions to develop several onsite scenarios to practice effective team communication from patient retrieval to initial triage and travel to the ED to intubation and eventual transfer of high-risk pediatric and adult patients with suspected COVID-19 out of the ED. Lastly, using dedicated observers during our simulated scenarios, we sought to uncover any latent safety threats that may have been present while conducting effective high severity clinical care for high-risk patients with presumed COVID-19.

Scenario design

To cover our objectives, our simulation team devised 4 onsite simulation-based cases: suspected COVID-19 in a stable infant that deteriorated once placed in a room, a hemodynamically unstable infant with known COVID-19, a stable adult patient with suspected COVID-19, and a critically ill adult patient with suspected COVID-19. A fifth scenario was added once cases of multisystem inflammatory syndrome in children began appearing in the literature. ,

We used Gaumard infant (Super Tory, Model S220, Newborn HAL, Model S3010) and adult mannequins (Victoria, Model S2200) for rapid deployment and removal in case the ED room in which the scenario took place was needed for a real patient. The patient’s vital signs were displayed on an electronic tablet running the SimMon application (Castle+Andersent ApS.2018), which was used to demonstrate changes in the patient’s condition. In addition, the technology operator included audible alarms to enhance emotional and environmental fidelity.

Each case deliberately included “tipping-points” in care when decisions were required that involved workflow changes, such as donning proper personal protective equipment prior to applying a nonrebreather oxygen mask for an acute fall in the oxygen saturation (Figures E1A and E1B, available online at http://www.annemergmed.com). To enhance realism, we used actual patient equipment, including pulse oximetry, ECG leads, blood pressure cuff, and peripherally inserted intravenous catheter connected to intravenous fluids. In addition, actual airway equipment was used, including the C-MAC for video laryngoscopy and an aerosol prevention intubation kit containing clear plastic drapes, an airway management checklist, and viral/bacterial filters for ventilation. Using authentic equipment in real ED patient rooms, we identified operational deficiencies and problems locating or acquiring the necessary equipment.

Scenario flow

Our simulation-based clinical systems tests were designed to take place onsite in the ED and last 60 to 90 minutes; Figure 1 , flow chart of COVID-19 testing simulation activities in the ED. Each simulation consisted of 5 phases: prebriefing, simulation case for testing, debrief, repeated case (up to 4 times), and finally, debrief and evaluation.

Figure 1

Flow chart of COVID-19 testing simulation activities in the ED. LST, latent safety threat; RRT, rapid response team; EARS, emergency ambulatory response system; PPE, personal protective equipment.

Simulation implementation

Simulations took place from March 29, 2020 to May 5, 2020, in the ED. Scenarios included physical retrieval of patients from outpatient settings as well as the transfer of patients in our ED’s respiratory isolation zone (created to isolate presumed cases of COVID-19) to ED resuscitation rooms and further transitions to either a general pediatric floor or the pediatric ICU. Unique to our institution is our ED Ambulatory Rapid Response Team. This team is responsible for the rapid assessment and transport of patients/parents/visitors to the ED from any noninpatient setting. These care transitions were deliberately included to test the process of transporting a patient with suspected COVID-19 through the hospital.

Before testing could begin each day, one study team member met with charge providers (physicians and nurses) to minimize disruption of the ongoing ED workflow and ensure there was available clinical space and staff to participate in the simulations. With participants and testing space designated, before each simulation, core team members, including content experts from relevant clinical groups, reviewed the most current Centers for Disease Control and Prevention-based hospital-issued COVID-19 practice guidelines and finalized the simulation plan, the case to be tested, and the testing priorities. , This included designating roles including simulated actors (parents of patients), a technology operator, videographer, observers, and facilitators. All simulations were time-stamped video recorded with consent from participating staff.

Each simulation session was designed to allow time for the case to be repeated in rapid cycles to consolidate the learning that occurred. If new guidelines were not followed during the initial cycles of a simulation, these were addressed in a rapid debrief, and the staff was encouraged to repeat the case to ensure adherence to protocols. If suggestions were made during debriefing related to missing equipment or improving communication, the staff were also permitted to repeat the scenario while implementing their suggestions. Every session involved at least 2 to 4 cycles depending on the participating staff’s needs and suggestions for improving clinical workflows.

After each simulation, participants evaluated this method of clinical system testing using a brief web-based survey that asked them to self-rate their knowledge of tested subjects (novice to expert), the use of this method of testing (strongly disagree to strongly agree), and suggestions for improvement (free text).

Debriefing Process and Observations

Debriefing development

For debriefing and logging compliance with participant actions, we adapted the observation tool we used in the operating room to include questions for debriefing that focused on the new ED system modifications and personal protective equipment usage aligned with COVID-19 clinical care guidelines. (Appendix E1, available online at http://www.annemergmed.com). With emergency preparedness and infection control input, we modified the Promoting Excellence And Reflective Learning in Simulation debriefing approach for the Systems Integration model. , This format was focused on the protocol steps to observe for omissions/workarounds taken by participants. Similar to the operating room study, debriefings occurred after each simulation round and tested the following 4 categories of issues: technology and equipment, isolation measures, job aids, and communication and personnel. If latent safety threats were identified during the simulation debriefing, participating staff were encouraged to provide input regarding potential solutions. When applicable, solutions could be implemented and trialed by participants when/if the case was repeated.

After each system test, participating staff were allowed to return to their clinical duties, whereas the core team and observers conducted postevent discussions to review latent safety threats data, which was then used to inform proposed clinical changes. These changes were prioritized based on the perceived risk from participating staff and expert observers before being communicated to the hospital COVID-19 response team and department leadership through circulated structured reports developed for this process.

High-risk priorities determined by the core team were quickly escalated through direct communication with hospital leadership. Each recommended change or action to system processes was vetted through infection control and emergency preparedness teams to obtain approval and limit unnecessary communication.

Measurements

Data Collection and Analysis by Outcome Measures

Latent Safety Threats:

To focus on compliance (staff errors and/or omissions) with new workflows and uncover potential unique latent safety threats, we modified the observation tool we used in the operating room for the ED. Three observers (CK, LE, and SW) underwent training sessions coordinated with scenario piloting. While observing each scenario, they specifically focused on identifying gaps/latent safety threats and, in debriefs, suggestions for remediation. These observers reviewed the recorded video and independently collected all participant comments, then each potential latent safety threat was entered into a secure database. Each of the 4 core investigators (CK, LE, MS, and SW) independently reviewed the complete list of latent safety threats using an inductive approach to ensure they were correctly categorized and established their list of unique latent safety threats. Next, to refine the list of unique latent safety threats and their frequency of occurrence, all 4 core investigators participated in a constant comparison session where results were compiled and reviewed, and we used consensus building to resolve any differences. Latent safety threats were then prioritized based on risk to patients and staff as previously described in debriefing development. High-priority latent safety threats, category, frequency, recommendations, and actions taken were tabulated.

Staff Evaluations Postsimulation Survey

To evaluate this simulation-based clinical systems test method, participants were asked to complete a short web-based survey (Qualtrics 2020) after completion. In addition, we revised the current simulation session evaluation to refocus on this method. This revised evaluation was piloted and revised based on input from staff of the disciplines represented. The survey was designed to evaluate the use, feasibility, and ability to manage airborne precautions based on this method using a 5-point Likert scale from 1=strongly disagree to 5=strongly agree, or 1=novice to 5=expert. Staff was also allowed to enter free-text comments related to the process. Complete survey is provided in Appendix E2 (available online at http://www.annemergmed.com).

Statistical Analysis

Frequencies (%) were calculated for each of the 4 latent safety threat categories. In addition, responses to Likert scale survey questions were analyzed using descriptive statistics. All statistical analyses were carried out with Statistical Package for the Social Sciences, IBM SPSS Statistics for Mac, version 25.0. (IBM Corp).

Results

Overall Scenarios Completed and Characteristics of Study Subjects

We conducted a total of 44 scenarios of onsite simulation-based clinical systems tests to test workflow and identify potential breaches in infection control measures, as well as to identify latent safety threats in our new COVID-19 ED and ED Ambulatory Rapid Response Team protocols. A total of 76 unique staff members participated in the clinical system tests, with 65 (86%) participants completing the posttraining evaluation. Table 1 lists the participants by job role and frequency in which they filled out the survey.

Table 1

Total participants by job role and evaluations completed N=76.

Clinical Role	Participants (% of Total)	Completed Evaluation	Total Staff Group Participants Who Completed Evaluation By Percent
Physician	22 (29)	19	86
Respiratory therapist	9 (12)	9	100
Nurse	30 (39)	24	80
Pharmacist	4 (5)	4	100
ED technician	9 (12)	8	89
Nurse practitioner	2 (3)	1	50
	76	65	86

Latent Safety Threats and Recommended Process Changes

From the 44 simulation sessions, staff identified 31 unique latent safety threats. The list of unique latent safety threats by category, rate of occurrence, and brief description are listed in Table 2 . The 31 unique latent safety threats were assigned to the categories with the following frequencies: job aids 9 (29%), isolation measures 8 (26%), communication and personnel 6 (19%), and technology and equipment 8 (26%). Eleven high-priority latent safety threats were escalated to leadership, and in most cases, actions were taken to address them within 2 days. The high-priority latent safety threats are listed in Table 3 , together with their categories, recommendations, and actions taken.

Table 2

Unique latent safety threats by category, number of occurrences, and abbreviated description.

Latent Safety Threat Category	Occurrences	Brief Description of Unique Latent Safety Threats
Job Aid (33)
	9	No bedside COVID-19 job aid for new intubation process
	8	Staff unsure of patient PPE during transport
	5	Lack of ED RRT job aid
	3	No transfer process for critical patient
	3	No use of adult patients needing transfer to an adult facility
	2	Transfer pathway maximized patient exposure
	2	Staff unsure of their own PPE during transport
	1	Staff unclear of an adult patient needing transfer to adult
Technology and equipment (18)
	5	Advanced respiratory airway equipment missing from the respiratory zone and negative pressure rooms
	5	Staff unaware of COVID-19 intubation supplies location
	2	Bacterial/viral filter missing from respiratory zone rooms
	1	Bacterial/viral filter missing from resuscitation room
	1	Pharmacy had no spot to mix medications
	1	Lack of adult medicine dose familiarity
	1	Pharmacy not physically present for resuscitation
	1	Lack of ACLS card
	1	Staff lacked knowledge of new negative pressure room functions
Communication and personnel (20)
	8	Difficulty hearing inside because of PPE
	5	Establishing Communication with pharmacy
	3	Difficulty hearing between providers inside and out
	2	Notification of needs PPE
	1	Gap in knowledge
	1	Clearing a path for transport
Isolation (32)
	8	Separating exposed and clean staff
	7	Comfort level
	5	Negative pressure and aerosol containment
	4	ED RRT process
	3	PPE team configuration around resuscitation
	2	Signage unavailable
	2	Delay in care and/or potential staff exposure
	1	Notification that PPE change is needed

ACLS, Advanced cardiac life support.

Table 3

High-priority latent safety threats by categories, recommendations, and actions taken.

Latent Safety Threat (Grouped by Categories with Description)	Recommendation to Leadership/Administration	Actions Taken
Communication:Limiting the number of staff inside the patient room impairs communication with the staff outside the room.	Facilitate communication by using cordless telephones with speakerphone/walkie-talkies with hands-free mode.Room telephone number placed outside the door	Speakerphone used at the head of the patient bed.Walkie-talkies deployed to the pharmacy as a lead person outside of the room
Communication:Staff in the patient room expressed difficulty hearing each while in full PPE	Limit staff entry to minimize noise levelEncourage staff to ask each other to speak louder if inaudibleReminders to speak louder while in PPE during daily staff huddles	Limit staff entry, and decreased noisePerform “sound/hearing checks,”Reminder posted on daily huddle board
Equipment missing:New COVID-19 guidelines restructured clinical workspaces leading to missing equipment and supplies in patient rooms.“Sick adult” medication dosing was missing from resuscitation rooms.	Airway resuscitation should be added to negative pressure isolation rooms· Provide copy of adult ACLS medication dosing to the ED pharmacist	Rooms restocked and retrieval equipment updated with locations labeled and reviewed with staff during daily shift huddlesEasy to identify COVID-19 intubation bags added to all resuscitation rooms and locations labeledPharmacists given ACLS card
Isolation measures:Changes to hospital-wide PPE guidelines resulted in a patient care delay because of uncertainty around required PPE for acutely deteriorating patients	Simplify PPE guidelines by requiring donning of airborne PPE plus N95 masks for all possible COVID-19 patients, trauma activations, and anticipated aerosol-generating proceduresPlace signage outside the room to indicate PPE required, time of last aerosol-generating procedures, and when aerosol should be cleared	ED leadership and infection control teams clarified PPE guidelines, particularly around acutely deteriorating patientsSimplified PPE recommendations circulated to all staff in daily huddles and COVID-19 daily email updatesClarifying PPE signage placed outside examination room doors
Isolation measures:Delay in patient care because of need for staff in patient rooms to doff and re-don PPE in compliance with current infection control practices when sending laboratory specimens or requiring additional supplies or equipment located outside the room	Have a “clean person” either stationed outside isolation rooms or immediately available by phone to get supplies or send lab specimensPlace room telephone number outside the door for those outside to call in When the room requires airborne precautions, a mini-huddle should take place to notify staff assigned to that clinical area	Recommended process was adopted, and information circulated to all staffSigns on room entrances created to designate airborne isolation rooms and display room telephone numbersProcess subsequently revised to say that if a patient is deteriorating that the “clean person” stays right outside the patient’s room
Isolation measures:Doors to airborne isolation rooms were being left open with loss of negative pressure and potential contamination of adjacent areas or exposure of staff outside the room	The designated “clean person” should also serve as a door monitor during resuscitations to minimize door opening and remind staff of PPE requirements prior to entering the room	This process was outlined, and the information circulated to all staff
Isolation measures:Staff uncertain of PPE requirements for patients and staff when transporting patients through the hospital	Simplify and distribute PPE guidelines for transporting patientsAt least one member of the transport team should remain in full PPE for transport if a patient intervention is anticipatedTransport team should perform a pretransport time-out to review job aids and discuss COVID-19 processes	Streamlined guidelines and job aids developed and posted to the hospital COVID-19 resource web page
Isolation measures:Staff unfamiliar with functioning of new negative pressure rooms	Install audible alarm when room door left open and negative pressure integrity is lostED staff should perform an on-shift review of the negative pressure room functionDoor monitor to limit door being left open	Audio alarm activation request forwarded to negative pressure room manufacturerNegative pressure room briefings were designed and conducted by ED nursing educatorsStaff assigned to be door monitors
Job aids:Staff unclear on how to move critically ill patients as existing hospital-wide patient transfer job aids only applied to noncritically ill patients not requiring an aerosol-generating procedures.	Develop clearer guidelines around moving critically ill patients	Specific and streamlined guidelines developed, tested, and refined through subsequent simulationsFinal guideline posted on the institutional COVID-19 web resource page and circulated to all staff as part of daily updates and during huddles
Job aids:Staff unclear about how hospital-wide COVID-19 measures changed the Emergency Ambulatory Rapid Response Team (ED RRT) process.	Clarify and provide staff education on updated ED RRT guidelines, including:If staff used Powered Air Purifying Respirator only (could not be successfully fit tested for N95), they should not be on the ED RRT team	A consensus call with ED RRT leadership resulted in the development of revised guidelines and job aid that were tested and refined in multiple subsequent simulationsFinal guideline posted on EARS pack, the institutional COVID-19 web resource page and circulated to all staff as a PowerPoint presentation and best practice video, and reiterated in daily updates and huddles
Job aids:Staff unfamiliar with newly developed COVID-19 intubation guidelines (ie, minimizing BVM, covering patient, inflating endotracheal tube cuff, video laryngoscopy use)	Create a streamlined COVID-19 intubation checklist and consolidate infection prevention intubation supplies into a “COVID intubation kit”Create a best practice video for review if intubation anticipatedHold additional practice sessions until at least 50% of the faculty was comfortable practicing new guidelines	An “Intubation of the Suspected COVID-19 Patient Checklist” was developed by multiple rounds of consensus building with expert staff, then refined through several rounds of testing and placed in new intubation kitsBest practice video created and link circulated to staffSimulations continued in the ED until at least 50% of faculty/fellows were trained
Job aids:Knowledge gaps of COVID-unrelated standard care processes identified, including epinephrine dosing in the shock patient and PPE availability among ED RRT supplies	Ensure ED staff are aware of epinephrine dosing in shock patientsReview available and uncommonly used equipment on the ED RRT gurney with team members	ED pharmacist clarified Epi dosing with all staffPPE location in ED RRT supplies was added to ED RRT job aid and reviewed in subsequent simulations

BVM, bag-valve-mask.

More common latent safety threats identified by staff include the following: confusion regarding the new intubation process and lack of a quick reference with the required steps (job aids), confusion around the timing and types of personal protective equipment required when entering a patient’s room (isolation measures), necessary supplies not usually stocked in patient rooms required staff to doff personal protective equipment to retrieve it (technology and equipment), and too many staff in the patient room during a resuscitation made communication difficult, especially in full personal protective equipment (communication and personnel).

Staff Evaluations

Eighty-five percent (65/76) of participating staff evaluated this method for usability, feasibility for learning, and testing the system. The survey questions and descriptive statistics results are reported in Table 4 . Overall, staff strongly agreed that this method was: worth the time it took (86%), an acceptable way of improving staff readiness/knowledge (92%), and an effective way to test changes/provide solutions (92%). Most participants also strongly agreed that the debriefing process allowed staff to share ideas (85%). Interestingly, when asked about which types of personal protective equipment to use in different situations, 92% of staff rated themselves as proficient or expert.

Table 4

Descriptive statistics summarizing ED responses to posttraining evaluation.

Item (N=65)	Mean	SD	Median	Min	Max
Each question in this block began with “This sim-based approach was…” (These questions used a 5-point Likert scale 1=Strongly Disagree to 5=Strongly Agree)
1. Worth the time it took	4.8	0.44	5	3	5
2. An acceptable way to improve system readiness and staff knowledge	4.9	0.43	5	3	5
3. An effective way to test changes and provide solutions	4.9	0.39	5	3	5
4. The debriefing process allowed staff to share ideas for improvement	4.8	0.47	5	3	5
5. Improved our team functioning	4.3	0.74	4	3	5
Each question in this block used a 5-point Likert scale (1=Novice to 5=Expert)
1. I know when to use which types of PPE to use in different situations	4.1	0.70	4	3	5
2. I know how my clinical practice has changed due to COVID-19 cases in my unit.	4.2	0.65	4	3	5

Max, maximum; min, minimum; SD, standard deviation.

Limitations

There were several limitations to this study. We recognize that job aids and checklists need to be optimized for deployment in the clinical setting with methods such as those from human factors engineering, and this did not occur. The level of rigor of this systems test was limited to specifically uncovering deviations from the newly developed protocols. Implementing a project of this scale may be difficult to conduct in the face of other operational priorities and competing interests that require time and resources. The need for a considerable time, simulation expertise, a strong existing relationship between ED staff and simulation team, and resources may limit the feasibility and generalizability of carrying out this type of testing effectively. Rapidly implemented simulation cannot uncover or resolve all the challenges that arise during preparation for a disaster such as a pandemic but can discover problems that less intensive means of preparation do not address. Although administrative planning involves the conceptualization of work, this exercise is often ineffective in predicting all of the complexities that will occur when taking care of patients.11, 12, 13

There are other additional limitations, which are critically important. We were not able to establish the impact on clinical outcomes from our intervention. We have no counterfactual comparator, so it is unclear to what extent our findings actually represent improvements in patient care or staff protection.

Because of significant time constraints, the simulation sessions did not include full-scale testing to include the many potential situations such as actual patient transfer to and from the adult EDs or involving ambulance crews. We were also not able to train all ED staff, because not all staff were present or able to participate during the scheduled simulation dates. Although the attending and fellow physicians trained approached 50% of all faculty, the numbers are limited, so extrapolating this method for much larger numbers of providers remains unclear.

Discussion

In preparation for a potential surge of patients from a looming highly contagious infectious agent such as during the COVID-19 pandemic, our urban tertiary care children’s hospital quickly made many changes to our existing processes of care to be consistent with Centers for Disease Control and Prevention recommendations. Most of these changes were based on limiting exposure to staff and patients, but they affected all aspects of ED care. Decisions such as when and which type of personal protective equipment to wear as well as limiting aerosol spread challenged emergency preparedness and ED staff. Compounding the problem was limited personal protective equipment mandating strict stewardship of isolation materials. By rapidly developing and deploying the simulation team in the ED, we found that, similar to other authors, onsite simulation is a good way to prepare for disease outbreaks.45, 46, 47, 48

In contrast to our operating room COVID-19 simulation-based clinical systems test, in which we trained only 2 anesthesia staff members at a time with a single scenario, the ED is a much more complex system requiring testing with 5 scenarios, a larger number of providers at a time, and more detailed observation. In both cases, our novel method combined rapid-cycle simulation training with a simulation-based clinical systems testing approach to define latent safety threats and develop responses. During this outbreak, the onsite simulation process was modified to take on a systems-based testing perspective. In the ED, this approach allowed for an evaluation of the many department-specific changes. Examples of the changes to the ED systems included: altering the response for the ED Ambulatory Rapid Response Team to retrieve children/adults, changing ED patient flow to separate patients into a respiratory zone, adding negative pressure rooms, and keeping more staff outside the room during care escalation or resuscitation. Based on the staff perceptions from the postdebriefing surveys, this method was highly rated and worth the time it took. This would indicate that this method could/should be strongly considered whenever a highly complex degree of change is being developed. In addition, the rapid turnaround time and development of testing methods were highly valued by the hospital leadership COVID-19 response team.

These exercises enabled staff to identify and make suggestions to address unexpected problems that were not obvious during the formulation of the protocols and allowed for changes to be made without risk to the patient or infectious risk to the staff. Furthermore, efforts were made to mitigate the identified latent safety threats. Some examples of these efforts include: placing donning and doffing posters outside of rooms and adding an intubation job aid on a laminated card on the video laryngoscopy cart, which helped the staff minimize missed steps. In addition, multiple attempts were made to address the communication inside and outside the room during resuscitations. Included in these attempts were having staff outside the room call the telephone inside the room as well as using walkie-talkies to facilitate communication. This enhanced communication provided a means for staff inside the room to access additional supplies, which minimized the risk of staff exposure from entering the resuscitation bay too early, minimized personal protective equipment usage, and limited the unnecessary contamination of equipment.

Before this method can be widely implemented for any/every protocol being developed, several considerations must be addressed. To assist with this line of thinking, we have outlined considerations and challenges faced by our staff that had to be overcome and how this method could be adapted for other potential applications. These are listed in Figure 2 , considerations for successful simulation-based clinical systems test implementation in settings other than COVID-19. First, among these considerations is that our ability to deploy this method rapidly was built on the distinguished relationship between ED leadership and simulation staff.

Figure 2

Considerations for successful simulation-based clinical systems testing implementation in settings in other than COVID-19.

This simulation-based approach allowed staff to experience the proposed workflow changes, improved the ability to train ED staff in interprofessional teams, and provided a comprehensive understanding of potential latent safety threats from multiple perspectives. The rapid deployment of this testing was due primarily to all elective procedures and subspecialty visits being canceled as well as an approximately 75% decrease in ED patient volumes so that both staff and ED space were widely available. In addition, the ED staff’s willingness to fully participate and engage in the exercise as adult learners and share their insights greatly enhanced the process. Although personal protective equipment limitations were significant, the simulation staff managed to find workarounds, including recycling face shields, using washable gowns, and staff bringing their already assigned N95 masks. The net effect was that simulations did not compromise the hospital’s limited personal protective equipment supplies.

In conclusion, the simulation-based clinical systems tests of our COVID-19 ED care processes combined with rapid deployment, targeted repetition, standardized feedback, and debriefing allowed us to rapidly identify and propose potential solutions to problems not foreseen during the protocol planning stages. Overall, 31 unique latent safety threats were identified, and mitigation strategies were developed. The simulation-based clinical systems test method improved participant familiarity with the clinical protocols and increased their confidence level in completing the clinical tasks using these new processes. Combining the simulation-based clinical systems test methodology with rapid-cycle deliberate practice of onsite simulations for each group of participating staff further augmented the ability to test and improve suggestions for remediation of latent safety threats identified during the new COVID-19 protocol testing. This method created open lines of communication with leadership and the creation of high-priority suggestions for improvement. This method should be considered to train staff and evaluate the implementation of new protocols in a short time.