Does utilization of an intubation safety checklist reduce omissions during simulated resuscitation scenarios: a multi-center randomized controlled trial.

OBJECTIVES: Checklists have been used to decrease adverse events associated with medical procedures. Simulation provides a safe setting in which to evaluate a new checklist. The objective of this study was to determine if the use of a novel peri-intubation checklist would decrease practitioners' rates of omission of tasks during simulated airway management scenarios.
METHODS: Fifty-four emergency medicine (EM) practitioners from two academic centers were randomized to either their usual approach or use of our checklist, then completed three simulated airway management scenarios. A minimum of two assessors documented the number of tasks omitted and the time until definitive airway management. Discrepancies between assessors were resolved by single assessor video review. Participants also completed a post-simulation survey.
RESULTS: The average percentage of omitted tasks over three scenarios was 45.7% in the control group (n = 25) and 13.5% in the checklist group (n = 29)-an absolute difference of 32.2% (95% CI 27.8, 36.6%). Time to definitive airway management was longer in the checklist group in the first two of three scenarios (difference of 110.0 s, 95% CI 55.0 to 167.0; 83.0 s, 95% CI 35.0 to 128.0; and 36.0 s, 95% CI -18.0 to 98.0 respectively).
CONCLUSIONS: In this dual-center, randomized controlled trial, use of an airway checklist in a simulated setting significantly decreased the number of important airway tasks omitted by EM practitioners, but increased time to definitive airway management.

RCT Entities:   Population Interventions Outcomes

Clinician’s capsule

Introduction

Over the last decade, there has been substantial interest in the use of patient safety checklists to mitigate risk of adverse events in healthcare settings [1-14]. Checklists became popular following Berenholtz’s study demonstrating a decrease in catheter-associated infections with the use of a checklist for central line insertion in the intensive care unit (ICU) [15]. Subsequently, the World Health Organization’s Surgical Safety Checklist study reported a 36% average reduction in postoperative complications and reduced mortality across all participating sites [9]. Marked interest and widespread application of procedural checklists, including airway management checklists, has followed.

Many adult patients requiring acute airway management in the emergency department (ED) undergo rapid sequence intubation, which is considered a high-risk event [16]—particularly when performed emergently for unstable patients. In the ICU, it has been reported that 28% of intubations are associated with serious complications, including hypoxemia, hemodynamic collapse, and cardiac arrest [17]. Given the risks associated with intubation, patient safety checklists have been considered for airway management. However, the majority of research in this field consists of observational studies in operating theatres [8, 18], or before-after studies that focus on specific patient populations such as the ICU [7], pediatric ED [19], or trauma patients [13]. Despite heightened interest in pre-intubation checklists, few randomized controlled trials (RCTs) have been performed [10, 20], and none focused on adult ED intubations.

This study aimed to determine if there was a difference in the proportion of omitted tasks observed during simulated airway management scenarios when emergency medicine (EM) practitioners utilized a new intubation safety checklist, versus their usual resuscitation practice. Since a novel process improvement tool implemented directly into the clinical setting has the potential to delay definitive airway management and cause harm, theatre-based simulation scenarios were considered most appropriate to evaluate the checklist without compromising patient safety. It was hypothesized that checklist use would decrease the rate of omitted tasks surrounding intubation without negatively impacting time to definitive airway management.

Methods

Study design

This was a dual-center RCT of a novel peri-intubation checklist utilized by EM practitioners in a theatre-based simulation environment. All EM physicians and residents from two Ontario centers—London Health Science Centre and University Health Network (UHN) in Toronto—working more than 20 h per month in any ED, were invited to participate via email. The study was approved by both institutions’ research ethics boards (REB): Health Sciences REB at Western University, and UHN REB.

Checklist and assessment tool development

A novel airway checklist was created for this study because the investigators were not aware of any validated peri-intubation checklists in existence. Additionally, after trialing several existing EM airway checklists in a simulation theatre, the investigators found these tools challenging and not suited to the local ED setting. Thus, existing checklists from published and grey literature [21-24] were modified using iterative revisions by experienced EM practitioners and local EM airway experts to achieve consensus on the final 29-item checklist consisting of pre- and post-intubation sections. The checklist was designed to be comprehensive, yet intuitive for practitioners of all experience levels (Fig. 1). The assessment tools, which included overlapping sets of expected actions (some case specific) and tasks from the checklist deemed important by investigators, also underwent iterative revisions to achieve consensus (Appendix 1).

Fig. 1

Peri-intubation checklist

Participants were randomly assigned to the checklist (intervention) or usual practice (control) group. Block randomization was performed using a computer-based random number generator. Participants and investigators were blinded to participants’ group allocation until a sealed envelope was opened upon participant arrival at their simulation session. Participants provided signed consent after reading the letter of information, which disclosed that this was an RCT of an intubation checklist. The intervention group was given the checklist, then shown a video demonstrating the simulation environment, and the checklist being read in a “do-confirm” fashion by the nurse pre-intubation [21]. Control group participants watched a video of the same scenario and actions, without a checklist. All participants then completed three, 10-min simulation cases. Copies of the checklist were left in the simulation room and participants were instructed to avoid sharing study specific details with colleagues.

Simulated scenarios included: (1) benzodiazepine overdose causing respiratory failure, (2) chronic obstructive pulmonary disease exacerbation failing non-invasive ventilation, and (3) angioedema secondary to anaphylaxis (Appendix 2). Cases were performed in this order, with the first two cases requiring endotracheal intubation and the third necessitating cricothyrotomy. Participants were accompanied by two trained research personnel acting as a respiratory therapist and a registered nurse. Allied health workers were instructed to only perform tasks when directed. Though they could prompt participants to use the checklist, they would only read it out loud if explicitly directed to do so.

Post-simulation, a study investigator provided an optional debrief. Participants then completed a survey comprised of multiple choice, Likert, rank-list and open-ended questions. Survey questions included: 12 questions regarding demographic characteristics and current practice; 13 questions regarding the utility of simulation; and 1 question about presumed airway checklist utility (control group), or 10 questions concerning the utility of the provided peri-intubation checklist (intervention group).

Outcomes

The primary outcome was the proportion of omissions observed during three simulated scenarios. Secondary outcomes included time to definitive airway management, and EM practitioners’ perceptions of the checklist and the utility of simulation. Definitive airway management was defined as successful insertion of an endotracheal tube or cricothyrotomy catheter.

Data collection

At least two study investigators observed each simulated scenario and concurrently completed the assessment tool. Tasks were considered ‘omitted’ if they were neither verbalized by the participant, nor observed being performed by the assessors. All scenarios were video recorded. A single study investigator reviewed videos for discrepancies between assessments. Data were entered directly into a study-specific Microsoft Excel database (Microsoft Corporation, Redmond, WA.).

Sample size

Primary analysis tested the null hypothesis that there was no difference in the proportion of errors with checklist use compared to usual care. A sample size of 146 participants (73 per group) was calculated to detect a difference of 15% between groups in error rate (estimated 20% omission rate in the control group and 5% in the checklist group), using a two-sided Z test of the difference between proportions with 80% power and a 5% alpha [25]. Estimated omission rates were based on findings of previous studies [8, 18].

Due to substantial challenges with participant recruitment the study was ended early. After 48 participants were recruited an unplanned, interim analysis was performed using a difference in omission rates of 30%. This calculation yielded a sample size of 54 participants. Thus, the study ended early at 54 participants.

Data analysis

Data analyses on SPSS 21.0 (IBM Corporation) software followed the intention-to-treat principle. Standard descriptive statistics were summarized using means and standard deviations. Time was reported as medians and interquartile ranges. Differences in median time to definitive airway management were calculated using the Independent-Samples Median Test with Hodges–Lehman estimate with a 95% confidence interval. Differences in proportions were assessed by Pearson chi-squared statistics and continuous data were analyzed using independent t tests, where appropriate.

Results

A total of 54 EM practitioners at two academic centers participated in this study. Participant characteristics were similar between control (n = 25) and intervention (n = 29) groups. Most participants were EM residents or recent graduates (n = 28, 51.9% and n = 13, 24.1% respectively) (Table 1).

Table 1

Participant characteristics

Participant characteristics	Control (n = 25)	Intervention (n = 29)
Mean (SD) age (years)	35 (11)	37 (11)
Male, n (%)	16 (64)	21 (72)
Years of practice, n (%)
Resident	15 (48)	13 (45)
< 10 years	6 (24)	7 (24)
10–20 years	2 (8)	3 (10)
> 20 years	2 (8)	6 (21)
Frequency of intubation, n (%)
1/month	3 (12)	2 (7)
1–4/month	9 (36)	14 (48)
6–12/years	8 (32)	7 (24)
5 or less/year	5 (20)	5 (17)
> 1 year since	0 (0)	1 (3)
Frequency of surgical airway, n (%)
6–12/year	0 (0)	0 (0)
1–5/year	0 (0)	1 (3)
> 1 year since	4 (16)	8 (28)
Never	20 (80)	20 (69)
Blank	1 (4)	0 (0)
Do you regularly participate in simulation scenarios to practice/refine previously learned skills? n (%)
Yes, regularly (1/month)	8 (32)	7 (24)
Often (1/3–4 months)	6 (24)	4 (14)
Sometimes (1–2/year)	7 (28)	13 (45)
Never	4 (16)	5 (17)

The percentage of omitted airway management tasks was significantly higher in the control group compared to the checklist group. Overall, the control group failed to perform 45.7% of tasks deemed important in airway management compared to 13.5% in the intervention group (Δ32.2%; 95% CI 27.8 to 36.6%). These results were consistent for each of the three scenarios (Table 2).

Table 2

Mean percentage of omitted items per participant

	Control (% omitted, SD)	Intervention (% omitted, SD)	Absolute difference (95% CI)	p value
Scenario 1	43.6 (12.5)	12.5 (10.6)	31.0 (24.7 to 37.3)	< 0.0001
Scenario 2	46.3 (11.0)	13.6 (7.9)	32.7 (27.6 to 37.9)	< 0.0001
Scenario 3	47.2 (8.9)	15.3 (9.5)	31.8 (26.8 to 36.9)	< 0.0001
Overall	45.7 (9.1)	13.5 (7.0)	32.2 (27.8 to 36.6)	< 0.0001

Time to definitive airway management was longer in the checklist group for the first two scenarios, but no statistical difference was detected in the third scenario where cricothyrotomy was required (Table 3).

Table 3

Median time to definitive airway (in s)

	Control (s, IQR)	Intervention (s, IQR)	Absolute difference (95% CI)	p value
Scenario 1	260.0 (186.5–339.5)	365.0 (308.0–339.5)	110.0 (55.0 to 167.0)	0.001
Scenario 2	306.0 (252.0–374.5)	398.0 (351.0–445.0)	83.0 (35.0 to 128.0)	0.03
Scenario 3	400.0 (321.5–476.5)	424.0 (371.0–507.0)	36.0 (− 18.0 to 98.0)	0.28

The most frequently omitted tasks in both groups included use of positive end expiratory pressure valve for bag-valve-mask ventilation, and identification of medical reasons for difficult intubation, allergies and past medical history. Tasks most consistently performed included consideration of sedative medication, presence of endotracheal tubes and adjunct device, and confirmation of tube placement by auscultation and capnography. All tasks, except identification of paralytic (including consideration of contraindications), were performed with greater frequency in the checklist group (Table 4).

Table 4

Frequency of airway management task omission (%)

Variables	Control (% omitted)	Intervention (% omitted)	Absolute difference
Identify operator's assistant	92	14.9	77.1
Identify cricothyroid landmarks	73.3	6.9	66.4
Identify next call for help	72	5.7	66.3
Any allergies noted	89.3	24.1	65.2
Identify medical reasons for difficult intubation	94.7	32.2	62.5
PPE applied	74.7	12.6	62
Consideration of post-intubation sedation	61.3	1.2	60.2
Verbalize plan for failure	68	11.5	56.5
Depth of ET tube confirmed	61.3	8	53.3
Oral and/or nasal airway available	64	12.6	51.4
BiPap connected to machine/oxygen	56	5.2	50.8
Patient head and body positioned appropriately	54.7	5.7	48.9
Identify mechanical reasons for difficult intubation	44	5.7	38.3
Pertinent medical history noted	68	29.9	38.1
Verbalize initial plan	44	6.9	37.1
Verbalize backup plan	37.3	1.1	36.2
Suction available and functioning	40	5.7	34.3
Cuff inflated	40	5.7	34.3
Identify reasons for difficult BVM ventilation	62	31	31
Monitors on patient and functioning	37.3	8	29.3
BVM connected to PEEP	80	51.7	28.3
Bed appropriately prepared	33.3	5.7	27.6
Diagnostic imaging ordered to confirm placement	26.7	3.4	23.2
IV fluids running well	32	9.2	22.8
Tube secured	61.3	39.1	22.3
Rescue device available	36	17.2	18.8
Patient optimally pre-oxygenated	20	5.7	14.3
Verbalized indication for intubation	20	6.9	13.1
Tube placement confirmed (capnography)	13.3	1.2	12.2
Adjunct device available	13.3	1.1	12.2
Air entry confirmed	10.7	0	10.7
Endotracheal tubes with stylet and 10 cc syringe	2.7	0	2.7
Laryngoscope present and light functioning	4	2.3	1.7
Sedative medication identified (or considered)	0	0	0
Paralytic identified (contraindications considered)	21.3	41.4	-20

Participants were observed for completion of thirty-three airway management tasks deemed important by study investigators. Results from the three scenarios were combined to determine how often each task was omitted in each group (mean %). Tasks are listed in order of absolute difference in performance frequency between control and intervention groups

ET endotracheal tube, PPE personal protective equipment, PEEP positive end expiratory pressure, BVM bag-valve-mask, IV intravenous

In the post-simulation survey (Appendix 3), most control group participants (n = 24, 96.0%) indicated an airway checklist would have been helpful for the scenarios. Amongst intervention group participants, 82.8% were satisfied with the peri-intubation checklist and indicated it was helpful for equipment preparation (n = 27, 93.1%), patient preparation (n = 26, 89.6%), and post-intubation care (n = 24, 82.8%). They believed that checklist use would reduce errors during intubation (n = 27, 93.1%), and that the simulated scenarios were beneficial for checklist adoption (n = 28, 96.6%). Conversely, they thought the checklist delayed definitive airway management, and was not helpful for airway assessment, medication selection, or choosing to perform a cricothyroidotomy.

Discussion

Interpretation

In this RCT, use of a novel peri-intubation checklist significantly decreased the rate of omissions during airway management in a simulated setting from 45.7% to 13.5%; however, there was an initial increase in time to intubation with checklist use. Participants believed that peri-intubation checklists are helpful and improve patient safety, and that simulation is a good modality for learning how to use checklists.

Prior studies

This study is a unique RCT looking at the use of a checklist for adult ED patients with three different, non-traumatic, indications for intubation. The results of this study are consistent with previous studies in operating rooms, ICUs and pediatric EDs that have shown a decrease in omissions of intubation equipment with the use of an intubation checklist [8, 18, 20]. These consistent results are promising since several studies have demonstrated improved patient outcomes with use of protocols that include pre-procedure and intubation checklists [7, 19]. In particular, Smith et al., demonstrated a reduction in complications from 9.2 to 1.5% with use of a pre-procedural checklist [13], and Fogg et al. demonstrated an increase in first pass intubations (83–93%), and a drop in complications (29–19%) and desaturation events (16–11%) with implementation of a safe operating procedure including an intubation checklist [26].

While most studies, including this RCT, favor the use of pre-intubation checklists, two recent studies have shown contrary results. A single-center before-after review of trauma patients intubated in the ED with the aid of a pre-intubation checklist resulted in no difference in total intubation attempts, hemodynamic stability, ventilator days, length of stay, or mortality [5]. Similarly, an RCT by Janz et al., showed no significant difference in lowest median systolic blood pressure or oxygen saturation with the use of a pre-intubation checklist in the ICU versus usual care [10]. These studies had small sample sizes and their total adverse event rates were low. Consequently, they may have failed to capture the benefit of a checklist in more complex cases or with less experienced operators.

Implications

This study provides unique insights into checklist implementation and the role of simulation. Notably, the study population consisted largely of residents and recent graduates who likely had a prior interest in simulation (85.2% had previously participated in simulation activities). It must be considered that the difference in omissions between groups may have been augmented by the fact that less experienced practitioners may benefit more from a comprehensive checklist. Although it is unclear if this decrease in omissions would improve patient safety, this correlation does suggest that there may be a role for the checklist as a training tool for airway management.

Interestingly, practitioners employed the checklist in a variety of different ways. Some intervention group participants abandoned the checklist mid-scenario and one participant in the control group used their own checklist as this was a part of their usual practice. This may have affected the results since data were analyzed using the intention-to-treat principle. Furthermore, despite the introductory video demonstrating a “do-confirm” technique for checklist use, several participants used the checklist as a “read-do” tool [21]. Failure to use the checklist in a “do-confirm” manner may have increased time to definitive airway management. Most importantly, the inconsistent manner in which participants utilized the checklist, highlights the importance of training to ensure proper checklist use prior to clinical implementation. Simulation could play an important role in such training activities.

Delay to airway management is commonly discussed as a possible disadvantage of airway checklist use. Interestingly, even in the scenario with the greatest difference in time between groups, use of the checklist only increased time to definitive airway management by 110 s—the clinical significance of which is unclear. Furthermore, this increase in time to intubation was only present in the first two scenarios. In the third case, there was no difference in time to definitive airway management between groups. Possible explanations for this include: (1) as participants became more familiar with the checklist their efficiency improved (noting that the time difference between groups decreased by 30–50 s between scenarios); (2) the checklist appropriately prepared practitioners for a failed airway; or (3) practitioners rushed though the checklist given the urgency of impending airway in the third scenario. Ultimately, this small, transient effect of checklist use on time to definitive airway management should not detract from trialing a checklist in a clinical setting.

Limitations

This study had several important limitations. First, recruiting volunteers was challenging and resulted in the study ending early due to lack of feasibility. It is acknowledged that results may have been different had the study reached the original sample size. Second, the inability to use a previously validated checklist and assessment tool limits the generalizability of these results. Third, use of theatre-based simulation both prevented the assessment of patient-centered outcomes, and likely impacted performance due to reduced realism. Some tasks, which are almost instinctive in a clinical setting (e.g. securing the endotracheal tube post intubation), were frequently missed, particularly in the control group. This may have overestimated the benefits of checklists. Additionally, a lack of comfort with simulation and verbalization of actions may have caused some participants to seemingly ‘omit tasks’ such as ensuring monitors were functioning or identifying reasons for difficult intubation. These tasks may have been performed mentally by the participant but would only be assessed as ‘complete’ if verbalized or physically observed.

Although the use of a simulated setting may have impacted study results and limits generalizability to a clinical setting, the survey results indicate that simulation may be an effective modality to train practitioners to use checklists prior to clinical application. User satisfaction is crucial to the successful implementation of a new clinical tool. Overall, participants were pleased with the checklist. Many did comment that it was long, however, a 29-item checklist is common in the literature [18, 22–24]. Despite checklist length, participants generally had positive opinions regarding the utility of the airway checklist.

Conclusion

Overall, in a simulated setting, use of a peri-intubation checklist significantly decreased the omission rate of airway management tasks, however, time to definitive airway management was increased. These findings support the notion that checklist use in the ED could decrease errors of omission, thereby potentially decreasing adverse events during airway management.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 670 kb)

What is known about this topic?

Airway management in the ED is a high risk event, and checklists are known to improve safety during medical procedures.

What did this study ask?

Does use of a peri-intubation checklist decrease the number of omitted tasks during airway management in simulated emergency medicine scenarios?

What did this study find?

This multi-centre randomized controlled trial found an average absolute decrease in omitted tasks of 32.2% during three simulation scenarios.

Why does this study matter to clinicians?

Peri-intubation checklist use in the ED would result in fewer errors of omission and may decrease adverse events during intubation.