Sustained Improvement in the Performance of Rapid Sequence Intubation Five Years after a Quality Improvement Initiative.

Many quality improvement interventions do not lead to sustained improvement, and the sustainability of healthcare interventions remains understudied. We conducted a time-series analysis to determine whether improvements in the safety of rapid sequence intubation (RSI) in our academic pediatric emergency department were sustained 5 years after a quality improvement initiative.
METHODS: There were 3 study periods: baseline (April 2009-March 2010), improvement (July 2012-December 2013), and operational (January 2014-December 2018). All patients undergoing RSI were eligible. We collected data using a structured video review. We compared key processes and outcomes with statistical process control charts.
RESULTS: We collected data for 615 of 643 (96%) patient encounters with RSI performed: 114 baseline (12 months), 105 improvement (18 months), and 396 operational (60 months). Key characteristics were similar, including patient age. Statistical process control charts indicated sustained improvement of all 6 key processes and the primary outcome measure (oxyhemoglobin desaturation) throughout the 5-year operational period.
CONCLUSIONS: Improvements in RSI safety were sustained 5 years after a successful improvement initiative, with further improvement seen in several key processes. Further research is needed to elucidate the factors contributing to sustainability.

INTRODUCTION

A common criticism of quality improvement (QI) initiatives is the lack of evidence that improvement is sustained after the project ends. Real advances in healthcare delivery are only possible if change efforts are both initially effective and ultimately sustained. If healthcare systems do not sustain successful change efforts, they waste resources, and clinicians will be less motivated to participate in future efforts.

Our knowledge of whether and how improvements in healthcare delivery are sustained is limited. Most published studies lack an explicit, standard definition of sustainability and often fail to distinguish between sustainability processes and outcomes and sustainability capacity.[1] Many studies either rely on self-reporting or fail to measure outcomes, and rigorous assessment of adherence to the intervention is uncommon.[2] Few published studies had rigorous methodology, and of these studies few reported improvements were sustained more than 2 years after implementation.[2] To justify the resources involved and to inform the design of future interventions, QI researchers must address this gap in our knowledge. We must improve our understanding of both whether improvements are sustained and what drives sustained improvement.

In December 2013, we completed the active improvement phase of a successful QI initiative to improve the safety of rapid sequence intubation (RSI) in a pediatric emergency department (PED).[3] Over 18 months, after introducing a bundle of interventions, the performance and safety of RSI improved. We did not investigate whether these improvements were sustained after initial improvement. The current study’s objective was to address part of the above knowledge gap by determining whether improvements in RSI safety in a high-volume, academic PED were sustained five years after the improvement initiative ended.

METHODS

Design

We completed a time-series study of key RSI processes and outcomes. There were 3 study periods: baseline (April 2009 through March 2010; 12 months), improvement (July 2012 through December 2013; 18 months), and operational (January 2014 through December 2018; 60 months). The baseline period represents our original description of the safety and performance of RSI in our PED.[4] We include in this report only methodology relevant to the current study; we have provided more detailed descriptions of our data collection methods and definitions in previous publications.[3,4] Our institutional review board reviewed the current study before commencement and determined it to be non-human subject research. We composed the current report to be consistent with the SQUIRE 2.0 guideline.[5]

Setting

The study setting was the resuscitation area of a high-volume academic PED. The parent institution is a level I trauma center, with more than 90,000 annual PED visits. The institution’s James M. Anderson Center for Health Systems Excellence provides a mature infrastructure for QI. Improvement science is an established part of both the hospital and the emergency department (ED) culture, and QI consultants and analysts assist with many ED projects. Many ED faculty have advanced training in improvement science and have led QI initiatives. ED faculty and staff participated in numerous quality initiatives before and after the RSI improvement project.

Operational Period

Our original RSI improvement initiative’s primary aim was to reduce the frequency of oxyhemoglobin desaturation during RSI. Oxyhemoglobin desaturation causes both direct injury, exacerbates existing disease, and increases the risk of bradycardia and pulseless arrest. Our original theory of improvement was that interventions designed to optimize patient preparation, minimize the duration of apnea, and facilitate early recognition of failing intubation attempts would reduce the frequency of oxyhemoglobin desaturation during RSI.[3] We believed these interventions would also improve the performance of RSI in general and reduce the frequency of other adverse events, like esophageal intubation. However, this was not the primary aim of the project.

We based our theory primarily on studies of RSI in our ED.[4,6] Both studies were video-based investigations and included the same 114 patients undergoing RSI in our pediatric ED over 12 months. In the first study, we found a high proportion of attempt failure, with one-quarter of patients having 3 or more attempts before successful intubation. Sixty percent of patients had at least one adverse event. A third of the patients experienced oxyhemoglobin desaturation during RSI. Most of the desaturation episodes were not minor, with most patients with data available dropping to less than 80%. We found problems with all aspects of the RSI process, from medication selection and administration to the confirmatory process.

In the second study, prolonged attempts were also common, with one-quarter of all attempts longer than 55 seconds, and attempt duration was independently associated with oxyhemoglobin desaturation.[6] Younger patient age and esophageal intubation were also independently associated with desaturation.

In July 2012, we began testing a bundle of 4 interventions: (1) an RSI checklist, (2) checklist execution by a second physician, (3) use of a video laryngoscope for all first attempts (Storz CMAC; Karl Storz SE & Co. KG, Tuttlingen, Germany), and (4) performance of attempts by approved providers only (essentially excluding pediatric residents). Before the initiative, our ED did not have a standard RSI protocol. Employing the video-based data collection methods used during our initial studies, we tracked the performance of 6 key process measures (see below) and our primary outcome, oxyhemoglobin desaturation during RSI (<90%). We sent a structured feedback form (see Appendix A, which displays rapid sequence intubation checklist, ) to the following care team members: lead physician and nurse, respiratory therapist, bedside nurses, pediatric resident, and anesthesiology/subspecialist involved in airway management (if present).

In December 2013, the multidisciplinary improvement team disbanded, and ED leadership determined the RSI interventions to be the standard of care. All team members continued to work clinically in the ED. The physician project leader and a research coordinator maintained the following activities throughout the operational period: case identification, video-based data collection, and feedback form completion. We did not substantially alter the interventions during the operational period. The physician project leader occasionally revised the checklist wording (see Appendix B, ). Since 2015, we have approved a small number of pediatric residents to intubate in the ED, after completing additional training with the project leader.

Subjects and Sampling

Only patients undergoing RSI were eligible for the time-series analysis. Patients undergoing other forms of tracheal intubation were not eligible; we specifically excluded “crash” or no-medication intubations for patients in cardiac arrest. We made no other exclusions based on patient age or other patient/process characteristics. Our methods of patient identification/sampling were identical to those of the previous work.[3,4,7] We used daily reports automatically generated from the electronic health record, followed by a manual review of the electronic and video records to confirm that the care team performed RSI. Each of the 4 resuscitation bays in our ED is equipped with an audiovisual recording system, which records continuously. Recordings are available for review using a proprietary software program (LiveCapture, BLine Medical, Washington, D.C.). We have previously identified nearly all eligible cases of RSI using the same approach.[4] RSI was defined as the successive administration of a neuromuscular blocking agent to facilitate tracheal intubation.[8]

Sustainability

We distinguish between sustainability capacity and sustainability outcomes. Sustainability capacity may be defined as “the existence of structures and processes that allow a program to leverage resources to implement and maintain evidence-based policies and activities effectively.”[9] For the current study, we define sustainability outcomes (hereafter sustainability) as sustained improvement in critical measures’ performance over time.

We considered sustainability capacity when designing the interventions. One example is the second pediatric emergency medicine (PEM) attending’s promotion of the checklist–the second attending role was explicitly designated to encourage the checklist completion during and after the improvement project. Another example is including various care providers on the improvement team to help determine whether proposed interventions would be adopted and performance sustained. We also intended for these providers to support the initiative as additional project champions, acting as experts within their social networks in the emergency department. Before transitioning to the operational period, we focused on ongoing leadership buy-in and active engagement of staff, maintaining video-based data collection and communication, and continued use of the RSI feedback forms. Finally, as noted above, during the operational period, ED leadership provided 5% full-time equivalent clinical support to the physician project leader and 10% full-time equivalent for a research coordinator to continue video-based RSI data collection.

RSI Processes and Outcome

The 6 key processes were (1) duration of the first laryngoscopy attempt <45 seconds, (2) verbalization of the presence/absence of end-tidal carbon dioxide within 30 seconds after insertion of the endotracheal tube, (3) use of the RSI checklist, (4) use of the video laryngoscope on the initial attempt, (5) approved intubating provider performs all attempts, and (6) correct approach to preoxygenation. The primary outcome was oxyhemoglobin desaturation, defined as pulse oximetry dropping to <90% after administering the RSI sedative and before the endotracheal tube was secured on the final attempt. If the pulse oximetry was below 90% before the RSI sedative, then desaturation was defined as a drop of 10% or more. For the baseline period, we determined desaturation by video evidence of an event occurring, primarily via verbalization by team members. For the improvement and operational period, we determined desaturation by direct visualization of the patient monitor view included in the video recording.

Data Collection

We maintained the patient capture and video-based data collection activities used during the baseline and improvement periods during the operational phase.[3,4] Briefly, for all patient encounters with RSI performed (confirmed by video review), a single investigator completes structured data collection from the video recordings. During the improvement and operational periods, a patient monitor view was added to the recordings, so that oxyhemoglobin desaturation and other physiologic data could be collected directly.

Analysis

We used p-charts and accepted rules to determine common and special cause variation[10] to assess changes in the performance of critical processes and the primary outcome. We used groups of 10 consecutive RSI cases, as in the original study. We did not perform reliability analyses, as we have previously reported these for the same data.[4,7]

RESULTS

Subjects

We included 615 of 643 (96%) patient encounters with RSI performed across the 3 study periods: 114 baseline, 105 improvement, and 396 operational. All 28 excluded patient encounters either had no video recordings available, or the recording was limited in some way that precluded collection of some but not all data elements. There were 9 such encounters in the baseline period, 10 in the improvement period, and 9 in the operational period. For all 3 periods, missing data were uncommon (Table 1 and Fig.).

Table 1.

Characteristics for Patients Undergoing Rapid Sequence Intubation in a Pediatric Emergency Department over 3 Study Periods*

	Baseline (114)†	Improvement (105)‡	Operational (396)
Age (median, IQR)	2.4 (0.4, 10.1)	3.0 (0.4, 10.8)	2.3 (0.4, 10.6)
Younger than 24 mo	53 (46)	43 (41)	186 (47)
Diagnostic category
Neurologic	39 (34)	33 (31)	141 (36)
Respiratory	29 (26)	13 (13)	101 (26)
Trauma	21 (18)	22 (22)	91 (23)
Shock	13 (11)	21 (21)	27 (7)
Other	12 (11)	13 (13)	36 (9)
Attempt success§
First	59 (52)	66 (63)	266 (67)
First or second	84 (74)	92 (90)∥	335 (85)
Cardiac arrest	2 (1.8)	1 (1.0)	4 (1.0)

N (%) shown, unless indicated.

*Baseline, April 2009 through March 2010; Improvement, July 2012 through December 2013; Operational, January 2014 to December 2018.

†Nine patients in baseline period excluded from all data collection due to lack of adequate videos. In the improvement and operational periods, patients without videos were included in the total and for variables, with data extracted from the electronic record for Table 1. Data missing due to lack of video are indicated in the Figure.

‡Diagnostic category data missing for 3 patients.

§Attempt defined as insertion of the laryngoscope blade, whether or not endotracheal tube insertion was attempted.

∥Second attempt data missing for 3 patients.

Patient characteristics across the three periods were generally similar (Table 1). There were notably more shock and fewer respiratory cases in the improvement period and an increasing percentage of trauma cases across the 3 periods. The difference in respiratory cases was likely due to the improvement period, including only one winter “viral respiratory” season. The first attempt success was higher in the improvement and operational periods than in the baseline. Cardiac arrests were similarly uncommon in all 3 time periods (<2%).

RSI Processes and Main Outcome

The p-charts indicated generally sustained performance for all 6 key process measures and the primary outcome (desaturation) during the 5-year operational period (Figures); ED care teams performed each critical process for approximately 90% of the patients. We detected additional improvement during the operational period (eight or more consecutive points below the centerline) for 3 measures: use of the video laryngoscope on the first attempt (Fig. 1B, centerline adjusted April 2016), the performance of preoxygenation (Fig. 2A, centerline adjusted March 2015), and performance of laryngoscopy attempts by an approved provider (Fig. 3A, centerline adjusted November 2013). There was special cause variation suggesting deterioration for 2 key processes and the primary outcome: one for adequate preoxygenation (Fig. 2A, one point above the upper limit in September 2017), 4 for attempts by an approved provider (Fig. 3A, one point above the upper limit in July 2014, December 2016, March 2017, and December 2017), and 1 for desaturation (Fig. 4, eight points above the centerline, September 2017 to May 2018).

Fig. 1.

Statistical process control (P) charts for 2 key processes. A, Failure to use the rapid sequence intubation checklist; and B, failure to use the video laryngoscope on the first attempt at laryngoscopy. Each dot represents the percentage of 10 patients. Missing data: 7 patients for the usage of the checklist and 10 patients for the video laryngoscope all in the operational period, and due to lack of an adequate video.

Fig. 2.

Statistical process control (P) charts for 2 key processes. A, Failure to perform adequate preoxygenation before the first attempt; and B, Failure to acknowledge end-tidal carbon dioxide (ETCO2) within 30 seconds of endotracheal tube insertion. Each dot represents the percentage of 10 patients. Missing data: 10 patients for preoxygenation and 18 patients for ETCO2, all in the operational period, and due to lack of an adequate video.

Fig. 3.

Statistical process control (P) charts for 2 key processes. A, Laryngoscopy attempt by a nonapproved provider; and B, first attempt at laryngoscopy longer than 45 seconds. Each dot represents the percentage of 10 patients. Missing data: 8 patients for nonapproved provider and 11 patients for the first attempt longer than 45 seconds, all in the operational period, and due to lack of an adequate video.

Fig. 4.

Statistical process control (P) charts for the primary outcome measure, oxyhemoglobin desaturation. Each dot represents the percentage of 10 patients. Missing data: 1 patient in the interventional period and 19 in the operational, all due to lack of an adequate video.

RSI Feedback Forms

In the operational period, we sent RSI feedback forms (email) for 344 of 396 patient encounters (87%). The median time to feedback email was 32 days (interquartile range, 18, 49; overall range 1–132 days). The patient encounters without feedback were due to a lack of video or staffing issues.

DISCUSSION

Comparison to Previous Studies

In a systematic review of 125 studies of health interventions’ sustainability, although two-thirds of studies examined outcomes more than 2 years after the intervention, few studies demonstrated high levels of sustainability.[2] Moreover, few of these studies used rigorous methodology; most relied on self-reports and few measured changes in critical processes and outcomes. We sought to address these deficiencies by studying sustainability over 5 years, using a well-defined set of process and outcome measures, employing a rigorous, refined approach to video-based data collection, and explicitly defining sustainability. We also distinguished between measuring sustained performance/outcomes and sustainability capacity.

Our findings are also generally consistent with the literature on improving the performance of emergency intubation.[11-17] These studies took a similar, checklist- or algorithm-based approach, and the magnitude of improvement was comparable. None of these studies, however, examined performance for more than 24 months.

Theory of Improvement and Sustainability

The findings from the current study strengthen our belief in the original theory. As our objective was to determine whether the initial improvements were sustained, that is, the sustainability of RSI performance and outcomes, we can only hypothesize which aspects of the improvement effort contributed to sustainability capacity. There are likely numerous factors that impact the sustainability capacity of improvement initiatives. In a systematic review of available models of sustainability, Lennox et al[18] found 40 separate constructs involved in the concept of sustainability. Some of these factors are likely essential, and others apply only to particular projects or settings. We speculate that the combination of an interprofessional approach and detailed process mapping with video review facilitated the design of interventions for which care providers’ internal and external motivation was exceptionally high. We also speculate meaningful and accurate video-based feedback combined with higher provider motivation to increase the sustainability capacity of the interventions. We did not attempt to determine individual providers’ motivations or perspectives; however, future studies should include a qualitative component to assess these factors, in addition to determining which aspects of our improvement bundle and infrastructure were most important.

It is important to emphasize that our system of video recording and data collection is unique and expensive, a resource that will not be feasible for many centers. However, we have found that the sustainability of our intervention was robust to at least a moderate level of resource disruption. There is no backup staff member to conduct video reviews. At times, competing factors have led to periods of a long delay or absent feedback forms, without any associated reduction in outcomes being met. While some funding and resources are essential, substantial additional funding may not be needed to sustain improvements if care providers broadly adopt key processes into daily work.

Additional Improvement

The performance of 3 fundamental processes improved further during the operational period: use of the video laryngoscope, preoxygenation, and attempts by an approved provider. We speculate that the video laryngoscope’s use improved further as subsequent classes of PEM fellows were trained primarily on this device, and ED staff grew to accept the video laryngoscope as the preferred option. The latter effect may directly result from the power of the ability to see the patient’s airway on video during a tense procedure. We believe preoxygenation improved primarily through independent efforts of specific respiratory therapists to perform the recommended approach, prompted mainly by the video-based RSI feedback. Finally, we believe that an approved provider’s attempts improved further as this crucial process is especially straightforward to enforce, and PEM fellows performed an even more significant proportion of initial attempts, with greater success.

Several instances of special cause variation suggested temporarily worse performance—of preoxygenation, attempts by an approved provider, and the primary outcome, desaturation. After reviewing the relevant cases, we believe the variation in preoxygenation was due to a temporary increase in non-ED respiratory therapists working staffing the ED. For approved providers performing the first attempt, these 4 instances were due to an attending PEM physician letting a senior pediatric resident attempt intubation for a small number of patients. We could not determine an exact cause of the increase for desaturation, although each of the runs of 8 data points was only 20%, just 4% higher than the baseline. There was also no deterioration in any of the critical processes corresponding to the variation in desaturation frequency.

Limitations

There are several significant limitations to our study. As noted in our report of the original improvement initiative, there was a 2-year gap between our baseline and improvement periods. Other unmeasured factors may have impacted RSI performance during this gap. However, we reviewed all cases during this gap, and there were no significant changes to the RSI process during this period. Second, we bundled the interventions and, therefore, cannot determine which aspects were more important, nor can we determine the relative impact of aspects of the interventions on specific parts of the RSI process. We were not attempting to identify specific, independent associations; however, we do not believe that optimization of RSI performance will involve one change. A multimodal approach is likely essential to improvement, as has been repeatedly demonstrated in the literature. Third, we did not attempt to determine the impact of improvement on patient outcomes, most importantly, peri-intubation cardiac arrest. We also did not examine serious adverse events other than hypoxemia and arrest.[19] Many of the adverse events associated with tracheal intubation were too uncommon after our initial improvement initiative to study in a single center, including esophageal intubation.[3] We believe hypoxemia and frequency of prolonged attempts are the best measures of the quality of the RSI process and the major driver of adverse events.[20,21] We are also currently working on additional improvement initiatives to study and reduce cardiac arrest frequency during tracheal intubation. Finally, this was a single-center initiative, and we do not know the generalizability of sustained improvement to other settings. We also acknowledge that this work is resource-intensive. As noted, successful initiatives with similar interventions have been reported in other settings, albeit over a shorter time frame.[11,12]

Next Steps

We are currently working on several projects related to RSI improvement. First, although we demonstrated the sustainability of the RSI interventions, we do not understand the sustainability capacity of the interventions. We are currently planning a mixed-methods study of ED staff to investigate the key factors contributing to sustainability capacity, particularly provider motivation. Second, our original aim was to reduce oxyhemoglobin desaturation to less than 10% of patients undergoing RSI. The margin for improving standardization of the RSI process is minimal, with ED care teams performing each essential process for more than 90% of patients. Our focus now is on enhancing provider performance of tracheal intubation. We study proceduralist mental workload during tracheal intubation in the PED, using a combination of biometrics and qualitative interviews. We believe that understanding mental workload will facilitate the design of interventions to enhance skill acquisition and translation to the clinical environment. Finally, we are designing a study to standardize the tracheal intubation procedure’s performance, particularly the choreography between the proceduralist and the primary assistant.

CONCLUSIONS

In conclusion, improvements in RSI safety in a PED were sustained 5 years after the end of a successful QI initiative, with additional improvement observed in several key processes. Although video review was likely essential, we need further research to elucidate the factors contributing to the sustainability capacity of successful improvement.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.

ACKNOWLEDGMENTS

We acknowledge our colleagues in the emergency department at Cincinnati Children’s Hospital, especially the many nurses, respiratory therapists, paramedics, patient care attendants, and physicians who did the real work to improve the safety of rapid sequence intubation. Without their support, motivation, and willingness to change, our work would have failed. We thank our leadership, especially Rich Ruddy, Steve Porter, Hamilton Schwartz, Scott Reeves, and Elena Duma, for their consistent and ongoing support. Finally, we thank Michelle Caruso, Jenna Dyas, and the Center for Simulation and Research for their work on the original improvement project. We completed this work with internal support only.