Audit-and-Feedback and Workflow Changes Improve Emergency Department Care of Critically Ill Children.

INTRODUCTION: Children with severe infection have improved outcomes when they received antibiotics promptly. Positive cultures help guide physicians in antibiotic selection. In 2011, 30% of children intubated in the emergency department received antibiotics and had respiratory culture collected within 60 minutes of intubation. Knowing the risk of delaying appropriate antibiotics, we charted a quality improvement team to improve compliance with 80% of intubated patients receiving both.
METHODS: The team evaluated all children intubated with concern for infection in the emergency department. Using a multidisciplinary team and employing quality improvement methods, we implemented multiple plan-do-study-act cycles to improve time to antibiotics and respiratory cultures. The team continued to implement successful interventions and restarted interventions directly affecting improvement.
RESULTS: While multiple interventions had small effects on the baseline of 30% compliance, 2 interventions appeared more influential than others. Workflow changes and audit-and-feedback created the largest, persistent positive changes. The importance of audit-and-feedback became very obvious when the project entered sustain mode. An abrupt decrease in compliance occurred when audit-and-feedback stopped. Complete recovery in compliance to greater than 80% occurred with the resumption of the audit-and-feedback intervention.
CONCLUSIONS: Workflow changes and audit-and-feedback interventions resulted in large improvements. Loss of compliance with cessation of the audit-and-feedback and resumption demonstrated the importance of this intervention. Recovery to >80% compliance with the renewal of the audit-and-feedback program indicates its strength as a positive intervention.

INTRODUCTION

Problem Description

Delays in the administration of appropriate antibiotics for critically ill children with suspected infection can result in poor outcomes.[1,2] Obtaining cultures from the sites of suspected infection is a hallmark of infection management and is vital for guiding antibiotic stewardship, including the de-escalation of empiric antibiotics in critically ill patients.[3] Despite this knowledge, in the pediatric emergency department (ED), long delays may occur between the time of endotracheal intubation, antibiotic delivery, and respiratory culture collection in patients with suspected infection who require intubation. In some cases, patients may not even receive antibiotics or have respiratory cultures collected in the ED before intensive care unit (ICU) admission.

Available Knowledge

Previous studies associate delays in the administration of antibiotics with increased morbidity and mortality.[2] Patients who receive timely antibiotics experience decreased intubation time, shorter ICU and hospital stays, and lower risk of death.[2,4-9] Because critically ill children frequently require broad-spectrum empiric antibiotics for suspected infections due to antibiotic-resistant bacteria, cultures are vital for guiding antibiotic stewardship and de-escalation of definitive antibiotic therapy.[1,3,5,10] Without cultures, children may receive prolonged courses of broad-spectrum antibiotics, contributing to growing difficulties with antibiotic resistance.

Rationale

Recent studies from this institution identified significant delays in the time to initial empiric antibiotics in many critically ill children treated in the local ED.[1,10] In many cases, cultures, particularly lower respiratory cultures, in patients requiring intubation, were delayed, resulting in delays in antibiotic administration, or were obtained after receiving antibiotics, leading to challenges with antibiotic management, particularly de-escalation. This problem occurred even in patients with community-acquired pneumonia.[1]

Patients in the ED with suspected infection who require endotracheal intubation represent the most critically ill patients with a tendency to have or develop sepsis. Even those with respiratory failure from viral infections are at risk for bacterial co-infection.[11] These patients are most likely to benefit from interventions to decrease the time to empiric antibiotics and subsequent high-quality antibiotic management. Historically, in this ED, staff collected blood, urine, and cerebral spinal fluid cultures regularly before antibiotics, but lower respiratory cultures collection occurred uncommonly, even when pneumonia was suspected.[1] Thus, the ED decided to focus quality improvement (QI) efforts on this population to increase the frequency of lower respiratory cultures via endotracheal tube aspirate and reduce the time from intubation to initial empiric antibiotic and respiratory culture collection. Although interventions implemented varied, here 2 key interventions are examined: changes in workflow for support staff and audit-and-feedback (AF) for providers.

The baseline rate of patients who had both lower respiratory cultures obtained and antibiotics administered within 1 hour of intubation was low, averaging 30% per month. The initial goal was to obtain 80% compliance in ED nontrauma patients intubated for suspected infection and sustain it for 6 months.

METHODS

Context

The ED at Nationwide Children’s Hospital in Columbus, Ohio, is part of a freestanding, tertiary care, urban, pediatric hospital with an annual ED volume of >90,000 patients. For inclusion in data analysis, the team analyzed the charts of all children intubated in the ED from February of 2012 through March of 2017. We excluded patients intubated by outside hospitals or by first responders since these actions could not be directly verified. We also excluded patients intubated for trauma, ingestion, or afebrile status epilepticus as these children were without infectious concerns. All other patients intubated for respiratory failure and concern for infection where antibiotics may be indicated were included. The study was deemed a QI project for improving the current standard of care and therefore did not require internal review according to the institutional review board policy.

Interventions

The established QI team implemented the Institute for Healthcare Improvement Model for Improvement using Plan-Do-Study-Act (PDSA) cycles to apply and study interventions. The QI team members comprised various roles within the ED including emergency medicine physicians, nursing staff, unit coordinators, pharmacists, respiratory therapists (RTs), and QI service-line coordinators.

Standard interventions included the introduction of the project, and specific education sessions promoting the importance and relevance of the project to the participating physicians, nurses, RTs, and pharmacists. These sessions occurred both in person and via e-mail by the lead physician and highlighted a literature review, staff expectations, and project goals. Although the study incorporated multiple PDSA interventions (Table 1), this project highlighted 2 interventions: changes in workflow and AF.

Table 1.

PDSA Cycles Completed during the Project and Associated Implementation Dates

The first significant intervention transferred some physician responsibility into support-staff workflow (ie, pharmacists and RTs). Several different QI project teams, both in the ED and the hospital pharmacy department, suggested an ED-specific pharmacist would improve patient care. In 2012, the hospital incorporated a dedicated pharmacist in the ED during peak hours to facilitate demand and collaboration. This change in workflow was maximized by having the pharmacist collaborate directly with the prescribing physician regarding antibiotic selection both before and after intubations, depending on the patient’s needs and presentation. The RT’s on the team conceived and initiated their workflow change to incorporate respiratory culture collection via endotracheal aspirate into their workflow as part of routine postintubation care along with the normal suctioning and securing of the endotracheal tube. After culture collection, the RT was also responsible for reminding both physician and nursing staff to place the antibiotic orders.

AF, the second significant intervention, involved auditing recorded care and providing feedback to both the physician and the RT. Within 1 week of each intubation, the QI project leader sent physicians and RTs involved in patient care a secure e-mail regarding time from intubation to antibiotic and respiratory culture. The team leader acknowledged and commended successes, but recognized failures with requests for the circumstances surrounding the delay (see Supplemental Digital Content, available at http://links.lww.com/PQ9/A59). After two and a half years of regular AF, this intervention was discontinued to prepare the project for sustain mode. Over the next 6 months, a significant drop occurred in the percentage of respiratory cultures collected and antibiotics received within 60 minutes, prompting us to resume AF.

Measures

Data from 78 patients contributed to the baseline from January to December 2011. Study data, from January 2012 through December 2016, included 409 patients, an average of 78 patients per year. The main outcome measure was the percentage of patients with respiratory cultures obtained and antibiotics administered within 60 minutes of intubation. We selected 60 minutes to antibiotics based on time to antibiotic recommendations in septic patients.[5] We chose 60 minutes to respiratory culture to minimize the risk of interpreting positive cultures as hospital-acquired infections and to maintain uniformity with antibiotic delivery. Process measures included median time to antibiotic and respiratory culture. Percent of febrile neutropenic patients receiving antibiotics within 60 minutes of ED arrival represented the balancing measure.

Nursing note documentation determined the time of intubation unless no note existed; in these cases, the ventilator flowsheet record indicated the time of intubation. The antibiotic initiation time recorded by the electronic medication administration record indicated the time of antibiotic delivery. If antibiotic delivery happened before intubation, a negative “time to antibiotics” from intubation occurred. The team accepted these negative time to antibiotics due to patients frequently being identified as septic, with blood and urine cultures collected and antibiotics appropriately delivered before progressing to respiratory failure and requiring intubation.

The time and date recorded by the laboratory upon specimen receipt documented the time of respiratory culture. When this time exceeded 60 minutes from intubation, the team examined the progress notes for an RT’s note indicating the time of respiratory culture collection. The earlier time served as the data point since delays in specimen submission occasionally occur. The team accepted that antibiotic delivery before respiratory culture represented a patient’s clinical need and minimally impacted respiratory secretion bacterial load in short time intervals.

Analysis and Study of the Interventions

The team employed statistical process control methods to determine if changes in the process resulted in changes in outcome measures.[12] For analyses of the percentage of patients receiving antibiotics and respiratory culture promptly, the team calculated and displayed centerline (percent) and control limits (+ 3 sigma) from January 2011 through December 2016. The upper and lower control limits, which demonstrate the limits of the expected inherent variation in the quarterly data, were added. The team evaluated control charts for significant change by using the Nelson rules.[13] When notable process shifts occurred, a new centerline was established and then the process repeated until the project achieved goal attainment.

RESULTS

After establishing a baseline of 30% compliance using 2011 data, the team observed an overall rise in compliance (66.7%) by fourth quarter of 2012 (Fig. 1). Notable interventions included AF, the establishment of the full QI team, and scheduled QI team meetings (Table 1). At the time of project initiation (January 2012), physicians received AF on their compliance, leading to a modest increase in compliance (42.9%, first-quarter 2012; Fig. 1). Decreased compliance in second-quarter 2012 (28.6%) correlated with the department relocating to a new hospital building—a move that doubled the bed capacity and quadrupled the ED’s physical size, resulting in multiple new staff members and drastic workflow changes. Increased compliance seen during third-quarter 2012 (61.5%) was associated with the start of the dedicated ED pharmacist and the formal creation of the QI team. These contextual changes, coupled with establishing regular meetings in third and fourth-quarters, led to incremental improvements in compliance through fourth-quarter 2012 (66.7%; Fig. 1).

Fig. 1.

Statistical process control chart for respiratory culture and antibiotics within 1 hour. [1] MD feedback PDSA; [2] Pharmacist in ED, QI team recruited; [3] First team meeting; [4] Pharmacist work-flow change, many new RTs; [5] RT work flow-change, culture education PDSA; [6] Education PDSA expanded; [7] RT feedback PDSA; [8] RT and MD update; [9] Stopped audit/feedback; [10] Resumed audit/feedback.

Significant changes in RT staff composition during first-quarter 2013 correlated with decreased compliance (54.5%). This change prompted several educational PDSA cycles in 2013 for both physicians and RTs, emphasizing the importance of obtaining cultures as well as the addition of the respiratory culture to the RT’s workflow (Table 1). Another PDSA incorporated the dedicated ED pharmacist into physician antibiotic selection and ordering. Finally, in response to the improved compliance in 2012 associated with AF to the physicians, the team initiated AF to the RTs. Individually, time to antibiotic consistently achieved 80% by second-quarter 2013 and time to respiratory culture consistently achieved 80% by third-quarter 3 (Fig. 2). The team achieved and exceeded the initial goal (80% of patients receiving antibiotics and respiratory cultures within 60 minutes of intubation) by fourth-quarter 2013 (91.7%) due to improvements implemented throughout 2013.

Fig. 2.

Combined run chart for individual data for respiratory culture and antibiotics within 1 hour. [1] MD feedback PDSA; [2] Pharmacist in ED, QI team recruited; [3] First team meeting; [4] Pharmacist work-flow change, many new RTs; [5] RT work-flow change, culture education PDSA; [6] Education PDSA expanded; [7] RT feedback PDSA; [8] RT and MD update; [9] Stopped audit/feedback; [10] Resumed audit/feedback.

The project maintained improvements (92.0% compliant; Fig. 1) for first-quarter 2014. By this time, the RTs had completely incorporated respiratory culture collection into their workflow and pharmacy collaborated with physicians for antibiotic administration to intubated patients. The team continued to provide AF to physicians and RTs but transitioned from scheduled meetings to e-mail updates. Due to a resulting dip in compliance (73.3%) by third-quarter 2014, leaders repeated the educational module for the physicians and RTs fourth-quarter, anticipating improved compliance as seen in 2013. Slow but steady improvement toward the 80% compliance goal occurred during first- and second-quarters, 2015 (78.9% and 87.5%, respectively).

At this point, due to upcoming changes in resource allocation, leaders anticipated placing the project in sustain mode in January of 2016. The AF program stopped in third-quarter, 2015. An immediate decrease in compliance occurred (66.7%), resulting in a special cause event fourth-quarter, 2015 (Fig. 1). With only 52.6% of patients receiving both antibiotics and respiratory cultures within 60 minutes), the data were outside the control limits of the control chart (Fig. 1). AF resumed first-quarter, 2016, with immediate improvement in the percentage of patients receiving antibiotics and respiratory cultures within 60 minutes of intubation. The project achieved >80% compliance in third-quarter, 2016 and sustained this through first- quarter 2017 without any further interventions.

The process measures mirrored the above changes. The median time to respiratory culture shifted from 161.7 minutes during the baseline period to 27 minutes during the last 6 months of the project. The median time to antibiotics shifted from 57.9 minutes during the baseline to 3 minutes during the last 6 months. Many of these patients received antibiotics before intubation due to concerns for acute infection, resulting in a possible skewing of the median time to antibiotics.

A concurrent QI project followed the percent of febrile neutropenic patients receiving antibiotics within 60 minutes of ED arrival. With similar goals, competition for resources may have occurred leading to decreases in both project’s time intervals, making it a perfect balancing measure. Fortunately, the febrile neutropenic project also saw an increase in patients receiving antibiotics within 60 minutes from a baseline of 12–70% within the same time interval.

DISCUSSION

Summary

Improvements in the time to antibiotic delivery and respiratory culture collection for children intubated in the ED for respiratory failure with concern for infection resulted from specific and measurable interventions. The team achieved and sustained the goal of antibiotic delivery and respiratory culture collection in 80% of patients intubated with concern for infection. These improvements associated most with changes in workflow and AF interventions. The AF intervention was associated with the greatest impact, as demonstrated by special cause change when the intervention stopped with regained improvement upon reimplementation.

Interpretation

As typical for QI projects, the team employed many interventions and saw positive cumulative results from all. Small steps, such as developing the QI initiative and simply meeting as a team led to small positive change. Educational modules for physicians and respiratory therapy were associated with slow, steady improvement while more sweeping changes, such as pharmacy’s and respiratory therapy’s workflow-restructuring and AF efforts, were associated with greater improvements. Determining which intervention has had the greatest impact is challenging.

Individually, antibiotic delivery and respiratory culture collection achieved 80% compliance before the primary goal reaching 80%. Possibilities for antibiotic delivery occurring first include target times for antibiotic administrations were already common in the ED in septic or febrile neutropenic patients. Furthermore, respiratory culture represents a new task that had previously been less frequently done and for a smaller number of patients or disease processes. The culture rate also dropped more vigorously once AF stopped as compared with the antibiotic delivery rate. This outcome may result from high RT staff turnover and nonexistent pharmacy staff turnover. These results also emphasize that dual component objectives are inherently more difficult to attain.

The initiation of AF for physicians in January 2012 lead to a modest improvement in patients achieving the goals from baseline. Workflow changes were followed by further improvements, with pharmacy changes having a larger impact than RT changes. Only after RTs started to receive AF did compliance increase above 90%. Since interventions added cumulatively, one could not tell if individual interventions or the combination of AF and changes in workflow led to improvement. Surprisingly, given the limited impact initially seen, when the team eliminated the AF program under the assumption of sustainability, an immediate decrease in compliance occurred that only improved with the resumption of this intervention.

This study distinctly demonstrates the ability of AF to impact and sustain QI programs positively. Although the available literature shows variable effects of AF on provider behavior, most studies support its utility in a wide variety of settings.[14-20] The striking improvement in this study supports the positive nature of AF and contributes to the audit and feedback literature in 2 additional aspects. First, this project took place in a pediatric ED. A preponderance of evidence exists in the adult literature, with a focus in surgical and ICU settings.[15,16,20-22] This study demonstrates AF’s generalizability to both the pediatric and ED settings. Second, this inquiry specifically demonstrates the power of AF without other influences. No additional interventions occurred after AF reinitiation in 2016. The dramatic drop in compliance when AF stopped followed by a rebound after reinitiating and throughout 2016 substantiates the influence of consistent AF.

In a review of the Cochrane database, Ivers et al.[23] demonstrated the feedback method might be especially influential. A previous ICU study notes that well-timed and specific feedback from a peer to be instrumental in creating positive change.[21] In this study, likely contributions to improvement included the team leader, who did not have a supervisory or administrative role, providing feedback to RTs and physicians within 1 week of the encounter. These interactions gave identifiable data points and asked specific questions regarding reflection on the experience. Although the team did not directly evaluate the effect of changing the timing of the feedback or who provided the feedback, prompt feedback (within a week) and peer-to-peer feedback lead to a positive response.

There may be unintended and potentially negative consequences of the intervention. For example, the increased focus on obtaining cultures from patients intubated in the ED may lead to culture collection in patients without suspected infection. This outcome could result in increased cost and potentially unnecessary antibiotic courses for some patients, including those with isolated viral infections. However, previous data show 30–50% of patients with the respiratory syncytial virus have bacterial co-infection, making a short course of empiric antibiotics reasonable.[11] Furthermore, this project focuses primarily on process improvements in the ED, where viral testing and bacterial culture results are frequently unavailable, and decisions require real-time action. Thus, the goal was a focus on the process improvements of timely and appropriate antibiotic prescription in critically ill patients while also improving the microbiologic data that could allow later antibiotic de-escalation. Research is ongoing to investigate the impact of these interventions on patient outcomes and antibiotic utilization.

Limitations

Several limitations to the study warrant discussion. As a tertiary free-standing pediatric facility, resource availability may differ from other locations. The hospital employs QI specialists to support these projects. Staffing in the ED included up to 2 pediatric emergency medicine trained physicians, 2 general pediatricians, 3 RTs, 1 pharmacist, and more than 20 nurses during peak times. The cost of staffing RTs and pharmacists in ED may be prohibitive; therefore, changes in workflow might not be possible. Also, the project champion used administrative time to work on the project. Without this, providing AF is time-consuming and frequently unsustainable. Even at this institution, investigating ways to sustain this timely, individual, peer-to-peer feedback efficiently exist. It is possible that some simultaneous unidentified process resulted in the improvements identified, but it is likely that the cumulative improvements seen throughout this project, especially associated with AF, directly resulted in these improvements. These findings align with many, if not most, of the criteria proposed for attributing causality in QI research.[24]

Some variability in the timing of feedback existed. Although the standard response occurred within a prescribed amount of time (1 week), sometimes feedback was given 1 day after the patient encounter and sometimes 7 days after. The team did not specifically investigate this variability and others have called for the specific evaluation of AF with standardized timing and delivery method.[21] However, the significant positive response to AF, regardless of the timing, suggests that the timing of feedback within the range observed had no significant influence on the overall impact of AF.

CONCLUSIONS

The team successfully achieved the goal of respiratory culture collection and antibiotic delivery within 60 minutes of intubation by highlighting the importance of changes in workflow and the positive impact of AF. This study supports the growing body of evidence regarding the power of AF and demonstrates its further generalizability. Next steps include developing systems to create more efficient AF.

ACKNOWLEDGMENTS

The authors thank Don Buckingham, MBOE, for his assistance in generating the figures.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.