Improving Knowledge of Active Safety and QI Projects Amongst Practitioners in a Pediatric ICU.

Introduction: The success of quality improvement (QI) projects depends on many factors, with communication and knowledge of project-specific practice change being fundamental. This project aimed to improve the knowledge of active safety and QI projects.
Methods: Two interventions were trialed to improve knowledge: paired email and meeting announcements followed by a daily huddle to review ongoing projects. Knowledge, measured as the ability to recall a project and its practice change, was the primary outcome. The frequency and duration of the Huddle were process and balancing measures, respectively.
Results: Seven days after a meeting/email announcement, 3 of 13 (23%) faculty and fellows recalled the announced practice change. Investigators then tested the effects of the Huddle by assessing practitioners' knowledge of safety and QI project-related practice changes on the first and last day of a service week. The average percentage of items recalled increased from the beginning to end of a service week by 33% [46% to 79%, 95% confidence interval (CI) 12-53] for faculty and 27% (51% to 77%, 95% CI 13-40) for fellows. The Huddle occurred in four of seven (interquartile range 2-5) days/wk with a mean duration of 4.5 (SD 2) minutes. Follow-up assessment 2 years after Huddle implementation demonstrate sustained increase in item recall [faculty +36% (95% CI +13% to 40%); fellows +35% (95% CI +23% to 47%)]. Conclusions: A daily huddle to discuss safety and QI project-related practice change is an effective and time-efficient communication method to increase knowledge of active projects.

INTRODUCTION

Despite increased interest in quality improvement (QI) and research into QI methods, many QI efforts fail to reach their goals, with some studies noting that fewer than half of QI efforts succeed.[1-4] Multiple implementation and QI studies and frameworks examining factors associated with implementation success or failure have noted the importance of communication of a practice change.[4-9] In a study of causes of poor guideline compliance, Cabana et al[7] noted that a lack of knowledge of a practice change, defined as awareness and familiarity with a specific practice change, was a commonly identified cause of guideline nonadherence. Further, researchers involved in the Welsh QI collaborative, “The 1,000 Lives Campaign” noted that a structured communication strategy increased knowledge of practice change and contributed to project success.[10]

Although there is a growing appreciation of the importance of communicating practice changes, little research identifies which communication methods optimize knowledge of active practice changes. Current communication methods to develop knowledge include preimplementation training sessions, visual aids such as posters, announcements during meetings, general emails, and audit/feedback systems. However, no method or combination of methods is proven superior. As knowledge of practice change is associated with implementing a new practice, increased understanding of how to improve knowledge of practice change may lead to a higher number of successful QI efforts.

This project aimed to increase the knowledge of QI-related practice changes in a pediatric intensive care unit (PICU). Iterative Plan-Do-Study-Act (PDSA) cycles culminated in developing and piloting a daily “Safety and QI Huddle.”

METHODS

Setting and QI Team

The hospital is a 289-bed, freestanding university-affiliated, tertiary pediatric hospital with 13,000 general admissions and 2,600 PICU admissions annually to its 28-bed PICU unit. The clinical team receiving the intervention included a pediatric intensivist, a pediatric critical care fellow, and three to five resident physicians from pediatrics and emergency medicine. Intensivist faculty average one week of service per month, fellows average 1–2 weeks/mo, and residents rotate for 2–4 weeks/year. At least one member of the team changes weekly. At the start of the QI project, there was no standard communication method. Prior QI efforts used a combination of posters, announcements during meetings, and emails to disseminate information regarding practice change. Three faculty and three fellows composed the QI team. The University of Utah IRB reviewed the project and determined it did not require IRB approval.

QI Method

The project utilized the Model for Improvement as the QI framework.[11] Our unit has an established QI committee (QIC) comprised of faculty with QI methods training, fellows, nurse practitioners (NPs), and PICU nurses to develop and implement QI projects. The QIC served as an expert panel. The critical care faculty and fellows and “practitioners” served as a user panel to identify and customize preferred communication methods for the intervention. The QI team obtained practitioner input via survey using the REDCap internet interface.

Interventions

The QI team performed a cause-and-effect analysis to identify potential interventions. Figure 1 displays the “Cause and Effect” (A) and “Key Driver” (B) diagrams developed for this project. The QI team noted that the lack of standard communication methods to create and sustain knowledge was a significant cause of low project knowledge and chose to focus improvement efforts on developing standard communication methods. This project tested two interventions (1) email/meeting announcements and (2) a huddle process.

Fig. 1.

Summary of baseline analysis. A, Cause-and-effect diagram showing potential causes of low knowledge of projects. B, Key driver diagram showing project goal, driver of change, and potential interventions. * = intervention used in the project.

Email/Meeting Announcement

The first intervention, email/meeting announcements, leveraged existing communication methods. The meeting announcements occurred at a twice-monthly Morbidity and Mortality (M&M) Conference, attended by most practitioners. All practitioners received an email detailing a practice change the same day as the M&M conference with identical information presented at the conference.

Huddle

Poor knowledge retention from our initial intervention led to developing a second intervention, entitled “Pre-Round Huddle,” or “Huddle,” for short. A PICU service team member presented a brief description of active QI-safety projects and specific practice changes. Based on user feedback, a list of medication shortages with treatment alternatives and a description of active research projects requiring prospective patient enrollment with screening criteria and site PI contact information were added to the Huddle. To assist in remembering the items to discuss, the QI team created a cognitive aid known as the “Huddle Sheet” to read during the Huddle (Fig. 2). Practitioners performed the Huddle daily immediately before starting patient care rounds. The Huddle Sheet was updated weekly by a QI team member who engaged in multi-disciplinary communication to identify items to add to the Huddle Sheet.

Fig. 2.

The Huddle Sheet is an 8.5 × 11 inch sheet of green, laminated paper attached to the mobile computer used by faculty and fellows for rounds.

Outcomes and Measures

Our primary outcome was “knowledge” of projects, defined as project awareness and familiarity with specific practice changes. Our Specific, Measurable, Attainable, Relevant, and Time-bound (SMART) goal was to achieve >70% knowledge of active projects amongst PICU faculty and fellows over 6 months. We also measured the improvement in compliance with ongoing QI projects.

Email/Announcement-specific Measures

To assess the effectiveness of email and meeting announcement communication, practitioners received an email and meeting announcement regarding a new practice to reduce the frequency of obtaining surveillance blood cultures on patients supported with extracorporeal membrane oxygenation (ECMO). The cultural practice changed from obtaining daily surveillance cultures from two sites to obtaining cultures from a single site every other day. One week after announcing the new practice, practitioners responded to an email asking them to describe the blood culture practice change discussed in the announcement. The QI team also reviewed blood culture use for ECMO patients four weeks before and after the announcement to assess the impact on inappropriate blood culture practice.

Huddle-specific Measures

Before implementation, the QI team presented the Huddle at a meeting and via email. In addition, practitioners provided feedback on the Huddle via a survey. The survey contained questions to evaluate the acceptability, appropriateness, and feasibility of the Huddle method using the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM).[12] Acceptability, appropriateness, and feasibility are leading implementation measures and may identify interventions that need further adjustment before implementation (Table 1).[13] The AIM, IAM, and FIM consist of four questions scored on a 5-point Likert scale (1, completely disagree to 5, completely agree), with higher scores indicating greater acceptability, appropriateness, and feasibility. The survey also included a comments section to elicit questions and feedback regarding the Huddle.

Table 1.

Definition and Leading Measure of Preimplementation Acceptability, Appropriateness, and Feasibility

Item1	Definition*	Measure	Average Score (SD)
Acceptability	Perception that a given treatment, service, practice, or innovation is agreeable, palatable, or satisfactory	AIM	15.2 (2.6)
Appropriateness	Perceived fit, relevance, or compatibility of the innovation or evidence-based practice for a given practice setting, provider, or consumer; and/or perceived fit of the innovation to address a particular issue or problem	IAM	15.6 (2.8)
Feasibility	Extent to which a new treatment, or an innovation, can be successfully used or carried out within a given agency or setting	FIM	16.1 (2.2)

*Definitions taken from Proctor et al.[13]

The QI team measured the effects of the Huddle on knowledge for 12 weeks. A QI team member independently approached practitioners at the beginning and end of a service week and asked them to identify and describe as many items from the Huddle Sheet as possible. Percent knowledge at the end of a service week, calculated as the number of accurately recalled items divided by the total number of items multiplied by 100%, served as the primary outcome. The additional outcome included change in knowledge, calculated as percent knowledge postservice minus percent knowledge preservice. The QI team assessed sustainability by measuring percent knowledge at the end of a service week and change in knowledge for 12 weeks 2 years after implementing the Huddle. A paired t-test compared the prepercent and postpercent knowledge.

As a process measure, the PICU fellows self-reported the number of days the Huddle occurred during the service week. The goal was >50% (at least 4/7 days) performance. The balancing measure was the time to complete the Huddle as assessed by fellow self-report of the average time taken to complete the Huddle during their service week. The goal was <5 minutes.

The QI team assessed the effects of the Huddle on an active QI project by tracking compliance with a practice change associated with a QI project to reduce the proportion of hospital code events missing documentation in the electronic health record (EHR). The practice change required practitioners to place a specific “Code Note” in the EHR. In week 1, practitioners received an email with a note template to “copy and paste” into the EHR to document code events. In week 7, the template was embedded into the EHR as a searchable “dot phrase” to ease use. In week 13, the template was updated to make the format consistent with code notes used in other hospitals in the healthcare system of which PCH is a member. Concurrent with this update, the need to place a “Code Note” in the EHR and instructions for using the template was added to the Huddle. The QI team calculated the proportion of code events without documentation for the 12 weeks before and 8 weeks following the addition of the code note practice change to the Huddle. A p-chart and Fischer exact test compared the proportion of missing notes before and after listing the code note practice change in the Huddle.

After 12 weeks of Huddle use, practitioners responded to a survey assessing their perception of the effectiveness and functioning of the Huddle using a 5-point Likert Scale (1, completely disagree to 5, completely agree). As “QI” and “Safety” projects had separate sections in the Huddle, practitioners responded to separate questions regarding the Huddle’s effect on “QI” and “Safety” projects.

RESULTS

Identifying Potential Communication Methods

The QIC identified six potential communication methods of which practitioners selected their two preferred methods. The survey response rate was 50% (13/26). Table 2 summarizes the number and percent of respondents choosing each method, with a monthly email and announcements at the M&M conference chosen most frequently.

Table 2.

Preferred Communication Method

Communication Method	No. Votes	Percent of Respondents Selecting Method (n = 13)
Monthly newsletter email	8	62
Announcement at M and M	6	46
Physical display board	4	31
Virtual bulletin board	3	23
Morning preround Huddle	3	23
Dedicated monthly QI meeting	2	15

Email and Meeting Announcements

The initial PDSA cycle involved paired email and meeting announcements. Seven days after announcing a change to ECMO surveillance blood culture practices, 13 of 26 (50%) practitioners responded to an email asking them to describe the practice change. Only three of /13 (23%) respondents recalled the practice change. Four patients were on ECMO for a combined 29 patient ECMO days in the 4 weeks before the announcement, with inappropriate cultures obtained on seven of 29 (24%) ECMO days. One patient was on ECMO 4 weeks after the announcement, with inappropriate cultures obtained on two of four (50%) of days.

Preround Safety and QI Huddle

Preimplementation Survey

The survey response rate was 54% (14/26). Table 1 summarizes the mean AIM, IAM, and FIM scores. Overall, there was a favorable opinion of the acceptability, appropriateness, and feasibility of implementing the Huddle. Seven respondents included a comment; three were concerned with the Huddle’s timing and duration, with one suggesting a 5-minute limit.

Postimplementation

Due to missing prepercent or postpercent knowledge data, the QI team collected data for 14 weeks to allow 12 weeks of complete preweek and postweek data for analysis. Over the 14 weeks, the Huddle Sheet contained 12 items [interquartile range (IQR) 11–12]. Table 3 summarizes the pre- and postpercent knowledge of faculty and fellows. The mean postweek recall was >70% for both faculty and fellows and both groups achieved >70% recall during 10 of the 12 (83.3%) weeks postimplementation. The Huddle occurred a median of 4/7 (IQR 2–3) days with a mean time of 4.5 (SD 2) minutes. Follow-up assessment 2 years after implementation showed a mean postweek recall >70% for both groups and a significant increase in percent knowledge [faculty +36% (95% confidence interval [CI] +13% to 40%); fellows +35% (95% CI +23% to 47%] (Table 3). The Huddle occurred a median of 5.5/7 (IQR 4–7) days with a mean duration of 2.7 (SD 1.1) minutes.

Table 3.

Comparison of Pre and Post Service Week Knowledge at Huddle Implementation and 2-year Follow-up

	Position	Pre Service Mean Percent Recall (SD)	Post ServiceMean Percent Recall (SD)	% Change (95% CI)
Initial implementation	Fellow	51 (17)	77 (19)	+ 27 (13–40)*
	Faculty	46 (27)	79 (20)	+ 33 (12–53)*
2-year postimplementation	Fellow	50 (26)	85 (14)	+ 35 (23–47)*
	Faculty	45 (14)	81 (11)	+ 36 (29–44)*

*For pre-post comparison P < 0.01.

The postimplementation survey response rate was 73% (19/26). Of the respondents, 89% (17/19) agreed or completely agreed that the Huddle improved knowledge of safety-related practice change, and 74% (14/19) agreed that the Huddle improved knowledge of QI project-related practice change. Only 11% (2/19) of respondents thought the Huddle took too long.

Figure 3 summarizes the results of the effect of adding the code documentation project to the Huddle. Again, special cause variation is seen on the process control chart (Fig. 3), with all eight points following the addition of the code documentation project to the Huddle falling below the initial centerline established by weeks 1–12.[14] Adding the code documentation project to the Huddle was associated with a 28% reduction in missing code documentation (Pre Huddle = 51% missing, Post Huddle = 23%, P = 0.002).

Fig. 3.

The p-chart includes data for 20 months from a project to reduce the proportion of code events missing documentation in the electronic health record. Months 2, 3, and 4 had no codes. The project was added to the Huddle after week 12, decreasing missing documentation is noted in weeks 13–20.

DISCUSSION

This study describes a QI project to improve knowledge of practice changes related to QI and safety improvement efforts in a high-volume PICU. Creating and implementing a daily huddle to review a list of active QI and safety projects and their required practice changes led to >70% knowledge of practice changes. In addition, this intervention was time-efficient and led to measured improvement in compliance with project-associated practice change. Furthermore, the effects of the Huddle are durable, with improvement in knowledge sustained 2 years after initial implementation. This study also found that email/meeting announcements had minimal effect on knowledge.

Knowledge of a practice change is a critical component of QI. If one does not know a new practice exists or how to perform the new practice, then the practice is unlikely to occur. The importance of knowledge of practice change is based on the Theory of Diffusion of Innovation. This theory notes the first stage of adopting a new practice requires one to obtain knowledge that a new practice exists and how the practice is performed. The importance of practice change knowledge is also included in several frameworks identifying barriers and facilitators to implementation, including the Consolidated Framework for Implementation Research and the Theoretical Domains Framework. Studies specifically noted lack of knowledge as a barrier to implementing clinical practice guidelines. Although knowledge of practice change is needed, there is little investigation into which communication methods facilitate increased knowledge and compliance with practice change. Our project investigated potential communication methods to increase knowledge of practice change.

Our unit did not have specific or consistent methods to communicate practice changes at baseline. We used a series of expert and user review panels to identify potential ideas to implement (expert review) and select the most desirable ones for implementation (user review).[9,15,16] We initially trialed paired email/meeting announcements. Email/meeting announcements are efficient methods to reach a large audience, and the use of email allows practitioners not in attendance at a meeting to receive similar information. Some studies have shown improved adherence to clinical guidelines with emailed information.[17-20] However, we found that email/meeting announcements did not increase knowledge of practice change or compliance with practice change. Potential reasons for the failure of these methods are inattention to email, lack of attendance or attention during meetings, or forgetting the practice change in the duration from the announcement until an individual is in a position to deploy the new practice.[20,21] Furthermore, qualitative data has noted that email and meeting announcements may be poor methods of disseminating time-sensitive safety information and may contribute to communication overload.[20]

Our second intervention involved the creation of a daily huddle involving the on-service physician team. Huddles are brief meetings to create situational awareness of unit census, staffing levels, and patient plans for the upcoming shift.[21-26] As the user panel did not rank the Huddle as a top communication method, we sought additional input to predict the Huddle’s potential success and identify barriers to its success. We determined that a focused, pre rounding huddle would be well-received using the AIM, IAM, and FIM. While the AIM, IAM, and FIM do not yet have score breakpoints to predict success, the overall high rating on these measures made us comfortable proceeding with implementation.[12,27] The comment section of the survey did identify a potential barrier to prolonged Huddles, and this allowed us to design the Huddle to be performed in under 5 minutes and to develop a balance measure to examine Huddle duration. We also added sections on medication safety and clinical research projects in response to user feedback.

A brief huddle to increase knowledge of practice change has several advantages compared to other communication tools. By occurring daily and including only those needing to perform the practice, the Huddle limits the potential of information overload and forgetfulness with periodic meeting and email announcements. Given the frequency of the Huddle and specificity of the information it transmits, it could be considered a type of “just in time” reminder.[28] This type of reminder often takes the form of an EHR “pop-up” guiding a practitioner toward a new or evidence-based practice.[17,28-32] While our Huddle may not be as temporally associated with a task as a pop-up, it does have several potential advantages. First, individuals will receive the reminder daily. Still. It may not be so frequent and intrusive to cause “pop up” fatigue where the message is frequently ignored.[33] Second, a paper-based reminder is easy to update and change. It does not have the lag time or technical expertise needed to create and edit an EHR reminder.

Our study has several limitations. First, an extended test of the email/meeting announcement did not occur. Knowledge may slowly increase over several repetitions of announcements. However, given the poor initial performance of this intervention to increase knowledge and adherence to practice change and the goal of rapidly increasing knowledge of practice change, we felt that announcements were ineffective, and efforts moved on to another intervention. Second, participants were asked to self-report the number of times per week the Huddle occurred and the duration of the Huddle. This approach could lead to reporting bias with over-reporting Huddle performance and under-reporting duration. Although this is possible, we felt that having a research team member observe daily rounds would act as an artificial reminder to perform the Huddle quickly and bias the results. To better understand if the Huddle was taking too long, the postimplementation survey included a question about the duration of the Huddle, and an overwhelming majority of respondents found the Huddle did not take too long. Future studies of the Huddle could include using of a “secret shopper” to discretely time the Huddle.

CONCLUSION

Communication of practice change is an important but under-researched factor in improvement project success. Implementing a daily huddle is an efficient and effective communication tool to increase knowledge and compliance with practice changes related to safety and QI projects. Future implementation of a daily huddle is one component that may lead to more projects reaching their improvement goal.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.

Portions of this work were presented in poster form at the 2021 Society of Critical Care Medicine Congress.