Pediatric Quality Improvement (QI) Virtual Practicum: Adapting a QI Simulator.

Introduction: Pediatric residencies are expected to arm trainees with skills in quality improvement (QI) that allow trainees to systematically enhance their own practice. Simulation has been shown to be effective in teaching QI, but there are no published QI simulation tools that target pediatric learners.
Methods: We adapted a previously developed QI simulation to include a case relevant for pediatric residents. Participants devised interventions using basic QI principles with iterative feedback from facilitators with knowledge of QI methodology. Changes in resident knowledge, attitudes about the curriculum, and depth of engagement in QI were assessed using pre- and posttests, surveys, and assessment of independent QI activities performed prior to graduation, respectively.
Results: Eighty-two residents completed the simulation. Of the 76 residents who completed both the pre- and posttests, which each had a total possible score of 28 points, 68% had improved posttest scores, with an average score increase of 2.6 points (SD = 0.6, p < .001). Improvements were most pronounced for residents that scored in the lowest quartile on the pretest. After the simulation, residents reported greater confidence in and likelihood of completing a QI initiative. There was no difference in the level of involvement in future independent QI activities completed by residents who were simulation participants compared with nonparticipants. Discussion: Adapting a previously published QI simulation for pediatric residents was feasible and effective, and the QI simulation was well-liked by learners. Those with lower baseline QI knowledge may have the most to gain from this simulation.

Educational Objectives

By the end of this session, learners will be able to:

Describe the basic principles of quality improvement (QI), including the plan-do-study-act (PDSA) cycle.

Apply QI principles to design meaningful interventions for a common problem seen in pediatric settings.

Test QI interventions in a simulated environment with immediate feedback from a facilitator.

Introduction

Quality Improvement (QI) research is growing in importance in the medical field. Two pediatrics residency milestones set by the Accreditation Council for Graduate Medical Education directly relate to resident participation in QI. Thus, training programs are faced with the task of teaching QI to residents. While there is a growing body of literature related to QI education, there remains extreme variability in the curricula and methods used for QI education nationwide. Pediatrics residency program directors have expressed dissatisfaction with QI education, which has been largely didactic.[1] While most residents are satisfied with their QI education, over one-third lack confidence in conducting or leading a QI project in practice.[2] Didactic sessions may convey QI principles sufficiently, but in order to help residents apply their knowledge of QI for practical use, experiential learning techniques should be employed.

The technique of simulation is used at all levels of medical training to imitate situations that are rare, complex, and/or critical.[3] The efficacy of simulation is tied to its ability to engage the adult learner to be actively involved in the process of medical learning.[4] Simulation offers the learner a team-based, time-sensitive, and realistic opportunity to gain experiential learning in QI, and simulation can be used to demonstrate the feasibility of participating in a QI initiative that is integrated into residents’ existing daily responsibilities.

Prior work has demonstrated the feasibility of using simulation for QI education while improving participants’ self-reported understanding of and attitudes toward QI.[5] While a QI simulator tool published in MedEdPORTAL by Worsham and colleagues was designed for universal use among medical trainees, the case employed in the simulation is a far reach from the real-time clinical challenges facing pediatric residents, thus making it difficult for this population to be fully engaged with the case. We thus chose to create and implement a pediatrics-specific adaptation that utilizes an adverse patient event within a familiar pediatric clinical setting to identify pathways to optimize care for families with limited resources. Additionally, we developed the session with a unique lens on how structural determinants of health, such as disparities in health literacy, poverty, and language barriers, can be approached with a QI framework. We evaluated our curriculum to identify important knowledge, skill, and attitude gains associated with participation in the adapted simulation experience. Pediatrics residency programs around the country may choose to utilize this tool to train their residents to think like QI specialists and positively impact patient care for kids.

Methods

Development

A team of pediatric residents and one faculty member with experience in QI education adapted the simulation from Worsham and colleagues using a case created specifically for pediatric interns with limited clinical experience. Our case focused on a school-aged patient at a fictional hospital that returned to the emergency department within 24 hours after being treated for an asthma exacerbation. We paid particular attention to preserving the qualities of the simulation from Worsham and colleagues that allow it to be accessible to learners who are new to QI, as well as to individuals across multiple health professions and members of the interdisciplinary clinical team.

Preparatory Work

Each pediatric intern in our residency program completed the simulation as a part of a longitudinal, integrated advocacy curriculum. One week prior to the simulation, interns received an email describing presimulation assignments. We recommended that learners read two articles that provide a foundational knowledge of patient safety and QI vocabulary and concepts.[6,7] Interns were also asked to read an article by Kavanagh and colleagues, which provided an example of a real-world application of QI methods in a pediatric care setting.[8] Simulation leaders interested in evaluating knowledge improvements after the simulation may choose to have learners complete an optional, open-book pretest using a single case from the Assessment of Quality Improvement Knowledge and Skills (AQIKS), which is a is a nine-question, pediatric case-based assessment published in MedEdPORTAL. The AQIKS asks learners to identify quality problems and conceptualize a QI intervention based on the principles outlined in the Model for Improvement, which specifically includes quality aims, driver diagrams, quality measures, and run charts.[9] The AQIKS has been shown to detect increases in knowledge and skills among pediatric residents exposed to a QI curriculum and was thus ideally suited as an evaluation tool for our curriculum. Our interns completed AQIKS Case 3. They did not receive scores or feedback about their performance prior to the simulation.

Equipment, Environment, and Personnel

All of the equipment, environmental needs, and personnel were comparable to those described by Worsham and colleagues' in their simulation instructions.[5] The materials listed below were updated to be relevant for this pediatrics case.

A room suitable for groups of four to six is needed, with sufficient space at tables for each group to work separately.

A projector is needed for the introduction slides.

One opaque envelope or folder is needed for each group.

Pens or dry erase markers are needed, depending on whether paper or laminated copies of the simulation materials described in Appendix A and Appendices D–P are provided to learners. Laminated materials can be reused for subsequent sessions.

The simulation leader guide and agenda (Appendix B) explains the preparation work for setting up the simulation and provides a time line for the simulation.

Appendices Q and C provide introductory materials for the simulation.

The QI simulation introduction (Appendix Q) is a PowerPoint presentation, which was adapted from Worsham and colleagues,[5] that facilitators can use to introduce the case and the basic structure of the simulation.

The tips for the QI facilitator document (Appendix C) provides facilitators with additional information and responses to commonly proposed interventions.

The small-group materials document (Appendix A), which should be provided to each team at beginning of the simulation, contains information for the groups to start the simulation.

The pediatric quality improvement virtual practicum document provides participants with a basic outline of how the simulation runs.

The meet the stakeholders document provides further information and definitions of the various stakeholders involved in the case.

The item menu lists the various data that teams can request. Teams choose multiple items and receive them all at once from the prix fixe menu. After completion of each plan-do-study-act (PDSA) cycle, teams may pick up additional items from the a la carte menu.

The run chart is used to track a group's data over the course of the simulation. The QI facilitator adds three months of data onto a group's chart after each PDSA cycle. Teams use that data to develop their next steps.

The process map is a basic outline of some of the relevant hospital processes. Groups that choose to map out a process can use this map as a starting point, as well as add additional information to the map.

The blank fishbone diagram is a tool used to systematically brainstorm reasons for the problem at hand.

The feasibility chart is given to the groups as a way to organize and prioritize their proposed interventions.

Menu items (Appendices D–P) are given to each group upon request, with one copy of each item per group kept in a folder until requested.

Appendix D contains the Gemba Walks document, which includes items 1A-1C.

Appendices E–O contain data.

Appendix E is Item 2A, which includes information on QI funding opportunities.

Appendix F is Item 2B, which includes information on available technological tools/solutions.

Appendix G is Item 2C, which includes staffing, unit, and census information for the pediatric emergency department (ED) at Best Medical Center (BMC), a fictional hospital.

Appendix H is Item 2D, which includes the BMC pediatric ED asthma pathway and follow-up guidelines.

Appendix I is Item 2E, which includes the asthma and discharge plan pathway.

Appendix J is Item 2F, which includes the asthma home visit collaborative options and referrals.

Appendix K is Item 2G, which includes the map of neighborhood clinics and clinic hours.

Appendix L is Item 2H, which includes patient housing development information and public transportation options to the hospital.

Appendix M is Item 2I, which includes the conversation with the pediatric ED electronic health record specialist about solutions related to the case.

Appendix N is Item 2J, which includes literature search results on asthma bounce backs.

Appendix O is Item 2K, which includes a thesis excerpt about a business case for a discharge medication delivery service.[10]

Appendix P contains correspondence emails from various stakeholders related to the case.

Item 3A is an email from the patient's parent.

Item 3B is an email from the patient's primary care doctor.

Item 3C is an email from the BMC division chief of pediatric emergency medicine.

Item 3D is an email from the follow-up nurse in the pediatric ED.

Item 3E is an email from a representative at a public health insurer.

Appendices R–T are optional tests and surveys.

Appendix R is the AQIKS Cases and Grading Scale[9] (the pretest from AQIKS Case 3 and the posttest from AQIKS Case 6).

Appendix S is the AQIKS Instructor Guide.[9]

Appendix T is the QI Simulation Feedback Survey.

The simulation leader gives the introduction presentation, briefs QI facilitators on their roles, keeps time, and assists the facilitators. This person may also act as a QI facilitator and the CEO, as described below.

There should be one QI facilitator for every 8–12 participants. The facilitators should be well-versed in basic QI principles, methods, and tools so that they can guide teams through the development of practical interventions that address the primary aim of the simulation, as described in Appendix C. Physician faculty, fellows, resident physicians, hospital project managers, and administrators have all been used successfully as facilitators.

The CEO is a QI facilitator who authorizes or denies expensive interventions proposed by the teams during the simulation.

Implementation

First, the setting and clinical scenario were presented to the large group via PowerPoint presentation (Appendix Q). The large group was then divided into small groups of four to six interns, with attention paid to dividing up interns with prior experience in QI. Each group received the small-group materials (Appendix A) and worked together to decide what to request from the menu, which included Gemba Walks (Appendix D), data (Appendices E–O), and correspondences (Appendix P). Given the volume of data available to each team and the known time constraints, teams were advised to divide up materials and then summarize main takeaways to one another. Teams then worked together using fishbone diagrams, process charts, and feasibility charts to prepare a pitch of their proposed first intervention to the QI facilitator. Importantly, using the divide and conquer strategy of data review, teams rarely ran out of time or were unprepared to pitch their idea(s); teams that were time-crunched received feedback to divide tasks in subsequent rounds. QI facilitators then asked clarifying questions and provided run chart data with verbal feedback representing what the facilitator felt would realistically happen if the pitched intervention were implemented. Over time, teams gathered new data (a maximum of one Gemba Walk, one correspondence, or one piece of datum per round) and refined and expanded upon their prior interventions. Teams made two additional pitches and received run chart data and feedback from the QI facilitator after each pitch. The 1.5- to 2-hour simulation was completed after each team finished three pitches (i.e., PDSA cycles).

Debriefing

After the simulation, the QI simulation leader led a 15- to 30-minute, large-group debrief with all participants that explored lessons learned from, challenges faced during, and reflections about the experience.

Assessment

Interns were asked to complete an anonymous, eight-question, postsimulation survey, which assessed residents’ attitudes about the simulation as well as the simulation's impact on the likelihood of their participation in future QI initiatives (Appendix T). Seven questions were scored on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). Qualitative feedback about the curriculum was gathered through an open-ended question at the end of the survey. The AQIKS pre- and posttest questions assessed resident knowledge and skills related to QI. Responses were de-identified and assigned a randomized study ID number and then graded independently by two of the three authors based on the AQIKS scoring rubric.[9] Grades were compared, and any discrepancies were resolved through discussion to arrive at a consensus final score. Scores were analyzed using a paired student's t-test to compare pre- and posttest performance.

All trainees in our residency program are required to complete an independent QI activity prior to graduating; this activity can include giving a morbidity and mortality (M&M) presentation at a departmental meeting, participating in an ongoing QI project, leading a novel QI initiative, or participating in an institutional house staff council. We analyzed the types of QI projects completed by simulation participants versus a historical comparison group of nonparticipants, all of whom were recent graduates. Importantly, since the simulation was held during the intern year, only a minority of simulation participants (n = 19) had reported their QI activity at the time of analysis. Nonetheless, this comparison offered an early glimpse into whether participation was associated with completion of more meaningful QI activities that required deeper, longitudinal involvement or leadership during residency. Resident projects were assigned a score of 1 (M&M), 2 (existing QI), 3 (novel QI initiative), or 4 (house staff council) in order of presumed increasing involvement. A chi-square test was used to assess whether the QI intervention impacted final QI projects.

Results

Eighty-two interns completed the QI simulation curriculum at the Boston Combined Residency Program (BCRP) between June 2015 and December 2017, of which 76 completed both pre- and posttests (93%). The total possible score was 28 points, with each of the nine questions counting for 1 to 5 points. The median AQIKS score increased by 3 points from 17 to 20 points (M = 2.6, SD = 0.6, p < .001). The score increase was driven primarily by improvements in responses related to two questions, one of which aimed at identifying primary and secondary drivers (question 3) and the other of which required generating a hypothetical run chart (question 8). Residents who scored in the lowest quartile on the pretest had a significantly larger mean score increase (M = 4.8) compared to residents who scored in the highest quartile on the pretest (M = 0.2, p < .001). The data are presented by quartile based on pretest scores in the Figure.

Figure.

Average change in AQIKS test score from the pretest to the posttest by quartile based on pretest scores.

Data obtained from resident responses to the postsimulation survey is presented in the Table. Overall, survey data indicated that interns initially lacked confidence in their abilities to lead a QI initiative, but after completing the simulation interns expressed that they had gained valuable experience in initiating a QI pilot. Interns also agreed that they were more likely to start their own QI initiative after the simulation. Qualitative feedback about the curriculum was positive. Some representative examples of comments from the survey include, “I thoroughly enjoyed this activity, as it introduced us to the world of QI and was an excellent forum to work as a team and share ideas,” and “I think that the format was great, getting to interact with real QI experts was phenomenal, and the information provided was awesome!”

Table.

Resident Responses to QI Simulation Feedback Survey

Data on independent QI projects were available for 158 residents, of which 19 had participated in the QI simulation. Most simulation participants were still residents and had not yet participated in QI projects at the time of data analysis, which resulted in the small sample size. Based on the current data set, simulation participants were not more likely to initiate a novel QI project or participate in a house staff council than nonparticipants. With our small sample size, there was no relationship between participation in the QI simulation and the level of involvement in future QI projects (p = .92).

Discussion

Engagement in QI allows residents to think critically about their health care environment and provides a reliable framework to make effective change. Delivering QI education in a timely, effective, and engaging way is a challenge pediatric residency programs must overcome. This study is an example of how educational tools can be adapted to specific learner populations while maintaining educational efficacy. Specifically, by adapting a previously developed simulation, we were able to teach QI principles in a way that was engaging and targeted toward the specific goals of pediatric residents, including the relationship between structural determinants of health and improved health care quality.

Our findings demonstrate that residents who completed the simulation had improved knowledge of QI principles, with residents who came into the exercise with the least QI knowledge showing the most improvement. Given that each question in the AQIKS focused on a critical, core QI concept, a 3-point median increase in scores represents an educationally significant increase in knowledge and/or skill. For example, a 3-point score increase represented whether a participant understood how to identify a root cause, track the success of an intervention in a run chart, or develop an optimal aim statement. Furthermore, residents noted that they were more confident in their ability to lead a QI initiative and indicated they were more likely to lead a QI initiative in the future. Together, these results indicate that this QI simulation may be a particularly good tool for those with less experience and baseline QI knowledge. Importantly, the small-group format allowed peers with more QI experience to lead and teach others through the process, which likely solidified their own QI competencies.

Resident feedback was critical to ensuring that the simulation was an effective teaching tool. While the vast majority of survey responses were positive and encouraged future use of the simulation, a few interns provided suggestions for improvement, including shortening the preparatory work and providing a summary handout with key takeaway points. Other suggestions included placing the simulation earlier in resident training to facilitate participant involvement in independent QI work at an earlier stage.

In assessing the residents’ independent QI projects, our study did not show that simulation participation had an impact on the quality of future QI projects. This was likely due to the small sample size, but another possible explanation is that the simulation does not lead to changed behavior over time. A single-day intervention may not be enough to change resident project choices over the three-year residency period, and a more longitudinal intervention could be more effective in influencing involvement in future QI projects. Factors that impact scholarly activities include mentorship and clinical experiences, which may be more important factors for residents in choosing projects. Future research should aim to delineate resident motivation in completing their independent QI activities to help inform interventions that can increase the depth of resident involvement in QI activities.

An important goal of this project was to show the potential for QI simulator adaptations to target a specific audience while maintaining the same positive impact on participants. This idea can be carried forward to a range of medical professionals interested in QI education. Future projects could focus on expanding QI simulations for various specialties and clinical environments.

Limitations

A limitation of this simulation is that QI facilitators must have enough basic training in QI to address resident questions and provide meaningful feedback on proposed interventions. Although the ideal situation is to recruit facilitators who have ongoing practical experience in QI, we have found that upper-level residents or chief residents with QI experience who have participated in this simulation at least once can also be excellent facilitators. The resources needed to complete the simulation, including time, space, and preparation, as well as the complex nature of the simulation and reliance on active participation by learners, are additional limitations. Educators may find success with integrating this simulation into their advocacy curricula, given QI is an excellent tool for creating meaningful improvements in care for patients. Additionally, we have found that comfort and experience with the simulation over time allow facilitators to be develop creative ways to ensure engagement with the material across all groups.

Conclusion

This educational tool is yet another example of how simulation can be successfully used to improve knowledge, skills, and attitudes related to QI among a population of pediatric trainees. Engaging trainees in exciting, hands-on, realistic, team-based learning through QI simulation has the potential to inspire participants to provide leadership in health care improvements over time.

Small-Group Materials.docx

Simulation Leader Guide and Agenda.docx

Tips for QI Facilitators.docx

Gemba Walks.docx

QI Funding Opportunities.docx

Available Technologies.docx

Staffing and Unit Information for Pedi ED.pdf

BMC Pedi ED Asthma Pathway.docx

Competing Hospital Asthma and Discharge Plan Pathway.docx

Asthma Home Visit Collaborative Referral Information.docx

Map of Neighborhood Clinics and Clinic Hours.docx

Housing Development and Public Transit.docx

Conversation with ED EMR Superuser.docx

Literature Search.docx

Discharge Medication Delivery.pdf

Correspondences.docx

Pediatric Virtual Practicum Introduction.pptx

AQIKS Cases and Grading Scale.pdf

AQIKS Instructor Guide.pdf

QI Simulation Feedback Survey.docx

All appendices are peer reviewed as integral parts of the Original Publication.