A Project-Based, Resident-Led Quality Improvement Curriculum Within a Pediatric Continuity Clinic.

Introduction: Participation in quality improvement (QI) projects is required of pediatric residents, and evidence-based medicine has highlighted the importance of providing residents with experiential practice in this realm. Embedding QI projects within a continuity clinic provides residents an opportunity for meaningful involvement in QI efforts.
Methods: A QI curriculum was implemented within a pediatric residency program that included an introductory lecture on QI principles and participation in resident-led, team-based QI projects at an outpatient clinic. Residents designed, implemented, and analyzed projects beginning in their intern year. Projects operated on an accelerated, 6-month time frame, allowing residents to complete multiple projects over the course of their residency. Resident QI knowledge was assessed before and after an introductory lecture with the Quality Improvement Knowledge Application Tool (QIKAT). Resident feedback was solicited 1 year following curriculum implementation via anonymous online surveys.
Results: Residents completed four QI projects that produced meaningful improvements in clinic processes and patient care. QIKAT scores significantly increased after the introductory lecture. Residents reported that the curriculum afforded them increased confidence to implement plan-do-study-act cycles and improve patient care in their future practices. Qualitative feedback highlighted the team-based structure, participation in multiple projects, and visible direct impacts on patient care as strengths of the curriculum. Increased involvement of clinic staff, scheduling concerns, and improved communication were areas for improvement. Discussion: Our model for integrating resident-led QI projects into an ambulatory clinic rotation is feasible and has been well received by residents and impactful on clinic processes and care.

Educational Objectives

After completing this curriculum, learners will be able to:

Understand basic quality improvement principles.

Identify areas for improvement within their clinic as potential quality improvement projects.

Design meaningful tests of change to address areas for improvement.

Implement iterative plan-do-study-act cycles to test changes within their clinic.

Analyze results of plan-do-study-act cycles using run charts.

Implement and spread successful changes in order to improve clinic processes and the quality of care delivered.

Introduction

Quality improvement (QI) has become an increasingly important domain within medical practice, but finding ways to meaningfully integrate QI education into residency training is easier said than done. The Accreditation Council for Graduate Medical Education (ACGME) has mandated that QI instruction and participation be included in residency education, requiring residents to “systematically analyze practice using quality improvement methods, and implement changes with the goal of practice improvement.”[1] Furthermore, the ACGME's Clinical Learning Environmental Review (CLER) includes patient safety and QI as two of the six domains that it examines in its assessment of resident and fellow learning environments.[2] However, several studies have shown wide variation in the content and degree of exposure to QI principles and participation in QI projects among residency programs, and early findings from the CLER have suggested that meaningful participation in QI is lacking in most training programs.[3-6] Studies have called for the implementation of QI curricula that involve more active participation of trainees in QI efforts and for more institutional support for such.[7,8]

Several residency programs have instituted longitudinal, project-based QI experiences that are interspersed throughout the year in educational conference time or conducted as stand-alone rotations.[9-12] Alternatively, since continuity clinics occur throughout residency, they also provide an opportune venue for resident participation in QI. A few studies have detailed individual programs' experiences in integrating a QI curriculum within their continuity clinic, each employing a slightly different programmatic structure.[13-15] In MedEdPORTAL, prior publications have described a longitudinal QI curriculum consisting of learning modules and guidance for completion of an individual QI project spanning each learner's residency,[16] a QI curriculum embedded in an ambulatory care rotation that culminates in development of a project proposal,[17] and two inpatient-focused QI curricula.[18,19] Others have provided didactic educational modules or facilitated sessions on QI topics[20,21] and an integrated curriculum that emphasizes the patient safety elements of QI and prepares residents to lead a morbidity, mortality, and improvement conference.[22]

Our curriculum is different from the previously described models in that residents participate in team-based QI projects in concentrated blocks while they are assigned to clinic months. Our curriculum adopts a blended didactic and hands-on approach to resident QI education but is novel in that, following a single introductory lecture, it guides residents to take more ownership for QI projects than most prior models. Residents take responsibility for planning, implementing, analyzing, and reporting results of QI projects starting immediately in their first year and continuing throughout their residency. Furthermore, these clinic-based projects operate on a more accelerated time line than in prior published curricula, thereby allowing residents to participate in more projects over the course of their residency. Our curriculum provides detailed guidance on each step of mentoring residents to complete such projects. Additionally, greater emphasis is placed on process improvement aspects of QI than in prior published curricula, including more details of data analysis, the process of hardwiring changes into a preexisting clinic structure, and the concept of spreading changes to other settings.

Methods

Participants

All 48 pediatric residents at the combined Wright State University/Wright-Patterson Medical Center (WPMC) Pediatric Residency Program participated in this QI curriculum. The residents were 50% civilian and 50% military, and all military residents were assigned to WPMC as their continuity clinic location. Residents attended their assigned continuity clinic 1 half-day per week, but all residents, both civilian and military, also rotated through the WPMC clinic for monthlong outpatient clinic blocks during all 3 years of their residency. Six to eight residents were usually on the clinic block and participating in the projects each month. Most residents spent 2–3 months at WPMC each year, so they participated in multiple QI projects throughout residency. All general pediatricians at WPMC were active-duty Air Force physicians. Medical students rotating through the WPMC pediatric clinic were invited but not required to participate in QI efforts.

Curriculum Structure

Introductory lecture

Beginning in 2015, residents participated in a 1-hour introductory lecture on basic principles of QI during their intern-year orientation (Appendix A; talking points for slides in Appendix B). During the first year of the curriculum, portions of the talk were also given to residents during morning lecture periods on their clinic blocks so that senior residents would also be exposed to the material. The introductory talk described the importance of QI to the medical field and focused on process improvement principles derived largely from the Institute for Healthcare Improvement's (IHI's) Open School. All slides were reviewed and approved by the IHI's content editor. The lecture also introduced the structure of the experiential portion of the QI curriculum in which the residents would be participating during their WPMC clinic blocks. Residents were given information on how to enroll in the IHI Open School to complete modules and explore these topics in greater depth. Other programs may wish to require some of these modules to improve the depth of resident training.

Resident-led QI projects

Clinic-based resident QI projects allowed residents to design, implement, analyze, and report on team-based QI projects of their own choosing over six 1-month time blocks, outlined as Blocks 1–6 below. The residents assigned to the WPMC clinic for the entire month, but not those coming in for just a continuity clinic half-day, were tasked with moving the project forward for that month. Continuity clinic residents coming in only for weekly half-day clinics were welcome to provide input on project efforts but were not assigned specific tasks for completion. Each month's team was assigned specific jobs in accordance with process improvement models, and the team members then handed off the work to the following month's residents at the end of their month. The process was ongoing: When one project was completed during Block 6, the next month's residents began anew at Block 1 with the planning of a new project. Only one project was conducted at a time so that all residents and clinic staff could focus on the targeted project.

During the first week of each month's rotation, the clinic residents participated in an hour-long, small-group, interactive planning session led by a faculty member trained in QI methodology who served as the mentor for the project. The faculty member reviewed progress on the current project to date, specifically outlined the steps to be completed for the month, and had residents assign themselves individual tasks to ensure fair division of labor. During the planning meeting, the faculty mentor strove to be a facilitator of resident discussion, as opposed to being directive. A second hour-long meeting was scheduled halfway through the month to review progress and provide additional working time. Residents had 30 minutes of administrative time each morning on their clinic rotation before morning lectures. They were able to use this time as needed to complete QI-related tasks but usually did not require more than 1–2 hours of time maximum per month to complete their work. Monthly tasks are outlined below. Worksheets to guide planning and recording of each block's tasks during the monthly planning sessions, as well as tips for facilitators specific to each block, are included in Appendix C.

Block 1 was entitled “Set an Aim and Establish Measures.” This first block of the project focused on planning efforts. The most important task to be accomplished during the month was for residents to identify an important, engaging, and feasible topic to target for improvement in the clinic. Residents were asked to brainstorm about inefficiencies, inconsistencies, deviations from guidelines, or concerns about care quality or safety that they had noticed in our clinic. Additionally, they were encouraged to compare processes in our clinic to those in others in which they had worked to identify potential areas of variation that could be improved. They also had access to some hospital-wide quality measures that were regularly reported (such as rates of spirometry among patients with asthma). The facilitator helped guide discussion to ensure that resident ideas would be appropriate and feasible given potential clinic-wide and larger institutional constraints. Once a topic was selected, residents were tasked to gather background information from medical literature (such as relevant evidence-based guidelines), query clinic staff regarding institutional background information (e.g., “Has this topic been addressed before in our clinic, and if so, what were the results?”), identify and meet with key stakeholders, and create a process map reflecting the current clinic flow. Finally, residents set a specific aim with measurable targets for improvement.

In Block 2, “Identify Changes and Collect Baseline Data,” the goals were to identify specific tests of change that could be trialed in subsequent months and to collect baseline data. Residents were directed to gather ideas from their own prior experiences, to seek information from key stakeholders on any similar prior interventions that had been tried or new ideas for change, and to analyze the current process map to identify potential points of intervention. For each potential test of change, they identified process and balancing measures, and they modified the outcome measure(s) as needed. Additionally, residents performed chart reviews and/or collected real-time data in clinic throughout the month to establish baseline measurements on the targeted topic.

Blocks 3 and 4 were “Test Changes” blocks. With many of the planning steps already completed, residents in Blocks 3 and 4 conducted the actual tests of change using the plan-do-study-act (PDSA) framework. They could select from changes proposed during the prior month but were also free to develop new ideas. Two months were devoted to this step since it was expected that multiple tests of change would be needed to learn from prior PDSA cycles in order to determine which interventions were most effective. A large emphasis was placed on the data-collection plan, and residents determined whether they would need to enlist other members of the clinic staff, such as nurses and medical technicians, to assist in the data-collection and recording efforts. Residents had to complete a minimum of one test of change per month, but they were encouraged to try to complete more than one PDSA cycle. Residents were reminded that more than one intervention could be tested simultaneously.

Of note, depending on the project aim and the specific tests of change proposed, this step of the process may need to be extended over more than 2 months. In such a case, the same framework could easily be used to continue working on PDSA cycles for 3 or 4 months (or more if needed) until the team was satisfied it had identified good interventions that could be scaled up and implemented on a larger scale for Block 5.

In Block 5, residents “Implement Changes.” Once several PDSA cycles had been completed, Block 5's residents analyzed results to date using run charts and decided which interventions should be adopted on a clinic-wide level. They focused on interventions that would embed changes within the normal clinic flow such that they were hardwired and therefore more likely to be sustainable. Importantly, residents worked with clinic leadership to ensure buy-in, monitor that changes were implemented as planned, and continue to collect data using the same collection tools as in Blocks 3 and 4 to verify that the outcomes were similar when changes were carried out on a large scale.

The final Block 6 was designed to “Spread Changes.” The testing phase of the project was now complete, and new processes were embedded within the regular clinic flow. Residents in Block 6 therefore analyzed and summarized findings to determine if there were ways to build upon the project—either within the clinic itself in spurring future QI efforts or more broadly through sharing lessons learned with other clinics within the hospital system. Residents performed a final analysis of run charts and put together a formal write-up of the project. At a minimum, residents created a PowerPoint presentation that was presented to the entire clinic at a monthly meeting, but they were also encouraged to share with larger audiences. To date, residents have also created abstracts and poster presentations and presented findings at local and regional meetings.

Faculty Training and Commitment

The primary faculty mentor for the QI projects obtained training in QI methodology through completing IHI Open School coursework and earning IHI certificates. Clinic leadership allowed the faculty mentor to block patient appointments as needed during the hour-long planning meeting and the mid-month follow-up meetings each month. On average, the faculty mentor spent 2 additional hours of work each month coordinating and following up resident efforts, and this time occurred during previously appointed administrative time. As faculty were active-duty military members, considerations such as financial compensation or full-time equivalents did not apply but could be pertinent in other practice settings.

Resident Assessment Using the Quality Improvement Knowledge Application Tool (QIKAT)

To assess residents' ability to apply QI principles to hypothetical clinical scenarios, interns completed a version of the QIKAT that had been adapted for pediatrics and validated in a prior study.[23] The QIKAT provided a clinical scenario and asked the learner to write an aim to target for improvement, a measurement, and a proposed test of change, with up to 2 points awarded for the aim, 1 point for the measurement, 1 point for the test of change, and 1 point for relatedness of the above answers, for a maximum of 5 points per scenario. Residents completed three scenarios as a pretest before the lecture and three scenarios afterward as a posttest. Participation in the QIKAT was voluntary and anonymous, but residents wrote an alphanumeric code of their choosing on the pre- and posttests so that responses could be matched by resident.

Resident Feedback

An anonymous online survey was sent to current residents and recently graduated residents 1 year following implementation of the new curriculum. Residents were asked about their exposure to QI in medical school and residency, their confidence in implementing QI methodologies in future practice, and their confidence in their ability to use QI principles to improve patient care. Additionally, they were asked for qualitative feedback on strengths and weaknesses of the QI curriculum. The survey was considered exempt by the Institutional Review Board of Wright-Patterson Medical Center.

Results

QI Projects

Four resident-led QI projects were completed within the first 2 years of implementation of the new curriculum. All residents from the 2015–2016 and 2016–2017 academic years participated in at least one of the projects (most participated in at least two or three), and all residents in the 2017–2018 year are continuing to participate in ongoing projects. Projects have focused on (1) the administration and documentation of the Ages and Stages Questionnaire and the Modified Checklist for Autism in Toddlers, Revised, both of which are standardized developmental screening questionnaires recommended to be given at specific well-child visits per the American Academy of Pediatrics; (2) the proper use and documentation of spirometry in children with asthma; (3) the identification and treatment of children with elevated blood pressure; and (4) the consistent administration and documentation of asthma action plans. All projects have yielded significant improvements in the process and quality of care delivered to clinic patients. Examples of specific interventions and outcomes from each project are outlined in the Table.

Table.

Key Interventions and Outcomes of Resident Quality Improvement Projects

Project	Key Interventions	Outcomes
ASQ/M-CHAT-R	• Resident education on AAP guidelines • Posting reminders in patient rooms • Changing process for distributing questionnaires, including them as packets along with check-in paperwork	• Proper administration and documentation of ASQ increased from 46% to 88% • Proper administration and documentation of M-CHAT-R increased from 38% to 77% • Coding for ASQ/M-CHAT-R increased from 9% to 83%
Spirometry	• Including spirometry as expected part of visit for all asthma follow-up visits and all well visits for children with asthma >6 years old • Techs obtaining spirometry prior to rooming patients • Establishing process for review of spirometry by pediatric pulmonologist • Establishing process for uploading spirometry into EMR • Obtaining new spirometry equipment (discovered old equipment was not pediatric-friendly and delivered unreliable results)	• Increased percentage of asthma patients who had spirometry obtained during asthma follow-up visits and well visits from 32% to 67% • Increased percentage of asthma patients with spirometry completed within the past year (key hospital metric) from 41% to 85% • Improved coding of spirometry from 64% to 71%
Blood Pressure	• Provider and technician education on AAP blood pressure guidelines • Posting blood pressure charts with percentiles based on age/height/gender in rooms • Techs including comment in vital signs section of EMR on whether blood pressure is normal or abnormal (since our EMR cannot flag abnormal values) • Techs automatically obtaining manual repeats of abnormal blood pressures	• Increased identification of abnormal blood pressure from 20% to 74% • Improved coding of abnormal blood pressure from 0% to 33% • Improved percentage of patients with elevated blood pressure who were given proper plan per AAP guidelines from 0% to 33%
Asthma Action Plans	• Provider education to give asthma action plan to patients at all visits in which asthma addressed, including well checks • Standardizing which version of an asthma action plan to use (multiple different versions used prior) • Standardizing process by which to upload asthma action plan into EMR	• Increased percentage of patients who received an asthma action plan at asthma follow-up visits and well checks from 37% to 44% • Increased percentage of patients with asthma who received as asthma action plan within the past year from 56% to 68% • Increased percentage of asthma action plans uploaded into EMR from 44% to 65%

Abbreviations: AAP, American Academy of Pediatrics; ASQ, Ages and Stages Questionnaire; EMR, electronic medical record; M-CHAT-R, Modified Checklist for Autism in Toddlers, Revised.

QIKAT Results

All 16 interns completed the QIKAT before and after the introductory QI lecture, and a two-tailed, paired t test was performed to assess the difference in matched resident pre- and posttest scores. Two independent reviewers scored each QIKAT, with scoring discrepancies resolved by consensus. The mean resident score was significantly higher on the posttest scenarios (11.31 ± 1.40) than the pretest scenarios (9.94 ± 1.39), p = .0018, t15 = 3.78.

Resident feedback on the new QI curriculum was largely positive. Forty-seven of 64 (73%) residents responded to the online survey. The majority of residents reported minimal or no exposure to QI prior to residency, including patient safety (50%), QI theory (67%), QI processes (67%), and involvement in any QI project (78%). Residents reported that the new curriculum increased their exposure to patient safety principles (60%), QI theory (80%), QI processes (80%), and involvement in QI projects (80%). Additionally, 50% felt confident or very confident that they could implement PDSA cycles on their own in future practice, and 67% were confident or very confident that participation in the QI curriculum would help them improve patient care in their practices.

Qualitative feedback revealed that common themes regarding strengths of the program included the clear delineation of tasks, the ability to be involved in multiple projects throughout residency, the team-based structure and division of labor that required relatively little additional work from each individual resident, the faculty mentorship and guidance, and the direct impact residents could see the projects having on clinic processes and patient care. Areas for improvement included more involvement of medical technicians and nurses in the planning process, difficulties in division of labor when certain residents were not in clinic for the entire month due to scheduling concerns (e.g., vacations, night coverage), better communication regarding QI project updates with residents who were not on a clinic month, and more dedicated time to work on the projects throughout the month. Based on this feedback, our clinic has included some additional protected time for QI work and review throughout the month (usually one additional morning lecture block per month), created a whiteboard with QI updates so current projects are better publicized throughout clinic, and started to send monthly QI project updates to all residents so that those not in clinic for the month can still be aware of current efforts.

Resident Scholarship

To date, results from our first four resident projects have been reported as an abstract and poster presentation at one regional and one international pediatric conference. Results from a subsequent project have been accepted for presentation at a national pediatrics conference later this year.

Discussion

Finding methods by which to meaningfully involve residents in QI efforts should be a key goal of residency programs. However, as all residency programs have different institutional priorities and limitations in terms of scheduling and implementing such activities, it is important to provide options to programs that wish to implement or improve QI curricula. Our experience highlights a model for integrating resident-led QI projects into an ambulatory clinic rotation that is feasible, is generally well received by residents, and can lead to meaningful improvements in clinic processes and patient care. A single introductory lecture that can be given during orientation provides an adequate foundation for residents with no prior background in QI to begin immediately participating in QI projects during their clinic month rotations, and the fast pace of the projects allows residents to participate in multiple projects throughout the course of their residency. The strong emphasis on resident ownership of these projects makes the process engaging, and to steal a principle from QI methodology, the embedding of the projects within the structure of the clinic rotation has become a hardwired part of the rotation in which all residents, attendings, and clinic staff expect to be participating.

Over the past 2 years, we have learned important lessons from experience and from resident feedback. In early iterations of the monthly planning meetings, we did not allot adequate time to clearly delineate roles among residents but instead asked them to divide tasks among themselves after the meeting. Consistently, we found this did not happen, and one or two motivated residents ended up taking responsibility for nearly all of the monthly tasks, with others not participating. Additionally, we did not initially schedule a mid-month follow-up meeting, and therefore, some tasks were not being completed as expected, with faculty not aware until the end of the month. We cannot emphasize enough the importance of ensuring that residents assign themselves specific tasks, that this division of labor is documented, and that at least one midpoint meeting to review progress is scheduled during the initial monthly planning. Communication with clinic staff, including leadership, nurses, and medical technicians, is also key to ensuring adequate buy-in and to eliciting their important feedback regarding tests of change. Finally, to promote resident ownership of the projects, faculty must resist the temptation to be overly directive throughout the process. Faculty knowledgeable in QI methodology and clinic procedures need to guide resident efforts, but residents themselves should be choosing the projects, designing the interventions, and leading the efforts to carry out the tests of change. Faculty must also resist the urge to fix problems that inevitably arise in order to allow residents to problem solve, even if that means letting an intervention fail. Guiding residents through the process of analyzing failures is not only in keeping with QI principles but also sends a strong message to residents that they are responsible for their project.

Limitations of this curriculum include that it is not designed to encompass all aspects of QI. While its strong focus on process improvement overlaps with other areas of QI such as patient safety, methodologies such as root cause analysis are not included. As such, this curriculum could be used in conjunction with other QI resources found on MedEdPORTAL, the IHI Open School, or elsewhere. It is possible that this structure, in which residents participate in the project for only 1 month at a time and at varied points throughout the year, could lead to residents feeling a lack of ownership over the projects or having knowledge gaps from certain phases of the projects in which they did not participate. We hope the fact that residents take such a high degree of responsibility for their steps of the project and that they are in clinic several months per year would mitigate these concerns, but such shortfalls are possible. Finally, this curriculum works well in our program, in which six to eight residents are generally assigned to clinic each month, but smaller programs or those in which residents are more widely dispersed over multiple clinic locations may not have enough residents in clinic at one time to expect them to accomplish such projects without more significant faculty involvement.

Future avenues of study could include following graduated residents longitudinally to see how often they lead or participate in QI efforts as attendings. Also, while this curriculum was implemented in a pediatric clinic, the model could be tested in other primary care settings or perhaps even adapted to subspecialty clinics. Lastly, since this curriculum does not cover all aspects of QI, future curricula could combine this structure with more didactic instruction and hands-on experiences in other realms of QI, such as root cause analysis.

A. Resident QI Curriculum Presentation.pptx

B. Talking Points.docx

C. Monthly Planning Meeting Templates and Tips.docx

All appendices are peer reviewed as integral parts of the Original Publication.