Promoting Quality Improvement in Primary Care Through a Longitudinal, Project-Based, Interprofessional Curriculum.

Introduction: Health professionals must demonstrate competencies in quality improvement (QI) and interprofessional (IP) practice. Yet few curricula are designed to address these competencies in an integrated, longitudinal way. Our experiential IP QI curriculum addresses this gap.
Methods: The IP QI curriculum was part of a San Francisco VA Health Care System training program for second-year internal medicine residents and adult gerontology primary care nurse practitioner students, pharmacy residents, and postdoctoral psychology fellows. Trainees worked in mentored IP teams to select, design, implement, evaluate, and present a project as part of a 9-month curriculum. Teaching methodologies included didactics and project-based skills application. Curriculum evaluation included trainees' QI knowledge and skills self-assessments, trainee satisfaction, mentor appraisals, and project results and impact assessments.
Results: From 2011-2012 to 2017-2018, 242 trainees completed the curriculum and 41 QI projects. Trainees reported high satisfaction with the introductory sessions (M = 4.4, SD = 0.7). They also reported improvement in comfort with QI knowledge and skills by the curriculum's completion. QI mentors (n = 23) observed growth in trainees' QI knowledge and skills, felt confident in trainees' ability to orchestrate a QI initiative, and believed their mentored QI projects added value to the organization. Thirty-eight projects resulted in system modifications. Discussion: This IP QI curriculum offers team-based, workplace experiences for trainees to learn and apply QI knowledge and skills. Leading factors for successful implementation included attention to team-building and faculty development. Challenges included reliably collecting evaluation data, accurately measuring ongoing systems changes, and variable trainee engagement.

Educational Objectives

Upon completion of the quality improvement (QI) curriculum, trainees will be able to:

Express confidence in applying QI processes and methods.

Demonstrate an ability to apply QI knowledge and skills.

Acknowledge contextual factors that can influence QI projects.

Work with colleagues from multiple professions to design, implement, and evaluate a QI project.

Introduction

As health care systems embrace quality improvement (QI) as a key element of practice, health professions educational programs aim to prepare trainees capable of participating in QI initiatives as part of their routine practice.[1-6] As a result, many programs include QI projects as part of a required curriculum during clinical placements.[7-11]

There is a potential added benefit in interprofessional (IP), experiential QI learning. The Accreditation Council for Graduate Medical Education (ACGME) Clinical Learning Environment Review program noted that “IP, team-based quality improvement efforts… provide residents… with experiential learning that goes beyond basic QI methods to include developing skills and behaviors in shared leadership, communications, systems-based thinking, change management, and professionalism.”[3] Despite this promoted ideal, most ACGME programs provide limited opportunities for residents to participate in IP QI teams.[12-17] While numerous QI curricula for internal medicine (IM) residents,[10,18-28] graduate nursing students,[29-33] and other health professional trainees[34-36] have been published, we identified only one longitudinal, project-based curriculum for IP trainees in a primary care setting.[37]

Comparable to the IP QI curriculum depicted by Hunt and colleagues, we created an experiential QI curriculum for primary care trainees that drew on best practices from profession-specific QI curricula,[1,12,18,21,37-47] and our experiences as QI and IP educators. Key attributes of our curriculum included being (1) authentic (provide opportunities to learn QI knowledge and skills with projects that address real workplace problems), (2) team based (trainees work in IP QI teams, often with clinic staff), (3) mentored (team mentors have expertise in QI and local clinic context), and (4) integrated (formal teaching of QI concepts and tools occurs alongside project work).

Our IP QI curriculum for IM and pharmacy residents, nurse practitioner (NP) students, and psychology fellows working in a primary care clinic was implemented and continuously improved over 7 years as part of the San Francisco VA Health Care System's (SFVAHCS) Center of Excellence in Primary Care Education (CoEPCE) program. This publication includes curriculum components, such as learning activities, curricular materials, strategies to involve stakeholders, evaluation methodologies, and related results, which to the best of our knowledge have not been published to date in such a comprehensive manner. Educators can apply this content to implement an IP QI curriculum for trainees and/or staff in primary care or other continuous care sites.

Methods

Curricular Context

QI was a core curricular domain of the SFVAHCS CoEPCE program, established in 2011 to teach IP trainees how to deliver team-based, patient-centered care.[11,48,49] Prior to and throughout the QI curriculum, trainees participated in several learning activities related to team-building and IP collaboration to help them in the development of skills (building relationships, establishing team goals, clarifying roles, and learning communication skills) that support successful QI teamwork.[2,11,38,50,51] IP teams of trainees, staff, and mentors performed QI activities in the setting in which they worked, the main academic primary care clinic or one of two community-based outpatient clinics.

Trainees were second-year adult gerontology primary care NP students, and second-year IM residents serving as primary care providers in their assigned clinics 3 half-days per week. IM residents rotated monthly between outpatient and inpatient rotations during which they did not participate in QI team activities. After 2 years, we began including first-year pharmacy residents and postdoctoral psychology fellows, who each worked 1 full day per week in their assigned clinic. Trainees had worked in the clinical setting from 0–15 months and generally had limited experience with QI.

Curriculum Development

Our curriculum was developed by the core QI leadership team composed of NP and MD co-directors, and an annual Quality and Safety IM chief resident and NP resident(s) with an interest in QI. Pharmacy and clinical psychology faculty also provided input. QI leadership team members and mentors had variable training and background in QI and IP teamwork. Most completed Institute for Healthcare Improvement (IHI) Open School online courses[42] and Lean yellow or green belt training[52] to attain a common set of knowledge and skills. The curriculum co-directors and two senior faculty mentors completed TeamSTEPPS training[53] to learn teamwork skills and principles. Other mentors were past graduates of our CoEPCE curriculum, had completed 1 year of the VA Quality Scholars program, or had completed the 1-week VA Quality and Safety Chief Resident boot camp.

Curriculum development followed our key attributes (authentic, team-based, mentored, and integrated) and evolved based on continuous feedback from trainees and faculty. We used a Logic Model (Appendix A) to plan and evaluate our work.[54-56]

Curriculum Overview

The formal curriculum took place over 9 months each academic year (2011–2012 through 2017–2018) beginning in September to allow almost all trainees to have at least 3 months experience working in the clinical setting. The curriculum highlighted concepts and related tools from the model for improvement[57] and Lean model for transformation.[58] The time line is presented in Appendix B.

In the first month of the curriculum, we introduced core QI concepts and skills in 1-hour sessions. With the exception of the first two sessions, the introductory sessions were separated by QI team and included time to apply QI methodology towards the trainees' selected project.

Introduction to QI: a 1-hour didactic overview of QI curriculum structure, QI methodology, and commonly used tools.

Performance Measures and Facility Quality Gaps: a 40-minute didactic and 20-minute facility data review to learn about quality measurement and identify potential project ideas.

Selecting a Project Theme: a 1-hour group discussion to determine project theme by completing an effort/yield analysis and drafting SMART aims for potential ideas.

Microsystem/Gap Analysis: brief didactic on process mapping and fishbone diagrams followed by 50 minutes of teamwork on initial drafts of each.

Approach to System Change: Teamwork and Plan-Do-Study-Act (PDSA) Cycles: team meeting using PDSA template as a guide to plan initial PDSA cycle (Appendix C).

We also recommended that trainees complete several IHI Open School online courses concurrently to reinforce key QI concepts.[42]

The curriculum leadership team assigned trainees to teams based on their clinical schedules and to maximize interprofessionalism. Activities to promote team formation and cohesion included establishing and observing ground rules and beginning all sessions with check-ins to share personal and professional background and interests.[59] Teams and mentors then met twice per month to work on project implementation. During these meetings, mentors emphasized the importance of collaborative QI teamwork and contextual factors (external environmental, organizational, infrastructural, microsystem, and QI team factors) that could influence QI success.[43,60-62] Learning strategies included group discussions, shared leadership (i.e., rotating roles of meeting facilitator, scribe, and timekeeper), debriefs regarding project progression, and team meeting synopses via minutes.[47,59] In the sixth month, the teams delivered work-in-progress presentations to their peers and mentors for the purpose of obtaining feedback to guide ongoing work. In the final months, the teams reported their work to date, lessons learned, and next steps for sustaining their work in a poster presentation. Since 2016, all teams also completed an abstract.

Resources

Our IP QI curriculum required the following resources:

People, responsibilities, and time:

The core IP QI leadership team met monthly for 30–60 minutes to plan for and review the curriculum. Each member spent an additional 2–4 hours monthly on curriculum planning.

Each of the six annual QI trainee teams had at least two project mentors. Each novice mentor (e.g., NP residents, Quality and Safety IM chief residents, and VA Quality Scholar fellows) was partnered with an experienced mentor. Some teams had more than two mentors due to recruitment of mentors for topic or method expertise. The average monthly mentor commitment was 4 hours: 2 hours of team meetings, a 1-hour mentor meeting, and 1 hour of consultation with team members or stakeholders.

A staff member with knowledge of data sources (e.g., facility data warehouse or data dashboard tools) retrieved and helped analyze data for the projects. Hours per month varied depending upon the project.

An administrative staff member assisted with scheduling, room reservations, preparing curriculum materials, IT support, poster production, and curriculum evaluation materials—on average, a 2-hour time commitment per month.

The estimated full-time equivalents of staff to organize and teach the curriculum are as follows:

MD lead: 0.1.

NP co-lead: 0.05.

Quality and Safety IM chief resident: 0.05.

QI mentors: 0.019/mentor (n = 12).

Administrative staff: 0.0125.

Analytic staff: 0.0125.

Materials:

Secure network drive for QI teams' shared work.

Lesson plans (Appendix C).

QI tools (Appendix D).

Potential project ideas (Appendix E).

Monthly QI newsletters (Appendix F).

Survey tools for curriculum evaluation (Appendices G–J).

PowerPoint presentations to augment the lesson plans (Appendices K and L).

Space:

Meeting rooms for QI teams.

Large conference room for all-team meetings.

Activities

The QI leadership team completed activities throughout the year to prepare, facilitate, deliver, and evaluate the curriculum. Two months prior to the start of the curriculum, leadership members recruited mentors and scheduled teaching sessions and team meetings. Leadership members identified institutional quality gaps and priority project themes (Appendix E) via review of facility performance data and input from clinic leadership and other key stakeholders. They prioritized potential project themes that had readily accessible data and/or invested stakeholders. Leadership members also designed and updated the curriculum schedule, lesson plans, and learning experiences (Appendices B and C) based on prior years' feedback and input from new members.

Throughout the curriculum, the leadership team coordinated and coached mentors and teams via monthly emails and mentor meetings. The newsletter emails (Appendix F) provided logistics and just-in-time QI education. In mentor meetings, mentors shared updates and received guidance from peers and leadership.

QI teams' formal 1-hour meetings occurred during the trainees' typical outpatient didactic time. The meeting format was coordinated by the mentors and predicated on work completed to date. In 2015, we increased the number of meetings from once to twice a month based on trainee and mentor feedback. Trainees also had up to 1.5 hours weekly to work on QI-related activities.

Evaluation Plan, Tools, and Analysis

The evaluation of our IP QI curriculum included (1) trainee satisfaction; (2) achievement of educational objectives; (3) impact on patient care, clinic systems, and stakeholders; and (4) QI mentor experiences and feedback. The Kirkpatrick four-level model of program outcomes informed the design of our evaluation approach.[54,63]

Trainees rated the overall quality of each introductory QI session using a 5-point scale (1 = poor, 5 = excellent) on an evaluation form collected at the end of each session (Appendix G). We calculated descriptive statistics (minimum, maximum, mean, standard deviation) for each item.

We evaluated the achievement of educational objectives via trainees' self-assessments (Appendix H), appraisal of team posters, and mentor perceptions of teams' performance (Appendix I). The pre/post self-assessment of trainees' comfort with QI processes and skills was a modified version of the Quality Assessment and Improvement Self-Assessment tool from the Quality Assessment and Improvement Curriculum Toolkit.[64] The original survey asked trainees to rate their comfort with 12 aspects of quality assessment and improvement using a 4-point Likert scale (1 = not at all, 2 = slightly, 3 = moderately, 4 = extremely). We added three items to assess attitudes toward QI, confidence in ability to make change, and perceived opportunity to apply QI skills, each rated on a 5-point scale (1 = strongly disagree, 5 = strongly agree). Trainees completed the self-assessment on day one of the curriculum and again at the end. We calculated descriptive statistics for each item and used paired samples t tests to compare pre/post self-assessment ratings for all trainees who completed both assessments. Authors JoAnne M. Saxe and Maya Dulay analyzed project posters to assess the trainees' ability to apply what they had learned, for example, to (1) document aim statements, PDSA cycles, data sources, measures, and tools; (2) highlight influencing contextual factors[43,62]; and (3) acknowledge IP collaboration.

In our project tracking record, we examined project impact by assessing improvements in processes and outcomes of care, as well as the products associated with each project (Appendix J). We assessed whether the projects were fully, partially, or not adopted (Appendix J) based on clinic medical directors' input on current clinic functions. We defined fully adopted projects as those where the systems change (e.g., process change, materials produced) implemented by the trainee team was sustained as of May 2019. Partially adopted projects included those in which some of the primary systems changes were adopted and other elements were abandoned and/or projects in which initial interventions were foundational but continued to be adapted by staff or other trainee teams. Not-adopted projects included those that did not result in systems improvements or were deemed too resource intensive to continue. Additionally, we obtained information regarding project impact from a mentor survey.

We designed the mentor survey (Appendix I) to collect feedback on our QI curriculum, including perceptions of trainees' achievement of educational objectives, and on the benefits, challenges, and impact of the curriculum on the clinic and the mentors themselves. Mentors used a 5-point scale (1 = strongly disagree, 5 = strongly agree) to rate each survey item. We administered the survey at the end of the 2017–2018 academic year via an online platform to mentors from 2016–2017 and/or 2017–2018. We calculated descriptive statistics for each item.

Results

Characteristics of Participating Students, Residents, Faculty, and Mentors

From academic years 2011–2012 to 2017–2018, 149 IM residents and 52 NP students completed the curriculum. Twenty-two pharmacy residents and 17 psychology fellows have completed the curriculum since 2013. Sixty-eight percent of teams were based at the main academic primary care clinic and 32% at one of two community-based clinics.

Twenty physicians (10 attending physicians, nine chief residents, and one senior resident), 27 NPs (including 17 NP residents), two pharmacists, and one psychologist served as QI mentors from 2011 to 2018. Based on the survey of 2016–2017 and 2017–2018 mentors (88% response rate, 23 out of 26), most mentors had formal training in QI (74%, n = 19). Fourteen mentors (54%) had 1 or 2 years of experience as a QI mentor; the rest had 3 or more years of experience (46%).

IP Trainee Satisfaction

Trainees rated the overall quality of QI sessions as very good to excellent (M = 4.4, SD = 0.7, n = 429, approximately 36% response rate based on five introductory sessions with 240 trainees). Trainees rated the match between the content discussed during the session and their learning needs being met as very good to excellent (M = 4.5, SD = 0.7, n = 112; asked only in 2016–2017 and 2017–2018; Table 1).

Table 1.

QI Curriculum Educational Objectives Mapped to Data Sources and Results

Achievement of Educational Objectives

We evaluated achievement of objectives via trainees' self-assessments (Tables 1 and 2), appraisals of the teams' posters (Table 1), and mentor perceptions of trainees' achievement of objectives (Tables 1 and 3).

Table 2.

Items From the Modified Quality Assessment and Improvement Self-Assessment

Table 3.

Results From QI Mentor Survey (n = 23)

Educational objective 1: express confidence in applying QI processes and methods

Sixty trainees (31 IM residents, 22 NP students, four pharmacy residents, and three psychology fellows) completed both the pre/post Quality Assessment and Improvement Self-Assessment surveys (25%, n = 60 out of 240; Tables 1 and 2). Trainees rated their comfort with all 12 aspects of quality assessment and improvement as significantly improved. On average, trainees indicated slight to moderate comfort with most items prior to the QI curriculum and moderate to extreme comfort at the end of the curriculum. Trainees' confidence in their ability to make a change to improve health care in a local setting increased significantly (M = 3.1, SD = 0.8, to M = 3.6, SD = 0.7).

Educational objective 2: demonstrate an ability to apply QI knowledge and skills

Trainee survey respondents noted that their team project provided them an opportunity to apply their QI skills (M = 4.0, SD = 0.8, n = 46). All teams were expected to document aim statements, PDSA cycles, process maps, and cause-and-effect (fishbone) diagrams in meeting minutes and/or posters, as well as to compose work-in-progress reports and posters. Posters reflected team members' ability to use common QI tools and display data. These included histograms (n = 24), Pareto charts (n = 2), pie charts (n = 17), process maps (n = 11), and run charts (n = 13). Data sources included VA performance dashboards (n = 18), locally developed surveys (n = 8), chart reviews (n = 17), and/or data extractions from the regional or national data warehouses (n = 10). More than half the teams referenced performance measures (e.g., percentage of patients with a glycohemoglobin less than 9% and 30-day hospital readmissions).

All teams demonstrated a broader application of QI knowledge and skills by presenting posters at a local conference. Additionally, five teams presented at regional meetings, eight presented at national meetings, and two published articles about their projects.[65,66]

The mentors surveyed also agreed that projects they mentored enhanced the trainees' knowledge and skills in QI (M = 4.2, SD = 0.8; Table 3).

Educational objective 3: acknowledge contextual factors that may influence QI projects

All teams (n = 41) acknowledged on their posters at least one contextual factor that favorably influenced or posed a significant barrier to project selection, implementation, and/or future sustainability. In Table 4, we highlight the most commonly identified contextual factors and their respective effects on QI projects.

Table 4.

Commonly Identified Contextual Factors and Influence on Projects

Educational objective 4: work with colleagues from multiple professions to design, implement, and evaluate a QI project

Our review of the 41 QI posters and team rosters showed that teams were multiprofessional, with a range of two to five different professions per team. Of the trainee professions represented on the teams, 100% included at least one MD resident, 95% NP trainee(s), 37% pharmacy resident(s), and 44% postdoctoral psychology fellow(s). Nineteen teams (46%) formally included staff as team members, including registered nurses, licensed vocational nurses, medical support assistants, pharmacists, and social workers. Staff typically did not have formal QI training. Seven teams identified professional diversity among team members as crucial factors for project outcomes. Our mentor survey showed that mentors felt confident that trainees could work with others to design, implement, and evaluate a QI initiative (M = 4.0, SD = 1.1; Table 3). One mentor commented, “It was great to see what a team could accomplish in a short amount of time. I appreciated being able to bear witness to the growth in leadership skills within my team.”

Impact on Patient Care, Clinic Systems, and Stakeholders

We evaluated impact on patient care, clinic systems, and key stakeholders (staff and mentors) through a review of the project tracking record (Appendix J) and mentor survey responses (Table 3). Thirty-eight IP teams' QI project work favorably influenced care processes (e.g., providing after-visit care summaries to patients, use of written opiate care agreements) that led to systems changes beyond the time frame of the formal curriculum. Forty-three percent of the projects' process changes and/or products were fully adopted, and 49% were partially adopted. Mentors favorably viewed the impact of the QI projects, endorsing the statement that the QI projects they mentored added value to the medical center (M = 4.4, SD = 0.7) and agreeing that clinic staff were contributing members to these projects (M = 4.0, SD = 1.1; Table 3).

QI Mentoring Experiences: Benefits and Challenges

Mentors indicated that team mentorship provided positive experiences, including deepening their QI and mentoring knowledge and skills (M = 4.3, SD = 0.8, for both) and getting to know trainees (M = 4.5, SD = 0.5) and staff better (M = 3.8, SD = 1.1). One mentor commented, “I loved working with an interdisciplinary team.” Another expressed appreciation for “affecting my workplace in a positive manner and changing my clinical practice based off QI findings.”

Mentors highlighted several challenges, such as finding the optimal level of involvement, particularly when mentors gave substantial guidance and had more direct involvement “to keep the project moving along.” Mentors also noted the complexity of involving staff as collaborative partners, balancing encouraging staff to “take action” without “taking away somewhat from trainees' learning and ownership of the QI project” and potentially “reducing trainee buy-in.” Mentors mentioned the IM resident schedules (alternating months in clinic) as sometimes challenging for project momentum and trainee engagement. Conflicts noted in team dynamics were noncollaborative behaviors, insufficient leadership, and difficulty ensuring equitable distribution of work among trainees.

Discussion

We designed an experiential, longitudinal (9-month) QI curriculum for IP teams working in a primary care setting. Our results from project data (41 projects completed over 7 years by 240 participants with two to five professions per team), trainee surveys (increased reported confidence in QI knowledge and skills), and positive perceptions of the curriculum by trainees and QI mentors indicate that the curriculum leads to trainees' successful attainment of educational objectives and systems-level improvements. Our reflections on the curriculum and the improvements made each year suggest several factors that contribute to the success of the curriculum as well as potential barriers others might encounter when attempting to deliver the curriculum elsewhere.

We aimed to give trainees realistic experiences to develop their QI and IP skills. Trainees, faculty, and mentors identified projects that aligned with gaps in clinic performance and leadership priorities. We also formally recruited staff to join the teams. The positives of staff involvement include diversity of perspective and expertise, involvement of potential process owners, identifying sustainable champions, and promoting the identity of staff as teachers. Trainees generally appreciated staff being part of the QI teams and felt their involvement favorably influenced project impact and sustainability. Programs adopting this model should be attentive to concerns about ownership of the project and how staff's involvement influences the trainees' learning experience.

Attention to the IP team learning environment also contributed to our QI projects' success and supported the development of trainees' IP competencies.[2] We integrated IP learning experiences in our introductory sessions and team meetings. Activities such as regular queries/check-ins about team members' role, interests, accomplishments, and challenges and mentor-guided IP teamwork were important for building cohesion and role clarity. A trainee-guided project selection process that led to the creation of SMART aim statements established clear team goals. Clearly stated ground rules, reflection on team function during meetings, and meeting minutes enhanced oral and written communication skills, respectively. We anticipate that these learning opportunities can be readily operationalized into other QI curricula.

Sites without an established IP training program can apply our curriculum to teams of physician residents and IP staff. Including staff who are also in a learning stage can have an additional sustained benefit to the clinical learning environment. Alternatively, a QI curriculum may be a key activity around which to build an IP teaching collaboration in a clinical setting that already hosts trainees of different professions. Attention to team-building and a safe learning climate may be especially important if working in an IP group is novel and team members have not formed relationships.[63,67,68]

Since our curriculum was embedded in a funded IP VA CoEPCE, we likely had more resources and access to data than typical training programs. Demonstrating the positive impact of the curriculum on patient care and developing QI expertise of faculty and staff could help others adopting this curriculum build a compelling argument to secure needed resources.[69]

We invested effort in faculty development for QI mentors, using multiple strategies to support effective mentorship and retention of experienced mentors. Our mentorship structure was purposely designed for each team to have more than one mentor, ideally from more than one profession. While using more faculty resources, this increased the opportunity for trainees to observe role models within their own profession and from other professions. The redundancy also helped teams avoid mentorship gaps due to planned and unplanned mentor absences from sessions throughout the year. Partnering junior faculty with senior faculty also built a more proficient group of mentors over time. QI team mentorship was a requirement for a number of junior mentors, specifically, NP residents and MD chief residents, which insured a reliable mentor pool. Junior mentors often had more time and bandwidth to support their teams between meetings, allowing senior mentors to focus on sharing their QI, IP, and education expertise during team sessions. Additionally, we recruited five NP residents to subsequently join our faculty, and they eventually became senior mentors. Attending MD faculty often had concurrent administrative roles, for example, in clinic leadership, and working on projects that aligned with responsibilities in their other role(s) was a win-win. Sites with limited faculty resources might consider designating a QI faculty champion from each profession instead of embedding multiple profession-specific faculty as team mentors. We used this model, for example, with pharmacy and pyschology faculty. These champions need not be QI content experts themselves. They can support trainees in their own profession by identifying profession-specific learning needs, promoting buy-in, and supporting conflict resolution.

Lastly, we created an integrated learning structure by teaching QI and team engagement content in sync with project work in a developmentally appropriate manner (Appendix C), which included access to IHI modules (average time for completion per module: 90 minutes) for trainees with less QI background to independently learn the basics via optional interactive online modules.[42]

While we appreciate the strengths and accomplishments of our curriculum, we do recognize its limitations. We have opportunities for ongoing improvement in the areas of trainee engagement, trainee assessment, curriculum evaluation, and measuring long-term impact.

Trainees' schedules and competing demands can pose challenges to engagement. When trainee involvement is reduced due to other training requirements and missed meetings, it is difficult to assure that each team member equitably and meaningfully contributes. Trainee absences may lead to more direct involvement by staff and mentors, thus making it difficult to distinguish the amount of input on a project from trainees versus mentors/staff. The staff may be able to more quickly advance an intervention because they are full time in the clinical setting. While this is overall advantageous, it can reduce trainees' buy-in and engagement if they have less personal responsibility in day-to-day project success. We found debriefing about project progression and reflecting on achievements and challenges faced by different team members during team meetings and in meeting minutes allowed those who did not directly participate in a specific project task to still learn from the activity. We also recognize the need to continuously seek feedback from trainees and mentors to address barriers to team engagement.

QI and IP competencies develop over time and at different rates for trainees depending on project progress, IP exposure, and teamwork. Our outcomes related to IP competency are at present indirectly measured through mentor assessment and project achievement (successful projects are likely associated with effective teamwork). We are also limited in our direct assessment of individual trainees' knowledge and skills beyond self-reported comfort and confidence levels. Thus, an individual trainee assessment process would improve our ability to gauge each participant's achievement of the learning objectives and, consequently, potentially better address the trainees' unique learning needs and influence team dynamics.

In order to minimize survey burden, our CoEPCE program focused IP competency assessments within our clinical teams rather than our QI teams. For this reason, a stand-alone IP QI curriculum evaluation should include a quality assessment and improvement tool (e.g., Quality Assessment and Improvement Self-Assessment tool[62]), as well as instruments to assess team functioning (e.g., the Team Development Measure[70]) and IP competencies (e.g., Interprofessional Education Collaborative Competency Self-Assessment survey[71]) in the context of QI work.

While our curriculum evaluation was multifaceted, the low survey response rates and reporting biases associated with surveys and posters limited our curricular assessment. Our program size and duration helped in part to offset these limitations. Analyzing posters may have underestimated QI skill acquisition since posters summarize and highlight specific aspects of a project but are not comprehensive of the whole endeavor. There may also have been a positive reporting bias, where authors were more likely to report their successful interventions and not include details of those PDSA cycles that were unsuccessful.

Almost all of the trainee QI projects reported positive outcomes and systems change by the end of the curriculum. Other than giving the project synopsis (Apprendix K), reporting these outcomes in detail was beyond the scope of this report as each project had its own context, methodology, and both qualitative and quantitative results. We found that long-term project impact was especialy difficult to codify and track, requiring careful planning and reliable systems to follow up on projects. As our curriculum is embedded within the complexities of the dynamic clinical systems with a culture of continuous QI, many of our trainee QI projects further evolved and were adapted once handed off to staff or another set of trainees. Some projects were no longer relevant given changes in clinical guidelines or institutional policies. At times, outcome data were collected in a labor-intensive manner that could not be sustained. Given these conditions, we could not report long-term outcome data and have relied on sustatined processes, products, and observable behavioral change as reported by clinical medical directors as evidence of long-term impact. These are important factors for IP QI educators to consider when designing and evaluating their QI curricula.

Conclusion

Our IP project-based QI curriculum offers trainees rich workplace learning experiences and, by focusing on local quality gaps, promotes care delivery changes and a culture of improvement in clinical learning environments. Key attributes include attention to building high-functioning QI teams, completion of scholarly projects, and regular inclusion of stakeholder feedback and assessment to guide curriculum adaptations. Our curriculum accomplishes the development of shared IP and QI competencies for health professionals and could be implemented in varied longitudinal inpatient or outpatient clinical training programs.

Logic Model for Interprofessional QI Curriculum.docx

QI Curriculum Schedule.docx

Lesson Plans.docx

QI Tools.docx

Sample of Potential Project Ideas.docx

QI Newsletters.docx

Trainee Evaluation of QI Introductory Sessions.docx

QI Self-Assessment Survey.docx

Mentor Survey.docx

QI Project Tracking Summary.docx

Intro to QI.pptx

Performance Measures.pptx

All appendices are peer reviewed as integral parts of the Original Publication.