Learning by Doing: Design and Evaluation of a Quality Improvement Curriculum for Pediatric Hospitalists.

Quality improvement (QI) is a core competency for Pediatric Hospital Medicine (PHM) and required for maintenance of certification, but many hospitalists lack QI training. This project set out to increase a PHM faculty's QI knowledge and comfort participating in QI projects, while concurrently applying the skills learned to a QI project in the hospital.
METHODS: We designed a 4-session curriculum utilizing principles of adult learning. Faculty immediately applied learned concepts to a QI project to increase the percentage of patients who were seen by an attending and billed for on the same day as admission to the PHM service. Attitudinal data and scores on the validated Quality Improvement Knowledge Application Tool- Revised knowledge assessment were compared precurriculum and postcurriculum. A manifest content analysis was carried out for qualitative questions.
RESULTS: Twenty faculty (83%) completed the preassessment; 15 (63%) completed the postassessment. Respondents showed statistically significant improvements in their perceived ability to participate in QI projects and their Quality Improvement Knowledge Application Tool- Revised scores. The group completed a QI project that increased revenue for the division. Faculty appreciated that the curriculum was applied to a real QI project and felt they would use the new skills in their daily practice. DISCUSSION: This curricular model based on adult learning theory, with immediate application to a real QI project, conclusively showed attitudinal, knowledge-based, and hospital system-level improvements, and was well received by faculty.

INTRODUCTION

Quality improvement (QI) is a core competency for pediatric hospitalists,[1] and meaningful participation in a QI project is a requirement for maintenance of certification (MOC) by the American Board of Pediatrics (ABP).[2] More importantly, QI is an essential skill for improving patient care and the efficiency of hospital systems.[3,4] Increasingly, QI is becoming recognized as a means for scholarship, administrative leadership, and academic promotion.[5] However, in a national survey of practicing pediatric hospitalists, few reported having formal QI training.[6] Sixty-five percent of respondents had an interest in conducting QI research. However, many identified a lack of mentorship as a barrier, and 90% stated they would find workshops on QI methodology to be helpful.[6] Another survey of recent Pediatric Hospital Medicine (PHM) fellowship graduates identified a similar perceived need for further QI training, despite the additional QI exposure during most fellowships.[7] Need for faculty training in QI is magnified at academic medical centers, where a lack of faculty mentors is a frequently cited barrier to teaching QI to residents and medical students.[3,4,8-10]

Much of the existing research on QI curricula focuses on teaching QI to medical students and residents.[11-15] One systematic review of QI curricula for clinicians identified only 6 studies focused on nontrainees that evaluated educational outcomes, with mixed results.[16] None of these studies utilized validated knowledge assessments. Several subsequent systematic reviews have centered exclusively on curricula for trainees.[8,17,18] The most recent systematic review of QI curricula that included studies of attending physicians called for more curricula that assess the application of QI principles to the real-life delivery of healthcare to patients.[19]

In the authors’ PHM division, many faculty members anecdotally expressed a lack of comfort with QI. Increasing the group’s QI involvement was a priority for divisional and departmental leadership. Knowing this, the authors set out to utilize adult learning principles and a peer coaching model to educate the group about the basic principles of QI while concurrently carrying out a QI project. The curriculum’s objectives were to (1) increase faculty members’ QI knowledge, as measured using a validated instrument; (2) increase faculty members’ confidence in participating in a QI project independently; and (3) successfully apply the QI skills learned from the curriculum to a real QI project in the hospital.

METHODS

Setting

The authors’ 26-member Division of PHM is based out of a 315-bed free-standing tertiary care center. The division staffs multiple service lines with house staff and medical students, as well as a direct-care service. All members of the division have academic appointments beginning at the assistant professor level.

Curricular Design

We delivered the 4-session curriculum every 6–8 weeks during an hour-long division-wide meeting over the noon hour. The curriculum utilized the Model of Improvement[20] as its QI framework, with curricular content based on the Institute for Healthcare Improvement Open School curriculum,[21] adapted with cases relevant to pediatric hospitalists, and additional content based on the authors’ QI experience. The authors, who had more QI experience than the rest of the group, delivered the curricular content. One author had recently completed the Intermediate Improvement Science Series through the James M. Anderson Center for Health Systems Excellence, and the other had experience as a project leader of multiple divisional QI projects and directs the QI working group for the division.

The session topics were (1) basics of QI and writing a SMART aim; (2) process mapping and failure mode effect analysis (FMEA); (3) Plan-Do-Study-Act cycles; and (4) run charts and data analysis. The sessions included 20–30 minutes of didactic content with the remainder of the hour reserved for small group activities, discussions, and other active learning forms. The curriculum was designed around principles of adult learning and utilized all nine of the principles of adult learning identified by Boonyasai et al[16] in their systematic review (Table 1).

Table 1.

Implementation of Principles of Adult Learning into a Curriculum to Teach QI to Pediatric Hospital Medicine Faculty

Principle of Adult Learning	Method of Implementation into Curriculum
Facilitates learners as active contributors	At least half of each session dedicated to non-didactic active learningLearners immediately apply session content to group QI project
Relates to learners’ current experiences	Examples relevant to pediatric hospitalists integrated into each didactic sessionLearners immediately apply session content to group QI project
Uses needs assessment to tailor curriculum	Writing a SMART Aim identified as area of weakness on preassessment; an emphasis to reinforce this topic at each session
Helps learners to self-direct learning	Working group allows opportunity for interested faculty to become more involved and receive more individualized coaching from faculty experts between quarterly curricular sessions
Allows learners to practice new skills	Learners immediately apply session content to group QI project
Offers faculty support during self-directed learning	Working group allows opportunity for interested faculty to become more involved and receive more individualized coaching from faculty experts between quarterly curricular sessions
Offers feedback	Group received real time data updates at each session, as well as during other faculty meetings not dedicated to curricular sessions, on progress towards achieving our group QI project’s SMART aim
Facilitates self-reflection	Homework assignments encouraged group to apply QI principles to their daily practice
Provides role modeling of behaviors	Each session provided coaching and application of principles by instructors

The curriculum aimed to provide learners the opportunity to apply the skills they had learned to a real QI project concurrently, providing spaced reinforcement of the material and illustrating how the content could be directly applied to a hospitalist’s daily work. To improve the care delivered to patients and the revenue generated for the division, the group chose a QI project focused on increasing the percentage of patients evaluated and billed for by an attending on the same calendar day as admission. To integrate the project and curriculum, we followed each curricular session with a “homework assignment” involving applying the skills learned during the session to the division QI project. For example, after the process mapping and FMEA session, each faculty member completed and submitted a simplified FMEA to identify interventions for the QI project. We created a voluntary 13-member “working group” to meet between curricular sessions and carry out the QI project. For example, the working group compiled all the FMEA responses submitted by the division and chose two interventions for the initial Plan-Do-Study-Act ramps.

The curriculum also utilized peer-coaching. Peer-coaching is a process of faculty development by which “faculty voluntarily assist each other within an atmosphere of collegial trust and candor.”[22] Experts suggest a peer-coaching model as a more effective means for faculty to translate learned skills to the workplace than traditional classroom teaching.,[23] Studies have shown that participants in peer-coaching find that learning from a trusted peer enhances comfort and safety.[22] A peer-coaching model seemed optimal for this project. Having the sessions run by peers of similar academic rank (as opposed to hospital administrators or external “QI experts”) made the process less intimidating and it role-modeled the application of the content to the faculty’s routine practice.

Assessment

An anonymous preassessment and postassessment was carried out using Qualtrics (Qualtrics, Provo, Utah). The preassessment was administered before the first curricular session and served as a needs-assessment. It assessed respondents’ demographic data, experience with QI education, and comfort participating in and leading a QI project using a 5-point Likert scale. The assessment also included the Quality Improvement Knowledge Application Tool- Revised (QIKAT-R), a validated instrument for assessing QI knowledge and application.[24] The QIKAT-R provides respondents with 3 scenarios based on real clinical practice. For each scenario, respondents develop an aim statement, identify measures for that aim, and propose an intervention to achieve their aim. The scenarios are scored based on a rubric with a maximum of 3 points for each component (aim, measure, and change), 9 points for each scenario, and a maximum total score of 27. There is no “passing score”; the tool is designed to be repeated as a preassessment and postassessment to show improvement in scores over time.

We administered the postassessment following the completion of all 4-curricular sessions. The postassessment was identical to the preassessment, except for using 3 different scenarios for the QIKAT-R and adding several qualitative questions regarding the curriculum.

QIKAT-R Scoring

The authors each independently assessed the same 10 preassessment QIKAT-Rs and achieved a Pearson’s coefficient of 0.6, which was similar to that achieved in the validation study of the QIKAT-R.[24] Differences in scoring between the authors on these initial 10 assessments were discussed and a final score agreed upon by both authors. The authors then divided the scoring of the remaining QIKAT-Rs between themselves, with each author scoring half of the remaining QIKAT-Rs independently.

Quantitative Statistical Analysis

All quantitative analysis was completed utilizing Microsoft Excel (Microsoft, Seattle, Wash.) statistical software. We compared preassessment and postassessment QIKAT-R scores using a dependent paired t test. Presurvey and postsurvey attitudinal changes were compared using a Wilcoxon Signed Rank test. Due to drop out between preassessment and postassessment, a bootstrapping analysis was utilized to balance the 2 sample sets to preform paired parametric and nonparametric testing. A Spearman’s Rank-Order Correlation was performed to look for a correlation between the number of self-reported curricular sessions attended and a respondent’s total QIKAT-R score.

Qualitative Analysis

For qualitative responses, a simple manifest qualitative content analysis was performed as described by Bengtsson.[25] Responses were reviewed by one of the authors and broken down into meaning units. The other author reviewed these results to limit bias and agreed that they accurately reflected the original data.

Ethical Considerations

This project was judged by the IRB to be quality improvement work and was exempted from human subjects review. The project was approved by the authors’ institution’s Quality Improvement Review Committee.

RESULTS

Participants

Out of 24 PHM faculty members in the division (the authors abstained from participating in any of the assessments), 20 (83%) completed the preassessment, and 15 (63%) completed the postassessment. Demographic data are summarized in Table 2. The majority of respondents were female, had been practicing as an attending for less than five years, had the academic rank of assistant professor, and had never participated in a QI curriculum previously. Ten of the 15 postassessment respondents had attended 3 or 4 of the 4 curricular sessions. The average attendance at the 4 sessions was 15.25 faculty members per session, with a range of 13–17 faculty members.

Table 2.

Demographic Data, Past QI Experience, and Attendance at QI Curricular Sessions by Pediatric Hospital Medicine Faculty

Question	Preassessment (n = 20)	Postassessment (n = 15)
Gender
Male	5 (25%)	3 (20%)
Female	15 (75%)	12 (80%)
Years in practice as an attending physician
0–5 y	11 (55%)	7 (47%)
6–10 y	2 (10%)	3 (20%)
11–20 y	3 (15%)	2 (13%)
>20 y	4 (20%)	3 (20%)
Academic rank
Assistant professor	14 (70%)	10 (67%)
Associate professor	5 (25%)	3 (20%)
Professor	1 (5%)	2 (13%)
Have you participated in a quality improvement educational curriculum previously?
Yes	9 (45%)	NA
No	11 (55%)	NA
No. QI curricular sessions attended (of 4 total sessions)
0	NA	0 (0%)
1	NA	1 (6%)
2	NA	4 (27%)
3	NA	7 (47%)
4	NA	3 (20%)
NA, not applicable.

Attitudinal Results

The attitudinal results are summarized in Table 3. Faculty members showed statistically significant improvements in their self-perceived ability to participate in or lead a QI project. All respondents reported being either extremely or somewhat satisfied with the QI curriculum.

Table 3.

Changes in Attitudes of Pediatric Hospital Medicine Faculty Following a QI Curriculum, as Measured Using a 5-point Likert Scale

Question	Preassessment (n = 20)	Postassessment (n = 15)
How comfortable are you with participating in a QI project?
Extremely or somewhat comfortable	7 (35%)	12 (80%)
Neither comfortable nor uncomfortable	6 (30%)	1 (7%)
Extremely or somewhat uncomfortable	7 (35%)	2 (13%)
Average score (1= extremely uncomfortable, 3=neutral, 5= extremely comfortable)	3.0	4.1*
How comfortable are you with leading a QI project?
Extremely or somewhat comfortable	3 (15%)	6 (40%)
Neither comfortable nor uncomfortable	3 (15%)	4 (27%)
Extremely or somewhat uncomfortable	14 (70%)	5 (33%)
Average score (1= extremely uncomfortable, 5= extremely comfortable)	2.15	3.1*
What was your overall satisfaction with the quality improvement curriculum?
Extremely satisfied	NA	13 (87%)
Somewhat satisfied	NA	2 (13%)
Neither satisfied nor unsatisfied	NA	0
Somewhat unsatisfied	NA	0
Extremely unsatisfied	NA	0

*P < 0.01, Wilcoxon Signed Rank test.

NA, not applicable.

QIKAT-R Results

Results of the QIKAT-R assessments are shown in Figure 1. Faculty members showed statistically significant improvements in their overall QIKAT-R scores, as well as in each component of the overall score. The faculty scored lowest on the aim component of the preassessment, and as such, writing a SMART aim was an area of focus for the curriculum. The aim component of the QIKAT-R showed the largest relative improvement when comparing the groups’ average preassessment and postassessment scores. There was a statistically significant correlation between a respondent’s self-reported number of curricular sessions attended and their total QIKAT-R score, with Spearman’s rs = 0.84 (P < 0.001), indicating a strong positive correlation.[26,27]

Fig. 1.

QIKAT-R total and component scores for a group of pediatric hospital medicine faculty show improvement after completion of a QI curriculum. *P value < 0.001, dependent two-sample t test.

While the QIKAT-R is not designed to have a passing score, the faculty members’ average score improved by 6.2 points from the preassessment to postassessment, representing a 38% improvement over the initial average score.

Qualitative Results

Several “categories” emerged from the qualitative survey responses and are summarized in Table 4. Faculty members enjoyed the group-based format, relevance to practice, and the ability to apply what they learned to the group-wide QI project immediately. Faculty desired more cases (similar to those tested on the QIKAT-R) to help with generalization of the principles they had learned and a way to access the educational materials if they could not attend the curricular session. Faculty reported that the curriculum increased their awareness that they were already doing QI in their day-to-day practice and should consider being more systematic and formally studying their efforts.

Table 4.

Categories, with Representative Quotes, from a Manifest Content Analysis of Pediatric Hospital Medicine Faculty’s Qualitative Responses about Their Experience with a QI Curriculum

Question/Category	Representative Quote(s)
What did you like most about the QI teaching curriculum?
Direct, real-time applications of didactic sessions to group wide-QI project	“[I] Liked that we did a QI project as part of the curriculum—[it] made it easier to understand with the real-life example.”“Doing a project together made it more real.”
Group-based format	“I enjoyed getting to learn and practice our skills as a group.”
Relevance to daily practice	“[The curriculum] Used real life examples pertinent to [our] daily practice.”
What would you change about the QI teaching curriculum?
More repetition, featuring cases in different settings	“I would appreciate more examples to work through, like in the [QI-KATR] assessment questions. It was great to focus on this one project, but I find it helpful to apply this thought exercise in different settings and would have appreciated the feedback and guidance from our QI experts in working through these [other scenarios].”“More sessions and practice sessions.”
Accessibility options for those who cannot make it in person	“Off-hour physicians have trouble reliably attending noon conference. Videotaping the sessions may be helpful and could be later referenced if a QI question comes up.”“Would want to have the slides posted maybe on our shared group drive…. When [I] miss a session, have slides available and [the] chance to ask questions after reviewing them.”
After participating in the QI teaching curriculum, what has changed in your day-to-day practice of medicine?
Increased awareness of applications of QI to day-to-day practice, and that this represents an opportunity for scholarship	“It helps to see that we our doing QI in our daily practice, and therefore the opportunity for academic productivity opens up once you have seen the example set with this project.”“Be mindful when making a change to consider studying the impacts.”
Increased awareness of the system in which we practice	“Now that I better understand quality improvement I feel as though I better understand what changes can be made to workflow, clinical practice, etc.”
Increased desire to participate/lead QI	“[I am] more comfortable participating in QI projects, [and I plan on] looking for more projects to participate in.”

QI Project

The group was successful in increasing the percentage of patients evaluated and billed for by an attending on the same calendar day as admission from an initial baseline median of 55% of patients to a new median of 71% following our QI interventions. We tracked this data using a run chart, and the improvement represented a shift of the centerline, qualifying as special cause variation. There were no other system-wide changes or divisional interventions, outside of the QI project, which would account for these improvements. The group also followed the average revenue generated from attending billing charges per hospitalization in dollars, seeing a 75% increase from a baseline median of $400 to just over $700 per hospitalization. These data were also tracked using a run chart and represented a shift of the centerline, qualifying as special cause variation.

The author’s hospital is a MOC Pediatric Portfolio Sponsor through the ABP. All members of the division were deemed eligible for MOC Type 4 (Quality Improvement) from the ABP for their participation in the curriculum and project. Sixteen faculty members (62%) have received MOC Type 4 credit for their participation in this project.

DISCUSSION

Faculty development in QI is essential to provide the best care possible to patients, maintain certification, and enable those who work at academic centers to teach this critical topic to trainees. There is a lack of published studies of QI curricula targeting faculty; those published seldom use validated knowledge assessments or evaluate outcomes at the hospital system or patient level.[16,19] This curriculum was successful in increasing faculty’s QI knowledge, as measured by a validated assessment, and in improving faculty’s comfort leading and participating in QI projects. There was a strong positive correlation between the number of sessions attended by a given respondent and their total QIKAT-R score, suggesting a dose-response effect from the curriculum. The group was also able to complete a QI project of strategic importance to the division, increasing the percentage of patients evaluated by an attending on the same calendar day as their admission and increasing divisional revenue. All members of the division were eligible for MOC 4 credit from the ABP for their participation, which anecdotally is one of the more challenging and time-intensive components of MOC for faculty to achieve.

Starr et al[19] identify the importance of assessing higher-level Kirkpatrick[28] learning outcomes when evaluating QI curricula. This curriculum showed improvements at level 1 (learner satisfaction), level 2a (attitudes/perceptions), level 2b (acquisition of skills and knowledge), and level 4 (change in organizational practice and benefits to patients/clients). Faculty reported plans to become more involved in QI initiatives and projects in the qualitative data, which would be a level 3 outcome (behavioral changes). The authors plan to track group members’ participation in QI projects in the future to evaluate this outcome further.

Some of the identified best practices (and challenges) of teaching QI to medical students and residents also proved applicable to this project focused on faculty. The utilization of well-established adult learning techniques is crucial to the success of a curriculum and learner engagement. This curriculum utilized all of the adult learning techniques identified in a previous systematic review,[16] and the qualitative analysis identified the ability to apply what was learned in each session to the group QI project, as well as to faculty’s day-to-day practice in general, as significant strengths of the curriculum.

Finding time to fit QI education into clinical demands is a challenge common to both faculty and trainees. By building the curriculum into already scheduled time for faculty meetings, this curriculum maximized attendance while not infringing on time for patient care. However, it is a fact of hospital medicine that some faculty members will often be working off-hour shifts or may have direct patient care responsibilities that prevent attendance at group meetings. The survey respondents expressed the desire for alternate accessibility options in their qualitative comments, such as videotaping sessions or posting the materials to a shared hard drive, so that those who could not attend in person could still experience the curriculum. These recommendations are an essential consideration to build into curricula targeting faculty, specifically hospitalists.

This study has limitations. First, this is a single-center study. The findings in this study may not be generalizable to other institutions. However, the groups’ general demographics and lack of prior experience with QI are similar to previously published national surveysr of PHM faculty. Second, there was a dropout of respondents between the preassessment and postassessment. This dropout group may represent faculty members who did not participate in the curriculum; it is also possible that those respondents who did not complete the post-assessment were those faculty members who did not find the curriculum to be helpful, thus skewing the results to appear more favorable. Utilizing a de-identified study ID to allow for matching of individual preassessments and postassessments would allow better characterization of curricular effectiveness and drop-out in future studies while maintaining respondent anonymity. Third, the study did not utilize a control group.

Given that about half of the sample from each assessment had already been in practice for at least 5 years, it seems unlikely that the improvement in QI skills was due to a maturation effect, but we cannot state this definitively. The authors are unaware of any other system-wide QI education that was occurring during the project. The strong correlation between the number of self-reported curricular sessions attended and the final QIKAT-R score would suggest that the curriculum played a significant part in these improvements. However, correlation is not synonymous with causation. Fourth, the level 4 Kirkpatrick outcome was a process-based hospital-level improvement, as opposed to a patient-level improvement. This initial iteration of the curriculum focused on a project to increase billing for the PHM division. The authors felt this provided the best potential to have a successful project, and solidify basic QI principles for the learners, as this was a process that the PHM division had primary control over. However, this led to learners having limited exposure to some essential QI principles, such as interdisciplinary collaboration and engaging outside stakeholders. These areas will be points of specific focus for future iterations of the curriculum. A more patient-focused outcome may also prove even more salient to learners.

This curriculum provides a model for faculty development in QI that would likely be successful at other institutions and in other disciplines. By making the curriculum focused on adult learning principles, tied to a QI project of organizational significance, and based in a peer-coaching model, this project was able to engage and educate faculty on QI successfully. The goal is that faculty members will become more active in QI going forward, and in so doing, provide opportunities for mentorship of residents and medical students. To encourage this, the next iteration of the curriculum will allow time in each session to have members of the group present their QI works in progress and solicit feedback and peer-coaching from the group. In addition to giving faculty a chance to showcase their QI efforts, this will provide more opportunities for spaced repetition of QI theory in different contexts. The authors believe this model will build a culture that supports QI as a means for the academic and professional advancement of pediatric hospitalists, improves the care provided to patients, and role models to trainees that QI is a part of every physician’s job.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.

ACKNOWLEDGMENTS

The authors would like to acknowledge the other members of the Paul C. Gaffney Division of Pediatric Hospital Medicine at UPMC Children’s Hospital of Pittsburgh for their support and enthusiasm throughout this project.