Improving the Resident Educational Experience in a Level IV Neonatal/Infant Intensive Care Unit.

The neonatal/infant intensive care unit (N/IICU) at the Children's Hospital of Philadelphia is a 98-bed, level IV unit through which second-year pediatric residents rotate monthly. We developed a quality improvement project to improve the resident educational experience using goal setting. Primary objectives were to increase resident educational goal identification to 65% and goal achievement to 85% by June 2017. Secondary objectives were to (1) increase in-person feedback from fellows and/or attendings to 90% by June 2017 and (2) sustain improvements through June 2018.
METHODS: The quality improvement team developed a driver diagram and administered a baseline survey to 48 residents who had rotated through the N/IICU in the 18 months before the project. Plan-Do-Study-Act cycles targeted project awareness and trialing of 3 different methods to elicit goals and track feedback, from July 2016 through June 2018.
RESULTS: The baseline survey response rate was 52% (n = 25). Among 60 rotating residents, the median resident-reported rate of goal achievement increased from 37.5% to 50%, and residents receiving in-person feedback increased from 25% to 50%. Of the 63% (n = 38) of residents who participated in data collection, goal identification and achievement increased from 38% to 100% between academic year 2016 and academic year 2017, and in-person feedback increased from 24% to 82%.
CONCLUSIONS: Instituting a goal-setting framework for residents during their N/IICU rotation increased goal achievement and in-person feedback. Consistent resident participation in postrotation data collection made measuring project outcomes challenging. These data support goal-oriented learning as an approach to enhance learner engagement and improve goal achievement.

INTRODUCTION

Resident education on clinical rotations is a core component of training across medical specialties. During rotations, residents interact directly with patients and attending physicians allowing for hands-on learning and real-time feedback. These opportunities are especially important in the intensive care unit, which poses unique challenges in resident education, given patient acuity, volume, complex multidisciplinary teams, and associated emotional demands.[1] Currently, pediatric residents are required to have exposure to 2 educational units (4 week or 1 month block) of neonatal intensive care during training, limiting resident exposure, clinical experience, and education in neonatal care.[2] These challenges impact educational opportunities for residents and are compounded by significant reductions in the number of hours residents work in the neonatal intensive care unit annually.[3,4]

Examples of published studies of resident-focused educational interventions have included the use of team-based learning strategies,[5] frameworks for teaching communication skills,[6] and identification of the barriers and enablers to effective feedback.[7] Team-based learning is an essential educational improvement strategy in graduate medical education, which is associated with improved resident educational engagement and increased satisfaction.[5,8] The communication skill-based educational framework described by Back et al[6] highlights the importance of integrating intentional goal setting with learners. Similarly, an interested educator (ie, fellow) is an essential factor in the resident educational experience.[9] Feedback is critical to reinforce and improve resident performance and is a requirement of the Accreditation Council for Graduate Medical Education (ACGME).[10] A qualitative study of residents across various subspecialties identified 5 major themes for useful feedback, including the educational context. These themes may extend to different locations of care or resident rotations (eg, outpatient clinic or intensive care unit).[7] Though these studies highlight educational engagement themes, they have not adequately quantified the effect of interventions targeting these essential concepts.

In our neonatal/infant intensive care unit (N/IICU), we perceived that the combination of patient acuity, volume, work hours, and the educational structure did not meet the educational needs of rotating residents. We hypothesized that identifying and communicating goals would improve goal achievement and facilitate feedback for residents. Our goals for this initiative were to facilitate the identification and achievement of N/IICU-specific learning objectives and to increase rates of in-person feedback both midway and at the end of the rotation. The specific aim was to increase N/IICU rotating resident goal identification to 65%, goal achievement to 85%, and in-person feedback from a fellow and/or attending to 90% over 12 months (July 2016 to June 2017). We aimed to sustain these improvements for the following academic year (AY; July 2017 to June 2018).

METHODS

Context

This project was conducted at the Children’s Hospital of Philadelphia, an urban academic 494-bed hospital, from July 2016 to June 2018. The N/IICU is a quaternary, 98-bed unit with over 1,200 admissions annually. At the time of this project, 2 to 3 residents rotated each month through the N/IICU. The clinical team composition changed every 4 weeks for residents and fellows and every 2 weeks for attending physicians. We assembled a dedicated quality improvement (QI) team of fellows, attendings, a QI supervisor, and interested residents within the Division of Neonatology to optimize resident education in the Children’s Hospital of Philadelphia N/IICU through the design and implementation of a QI project. Institutional Review Board approval was not required for this QI effort.

We designed this project to create a framework for eliciting, addressing, and providing feedback on the individual resident’s goals. Before this project, resident evaluation and assessment were completed using the ACGME milestones through an online portal. Although encouraged, direct in-person feedback to the resident was not required. We conducted a needs assessment with residents who had rotated through the N/IICU in the prior 18 months. We asked residents if they had identified and achieved their educational goals during their N/IICU rotation (Supplemental Digital Content, which shows Needs Assessment Survey Items, http://links.lww.com/PQ9/A216). We compiled the baseline needs assessment survey data and reported results to the QI team and Division of Neonatology. These data informed preintervention numbers and assessments of change postintervention. We used the baseline survey to develop our driver diagram and inform the Plan-Do-Study-Act (PDSA) cycles (Fig. 1).

Fig. 1.

Driver diagram.

Interventions

Through our PDSA interventions, we standardized the approach to goal identification, which started with asking all residents to identify 3 different goals. Our interventions to standardize educational goal identification and documentation are detailed in Table 1, specifically interventions 2, 3, and 4A–4E. We also worked to standardize feedback timing, which started with the addition of midrotation and end-of-rotation feedback encounters. Midrotation feedback was added to allow residents to apply feedback while still rotating in the N/IICU. Interventions 2 and 4A–4D in Table 1 describe feedback related changes in detail.

Table 1.

PDSA Interventions

Intervention	Change
1. Involvement of Fellow-run Teaching Committee	Assess and identify the teaching and learning opportunities for Neonatology fellows, specifically as they pertained to the residents who rotated through the CHOP N/IICU.
2. Presentation at Division of Neonatology faculty meetings	At the beginning of the study the results of the baseline survey of residents who had rotated through the unit were presented to faculty, we highlighted resident-reported rates of goal identification, achievement, and feedback. Throughout the project, the QI team provided regular updates to the faculty on new interventions, presented the data, and solicited additional feedback and suggestions.
3. Resident orientation	Resident orientation to the NICU rotation was transitioned from an attending to a fellow responsibility. A written orientation outline was developed and provided to the fellows. The orientation content was changed and the outline included prompts for fellows to help residents discuss, identify, share, and document 3 educational goals on the first day of the rotation. Goals were shared with the fellow and attending.
4A. “Goal Card” (written educational goal identification)	The front side of the goal card provided a space for residents to write down their identified learning objectives for the rotation as well as other topics of interest. The back side of the card provided space for residents to indicate which of their goals they did or did not achieve as well as list factors that facilitated or hindered goal achievement. There was also a place to indicate the receipt of midpoint and end-of-rotation feedback.
4B. Recording and discussion of goals and feedback	To standardize and promote discussion of resident goals and stimulate feedback and completion of the back side of the goal card, we sent email reminders to the residents, fellows, and attendings on the team at the beginning, middle, and end of the rotation.
4C. "The Whiteboard” (visible educational goals)	A white dry erase board in the resident workroom provided a common and visible location where residents with the fellows and attendings would write educational goals for the rotation. This replaced the goal card and served as a visual reminder of resident goals for all team members. The whiteboard also included an area to indicate the completion of midrotation and end-of-rotation feedback.
4D. Electronic postrotation survey	The paper-based goal card was replaced with a Qualtrics (free electronic platform) survey to collect resident goal identification, achievement, and feedback data. This intervention also included a 3-question survey for fellows and attendings regarding the use and utility of the whiteboard.
4E. Midrotation goal update	Based on attending feedback, a follow-up intervention to the introduction of the whiteboard included having a conversation to update resident goals on the whiteboard midway through the rotation, which coincided with a new attending taking over the team.

CHOP, Children’s Hospital of Philadelphia.

We designed our PDSA interventions to target the different drivers identified by our QI team. These interventions included:

involvement of the Fellow-run Teaching Committee;

discussion of the project at the Division of Neonatology faculty meetings;

changes in the resident rotation orientation; and

several cycles focused on eliciting and recording goals and feedback (interventions 4A-4E) (Table 1).

With the introduction of the electronic postrotation survey for residents (intervention 4D), we initiated a survey of fellow and attending physicians with questions regarding the implementation of “the whiteboard” (intervention 4C, a white dry erase board mounted in the resident workroom). These questions included (1) “Was the whiteboard used?” (2) “What were the obstacles to using the whiteboard?” and (3) “What was useful about the whiteboard?” The survey data were integrated into our PDSA interventions. The various PDSA cycles resulted in the use of a white dry erase board mounted in the resident workroom to record goals and document feedback and an electronic postrotation survey to track project measures.

Measures

The 3 primary measures were goal identification, goal achievement, and in-person feedback. Goal identification was defined as goals that were written by the resident (either on the goal card or “the whiteboard”) during the rotation. It was strongly encouraged to write down goals on the first day of the rotation. In discussion with the fellow, goals were identified based on the individual resident’s clinical interests and desired areas for growth. Goal achievement was based on the resident’s perception of goal achievement and was self-reported. We measured receipt of in-person feedback by resident self-report, including the frequency of feedback over the rotation (midpoint and/or end of the rotation). Receiving feedback at least once during the rotation was counted as a receipt of in-person feedback. Ongoing assessment of the educational experience and goals concerning the QI project was the fellow’s responsibility, with some input from the attending physician.

Analysis

We used run charts to determine if quantitative changes had been made in the primary measures over time.[11] Given the frequency of data (2 to 3 rotating residents per month), we monitored the progress of the interventions and data completeness every 2 to 3 months (ie, every 6 residents). We discussed potential changes among the QI team and the Fellow-run Teaching Committee. We assessed progress using data collected from the goal cards, the whiteboard, and the electronic postrotation survey. The number of residents who rotated through the N/IICU, regardless of participation in data collection (via the goal card or electronic survey), was recorded. There were no concurrent educational interventions or changes in the N/IICU rotation structure during this project.

RESULTS

The total possible number of baseline survey participants were the 48 residents who rotated through the N/IICU in the 18 months before the project. Of these residents, 52% (n = 25) completed the survey. Of all residents who received the survey, 42% (n = 20) had identified goals (5 specifically reported not identifying goals); 19% (n = 9) discussed their goals with a fellow or attending physician; 42% (n = 20) achieved their goals, and 33% (n = 16) received feedback. These percentages served as the preproject baseline values for 2015–2016. Looking only at the 25 residents who completed the survey: 80% (n = 20) identified goals, 36% (n = 9) discussed their goals with a fellow or attending physician; 80% (n = 20) achieved their goals, and 64% (n = 16) received in-person feedback at least once during the rotation.

Throughout the QI project, 60 residents rotated through the N/IICU, and 38 actively participated in data collection. There was no change in the rate of goal identification among all rotating residents (Fig. 2); however, the median rate of goal achievement increased from 37.5% to 50% (Fig. 3), and in-person feedback increased from 25% to 50% (Fig. 4). Including residents who completed a goal card or the electronic postrotation survey (66% in the AY 2016 and 61% in AY 2017) suggests that goal identification may promote more robust goal achievement. Both goal identification and achievement increased from 38% to 100% between AY 2016 and AY 2017, and in-person feedback increased from 24% to 82%.

Fig. 2.

Run chart of residents identifying educational goals, July 2016 to June 2018. Each resident group/point represents 6 residents.

Fig. 3.

Run chart of residents achieving educational goals, July 2016 to June 2018. Each resident group/point represents 6 residents.

Fig. 4.

Run chart of residents receiving in-person feedback, July 2016 to June 2018 (2015–2016 was the period of rotating residents included in the baseline survey). Each resident group/point represents 6 residents.

Of residents who provided data via the electronic survey during AY 2017, 100% (n = 17) identified educational goals, 88% (n = 15) had a conversation about their goals at the beginning of the rotation; 71% (n = 12) discussed their goals with both the fellow and attending physician, and 88% (n = 15) achieved their objectives most or all of the time. Based on the postrotation survey data, the whiteboard was used by 65% of residents on the first day and was used during the rotation by all but 2 residents. Eighty-two percent (n = 14) received in-person feedback, 58% (n = 10) only midway rotation feedback, 24% (n = 4) both midway and end, 6% (n = 1) only end, 63% (n = 10) received feedback from both fellow and attending.

The most commonly identified reason residents reported achieving their learning objectives was support from the fellow. In contrast, the most common barriers to achieving educational goals were lack of time and opportunity (often related to procedural goals dependent on patient clinical need and condition).

DISCUSSION

Summary

Engaging residents in educational goal setting is 1 approach that appears to be an effective way to increase resident goal achievement and provides a framework that increases in-person feedback. Before initiating this QI project, the average resident rating (on a scale of 1 to 5) of the emphasis placed on resident education during the N/IICU rotation was 4.55, which increased to 4.71 in the final year of the project. Similarly, the receipt of constructive feedback increased from 4.0 to 4.46. These data support better educational experiences for rotating residents. Based on the rates of goal identification, achievement, and in-person feedback in the subgroup of residents who completed the goal card or postrotation electronic survey and annual resident ratings of the rotation, we improved the educational experience of residents in a quaternary, academic N/IICU.

The ACGME Pediatrics Milestones, which outline core competencies for pediatric residents, include competencies that are relevant to goal identification and achievement: (1) identifying strengths, deficiencies, and limits in one’s knowledge and expertise and (2) to identify and perform appropriate learning activities to guide personal and professional development.[12] These milestones encourage residents to develop insight into their individualized educational needs and goals and to formulate plans to achieve them as independent, adult learners. Our baseline survey highlighted the residents’ educational priorities, which were consistently related to their fund of knowledge, followed by procedural goals, and then communication-based goals. Although goals based on “fund of knowledge” are likely to vary based on each resident’s preexisting knowledge, this type of goal is likely achievable using patient cases on the resident’s team, informal or formal lectures, simulation (if applicable), related articles, and studies or other web-based resources. Procedure-based goals, outside of simulated experiences, may be harder to achieve. For example, in the case of endotracheal intubation, the number of intubations performed in large academic centers by residents in the delivery room or N/IICU and pediatric intensive care unit is relatively low and decreasing, respectively.[13,14] The opportunity to achieve specific communication goals (eg, leading family meetings, delivering bad news) is likely related to the type of patients the resident cares for during their rotation and appropriateness of opportunities for such learning experiences.

The denominator for our run charts was all residents who rotated through the N/IICU. The discrepancy between our run chart denominator and project objectives based on responses to the needs assessment created optimistic and challenging goals for identification, achievement, and feedback. This difference explains the rationale behind our primary aim described in our driver diagram and the low rates of the primary measures relative to the reported baseline rates shown in the run charts.

Our hypothesis suggests that the increased frequency of in-person feedback may be secondary to the rotating resident’s identified goals and an associated opportunity to provide targeted and direct feedback. However, consistent delivery of in-person feedback was a challenge, as evident in the 12 consecutive residents depicted in the run chart who did not receive or did not report receiving in-person feedback during the rotation. The provision of feedback is a challenge and a separate area of research. Faculty feedback is often low quality. They are reluctant to provide constructive feedback, and they dominate conversations.[15] Challenges with data collection, especially related to feedback, contributed to low rates of recorded in-person feedback. This problem was first evident after reviewing the goal card PDSA cycle, during which the front of the goal card was completed much more frequently than the back, where we collected feedback data.

Limitations and Strengths

One of the limitations of this study is the lack of data from all residents who rotated through the N/IICU before and during the project period. Surveys, including our baseline survey, carry an inherent risk of recall and sampling bias, which may have biased our baseline estimates. The primary source of missing data during the project period was limited use of and compliance with the data collection tools, resulting in measurement bias. There were multiple efforts made to create succinct tools, provide timely reminders, and change tool formats. Attempts were made to address and improve resident participation throughout the study, through standardization of rotation orientation and email reminders, which increased in frequency throughout the project, and the introduction of the whiteboard.

The emphasis on individualized educational goals creates significant variation between the types of goals residents identify (eg, fund of knowledge versus procedural) and successfully achieve. The data regarding what type of goal residents did or did not achieve is not available, and differences between residents (eg, career aspirations) are unknown, which may impact the internal validity of the findings.

The PDSA interventions did not have a measurable or sustainable change on goal identification. Although much of the qualitative feedback received regarding the whiteboard was favorable in supporting a culture of goal identification, this was not measurable. Additionally, attending physicians liked updating goals midrotation. However, it did not translate into a quantifiable increase in goal identification. Other interventions to improve goal identification and documentation are needed to improve this measure.

As with many QI projects, the study context is essential to consider. We developed this project in response to resident needs in a quaternary N/IICU at a large academic hospital. Thus, this approach may not produce similar results in other contexts. One of the primary strengths of this QI project is the generalizability of the educational principles applied and the focus on previously identified educational themes in graduate medical education. The principles of goal-setting and feedback apply to a variety of educational settings in medicine and beyond.

CONCLUSIONS

This QI project demonstrated that implementing an individualized, low-cost educational initiative in a busy, acute setting is feasible and has measurable benefits for learners. Residents and other learners can apply this goal-setting framework and associated principles to other rotations or educational settings to increase the success of their educational experience.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.

ACKNOWLEDGMENTS

The authors would like to acknowledge Jeannie Carroll for guidance in the organization of this QI project and her contribution to the organization and review of the manuscript, Brenna Aredas and Ali Chandler for their contributions in manuscript editing and preparation, and all the participating residents, fellows, and attendings neonatologists in the Children’s Hospital of Philadelphia N/IICU.