Pediatric Resident Engagement With an Online Critical Care Curriculum During the Intensive Care Rotation.

OBJECTIVES: Residents are often assigned online learning materials as part of blended learning models, superimposed on other patient care and learning demands. Data that describe the time patterns of when residents interact with online learning materials during the ICU rotation are lacking. We describe resident engagement with assigned online curricula related to time of day and ICU clinical schedules, using website activity data.
DESIGN: Prospective cohort study examining curriculum completion data and cross-referencing timestamps for pre- and posttest attempts with resident schedules to determine the hours that they accessed the curriculum and whether or not they were scheduled for clinical duty. Residents at each site were cohorted based on two differing clinical schedules-extended duration (>24 hr) versus shorter (maximum 16 hr) shifts.
SETTING: Two large academic children's hospitals.
SUBJECTS: Pediatric residents rotating in the PICU from July 2013 to June 2017.
INTERVENTIONS: None.
MEASUREMENTS AND MAIN RESULTS: One-hundred and fifty-seven pediatric residents participated in the study. The majority of residents (106/157; 68%) completed the curriculum, with no statistically significant association between overall curriculum completion and schedule cohort at either site. Residents made more test attempts at nighttime between 6 PM and 6 AM (1,824/2,828; 64%) regardless of whether they were scheduled for clinical duty. Approximately two thirds of test attempts (1,785/2,828; 63%) occurred when residents were not scheduled to work, regardless of time of day. Forty-two percent of all test attempts (1,199/2,828) occurred between 6 PM and 6 AM while off-duty, with 12% (342/2,828) occurring between midnight and 6 AM.
CONCLUSIONS: Residents rotating in the ICU completed online learning materials mainly during nighttime and off-duty hours, including usage between midnight and 6 AM while off-duty. Increasing nighttime and off-duty workload may have implications for educational design and trainee wellness, particularly during busy, acute clinical rotations, and warrants further examination.

In busy clinical rotations such as the ICU, patients, diagnoses, and clinical acuity vary between rotations, and limited time and competing demands are common for both trainees and faculty (1). Supplementing the ICU rotation experience with a blended online educational curriculum may help improve knowledge and ensure consistent exposure to core content (2), but these resources are often superimposed on the demands of patient care and other educational experiences (3). For our ICU residents, we designed curricula that included short videos with pre- and posttests and hypothesized that residents would use these materials most frequently during breaks in patient care while on clinical duty. However, there is a lack of previously published data that describe when and to what extent resident physicians complete online materials during ICU rotations. Such understanding would help inform decisions about how to best implement educational interventions for residents rotating in the demanding, high-acuity clinical setting of the ICU.

MATERIALS AND METHODS

We conducted a prospective cohort study examining junior (post-graduate year 2) resident use of online curricula during their first rotation in medical-surgical PICUs at two large academic children’s hospitals (Boston Children’s Hospital and Cincinnati Children’s Hospital) that are similar in size, patient population, and resident scope of responsibility. This study was approved by the Institutional Review Boards at both sites. ICU resident rotation directors created individualized curricula for each site that covered core concepts in pediatric critical care medicine. Both sites also provided in-person educational experiences (bedside teaching, didactic lectures, and manikin-based simulations). Site 1 delivered in-person education between 07:30 and 08:15 every weekday, and Site 2 did so between 12:15 and 13:00 every Monday through Thursday. We administered the online curricula from July 2013 to June 2015 at Site 1 and June 2015 to June 2017 at Site 2. Neither site provided protected time for curriculum completion. Although residents were informed of the expectation to complete curricula by the end of their ICU rotation, there were no formal consequences for failing to complete. At both sites, residents were e-mailed 2 weeks before their rotation and instructed to complete the curriculum before the end of their ICU rotation, with e-mail reminders provided at rotation weeks 2, 3, and 4. Residents were excluded from the time-of-use analysis if their daily schedule data were unavailable.

Curricula were delivered on OPENPediatrics (www.openpediatrics.org), which is based at Site 1. Each lesson contained a pretest, video, and posttest. The curriculum contained 18 lessons at Site 1 and 21 lessons at Site 2. Individual lesson videos ranged in length from 4 to 36 minutes (average length 18 min). The total curriculum video duration at Site 1 was 5 hours, 12 minutes and at Site 2 was 6 hours, 50 minutes. The website required strictly linear progress; a pretest, then video, then posttest for each lesson needed to be completed before a resident could progress to the next lesson. Residents were only able to take the pretest once, but posttests could be attempted multiple times until the minimum passing score (≥80%) was achieved. The platform allowed residents to stop and restart within pre- or posttests, as well as within videos, if they did not complete a given item in one sitting.

During the study interval, residents worked within two different clinical schedules as part of a concurrent trial of resident physician work hours Randomized Order Safety Trial Evaluating Resident Schedules (4), where each schedule operated for 1 year of clinical rotations at each site. One schedule cohort involved daytime and nighttime work shifts limited to a maximum of 16 hours of duration, whereas the second involved traditional extended duration (>24 hr) work shifts, with daytime shifts alternating with extended duration work shifts every fourth night. On average, residents worked about 10% more hours per week on the extended duration work schedule (5). Throughout this article, we refer to these differing schedule cohorts as “short call” and “long call”.

We collected curriculum completion data for each resident and timestamps for every pre- and the first posttest attempt that occurred during the ICU rotation and in the 14 days preceding. We did not include test attempts occurring prior to the ICU rotation in the time-of-use analysis due to the significant heterogeneity in resident clinical rotations immediately prior to the ICU rotation. We only included the first posttest attempt to avoid over-representing a given time of day if a user attempted a posttest multiple times. Video viewing activity is captured only in aggregate, deidentified fashion on the platform, so individual video view timestamps were not available for specific residents. For test attempts during the ICU rotation, we cross-referenced timestamps with each resident’s schedule to determine whether or not they were scheduled for clinical service in the hospital. We calculated frequencies and percentages for resident and site characteristics and compared data between cohorts and between sites using chi-square tests of independence using a significance level of 0.05. Data were analyzed using Stata/SE 13.1 (StataCorp, College Station, TX) and Microsoft Excel (Microsoft Corp., Washington, DC).

RESULTS

During the study, 157 residents rotated through the ICU for the first time, and 92% (144/157) accessed the curriculum. Fifty-three percent of residents (83/157) accessed the curriculum during the ICU rotation at least once while on duty. Seven residents at Site 2 accessed the curriculum during the rotation but did not have daily schedule data available for analysis. We included 137 residents (137/157; 87%) in the time-of-use analysis. There were no statistically significant differences in demographic characteristics (gender or residency track) between the two sites or between the schedule cohorts within each site.

Curriculum Completion

Sixty-eight percent of residents (106/157) completed the curriculum (Table ). A greater percentage of residents completed the curriculum at Site 2 (53/60; 88%) compared with Site 1 (53/84; 63%), p value of less than 0.001. There was no statistically significant association between overall curriculum completion and schedule cohort at either site.

Time-of-Use Analysis

We included 2,828 test attempts by 137 residents from the two sites in the time-of-use analysis. Of the test attempts made during clinical duty, 40% (418/1,043) occurred during daytime shifts versus 60% (625/1,043) at night. Approximately two thirds of test attempts (1,785/2,828; 63%) occurred when residents were not scheduled to work, regardless of time of day. Approximately two thirds of all test attempts (1,824/2,828; 64%) occurred during nighttime hours (between 6 pm and 6 am), regardless of work status (Fig. ) (Supplemental Table 1, Supplemental Digital Content 1, http://links.lww.com/PCC/B389).

Time of use analyses. A, Distribution of test attempts during clinical rotation by hour of day and by on-duty versus off-duty time status. n = 2,828 test attempts. B, Distribution of test attempts in relation to day of rotation (range day –14 to +31). n = 3,176 test attempts (includes 348 attempts prior to ICU rotation).

Forty-two percent of all test attempts (1,199/2,828) occurred during nighttime hours when the residents were not scheduled to work. Nearly a quarter of all test attempts (647/2,828; 23%) occurred between midnight and 6 am, with 12% (342/2,828) of all test attempts occurring between midnight and 6 am while residents were not scheduled to work.

We observed an association between time-of-use patterns and schedule cohort at Site 1 but not at Site 2. At Site 1, residents in the long call cohort used the curriculum more during nighttime hours (428/553; 77%) compared with residents in the short call cohort (427/769; 56%), p value of less than 0.001. The long call cohort also used the curriculum more during times when they were not scheduled to work (461/553; 83%) compared with the short call cohort (512/769; 67%), p value of less than 0.001.

Figure displays the distribution of test attempts by day of rotation, ranging from 14 days prior to the start of the rotation to 31 days after the start date. For test attempts within the ICU rotation, a greater proportion occurred in the second half of the rotation (1,679/2,828; 60%) versus the first half (1,149/2,828; 40%). Three hundred and forty-eight additional test attempts occurred prior to the start of the rotation, the majority of which (264/348; 76%) were in the 2 days immediately prior.

DISCUSSION

Using 4 years of timestamped online learning data from two large pediatric residency programs, we found that the majority of residents in our study accessed a supplemental online curriculum during the ICU rotation. However, despite designing the ICU curriculum to include short lessons that could be completed during breaks from clinical work while on duty, only half of the included residents accessed the curriculum during clinical periods, and they frequently chose to use the materials during nighttime hours and when not scheduled to work clinically. Notably, 12% of test attempts occurred between midnight and 6 am while residents were not scheduled to work. Test attempts were made immediately prior to, and throughout the rotation, with a majority occurring in the second half of the rotation.

The inconsistent association of completion rate or time of use with schedule cohort makes it less likely that the type of schedule is a main contributor to online curriculum engagement. Because both sites provided in-person educational experiences during most workdays in addition to the online curriculum, it is possible that residents were biased against completing online materials while on-duty, since other materials were already being presented during on-duty hours. Other studies have supported the notion that medical learners often prefer online learning to be supplementary to in-person learning experiences (6, 7). The greater proportion of on-duty use at night compared with during the daytime may be a consequence of the greater volume of clinical care demands requiring resident attention during the day, including but not limited to rounds and scheduled admissions. The greater number of test attempts in the second half of the rotation may reflect residents catching up on incomplete lessons before the end of the rotation or may be related to residents focusing on getting comfortable in the clinical environment before turning their attention to self-directed learning.

Several studies have highlighted successful implementation and outcomes of online medical learning, noting improvements in knowledge (8) and perceived utility and satisfaction from clinicians and instructors (9). However, although residents have always incorporated self-directed learning at night and during off hours, blended learning models that increase trainees’ obligatory nonclinical workload outside of dedicated educational time may have a different impact. Despite our intention to provide short lessons that would provide education during breaks from patient care, residents accessed the curriculum more often during nonclinical hours. Previous reports have commented on the need to consider the distinct time constraints and service-education task conflicts in graduate medical education (10) and on the risk of creating information overload when educational content is shifted to the online environment (11). Increases in workload added to preexisting stressors of the clinical learning environment can contribute to resident physician burnout (12, 13) and sleep deprivation. Sleep deficiency is known to adversely impact resident clinical performance (14) and increases risk of physical harm, such as motor vehicle crashes (15) and needlestick/sharps injuries (16). Therefore, program and rotation directors of busy, inpatient rotations may need to consider alternative approaches to implementing online learning, such as incorporating protected time to complete curricula; making the curriculum optional or controlling the volume of content shifted to the asynchronous, self-directed setting (11, 17, 18).

Our study has several important limitations. As this was a purely an observational study, we did not qualitatively assess resident motivations for why they accessed the curricula at the times they did nor did we formally survey participant satisfaction with the curriculum or any potential impact on their wellness. These are important areas that warrant further investigation. Additionally, not all residents completed the online curricula, and rates of curriculum completion differed between the two sites despite having identical procedures to encourage completion. This may reflect differences in willingness to engage with online curricula during clinical rotations between residents and between the two sites, which may have led to a sampling bias. Interestingly, despite the fact that OPENPediatrics (https://www.openpediatrics.org/) is primarily based at Site 1, curriculum completion rate was lower at that site, suggesting that there was not increased pressure among residents to use the platform due to the shared institutional affiliation. Although we have no reason to suspect differences in resident roles or workload between the sites, we did not directly assess this either. Curricular length has also been described as a barrier to curriculum completion (19, 20); however, we observed higher completion rates by residents at the site with the longer curriculum.

CONCLUSIONS

Our data show that residents will engage with online learning materials during and immediately prior to their ICU rotation but do so most often at nighttime and when off-duty, with a portion of use occurring during midnight and 6 am while off-duty. This may have implications for resident well-being, including sleep, personal life disruptions, and/or burnout, and warrants further examination. Program and rotation directors will need further guidance on how best to implement blended learning models in busy clinical rotations, such as the ICU.

ACKNOWLEDGMENTS

We wish to thank the residents at Boston Children’s Hospital and Cincinnati Children’s Hospital for their participation in this study, as well as the administrative staff at both hospitals and at OPENPediatrics who provided coordination and support for this study.

TABLE 1.

Resident Learning Tracks and Curriculum Completion Rates by Cohort and Site

Study Participant Category	Site 1			Site 2			Overall
	Short Call, n (%)	Long Call, n (%)	p	Short Call, n (%)	Long Call, n (%)	p	Overall, n (%)	p (between sites)
Total residents, n	48	49	—	32	28	—	157	—
Residency track
Categorical	40 (83)	42 (86)	0.75	23 (73)	21 (75)	0.79	126 (80)	0.09
Medicine-Pediatrics	4 (9)	4 (8)	0.98	3 (9)	2 (7)	0.76	13 (8)	0.98
Pediatric-Neurology	3 (6)	2 (4)	0.63	3 (9)	3 (11)	0.86	11 (7)	0.25
Other combined tracksa	1 (2)	1 (2)	0.99	3 (9)	2 (7)	0.76	7 (5)	0.06
Curriculum access and completion
Accessed curriculum	41 (85)	43 (88)	0.73	32 (100)	28 (100)	—	144 (92)	0.003
Completed curriculum overall	29 (60)	24 (49)	0.26	30 (94)	23 (82)	0.16	106 (68)	< 0.001
Missing schedule data	0	0	—	0	7 (25)	0.003	7 (4)	0.001
Included in time-of-use analysis	41 (85)	43 (88)	0.74	32 (100)	21 (75)	0.003	137 (87)	0.75

aPediatrics-Anesthesia, Pediatrics-Genetics, Pediatrics-Physical Medicine and Rehabilitation, Pediatrics-Psychiatry-Child Psychiatry.

Dashes indicate a test of significance was not run for these items.