Return to the Reading Room: Implementation of a Hybrid Radiology Clerkship Model after Emergent Conversion to Remote Learning in the COVID-19 Pandemic.

RATIONALE AND
OBJECTIVES: The COVID-19 pandemic prompted the virtualization of historically in-person radiology rotations for medical students. As students return to in-person clinical education, there is an opportunity to reevaluate teaching strategies and incorporate best practices from the pandemic. We describe our experience with the conversion of a four-week radiology clerkship from an in-person (IP) to remote learning (RL) to hybrid model (HM) and its impact on student performance and satisfaction.
MATERIALS AND METHODS: Stratified by curriculum (Group 1 IP, Group 2 RL, Group 3 HM), student standardized final examination scores, final grades, lecture evaluation scores, and satisfaction scores were compared. Additional analysis was performed for Group 3 clinical divisions in which IP or RL models predominated.
RESULTS: A significant decrease in mean final exam score was noted in Group 2 (p < 0.0001). Average lecture rating decreased in Group 3 compared to Group 1 (p < 0.001). Group 3 students reported improved faculty (Group 1: 59, Group 2: 61, Group 3: 82; p < 0.001) and resident (Group 1: 76.5, Group 2: 68, Group 3: 90; p < 0.001) teaching effectiveness. Student-reported quantity and quality of formative feedback were also highest for Group 3 (Quantity; Group 1: 60.6, Group 2: 74, Group 3: 93; p < 0.001) (Quality; Group 1: 59.1, Group 2: 77, Group 3: 97; p < 0.001). Group 3 subanalysis demonstrated increased student-perceived usefulness of activities within IP divisions (p < 0.01) and a decrease for RL divisions (p < 0.05).
CONCLUSION: A hybrid curriculum resulted in improved student satisfaction and preserved student performance after an emergent conversion to remote learning.

academic year

hybrid model

in-person

picture archiving and communication system

remote learning

Introduction

Clinical rotations are a cornerstone of undergraduate medical education, constituting a large portion of medical school curricula. While only 16% of medical schools in the United States require a dedicated radiology clerkship, radiology is widely perceived as an important curricular component with 96% of medical schools incorporating radiology content in coursework.1, 2, 3, 4, 5, 6 Most undergraduate clinical radiology education occurs by way of traditional block rotations, which rely heavily on in-person, apprenticeship-style interactions between medical students and radiologists, trainees, and technologists, including participation in imaging interpretations and procedures. , 6, 7, 8

The abrupt onset of the COVID-19 pandemic has had a profound and ongoing effect on clinical undergraduate medical education, given the historic emphasis on face-to-face learning. On March 17, 2020, the Association of American Medical Colleges issued a statement recommending a temporary suspension of medical student participation on clinical rotations. Existing clinical experiences, including radiology electives10, 11, 12, 13, 14, 15 and clerkships,16, 17, 18, 19 had to emergently restructure, with medical educators forced to balance the constraints of COVID-19 mitigation policies with the requirements of undergraduate medical education. Our institution suspended second-year (i.e., the core clinical year) medical student participation in direct patient care activities on March 17, 2020. Our Department of Radiology rapidly transitioned from an in-person to a remote learning clerkship model, which was later converted to a hybrid model incorporating both in-person and remote learning components (Table 1 ).

Table 1

Timeline of the COVID-19-mediated evolution of the radiology clerkship.

March 17, 2020	Association of American Medical College press release addressing medical student participation on clinical rotations
March 17, 2020	Suspension of in-person clinical involvement for second-year medical students
March 17, 2020	First day of radiology clerkship under a remote learning model
June 15, 2020	Re-immersion of second-year medical students into in-person clinical environments
August 21, 2020	Final day of radiology clerkship under a remote learning model
August 24, 2020	First day of radiology clerkship under a hybrid model (ongoing)

As the thralls of the COVID-19 pandemic wane and medical students have returned to in-person clinical education, there is an opportunity to reevaluate existing teaching strategies to incorporate best practices learned from the pandemic. While numerous studies report positive learning outcomes within COVID-19-driven virtual radiology rotations, several emphasize the student- and faculty-perceived value of live instruction and the importance of in-person education in the reading room. , , , , Few, however, provide concrete guidance on the next best steps for “post”-COVID radiology curricula. Blended learning models, or the balanced integration of in-person and technology-assisted (i.e., e-learning) environments, offer a promising approach.20, 21, 22

In this article, we describe the COVID-19-mediated conversion of a radiology clerkship from an in-person (IP) to remote learning (RL) to a hybrid model (HM) and investigate the impact on student performance and satisfaction.

Materials and Methods

This retrospective study was deemed exempt by the Institutional Review Board.

Structure of the Traditional IP Clerkship

”Redacted” School of Medicine is an academic institution that has approximately 123 medical students per class who undergo core clinical rotations during their second year. The Radiology clerkship is housed within a large tertiary and quaternary care teaching hospital and is historically conducted as a required four-week in-person rotation. The student population included almost all second-year medical students and a select number of fourth-year medical students on a self-selected primary care track who took a near-identical clerkship in their fourth year rather than their second year.

Students participated in approximately 40 live in-person lecture hours and 140 in-person clinical hours across a student-dependent combination of nine subspecialty subrotations (e.g., breast imaging, nuclear medicine). Additional required didactic activities included resident teaching conferences, departmental conferences (e.g., Grand Rounds), multispecialty conferences (e.g., tumors boards), hands-on workshops (e.g., US-guided biopsy), and subrotation-dependent ancillary materials (e.g., guided review of archived cases). Evaluation of student performance relied on subrotation-specific assignments (quizzes and case presentations), subjective evaluations of clinical work, two presentations (one clinical case and one imaging utilization), and two multiple-choice exams (midterm and final), which culminated in an Honors/High Pass/Pass/Fail grade. The course director met with students at the midpoint of the clerkship to provide and receive feedback.

Structure of the RL Clerkship

Conversion of the traditional IP clerkship to a RL format necessitated an iterative restructuring of course content and delivery. While the overarching requirements were retained, the course was initially split into two phases: (1) a didactic phase, where all lectures and exams occurred remotely; and a (2) clinical phase, where a subrotation-dependent mixture of remote and/or in-person participation within the department occurred. Students participated in approximately 40 hours of live videoconference lecture hours and 140 remote clinical hours (videoconference readouts and conferences) with the opportunity to meet remotely with subspecialty radiologists during “office hours” as needed. The clinical component of videoconference readouts focused on the examinations reviewed by trainees (e.g., residents); however, the educational component was targeted to all learners in virtual attendance, including medical students. Videoconference readouts were scheduled throughout the workday at the discretion of the subspecialty attending radiologists. In subsequent rotations, a predominantly remote structure was maintained, however with reintegration of lecture hours and clinical hours. While the student population and methods of evaluation were maintained, due to the stressors of the COVID-19 pandemic, final student grades were converted to a Satisfactory/Unsatisfactory scale.

Structure of the HM Clerkship

In academic year (AY) 2020-2021, major revisions were implemented to the content and structure of the clerkship, including the addition of several interactive lectures. As restrictions on in-person medical student engagement were lifted, the clerkship was restructured to emphasize blended learning, combining the ease of virtual didactics with the return to completely in-person instruction in the reading room. Second- and fourth-year medical students participated in approximately 14 live videoconference lecture hours, 25 prerecorded lecture hours, 70 in-person clinical hours, and 70 remote clinical hours (videoconference readouts and conferences) across eight subspecialty subrotations. A sample student clerkship schedule is provided in Figure 1 .

Figure 1

Sample medical student schedule on HM radiology clerkship. Abbreviations: picture archiving and communication system (PACS), musculoskeletal (MSK), vascular and interventional radiology (VIR)

Evaluation of student performance under the HM clerkship was modified to include subjective evaluations of clinical work, one case presentation, one multiple-choice final exam, and one oral final exam, culminating in a Satisfactory/Unsatisfactory grade. A team of four third- and fourth-year medical student teaching assistants was also added as a separate innovation, who helped co-facilitate teaching sessions and provide logistical and educational support for clerkship students. A summary of changes between the IP, RL, and HM clerkships is provided in Figure 2 .

Figure 2

Summary of changes made between IP, RL, and HM radiology clerkship models. Abbreviations: computed tomography (CT), magnetic resonance imaging (MRI), gastrointestinal (GI), ultrasound (US), musculoskeletal (MSK)

Return to the Reading Room: IP Elements of the HM Clerkship

In-Person Subrotations

Students spent two one-week blocks on two student-prioritized subrotations; the remaining two weeks were divided between the other six subrotations. Of the 140 total clinical hours, approximately 70 were in-person and 70 were remote (videoconference readouts and conferences), with the precise division between IP and RL dependent on subrotation preference. Clinical responsibilities similarly varied, but included observation and participation in patient rounds, procedures, imaging studies, and case readouts.

Virtualizing Radiology Education: RL Elements of the HM Clerkship

Prerecorded Lectures and Virtual Didactics

Our institution provides students and staff with licensed accounts on the Zoom videoconference platform and free accounts on the WebEx videoconference platform. Medical students are also provided with instructions to download the institution's picture archiving and communication system (PACS) client to their personal laptops, which can be accessed at any time via an institutional virtual private network.

Orientation, a departmental tour, and PACS training were presented live over Zoom videoconference. PACS training included guidance on how to access patient information and academic folders and use key features (e.g., scroll, zoom, window/level adjustment). Of the required lectures, some lecturers presented their talks live over Zoom (14 hours); others pre-recorded content (25 hours) where recordings and other materials were made available to students via an institutional learning management system for self-directed study (Table A1). Where possible, resident teaching conferences, departmental conferences, pertinent multispecialty conferences, and other subrotation-dependent didactics occurred in-person; virtual alternatives, largely via Zoom, occurred at the discretion of the subrotation or specific conference.

Interactive Chest Workshops

Two interactive, half-day-long chest radiograph workshops were administered via Zoom by a senior faculty cardiothoracic radiologist. Prior to the first workshop, students accessed introductory educational material for independent review and completed a single mock dictation of a chest radiograph on PACS. The first workshop was spent establishing search patterns, reviewing anatomic structures, and discussing what vocabulary to use in chest radiograph interpretations. Prior to the second workshop, students completed several more mock dictations for commonly encountered pathologies, which were reviewed during the second workshop.

Interactive Mock Dictation Session

On the first day of the clerkship, students were assigned a group of ten unknown cases accessible via PACS and provided with blank dictation templates for each case (Figure 3 ). The selected cases included a variety of normal and pathologic studies across imaging modalities (XR, CT, and US) and patient ages. Mock dictations were to be completed prior to an end-of-week group meeting, during which the cases were discussed using Zoom's chat, screen sharing, and annotation features. This session was co-facilitated by teaching assistants and offered a dedicated opportunity for clerkship students to ask questions about the clerkship experience and share feedback.

Figure 3

Sample dictation template and representative ultrasound image. Abbreviations: medical record number (MRN), accession (ACC), ultrasound (US), right upper quadrant (RUQ)

Case Presentations and Mid-Clerkship Review

As a course assignment, students were required to create and give oral case presentations via Zoom on a patient and topic of their choice. Representative topics included imaging findings of particular pathologies (e.g., moyamoya), the role of imaging in patient treatment (e.g., tumor embolization), and appropriate imaging workups (e.g., for a breast mass). The week prior to the case presentation date, a clerkship co-director or teaching assistant gave an interactive case presentation preparation session live via Zoom that reviewed the assignment rubric and tips for creating effective presentations. Students were also paired via Zoom's breakout rooms feature to trade presentation drafts and provide live feedback on one another's slides. After all case presentations were complete, a co-clerkship director met individually with students over Zoom to provide and receive feedback.

Methods of Clerkship Analysis

During the last week of each clerkship block (AYs 2016-2021), a standardized multiple-choice written exam was administered. At the end of each clerkship, all students were required to complete an anonymous online course evaluation on the Qualtrics platform. Stratified by curriculum (Group 1 IP, Group 2 RL, and Group 3 HM), student standardized final written examination scores, final grades, lecture evaluation scores, and student satisfaction scores were compared. Additional analysis was performed for Group 3 clinical divisions in which IP or RL predominated. Two-sample t-tests and three-way analyses of variance were conducted for comparisons between different means. Tukey's test was performed for post hoc analysis. A p value of <0.05 was considered statistically significant. Student comments were also compiled.

Results

Student Enrollment

Between AYs 2016-2021, 549 total medical students participated in the core radiology clerkship (Table 2 ). 100% of students completed the course evaluation.

Table 2

Characteristics of Group 1 (IP), Group 2 (RL), and Group 3 (HM) students.

	Group 1	Group 2	Group 3
Date Range	August 22, 2016 – March 16, 2020	March 17, 2020 – August 21, 2020	August 24, 2020 – August 20, 2021
Total Student Enrollment	371	71	107
Number of Block Rotations	42	6	12

Abbreviations: in-person (IP), remote learning (RL), hybrid model (HM)

Course Outcomes

100% of students completed the final written exam. A significant decrease in mean final exam score was noted in Group 2 (Figure 4 ).

Figure 4

Average medical student final written exam score. Abbreviations: in-person (IP), remote learning (RL), hybrid model (HM)

No reports of student mistreatment (abuse or neglect) occurred in Groups 2 or 3, compared to two per year in Group 1. No students in any group had a Fail/Unsatisfactory grade or received an exam score that prevented them from passing the course.

No students in any group had a failing final exam score or received an Unsatisfactory grade.

Student Satisfaction

Average lecture rating decreased from 4.5 in Group 1 to 4.2 in Group 3 (p<0.001). The lecture rating in Group 2 was 4.0; however, due to the small number of students in Group 2, differences between Group 2 and other groups did not reach significance.

Group 3 students reported significantly improved faculty and resident teaching effectiveness (p<0.01) (Figure 5 ).

Figure 5

Percentage of medical students who rated faculty and resident teaching effectiveness "favorable” or “very favorable.” Abbreviations: in-person (IP), remote learning (RL), hybrid model (HM)

Student-reported quantity and quality of formative feedback was also significantly highest for Group 3 (p<0.01) (Figure 6 ).

Figure 6

Percentage of medical students who rated quantity and quality of formative feedback "favorable” or “very favorable.” Abbreviations: in-person (IP), remote learning (RL), hybrid model (HM)

Students were asked to rate the usefulness of clerkship activities in helping them meet clerkship goals (e.g., understanding imaging techniques) on a five-point Likert scale. Group 3 subanalysis demonstrated significantly increased student-perceived usefulness for IP-predominant subrotations (p <0.01) and a significantly decreased score for RL-predominant subrotations (p<0.05) (Figure 7 ).

Figure 7

Percentage of medical students who reported “favorable” or “very favorable” usefulness scores. Abbreviations: in-person (IP), remote learning (RL), hybrid model (HM)

Student Recommendations

Within the comments from Group 3 students, 73% of student suggestions for improvement were centered around remote readouts. Representative comments are shown in Table 3 .

Table 3

Representative anonymized student comments from Group 3 (HM) students.

I recognize that readout schedules vary in frequency and duration, but it felt disappointing when my only exposure to certain services was a brief virtual readout in which teaching/asking questions was not incorporated, and I primarily watched passively

It was difficult to stay engaged and learn over zoom. Sometimes attendings wouldn't remember to call the med students, or the readouts happened so quickly there weren't great learning opportunities.

the Webex readouts were often not an effective teaching tool because they are meant for the residents and there doesn't seem to be adequate time/space to teach medical students in that space

Virtual readouts are difficult. The one opportunity I was given to readout with the attending in person was so helpful to have someone point things out on the screen and directly engage with me. I felt safe doing this, as it was just two of us at a computer with masks on.

Abbreviations: hybrid model (HM)

Discussion

Our results demonstrate that a hybrid radiology clerkship curriculum improved student satisfaction and preserved student performance compared to purely in-person or remote-learning curricula. Group 3 (HM) students reported the highest ratings for teaching effectiveness and feedback quantity and quality, which may reflect the increased opportunities for meaningful student-instructor exchanges in in-person clinical settings and in several new interactive teaching sessions (e.g., mock dictations). Importantly, Group 3 (HM) students also demonstrated a significantly more favorable appraisal of subrotations that were predominantly in-person rather than remote. Several student comments cited the “helpful” nature of in-person clinical education, as opposed to “difficult” and less effective virtual readouts. Although average lecture ratings decreased significantly between Group 1 (IP) and Group 3 (HM) students, we believe this may represent the cumulative mental fatigue that accompanies extended periods of virtual learning. This concept was particularly applicable to our Group 3 (HM) students, as they began clinical rotations after over three months of virtual preclinical education. An additional contributor to lecture satisfaction may have been differences in lecture format: Group 1 (IP) students had exclusively live lectures whereas Group 3 (HM) students had a combination of live and prerecorded lectures, the latter of which has a reduced capacity for meaningful interaction with educators. Table A1

Table A1

Lecture concepts by organ system under the HM model.

Abdominal	Breast	Cardiothoracic	General Concepts
Abdominal lymphoma*Acute and inflammatory colitisAcute appendicitisAcute cholecystitis Acute diverticulitis Acute pancreatitis Biliary diseases and workup*Bowel obstruction vs. ileus Deep venous thrombosis Ectopic pregnancy Gallstones and renal stonesHepatic metastases*Pancreatic adenocarcinoma*Pneumatosis intestinalis Pneumoperitoneum Sigmoid volvulus	General overview of breast imaging modalities Screening, diagnostic, and interventional principles	Acute alveolar processes*Aortic aneurysmAortic dissection Atelectasis Congestive heart failure COPD Lines and tubes on CXRPleural effusion Pneumomediastinum*PneumoperitoneumPneumothorax Pulmonary hypertension Pulmonary foreign bodiesPneumonia vs. contusion Rib fracture SarcoidosisSigns on thoracic imaging*	Basic principles of CT, fluoroscopy, MRI, US, and XRBasic principles of pediatric imagingCross-sectional CT anatomy
Interventional	Musculoskeletal	Nuclear	Pediatric
BRTO*Gastrointestinal bleedingGeneral IR terminology and patient workup*Pulmonary emboliTIPS*	Fracture basicsSpine anatomy on radiographs*	General principles in nuclear medicine and PETMedical radiation physicsOverview of radiation oncology*	Annular pancreasBowel atresia and stenosis Bowel obstruction workup Hirschsprung diseaseHypertrophic pyloric stenosisIntussusception Meconium aspirationMeconium ileus Midgut volvulus Necrotizing enterocolitis NRDSPediatric lines and tubesPulmonary interstitial edemaSmall left colon syndrome

Exclusively presented in recorded lecture content.

Abbreviations: chronic obstructive pulmonary disease (COPD), chest x-ray/radiograph (CXR), computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), x-ray/radiography (XR), balloon-occluded retrograde transvenous obliteration (BRTO), interventional radiology (IR), transjugular intrahepatic portosystemic shunt (TIPS), positron emission tomography (PET), neonatal respiratory distress syndrome (NRDS)

Regarding course outcomes, the mean final exam score was restored in Group 3 (HM) after declining in Group 2 (RL). This pattern likely reflects the acute and unfamiliar stress that the COVID-19 pandemic placed on Group 2 (RL) students at its inception, compared to the adjustment period that Group 3 (HM) students were afforded. In addition, Group 2 (RL) students were required to learn four weeks of curricular material in a two-week period and had to take their final exams before any clinical experience was offered; clinical application of studied material would likely have helped solidify the content. Group 3 (HM) students additionally had the novel opportunity to take advantage of medical student teaching assistants, who provided peer support and an interactive examination review session.

Reports of student mistreatment (abuse or neglect) also declined from two per year in Group 1 (IP) to none in Groups 2 (RL) and 3 (HM). Due to the short time frame and smaller population, this decrease may not be significant, but the trend may suggest that students attributed possible negative interactions with faculty/residents to the stressful learning adaptations (i.e., learning environment differences due to COVID-19 policies) rather than mistreatment.

This study has several limitations. Although our overall sample size is adequate, the sizes of Groups 2 (RL) and 3 (HM) are small relative to Group 1 (IP), limiting the ability for us to reach statistical significance for some differences such as lecture ratings. Pre-/post-rotation assessments were also not performed; instead, our conclusions rely exclusively on end-of-rotation examinations and post-rotation surveys that may be influenced by the time of year or recall bias. Additionally, several unique variables were introduced in Group 3 (HM) that were not present in Groups 1 (IP) or 2 (RL), including new leadership and a new team of medical student teaching assistants.

Altogether, these findings suggest that a hybrid radiology educational model offers a unique opportunity to thoughtfully integrate the most successful aspects of traditional in-person and remote “COVID-era” methods. Future hybrid models should emphasize the maintenance of in-person immersive experiences (e.g., procedures, reading room participation) over videoconference readouts, which were important both for our students and those in existing literature. , , , In-person clinical education offers significant merits, including the opportunity to foster personal relationships with faculty and peers, directly observe the activities of radiologists, participate in interventions, and witness “live” cases unfold in the reading room. , Student-faculty mentorship can additionally influence the consideration of radiology as a career choice. As such, our current clerkship model exclusively features live participation in the reading room. Similarly, the flexibility and independence afforded by virtual didactics cannot be overlooked; the use of interactive teaching sessions (e.g., mock dictations, participatory or flipped lectures) is convenient while maintaining high-quality learning outcomes. , However, the balanced integration of the two formats is critical to prevent strained clinical environments or “Zoom fatigue.”

Conclusion

Our findings suggest that a hybrid radiology clerkship model can preserve medical student performance while improving student satisfaction relative to purely in-person or remote learning models. Hybrid educational models offer a unique opportunity to leverage the best practices of both in-person and remote learning methods.

CRediT authorship contribution statement

Alexis Musick: Conceptualization, Writing – original draft, Visualization. Deeksha Malhotra: Writing – original draft, Visualization. Robert French: Conceptualization, Methodology, Formal analysis, Investigation, Writing – review & editing, Supervision. Caroline Carrico: Investigation, Writing – review & editing. Jonathan Martin: Conceptualization, Methodology, Formal analysis, Investigation, Writing – review & editing, Supervision.