Resident Experiences With Virtual Radiology Learning During the COVID-19 Pandemic.

RATIONALE AND
OBJECTIVES: COVID-19 has disrupted radiology education and forced a transition from traditional in-person learning to a virtual platform. As a result of hospital and state mandates, our radiology residency program quickly transitioned to a virtual learning platform to continue dissemination of knowledge, maintain resident engagement, and ensure professional development. The goal of this study is to assess the strengths and weaknesses of the virtual learning platform at our institution using resident ratings.
MATERIALS AND METHODS: This institutional IRB-exempt study involved a survey of 17 questions which was electronically distributed to 45 radiology residents using SurveyMonkey. Questions encompassed resident satisfaction with teaching and professional development, scheduling changes, and engagement with the virtual platform. Answers to most questions were submitted on a Likert scale.
RESULTS: A total of 31 of 45 respondents completed the survey (response rate = 69%). Most residents were satisfied with the virtual platform with teaching activities identified as a strength and the incorporation of professional development as a weakness. The most frequent barriers to attending the virtual curriculum were technical difficulties (43%) and childcare (36%). Residents who reported experiencing barriers were less likely to adhere to the virtual curriculum (p = 0.004). Most respondents (81%) reported a desire to maintain elements of the virtual learning practice postpandemic.
CONCLUSION: The majority of residents reported high satisfaction with virtual learning during the COVID-19 pandemic. Teaching activities are a curricular strength. Weaknesses identified include the incorporation of professional development and extrinsic barriers, such as technical difficulties and family obligations, which require further support for trainees. Published by Elsevier Inc.

INTRODUCTION

The Coronavirus Disease 2019 (COVID-19) pandemic has disrupted radiology education, forcing radiology residency programs to transition from traditional in-person learning to a virtual platform. For example, changes to side-by-side, in-person readouts were necessary to comply with social distancing requirements during the pandemic. Many programs have described their experience with virtual read-outs, which include reviewing imaging via phone or using screen-sharing software (1, 2, 3, 4, 5, 6). Didactic lectures and multidisciplinary conferences also have transitioned to a virtual platform. Software-based conference room solutions have allowed for these activities to continue in a virtual setting (1, 2, 3, 4, 5,7).

The COVID-19 pandemic has brought additional challenges that residency programs have tackled using virtual strategies. Solutions to decreases in imaging volumes (8,9) include dividing residents into two clinical groups with one reporting to service and the other assigned to distance learning, promoting online learning resources, assigning scientific and/or review articles with multiple choice questions, and hosting interesting case conferences (1, 2, 3, 4). Additionally, studies have indicated that the pandemic has negatively impacted trainee well-being, including decreased resident morale (10,11). Promoting trainee wellness has been facilitated with the use of virtual strategies such as engagement in social media, encouraging resident meetings and/or huddles, hosting a virtual happy hour, and promoting institutional or other online well-being resources (2,3,4,10).

Advantages of virtual learning include increased accessibility of teaching material from anywhere at any time, increased schedule flexibility, and asynchronous discussions with peers. Increases in engagement, time spent answering student questions, and complexity of questions asked have also been described (12).

Our residency program quickly transitioned to a virtual learning environment at the beginning of the COVID-19 pandemic, in response to local hospital and state mandates. With the recent increase in COVID-19 cases in our state and nationwide, it is clear that the virtual curriculum is here to stay for the near future (13,14). Although many papers have offered suggestions on how to structure a virtual curriculum during the pandemic, or describe their experience with a specific curricular intervention, a current gap in the literature is resident feedback on the overall virtual delivery of the traditional educational curriculum. The purpose of this study was to assess the strengths and weaknesses of our virtual learning platform, using resident ratings, in order to optimize the learning environment for radiology trainees.

MATERIALS AND METHODS

This study was submitted to the Beth Israel Deaconess Medical Center Ethics Board and was approved for institutional review board exemption.

Structure of Virtual Curriculum

Our diagnostic radiology residency program is based at a large academic institution affiliated with Harvard Medical School, which currently trains 45 radiology residents (36 diagnostic radiology residents and 9 integrated interventional radiology residents).

At the start of the pandemic in March 2020, several curriculum and workflow modifications were implemented to comply with hospital social distancing guidelines and to accommodate decreased imaging volumes. For example, scheduling changes were made to minimize the number of attending radiologists required to be physically on-site, with many interpreting studies from home. Resident vacation was put on hold for March and April 2020. Residents were divided into two tracks which alternated every two weeks. One track was clinical and worked in-hospital in various sub-specialties within the radiology department. The other track was assigned to a virtual learning curriculum, which was comprised of various teaching modules, articles and online lectures to be done from home and designed by the individual radiology sub-specialties. The virtual content for each sub-specialty was decided by consensus by the attending radiologists within the division. Table 1 outlines an example of the daily virtual curriculum designed for residents rotating through Abdominal Imaging, Neuroradiology and Musculoskeletal Radiology. Residents continued rotating through different sub-specialties regardless of whether they were in-hospital or on a virtual block.

Table 1

Daily Virtual Curriculum in Different Radiology Sub-Sections

Abdominal Imaging	Neuroradiology	Musculoskeletal Radiology
Virtual resident teaching rounds 8-9 AM
-Watch online lectures by RSNA, ARRS and SAR-Assigned Radiographics articles curated by the ABR on Gastrointestinal, Genitourinary or Ultrasound Imaging. Attending radiologist provided examples of interesting cases based on selected reading material for residents to review on remote PACS viewer-Attend virtual inter-disciplinary rounds (e.g., Gynecology-Oncology Board, Liver Tumor Board)-Daily interesting case rounds-Weekly didactic presentation given by resident (three residents selected per week)-Complete 25–50 question bank questions (RADPrimer)-Complete prostate MRI simulation cases on free web platform	-Assigned Radiographics articles in Neuroradiology-Attend virtual inter-disciplinary rounds-Watch teaching presentations from neighboring institutions-Joint teaching sessions with Otorhinolaryngology trainees-Watch at least two online videos from major Neuroradiology national organizations' web-channels-Attendings sent residents interesting cases to review on remote PACS viewer, with more cases sent to first year residents for call preparation-Complete a minimum of 25 RadPrimer questions per day-Complete assigned RADPrimer teaching modules	-Residents assigned a daily topic to research-1 hour didactic lecture given by attending radiologist-30–45 minutes of attending-led teaching cases-30 minutes of attending-led interesting case rounds-Presentation of resident assigned topics-Assigned readings based on the ABR curriculum-Complete 25 RADPrimer questions per day

ARRS, American Roentgen Ray Society;

RNSA, Radiological Society of North America; SAR, Society of Abdominal Radiology.

A few radiology residents, all of whom had completed a 1-year clinical internship in Internal Medicine or General Surgery, were re-deployed for 2-week rotations to Internal Medicine due to the high number of COVID-19 cases at the peak of the pandemic in March and April 2020. Any first to fourth year radiology resident who volunteered was eligible for re-deployment. During re-deployment, residents completed all clinical duties remotely from home and helped with writing clinical notes, arranging consults and organizing patient disposition. No changes were made to the radiology resident call schedule.

Another component of our curriculum is the virtual read-out. Within our picture archiving and communication system (PACS), there is a “collaborate” function that allows trainees to share their viewing station with attending radiologists. The resident and attending both have the ability to scroll and control the screen, which simulates the traditional in-person reviewing while promoting social distancing. Residents and attendings can discuss cases over the phone or through virtual conference room software available at our institution. Read-outs occurred approximately four times per day, twice in the morning and twice in the afternoon. However, attending adiologists were always available for emergency cases, urgent questions or more frequent read-outs. Statistics on how virtual read-outs occurred were not collected since read-outs occur in various locations and at multiple times during the day, so that collecting these statistics accurately would be challenging. To provide some context, at the time that this study was conducted, Neuroradiology for example kept their read-outs completely virtual with use of the screen sharing function. About 80% of the read-outs in Abdominal Imaging were in person, with the rest occurring over the phone or with the screen share function.

Resident teaching rounds, which take place Monday through Thursday mornings, were transitioned to a virtual platform, with attendings encouraged to use interactive audience-response techniques to promote resident engagement. Weekly section rounds and interdisciplinary conferences also became completely virtual, and residents were encouraged to attend.

From a resident wellness perspective, our program hosted monthly town hall meetings where attending radiologists on the residency program committee would provide all residents with COVID-related workflow updates and address any resident concerns. Additionally, separate town halls were held with each residency cohort to address year-specific concerns, such as redeployment of residents to clinical medicine or cancellation of the American Board of Radiology (ABR) Core Examination for third-year residents.

At our institution, we have a two-year curriculum implemented by each division within radiology. The content for the 2020 academic year was formalized at the beginning of the academic year. The virtual learning curriculum described in this paper details the strategy for teaching the existing curriculum in a virtual format as a result of the pandemic. Since it was the method of delivery of the curriculum that was changed, and not curriculum content, we continued our traditional assessment metrics, which included rotations when residents were in-house and performance on call shifts. Additionally, learning from the virtual curriculum material was assessed individually by attendings during case read-outs and resident teaching rounds. The virtual delivery of our curriculum was instituted in March 2020, and has been on-going apart from the online two week at-home rotation that was discontinued in June 2020. All components of the curriculum were mandatory, and attendance was recorded. However, due to the challenges facing residents in their personal and professional lives during the peak of the pandemic in March-June 2020, punitive measures were not implemented. The residency leadership did not specify a minimum number of hours to be spent on the learning curriculum. Instead, the expectation was to go through the assigned learning material. Residents were required to submit to the educational office a log of all learning activities completed on a weekly basis which was regularly reviewed by the residency program committee to ensure residents continued to meet their learning milestones.

Data Collection and Analysis

Data was collected via an anonymous 17-question survey distributed to all current radiology residents between September to November 2020, after the curriculum had been implemented for six months. Surveys were distributed electronically via e-mail using Survey Monkey (San Mateo, CA). Three e-mail remainders were sent following distribution of the survey, and response collection ended after eight weeks.

Survey questions (Appendix A) related to resident satisfaction with the various components of the virtual curriculum including virtual readouts, virtual lectures and/or conferences, virtual teaching rounds, and scheduling changes. Satisfaction with professional development in a virtual setting, such as mentorship, career planning and research, was assessed. Additional questions evaluated stress and the possibility of work-related COVID exposure, as well as engagement with the virtual curriculum. Questions included answers with tick boxes, 5-point Likert scale, and open-ended questions.

Data was exported from Survey Monkey and analyzed using Microsoft Excel 2010 (Microsoft Corp, Redmond, WA) and Matlab version 9.9 (Mathworks, Natick, MA). Proportions, chi-squared, Kendall rank correlation, and 2-tailed Mann-Whitney U tests were used to analyze the quantitative survey data. p values ≤0.05 were considered statistically significant.

Data from the open-ended questions were reviewed by two study authors. Given the low response rate for the open-ended questions, the two main themes for recurring comments are presented in this study.

RESULTS

A total of 31 out of 45 eligible residents completed the survey, corresponding to a 69% response rate. Fourteen women (31%) and fourteen men (31%) responded, with three respondents (10%) preferring not to disclose their gender. The distribution of respondents by level of training was: 5 (16%) first year residents, 9 (29%) second year residents, 10 (32%) third year residents, and seven (23%) fourth year residents. Three surveys were incomplete, and completed data was included in the analysis.

Most residents were satisfied with the various facets of the virtual learning platform (Table 2 ). Nearly all residents (97%) were extremely to moderately satisfied with morning teaching rounds and the virtual delivery of interdisciplinary rounds. The majority of residents were also extremely to moderately satisfied with virtual read-outs and section teaching rounds (81% and 90%, respectively). However, one of the main themes identified from our qualitative data was that residents missed reading real cases on a PACS system (Supplementary Table 3).

Table 2

Survey Results

Survey Item	n, %
Level of training
R1	5, 16%
R2	9, 29%
R3	10, 32%
R4	7, 23%
Gender
Male	14, 45%
Female	14, 45%
Prefer not to disclose	3, 10%
Satisfaction with teaching activities	Extremely satisfied	Very satisfied	Moderately satisfied	Slightly satisfied	Not at all satisfied
Morning conferences	13, 42%	9, 29%	8, 26%	1, 3%	0, 0%
Interdisciplinary conferences	11, 35.5%	11, 35.5%	8, 26%	0, 0%	1, 3%
Virtual read-outs	9, 29%	9, 29%	7, 23%	4, 13%	2, 6%
Virtual section teaching rounds	11, 36%	10, 32%	7, 23%	2, 6%	1, 3%
Satisfaction with career planning activities
Mentorship	6, 19%	9, 29%	9, 29%	5, 16%	2, 6%
Career planning	4, 13%	8, 26%	8, 26%	5, 16%	6, 19%
Research activities	6, 19%	8, 26%	10, 32%	4, 13%	3, 10%
Satisfaction with schedules changes
Changes to learn from home	14, 48%	8, 28%	5, 17%	2, 7%	0, 0%
Social distancing at work	10, 35%	13, 45%	5, 17%	1, 3%	0, 0%
Vacation	6, 19%	5, 16%	9, 29%	3, 10%	8, 26%
	Strongly agree	Agree	Neutral	Disagree	Strongly disagree
I was stressed regarding the possibility of work-related exposure.	6, 19%	9, 29%	6, 19%	7, 23%	3, 10%
I fell behind in my radiology education during the pandemic.	5, 17%	5, 17%	5, 17%	7, 23%	8, 26%
After COVID-19, we should maintain some virtual learning.	14, 45%	11, 36%	2, 6%	2, 6%	2, 6%
	Always	Frequently	Sometimes	Seldom	Never
I was able to stick to the virtual curriculum.	11, 38%	12, 41%	4, 14%	2, 7%	0, 0%
How accessible were attendings?	12, 41%	14, 42%	3, 10%	0, 0%	0, 0%
How often are outside learning resources used?	8, 28%	11, 39%	5, 18%	3, 11%	1, 4%

The majority of residents were also satisfied with professional development activities, such as mentorship, career planning, and research activities, albeit to a lesser degree compared to teaching activities (Table 2). The percentage of residents who responded “slightly satisfied” or “not at all satisfied” with mentorship, career planning and research activities were 23%, 35%, and 23%, respectively.

With regards to scheduling changes (Table 2), 93% of residents were extremely to moderately satisfied with learning from home. Similarly, 97% of residents were extremely to moderately satisfied with schedule changes to allow social distancing at work. However, there were mixed responses concerning vacation time, with 35% responding they were slightly satisfied or not at all satisfied. Most residents (93%) reported that attending radiologists were always or frequently accessible during the pandemic.

Fifteen residents (48%) responded that they were stressed regarding the possibility of work-related exposure to COVID-19. Senior residents and women expressed significantly more stress regarding the potential for work-related exposure (p < 0.001 and p = 0.033, respectively).

Residents reported dedicating an average of six hours per day to the virtual learning curriculum. Twenty-three residents (79%) were able to consistently adhere to the curriculum; nonetheless, 33% of residents felt that they fell behind in their radiology education during the pandemic. Residents who felt they fell behind reported a significantly lower number of daily hours (p = 0.009).

Half of residents described barriers to attending the virtual curriculum from home. The most frequent were technical difficulties (43%), childcare (36%) and lack of an office space (21%). Residents who reported experiencing barriers were significantly less likely to adhere to the curriculum (p = 0.004) and felt that they fell behind in their education compared to peers who did not report barriers (p = 0.001). Furthermore, family obligations, such as childcare, was the second main theme identified in our qualitative data as a barrier to adhering to the virtual curriculum (Supplementary Table 3).

In addition to the virtual curriculum designed by our institution, 68% of residents frequently used learning resources outside of the recommended learning material. Fourth-year residents used outside resources somewhat less frequently compared to more junior residents (p = 0.097). Additionally, residents who used outside learning resources spent a higher number of hours on the virtual curriculum (p = 0.012).

Most respondents (81%) reported a desire to maintain elements of the virtual learning practice postpandemic. However, more junior residents, particularly first-year residents, were significantly less favorable to continuing postpandemic virtual learning (p = 0.024).

DISCUSSION

The COVID-19 pandemic has dramatically altered diagnostic radiology residency training, and it is clear that the virtual curriculum is here to stay for the foreseeable future. Our study provides trainee insight into the strengths and weaknesses of the virtual learning environment. This addresses a current gap in the literature, which is essential to improve its structure moving forward. Moreover, these results are important as learner evaluations and ratings form the primary basis for evaluation at most educational institutions.

Overall, residents in our study were satisfied with the virtual learning curriculum. For example, residents were highly satisfied with the delivery of online morning teaching conferences and this is a curricular strength. Our institution has encouraged lecturers to use polling software to promote audience engagement for virtual lectures, as the incorporation of audience participation has been correlated with higher lecture evaluations (15). Audience participation in lectures has similarly been advocated in virtual radiology curriculums at other institutions (2,4,7). Other teaching activities, such as the incorporation of interdisciplinary rounds and weekly section teaching rounds, were well-received by residents and are additional strengths.

Virtual read-outs also were well-received, though with less satisfaction than other components of the virtual curriculum, and is considered a strength, although this process could be improved. This finding was echoed in our qualitative data, where a major theme was residents missing the workstation reading of real cases. Possible reasons for lower ratings for virtual read-outs described by Matalon et al. are impacts on teaching, feedback, and the learning atmosphere (6). Daily read-outs are undoubtedly a key component of radiology residency training. Unfortunately, hospital mandates to increase social distancing have required profound changes to how read-outs are performed. Moreover, decreased imaging volumes have impacted the educational value of the daily read-out. A potential solution to these problems devised by Recht et al. was the creation of a simulated daily readout, in which providing a list of normal and abnormal cases for residents to interpret was well received by faculty and residents alike (16). Alternatively, programs can use web browsers or a virtual private network to allow residents to continue accessing and interpreting real cases.

Another area identified for improvement is the incorporation of such professional development activities as mentoring, career planning and research activities in the curriculum. These activities happened informally in our curriculum, such as during virtual read-outs. Mentoring likely has proven challenging during the pandemic, as it requires the development of a personal relationship between the mentor and mentee, which undeniably becomes much more difficult in a virtual setting. Continuing mentorship during the pandemic is important for trainees, as benefits of mentorship include personal development, career guidance, and increased research productivity (17). In order to improve this aspect of our curriculum, we plan to engage with the residency program committee to develop a strategy that promotes mentorship within our department, such as assigning residents a staff mentor and encouraging regular meetings. Regarding research activities, our hospital put a hold on all new non-essential research until approximately July 2020, which likely adversely affected the new opportunities available to residents and may have impacted resident ratings. Hopefully, the resumption of departmental research will open more doors for residents to be involved, and we will monitor this throughout the upcoming months.

Scheduling changes allowing periods of time when residents could work from home and to promote social distancing at work proved to be well-received by residents. We included an assessment of vacation in our virtual curriculum as we considered this a possible compromise for residents who had their vacations put on hold at the peak of the pandemic in March and April 2020, since the at-home rotation had a flexible schedule. Despite this, we found higher rates of dissatisfaction regarding vacation. Honoring resident vacation time should be prioritized in order to maintain resident well-being and prevent burn-out. Work by Veersauri et al. in the United Kingdom showed that nearly half of their study cohort, composed of radiology residents, reported worsened well-being compared to pre-pandemic (11). Similarly, Robbins et al. found that the pandemic negatively impacted resident morale (10). Undeniably, residents are at increased risk of burn-out and decreased well-being as a result of the pandemic, and protecting time off, as departmental workload allows, is of the utmost importance to mitigate this risk.

At the time of our study, the number of daily new COVID-19 cases in our state was 244 when our survey was begun in September, to 2700 when the survey closed in November. We found that almost half of residents were stressed regarding the potential of work-related COVID-19 exposure, especially among senior residents and women. A possible explanation is that more senior residents in our program have young families and may fear exposing them. Another reason could be that being forced to quarantine due to an exposure might lead to loss of clinical time in mini-fellowships, which residents find very valuable for future job preparation (18). The reason for increased stress reported in women physicians is supported in other studies where female frontline pandemic workers had a greater risk for developing adverse psychiatric outcomes, and this issue merits further investigation (19,20).

A third of residents felt that they fell behind in their radiology education during the pandemic and is a potential weakness. However, we believe this to be a questionable finding, since our clinical competence committee overseeing resident evaluations and performance has not seen significant changes in resident evaluations since the implementation of the virtual platform. More objective evaluation metrics, such as success rates on the ABR Core Examination, or scores on the American College of Radiology Diagnostic Radiology In-Training Examination, are not yet available to assess for changes in resident performance. Ultimately, more time and research is needed to see if virtual learning impacts resident promotion. Two possible reasons residents felt this way have been identified: not enough time spent on the virtual curriculum and barriers such as family obligations. A significant finding in our study is that residents who felt that they fell behind reported a lower number of daily hours on the virtual curriculum. Based on these findings, we recommend that residency programs suggest a minimum number of hours residents spend on virtual learning to potentially tackle this issue. Moreover, residents who experienced barriers were significantly less likely to adhere to the virtual curriculum. To address these barriers, we will liaise with residents and faculty to see if there are common technical problems that could be mitigated in the future. Additionally, we plan to work with the residency program committee to develop strategies to support residents who may experience difficulty balancing work and family commitments as a result of the pandemic. One example might be connecting residents with childcare centers offering discounts to healthcare workers.

Our study also found that approximately two-thirds of residents frequently used learning resources outside the recommended learning material provided in our virtual curriculum. This finding could be considered a potential weakness, as deviation from the assigned curriculum suggests that there are potential gaps in our learning material. However, this finding is also in line with current concepts of adult learning. Knowles’ well-established adult learning theory states that learners become more self-directed with age, and emphasized the role of the learner in selecting learning goals and directing their own learning (21). We believe our curriculum provided residents the framework needed for them to seek additional resources to meet their personal educational goals. This is reflected in our finding that residents who used outside resources spent a significantly higher number of hours on the virtual curriculum, suggesting that residents who deviated from the curriculum spent more time studying. We plan on informally surveying residents to find out which outside resources were commonly used to optimize our future curriculum.

Finally, the majority of residents reported a desire to keep elements of the virtual curriculum postpandemic. This reinforces learning preferences of the current generation of radiology residents, who expect the integration of technology in medical education and use web-based platforms as their main source of medical information (22). A hypothesis for why more junior residents, particularly first-year residents, were less favorable to continuing the virtual learning curriculum postpandemic may be related to less practical experience with image interpretation and troubleshooting of the daily radiology workflow, resulting in a desire for more in-person and hands-on practice. Given that our data suggests this sub-group of residents may be less adaptable to a virtual learning environment, our program will try to arrange that the initial rotation in each sub-specialty be in-house, so that the resident gains practical experience with one-one-one teaching at the PACS workstation.

Study limitations include a single institution survey with a small sample size, which nonetheless was considered adequate given the anonymous and voluntary nature of the survey. Selection bias may reflect that residents who are more favorable to virtual learning may have been more likely to participate. In our survey design, we opted to assess the subjective evaluation metric of resident satisfaction. Using more objective evaluation tools could help strengthen our study's results. Additionally, data regarding actual engagement with the virtual curriculum could not be collected and is self-reported, and participation/attendance in the various aspects of the curriculum was not strictly monitored. Another limitation is the heterogeneity in the curriculum experienced by trainees. For example, how daily read-outs were performed varied amongst different sub-specialties. Furthermore, resident ratings for the teaching and professional development components of the curriculum may have been impacted by their interaction with attending radiologists, and perhaps residents who worked with faculty who did not emphasize these components might have resulted in lower ratings. A potential confounder for the high rates of satisfaction seen for teaching activities, or the desire to maintain elements of the virtual curriculum postpandemic, is that during the months of March-June 2020, residents were able to partake in these activities from home. It is possible that residents were satisfied with the schedule flexibility associated with these teaching activities, which could have impacted resident ratings. Finally, the results of this study may not be generalizable to smaller institutions that potentially have scarcer resources available, or to institutions with different policies in place to deal with learning via social distancing.

The findings of our study provide a stepping stone for radiology education research during the pandemic. In our qualitative data, residents described missing real cases, however the reason for this was not elaborated. Exploring what residents’ value in traditional read-outs, whether it be teaching or mentorship, would be of value to ensure that this continues in a virtual environment. Moreover, our study serves as a platform to evaluate which aspects of virtual learning survive in the postpandemic era. For example, perhaps institutions may wish to continue virtual read-outs with senior radiology residents or with an online platform for resident teaching rounds.

CONCLUSION

Virtual learning was overall well received by radiology residents during the COVID-19 pandemic. Educational activities, such as resident teaching rounds, are a curricular strength. Weaknesses identified include the incorporation of professional development and extrinsic barriers, such as technical difficulties and family obligations, that require further support for trainees. Our study sheds light on important questions regarding virtual education that are important to medical educators, in order to continue to provide residents the training required to be successful radiologists despite these unprecedented times.