Radiation Oncology Virtual Education Rotation (ROVER) 2.0 for Residents: Implementation and Outcomes.

The COVID-19 pandemic catalyzed the integration of a virtual education curriculum to support radiation oncologists in training. We report outcomes from Radiation Oncology Virtual Education Rotation (ROVER) 2.0, a supplementary virtual educational curriculum created for radiation oncology residents globally. A prospective cohort of residents completed surveys before and after the live virtual webinar sessions (pre- and post-surveys, respectively). Live sessions were structured as complex gray-zone cases across various core disease sites. Resident demographics and responses were summarized using means, standard deviations, and proportions. Nine ROVER sessions were held from October 2020 to June 2021. A total of 1487 registered residents completed the pre-survey, of which 786 attended the live case discussion and 223 completed post-surveys. A total of 479 unique radiation oncology residents (of which 95, n = 19.8%, were international attendees) from 147 institutions (national, n = 81, 55.1%; international, n = 66, 44.9%) participated in the sessions. There was similar participation across post-graduate year (PGY) 2 through 5 (range n = 86 to n = 105). Of the 122 unique resident post-surveys, nearly all reported learning through the virtual structure as "very easy" or "easy" (97.5%, n = 119). A majority rated the ROVER 2.0 educational sessions to be "valuable or "very valuable" (99.2%, n = 121), and the panelists-attendee interaction as "appropriate" (97.5%, n = 119). Virtual live didactics aimed at radiation oncology residents are feasible. These results suggest that the adoption of the ROVER 2.0 curricula may help improve radiation oncology resident education.

Introduction

With the COVID-19 pandemic and mandated social distancing, disruption in traditional in-person teaching models led to the creation of alternate virtual educational platforms in residency training [1-8]. Radiation Oncology Virtual Education Rotation (ROVER) is a novel virtual education platform developed to support radiation oncology education for medical students and residents. ROVER was initially implemented in 2020 during the COVID-19 pandemic, when medical student clerkships were on hold, to improve medical student interest and knowledge in radiation oncology [9]. Initially, these virtual didactics were intended to serve as a temporary replacement for in-person classroom teaching; however, virtual learning offers unique advantages as a supplement to in-person teaching even in the post-pandemic setting [1, 4–6, 8]. In the previous meta-analysis, technology-enhanced stimulation in health profession education has shown to be effective in outcomes for knowledge, skills, and behavior [10].

Recent publications have emphasized the wide variation and potential deficiencies in the curricula of radiation oncology residency programs [11-13], with a desire for greater resource sharing between institutions, particularly during the pandemic [11]. As ROVER was found to be feasibly implemented, well attended, and well received, ROVER 2.0 was launched to enhance the education of radiation oncology residents. Live sessions were led by invited faculty from diverse institutions across the USA. ROVER 2.0 sessions focused on difficult cases to stimulate thoughtful discussion and emphasize practice variation in a virtual format, which facilitated increased connectivity. We report here the results of our multi-institutional case-based virtual sessions for radiation oncology residents.

Methods

ROVER 2.0 Design

ROVER 2.0 comprised a series of virtual educational panels tailored for residents. Each session covered gray-zone cases across disease sites representative of core rotations in a typical residency program (Table 1). Faculty members presented complex case variations and areas of clinical equipoise to stimulate discussion around institutional practice differences in radiation oncology.

Table 1

ROVER 2.0 schedule (2020–2021) and participant summary

Session topic	Dates	Registrants (n = 1487)*	Attendees (n = 786)*	Post-surveys (n = 223)*
Gastrointestinal malignancies	October 2, 2020	186	103	50
Genitourinary malignancies	November 6, 2020	159	83	29
CNS malignancies	December 4, 2020	140	58	19
Pediatrics malignancies	January 8, 2021	177	94	27
H&N malignancies	February 5, 2021	206	107	27
Gynecologic malignancies	March 5, 2021	157	82	17
Lymphoma and hematologic malignancies	April 2, 2021	156	101	25
Breast malignancies	May 7, 2021	181	86	15
Thoracic malignancies	June 4, 2021	125	72	14

*Number of registrants, attendees, and post-surveys is the cumulative sum of respondents across all nine ROVER 2.0 sessions

Zoom (Zoom Video Communications, San Jose, CA), a cloud-based video conferencing tool, was used to host the 1-h live sessions (12–1 PM PST). All sessions were password protected and required attendee registration. Zoom features, including poll questions and live chat Q&A, were used to facilitate engagement between the faculty and attendees. Each session was moderated by three to four radiation oncology faculty from different radiation oncology academic programs across the country; faculty were deliberately invited from different geographical areas to promote speaker diversity (Northeast, n = 4; Midwest, n = 7; South, n = 9; West, n = 9). Sessions were advertised on Twitter (Twitter Inc., San Francisco, CA) and distributed to mailing lists of groups including Association of Residents in Radiation Oncology (ARRO), American College of Radiation Oncology (ACRO), and Association of Radiation Oncology Program Coordinators (AROPC), with information to sign up along with fliers. The calendar invite reminder feature was included in the registration to encourage live attendance; however, all sessions were recorded and uploaded to our website archive for asynchronous viewing (https://www.radoncvirtual.com/past-rover-sessions).

Data Collection and Statistical Analysis

This study was exempted by our institutional review board. Resident pre- and post-session surveys (Supplementary Fig. 1) were collected using Research Electronic Data Capture (RedCap), a web-based application for capturing clinical research data hosted by our institution. Although pre- and post-survey results were linked by a respondent, responses were anonymized and aggregated. Residents had to complete the pre-session survey as part of the registration process to receive the Zoom link and password.

Pre- and post-session survey responses were summarized using mean, standard deviations (SDs), and proportions. Responder vs. non-responder characteristics were compared using the Chi-square test, Fisher’s exact test, and Mann–Whitney U test. Inductive thematic analysis was performed by a single user (author NS) to determine the common feedback themes [12]. Statistical analyses were performed using SAS software (version 9.4; SAS Institute, Cary, NC).

Results

Resident Characteristics

Nine ROVER 2.0 sessions were held from October 2020 to June 2021. A total of 786 registered residents attended the live case discussion, and 223 completed post-surveys. There were 479 unique radiation oncology resident attendees (of which 95, n = 19.8%, were international attendees) from 147 institutions (national, n = 81, 55.1%; international, n = 66, 44.9%) (Table 2). The five most represented institutions in the USA were Northwestern University in Illinois (n = 15), Beaumont Hospital in Michigan (n = 14), Johns Hopkins University in Maryland (n = 14), Stanford University in California (n = 13), and University of Miami in Florida (n = 12). International resident attendees (n = 95, 19.8%) were most commonly from institutions in Asia (n = 50, 52.6%), North America (excluding the USA, including the Caribbean; n = 18, 18.9%), Africa (n = 12, 12.6%), and Europe (n = 12, 12.6.%). Table 2 characteristics were also evaluated by international vs. national resident attendees (Supplementary Table 1).

Table 2

Resident attendee demographics and responses related to residency program

	Whole cohort (n = 479)*
Gender
Male	268 (55.9%)
Female	210 (43.8%)
Other	1 (0.3%)
Residency year
PGY-1	33 (6.9%)
PGY-2	86 (18.0%)
PGY-3	102 (21.3%)
PGY-4	105 (21.9%)
PGY-5	87 (18.1%)
Other†	66 (13.8%)
Institution‡
USA¶	354 (73.9%)
International§	95 (19.8%)
Unknown/prefer not to say	30 (6.3%)
Mock oral examinations by home programs
Yes	335 (69.9%)
Yearly	110 (32.8%)
Every rotation	107 (31.9%)
Depends on the rotation	94 (28.1%)
Unknown||	24 (7.2%)
No	144 (30.1%)
Number of hours dedicated to didactics/education for residents by home programs (mean ± standard deviation)	5.5 ± 4.4
Impact of COVID-19
Residency didactic curriculum
Fewer	121 (25.3%)
Unchanged	330 (68.9%)
More didactics	28 (5.8%)
Overall faculty engagement in didactic/teaching sessions
Less engagement	160 (33.4%)
Unchanged engagement	267 (55.7%)
More engagement	52 (10.9%)
Access to faculty
Less access	151 (31.5%)
Unchanged access	291 (60.8%)
More access	37 (7.7%)

*Unique attendees who attended at least one ROVER session were counted only once for the whole cohort sum

†Medical students, fellows, faculty, and international graduates outside the PGY system

‡Unique institutions: N = 147 (national, n = 81; international, n = 66)

¶Five most common institutions: Northwestern University in Illinois (n = 15), Beaumont Hospital in Michigan (n = 14), Johns Hopkins University in Maryland (n = 14), Stanford University in California (n = 13), and University of Miami in Florida (n = 12)

§Continental breakdown: Asia, n = 50; North America (excluding USA and including Caribbean), n = 18; Africa, n = 12; Europe, n = 12; South America, n = 2; Australia, n = 1

||No frequency response captured due to Redcap issue

Among all resident attendees, slightly over half were male (55.9%, n = 268), and representation was similar across PGY-2 through 5 (range of n = 86 to n = 105), with low representation from PGY-1 (6.9%, n = 33) (Table 2). Most residents (69.9%, n = 335) reported having mock oral examinations through their home programs, either yearly (32.8%, n = 110), every rotation (31.9%, n = 107), or rotation dependent (28.1%, n = 94). Residents reported an average of 5.5 h (± 4.4 SD) per week dedicated for didactic education by their home programs. Mock oral frequency was the only significant factor between post-survey responders vs. non-responders (p = 0.0022), with post-respondents reporting a higher value and still finding the ROVER sessions valuable. There were no other significant systematic differences between responders vs. non-responders. Approximately, one-third of the residents reported fewer didactics (25.3%, n = 121), less faculty engagement in didactics (33.4%, n = 160), and less access to faculty (31.5%, n = 151) due to the COVID-19 pandemic. Over half of residents reported that faculty engagement in teaching sessions remained unchanged (55.7%, n = 267), while a third (33.4%, n = 160) reported less overall faculty engagement due to COVID-19. Similarly, most residents reported access to faculty being unchanged (60.8%, n = 291), while almost one-third (31.5%, n = 151) reported less access to faculty due to the pandemic.

Interest in ROVER 2.0 and Virtual Learning

The top three reasons for signing up for sessions were (1) opportunity to hear from a diverse panel of experts (78.1%, n = 374), (2) interest in a new virtual learning platform (39.5%, n = 189), and (3) reduced didactic training due to COVID-19 (12.1%, n = 58). Most residents reported being interested in discussions centered on the following topics during the sessions (non-exclusive): treatment standard of care and existing evidence (70.8%, n = 339), contouring and treatment planning (70.4%, n = 337), case variations (70.0%, n = 335), emerging treatment options (70.0%, n = 335), and case work-up, including imaging review and staging (61.4%, n = 294). Over one-third of residents (38.0%, n = 182) viewed virtual education platforms as equal to in-person learning, while a similar proportion (36.7%, n = 176) reported in-person learning as superior but recognized the need for virtual platforms due to the pandemic.

ROVER 2.0 Perceived Effectiveness

Pre-session, most residents rated their expertise in a session’s disease site as “beginner” (34.0%, n = 506) or “competent” (37.5%, n = 558) (Fig. 1A). Of post-session respondents, 74.8% (n = 167) “agreed” or “strongly agreed” with the statement that they feel more confident in treating the specific disease site because of the session (Fig. 1B). A total of 122 unique residents completed 223 post-session surveys. Most residents (97.5%, n = 119) found learning through the virtual session format to be “very easy” or “easy.” The majority of residents rated the sessions as “valuable” or “very valuable” (66.4%, n = 81, and 32.8%, n = 40, respectively). Nearly all residents (99.0%, n = 121) were satisfied and would recommend the session to a fellow resident. There was no significant difference between post-survey responses for US (n = 31) vs. international unique residents (n = 74).

Fig. 1

Survey responses for disease sites. Responses are shown regarding registrants’ (A) subjective feelings of expertise in the disease site prior to the live session (pre-survey, n = 1487) and (B) confidence in treating the disease site immediately after the live session (post-survey, n = 223). GI, gastrointestinal; GU, genitourinary; CNS, central nervous system; HN, head and neck; GYN, gynecologic

There were three central themes in resident feedback: (1) reduced number of cases for more in-depth discussions, (2) interactive polls, and (3) effective panelist discussions (Table 3). First, reducing the number of cases to 2–3 per session would allow attendees to learn about clinical variation in depth from each panelist. Residents appreciated the integration of polls throughout the sessions to enhance engagement and panelist-attendee interaction. Residents also enjoyed the effective format with multi-institutional panelists shedding light on practice variation across institutions. Most residents (97.5%, n = 119) rated the level of interaction between panelists and attendees as “appropriate.” Finally, in terms of content feedback, 40.2% of residents (n = 49) wanted more time to be spent on discussion of treatment standard of care and supportive evidence, followed by 37.7% (n = 46) interested in dedicating more time to contouring and treatment planning. There were no distinct themes pertaining to different disease subsites.

Table 3

Resident feedback themes and future directions

Themes	Selected comments
Reduce number of cases to 2–3 per session	“Number of cases can be reduced so that discussion in detail can be done” “If there had been two or maybe three cases, we could have gone into a bit more detail to discuss variation in approaches” “Less cases so you can cover more areas of controversy”
Interactive polls	“Enjoyed the poll format which encourages critical thinking” “Continue polls, these are really helpful!” “Use audience polls regularly to increase engagement”
Effective multi-institutional panelist discussion	“I loved the variety of cases and that the panelists addressed how there are multiple ways to do things in our field and why different techniques/plans are more/less appropriate for difference scenarios!” “The discussion among panelist from different institutions are valuable for residents to understand the practice differences” “Excellent and detailed discussion, nuanced teaching points. Great enrichment to resident curriculum” “I felt one of the most valuable aspects of this session was having the panelists comment on how the treatment recommendations vary across institutions”

Discussion

Overall, the multi-institutional ROVER 2.0 program of live virtual case-based sessions was well attended and well received among radiation oncology residents. Notably, the sessions were attended by 479 unique residents from 147 unique institutions globally, with almost 20% representation from international residents (most commonly in Asia). There are an estimated 773 radiation oncology residents and 91 ACGME-accredited programs in the USA in 2020–2021 [14], thus indicating a high turnout rate for the ROVER 2.0 sessions. Additionally, there was a similar mix of all post-graduate levels (with the exception of PGY-1), suggesting the material was perceived as useful for all levels of training; the low PGY-1 participation is not unexpected given PGY-1 s have not yet started at their radiation oncology training program.

Importantly, over half (56–69%) of residents did not report a pandemic-related change in access to faculty, faculty engagement during didactics, or hours of the resident didactic curriculum with an average time of 5.5 h per week, similar to a prior study [11]. Despite the fact that in-person educational opportunities were unchanged for the majority of respondents, the live virtual sessions were still highly ranked and subjectively effective: Nearly all residents would recommend the session to a fellow resident and found the sessions to be valuable or very valuable, and the majority (84%) felt that the session improved their ability to treat the disease site in question. This finding emphasizes that virtual learning provides unique benefits that can supplement standard in-person learning. A common feedback point revealed a critical advantage of this platform: effective multi-institutional panelist discussion (Table 3), with an appreciation for discussion of practice variation and differing treatment approaches (a request of 70% of respondents on the pre-survey). Radiation oncology exhibits strong practice variation between geographies and institutions [15-17], partly related to differences in available technologies, but even beyond that based on training, varying adoption of new studies [17], practice setting (i.e., academic versus private practice [16]), and inherent clinical equipoise for many topics in our field. Thus, our panelists, which were deliberately constructed with an eye to speaker diversity, allowed for thoughtful discussion of varying treatment approaches. By removing typical barriers such as travel, virtual learning allows for multi-institutional (or even global) collaboration and sharing of knowledge/resources in a convenient and cost-effective manner, similarly reflected in other survey studies [1, 8, 18–21]. To this end, the use of gray-zone cases was particularly useful given there was no definitive “correct” answer, thus allowing for thoughtful discussion of different treatment approaches.

A potential concern of added virtual education is that residents may experience burn-out and fatigue with virtual presentations, especially if their didactics were unchanged during COVID. However, approximately 80% found the timing convenient (deliberately scheduled 12–1 PM PST, with strict time constraints), sessions were monthly (and avoided summer months and early in the academic year when residents are adjusting), and once again, nearly all respondents viewed the sessions as valuable despite the majority having unchanged didactics. It should also be emphasized that approximately one-quarter to one-third (25–33%) of residents reported a negative impact of the pandemic on either their dedicated didactic time, faculty engagement during didactics, or faculty access; thus, the virtual sessions may have unfortunately served more than a supplementary role for some.

Even pre-pandemic, there have been concerns regarding the wide variation of radiation oncology curricula between institutions [11-13]. The radiation oncology residency curriculum in the USA is not standardized, unlike the European Society for Radiotherapy and Oncology (ESTRO) Core Curriculum [20] or the European Society for Medical Oncology and American Society of Clinical Oncology (ESMO-ASCO) global core curriculum for medical oncology [21]. There have been recent successful attempts to standardize medical student radiation oncology clerkships by ROECSG [22]. A recently published national survey of program directors highlighted the need to similarly standardize the radiation oncology residency curricula [11, 23]. Notably, in this survey, no programs shared identical curricula content. Additionally, some institutions averaged as low as 20% faculty presence during didactics, and almost one-third of residents reported disruption of didactics by clinical responsibilities. Finally, there was a shared desire for increased faculty involvement in didactics, updated didactic content, introduction of novel evidence-based treatment techniques, and sharing of resources nationally across programs, particularly among small programs with limited resources. Thus, virtual learning can potentially improve and supplement the quality of programs with limited expert faculty or technology by providing high-quality didactics to cover these topics (i.e., brachytherapy, pediatric radiation oncology, lymphoma, proton therapy, CyberKnife/GammaKnife).

An invaluable advantage of traditional in-person didactics is the interactive atmosphere and comradery [1, 7, 8]; the hands-on nature of experiential learning is also hard to simulate virtually [1]. Thus, several studies have shown a desire to return to in-person learning but continuing supplemental virtual learning given the unique benefits as addressed above. A potential benefit of the virtual interactive atmosphere is that participants may feel more comfortable raising questions because of the relative anonymity and dismantling of the physical classroom space [1, 24]. Finally, the interaction was also enhanced by obtaining feedback from registrants on how to improve further sessions; in-person didactics would similarly benefit from obtaining and acting on resident feedback to enhance the learning experience.

We acknowledge several limitations of this study. Most notably, there may be selection bias in residents who participated in the surveys, such that those who are more favorable to virtual learning participated. The post-survey results with the low response rates may be especially impacted by this bias, an inherent limitation of any such survey study. Our pre- and post-surveys also captured subjective evaluations for the sessions rather than evaluating whether the sessions were objectively beneficial in improving resident understanding (alluding to Kirkpatrick’s evaluation of hierarchy in learning, which distinguishes between the perception of knowledge versus actual processing and acquisition of that knowledge [25]). Given the complexity of these sessions, which focused on gray-zone cases with no single correct answer, and discussions regarding subjective practice variation, subjective knowledge assessment and student evaluation of teaching were felt to be more reasonable metrics than objective quizzes. These metrics were also fitting, given the intention of this study was to evaluate the feasibility and reception of these nuanced virtual sessions, which are the first of their kind for radiation oncology residents. That being said, implementation of a more objective evaluation, such as a brief pre- and post-survey quiz, would be warranted to better characterize the effectiveness of the program. Finally, we only acquired the feedback from participants of the live session; registrants who did not attend the live session but viewed the recording at a later date were not quantified nor surveyed. On that note, an added benefit of virtual didactics is, in fact, the ability to record and archive sessions for future viewings, for registrants who could not attend due to scheduling constraints, or even for future residents (rather than repeating the live webinar annually).

Because live sessions were recorded and archived, and due to residents’ desire for more nuanced topics under-represented in their core curricula, such as safety/quality improvement, palliative care, and interviewing/job search [11, 13], sessions for ROVER 2.0 2021–2022 will expand to include the aforementioned topics, as well as other specialized topics like brachytherapy and advanced treatment modalities (i.e., proton therapy, stereotactic body radiotherapy [SBRT]). Resident feedback was vital in this first iteration of ROVER 2.0 to structuring the future 2021–2022 sessions.

Conclusion

Based on our results, virtual radiation oncology teaching is a viable tool for global resident learning that should continue to have a role post-pandemic. Approximately 99% of the residents reported this virtual education platform as valuable, with one-third reporting decreased faculty engagement due to COVID-19. Thus, the continued implementation of a virtual radiation oncology curriculum will serve as an adjunct to traditional resident education for learners globally.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 921 KB)