Implementation and Evaluation of a Formal Virtual Medical Student Away Rotation in Orthopaedic Surgery During the COVID-19 Pandemic: A Single Institution Pilot Experience.

Away rotations are a valuable experience for medical students when applying for residency. In light of the coronavirus disease 2019 pandemic, there has been significant interest in the development of virtually based substitutes. This study evaluates the utility of a formal virtual fourth-year medical student away rotation in orthopaedic surgery by surveying participants and provides recommendations for success.
METHODS: A 2-week virtual orthopaedic elective was offered to fourth-year medical students in lieu of traditional in-person away rotations. The course consisted of multiple components such as subspecialty case-based didactics, "happy hours" with residents, assigned resident mentors, student case presentations, and observation of resident lectures. After course completion, anonymous surveys were administered to participants to evaluate the rotation.
RESULTS: Twenty-three of 24 participating students (96%) completed the student survey, and 22 of 24 participating faculty and residents (82%) completed the resident/faculty survey. Most students were very (87%) or somewhat (9%) satisfied with their experience and found the rotation to be a very (35%) or somewhat useful (61%) substitute for an in-person rotation. Students indicated that the rotation very (91%) or somewhat positively (9%) influenced their perception of the program. All students indicated that the rotation was very educational. Most students (91%) reported that the rotation was very useful for learning about the program and culture, with subspecialty didactics and happy hours most useful. Faculty and residents indicated that the rotation was useful for getting to know the students (17% "very useful" and 83% "somewhat useful") and for assessing student characteristics, such as knowledge base and communication skills.
CONCLUSIONS: A formal virtual orthopaedic surgery away rotation can be a valuable experience for medical students which provides educational value, insight into program culture, and an opportunity to demonstrate interest in the program. The curriculum and recommendations presented in this study can be used as a preliminary template for others interested in creating successful virtual rotations.

The coronavirus disease 2019 (COVID-19) pandemic has led to significant ramifications for medical education. In an attempt to minimize virus spread and conserve personal protective equipment, efforts have been taken to limit unnecessary patient care contact for medical students. During the 2020 to 2021 residency application cycle, the Coalition for Physician Accountability, including the American Association of Medical Colleges, instituted recommendations discouraging medical students from performing clinical rotations at outside institutions (“away rotations”)[1].

Away rotations play an essential role for fourth-year medical students during the residency application process, particularly within competitive specialties such as orthopaedic surgery. Away rotations not only have an educational value, but they allow students to learn more about that training program and may improve their competitiveness in the match[2,3]. Given the need for alternative rotation offerings during the pandemic, there has been significant interest in the development of virtually based rotations as a substitute[4-7]. Despite hope for resumption of normal in-person activities in 2021, this has not yet been realized. As a result, the updated guidelines recommend limiting applicants to 1 away rotation and highly encourage virtually based alternatives for the 2021 to 2022 application cycle[8].

Because formal virtual rotations are a novel undertaking in medical student education and residency recruitment, there is a minimal understanding of their value or which components are most beneficial. The purpose of this study was to (1) describe a single institution's approach to a formal, virtually based, medical student away rotation in orthopaedic surgery, (2) evaluate the utility of this virtual rotation by surveying participants, and (3) provide recommendations for successful execution of a virtual rotation based on our initial experience.

Methods

This study was determined to be exempt by the Institutional Review Board at the University of Rochester.

Curriculum Format

A 2-week formal virtual orthopaedic surgery rotation was offered to fourth-year medical students from outside institutions in lieu of a traditional in-person rotation. Virtual rotation applicants were not subject to a screening process as is usually required for participation in our traditional in-person rotation. Participating students were placed in 1 of 2 sessions that were conducted consecutively from October to November 2020. All course events were conducted through online video conferencing (Zoom Technologies). Because of the potential for exposure to protected patient health information, all students were required to complete confidentiality agreements through the university registrar in a similar manner to traditional in-person rotators. The rotation consisted of multiple components intended to maximize educational value, provide insight into the program and culture, and allow interaction with a variety of residents and faculty members (Fig. 1).

Fig. 1

Orthopaedic surgery virtual rotation schedule.

Subspecialty-specific didactic sessions were conducted in the evenings, led by a faculty member from that division and a resident. Different faculty members and residents were present each evening, increasing the variety of connections students were able to make. Didactics were primarily case-based, and students were given a prereading assignment related to each evening's topic. Time was allotted at the end of each session for the resident to talk about their experience on that subspecialty rotation and answer questions without the faculty member present to facilitate candid discussion. Weekly “happy hours” allowed students to interact socially with residents. During these happy hours, participants were split into groups of 3 to 4 students with 2 to 3 residents to better engage in conversation.

At the end of their rotation, students gave a 10-minute case presentation to residents and faculty on an orthopaedic topic of interest, allowing students an opportunity to demonstrate their knowledge base and communication skills to a large number of individuals in the program. Students were prompted to choose an interesting case they had been involved with on a clinical rotation and use those details to teach the audience about the pathology (diagnosis, workup, classification, and treatment). This format was consistent with the presentations typically given by our in-person rotators to allow for comparable performance assessments.

While participating in the rotation, students were also encouraged to observe the regularly scheduled early morning residency lectures and daily “fracture sign-out” where consults from the previous day are discussed. In addition to the course curriculum, each student was assigned a resident mentor to personalize their experience and to serve as a primary contact they could bring questions to. Mentors and mentees were encouraged to set up meetings to “check in” and get to know each other better. Mentors also served as a useful resource to help choose topics and provide feedback for the students' case presentations.

Course Evaluation

After completion of each virtual rotation session, students were invited through email to participate in an anonymous survey providing feedback on their rotation experience. Similarly, after completion of both sessions, residents and faculty members directly involved in rotation events were invited to complete a separate anonymous survey about their experience. Survey questions included a mixture of multiple-choice and free-response questions to assess participants' opinions on various aspects of the rotation. Survey responses were analyzed and reported using descriptive statistics, including percentages, means, ranges, and standard deviations. Feedback obtained from free-response items was compiled and assessed in a qualitative manner to determine prevailing themes and make recommendations.

Interview Attainment

Participation in the virtual rotation was noted in the students' residency applications and taken into consideration when offering interview invitations. After completion of residency interviews for the 2020 to 2021 application cycle, departmental records were reviewed to determine the rate of interview attainment among virtual rotation participants. The interview rate for virtual rotators was compared with that of nonrotator applicants (neither virtual nor in-person) from the same year and with that of traditional in-person away rotators from the previous 3 application cycles (2017 to 2018, 2018 to 2019, and 2019 to 2020).

Results

Student Survey

A total of 24 students participated in the virtual rotation (12 per session), 23 of whom (96%) completed the student survey. Responses to various multiple choice questions from the student survey are detailed in Figure 2. Most of the students were very (87%) or somewhat (9%) satisfied with their experience and found the virtual rotation to be a very (35%) or somewhat (61%) useful substitute for an in-person rotation. Students also reported that the virtual rotation very (91%) or somewhat positively (9%) influenced their perception of the residency. Students indicated that they gained sufficient insight into faculty-resident dynamics (78% “strongly agree”) and were satisfied with the quality of their interactions with residents (96% “strongly agree”) and faculty members (83% “strongly agree”). To a lesser extent, students also felt that residents and faculty members were able to get to know them (52% “strongly agree” and 35% “somewhat agree”).

Fig. 2

Assessment of virtual rotation experience according to student survey responses.

Table I describes the reported educational value of virtual rotation components. All students (100%) indicated that the rotation overall was very educational. Student case presentations and the evening subspecialty didactic sessions were found to be the most educational, whereas observing morning resident conferences and fracture sign-outs were least educational. Most of the students (91%) reported that the virtual rotation overall was very useful for learning about the program and culture. Subspecialty didactic sessions and happy hours with residents were found to be the most useful, whereas fracture sign-out was the least useful. The reported utility of all virtual rotation components for learning about the program and culture is detailed in Table II.

Resident and Faculty Survey Responses

A total of 14 residents and 8 faculty members participated in virtual rotation events. The resident and faculty survey was completed by 18 (11 residents and 7 faculty members) of these 22 participants (82%). Residents and faculty members found the virtual rotation to be very (17%) or somewhat (83%) useful for getting to know the students. Resident mentoring was found to be the most useful, whereas morning conferences and fracture sign-outs were least useful (Table III). In addition, residents and faculty members felt that students improved their chances of matching at our residency program by participating in the virtual rotation (39% “strongly agree” and 50% “somewhat agree”).

The reported utility of the virtual rotation for assessing various student characteristics is detailed in Figure 3. Residents and faculty members reported that the virtual rotation was useful for assessing knowledge base (11% “very useful” and 61% “somewhat useful”) and interpersonal communication skills (11% “very useful” and 56% “somewhat useful”). The virtual rotation was not useful for assessing the students' work ethic (61% “not very useful” and 33% “not useful at all”).

Fig. 3

Utility of virtual rotation for assessing student characteristics as indicated by resident and faculty member survey responses.

Qualitative Feedback and Recommendations for Virtual Rotation Success

In the qualitative responses, several themes were highlighted as positive features of the virtual rotation, including the small group sizes, educational value of didactic sessions, quality of interactions with residents and faculty through didactic sessions, resident mentors, happy hours, opportunities for participation through the didactic sessions, and student case presentations. Limited student interaction and technical challenges during the early morning sessions (fracture sign-out and morning resident conferences) were commonly cited as areas for improvement. After review of the surveys and all qualitative feedback, recommendations were compiled as guidance for virtual rotation development (Table IV).

All 24 virtual rotators applied to our residency program during the 2020 to 2021 application cycle, and 63% of them were offered and attended an interview. Meanwhile, the interview invitation rate for nonrotators during the 2020 to 2021 application cycle was 12%. In comparison, the interview invitation rate for in-person away rotators over the previous 3 application cycles was 94%.

Discussion

Virtual rotations are a novel undertaking in medical student education and residency recruitment because of the COVID-19 pandemic. Given the continued recommendations for virtually based rotation offerings into the 2021 to 2022 application cycle[8], along with the limitation on away rotation numbers, it is important to better understand their value and which features will maximize their success. This study evaluated the utility of a formal virtual fourth-year medical student away rotation in orthopaedic surgery and provides preliminary recommendations for rotation success. In this study, students reported a high level of satisfaction with the virtual rotation and found it to be educational and useful for learning about the program and culture. These results support the findings of similar recent investigations in radiology, urology, otolaryngology, and internal medicine-pediatrics[9-13]. This suggests that a virtual rotation can function as a reasonable substitute when in-person rotations are not possible or are limited in number.

The results of this study suggest that both students and residency programs can benefit from virtual rotations. These rotations provide an opportunity for students to make an impression on programs and help set themselves apart among an increasingly large and competitive applicant pool[14]. They can also provide students with greater insight into a program and allow them to assess the educational quality and culture to help determine whether it would be a good fit for them. However, the virtual format does not impart all the advantages of in-person rotations. Although residents and faculty members felt that they were able to assess many qualities of value for a prospective resident, characteristics such as work ethic or technical skill were not adequately assessed. In addition, the virtual format may not provide students with the in-depth immersion and prolonged interactions desired to obtain quality letters of recommendation or adequately assess for concerns within the program. Although these limitations make it unlikely that virtual rotations could fully substitute for in-person rotations, they do provide more insight than could be obtained through review of a program's website or a student's application or during the limited interactions on an interview day. In the future, virtual rotations may remain an attractive adjunct, particularly among applicants for whom multiple in-person away rotations may be cost-prohibitive or those with scheduling constraints. Virtual rotations should be considered as an additional tool for residency recruitment that can increase access for disadvantaged applicants and encourage a more diverse applicant pool[15,16].

In this study, virtual rotators received interview invitations at a much higher rate than nonrotators, although not as high as traditionally seen among in-person away rotators. At our program, students who apply for in-person away rotations undergo a holistic screening process with consideration given to both academic merit and a requested statement of interest. Given the atypical circumstances related to the COVID-19 pandemic, no formal screening process was used for this virtual rotation so that any interested students were able to participate. Because screening and application review processes are highly variable and often subjective, our reported interview offer rates are not intended to provide students with specific expectations for their experience at other programs. However, these data support the notion that virtual rotations may enhance applicants' chances of receiving an interview offer, although it may not impart the full benefits traditionally associated with away rotations[2,14]. In the future, programs may consider a screening process for virtual rotations to maximize applicant benefit and allow those unlikely to receive an interview a chance to explore opportunities elsewhere. Furthermore, transparency from programs during this process is recommended so that applicants can make informed decisions about which virtual rotations will be the most beneficial use of their time and energy.

The feedback we obtained allowed us to develop multiple recommendations for successful virtual rotations. Both our results and those of other studies have suggested that the main priority for virtual rotations should be providing high-quality interactions between students and members of the residency program[10,17,18]. This was best accomplished during our rotation through small group sizes, mentoring and resident-only social activities, dedicated didactic sessions, and student case presentations. During our rotation, students were able to observe resident-specific conferences; however, most lectures were geared toward review for the residents' upcoming in-training examination, and the high trauma volume at our institution necessitates a rapid fracture sign-out, with little time for educational discussion or student participation. As a result, these components of the rotation were less useful compared with the other rotation events. We suggest creating opportunities for students to participate and ask questions whenever possible and to consider alternative activities that might be of greater value when unable to make significant changes. Furthermore, the timing of morning conferences was dictated by the residency program schedule, so attendance may have been challenging for students in different time zones or those completing clinical rotations concurrently. These factors should be taken into consideration when scheduling rotation events. If sessions must occur at potentially difficult times, we recommend making attendance optional so as to not overly burden the students. In addition, rotation schedules should be made available to students early so they can plan accordingly.

This study is not without limitations. First, this investigation was performed at a single institution, which may limit its generalizability to other residency programs. Although our program shares many similar qualities to other orthopaedic residencies, it will be important to consider how the unique characteristics of a program may influence the feasibility of certain curriculum elements. Second, despite the anonymous nature of the survey, there is a potential for response bias from students because surveys were administered in the midst of their application cycle and from some residents and faculty members who were involved in curriculum design. Finally, the virtual rotation has been completed by a relatively small number of participants to date, which limits the sample size of this study. Virtual rotations are a novel offering necessitated by the COVID-19 pandemic, and a few studies reporting on them to date have been subject to similar limitations[9-11,16,19,20]. However, further investigations among larger cohorts with longitudinal follow-up will be necessary. Despite these limitations, this study was the first of its kind in orthopaedic literature to provide valuable preliminary information to assess the utility of virtual rotations and help other programs successfully develop similar curriculum offerings.

Conclusions

A virtual orthopaedic surgery away rotation can be a valuable experience for medical students, which provides educational value, insight into a program's culture, and an opportunity to demonstrate interest in the program. Virtual rotations can also be a useful tool for residencies to share information about their program and learn more about prospective applicants. In the future, virtual rotations may remain a beneficial tool for residency recruitment that can increase access for disadvantaged applicants and encourage a more diverse applicant pool, although will not completely replicate the in-person experience. The curriculum and recommendations presented in this study can be used as a preliminary template for other residency programs interested in creating successful virtual rotations, and students can use them as a basis for comparison when choosing virtual rotations to participate in.

TABLE I

Student Ratings of Virtual Rotation Components According to Educational Value

Rank	Score*	Rotation Component
	1.00 ± 0.00 (1-1)	Overall
1	1.00 ± 0.00 (1-1)	Student case presentations
2	1.04 ± 0.21 (1-2)	Didactic sessions
3	1.77 ± 0.75 (1-3)	Morning conferences
4	2.27 ± 0.94 (1-5)	Fracture sign-outs

Student responses were recorded with range of values from 1= very educational to 5= not educational at all.

Represented by mean ± SD (range).

TABLE II

Student Ratings of Virtual Rotation Components According to Utility in Learning About the Program and Culture

Rank	Score*	Rotation Component
	1.09 ± 0.29 (1-2)	Overall
1	1.04 ± 0.21 (1-2)	Didactic sessions
1	1.04 ± 0.21 (1-2)	Happy hours with residents
3	1.39 ± 0.66 (1-3)	Student case presentations
4	1.52 ± 0.85 (1-4)	Resident mentors
5	1.55 ± 0.60 (1-3)	Morning conferences
6	1.86 ± 0.94 (1-5)	Fracture sign-outs

Student responses were recorded with range of values from 1= very useful to 5= not useful at all.

Represented by mean ± SD (range).

TABLE III

Resident and Faculty Ratings of Virtual Rotation Components According to Utility in Getting to Know Students

Rank	Score*	Rotation Component
	1.83 ± 0.38 (1-2)	Overall
1	1.40 ± 0.55 (1-2)	Resident mentors
2	1.71 ± 0.83 (1-3)	Student case presentations
3	1.88 ± 0.35 (1-2)	Happy hours with residents
4	1.92 ± 0.67 (1-3)	Didactic sessions
5	3.58 ± 0.51 (3-4)	Morning conferences/fracture sign-outs

Resident and faculty responses were recorded with range of values from 1 = very useful to 4 = not useful at all.

Represented by mean ± SD (range).

TABLE IV

Recommendations for a Successful Virtual Rotation Based on our Initial Experience and Participant Feedback

Minimize burden of time commitment

Limit required rotation activities to a suggested 1-2 hours per day and an overall duration of 2 weeks

Thoughtfully schedule activities (consider differences in time zones and concomitant in-person clerkships)

Low threshold to make events optional if not geared toward students or at inconvenient times

Maximize networking opportunities and insight into program culture

Maintain small group sizes (breakout rooms can be a useful adjunct)

Include a variety of faculty and resident participants

Provide multiple opportunities for resident-only interactions

Incorporate social activities

One-on-one mentorship (resident and/or faculty)

Clear guidance on their role before rotation start date

Dedicate times for check in at start and/or middle of rotation

Didactic sessions geared toward student education

Case-based format for enhanced student participation

Provide prereading questions or learning objectives for each session

Consider involvement from both faculty and residents for greater insight into faculty-resident dynamics

Student case presentations for formal evaluation of student characteristics

Maintain similar format to presentations by in-person rotators (if applicable) for more comparable assessments

Provide clear guidance on format at the beginning of rotation

Accommodate limitations to medical record access at home institution

General/other considerations

Distribute schedule to participants as early as possible

Provide clear expectations for attendance and instructions on how to address schedule conflicts

Consider unique program characteristics that may influence feasibility and optimal format of rotation activities

Maximize virtual engagement by encouraging camera use and confirm technology is working appropriately