Evaluation of an interactive virtual surgical rotation during the COVID-19 pandemic.

OBJECTIVE: To evaluate medical student and attending surgeon experiences with a novel interactive virtual Otolaryngology - Head and Neck Surgery (OHNS) medical student elective during the COVID-19 pandemic. STUDY
DESIGN: A virtual OHNS elective was created, with three components: (1) interactive virtual operating room (OR) experience using live-stream video-conferencing, (2) telehealth clinic, (3) virtual didactics.
SETTING: OHNS Department at the University of Pennsylvania (May 2020 to June 2020).
METHODS: Six medical students from the University of Pennsylvania; five attending otolaryngologists. Two surveys were designed and distributed to participating medical students and attending surgeons. Surveys included 5-point Likert scale items, with 1 indicating "not at all" and 5 indicating "very much so".
RESULTS: Response rate was 100% for both surveys. Students on average rated the educational value of the telehealth experience as 4.2 ± 1.2, and the virtual OR experience as 4.0 ± 0.6. Most students (n = 5, 83%) indicated that they had enough exposure to faculty they met on this rotation to ask for a letter of recommendation (LOR) for residency if needed, while attending surgeons had an average response of 3.0 ± 1.0 when asked how comfortable they would feel writing a LOR for a student they met through the rotation. A majority of students (n = 4, 67%) felt they connected enough with faculty during the rotation to ask for mentorship. Half the students (n = 5, 50%) indicated that the rotation allowed them to evaluate the department's culture either "extremely well" or "somewhat well".
CONCLUSIONS: Overall, participating students described this innovative virtual surgical rotation as an educationally and professionally valuable experience. With the continued suspension of visiting student rotations due to the COVID-19 pandemic, this virtual model may have continued relevance to medical education.

INTRODUCTION

The novel coronavirus 2019 (COVID‐19) pandemic has disrupted every facet of the United States (US) healthcare system, including undergraduate medical education. On March 17, 2020, the American Association of Medical Colleges released recommendations that in‐person clinical learning for medical students across the country should be halted. This suspension continued from March 2020 until June 2020, when hospitals slowly began to re‐integrate students into clinical care teams. During this prolonged suspension of clinical rotations, many institutions developed virtual curricula for medical students. , ,

Despite the return of medical students to clinical rotations at their home institutions, visiting student learning opportunities (VSLOs), rotations which for many specialties usually play a critical role in the residency application process, , remain indefinitely suspended. Hence, research examining the implementation and effect of virtual medical education remains of high importance. For surgical specialties, these virtual curricula have included learning tools such as targeted readings, case‐based discussions, simulated or real patient interactions, and instructional surgical videos. , , , , However, literature evaluating the effectiveness of these interventions is limited, and few studies have addressed the significant problem of lack of student access to the operating room (OR) for interactive learning. One study described a virtual surgical subinternship that included simulated surgical skills workshops, and another presented a case report of live‐streaming from the OR for medical student education, but neither have yet evaluated the educational value of these virtual experiences from student or faculty perspectives.

The authors present a single institution's critical evaluation of an innovative virtual Otolaryngology ‐ Head and Neck Surgery (OHNS) medical student elective. This previously described virtual curriculum emphasizes active participation and includes an innovative and interactive live‐stream OR experience, along with virtual clinic and didactic components. ⁠ This survey‐based study aims to analyze the value of our virtual curriculum to medical students, both educationally and professionally, as they choose a specialty and prepare to apply to residency.

MATERIALS AND METHODS

Virtual elective structure

A 2‐week virtual elective in OHNS was offered to medical students at our institution from May 2020 to June 2020. Four sessions of the course were completed, each with 1‐2 students enrolled, for a total of 6 students over two months. Five attending surgeons participated in the course. The course comprised three major components, which we have previously described in greater detail :

Virtual OR: Attending surgeons wore a head‐mounted GoPro® camera which live‐streamed video of the surgical field via a Health Insurance Portability and Accountability Act (HIPAA) compliant, video‐conferencing platform, BlueJeans (BlueJeans, Verizon Enterprise Solutions LLC, Mountainview, California). Students on the video conference were then able to, in real time, view the operation and converse with attending surgeons, who were wearing wireless headphones connected to the call.

Telehealth: Students joined attending surgeons’ video‐conferencing visits with patients, conducted via BlueJeans or Doximity (Video Dialer Beta, Doximity Inc., San Francisco, California). For some visits, students performed an independent history and virtual “physical”, presented the patient to the attending, and then returned to the video call with the attending for the full visit.

Virtual Didactics: Medical students presented patient cases at virtual multidisciplinary head and neck tumor board and attended virtual OHNS grand rounds and resident didactics. Students participated in small group discussions and lectures with attending otolaryngologists covering a wide variety of OHNS topics.

Study outcome measures

The value of the course was determined by six (6) outcome measures, assessed both through student perceptions (items 1‐3 below) and attending surgeon perceptions (items 4‐6 below). See Appendix A for an outline of which survey items correspond to which of these six outcome measures.

Student‐related outcome measures: ① Feasibility, ② Educational benefit, ③ Professional benefit.

Attending‐related outcome measures: ④ Feasibility, ⑤ Ability to provide professional support to students, ⑥ Overall impressions.

Of note, content‐based learning objectives were provided to students upon enrollment. These served as a guidepost for student learning throughout the course. However, these learning objectives should be distinguished from the course objective, which was to introduce students to the subspecialty of Head and Neck oncologic surgery and to offer the opportunity to interact directly with attending surgeons through live‐streamed surgeries, telemedicine, and case‐based discussions. Students’ perception of the educational value of the course components, as assessed by our study, was likely based on the content‐based learning objectives, which were:

Understand the risk factors and epidemiology of head and neck cancer

Create a management algorithm for patients presenting with a neck mass

Identify key portions and pitfalls of head and neck surgical operations (i.e. neck dissection, parotidectomy)

Survey design, distribution, and analysis

Two surveys were created to evaluate student and attending experiences, respectively [see Appendices B and C]. The surveys were designed to assess the outcome measures of the study (see Section 2.2), and underwent at least 3 rounds of edits with different members of the research team to ensure they achieved this goal. For some questions, a 5‐point Likert scale was used with 1 indicating “not at all” and 5 indicating “very much so”. Optional comment boxes were included in both surveys for gathering of qualitative data. Surveys were distributed to students and attendings by email. Study data were collected and managed using REDCap electronic data capture tools hosted at the University of Pennsylvania. , ⁠ Survey data were analyzed using mean and standard deviation (SD). All statistics were calculated using Numbers [version 4.1.1 (4338), Apple iWork Suite, Cupertino, California]. This project was reviewed and granted exemption by the University of Pennsylvania Institutional Review Board.

RESULTS

Response rate was 100% for both the attending (n = 5) and student (n = 6) post‐course surveys. Survey results regarding student and attending perceptions of course components (telehealth, operating room, didactics, tumor board) are presented in Table 1.

Table 1

Student and attending perceptions of course components

Items	Mean [±SD, (range)], 5 point Likert scale a
Course component	Students:	Attendings:
	How educational was your experience with …	How manageable was it to have students participate in…
…telehealth?	4.2[±1.2, (2~5)]	3.8 [±1.3, (2~5)]
…virtual operating room?	4.0[±0.6, (3~5)]	4.0 [±0.8, (3~5)]
…virtual didactics (all parts)?	4.5 [±0.5, (4~5)]	‐‐
…presenting at tumor board?	4.3 [±0.5, (4~5)]	‐‐

1 = not at all; 5 = very much so

Telehealth

During the telehealth portion of the course, all but one student (83%, n = 5) had the opportunity to perform an independent history and virtual “physical exam” and formally present the patient to the attending surgeon prior to observing the attending's visit with the patient. All students (n = 6, 100%) reported an increase in their level of comfort with telehealth from before beginning the course to after completing it. Reported comfort levels in telehealth increased from a mean of 2.3 ± 0.8 to a mean of 4.0 ± 0.6. Students on average rated the educational value of the telehealth experience as 4.2 ± 1.2 and attending surgeons on average ranked the experience of having medical students in telehealth clinic as 3.8 ± 1.3. Table 2.

Table 2

Perceived potential for networking and evaluation

Comfort with seeking mentorship or research
Items	Yes	No	Unsure	‐	‐
Students: During this rotation did you connect with residents or faculty enough to engage in research?	3 (50%)	2 (33%)	1 (17%)	‐	‐
Students: Did you connect enough with residents to ask for advice or mentorship?	3 (50%)	1 (17%)	2 (33%)	‐	‐
Students: Did you connect enough with faculty to ask for advice or mentorship?	4 (67%)	0	2 (33%)	‐	‐
Evaluation of department and applicants
	Responses, number(%)
Items	Extremely well	Somewhat well	Neutral	Not very well	Not at all
Students: How well were you able to evaluate department culture/fit during this rotation?	1 (17%)	2 (33%)	3 (50%)	0	0
Items	Mean[±SD, (range)], 5 point Likert scale a
Attendings: How well were you able to get to know medical students during this rotation?	3.2 [±0.8, (2~4)]
Letters of recommendation (lors) for residency applications
	Responses, number (%)
Items	Yes, definitely	Maybe	Only if I really needed it	Definitely not	‐
Students: Did you have enough exposure to faculty (met for the first time on this rotation) to ask for a LOR?	0	0	5 (83%)	1 (17%)	‐
Items	Mean [±SD, (range)], 5 point Likert scale a
Attendings: How comfortable would you feel writing a LOR for a student (met for the first time on this rotation)?	3.0 [±1.0, (2~4)]

1 = not at all; 5 = very much so

When asked for additional comments, students voiced particular appreciation for the telehealth component of the course. One commented that “‘seeing’ the patient on my own first, and then presenting to the attending…… really helped with my comprehension”, while another similarly noted that “the best learning experiences were seeing patients on my own and presenting to the attending…… this prepared me better……to see how the attending proceeded with the patient”. A third student remarked that this rotation was a great “primer” for their subsequent in‐person OHNS elective. Students described the experience as “enjoyable” and “educational”.

Virtual operating room

All students (100%, n = 6) reported that, for the virtual operating room (OR) experience, their level of interaction with the attending surgeon was “about the right amount”. On average, students rated the educational value of the virtual OR experience as 4.0 ± 0.6. All but one student (n = 5, 83%) were either “somewhat likely” or “very likely” to recommend the virtual OR experience to a classmate. Attending surgeons on average ranked the experience of having medical students in the virtual OR as 4.0 ± 1.3.

Virtual didactics

The majority of students (n = 5, 83%) reported that the amount of time spent on virtual didactics was “just right”; the remaining student thought it was “too little” (n = 1, 17%). Out of the three didactic components (student‐only didactics, resident didactics, and tumor board), a majority (n = 5, 83%) enjoyed student‐only didactics the most. Within the student‐only didactics, 100% of students (n = 6) preferred the interactive case discussion format over the lecture format.

Professional implications

When asked whether this virtual elective helped them make a decision on whether to apply into OHNS, four students (67%) said “no”, and two (33%) said “yes”. Of those that said “no”, the majority (n = 3, 75%) had already decided to apply prior to the start of the course, and the rest (n = 1, 25%) decided based on other factors.

When asked about letters of recommendation (LORs) for residency applications, most students (n = 5, 83%) indicated that they had enough exposure to faculty they met for the first time on this rotation to ask for a LOR “only if ‘they’ really needed it”. Attending surgeons were also asked how comfortable they would feel writing a LOR for a student they met for the first time through the virtual elective; the average response was 3.0 ± 1.0.

The majority of students (n = 4, 67%) felt they connected enough with faculty during the rotation to feel comfortable asking for advice or mentorship. When asked the same question about residents, half said they would feel comfortable (n = 3, 50%). When asked how well they were able to evaluate the department's culture and fit, half of students selected either “extremely well” (n = 1, 17%) or “somewhat well” (n = 2, 33%).

According to student responses, difficulties with technology were present in both the telehealth and virtual OR sections of the course but were more prevalent in the latter (Table 3). One attending commented that this elective, and the virtual experience in general, would likely be “much easier” a second time if a resurgence of the pandemic necessitates it. Another thought the virtual elective was an “excellent option” given the situation but that it “cannot and should not replace a real rotation”.

Table 3

Prevalence of technology challenges

Items	Responses, number (%)
Students:What percentage of the time did you experience technical difficulties with…	0~20%	21%~40%	41%~60%	61%~80%	81%~100%
…telehealth?	3 (50%)	3 (50%)	0 (0%)	0 (0%)	0 (0%)
…virtual operating room?	1 (17%)	3 (50%)	0 (0%)	2 (33%)	0 (0%)
Students:For the virtual operating room experience, how was the quality of the…	Superb	Good	Okay	Poor	Very Poor
…audio?	0	3 (50%)	3 (50%)	0	0
…video?	1 (17%)	0	4 (67%)	1 (17%)	0

DISCUSSION

In this study, the authors describe a single institution's experience with and evaluation of an innovative and interactive 2‐week virtual OHNS surgical elective utilizing head‐mounted GoPro® video technology and live HIPAA compliant two‐way audiovisual video conferencing. Overall, participating students described the rotation as an educationally and professionally valuable experience.

In this study, student perception of the educational value of the interactive telehealth experience was high. This is supported by existing literature, in which learning about and experiencing telemedicine has been well‐received by medical students both during and prior to the COVID‐19 pandemic. , , ⁠ Satisfaction with the virtual OR experience was also high among our respondents. Data against which to compare these results is limited, as few virtual OR experiences have been described. One study of an ophthalmology course for pre‐clerkship students suggested that virtual surgical simulation was well‐received. Recent publications discuss use of intraoperative videos for subsequent surgical teaching, virtual suturing and laparoscopic skills workshops, or even a case report of an OR live‐stream being used to teach medical students, , , ⁠ but their effect on learners has not been investigated. Finally, our didactic component was also well received. The body of literature on virtual didactics during the COVID‐19 pandemic is more robust than for other methods. Studies of didactic‐based virtual electives show high student satisfaction as well as significant improvement in medical student performance in post‐ versus pre‐tests on specialty‐specific material. ,

Our results are particularly relevant given the aforementioned continued suspension of visiting student learning opportunities (VSLOs), or colloquially, “away rotations”. ⁠ Participation in VSLOs is especially prevalent in highly competitive surgical subspecialties such as otolaryngology, plastic surgery, orthopedic surgery, and urology. In many of these fields, more than half of residency applicants ultimately match at an institution in which they completed an away rotation. ⁠ While many institutions have created “virtual sub‐internships” geared toward students from other institutions, these experiences have focused on providing information about a program's features, creating a platform for students to interact with faculty and residents, and offering virtual access to departmental conferences. ⁠ Efforts at improving virtual surgical elective options may remain relevant even after the COVID‐19 pandemic fades, especially for students without home programs in their chosen specialties, who are less likely to match at their top choice program despite being equally competitive applicants.

Limitations of this study included small sample sizes, subjective‐only measurement of educational value (ie, lack of content‐based pre‐ and post‐test), as well as technology challenges. For the latter, in the authors’ experience with the course, these technology issues demonstrated a learning curve and were most prevalent in the first iterations of the course, with substantial improvement as problems were identified and corrected. Frequent feedback was sought from students for this purpose. Another somewhat inevitable limitation was the inability for faculty to assess students’ surgical skills due to the virtual nature of the OR experience. Further development of this course could include creative solutions to this problem, such as a virtual attending‐ or resident‐led skills workshop using university‐issued suture kits.

Though a virtual elective should not replace an in‐person OR experience, our model for a surgical elective provides a feasible alternative that allows students to gain some insight into the culture and fit of the program. Furthermore, given the void left by the suspension of VSLOs, our surgical curriculum could serve as a basis for developing a virtual VSLO with higher fidelity to an in‐person rotation than the options currently offered by surgical residency programs.

CONCLUSIONS

Traditional paradigms in medical student surgical education have shifted due to the COVID‐19 pandemic. Though students have largely returned to home clinical rotations, the ability to rotate at outside institutions through VSLOs remains suspended. Our two‐week interactive virtual surgical elective, including an innovative live‐stream OR experience, has been well‐received by our home institution students and faculty during its preliminary iterations. Students reported that the virtual OHNS elective was not only educationally valuable but also gave them the opportunity to connect with department faculty. This study provides a promising model for a virtual surgical elective. As VSLOs remain suspended indefinitely, the implementation of virtual surgical curricula enhances medical students’ understanding of the specialty and builds relationships within the department.

FINANCIAL DISCLOSURES

None.

DECLARATION OF COMPETING OF INTEREST

None.