Adapting to the Educational Challenges of a Pandemic: Development of a Novel Virtual Urology Subinternship During the Time of COVID-19.

OBJECTIVE: To design, implement, and evaluate learner attitudes of a virtual urologic surgery clinical rotation for medical students.
METHODS: Ten senior medical students at the Perelman School of Medicine at the University of Pennsylvania were enrolled. Students were administered a precourse test on their perceived confidence of their urologic knowledge, confidence in identifying urologic conditions, comfort with performing urologic evaluations, and confidence placing consults for urologic issues. Students participated in a 2-week curriculum that included both asynchronous and synchronous content. Asynchronous content included prerecorded lectures, self-paced problem-based learning modules, directed reading and video content, and an online discussion board. Synchronous content included real-time videoconferences covering case discussions, simulated patient presentations, and critical literature reviews. At the conclusion of the course, students were administered the postcourse survey evaluating changes in their ability to identify and understand urologic conditions.
RESULTS: The postcourse survey demonstrated this course significantly increases students' scores in: self-perceived urologic knowledge, confidence in naming urologic conditions, comfort with performing urologic evaluations, and confidence placing consults for urologic conditions (P <.05).
CONCLUSION: Virtual medical student rotations are scalable and effective at delivering surgical material and can approximate the interpersonal teaching found in clinical learning environments. They may be a useful tool to supplement or augment clinical learning in select situations.

The COVID-19 pandemic has affected every aspect of the healthcare system including medical education. In accordance with social distancing guidelines, in March 2020, medical students were removed from all clinical care at institutions across the United States. , This major shift had a significant effect on surgical upper-level students whose educations heavily rely on direct patient care in clinical spaces. In fact, a similar response was also observed for Urology resident learners, drastically reducing their direct patient care, and thus curtailing their education. In response, multi-institutional coalitions like the UCSF Collaborative Online Virtual Didactics (COViD) and New York Educational Multi-institutional Program for Instructing Residents (EMPIRE) lecture series swiftly pivoted to delivering virtual didactics to residents and medical students. Although filling some educational gaps, these didactics failed to recapitulate the benefit of networking, mentorship, and personable feedback that come from medical students interacting in-person with residents and faculty. We hypothesized that this may be approximated by an online subinternship that combines flexible, web-based, modular content with real-time interactions between medical students, residents, and faculty. Recognizing the urgent call to action, we describe our experience developing this virtual surgical subinternship and present early programmatic evaluation outcomes.

Following the decision to suspend all student clinical activity, Urology faculty and 2 senior medical students at the University of Pennsylvania Perelman School of Medicine developed a novel virtual urologic curriculum. All upper-level, postclerkship medical students were eligible to register for this 2-week virtual Urology subinternship over the course of 1 month. The course was advertised as a fundamental-level course with the only perquisite requirement being the completion of the Surgery core clerkship.

The primary goal of this curriculum was to expose students to Urology while advancing their foundational knowledge of urologic evaluation and management. The secondary goal was to innovate methods for implementing a virtual curriculum that is scalable, sustainable, and adaptable to the needs of learners and instructors across surgical disciplines.

By the end of the course, learners were to achieve the following learning objectives: (1) Describe the steps to performing a urologic evaluation; (2) deliver urologic history, exam, assessment and plan in an oral case presentation; (3) list and describe the common conditions that urologists manage; (4) name key anatomical structures relevant to urological pathologies and urinary drainage; and (5) demonstrate ability to appraise urologic literature that intersects with other medical and surgical specialties.

Methods

Curriculum Format

The curriculum used both asynchronous and synchronous content (Table 1 ). The online learning management system, Canvas (Instructure, Salt Lake City, UT) hosted asynchronous readings, lectures, and discussion boards. BlueJeans (BlueJeans Network, San Jose, CA) videoconferencing was utilized for synchronous discussions and case presentations.

Table 1

Breakdown of synchronous and asynchronous materials

Overview of Penn Medicine Virtual Urology Curriculum
	Type of Activity	Learner Hours Per Course	Additional Educator Hours Per Course	Description
AsynchronousMaterial	Recorded lectures	20	2	Faculty recorded 30 min lectures in benign, pediatrics, and oncology
	Online cases	10	0	1-2 problem-based learning cases accompanied each lecture
	Additional readings	15	0	Supplemental reading from AUA medical student curriculum, Cambell-Walsh-Wein, Penn Urology Handbook, and landmark articles
	Discussion board post	15	0.5	Students posted one question, comment, or article a day.Students replied to at least one of their peersFaculty and resident provided comments on board
	Cumulative	60	2.5
Synchronousmaterial	Live discussion sessions	10	1.5	Faculty and resident facilitated on rotating scheduleReview of online cases with students
	Live lecture- urology department grand rounds(once per week)	3	0	Students were invited to attend weekly virtual grand round meetings
	Live student presentations(once)	2	2	Students delivered final evidence-based medicine presentations to department
	Cumulative	15	3.5
Cumulative course total(2-wk)		75	6

Asynchronous material accommodated the uncertainty of faculty clinical schedules. Faculty members prerecorded, introductory lectures on core topics for on-demand viewing by learners. Topics included medical and surgical management of benign, oncologic, and pediatric Urology and aligned with the American Urological Association Medical Students Curriculum. The majority of lectures were repurposed from pre-existing educational content created by each faculty member. Supplemental material included surgical videos, directed readings, and illustrations referenced from the AUA video library, Urology by Campbell's-Walsh-Wein, and the Penn Clinical Manual of Urology (Fig. 1 ) . Following completion of the lecture and supplemental materials, a discussion board opened to facilitate asynchronous communication and feedback about lecture contents among students, residents, and faculty. Learners were required to post one comment per day, with their activity identified by a distinct username. This served to increase interaction and participation with personal feedback on individual topics of student interest (Fig. 2 ).

Figure 1

Two weeks schedule of virtual urology subinternship. (Color version available online.)

Figure 2

Example of discussion board thread.

Medical students were instructed to post a brief comment on a topic of interest from each day's lectures. Canvas discussion board threads allowed for asynchronous interactions among medical students, residents, and faculty.

Synchronous material included daily virtual discussion sessions focused on the day's lecture topic. Similar to an in-person rotation, facilitators of these sessions rotated daily to provide flexibility for schedules and expose students to multiple faculty and residents. During these sessions, participants discussed surgical decision-making through problem-based learning (PBL) cases. PBLs were either written de novo or, when available, adapted from the AUA medical student case slides. Facilitators used Socratic teaching methods to approximate what is observed in our clinical environment during a surgical rotation. Learners also prepared a surgical SOAP note from PBL cases and presented a simulated patient to the group for feedback by faculty and fellow students. At the conclusion of the course, learners gave an evidence-based medicine capstone presentation on a urologic topic of interest to the entire Urology division in a Grand-Rounds-style presentation.

In accordance with our institution's grading guidelines for online courses, students were graded Pass/Fail based on completion of course requirements as outlined in Table 1.

Programmatic Evaluation

The Kirkpatrick model for programmatic evaluation was used. We focused our outcomes on Kirkpatrick Level 1 (participation) and Level 2a (attitudinal). To evaluate Level 2a outcomes, we developed a pre- and post-course questionnaire assessing participants perceptions of Urology using a 5-point Likert scale (1 = poor/strongly disagree and 5 = excellent/strongly agree) (Supplemental 1). Areas for evaluation of student attitudes included self-perceived urologic knowledge, confidence in naming urologic conditions, comfort with performing urologic evaluations, and confidence placing consults for urologic conditions. Students also provided end-of-course open-ended comments.

Table 2

Median scores in learner attitudes

Topic	Median Baseline Score (IQR)	Median Post-course Score (IQR)	P
Overall knowledge of urology	3 (2-3)	4 (4-4)	0.01
Naming urologic conditions confidence	3 (2-3.75)	4.5 (4-5)	0.01
Urologic evaluation confidence	2 (2-2.75)	3.5 (3-4)	0.02
Urology consult confidence	3 (2.25-4)	5 (4.25-5)	0.01

5-point scale: 1 poor, 5 excellent.

n = 10.

Median scores with interquartile ranges of anonymized pre- and postcourse surveys were calculated. A paired Mann-Whitney U test compared pre- to postcourse responses for each item using RStudio software, v. 1.2.5033 (RStudio Inc, Boston, MA). A P-value <.05 was considered statistically significant.

Qualitative content analysis on open-ended answers was performed using conventional content analysis methodology. Thirty comments of written feedback were compiled, read in entirety, and analyzed inductively. Codes were then derived from prevailing themes captured among responses. Each comment was read word for word and assigned to its appropriate theme. Coding was performed separately by 2 investigators, C.W. and O.F., to minimize subjectivity. Disagreement among investigators regarding theme titles was resolved by discussion between the 2 coders. This study was considered quality improvement by the University of Pennsylvania Institutional Review Board, and informed consent was waived.

Results

Kirkpatrick Level 1 outcomes (participation) were a total of 10 senior medical students over 2 separate 2-week subinternship iterations occurring during May 2020. Participant's specialty of interest was recorded at the start of class and included Urology (3), General Surgery (1), OB/GYN (3), Emergency Medicine (1), Orthopedic Surgery (1), and Plastic Surgery (1). Twenty of 26 (77%) division faculty participated in lectures, discussion boards and/or discussion groups. Faculty reported that the creation of lecture material and participation in the discussion board on average was a cumulative 2.5 hours (Table 1). Faculty participation in synchronous learning activities, including discussion sessions and final capstone presentations, was a total of 3.5 hours.

Kirkpatrick Level 2a outcomes include learner attitudes. Prior to taking the course, median student scores for all items ranged from 2 to 3 (Table 2 ). By the end of the curriculum, median scores significantly improved to greater than 3 in every topic area (P <.05). At the completion of the course, 2 students reported a shift of specialty commitment to Urology.

In qualitative content analysis of 30 end-of course feedback comments, themes included that the course had high perceived educational value, and that the course increased appreciation of Urology among students who were not previously considering the field. A representative comment stated that “[the] surgical patient presentations and capstone project were fun! It allowed me to think critically about a patient and get real time feedback on my surgical decision making.” Student feedback also highlighted the theme that discussion sessions that were didactic in nature were less favored than sessions that took a question and answer format. An additional theme was the convenience and appreciation of pre-recorded lectures and varied interaction with faculty members. One student remarked, “The flexibility of the course was great! The prerecorded lectures were succinct but with all the high yield facts I needed. I really liked that I could watch them on my own time.” Another student reported, “the daily discussion groups were my favorite part of the course. It was nice interacting with all of the Urology team.”

Feedback volunteered by facilitators revealed that faculty and residents enjoyed the opportunity to continue teaching medical students during social distancing, and they appreciated the practicality of now having pre-recorded lectures for other teaching opportunities.

Comments

We provide a framework that allows for resuming urologic education for medical students while social distancing. Although there is no replacement for in-person, direct patient care for senior medical students, a virtual surgical subinternship is an effective and feasible alternative that has been recently explored in other surgical specialties. , Our approach is unique in recreating an online version of the interpersonal aspects of clinical learning. These initial programmatic outcomes demonstrated an improved perception of urologic knowledge during the rotation. Although this was a temporizing innovation during the COVID-19 pandemic, this curriculum has the potential for lasting improvement of the future of urologic undergraduate medical education. Considering the AAMC recommendation to suspend visiting student rotations, the Society of Academic Urologists has encouraged Urology programs nationally to develop online virtual rotations. Unfortunately, no best practices yet exist. Our curriculum can model this adaptation for other Urology and surgery residency programs.

Beyond the pandemic period, virtual course materials may augment or replace traditional curricula. An organized virtual platform is well-suited to expose “visiting” students to more programs by providing flexible scheduling and reducing the cost of visiting rotations. For students uncertain about their interest in a specialty, a virtual format with synchronous and asynchronous learning methods may allow them to learn about the specialty and its culture before deciding on a traditional rotation. Additionally, with the incorporation of students into outpatient telemedicine visits, there is potential for virtual subinternships to include patient care. The use of live video streaming of cases from the operating room may allow students to experience surgical decision making and operative techniques in real-time. Finally, this format could correct the pervasive lack of Urology (and surgical subspecialties) within preclinical curriculums. Prerecorded lectures and case modules created for virtual subinternships and stored on open-access Canvas sites are easily scalable and adaptable for a broader preclinical audience with minimal additional educator time. Furthermore, the learner-driven discussion board can become a portal to activate wider learning by empowering students to ask specific questions and learn from each other.

We acknowledge limitations with this reported experience. Our programmatic evaluation is based on 2 course-sessions with a relatively small number of participants. Additionally, 2 students involved with curriculum design participated in one iteration of this study, introducing potential bias of the evaluations. Areas for improving of the course include more interaction with residents and more exposure to actual patients. In addition to accruing more participants, future directions include a postcurriculum exam assessing knowledge acquisition of urological principles (Kirkpatrick level 2b) and longitudinal follow-up to determine student's choice to enter Urology residency (Kirkpatrick level 4).

Conclusion

Urology has always been on the forefront of healthcare innovation, and here we apply this same spirit to medical education. By integrating didactic lectures, problem-based cases, and discussion sessions, we provide a model for recapitulating the interpersonal relationships of a subinternship experience in the virtual environment. With continued healthcare and educational innovation, we have an opportunity to embrace online learning with virtual subinternships and continue to push our collective mission of educating the next generation of Urologists during these unprecedented times.