Where Do We Go From Here? Assessing Medical Students' Surgery Clerkship Preparedness During COVID-19.

INTRODUCTION: The impact of COVID-19 on surgical education has been profound, and clinical learning experiences transitioned to virtual formats. This study investigated the impact of virtual experiences created to facilitate learning during the pandemic for medical students.
METHODS: We performed a cohort study to determine the perceived clinical preparedness for medical students enrolled in the preclinical surgery pilot course, surgical Extended Mastery Learning Rotation (EMLR), and longitudinal surgical clerkship (LC). The preclinical surgery pilot course took place before COVID-19 disruptions, and the EMLR and LC experiences took place virtually. Specialty choice was examined in the EMLR and LC cohorts. Performance on the NBME surgical assessments was analyzed among students enrolled in the traditional clerkship and pandemic-disrupted courses and compared to national data using a two-sample t-test.
RESULTS: Compared to preclinical students, EMLR and LC students demonstrated improvements in their perceived surgical clerkship readiness. After the 3-week EMLR course, in the setting of completing only one-third of the clerkship year, students had an average NBME Surgical Self-Assessment Exam score of 72 (SD 12), comparable to the national average of 71 (SD 9) p = 0.33. The average shelf exam score for students (N = 24) enrolled in the traditional clerkship (block 1), prior to COVID-19, disruptions was 66 (SD 9) compared to an average score of 69 (SD 9) for the longitudinal clerkship students (N = 20) that took the shelf exam later in the year (p = 0.36). COVID-19 disruptions did not affect specialty choice. All LC students have decided on a specialty; 50% nonsurgical and 50% surgical. From the EMLR cohort, 36% and 38% plan to pursue surgical and nonsurgical specialties, respectively, with 26% still undecided.
CONCLUSIONS: Courses were well-liked and will be implemented in future clerkships. Surgical educators demonstrated flexibility and creativity in the development of the EMLR. Despite COVID-19 disruptions, medical students made progress in their clinical skills and foundational science knowledge. COVID-19 disruptions did not appear to impact specialty choice.

Introduction

The World Health Organization classified the COVID-19 outbreak as a pandemic on March 11, 2020. The pandemic has altered many educational experiences. The effect of COVID-19 on medical education has been profound, leading to the replacement of in-person learning with virtual experiences. , Institutional responses have varied and are driven primarily by regional pandemic trends, the availability of personal protective equipment, testing, and clinical experiences. A recent study predicted a decline in the duration of future surgery clerkship experiences to enable the completion of all required rotations during pandemic disruptions. The reduction of surgical clinical experiences should not be taken lightly, as it has the potential to exacerbate longstanding concerns around limited student exposure to surgical faculty and skill development. In light of COVID-19, the future of surgical clerkships remains unclear. It is crucial for surgical educators to be agile and creative in managing the rapidly-evolving challenges posed by COVID-19 to maximize learning experiences for students.

All students enrolled in the last preclinical block transitioned to enroll in either the traditional or longitudinal clerkship from January to December of a calendar year. Traditional clerkships are typically conducted in six 8-week blocks while longitudinal clerkships are intermittent surgical experiences throughout the clerkship year. In response to the pandemic, education leaders at the University of California, San Francisco (UCSF) eliminated all in-person clinical experiences for third-year medical students (MS3s) from March 9, 2020, to July 13, 2020. In response to the current milieu, the Department of Surgery education leadership team designed and implemented a 3-week Extended Mastery Learning Rotation (EMLR), with the primary objective of preparing students, previously enrolled in traditional clerkships, for their return to in-person experiences through the solidification of foundational and clinical science knowledge. In addition to completing the EMLR, students also were required to take a surgical self-assessment test designed by the National Board of Medical Examiners (NBME) that are typically used to prepare students for the traditional surgical shelf exam. Completion of the EMLR course was a prerequisite to returning to clerkships. Prior to COVID-19 disruptions, students in longitudinal clerkships had primarily outpatient surgical clinic experiences interspersed with inpatient time and operating room days. As a result of the pandemic, in-person clinical experiences were halted for the longitudinal clerkship students, and they were transitioned to assist with primary care telemedicine visits complemented with attending-led virtual didactic sessions. At the end of the clerkship year, longitudinal clerkship students were required to complete the surgery shelf exam.

In light of these significant changes, surgical educational leadership continued to track student progress at frequent time intervals and adjusted future clerkship experiences accordingly. This study aimed to explore the efficacy of the EMLR in preparing students for future in-person surgical experiences. Specifically, we compared perceived preparedness among the following cohorts of students: (1) students enrolled in a preclinical surgical course prior to the start of their clerkship year, (2) students enrolled in the EMLR course, and (3) students enrolled in the longitudinal clerkship. We analyzed the exam performance of students enrolled in traditional clerkship prior to the pandemic (block 1), EMLR, and longitudinal cohorts and compared exam scores between the block 1 and longitudinal cohorts, and compared all cohort scores to national data. We hypothesized that this study would serve as a curricular needs assessment with the aim of consolidating future clerkship experiences by identifying areas of strength and remediation before students return to the in-person clinical environment.

Methods

Participants and Data Sources

This study included students in the preclinical surgical course, individuals from the prepandemic traditional clerkship (block 1), EMLR, and longitudinal clerkship cohorts voluntarily participated. Before starting clerkships (December 2019), the Department of Surgery clerkship leadership team selected senior medical students and general surgery residents with prior teaching experience and interest in surgical education to teach 3 voluntary sessions during the preclinical surgical course to help prepare students for the surgery clerkship. Before the course, instructors met with the clerkship director and coordinator to review course objectives, pedagogic approaches, and content. Sessions consisted of case-based learning, resident panels, and didactics. Sessions discussed key differences between internal medicine and surgery patient presentations, gave advice for surgical note writing, outlined operation room etiquette and medical student roles, and provided general tips for success and approaches to common surgical problems (i.e., appendicitis, cholecystitis, and postoperative fever). Students went on to commence their traditional or longitudinal clerkship. In response to the pandemic, students enrolled in the block 2 were transferred from their traditional clerkship to the EMLR curriculum. The EMLR course was created and comprised of a series of virtual didactics and interactive sessions designed and taught by faculty, general surgery residents, and senior medical students. Students reviewed common surgical patient presentations, fundamental anatomy, physiology, pathophysiology principles, and discussed management plans. Students who did not commence their surgery clerkship before COVID-19 disruptions were enrolled in the EMLR. The EMLR course was taught four times to different sets of students originally enrolled in the next four surgical clerkship rotations (blocks 3-6). Student perceptions were captured with a clerkship readiness survey, and sessions were assessed with a course evaluation survey.

Longitudinal students were eligible to participate in the preclinical surgical course but did not participate in the EMLR. After COVID-19 disruptions, they continued their learning through outpatient virtual surgical clinic experiences. Of note, all longitudinal students had previously performed a 2-week inpatient surgical experience and worked with surgical preceptors weekly for 3 months before being surveyed. Student perceptions and clinical readiness were assessed for the longitudinal cohort halfway through their clinical year (Fig. 1 ).

FIGURE 1

Curricular layout in light of COVID-19 pandemic disruptions. EMLRs, Extended Mastery Learning Rotations; NBME, National Board of Medical Examiners.

Students enrolled in block 1, prior to COVID-19 disruptions, completed the clerkship and took the shelf exam. EMLR students had the option of taking the surgical self-assessment test at any time before the end of their clinical year. Longitudinal clerkship students did not take this self-assessment and took the traditional shelf examinations at the end of the clinical year. Students were required to take all examinations under standard-paced conditions unless student-specific accommodations required the use of the self-paced mode. Under standard-paced conditions for the self-assessments, students are provided up to 1 hour and 15 minutes to complete all the exam's 4 sections consisting of 50 questions each. Under standard-paced conditions for the surgery shelf exam, students have up to 2 hours and 45 minutes to complete the 110-question exam. Self-paced mode allowed eligible students to take the exam at the allotted pace approved by student support specialists. All exam conditions prohibited the use of outside materials (i.e., notes, textbooks, and reference material).

Data Analysis

Preclinical surgical pilot course students were surveyed using a modified version of the Readiness for Clerkship Student Survey. The survey was adapted by omitting 10 questions that did not pertain to clinical skills that resulted in utilizing 34 five-point Likert-style questions. The survey also was modified by substituting the term “medical” with the word “surgical.” Upon completion of the course, students who attended the preclinical surgical pilot course were asked to assess how well the existing preclinical curriculum prepared them for their inpatient surgical clerkship as well as their level of competence in performing various tasks, such as taking a full surgical history, performing a physical examination, developing a problem list, and patient communication. Students evaluated their level of competence on the following scale: Unable to rate/not applicable, 1- an unacceptable level of competence, 2 - a marginal level of competence, 3 - a satisfactory level of competence, 4 - a high level of competence, and 5 - an extremely high level of competence.

The same survey also was administered to both students enrolled in the EMLR and longitudinal clerkship. For these groups, additional survey questions were added that asked students to rate their level of agreement with statements such as the following: How do you rate your current interest in surgery? COVID-19 has influenced my decision to go into a surgical specialty, and I would like to see the EMLR (or virtual clinic teaching for longitudinal students) implemented into all future surgical clerkships. These questions were evaluated on the following scale: 1- strongly disagree, 2 - somewhat disagree, 3 - neither agree nor disagree, 4 - somewhat agree, 5 - strongly agree.

Average Likert scores and standard deviations were determined for the preclinical surgical pilot course, EMLR, and longitudinal clerkship cohorts using Microsoft Excel (OS 10.3) and compared across all groups. Survey data scores were compared using an independent sample t-test and Cohen's effect size evaluation using Stata Version 16 (College Station, TX).

The student self-assessment and shelf exam scores were anonymized by the surgery clerkship administration. The self-assessment scores were converted to approximate subject examination scores utilizing the conversion table provided by the NBME and the averages and standard deviations were calculated using Microsoft Excel (OS 10.3 Redmond, WA). The NBME shelf exam scores of students enrolled in the traditional clerkship prior to pandemic disruptions were compared to longitudinal cohort using a two-sample t-test. A two-sample t-test was also utilized to compare institutional student self-assessment and shelf exam scores to the respective national average scores. The Institutional Review Board deemed this study as exempt.

Results

Participants

All preclinical medical students (N = 154) enrolled at the main UCSF Parnassus campus were eligible to participate in the preclinical surgery pilot course. Participants of the preclinical surgical course represented 27% (41/154) of the eligible participants. All preclinical students (N = 154) plus students from the UC Berkeley-UCSF Joint Medical Program (N = 7) enrolled in either the traditional or longitudinal clerkships (N = 161). Prior to COVID-19 disruptions, 32% (51/161) of enrolled students participated in the traditional surgery clerkship. Students participating in the EMLR and longitudinal clerkship cohorts represented 51% (82/161) and 17% (28/161) of enrolled MS3s, respectively. All students that enrolled in the preclinical surgical pilot course enrolled in either the traditional clerkship or the longitudinal clerkship. Survey completion rates were 51% (21/41), 60% (49/82), and 36% (10/28) for the preclinical surgical pilot, EMLR, and longitudinal clerkship cohorts, respectively. Two longitudinal clerkship participants answered “unable to rate/not applicable” for the Likert-style survey questions pertaining to clinical preparedness.

Student Perceived Preparedness for Inpatient Surgical Clerkship

Student perceived preparedness increased among the MS3 cohort as the year progressed. There was significant improvement appreciated as they transitioned from preclinical surgical pilot sessions to the clinical environment (EMLR and longitudinal clerkship). A statistically significant increase was appreciated in clinical skills, patient care, fund of knowledge, clinical reasoning, and management. The differences in a majority of these domains were not significant between EMLR and longitudinal clerkship students; however, longitudinal clerkship students felt more prepared to verbally present their findings to their resident or preceptor, and EMLR students felt more skilled at identifying illness severity in a surgical patient (Table 2).

TABLE 2

Student Perceived Preparedness for Surgery Clerkship

	PSPCN = 31	EMLRN = 49	LCN = 8
Question	Average Likert (±Std)	Average Likert (±Std)	Average Likert (±Std)	P-Value*	p-Value⁎⁎Effect SizeEF 95% CI	p-Value⁎⁎⁎Effect SizeEF 95% CI
Q1-Take an appropriate history of the current surgical problem.	1.6 (0.6)	2.5 (0.8)	2.5 (1.1)	<0.01*	1.00(−0.7 to 0.8)	0.003*−1.2(−2.1 to −0.4)
Q2-Take a full surgical medical history.	1.7 (0.7)	2.1 (0.8)	2.5 (1.1)	0.03*	0.220.5(−0.3 to 1.2)	0.01*−1.0(−1.8 to −0.2)
Q3-Formulate a problem list in a surgical patient.	1.7 (0.6)	2.6 (0.9)	2.5 (0.9)	<0.01*	1.00(−0.7- 0.8)	<0.01*−1.2(−2.0 to −0.4)
Q4-Perform a full physical examination in a surgical patient.	1.8 (0.7)	1.9 (0.7)	2.0 (0.9)	0.5	0.70.1(−0.6 to 0.9)	0.5−0.3(−1.0 to 0.5)
Q5-Interpret relevant key lab results obtained on surgical patients.	1.6 (0.6)	2.8 (0.7)	2.6 (0.9)	<0.01*	0.5−0.3(−1.0 to 0.5)	<0.01*−1.5(−2.3 to −0.6)
Q6-Interpret relevant imaging reports for common health problems of surgical patients.	1.6 (0.7)	2.6 (0.7)	2.1 (1.0)	<0.01*	0.08−0.7(−1.4 to 0.1)	0.1−0.7(−1.4 to 0.1)
Q7-Explain the underlying pathology and pathophysiology of key surgical problems.	1.7 (0.6)	2.8 (0.7)	2.5 (1.1)	<0.01*	0.3−0.4(−1.1 to 0.4)	<0.01*−1.2(−1.9 to −0.3)
Q8-Verbally present your findings to the resident or preceptor.	1.9 (0.7)	1.9 (0.9)	3.0 (0.9)	1.0	<0.01*1.2(0.4-2.0)	<0.01*−1.5(−2.3 to −0.6)
Q9-Demonstrate a clear understanding of anatomy in the context of physical exams and interventions.	1.8 (0.6)	2.4 (0.8)	2.4 (1.1)	<0.01*	1.00(−0.7 to 0.7)	0.04*−0.8(−1.6 to −0.3)
Q10-Propose a differential diagnosis consisting of more than one reasonable alternative based on the history, laboratory, and other test, and the results of the physical examination in a surgical patient.	1.9 (0.7)	2.9 (0.7)	2.4 (0.9)	<0.01*	0.08−0.7(−1.4 to 0.1)	0.1−0.7(−1.4 to 0.1)
Q11-Identify medications appropriate to treat a surgical patient's problems.	1.3 (0.5)	2.3 (0.7)	2.0 (0.8)	<0.01*	0.3−0,4(−1.2 to 0.3)	<0.01*−1.2(−2.0 to −0.4)
Q12-Propose a basic short-term management plan for a surgical patient's major problems.	1.4 (0.6)	2.5 (0.8)	2.1 (0.8)	<0.01*	0.2−0.5(−1.3 to 0.3)	<0.01*−1.1(−1.9 to −0.3)
Q13 - Explain the short- intermediate- and long-term management plans that were developed for the surgical patients under your care.	1.4 (0.5)	2.2 (0.8)	1.9 (0.8)	<0.01*	0.3−0.4(−1.1 to 0.4)	0.03*−0.9(−1.7 to −0.1)
Q14 - Identify if a surgical patient is seriously ill and requires immediate assessment and treatment.	1.8 (0.6)	3.2 (0.8)	2.6 (0.8)	<0.01*	0.05*−0.8(−1.5 to 0.01)	<0.01*−1.2(−2.1 to −0.4)
Q15- Pursue opportunities to learn the required surgical technical skills.	2.0 (0.9)	1.7 (0.8)	1.7 (1.0)	0.12	1.00(−0.7 to 0.7)	0.40.3(−0.5 to 1.1)

EMLR, Extended Mastery Learning Rotation; LC, Longitudinal Clerkship; PSPC, Preclinical Surgery Pilot Course.

p-value from t-test evaluation of EMLR vs Preclinical clerkship students.

p-value from t-test evaluation of EMLR vs Longitudinal students.

p-value from t-test evaluation of Longitudinal students vs Preclinical Students.

Student NBME Exam Performance

Ninety-three percent (76/82) of EMLR students selected to take the surgery self-assessment examination before returning to clerkships; 74 students took it in standard-pace mode, and 2 students in self-paced mode. The average score was 72 (SD 12). This was similar to the national average surgery self-assessment score of 71 (N = 16,460 SD 9) p = 0.33. Six students have not completed the self-assessment and plan to complete the exam prior to enrolling in their surgery clerkship.

The average shelf exam score for students (N = 24) enrolled in the traditional clerkship (block 1), prior to COVID-19, disruptions was 66 (SD 9) compared to an average score of 69 (SD 9) for the longitudinal clerkship students (N = 20) that took the shelf exam later in their clerkship year (p = 0.36). The longitudinal students scored lower than the national average shelf score of 75 (N = 17,798 SD 8) (p = 0.001).

COVID19 Influence on Medical Student Specialty Choice

The specialties of choice for EMLR students were as follows: 37% (n = 18/49) desired a surgical specialty, 39% (n = 19/49) desired a nonsurgical specialty, and 26% (n = 13/49) were undecided. The specialties of choice for longitudinal students were as follows: 50% (n = 5/10) desired a surgical specialty and 50% (n = 5/10) desired a nonsurgical specialty (Fig. 2 ). There were no students in the longitudinal clerkship who were undecided about their specialty choice. More students wanted to pursue a surgical specialty among the longitudinal participants, but the difference was not significant (Table 1 ). Differences in surgical specialty interest among EMLR and longitudinal participants did not reach significance (Table 1). COVID-19 disruptions did not seem to influence students' decision to pursue a surgical specialty (Table 1). Longitudinal clerkship students expressed interest in seeing EMLR content in future didactics. Most longitudinal students agreed that their surgical preceptor inspired them to pursue a surgical specialty; 20% of longitudinal students communicated with their preceptor weekly, 20% communicated with their preceptor bi-weekly, 50% of longitudinal students communicated with their preceptor monthly, and 10% communicated with their preceptor at a different frequency.

FIGURE 2

Student specialty interest EMLR vs. LC. *EMLR, Extended Mastery Learning Rotation; LC, Longitudinal Clerkship.

TABLE 1

Student Interest in Surgical Specialties

	EMLRN = 49	LCN = 10
Question	Average Likert (±Std)	Average Likert (±Std)	p-ValueEffect SizeEF 95% CI
How do you rate your current interest in surgery?	3.7 (1.1)	4.0 (1.1)	0.4
			−0.3
			(−1.0 to 0.4)
COVID19 has influenced my decision to go into a surgical specialty.	2.8 (0.9)	2.9 (0.9)	0.8
			−0.1
			(−0.8 to 0.6)
I would like to see the EMLR (or virtual clinic teaching for longitudinal students) implemented into all future surgical clerkships.	3.4 (1.1)	2.9 (1.4)	0.2
			0.4
			(−0.3 to 1.1)
I wish that longitudinal clerkship students were given the opportunity to participate in EMLR didactics.	N/A	2.7 (1.4)	N/A
My designated surgery mentor has inspired me to pursue a surgical specialty.	N/A	3.6 (1.2)	N/A

EMLR, Extended Mastery Learning Rotation; LC, Longitudinal Clerkship.

Conclusions

COVID-19 disruptions have altered the course for third-year clerkships. Achieving progress can seem daunting during unprecedented and challenging times. This study sheds light on the effectiveness of rapid adjustments made by a surgical department at an academic medical center to create beneficial experiences for medical students. Although in-person patient care experiences were limited, EMLR and longitudinal clerkship students made appreciable strides in developing patient care skills and medical fund of knowledge. Since taking the preclinical surgical pilot, students made significant improvements in perceived clerkship preparedness. Students also may be more prepared for future licensing examinations, as evidenced by their average student self-assessment scores mirroring the national average after only completing one-third of their third-year clerkships. Their progress is ahead of findings in the literature that state that students tend to do better on shelf exams after completing more clerkships. The longitudinal students’ performance on the shelf exam reinforced these projections. Although the difference was not statistically significant, longitudinal students had a higher average shelf exam score than block 1 students when this was taken toward the end of the year, presumably after gaining overall experience. This phenomenon demonstrates that students made strides in their knowledge acquisition throughout the year, even during altered clinical curricula. Despite the progress, COVID-19 has impacted knowledge acquisition in this cohort, given the fact that students in this group performed lower than the national averages that incorporated 75% of examination scores reported prior to COVID-19 disruptions.

The results of this study have served as a useful needs assessment for educational leadership. These data have encouraged clerkship leaders to shift future clerkship focus from solidifying foundational science knowledge concepts toward an increased emphasis on patient care and operative management. As a result of student feedback and survey results, weekly scheduled didactics have been removed from future clerkships to free up more time for clinical experiences. Although the EMLR increased overall preparedness, many students identified some insecurities that alerted clerkship directors about areas for remediation. Students expressed a need for more suturing opportunities and anatomy review; they wished for more hands-on experiences with the hope that in-person experiences will provide more opportunities to reconnect with patients and solidify foundational knowledge. The Department of Surgery education leadership team is currently working to incorporate these suggestions into future clerkship iterations.

Even though the longitudinal students did not participate in the EMLR, they also made significant strides in patient care, and they appreciated having formal surgical mentorship. Compared to the EMLR cohort, their telemedicine experiences seemed to increase preparedness in their communication skills with attendings, residents, and patients. Longitudinal students appeared to be less confident than EMLR patients in identifying disease severity; this could be attributable to longitudinal patients being restricted to telemedicine visits in the outpatient setting, where patients tend to be clinically stable overall. This was an interesting finding because longitudinal students had a 2-week inpatient surgical experience prior to the COVID-19 pandemic. Clerkship leadership should consider developing didactic sessions or interactive modules to help longitudinal students assess illness severity. The lack of EMLR participation by longitudinal students should not hinder future clinical success. The literature has found that longitudinal students demonstrate clinical competence and perform better than traditional clerkship students.

In the mentorship realm, the longitudinal model provided more formal opportunities to meet with mentors from each of their respective clerkship rotations. This appeared to give students the chance to solidify career aspirations since longitudinal students were more decisive with their career choice. On the other hand, 26% of EMLR students were undecided around a future specialty choice. This indecision presents itself as an opportunity for the surgical residents and faculty to inspire a cohort of future surgeons.

Preclinical students enrolled in the preclinical pilot expressed interest in having more exposure to surgical concepts. The success of the pilot has led to the integration of surgical concepts into the formal curriculum and the development of more surgical electives since increased clinical exposure is correlated with increased success in the surgical clerkship. For example, several studies noted that the implementation of preclinical operative experiences have led to increased exposure to surgical faculty, confidence in surgical skills, and interest in pursuing a surgical residency.10, 11, 12 We propose an increased level of engagement of surgical educators in formal preclinical didactics. We also opine that the influence of surgical educators should extend into positions within the undergraduate medical education leadership structure (i.e., deans of student affairs, medical education, curriculum, assessment, and admissions). Increased integration of surgical educators into the fabric of an institution will naturally lead to more opportunities for role modeling, mentorship, and influence at earlier points in the medical student educational experience.

This study has several limitations. It contains a relatively small student sample overall, particularly for the longitudinal clerkship; 32% (51/161) of the MS3s were not captured in the EMLR and longitudinal cohorts because these students had already begun or completed their surgery clerkship before COVID-19 disruptions. Preclinical pilot sessions were voluntary while the EMLR course and longitudinal clerkship were mandatory. This selection bias may have skewed the perceived clerkship readiness scores, as preclinical students may have self-selected as feeling less prepared and less confident in surgical clerkship-related competencies.

The data in this study were meaningful and informative because we captured student perspectives at critical points in their clinical growth. Investigating the clerkship readiness of students enrolled in the preclinical pilot course was a natural starting point for this study because it enabled us to assess progress over time as they transitioned to clerkships (traditional or longitudinal) and subsequently altered clinic experiences due to the pandemic. This study demonstrated the ingenuity and collaborative spirit of a surgical department in adapting rapidly and developing a robust and successful curriculum that provided students with opportunities for growth during this unprecedented time when in-person clinical opportunities were unavailable. The information gained during this period will be incorporated to enrich future clerkship educational opportunities.

Declaration of Competing Interest

Julie Ann Sosa, MD, MA, is a member of the Data Monitoring Committee of the Medullary Thyroid Cancer Consortium Registry supported by GlaxoSmithKline, Novo Nordisk, Astra Zeneca, and Eli Lilly. She receives institutional research funding from Exelixis and Eli Lilly.