Innovation and Missed Opportunities in Internal Medicine Undergraduate Education During COVID-19: Results from a National Survey.

BACKGROUND: COVID-19 disrupted undergraduate clinical education when medical schools removed students from clinical rotations following AAMC recommendations. Clerkship directors (CDs) had to adapt rapidly and modify clerkship curricula. However, the scope and effects of these modifications are unknown.
OBJECTIVE: To examine the effects of the initial phase of COVID-19 on the internal medicine (IM) undergraduate clinical education.
DESIGN: A nationally representative web survey. PARTICIPANTS: IM CDs from 137 LCME-accredited US medical schools in 2020. MAIN MEASURES: Items (80) assessed clerkship structure and curriculum, assessment in clerkships, post-clerkship IM clinical experiences, and CD roles and support. The framework of Understanding Crisis Response (Royal Society for Encouragement of Arts, Manufactures, and Commerce) was used to determine whether curricular modifications were "amplified," "restarted," "let go," or "ended." KEY
RESULTS: Response rate was 74%. In response to COVID-19, 32% (32/101) of clerkships suspended all clinical activities and 66% (67/101) only in-person. Prior to clinical disruption, students spent a median of 8.0 weeks (IQR: 2) on inpatient and 2.0 weeks (IQR: 4) on ambulatory rotations; during clinical re-entry, students were spending 5.0 (IQR: 3) and 1.0 (IQR: 2) weeks, respectively. Bedside teaching and physical exam instruction were "let go" during the early phase. Students were removed from direct patient care for a median of 85.5 days. The sub-internship curriculum remained largely unaffected. Before the pandemic, 11% of schools were using a pass/fail grading system; at clinical re-entry 47% and during the survey period 23% were using it. Due to the pandemic, 78.2% of CDs assumed new roles or had expanded responsibilities; 51% reported decreased scholarly productivity.
CONCLUSIONS: Curricular adaptations occurred in IM clerkships across US medical schools as a result of COVID-19. More research is needed to explore the long-term implications of these changes on medical student education and clinical learning environments.

INTRODUCTION

The COVID-19 pandemic disrupted clinical experiences for medical students across the country. On March 17, 2020, the Association of American Medical Colleges (AAMC) issued the recommendation to remove medical students from clinical rotations due to the pandemic.[1] Medical educators were called upon to develop educational strategies and provide alternative clinical experiences that could still support meaningful clinical learning despite the immediate disruption.[1]

Clerkship directors (CDs) needed to adapt rapidly to the changing situation and modify the design and delivery of the clerkship curriculum in response to the AAMC recommendations. However, the scope and effects of these modifications are not known. In this study, we examine the impact of the initial phase of the COVID-19 pandemic (i.e., pandemic) on internal medicine (IM) clinical undergraduate medical education (UME) at US medical schools through a national survey of IM core CDs.

In our study, we use the model of Understanding Crisis Response from the Royal Society for the Encouragement of Arts, Manufactures, and Commerce[2] (Fig. 1) as a framework.[2] Although crisis management models were primarily developed for large-scale organizational crises, we found them relevant to the global crisis due to the COVID-19 pandemic. Boin suggests “crisis management efforts are effective when operations are sustained and resumed, losses are minimized, and learning occurs so that lessons are transferred to future incidents.”[3] We chose the Understanding Crisis Response framework[2] that has been directly applied to the pandemic; it provides a means to reflect on what was stopped and started in response to a crisis and determine whether a response was “amplified,” “restarted,” “let go,” or “ended.” Reflecting on these areas can be a catalyst for positive change.[2]

Figure 1

The model of Understanding Crisis Response from the Royal Society for the Encouragement of Arts, Manufactures, and Commerce.

METHODS

The Clerkship Directors in Internal Medicine (CDIM) conducts an annual, voluntary survey of IM core CDs from US/US territory-based medical schools with CDIM membership and full LCME accreditation. Only one possible respondent per school (i.e., designated IM core CD) was surveyed. The 2020 CDIM survey focused on the effects of the pandemic on the IM clerkship. At survey launch, there were 142 survey-eligible medical schools, and the number was adjusted to 137 (96.5%) after removing two schools whose CDIM membership had lapsed, and three with preliminary LCME accreditation. The survey was deemed exempt by Pearl IRB ((#20-AAIM-112; U.S. DHHS OHRP #IRB00007772).

The CDIM Survey and Scholarship Committee, representing UME leaders, developed the survey questions/content and the CDIM Council reviewed and modified the questions for construct validity through several iterations. The final survey consisted of 80 items, including multiple choice, Likert scale, numeric-only, and open-text response options (Supplemental Appendix A). Due to conditional logic or item-nonresponse, denominators for some questions do not sum to the total number of survey respondents. We administered the survey electronically via Qualtrics[4] from August 19 to October 20, 2020. Only MK had access to the survey data. Prior to de-identifying the dataset for analysis, MK merged respondents’ records with the complete survey population file to include demographics and medical school characteristics.

We asked the survey questions in relation to the following time periods (i.e., phases):

Pre-pandemic: prior to March 17, 2020

Clinical interruption: March 17 to June 1, 2020, when students were removed from the clinical learning environment

Clinical re-entry: when students re-entered the clinical learning environment after June 1, 2020

Current (i.e., relative to the survey period): August 19 to October 20, 2020

Statistical Analysis

We used descriptive statistics to report the summary results, and Pearson’s chi-square (with Sidak-adjusted p values, where applicable, to minimize Type-I and Type-II errors), or Fisher’s exact test to conduct bivariate (two-sided) tests for associations or goodness-of-fit between categorical variables (a=0.05). We conducted multivariate tests of means (Hotelling’s t-squared) and paired t tests to compare differences in continuous variables for multiple time periods defined in the survey (a=0.01), reporting medians and interquartile ranges (IQRs), or means and standard deviations. We used Stata 16.1 (StataCorp. 2019) for the analysis. Open-ended comments were reviewed (IA, ATP) and classified according to themes following an inductive, iterative approach.

RESULTS

The survey response rate was 74% (101/137). There were no statistically significant differences between respondents and non-respondents with regard to medical school type (public/private), US Census Bureau region, class size, and gender of respondent (Supplemental Appendix B).

Clerkship Structure

In response to the pandemic, 32% (32/101) of clerkships suspended all clinical activities (i.e., virtual and in-person), whereas 66% (67/101) suspended only in-person. Two clerkships continued all in-person clinical activities during the pandemic. By survey closure, 98% (97/99) of clerkship students had returned to in-person clinical learning, and for the remaining 2%, a return date had been set. Among clerkships that removed students during clinical interruption, 19% (18/96) resumed in-person clinical activities in May, 58% (56/98) in June, 18% (17/98) in July, and 5% (5/98) in August. Students were removed from in-person learning for a median of 85.5 days (IQR: 30) across clerkships.

The overall IM clerkship structure (i.e., longitudinal integrated, traditional) did not change during the various phases, but the duration of clerkships changed, such as the number of weeks on inpatient and ambulatory rotations. Pre-pandemic students were spending a median of 8.0 weeks (IQR: 2) on the inpatient general medicine and 2.0 weeks (IQR: 4) on ambulatory medicine. During the initial clinical re-entry phase, the median number of weeks decreased to 5.0 inpatient (IQR: 3) and 1.0 ambulatory (IQR: 2). The median number of weeks on inpatient service rebounded partially at the time of the survey (6.0 weeks, IQR: 4), but it did not for the ambulatory medicine rotations (1.5 weeks, IQR: 2). The changes in median numbers of weeks spent between each phase (e.g., 8 weeks inpatient pre-pandemic to 5 weeks re-entry) were statistically significant at p<0.01.

Moreover, there were changes in the number of clinical sites. Although the number of inpatient sites remained the same, available outpatient sites decreased from a median of 4.0 (IQR: 10) sites pre-pandemic to 2.0 (IQR: 5) during re-entry and to 2.0 (IQR: 8) at the time of the survey (p<0.001 for all comparisons). During clinical interruption, students from medical schools that paused in-person learning but continued clerkship education (n=60) spent a mean of 12% of their time (SD: 22) on inpatient care via video/telephone conferencing compared to 0.5% (SD: 2.6) pre-pandemic (p<0.001), and 84% (SD: 27.1) on virtual cases in lieu of patient care compared to 0.7% pre-pandemic (p<0.001). Of 50/101 (49.5%) respondents whose IM clerkship had an ambulatory component, 36/50 reported the percentage of time for ambulatory telehealth experiences. The median percent of time pre-pandemic was 0 (IQR: 0), 20 (IQR: 32) during re-entry, and 10 (IQR: 20) at the time of the survey (p<0.001 for all comparisons).

Curriculum

Sixty-three percent (62/99) of CDs who reported removing students from in-person clinical activities during COVID-19 reported adding new curricular material at clinical re-entry, including COVID-19 education (84%, 52/62), social determinants of health (27%, 17/62), narrative medicine (23%, 14/62), and medical ethics related to telehealth (16%, 10/62); in-person didactics were replaced by synchronous or asynchronous online content during the clinical interruption and after re-entry (Table 1).

Table 1

Method of Delivery of Internal Medicine Clerkship Didactics over Time

	Pre-pandemic (n=101)	Clinical interruption (n=66)	Clinical re-entry (n=93)	Currently (relative to survey launch) (n=95)
	Number of responses (%)				p value*
In-person
Lectures	95 (94.1)	1 (1.5)	9 (9.7)	22 (23.2)	<0.001
Small groups	89 (88.1)	2 (3)	27 (29)	38 (40)	<0.001
Virtual
Lectures
Synchronous (live-streamed)	11 (10.9)	54 (81.8)	79 (85)	77 (81.1)	<0.001
Asynchronous (previously recorded)	22 (21.8)	34 (51.5)	39 (41.9)	41 (43.2)	0.019
Small groups	7 (6.9)	42 (63.6)	59 (63.4)	54 (56.8)	<0.001
Morning reports	3 (3)	28 (42.4)	44 (47.3)	46 (48.4)	<0.001
Independent readings	65 (64.4)	50 (75.8)	67 (72)	65 (68.4)	0.609
e-learning platforms	39 (38.6)	47 (71.2)	58 (62.4)	51 (53.7)	0.035
Podcasts	9 (8.9)	18 (27.3)	19 (20.4)	18 (20)	<0.001
Social media group	2 (2)	3 (4.6)	4 (4.3)	5 (5.3)	0.224
Other**	1 (1)	1 (1.5)	1 (1.1)	2 (2.1)	0.532

**Responses included “Virtual Grand Rounds” and “Discovery (bedside) Rounds”

*Pearson chi-square (1 degree of freedom) with Sidak-adjusted p values: tests for association between “Pre-pandemic” and “Currently”

Engaging students during virtual sessions was a new challenge for many CDs. Techniques to increase student engagement included calling on students during lectures (85%, 79/93), requiring cameras to be on (65%, 60/93), using breakout rooms (51%, 47/93), and polling students (48%, 45/93). In open-ended comments, many CDs noted that use of virtual education platforms was highly convenient and allowed for standardized curriculum delivery across multiple sites; and that they continued using virtual platforms for teaching even after resuming in-person activities. More than half of the respondents reported the pandemic adversely affected the clinical learning environment (Table 2).

Table 2

Overall Impact of the COVID-19 Pandemic on the Clinical Learning Environment Across all Internal Medicine Clerkship Sites (n=99)

Activity	Decreased, no. of responses (%)	Unchanged, no. of responses (%)	Increased, no. of responses (%)	Don’t know/Unsure, no. of responses, n (%)
Number of patients followed	35 (35.4)	62 (62.6)	1 (1)	1 (1)
Diversity of patient pathology	26 (26.3)	69 (69.7)	3 (3)	1 (1)
Time at patient bedside	53 (53.5)	38 (38.4)	0 (--)	8 (8.1)
Physical exam opportunities	66 (66.7)	27 (27.3)	0 (--)	6 (6.1)
Medical student autonomy	21 (21.2)	76 (76.8)	1 (1.0)	1 (1)
Number of medical students per team/faculty	21 (21.2)	55 (55.6)	23 (23.2)	0 (--)
Quantity of teaching time	37 (37.4)	57 (57.6)	2 (2.0)	3 (3)
Availability of clinical teaching space*	66 (67.4)	28 (28.6)	2 (2.0)	2 (2)
Availability of clinical workspace	60 (60.6)	35 (35.4)	1 (1.0)	3 (3)
Student night-time clinical work**	10 (13.5)	50 (67.6)	13 (17.6)	1 (1.4)

*n=98; **n=74 (an additional 25 respondents reported “not applicable”)

Note: Questions were presented to 97 respondents who reported that students returned to in-person clinical aspects of the internal medicine clerkship and to two respondents who reported that “the clerkship was not suspended, and students were not removed from all in-person clinical rotations”

Assessment in the IM Clerkships

Approximately half (46%, 46/100) of respondents reported their medical school changed the clerkship grading system in response to the pandemic. Pre-pandemic, most schools (89%, 90/101) utilized some form of tiered grading system, whereas only 11% (11/101) of schools were using a pass/fail system. At the time of clinical re-entry, 47% (46/97) of schools were using a pass/fail grading system, whereas, at the time of the survey, 23% (23/100) were using a pass/fail system (p<0.001 for all comparisons). Moreover, methods of assessment changed. The use of in-person Observed Structured Clinical Exams decreased at the time of clinical re-entry. There was minimal incorporation of telemedicine observation as assessment (Table 3).

Table 3

Type of Assessment Contributing to the Final Grade in the Internal Medicine Clerkship

	Pre-pandemic (n=99), no. of responses (%)	Clinical re-entry (n=95), no. of responses (%)	Currently (n=98), no. of responses (%)	p value*
Oral exam	6 (6.1)	3 (3.2)	4 (4.1)	0.315
Faculty evaluations	97 (98)	90 (94.7)	96 (98)	0.260
Evaluations from non-physician team members	4 (4)	4 (4.2)	4 (4.1)	0.999
Evaluations from residents	77 (77.8)	73 (76.8)	76 (77.6)	0.611
Assessment from patients	2 (2)	2 (2.1)	3 (3.1)	0.999
In-person OSCE**	45 (45.5)	13 (13.7)	22 (22.5)	<0.001
Virtual OSCE	1 (1)	28 (29.5)	25 (25.5)	<0.001
Other simulation examinations	8 (8.1)	5 (5.3)	8 (8.2)	0.611
Observed bedside encounters	45 (45.5)	38 (40)	46 (46.9)	0.162
Telemedicine observation	0 (--)	6 (6.3)	5 (5.1)	0.039
MCQs other than NBME Shelf exam	23 (23.2)	19 (20)	24 (24.5)	0.101
NBME Shelf exam	92 (92.9)	84 (88.4)	89 (90.8)	0.064
Participation in group didactics	35 (35.4)	31 (32.6)	33 (33.7)	0.225
Online modules	23 (23.2)	34 (35.8)	30 (30.6)	0.001
Notes/write-ups	61 (61.6)	54 (56.8)	58 (59.2)	0.152
Graded reflections	16 (16.2)	12 (12.6)	13 (13.3)	0.135
Other	15 (15.2)	17 (17.9)	16 (16.3)	0.372

*Pearson chi-square (2 degrees of freedom) with Sidak-adjusted p values; tests for goodness-of-fit between all three periods

**Observed Structured Clinical Exam with standardized patients in-person

MCQ, multiple choice question; NBME, National Board of Medical Examiners

n, total number of respondents/phase (i.e., pre-pandemic, clinical re-entry, and currently (in relevance to survey)

Medical school leadership and CDs contributed to the decision to change the grading system; 36% (16/96) of schools considered student input in the decision. Among the 45 CDs at schools that changed the grading system, 78% (35/45) agreed or strongly agreed with the decision. CDs expressed concerns about the effect of grading system changes on residency applications (71.6%, 48/67), ability to assess students (59.7%, 40/67), student learning (29.9%, 20/67), student engagement (29.9%, 20/67), the quality of letters of recommendation (25.4%, 17/67), and effects on underrepresented student groups (17.9%, 12/67).

Post-clerkship Internal Medicine Clinical Experiences

Sixty-six percent (65/99) of medical schools did not require additional IM clinical experiences in the fourth year to replace IM clerkship time that was postponed or missed due to the pandemic. For 27% (27/99), the make-up time was between 2 and 6 weeks in duration. In 48% (46/96) of schools, students completed their first sub-internship in the summer, and in 40% (38/96) in the fall. Most CDs did not report a change in capacity, patient volume, duration of the sub-internship, or night-time experience. Only 13% (13/100) of respondents created new sub-internships in response to the pandemic, such as adding new clinical sites less affected by COVID-19 or allowing students to complete their rotation on services not previously utilized.

With respect to patient contact during fourth-year rotations, 53% (44/83) of medical schools permitted students to see asymptomatic patients who had been tested and were awaiting test results. Thirty-nine percent of schools (32/83) allowed students to care for symptomatic patients once the COVID-19 test returned negative; 13% (11/83) gave students the option to care for COVID-19-positive patients.

Clerkship Director Roles and Support

CDs found themselves rapidly modifying curricula for online delivery and creating new curricular materials. Seventy-six percent (77/101) of respondents reported curricular modifications for online delivery being the most time-intensive responsibility. Scheduling faculty presentations, reviewing and editing online resources, and modifying assessment processes were additional time-consuming responsibilities. From the 79 respondents (78.2%) who assumed new or expanded responsibilities due to the pandemic, 72% (57/79) reported having increased teaching responsibilities in the clerkship. Other new roles included serving on curriculum (30%, 24/79) or institutional committees (24%, 19/79), and participating in faculty development (30%, 24/79). Clinical responsibilities increased for 40% (40/100) of CDs, remained the same for 47% (47/100), and decreased for 9% (9/100).

Sixty-five percent (34/52) of respondents reported that resources for operating the clerkship were not modified during the pandemic. Among 52% (52/101) of respondents who were asked about resources needed, only 27% (14/52) reported that resource allocation changes were made. The mean percent full-time equivalent support for the CD position was 36.1 (SD: 15.2) during all time periods. CDs reported need for more support, including additional time for CD duties (55%, 50/91), development of virtual curricula (54%, 49/91), new assessments (33%, 30/91), and new ambulatory sites (37%, 34/91), and for faculty development for virtual curricula (55%, 50/91) and time support for clerkship coordinators (41%, 37/91).

For 51% (51/100) of respondents, scholarly productivity decreased, and 66% (66/100) reported decreased ability to present their work. Self-reported explanations included not enough time to write, increased time demands for creating new clerkship curricula and assessments, and online instruction. At the time of the survey, 34% (33/98) of CDs screened positively for burnout using the 2-item Maslach Burnout Inventory,[5] and 47% (46/98) reported to have considered resigning from their position in the past year.

DISCUSSION

The COVID-19 pandemic had a significant impact on clerkship training, spurring rapid transformation in medical education. To our knowledge, this is the first study to describe the curricular changes to IM undergraduate clinical education during the early phase of the pandemic, and the outcomes of these changes upon re-entry to in-person clinical activities. We used the model of Understanding Crisis Response from the Royal Society[2] as a framework to organize our findings based on whether the clerkship education components were “amplified,” “restarted,” “let go,” and “ended” (Fig. 2).

Figure 2

Responses applied to the model of Understanding Crisis Response from the Royal Society for the Encouragement of Arts, Manufactures, and Commerce.

“Amplified”

The transition from in-person to virtual instruction was a significant modification that was amplified during the pandemic. CDs noted the convenience of delivering content online and the consistency of teaching across training sites which may account for the continued use of online instruction after clinical re-entry. Ruiz et al.[6] have suggested that online learning technology may offer learners control over content to tailor their experiences, but cannot replace in-person instruction methods. Our early findings suggest that virtual clerkship education has been amplified, although the impact on student learning and academic performance is uncertain. Additionally, a section of our survey (reported separately)[7] observed the introduction and amplification of telehealth in the IM clerkship, particularly in the ambulatory setting.

During the pandemic, the responsibilities of CDs were also “amplified” to include additional responsibilities. The strained CDs did not have supplementary resources which adversely affected their scholarly activity. According to the Alliance for Clinical Education, scholarly activity in the clerkship is an important measure of productivity toward academic promotion, necessary for maintaining longevity in a CD position, and leading a successful clerkship.[8] Furthermore, more than one-third of CDs (34%) screened positively for burnout, an increase compared with 25% from the 2019 CDIM Annual Survey.[9] Moreover, almost half of CDs (47%) had considered resigning from their position which is higher compared with 35% in 2018 and 2019.[9] These findings are concerning and underscore the need to provide additional support to CDs, particularly during times of rapid curricular transformation.

“Restarted”

Overall, medical students returned to direct patient care on the IM clerkship after approximately 3 months (median of 85.5 days). Despite the restarting of clinical training through direct patient care, inpatient time increased but did not fully rebound to pre-pandemic levels. The impact of these modifications on medical students’ development of clinical skills and professional identity formation is unknown.[10]

In contrast with the clerkships, sub-internship IM experiences were “restarted” and remained similar to pre-pandemic experiences. As we focus more on the UME to GME transition, this may represent a missed opportunity to improve post-clerkship clinical education.

“Let Go”

Bedside teaching and physical exam instruction were “let go” during the early phase of the pandemic. This was likely due to social distancing and efforts to mitigate risk. This may have adversely affected clerkship student education in history taking, physical examination, and development of communication skills.[11] A survey of final-year medical students from 33 medical schools across the UK revealed that disruptions of direct patient care during the pandemic had the most significant effect on student confidence and preparedness.[12] Assessing the impact of losing this valuable time at the bedside may be an important next step to address potential gaps in student knowledge, clinical, or communication skills. It is unclear at this time to what extent bedside teaching will be restarted.

The pandemic did not affect the overall IM clerkship structure (i.e., longitudinal integrated, traditional), but led to an “amplified” change in the duration of the ambulatory medicine rotations that remained low at clinical re-entry and the time of the survey at 1.5 weeks on average. This is concerning and may have implications for student preparedness to practice in outpatient settings during residency and student interest in general IM and ambulatory-based specialties. Prospective studies are needed to show the long-term impact of the pandemic on ambulatory education.

“Ended”

In response to the pandemic, approximately half of medical schools changed their grading system, with more schools transitioning to a pass/fail system. This decision was likely driven by a sudden change in the ability to assess students equitably given the disruption in clinical opportunities for assessment. Additionally, considering the significant effect of the pandemic on student lives and emotional well-being, a more forgiving grade structure could have the potential to alleviate some of the anxiety students were experiencing.[13] For many of these clerkships, however, the change from a tiered to a pass/fail grading system was temporary and “ended.” The “ending” of assessment changes may represent a missed opportunity for grading reform given the concerns about potential structural bias in the clerkship grading process that can lead to inequities and disproportionately impact underrepresented in medicine students.[14,15]

The study is not without limitations. Although, the survey response rate was representative of the population (74%), there may have been some nonresponse bias or measurement error as only designated IM CDs were surveyed, and not the entire IM clerkship leadership. Moreover, the survey was limited to clerkship training experiences during a specific timeframe. Our study was limited to survey results which did not allow further in-depth exploration. Additionally, we captured data during the initial phase of the pandemic; longitudinal assessment of the impact of the pandemic on student learning and IM clerkships will be imperative.

Overall, our results illustrate how IM clerkships across the country responded to the AAMC guidance to remove students during the COVID-19 pandemic. These findings highlight areas clinical medical educators may need to focus on for the future: first, capitalizing on the “amplification” of virtual clinical education and thoughtfully designing didactic curricula that optimize medical student learning outcomes and engagement; second, reflecting on the temporal “letting go” of bedside teaching of physical exam and how to “restart” this important part of clinical training to best prepare students for clinical practice; third, the “ending” of the rapid transformation in clerkship assessment during the pandemic with the shift to a pass/fail system may represent a missed opportunity to create more robust and equitable assessment systems.[16] Last, providing support and resources to clinical educators during times of rapid curricular transformation is critical for faculty well-being and retention.

CONCLUSION

Our study showed how IM CDs adapted their curricula in response to the AAMC recommendations.[1] These changes primarily affected clerkship students, whereas the sub-internship curriculum remained largely unaffected. Future research needs to explore the long-term implications of these changes on student education and the post-pandemic clinical learning environment.

(DOCX 427 kb)