Emergency medicine clerkship director experience adapting emergency remote learning during the onset of COVID-19 pandemic.

Objectives: The recent outbreak of the COVID-19 altered the traditional paradigm of clinical medical education. While individual clerkships have shared their curricular adaptations via social and academic networking media, there is currently no organizational standard in establishing a nonclinical, emergency medicine (EM) virtual rotation (VR). The primary objective of this study was to describe EM clerkship directors' (CDs) perspectives on their experience adapting an EM VR curriculum during the onset of the COVID-19 pandemic.
Methods: A 21-item survey with quantitative and qualitative questions was disseminated between June and August 2020 to EM CDs via the Clerkship Director of Emergency Medicine Listserv to describe their experience and perspectives in adapting a VR during spring 2020.
Results: We analyzed 59 of 77 EM clerkship survey responses. Among respondents, 52% adapted a VR while 47.5% did not. Of those who adapted a VR, 71% of CDs had 2 weeks or less to develop the new curriculum, with 84% reporting usual or increased clinical load during that time. Clerkships significantly diversified their asynchronous educational content and utilized several instructional models to substitute the loss of clinical experience. Reflecting on the experience, 71% of CDs did not feel comfortable writing a standardized letter of evaluation for students based on the VR, with the majority citing inability to evaluate students' competencies in a clinical context.
Conclusion: A crisis such as COVID-19 necessitates change in all facets of medical education. While EM educators demonstrated the ability to create emergency remote learning with limited time, this was not equivalent to the formal development of preplanned VR experiences. Future faculty development and curriculum innovation are required to fully transition an in-person immersive experience to a noninferior virtual experience.

INTRODUCTION

Emergency medicine (EM) is a unique, exciting medical specialty in which emergency physicians are given a brief period of time to interact with, examine, diagnose, and treat patients who present with a wide array of undifferentiated pathologies. The skills of the emergency physician and an understanding of the capabilities and limitations of the emergency department (ED) have been shown to be valuable to medical student education regardless of the student's chosen specialty. The argument has been made for many years that EM should be a required portion of every medical school curriculum. , , Moreover, the EM clerkship experience is integral for students being able to pursue EM as their chosen specialty, because it can generate a standardized letter of evaluation (SLOE) required by EM residency programs to consider students for an interview. While each EM rotation experience may vary, the goals of the clerkship have always been consistent: (1) to expose medical students to a wide array of acute pathologies within the ED clinical environment; (2) to teach them to work as part of a complex, interprofessional team that provides the highest quality unscheduled health care; and (3) to evaluate their clinical skills as they extend from basic history gathering and physical examination to diagnosis and applied management. ,

In response to the unprecedented outbreak of COVID‐19, the Association of American Medical Colleges released an announcement on March 17, 2020, strongly encouraging all medical schools to pause student clinical rotations. Institutions across the United States immediately altered the traditional paradigm of clinical medical education, employing educational platforms such as video conferencing and virtual simulation to allow for distance learning to preserve resources and reduce COVID‐19 transmission. , Schools and educators were forced to focus on how to minimize the loss of student learning due to reduced clinical exposure. Faculty from all specialties struggled to find ways to convert the traditional bedside learning obtained by students naturally over the course of a clinical rotation to specific virtual learning exercises and experiences.

In the aftermath of the emergent switch to remote learning, EM faculty are still struggling to discover the impacts this might have had on students. While individual clerkships have independently shared their curricular adaptations via social and academic networking media, there is currently no aggregate collection or review on how EM clerkship directors (CDs) as a whole have adapted their individual curriculum with the new pandemic restrictions. This knowledge can provide the impetus to create a lasting shift in the traditional clinical bedside learning for our students, thus ultimately affecting their knowledge acquisition and essential clinical skills for future global pandemics.

The primary objective of this study was to describe EM CDs’ perspectives on their experience adapting an EM virtual rotation (VR) curriculum during the onset of the COVID‐19 pandemic. Understanding insights from CDs and impacts these curricular changes had on students will help stakeholders better prepare future virtual learning experiences in the face of similar disasters.

METHODS

A 21‐item survey was constructed with quantitative and qualitative questions on conducting a VR during the COVID‐19 pandemic from the perspective of CDs. The questionnaire was divided into sections based on clerkship demographics (institution name and location), objectives and curriculum development, content delivery, assessment, and reflections. The questionnaire was reviewed and revised by internal study investigators and external, select Council of Residency Directors in Emergency Medicine (CORD) Advising Students Committee in Emergency Medicine (ASC‐EM) members, with education background ranging from deans, CDs, and program directors in a sequential group review process to obtain survey consensus. A pilot survey was administered in June 2020 to ASC‐EM away rotation and COVID‐19 team members (N = 30) through Google Forms for feedback on survey content, flow, and additional comments or edits. Their responses and feedback were reviewed and minor changes were incorporated into the final survey.

Inclusion criteria were: (1) age greater than 18 years old and (2) active CD at an accredited U.S. EM residency program. Programs that did not answer whether they offered a VR or not were excluded from the final analysis.

The survey was administered from late June to August 2020 and distributed via email to all members of Clerkship Directors in the Emergency Medicine (CDEM) Listservs for dissemination to a convenience sample of all potential current EM CDs. Specific consideration was made to exclude any mention of “virtual rotation” in the recruitment email to promote greater survey participation among non–VR adaptees. Given the known redundancies in membership within the CDEM listserv (N = 383), the total eligible subject pool was extrapolated from the number of Liaison Committee on Medical Education–accredited medical schools (n = 155).

Eligible participants received a brief study summary that included details on enrollment, inclusion/exclusion criteria, data collection, result dissemination, and an anonymous link to the Qualtrics (https://qualtrics.com) survey for interested participants. Personal identifiers were not directly collected for data analysis; however, surveyed participants could indicate willingness to participate in a phone interview separate from the survey study for details regarding their curriculum. Emails were sent to the CDEM Listserv every 2 weeks to encourage survey responses. The survey study was exempted by the Thomas Jefferson University Institutional Review Board (IRB study #20E.550).

Quantitative data were analyzed with descriptive statistics using Excel 2016. Thematic analysis using a constructivist research paradigm was performed on the qualitative comments. Authors performing the analysis were all EM faculty involved in undergraduate medical education at their respective medical schools. Their experience as faculty ranged from less than 1 to 17 years and roles varied, including clinical faculty, CD, and assistant dean for undergraduate medical education. This diversity of experiences helped guard against potential gaps in analysis. The authors had no contact with the respondents and were not affiliated with their respective institutions. This helped prevent undesired subjectivity during analysis such as interpreting certain results with reference to the local institutional context. Two authors (XCZ, RR) independently generated common themes from open coding of free‐response survey narratives. All discrepancies were arbitrated by third and fourth authors (KP, ME). Final results were reviewed and approved by the entire research team.

RESULTS

The survey received a total of 77 responses; 15 were excluded (four did not meet the inclusion criteria, 11 were excluded for not answering whether they adapted a VR or not). Of the remaining 62 responses, three were duplicates that were merged after checking for contradictions. This resulted in 59 analyzed EM clerkships (37 required rotations, 22 elective rotations).

Among these, 52.5% (n = 31/59) clerkships adapted a VR while 47.5% (n = 28/59) did not. Of those conducting VRs, 22.6% (n = 7/31) were for MS3 only, 61.3% (n = 10/31) were for MS4 only, and 16.1% (n = 5/31) were offered to both. Statistical calculation for the remainder of the data was based on completed optional survey questions. Respondents’ experiences adapting and running a VR are described in Table 1.

TABLE 1

Clerkship experience adapting EM VR (N = 31)

	% Respondents	n
Time available to develop VR (weeks)
<1	32.26%	10
1–2	38.71%	12
2–4	22.58%	7
>4	3.23%	1
Time spent developing VR (h)
<12	12.90%	4
12–24	38.71%	12
24–72	32.26%	10
≥72	12.90%	4
Clinical load during VR development
Reduced	12.90%	4
Usualload	67.74%	21
Increasedload	16.13%	5
Grading scheme utilized
Ordinal (i.e., A, B, C, D)	12.90%	4
Pass/fail	74.19%	23
Faculty interaction with students outside clinical shifts
Increased	41.94%	13
No change	6.45%	2
Decreased	38.71%	12
I am able to get to know the student as an individual better in a VR
Strongly disagree	41.94%	13
Somewhat disagree	22.58%	7
Neither agree or disagree	6.45%	2
Somewhat agree	16.13%	5
Strongly agree	0.00%	0
I am able to evaluate the student's clinical competencies better as specified by the SLOE in a VR
Strongly disagree	58.06%	18
Somewhat disagree	19.35%	6
Neither agree or disagree	3.23%	1
Somewhat agree	3.23%	1
Strongly agree	3.23%	1
Comfort writing SLOE based on VR
Comfortable	16.13%	5
Not comfortable	70.97%	22
Interest in offering current iteration of VR in the future
Offer in addition to clinical rotation	9.68%	3
Only as backup to clinical rotation	67.74%	21
Need significant modification	3.23%	1
Never offer again	3.23%	1

Abbreviations: SLOE, standardized letter of evaluation; VR, virtual rotatioin.

Regarding the VR curricular content, respondents estimated that an average of 28.89% was adapted from the existing in‐person clerkship, 35.37% was newly created by the program, and 35.74% was adapted from outside sources such as ALiEM Bridge to EM. Respondents were asked about changes in their utilization of several instructional methods while adapting from an in‐person to a VR. Figure 1 describes this change in utilization on a 3‐point scale (–1 = decreased, 0 = did not change, +1 = increased). Clerkships significantly increased the utilization of free and paid online educational content, question banks, lectures, and problem‐based learning. Simulation and mock oral boards, individual or group, showed the most variance in change in utilization from in‐person to virtual.

FIGURE 1

Mean changes in utilization of instructional methods from in‐person rotation to VR on a 3‐point scale (–1 = decreased, 0 = did not change, +1 = increased). Brackets represent margin of error based on a 95% confidence interval. JC = journal clubs; PBL = problem‐based learning; Qbank = question banks; Student Pres = student presentations

Respondents were also asked to reflect upon their spring VR experience and how it might impact their future EM rotations through a series of free‐response questions. Major themes, their frequency, and their example quotes are described in Table 2. Finally, respondents were asked to provide some resources that they found useful while conducting the VR. They are described in Table 3, arranged by instructional category and cost.

TABLE 2

Themes, frequency, and example quotes from the qualitative survey questions

Survey question	Theme	Frequency	Example quote
If you feel comfortable writing their SLOEs, how do you plan to address the clinical competencies included in the SLOE?	Using available information	3/5	“Using what info I have from the oral presentations and cases.”
	Writing a modified or limited SLOE	3/5	“I do feel that professionalism, and to some degree, teamwork, and problem solving, as well as other personal characteristics were reliably assessed.”
What are the factors keeping you from writing a SLOE for the virtual EM rotation?	Lack of clinical exposure	14/18	“No actual patient care and clinical experience.”
	Lack of time with students	3/18	“Not enough interaction to be able to evaluate.”
How might you change your VR to account for these factors (limitations keeping you from writing a SLOE for VR)?	Could not be done	7/13	“I don't believe it can be done.” “Won't write a SLOE unless we get to work clinically with them.”
	Modify the learning experience	4/13	“Try to incorporate some type of oral board or simulation case.”
Describe one thing from the VR that you might adapt to the in‐person rotation.	Adding new course activities	9/19	“We're keeping the Mock Oral Boards exam and adding it to our in‐person rotation.”
	Supplementing the in‐person course with virtual learning	8/19	“Some Zoom didactics to accommodate people at home after night shifts.” “We created some virtual interactive cases we hope to use going forward.”
	Increasing small‐group activities	4/19	“Increased and enhanced time with learners in a small group setting.”

Abbreviations: SLOE, standardized letter of evaluation; VR, virtual rotatioin.

TABLE 3

Recommended instructional tools for VRs

	Always free	Can be freea	Paid only
Asynchronous core content	ALiEM U CDEM M4 Curriculum CORD Education and Curricula Foundations of EM	ALiEM Bridge to EM EM:RAP C3	EM Coach Society for Critical Care Medicine Virtual Critical Care Rounds
Blogs	RebelEM Taming the SRU
Podcasts	EM Basic	EM:RAP
ECG learning	Wave‐Maven ECG Stampede		MonitorTech.org
Simulation practice		Full Code
Questions			Rosh Review
Accessibility		Canvas Google Classroom Slack Zoom

aResources listed under “Can be Free” include software or websites with limited free access (with a comprehensive paid option) or that are only free with academic/organizational affiliations.

DISCUSSION

A crisis such as the current COVID‐19 pandemic necessitates change and challenges all facets of medical education, including the EM clerkship experience. EM is a unique clinical experience and a profession in which many aspects of society and medicine merge together in often unparalleled acuity, making it particularly difficult to replicate the experience in any other venue. Despite these challenges, just over half of our survey respondents utilized a virtual curriculum, the majority of which were oriented toward fourth‐year medical students. The ability to create and deliver such curricula in a time‐compressed fashion is an example of adaptive expertise in medical education. Experienced educators are able to apply their knowledge and skills to adapt to a novel situation.

The timeline for VR curriculum development was particularly short when compared to the usual timeline of months to years for individual courses or larger institutional curriculum reform. , Nearly all virtual curricula were generated in less than 4 weeks with most in less than 2 weeks (Table 1). Online course development typically requires more time investment than in‐person course development and CDs who were developing VR curricula overwhelmingly had the same or increased clinical responsibilities. This emphasizes the need to distinguish between emergency remote learning (e.g., a sudden need to transition to online only or virtual learning) versus the development of planned online educational experiences. While EM educators demonstrated that they can accomplish emergency remote learning, this is not equivalent to formal development of preplanned VR experiences.

A strong majority of faculty felt that the virtual format limited the ability to interact and connect with students and also felt that it was more difficult to evaluate students (Table 1). We suspect a direct link between these two elements. Although a causal relationship cannot be formally established here, an education trend in EM prior to COVID‐19 pandemic stressed the importance of direct observation in an authentic clinical environment. We suspect that the limitations on both formal and informal student–faculty interaction in the workplace challenged faculty's ability to assess these learners, leading to their reluctance to formulate a SLOE.

The narrative data also identified areas that might be amenable to assessment during VRs, namely, teamwork, professionalism, and problem solving. This is consistent with reports in the literature of these qualities being assessed in online health professions course work. Furthermore, for the areas of teamwork, professionalism, and problem solving, respondents suggested that the assessment methods, rather than the assessed content, were the real limitations in their ability to evaluate students. Because assessments typically used in the clinical setting, such as direct observation, may not translate easily to a VR, it may be prudent for medical educators to consider this when designing future programmatic, learning, and clerkship objectives. Prospective attention to this matter may help with the integration of virtual learning within the medical school.

Further, a lack of confidence in an online assessment tool is endorsed by the trend of shifting grading schemes to pass/fail (74%), a significant increase from 12.7% reported in literature. A lack of confidence in assessment tools likely yielded the use of binary grading to not penalize students for limited assessment methods and to protect faculty from discerning too much detail from limited interactions.

These results are not unexpected, because EM clerkship faculty and directors would not be expected to routinely assess students in a virtual platform and therefore not have robust or well‐developed virtual assessment tools. It stands to reason that faculty development targeted at the creating and utilizing virtual assessment tools would be a worthwhile investment should this type of education continue or be forced upon institutions during a critical sub internship season (the summer months for EM).

Survey respondents overwhelmingly felt that a virtual clerkship held in the future should solely be utilized as a “backup” resource, with a small minority suggesting that some combination of live and virtual experiences might be beneficial in the future. Given the limitations acknowledged by our respondents, one could speculate that the EM virtual curriculum, without the accompanying clinical experience, is not ready for “prime time.” However, as the country continues to experience increased COVID‐19 cases and face potential for other pandemics or natural disasters that may impact in‐person learning in the future, one might posit that the development of a virtual curriculum, even as a backup, requires further investment.

Our study group doubts that a virtual clerkship would replace a fully immersive in‐person EM experience. However, this study reflected the underlying flexible nature of EM, as educators sought innovative ways to teach, assess, and evaluate learners through a critical time period.

LIMITATIONS

As is common with survey data, the primary limitation of this study is in the response rate (37%) from available EM clerkships nationwide despite the email reminders. There may be selection bias toward those adapting VR though we made specific efforts to avoid leading words, such as “virtual rotation,” in the recruitment emails. Our survey also did not distinguish between elective and required rotation experiences that may have had differing curricular design due to their respective learning priorities and expectations. While the qualitative data were insightful, the responses were often concise and limited by our inability to ask follow‐up questions within the survey.

The survey evaluated the spring EM clerkship experience during the height of restricted learner presence within the clinical environment, a time in which most EM‐bound students had already completed their subinternship or away rotations. While our survey questions about SLOEs do not fully reflect true perspectives from subinternship experiences later in the summer, we intentionally chose to sample the earlier experience as all EM clerkships across the country were forced to ban in‐person learning during this time. We hypothesize that by summer, many hospitals may have instituted rigorous protocols to reintroduce medical learners into the ED, and this availability of choice would impact perspectives on SLOE writing for a virtual‐only rotation. More data are needed to assess the subinternship experience of students during the summer and fall 2020 with consideration to differences between virtual and in‐person rotation experiences and changes in available in‐person clinical experiences for students given ongoing COVID‐19 restrictions from medical schools. It is also concerning that feedback suggests that education directors may not accept a VR SLOE as a viable assessment of a student's clinical skills. Specific requirements for acceptable evaluation need to be defined within our community to allow for beneficial rotation experiences for students.

Many of our survey respondents had limited or no experience in designing a virtual curriculum prior to being required to institute one. Although we sincerely hope that we may never encounter a situation necessitating student withdrawal from EDs again, it is not unimaginable to picture a future where this may happen. Our education faculty should have access to appropriate training to allow for them to feel more qualified and comfortable instituting a VR experience.

CONCLUSION

A crisis such as COVID‐19 necessitates change in all facets of medical education. During this emergency, emergency medicine educators demonstrated the ability to quickly adapt their in‐person clinical rotations to accommodate distanced learning by utilizing a myriad of asynchronous content. However, it is imperative for stakeholders to recognize that emergency remote learning is not equivalent to a formal, preplanned virtual learning experience. A majority of respondents did not feel comfortable evaluating students based on a virtual rotation alone and would only utilize it as a back up to a clinical rotation. Considering the ongoing pandemic as well as other financial or personal constraints to student travel, there is an unprecedented need within the medical education community for further faculty development in virtual education and to develop a consensus in adapting to the new, postpandemic norm.

CONFLICT OF INTEREST

The authors have no potential conflicts to disclose.

AUTHOR CONTRIBUTIONS

Ronnie Ren and Xiao Chi Zhang conceptualized the study, acquired the data, and disseminated the study to eligible participants. Kendra Parekh, Ronnie Ren, Mark Olaf, and Xiao Chi Zhang reviewed and analyzed the data and provided critical revisions of the manuscript based on the study findings. Molly Estes and Melanie Camejo conducted research on the COVID pandemic and how current medical education adapted to this new paradigm shift. Doug Franzen and Xiao Chi Zhang mentored the writing process and provided additional study supervision. All authors contributed meaningfully to the manuscript write‐up.