Planning virtual residency interviews as a result of COVID-19: insight from residency applicants and physicians conducting interviews.

BACKGROUND: In response to the COVID-19 pandemic, the 2020-2021 residency interview process will undergo significant changes. Residency programme stakeholders would benefit from knowledge on what students and physicians expect from this process.
OBJECTIVE: The purpose of the study was to describe and contrast the perspectives of student applicants and interviewing physicians related to the residency programme virtual interview process.
METHODS: A survey consisting of 24 Likert statements was administered across listservs in summer 2020 to physicians (attendings and residents who interview medical students). Medical students also received an anonymous survey and were recruited via email to participate.
RESULTS: A total of 155 individuals (104 fourth-year medical students and 51 physicians) completed a survey. Results showed students would prefer in-person interviews over virtual. Residency applicants had high agreement on the limited ability to fully assess the programme and city due to virtual interviews. Individuals with lower step 1 scores had higher agreement on preferring in-person interviews. Individuals in the lowest and highest scoring groups appear more worried about the representation of themselves as a result of virtual interviews. Furthermore, applicants feel that more weight will be placed on steps 1 and 2 scores and class ranks, and they may not be able to fully demonstrate their personality compared with interviewers.
CONCLUSION: The result of COVID-19 has created challenges and subsequent reshuffling of medical education requiring careful preparation and planning. This study provides insight for residency programmes to better understand the applicants' expectations for the 2020-2021 residency interview and matching process. © Author(s) (or their employer(s)) 2022. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

During the COVID-19 pandemic, concerns related to exposure and personal protective equipment have led to the restructuring of medical education. During this time, US undergraduate medical education leaders have discussed methods to provide safe but effective clinical experience for students. In the standard residency application process, students are interviewed in person at 10–20 (or more) programmes around the country.1 This involves significant travel planning, time-off from rotations, and cost to students and residency programmes.

In response to the pandemic, the interview process in 2020–2021 will undergo significant changes. In attempts to limit spread of COVID-19, new policies that restrict visiting ‘away rotation’ clerkships have also transitioned the residency interview process to virtual interviews.2 As a result of this shift, medical student applicants may express concern on their ability to engage, present themselves and feel fully prepared for these interviews.3

As residency programmes prepare for this unique upcoming interview season, residency programme stakeholders would benefit from knowledge on what the students expect from this process. Though some published data have described the perspective of those conducting the interviews (physicians), there is a gap in understanding of student insight, and how these perspectives differ. The purpose of the study was to describe and contrast the perspectives of student applicants and interviewing physicians related to the residency programme virtual interview process.

Methods

After Institutional Review Board (IRB) approval by the University of Louisville, an anonymous survey was administered in summer 2020 across listservs to physicians (attendings and residents who interview medical students for residency) via REDCap, a secure, encrypted online database. Medical students also received a similar anonymous survey and were recruited via email to participate; for this, author ST sent an email to all medical students regardless of which medical residency specialty they were applying to within her medical school class in order to gain their respective feedback. Participation in the survey was voluntary for all groups.

The survey consisted of demographic questions as well 24 Likert statements in which respondents rated their level of agreement from 1 (strongly disagree) to 5 (strongly agree). Descriptive statistics were examined to provide sample characteristics. Next, individuals were categorised to determine if differences existed among gender identity or step 1 score group and survey statement. T-tests were conducted to compare gender differences and distinctions between interviewers and students. Analysis of variances (ANOVAs) were employed for comparing responses among step 1 score groups. For any statistically significant findings from the ANOVAs, Games-Howell post hoc analyses were employed to determine the underlying root cause(s) of the significant differences. Statistical significance was set by convention at p<0.05. For comparisons related to agreement differences, 95% CIs are reported.

Results

A total of 155 individuals completed a survey: 104 fourth-year medical students who will apply to residency this year and 51 physicians who conduct residency interviews. Medical student respondents noted they were applying to a variety of specialties including: anaesthesiology (n=8), child neurology (n=2), dermatology (n=4), emergency medicine (n=17), family medicine (n=10), internal medicine (n=7), medicine-paediatrics (n=4), OB-GYN (n=9), ophthalmology (n=1), orthopaedics (n=3), otolaryngology (n=1), pathology (n=2), paediatrics (n=22), psychiatry (n=5), surgery (n=8), or unclear (n=1). Table 1 provides demographic breakdowns for the respective groups.

Table 1

Demographics

	Medical students(n=104)	Physicians(n=51)
Gender identity (%)
Female	51 (49)	21 (41.2)
Male	53 (51)	30 (58.8)
Race/ethnicity (%)
Asian or Pacific Islander	5 (4.8)	4 (7.8)
Black or African American	3 (2.9)	1 (2)
Hispanic or Latino	3 (2.9)	2 (3.9)
White	92 (88.5)	43 (84.3)
Prefer not to answer or unknown	1 (1)	1 (2)
Step 1: score range (%)
191–220	37 (35.6)
221–240	37 (35.6)
241+	29 (27.9)
Unknown	1 (1)
Faculty rank (%)
Resident		4 (7.8)
Instructor		2 (3.9)
Assistant professor		13 (25.5)
Associate professor		19 (37.3)
Professor		10 (19.6)
Other or unknown		3 (5.9)

Medical student feedback

The medical students responded to the 24 Likert questions and provided insight on the key concepts related to virtual interviews and recruitment strategies for residency programmes (table 2). Highest agreement averages were within the themes of (1) virtual interviews not allowing the students to fully understand the city/programme and (2) students would prefer in-person interviews over remote. Lowest agreement was found in students considering applying to more than one specialty as a result of virtual interviews. Table 2 provides results for all statements.

Table 2

Medical student survey results

Statement	Mean	SD
I am comfortable taking part in remote interviews for residency.	3.26	1.00
I would prefer in-person interviews over remote interviews.	4.46	0.75
I would prefer remote interviews over in-person interviews.	1.94	0.80
I would prefer individual video interviews.	3.98	1.01
I would prefer group video interviews.	1.74	0.84
I believe as a result of remote interviews I will experience less stress than I would if I had to travel for interviews.	2.83	1.11
I think remote interviews will limit my ability to fully assess the programme(s).	4.57	0.52
I think remote interviews will affect my ability to demonstrate my personality.	4.14	0.82
I think remote interviews will affect my ability to fully understand/experience the city of the programme.	4.65	0.50
Due to interviews being conducted remotely, I plan to apply to more programmes.	3.14	1.13
Due to interviews being conducted remotely, I plan to interview at more programmes.	3.38	1.09
Due to interviews being conducted remotely, I plan to rank more programmes.	3.23	1.06
Due to interviews being conducted remotely, I plan to apply to more than one specialty.	1.60	0.76
I fear/think the virtual interviews will not present me in the best light.	3.50	1.04
I believe match success will go up as a result of remote interviews.	2.34	0.83
Due to interviews being conducted remotely, I believe more weight will be placed on my USMLE step 1 score.	3.84	0.97
Due to interviews being conducted remotely, I believe more weight will be placed on my USMLE step 2 scores.	3.68	1.04
Due to interviews being conducted remotely, I believe more weight will be placed on my class rank.	3.39	0.92
Residency programme websites are an important part of my process of evaluating programmes.	4.09	0.81
Residency programme Twitter presence is an important part of my process of evaluating programmes.	2.03	1.07
Residency programme Facebook presence is an important part of my process of evaluating programmes.	2.03	0.96
Residency programme Instagram presence is an important part of my process of evaluating programmes.	2.27	1.13
I would watch a video tour of the city of the programme if provided by the residency programme.	4.29	0.82
I would watch a video montage regarding the residency programme if provided.	4.51	0.64

Fourth-year medical students (n=104) were asked to respond to statements related to residency interviews moving to virtual and other facets of the residency interview process from strongly disagree (1) to strongly agree (5).

USMLE, US Medical Licensing Examination.

Differences by gender identity

T-tests were conducted to determine differences between women and men on the survey statements. Results showed similar agreement levels among 22 of the 24 items with two items showing statistically significant differences. Women had a slightly higher agreement (although the sample showed more total disagreement) average (M=2.49, SD=1.16) on ‘Residency programme Instagram presence is an important part of my process of evaluating programmes’ compared with men (M=2.06, SD=1.06), showing a mean difference of 0.43 (95% CI: 0.00 to 0.87), p=0.049. Women also had a higher average (M=4.53, SD=0.67) on ‘I would watch a video tour of the city of the programme if provided by the residency programme’ compared with men (M=4.06, SD=0.89), showing a mean difference of 0.47 (95% CI: 0.17 to 0.78), p=0.003.

Differences by step 1 score group

ANOVAs were used to determine differences between step 1 score group [(191−220 (n=37); 221−240 (n=37); 241+(n=29)] and survey responses. Results showed that the majority of statement items (21 out of 24) had similar agreement across step 1 score cohorts, with three items showing statistically significant differences. For these three items, Games-Howell post hoc analyses were conducted to determine differences. For the first item, ‘I would prefer in-person interviews over remote interviews’, there were statistically significant differences (p=0.045). Post hoc analyses showed the group with the lowest step 1 score (191–220) had the highest mean (M=4.68, SD=0.53) which was higher than the highest score step 1 group (M=4.45, SD=0.78) and statistically significantly higher than the group scoring 221–240 (M=4.24, SD=0.86) showing a mean difference between these two groups of 0.43 (95% CI: 0.03 to 0.83), p=0.031.

The next item, ‘I believe as a result of remote interviews I will experience less stress than I would if I had to travel for interviews’, showed statistically significant differences (p=0.007). The lowest step 1 score group had lower average agreement (M=2.59, SD=1.21) compared with the middle group (M=2.67, SD=1.03) and statistically significantly lower scores than the highest scoring group (M=3.38, SD=0.86); the mean difference between the lowest and highest scoring step 1 groups was −0.78 (95% CI: −1.40 to –0.17), p=0.009.

Finally, the third item ‘I fear/think the virtual interviews will not present me in the best light’ showed statistically significant differences (p<0.001). On this item the lowest scoring group had the highest average agreement (M=3.84, SD=1.01) compared with the middle scoring group (M=2.97, SD=1.04) and the highest scoring group (M=3.69, SD=0.81). Post hoc analyses revealed that the middle scoring group had statistically significant lower scores than the lowest step 1 score group with a mean difference of −0.86 (95% CI: −1.44 to –0.29), p=0.002. Furthermore, the middle scoring group also had lower scores than the highest scoring group with a mean difference of −0.72 (95% CI: −1.27 to –0.17), p=0.007. On this item, there were no differences between the lowest and highest scoring respective groups.

Physician insight on virtual interviews

Next, we examined physicians’ responses to 24 Likert questions related to virtual interviews and recruitment strategies for residency programmes (table 3). Highest agreement averages occurred along the theme of virtual interviews not allowing the students to fully understand the city and the programme. Lowest agreement was noted in (1) students preferring group video interviews and (2) that students would prefer remote interviews versus in-person. Table 3 provides results for all statements.

Table 3

Physician survey results

Statement	Mean	SD
Medical students will be comfortable taking part in remote interviews for residency.	3.30	0.89
Students would prefer in-person interviews over remote interviews.	4.10	0.85
Students would prefer remote interviews over in-person interviews.	2.12	0.82
Students would prefer individual video interviews.	4.08	0.82
Students would prefer group video interviews.	1.92	0.77
I believe as a result of remote interviews students will experience less stress than if they had to travel for interviews.	2.86	0.98
I think remote interviews will limit the students' ability to fully assess the programme(s).	4.47	0.67
I think remote interviews will affect students' ability to demonstrate their personality.	3.65	0.91
I think remote interviews will affect the students' ability to fully understand/experience the city of the programme.	4.49	0.61
Due to interviews being conducted remotely, students will apply to more programmes.	4.18	0.68
Due to interviews being conducted remotely, students will interview at more programmes.	4.00	0.72
Due to interviews being conducted remotely, students will rank more programmes.	3.96	0.75
Due to interviews being conducted remotely, students will apply to more than one specialty.	2.53	0.86
Students fear/think the virtual interviews will not present them in the best light.	3.78	0.71
I believe match success will go up as a result of remote interviews.	2.25	0.63
Due to interviews being conducted remotely, I believe more weight will be placed on USMLE step 1 scores.	3.18	0.99
Due to interviews being conducted remotely, I believe more weight will be placed on USMLE step 2 scores.	3.24	1.03
Due to interviews being conducted remotely, I believe more weight will placed on class ranks.	3.00	1.08
Residency programme websites are an important part of students' evaluation of programmes.	4.33	0.52
Residency programme Twitter presence is an important part of students' evaluation of programmes.	3.29	0.88
Residency programme Facebook presence is an important part of students' evaluation of programmes.	2.73	0.96
Residency programme Instagram presence is an important part of students' evaluation of programmes.	3.43	1.02
Students would watch a video tour of the city of the programme if provided by the residency programme.	4.27	0.57
Students would watch a video montage regarding the residency programme if provided.	4.28	0.54

Physicians (n=51) were asked to respond to statements related to residency interviews moving to virtual and other facets of the residency interview process from strongly disagree (1) to strongly agree (5).

USMLE, US Medical Licensing Examination.

Comparing the interviewers with the interviewees

While the surveys were slightly different (see table 2 vs table 3), the underlying measures were similar enough to allow comparison of perception differences between residency applicants and interviewers. Table 4 shows multiple statistically significant differences in responses.

Table 4

Differences between applicant survey responses and interviewer survey responses

Residency virtual interviews survey concept	Diff	95% CI	T	P value
Students are comfortable with remote interviews	0.04	−0.29 to 0.24	0.37	0.807
Students prefer in-person interviews	−0.36	−0.63 to −2.71	−0.10	0.008
Students prefer remote interviews	0.18	−0.10 to 1.28	0.45	0.203
Students prefer individual video interviews	0.10	−0.22 to 0.60	0.42	0.551
Students prefer group video interviews	0.18	−0.09 to 1.30	0.46	0.196
Students will experience more stress from virtual interviews	0.04	−0.33 to 0.20	0.40	0.845
Students’ ability to fully assess the programme will be limited*	−0.10	−0.31 to −0.90	0.12	0.369
Students’ ability to fully demonstrate their personality will be affected	−0.50	−0.78 to −3.42	−0.21	0.001
Students’ ability to fully understand the city of the programme will be affected*	−0.16	−0.36 to −1.66	0.03	0.101
Students will apply to more programmes*	1.04	0.75 to 7.07	1.33	<0.001
Students will interview at more programmes*	0.63	0.33 to 4.25	0.92	<0.001
Students will rank more programmes*	0.73	0.43 to 4.89	1.02	<0.001
Students will apply to more than one specialty	0.93	0.67 to 6.90	1.20	<0.001
Students fear virtual interviews will not present them in the best light*	0.28	0.00 to 1.96	0.56	0.053
Match success will go up as a result of remote interviews*	−0.08	−0.32 to −0.68	0.16	0.497
More weight will be placed on step 1 scores	−0.66	−0.99 to −3.96	−0.33	<0.001
More weight will be placed on step 2 scores	−0.45	−0.80 to −2.53	−0.10	0.012
More weight will be placed on class rank	−0.39	−0.72 to −2.37	−0.07	0.019
Programme websites are an important part of students’ evaluation of programmes	0.25	0.00 to 1.99	0.49	0.049
Twitter presence is an important part of students’ evaluation of programmes	1.27	0.92 to 7.34	1.61	<0.001
Facebook presence is an important part of students’ evaluation of programmes	0.70	0.37 to 4.24	1.02	<0.001
Instagram presence is an important part of students’ evaluation of programmes	1.16	0.79 to 6.22	1.53	<0.001
Students would watch a video tour of the city of residency programme*	−0.01	−0.24 to −0.12	0.21	0.902
Students would watch a video montage of the residency programme	−0.23	−0.44 to −2.20	−0.02	0.030

Physicians (n=51) and medical residency applicants (n=104) were asked to respond to statements related to residency interviews moving to virtual and other facets of the residency interview process from strongly disagree (1) to strongly agree (5).

Diff=mean difference. A positive difference indicates that the physicians had a higher average than the students, a negative difference reveals students had a higher average than the physicians.

T=t-test statistic.

*Indicates equal variances not assumed.

Discussion

This study provides insight for student applicants, residency directors and administrators who guide medical students through the interview process. Scant data have addressed optimal recruitment strategies, student mindsets on virtual interviews and plans for the residency application season. This survey showed that students would prefer in-person interviews over virtual. However, with COVID-19 restrictions mandating virtual interviews, residency applicants would prefer individual interviews as opposed to group interviews. Residency applicants had high agreement on the limited ability to fully assess the programme and city as a result of virtual interviews. Individuals with lower step 1 scores had higher agreement on preferring in-person interviews, and as a result of virtual interviews, may be more stressed. Individuals in the lowest and highest scoring groups appear more worried about the representation of themselves as a result of virtual interviews compared to the middle scoring group. Comparisons between interviewers and applicant students yielded multiple differing viewpoints. Applicants feel that they may not be able to fully demonstrate their personality compared with physicians’ perspectives. Also, applicants feel more weight will be placed on steps 1 and 2 scores and class ranks as compared with interviewers. Interviewers were more likely to believe students would apply to and rank more programmes, compared with applicants. Finally, viewpoints differed on recruitment strategies, as social media presence may not be as highly valued by applicants as residency programmes thought.

The complexity of the pandemic and subsequent restructuring of medical education necessitates critical planning. This difficulty especially affects fourth-year medical students, most of whom had to reschedule high-stakes examinations, as they finalise required assessments and seek residency positions.4 Residency programmes must use evidence to guide virtual interview logistics. Residency programmes (including their faculty, residents and so on) may engage in virtual networking events and expand their virtual/social media presence.5

While the challenge of residency interview planning is unique, it is one of many transformations medical education is currently confronting. Proper procedures for administration of offsite examinations, the assurance of students’ exposure to a variety of required diagnoses, the overhaul (and effective cancellation) of many electives and internship opportunities, trends toward video-conferencing for meetings and lectures, avoidance of simulation labs, and many more alterations have been implemented. While we must not forget the main focus of patient care, it is critical to continue to provide the best medical educational experiences, including residency interviews. With the 2020–2021 residency interview season approaching, programmes must continuously evaluate what works and what does not, making changes along the way.3 We can provide positive experiences for students while adhering to rigorous standards.6

Though challenging, the overhaul of the accustomed model of residency application may present opportunities for programmes. Virtual interviews could provide more access and equity for all students.7 With a substantial amount of time and resources required for in-person or virtual interviews, leaders will benefit from making evidence-based decisions on how to allocate resources. Thus, we can ensure that interviews benefit programmes, students and eventually patients.8

In consideration of the findings within this study, some recommendations for residency programmes as they prepare for virtual interviews in the future include:

Provide individual interviews over group interviews when possible.

Increase opportunities for applicants to fully understand the programme such as development of marketing materials on the programme and city of programme that are easily accessible for the applicants to gain better perspective on what the programme offers and how it differs from others.

Understand that individuals may have varying levels of stress as they approach virtual interviews compared with in-person interviews.

Be transparent with applicants on the weighing of application measures such as test scores.

Consistently seek out and use feedback from applicants on what is working well and what needs improved on to optimise the quality of the (virtual) interview process.

Limitations

Our sample for this survey research was modest, lacking power, and most students were from one institution, limiting generalisability. While we did include students applying to and physicians interviewing for a variety of specialties, we did not make comparisons to see if differences occurred by specialty or by faculty rank; thus, future work could examine medical discipline associations or by faculty rank. Additionally, there were interesting findings such as those scoring highest on step 1 feeling that interviews may not present them in the best light which cannot be fully addressed within this work. Future work, using a qualitative approach, may yield important insight as to why these perceptions occur. Finally, our study sample did not include strong representation across different race/ethnicity groups. Future work should include a larger and more diverse sample to determine differences across groups given that individuals experience uniquely different challenges during interviews.

Conclusion

This study provides insight into how applicants, programme directors and physicians feel virtual interviews will impact the residency interview and matching process. The result of COVID-19 has created numerous challenges and subsequent reshuffling of medical education which requires careful preparation and planning. By understanding the perceptions outlined in this study, residency programmes can better understand the applicants’ expectations for the 2020–2021 residency interview and matching process.

COVID-19 has impacted residency interview season.

Applicants’ and interviewers’ perspectives differ on many items.

Knowing how these perspectives differ can help plan accordingly for interview season.

Describe the perspectives of student applicants and interviewing physicians related to the residency programme virtual interview process.

Determine if there are differences between the perspectives.

COVID-19 has impacted many aspects of medical education.

Residency interviews have moved to virtual.