Internal medicine residents' and program directors' perception of virtual interviews during COVID-19: a national survey.

Purpose: Due to the coronavirus disease 2019 pandemic, all Canadian Resident Matching Service interviews for internal medicine subspecialty programs were conducted virtually for the first time. This study explored the perceptions and experiences of internal medicine residents, subspecialty medicine program directors, and interviewers during virtual interviews.
Methods: We invited all Canadian third-year IM residents, subspecialty program directors, and interviewers who participated in the 2020 medical subspecialty medicine interviews to complete a branching survey with a section for residents and one for program directors and interviewers. We distributed the anonymous survey after the submission of the rank order lists, to not affect residency match outcomes. Qualitative data were open-coded thematically and quantitative data were cleaned and then statistically analyzed using descriptive statistics and Analysis of Variance tests.
Results: 62 residents, 59 program directors, and 113 interviewers responded to the survey with representation from almost all Canadian medical faculties and medical subspecialties. Strengths of virtual interviews included reduced cost, stress, pandemic infection risk, and carbon footprint. Weaknesses of virtual interviews included decreased ability to connect personally and informally, and inability to tour medical facilities and cities. A majority of both resident respondents (59.6%) and program directors/interviewer respondents (54.6%) supported conducting interviews virtually in the future. Conclusions: This study showed that the majority of both sampled residents and program directors/interviewers would prefer to conduct medicine subspecialty match interviews virtually in the future, and provides suggestions on how to improve the virtual interviews for the next iteration.

Introduction

The coronavirus disease 2019 (COVID-19) pandemic has imposed travel restrictions and physical distancing policies that make conducting in-person interviews logistically difficult and potentially illegal. Interviews are an important part of the process by which medical students and residents and the programs to which they have applied explore each other for a potential fit for future residency and sub-specialty positions. The Canadian Residency Matching Service process (CaRMS) manages these processes. As such, in 2020, for the first time all residency programs conducted their CaRMS match interviews in a virtual format.

We frame this study as an exploration of this sudden change as a means for determining and establishing more effective practices. Studies on virtual interviews in medical residency show conflicting results with respect to resident and program satisfaction.[1]–[6] A study that incorporates both resident and interviewers perspectives on the same issues in this understudied population would enhance our understandings and show differences in the perspectives between these two co-operating and interacting groups such that both groups’ needs can be understood and met during this highly-stressful and consequential process.

This study investigates the experiences of internal medicine residents, subspecialty program directors (PDs), and interviewers throughout the 2020 virtual sub-specialty position matching process during the COVID-19 pandemic.

Methods

We decided to use a cross section survey study consisting of a voluntary, anonymous survey that was emailed to Canadian third-year (PGY-3) core internal medicine residents and sub-specialty PDs and interviewers who participated in the 2020 CaRMS subspecialty medicine interviews. This survey approach used easily answerable Likert, yes/no, and open-ended questions that collected a wider sub-section of the experiences and better established a varied sample for useful extrapolation to other contexts. Interviews and focus group would have provided more depth with fewer respondents, but not had the same potential for application to other contexts through aggregation as the cross-sectional survey approach. Program directors were invited directly by email by the study authors, and were asked to share the survey with their residents in the sample frame.

The tool combined two separate surveys. It first asked the participants to identify themselves and then, depending on the response, the survey opened to questions relevant to residents or to interviewers. The branch of the survey for residents consisted of 23 5-point Likert and yes/no items and 16 open-response items. The branch of the survey for interviewers/program directors included 11 5-point Likert and yes/no items and 10 open response items. Both branches of the survey took approximately 10 minutes to complete.

This study was reviewed for ethical compliance by the Queen’s University Health Sciences and Affiliated Teaching Hospitals Research Ethics Board. No personal identifying information was collected to ensure anonymity of participants.

The author team reviewed and cleaned the data for completeness and for missing data in Microsoft Excel and then imported to SPSS v25. No participants were excluded. We ran descriptive tests for displaying the results of the participant responses, and inferential tests such as Analysis of Variance (ANOVA) to explore differences between demographic groups and applications per year on responses to normally distributed items. The open responses were categorized by participant type, sorted into clusters based on aspect of the virtual interview experience,[8],[9] and then nested with the quantitative findings.[10] We thereby created a systematic display of the descriptive and statistical trends followed by the narrative and thematic insights provided by the qualitative items.

Results

Resident responses

62 PGY-3 internal medicine residents responded to the survey out of a total potential sample of 338 residents who received the invitation to participate for a total response rate of 18.4%. Responses were received from across Canadian universities and the majority of medical sub-specialties (See Figure 1).

Figure 1

Distribution of Resident sample across institutions

35.4% of resident respondents believed that COVID-19 impacted their chances of matching to their first-choice program. 33.8% of resident respondents applied more broadly to residency programs than they otherwise would have before COVID-19, and 61.4% did not. 87.0% of resident respondents agreed that COVID-19 impacted their ability to do electives away from their home school.

61.3% of resident respondents said that programs should have offered applicants alternative ways to liaise with the program in lieu of cancelled elective time, however only 9.6% of residents reporting having this opportunity. A few residents reported presenting at divisional journal club as being helpful in connecting with program faculty and residents in the absence of on-site electives.

Before the virtual interview, 48.3% of resident respondents were worried about the interview occurring virtually. 41.9% of resident respondents felt that the sub-specialty match process has been more stressful than the R1 match process.

The virtual interviews were structured as panel interviews (67%) and multiple mini-interviews (32%). A common theme, regardless of the type of interview, was that it was easier to connect virtually with small numbers of interviewers than a larger group, and that this resulted in a superior interview experience.

62.9% of resident respondents reported that programs offered virtual opportunities around (including before/during/after) the interview process to connect with faculty and residents outside of the interview itself (See Figure 2).

Figure 2

Media reported by residents to be the most helpful in providing information about residency programs for the 2020 CaRMS subspecialty medicine match.

Resident respondents report that the strengths of virtual interviews included: reduced cost, ease of scheduling, reduced stress overall, and reduced carbon footprint. All of the resident respondents agreed that the virtual CaRMS interview process saved money.

Resident respondents reported that difficulty understanding institutional culture as a weakness of virtual interviews. Residents “could not tour the facilities, engage with current residents”. Resident respondents reported that virtual interviews “felt less personal than in-person” interviews, and that it was “difficult making eye contact, and reading body language”. Technical issues were reported as a weakness of virtual interviews, and 37.0% of resident respondents experienced technical issues.

Given the opportunity next year, 59.6% of resident respondents would prefer to conduct interviews virtually. Resident respondents stated that virtual interviews could be improved by a) standardization across programs including: information sessions, utilizing one virtual conference platform (zoom was favoured by over 70% of resident respondents), and time off for interviews, b) small panel sizes, and c) local program-provided interview space and equipment to minimize distractions and obstacles.

PDs and Interviewer Responses

Fifty-nine program directors and 113 interviewers (172 total) responded to the survey, with representation from all 17 Canadian medical faculties and nearly all medical subspecialties. Response rate for Program directors was 44.3%, whereas the response rate for total interviewers was impossible to calculate as schools defined interviewers differently on the survey.

Nearly all faculty responses indicated that the same criteria were used this year to offer interviews compared to previous years, regardless of the COVID-19 pandemic. Most programs did not require an on-site elective to offer an interview, which is largely unchanged from previous years. Responding PDs reported a total of 1272 CaRMS applications in 2019 and 1283 in 2020. The lone ANOVA test revealed that there was no significant difference (F = 0.005, df = 1,106, p = 0.94) between 2019 (m = 23.56, SD=13.56) and 2020 (m = 23.76, SD = 14.53) applications.

A majority (80.9%) of PDs and interviewers felt that the virtual CaRMS interview process was excellent overall. 82.5% of PDs and interviewers were able to easily evaluate residents’ suitability for their subspecialty program during the virtual interview, and 84.3% felt that they were able to articulate the strengths of their residency program through the virtual interviews.

Twenty-five percent of PDs and interviewers experienced technical difficulties during virtual interviews, which were largely minor issues (i.e., freezing, delay in interview start time) which did not impact the virtual interview overall.

Reported strengths of the virtual interviews by PDs and interviewers include a) cost and time savings for residents, b) scheduling convenience for both interviewers and resident respondents, c) decreased carbon footprint, d) less time away from clinical duties for both resident respondents and interviewers, e) reduced risk of COVID-19 infection, and f) cost savings for the program.

The most common reported weakness of the virtual interviews by PDs and interviewers was a decreased ability to connect with resident respondents personally. Notably and potentially for the best given the resident respondent-reported potential for implicit bias the virtual interviews prevented for “observation of resident’s interactions with current resident respondents in the program, faculty, administrative staff, and even other candidates” that are helpful in gauging ‘fit’ and interest in the program.

Other faculty reported weaknesses of virtual interviews are difficulty promoting the program facilities and city; and difficulty determining resident respondents’ true interest in a program virtually, as “desire to physically present to a location is in itself a demonstration of interest”.

If given the choice of virtual or in-person interviews next year, 54.6% of PDs and interviewers would choose to have virtual interviews. PDs and interviewers suggested the following to improve future virtual interviews: a) standardization across programs (video conference platform (zoom was favoured by over 80% of respondents), information videos), b) technical support available to trouble shoot issues in real time, c) informal virtual gatherings to get to know applicants outside of the structured interview setting.

Discussion and conclusions

We did not find any other national study to describe the impact of the COVID-19 pandemic on virtual medical residency interviews. Despite the weaknesses identified, the majority of resident respondents and PDs/interviewers supported continuation of virtual medical subspecialty interviews in the future.

While there are possible benefits of virtual interviews such as reduced resident financial burden and scheduling conflicts, potential challenges for residency programs include reduced informal interactions with resident respondents to determine compatibility, and perhaps increased acceptance of interviews due to fewer scheduling barriers, making it difficult to discern genuine interest in a program.[7]

In our study, the most frequently reported benefit of virtual interviews by both resident respondents and interviewers was resident cost savings, which although well-noted in the American literature the exact nature of this pressure in Canada continues to be defined and calculated. Given the similar structures of residency application and largely comparable costs in the United States and Canada, we posit that these pressures exist in similar forms and magnitudes. One study of surgical fellowship candidates estimated that virtual interviews saved each candidate approximately $6,000USD in interview travel expenses.[11] Given that the Association of American Medical Colleges (AAMC) reported the mean education debt for indebted medical school graduates was $200,000[12], continuing virtual interviews would help alleviate some financial burden on medical resident respondents. We posit that this burden would logically contribute to the stress and well-being of resident respondents. Addressing this early and exacerbating burden on existing financial difficulties would potentially contribute to a healthier practice of medicine.

This study highlighted a number of inequities in the virtual interview process that should be addressed in future iterations. First, programs should consider providing a neutral, distraction-free space for residents as some reported it was “difficult to get away from [children] and pets.” In addition, high-quality internet and video conferencing equipment should be made available for residents without this resource.

Second, national standardization of several components of virtual interviews should be undertaken to further harmonize the process, including adopting Zoom as the video conferencing platform of choice, given its popularity among both applicants and interviewers, as well as dedicating available support staff for technical difficulties. In addition, expectations for residents’ return to clinical service post interview should be standardized. A few residency programs-imposed time restrictions whereby residents were mandated to promptly return to service after their interview, or to complete interviews post-call, which was reported to exacerbate interview day stress. We suggest that internal medicine program directors discuss this nationally and reach a consensus in order to provide their graduating residents the best interview process possible.

Third, residency programs should offer virtual “meet and greets,” as residents reported this to be the most helpful platform to obtain information about subspecialty programs and the culture of medicine at each institution. This would also increase the opportunities for informal networking between applicants and program representatives, which was a major pitfall of virtual interviews identified in this study.

Lastly, residents in this survey clearly expressed that in-person away electives are highly valued in the subspecialty medicine match process. Many residents stated that virtual interviews would be sufficient if they are able to complete away electives, as they would then be able to network with programs of interest and tour the hospital and city during elective time.

This study has a few limitations. Response rate for interviewers was impossible to calculate as responding schools defined and selected interviewers in different ways. The survey takes place in a Canadian setting and for the internal medicine residency group. We propose that our findings and their implications may apply to many other healthcare settings with similar pressures and training paradigms, including international and other health professional training contexts. However, surgical and technical skill-reliant programs may face different challenges with virtual interviews due to the inability and difficulties of assessing technical skills in a virtual setting, which should be addressed in future studies. We also note that an online survey has a participant selection bias that may have affected the results.

In conclusion, residents and PDs/interviewers reported similar strengths and weaknesses of virtual subspecialty medicine interviews and the majority in both groups would choose virtual interviews as opposed to in-person interviews in the future.