Experiences, perspectives, and advice for using virtual interviews in post-graduate trainee selection: a national survey of CCFP (EM) program directors.

OBJECTIVE: Due to the COVID pandemic, restrictions were put in place mandating that all residency interviews be transitioned to a virtual format. Canadian CCFP(EM) programs were among the first to embark on this universal virtual interview process for resident selection. Although there have been several recent publications suggesting best practice guidelines for virtual interviews in trainee selection, pragmatic experiences and opinions from Program Directors (PDs) are lacking. This study aimed to elicit the experiences and perspectives of CCFP(EM) PDs after being amongst the first to conduct universal virtual interviews in Canada.
METHODS: A 17-item online survey was created and distributed to all CCFP(EM) PDs (n = 17). It explored the virtual interview format employed, perceived advantages and disadvantages of a virtual configuration, confidence in determining a candidate's rank order, and PD preference for employing a virtual interview format in the future. It also elicited practical advice to conduct a smooth and successful virtual interview day.
RESULTS: The survey response rate was 76.5% (13/17). Nine respondents (69.2%) agreed that the virtual interview format enabled them to confidently determine a candidate's rank order. With respect to preference for future use of virtual interviews, 23.1% agreed, 38.5% disagreed and 38.5% neither agreed nor disagreed. Inductive thematic analysis of free text responses revealed themes related to virtual interview advantages (time, financial, and resource costs), disadvantages (difficulty promoting smaller programs, getting a 'feel' for candidates and assessing their interpersonal skills), and practical tips to facilitate virtual interview processes.
CONCLUSION: Once restrictions are lifted, cost-saving advantages must be weighed against suggested disadvantages such as showcasing program strengths and assessing interpersonal skills in choosing between traditional and virtual formats. Should virtual interviews become a routine part of resident selection, the advice suggested in this study may be considered to help optimize a successful virtual interview process.

Clinician’s capsule

Introduction

The COVID-19 pandemic has created several challenges for medical education including recruitment and selection of trainees. Traditionally, residency programs in Canada have conducted in-person interviews to aid in candidate selection. In May 2020, it was mandated that all program interviews be transitioned to virtual format [1]. Advantages to virtual interviews include improved convenience and flexibility and reduced financial and resource costs [2-5]. Posited disadvantages include potential technological difficulties, lack of personal interactions which may be useful for gleaning information about candidates’ interpersonal skills and professionalism, and a loss of opportunity for applicants to observe the facilities and culture of a program directly [3, 4].

In November 2020, family medicine/emergency medicine enhanced skills (CCFP(EM)) programs across Canada were among the first to embark on this universal virtual interview process for resident selection. There are many “best-practices” for conducting interviews which include utilizing structured, standardized, and blinded interview processes [6]. PDs place great weight on interviews during trainee selection [7]. In 2013, a working group produced a report of Best Practices in Application and Selection which included recommendations pertaining to candidate interviews [8]. This report emphasized the need for standardized and objective interview processes but did not suggest a need for interviews to occur-in person. Although there have been several recent publications suggesting theoretical best practice guidelines for virtual residency interviews [2-4], pragmatic experiences and opinions from Program Directors (PDs) on conducting such interviews are lacking.

The purpose of this study was to elicit the experiences and perspectives of CCFP(EM) PDs and propose experience-guided advice regarding virtual interview processes. Results of this study may assist both undergraduate and post-graduate training programs to implement and conduct future virtual interviews.

Methods

Design

This was a single cohort survey of CCFP(EM) PDs. This study received ethics exemption from the Ottawa Health Science Network Research Ethics Board.

Participants and recruitment

All 17 CCFP(EM) Program Directors received an email invitation in February 2021 to voluntarily participate in this anonymous survey. They were selected as they were amongst the first to conduct universal virtual interviews for resident selection in Canada and therefore be able to offer first-hand insights and experience.

Survey content and design

Survey methodology was chosen for the convenience and timeliness of survey distribution, cost effectiveness of data collection, and ease of respondent participation. The designed survey questions allowed the study team to gather data about the format and structure of virtual interviews used by each participant and how these differed from traditional in-person interviews, as well as the perceived challenges and advantages of conducting virtual interviews.

A 17-item electronic survey was created based on a review of existing literature of postulated best practices for virtual interviews and overall resident selection (Appendix 1). Recent virtual interview experiences of research team members were also considered in the development of the survey.

The survey was piloted with local enhanced skills PDs to solicit feedback regarding question clarity and readability.

Data analysis

Quantitative data were analyzed using descriptive statistics by calculating frequencies and percentages. Due to the small sample size, analysis of Likert-Scale responses was conducted by agreement rather than strength of the agreement. Positive indicators (strongly agree and agree) and negative indicators (strongly disagree and disagree) were collapsed. Inductive thematic analysis of qualitative data was conducted following the approach described by Braun and Clark [9]. Free text responses were reviewed by two study team members (JL and AN) and coded line-by-line. Codes were then collated and synthesized to identify themes. Of note, two authors (AN and JL) are CCFP(EM) PDs and the third author WJC) is highly involved in an FRCPC EM residency program.

Results

The survey was completed by 13 CCFP(EM) PDs (response rate of 76.5%).

Most interview members connected to the platforms individually from different locations (69.2%), citing local COVID restrictions as the reason for this format. Four (33.3%) programs conducted more interviews than in previous years. Two (15.4%) programs modified their traditional interview format by adding clinical knowledge stations to better discriminate between candidates as these could not be directly assessed due to cancellation of visiting electives.

Most respondents indicated that their interview day ran smoothly with minimal technological interruptions (92.3%). One respondent (7.7%) indicated that interviewees experienced internet connectivity issues, while two programs (15.4%) reported internet problems for interviewers.

Nine respondents (69.2%) agreed that the virtual interview format enabled them to confidently determine a candidate’s rank order. With respect to preference for future virtual interviews, responses were mixed. Three respondents agreed (23.1%), five respondents disagreed (38.5%) and five respondents neither agreed nor disagreed (38.5%).

Inductive thematic analysis of free text responses revealed several themes and are summarized in Appendix 2. Themes included: (1) advantages and disadvantages to conducting virtual interviews; (2) technological considerations; and (3) experience-based advice to facilitate the virtual interview process. Experience-based advice elicited can be found in Table 1.

Table 1

Experience-based recommendations to conduct successful virtual interviews

Do a Practice Run Conducting a trial run in advance to ensure all interviewers are trained and familiar with the technology being used is strongly recommended. Familiarity with technology through advanced practice can help programs appear more professional during virtual interviews which can have a positive impact on candidates’ perceptions of a program.

Have a Backup Plan Given the high stakes nature of residency interviews coupled with the complete reliance on internet connectivity, a backup plan is imperative. Obtaining cell phone numbers from interviewers, candidates, and support team members ahead of time is advisable.

Be Careful with Time Zones Consider potential time-zone differences for candidates when creating the interview schedule and include reminders to candidates in any pre-interview communications.

Have a Support Team Ensure adequate IT and/or administrative support both prior to and during the actual interview day(s). Virtual interviews take more time to plan and prepare ahead of time than in-person interviews. During the virtual interviews, having adequate support will help to ensure candidates are moved appropriately through virtual rooms and can help troubleshoot any technological issues on the day. Having more than one support staff present is advisable so if one individual is troubleshooting an issue, there is an alternate on standby to address any additional challenges.

Group Interviewers Together Have groups of interviewers in the same geographical location when possible, respecting local guidelines on gatherings. This may help to facilitate post-interview discussion and deliberation regarding candidate ranking.

Advantages to virtual interviews included convenience, decreased costs, and improved time management on the actual interview day. Disadvantages included difficulty promoting smaller and more remote programs, difficulty getting a “feel” for candidates in an artificial environment, potential for technological disruptions, and more work to organize. One respondent stated, “…there is something profoundly lost when interviewing virtually. The nonverbal nuances are not there to make meaningful connections”. With respect to the challenge of showcasing programs and facilities, one respondent wrote, “it’s the people that make smaller programs worth coming to. If you can’t make those connections in person, applicants aren’t going to rank your program highly”.

Discussion

Interpretation and previous studies

This is the first Canadian study looking at the experiences of PDs in conducting mandatory virtual interviews for post-graduate trainee selection in a CaRMS match.

The advantages to a virtual interview format reported in this survey included improved time management on interview day and convenience for both interviewers and candidates. Numerous respondents commented on financial, temporal, and environmental cost savings associated with conducting interviews virtually. Both of these findings are supported in the literature [5, 10]. One recent study of Canadian general surgery applicants found that the mean travel costs for in-person interviews was $4866 and 1.82 tonnes of CO2 emissions per applicant [5].

The disadvantages of virtual interviews reported in our survey are also consistent with the literature. Some respondents indicated that they found virtual interviews to be cold and artificial. Another theme expressed by participants was the loss of meaningful personal connection and the inability to get a good “read” or “feel” for candidates [3]. This concept is supported by a recent study where PDs indicated that they found it more difficult to establish rapport with candidates in a virtual interview format [11]. Losing this personal interaction could detract from gaining important insights into important interpersonal skills and professional behaviour of candidates [4].

Our survey respondents made numerous comments about the challenges of showcasing programs and facilities through a virtual interview format, especially for smaller and more remote programs. This aligns with previous literature which has reported concerns of applicants not getting a “feel” for a site [11] as well as difficulty showcasing the camaraderie amongst trainees and faculty, factors shown to be highly valued by EM applicants [3, 12, 13]. In one recent survey of CCFP(EM) applicants, collegiality between faculty and residents was reported as being highly influential in first-choice program selection [14]. Novel recruitment strategies such as video-tours and promotional videos have been recommended to navigate these challenges [2], but it is unclear if these are as effective as in-person interactions.

In our study, most respondents indicated that they were able to confidently determine a candidate’s rank order. One study comparing a web-based to traditional in-person interview process found that faculty members felt less comfortable ranking candidates who were interviewed using a web-based platform compared to those interviewed in-person [10]. It is unclear if the lower confidence and comfort in determining rank-order by PDs is due to inherent differences in interview formats or related to an uneasiness with a novel process. Perhaps more experience with virtual processes will bolster confidence in ranking candidates. This is an area that would benefit from further exploration.

Implications

Survey respondents offered a lot of advice for conducting virtual interviews based on their recent experiences which is summarized in Table 1.

With respect to the use of future virtual interviews, responses in this survey were heterogeneous. Only three (23.1%) respondents indicated that they would prefer virtual interviews even if in-person interviews were permitted. This is less than reported in another study which found 55% of PDs agreed that future interviews should be held virtually, regardless of pandemic restrictions [15]. Some studies have suggested that virtual interviews should be considered as an adjunct instead of a replacement to traditional methods [10, 15]. This could lead to potential bias as the effect of in-person vs virtual interviews on candidate selection and rank order has not been fully elucidated. However, one small single-centre study suggested that interview type did not impact the likelihood of a prospective candidate being ranked or matched [16].

Strengths and limitations

While our survey had a strong response rate, the major limitations of this study are those inherent to any survey design including recall and response bias. Additionally, there was limited ability to explore participants’ responses in greater depth. Furthermore, this study surveyed PDs from a single subspeciality, which may limit the generalizability and transferability of results.

Future directions

Future directions include further exploration regarding differences in assessment of interpersonal skills through virtual interviews compared to traditional in-person methods, assessment of PD satisfaction of match results following use of a universal virtual interview process, and evaluation of advantages and disadvantages of a hybrid model of virtual and in-person interview methods.

Conclusions

Once restrictions mandating virtual interviews are lifted, advantages and disadvantages must be weighed in choosing an appropriate way forward. CCFP(EM) PDs are divided on their preference in conducting future virtual interviews. Advice suggested in this study may be considered by future interviewers to facilitate a successful virtual interview process.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 26 kb)

What is known about the topic?

Perspectives, experiences, and advice from Canadian Program Directors (PDs) about conducting virtual resident interviews are lacking.

What did this study ask?

What were the experiences and perspectives of CCFP(EM) PDs after being amongst the first to conduct universal virtual interviews in Canada?

What did this study find?

Promoting smaller programs and judging interpersonal skills through virtual interviews present challenges. Experience-guided advice is also suggested.

Why does this study matter to clinicians?

Advantages and disadvantages must be considered in choosing between formats. Virtual interviews may be informed by experience-guided advice suggested.