Survey of surgical critical care applicant and program director views on virtual interviews for fellowship training: a Surgical Critical Care Program Directors Society sponsored study.

Background: The COVID-19 pandemic forced postgraduate interview processes to move to a virtual platform. There are no studies on the opinions of faculty and applicants regarding this format. The aim of this study was to assess the opinions of surgical critical care (SCC) applicants and program directors regarding the virtual versus in-person interview process.
Methods: An anonymous survey of the SCC Program Director's Society members and applicants to the 2019 (in-person) and 2020 (virtual) interview cycles was done. Demographic data and Likert scale based responses were collected using Research Electronic Data Capture.
Results: Fellowship and program director responses rates were 25% (137/550) and 58% (83/143), respectively. Applicants in the 2020 application cycle attended more interviews. The majority of applicants (57%) and program faculty (67%) strongly liked/liked the virtual interview format but felt an in-person format allows better assessment of the curriculum and culture of the program. Both groups felt that an in-person format allows applicants and faculty to establish rapport better. Only 9% and 16% of SCC program directors wanted a purely virtual or purely in-person interview process, respectively. Applicants were nearly evenly split between preferring a purely in-person versus virtual interviews in the future. Discussion: The virtual interview format allows applicants and program directors to screen a larger number of programs and applications. However, the virtual format is less useful than an in-person interview format for describing unique aspects of a training program and for allowing faculty and applicants to establish rapport. Future strategies using both formats may be optimal, but such an approach requires further study. Level of evidence: Epidemiologic level IV. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Virtual interviews are commonly done in the business sector, but there is little information on their utility for recruitment of medical trainees.

This study found that the interpersonal aspects of the interview process are lost in the virtual environment, but the virtual setting allows for better initial screening of training programs and trainees by each party.

This study suggests that a two-phase process using the virtual setting to screen candidates and programs followed by an in-person setting for selected candidates and programs may offer the most benefit to all parties.

Introduction

On March 11, 2020, the WHO officially classified COVID-19 as a global pandemic.1 Resultant institutional travel bans rendered in-person interviews for postgraduate training positions impossible. Therefore, institutions were forced to move to a virtual interview format for the 2020 application cycle.2–6 The sudden and complete transition from an in-person to virtual interview platform provided a unique opportunity to compare these formats.

Even prior to the pandemic, there was interest in use of virtual interviews for postgraduate interviews given the cost and time burden associated with in-person interviews.7 Now there is debate regarding whether to continue with virtual interviews, resort back to in-person interviews or offer a hybrid model once the pandemic has subsided. The purpose of this project was to inform this debate by surveying surgical critical care (SCC) fellowship applicants and program directors regarding their in-person versus virtual interview experiences. We hypothesize that a hybrid model would be preferred by both groups.

Methods

After obtaining IRB approval from the University of California at San Francisco Fresno (approval number 2021006), the Surgical Critical Care Program Directors Society (SCCPDS) sent an anonymous online survey invitation to 2019 and 2020 SCC fellowship applicants and also to program directors who are members of the SCCPDS. The survey was developed by the authors but was not validated prior to the study’s start. The SCCPDS consists of over 100 programs and their directors in the USA. One of its services is coordination of the SCC fellowship interview process.

Invitations to participate were sent from February 26, 2021 to April 26, 2021. Applicants to the 2019 (in-person interview) and 2020 (virtual interview) application years and current members of SCCPDS were contacted by email. The survey was administered using the Research Electronic Data Capture application, and responses were tabulated using Microsoft Excel. Because the questions asked of applicants were different than those asked of faculty, we could not statistically analyze responses comparing the two groups. Within each cohort, statistical significance was assessed using the Mann-Whitney U test for continuous variables, and significance was set at p<0.05.

Fellowship applicants were asked about sex, age, the number of SCC programs to which they applied, the number of interview offers received and the number of programs ranked. Each applicant was then asked to evaluate 12 statements on a Likert scale of 1–5 with one being strongly agree and five being strongly disagree (online supplemental questionnaire). Applicants were asked about their experiences regarding cost, time for travel, ability to get a personal feel for the program/hospital/city and the impact that the interview had on their final decision. Applicants for 2020 were also asked to rank their virtual experience as compared with their previous experiences with in-person interviews.

SCC program directors ranked their views using a similar Likert scale of 1–5 with one being strongly agree and five being strongly disagree (online supplemental questionnaire). Questions assessed the number of applications they received, number of interviews offered, their ability to present their program and their ability to assess the applicants using a virtual format versus an in-person format. They were also asked to comment on their views regarding continued use of a virtual-only format, a hybrid model consisting of both in-person and virtual interviews or reverting to solely in-person interviews in the future.

Results

Applicant demographic and responses

A total of 137 SCC fellowship applicant responses were included out of 550 contacted for the survey (25% response rate). Response rates by year were 18% (46/251) for 2019 and 30% (91/299) for 2020. The average age of respondents was 34±3.7 years old, and 63% (n=86) were male (table 1).

Table 1

Fellowship applicant demographics and interview format (n=137)

Variable	2020 application year, n=91	2019 application year, n=46	P value
Male n (%)	56 (62%)	30 (65%)
Median age (25th, 75th IQR), years	33 (31, 35)	34 (33, 36)	0.011
Median number of programs applied to (25th, 75th IQR)	25 (18, 40)	18 (8, 25)	0.003
Median number of interviews offered (25th, 75th IQR)	15 (18, 40)	12 (8, 25)	0.135
Median number interviews attended (25th, 75th IQR)	13 (8, 16)	11 (5, 15)	0.006
Type of interview offered
In-person only	2	43	<0.0001
Virtual only	86	2	<0.0001
Hybrid	3	1	0.70
Interview agenda (virtual only) (n, %)
Group interview with PD	41 (45)	1 (2)	<0.0001
Group interview and separate interview with PD	46 (51)	0
Individual interviews	85 (93)	1 (2)	<0.0001

PD, program director.

Overall, applicants in the 2020 (virtual) application cycle applied to more programs, attended more interviews and ranked more programs than those in the 2019 cycle (table 1). In 2020, applicants applied to an average of 30±22 programs as compared with an average of 21±17 programs in 2019 (p=0.008). There was no difference in the average number of interviews offered to each applicant, but applicants in the 2020 cycle accepted significantly more interviews (13±7 vs 10±6, p=0.014). The number of programs ranked was also significantly higher in the 2020 application cycle (12±6 vs 10±5, p=0.028).

In analyzing the results based on in-person versus virtual interview, 45 applicants (43 applicants from 2019 and 2 applicants from 2020) had an in-person interview only, 88 applicants (2 applicants from 2019 and 86 applicants from 2020) had a virtual interview only, and four applicants (one applicant from 2019 and three applicants from 2020) had a hybrid interview consisting of both in-person and virtual elements. Of applicants who had a virtual interview only (n=88), 31% (n=27) had group interviews with multiple attendings at once including the program director, 34% (n=30) had group interviews with multiple attendings and a separate interview with the program director and 63% (n=55) had separate individual interviews with attendings and with the program director and 91% (n=80) were offered a virtual meeting with current fellows and residents. Sixty-nine percent (n=61) had a virtual tour of the intensive care unit (ICU) and/or the hospital.

Applicant responses were mixed as to whether the cost of travel limited the number of interviews accepted (table 2). The median response was 3 (neutral) with 32% either strongly agreeing or agreeing compared with 47% strongly disagreeing or disagreeing that the cost of travel limited the number of interviews that the applicant could accept. The distribution was similar regarding the difficulty of scheduling time away from residency to attend in-person interviews, with the median response again being 3 (neutral) and 46% either strongly agreeing or agreeing as compared with 46% either strongly disagreeing or disagreeing that this was an obstacle. Applicants in the 2020 virtual interview cycle agreed (median response was 2) that they accepted more interview offers than they would have if they had to travel (55% strongly agree or agree vs 31% strongly disagree or disagree).

Table 2

Fellowship applicant responses (n=137)

Question	1 (strongly agree) (%)	2 (%)	3 (%)	4 (%)	5 (strongly disagree) (%)	Median (25th, 75th IQR)
In-person interview (n=46)
The cost of travel for interviews limited the number of interviews I accepted	17.02	14.89	21.28	25.53	21.28	3 (2, 4)
The difficulties of scheduling time away from residency responsibilities limited the number of interviews that I accepted	23.91	21.74	8.70	19.57	26.09	3 (2, 4.75)
An in-person interview helped me to get a much better sense of the program than I would have otherwise	53.19	27.66	12.77	4.26	2.13	1 (1, 2)
Visiting the program was critical to seeing the hospital and educational environment	43.48	30.43	15.22	8.70	2.17	2 (1, 2.75)
As a result of the in-person interview, I ranked this program more highly than I would have without a visit	44.68	23.40	21.28	8.51	2.13	2 (1, 3)
Virtual interview (n=91)
I applied to more SCC fellowship programs than I would have if all interviews were in-person	25.00	14.13	16.30	15.22	29.35	3 (1.75, 5)
I accepted more SCC fellowship interview offers than I would have if all interviews were in-person	41.30	14.13	13.04	8.70	22.83	2 (1, 4)
I was able to get a good feel for the program via the virtual interview process and any other research I did for each program	5.43	20.65	34.78	35.87	3.26	3 (2, 4)
An in-person interview would have allowed me to gain a better understanding of the educational environment and offering of each program	21.74	43.48	18.48	13.04	3.26	2 (2, 3)
An in-person interview would have allowed me to gain a better understanding of the comradery and culture of each program	44.57	34.78	7.61	7.61	5.43	2 (1, 2)
I liked the virtual interview process overall	18.48	32.61	33.70	11.96	3.26	2 (2, 3)
As compared with the previous in-person interviews I have had (eg, residency), I would prefer having a virtual interview in the future rather than an in-person interview	14.29	21.98	24.18	23.08	16.48	3 (2, 4)

SCC, surgical critical care.

In comparing in-person interview to virtual interviews (table 2), 2019 applicants strongly indicated that the in-person interview allowed them to get a much better sense of the program, with the median response being 1 (strongly agree). Eighty-one percent strongly agreed/agreed with this statement. This was compared with a neutral median response for applicants in the 2020 cycle, only 26% of whom strongly agreed/agreed that the virtual interview allowed them to get a good sense of the program. Notably, 40% of this cohort strongly disagreed/disagreed that the virtual interview format allowed them to get a good sense of the program. In-person applicants also felt seeing the hospital and assessing the educational environment was critical with a median response 2 (agree) and 74% of respondents strongly agreeing or agreeing with this statement. When asked to compare how the virtual interview process impacted their understanding of the educational environment and culture of the institution with their previous experiences using in-person interviews, the 2020 applicants strongly preferred an in-person experience to a virtual experience. Sixty-five percent strongly agreed/agreed that the in-person format allowed them to gain a better understanding of the educational environment (median response 2) and 80% strongly agreed/agreed that the in-person interview format allowed them to better gauge the culture of the institution (median response 2). Similarly, 68% of the 2019 applicants strongly agreed/agreed that the in-person experience caused them to rank programs higher than they otherwise would have without a visit.

Program director/assistant director demographic and responses

Eighty-three out of 143 (58%) invited SCCPDS members filled out the survey. Prior to the 2020 application cycle, 94% of interviews were conducted exclusively in-person. In 2020, 84% of programs used separate virtual interviews with faculty, 9% had group virtual interviews with multiple attendings at once and then a separate session with the program director, and 4% had a group virtual interview with multiple attendings including the program director. Forty-four percent of programs offered a virtual tour of the hospital and ICU, 3% offered a tour of the ICU only and 46% did not offer a virtual tour at all.

As with the applicant responses, program directors/assistant directors also did not feel that technological limitations and glitches hindered the interview process (table 3). The general trend of responses favored in-person interviews as compared with virtual. The majority (63%) of respondents felt in-person interviews offered an advantage to the candidate to present themselves and develop rapport with the interviewer, with only 9% strongly disagreeing/disagreeing with this statement. Fifty-seven percent of directors strongly agreed/agreed that they were able to adequately assess the candidates in a virtual format, but 43% were neutral or disagreed with this statement. Interestingly, no respondent strongly disagreed with this statement. Respondents also felt in-person interviews offered a better way to describe the learning environment compared with virtual interviews, with 70% strongly agreeing/agreeing and only 15% strongly disagreeing/disagreeing with this statement. Only a slight majority (52%) felt that the virtual format allowed them to adequately present their program to the applicants, while 15% of the faculty strongly disagreed/disagreed with this statement.

Table 3

Surgical critical care program director responses (n=81)

	1 (strongly agree) (%)	2 (%)	3 (%)	4 (%)	5 (strongly disagree) (%)	Median (25th, 75th IQR)
Technology limitations and glitches made the interview process challenging	2.50	15.00	21.25	46.25	15.00	4 (3, 4)
Our program received more applications in 2020 as compared with 2019	57.50	22.50	12.50	6.25	1.25	1 (1, 2)
Our program interviewed more applicants in 2020 as compared with 2019	50.00	18.75	18.75	10.00	2.50	1.5 (1, 3)
As compared with virtual interviews, in-person interviews offer an advantage to the candidate to present themselves and develop rapport with the interviewer	26.25	36.25	28.75	6.25	2.50	2 (1, 3)
As compared with virtual interviews, in-person interviews offer an advantage to the program to present the learning environment	38.27	32.10	14.81	9.88	4.94	2 (1, 3)
We were able to provide a good representation of our fellowship program in a virtual format	14.81	37.04	33.33	13.58	1.23	2 (2, 3)
We were able to adequately interact and assess the candidates in a virtual format	9.88	46.91	27.16	16.05	0.00	2 (2, 3)
I liked the virtual interview process	20.99	45.68	16.05	14.81	2.47	2 (2, 3)
	Virtual (%)	In-person (%)	Hybrid (%)
When the COVID-19 pandemic goes away, I recommend that interviews be:	8.64	16.05	75.31

Regarding the number of applications submitted, 39% of applicants strongly agreed/agreed compared with 45% who strongly disagreed/disagreed that they applied to more SCC fellowship programs than if interviews had been in-person. The faculty leaders, however, stated more confidently that they received more applications in 2020 as compared with 2019 with 80% strongly agreeing/agreeing and only 8% strongly disagreeing/disagreeing with this statement. Eighty-three percent of applicants strongly agreed/agreed that they accepted more interview offers due to the virtual format and 69% of faculty concurred that more fellowship applicants accepted interview offers in 2020. This compared with 32% and 13% in the applicant and faculty cohorts, respectively, who strongly disagreed/disagreed with these statements (tables 2 and 3).

Lastly, in terms of each group’s overall evaluation of the in-person versus virtual experience, a slight majority (51%) of the 2020 applicants strongly liked or liked the virtual interview format as compared with 16% who strongly disliked/disliked it. Similarly, 67% of faculty respondents liked the overall virtual interview experience as compared with 17% who did not. The 2020 respondents were nearly equally divided on whether they would want to interview in-person or virtually in the future with approximately 40% strongly agreeing/agreeing or strongly disagreeing/disagreeing with this statement. Seventy-five percent of the SCCPDS respondents preferred a hybrid interview process consisting of both virtual and an in-person components in the future. Only 9% wanted purely virtual and 16% wanted purely in-person formats.

Discussion

The purpose of this study was to evaluate the opinions of SCC fellowship applicants and program leaders regarding the benefits and shortcomings of a virtual as compared with in-person fellowship interview process in order to inform the debate as to what future interview processes should be once the present pandemic has subsided. Whereas the virtual format has advantages in terms of cost savings, ability to reach a broader audience and ease of scheduling, it also has potential shortcomings, including limited opportunity to demonstrate intangible aspects that may allow one to value one program over others.

Prior to the pandemic, virtual interviews in the medical community were considered due to the high financial costs related to travel and lodging as well as the difficulty with applicants taking time away from work. Recent work from the cardiothoracic fellowship cycle estimated the cost of an in-person interview to be $600 per interview and that fourth year (PGY-4) general surgery residents spent 15% of their academic year away from their program during the interview process.6 Collectively, the total cost of interviewing for postgraduate surgical training can be as high as $6000.8 Given residency staffing issues, in particular, many applicants are often forced to use vacation time and/or limit the number of fellowship interviews they accept.7 9 However, a host of perceived disadvantages of the virtual format, such as the inability to meet faculty in-person, fully assess the program and city and adequately present oneself to the program, made use of a virtual format very rare prior to the COVID-19 pandemic.10 11

Our study found that the views of the fellowship applicants and the program leadership were mostly congruent with one another. The majority of applicants (57%) and program faculty (67%) strongly liked/liked the virtual interview format. However, both cohorts felt that the in-person format offered significant advantage in terms of allowing the applicant to understand the culture of the institution and in terms of allowing the applicant and faculty to establish rapport. The lack of in-person interactions has been well described as a major disadvantage in an entirely virtual format.11 12

Interestingly, the applicants’ perception of the impact that the virtual interview process had on their application is not congruent with actual statistics regarding the match. Applicants felt that savings related to travel and cost did not impact on their decision regarding number of programs to which they applied, although they stated that it did impact on the number of interviews they accepted. Yet, data on the 2021 SCC Match showed a 5% decrease in the number of applicants but a 10% increase in the total number of applications received by each program and a 6% increase in the average number of applicants per program (W Chiu, MD, personal communication 2022). Based on responses received, it is likely that a portion of this growth in applications per site is related to the decreased cost and time associated with the virtual interview process.8 Factoring in the faculties’ opinions favoring a hybrid interview process moving forward, one could suggest a two-phase interview process using a virtual format as a first step followed by in-person interview for a select cohort. This would allow applicants to screen a wide number of programs while also allowing those who both parties feel may be a good match to reap the benefits of an in-person encounter, but such an approach would also increase the overall work and time required for faculty.

Moving forward, only 9% and 16% of SCC program directors wanted a purely virtual or purely in-person interview process, respectively. A large majority (75%) preferred a hybrid model. Applicants were nearly evenly split between preferring a purely in-person versus virtual interviews in the future.

This study has limitations. Survey studies are subject to recall bias. The response rate was also limited. The study did not assess the specific costs associated with travel, time-off from residency training or other specific burdens placed on applicants in 2019. The questionnaire that was used was not validated in a focus group prior to dissemination to participants.

Conclusion

This study demonstrates that interpersonal aspects of the interview process are met better using an in-person interview format, but both applicants and faculty also acknowledge some beneficial aspects to the virtual setting. Future strategies using both formats may maximize benefit, but such an approach requires further study.