Virtual Interviews: Challenges and Opportunities for Pulmonary Disease and Critical Care Medicine Fellowship Programs.

The coronavirus disease (COVID-19) pandemic brought profound change to the medical education system, and residency and fellowship recruitment was not spared. Many of the activities required for recruitment of new fellows (e.g., airline travel and face-to-face meetings) were not able to be safely done. The rapid shift to all-virtual interviewing brought logistical challenges but, as the season concluded, called into question the value and validity of prior protocols. Our institutions (University Hospitals Cleveland Medical Center and MetroHealth Medical Center in Cleveland, Ohio) designed surveys to collect both applicants' and interviewers' perspectives on the virtual interview process for the 2020-2021 recruitment season to identify the challenges virtual interviews may bring to the current paradigm and what that may mean for the value of the traditional in-person model. Our results show that the absence of certain aspects of in-person interviews (e.g., travel costs and time required off-service) were welcome changes to both applicants and interviewers. However, there were new challenges identified, such as lack of formal training for virtual interviews and a shift in applicants' attention to fellowship program websites. We discuss how these observations could inform best practices for programs and applicants in the future.

In an unprecedented time of change and new standards, even medical fellowship interviews were challenged to adapt to the restrictions imposed as a result of the coronavirus disease (COVID-19) pandemic. For the academic year 2020–2021, both the Association of American Medical Colleges and the Alliance for Academic Internal Medicine strongly encouraged medical school and teaching hospital faculty to conduct all interviews with potential students, residents, fellows, and faculty in a virtual setting via phone or video conferencing (1). More recently, the Alliance for Academic Internal Medicine released recommendations for the 2021–2022 internal medicine (IM) fellowship application cycle in response to the continued challenges of the COVID-19 pandemic (2). This includes a strong recommendation for conducting virtual interviews for all applicants and a strong recommendation against open houses or in-person single/group visits. Although the importance of interview day has been studied before (3), there is little published data on the experience of applicants and interviewers with virtual interviewing for fellowship (4–6).

The future of the pandemic continues to depend on many unknowns. Although vaccine distribution has brought a degree of normalcy back to life in our society, concerns for the safety of travel and in-person group meetings remain. As such, both programs and applicants must prepare to conduct and participate in interviews virtually.

We decided to gather information from applicants and interviewers regarding their virtual interviewing experience during the 2020 interview cycle for pulmonary disease and critical care medicine (PCCM) fellowship. Our observations were gathered and perspectives formed in the postpandemic period. There was a clear need to understand the advantages and disadvantages of virtual interviews, as programs and applicants will continue participating in this format for at least the 2021–2022 season and possibly beyond.

What We Did

Our research team developed two separate questionnaires: one for applicants (applicants’ questionnaire [AQ]) and one for interviewers (interviewers’ questionnaire [IQ]). The questionnaires and electronic survey format were developed by the research team, starting with a review of previously published literature on the residency/fellowship interview process as an initial framework (3–6). We conducted in-depth interviews and focus groups with members of the most recent applicant pool (first-year PCCM fellows), potential future applicants (third-year IM residents), and interviewers (PCCM faculty) to identify the domains to be surveyed.

For the AQ, we conducted in-depth interviews and focus groups with first-year PCCM fellows who interviewed for positions in the 2019–2020 cycle and third-year IM residents who were preparing their applications to PCCM fellowship for the 2020–2021 cycle. We identified the following domains: IM residency characteristics, advantages of virtual interviews, disadvantages of virtual interviews, programs’ efforts during virtual interviews, and overall experience and future recommendation. For the IQ, we conducted in-depth interviews and focus groups with our PCCM faculty. We identified the following domains: advantages of virtual interviews, disadvantages of virtual interviews, programs’ efforts during virtual interviews, and overall experience and future recommendation. For both questionnaires, questions (a combination of multiple-choice questions and five-point Likert scale items) were formulated within the domains. The focus groups were asked to evaluate the relative merit of the questions by providing a binary response (include/exclude) with an option for comments. Edits and suggestions were evaluated and incorporated.

Participants for the AQ included all applicants to the PCCM fellowship programs at either Case Western Reserve University/University Hospitals Cleveland Medical Center (492 total applicants) or Case Western Reserve University/MetroHealth Medical Center (452 total applicants) for the 2021 appointment year. The total number of unique applicants to both programs was calculated at 540. Participants for the IQ included the 2 chief fellows and 20 faculty members who conducted interviews for the aforementioned fellowship programs for the 2021 appointment year.

The questionnaires were converted to electronic format, and responses were anonymously collected and stored using REDCap electronic data capture tools hosted at Case Western Reserve University (7, 8). Responses to the AQ were collected between November 25, 2020, and December 2, 2020. These dates were chosen to ensure data collection occurred after participants had submitted their rank-order lists to the National Residency Matching Program but before the match results were available. The IQ was sent to the eligible fellows/faculty, with responses collected starting December 3, 2020, and ending December 13, 2020, after all potential participants completed the survey.

The analysis of the collected responses included descriptive statistics for each questionnaire item. To assess representativeness of the sample population of respondents to the AQ, the collected demographic information (program size and geographic region) was compared with publicly available statistics provided by the Accreditation Council for Graduate Medical Education . For the questions assessing positive/negative factors regarding virtual interviewing, the responses were averaged and then ranked from least to most important in addition to computation of 95% confidence intervals (CIs). We also performed a subgroup analysis for participants in the AQ who selected a future preference for either “all virtual” or “all in-person” interviews, with unpaired t tests performed on each survey item to identify significant between-group differences (reported separately in the data supplement).

Applicants’ Perspective

Eighty-seven (16.1%) participants filled out the AQ. The majority of participants were from IM residency programs in the Northeast (39%) and the Midwest (33%). Forty-nine of the participants (57%) had not received any formal training on virtual interviewing (Table 1). Forty-nine participants (57%) agreed that virtual interviewing provided the chance to adequately represent themselves, and 53 participants (61%) agreed that their questions were answered during virtual interview day. Forty-three applicants (49%) believed that virtual interviewing would hurt their chances of getting into their top programs (Table 2). Fifty-one participants (59%), if given the chance in the future, would favor a hybrid model, whereas 21 participants (24%) would favor an all in-person model, and 15 (17%) participants would favor an all-virtual model (Figure 1). In terms of advantages of virtual interviewing, the two highest scoring factors were saving on cost (4.1) and not having to arrange for clinical coverage (4), with 95% CIs of 3.9–4.4 and 3.8–4.3, respectively. In terms of disadvantages of virtual interviewing, the highest scoring factor was not being able to tour the city (3.9; 95% CI, 3.7–4.2). In terms of program preparation, the highest scoring factor was program website (4.4; 95% CI, 4.2–4.6) (Figure 2).

Table 1.

Demographics and characteristics of participants

Applicants (N = 87)	n (%)
Geographic region of residency program
Northeast	34 (39%)
Midwest	29 (33%)
Southeast	14 (16%)
West	7 (8%)
Southwest	3 (3.4%)
Number of categorical residents in residency program
45 residents or less	41 (47%)
46–74 residents	21 (24%)
75 residents or more	22 (25%)
Unsure	3 (3.4%)
Number of virtual interviews the applicant participated in
Between 1 and 5	29 (33%)
Between 6 and 10	21 (24%)
Between 11 and 15	20 (23%)
Between 16 and 20	11 (13%)
More than 20	6 (7%)
Amount of training on virtual interviewing received
None	49 (57%)
Limited (under 30 min)	23 (27%)
Extensive (30 min or more)	14 (16%)
Interviewers (N = 22)
Numbers of years of interview experience
First year	4 (18%)
Less than 3 yr	2 (9%)
3–5 yr	4 (18%)
More than 5 yr	12 (55%)
Number of applicants the interviewers virtually interviewed
1–10	5 (23%)
11–20	9 (41%)
21–30	6 (27%)
31–40	0 (0%)
More than 40	2 (9%)

Table 2.

Applicants’ and interviewers’ opinions

Applicants (N = 87)	n (%)
Did you feel virtual interviews gave the chance to adequately present yourself?
Strongly disagree	6 (7%)
Disagree	10 (12%)
Neither agree nor disagree	19 (22%)
Agree	49 (56%)
Strongly agree	3 (3%)
In general, did you feel that your questions were answered during your interviews?
Strongly disagree	2 (2%)
Disagree	4 (5%)
Neither agree nor disagree	14 (16%)
Agree	53 (61%)
Strongly agree	14 (16%)
Do you believe virtual interviews will hurt or help your chances of getting into your top choice programs?
Strongly hurt	12 (14%)
Slightly hurt	43 (49%)
Have no effect	26 (30%)
Slightly help	5 (6%)
Strongly help	1 (1%)
Interviewers (N = 22)
Virtual interviews gave me the chance to adequately represent myself
Strongly disagree	0 (0%)
Disagree	1 (4%)
Neither agree nor disagree	3 (14%)
Agree	14 (64%)
Strongly agree	4 (18%)
Virtual interviews gave me the chance to adequately represent my program
Strongly disagree	1 (4%)
Disagree	4 (18%)
Neither agree nor disagree	2 (9%)
Agree	12 (55%)
Strongly agree	3 (13%)

Figure 1.

Preference for in-person, virtual, or hybrid interview model.

Figure 2.

Ranked factors of importance.

A subgroup analysis was completed comparing the responses of applicants who chose that they would prefer an all-in-person (n = 21) or all-virtual (n = 15) interview model over a hybrid model (Table E1 in the data supplement). Applicants who preferred all-in-person interviews were most likely to come from residency programs in the Midwest or Southeast, whereas those who preferred all-virtual interviews were from programs in the Northeast. Not surprisingly, those who preferred all-virtual interviews were more likely to cite decreased travel costs and time required off-service as positive aspects of the virtual interview experience. Of the negative aspects of the virtual interview experience queried, not meeting faculty or fellows in person had the biggest impact for applicants who preferred an all-in-person model.

Finally, most applicants (57%) reported having received no training on virtual interviewing and only 16% having received “extensive training” defined as more than 30 minutes.

Interviewers’ Perspective

Twenty-two (100%) participants filled out the IQ. Twelve (55%) interviewers had more than 5 years of experience interviewing (Table 1). Fourteen of our interviewers (64%) agreed they were able to present themselves adequately, and 12 interviewers (55%) agreed that virtual interviews allowed them to present their programs adequately (Table 2). The majority (64%) favored considering a hybrid interviewing model in the future, whereas a preference for strictly virtual interviews or in-person interviews was evenly split at 18% of faculty favoring each (Figure 1). Location flexibility was the highest scoring factor (4.2; 95% CI, 3.8–4.7), whereas the capacity to judge an applicant’s interpersonal skills scored the lowest (3.2; 95% CI, 2.8–3.7) (Figure 2).

Limitations

The collected data are not without limitations. This survey was limited to two fellowship programs in Northeast Ohio. However, the geographical distribution of the applicants’ IM residencies and the average number of residents in these programs mimics the general U.S. distribution of IM residency (detailed in the data supplement), suggesting that the results may be applicable to the pool of PCCM applicants. The response rate was 16.1% for the applicants. We suspect the low response rate to be due, in part, to the pandemic strain on the participants’ time and effort but also to the increasing number of electronic surveys and survey fatigue. Given the short interview season, multiple survey reminders were not possible. Our questionnaires were kept purposefully short to encourage responses. Therefore, our survey did not address the sex difference in answers or if the experience would be different for underrepresented minorities. Finally, our survey did not address technical issues and their impact on participants’ and interviewers’ experience with virtual interviews

Future Directions

Virtual interviewing, imposed by the COVID-19 pandemic, has created a paradigm shift in fellowship recruitment. The majority of our applicants and interviewers felt that the virtual interviewing process allowed them to adequately represent themselves despite the limitations of a web-camera only. The applicants overall felt their questions were answered during their interview as well. Saving on the cost of travel and not having to arrange for clinical coverage were the most important factor for applicants. The flexibility to interview from different locations was the most important factor for the interviewers.

For programs who will continue to conduct virtual interviews as part of their recruitment process, our study highlighted a few important factors. Applicants were most interested in a program’s website, and dedicating the time to updating a program’s website may help attract more applicants. A virtual tour of the program’s facilities was another important factor and was weighed more favorably than a program’s presence on social media. Applicants reported that having the opportunity to visit a city and meet fellows in person were more important than meeting faculty in person.

Furthermore, our survey highlighted that most applicants have not received any formal training on virtual interviewing during their residency. Offering training and “best practices” for virtual interviewing has the potential to improve applicants’ experience.

Although the majority of applicants felt that virtual interviewing allowed them to adequately represent themselves, the majority of applicants also felt that the virtual model interview would hurt their chances of matching into their top choice programs. The reason behind this discrepancy was not evaluated in our survey. One possible explanation is the anticipated increase in applicant pool given the convenience of virtual interviews; however, more evaluation is needed. Furthermore, interviewers did not have great confidence in their capacity to assess applicants’ interpersonal skills in an all-virtual format.

Conclusions

Virtual interviews offer many advantages to both applicants and programs and are likely to continue to be part of the recruitment season even once in-person interviews are determined to be safe to conduct again. Training and guidelines for best practices regarding virtual interviewing should be developed, and PCCM society engagement and leadership on this issue is urgently needed.