The Virtual Interview Experience: Perspectives of Pulmonary and Critical Care Fellowship Applicants.

Background: Because of the coronavirus disease (COVID-19) pandemic, graduate medical education programs adopted virtual interviews (VIs) as the default modality for the 2020 recruitment season. It is unknown whether VIs allowed applicants to effectively evaluate programs, and the best interview format for the future is unclear. Objective: To 1) assess pulmonary and critical care applicants' perceived ability to evaluate programs using VIs, 2) determine the attitudes of applicants toward the components of VIs, and 3) identify applicants' preferences for the future fellowship interview format.
Methods: After the National Residency Matching Program medical subspecialty match, an electronic survey was sent to 1,067 applicants to pulmonary and critical care medicine programs asking them to compare their fellowship VI experience with their residency in-person interview (IPI) experience.
Results: Three hundred six (29%) applicants responded to the survey, and 289 completed it (27%). There were 117 (40%) women and 146 (51%) White individuals. Most respondents believed that VIs hindered their ability to evaluate programs' culture, faculty-fellow relationships, location, facilities, and their own fit within the program. They believed they were able to evaluate the clinical experience, curriculum, and potential for academic development equally well compared with IPIs. The most helpful elements of VIs were the interview with the program director, meetings with the fellows, and interviews with faculty members. Less helpful elements included conference access, prerecorded program director presentations, virtual hospital and city tours, and video testimonials. One hundred twenty-three respondents (43%) chose VIs with an optional visit as their preferred future interview format, 85 (29%) chose IPIs, 54 (19%) wanted a choice between VIs and IPIs, and 27 (9%) chose VIs only.
Conclusion: Most pulmonary and critical care medicine applicants preferred future interviews to include both VIs and the option of an in-person visit or interview. This study can assist programs in designing their future interview formats in a trainee-centric fashion.

The pulmonary and critical care medicine (PCCM) fellowship recruitment season coincided with the second wave of the coronavirus disease (COVID-19) pandemic in the United States in the summer and fall of 2020, leading to national organizations strongly recommending that interviews be conducted virtually (1, 2). Heeding the call to conduct this recruitment season in a safe manner, graduate medical education programs nationwide converted their traditional in-person interview (IPI) format to a virtual interview (VI) format. Although some graduate medical education programs had used VIs in the past, few used them as their standard format; therefore, most programs created VI processes anew. Most fellowship program applicants had not interviewed in a virtual format before: this process was novel for both programs and applicants.

VIs presented opportunities for innovation. Programs avoided the expense of interview meals (3), and applicants evaded the significant travel and lodging expenses associated with IPIs (4). Some posited that removal of these barriers may increase the diversity of applicants to a given program, allowing those with lesser financial means to interview at places they otherwise might not be able to interview (5). Finally, VIs allowed applicants more flexibility in scheduling, with less time away from busy residency schedules than with IPIs (6).

Choosing a fellowship program is an important decision for applicants. It is unclear whether the virtual experience allowed PCCM applicants to get a true sense of the cultures and learning environments of the institutions at which they interviewed. We sought to assess the applicants’ experiences with VIs, assess their perceptions of effectiveness, and gather their input regarding the future format of fellowship interviews.

The Association of PCCM Program Directors (PDs) convened a working group of 15 volunteer PDs and associate PDs to survey PCCM fellowship applicants for the 2020 match about their experience with VIs. This applicant cohort is unique: they experienced IPIs for a residency position, followed by VIs for a fellowship. As such, they are well positioned to compare the two modalities. The goals of the survey were to 1) assess the applicants’ perception of effectiveness of VIs in their ability to adequately evaluate a training program, as compared with IPIs; 2) determine the attitudes of applicants toward components of a VI experience; and 3) identify the applicants’ preferences for the future format of fellowship interviews.

Methods

This survey study was approved as exempt by the institutional review boards of Emory University and Indiana University. Although it was human subject research, it was exempt from further institutional review board review and approval, as it met the criteria for exemption under Title 45, Part 46.104(d)(ii) of the U.S. Code of Federal Regulations.

We aimed to survey all applicants to PCCM fellowships within the United States. As a list of all applicants is not made available through the Electronic Residency Application System (ERAS), we identified applicants by combining ERAS applicants’ e-mail addresses from 13 U.S. PCCM fellowship programs represented by the authors and contributors. Approval to share these e-mail addresses with the first author for the purpose of this study was obtained from ERAS. The survey was anonymous, and results did not include any identifying applicant information.

A modified Delphi method was used to create the survey tool (7). After an initial meeting to review the background and goals of the survey, questions were created collaboratively by the authors on an electronically shared document. The authors were then surveyed anonymously and asked for each question whether it should be included, deleted, or modified. There was a predetermined threshold of 80% consensus required for inclusion. After a first round with 100% participation, 21 questions were carried over and circulated again in a shared document for final edits and comments. The survey was then tested by the authors, and further deletions were made to optimize its length. The resulting 16-question survey was piloted on 11 internal medicine residents from the authors’ home institutions who were applying to other internal medicine subspecialty fellowships. Minor modifications were made in response to their feedback, and the final survey contained 15 questions. The survey tool can be found in Appendix E1 in the data supplement.

The initial survey request was sent on December 7, 2020—4 days after the National Residency Matching Program (NRMP) medical subspecialty match. This initial request was followed by three reminder e-mails on December 11, December 15, and January 4, 2021.

Statistical analysis was performed by using JMP Pro 15 software (SAS Institute Inc.). Continuous variables were summarized by using the median and interquartile range (IQR). Categorical variables were summarized by using percentages that were rounded to the nearest whole number. A Pearson chi-square test was used to compare values.

Results

One thousand sixty-seven unique applicants applied to the 13 programs represented by the authors. Three hundred and six (29%) applicants responded to the survey, with 289 completing the survey (27% overall completion rate). Table 1 compares our respondents with the 2020 ERAS PCCM applicant pool. Respondents were 40% female, 51% White, and relatively spread out geographically across the country. Forty-seven (16%) were from groups underrepresented in medicine (UIM), defined as those who identify with at least one of the following groups: American Indian or Alaska Native; Black or African American; Hispanic, Latinx, or of Spanish origin; and Native Hawaiian or Pacific Islander. The median number of programs applied to by applicants was 37, with 35% of respondents having applied to more than 50 programs.

Table 1.

Demographics and geographic distribution of applicants who completed the survey

	Our Sample (N = 289)	2020 PCCM ERAS Applicants (N = 1,295)
Sex, n (%)
Male*	167 (57.8)	860 (66.4)
Female	117 (40.5)	435 (33.6)
Prefer not to answer	5 (1.7)
Self-identified race and ethnicity, n (%)
American Indian or Alaska Native	0 (0)	2 (0)
Asian	73 (25.2)	323 (24.9)
Black or African American	13 (4.4)	56 (4.3)
Hispanic, Latinx, or of Spanish origin	29 (10.0)	72 (5.6)
Native Hawaiian or other Pacific Islander	2 (0.7)	1 (0)
White	146 (50.5)	499 (38.0)
A race/ethnicity not listed here	15 (5.2)	53 (4.1)
Unknown	12 (4.2)	58 (4.5)
Region, n (%)
Northeast†	105 (36.3)	Data unavailable
South‡	67 (23.2)
Midwest§	58 (20.1)
West‖‖	57 (19.7)
Outside of the United States	2 (0.7)

Definition of abbreviations: ERAS = Electronic Residency Application System; PCCM = pulmonary and critical care medicine.

ERAS 2020 data are available from https://www.aamc.org/media/41121/download. In both our sample and the ERAS sample, data display applicants who reported self-identified race and/or ethnicity alone or in combination with other races and/or ethnicities. Therefore, an individual may be counted in multiple race and/or ethnicity categories.

Significantly different between our sample and 2020 ERAS applicants (P = 0.01).

CT, DE, MD, ME, MA, NJ, NH, NY, PA, RI, VT.

AL, AR, DC, FL, GA, KY, LA, MS, NC, OK, SC, TN, TX, VA, WV.

KS, IA, IN, IL, MI, MO, ND, NE, MN, OH, SD, WI.

AZ, AK, CA, CO, HI, ID, MT, NM, NV, OR, UT, WA, WY.

Respondents participated in a median of 9 (IQR, 5–12) VIs. Thirty-two percent estimated that they applied to more programs as a result of the VI format, and 30% believed they participated in more interviews as a result of the virtual format. Most respondents believed that VIs hindered their ability to evaluate a program’s facilities, location, culture, and faculty–fellow relationships, as well as their own fit within the program. On the other hand, most believed that they were able to evaluate the clinical experience, curriculum, and the potential for research and academic development as well as they had been during IPIs. Attitudes toward the ability to evaluate diversity and inclusion and fellow satisfaction were mixed. None of the program factors were believed to be better evaluated through VIs than through IPIs (Figure 1). There was no difference in the responses between UIM and non-UIM applicants.

Figure 1.

How well applicants were able to evaluate program factors with virtual interviews compared with in-person interviews.

When asked about the importance of specific factors (location, proximity of support system, previous experience with the program, program reputation, and interview day experience) in ranking of a program after VIs compared with IPIs, most applicants believed the VI process did not affect the importance of any of these factors in their ranking. Two factors were considered by a minority of respondents to be more important in VIs: program location (39%) and proximity of a support system (34%). Applicants were split on the importance of a program’s social media presence in the context of VIs versus IPIs, with 44% stating it was more important in the setting of VIs and 40% believing it was less important. Detailed data can be found in Figure E1. UIM applicants ranked the interview day experience as more important in their ranking decisions more often than did non-UIM applicants (36% vs. 23%; P = 0.02); they considered the presence and images of the program on social media as less important than did non-UIM applicants (53% vs. 38%; P = 0.01). Fifty-five percent of respondents did not believe that VIs affected how highly they ranked their home institution.

When asked about how helpful particular elements of the VI were in evaluating the program, the highest-rated elements included the interview with the PD (very helpful, 76%), informal meetings with the fellows (71%), and interviews with faculty members (70%) (Figure 2). Other highly rated factors included a live presentation by the PD, interviews with other division leaders, and the program website. Conference access, prerecorded PD presentations, virtual hospital tours, video testimonials, and virtual city tours were rated as somewhat helpful by most respondents, with 18–28% finding them unhelpful. Open houses, when offered, received mixed reactions, with 16% finding them very helpful, 20% finding them somewhat helpful, and 20% finding them not helpful (Figure 2). A significantly larger proportion of UIM respondents found the prerecorded PD presentations and the virtual tours to be very helpful compared with their non-UIM counterparts (53% vs. 22% [P = 0.0002] and 38% vs. 20% [P = 0.01], respectively).

Figure 2.

Importance of elements of the virtual interview in candidates’ ability to evaluate a program. PD = program director.

When asked about their choice of interview format for the future, 43% preferred VIs with an optional in-person visit, 29% preferred IPIs only, 19% preferred to be offered a choice between VIs and IPIs, and only 9% chose exclusively VIs (Figure 3). There were no significant differences in future interview format preferences between UIM and non-UIM applicants. Eighty-nine percent of respondents expressed concern that, if offered a choice between VIs and IPIs, their choice would impact how they were ranked by a program.

Figure 3.

Applicants’ preferred interview format in the future. IPI = in-person interview; VI = virtual interview.

The factors that would lead applicants to choose VIs over IPIs included travel cost (87%), travel time (85%), geographic location (53%), familiarity with the program (41%), and less interest in the program (39%), whereas more interest in the program was cited as a factor in choosing VIs by only 8% of respondents. One applicant commented that “what other applicants are opting for” would also matter in choosing VIs over IPIs.

Sixty-three respondents (22%) provided additional narrative comments. Details and examples of these comments can be found in Appendix E2.

Discussion

This study is the largest to date evaluating fellowship applicants’ perception of the VI process and sampled the national population of PCCM applicants. We found that most respondents (91%) would choose a future interview format that would include at least the option of an IPI or in-person visit, with only 9% choosing VIs exclusively. The most popular option (43%) was a VI followed by an optional in-person visit.

These results are similar to those of other studies from surgical programs in which applicants believed that VIs should continue to be offered, although not as the only modality (8, 9). Similarly, a recent NRMP survey assessing residency applicants’ experiences with interviews revealed that, despite financial and scheduling benefits with VIs, the majority of respondents (51%) preferred IPIs, whereas 21% preferred VIs. The remaining respondents had no preference (11%) or were undecided (17%) (10).

Because ERAS does not share applicant information, we used a convenience sample gathered through combining ERAS application e-mails from programs represented by the authors and contributors. We hypothesized that by including applicants from several large and geographically diverse programs, we would capture most of the PCCM applicant pool. Indeed, the total number of applicants who applied to a PCCM program through ERAS in 2020 was 1,387 (11), and our survey was sent to 1,067. Furthermore, the number of applicants who registered for the NRMP match in 2020 was 1,023. The difference between the ERAS number and the NRMP number likely reflects applicants who did not receive interview invitations. As a result, we are confident that we were able to send surveys to the vast majority of applicants in the 2020 PCCM cycle who received interview invitations.

About a third of applicants perceived that they applied to and interviewed with more programs as a result of the virtual format. At first glance, this could be thought of as a benefit of VIs, increasing trainees’ access to interviews and improving their chances of matching. In reality, given the fixed number of available PCCM positions, the increasing ratio of applicants per training position, and the nearly 100% fill rate in the past 5 years (12), the effect is likely to be a net increase in application volumes to programs, without an increase in match rates. This will place a higher workload on programs and may contribute to interview allocation disparities in which the most traditionally competitive applicants receive most of the interview invitations.

Applicants perceived VIs to be as effective as IPIs for evaluating a program’s clinical experience, curriculum, and the support for research and academic development. This is not surprising, as this information is made available to applicants in similar ways during both VIs and IPIs, either via a website or by direct communication from PDs or their representatives during the interview day. Applicants also believed that the virtual process was less conducive to evaluating facilities and program location. Program location was still considered to be “as important” (53%) or “more important” (39%) in program ranking for applicants. However, virtual hospital and city tours were not deemed very helpful when offered. It remains unclear whether virtual tours were not helpful because they were of poor quality, did not focus on elements that applicants found important, or simply do not represent a useful source of information to candidates trying to discriminate the best programs.

Not surprisingly, applicants felt less able to evaluate intangible factors such as program culture, faculty–fellow relationships, their own fit within a program, and, to a lesser extent, fellow satisfaction and diversity and inclusion with VIs. Interestingly, several of these items, including overall fit and geographic location, are top-ranked factors affecting program ranking by applicants (13). This may explain why most respondents chose future interview options that include an in-person component. These findings mirror those of the NRMP survey of residents’ experiences with VIs—most respondents found it slightly to moderately challenging to assess the program curriculum but found it moderately to very challenging to assess program culture and fit (10).

This year’s VI process was borne from extraordinary circumstances and implemented quickly. VIs, undoubtedly, have tremendous advantages, including financial and time savings for applicants (3) and programs (4), decreasing the significant carbon footprint of the medical subspecialty interview process (14), and potentially increasing the diversity of a program’s applicant pool. In fact, 71% of respondents opted to keep VIs as an option in future interview seasons.

For programs that are considering continuing VIs in the future, these data should inform the process. Given our findings, programs should consider focusing their attention on the most helpful elements of VIs identified by applicants. These included interviewing with the PD and faculty and division leaders, meeting with fellows, a live presentation by the PD, and a robust program website. Less helpful elements were conference access, prerecorded PD presentations, virtual hospital and city tours, video testimonials, and open houses (informal virtual gatherings with fellows and/or faculty on days other than the interview day). Our study did not allow us to further evaluate the aspects of these interview elements that made them unhelpful. For example, were virtual tours unhelpful because they were not professionally made? Could open houses be more helpful if better designed and more conducive to casual conversations, such as by limiting the number of participants? These remain open questions and areas that programs can, and should, evaluate to determine how best to improve applicants’ ability to evaluate their programs.

The results of this study also present a difficulty for PDs in designing interview processes going forward. Although a majority of applicants (62%) preferred some sort of choice regarding an IPI, they (89%) were also concerned that a decision to interview in person or virtually may be interpreted as a signal of interest in the program. This is a valid concern—in reality, a decision to interview in person or virtually is complex and may be informed by financial and time constraints that are out of an applicant’s control, although 39% of respondents did cite less interest in a program as one of the factors leading them to choose VIs over IPIs.

Given these facts, programs should be careful to build equity into this process. This may include blinding program recruiting leadership to decisions regarding whether or not a person chooses an in-person visit. Alternatively, it could also be a process in which a program completes its rank order list before a visit day (which would occur before the rank list due date to allow applicants to use their experience to inform their decisions). Each of these suggestions carries its set of challenges. It could be difficult to completely blind program leadership from in-person visit attendance. Furthermore, suggesting that all programs complete their rank lists before offering in-person visits would condense these visits to a very narrow temporal window, practically limiting most applicants to one in-person visit, negating the utility of such visits, and potentially disadvantaging applicants with limited resources and scheduling flexibility. Uncoupling of the visit from the rank order creation may be feasible if deadlines for rank order lists for applicants and programs are separated, allowing more time for candidates to schedule their visits and ensuring their choice would not affect their rank (15). The challenges in designing a truly equitable mixed VI and IPI format in the current recruitment timeline are further delineated in the Association of PCCM PDs’ recommendation to proceed exclusively with VIs for the upcoming recruitment season, and to consider doing so indefinitely (16). The recommendation also calls for uniformity in the interview format across programs, which will be of utmost importance to ensure order and equity.

Limitations

Our study is limited by the completion rate of 28%, which introduces selection bias, and the opinions represented may not be a true reflection of the entire PCCM applicant population. In addition, the percentage of female respondents outweighed the percentage of female applicants. It is possible that this influenced the data collected, but we believe that this is unlikely. The response rate of this study is in the range of previously published e-mail survey response rates among physicians (17). Although e-mail surveys do not usually result in a high response rate, it was the only practical way to reach applicants nationally and was recently used by others with similar completion rates (10). Our study is also limited to the applicants to a specific medical subspecialty, which may limit external validity, although our results were similar to those of the NRMP survey of residency applicants (10).

Conclusions

The COVID-19 pandemic has ushered in a new era for training program recruitment and has demonstrated many advantages of VIs. Our study shows that most applicants prefer future interview formats to include both VIs and the option of in-person visits or interviews. It also identifies the most and least helpful elements of VIs and can assist programs in structuring their interviews in a trainee-centric way. Future recruitment seasons will have to be designed with fairness and consideration toward all applicants in mind and will require a continued dialogue among all stakeholders.