Virtual Recruitment Is Here to Stay: A Survey of ID Fellowship Program Directors and Matched Applicants Regarding Their 2020 Virtual Recruitment Experiences.

BACKGROUND: Graduate Medical Education training programs transitioned to all-virtual recruitment in 2020. Limited data have been published regarding the consequences of this transition. We aimed to understand (1) infectious diseases (ID) fellowship programs' recruitment efforts and the effect of virtual recruitment on application and interview numbers and (2) the number of programs to which matched applicants applied and interviewed and applicants' perspectives on virtual recruitment.
METHODS: In 2020-2021, we surveyed all US ID fellowship program directors (PDs) and matched applicants. Descriptive data analysis was performed on quantitative survey items. Free-text responses were analyzed through a quantitative content analysis approach.
RESULTS: The PD response rate was 68/158 (43%); the applicant response rate was at least 23% (85/365). PDs reported a 27% increase in mean number of applications received and a 45% increase in mean number of applicants interviewed compared with the previous year. Applicants especially valued the online program structure information, PD program overview videos, didactic and curriculum content, and fellow testimonials and profiles. Most applicants preferred interviews lasting no more than 40 minutes and interview days lasting no more than 5 hours. Nearly all (60/64, 94%) PDs adequately learned about candidates; most (48/64, 75%) felt unable to showcase their program as well as when in-person. Most PDs (54/64, 84%) and applicants (56/73, 77%) want an option for virtual recruitment.
CONCLUSIONS: Virtual recruitment enabled programs to accommodate more applicants and highlighted applicants' preferences for programs' augmented online presences and time-limited interview days. Most programs and applicants want an option for virtual interviews.

Due to the coronavirus disease 2019 (COVID-19) pandemic, the Coalition for Physician Accountability recommended that all residency and fellowship programs “commit to online interviews and virtual visits for all applicants, rather than in-person interviews for the entire [2020-2021] cycle” [1]. As a result, programs were compelled to establish virtual interviewing processes and platforms while concomitantly trying to upgrade their online presences. Meanwhile, applicants needed to adjust their expectations for interviewing and learning about new programs and cities from afar.

To guide applicants and programs, a number of perspective pieces, editorials, viewpoints, reviews, and single-program experiences were published before, during, and after the medical specialty fellowship program recruitment season [2-19]; 1 review provided evidence-based best practices [20]. Other contributions discussed the importance of mindfulness regarding potential biases against applicants under-represented in medicine and the potential role of social media [4, 21–23].

Prior specialty-wide study of virtual recruitment has largely been limited to surgical specialties, many of which transitioned from in-person to virtual recruitment in the middle of their 2020 recruitment seasons. In a survey of complex general surgical oncology program directors (PDs) and applicants, most PDs felt that virtual interviews permitted accurate portrayals of programs and applicants [24]. Roughly half of applicants felt that virtual interviews allowed applicants to accurately portray themselves; nearly half had a neutral view regarding programs’ ability to accurately represent themselves. In another report, >85% of surveyed female pelvic medicine and reconstructive surgery fellowship PDs reported satisfaction with virtual interviews and found them effective in assessing applicants; 31% preferred virtual interviews, and 60% reported being likely to offer virtual interviews in the future [25]. Surveys of applicants to and faculty of other programs found that applicants and faculty preferred in-person interviews, felt that they did not get to know one another as well virtually as in-person, and were less able to understand program culture and make an informed rank list [26-28]. To our knowledge, there are no published data regarding efforts undertaken by programs in advance of an entirely virtual recruitment season, the effect of all-virtual recruitment on application numbers, or perspectives of PDs and applicants on all-virtual recruitment within nonsurgical specialties.

Through a survey of infectious diseases (ID) fellowship PDs, we aimed to understand the impact of virtual recruitment on the number of applications received by programs, the number of interviews offered and conducted, and the recruitment resources developed by programs. Through a survey of matched applicants to these programs, we aimed to understand the number of programs to which applicants applied and interviewed and applicants’ perspectives on discrete components of virtual recruitment.

METHODS

Design and Setting

Based on a discussion of virtual recruitment experiences by members of the Infectious Diseases Society of America Training Program Directors’ Committee (IDSA TPDC) during their September 2020 meeting, 1 author (M.T.M., ID fellowship PD and TPDC Chair) established project goals and drafted initial survey items consistent with published survey guidance [29]. In October 2020, the other authors (D.R., ID fellow; C.A.B., B.G.B., L.M.C., V.P.L., and R.R.R., ID fellowship PDs and TPDC Members; and V.J.L.K., IDSA Fellows’ Subcommittee representative to the TPDC) reviewed and proposed edits to the survey items. One author (M.T.M.) then revised the surveys and sent them to the other authors for final review. In order to permit timely institutional review board review and send survey requests to PDs and applicants as close to Match Day as possible, focus group interviews, cognitive testing, and pilot testing of the surveys were not feasible.

Instruments

We developed a 20-item mixed qualitative and quantitative survey that included single-line open response items, a matrix question, multiple choice questions, and prose-based free-text questions for the PDs (Supplementary Appendix 1) and the matched applicants (Supplementary Appendix 2). Surveys were managed in Qualtrics (Provo, UT, USA).

Procedures

Using a list of US ID fellowship programs, their PDs, and PD email addresses provided by IDSA staff, on December 14, 2020, 1 author (M.T.M.) emailed each ID fellowship PD requesting their participation in the PD survey. These emails were sent 12 days after Match Day and contained a hyperlink to the survey.

Because there is no central repository containing the names and contact information for all applicants who matched into ID fellowship programs, we asked each PD to email the applicant survey request to applicants who matched into their program. PDs were sent a draft email to be sent to each matched fellow requesting their survey participation. This draft email contained a hyperlink to the applicant survey. We asked applicants to reflect not only on their experiences with the program with which they matched, but also on their collective recruitment experiences. We were not able to contact unmatched applicants.

These emails were sent weekly for 4 weeks. To provide reminders and help ensure receipt of our messages, the day after each email was sent, 1 author (M.T.M.) sent a message through the Training PD Community (ie, listserv) within the online MyIDSA platform of the IDSA.

All responses were anonymous. In order to protect and maintain participant anonymity, we did not solicit demographic data about program directors, programs, or applicants.

Analysis

Descriptive data analysis was performed on quantitative survey items. Questions that asked for free-text input of numerical data occasionally elicited impossible values (eg, >100% of applicants were interviewed). These values were excluded from analysis as noted in each table. We used t tests to compare differences in mean applications, interview invitations, and interviews comparing 2020 with 2019 in the PD surveys. Analyses were performed using Stata (StataCorp LP, College Station, TX, USA). Because responses to open-ended survey prompts are typically not appropriate for formal qualitative approaches to analysis, free-text responses to questions regarding aspects of virtual recruitment to retain or change (questions 19 and 20 on each survey) were analyzed through a quantitative content analysis approach [30]. A single author (M.T.M.) reviewed all responses and coded them. These codes and frequencies were reviewed by an additional author (D.R.); discrepancies were resolved through discussion.

RESULTS

Program Director Survey

The survey was sent to all 158 US adult ID fellowship PDs. Sixty-eight (43%) responded.

PDs reported a 27% increase in the mean number of applications (SD) submitted to their programs (89 [516] vs 70 [48]; P = .03), a 23% increase in the mean number of interview invitations offered (42 [17] vs 34 [14]; P < .01), and a 45% increase in the mean number of applicants interviewed (35 [17] vs 24 [14]; P < .01) in 2020 as compared with 2019 (Table 1). The majority (48/67, 72%) of PDs anticipate interviewing the same number of applicants in 2021 as in 2020, with only 8 (12%) and 11 (16%) planning to interview fewer or more applicants, respectively.

Table 1.

Application and Interview Data Reported by ID Fellowship Program Directors for the 2020 Recruitment Season

	No.	Mean	SD	Min	Max	P a
How many applications submitted to your program in 2020?	68	89	51	3	195	.03
How many applications submitted to your program in 2019?	63	70	48	3	176
How many applicants did you invite for interviews in 2020?	67	42	17	4	83	<.01
How many applicants did you invite for interviews in 2019?	63	34	14	4	65
How many applicants did you actually interview in 2020?	67	35	17	4	82	<.01
How many applicants did you actually interview in 2019?	63	24	14	1	56
Percentage of applicants invited to interview in 2020b	61	56%	23%	20%	97%	.419
Percentage of applicants invited to interview in 2019b	61	60%	25%	23%	100%
Percentage of invited applicants who interviewed in 2020c	61	81%	18%	27%	100%	<.01
Percentage of invited applicants who interviewed in 2019c	61	68%	23%	8%	100%

Abbreviation: ID, infectious diseases.

aAll P values correspond to unpaired t tests comparing means from 2020 vs 2019.

bExcludes 2 responses wherein the reported number of applicants invited to interview exceeded the reported number of applicants.

cExcludes 2 responses wherein the reported number of applicants who interviewed exceeded the reported number of applicants invited to interview.

The proportion of programs that generated, modified, or maintained online recruitment-related content is detailed in Figure 1A. Most (57/67, 85%) programs required additional resources (eg, faculty and/or staff time, monetary expenses) to facilitate the transition to a virtual format. For these programs, the amount of additional faculty and staff time varied (Table 3). The majority (28/55, 51%) of programs did not incur additional monetary expenses; these costs ranged from no additional monetary cost to a maximum of $25 000 for 1 program (Table 4). Half (32/64) of respondents said they will require fewer resources to support virtual recruitment should it be needed in 2021, whereas 26 (41%) will need similar and 6 (9%) more resources.

Figure 1.

A, Information made available by programs to applicants.a B, Matched applicants’ views of different features of programs’ websites. aOn either a public-facing website as of 8/12/20 (the date on which applications became available to PDs) or otherwise. Abbreviations: GME, Graduate Medical Education; PD, program director.

Table 3.

Self-Reported Additional Time Spent on Virtual Recruitment by Program Faculty and Staff

	None		<10 h		10–19 h		20–29 h		30–39 h		40–49 h		>50 h		Total
For the 2020 recruitment season	No.	%	No.	%	No.	%	No.	%	No.	%	No.	%	No.	%	No.	%
How much additional PD + APD time was required beyond time usually spent during recruitment?	1	2	3	6	13	24	16	30	4	7	11	20	6	11	54	100
How much additional total faculty time (other than PD + APD) was required beyond time usually spent during recruitment?	8	15	26	47	9	16	7	13	2	4	1	2	2	4	55	100
How much additional total staff/personnel time was required beyond time usually spent during recruitment?	3	6	6	11	19	35	18	33	2	4	4	7	3	6	55	100

Abbreviations: APD, associate program director; PD, program director.

Table 4.

Self-Reported Monetary Cost of Virtual Recruitment to Programs

	No.	%
No additional monetary costs were incurred	28	51
<$100	2	4
$100–$249	5	9
$250–$499	4	7
$500–$999	6	11
>$1000	10a	18
Total	55	100

aSix respondents entered individual cost values: $1500, $2500, $3500, $4000, $10 000, $25 000.

Nearly all (60/64, 94%) PDs felt they were able to sufficiently learn about each candidate virtually, with 25 (39%) feeling they learned about applicants sufficiently but not as well as in-person, 31 (48%) equally well as in-person, and 4 (6%) better than in-person. Most PDs (48/64, 75%), however, felt they were either unable to adequately showcase their program (8, 12%) or were able to showcase their program adequately but less well than with in-person recruitment (40, 62%). Only 12 (19%) PDs felt they were able to showcase their program as well as in-person, and 4 (6%) better than in-person. Despite these concerns, most (54/64, 84%) PDs want to at least have the option for virtual recruitment moving forward, with 37 (58%) preferring face-to-face with an option for virtual, 9 (14%) preferring virtual with an option for face-to-face, and 8 (12%) preferring virtual. Only 10 (16%) prefer in-person interviewing with no virtual option.

When asked to describe the aspect(s) of virtual recruitment/interviewing they are most likely to retain moving forward, 47/79 (59%) of PDs provided at least 1 response; this reflects 30% (47/158) of the total PD population. Of these 47, 28 (60%) plan to retain and/or improve or expand their reservoir of prerecorded online resources. The second most common response (16, 34%) was a plan to retain the option for either primary or secondary virtual visits, such as for additional meetings with faculty or asynchronous conference viewing. Other responses provided more than twice included maintenance of conversations or “happy hours” with current fellows (5, 11%), an emphasis on diversity, inclusion, and/or avoidance of bias (3, 6%), and a modified interview day structure or format (3, 6%).

Of the 38 PDs who provided at least 1 aspect of recruitment they are likely to change, 14 (37%) plan to augment or improve their portfolio of prerecorded online resources. The second most common response was to make no changes (7, 18%).

Matched Applicant Survey

There were 85 unique responses from matched applicants. The number of matched applicants who were sent the survey is not known; 365 positions were filled through the Match, so the response rate was at least 23% (85/365). These matched applicants applied to a mean of 20 programs (Table 2). For 47/85 (55%) applicants, this number was not affected by the virtual nature of the recruitment season. Of the 38 for whom this number was affected, 13 (34%) applied to 1–3 additional programs, 18 (47%) applied to 4 or more additional programs, 4 (11%) applied to fewer programs, and 3 (8%) did not answer. In retrospect, the majority (52/73, 71%) of respondents would have applied to the same number of programs were they to repeat the experience, 19 (26%) would have applied to fewer programs, and few (2/73, 3%) would have applied to more programs.

Table 2.

Application and Interview Data Reported by Matched ID Fellowship Program Applicants for the 2020 Recruitment Season

	No.	Mean	SD	Min	Max
To how many programs did you apply in 2020?	82	20	15	1	91
How many interview offers did you receive in 2020?	82	14	10	1	56
How many interview days did you actually attend in 2020?a	81	11	6	1	30

Abbreviation: ID, infectious diseases.

aOne response excluded (respondent reported 0 interviews attended).

Matched applicants received a mean of 14 interview offers and attended a mean of 11 interview days (Table 2). Of the 48/82 (58%) of applicants whose decision to interview at programs was affected by the pandemic, 40 responded to a question quantifying this effect; half (20/40) of these respondents reported interviewing at 1–3 more programs than they otherwise would have, and half (20/40) at 4 or more. In retrospect, the majority (47/73, 64%) of respondents would have interviewed at the same number of programs were they to repeat the experience, and 26 (36%) would have interviewed at fewer; none would have interviewed at more programs.

Matched applicants’ views of online materials provided by programs are summarized in Figure 1B. There was near consensus regarding interview day structure, including individual interview duration, total interview day duration, and number of faculty interviews per day (Table 5). The majority (48/73, 66%) of applicants preferred 5–9 minutes between interviews, with 13 (18%) preferring <5 minutes and 12 (16%) 10–14 minutes. Nearly all (65/73, 89%) matched applicants preferred a single, consolidated interview day. Only 3/72 (4%) of applicants reported changing their rank order list based upon gift boxes or vouchers for food provided by programs; 31 (43%) viewed these favorably but did not change their rank order list as a result, 36 (50%) had a neutral view, and 2 (3%) had an unfavorable view.

Table 5.

Matched Applicant Perspectives on the Structure and Duration of the Interview Day

Ideal Number Interviews/Day			Ideal Interview Duration			Ideal Virtual Interview Day Duration
	No.	%		No.	%		No.	%
2	4	6	<30 min	47	64	>1 and ≤2 h	2	3
3	28	38	30–39 min	23	32	>2 and ≤3 h	6	8
4	33	45	40–49 min	3	4	>3 and ≤4 h	30	41
5	5	7	Total	73	100	>4 and ≤5 h	26	36
6	2	3				>5 and ≤6 h	6	8
7	1	1				>6 and ≤7 h	3	4
Total	73	100				Total	73	100

Most matched applicants (52/73, 71%) felt that they learned about programs somewhat (47, 64%) or much (5, 7%) less well than had recruitment been in-person; 20 (27%) felt they learned about programs equally well, and 1 (1%) somewhat better. Despite this perspective, most (56/73, 77%) applicants want to at least have the option for virtual recruitment moving forward, with 32 (44%) preferring face-to-face with an option for virtual, 15 (21%) preferring virtual with an option for face-to-face, and 9 (12%) preferring virtual. Only 17 (23%) prefer in-person interviewing with no virtual option.

When asked to describe aspects of virtual recruitment/interviewing they would most like to see retained moving forward, 37/85 (44%) matched applicants responded; this reflects 10% (37/365) of the total matched applicant population. More than half (20/37, 54%) of these respondents would like programs to retain their expanded online content. The second most common response, described by 11 (30%), was a preference to retain virtual interviews, whether for primary or secondary visits. Other responses provided more than twice included shorter interview days and/or interview duration (5, 14%) and spending time with fellows, including meetings with smaller groups of fellows (3, 8%).

Of the 28 matched applicants who described aspects of virtual recruitment/interviewing they would most like to see changed moving forward, 15 (54%) indicated a preference for different aspects of the timing or structure of the interview day, including 6 (21%) who preferred shorter interview days, 3 (11%) limits upon the total number of interviews and/or their duration, 2 (7%) sufficiently long breaks between interviews, 2 (7%) consolidation of all interviews into a single day, 1 (4%) interviews spread over the course of the week, and 1 (4%) time for lunch. There were 9 (32%) comments pertaining to time spent with fellows with no consensus message.

DISCUSSION

While the majority of surveyed PDs and matched applicants prefer in-person recruitment, most want at least the option of virtual recruitment, and nearly all PDs felt that they adequately learned about candidates virtually. Additional attributes of virtual recruitment identified by our study include programs’ and applicants’ abilities to accommodate increased numbers of applications and interviews, the value applicants place on specific aspects of programs’ expanded online profiles, and applicants’ preference for time-limited interview days.

Whereas most other internal medicine subspecialties typically fill at least 90% of their open fellowship program positions, ID has had less success filling its positions in the recent past [31]. The 2020 ID fellowship application numbers and Match results, however, were notably improved [31, 32]. In 2020, there were 504 ID fellowship applicants to the Electronic Residency Application Service, up from 385–444 over the preceding 4 years [32]. These 504 individuals, including 100 applicants who did not submit rank order lists to the National Resident Matching Program (NRMP), applied to an average of 23 programs, for an average of 76 applications per program, comparable with the matched applicant data from our survey [31, 32]. Many other IM fellowship programs also experienced increases in these numbers [33]. Of the 404 applicants who participated in NRMP, 365 (90%) matched into an ID fellowship, such that 365/414 (88%) offered positions filled, up from 79% to 81% over the preceding 4 years [31]. In concert with these Match results, our survey data suggest that ID fellowship programs were able to effectively pivot to virtual recruitment.

During meetings of ID fellowship PDs in advance of 2020 recruitment, many expressed an expectation that virtual recruitment would be associated with increased application numbers. Our data support this hypothesis, with programs reporting a greater number of applications, interview offers, and interviews than in 2019. While the majority of applicants reported applying to the same number of programs as they would have had recruitment been in-person, most who modified this number due to virtual recruitment applied to more programs than they otherwise would have. Although associated with increased faculty time demands, the system clearly appears to have had capacity for these increased numbers, and the majority of programs and applicants seem comfortable with them, as most plan to maintain them next year (programs) or would not have changed them if they could (applicants). With the caveat that our study was not designed to address this question as we surveyed only matched applicants, applicants nationwide do not appear to have been disadvantaged by the increased number of applicants and applications, as a higher proportion of applicants matched than in recent years, and only 10% of applicants went unmatched, as compared with 7%–8% over each of the prior 4 years [31].

Similar to previous reports, our data suggest that we are unlikely to return to the former all-in-person recruitment status quo [24, 25]. PDs are urged to consider how they will accommodate a mixture of in-person and virtual recruitment once the former again becomes possible, including ways to guard against potential biases toward applicants who choose one format over the other. Our data also provide important perspectives on recruitment preparations. While all online resources were helpful to some proportion of matched applicants, some were rated as critically or very important by at least 60% of applicants, including prose- and photo-based content on program structure, PD program overview video, prose- and photo-based content on didactics and curriculum, and fellow testimonials and profiles. Programs planning to revise, update, or newly create web content may wish to focus their efforts on these highest-yield areas. When undertaking these preparations, programs should be cognizant of the substantial time investment required to prepare for virtual recruitment, with half of programs anticipating a need for similar resources should virtual recruitment be incorporated into the next recruitment season. These data may help PDs who aim to maintain or increase the support they receive for their PD work, especially in light of the increased number of applicants in 2020 and the expectation that these numbers will be maintained should virtual recruitment continue.

Matched applicants expressed clear preferences regarding the interview day duration and structure. Most prefer 3–4 interviews lasting no more than 40 minutes each and a total interview day duration of 3–5 hours. We did not solicit input from PDs on these items, but there is a notable difference in the frequency with which these topics were discussed in PD and matched applicant open-ended responses regarding innovation opportunities for the future. PDs are encouraged to be mindful of this input as they plan future recruitment seasons.

The limitations of our study include our PD response rate of 43%. While incomplete, this response rate to an unincentivized survey in the midst of the ongoing COVID-19 pandemic, particularly during the surge in cases between mid-December 2020 and mid-January 2021, was greater than we anticipated. Although our matched applicant response rate was lower, we are not able to quantify this rate with certainty as the total number of matched applicants who were sent the survey is not known. For both groups, our results are at risk of nonresponse bias, such that respondents may not be representative of the entire population of PDs and candidates; candidate responses and nonresponses may also not be evenly distributed across different program demographics if some PDs were less likely to send the survey to matched applicants than others [34, 35]. As such, our responses may not be generalizable to all programs and all matched applicants. Because our surveys were anonymous, we were unable to determine whether respondents differed from the entire survey population [35]. Survey incentives, sending the survey after rank order lists are submitted to NRMP and before Match Day, sending the survey at a time when COVID-19 cases are not cresting, and working with stakeholders to develop a centralized database of contact information for all ID fellowship program applicants might improve future response rates. PD reports of the additional faculty and staff time spent on recruitment activities in 2020 were likely best estimates. The majority of matched applicants only go through the ID fellowship match once and never went through an in-person ID fellowship match; their estimated number of applications and interviews had recruitment been in-person may be different from what would have transpired with an all-in-person recruitment season. We do not have data on the extent to which virtual recruitment permitted applicants with more limited resources to match at programs to which travel costs might have been prohibitive had recruitment been undertaken in-person. Drawing conclusions from free-text responses to survey questions can be misleading, and formal qualitative approaches to analysis of such responses is often inappropriate [30]. Because these responses complemented the data from responses to Likert-scale style questions, however, we have reported those results through a quantitative content analysis approach. We are unable to exclude the possibility of recall bias and unintentional data entry errors by participants.

Future research efforts should study the perceptions and ramifications of hybrid in-person and virtual recruitment seasons, including strategies for mitigating bias for or against applicants who choose one interview format over the other, as well as cost considerations for applicants and programs. Also important will be to study whether application numbers per program continue to increase and how programs balance interview capacity with demand.

CONCLUSIONS

Our survey data and the Match results indicate that the 2020 ID fellowship virtual recruitment season was a remarkable success. With virtual recruitment likely here to stay in some capacity for all specialties and programs, applicants’ views regarding essential aspects of programs’ online portfolios and their preference for time-limited interview days should help inform future recruitment efforts to the benefit of programs and applicants alike.

Click here for additional data file.

Click here for additional data file.