Virtual Recruitment: Experiences and Perspectives of Internal Medicine Program Directors.

Virtual recruitment is new to nearly all internal medicine residency programs.

The majority of internal medicine program directors were equally or more satisfied with their 2020-2021 incoming intern class compared with the prior 3 intern classes.

Cost savings for residency programs are substantial for virtual interviews, compared to in-person interviews.

Internal medicine program directors express varying opinions about the future of virtual and in-person recruitment.

Alt-text: Unlabelled box

Introduction

Due to the coronavirus disease 2019 (COVID-19) pandemic, internal medicine residency recruitment in fall 2020 was exclusively conducted virtually. This change represented a major shift in recruitment strategies and operations for US training programs, with limited data on best practices for virtual interviews. , Prior to COVID-19, applicants routinely traveled for in-person interviews with program faculty, met with program leadership and current residents, and visited the location and health care facilities. The in-person interview process provided applicants the benefit of learning about the culture and multiple aspects of the program and its location and had a major influence on how applicants ranked programs. In 2020, conditions associated with the pandemic demanded a change to a virtual interview process with preparation occurring in a compressed timeline that coincided with unpredictable COVID-19 surges.

Intern recruitment is a core responsibility for internal medicine program directors and a sizable undertaking with significant investments of time and resources by residency programs and applicants. Prepandemic, internal medicine programs spent a median of $14,162 per filled intern position with an overall median cost of recruitment per program at $148,000. Added to the costs, students, medical schools, and program directors have grappled with escalating numbers of residency applications for several years. In 2020, the mean number of Electronic Residency Application Service (ERAS) applications submitted for each applicant to internal medicine programs had risen to a high of 70.7 from 59.3 in 2016, representing approximately $1500 in direct application costs per applicant, which does not include travel. In response to this “application inflation,” some program directors have increased the number of invitations extended and interviews conducted and have raised their screening standards for interview offers. , Virtual recruitment imposed another dimension of uncertainty in an already cost- and labor-intensive process.

Limited data are available about the effects of the wholesale shift to virtual recruitment and how it might affect future recruitment. To better understand this, we conducted a national survey of internal medicine program directors about their experiences and program outcomes in the first completely virtual recruitment season. Further, we solicited program director perspectives about application inflation as well as guiding principles for future recruitment seasons.

Methods

The Association of Program Directors in Internal Medicine (APDIM), a professional organization that represents internal medicine residency program directors, core faculty, and program administrators, is a charter organization of the Alliance for Academic Internal Medicine (AAIM). The APDIM Survey Committee regularly surveys internal medicine program directors on a variety of critical topics in graduate medical education (GME).

Survey Instrument

In fall 2020, the APDIM Survey Committee recognized the need to conduct a survey of internal medicine program directors regarding the residency virtual interview and recruitment process. The tentative subject matter for the survey was curated from exchanges via the APDIM Discussion Forum (an email listserv including more than 4000 GME faculty and administrators) and the final topics were identified via committee consensus. The initial survey sections were drafted by a subset of committee members, and the complete survey was reviewed and revised by the entire committee. The instrument was programmed in the Qualtrics Surveys platform by AAIM staff and was pilot-tested by the committee as well as 6 GME experts outside of the committee. The instrument consisted of 25 questions with conditional (skip or display) logic and included multiple-choice-select-one, 5-point Likert scale, numeric entry, and open-text response questions.

In March 2021, the survey was distributed via a personalized email invitation to 436 internal medicine residency program directors, representing 82% (436 of 532) of US internal medicine residency programs with Accreditation Council for Graduate Medical Education (ACGME) accreditation prior to July 1, 2019 (ie, at least 1 year prior to the academic year to which the survey applied). Four email reminders to nonrespondents were sent, and the survey closed in May 2021. The study (21-AAIM-117) was deemed exempt by Pearl Institutional Review Board (US DHHS OHRP #IRB00007772) under FDA 21 CFR 56.104 and 45CFR46.104(b)(2).

Data Analysis

Data analysis was conducted in Stata 16.1 SE by AAIM staff. Before deidentifying the final responses for analysis, the study data set was appended with data from external sources that explained the most survey population variance, including US Census Bureau geographic region. Residency program characteristics such as number of approved positions were obtained from the Accreditation Council for Graduate Medical Education Accreditation Database System (Public). Program type and other program characteristics were obtained from the American Medical Association Fellowship and Residency Electronic Interactive Database Access System Online. American Board of Internal Medicine (ABIM) rolling 3-year residency pass rates were provided by the board.

Summary statistics included frequencies and percentages for categorical variables and measures of central tendency (eg, mean, median, standard deviation, interquartile range) for continuous variables. To assess the statistical representativeness of the responses, essential program characteristics from the data sources described previously were used to compare respondents and their programs to the survey-eligible population. We tested for goodness-of-fit or associations between categorical variables using the Adjusted Wald (Pearson) χ2 test (1 degree of freedom); a design-based Pearson χ2 (2 degrees of freedom) was used to assess goodness-of-fit when both categorical variables exceeded 2 categories. Due to unequal variances, we used the Welch t-test to compare the means of continuous variables against dichotomous variables. Statistical significance was designated with an alpha level of P ≤ .05. Due to conditional logic or item nonresponse, some denominators do not equal the total number of survey respondents.

Results

The response rate was 61.9% (270 of 436 responses). There was no statistical association between respondents and nonrespondents based on essential characteristics that defined the complete survey population (Table 1 ). Although there was slight over-representation of university-based programs (39.3% among respondents; 31.9% for the population), that difference was not statistically significant (P = .053).

Table 1

Essential Program Characteristics of Respondents and Nonrespondents: 2021 APDIM Spring Survey of Internal Medicine Residency Program Directors

	Respondents (n = 270)	Nonrespondents (n = 166)	Total (n = 436)
Characteristic	No. (Column %)	No. (Column %)	No. (Column %)	P Value*
Program Type (AMA-FREIDA)
University-based	106 (39.3)	33 (19.9)	139 (31.9)	.053
Community-based	37 (13.7)	43 (25.9)	80 (18.4)	.109
Community-based, university-affiliated	122 (45.2)	89 (53.6)	211 (48.4)	.324
Military-based	5 (1.9)	1 (0.6)	6 (1.4)	.349
Census Region (US Census Bureau)†
Northeast	83 (30.7)	51 (30.7)	134 (30.7)	.998
Midwest	60 (22.2)	38 (22.9)	98 (22.5)	.922
West	41 (15.2)	27 (16.3)	68 (15.6)	.769
South	86 (31.9)	50 (30.1)	136 (31.2)	.842
VA Affiliation: Yes (ACGME)	107 (39.6)	47 (28.3)	154 (35.3)	.096
Accreditation Status (ACGME)
Continued or continued with warning	260 (96.3)	152 (91.6)	412 (94.5)	.099
Initial or initial with warning	10 (3.7)	14 (8.4)	24 (5.5)

ABIM = American Board of Internal Medicine; ACGME = Accreditation Council for Graduate Medical Education; AMA-FREIDA = American Medical Association Residency and Fellowship Database; APDIM = Association of Program Directors in Internal Medicine; SD= standard deviation; USMLE = US Medical Licensing Examination; VA = Veterans Affairs.

Bivariate (Adjusted Wald [Pearson]) test of association with 1 degree of freedom).

Excludes programs from 2 US territories, due to small cell sizes/data confidentiality.

Welch t-test with unequal variances.

Interquartile range test (Welch t-test).

Ninety percent of programs (243 of 270) reported that virtual interviews were not employed routinely prior to the 2020-2021 recruitment season. Compared to the 2019-2020 recruitment season, a majority of program directors reported receiving more applications from US medical school students (59.1%, 159/269) and offering more interviews (61.5%, 166/270). The median percentage increase in categorical position interviews offered was 20% (mean: 22, standard deviation: 16.3). Sixty-five percent of respondents (174 of 270) reported that fewer applicants declined or canceled interviews compared to the prior year.

Applicant favorability varied, defined by asking where on the rank list their last matched applicant was compared to their program's average match over the past 3 years. Of program directors who could answer this question, 32.8% (87/265) reported that the last matched position was more favorable, 37.7% (100 of 265) reported it was about the same, and 29.5% (78 of 265) reported it was less favorable. Reported favorability of the last matched position varied by geographic region (Table 2 ) but did not vary by median program size, type, or program director's tenure (median years in position) (data not shown). For program directors who reported less favorable matches, most (85.9%, 67 of 78) attributed that outcome to a “moderate” or “great” extent to virtual recruitment. For program directors who reported more favorable matches, a minority (27.6%, 24 of 87) attributed that outcome to a “moderate” or “great” extent to virtual recruitment. Overall, 81.2% of program directors (216 of 266) reported that they were similarly or more satisfied with their incoming intern class compared with the past 3 years; the responses of which did not vary by program type (Appendix A, available online).

Table 2

Regional Differences in the Location of Last Matched Applicant on the Rank List (2020-2021 Recruitment Cycle) Compared to the Previous 3 Years

Residency Program US Census Region	Much More Favorable/More Favorable (n = 87)	About the Same (n = 100)	Less Favorable/Much Less Favorable (n = 78)	Total (n = 265)	P Value*
Midwest	18 (20.7)	21 (21.0)	21 (26.9)	60 (22.6)	.552
Northeast	26 (29.9)	42 (42.0)	12 (15.4)	80 (30.2)	.035
South	36 (41.4)	22 (22.0)	27 (34.6)	85 (32.1)	.035
West	7 (8.1)	15 (15.0)	18 (23.1)	40 (15.1)	.019

Note: For 265 of 270 respondents; an additional 5 respondents reported “Unsure or Not applicable.”

Design-based Pearson χ2 goodness-of-fit test with 2 degrees of freedom.

Virtual recruitment affected what program directors were able to convey about their programs and what they learned about applicants. Most program directors indicated that virtual recruitment reduced their ability to showcase their program's facilities (89.9%, 240/267), city (87.7%, 235/268), and esprit de corps (82%, 219/267). Program directors (71.3%, 189/265) reported a decreased ability to determine which applicants had a sincere interest in their program. In contrast, the majority of program directors reported that the ability to highlight their program's defining features was unchanged (50.3%, 135 of 268) or had increased (14.6%, 39 of 268).

Excluding program directors who responded “unsure or not applicable,” almost all (94%, 250 of 266) reported a somewhat or substantial decrease in recruitment costs, and 40.9% of program directors (108 of 264) reported increased ability to get faculty to conduct interviews. Slightly less than half of program directors (46.6%, 125 of 270) reported investing more of their time for interview days.

Although a majority of program directors (74.4%, 200 of 269) reported to favor or strongly favor a guiding set of principles on whether internal medicine recruitment should be conducted virtually, in-person, or in a combined fashion, there was not broad consensus about what those principles should be (Appendix B, available online). A minority of program directors endorsed a fully virtual (5.6%, 15 of 269) or a fully in-person (7.4%, 20 of 269) recruitment process. Twenty-one percent (55 of 269) of program directors expressed that programs should make their own determinations and the remainder (28.6%, 164 of 269) preferred a hybrid option. Excluding respondents who reported “unsure,” more than half reported that if unrestricted travel and hospital visitors are allowed for the next recruitment season, they would retain virtual interviews with faculty (59.7%, 151 of 253), videos featuring the program or city (88.5%, 224 of 253), and a fully virtual interview option (61.3%, 155 of 253).

Program directors reported that virtual recruitment contributed to greater (79.5%, 214 of 269) application inflation. Two-thirds (68%, 183 of 269) agreed with the statement that internal medicine residency programs should pursue voluntary preference signaling, whereas 11.2% (30 of 269) reported “neutral,” 16% (43 of 269) disagreed, and 4.8% reported “unsure.”

Discussion

This national survey of internal medicine program directors demonstrated that although virtual recruitment was new to most residency programs, the vast majority (more than 80%) of respondents indicated satisfaction with their incoming class of interns. Respondents reported more applicants to their programs (compared to the previous recruitment season), an increased number of interviews conducted, fewer canceled or declined interviews, and the perspective that virtual interviewing contributed to application inflation. Among program directors who reported less favorable match outcomes, virtual recruitment was perceived to be a contributing factor and, conversely, less so for program directors who reported more favorable match outcomes. Benefits of virtual recruitment were several and included lower costs for nearly all programs and increased ease of faculty participation in recruitment. However, almost half of responding program directors invested more of their time on the interview day. Program directors indicated that it was more challenging to communicate key aspects of their program such as esprit de corps. Although program directors did not express agreement about what principles should shape recruitment in the future, many planned to retain some aspects of virtual recruitment.

Our results are in keeping with the results of a 2021 National Residency Matching Program (NRMP) cross-specialty survey and preliminary 2021 ERAS data on internal medicine recruitment. Broadly, approximately one-third of program directors indicated that their programs received more applications, invited more applicants, and had fewer cancellations during the 2020-2021 recruitment cycle. For internal medicine residency and many internal medicine subspecialty fellowships, the number of applicants further increased in 2021. Internal medicine applications rose by 6% and the mean number of applications submitted by an internal medicine applicant increased by 11% compared to the prior year.

Similar to internal medicine program directors perspectives in our study, about three-quarters of all residency applicants in a 2021 National Residency Matching Program survey found it challenging to develop an understanding of program culture and degree of fit. It remains to be seen what effect this has on resident satisfaction over time. Despite these issues, internal medicine program directors reported satisfaction with their match results. Program directors of colon and rectal surgery programs noted a high degree of satisfaction with match outcomes. Otherwise, few studies have reported program directors perceptions of their match results.

It is interesting that program directors who reported less favorable matches attributed it to virtual recruitment in contrast to those who reported more favorable matches and perceived that virtual recruitment played less of a role. Attributing performance of the virtually recruited intern classes solely to virtual recruitment (favorably or unfavorably) overlooks many confounding factors and likely focuses on an obvious but not comprehensive assessment of trainees.

Virtual recruitment likely benefits some programs and not others. It may also explain the variety of opinions reported by program directors about best practices moving forward: For program directors who believed there should be a national policy, only 5.6% expressed a preference for a purely virtual process, suggesting that getting to know applicants and “selling” the culture, facility, and city is important.

Cost savings for applicants and programs are one of the major advantages of virtual recruitment. In this study, virtual interviewing was not associated with a decrease in program directors time spent on interview days, which may be due to the newness of virtual recruitment for 90% of programs. Additionally, the environmental footprint of recruitment was substantially reduced as carbon emissions are estimated to be 51,665 metric tons of carbon dioxide per year for residency interviews across the United States.

A majority of internal medicine responding program directors favored implementation of some form of signaling preference pathway in response to ongoing application inflation, a problem seemingly exacerbated by virtual recruitment. A signaling preference pathway—proposed as a solution to application inflation—was piloted by otolaryngology in 2020-2021 through a voluntary, free preference-signaling program. A previous study demonstrated that two-thirds of internal medicine program directors were interested in having a mechanism for applicants to indicate a high level of interest. For the 2022 recruitment season, the Association of American Medical Colleges (AAMC) is piloting a voluntary supplemental ERAS application in surgery, dermatology, and internal medicine that will include preference signaling for up to 5 programs.

To our knowledge, this study is the first to describe internal medicine program director experiences with virtual recruitment and to study their opinions about program outcomes and future plans. However, our study has limitations. The 2020-2021 recruitment season was the first conducted virtually; practices and program directors opinions may change substantially over time. Additionally, not only were program directors modifying their recruitment practices, but they were also responding to a pandemic in which their trainees were on the front lines of care. This pressure may have significantly influenced their experience with and the implementation of virtual recruitment. Although the survey response rate was generally representative of the underlying population, possible nonresponse bias does not allow the results to be completely reflective of the perspectives and experiences of all internal medicine program directors at the time of the survey.

Now that there is more planning time and given that a hybrid recruitment approach may be possible in the future, further studies are needed as program directors refine their approach to recruitment and incorporate virtual and in-person elements. Further, virtual recruitment's impact on diversity and equity in internal medicine residency selection is not yet known.25, 26, 27

In summary, virtual recruitment offers significant cost savings for residency programs and applicants as well as reduced environmental impact, and most program directors reported satisfaction with the outcome. However, it may be more difficult with virtual recruitment to convey all program traits and to assess the extent of applicant interest. Future internal medicine recruitment will be shaped by the experiences of the 2020-2021 recruitment season. Like some other specialties,28, 29, 30, 31, 32, 33 many internal medicine program directors plan to incorporate virtual interviews into future recruitment cycles. This shift suggests that a hybrid approach may become the norm even after the COVID-19 pandemic ends. Application inflation remains a major challenge for both internal medicine programs and applicants and ostensibly was exacerbated by virtual recruitment. The upcoming ERAS supplemental application pilot is an important effort to address this challenging issue, although multiple strategies are likely needed.

	University-based (n = 105)	Community-based (n = 37)	Community-based, university-affiliated (n = 120)	Total (n = 262)	P Value*
Much more/somewhat more satisfied	39 (37.1)	14 (37.8)	44 (36.7)	97 (37.0)	.989
About the same	49 (46.7)	12 (32.4)	54 (45.0)	115 (43.9)	.229
Somewhat more/much more dissatisfied	17 (16.2)	11 (29.7)	22 (18.3)	50 (19.1)	.104

For 262 of 270 respondents (4 nonrespondents). An additional 4 respondents reported “Unsure/Unable to answer.” Excludes 4 military-based programs due to small cell sizes/data confidentiality.

Design-based Pearson χ2 goodness-of-fit test with 2 degrees of freedom

	n = 269
Recruitment should be only in person (if safe due to COVID-19)	20 (7.4)
Virtual option should be offered even if in-person is the norm again	77 (28.6)
Recruitment should remain virtual with option for subsequent onsite visit	87 (32.3)
Recruitment should be only virtual	15 (5.6)
Each program should be allowed to make its own determination	55 (20.5)
Unsure	15 (5.6)

Notes: For 269 of 270 respondents; 1 nonrespondent.

The survey question asked “If there is general agreement in favor of a guiding set of principles, which of the following would you endorse?”