The Case for Capping Residency Interviews.

OBJECTIVE: To determine 2020 residency cycle application practices and to model potential consequences in the 2021 cycle if (1) applicants scheduled an uncapped number of interviews; (2) applicants were capped to schedule 12 interviews; (3) residency programs kept their number of interview offers unchanged; and (4) programs increased their interview offers by 20%. DESIGN AND
SETTING: The authors sent an anonymous survey to all obstetrics and gynecology applicants registered through the Electronic Residency Application Service in February 2020 asking respondents to share demographics and number of interview offers received and completed. Based on prior estimates that 12 interviews are needed to match in obstetrics and gynecology, respondents were divided into Group 12+ (those receiving ≥12 interview offers) and Group <12 (those receiving <12 offers). Model assumptions were (1) applicants can complete all interviews they are offered because they are virtual; (2) interview offers that applicants in Group 12+ decline are subsequently offered to applicants in Group <12; (3) the proportions of interviews offered to Group 12+ and Group <12 will remain the same if programs chose to increase their total number of interview spots. PARTICIPANTS: Among 2508 applicants, 750 (30%) provided the number of interview offers received and completed: 417 (56%) in Group 12+ and 333 (44%) in Group <12.
RESULTS: In models where applicants are uncapped in the number of interviews, Group <12 applicants receive <1 interview offer, even if programs increase the number of interviews offered and performed. If applicants are capped at 12 interviews, Group <12 applicants will receive 9 interview offers on average and will reach 12 if programs increase the number of interviews offered by 20%.
CONCLUSIONS: This work highlights how current inefficiencies may lead to negative consequences with virtual interviews. Interview caps and preference signaling systems need to be urgently considered.

INTRODUCTION

As a result of the COVID-19 pandemic, residency programs will make an abrupt shift to virtual interviews in the 2021 residency application cycle. This change adds to the existing uncertainty and insecurity for all stakeholders in the application process. Medical students, medical schools, and residency programs have needed to react to sudden developments such as cancelled clinical electives, delayed or cancelled United States Medical Licensing Exams (USMLE), significant limitations on visiting student elective and sub-internship rotations, and changes in Electronic Residency Application Service (ERAS) timelines. Given this context, applicants may opt to increase their total number of residency applications as well as interviews accepted and completed, especially since they will no longer be limited by travel and cost deterrents.5, 6, 7 Likewise, residency programs are no longer logistically restrained to configure an applicant's interview schedule on a single day, and will have the ability to schedule interviews throughout multiple days and during non-business hours. In-person interactions provided by traditional interview day experiences have historically weighed heavily in determining mutual compatibility , ; thus, both stakeholder groups will be looking to raise their chances of finding a match, including potentially increasing the number of interviews.

This is particularly troublesome given the current state of residency application processes, fraught with misaligned stakeholder incentives.10, 11, 12, 13, 14, 15 Although the ratio of positions per applicant is higher now than ever before, the number of applications per applicant have risen. , These numbers have increased rapidly in certain specialties, with the mean number of applications per applicant in obstetrics and gynecology (OBGYN) rising from 28 in 2010 to 66 in 2019. The consequences of application inflation are numerous and include decreased abilities for residency programs to perform holistic review of applicants with increased reliance on metrics such as USMLE scores. Residency programs also need to devote significant faculty and administrative time for the interview processes. The consequence of application inflation that will be of crucial importance this application cycle is the growing awareness that a small percentage of applicants has been receiving a disproportionate percentage of interview offers. , In the era of virtual interviews, if these applicants choose to schedule all of their interview offers, there is a real potential for detrimental downstream effects to other applicants. This may also lead to a greater number of unfilled residency spots, with a larger number of programs and applicants required to enter into the Supplemental Offer and Acceptance Program. Given the paucity of data to inform best practices, there is a pressing need to model the potential effects of current application processes and applicant strategies in this disrupted application cycle.

The goals of this project were to determine 2020 application practices for OBGYN applicants and to model potential consequences to the entire applicant pool in the 2021 application cycle if: (1) applicants were able to schedule an uncapped number of interviews; (2) if applicants were capped to schedule 12 interviews; (3) if residency programs kept their number of interview offers unchanged; and (4) if programs increased their interview offers by 20%.

MATERIAL AND METHODS

In 2020, the American Medical Association (AMA) awarded the Association of Professors of Gynecology and Obstetrics and the Council on Resident Education in Obstetrics and Gynecology organizations a Reimagining Residency grant to improve the medical school to residency transition for applicants entering into OBGYN residencies. The grant team developed a needs assessment surveys to query applicants on their perspectives and experiences pertaining to residency application processes. This survey was anonymous and asked respondents to share application profile information (including USMLE scores and participation in the Couples Match), how many interview offers they received, and how many interviews they completed (the full survey is available Appendix A). This survey was sent in February 2020 to all OBGYN applicants registered through ERAS. The University of Michigan Institutional Review Board deemed the study to be exempt from regulation (HUM00177624, February 2020).

Modeling Description

Based on prior data that applicants with 12 programs on their rank order list are highly likely to match into an OBGYN Postgraduate Year One (PGY-1) position for both U.S. Allopathic and Osteopathic Seniors, , we divided respondents into Group 12+ (those receiving ≥12 interview offers) and Group <12 (those receiving <12 interview offers). The number of interviews completed served as a proxy marker for the number of programs ranked since applicants rank most programs where they interview.

Total Number of Interview Offers (TNIO) is the sum total number of interview offers reported. Total Number of Interviews Completed (TNIC) is the sum total number of interviews completed reported.

Modeling Assumptions

Given that residency programs have a set number of interview spots, we assume that any interview declined by individuals in Group 12+ is then offered to individuals in Group <12. In a virtual interview scenario, we assume that applicants can accept every interview offered (i.e., TNIC can equal TNIO). If residency programs begin offering interviews in mid-November, then there are 48 possible interview days from November 23rd 2020 to January 15th 2021. Even if applicants schedule only 1 interview per day, then it is theoretically possible for applicants to accept every interview offered. If applicants in Group 12+ are uncapped in the number of interviews they are able to accept, then TNIO (Group 12+) will remain unchanged and we will assume that TNIC will equal TNIO (Group 12+). If applicants are capped in the number of interviews they can accept, then the TNIC (Group 12+) will equal the interview cap times the total number of applicants in Group 12+. The Interview Differential will be the difference between the TNIC (Group 12+) specific for the model and the TNIC (Group 12+) baseline. Assuming that the overall TNIO remains unchanged, the TNIO for Group <12 will either increase or decrease by the Interview Differential.

If programs are able to increase their total number of interview spots by 20%, we assume that the proportion of interviews offered to Group 12+ and Group <12 will remain the same. Each interview “day” completed will involve, at bare minimum, 2 hours of faculty time.

RESULTS

Demographics and Interview Behaviors

Of the 2508 OBGYN applicants, we received responses from 904 (36% response rate [RR]). Of those, 750 respondents provided information about the numbers of interview offers received and accepted (RR: 30% overall, 83% of survey respondents). Mann-Whitney U tests demonstrated minimal differences in distribution between respondents to the survey and participants in the National Residency Matching Program (NRMP) survey according to USMLE/Comprehensive Osteopathic Licensing Exam score and applicant status (e.g., U.S. Senior MD, DO, etc.) and are included in Table 1 . Approximately 10% of respondents (75) participated in the Couples Match.

TABLE 1

Demographics of Survey Respondents Compared to National Resident Matching Program Data

Question	Survey Respondents	NRMP	Mann-Whitney U	p Value
Please check your USMLE Step 1 or COMLEX score*			12.5	1.00
USMLE<200	33 (4)	74 (5)
USMLE 200-220 or COMLEX 488-575	189 (25)	433 (29)
USMLE 221-240 or COMLEX 576-660	337 (44)	644 (43)
USMLE 241-260 or COMLEX 661-742	186 (25)	318 (21)
USMLE >260 or COMLEX >742	15 (2)	24 (2)
Please check the status that applies to you when you submitted your application in the 2019-2020 residency application cycle†			24.5	.442
US Senior MD	510 (67)	1292 (64)
US Alum MD	13 (2)	NA
US Senior DO	130 (17)	338 (17)
US Alum DO	3 (0)	NA
IMG US	48 (6)	122 (6)
IMG US Alum	4 (1)	NA
IMG Non-US	42 (6)	109 (5)
IMG Non-US Alum	12 (2)	NA

Data presented as n (%).

NRMP data from 2018; includes US MDs, US DOs, US IMGs, and non-US IMGs (USMLE data only).

NRMP data for from 2020 data; % from total number of applicants from 2014 data.

Abbreviations: NRMP, National Resident Matching Program; USMLE, United States Medical Licensing Exams; IMG, international medical graduate; NA, not available.

The mean number of interview offers per respondent was 13.3, and the mean number of interviews completed was 10.8. Applicants participating in the Couples Match had higher mean numbers of interview offers and interviews completed than non-Couples Match participants (17.5 vs 12.8 and 13.3 vs 10.5 respectively). Responses divided by Couples Match participation status are described in Appendix B.

More than half (417, 56%) received ≥12 interview offers (Group 12+), with the remaining (333, 44%) receiving <12 interview offers (Group <12). A small percentage of respondents received a very large number of interview offers (Table 2 ). For example, the 6% of respondents who received 30 to 45 interview offers received 15% of all interview offers and completed 11% of all interviews completed. These individuals completed a mean number of 19.7 interviews. Respondents from Group 12+ completed a mean of 14.8 interviews per applicant, while respondents from Group <12 completed a mean of 5.8 interviews per applicant.

TABLE 2

Self-Reported 2019 to 2020 Residency Cycle Interview Offers and Completions by Survey Respondents (n = 750)

Number of Interviews Offered	Number of Respondents	TNIO	TNIC	Mean Number of Interviews Completed
<3	71 (9)	74 (1)	71 (1)	1.0
3-5	71 (9)	298 (3)	292 (4)	4.1
6-9	122 (16)	911 (9)	874 (11)	7.2
10-14	185 (25)	2237 (22)	2085 (27)	11.3
15-19	143 (19)	2392 (24)	2063 (27)	14.4
20-24	80 (11)	1727 (17)	1248 (16)	15.6
25-29	34 (5)	877 (9)	596 (8)	17.5
30-45	44 (6)	1453 (15)	865 (11)	19.7
Total	750	9969	7731	–

Data presented as n (%) or n unless otherwise noted.

Abbreviations: TNIO, total number of interviews offered; TNIC, total number of interviews completed.

Modeling for 2020 to 2021 Cycle

Given that the respondents of our survey were generally representative of the overall OBGYN applicant pool, we scaled our 30% RR data (n = 750) to represent the entire pool of OBGYN applicants in ERAS (n = 2500). Based on this scaling, the TNIO for the entire applicant pool would be 33,230 and the TNIC would be 26,980. Given that nationally there were 1443 PGY-1 OBGYN applicant positions, the number of interview offers per PGY-1 position was 23 and the number of interviews completed per PGY-1 position was 18.7. See Table 3 for scaled modeling to the entire applicant pool.

TABLE 3

Scaled Calculations for the 2020 Application Cycle and Models 1 to 4 for the 2021 Application Cycle

Group	2020 Cycle	Model 1*	Model 2†	Model 3‡	Model 4§
<12║
TNIO	6703	766	10,613	920	16,072
TNIC	6430	766	10,613	920	13,320
Interviews Offered/Applicant	–	0.7	9.6	0.8	14.5
Interviews completed/applicant	–	0.7	9.6	0.8	12
12+¶
TNIO	26,527	26,527	26,527	31,832	31,882
TNIC	20,590	26,527	16,680	31,832	16,680
Interview differential	–	−5937	+3910	−7124	+8028
Interviews offered/applicant	–	19.1	19.1	22.9	22.9
Interviews completed/applicant	–	19.1	12	22.9	12

Applicants Uncapped, Programs Unchanged.

Applicants Capped at 12, Programs Unchanged.

Applicants Uncapped, Programs Increased Interviews by 20%.

Applicants Capped at 12, Programs Increased by 20%.

<12 interviews offered.

≥12 interviews offered.

Abbreviations: TNIO, total number of interviews offered; TNIC, total number of interviews completed.

Models 1 and 2 assume that programs offer an unchanged number of interviews compared to the prior year. For Model 1, if the number of interview offers that may be completed is not capped, then we assume that applicants in Group 12+ complete 100% of their interview offers. The Interview Differential will be 5937 fewer interview offers available to applicants in Group <12. Model 2 examines the consequences if applicants are capped at completing 12 interviews. The Interview Differential in this situation will be 3910 additional interview offers available to Group <12.

Models 3 and 4 examine consequences if programs opt to increase their number of interview offers by 20%. The TNIO would then be 39,876 and the TNIC would be 32,376. The number of interview offers and interviews completed per PGY position would be 27.6 and 22.4, respectively. Model 3 demonstrates that if applicants are uncapped in the context of this 20% increase, and assuming that applicants in Group 12+ complete 100% of their interview offers, the Interview Differential will be 7124 fewer interview offers available to Group <12. Model 4 demonstrates that if applicants are capped at 12 in the context of the 20% increase in interview offers, the Interview Differential will be 8028 additional interviews available to Group <12. Group 12+ applicants and Group <12 applicants will then all be able to complete 12 interviews. See Figure 1 for Models 1 to 4.

FIGURE 1

Four models for the 2020 to 2021 application cycle.

In Model 3, the TNIC of 32,752 is 5732 additional interviews compared to the 2019 cycle. This translates into each of the 250 OBGYN residency programs performing an average of 22.9 additional interviews. Assuming that each interview will involve a minimum of 2 hours of faculty time, each program will need an additional 46 hours of faculty time. Using the median salary of $287,000 for an Assistant Professor of OBGYN and assuming a 58-hour faculty work week, this translates to $95 per hour of faculty time. Thus, 23 more interviews would translate to an additional $4377 per program and a collective $1,094,330 for all residency programs. In Model 4, the TNIC of 30,000 is 2980 additional interviews compared to the 2019 cycle. This would translate to an additional 12 interviews and $2280 per program, with a collective $570,000 for all residency programs. Moreover, our calculation does not take into account the losses in faculty productivity and fixed clinic costs that result from the allocation of this additional faculty time to residency interviews.

Models 1 and 4 assume that applicants in Group 12+ complete 100% of their interview offers, however it is possible that applicants may not be able to actually complete all offers secondary to logistical constraints such as conflicting interview offers on the same day. For Model 1, if applicants in Group 12+ completed 90% of their interview offers, then the Interview Differential would be 323 fewer interview offers available to applicants in Group <12, and Group 12+ would complete 17.1 interviews/applicant, and Group <12 would complete 2.8 interviews/applicant. In Model 4, if residency programs increased their interview offers by 20%, and applicants in Group 12+ completed 90% of their interview offers, then Group 12+ would complete 20.6 interviews/applicant, and Group <12 would complete 3.4 interviews/applicant.

CONCLUSIONS

Using responses from OBGYN applicants in the 2020 residency application cycle helps to shed light on potentially serious negative implications of an unrestricted virtual application cycle. In our 2 models in which the number of interviews an applicant can complete are uncapped, nearly half of applicants could receive less than 1 interview on average. This was true even if programs chose to respond to the uncertainties of this application cycle by increasing their number of interviews offered.

Our data from the 2020 residency application cycle highlight the inefficiencies for applicants and residency programs even prior to the COVID-19 disruption. Applicants who received a large number of interview offers were completing more interviews than needed. This was occurring despite ample evidence of the diminishing returns of additional interviews above certain thresholds.22, 23, 24 The problematic consequence of these actions is that residency programs are interviewing this same small pool of competitive applicants. Our work is consistent with published literature regarding the Otolaryngology Match , ; in that specialty, the record number of unfilled residency positions in 2017 and 2018 was partially attributed to programs all vying for the same very narrow pool of candidates. Although NRMP data demonstrates that OBGYN residency programs need to rank 4 positions per PGY-1 spot, our work shows that programs are completing 18.7 interviews per PGY-1 position. These practices are not only detrimental to the entire application process, but are also costly in terms of the faculty time required for interviews. , For 2021, programs will no longer need to provide applicants with welcome receptions and interview day meals; however, these offerings make up only a small portion of the overall interview day costs to programs. Our models demonstrate the bare minimal financial implications of the increased faculty time that would be necessary for residency programs to increase interviews, which needs to be taken into account—especially during these times of economic challenges for clinical departments.

All stakeholders urgently need equitable solutions that address both individual and systems-level problems for this coming application cycle and beyond. Capping the number of interviews that an applicant can schedule could remedy 1 pressing flaw in current application interview processes. Implementing caps at the interview scheduling stage is preferable to capping at the application stage given the multiple complexities that must be considered such as DO and IMG status, and overall competitiveness. In addition, exceptions may need to be made for individuals participating in the Couples Match. The potential legal implications of mandatory interview caps are in the infancy of exploration. Capping interviews would likely not violate anti-trust laws given that applicants would still have the choice of where they would like to interview, however these issues would need to be further investigated. New measures such as preference signaling mechanisms30, 31, 32 need to be urgently considered in order for programs to be able to prioritize whom to offer interviews. The use of “tokens” would enable applicants to be able to convey interest to a set number of programs; this has been well-described in graduate PhD economics education literature. It will be imperative for “fit” to not become a proxy for decisions guided by unconscious bias, but instead, for principles of equity and inclusion to guide change during this time of accelerated change.

The provocative nature of our modeling assumptions, as well as the limitation of our models, needs to be acknowledged. Applicants in Group 12+ may not complete 100% of their offered interviews, either due to logistical constraints or by choice. Given that the applicants who received a large number of interview offers in 2020 did indeed complete a disproportional number of interviews, this assumption may prove to be accurate in 2021. Regardless, it is important to model this possibility early in this 2021 application cycle in order to inform conversations both locally with individual applicant counseling, and nationally for specialty-specific and large medical education stakeholder bodies. We also assume that programs will keep the number of interview offers stable, and that increased offers completed by Group 12+ would directly result in decreased interview offers available for Group <12. This modeling did not take into account the strategies that individual programs may use to try to decide which applicants to offer interviews. For example, programs may consider factors such as geography in interview offer decision-making that could lead to more offers for individuals in Group <12. Given our overall 30% response rate, response bias was likely present, and individuals receiving either large or small numbers of interviews may have been more motivated to complete our survey. However, given the similarity of their academic profiles to the NRMP data for the same cycle, it is likely that responses are representative of OBGYN applicants. In addition, we chose to administer the survey before the Match in order to capture responses before individuals were aware of their match results. Therefore, our assumptions were made based on interview offers and probability of matching based on retrospective data, but not on the match results of this group. Future work will need to link data such as ours to actual match results. Future modeling will also need to investigate potential benefits or detrimental effects of changes to applicants participating in the Couples Match.

In conclusion, the burden of responsibility in implementing and enforcing interview limitation guidelines needs to be shared between specialty societies, medical schools, national organizations, and applicants themselves. Data from this work can inform conversations for the short term, as well as longer-term discussions about how to meaningfully improve the application processes for the future.