COVID-19's Impact on the 2020-2021 Resident Match: A Survey of Otolaryngology Program Directors.

OBJECTIVE: To determine the impact coronavirus disease of 2019 (COVID-19) will have on the 2020-2021 otolaryngology (OTO-HNS) resident application cycle.
METHODS: A cross-sectional survey targeting OTO-HNS program directors (PD) was created and disseminated via email to PDs on May 28th 2020. Descriptive analyses of the 19-question survey was performed, and free text responses for certain suitable questions were thematically categorized into groups determined to be relevant during analysis.
RESULTS: Twenty-nine of 123 solicited PDs (23.6%) completed the survey. Nineteen (65.5%) respondents indicated they would not host away rotations (AR) in 2020, and 9 (31.0%) reported that they would consider away rotators without home programs. Regarding the historical importance of AR, 21 (72.4%) PDs stated they were either "extremely" or "very" important in evaluating candidates. Sixteen (55.2%) PDs stated that virtual interviews would impact their ability to properly gauge candidates and 12 (41.4%) were unsure. Eight PDs (27.6%) stated their evaluation of candidates will likely change, with a shift toward an increased reliance on letters of recommendation, research involvement, and clerkship grades. The large majority of PDs-25 (86.2%)-were not worried that the COVID-19 pandemic would affect the abilities of new interns beginning in 2021.
CONCLUSION: Virtual interviews and engagement activities will mostly supplant sub-Is and AR for the 2020-2021 OTO-HNS application cycle. Surveyed PDs largely believe these will be insufficient in providing a comprehensive assessment of candidates, and will similarly limit applicants' ability to gauge residency programs. Criteria utilized to evaluate students is expected to change.

Introduction

Otolaryngology—head and neck surgery (OTO-HNS) is among the most competitive residencies in the United States.[1,2] Demands on applying students have included high United States Medical Licensing Examination (USMLE) step 1 scores, program-specific letters of interest, and rising levels of scholarly output, which has contributed to an “impossible to match” stereotype for OTO-HNS residency.[2-6] As a response, medical students have begun to apply more broadly, with a recent survey showing that candidates submit a median of 60 applications through the Electronic Residency Application Service (ERAS), the centralized service through which applicants and programs communicate for matching into residency in the United States.[7] Furthermore, there is evidence that interviews are being amassed among relatively few applicants,[8] resulting in potentially worse match outcomes for both students and programs.[9]

The coronavirus disease of 2019 (COVID-19) pandemic has produced a dramatic transformation in healthcare in an attempt to protect both patients and providers.[10,11] These changes have extended to medical education, with recent recommendations by the Association of American Medical Colleges (AAMC) and otolaryngology governing bodies encouraging restricted student-patient interactions, recommending against away sub-internships (aways) at outside medical institutions, and urging for virtual residency interviews.[12,13] The impact on the 2020-2021 (henceforth referred to as 2020) application cycle remains unclear, though traditional methods of ascertaining an applicant’s candidacy will undoubtedly be altered.

The primary objective of this study is to investigate COVID-19’s perceived impact on the upcoming 2020 application cycle from the residency program perspective using a cross-sectional survey. Secondarily, we aimed to determine changes to programs that have already occurred secondary to COVID-19. In elucidating residency program directors’ approaches to this unique application cycle, as well as documenting changes and concerns of significance to programs, we hope to promote a more uniform and transparent application process for residency programs and medical students alike.

Methods

Survey Creation and Content

We utilized the Yale Qualtrics Survey Tool (Qualtrics, Provo, UT) to create a cross-sectional survey (Supplemental Appendix A). Our target population included OTO-HNS residency program directors (PDs). The Yale Human Investigations Committee deemed this study exempt due to the anonymous nature of the survey.

In brief, the 19-question survey (estimated completion time of 5 minutes), consisted of questions investigating the impact of the COVID-19 pandemic on their program and their current residents, whether their program was planning on hosting away rotators, the historical importance of away rotations, their approach to candidate evaluation for the coming cycle, and their strategies for conducting virtual interviews. Finally, we asked for any concerns they envision for this application cycle, and whether they had advice for applicants. Seventeen (87.5%) questions were multiple choice, with 10 of these (58.8%) including an optional free text box to expand. Two (12.5%) questions were optional free text responses.

Survey Dissemination

The survey was disseminated via email by the senior author (R.P.M.) on May 28th, 2020, with a follow-up email on June 8th, 2020. PD emails (or, in some cases, that of their administrative assistant) were obtained from the FREIDA online database. The survey was closed June 12th, 2020, open for approximately 15 days.

Survey Analysis

We provide descriptive analyses on each question. After survey closure, free text responses for certain suitable questions were thematically categorized into groups determined to be relevant during analysis by 2 separate authors (S.J.T. and S.I.). Any disputes were settled via discussions. No significance testing was performed.

Results

Characteristics of Programs Surveyed

Thirty-eight individuals clicked past the first page (intro). Of these, 29 (76.3%) completed the survey. The remaining respondents either did not complete the survey or indicated they were not PDs. As there were 123 programs listed in FREIDA, 29 responses accounts for 23.6% of all PDs.

Of 29 PDs, all (100.0%) indicated that their program was affiliated with an academic institution and have historically offered away rotations to visiting sub-interns. Most responding PDs (15 [51.7%]) were from midsized programs (admitting 3-4 interns). When asked to self-report whether they felt their program was heavily impacted by the COVID-19 Pandemic, 13 (44.8%) felt their program was heavily impacted. In a thematic grouping of the ten who felt that their program was affected and provided free text responses, all 10 (100.0%) indicated decreased volume (Table 1).

Table 1.

Program Characteristics.

	All Participants (n = 29)
Affiliated with an academic institution
Yes	29 (100.0%)
No	0 (0.0%)
Number of interns to join following 2020-2021 application cycle
1-2	10 (34.5%)
3-4	15 (51.7%)
≥5	4 (13.8%)
Historically offered away rotations to visiting sub-interns
Yes	29 (100.0%)
No	0 (0.0%)
Program heavily impacted by COVID-19 (self-reported)
Yes	13 (44.8%)
No	14 (48.3%)
Unsure/Prefer not to answer	2 (6.9%)
Thematic groupings for those who answered “Yes” and utilized the free text box to explain why (n = 10)
Decreased volume (clinical and/or surgical)	10 (100.0%)
Redeployed residents	2 (20.0%)
Ill residents	1 (10.0%)

Status of Away Rotations and Their Historic Importance

Nineteen (65.5%) respondents indicated they would not host aways for the 2020 application cycle. Eleven of these PDs elaborated with free text with 2 (18.2%) indicating this may change in the future, and 8 (72.7%) indicated this was due to institutional policies. Nine (31.0%) reported that they would consider away rotators without home programs, while only 1 (3.4%) reported their program was open to all students (Figure 1A). When asked about the historic importance of aways in gauging candidacy, 13 (44.8%) stated they were extremely important, 8 (27.6%) stated they were very important, and 5 (17.2%) stated they were moderately important (Figure 1B). When asked if prohibition of aways would affect their ability to gauge applicants, the majority (16 [55.2%]) answered “yes” (Figure 1C). Two of those who answered “no” utilized the free text box to expand, and both stated that they have not had many away rotators in recent years.

Figure 1.

(A) Institutional plans for away rotations. (B) Historic importance of away rotations. (C) Limitations in light of lack of away rotations.

Virtual Interview Limitations and Number of Interview Invites

Sixteen (55.2%) PDs stated that virtual interviews would impact their ability to properly gauge candidates, while 12 (41.4%) were unsure (Figure 2A). The single respondent who stated it would not affect their ability to evaluate candidates elaborated with free text and stated, “we had a positive experience in virtual interviews during the COVID-19 period.” Twenty PDs (69.0%) felt that virtual interviews would limit the ability of applicants to properly gauge the merits of their program (Figure 2B). Fourteen of these 20 utilized the free test box to expand; 11 (78.6%) felt that it would be difficult for applicants to evaluate “fit” among residents and faculty, and 3 (21.4%) indicated it would be difficult for applicants to evaluate the program’s city. In light of virtual interviews, 7 PDs (24.1%) said their programs would interview more applicants. Two of these PDs elaborated with free text, and both stated that this will be due to decreased interview conflicts and decreased expense and time commitment; 1 stated that they “may be able to capture more interviews,” while another stated “we will need to interview more people since applicants will likely interview at more programs.” No program said they would interview fewer people virtually, and 19 (65.5%) stated that their interview invitation numbers will remain unchanged (Figure 2C).

Figure 2.

(A) Limitations of virtual interviews for programs. (B) Limitations of virtual interviews for applicants. (C) Change in number of interview invites.

Plans for Virtual Interviews

Twenty-two (75.9%) PDs indicated that they would offer virtual facility tours, 20 (69.0%) would offer Q&A sessions with the PD/faculty, and 18 (62.1%) would offer virtual chats with residents. Sixteen (55.2%) stated they would offer pre-application engagement activities (such as virtual sub-Is; Figure 3).

Figure 3.

Institutional plans for virtual activities.

Judging Applicants During the COVID-19 Pandemic

In light of the COVID-19 pandemic, 8 PDs (27.6%) stated their evaluation of candidates will change; 7 of these PDs elaborated with free text with considerable variety in responses—from “cannot assess as well without meeting live” and “we [will] add the program specific paragraph back” to “we typically do skills testing, it is unclear if/how we will do that” and “we will rely more heavily on letters of recommendation and letters from medical school deans, as well as personal communications with students’ mentors.” Eleven (37.9%) stated evaluations would not change; 2 utilized the free text box and stated: “we will still try to assess the same qualities we looked for” and “not consciously. However, I imagine that our perception of a candidate may be different compared to a live interview event.” Ten PDs (34.5%) were unsure.

When PDs were asked to select the 3 most important criteria in selected an OTO-HNS candidate in a regular year, 69.0% indicated letters of recommendation in the specialty, 44.8% indicated audition rotations, 41.4% indicated personal prior knowledge of applicants, and 37.9% indicated USMLE Step 1/COMLEX Level 1 scores. When asked the same question instead for a COVID-19-affect cycle, letters of recommendation in the specialty rose to 82.8%, audition rotations fell to 0.0%, personal prior knowledge of applicants fell to 31.0%, and USMLE Step 1/COMLEX Level 1 scores rose to 41.4%. Demonstrated interest and involvement in research rose from 20.7% to 31.0% (Figure 4).

Figure 4.

Three most important factors in judging candidate in a normal and COVID-19-affected year.

PDs View on the Effect of the COVID-19 on New Interns and Current Residents

The large majority of PDs—25 (86.2%)—were not worried that the COVID-19 pandemic would affect the abilities of new interns beginning in 2021. Three (10.3%) were unsure, while only 1 (3.4%) did indicate some concern. Using free text, this PD stated, “I worry about basic exposure to ICU-type learning, and some general surgery procedures/management.”

With regards to current residents, 14 PDs (48.3%) felt that the training and careers of their current residents were affected by the COVID-19 pandemic, 6 (20.7%) were unsure, while 9 (31.0%) did not feel their residents were significantly impacted. Using thematic grouping of the 17 PDs who stated “yes” or “unsure” and utilized the free text box, 12 (70.6%) indicated a potential impact on surgical skills, 1 (5.9%) indicated impact on resident research, and 8 (47.1%) indicated an impact on fellowship interviews or career planning.

Other Difficulties or Concerns, and Advice for Applicants

Nine (31.0%) chose to utilize the free text box to expand on other difficulties they envision during the upcoming application cycle. Using thematic analysis, 4 PDs (44.4%) expanded on how they envision great difficulty in evaluating applicant fit, and 5 (55.6%) had great concerns for applicant interview hoarding, whether or not application limitations should be implemented, and how to evaluate true applicant interest.

Nine (31.0%) PDs, not necessarily the same 9 as above, chose to utilize the free text box to give advice to students. In a thematic grouping, 4 PDs (44.4%) indicated applicants should find a way to signal their preferences to particular programs, 2 (22.2%) indicated applicants should focus on optimizing letters of recommendations, and 3 (33.3%) said applicants need to practice selling themselves and conveying their positive qualities.

The full responses for all concerns and advice may be found in Table 2, and we emphasize that each item represents the view of an individual PD.

Table 2.

Free Responses for Other Envisioned Concerns, and Advice for Applicants.

What Outstanding Difficulties or Concerns Do You Envision for the Upcoming 2020/2021 Application Cycle?	What Would You Advise Applicants to Consider in Order to Maximize Their Success in the Upcoming 2020/2021 Application Cycle?
Finding the right fit.	Good question—at this point I really do not know. They need to truly sell themselves during the virtual interview and may need some practice sessions on virtual interviewing.
Applicants will apply to every program in the control. We all need to agree on a way to limit this potential chaos by decreasing the number of programs an applicant can apply to.	Signal programs of particular interest through emails/calls from supportive faculty or other methods.
Concern for applicant interview hoarding.	Maximize the effectiveness of their LORs.
Appropriateness of a fit within our program.	Try to make personal contacts.
Our program looks for residents who are compatible with the group, and that will be very difficult to assess.	It has been suggested by the OPDO, that they maximize their LOR’s, obtaining in-depth contact with mentors who will be able to provide information about them.
Ability to attract applicants. I fear that there also will be an increased number of submitted applications per applicant, congesting the review process.	Since away rotations are not likely, I am recommending all applicants at our institution complete at least 8 wk of ENT sub-I.
Need to interview more applicants; harder to assess applicant interest.	Reach out to programs that are of particular interest early in the process.
It will likely be [an] increased number of applications sent out per applicant. Residency programs will receive on average, more applications. It would be more difficult to determine how to pick residents.	Contact the top 5 programs you are most interested in early and tell us who you are in your personal statement. What are you passionate about? How are you a hard worker?
Feeling of inadequate gauge of fit for program in both directions.	Understand your screens and what you can bring to the program, as you sell your candidacy.

Note. Answers in columns do not correlate (ie, the PD who wrote the third concern is not the same as who wrote the third piece of advice). Answers were also slightly edited so that they may be grammatically correct. It is important, especially with regards to the advice, to remember that these represent the opinions of a single PD, and may not be representative of broader preferences.

Discussion

COVID-19 has drastically altered healthcare, translating to a reformed patient, clinician, and medical student experience. Students are participating in less clinical activity with decreased exposure to residents and faculty.[12,13] Applicants in previous years, particularly those without OTO-HNS home programs (HP), have reported an increased utilization and reliance on aways, which have been predominantly eliminated during the 2020 application cycle.[13,14] As OTO-HNS is among the most competitive residencies, the decrease in mutual exposure exacerbates the burden for both medical students and residency programs to attain a mutually optimal match. Thus, our goal was to gather perspectives from residency leadership to provide insight into the ongoing adjustment process for residencies and to better guide medical students.

In light of national guidance and recent recommendations from the OTO-HNS governing bodies to limit aways, 65% of responding PDs stated they would not be hosting aways, mostly secondary to institutional policies (Figure 1A). Nine (31%) plan to host aways that do not have a home institution and only 1 program will be “open to all students.” Although one objective of the national OTO-HNS guidance statements was to offer away opportunities for applicants without a HP, potential limits on these opportunities, including a need to maintain close proximity to their home institution, may remain a barrier for such applicants. Moreover, that the majority of PDs indicated they would not consider hosting students without HP suggests that these students will be limited in acquiring the requisite exposure and support necessary for a successful OTO-HNS match.

Contrasting the above with our finding that 72.4% of respondents believe that aways are either “very” or “extremely” important in gauging the candidacy of an applicant, and that over half (55.2%) of the PDs believed that a lack of aways will limit their ability to evaluate a prospective resident, accommodating the lack of exposure that comes with aways will be important (Figure 1B and 1C). One proposed mechanism to manage decreased student exposure is via a virtual sub-I. In addition to facilitating program and student evaluation of their compatibility, this would allow for more individualized mentoring and increased OTO-HNS-based clinical experience, potentially easing the transition into internship.[15] While a virtual sub-I may be inherently limited secondary to its tele-communicative nature, benefits include the opportunity to convey mature interpersonal aptitude and demonstrate clinical acumen over a prolonged period.

As mentioned previously, the potential for few applicants to collect an outsized share of interviews, particularly now without geographic or cost limitations, is a notable concern.[8] Interestingly, although most (55.2%) PDs reported that virtual interviews will negatively impact their ability to assess candidates and limit applicants’ aptitude to “properly gauge the merits of individual programs,” only 24.1% plan to extend more interview invitations, with 65.5% planning to interview the same number of candidates as in previous years. Consequently, applicants hoarding even more invitations may limit the diversity of matriculants, and possibly affect the OTO-HNS applicant match rate and program fill rate. Alternatively, though no PD indicated they would interview fewer applicants, considering limitations in ability to distinguish applicants from one another in a virtual setting, more thorough interviewing of fewer applicants may be beneficial in informing the rank process. Regardless, our findings highlight that, absent application or interview caps or some form of program preference signaling, interviewing the same or fewer number of applicants may lead to match inefficiencies.[16,17] To complement virtual interviews, virtual activities including virtual facility tours, question and answer sessions with faculty, and virtual chats with residents have all been considered (Figure 3).

How programs will synthesize and incorporate the information obtained from a new interview format remains to be seen. Reflecting the restructuring of applicant and program interaction, 27.6% of PDs anticipate integrating changes in the relative importance of historically used metrics for candidate evaluation. For example, an increased significance could be given to letters of recommendation and clerkship grades as opposed to personal knowledge of applicants (Figure 4). Furthermore, relative to previous application cycles, PDs ascribed more importance to “interest and involvement in research,” something which has previously demonstrated to positively affect candidate admission, and is theoretically easier to quantify and compare between applicants.[18] Optimistically, the vast majority (86.2%) of PDs reported no concerns regarding incoming residents training or ability to perform as a result of COVID-19.

Surveying PDs on the impact of COVID-19 on their institutions, approximately 44% responded they had been “heavily impacted,” though nearly half stated they had not been affected (Table 1). Of those who responded their programs had been impacted and chose to expand in a free text box, 100% indicated there had been a decrease in volume, 2 (20.0%) indicated redeployment of residents, and 1 (10.0%) indicated ill residents (Table 1). While decreased clinical volume for OTO-HNS services is an expected outcome of COVID-19, that nearly half of PDs thought the virus had minimal effect on their program is somewhat surprising. In regards to COVID-19’s impact on OTO-HNS residents specifically, almost 50% (14) of PDs replied it had affected residents, including their surgical skill/training, fellowship interviews, and career planning.

Utilizing an opportunity to voice concerns and advise applicants, apprehensions regarding applicants broadly applying and “interview hoarding” was a common theme. (Table 2). Assessing and attracting appropriate applicants was another, unsurprising, concern. Signaling their heightened interest in a particular program, maximizing their letters of recommendation, and practicing interviewing virtually were some of the practical pieces of advice offered to candidates. Reviewing PD concerns and advice would be prudent as applicants prepare for the 2020 application cycle (Table 2).

While our findings are derived from the views and experiences of OTO-HNS PDs, these findings may be germane to other competitive surgical subspecialties facing similar challenges in light of COVID-19. Commentary articles offering prospective guidance from surgical subspecialty leaders for navigating the 2020 application cycle had anticipated many of the challenges highlighted in our study. Importantly, the risk of limited exposure available to both applicants and programs was raised early, although these concerns were offered without the benefit of cross-sectional data.[19] Identification of students without home OTO-HNS programs as being particularly vulnerable in the upcoming cycle was also reported early during the pandemic, and our data highlighting the importance of aways and of letters of reference from established mentors in the field of OTO-HNS in the upcoming cycle is in agreement with this.[20]

The major limitation of this study is response bias. While an effort was made to contact all otolaryngology programs nationally, it is possible we captured a sample not representative of all otolaryngology residency leadership. This was also evident in our response rate, which only captured about 23.6% of all PDs, potentially limiting the generalizability of our results. However, this may be partially mitigated by fairly clear consensus among some survey questions. We were also limited by the answer choices posed, though we attempted to alleviate this by thematically grouping free text responses. Still, not everyone chose to elaborate in the free text box, and the nature of free text prevents assessing how many participants believe in a thematic option. For example, that 4 of 29 (13.8%) respondents report they will have difficulty assessing applicant fit with the program does not mean that 86.2% do not.

Conclusion

The impact of COVID-19 on the 2020 OTO-HNS resident application cycle has yet to be fully elucidated. Virtual interviews and engagement activities are likely to be implemented, and away rotations, although previously highly emphasized in evaluating candidates, will mostly be eliminated. Surveyed PDs largely believed these limitations would hinder their ability to gauge applicants, and similarly assumed applicants will be limited in their ability to assess residency programs. Although candidates may theoretically be able to attend more interviews, most PDs are not currently planning on increasing interview capacity. PDs reported that criteria used for applicant evaluation may change this year, with particular reliance on letters of recommendations.

Click here for additional data file.

Supplemental material, Appendix_A_3 for COVID-19’s Impact on the 2020-2021 Resident Match: A Survey of Otolaryngology Program Directors by David A. Kasle, Sina J. Torabi, Said Izreig, Rahmatullah W. Rahmati and R. Peter Manes in Annals of Otology, Rhinology & Laryngology