Otolaryngology residency programs' perspectives on virtual interviews during the COVID-19 pandemic.

Objective: To evaluate perceptions of otolaryngology residency program directors (PDs) and department chairs (DCs) on virtual interviews (VIs) at the conclusion of the 2021 residency match. Study design: Cross-sectional survey study.
Methods: An anonymous and voluntary survey was administered to PDs and DCs of U.S. otolaryngology residency programs from March 1, 2021 to April 11, 2021. Utilizing Likert scale ratings, the survey measured overall satisfaction with the VI format, perceived ability to assess key candidate attributes, relative importance of traditional candidate metrics, and likelihood to implement VIs in future cycles.
Results: Ninety-four surveys were completed by PDs (48.9%) and DCs (51.1%). Sixty (63.8%) respondents found the VI experience to be satisfactory or strongly satisfactory. Sixty-six (70.2%) respondents felt confident in their ability to assess a candidate's professionalism and communication skills through VIs; however, confidence in gauging an applicant's clinical skills/knowledge (41.2%) and overall program fit (47.3%) was lower. Regarding candidate metrics, 64 (68.1%) respondents believed that personal prior knowledge of the applicant gained increased importance with VIs. Forty-four (46.8%) and 45 (47.9%) participants believed that letters of recommendation in the specialty and perceived commitment to their program similarly attained increased significance in evaluating applicants, respectively.
Conclusion: Most PDs and DCs were satisfied with their VI experiences and expect the quality of interns to be unaffected by virtual assessment modalities. Majority opinion on the relative importance of traditional applicant metrics remained largely unchanged, the exception being an increase in importance of prior knowledge of the applicant.Level of Evidence: N/A.

INTRODUCTION

Otolaryngology has consistently ranked among the more competitive residencies to apply to in the United States. , In light of the coronavirus disease 2019 (COVID‐19) pandemic, the residency application process has undergone drastic shifts from traditional in‐person interviews to exclusively virtual assessment modalities. For the 2020 to 2021 interview cycle, residency programs were required to adapt to a virtual format to follow social distancing guidelines and limit travel during the pandemic. As expected, this major transition has raised many concerns among both programs and applicants regarding the outcomes of the most recent residency match. In a survey of program directors (PDs) conducted prior to the start of the 2020 to 2021 application cycle, over half of the respondents believed that virtual interviews (VIs) would impact their ability to properly assess candidates. This concern appeared to stem largely from the decision to suspend away rotations, which over 70% of surveyed PDs believed to be either “very” or “extremely” important in gauging the candidates' overall competence and program fit.

As the cross‐sectional study by Kasle et al assessed PDs' impressions of the VI process several months prior to the start of the 2020 to 2021 application cycle, it is possible that many of these perceptions have changed over the course of the pandemic. With one VI cycle now complete, it is important to reflect and re‐evaluate these perceptions so as to identify any benefits, challenges, and major obstacles faced. Lessons learned should help improve the process for future application cycles from the perspective of both residency programs and applicants. Using a cross‐sectional survey‐based study, we aim to elucidate both the major strengths and limitations otolaryngology PDs and department chairs (DCs) face in assessing applicants through the virtual format.

MATERIALS AND METHODS

Following approval from the Institutional Review Board of the University of California, Irvine, an online and secure survey was created using REDCap (Nashville, Tennessee). The introductory page of the survey included an informed consent, which was signed electronically by the respondents. The majority of survey questions focused on individuals' perceptions of the VI format; particularly, its effect on their ability to evaluate applicants and convey program qualities to applicants, in comparison to traditional in‐person interviews. PDs and DCs were also asked about the specific criteria used to evaluate an applicant's candidacy, and which of these criteria they perceived to change in weight of relative importance as a result of the VIs. Additionally, respondents were queried regarding the VI process itself, including any challenges encountered during the process. Last, participants were surveyed on specific aspects of their program and their ability to communicate these features to applicants through VIs. The total number of entering interns in the respective otolaryngology programs was queried with the following options: 1 to 2 interns, 3 to 4 interns, and >4 interns. For statistical analysis, this was categorized as small (1‐2 interns), intermediate‐sized (3‐4 interns), and large programs (>4 interns). To prevent missing values, most survey items required an answer (ie, forced response) before the survey could be submitted. The survey, which was voluntary and anonymous, was disseminated by our program coordinator, PD, and senior author (S.I., T.T., and M.A., respectively) to PDs and DCs from all U.S. otolaryngology residency programs listed on Doximity. Responses were collected from March 1, 2021 to April 11, 2021. Presumptive program response rate was calculated as a range based on the minimum and maximum probable representation of eligible otolaryngology programs among our responding cohort of PDs and DCs. PASW Statistics 18.0 software (IBM, Chicago, Illinois) was utilized for descriptive analysis and Pearson's Chi‐squared test of independence, where a P value <.05 was considered significant.

RESULTS

From the 123 eligible otolaryngology programs, 94 individuals completed the survey, with 46 (48.9%) PDs and 48 (51.1%) DCs providing responses. The presumptive program response rate was 39% to 76%. Demographic variables are summarized in Table 1. Eighty‐seven (92.6%) participants represented programs affiliated with an academic institution. The cohort consisted of 32 (34.0%) representatives from programs in the Northeast U.S., 13 (13.8%) in Midwest, 19 (20.2%) in Southeast, 13 (13.8%) in Southwest, and 17 (18.1%) programs in the West U.S. Thirty‐five (37.2%) participants represented small‐sized programs, 46 (48.9%) from intermediate‐sized, and 13 (13.8%) from large programs based on the number of interns to join in 2021. When asked whether their program was heavily impacted by COVID‐19, 61 (64.9%) respondents stated “yes” and 28 (29.8%) stated “no.” Five (5.3%) respondents were either unsure or preferred not to answer. The proportion of surveyed individuals reporting “yes” to this question was significantly different by geographic location (Northeast [75.0%], Midwest [38.5%], Southeast [47.3%], Southwest [61.5%], West [88.2%], P = .04).

TABLE 1

Respondent demographics (n = 94)

Variable	No. respondents
Appointment
Program director	46 (48.9%)
Department chair	48 (51.1%)
Affiliation with an academic institution
Yes	87 (92.6%)
No	7 (92.6%)
Location of program
Northeast	32 (34.0%)
Midwest	13 (13.8%)
Southeast	19 (20.2%)
Southwest	13 (13.8%)
West	17 (18.1%)
Size of program
Small	35 (37.2%)
Intermediate	46 (48.9%)
Large	13 (13.8%)
Program heavily impacted by COVID‐19 pandemic?
Yes	61 (64.9%)
No	28 (29.8%)
Unsure/prefer not to answer	5 (5.3%)

Seventy‐five (79.8%) participants stated that they felt either prepared (n = 50, 53.2%) or very prepared (n = 25, 26.6%) prior to conducting the VIs, with 55 (58.5%) reporting that their program conducted mock VIs prior to conducting official interviews with applicants. Additionally, 73 (77.7%) individuals stated that their program hosted virtual open houses for applicants during the 2020 to 2021 cycle. Twenty‐eight (29.8%) respondents reported that their program offered virtual sub‐internships to students. While 11 (11.7%) respondents reported that their program experienced major technical difficulties at some point during the VI, none of the participants (0%) seemed to deny that their program dealt with technological issues promptly and effectively whenever they occurred. Eighty‐four (89.4%) respondents expressed having no concerns regarding data security or privacy at any point during the VI process and no (0%) individual reported experiencing difficulty in maintaining a quiet interview space, free of disturbances. Concerning applicants, 18 (19.2%) respondents believed that some candidates were at a disadvantage due to limited personal technological and financial resources (eg, poor internet connection affecting video/sound, dropped calls).

When compared to previous in‐person interviews, the time length of each VI session with each interviewer was shorter according to 24 (25.5%) participants, longer according to 10 (10.6%), and the same length according to 60 (63.8%) respondents. Sixty (63.8%) respondents stated that they were either satisfied or strongly satisfied with the overall VI experience and 27 (28.7%) individuals were neither satisfied nor dissatisfied. Seven (7.4%) participants reported being dissatisfied with the VI experience. When asked how likely (0% = highly unlikely and 100% = highly likely) their program was to continue utilizing VIs over in‐person interviews in future application cycles the responses were widely distributed with a mean (range, SD) likelihood of 51.9% (0‐100, 24.37), as shown in Figure 1. Thirty‐one (33%) respondents either agreed (n = 23, 24.5%) or strongly agreed (n = 8, 8.5%) with the statement that VIs allowed them to dedicate more time to their clinical obligations and 31 (33%) either disagreed (n = 28, 29.8%) or strongly disagreed (n = 3, 3.2%) with this statement.

FIGURE 1

Respondents' self‐reported likelihood of continuing virtual interviews in future application cycles. “×” indicates mean

Seventeen (18.1%) respondents believed that the VIs would have a positive impact, while 25 (26.6%) predicted a negative impact on the quality of incoming interns. Fifty‐two (55.3%) participants believed that VIs would have no effect on the quality of incoming interns. In total, 66 (70.2%) surveyed individuals felt either confident or very confident in their ability to assess an applicant's professionalism and communication skills and 70 (74.5%) felt either confident or very confident in assessing an applicant's personal narrative through the VI. When asked about their ability to gauge an applicant's clinical skills/knowledge and overall “fit” with the program, the responses were more varied, with >50% of respondents expressing either a neutral or negative opinion for both measures (Figure 2). This proportion was similarly higher in respondents from programs that conducted mock VIs compared to those that did not (44.4% vs 30.8%, P = .04). The proportion of participants who expressed confidence in assessing a candidate's professionalism and communications was higher among respondents from programs that offered virtual sub‐internships compared to those that did not (85.7% vs 63.6%, P = .04).

FIGURE 2

Respondents' level of agreement with confidence in assessing candidate attributes

Table 2 displays the respondents' perceived change in the significance of various metrics traditionally used in evaluating an applicant's candidacy. Sixty‐four (68.1%) participants believed that personal prior knowledge of the applicant gained increased importance as a result of the transition to VIs. Forty‐four (46.8%) respondents also believed that letters of recommendation in the specialty and 45 (47.9%) believed perceived commitment to their program gained increased importance in evaluating candidates. Additionally, 37 (39.4%) and 31 (33.0%) surveyed individuals perceived USMLE Step 1/COMLEX Level 1 and USMLE Step 2 CK scores to increase in importance, respectively. Table 3 shows respondents' level of agreement with the statement that they felt confident in the ability to convey specific program characteristics through VIs. Apart from program facilities (35.1%) and geographic features (34%), most participants expressed confidence in their ability to communicate their program's quality of medical education and training (74.4%), research opportunities (71.3%), program culture (56.4%), program diversity (58.6%), and resident resources (55.8%). Responses to miscellaneous survey questions regarding the VI experience are tabulated in Supplemental Table 1.

TABLE 2

Respondents' perceived change in relative importance of metrics used to evaluate candidates

Metric	Increase	Decrease	No change
Audition/virtual rotation	24 (26.1%)	16 (17.4%)	52 (56.5%)
USMLE Step 1/COMLEX Level 1 score	37 (39.4%)	0 (0.0%)	57 (60.6%)
USMLE Step 2 CK score	31 (33.0%)	1 (1.1%)	62 (66.0%)
Letters of recommendation in the specialty	44 (46.8%)	1 (1.1%)	49 (52.1%)
Personal statement	22 (23.4%)	1 (1.1%)	71 (75.5%)
Required clerkship grades	20 (21.3%	2 (2.1%)	72 (76.6%)
Perceived commitment to specialty	16 (17.6%)	5 (5.5%)	70 (76.9%)
Personal prior knowledge of applicant	64 (68.1%)	4 (4.3%)	26 (27.7%)
AOA membership	13 (14.0%)	0 (0.0%)	80 (86.0%)
Perceived commitment to program	45 (47.9%)	6 (6.4%)	43 (45.7%)
Graduate of a highly regarded medical school	19 (20.2%)	4 (4.3%)	71 (75.5%)
Research interest and involvement	19 (20.2%)	1 (1.1%)	74 (78.7%)

Abbreviations: AOA, Alpha Omega Alpha; CK, Clinical Knowledge; COMLEX, Comprehensive Osteopathic Medical Licensing Examination; USMLE, United States Medical Licensing Examination.

TABLE 3

Respondents' level of agreement with confidence in ability to convey various program characteristics to candidates

Characteristic	Strongly agree	Agree	Neutral	Disagree	Strongly disagree
Quality of medical education and training	24 (25.5%)	46 (48.9%)	18 (19.1%)	5 (5.3%)	1 (1.1%)
Research opportunities	26 (27.7%)	41 (43.6%)	24 (25.5%)	2 (2.1%)	1 (1.1%)
Program culture	16 (17.0%)	37 (39.4%)	25 (26.6%)	14 (14.9%)	2 (2.1%)
Program diversity	15 (16.0%)	40 (42.6%)	28 (29.8%)	10 (10.6%)	1 (1.1%)
Wellness initiatives	18 (19.6%)	36 (39.1%)	32 (34.8%)	5 (5.4%)	1 (1.1%)
Resident resources	18 (19.6%)	34 (36.2%)	35 (37.2%)	6 (6.4%)	1 (1.1%)
Facilities (hospital, clinics, etc.)	8 (8.5%)	25 (26.6%)	27 (28.7%)	29 (30.9%)	5 (5.3%)
Geographic location (city, housing, etc.)	10 (10.6%)	22 (23.4%)	27 (28.7%)	25 (26.6%)	10 (10.6%)

DISCUSSION

To our knowledge, this cross‐sectional survey‐based study is the first to describe PDs' and DCs' impressions on the widespread use of VIs among otolaryngology residency programs in retrospect of the 2020 to 2021 interview cycle. Nearly two‐thirds of all survey participants found the VI format to be satisfactory or strongly satisfactory. As the mean estimated likelihood to continue using the VI format in future application cycles was essentially 50% and based on a wide‐ranging distribution of responses however, it remains difficult to predict whether VIs will supplant in‐person interviews in following cycles. This theme of overall satisfaction yet mixed hesitancy and optimism for VIs in future residency cycles is not unique to otolaryngology residency programs. PDs of colon and rectal surgery residency programs have also been demonstrated to be highly satisfied in both their experiences of the 2020 to 2021 VI‐facilitated application cycle and their results in The Match. Yet similarly, only 55% of PDs surveyed agreed that regardless of the level of COVID‐19 concerns, the following cycle's interviews should be conducted virtually.

The majority of respondents in our study also indicated that they did not believe the quality of interns would be impacted by their VI assessments. Possibly contributing to these positive attitudes include the findings that more than two‐thirds of all respondents felt confident in their ability to ascertain an applicant's professionalism and communication skills as well as their personal narrative. However, individuals were hesitant when it came to their ability to assess applicant clinical skills and knowledge or overall “fit” with the program and most expressed a neutral or negative impact on those abilities. These findings were particularly interesting when considered against the backdrop of the drastic decrease in ability to offer away rotations this past cycle, a necessity of pandemic considerations, as historically away rotation performance has been an extremely important indicator of applicant quality. In an effort to begin courting applicants ahead of their VIs, a strong majority of our respondents reported that their programs provided mock VIs, virtual open houses, or both, important developments that have been demonstrated to be extremely effective for mediating the gap left by the lack of in‐person tours of programs in the general surgery iteration of the 2020 to 2021 residency application cycle. ,

Considering the unique challenges of a wholly virtual approach to gauging applicant quality, the absence of significant adaptive shifts in the relative importance of various applicant metrics was notable. Of the 12 metrics included on the survey, only one was considered by a majority of participants to have had increased relative importance during this past application cycle: personal prior knowledge of the applicant, a criterion that was actually anticipated to decrease in relative importance at the outset of the cycle. A significant proportion, though not a majority, of respondents in our study also witnessed measured increases in relative importance in perceived commitment to the program, letters of recommendation from within the specialty, and the USMLE Step 1/COMLEX Level 1 score, observations that closely fulfilled precycle expectations described by otolaryngology‐focused studies such as Kasle et al.

Whether or not the increased emphasis on Step 1 scores is a shift away from the recent trend toward more holistic approaches to applicant review should be a topic of further reflection and discussion. Its emergence alongside the current VI format which already suffered from limitations due to the lack of opportunities to demonstrate clinical skills or obtain LORs via away rotations, may place further disadvantage on candidates who are underrepresented in medicine. It also remains to be seen how the impending transition to a pass/fail scoring system for the USMLE Step 1 exam will affect the outlook of the use of VIs. In a recent study, Goshtasbi et al found that academic faculty within the field of otolaryngology deem performance on the Step 1 exam to be the single most heavily weighted metric for interview offer decisions and correlate it with resident medical knowledge and in‐service performance. The increased relative weight placed on Step 1 performance that the VI format appears to require may render the VI somewhat less attractive or effective with the adoption of the pass/fail Step 1, although other metrics such as applicant Step 2 CK score, grades in core clerkships, and research experience are expected to increase in importance as a compensatory effect.

VIs present a unique set of logistical challenges to programs, including background noise and other external disturbances, the availability of the pertinent technology and hardware resources, the lack of a standardized video‐conferencing platform that could equalize in‐interview technical expectations for applicants and programs, coordination of interview times while being mindful of different time zones, and videoconferencing fatigue for both applicant and program. Yet the vast majority of respondents in the present study felt adequately equipped to conduct their VIs. Notably, participants in our study felt unanimously that any technical issues that arose throughout the course of a VI were addressed effectively and in a timely manner and major technical difficulties were a relatively rare disruption among our cohort. Though not a predominant opinion, some respondents expressed concern about the potential for VIs to further disadvantage low‐resourced applicants who could not access high‐speed internet or the technological hardware to facilitate sufficient audiovisual quality. However, it should be noted that this specific concern may have been mitigated to some extent by medical schools that provided applicants with interview rooms equipped with the necessary capabilities. Furthermore, some believe that in comparison to in‐person interviews, videoconferencing by nature may reduce the impact of biases related to physical appearance, ethnicity, gender identity, and other relevant personal factors as well as provide an expanded potential to provide “blinded” interviews, should programs choose to pursue such an approach. Whether or not these effects in addition to direct measures, such as optional mock interview sessions for applicants to receive technical feedback or implicit bias trainings for interviewers, can sufficiently outweigh the effects of unequal access to technology remains to be seen. ,

Investigation into otolaryngology residency applicant perspectives on the VI formats used during the past application cycle is also warranted although not altogether unprecedented. VIs were piloted at one institution during the 2018 to 2019 and 2019 to 2020 interview cycles, and it was found that candidates were able to save an estimated $500 to $1000 in expenses per interview. Most applicants in their cohort had no prior experience in virtual interviewing and reported difficulty with eye contact and audiovisual lag affecting conversational flow as downsides of their experiences. Additionally, a study of fourth‐year medical students across 15 intended specialties at the outset of the 2020 to 2021 application cycle found that irrespective of intended specialty, wariness toward VIs could stem from expectations of VI‐associated increases in Step 1 emphasis that were on average significantly stronger than residency interviewers' projections. Notably, students with lower Step 1 scores demonstrated higher agreement with the sentiment that VIs would fail to present their candidacy in the best light compared to students with higher Step 1 scores. However, openness to VIs among applicants who interviewed face‐to‐face had been trending upward over time prior to the pandemic, as a significantly higher proportion of candidates in the 2019 to 2020 cycle reported that they would have chosen to interview virtually if given the option, compared to those from the 2018 to 2019 cycle. With the widespread use of online face‐to‐face interactive media formats this past year, we may expect that familiarity with and receptiveness for VIs has increased dramatically. Further studies are warranted to update and expand on these findings.

As we proceed into the second iteration of VIs necessitated by the continued pandemic, we hope that the results of this study may aid both programs and applicants in adopting strategies to address some of the challenges encountered in the previous cycle. Given the relatively low confidence expressed by programs in conveying their culture, diversity, and resources for trainees, we recommend that an emphasis be placed on communicating program characteristics through both resident and faculty‐led sessions dedicated to these specific topics. We further advocate for interviewers to directly engage with applicants during these sessions to simultaneously improve assessment of applicant “fit,” another important measure that most programs expressed difficulty gauging through VIs. Owing to the resumption of away rotations for the 2021 to 2022 application cycle, we expect both prior knowledge of the applicant and recommendation letters in the field to gain even greater importance than indicated in the present study. While this may serve to benefit the majority of applicants and particularly those without home programs, consideration should continue to be given to economically disadvantaged applicants through a more holistic review. Finally, facility tours and geographical experience remain difficult to replicate and convey through the virtual format. From the applicant perspective, these aspects are likely important factors when comparing programs and therefore justify the allocation of funds for creation of media content highlighting these key features (eg, virtual tours, photo/videos of program city, and resident life).

This study has several limitations that should encourage cautious interpretation of the results. While an effort was made to elicit responses from a high proportion of PDs and DCs, participation was voluntary and response bias may have skewed the data represented in our survey toward stronger opinions. Although our survey software of choice was unable to prevent respondents from submitting more than one response, attempts to respond multiple times were considered unlikely as such would require substantial effort and time. Over‐ and under‐representation of specific residency programs was a minimal concern in this survey as recipients were limited to only PDs and DCs, and while disproportion in representation by geographical region was not controlled for, it was well described. Further studies are required to investigate the ability to assess applicants holistically via a VI format, determine potential biases and disadvantages against lower‐resourced applicants and measures to address them, and to compare overall costs and benefits between VIs and face‐to‐face interviews for both applicants and programs in a postpandemic environment. As demonstrated by the present study however, the circumstances surrounding the COVID‐19 pandemic offer an invaluable glimpse into the value and feasibility of VIs as a potential future of otolaryngology residency candidate assessments.

CONCLUSION

At the conclusion of the 2020 to 2021 application cycle, a survey of otolaryngology PDs and DCs found that while opinions on the effectiveness of VIs in assessing specific candidate attributes were mixed, a strong majority of respondents were satisfied with their VI experiences. Overall likelihood that programs would continue to use the VI format in future application cycles was approximately 50%, and the majority of respondents expect the quality of interns to be unaffected by their programs' reliance on virtual assessments. Majority opinion on the relative importance of most traditional applicant metrics remained largely unchanged, the notable exception being an increase in the importance of personal prior knowledge of the candidate. Further investigations into the effects of virtual residency interviewing on applicant experience, ability to evaluate holistically, and outcomes of low‐resourced applicants, are warranted to guide the future of VIs.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

Table S1 Miscellaneous impressions of the virtual interview cycle (n = 94).

Click here for additional data file.