Opportunity for Improved Transparency on Otolaryngology Residency Websites for Prospective Applicants.

OBJECTIVE: Otolaryngology residency applicants often struggle to gauge their competitiveness at programs due to the lack of information available, resulting in a rising number of applications. We aimed to evaluate otolaryngology websites for information pertaining to prospective applicants. STUDY
DESIGN: Systematic content analysis.
SETTING: Web.
METHODS: We reviewed 50 otolaryngology websites from June to July 2021. We searched for information pertaining to the application process, including requirements, screening and/or selection processes, and average interviewee or matched resident statistics.
RESULTS: All websites had a page for prospective applicants. Under half (n = 24, 48%) explicitly listed required application components. Only 23 (46%) mentioned the desired number of letters of recommendation, and only 2 (4%) noted the need for a letter from the department chair. The majority (n = 35, 70%) provided no information regarding the number of applications received or interviews granted. Most (n = 35, 70%) did not mention how candidates are evaluated. A minority (n = 14, 30%) provided very general metrics on which candidates are scored or ranked. Almost all (n = 49, 98%) did not mention screening processes in place to select applicants for interview. None provided information about the academic characteristics or demographics of their interviewed applicants, and only 1 (2%) included this information for matched applicants.
CONCLUSION: Otolaryngology websites contain limited information pertaining to the residency application process for prospective applicants, making it difficult for candidates to discern their competitiveness at programs and potentially contributing to match inefficiency.

In recent years, there has been a sustained increase in the number of residency applications per student, particularly for competitive specialties such as otolaryngology. During the 2020-2021 residency application cycle, an average of 77.69 Electronic Residency Application Service (ERAS) applications were submitted per US MD otolaryngology applicant, as opposed to 61.15 in the 2019-2020 residency application cycle. During the same period, otolaryngology residency programs received an average of 345.02 ERAS applications from US MD applicants versus 297.86 in 2019-2020. These figures contradict guidance from the Association of American Medical Colleges (AAMC) that suggests a point of diminishing returns for matching with submission of additional applications based on one’s United States Medical Licensing Examination (USMLE) Step 1 score. For otolaryngology US MD applicants, that point of diminishing returns is 28 applications with a Step 1 score ≥252, 45 with a score from 241 to 251, and 48 with a score ≤240. In addition to being costly, the increased number of applications does not change the annual match rate and thus contributes to match inefficiency.

The surge of otolaryngology residency applications may in part be due to candidates struggling to identify the factors that make them competitive. Their perception of important factors and the relative weight of those factors in the application process often misaligns with the perception of residency selection committees. This lack of clarity fuels a fear of not matching and fosters the belief that more applications are better. In an effort to end this “arms race,” the AAMC Undergraduate Medical Education to Graduate Medical Education Review Committee recently penned a report with recommendations including the guidance for programs to provide more information so that candidates may better understand where they have a chance for interview or acceptance.

Applicants often turn to the internet to seek information about otolaryngology residency programs, a trend expected to increase in the setting of virtual application cycles resulting from the COVID-19 pandemic. While research on this topic area is limited in otolaryngology, a survey of emergency medicine applicants demonstrated that >75% of respondents noted that information found on residency programs’ websites influenced their decision to apply to those programs. Additionally, a survey of interventional radiology applicants discovered that the most important source of information for applicants was the program’s website, even ahead of information from physicians and mentors.

Previous work in otolaryngology has examined the content areas on residency program websites, including clinical training, research opportunities, didactics, and incentives. However, this information is less helpful for applicants wanting to assess their competitiveness at particular residency programs. Our objective was to evaluate otolaryngology residency programs’ websites for information pertaining to the application process, including requirements, screening and/or selection, and average interviewee or matched resident statistics.

Methods

We conducted a systematic content analysis of the websites of the top 50 otolaryngology programs as defined by reputation on Doximity Residency Navigator from June to July 2021. Two researchers (N.M.M. and B.A.G.) independently evaluated each website and collected information into separate data extraction forms ( ). We assessed each otolaryngology department’s home website for direct links or drop-down menu options for education and/or residency program information for applicants. Data extraction forms were compared to ensure consistency. Any discrepant data resulted in a return to the website and a discussion among researchers until consensus was met. The University of Michigan institutional review board deemed this study not regulated, as human subjects were not involved.

Table 1.

Data Extraction Form.

Program name
Steps executed to get to otolaryngology residency program website	Yes	No
1. Does the website have a tab or section for prospective applicants?
2. Does the website mention application requirements? • If yes, provide exact text for application requirements:
3. Does the website mention how applications are reviewed? • If yes, provide exact text for review process:
4. Does the website mention use of a screening process? • If yes, provide exact text for screening process: • If no, does the website explicitly state there is NO screening process? Provide exact text.
5. Does the website mention: • Number of applications reviewed? • Number of interviews offered? • Number of residency positions available?
6. Does the website mention average interviewee statistics? • If yes, provide exact text for interviewee statistics:
7. Does the website mention average resident statistics? • If yes, provide exact text for resident statistics:
8. When was the website last updated or published?
9. What additional links or resources for prospective applicants are available?
10. Other comments:

Results

All otolaryngology residency programs analyzed (N = 50) had a website dedicated to prospective applicants through a link on the homesite to “education,”“residency program,”“application process,” or a variation and/or combination of these elements. The websites had copyrights of 2021 (n = 43, 86%), 2020 (n = 2, 4%), and 2018 (n = 1, 2%). Copyright was not provided in 4 cases (8%).

Just under half the websites (n = 24, 48%) explicitly listed required application components (eg, ERAS common application, personal statement, medical school transcript), while the remainder (n = 26, 52%) simply stated that applications were accepted through ERAS. Just 23 websites (46%) mentioned the desired number of letters of recommendation, and only 2 (4%) noted a specific need for a letter written by the chair of the applicant’s home otolaryngology department ( ). One website mentioned that a chair’s letter was not necessary.

Table 2.

Excerpts for Website Content Areas Assessed.

Content area	Websites with content (N = 50) a	Excerpt
Required application components	24 (48)	“Applicants must apply through ERAS by submitting a completed application form, curriculum vitae and personal statement. Medical school transcripts, dean’s letter and three letters of reference should be submitted directly from the medical schools.”Source: University of Pennsylvania Health System
Required letters of recommendation	23 (46)	“Three strong letters of recommendation (one by Chairmen of Otolaryngology–Head & Neck Surgery).”Source: Medical University of South Carolina
Number of applications received; interviews granted	9 (18); 15 (30)	“In the last three years we received between 350 and 400 submissions. Approximately fifty applicants are invited to interview each year, although we increased the number of interview spots during the pandemic.”Source: University of Kansas School of Medicine
General overview of metrics desired or evaluated	14 (30)	“All applications are reviewed and interviews for prospective candidates are arranged on the basis of their medical school record, research experience, extracurricular activities and letters of recommendation. . . . Applicants are ranked on the basis of high moral, ethical and professional qualifications by the Resident Selection Committee for final selection in the Otolaryngology Matching Program.”Source: Johns Hopkins University“Historically, this Department has been dedicated to training academic otolaryngologists who not only have excellent clinical skills, but will also become teachers and researchers in the specialty. We are looking for applicants with an interest in academic medicine, research experience and a strong academic record.”Source: University of Minnesota
Specific overview of selection process	1 (2)	“A screening committee initially reviews the applications. The committee forwards recommendations to the chair who selects a subset of appropriate applicants for personal interviews. Interviewers will rate applicants on a variety of criteria, including medical school performance, personal attributes, exposure to the field, and goals, including research and clinical interests. Interviewers will use both the application materials and their interactions with the applicants to assign the ratings or make summary comments. Development of the rank list for the match involves all members of the residency selection committee. Each member will develop his or her own rank list. Criteria that may be considered include medical school performance, letters of recommendation, interview performance, research interest, career goals, and input from residents or other members of the staff who met the applicant. The residency selection committee will develop a consensus of the rank order. The program director and chair may review and revise the final list if needed.”Source: University of Washington
Mention of NO screening process	6 (12)	“Our program does not have a cut-off year—a maximum number of years since graduation from medical school—and does not have a score requirement on the United States Medical Licensing Examination® (USMLE®); however, the average score of our applicants is 235.”Source: NYU Grossman School of Medicine
Mention of screening process	1 (2)	“The Vanderbilt Otolaryngology–Head & Neck Surgery Residency Program receives well over 400 applications annually. Each application received by the deadline is reviewed thoroughly and, while there is no USMLE board score requirement, the score is used to narrow down the number of applications, given initial volume.”Source: Vanderbilt University Medical Center
Interviewee characteristics; statistics	1 (2); 0 (0)	“ENT residency spots are typically competitive. Applicants tend to have higher than average USMLE Step 1 scores, strong clinical achievement, and have participated in research.”Source: Oregon Health and Science University
Resident characteristics; statistics	2 (4); 1 (2)	“Applicants selected for our Program usually graduate in the top 10 percent of their medical school class.”Source: University of Texas Southwestern Medical Center“Department of Otolaryngology–Head and Neck Surgery Residency Matched Application Statistic Averages (2015 – 2020) Step 1: 253 (range = 217-275), Step 2: 260 (range = 240-279), AOA Induction: 69%, Clinical Honors: 6.9 (range = 4-12), Research Projects: 8.0 (range = 2-16), Publications: 3.0 (range = 0-8), Oral Presentations: 2.1 (range = 0-7), Poster Presentations: 3.3 (range = 0-14), Leadership Positions: 2.4 (range = 0-6), Volunteer/Extracurriculars: 6.5 (range 2-15).”Source: The Ohio State University Hospital

Data are expressed as No. (%).

All websites included the number of residency positions available. However, just 9 websites (18%) provided the number of applications received and the number of interviews granted. These numbers typically ranged between 300 and 400 applications received and 40 to 50 interviews granted for 3 to 5 residency positions. An additional 6 websites (12%) provided only the number of interviews granted.

Most websites (n = 35, 70%) did not describe the process that the residency selection committee uses to evaluate candidates for residency positions. A minority of programs (n = 14, 30%) provided very general information for metrics on which candidates are scored or ranked in the application process (eg, medical student performance, research interests). This content ranged from broad professional and personal qualities to more specific career goals and interests. One exception was the University of Washington, which provided a relatively detailed overview of its residency selection process. Nearly all websites (n = 49, 98%) did not mention the screening processes in place to select applicants for interview. Only 6 websites (12%) explicitly mentioned that USMLE Step score cutoffs are not used, while 3 others (6%) made vague statements indicating that every ERAS application is reviewed. For 2 of the 6 websites indicating that USMLE Step score cutoffs are not used, one stated that the score was used as a factor in the selection process, and the other stated that its applicants have an average Step 1 score of 235. One program mentioned the use of USMLE scores to narrow down the applicant pool; however, it did not provide the exact score that it used to screen applicants in or out of consideration.

Zero programs provided information about the demographics or academic characteristics of their interviewed applicants (eg, medical student performance, USMLE Step scores, research experience). One program made a general statement regarding the usual profile of an otolaryngology applicant. Most websites did not provide any information regarding the demographics or academic characteristics of their residents. Two websites (4%) provided general statements regarding the characteristics of the applicants chosen for residency at those programs. The exception to the trend described was The Ohio State University, which provided matched residents’ application statistic averages for the preceding 5 years from 2015 to 2020.

The most common external links were to the websites for ERAS (n = 29, 58%) and/or the National Resident Matching Program (n = 24, 48%). Boston University Medical Center provided a general section titled “How Do I Match in Otolaryngology?” with an external link to the American Academy of Otolaryngology–Head and Neck Surgery Foundation with additional information.

Discussion

Our study demonstrates the limited information available on departmental otolaryngology websites pertaining to the residency application process, highlighting opportunities for improvement ( ). First, a minority of programs provided candidate metrics desired or evaluated by the residency selection committee. When stated, such metrics were broad professional and personal qualities, such as “record of academic excellence” and “good communication skills,” rather than specific qualifications. A few select program websites stated a preference for strong research backgrounds and a goal of training academic otolaryngologists. While acknowledging that the vagueness or outright lack of information is likely a consequence of the holistic review process in place at many programs, institutions still may provide the desired qualifications for residents selected to train at their programs in line with their programs’ missions. This practice would be particularly valuable for programs with unique aims, such as producing physician-scientists, promoting diversity, or serving certain patient populations (ie, urban or rural health).

Table 3.

Identified Areas of Improvement for Otolaryngology Residency Program Websites.

Proposed changes

• Provide required application components, specifically the number of letters of recommendation required and if any of those letters must be from the chair.• List the number of applications received, the number of interviews granted, and the number of positions available.• Note the desired qualifications for candidates and the metrics by which candidates are evaluated.• Describe the use of any screening processes.• Give average interviewee and/or matched resident statistics.

Second, most programs made no mention of the screening processes in place to narrow the initially large pool of applications to one that is more manageable for the residency selection committee. While not a practice at all residency programs, it is not uncommon for applications to be screened on factors such as USMLE Step 1 scores, geographic biases, or international medical graduate and reapplicant statuses. One survey study of otolaryngology departmental chairs, program directors, associate program directors, and faculty reported that 31.1% of respondents endorsed use of a numeric USMLE Step 1 score screening process at their institution with an average cutoff of 230.5 ± 8.8. If such screening practices are in place, it would be beneficial for programs to advertise this number to prospective applicants, who may be inadvertently wasting limited time and financial resources to apply to programs at which they will be screened out of further consideration. Even with the transition to a pass/fail Step 1, it is likely that another scored metric, such as Step 2, will continue to be used for this purpose.

Finally, except for 1 institution, no programs provided data for interviewed or matched applicants. While some matched resident data may be obtained through the National Resident Matching Program’s Charting Outcomes in the Match, this information is not separated by individual otolaryngology residency programs. Moreover, resources such as the Texas Star Dashboard require institutional enrollment. Providing matched resident data on individual otolaryngology websites improves accessibility, promotes transparency, and allows applicants to assess their competitiveness at different programs in line with recommendations from the AAMC. In turn, this change may enable students and their faculty advisors to make more informed decisions, by strategically targeting residency programs and thereby reducing the number of potentially unnecessary applications. Future prospective work would be valuable to determine whether providing such information influences the application rates to individual programs.

Our study is limited in that we restricted our analysis to the top 50 otolaryngology residency programs as defined by reputation on Doximity Residency Navigator. We chose to use this resource as it is a tool used by many prospective applicants. Previous research suggests that otolaryngology websites for “large” programs (≥3 residents per year) are more comprehensive than those for small programs. As our list includes mostly large programs, it is therefore unlikely that smaller programs not in this study would contain more robust information than that observed here.

Conclusion

The results of this study demonstrate that otolaryngology department websites contain limited information pertaining to the residency application process for prospective applicants, such as requirements, screening and/or selection, and average interviewee or matched resident statistics. This lack of transparency may make it difficult for candidates to discern their competitiveness at programs and potentially contribute to match inefficiency.

Author Contributions

Nicole M. Mott, design, conduct, analysis, writing, approval and responsibility; Bhavna A. Guduguntla, conduct, analysis, revising writing, approval and responsibility; Lauren A. Bohm, design, revising writing, approval and responsibility, oversight

Disclosures

Competing interests: None.

Sponsorships: None.

Funding source: None.