Overemphasis of USMLE and Its Potential Impact on Diversity in Otolaryngology.

OBJECTIVE: Applicant demographics during the 2019-2020 residency cycle were evaluated to determine if strict utilization of United States Medical Licensing Examination (USMLE) scores in applicant selection could lead to a restriction in diversity. STUDY
DESIGN: Cross-sectional study.
SETTING: Otolaryngology residency applicants to a single institution.
METHODS: A total of 381 applicants were analyzed by age, gender, applicant type, race/ethnicity, USMLE scores, permanent zip code, and graduating medical school.
RESULTS: Among applicants, 37% were women; 9% were ≥30 years of age; 12% were underrepresented minorities (URMs); 71% to 81% had above-average socioeconomic surrogate markers; 22% were from a top 25 US News & World Report-ranked institution; and 81% were from an institution with an otolaryngology residency program. There was no increase in applicants who identified as URM from the 2015-2020 cycles. Multivariable regression analysis showed that applicants who were international medical graduates, URMs, and ≥30 years of age had lower Step 1 and Step 2 scores (P < .05). Applicants who identified as women had a lower Step 1 score, and those from top 25 National Institutes of Health-funded institutions had a higher Step 1 score; however, there was no difference when Step 2 scores were compared (P > .05).
CONCLUSION: Our data suggest that in the pre-USMLE Step 1 pass/fail setting, strict adherence to USMLE scores may lead to disproportionally low recruitment of applicants who are women, ≥30 years of age, URMs, and from institutions without an otolaryngology residency program. We must implement measures against overemphasizing the absolute values of USMLE scores for a true holistic review of applicants, specifically to prevent an overemphasis on the USMLE Step 2 score.

In 2020, the United States Medical Licensing Examination (USMLE) announced that Step 1 will become pass/fail starting in 2022. The Federation of State Medical Boards and the National Board of Medical Examiners believe that this new change in policy will help reduce the current overemphasis on USMLE performance and is an “important first step toward facilitating broader, system-wide changes to improve the transition from undergraduate to graduate medical education.” Also in 2020, the National Resident Matching Program reported a record-high 44,959 registered applicants for 37,256 residency positions. In otolaryngology, there were 505 applicants for 350 residency positions.

Many residency selection committees among the more competitive specialties have traditionally used the USMLE Step 1 score to filter through large applicant pools vying for limited residency positions. Consequently, medical students without a “competitive” Step 1 score may prematurely abandon plans to pursue a given specialty.[4,5] This overemphasis on Step 1 scores for resident selection may have unintended consequences of reducing diversity in competitive specialties based on gender, age, and distribution of underrepresented minorities (URMs). Various studies have reported the positive effect on patients and communities as a whole when a diverse physician workforce is in place.[7-9] Otolaryngology lags behind other medical specialties in the representation of URMs and women. This study evaluates the demographics of the most recent pool of residency applicants to determine whether a correlation exists between the distribution of USMLE scores and the diversity of applicants.

Methods

After institutional review board approval for this study was obtained from the University of California–Davis, we collected all Electronic Residency Application Service (ERAS) applications submitted to the university’s otolaryngology–head and neck surgery residency program for the 2019-2020 cycle. The following data were obtained: age at the time of application submission, gender, applicant type, race/ethnicity, Step 1 score, Step 2 score, permanent zip code, and name of the medical school of graduation.

Age was categorized as <30 versus ≥30 years; gender as man versus woman; applicant type as US allopathic, US osteopathic, or international medical graduate (IMG); and race/ethnicity as URM versus non-URM. URM was defined as African American/Black, Hawaiian/Pacific Islander, Hispanic/Latino, or Alaskan Native/Native American, and non-URM was defined as any race/ethnicity not in the URM definition, including Caucasian, Asian, and prefer not to state. The Step 1 cutoff score was set at ≥240 for in-depth Step 2 univariate and multivariable regression analyses and labeled Step 1 ≥240 .

Applicants were excluded from zip code data analysis when their reported permanent zip codes matched their current locations and the sites of their medical schools of graduation. Applicants with zip codes outside the United States or Puerto Rico were excluded. These exclusions were created to optimize the permanent zip code analysis by capturing the applicants’ residences independent of their medical schools of graduation. Utilizing the American FactFinder of the US Census Bureau, we defined the following variables in a binary format based on publicly available national data: median household income ($63,179), percentage below the poverty line (11.8%), high school graduate >25 years of age (90%), and age >25 years with a graduate or professional degree. The percentage of individuals with a graduate or professional degree per zip code were stratified in an ascending-tier grading system: ≤11.8% = 0, >11.8% to 23.6% = 1, >23.6% to 35.4% = 2, >35.4% = 3.[13,14]

Medical schools were classified as top 25 US News or top 25 NIH based on the 25 highest-ranked schools in the 2021 US News & World Report and the 25 institutions with highest funding from the National Institutes of Health (NIH). Additionally, each applicant was categorized by home ENT program (yes vs no) based on whether the medical school of graduation had an affiliation with an otolaryngology residency program. The race/ethnicity of the 2018 US population was collected from American FactFinder of the US Census Bureau. The race/ethnicity of the 2019 US medical school graduates and 2015-2019 US otolaryngology applicants was collected from the National Resident Matching Program.[3,17]

Univariate statistical analyses of difference in mean Step 1 and Step 2 scores were performed with a 2-sample t tests and analysis of variance for the explanatory variables listed in and . A multivariable linear regression analysis was conducted modeling Step 1 and Step 2 scores versus predictor variables of applicant type, URM, gender, age, top 25 NIH, home ENT program, and Step 1 ≥240. For nominal data, we performed a 2-sample Z test for the difference between proportions. All statistical tests were 2-sided and evaluated at a significance level of 0.05. Statistical analyses were conducted with R version 4.0 (R Project for Statistical Computing).

Table 1.

Age and USMLE Score Distribution Among Applicants (N = 381).

	No. (%)	Mean	Median	Mode	Range	SD
Age, y	379 (99)	27.4	27	27	23.8-38.8	2.1
Step 1 a	381 (100)	245.4	247	249	197-269	12.5
Step 2 b	364 (96)	253.4	255.5	259	203-279	12

Abbreviation: USMLE, United States Medical Licensing Examination.

Two applicants did not report age and were excluded from analysis.

Seventeen applicants did not report Step 2 and were excluded from analysis.

Table 2.

Univariate Analysis of Step 1 and Step 2 Scores.

	Step 1 score				Step 2 score
	No. (%)	Mean	SD	P value	No. (%)	Mean	SD	P value
Gender
Men	241 (63)	246.3	12.8	.068	228 (63)	253.2	12.7	.685
Women	140 (37)	243.9	11.8		136 (37)	253.7	10.9
Age ≥30 y
Yes	35 (9)	239.1	15.8	.008	31 (9)	244.7	16.7	.002
No	344 (91)	246.3	11.8		331 (91)	254.3	11.1
Applicant type
US allopathic	352 (92)	246.1	11.7	<.001	336 (92)	254.2	11.1	<.001
US osteopathic	14 (4)	243	15.1		13 (4)	250.6	15.5
International medical graduate	15 (4)	232.3	19.7		15 (4)	237.7	17.3
Underrepresented minority a
Yes	47 (12)	238.8	14.7	.001	46 (13)	248.5	14.6	.016
No	334 (88)	246.4	11.9		318 (86)	254.1	11.5
Home otolaryngology program
Yes	310 (81)	246.1	11.9	.045	295 (81)	255.1	11.1	.004
No	71 (19)	242.4	14.2		69 (19)	248.9	14.7
Top 25 National Institutes of Health
Yes	79 (21)	248.5	10.12	.005	74 (20)	255.2	9.9	.092
No	302 (79)	244.6	12.9		290 (80)	252.9	12.5
Top 25 US News & World Report
Yes	82 (22)	247.7	10.3	.035	77 (21)	255.4	9.4	.052
No	299 (78)	244.8	12.9		287 (79)	252.8	12.6
Step 1 ≥240
Yes	281 (74)				269 (74)	256.8	9.4	<.001
No	100 (26)				95 (26)	243.8	13.5
Household income <$63,179 b
Yes	79 (29)	244.5	12.4	.798	78 (29)	254	11.2	.502
No	195 (71)	245	13		187 (71)	252.9	12.7
Poverty line >11.8% b
Yes	73 (27)	244.5	13.7	.806	73 (28)	252.1	13.4	.367
No	201 (73)	245	12.5		192 (72)	253.7	11.8
High school degree <90% b
Yes	54 (20)	243.4	14.3	.400	52 (20)	251.2	12.2	.187
No	220 (80)	245.2	12.4		213 (80)	253.7	12.2
Graduate or professional degree, % c
<11.8	51 (19)	242.9	12.5	.472	50 (19)	251.4	12.7	.610
>11.8-23.6	107 (39)	244.4	13.4		102 (38)	253.6	12.2
>23.6-35.4	83 (30)	246.2	12.6		81 (31)	254.2	12.5
>35.4	33 (12)	246.1	12.2		32 (12)	252.7	11.5

Includes African American/Black, Hawaiian/Pacific Islander, Hispanic/Latino, and Alaskan Native/Native American.

Variable information extracted from zip code data.

Graduate or profession degree is stratified by percentage in zip code.

Results

Demographic information was collected from 381 applicants who applied to the University of California–Davis during the 2019-2020 cycle. contains mean, median, and mode of the age, Step 1, and Step 2 variables for all applicants analyzed. Step 1 and 2 scores were 245 ± 12.5 and 253 ± 12 (mean ± SD), respectively. The applicant pool comprised 352 US allopathic medical school graduates (92%), 14 US osteopathic medical school graduates (4%), and 14 IMGs (4%; ). The typical applicant was a man (63%), <30 years of age (91%), and of non-URM status (88%). The majority of the applicants self-identified as Caucasian (50.5%), followed by Asian (29.6%), Hispanic/Latino (6.3%), African American/Black (5.8%), and Native Hawaiian/Pacific Islander (0.5%) ( ). shows the percentage of otolaryngology applicants by race/ethnicity from 2015 to 2020. There was no significant increase in URM otolaryngology residency applicants from 2015 to 2020.

Figure 1.

US population, medical school graduates, and otolaryngology residency applicants by race/ethnicity. 2015 data were from ERAS Statistics and 2020 data from this study. US population is from 2018 and US medical school graduates from 2019.

Figure 2.

Otolaryngology applicants from 2015 to 2020 by self-identified race/ethnicity. The 2015-2019 data were from ERAS Statistics and 2020 data from this study. There was no significant increase in underrepresented minority applicants.

Univariate analysis ( ) showed that Step 1 scores were significantly different by applicant type (P < .001). Scores were significantly higher for applicants belonging to any of the following categories: top 25 NIH (P = .005), top 25 US News (P = .034), home ENT program (P = .045), ≤30 years of age (P = .008), or non-URM (P = .001). A total of 274 and 265 applicants were in the Step 1 and Step 2 univariate permanent zip code analysis, respectively. There were no significant differences in Step 1 scores when comparing across categories of permanent zip code: household income, poverty line, high school graduates, and graduate or professional degree.

Univariate analysis ( ) showed that Step 2 scores were significantly different for applicant type (P < .001). Scores were significantly higher in home ENT program applicants (P = .041), those ≤30 years of age (P = .002), and non-URM applicants (P = .016). Applicants who scored ≥240 on Step 1 had a significantly higher Step 2 (P < .001). There were no significant differences in the Step 2 scores of otolaryngology applicants when comparing means across categories for top 25 NIH, top 25 US News, gender, and permanent zip code data.

On multivariable linear regression analysis ( ) for Step 1 and Step 2 as response variables, Step 1 scores were lower for IMG applicants than US allopathic applicants (P = .017), URMs versus non-URMs (P < .001), women versus men (P = .017), and those aged ≥30 versus <30 years (P = .006). Conversely, a statistically significant increase in Step 1 scores was observed in applicants who graduated from a top 25 NIH-funded medical institution (P = .015), but this did not hold true for Step 2 (P = .757). There was no significant difference in Step 1 or Step 2 scores for home ENT program applicants after multivariable analysis. Multivariable regression analysis for Step 2 scores showed significantly lower scores in applicants who were IMG (P = .007), URM (P = .007), ≥30 years of age (P = .010), and scored <240 on Step 1 (P < .001). While gender was a predictor of Step 1 performance, no difference was found for Step 2 scores (P = .502).

Table 3.

Multiple Linear Regression Analysis of USMLE Step 1 and Step 2 Scores.

	Step 1 score			Step 2 score
	Estimate	SE	P value	Estimate	SE	P value
Intercept a	248.4	1.7	<.001	256.5	1.6	<.001
US osteopathic	−3.4	3.3	.324	−3.1	3.0	.303
International medical graduate	−8.6	3.6	.017	−5.3	3.2	.098
Underrepresented minority	−7.7	1.8	<.001	−4.4	1.6	.007
Women	−2.9	1.2	.018	0.74	1.1	.502
Age ≥30 y	−5.9	2.2	.007	−5.3	2.0	.010
Top 25 National Institutes of Health	3.7	1.5	.015	0.43	1.4	.757
Home otolaryngology program	−0.61	1.8	.737	0.99	1.6	.542
Step 1 ≥240	248.4	—	—	−11.2	1.3	<.001

Abbreviation: USMLE, United States Medical Licensing Examination.

Intercept represents baseline categories of the predictors: US allopathic applicants, non– underrepresented minority, male, <30 years of age, non–top 25 National Institutes of Health, non–home otolaryngology program, and Step 1 cutoff <240.

Discussion

Otolaryngology currently faces a significant representation deficit of URMs and women, even though research in various industries, such as finance, technology, and government-run agencies, has shown that the deliberate creation of a diverse workforce selects for increased success and innovation.[18-20] Editorials in otolaryngology have emphasized the idea that diversity in otolaryngology should be sought, as the power of diversity is critical for breaking down bias, improving innovation, and preparing for success.[18,21-24] To that end, diversity is multifaceted, as it encompasses areas such as gender, age, educational background, and social background.

Currently, there are 155 allopathic and 36 osteopathic accredited medical schools in the United States.[26,27] However, only 120 otolaryngology programs are listed in the 2020-2021 ERAS, which leaves many students with limited access to mentorship and guidance. Our data demonstrate that we are mostly recruiting students from institutions with affiliated otolaryngology programs (81%; ). Even with several initiatives to recruit and promote diversity in the field with scholarships, lecture series, and diversity programs,[29-33] our cohort breaks down as follows: 37% women, 9% aged ≥30 years, 4% US osteopathic, 4% IMG, and 12% URM.

The 2018 US population of URMs was 32.1%, yet only 13.5% of 2019 medical school graduates identified as such. Just 5.6% of the 2018 US population identified as Asian, as compared with 21.5% of 2019 medical graduates and 29.6% of 2020 otolaryngology applicants ( and ). Even when all URM applicants are combined (African American/Black, Hispanic/Latino, Hawaiian/Pacific Islander, and Alaskan Native/Native American), their percentage is just 15.7% of the Asian/Caucasian 2020 otolaryngology applicant pool. Recent work by Smith et al showed that only 19% of otolaryngology programs have department-level diversity and inclusion initiatives, which may expose an avenue to increase recruitment of URMs into otolaryngology. There is still significant work to be done to expand the pipeline for diverse candidates and create a sustainably diverse racioethnic field within otolaryngology that truly reflects our country’s population.[29-33]

Simply increasing the number of applicants does not guarantee an increased number of URMs obtaining residency interviews or matching into otolaryngology. The issue that we face is 2 pronged: the needed increase of URM applicants and URM matriculates in otolaryngology residency. Research in applicant trends has shown that stringent Step 1 expectations may limit future talented physicians from entering the field of otolaryngology while limiting diversification of the field.[4,5,29] The 2018 National Resident Matching Program’s Charting the Outcomes showed that otolaryngology had an average Step 1 of 248 and Step 2 of 254, which is well above the national average. The 2018 Program Director Survey highlights the importance that program directors place on Step 1 in making decisions for offering interviews and ranking applicants. However, a clear link between Step 1 and resident performance has never been fully established.[36,37] Our study demonstrates that applicants who are ≥30 years of age or identify as URM have lower Step 1 and Step 2 scores as compared with their younger or non-URM counterparts ( and ). Given the historical value placed on Step 1 in otolaryngology, we advise against overemphasizing the absolute values of the USMLE scores to stratify applicants, as this may continue to adversely affect URMs to a greater degree. With the current COVID-19 pandemic, these disparities may be exacerbated. Students coming from institutions without affiliated otolaryngology programs will face challenges garnering letters of recommendations and valuable face time with programs, which could lead to an even greater emphasis on the USMLE.

In 2017 the Association of American Medical Colleges reported that for the first time, more women than men matriculated into medical school. Despite that great achievement, the association noted that of the 1542 otolaryngology residents in 2017 to 2018, just 558 (36.2%) were women. Additionally, an estimated 14.5% of practicing otolaryngologists are women. Our data highlight the need for increased recruitment of otolaryngology applicants who are women, as only 37% of applicants in our cohort are women ( ). Women in our cohort averaged slightly lower Step 1 scores than men; however, our data show that for Step 2, the mean score difference between genders did not persist ( ). Similar to URM applicants, our results suggest that further work is needed for equal gender representation in our applicant pool. We highlight that an overemphasis of the Step 1 score has the potential to contribute to the gender gap seen in otolaryngology resident selection.

Leaders in academic medicine have begun to advocate against implicit or subconscious bias by promoting training and education to create a diverse workforce that reflects the population that we treat.[43-45] When Step 1 transitions to pass/fail, there exists the potential for applicants coming from lower-tier or newer medical schools to face a subconscious bias from resident selection committees as compared with their colleagues attending a school labeled as top 25 NIH or US News or with a home ENT program. Chatterjee et al recently explored how subconscious biases are prevalent in the residency interview trail, while Hauge et al showed how a “closed file review” may affect how interviewers rate applicants. These studies suggest that efforts are needed to limit biases based on “educational pedigree” and/or background. After adjusting for covariates, applicants coming from top 25 NIH schools have significantly higher Step 1 scores (3.7-point estimate); however, this is no longer significant in their Step 2 scores ( ). It is unclear why a difference exists between these groups, although it could be explained by the low percentage of URMs, women, and those aged ≥30 years in our cohort who come from top 25 NIH medical schools (12.7%, 39.2%, 10.1%, respectively). Therefore, we propose that subconscious bias training be implemented for all application reviewers and interviewers, while subconscious bias–prone variables such as educational pedigree be omitted from application review. Additionally, we propose that closed file interviews be undertaken to limit bias even before the beginning of the interview, with the goal of promoting diversity of selection, especially because 78.7% of URMs, 77.9% of women, and 77.1% of those aged ≥30 years in our data come from medical schools not labeled top 25 NIH.

Area-based socioeconomic research has used zip code data to evaluate outcomes in health disparities. Using similar zip code methods, we utilized surrogate markers for socioeconomic status to compare the influence on USMLE examinations on performance and found no significant difference in any of the variables ( ); however, we excluded multiple applicants as described in the methods. The exclusion limited our power for an in-depth area-based socioeconomic analysis in our cohort and could be the reason for lack of significance. Interestingly, the majority of applicants in the analysis reported living in a permanent zip code above the national mean or median for household income (71%), were high school graduates >25 years of age (80%), had a graduate or professional degree (81%), and fell below the national mean in poverty line (73%). Further investigations are needed to evaluate how socioeconomic background affects performance in the USMLE and diversity in otolaryngology, as this could help validate increased support to students who come from backgrounds limited in funding, education, and preparation.[49,50]

This study has several limitations and strengths that should be considered when evaluating the data. All variables were self-reported by the applicants in their ERAS application, including race/ethnicity, permanent zip code, and gender. Additionally, applicants had the option prefer not to state, which could have affected the analysis of this study. Our study captured 75.4% of all otolaryngology residency applicants during the 2019-2020 cycle; thus, there is a possibility that the demographics of the applicants not reviewed could affect our results and not be representative of the entirety of the otolaryngology applicant pool for this cycle. However, given the robust significance of our data and the affinity of our data to those seen in other residency applicant years,[10,34] we trust that our applicant cohort demonstrates an acceptable portrayal of the otolaryngology applicant pool as a whole.

With the change to a pass/fail Step 1, the question is whether there will be an inevitable shift in emphasis to the absolute value of the Step 2 score in differentiating applicants. In our data 100% of the applicants passed Step 1, and while the scoring changes may remove the emphasis in that factor, it leaves Step 2 as the sole objective variable for consideration. Even though our multivariable regression analysis showed that applicants who scored ≥240 on Step 1 had a significantly higher Step 2 score, we advise against the future overemphasis of Step 2 in applicant selection as a surrogate for Step 1 ( ). Specifically, our data demonstrate that the currently underrepresented groups in otolaryngology would continue to be at a significant disadvantage if an overemphasis on Step 2 were to develop. Furthermore, using other metrics as filtering tools, such Alpha Omega Alpha and Gold Humanism Honor Society, holds intrinsic limitations due to the lack of standardization in criteria for nomination, selection, and even availability of honor society chapters at institutions. Using publication numbers or impact factor of published scholarly work would likely negatively affect applicants with limited access to research opportunities, as such opportunities and funding are often concentrated at major academic centers with a plentitude of resources and faculty dedicated to scholarly activity. Therefore, we highly encourage a more holistic review of applicants to assess motivation, passion, academic rigor, personality, and other skills irrespective of factors external to their control, as this may encourage URMs to consider otolaryngology and it may increase the number of URMs being interviewed and matched.[4,5,29,34,35] As Step 1 moves to pass/fail, otolaryngology residency selection committees—and graduate medical education as a whole—should take clear preventative measures to avert the unwarranted overemphasis of Step 2 scores in filtering applicants.

Conclusion

Utilization of the Step 1 score absolute value in resident selection may lead to disproportionally low recruitment of applicants who are women, ≥30 years of age, URM, socioeconomically disadvantaged, or from institutions without an affiliated otolaryngology residency program while holding true for Step 2 for applicants who are URM and ≥30 years of age. Therefore, overemphasis of USMLE scores may perpetuate the critical lack of diversity in our field.

Author Contributions

Pompeyo R. Quesada, substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved; Roberto N. Solis, substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved; Macaulay Ojeaga, substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved; Nuen T. Yang, substantial contributions to analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved; Sandra L. Taylor, substantial contributions to analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved; Rodney C. Diaz: substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Disclosures

Competing interests: None.

Sponsorships: None.

Funding source: This project was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through grant UL1 TR001860.