Implications of Pass/Fail Step 1 Scoring: Plastic Surgery Program Director and Applicant Perspective.

As early as 2022, United States Medical Licensing Examination Step 1 results will be reported as pass or fail, rather than as 3-digit numeric scores. This survey examines the perspectives of plastic surgery applicants and program directors (PD) regarding this score reporting change.
METHODS: A 24-item survey was distributed to integrated applicants from the 2018-19 and 2019-20 application cycles. An analogous 28-item survey was sent to integrated and independent plastic surgery training program directors. Data were analyzed using summary tables and marginal homogeneity tests.
RESULTS: 164 applicants (33.2%) and 64 PDs (62.1%) completed the survey. Most applicants (60.3%) and PDs (81.0%) were not in favor of the score reporting change. As a result of binary scoring, a majority of respondents anticipate that residency programs will use Step 2 CK scores to screen applicants (applicants: 95.7%, PDs: 82.8%), prioritize students from more prestigious medical schools (applicants: 91.5%, PDs: 52.4%), and that dedicated research time will become more important (applicants: 87.9%, PDs: 45.3%). Most applicants (66.4%) and PDs (53.1%) believe that there will be an increase in plastic surgery applicants. Applicants and PDs anticipate that the top 3 metrics used by programs when deciding to offer an interview will change as a result of binary Step 1 scoring.
CONCLUSIONS: Most plastic surgery applicants and PDs do not support the change in United States Medical Licensing Examination Step 1 scoring to pass or fail. The majority believe that other metrics (such as Step 2 CK scores, research experience, and medical school reputation) will become more important in the application process.

INTRODUCTION

United States Medical Licensing Examination (USMLE) Step 1 scores are an important component of the residency application process, particularly in competitive specialties.[1] Step 1 scores have been positively correlated with receiving plastic surgery residency interview invitations and with an applicant’s success in matching.[2-5] In turn, plastic surgery training programs routinely match applicants with Step 1 scores approximately 1 SD above the national mean.[1,2]

Given its significance in residency matching, Step 1 performance creates considerable stress and anxiety.[6] There has been increasing emphasis on Step 1 preparation in undergraduate medical education, with students purchasing costly resources and dedicating additional time to study for the Step 1 examination.[7] Unfortunately, socioeconomic status disparities in accessing resources may contribute to demographic differences in Step 1 performance by race and gender.[8-10] Consequently, Black and Latinx medical students may be disproportionately excluded from surgical careers, as many surgical training programs use minimum Step 1 scores for screening applications.[11-15]

To address the overemphasis of Step 1 performance in residency selection and minimize demographic differences in Step 1 performance, the Federation of State Medical Boards and the National Board of Medical Examiners moved to change score reporting for the USMLE Step 1 to pass/fail (P/F)[16] beginning as early as January 2022. This paradigm shift will have significant ramifications for the plastic surgery residency application process. This study characterizes the perspectives of plastic surgery program directors (PD) and recent applicants regarding how this change will affect the plastic surgery residency application process.

METHODS

Survey Design

Two analogous survey instruments were designed to assess applicant and PD perspectives regarding P/F USMLE Step 1 scoring (Table 1). The surveys were then pilot tested and internal validity was assessed. The final survey instruments utilized a combination of Likert scales, rank orders, multiple-choice, and free text questions. The surveys were distributed by email, using the Qualtrics (Provo, Utah) and REDCap (Research Electronic Data Capture, Nashville, Tenn.) platforms. Review of the electronic consent form and successful completion of the survey was considered consent for participation. This study received institutional review board (IRB) exemption from the Ohio State University Office of Responsible Research Practices (IRB #2020E0286) and the Vanderbilt University IRB (IRB #200355).

Table 1.

Survey Questions

Part I: Changing the USMLE Step 1 score reporting to P/F
Is a good idea Will increase the importance of Step 2CK scores in selecting applicants Will put international medical graduates at a disadvantage* Will positively impact the application process† Will make objective evaluation of applicants easier Will improve medical student well-being Will decrease socioeconomic disparities in the application process Will decrease medical student knowledge of basic sciences* Will prioritize students from higher-ranked medical schools
Part II: Currently, what do you believe are the top 3 metrics a program uses when deciding to extend a residency interview?‡
Applicant’s medical school	Dean’s letter
Letters of recommendation	Doing a rotation at a program
Research	Step 1 score (numeric)
Step 2 CK score (numeric)	Clerkship grades
Society memberships	Leadership experience
Part III: Following the USMLE Step 1 score reporting change to P/F, what do you believe will be the top 3 metrics a program will use when deciding to extend a residency interview?†
Applicant’s medical school	Dean’s letter
Letters of recommendation	Doing a rotation at a program
Research	Step 1 score (P/F)
Step 2CK score (numeric)	Clerkship grades
Society memberships	Leadership experience
Part IV: As a result of changing USMLE Step 1 score reporting to P/F
Step 2CK will be required as part of the residency application process
An applicant’s medical school will be more important in screening and selection*Programs are likely to screen applicants based on Step 2CK scores†
Step 2CK will also be changed to P/F
A new standardized examination will be created for evaluating applicants†
There will be more medical students applying to integrated plastic surgery residency
More applicants will apply to at least one other specialty in addition to plastic surgery†
It will be more important to take time off from medical school to participate in research
My application will be more competitive for integrated plastic surgery residency†
Part V: Demographic Information (age, gender identity, medical school type, race, ethnicity, dedicated research experience, cumulative medical school debt)

*Only in program director survey.

†Only in applicant survey.

‡Rank order.

Survey Respondents

Email addresses for PDs of integrated (n = 80) and independent (n = 54) plastic surgery residency programs were gathered from publicly-available ACGME documents and institutional websites. PDs who direct both program types (n = 31) were included only once. Three survey rounds were issued to 103 PDs over a 3-week period.

Email addresses of all integrated plastic surgery applicants (n = 493) from 2 consecutive integrated plastic surgery residency application cycles (2018–2019 and 2019–2020) were compiled. Four survey rounds were issued over a 2-week period.

Statistical Analysis

Continuous variables were analyzed for mean, median, mode, SD, and quartiles. Ordinal and categorical variables were summarized in tables. Marginal homogeneity tests were utilized to detect changes in metrics for interview selection. Statistical analysis was performed using Microsoft Excel (Redmond, Wash.) and IBM SPSS Version 25.0 (Armonk, N.Y.), and P < 0.05 was considered statistically significant.

RESULTS

Program Director Response Rate and Demographics

Sixty-four PD responses were obtained (response rate = 62%), including 40 integrated PDs, 8 independent PDs, and 16 dual-program PDs. Mean PD age was 49 years, and 31% were identified as women. The mean tenure as PD was 6.4 years.

Applicant Response Rate and Demographics

A total of 164 applicants (33.2%) completed the survey. Mean applicant age was 28 years, and 48% identified as female. An estimated 69.5% of applicants self-identified as White or Caucasian, 18.9% as Asian, 3.7% as Black or African American, and 5.5% as Hispanic or Latino. Almost all applicants (98.2%) graduated from allopathic US medical schools. Over a third (34.8%) had participated in a dedicated research experience, defined as a contiguous period no less than 12 months, where one is conducting research (basic, translational, or clinical) related to biomedicine or healthcare, but not actively pursuing a health professional degree. In total, 25.6% of applicants had no cumulative medical-school–related debt; 54.3% reported debt greater than $100,000, and 17.1% reported debt greater than $300,000 (Table 2).

Table 2.

Applicant Demographics (n = 164)

Category		n	%
Gender identity	Female	78	47.6
	Male	85	51.8
	Gender-variant/non-conforming	1	0.6
Type of medical school	Allopathic (M.D.)	161	98.2
	Osteopathic (D.O.)	3	1.8
Race	American Indian	1	0.6
	Asian	31	18.9
	Black or African American	6	3.7
	White or Caucasian	114	69.5
	Other	12	7.3
Ethnicity	Hispanic and/or Latino	9	5.5
Dedicated research experience	Yes	57	34.8
Cumulative medical education debt	No debt	42	25.6
	Less than $50,000 USD	21	12.8
	$50,001–$100,000 USD	12	7.3
	$100,001–$200,000 USD	31	18.9
	$200,001–$300,000 USD	30	18.3
	>$300,000 USD	28	17.1

Applicant and PD Perspectives regarding Binary Step 1 Scoring

Most respondents viewed the score reporting change unfavorably (applicants: 60.3%, PDs: 81.0%) (Tables 3 and 4). Most PDs (82.8%) believe the change will make objective comparison of applicants more difficult. Nearly all applicants (91.5%) agreed the change will benefit students from more prestigious medical schools, and most PDs (52.4%) agreed that an applicant’s medical school will become more important in screening and selection. While most applicants (87.9%) believe the change will increase the importance of dedicated research, only some PDs (45.3%) shared this belief. Most respondents felt this change will increase the importance of numeric Step 2 CK scores (applicants: 73.2%, PDs: 87.5%), and most anticipate that Step 2 CK will become a required application component (applicants: 96.3%, PDs: 82.8%). Furthermore, 95.7% of applicants feel programs will use Step 2 CK to screen applicants for interview invitation. Most disagreed that socioeconomic disparities in the application process will diminish as a result of this change (applicants: 56.1%, PDs: 57.8%). Opinions were split on whether this change will improve student well-being. Most respondents felt there will be more applicants to integrated plastic surgery as a result of the reporting change (applicants: 66.4%, PDs: 53.1%).

Table 3.

Applicant Sentiments Regarding Step 1 Score Reporting Change

Statement	Strongly Agree	Agree	Neither Agree Nor Disagree	Disagree	Strongly Disagree
Changing the USMLE Step 1 score reporting to P/F…
Is a good idea	9.8%	13.4%	16.5%	34.1%	26.2%
Will increase the importance of Step 2CK scores in selecting applicants	73.2%	21.3%	1.2%	2.4%	1.8%
Will positively impact the application process	4.9%	7.9%	26.2%	42.1%	18.9%
Will prioritize students from higher ranked medical schools	62.8%	28.7%	6.1%	0.6%	1.8%
Will make objective evaluation of applicants easier	1.2%	4.9%	9.8%	47.0%	37.2%
Will improve medical student well-being	15.9%	28.0%	23.2%	26.8%	6.1%
Will decrease socioeconomic disparities in the application process	6.7%	12.2%	25.0%	33.5%	22.6%
As a result of changing USMLE Step 1 score reporting to P/F…
Step 2CK will be required as part of the residency application process	64.0%	32.3%	1.8%	1.2%	0.6%
Programs are likely to screen applicants based on Step 2CK scores	62.2%	33.5%	2.4%	0.6%	1.2%
Step 2CK will also be changed to P/F	8.5%	12.8%	33.5%	36.6%	8.5%
It will be more important to take time off from medical school to participate in research	47.0%	40.9%	7.9%	3.7%	0.6%
A new standardized examination will be created for the purpose of evaluating applicants	1.8%	12.8%	29.3%	42.7%	13.4%
There will be more medical students applying to integrated plastic surgery residency	26.2%	40.2%	19.5%	14.0%	0.0%
More applicants will apply to at least one specialty in addition to plastic surgery	15.2%	41.5%	33.5%	9.1%	0.6%
My application will be more competitive for integrated plastic surgery residency	5.5%	14.0%	26.2%	36.6%	17.7%

Table 4.

Program Director Sentiments regarding Step 1 Score Reporting Change

Statement	Agree	Neutral	Disagree
Changing the USMLE Step 1 to P/F:
Is a good idea	4.8%	14.3%	81.0%
Will make it more difficult to objectively compare applicants	82.8%	6.3%	10.9%
Will increase emphasis on Step 2 CK scores in selecting applicants for my program	87.5%	3.1%	9.4%
Will put international medical graduates at a disadvantage	43.5%	45.2%	11.3%
Will decrease socioeconomic disparities in the application process	4.7%	37.5%	57.8%
Will decrease medical student knowledge of the basic sciences	29.7%	50.0%	20.3%
Will improve medical student well-being	15.6%	37.5%	46.9%
As a result of changing USMLE Step 1 to P/F:
I will now require applicants to submit Step 2 CK scores with ERAS	82.8%	7.8%	9.4%
Where an applicant goes to medical school will be more important in screening and selection for my program	52.4%	22.2%	25.4%
There will be more students applying to plastic surgery residency	53.1%	29.7%	17.2%
It will be more important for applicants to take time off from medical school to participate in research	45.3%	34.4%	20.3%
Step 2 CK should also be changed to P/F	4.8%	9.5%	85.7%

Current and Future Metrics for Selecting Applicants for Residency Interviews

Respondents overwhelmingly selected numeric Step 1 scores (n = 185), letters of recommendation (n = 184), and away rotations (n = 136) as the current top 3 most important metrics (MIMs) that programs would consider when offering a residency interview (Table 5). Research was the 4th most selected MIM (n = 77). When asked to rank the top 3 MIMs following the USMLE Step 1 scoring change, respondents selected letters of recommendation (n = 189), away rotations (n = 134), and numeric Step 2 CK scores (n = 129) (Table 5). Research (n = 87) and an applicant’s medical school (n = 75) were the 4th and 5th most selected MIMs after the score reporting change. The MIMs before the change differed significantly from those after the change (P = 0.005). After the change to P/F scoring, respondents ranked Step 1 score as an expected top 3 MIM 184 less times (−26.9%) and ranked numeric Step 2CK scores as an expected top 3 MIM 124 more times (+18.1%).

Table 5.

Perceived Top Three Metrics for A Residency Interview before and after the Step 1 Score Reporting Change*

Most Important Metrics with Numeric Step 1 Score								Most Important Metrics with P/F Step 1 Score
Metric (in descending frequency)	Total (n)	#1		#2		#3		Metric (in descending frequency)	Total (n)	#1		#2		#3
		n	%†	n	%†	n	%†			n	%†	n	%†	n	%†	Δ
Letters of recommendation	185	57	25.0	91	39.9	37	8.0	Letters of recommendation	189	74	32.5	91	39.9	24	10.5	+4
Step 1 score (numeric)	184	83	36.4	54	23.7	47	20.6	Away rotation	134	73	32.0	37	16.2	24	10.5	−2
Away rotation	136	68	29.8	34	14.9	34	14.9	Step 2CK score (numeric)	129	45	19.7	36	15.8	48	21.1	+124
Research	77	0	—	18	7.9	59	25.9	Research	87	6	2.6	24	10.5	57	25.0	+10
Applicant’s medical school	28	5	2.2	10	4.4	13	5.7	Applicant’s medical school	75	17	7.5	22	9.6	36	15.8	+47
Clerkship grades	27	0	—	9	3.9	18	7.9	Clerkship grades	35	4	1.8	8	3.5	23	10.1	+8
Step 2CK score (numeric)	5	1	0.4	4	1.8	0	—	Step 1 score (P/F)	1	0	—	0	—	1	0.6	−184

*Metrics selected < 5 times for both pre- and post-reporting change are not represented.

+column percentage.

Δ = Number of selections with P/F Step 1 score – number of selections with numeric Step 1 score.

DISCUSSION

In February 2020, the Federation of State Medical Boards and the National Board of Medical Examiners announced that USMLE Step 1 results will be reported as P/F as early as 2022. Key drivers of this change were to reduce overemphasis of Step 1 scores in the residency application process and to minimize the influence of demographic differences in Step 1 performance.[17] As a result, the pathway to plastic surgery training will significantly change. Our survey describes key stakeholders’ perspectives on this change to the plastic surgery application process.

Plastic surgery is one of the most competitive specialties[2,4,18] with match rates for US seniors of 74% in 2019[19] and 62% in 2020.[20] Matched integrated applicants often have high USMLE scores, multiple research publications, and above-average rates of Alpha Omega Alpha membership.[3,4,21] USMLE scores even correlate with success in the independent applicant match, despite being nearly a decade old.[22] Nearly all PDs (94%) currently use Step 1 scores for residency interview selection, with a mean importance rating of 4.1/5.0.[2] Therefore, it is crucial to understand how plastic surgery PDs and applicants will respond to binary USMLE Step 1 scoring.

The USMLE scoring change is unlikely to reduce the competitiveness of the integrated plastic surgery match. Meanwhile, most applicants and PDs are not in favor of the scoring change.[23] With 94% of US medical students passing the USMLE Step 1 on the first attempt,[24] PDs will need to use other metrics to distinguish accomplished applicants. Most applicants suspect—and PDs agree—that numeric Step 2 CK scores will replace Step 1 as an objective measure for programs to screen and compare applicants. This delays the stress of high-stakes testing for medical students. Overall, applicants’ and PDs’ perspectives captured by our survey suggest the proposed goal of reducing “…the current overemphasis on USMLE performance” will not be accomplished by this scoring change alone.

A priority of the Invitational Conference on USMLE Scoring was to address demographic differences that exist in USMLE performance. Historically, Step 1 has produced statistically significant gender differences, with men scoring higher than women, even when controlling for undergraduate performance measures.[10] Furthermore, Black and Latinx students score lower on average than their White and Asian counterparts.[10,15] In plastic surgery, it is common to use Step 1 “cut-off” scores to screen applications, with one study of plastic surgery PDs revealing that approximately 50% utilize a median cut-off score between 220 and 229[11]. By eliminating the Step 1 numeric score, most respondents feel that the number of applicants to integrated plastic surgery will increase. As the mean USMLE Step 1 score for matched integrated plastic surgery applicants has remained steadily above the 80th percentile for over a decade,[2,25,26] the scoring change may result in a more diverse set of applicants applying for plastic surgery. While the ultimate influence on applicant diversity remains unclear, we are hopeful that the scoring change can open the field of plastic surgery to applicants from more diverse backgrounds.

The reporting change could, however, magnify socioeconomic disparities in the application process. Most respondents believe that programs will prioritize applicants from more prestigious medical schools after Step 1 is reported as P/F. Students from higher socioeconomic status are more likely to attend prestigious medical schools,[27] suggesting the scoring change may benefit this group at the expense of students from lower socioeconomic status. Students from less prestigious medical schools, as well as international medical graduates, may be at a greater disadvantage. As one PD replied, “This [change] is going to hurt [applicants] from lower income backgrounds, who have the knowledge and skills to do well, but don’t come from a top-tier program and thus can’t get letters from big names and can’t afford the cost of multiple away rotations.”

Several factors already impose financial burdens on plastic surgery applicants, including completing away rotations, applying broadly, and dedicating time to research. While none of these factors are mandatory, all correlate with a successful match.[28-31] Integrated applicants already complete a mean of 2.1 away rotations, spending an average of $3,591 on each rotation[28,30,32] and may feel obligated to pursue more away rotations if more students enter the applicant pool. Second, applicants may feel obligated to apply more broadly to maximize their chances of a successful match. In 2018, matched applicants ranked 13.7 programs,[2] and as a result, the plastic surgery residency interview process can regularly cost $10,000 or more.[31] Lastly, respondents feel that a dedicated research experience will become more important.[11,29] Limited access to dedicated research experiences due to time and cost may exclude diverse applicants who do not have the means or ability to pursue such opportunities. The resulting financial burden of these secondary effects cannot be ignored, especially as these costs may exacerbate socioeconomic disparities. The plastic surgery community should mitigate financial barriers to entry, or else risk excluding well-suited candidates from socioeconomically diverse backgrounds. Ultimately, additional reforms are needed to address the racial and socioeconomic disparities present in the residency selection process.

Several limitations may lessen the applicability of these results. First, our results may not represent all integrated plastic surgery applicant population, as the applicant response rate was 33.2%. Respondents identifying as White or Caucasian were moderately over-represented (69.5% compared with 53% from the Electronic Residency Application Service (ERAS) 2015–2019 demographics data). Respondents identifying as Black or African American or Hispanic, Latino, or Spanish origin were moderately under-represented (survey = 3.7% and 5.5% respectively, compared with 6.2% and 8.4% respectively from ERAS demographics data). Additionally, women were modestly over-represented (47.6% compared with 41.2% from ERAS demographics data). Second, the applicant survey did not differentiate between US graduates and international medical graduates or between matched and unmatched. Responses may differ between these groups, though the purpose of our study was to characterize general sentiments among the entire applicant pool. Finally, our data are limited by the applicability of prospective psychometric data assessing opinions, attitudes, and forward-looking perspectives of recent applicants. These conclusions will need to be investigated longitudinally to assess the ultimate outcomes of this scoring change.

Historically, USMLE Step 1 numeric scores have played a major role in the plastic surgery training pathway.[33] As scores will be reported as P/F, it is critical for plastic surgery programs and future applicants to consider other metrics for distinction. Applicants and PDs both anticipate that Step 2 CK, research, and an applicant’s medical school will replace the Step 1 numeric score in applicant evaluation. While it remains to be seen whether this paradigm shift will improve residency selection, we are hopeful that additional reforms can produce a more equitable plastic surgery application process.

ACKNOWLEDGMENT

The authors thank Benjamin Sarac for his assistance in identifying eligible applicant survey respondents.