The Change of USMLE Step 1 to Pass/Fail: Perspectives of the Surgery Program Director.

OBJECTIVE: This study sought to evaluate the perspectives of surgical program directors regarding the change of USMLE Step 1 to pass/fail grading.
DESIGN: Validated electronic survey.
SETTING: Vanderbilt University Medical Center Department of Plastic Surgery. PARTICIPANTS: Program directors of all ACMGE-accredited General Surgery, Integrated Vascular Surgery, Integrated Thoracic Surgery, and Integrated Plastic Surgery residency programs.
RESULTS: The overall response rate was 55.5%. Most PDs (78.1%) disagreed with the scoring change. Only 19.6% believe this change will improve medical student well-being. For 63.5% of PDs, medical school pedigree will become more important, and 52.7% believe it will place international medical graduates at a disadvantage. Only 6.2% believe Step 2 CK should also be pass/fail, while 88.7% will increase the weight of Step 2 CK and 88.4% will now require Step 2 CK score submission with the electronic residency application service.
CONCLUSIONS: While well-intentioned, changing USMLE Step 1 to pass/fail may have unintended consequences and may disadvantage certain groups of applicants. The emphasis on Step 1, and resulting test-taking apprehension, will likely shift to Step 2 CK. Proponents of equitable evaluation should direct their efforts toward increasing, not decreasing, the number of objective measures available for student assessment. Published by Elsevier Inc.

INTRODUCTION

The 3-part US Medical Licensing Examination (USMLE) was designed as a universal assessment tool for states to determine candidacy for medical licensure.1, 2, 3 Although licensing outcomes are binary, the Step 1, Step 2 CK, and Step 3 exams all report 3-digit scores. The Step 1 exam is taken during medical school, before residency applications are submitted, and is used to assess understanding of the “scientific principles required for maintenance of competence through lifelong learning.” Teams of scientists, educators, and clinicians ensure the validity of Step 1 questions, and the accompanying scores are critical to the surgical residency application process.5, 6, 7, 8, 9

Aside from its intended use, the Step 1 examination has evolved into a screening tool for residency programs across many specialties, , 10, 11, 12, 13, 14 and many surgical residency programs set target scores to select competitive interviewees.5, 6, 7, 8, 9 Surgical program directors (PDs) also use these scores to formulate rank lists , and to differentiate between applicants. , , Scores not only reflect an applicant's basic medical knowledge, but have also been correlated with academic performance during residency across surgical and nonsurgical fields. , 17, 18, 19, 20, 21, 22, 23, 24

As many US medical schools have transitioned to pass/fail preclinical grading systems, the numeric scoring of the USMLE Step 1 has been challenged. In addition, the test is known to cause substantial anxiety and has been implicated in fostering racial and demographic disparities. , To address these concerns, the Invitational Conference on USMLE Scoring (InCUS) was held in March of 2019. Intent on facilitating “broader, system-wide changes to improve the transition from undergraduate to graduate medical education.” One year later, in February 2020, the Federation of State Medical Boards and National Board of Medical Examiners made the decision to change USMLE Step 1 scoring to pass or fail. ,

A change of this magnitude carries important ramifications for surgical residencies that rely on the value of Step 1 scores. A recent national survey of all residency PDs demonstrated that the majority did not believe the change to pass/fail grading was a good idea. However, the viewpoints of surgical PDs have not been presented. The aim of this study was to survey PDs of all General Surgery and General Surgery-affiliated integrated residency programs regarding their attitudes toward the change in USMLE Step 1 scoring.

METHODS

After receiving exemption from the Vanderbilt Medical Center University Institutional Review Board (IRB# 200355) an electronic survey was developed in the Research Electronic Data Capture (REDCap) program (Nashville, TN). The survey was validated through prepilot and pilot testing prior to distribution. Internal validity was assessed (Cronbach's alpha = 0.87).

The American Council for Graduate Medical Education (ACGME) listing of accredited residency programs was queried for all active General Surgery residency programs, as well as all Integrated Thoracic, Vascular and Plastic Surgery residencies. This list yielded a total of 506 programs (28 Integrated Thoracic, 63 Integrated Vascular, 82 Integrated Plastic, 333 General Surgery). Contact information was unavailable for five General Surgery PDs and one Integrated Plastic Surgery PD. One PD declined to participate and 2 General Surgery PDs each directed 2 separate residency programs. In total, 497 unique PDs were surveyed (28 Integrated Thoracic, 62 Integrated Vascular, 81 Integrated Plastic, and 326 General Surgery).

The 19-item survey consisted of 8 questions regarding the change of Step 1 scoring, 4 questions regarding changes made as a result, 5 demographic questions, and 1 free response section. Four rounds of survey distribution proceeded over the course of 8 weeks and responses were recorded anonymously. Responses were analyzed with descriptive statistics and all values are listed as discrete numbers, means ± standard deviation (SD) or percentages with 95% or 99.9% confidence intervals (CI). Statistical significance in response plurality was determined by nonoverlapping confidence intervals.

RESULTS

Of the 497 unique PDs surveyed, 276 responded (response rate 55.5%). Individual response rates were as follows: General Surgery 54.3% (177/326), Integrated Plastic Surgery 65.4% (53/81), Integrated Vascular Surgery 56.5% (35/62), and Integrated Thoracic Surgery 39.3% (11/28). Overall, the mean PD age was 50.7 ± 7.9 years and respondents were mostly male (78% male, 21% female, 1% nonbinary). Average PD tenure was 6.6 ± 5.26 years. Programs were located in the Northeast (27%), South (33%), Midwest (25%), and West (15%).

Overall, 78.1% of PDs did not believe changing USMLE Step 1 to pass/fail was a good idea. Eighty-eight percent believe that this change will make it more difficult to objectively compare residency applicants, and 85.4% believe it will make applicant screening more arduous. The majority (52.7%) believe this change will place International Medical Graduates (IMGs) at a disadvantage. Most (53.6%) do not believe the change will decrease socioeconomic disparities in the application process. PDs were largely neutral regarding its effect on basic science knowledge. Only 19.6% believe this change will improve medical student well-being. While most PDs are neutral regarding the addition of a supplementary application, 88.4% will now require applicants to submit Step 2 CK scores with the electronic residency application service (ERAS). This contrasts with only 5% (25/497) of PDs who already do this. The majority of respondents (88.7%) will increase the weight of USMLE Step 2 CK in the application process and only 6.2% believe Step 2 CK should also be pass/fail. Finally, 63.5% of PDs agree that the importance of which medical school an applicant attends will become more important during the screening process (Table 1 ).

Table 1

Perspectives of Surgery Residency Program Directors Regarding the Change of USMLE Step 1 to Pass/Fail

Statement	Disagree	Neutral	Agree
Changing the USMLE Step 1 to pass/fail
Is a good idea	78.1% [69.9-86.4]*	17.4% [9.8-25.0]	4.4% [0.3-8.6]
Will make it more difficult to objectively compare applicants	5.8% [1.2-10.5]	5.8% [1.2-10.5]	88.3% [81.9-94.7]*
Will increase emphasis on Step 2 CK scores in selecting applicants for my program	4.4% [0.3-8.4]	6.9% [1.9-11.9]	88.7% [82.5-95.0]*
Will put IMGs at a disadvantage	12.8% [6.2-19.5]	34.4% [25.0-43.9]	52.7% [42.8-62.7]
Will make applicant screening more arduous	6.9% [1.9-12.0]	7.7% [2.4-13.0]	85.4% [78.4-92.4]*
Will decrease socioeconomic disparities in the application process	53.6% [43.7-63.5]	40.2% [30.5-49.9]	6.2% [1.4-10.9]
Will decrease medical student knowledge of the basic sciences	21.1% [13.0-29.2]	45.8% [35.9-55.7]	33.1% [23.8-42.4]
Will improve medical student well-being	44.6% [34.7-54.5]	35.9% [26.4-45.4]	19.6% [11.7-27.4]
As a result of changing USMLE Step 1 to pass/fail
I will add a supplementary application for my residency program	36.1% [26.4-45.9]	42.6% [32.6-52.6]	21.3% [13.0-29.6]
I will now require applicants to submit Step 2 CK scores with ERAS	4.8% [0.4-9.3]	6.8% [1.6-12.1]	88.4% [81.7-95.0]*
Where an applicant goes to medical school will be more important in screening and selection for my program	12.4% [5.8-19.1]	24.1% [15.4-32.7]	63.5% [53.8-73.2]*
Step 2 CK should also be changed to pass / fail	84.0% [76.7-91.3]*	9.8% [3.9-15.7]	6.2% [1.4-11.0]

USMLE, United States Medical Licensing Examination; Step 2 CK, Step 2 Clinical Knowledge; IMG, International Medical Graduate; ERAS, Electronic Residency Application Service.

Responses are recorded in percentages, followed by respondents over total responders and then 99.9% confidence intervals.

Indicates a statistically significant (p < 0.01) plurality of responses by nonoverlapping 99.9% confidence intervals. Bolded values indicate majority responses in the setting of a statistically significant plurality of responses.

Responses were compared among PD age, gender, tenure, program region and program specialty (Table 2 , Appendices A-D). Responses did not differ between demographic groups. The free response rate was 32% (88/276) and responses focused on disapproval of the change, citing Step 1 as a valuable piece of objective data. Many respondents were concerned with the shift of testing burden onto Step 2 CK and the negative impact on IMGs, Osteopathic applicants, those from less well-known medical schools and those from lower socioeconomic backgrounds.

Table 2

Perspectives of Surgery Residency Program Directors Regarding the Change of USMLE Step 1 to Pass/Fail – By Specialty.

Statement	Disagree				Neutral				Agree
	GS	PRS	VASC	THOR	GS	PRS	VASC	THOR	GS	PRS	VASC	THOR
Changing the USMLE Step 1 to pass/fail
Is a good idea	79.3% [73.3-85.3]*	82.7% [72.4-93.0]*	72.7% [57.5-87.9]*	54.5%	17.2% [11.6-22.9]	13.5% [4.2-22.7]	21.2% [7.3-35.2]	27.3%	3.4% [0.7-6.2]	3.8% [0.0-9.1]	6.1% [0.0-14.2]	18.2%
Will make it more difficult to objectively compare applicants	2.8% [0.4-5.3]	11.3% [2.8-19.9]	9.1% [0.0-18.9]	18.2%	5.6% [2.2-9.1]	3.8% [0.0-8.9]	6.1% [0.0-14.2]	18.2%	91.5% [87.4-95.6]*	84.9% [75.3-94.5]*	84.8% [72.6-97.1]*	63.6%
Will increase emphasis on Step 2 CK scores in selecting applicants for my program	4.0% [1.1-6.9]	7.5% [0.4-14.7]	2.9% [0.0-8.4]	0%	4.5% [1.5-7.6]	1.9% [0.0-5.5]	20.0% [6.7-33.3]	27.3%	91.5% [87.4-95.6]*	90.6% [82.7-98.4]*	77.1% [63.2-91.1]*	72.7%
Will put IMGs at a disadvantage	9.7% [5.3-14.1]	11.5% [2.9-20.2]	25.7% [11.2-40.2]	27.3%	34.3% [27.3-41.3]	40.4% [27.0-53.7]	31.4% [16.0-46.8]	18.2%	56.0% [48.6-63.4]*	48.1% [34.5-61.7]	42.9% [26.5-59.3]	54.5%
Will make applicant screening more arduous	4.5% [1.5-7.6]	13.5% [4.2-22.7]	5.7% [0.0-13.4]	18.2%	7.4% [3.5-11.3]	11.5% [2.9-20.2]	5.7% [0.0-13.4]	0%	88.1% [83.3-92.9]*	75% [63.2-86.8]*	88.6% [78.0-99.1]*	81.8%
Will decrease socioeconomic disparities in the application process	50.8% [43.5-58.2]	54.7% [41.3-68.1]	60.0% [43.8-76.2]	72.7%	41.8% [34.5-49.1]	39.6% [26.5-52.8]	37.1% [21.1-53.2]	27.3%	7.3% [3.5-11.2]	5.7% [0.0-11.9]	2.9% [0.0-8.4]	0%
Will decrease medical student knowledge of the basic sciences	18.2% [12.5-23.9]	22.6% [11.4-33.9]	28.6% [13.6-43.5]	36.4%	47.7% [40.3-55.1]	50.9% [37.5-64.4]	37.1% [21.1-53.2]	18.2%	34.1% [27.1-41.1]	26.4% [14.5-38.3]	34.3% [18.6-50.0]	45.5%
Will improve medical student well-being	44.6% [37.3-52.0]	45.3% [31.9-58.7]	40.0% [23.8-56.2]	54.5%	36.7% [29.6-43.8]	35.8% [22.9-48.8]	34.3% [18.6-50.0]	27.3%	18.6% [12.9-24.4]	18.9% [8.3-29.4]	25.7% [11.2-40.2]	18.2%
As a result of changing USMLE Step 1 to pass/fail
I will add a supplementary application for my residency program	25.7% [19.1-32.4]	56.9% [43.3-70.5]	50% [33.2-66.8]	54.5%	47.3% [39.7-54.9]*	33.3% [20.4-46.3]	35.3% [19.2-51.4]	36.4%	26.9% [20.2-33.7]	9.8% [1.6-18.0]	14.7% [2.8-26.6]	9.1%
I will now require applicants to submit Step 2 CK scores with ERAS	2.0% [0.0-4.2]	9.4% [1.6-17.3]	6.3% [0.0-14.6]	18.2%	5.9% [2.2-9.6]	7.5% [0.4-14.7]	12.5% [1.0-24.0]	0%	92.2% [87.9-96.4]*	83.0% [72.9-93.1]*	81.3% [67.7-94.8]*	81.8%
Where an applicant goes to medical school will be more important in screening and selection for my program	10.1% [5.5-14.6]	26.9% [14.9-39.0]	2.9% [0.0-8.6]	9.1%	26.6% [20.0-33.3]	17.3% [7.0-27.6]	17.6% [4.8-30.5]	36.4%	63.3% [56.0-70.6]*	55.8% [42.3-69.3]*	79.4% [65.8-93.0]*	54.5%
Step 2 CK should also be changed to pass / fail	85.8% [80.6-91.0]*	86.8% [77.7-95.9]*	74.3% [59.8-88.8]	72.7%	9.1% [4.8-13.3]	11.3% [2.8-19.9]	14.3% [2.7-25.9]	0%	5.1% [1.9-8.4]	1.9% [0.0-5.5]	11.4% [0.9-22.0]	27.3%

GS, General Surgery; PRS, Plastic and Reconstructive Surgery; VASC, Vascular Surgery; THOR, Thoracic Surgery; USMLE, United States Medical Licensing Examination; Step 2 CK, Step 2 Clinical Knowledge; IMG, International Medical Graduate; ERAS, Electronic Residency Application Service.

Responses are recorded in percentages, followed by respondents over total responders and then 95% confidence intervals.

Indicates a statistically significant plurality of responses by nonoverlapping 95% confidence intervals. Integrated Thoracic Surgery program data were not analyzed for response plurality due to limited sample size. Bolded values indicate majority responses in the setting of a statistically significant plurality of responses.

DISCUSSION

Over 90% of US medical graduates pass the USMLE Step 1 on the first attempt. , As medical schools transition to pass/fail preclinical grading systems, , students have fewer objective measures by which to differentiate themselves from their peers. The Step 1 scoring change was proposed to mitigate USMLE-based applicant screening and encourage a more balanced assessment of interviewees. In 2019, the InCUS conducted a large survey to obtain feedback on their 3 core recommendations. Contrary to most assistant/associated deans and course/clerkship directors, only 26% of residency PDs agreed that a scoring change would reduce USMLE-mediated residency screening and selection practices. Our study suggests that surgical PDs have concerns regarding the impact of this change. Most (78.1%) PDs disagree with this change and 88.3% believe it will make it more difficult to objectively compare applicants.

In the field of General Surgery, USMLE Step 1 scores not only aid in applicant selection and ranking, but have also been correlated with resident academic performance. Some studies have portrayed Step 1 scores as predictors of resident matriculation, clinical and academic performance during General Surgery residency, and even manual dexterity. , The test is also associated with performance on the American Board of Surgery In-Service Training Examination (ABSITE) more so than Step 2 CK or Step 3—as well as performance on the American Board of Surgery Qualifying Examination (QE). , , 33, 34, 35, 36, 37, 38, 39 As long as QE performance is considered by the ACGME during program accreditation, the correlation with USMLE performance will be considered by PDs. Integrated Plastic Surgery, Vascular Surgery, and Thoracic Surgery , residency programs all report using Step 1 scores to assist in applicant selection as well, and a recent study demonstrated that 81% of Plastic Surgery PDs disagree with the scoring change. Interestingly, Step scores have even been associated with physician professionalism ratings and risk for state board disciplinary action.

Although the 3-digit score has proven useful in many ways, it has also been criticized as a catalyst for medical student burnout and depression. While the pass/fail change was aimed at mitigating the psychological consequences of high-pressure testing, only 19.6% of surgery PDs believe the change will improve student wellness. Most expect Step 2 CK will ultimately replace Step 1 with regards to applicant screening, simply shifting the test-taking anxiety to a different exam. Furthermore, only 6.2% felt that Step 2 CK scores should also be changed to pass/fail, further emphasizing the importance of objective measures during consideration for surgical training positions.

In a positive light, the pass/fail scoring change may encourage more students to apply for traditionally “competitive” training positions. Those concerned with obtaining prohibitively low scores may now have the opportunity to build a more well-rounded application and focus on the Step 2 CK, which is predominantly centered on clinical medicine as opposed to its basic science counterpart, Step 1.

Surgery PDs expressed concerns that pass/fail exam scoring may worsen socioeconomic status (SES) disparities in the application process. In the United States, SES and race are intimately linked, and those from lower SES backgrounds may opt to attend a less prestigious medical school for financial reasons. Without objective measures of achievement like Step 1 scores, these students lose an opportunity to distinguish themselves from students graduating from more prestigious schools. The results of this study support that concern, as 63.5% of respondents agree that medical school pedigree will become more important. This is particularly important for IMGs as foreign medical schools are uniformly less well-known than US schools, and IMGs rely on Step scores to distinguish themselves from their US counterparts. Most (52.7%) surgery PDs reported that the Step 1 scoring change will place IMGs at further disadvantage. With the decline of measures for objective assessment, the importance of visiting rotations—an out-of-pocket applicant expense , —will likely increase. Other subjective measures may also become more important, such as the Medical Student Performance Evaluation and letters of recommendation. Letters of recommendation are also prone to inequity as students may not have access to prominent letter writers, especially in the setting of visiting rotations that were unaffordable or have since been cancelled due to COVID-19. Indeed, only 13.4% of surgery PDs believe the scoring change will decrease socioeconomic disparities in the application process. Taken together, these findings raise concern that the change may exacerbate disadvantages that applicants from lower socioeconomic backgrounds already face.

Limitations of this study include the use of psychometric data. This study reports PD opinions that are constitutionally biased (i.e., perceptions). Future studies should focus on capturing objective data after the change of USMLE Step 1 scoring to pass/fail. While our data is subjective, it represents the opinions of those personally involved in the residency selection process. We hope this study helps both PDs and applicants understand the potential implications of this change in medical education. The Coalition for Physician Accountability, whose goal is to assist in the transition from undergraduate to graduate medical education, will play a critical role in managing the effects of the scoring change along with other hardships that occur during this transition, all of which will be augmented by the ongoing viral pandemic.

CONCLUSIONS

The USMLE and surgical residency application processes are far from perfect. A student's cognitive abilities cannot be represented by a single number, and yet, a scarcity of objective data has likely contributed to overemphasis of Step 1 scores. With the change to pass/fail scoring, the emphasis on Step 1, and associated test-taking apprehension, will likely shift to Step 2 CK. This change begins the long-awaited evolution of the transition from undergraduate to graduate medical education; however, it is just the commencement and further refinements are needed. Proponents of equitable applicant evaluation should direct their efforts towards increasing, not decreasing, the number of objective measures available for comprehensive assessment.

AUTHOR CONTRIBUTIONS

Study conception and design: Pontell, Makhoul, Kumar, Drolet

Acquisition of data: Pontell, Makhoul, Kumar, Drolet

Analysis and interpretation of data: Pontell, Makhoul, Kumar, Drolet

Drafting of manuscript: Pontell, Makhoul, Kumar, Drolet

Critical revision: Pontell, Makhoul, Kumar, Drolet