Pass/Fail USMLE Step 1 Scoring-A Radiology Program Director Survey.

RATIONALE AND
OBJECTIVES: In February 2020, administrators for the US medical licensing examination (USMLE) announced that Step 1 reporting would change to pass/fail in hopes of reducing the overemphasis of USMLE performance on the residency selection system and improving medical student well-being. Our objective was to determine the perspectives of diagnostic radiology (DR), interventional radiology (IR), and nuclear medicine (NM) program directors (PDs) regarding pass/fail USMLE Step 1 scoring.
MATERIALS AND METHODS: A survey composed of thirteen questions on a three-point Likert scale, five demographic questions, and a free-text question was distributed to 179 DR, 84 IR, and 34 NM PDs from ACGME-accredited residency programs.
RESULTS: In total, 140 unique responses were obtained (response rate = 47.1%). The PD respondents had a male predominance of 79.1%, average age of 46 ± 7.2 years, and average tenure of 5.9 ± 5.2 years. A majority of PDs (69.6%) disagreed that the change is a good idea, and a minority (21.6%) believe the change will improve medical student well-being. Further, 90.7% of PDs believe a pass/fail format will make it more difficult to objectively compare applicants and most will place more emphasis on USMLE Step 2 scores and medical school reputation (89.3% and 72.7%, respectively).
CONCLUSION: The lasting impact of pass/fail Step 1 scoring are uncertain and many radiology PDs do not support this change. While the central motivations to reduce the overemphasis on USMLE Step 1 performance and improve medical student well-being are admirable, it remains to be seen if pass/fail scoring will accomplish these goals.

Introduction

Residency programs will soon be without one the most historically used objective metrics for residency applications. Administrators of the US medical licensing examination (USMLE) announced in February 2020 that Step 1 reporting will change from a three-digit numerical score to a pass/fail outcome starting as early as 2022. This change was founded in discussions occurring at the invitational conference on USMLE scoring (InCUS), a multi-stakeholder assembly in March 2019. InCUS explored issues in medical education and considered alternatives to the current USMLE exam structure and scoring. Specific recommendations resulting from InCUS included a reduced emphasis on USMLE scores, accelerated research on correlations between USMLE scores and clinical performance, and minimizing racial differences in USMLE scores. In the subsequent months, USMLE sponsors engaged educators, regulators, test-takers, and the public to provide feedback on the InCUS recommendations and reached the decision to convert Step 1 to a pass/fail outcome (1,2). This announcement highlights a system-wide change to improve the transition from undergraduate to graduate medical education and parallels the American Medical Association's “Reimagining Residency” initiative in supporting educational innovations in medical schools, residency programs, and health systems (1,3).

A major concern in medical education includes medical student and resident burnout, with reported prevalences of nearly 50% in fourth year medical students and second year residents (4,5). The outlook beyond residency training is correspondingly poor with 46% of radiologists reporting burnout, above the national physician average of 42% (6). Further, career choice regret in radiology is significantly higher than in internal medicine (4,7). Standardized tests, including the Maintenance of Certification and USMLE Steps 1 and 2 exams, are under scrutiny for possible contributions to student and physician burnout (8).

The USMLE sponsors believe a pass/fail format will reduce the overemphasis of Step 1 performance on the residency selection process and improve medical student well-being. Central to this decision was the fact that USMLE examinations were developed for medical licensure purposes and not evaluation of academic achievement (1). Further, medical student debt has been shown to influence an applicant's chosen residency specialty and the increasing costs of Step 1 registration and preparation have contributed to this debt (9).

In the last few decades, numerical Step 1 scores have played a significant role in screening and evaluating medical students for diagnostic radiology (DR), interventional radiology (IR), and nuclear medicine (NM) residency programs. In 2018, Step 1 scores were cited as the most prevalent factor in selecting applicants to interview, with 95% of DR programs directors (PD) and 100% of IR PDs citing the score (10). As many schools transition to pass/fail course grading and eliminate class rankings, residency programs may place more emphasis on the few remaining objective measures, such as USMLE scores.

In response to InCUS recommendations, the Association of Program Directors in Radiology announced in August 2019 their opposition to a pass/fail format despite agreement that numerical Step 1 scores should not be used as a screening tool for applicant comparison (11). Although additional stakeholders have expressed their opinions on this change, including radiology PDs, it remains unclear how binary Step 1 scoring will affect the residency application process and medical student well-being (12). This study was designed to determine the perspectives of DR, IR, and NM PDs regarding USMLE Step 1 pass/fail scoring.

Materials and Methods

This study is an ancillary report from an aggregate study investigating PD opinions from all specialties throughout the country (13). A nineteen-item survey was developed and validated through phases of pre-pilot and pilot testing. The final survey was distributed by the Research Electronic Data Capture platform, a secure data collection tool meeting HIPAA compliance standards. The anonymous survey consisted of thirteen questions on a three-point Likert scale, five demographic information questions (age, gender, tenure duration, program geography, and specialty), and one free text question. A link to the survey can be found in the supplementary materials. Free text responses were manually analyzed using a content-analysis thematic approach. Multiple themes could be applied to a single response.

After receiving exemption status from the Institutional Review Board, we invited PDs from 179 ACGME-accredited DR residency programs, 84 ACGME-accredited IR residency programs, and 34 ACGME-accredited NM residency programs to participate in the anonymous electronic survey. The PD list and contact information were generated from publically available information on the ACGME Accreditation Data System. Over the course of four weeks, four individualized requests for participation were sent. Data was analyzed on Microsoft Excel software with 95% confidence intervals and statistical significance levels of 0.05.

Results

In total, 140 unique responses were obtained (response rate = 47.1%). The average PD age was 46 ± 7.2 years with an average tenure as PD of 5.9 ± 5.2 years. The respondents had a male predominance of 79.1%. A majority of PDs (69.6%) disagreed that the change is a good idea, and a minority (21.6%) believe the change will improve medical student well-being. Further, 89.3% of PDs expect an increase in emphasis on USMLE Step 2 Clinical Knowledge (CK) scores. While only 5.0% of programs currently require Step 2 CK scores, 82.6% will require Step 2 CK scores with their Electronic Residency Application Service (ERAS) application when Step 1 is pass/fail. Only 7.2% believe USMLE Step 2 CK should also be changed to pass/fail. Last, 72.7% of PDs will place more importance on an applicant's medical school reputation (Table 1 ).

Table 1

Diagnostic Radiology, Interventional Radiology, and Nuclear Medicine Program Directors’ Perspectives of Reporting Step 1 as Pass/Fail

Statement	Disagree	NeutralPercent (95% Confidence Interval)	Agree
Changing the USMLE Step 1 to pass/fail:
Is a good idea	69.6 (61.9–77.2)*	22.5 (15.5–29.4)	8.0 (3.5–12.5)
Will improve medical student well-being	32.4 (24.6–40.2)	46.0 (37.8–54.3)	21.6 (14.7–28.4)
Will make it more difficult to objectively compare applicants	2.1 (0.0–4.5)	7.1 (2.9–11.4)	90.7 (85.9–95.5)*
Will make applicant screening more arduous	3.6 (0.5–6.6)	13.6 (7.9–19.2)	82.9 (76.6–89.1)*
Will increase emphasis on Step 2 CK scores in selecting applicants for my program	2.1 (0.0–4.5)	8.6 (3.9–13.2)	89.3 (84.2–94.4)*
Will put international medical graduates at a disadvantage	7.9 (3.4–12.3)	37.1 (29.1–45.1)	55.0 (46.8–63.2)*
Will decrease socioeconomic disparities in the application process	42.9 (34.7–51.1)	45.7 (37.5–54.0)	11.4 (6.2–16.7)
Will decrease medical student knowledge of the basic sciences	11.4 (6.2–16.7)	42.1 (34.0–50.3)	46.4 (38.2–54.7)
As a result of changing USMLE Step 1 to pass/fail:
Step 2 CK should also be changed to pass / fail	76.3 (69.2–83.3)*	16.5 (10.4–22.7)	7.2 (2.9–11.5)
I will now require applicants to submit Step 2 CK scores with ERAS	2.3 (0.0–4.8)	15.2 (9.0–21.3)	82.6 (76.1–89.0)*
Where an applicant goes to medical school will be more important in screening and selection for my program	6.5 (2.4–10.6)	20.9 (14.1–27.6)	72.7 (65.3–80.1)*

CK, clinical knowledge; ERAS, electronic residency application service.

Indicates a statistically significant (p < 0.05) plurality of responses by nonoverlapping 95% confidence intervals.

Of the 140 respondents, 43 PDs provided additional free response feedback on the USMLE change. The most common themes were decreased ability to both evaluate applicants and predict candidates who may struggle to successfully pass the American Board of Radiology Core Exam. Themes contained in the free text responses are found in Table 2 .

Table 2

Themes Contained Within Free Text Feedback Responses

Theme	n (%)
Decreased ability to evaluate applicants	18 (42%)
Inability to predict passing boards	11 (26%)
Less likely to accept applicants from osteopathic schools or international schools	9 (21%)
Decreased quality of applicants and profession	6 (14%)
Increased emphasis on medical student performance evaluations, class rankings, and letters of recommendation	4 (9%)
Increased emphasis on Step 2 CK	4 (9%)
Disservice to residents matching into programs and specialties and unable to meet the demands	3 (7%)

Percentages do not add up to 100% as multiple themes may have been present in each free text submission. Themes addressed by less than three respondents are not included.

CK, clinical knowledge.

Discussion

Many DR, IR and NM PDs disagree with shifting Step 1 to a pass/fail format, and a majority believe this change will decrease their ability to objectively evaluate applicants. Meanwhile, Step 2 CK will likely have greater emphasis as the only remaining objective data point to compare applicants. While this metric is currently only required by a minority of radiology programs, completion of Step 2 CK may soon be commonly required for ERAS submission.

Despite the intended goal of improving student well-being, most PDs do not believe a pass/fail format will achieve this goal. In fact, some PDs believe this change will redirect the stress of Step 1 to Step 2 CK, which often falls near the start of a student's residency application cycle. As many Step 2 CK scores are released shortly before residency applications are due, a poor score may have significant mental health consequences and may even require a shift in career trajectory.

Another concern of PDs is the decreased ability to predict applicant success in passing the American Board of Radiology Core Exam as low scores on USMLE Steps 1 and 2 have been correlated with Core Exam failure (14). Further, some PDs believe this change will be a disservice to residents who match into a specialty but subsequently are unable to meet the demands, requiring corrective action plans or risking failure to progress in their medical training.

One positive impact of this change may be an increase in underrepresented minority (URM) students matching into radiology. When Step 1 scores are utilized for screening applicants, Black, Hispanic, and female applicants are offered fewer interviews and residency spots (15). The field of radiology has struggled with racial, ethnic, and gender diversity, with URM and women accounting for approximately 7% and 20% of radiologists, respectively. Despite a more diverse medical student population, many radiology programs have failed to develop diversity and inclusivity (16, 17, 18). However, our data suggest that pass/fail Step 1 scoring will merely shift focus to Step 2 CK scores, which suffer from the same biases against URM students.

Another unforeseen consequence of a pass/fail Step 1 may be decreased access to radiology residency positions for students from osteopathic and international medical schools. This discrepancy may reinforce medical school hierarchy (i.e., reputation) and further disadvantage URM medical students. Research experiences and clinical electives may soon have a higher weighted importance, which may favor students from medical schools with greater funding and progressive curricula that enable early electives and longitudinal research opportunities. In the current medical education climate surrounding the coronavirus pandemic, there is a lack of opportunity for away rotations, which may further disadvantage applicants from osteopathic and international medical schools as well.

Survey research has important limitations, including nonresponse bias. PDs with negative perceptions of the scoring change may have been more motivated to complete the survey than those in support or neutral of the change. Nevertheless, this survey represents perspectives of nearly half of all radiology PDs in the United States. Likert scales are subject to the central tendency bias, in which people avoid extreme responses. Use of a larger scale may have minimized this bias and offered greater opportunity for statistical significance but may have resulted in a lower response rate. Directions for future research include medical student and radiology resident physician perspectives on the pass/fail reporting system and these data will benefit the radiology medical education community.

The lasting impacts of pass/fail USMLE Step 1 scoring are uncertain and many DR, IR, and NM PDs do not support this change. While the central motivations of the change to reduce the overemphasis on USMLE Step 1 performance and improve medical student well-being are admirable, it remains to be seen if this change will accomplish these goals.