Internal medicine residency program director perceptions of USMLE Step 1 pass/fail scoring: A cross-sectional survey.

Introduction

Background/rationale

On February 12, 2020, the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME) announced that score reporting for Step 1 of the United States Medical Licensing Examination (USMLE) would change from the three-digit numeric score to reporting a pass/fail outcome–starting no earlier than January 2022. At this time, Step 1 is one of the most important factors used to screen applicants by internal medicine residency programs, due to the high volume of applicants.[ In 2020, internal medicine programs received the most number of applications among all specialties, with 13,118 applicants competing for 8697 positions.[ The number of applicants is projected to continue to grow in future years.[ With the absence of a scored Step 1, internal medicine residency programs will need to utilize other methods to screen applicants for an interview.

Objectives

The purpose of this study was to investigate the perceptions among internal medicine residency program directors (PDs) on the change of Step 1 from a graded to a pass/fail examination. We also aimed to investigate how the change would impact the other factors normally considered for internal medicine residency selection.

Methods

Study design

This study was a multi-center, cross-sectional, 27-item electronic research survey. The College of Medicine's Institutional Review Board granted exempt status (ID#: STUDY00015130).

Setting

We obtained publicly available contact information for PDs at all active Accreditation Council for Graduate Medical Education (ACGME) internal medicine residency programs through the American Medical Association's Fellowship and Residency Electronic Interactive Database (FREIDA Online).

Participants

We identified and contacted 554 U.S. internal medicine residency PDs. A survey invitation email with the appropriate informed consent information was sent to each PD. Submission of the survey indicated the respondents’ consent to participate. The electronic survey was distributed on July 8, 2020. Follow-up emails were sent 2 and 4 weeks after the initial submission. The survey was closed on August 5, 2020.

Variables

This anonymous survey was developed, housed, and distributed through REDCapTM. The survey was developed using criteria analyzed in the 2018 National Resident Matching Program (NRMP) PD survey, which investigated the factors involved in selecting applicants to interview.[

Data sources/measurement

Our survey questions used single-answer multiple-choice, multiple-answer multiple-choice, a Likert scale ranging from 1 (strongly disagree or least important) to 5 (strongly agree or most important), and free text formats.

Bias

Prior to distribution, the questions were pretested and tested with subsets of medical students and physicians, then adjusted and readjusted for validation. It was then sent to PDs of a different specialty before final revisions were made. In total, the survey underwent five revisions prior to distribution. All PDs were contacted at verified email addresses, and responses were recorded anonymously.

Study size

All 554 U.S. internal medicine residency PDs were contacted via email.

Quantitative variables

We collected the PDs’ answers from the anonymous survey and conducted statistical analysis to determine their perceptions of USMLE Step 1 changing from a scored to pass/fail examination.

Statistical methods

Responses were analyzed with descriptive statistics and percentages. We defined statistical significance (P < .01) by non-overlapping 99.9% confidence intervals.

Results

Participants and descriptive data

About 554 programs were contacted, and 548 programs were included in this study due to invalid email accounts. The survey had 123 respondents, for a response rate of 22.4%. 58 programs responded after the first email, 47 after the follow-up email, and 18 programs after the final email. The survey is shown in Supplement 1.

Outcome data and main results

How do internal medicine PDs feel about the pass/fail Step 1?

The majority of PDs do not support the change of Step 1 to pass/fail (74.0%), and feel that the pass/fail transition was not transparent (78.9%). Yet, only 39.0% of PDs think that a graded Step 1 adequately measures an applicant's ability to succeed in internal medicine. See Table 1.

Table 1

Internal Medicine Program Directors’ response to the pass/fail Step 11.

	Agree	Neutral	Disagree
I support the change to pass/fail Step 1	12.2 (2.5–21.9)	13.8 (3.6–24.1)	74.0 (61.0–87.0)∗
Decision to transition to pass/fail Step 1 was transparent and involved stakeholders	6.5 (0–13.8)	14.6 (4.1–25.1)	78.9 (66.7–91.0)∗
A graded Step 1 adequately measures ability of applicant to succeed in internal medicine	39.0 (24.6–53.5)	30.9 (17.2–44.6)	30.1 (16.5–43.7)
Pass/fail Step 1 will help to create better future physicians	10.6 (1.4–19.7)	32.5 (18.6–46.4)	56.9 (42.2–71.6)
Pass/fail Step 1 will make it easier for residency programs to select which applicants to interview and accept	24.4 (11.6–37.1)	7.3 (0–15.0)	68.3 (54.5–82.1)∗
Pass/fail Step 1 will allow the match process to be fair and meritocratic	22.0 (9.7–3.4)	18.7 (7.1–30.3)	59.3 (44.8–73.9)∗

How will a pass/fail Step 1 impact internal medicine residency programs?

A majority feel that the pass/fail Step 1 will make it harder for residency programs to select which applicants to interview (68.3%), and that the pass/fail Step 1 will not allow the match process to be fair and meritocratic (59.3%). See Table 1.

How will the pass/fail change impact the importance of the factors reviewed by internal medicine residency programs when assessing applicants?

A majority of PDs believe that the pass/fail Step 1 exam result will be less important as a selection factor (71.5%). They believe that a greater importance will be assigned to Step 2 Clinical Knowledge (CK) exam results (87.8%), personal knowledge of the applicant (66.7%), grades in required clerkships (66.7%), and audition elective/rotation with their own department (64.2%). About half of PDs believe that letters of recommendation from known medicine faculty (56.1%), Medical Student Performance Evaluation (MSPE)/Dean's Letter (52.0%), and graduation from a highly-regarded U.S. medical school (51.2%) will rise in importance.

Most PDs believe that there will be no change in the impact of a student's research experience (81.3%), personal statement (78.0%), letters of recommendation from non-medicine faculty or unrecognized medicine faculty (78.0%), volunteer/extracurricular experience (71.5%), Gold Humanism Society membership (71.5%), Alpha Omega Alpha status (70.0%) or Step 2 Clinical Skills (CS) exam result (53.7%). See Table 1.

How will various applicant groups be affected by the change to a pass/fail Step 1?

A majority of PDs believe that MD students who attend highly regarded medical schools will have an advantage (55.3%) and that international medical graduates (82.9%), MD students who do not attend highly regarded medical schools (63.4%), and osteopathic (DO) students (52.0%) will be at a disadvantage. See Table 1.

How will changing to a pass/fail Step 1 affect medical students interested in internal medicine?

A majority of PDs do not think that changing the Step 1 to pass/fail would have any effect on medical students’ research interests, extracurricular/leadership involvement, hobbies, number of away rotations or applications to other specialties. However, a majority of PDs believe that a pass/fail Step 1 will encourage students to apply to more internal medicine residency programs (56.9%). See Table 2. A statistically significant majority of PDs also believe that there should be a cap on the number of residency applications a medical student can submit (54.5%), with suggestions ranging from 6 to 50 residency applications per student. See Table 1.

Table 2

How do you think changing to a pass/fail Step 1 will affect medical students interested in internal medicine?.

	n	%
Allow students to focus more on learning medicine rather than studying for Step 1	32	26.0
Encourage more research experiences	12	9.8
Encourage more extracurricular/leadership involvements	23	18.7
Allow students to pursue more hobbies or self-development	26	21.1
Encourage students to attend more rotations in medicine (away or sub-internships)	36	29.3
Encourage applicants to apply to more residency programs	70	56.9
Encourage applicants to consider or apply to other specialties in addition to internal medicine	48	39.0

What are the future implications on residency applications and medical education?

Because Step 1 is going to be pass/fail, about half of the PDs believe that medical schools should adopt a graded pre-clinical curriculum (51.2%).

Discussion

Key results

This study reveals that most internal medicine PDs do not support the Step 1 change to pass/fail and expect the selection process to be more difficult and less fair and meritocratic. They expect Step 2 CK, clerkship grades, personal knowledge of the applicant and audition electives to rise in importance. The PDs also believe that allopathic medical students not attending highly regarded medical schools, osteopathic students and international medical graduates will be disadvantaged with a pass/fail Step 1. The majority of PDs recommend limiting the number of residency applications per medical student to < 50.

Interpretation and generalizability

The USMLE Step 1, as developed in 1992, is the first of a three-part series created for licensing eligibility.[ However, with a number of additional conglomerate variables, the numeric scoring of this national exam, has resulted in unintended consequences. A few major changes in medical education and residency training since 1992 have forced the over-magnification of Step 1 numeric scores in assessing residency applicants. First, many medical schools have adopted a pass/fail pre-clinical curriculum.[ Second, there has been an increase in medical school matriculants, without a corresponding increase in the number of federally funded residency positions. And third, the Match process requires PDs and medical students to submit ranking lists.[ These changes have resulted in student fear of “not matching”. As a result, the average number of residency applications has increased from single digits to 60.3 applications per medical student.[ With fewer standardized objective criteria, and more applicants, 95% of PDs use Step 1 scores to screen and filter applicants for residency interviews with 70% of internal medicine residency programs enforcing a cutoff score.[ There is moderate correlation between failing Step 1 or scoring in the lowest quartile and having difficulty passing the American Board of Internal Medicine.[ Despite the limited evidence supporting a numerically scored Step 1's ability to screen for successful residents or future successful physicians, internal medicine residency programs have placed a strong emphasis on Step 1 scores.[ Currently, internal medicine PDs use USMLE Step 2 CK scores, internal medicine clerkship grades and USMLE Step 1 scores as their top three criteria for deciding to offer applicant interviews.[

The trickle-down effect of the emphasis placed on Step 1 scores is the unanticipated parallel curriculum which is time consuming and expensive for the students.[ Because of minimal standardizations in grades, clerkship grade inflation, and lack of class rank in medical schools, PDs place a greater emphasis on Step 1 scores as one of the few objective measures to assess applicants.[ Our study also reflected PDs’ desires for concrete evaluation metrics, with a statistically significant majority of internal medicine PDs not supporting the pass/fail change, and over half of them advocating for a graded pre-clinical curriculum.

The Invitational Conference on USMLE Scoring and numerous commentaries have asserted that the change to a pass/fail Step 1 may simply lead to the scored Step 2 CK becoming the new screening metric.[ A significant majority of internal medicine residency PDs in our survey agree that the Step 2 score will become more important, replacing the emphasis on Step 1. The literature suggests that there is a better correlation between Step 2 scores and clinical performance than there is between Step 1 scores and clinical performance.[ In addition, internal medicine programs will place a greater importance on personal knowledge of the applicant, grades in required clerkships, and audition elective/rotation with their own department. It seems that one of the goals of changing Step 1 to pass/fail is to develop a more holistic approach to residency applications. But interestingly, PDs do not plan to increase their emphasis on a student's research experience, involvement in the Alpha Omega Alpha Society, Gold Humanism Society, or volunteer and leadership extracurricular activities. PDs do not plan on adjusting the impact of the personal statement, or letters from non-medicine or unrecognized faculty.

In our study, a majority of internal medicine residency PDs believe that the pass/fail transition will disadvantage MD students who do not attend a highly regarded medical school, DO students, and foreign medical students. Citing school reputation and personal relationships as increasingly important in residency selection seems at odds with meritocracy and holistic consideration of applicants.

The majority of PDs believe that medical students will adapt to the pass/fail transition by applying to even more residency programs. More applications, and less objective criteria, can potentially make residency admissions decisions even more difficult. More than half of internal medicine residency PDs would support a cap/limit on the number of residency applications each medical student can submit.

Understanding that adaptations are necessary to adjust to the pass/fail Step 1 change, the FSMB and NBME do not plan to implement this change until sometime after January 2022. The disagreement about making Step 1 pass/fail is largely based on losing a universal standardized metric by which to expeditiously select a few candidates from thousands of applicants. The FSMB and NBME are allowing time for medical schools and graduate medical education programs to develop new metrics of student evaluation. Successful transition to a more holistic review of each applicant requires medical schools to be more transparent about clinical experiences, provide clearer criteria for clerkship grading and a better skills handoff method from undergraduate medical education to graduate medical education.

Conclusion

In conclusion, the majority of internal medicine PDs do not support the transition of USMLE Step 1 to a pass/fail scoring system. Internal medicine residency programs will have to reevaluate how they assess applicants for an interview. Other factors will now be emphasized including Step 2 CK, personal knowledge of the applicant, grades in required clerkships, and audition rotations. There is time, before Step 1 becomes pass/fail, to make adjustments to the undergraduate and graduate medical education evaluation process. Changing Step 1 to pass/fail is just the first step.

Limitations

Our study had several limitations. Firstly, we were unable to obtain responses from all internal medicine program directors. However, we received responses from 123 PDs (22.4%), which we believe is reasonable and representative. We sent the survey out three times because we believed that subsequent emails would not have a significant impact on response rate. Secondly, our study may be limited by sampling bias and non-response bias. PDs who felt strongly about the change in Step 1 may have been more likely to respond. Finally, some contact information provided by FREIDA Online did not belong to the PD, but instead the program's coordinator or administrative assistant. Our email requested program coordinators and administrative assistants to send the survey to the PD.

Author contributions

Conceptualization: Frederick Mun, Alyssa R Scott, David Cui, Alia Chisty, William L Hennrikus, Eileen F Hennrikus.

Data curation: Frederick Mun, Alyssa R Scott, David Cui, Alia Chisty, William L Hennrikus, Eileen F Hennrikus.

Formal analysis: Frederick Mun, Alyssa R Scott, David Cui, Alia Chisty, William L Hennrikus, Eileen F Hennrikus.

Funding acquisition: Eileen F Hennrikus.

Investigation: Frederick Mun, Alyssa R Scott, David Cui, William L Hennrikus, Eileen F Hennrikus.

Methodology: Frederick Mun, Alyssa R Scott, David Cui, William L Hennrikus, Eileen F Hennrikus.

Project administration: Frederick Mun, Alia Chisty, William L Hennrikus, Eileen F Hennrikus.

Resources: Frederick Mun, Eileen F Hennrikus.

Supervision: William L Hennrikus, Eileen F Hennrikus.

Validation: Frederick Mun, Alyssa R Scott, Alia Chisty, William L Hennrikus, Eileen F Hennrikus.

Visualization: Frederick Mun, Alyssa R Scott, Alia Chisty, William L Hennrikus, Eileen F Hennrikus.

Writing – original draft: Frederick Mun, Alyssa R Scott, David Cui, Alia Chisty, William L Hennrikus, Eileen F Hennrikus.

Writing – review & editing: Frederick Mun, Alyssa R Scott, David Cui, Alia Chisty, William L Hennrikus, Eileen F Hennrikus.

Correction

A sentence in the abstract, “Contact information from the American Medical Association's Fellowship and Residency Electronic Interactive Database was used”, was repeated when the article was originally published. The duplicate sentence has been removed.