Selecting the Next Generation of Surgeons: General Surgery Program Directors and Coordinators Perspective on USMLE Changes and Holistic Approach.

INTRODUCTION: The United States Medical Licensing Examination (USMLE) was designed as a universal assessment tool for states to determine physician's medical licensure's candidacy. Recent changes in the USMLE exam have changed the way future surgical residency candidate applications will be reviewed. The survey aimed to assess the effect of changes in USMLE exams-USMLE Step 1 pass/fail, complete dissolution of USMLE clinical skills exam, and the role of holistic review in future surgical residency candidacy selection.
METHODS: An anonymous online survey was created and distributed to general surgery program directors and coordinators across the USA. The survey aimed to assess attitudes toward changes to USMLE exams and the potential changes with a holistic review of candidate applications.
RESULTS: The response rate was 63.7%. Most program directors and coordinators disagree with changing USMLE Step 1 to a pass/fail scoring system. The majority felt that contacts, the medical school's name, and performance in clinical electives and sub-internships would hold more significance. They also believe that a holistic review of application will decrease socioeconomic discrepancies and promote a more diverse and inclusive resident cohort.
CONCLUSION: Step 2 clinical knowledge (CK) will gain more importance in future residency matches because of the change in the scoring system of Step 1. The medical school's name, personal contacts, and clinical performance in rotations will hold more significance.

Introduction

On January 26, 2021, the Federation of State Medical Boards (FSMB), and NBME (National Board of Medical Examiners), co-sponsors of the United States Medical Licensing Examination® (USMLE®), announced that they are discontinuing the work to relaunch a modified Step 2 Clinical Skills examination (Step 2 CS) [1]. A year earlier, on February 12, 2020, the FSMB and the NBME had announced changes to the USMLE Step 1, which will now be a pass/fail exam rather than a score-based exam [2]. These changes will inevitably alter the metrics that are submitted to residency programs by prospective surgical trainees.

For years, programs have used the USMLE Step 1 scores to filter applicants to offer an interview [3]. Aside from its intended use of determining medical license eligibility, the USMLE exams have evolved into a screening tool for residency program applications. Many surgical residency programs set target scores to select competitive candidates [4]. Surgical program directors (PDs) also use these scores to formulate rank lists and differentiate between applicants. Scores not only reflect an applicant's basic medical knowledge but have also been correlated with the academic performance during residency and serve as the best objective data available at the time of candidate review [5].

This magnitude change carries important ramifications for surgical residencies that rely on the value of Step 1 scores and Step 2 CS as a measure of medical knowledge and clinical skills. Secondly, there has been an emphasis on a holistic approach toward resident selection. The holistic approach emphasizes the importance of considering every applicant. It provides operational guidance for developing mission-driven, diversity-oriented processes; and encourages applying a balanced approach to assess the experiences, attributes, competencies, and academic metrics of each candidate [6, 7]. This study aimed to survey the program directors and program coordinators of all General Surgery residency programs regarding their attitudes toward the changes in USMLE exams and holistic approach for resident selection.

Methods

An anonymous 24-question multiple-choice survey was developed using Qualtrics, an online survey platform [8]. All general surgery residency program directors and coordinators in the USA were invited to participate.

Survey development and distribution

We developed the survey questions after a comprehensive review of current literature and validated instruments. The survey queried their program location, training program type, and survey participants’ years of experience associated with residency education and leadership. The questions were related to attitudes toward USMLE changes—Step 1 being a pass/fail test, cancellation of Step 2 CS, the significance of Step 2 CK, and Step 3 after these changes. We also assessed the importance of the medical school's name, personal contacts, electives, and clinical experience once the changes in USMLE are implemented. The questions also evaluated whether programs employ a holistic review of applications and their opinions on a holistic approach for resident candidacy selection. There was a free text option available for program directors or coordinators who had any concerns about the holistic approach.

The questions were converted to an online format and beta tested to ensure brevity, clarity, relevance, and consistent interpretation. The survey was then refined, incorporating feedback from the beta-tested survey. The survey was preceded by a statement explaining that the survey was for research purposes and that data would be de-identified before analysis. A definition also preceded the survey for terms used in the survey to make the questions clearer. The holistic review was defined as mission-aligned admissions or selection processes that consider applicants' experiences, attributes, and academic metrics and the value an applicant would contribute to learning, practice, and teaching. “Personal contacts” were defined as mentors or academic acquaintances a candidate had built due to clinical work. Past “non-academic struggles” were defined as life adversities or stressful situations such as personal health issues, being a single parent, death of any family member, supporting a family financially at a younger age, lack of resources, financial restraints, lack of support systems, or previous health problems. We wanted to assess if program directors would consider such personal stressors in the applicants' lives and the mechanism by which they responded or recovered from those challenges.

There were no incentives or disincentives to participate. This study was deemed exempt by the Institutional Review Board.

A link to the survey was emailed to 310 general surgery program directors (PDs) and program coordinators (PCs). The emails of PDs and PCs were obtained from the Association of Program Directors in Surgery website (www.apds.org) [8, 9]. Responses were captured anonymously. The questions asked in the survey are reported in Appendix A1. Descriptive statistical analysis was performed to summarize the data. A response rate was calculated by the number of responses from a program (either by PD or PC) divided by the total number of programs we approached. The survey was sent in April 2021. We sent a total of 4 reminders regarding survey completion, which were sent one week apart.

Outcomes measured and statistical analysis

The primary purpose was to assess program directors' and coordinators' viewpoints regarding the recent USMLE exam changes and evaluate their opinions on resident applicants' holistic review. We did two sub-analyses: 1—to assess if there was any difference in opinions of the program coordinators compared to the program directors and 2—to assess if there is any difference in opinion between program directors with five or more years of experience compared to program directors with 1–4 years of experience.

The data were reported as proportions. The Chi-square test was utilized to assess any difference in responses between the groups. A p value ≤ 0.05 was considered statistically significant.

Results

Out of 310 general surgery programs emailed, 40 programs declined to participate, and 30 programs had older or invalid emails. A total of 185 responses were collected—119 from the program directors and 66 from the program coordinators. The overall response rate was 63.7% (185/290). Table 1 highlights the demographics of our respondents.

Table 1

Survey demographics

Demographics of respondents	(%)
Q2. Proportion of respondents
Program directors	64.7
Program coordinators	35.3
Q3. Number of residents per class
2–3	11
4–6	60.2
7 +	27.7
Q4. Type of hospital
University hospital	65.5
Community with university affiliation	29.5
Non-University affiliated hospital	11.1
Q5. Experience in their current role
1–2 years	10.3
3–4 years	33.2
> 5 years	55.8

USMLE changes

70.9% of the program directors did not agree with the change of USMLE Step 1 to a pass/fail exam. 86.2% of the program directors felt that USMLE Step 2 will gain more significance and will now be utilized instead of Step 1 for granting residency interviews and ranking candidates. 80.9% of the respondents stated that residency programs should make USMLE Step 2 CK scores a mandatory requirement to submit their application. We found mixed results regarding the discontinuation of Step 2 CS: 37.8% of the program directors were neutral about this change while 30.2% agreed with it. 40.2% of program directors felt that some clinical skill competency assessment is required. Regarding USMLE Step 3, 30.6% of the program directors felt that Step 3 would have a more critical role in the resident selection process, while 42.9% felt its role would remain unchanged. 74.9% of respondents stated that resident selection will now be more challenging because of the recent changes in the USMLE structure.

International medical graduates

48.7% of the program directors stated that matching international medical graduates (IMGs) into general surgery programs will be more challenging with the USMLE exams' changes. However, 20.6% disagreed with this, while the remainder (30.7%) reported a neutral opinion.

Role of medical school, clinical rotations, and contacts

42.5% of the program directors stated that the significance of personal statements would increase. 85.4% agreed that the reputation of the candidates’ medical school would gain more importance. 85.4% of the responding program directors agreed that an applicant's clinical rotations and electives would hold more significance than before. Table 2 highlights the responses of program directors in the survey.

Table 2

Survey questions and responses—program directors

Question	Agree (%)	Neutral/not sure (%)	Disagree (%)
Changing USMLE Step 1 to pass/fail is a good idea	15.1	12.8	70.9*
Significance of USMLE Step 2 CK will be increased now that USMLE Step 1 is pass/fail	86.2*	6.4	6.1
Removal of USMLE Step 2 CS is a good idea	30.2	37.8	31.1
Some form of clinical assessment exam should exist	40.2	30.8	27.8
USMLE Step 3 will carry more importance than before	30.6	25.6	42.9
Recent changes in Step 1 and Step 2 CS will make resident selection more difficult than before	6.2	17.9	74.9*
Your program is moving toward a “holistic” approach to resident selection	40.1	44.8	13.2
A minimum cutoff score for USMLE Step 2 CK will be required for resident selection	66.6*	16.6	16.5
“Personal contacts” matter more than before	73.2*	16.6	8.4
Name of medical school will matter more than before	66.5*	16.6	15.9
Clinical experience such as away rotations and electives matter more	85.4*	8.9	4.8
IMGs would be at a disadvantage with recent changes	48.7	30.7	20.6
Personal statement become more important	42.5	38.9	17.9
Holistic review decreases the socioeconomic disparities	43.3	27.2	28.5
Holistic review promotes inclusion and diversity	46.6	26.6	26.6

Responses are recorded in percentages

*Indicates a response greater than 50%

Holistic review

Regarding a holistic approach for residency candidates, 45.2% of the program directors stated that their program is moving toward a holistic approach. In comparison, 15.1% of the programs will continue with their current practice for resident selection. Interestingly, 39.7% of the programs remain undecided about how they will be selecting residents due to the recent changes in USMLE exams. 43.3% of the responding PDs stated that they believed a holistic review would decrease socioeconomic disparities among applicants. 46.6% agreed that a holistic review of applicants would promote diversity and inclusion in their program. When asked about the significant anticipated outcomes from a holistic review, 33.6% of respondents felt a more fitting resident cohort would be recruited, 27.5% stated that it would lead to an increase in recruitment of people of color and ethnic minorities. In comparison, 20.4% said that this would allow them to find and recruit candidates with past, non-academic struggles. Only 18.3% of the respondents felt that applicants who have failed any component of USMLE have more success in the holistic review than the traditional match. Figure 1 highlights the free text comments of program directors and coordinators.

Fig. 1

Free text comments of Program Directors and coordinators regarding recent changes in USMLE and holistic review

PD vs. PC

Overall, the opinions of PDs and PCS were similar. However, some differences did exist in terms of the strength of the response. For example, 30.2% of the PDs agreed with removing Step 2 CS being a good idea compared to only 16.4 of the PCs (p 0.01). Similarly, 50% of the PCs thought some form of clinical assessment as a replacement for Step 2 CS is required compared to 40.2% of the PDs (p 0.01). Finally, 42.1% of the PDs disagreed that USMLE Step 3 will carry more importance than before compared to 16.1% (p 0.01). Table 3 highlights the comparison of opinions between the PDs and PCs.

Table 3

Comparison of Responses between Program directors and coordinators

Question	Agree	Agree	Neutral	Neutral	Disagree	Disagree
	PD (%)	PC (%)	PD (%)	PC (%)	PD (%)	PC (%)
Changing USMLE Step 1 to pass/fail is a good idea	15.1	14.1	12.8	11.6	70.9	74.3
Significance of USMLE Step 2 CK will be increased now that USMLE Step 1 is pass/fail	86.2	88.4	6.4	9.3	6.1	2.3
Removal of USMLE Step 2 CS is a good idea*	30.2	16.4	37.8	37.2	31.1	46.4
Some form of clinical assessment exam should exist*	40.2	50.0	30.8	33.3	27.8	16.7
USMLE Step 3 will carry more importance than before*	30.6	41.3	25.6	25.5	42.9	16.1
Recent changes in Step 1 and Step 2 CS will make resident selection more difficult than before	74.9	74.3	17.9	20.9	6.1	4.8
Your program is moving toward a “holistic” approach to resident selection	40.1	43.1	44.8	46.1	13.2	11.6
A minimum cutoff score for USMLE Step 2 CK will be required for resident selection	66.6	64.3	16.6	16.2	16.5	9.1
“Personal contacts” matter more than before	73.2	64.8	16.6	23.6	8.4	11.6
Name of medical school will matter more than before	66.5	58.3	16.6	25.5	15.9	16.2
Clinical experience such as away rotations and electives matter more	85.4	76.9	8.9	16.2	4.8	6.9
IMGs would be at a disadvantage with recent changes*	48.7	50	30.7	37.5	20.6	12.5
Personal statement become more important*	42.5	57.1	38.9	35.7	17.9	7.1
Holistic review decreases the socioeconomic disparities	43.3	42.7	27.2	38.1	28.5	19.2
Holistic review promotes inclusion and diversity	46.6	47.7	26.6	33.3	26.6	19.0

Responses are recorded in percentages

*Indicates a statistically significant (p < 0.01)

PDs with less than five years’ experience vs. PDs with five or more years of experience

Program directors with ≥ 5 years of experience were more likely to disagree with the USMLE Step 1 change to pass–fail than PD with < 5 years of experience (81.3% vs. 73.3%; p 0.01). PD < 5 years of experience were more likely to agree that removal of Step 2 CS is a good idea (46.7% vs. 37.3%; p0.01), that the name of the medical school will matter even more than before (83.4% vs. 67.5%; p0.01) than compared to the PDs ≥ 5 years’ experience. Table 4 highlights these outcomes.

Table 4

Comparison of responses between program directors based on experience

Question	Agree	Agree	Neutral	Neutral	Disagree	Disagree
	PD < 5 years (%)	PD ≥ 5 years (%)	PD < 5 years (%)	PD ≥ 5 years (%)	PD < 5 years (%)	PD ≥ 5 years (%)
Changing USMLE Step 1 to pass/fail is a good idea*	13.4	11.7	13.3	7.0	73.3	81.3
Significance of USMLE Step 2 CK will be increased now that USMLE Step 1 is pass/fail	86.7	86.4	10.0	4.6	3.3	9.3
Removal of USMLE Step 2 CS is a good idea*	46.7	37.3	33.3	32.5	20.0	30.2
Some form of clinical assessment exam should exist	40.0	50.0	50.0	35.0	10.0	15.0
USMLE Step 3 will carry more importance than before	13.4	14.1	33.3	23.2	53.3	62.7
Recent changes in Step 1 and Step 2 CS will make resident selection more difficult than before	80.0	83.8	13.3	13.9	6.7	2.3
Your program is moving toward a “holistic” approach to resident selection	43.4	46.6	43.3	37.2	13.3	16.2
A minimum cutoff score for USMLE Step 2 CK will be required for resident selection	63.4	62.9	23.3	16.2	13.3	20.9
“Personal contacts” matter more than before	86.7	74.5	10.0	11.6	3.3	13.9
Name of medical school will matter more than before *	83.4	67.5	3.3	13.9	13.3	18.6
Clinical experience such as away rotations and electives matter more	90.0	95.4	10.0	4.6	0.0	0.0
IMGs would be at a disadvantage with recent changes *	54.7	45.7	27.2	31.4	18.1	22.6
Personal statement become more important*	43.4	30.4	43.3	44.1	13.3	25.5
Holistic review decreases the socioeconomic disparities	30.0	37.3	30.0	30.2	40.0	32.5
Holistic review promotes inclusion and diversity	42.4	39.7	20.6	25.5	31.0	34.8

Responses are recorded in percentages

*Indicates a statistically significant (p < 0.01)

Discussion

We found that most surgical program directors and coordinators disagreed with the transition of USMLE Step 1 to a pass/fail exam. Our survey results indicate that they anticipate a more difficult resident selection process, with more emphasis on Step 2 scores than before. The USMLE exams have evolved into a screening tool for residency programs across many specialties, and many surgical residency programs set target scores to select competitive interviewees [9]. Surgical program directors also use these scores to formulate rank lists and to differentiate between applicants [10]. Specifically, there has been a heavy reliance on the USMLE Step 1 score in the selection process. In 2019, InCUS (Invitational Conference on USMLE Scoring) suggested changing USMLE Step 1 to pass/fail to reduce the adverse impact of the current overemphasis on USMLE performance in residency screening [11]. The survey showed that many assistant/associate deans and course/clerkship directors support this change. In contrast, only 26% of the PDs and PCs agreed with changing USMLE Step 1 to a pass–fail exam. With USMLE Step 1 becoming a pass/fail exam, the significance of Step 2 CK scores will increase and play a more integral role in selecting surgical residency candidates. This is evident by our survey results, as most of the program coordinators and directors believe that USMLE Step 2 CK's significance will change and become more critical in the future. Other organizations and surveys have shown similar findings and concerns that these changes hinder an objective applicant evaluation [12-14]. In a recent survey of urology program directors, 84.6% felt that the USMLE Step 1 change would make it more challenging to compare applicants objectively and place a great emphasis on Step 2 CK scores [15].

In our survey, the respondents believe that personal contacts, medical schools' names, surgical clerkship performance, and additional clinical rotations would all play a more prominent role in the selection criteria. The schools that will be best prepared to handle the increased emphasis on Step 2 CK are many of the top medical schools that require students to take Step 1 and Step 2 CK after completing their core clinical rotations during their third year, rather than at the beginning of their second year. Our findings are similar to a recent survey of otolaryngology program directors, which showed that recent changes in USMLE exams are anticipated to increase the significance of core clerkship grades and elective rotations at respective institutions, Alpha Omega Alpha and other awards, and letters of recommendation [16]. They also anticipate that this change will negatively impact international medical graduates, Osteopathic Medicine Doctors, and underrepresented students [16].

We found that 42% of surgery programs are moving from a traditional approach to a holistic approach for their upcoming residency selection process. An increasing emphasis is being placed on the diversification of residency training to reflect an expanding, diverse patient population [17, 18]. Incorporating a holistic review process, providing a broader assessment of applicants can play a pivotal role in increasing the proportion of women and underrepresented students involved in the general surgery recruitment process. Nehemia et al. have shown that after implementing a holistic review in 2017, a statistically significant increase in the proportion of women and underrepresented students ranked in their program compared to their prior "traditional" approach [19].

Our respondents believe that the pass/fail scoring change may encourage more students to apply to general surgery, which is traditionally considered to be "competitive." Those concerned with obtaining prohibitively low scores may now build a more well-rounded application and focus on Step 2 CK, predominantly centered on clinical medicine instead of its basic science-focused counterpart, Step 1. However, they believe that because of a lack of more objective data, applicants' names of medical school, personal statements, and contacts may play a more critical role in the match than before. By eliminating a core objective measurement of achievement such as Step 1, many students lose an opportunity to distinguish themselves from students graduating from more prestigious schools. Similarly, letters of recommendation are also prone to inequity as students may not have access to prominent letter writers, especially in the setting of unaffordable visiting rotations or have since been canceled.

Our study found that additional clinical rotations would now play a more prominent role in the selection criteria. COVID-19 has not only severely affected our patients but also our trainees and learners [8, 20, 21]. With the current pandemic, many medical students nationally and internationally have been prevented from completing away rotations. Secondly, many students have not been able to participate in extracurricular activities and experiences during this time or build “contacts” all of which will gain more significance in a holistic approach. The effect of change in USMLE and lack of opportunities because of stay-at-home have a huge implications for future applicants [8]. Hence, we believe that the timeline of implementation of these changes should be reconsidered given the possible compounded disadvantages it may give to underrepresented students and international medical students due to the pandemic.

We chose to survey the program coordinators in our study along with the program directors. We analyzed their opinion in comparison with the program directors and did not find any difference in opinion. We do understand that program directors make the final decision in resident selection. However, there is no denying that program coordinators play an essential role in residency programs and are well-versed in the thought process that governs a particular residency program. Similarly, we compared PDs who have been program directors for five years or more with those who had less than five years of experience. We found that the overall outcomes were similar. However, there were some points where PDs with more than five years were more decisive in their approval, such as the change of USMLE Step 1 to pass/ fail, while there were some topics where the PDs with less than five years of experience had a more robust approval, such as removal of Step 2 CS and the significance of the name of the medical school.

The findings of our study should be interpreted in the context of a possible response bias. Nearly one-third of recipients did not respond to the survey request. PDs and PCs with stronger opinions on the current changes may have been more likely not to participate in the study. The initial setup of the questions may also have produced some of the results seen in the study. Future studies should focus on capturing objective general surgery match data after the implementation of USMLE to evaluate its ramifications better [13]. While our data are subjective, they represent the opinions of those directly involved with the residency selection process. As the evaluation of surgical candidates changes, this survey could provide great insight for program directors and coordinators as they embark on this journey with a great sense of insecurity and lack of norm. Another limitation of the study is that only electronic recruitment methods via e-mails were employed. Had other methods such as social media, postal, or telephone surveys been employed, maybe those non-respondents who did not have up-to-date electronic contact information could have been adequately recruited.

Conclusion

Step 2 CK is anticipated to gain more importance in future general surgery residency matches because of the change in the scoring system of Step 1. The applicant’s medical school reputation, personal contacts, and clinical performance in rotations will hold greater significance. Many PDs and PCs believe that a holistic review of applicants will increase the recruitment of students of color and minorities and candidates with non-academic struggles.