Lower-Third SLOE Rankings Impede, But Do Not Prevent, A Match in Emergency Medicine Residency Training.

OBJECTIVE: Emergency medicine program directors (PD) value the standardized letter of evaluation (SLOE) as the most important aspect of a residency application when making both invitation and ranking decisions. This study aims to determine whether the presence of any lower-third in either SLOE global assessment (GA) question impacted the ability of an applicant to match into EM. We hypothesized that any lower-third ranking would be associated with increased odds of not matching into EM.
METHODS: We conducted a retrospective cohort study evaluating allopathic applicants from medical schools in the United States (US allopathic applicants) to a single EM residency program during the 2018/2019 match cycles. GA SLOE rankings from all applications were tabulated and compared to the applicant's National Resident Matching Program (NRMP) match outcome. Comparative analyses were conducted between SLOE groupings and odds ratios (OR) were calculated.
RESULTS: A total of 2,017 SLOEs from 781 US allopathic applicants were analyzed during the study period. Of the total, 277 (35%) applicants in our sample had any lower-third GA ranking, which significantly decreased an applicant's odds of matching in EM by 79% (OR 0.21, 95% CI, 0.12-0.34). Having more than one lower-third GA ranking did not further statistically decrease the odds of a successful EM match (OR 0.60, 95% CI 0.31-1.17). As a secondary finding of the study, results demonstrate that those applicants having no lower-third GA rankings had a nearly 5 times increased odds of an EM match (OR 4.84, 95% CI, 2.91-8.03).
CONCLUSION: Having any lower-third GA ranking significantly reduced an applicant's chances of matching into an EM program. Faculty advisors should be aware of the increased risk of not matching for any applicant with any lower-third GA ranking and advise students appropriately, while maintaining the integrity of the SLOE and not divulging the confidential information contained within.

Residency program directors (PDs) in emergency medicine (EM) are tasked with sorting through large volumes of applications in a short period of time to select the best applicants to interview, and subsequently rank, for a position in their program. Despite a steady increase in the number of first-year resident (PGY-1) positions in EM, medical students are applying to a larger number of EM residency programs. Recent data illustrates that the average United States (US) allopathic applicant, defined as an applicant attending an institution in the US accredited by the Liaison Committee on Medical Education (LCME), applies to 46 programs and must rank 16 or more programs to generally ensure a match in EM. This has contributed to a rising number of applications for PDs to review, with the average program receiving approximately 780 applications during the 2014-2015 application cycle.[1]

In addition to United States Medical Licensing Examination (USMLE) scores, academic transcripts, and the medical student performance evaluation (MSPE), the data available for EM PDs is unique in that it includes the Standardized Letter of Evaluation (SLOE). EM PDs consider the SLOE to be the most important component of the application when making decisions on which applicants to invite for an interview.[2,3] The SLOE aims to provide PDs with concise, objective, and comparative information regarding student performance on EM rotations, as well as their suitability for the specialty. The final section of the SLOE contains two global assessment (GA) questions asking the writer to compare the student to all students they have worked with in the past academic year and to estimate where they will place the student on their final rank list. Both questions ask the SLOE writer to place the student into tiers: top 10%, top-third, middle-third, and lower-third, with an additional ranking of “unlikely to be on our rank list” for the final GA question only.[3,4] Although prior work has demonstrated no clear correlation between an applicant’s SLOE GA rankings and the position of that applicant on a program’s rank list, PDs continue to rely heavily on the SLOE and these two GA data points in the interview selection process.[2,5]

Previous studies of GA rankings have focused on survey-based data of the potential negative impact of lower-third rankings on PD perception of applicants. SLOE writers report grade inflation to minimize the risk of applicants not matching, with nearly one-quarter of SLOE writers acknowledging that they “sometimes” or “frequently” inflate GA rankings.[6,7] Despite this, the actual impact of a lower-third GA ranking in the SLOE on ultimate match outcome is unknown. Our study is the first, to our knowledge, to directly evaluate the impact of a lower-third GA ranking upon successful match rates in EM using actual NRMP outcomes data. A data-based assessment of the actual impact of lower-third rankings on match success is needed to better inform these subjective perceptions of EM PDs, as this may help avoid grade inflation and maintain the integrity of the SLOE. We hypothesized that NRMP match outcomes data would confirm that a lower-third SLOE ranking is associated with an increased odds of not matching into EM. In order to evaluate potential performance confounders which could be associated with a lower-third GA ranking, this study also correlated performance on USMLE Step 1 and Step 2 CK with match outcomes. Given survey data demonstrating the importance that PDs place on these objective variables, we chose these secondary analysis points to determine if higher performance on these examinations could help an applicant overcome a lower-third GA ranking.[8]

Methods

Study design

This is a retrospective cohort study of all US allopathic applicants to a single EM residency program during the 2018 and 2019 match cycles. Applicants were excluded from the analysis if they were (1) non-US allopathic applicants or (2) US allopathic applicants without a SLOE in their application. SLOE rankings and USMLE scores were obtained from the standard information within each applicant’s Electronic Residency Application Service (ERAS) file. USMLE Step 1 and Step 2 CK scores were extracted from the USMLE score report in ERAS. If an applicant required multiple attempts to pass a USMLE examination, the initial failure(s) was recorded, and the highest score obtained was included for analysis. The SLOE was reviewed for the following 2 global assessment (GA) questions: (1) Compared to other EM residency candidates you have recommended in the last academic year, this candidate is in the (top 10%, upper-third, middle-third, or lower-third)?; and (2) How highly would you estimate the candidate will reside on your rank list?, with the same response options as question 1, with the addition of “unlikely to be on our rank list”. SLOEs from EM sub-specialty rotations (ie, pediatric EM, ultrasound, toxicology, etc) were included. The number of lower-third rankings in either GA response in all available SLOEs was recorded, with no distinction being made between whether the lower-third ranking(s) was assigned within a single SLOE or multiple SLOEs for each applicant.

After recording this application data, final match status for each applicant was determined with the possible match outcomes recorded as the following: EM, EM combined program (ie, emergency medicine/internal medicine, emergency medicine/family medicine, emergency medicine/pediatrics, etc), non-EM residency program, or no match. In this study, the EM combined programs were considered as a successful EM match. The outcomes of non-EM residency program or a no match were considered to be no match for data analysis. For applicants who were interviewed and subsequently placed on the final rank list by the residency program performing this study, final match status was determined by utilizing the “Match Results by Ranked Applicant” report provided by the NRMP at the conclusion of the match cycle. For those applicants who were not invited for interview or who were interviewed but not ultimately ranked by the program, the NRMP Applicant Match History database was queried for each individual’s match outcome. This database is available to institutional officials, program directors, and program coordinators. The database was queried using the applicant’s American Association of Medical Colleges (AAMC) identification numbers. The study protocol was reviewed and approved by the Institutional Review Board at the study site, with a waiver of informed consent.

Study setting and population

The residency program in this study has participated in the NRMP match since 1993 and regularly receives applications from all regions of the United States (US). Annually, the program received an average of 700 total applications for its 10 positions, consistent with the previously reported national program average of approximately 780.[1] Over half of the applications to this program are typically received from US allopathic applicants, which is consistent with the national average of 59% to 64% US allopathic applicants to emergency medicine in the 2018 and 2019 match cycles.[9]

In the 2018 and 2019 match cycles, there were 1,748 and 1,823 total US allopathic applicants, respectively, to emergency medicine. Of the total positions in EM [2,278 (2018) and 2,488 (2019)], 99% were filled through the match, with 65% of all EM positions in both study cycles filled by US allopathic applicants. Furthermore, 89% to 91% of all US allopathic applicants to emergency medicine successfully matched into the specialty in 2018-2019. In the most recent 2018 NRMP data, the mean USMLE Step 1 and Step 2 CK scores for applicants matched into EM were 233 and 247, respectively.[9]

Data analysis

Descriptive statistics were calculated for all variables. The extent to which the distribution of continuous variables departed from normality was assessed via the Shapiro-Wilk W test, with significant (P < .05) results indicative of a non-normal distribution. Nonparametric tests were conducted in the presence of non-normal outcome distributions.

Primary outcome

Contingency table analysis was employed to explore the association between matching into EM and the presence of any lower-third GA ranking on a SLOE, with outcomes expressed as odds ratios (OR). We calculated 95% confidence intervals around each OR estimate. Ratios that did not include 1 in the confidence interval were considered to be statistically significant associations.

Secondary outcomes

Due to non-normal distributions of the USMLE scores, the Wilcoxon rank-sum test was used to explore differences in average highest USMLE Step 1 and Step 2 CK scores by EM match within the entire cohort and the subgroup of applicants with at least one lower-third GA ranking. Furthermore, a logistic regression was performed in order to ascertain whether performance on USMLE Step examinations was a significant confounder in assessing whether the presence of a lower-third GA ranking had an association with the odds of an applicant matching into EM. Given survey data demonstrating the importance that PDs place on Step 1 and Step 2 CK performance for both interview selection and ranking,[8] performance on USMLE step scores were chosen as the variables for this secondary outcome logistic regression analysis in order to assess whether higher scores on either USMLE Step 1 or Step 2 CK could potentially mitigate the presence of any lower-third GA ranking(s) on a SLOE. An alpha of 0.05 was selected as the threshold for statistical significance in this secondary analysis. All analyses were performed using JMP® Pro 14.0 (SAS Institute, Inc., Cary, North Carolina).

Results

A total of 1,405 applicants applied to EM at the study site during the 2018 and 2019 match cycles. Of those, 798 were US allopathic applicants. This applicant population represents 22% of all US allopathic EM applicants nationally during the 2-year study period.[10] Of these 798 applicants, 17 (2%) had no SLOE in the ERAS application, and therefore were excluded from analysis. The remaining 781 applicants had a total of 2,017 SLOEs, averaging 2.5 SLOEs per applicant. These SLOEs were generated from 190 distinct EM program sites, representing 76.9% of all EM residency programs in the United States.[10,11] Information demonstrating the demographic characteristics of applicants and their associated medical schools, as well as institutions providing a SLOE are included in Figure 1.

Figure 1.

The demographic characteristics of the applicants, their respective medical schools, and the institutions providing SLOEs in the study cohort. Regions in this figure and in the remainder of the paper are derived from the Society for Academic Emergency Medicine residency directory: https://member.saem.org/SAEMIMIS/SAEM_Directories/ResidencyMap/SAEM_Directories/P/ResidencyMap.aspx?hkey=1e134970-ec57-4862-87fb-6971bad7a77b.

Of the total 781 applicants reviewed, 504 (64.5%) had no lower-third GA ranking and 277 had any lower-third GA ranking, with 96/277 (34.6%) having one lower-third GA ranking, and 181/277 (65.3%) having more than one lower-third GA ranking (Table 1).

Table 1.

Distribution of applicants in the study cohort with no lower-third GA rankings, 1 lower-third GA ranking, or more than 1 lower-third GA rankings.

Total applicants reviewed	781
Applicants with no lower-third GA	504 (64.5%)
Applicants with 1 lower-third GA	96 (12.3%)
Applicants with more than 1 lower-third GA	181 (23.2%)

Table 2 illustrates the match outcomes for all applicants, inclusive of those with no lower-thirds, those with any lower-third, and those with more than one lower-third. About 703 of the 798 (88.1%) total applicants in our study successfully matched into EM or a combined EM specialty. As a secondary finding, those applicants having no lower-third GA ranking on any SLOE gave them a nearly 5 times higher chance of successfully matching into EM (OR 4.84, 95% CI 2.91-8.03). Of those 78 who did not match into EM, 54 (69.2%) had one lower-third GA ranking, with 40/54 (74.1% of applicants with any lower-third) having more than one lower-third GA ranking.

Table 2.

Odds of matching into an emergency medicine residency for applicants with no lower-third GA ranking, any lower-third GA ranking, and more than 1 lower-third GA ranking.

	Applicants who did match (%)	Applicants who did not match (%)	Odds ratio (95%CI)
Overall	703 (90.0%)	78 (10.0%)
No lower-third GA	480 (95.2%)	24 (4.7%)	4.84 (2.91–8.03)
Any lower-third GA	223 (80%)	54 (20%)	0.21 (0.12–0.34)
More than 1 lower-third GA	141 (78%)	40 (22%)	0.60 (0.31–1.17)*

Odds of matching with more than 1 lower-third versus 1 lower-third ranking.

Despite this, 80% of applicants with any lower-third GA ranking and 78% of applicants with more than one lower-third GA ranking successfully matched, with both of these cohorts matching at a rate well below the overall match rate of our study population (88.1%), as well as the overall national match rate (88%-91%, 2018 and 2019, respectively).[9]

The majority of trainees with one lower-third ranking (223/277; 80%) and even trainees with multiple lower-third rankings (141/181; 78%) were still able to successfully match into EM. However, the presence of at least one lower-third significantly decreased the applicant’s odds of matching in EM by 79% (OR 0.21, 95% CI, 0.12-0.34). Although any lower-third GA ranking decreases an applicant’s chances of matching, more than one lower-third ranking does not appear to further negatively impact match outcomes. (OR 0.60, 95% CI, 0.31-1.17) (Table 2).

In the secondary outcome analysis (Table 3), applicants that matched into EM had significantly higher mean USMLE Step 1 scores, as compared to those who did not match (P ⩽ .001). Similarly, applicants that matched into EM had significantly higher mean scores on USMLE Step 2 CK (P = .0005) than their non-matching counterparts. In the logistic regression model, the decreased odds of an EM match with any lower-third GA ranking remained statistically significant after adjusting for both USMLE Step 1 and Step 2 CK scores (aOR = 0.22, 95% CI, 0.13-0.39). In the adjusted model, higher USMLE Step 1 and Step 2 CK scores were no longer associated with a successful match in applicants with any lower-third GA ranking (P > .05) (Table 4).

Table 3.

Mean USMLE Step 1 and Step 2 CK scores for matched and unmatched applicants for the overall cohort and those applicants with any lower-third.

	Applicants who did match (%)	Applicants who did not match (%)	P-value
All Applicants
Mean USMLE Step 1	223	215	<.0001
Mean USMLE Step 2 CK	240	233	.0005
Applicants with any lower-third
Mean USMLE Step 1	221	214	.0025
Mean USMLE Step 2 CK	236	232	.11

Table 4.

The results of the logistic regression model demonstrating the adjusted odds (aOR) for matching with any lower-third GA ranking, when adjusted for USMLE Step 1 and Step 2 CK performance. Table 3 demonstrated that applicants with any lower-third GA ranking who did match had higher mean USMLE scores compared to those who did not match. The logistic regression model (with results as illustrated in this table) was created to control for USMLE Step scores as a potential confounder and demonstrates that higher USMLE Step 1 and Step 2CK scores are no longer significant predictors of matching in EM for applicants with any lower-third GA ranking.

	P-value	aOR (95%CI)
Any lower-third	<.0001	0.22 (0.13–0.39)
USMLE Step 1	.06	0.23 (0.05–1.08)
USMLE Step 2 CK	.32	0.36 (0.04–2.74)

Discussion

Need for objective data

The landscape of undergraduate medical education is changing rapidly with the recent announcement from the National Board of Medical Examiners that USMLE Step 1 scoring will transition to pass/fail, diminishing the objective information available for PDs in applicant review.[12] Given prior survey data demonstrating that EM PDs place more importance on objective performance measures than other aspects of the application, the SLOE will likely become even more important for EM applicants in the coming years.[13] Despite the weight already given to the SLOE by EM PDs and the objective value they provide with measuring EM student performance relative to their peers, there has been no work done to directly measure the impact of a lower-third SLOE on EM match status. Our study is the first to quantify the effect of both single and multiple lower-third GA rankings on an applicant’s chances of successfully matching in EM.

Our data strongly suggests a direct correlation between any lower-third GA ranking and a significantly reduced odds of an EM match. Despite this correlation, the majority of US allopathic applicants with one or even multiple lower-third GA rankings do successfully match in the specialty. An understanding of the association between lower-third GA ranking and an increased risk of not matching is important for EM PDs, given their dual role in both selecting applicants for their residency program, while simultaneously advising EM-bound students. PDs that also serve as medical student advisors face a significant challenge in providing nuanced advice for medical students applying to EM with lower-third GA rankings. They must strongly advocate for the development of a parallel career plan, without presenting an overly pessimistic view of EM career opportunities or violating the confidentiality of the SLOE.

Skewed representation of students within the lower-third on SLOE rankings

Our study sample included a greater proportion of applicants (35%) with lower-third GA rankings than has been previously described in the literature.[4,7,14] However, we feel that this sample allows for a more confident assessment of match outcomes in this group.

Although the SLOE was intended to equally divide students into discrete tiers, SLOE writers nationally have not used the full spectrum of categories proportionately.[14] A review of all SLOEs during the 2016-2017 application cycle found that only 12% of SLOEs received a lower-third GA ranking for the question comparing students in the past academic year, and only 10% of SLOEs had lower-third GA rankings on the question “How highly would you estimate the candidate will reside on your rank list?”[4] Our findings, demonstrating the negative impact of lower-third GA rankings upon match outcome, support the assertions from prior authors that letter writers may be hesitant to use the entire spectrum of rankings for fear of limiting EM residency choices for applicants.[4,7,14] The overall match rate in our study sample (90%) was commensurate with the national cohort overall match rate during the study years (89% and 91% in 2018 and 2019, respectively) despite our sample having more students with lower-third GA rankings than the previous literature demonstrated.[4,7,9,14]

Rather than advocating for grade inflation or the complete avoidance of placing students into the lower-third rankings on the GA, we suggest that SLOE authors both follow the instructions provided by the Council of Residency Directors in Emergency Medicine and use the commentary portion of the SLOE for outlining why the student has been placed in the lower-third. Application reviewers should reflect upon this commentary provided by SLOE writers to discern the viability of the applicant for emergency medicine, rather than reflexively deciding not to interview an applicant or lowering them on the rank list simply based off the lower-third GA ranking. Utilizing the narrative comments in order to separate out the lower-third students who have potential in our specialty versus students who have grave flaws in clinical performance, professionalism, or interpersonal communications, and likely are not going to be successful in EM, is imperative to communicate to training programs.

In addition to robust narrative explanations as outlined above, if writers are hesitant to utilize the lower-third as it appears, it may be time for EM to consider alterations to the GA categories, such as a new bottom 10% category, that would allow for differentiation of the subset of the lower-third applicants who SLOE writers are concerned are unfit for EM. Although the SLOE currently has the category of “unlikely to match,” only 2% of SLOEs have an applicant placed in this designation.[4] Given the entirely negative connotation of this category as it currently stands, PDs may be extraordinarily hesitant to place applicants into this grouping. A lower 10% grouping would allow for PDs to more accurately categorize those at risk for not matching, while making the lower-third grouping more reflective of actual performance and illustrative of what is likely a teachable group of residency applicants.

Limitations

Study limitations include that the study is limited to a single site in the Southeast region of the United States. Though our study only included 22% of all US allopathic EM applicants between 2018 and 2019, our demographics demonstrate that there was proportional representation of applicant medical schools and SLOE writing institutions from all regions of the country. Although we feel that our study is a representative sample, future studies could include a multi-institution validation study sample from different regions of the country to demonstrate that these results are indeed generalizable.

Although our study examined USMLE performance as a potential confounder and found that USMLE Step 1 performance alone cannot mitigate the risk of not matching associated with any lower-third GA ranking, our study did not explore other applicant factors, such as the number of programs that applicants applied to or the number of programs ranked for each applicant. Thus, it is possible that applicants with lower-third GA rankings may mitigate potential effects of the lower-third rankings by applying more broadly and accepting a less desired position based on geographic needs, proximity to family, or leisure interests. Further work would be needed to specifically evaluate the impact of these applicant factors on the risk of not matching for students placed in the lower-third.

Conclusion

This is the first study to objectively quantify the effect of SLOE lower-third GA ranking(s) on an applicant’s chances of successfully matching in EM. We found that having any lower-third GA ranking significantly decreased the chances of matching into EM but does not absolutely preclude a successful EM match. This relationship remains significant even after adjusting for USMLE Step 1 and Step 2 CK scores, suggesting that even high USMLE scores do not mitigate the risk of not matching in the lower-third GA ranking group. Knowledge of the objective impact of a lower-third SLOE ranking can assist program leaders in providing career guidance and initiation of parallel planning for such at-risk applicants.