When (Almost) Everyone is Above Average: A Critical Analysis of American Orthopaedic Association Committee of Residency Directors Standardized Letters of Recommendation.

The American Orthopaedic Association introduced standardized letters of recommendations (SLORs) to improve on traditional letters of recommendations by "providing a global prospective on an applicant." However, no study has defined the utilization of SLORs, the distribution of applicant ratings in SLORs, or the impact of sex, race, or degree of involvement between the letter writer and applicant on SLOR domain ratings.
METHODS: One-hundred seventy-nine applications were randomly selected from all applicants submitted to a single, academic orthopaedic residency program. A single reviewer extracted both applicant characteristics and SLOR characteristics from applications. Descriptive statistics, chi-square tests, and nonparametric one-way analysis of variance analysis were conducted.
RESULTS: Six hundred twenty-eight letters of recommendation from 179 applicants were analyzed. Four hundred ninety-seven of 628 (79.1%) letters contained a SLOR. Mean percentile ratings were calculated for all the following domains: patient care (mean ± SD = 86.7 ± 8.7), medical knowledge (87.2 ± 8.6), interpersonal and communication (87.7 ± 9.3), procedural (86.6 ± 8.9), research (88.9 ± 9.0), ability to work within a team (89.6 ± 8.4), professionalism (90.8 ± 7.3), initiative and drive (90.6 ± 7.6), and commitment to orthopaedic surgery (91.1 ± 6.7). Forty-eight percent of applicants were indicated as "ranked to guarantee match." When compared with male applicants, female applicants demonstrated higher percentile ratings in patient care (88.6 ± 8.2 vs. 86.3 ± 8.7, p = 0.010), interpersonal and communication skills (90.6 ± 7.3 vs. 86.9 ± 9.6, p < 0.001), and ability to work within a team (91.3 ± 6.3 vs. 89.2 ± 8.8, p = 0.045). Higher United States Medical Licensing Examination step 1 (r = 0.08, p = 0.05) and step 2 scores (r = 0.10, p = 0.02) correlated with higher medical knowledge ratings. The number of publications (r = 0.3, p < 0.001) and presentations (r = 0.25, p < 0.001) correlated with research ratings.
CONCLUSION: SLORs demonstrated a profound ceiling effect, potentially limiting the utility of the instrument for the comparison of applicants. Future modifications to this instrument may include measures to better delineate between applicants.

Securing a residency position in orthopaedic surgery represents a formidable task, with 1,037 applicants vying for only 755 training positions[1]. This statistic fails to capture numerous additional applicants who self-select away from the field, daunted by the possibility of finishing medical school without obtaining a postgraduate medical training position, and facing the financial and emotional burdens of reapplication[2,3]. Concurrently, orthopaedic residency programs must find a manner in which to adequately evaluate an applicant’s fit for their particular environment, allowing the trainee to obtain success as a resident and beyond[4]. In addition, the field of orthopaedic surgery must determine an equitable manner in which to award entrance into this sought after profession, ensuring a qualified and diverse workforce to serve our patient population[5,6].

One component of the resident selection process is letters of recommendation (LOR). Eighty-nine percent of programs consider LORs when deciding to grant an applicant an interview, and LORs written by orthopaedic surgeons seem to hold a similar value in the resident selection process as Alpha Omega Alpha honor society status (outweighing other factors such as personal statements, research publications, and previous research experience)[7]. Despite the value placed on LORs, certain difficulties and variability lie in the interpretation of narrative letters of recommendations (NLORs)[8]. Moreover, “glowing reports”—overly positive evaluations that fail to portray the letter writer's true evaluation of the candidate—hinder the utility of letters to discriminate between applicants[8]. The American Orthopaedic Association (AOA) Council of Orthopaedic Residency Directors (CORDs), therefore, introduced standardized letters of recommendations (SLORs) in 2017 to “provide a global perspective on an applicant's candidacy that allows for easier and potentially more meaningful comparison to peers[9]” (Fig. 1)[10].

Fig. 1

American Orthopaedic Association Committee of Residency Program Directors standardized letter of recommendation.

Previous studies in emergency medicine and otolaryngology (ENT) indicate that SLORs reduce subjectivity and sex bias, when compared with NLORs[11-14]. Given the recent introduction of SLORs to the field of orthopaedic surgery, however, it is unclear whether this tool adds value to the residency selection process. Furthermore, no data exist indicating improved objectivity compared with NLORs[15] or the correlation between SLOR domains and known predictors of resident success. As such, the primary objective of this study was to define the utilization of SLORs and the distribution of applicant ratings in SLORs. The secondary objective of this study was to define the impact of self-identified sex and race, type and length of involvement between the letter writer and applicant, and traditional metrics of medical student achievement on SLOR domain ratings.

Methods

This study was submitted to the host institutions research protection office. Because no identifying information was collected for the purpose of this study, this activity was deemed nonhuman subject research.

A priori sample size calculation was conducted with the assistance of the performing institution’s division of biostatistics. Using a nonparametric Mann Whitney–Wilcoxon test, we assumed an average percentile ranking of 81.2% and standard deviation of 13.4%[16], with a 5:1 male-to-female applicant ratio[16]. Based on 3,000 Monte Carlo sample, 444 SLORs were required to achieve 80% to detect a 5% difference between male and female applicants. Assuming a 20% attrition rate and 3 SLORs per applicant, 179 applicants were selected for inclusion in this study.

All Electronic Residency Application Service (ERAS) applications submitted to a single orthopaedic residency program were eligible for inclusion in this study. No filter, screening criteria, or method of stratification was used before file selection. This program is a large (38 residents), academic residency program, currently ranked in the top 5 training programs by Doximity. This program did not require the SLOR to be included in submitted applications. Files were placed in alphabetical order and assigned contiguous number designations. Then—using a random number generator—the requisite number of applications were selected for review.

Applicant names were electronically redacted from all applications, whereas applicant birth date, letter writer name, and any further potential identifying information were manually redacted by a program manager who had previously reviewed all applications.

After redaction of all identifying information, a single reviewer (P.M.I.) extracted data from each ERAS application and associated LOR using Research Electronic Data Capture software. Self-identified race, sex, applicant type (i.e., MD, DO, Canadian, International), AOA membership, Gold Humanism Honor Society membership, United States Medical Licensing Examination (USMLE) step 1 Score, USMLE step 2 Score, number of peer reviewed publications/abstracts, and number of poster/oral presentations were recorded for each applicant.

Letters of recommendations were first reviewed for its type (SLOR only, SLOR with associated comments, SLOR and NLOR, or NLOR only). Letters of recommendation without an associate SLOR were excluded, as were early renditions of SLORs not containing a percentile rating system. Then, percentile ranking for each domain was recorded. For applications containing an electronically generated (X) in the middle of a decile, the applicant was given a percentile ranking averaging the higher and lower deciles (e.g., an “X” between the 80th and 90th percentile was recorded as the 85th percentile). For manually generated demarcations, a ruler was used to determine the percentile score assigned to the applicant[16]. This measurement was based on the inflection point of the demarcation (point of intersection in “x” and lowest point of “✓”).

Descriptive statistical analysis was performed for all domains listed on the SLOR. In addition, χ2 tests were performed for class-type data. For numerical data, Kruskal-Wallis rank sum tests (nonparametric one-way analysis of variance) analysis was performed. All statistical analysis was performed with SAS 9.4 Software.

Results

Cohort Characteristics

One hundred seventy-nine applicants were selected for inclusion in this study (Table I). One hundred and seventy-seven applicants (98.9%) analyzed attended a US allopathic school of medicine, 1 applicant attended a US osteopathic school of medicine, and 1 applicant attended an international school of medicine.

Sixty-nine (42.9%) applicants received honors in both medicine and surgery, 53 (32.9%) in either medicine or surgery, and 39 (24.2%) did not receive honors in either clerkship. Eighteen (10.1%) applicants did not receive clerkship grades or did not provide this information in the application.

Fifty-two (29.1%) applicants were members of AOA, whereas 17 (9.5%) were members of Gold Humanism Honors Society. The average applicant scored 247.9 ± 11.3 (mean ± SD) on USMLE step I and 254.2 ± 11.1 on USMLE step II (Tables I and II).

Letters of Recommendation

From the 179 applicants analyzed, 628 LORs were submitted (Fig. 2), with 494 LORs containing a SLOR. Three SLORs were excluded from analysis, given discordance between domain ratings (lowest first and second deciles) and summative rankings (ranked to match, high rank).

Fig. 2

Type of letters of recommendation submitted by applicants: Standardized Letter of Recommendation (SLOR) only (n = 11), Narrative Letter of Recommendation (NLOR) only (n = 138), SLOR with comments (200), SLOR, and NLOR (n = 279).

The length of contact between a letter writer and applicant was “<2 weeks” for 63 (12.8%) SLORs, “1 month” for 161 (32.6%) SLORs, “1 to 6 months” for 56 (11.3%) SLORs, “6 to 12 months” for 33 (6.7%) SLORs, “1 to 2 years” for 91 (18.4%) of SLORs, and “2 + years” for 77 (15.6%) SLORs. No length of contact was indicated on 13 (2.6%) SLORs. “Extended direct clinical contact” (n = 233 letters) was the most common nature of contact between letter writers and applicants, followed by “direct research contact” (n = 96), “limited direct clinical contact” (n = 83), “known indirectly through others” (n = 54), and “committee prepared letter” (n = 12). Of note, LORs could contain multiple or no answer for this portion of the instrument.

Mean percentile ratings were calculated for all the following domains (Table III): patient care (mean ± SD = 86.7 ± 8.7), medical knowledge (87.2 ± 8.6), interpersonal and communication (87.7 ± 9.3), procedural (86.6 ± 8.9), research (88.9 ± 9.0), ability to work within a team (89.6 ± 8.4), professionalism (90.8 ± 7.3), initiative and drive (90.6 ± 7.6), and commitment to orthopaedic surgery (91.1 ± 6.7). Forty-eight percent of applicants were indicated as “ranked to guarantee match.” For all domains, 75% of applicants received a rating of 85 of greater (Fig. 3). Moreover, of the 4,124 domain ratings recorded, 2 ratings (0.04%) were below the 40th percentile and 16 (0.3%) ratings were below the 50th percentile.

Fig. 3

Distribution of applicant domain ratings.

Impact of Sex and Race

Female applicants received statistically significant higher domain ratings in “patient care” (88.6 ± 8.2 vs. 86.3 ± 8.7, p = 0.01), “interpersonal and communication skills” (90.6 ± 7.3 vs. 86.9 ± 9.6, p = <0.001), and “ability to work within a team” (91.3 ± 6.9 vs. 89.2 ± 8.8, p = 0.048) domains when compared with male applicants to the field (Table IV).

Applicants self-identifying as “white” received statistically significant higher ratings in the “commitment to the field of orthopaedic surgery” domain than applicants not self-identifying as “white” (91.7 ± 6.2 vs. 90.2 ± 7.3, p = 0.025), but no difference was identified in other domains (Table V).

Impact of the Length and Nature of Relationship

Domain ratings in all categories—apart from “procedural skills”—increased with increasing length of letter writer-applicant relationship length (Table VI). The nature of relationship demonstrated a statistically significant impact on domain ratings, with letters generated following “extended direct clinical contact” or “direct research contact” demonstrating superior ratings than other natures of contact (p < 0.001). Average domain ratings from “committee prepared letters” were lowest for all categories.

Correlation Between SLORs and Known Predictors of Resident Performance

Both USMLE step I (r = 0.128, p = 0.004) and USMLE step II (r = 0.163, <0.001) demonstrated a weak, but statistically significant correlation with the “medical knowledge” domain ratings. In addition, the number of publications (r = 0.264, p < 0.001) and number of presentations (r = 0.234, p < 0.001) demonstrated a statistically significant positive correlation with the “research” domain rating. No statically significant correlation was noted between honors in surgery or medicine and “patient care” domain rating (p = 0.21).

Members of AOA received higher domains ratings in “medical knowledge” (88.1 ± 7.5 vs. 85.6 ± 9.2, p = 0.05), “interpersonal and communication skills” (87.8 ± 9.4 vs. 85.1 ± 10.6, p = 0.04), “procedural skills” (87.6 ± 8.5 vs. 84.4 ± 9.0, p < 0.001), and “initiative and drive” (91.6 ± 6.6 vs. 88.3 ± 10.2, p = 0.05) domains, when compared with applicants indicating Alpha Omega Alpha elections were held during senior year. No difference was noted between applicants elected to AOA and those not indicating status of AOA membership.

Discussion

The AOA CORD introduced SLORs with the purpose of simplifying and improving the utilization LOR in the orthopaedic residency application process. A requisite component of the AOA's goal is the widespread acceptance of the instrument, which has occurred based on the above findings: over three-quarters of the LOR analyzed contained a SLOR. As such, a critical analysis of the instrument becomes a vital asset to the orthopaedic community.

In this study, SLORs domain ratings are primarily confined to the top 10% to 20% of ratings, with means for “professionalism,” “initiative and drive,” and “commitment to orthopaedic surgery” fallings within the top decile. Such findings signify a profound ceiling effect for SLORs, limiting the utility of this tool for discriminating between applicants to the field. This outcome is expected because similar pursuits in otolaryngology[16] and emergency medicine were met[17] with similar elevated evaluations of applicants.

The concept that all applicants are in fact “above average” cannot—by definition—be true because the SLOR instructions recommend comparison “to other orthopaedic applicants.” Thus, the profound ceiling effect must be explained by alternative means. First, the applicants to the field of orthopaedics represent a uniquely qualified pool of medical students[1], creating challenges for discriminating between applicants. In addition, letter writers may compare the applicant to the average student, rather than average applicant, causing most orthopaedic applicants to accurately fall in the top deciles. Alternatively, distinguishing feature of applicants may be present in domains not evaluated by the SLOR or may be difficult to categorize. Moreover, the average writer of a letter of recommendation is only interacting with the student for 1 month in most cases—likely in the clinic setting or in the operating theater—and may not have the opportunity to evaluate the student in situations where applicants may distinguish themselves: nights, weekends, urgent or emergent settings, or interactions with support staff. Finally, we cannot discount the potential influence medical school administration may have on the letter writer to support an applicant. A medical school is, in part, defined by the success of its applicants in The Match[18]. Thus, a harmful letter of recommendation that impacts an applicant’s success may lead to displeasure among medical school leadership.

A potential remedy for this behavior may involve providing an increasingly granular manner to evaluate applicants within the top 2 or 3 deciles of applicants. In this way, a letter writer may distinguish the applicant in the top 1% from the applicant in the top 10%. In addition, emphasis may be placed in the instructions on the intended grading scale of the applicant: a comparison to applicants to the field of orthopaedic surgery rather than all fourth-year medical students.

This study also demonstrates that female applicants were likely to receive higher ratings in both communication and teamwork domains, contrary to previous work in ENT[13] and emergency medicine[17] that demonstrated no sex bias in SLORs. Although our methodology provides no explanation as to the cause of such discrepancy, the cause and ramifications of sex differences in SLORs must be strongly considered and investigated further.

Finally, the statistically significant positive association between known predictors of resident success and SLOR domain ratings may indicate the utility of SLORs in predicting resident performance. In the case of AOA status, our findings demonstrated a significant difference in domain ratings for members of AOA when compared with individuals indicating elections were held during their senior year, but not when compared with applicants not indicating any AOA status. Such findings are difficult to interpret but likely point to the variability of both AOA elections (competitiveness of candidates at a given school, number of students elected in the spring, eligibility criteria for election, and inclusion/exclusion of applicants changing graduation year to pursue a master or dedicated research) and indication of AOA status on ERAS applications (applicant not likely to be elected to AOA may either provide no answer or indicate elections are held during senior year). Regardless, our findings generally indicated SLORs grossly correlate with other predictors of resident success. Ultimately, each metric in resident selection is of value in-so-far because it serves to inform or predict future resident performance, thus correlation between applicant domain rating and subsequent measures of resident performance will ultimately define the value of SLORs in assisting in resident selection.

Limitations

Limitations in this study include the recording of domain ratings for applicants, given the variability with which these domain ratings were conveyed by letter authors. The SLOR is intended to utilize a decile rating system with electronic designations. However, many authors printed SLOR instruments and provided manual demarcations for applicant ratings at variable points along the domain scale. Given this actual utilization of the form, the domain ratings were treated as continuous variables during data collection and analysis. Importantly, such data collection would not impact conclusions detailed above but merely increase the sensitivity of detecting these effects. Finally, this study did not account for the impact of professorial rank on applicant ranking, although such data are available in the SLOR, but such data may not have significantly impacted the ceiling effect identified.

Our results come from a highly ranked, urban, and academic training program; thus, this pool of applicants and resultant findings may differ from the applicants to smaller, rural, or community-based programs. However, the academic characteristics of this cohort (USMLE step 1: 247.9, USMLE step 2: 254.2, 29.1% AOA) are similar to the academic characteristics of matches applicants to the field (USMLE step 1: 248, USMLE step 2: 255, 41.8% AOA)[19]. Consequently, these results remain generalizable to the applicant population.

Finally, given 10 domains are available for analysis on SLORs and numerous independent variables were selected, the possibility of type II error in our analysis exists. However, for sex-based differences, “patient care” and “interpersonal and communication skills” achieved p-values of <0.01, decreasing the likelihood of type II error in our findings.

Conclusion

SLORs were introduced in the orthopaedic residency selection process with the intent to “provide a global perspective on an applicant's candidacy that allows for easier and potentially more meaningful comparison to peers[9].” Although widely adopted by the 2018 to 2019 application cycle, this instrument demonstrates a profound ceiling effect with most applicants rated in the top 2 deciles. Moreover, rating discrepancies were noted between both female and male applicants and between white and self-identified minority applicants. Finally, weak correlation was noted between traditional measures of applicant performance and corresponding domain ratings. To improve the value of SLORs, the orthopaedic community must identify remedies for overly positive domain ratings, and further research is warranted to define the implications and impact of sex and racial discrepancies noted in SLORs. Ultimately, the value of SLORs will be determined by the ability of this instrument to predict resident performance, which may be defined as current applicant's progress through residency.

TABLE I

Demographic Data for 179 Applicants Analyzed

Applicant Characteristics
Gender
Male	141 (78.8%)
Female	38 (21.2%)
Race
Asian	32 (17.9%)
Black or African American	10 (5.6%)
Latino or of Spanish Origin	13 (7.3%)
White	107 (59.8%)
Other	1 (0.6%)
Unknown race/ethnicity	16 (8.9%)
Academic performance
USMLE step 1 (mean ± SD)	247.9 ± 11.3
USMLE step 2 (mean ± SD)	254.2 ± 11.1
No. of publications/abstracts (mean ± SD)	2.98 ± 3.93
No. of poster/oral presentations (mean ± SD)	6.94 ± 8.05
Member of Alpha Omega Alpha	52 (29.1%)
Member of Gold Humanism Honor Society	17 (9.5%)

TABLE II

Displays Cohort Academic Characteristic, When Compared with Successful and Unsuccessful Applicants in the 2016 Match

	Cohort	Matched Applicant	Unmatched Applicants
USMLE step 1	248.7	247	238
USMLE step 2	254.7	253	245
AOA member	29.10%	34.40%	12.20%

TABLE III

Mean, Median, and Interquartile Range of 497 Letters of Recommendation Analyzed

Domain	Mean ± SD	Median (Q1, Q3)
Patient care	86.7 ± 8.7	86.7 (85, 95)
Medical knowledge	87.2 ± 8.6	86.0 (85, 95)
Interpersonal and communication	87.7 ± 9.3	91.0 (85, 95)
Procedural	86.6 ± 8.9	85.0 (85, 95)
Research	88.9 ± 9.0	94.0 (85, 95)
Ability to work within a team	89.6 ± 8.4	95.0 (85, 95)
Professionalism	90.8 ± 7.3	95.0 (85, 95)
Initiative and drive	90.6 ± 7.6	95.0 (85, 95)
Commitment to orthopaedic surgery	91.1 ± 6.7	95.0 (85, 95)

TABLE IV

Impact of Gender on Domain Ratings*

Domain	Male, n = 392 (Mean ± SD)	Female, n = 102 (Mean ± SD)	p
Patient care	86.3 ± 8.7	88.6 ± 8.2	0.01
Medical knowledge	86.9 ± 8.8	88.5 ± 7.5	0.17
Interpersonal and communication	86.9 ± 9.6	90.6 ± 7.3	<0.001
Procedural	86.3 ± 9.2	87.7 ± 7.6	0.40
Research	88.5 ± 9.6	90.2 ± 6.9	0.33
Ability to work within a team	89.2 ± 8.8	91.3 ± 6.3	0.05
Professionalism	90.5 ± 7.7	91.9 ± 5.9	0.09
Initiative and drive	90.3 ± 7.9	91.8 ± 6.2	0.06
Commitment to orthopaedic surgery	91.0 ± 6.9	91.7 ± 5.8	0.66

Bolded entries represent statistical significance (p < 0.05).

TABLE V

Impact of Self-Identified Race on Council of Orthopaedic Residency Director Standardized Letters of Recommendation Domain Ratings*

Domain	Self-Identify White (Mean ± SD, n = 303)	Do Not Self-identify White (Mean ± SD, n = 191)	p
Patient care	86.3 ± 9.3	87.0 ± 8.3	0.65
Medical knowledge	87.0 ± 9.6	87.4 ± 7.8	0.68
Interpersonal and communication	87.7 ± 9.9	87.7 ± 8.9	0.39
Procedural	86.2 ± 9.7	86.9 ± 8.3	0.72
Research	88.3 ± 10.4	89.3 ± 8.1	1
Ability to work within a team	89.8 ± 8.9	89.6 ± 8.0	0.21
Professionalism	91.0± 7.4	90.6 ± 7.3	0.38
Initiative and drive	90.2 ± 8.3	90.8 ± 7.0	0.91
Commitment to orthopaedic surgery	90.2 ± 7.3	91.7 ± 6.2	0.025

Bolded entries represent statistical significance (p < 0.05).

TABLE VI

Impact of Length of Relationship on Standardized Letters of Recommendation Domain Ratings*

Domain	<2 Weeks (Mean ± SD)	1 Month (Mean ± SD)	1-6 Months (Mean ± SD)	6-12 Months (Mean ± SD)	1-2 Years (Mean ± SD)	> 2 Years (Mean ± SD)	p
Patient care	85.4 ± 9.4	85.5 ± 9.0	86.9 ± 8.4	86.6 ± 7.5	87.6 ± 8.2	89.9 ± 7.9	0.002
Medical knowledge	85.2 ± 9.4	85.8 ± 9.5	87.7 ± 7.9	88.2 ± 6.2	88.6 ± 6.6	89.8 ± 8.4	0.003
Interpersonal and communication	86.4 ± 9.4	86.1 ± 9.9	88.1 ± 8.7	88.2 ± 7.8	89.0 ± 8.6	90.7 ± 8.9	<0.001
Procedural	85.0 ± 9.9	85.8 ± 9.3	86.3 ± 7.8	86.3 ± 8.0	87.5 ± 8.5	88.9 ± 8.7	0.10
Research	87.4 ± 7.6	86.3 ± 10.4	89.8 ± 8.1	90.0 ± 6.1	88.1 ± 10.4	92.2 ± 6.7	<0.001
Ability to work within a team	87.5 ± 10.3	88.6 ± 8.9	90.5 ± 7.0	89.2 ± 6.7	90.2 ± 7.7	92.7 ± 6.9	<0.001
Professionalism	89.5 ± 7.0	90.0 ± 7.6	91.9 ± 5.8	91.2 ± 4.8	90.9 ± 8.2	92.7 ± 7.4	<0.001
Initiative and drive	89.0 ± 7.9	89.1 ± 8.7	92.2 ± 5.9	90.6 ± 6.8	91.2 ± 7.0	92.8 ± 6.2	<0.001
Commitment to orthopaedic surgery	90.1 ± 7.2	89.7 ± 7.5	91.7 ± 6.3	92.5 ± 4.1	91.4 ± 6.8	93.8 ± 4.3	<0.001

Bolded entries represent statistical significance (p < 0.05).