The Current Status of Medical School Clerkship Grades in Residency Applicants.

BACKGROUND: Two criteria that have been investigated for evaluating orthopedic surgery residency candidates are achieving an "honors" grade during a surgery clerkship and the total number of honors grades received in all clerkships. Unfortunately, the rate of honors grades given and the criteria for earning an honors grade differ between medical schools, making comparison of applicants from different medical schools difficult.
OBJECTIVE: We measured the rate of honors grades in clerkships at different medical schools in the United States to examine the utility of clerkship grades in evaluating orthopedic surgery residency applicants.
METHODS: Adequate data via the Electronic Residency Application Service were available for 86 of 142 Association of American Medical Colleges medical schools from the 2017 Match cycle. Descriptive statistics and Wilcoxon rank sum tests were performed to identify differences in grade distributions within each clerkship and in school ranking for research by U.S. News & World Report.
RESULTS: For the surgery clerkship, the median rate of honors grades given was 32.5% (range 5%-67%). There was a high rate of interinstitutional variability in all clerkships. We were unable to demonstrate a statistically significant relationship between research ranking and percentage honors grades given for individual clerkships.
CONCLUSIONS: A standardized method for grading medical students during clinical clerkships does not exist, resulting in a high degree of interinstitutional variability. Surgery clerkship grades are an unreliable measure for comparing orthopedic surgery residency applicants from different medical schools. Standardized measures of applicant evaluation might be helpful in the future. Accreditation Council for Graduate Medical Education 2020.

What was known and gap

Residency programs believe honors grades in clerkships are important when evaluating applicants, but a lack of consistent grading schemes between schools makes comparison of applicants from different medical schools difficult.

What is new

A description of the rate of honors grades in clerkships at various medical schools attended by orthopedic surgery applicants.

Limitations

The survey of grading practices included only 86 of 142 medical schools due to differences in grading schemes.

Bottom line

Surgery clerkship grades are an unreliable measure for comparing orthopedic surgery residency applicants from different medical schools.

Introduction

When evaluating applicants, residency programs can use a multitude of data points in their effort to predict future resident success. Despite several studies evaluating various criteria as predictors of success during residency, the invitation, interview, and ranking process are inexact—there is need for improvement in most if not all specialties. A survey of 156 orthopedic residency program directors across the United States1 found that 1 in 6 resident selections was considered to be “inappropriate,” while 1 in every 12 selections was deemed to be a “serious mistake.” Historically, an applicant's score on Step 1 of the United States Medical Licensing Examination (USMLE) has played a large part in determining whether a candidate is chosen for an interview.2,3 In addition to USMLE scores, orthopedic program directors have tended to place the highest value on class rank and American Orthopaedic Association status when evaluating applicants.1 The most recent survey of orthopedic surgery program directors published by the National Resident Matching Program in 2018 illustrates these are still given high value, along with letters of recommendation in the specialty.4 The emphasis on the domain of cognitive skills may be partially due to the fact that residency programs are expected to have higher than a 75% first-time pass rate on Part I of the American Board of Orthopaedic Surgery (ABOS) Examination in order to maintain their accreditation.1 The experience of orthopedic residency programs and the literature about matching into such programs can likely be applied to other competitive surgical subspecialties.

Multiple outcomes have been used to measure resident success, including Orthopedic In-Training Examination (OITE) scores, ABOS Part I success, appointment as executive chief resident, and number of publications and faculty evaluations. Studies evaluating the predictive value of medical school clerkship grades on these measures of resident success have failed to reach consensus. Raman et al5 reported that a higher number of honors grades in medical school clerkships had a moderate positive linear correlation with ABOS Part I scores and a weak positive correlation with senior resident OITE scores; however, there was no correlation between number of honors grades and subjective outcome measures (ie, faculty evaluations).

Harris et al6 showed similar average rates of giving honors grades in the different clerkship rotations. However, in evaluating the surgery clerkship, they found a high level of interinstitutional variability, with honors grades given in a range from 2% to 75%, with most schools falling between 15% and 40%. They concluded that it is difficult to compare honors grades between schools; thus, the grade of “honors” is less useful as a primary tool for assessing the aptitude of potential residents. Related studies7–9 in other medical school rotations have reached similar conclusions.

Our study reevaluates the utility of the grade of honors using more recent data from a larger number of medical schools. We predict that the rate of honors grades given remains disparate between schools. Although this study was performed with a focus on the surgery clerkship, this topic is relevant to all specialties interested in using clerkship grades as a measure of residency applicant potential.

Using more recent data from a larger number of medical schools, we examined the rate of medical school clerkship honors grades and evaluated the correlation between the rate of honors grades and medical school research ranking by U.S. News & World Report in order to determine whether or not schools that have a higher ranking are more likely to give their students an honors grade.

Methods

Information from the Electronic Residency Application Service was used to gather grade distribution data from applications submitted to our orthopedic residency for the 2017 National Resident Matching Program Match. Only allopathic schools within the United States were examined. Of 142 Association of American Medical Colleges–accredited medical schools, 133 different schools had students applying to our program and thus had data available for our review. Of these schools, 47 were excluded, including 20 that did not provide grade distribution data and 27 that used a grading scheme other than honors, high pass, and pass; this left 86 schools with data for analysis (Figure). The percent of honors grades given by each medical school for each core clerkship (family medicine, internal medicine, obstetrics and gynecology, pediatrics, psychiatry, and surgery) was extracted from the provided graphs or tables attached to the applicant's Medical Student Performance Evaluation letter. The same was done for the categories of “high pass” and “pass” when the information was provided. For medical schools that combined certain clerkships (ie, family medicine and internal medicine), their grade breakdowns were analyzed as reported.

Figure

Medical Schools Included in Analysis

The national average percentage of honors grades was calculated for each clerkship using the combined data from all schools with complete data. Descriptive statistics were performed, and interquartile ranges were identified.

A subanalysis was then performed to evaluate schools ranked highest for research by U.S. News & World Report. Our complete data for 16 of the top 25 medical schools as ranked by U.S. News & World Report were included for analysis, investigating the rate of overall percentage of honors grades given at these schools, compared with the other 70 schools with adequate data. This was then broken down by clerkship, and a 1-tailed Student's t test was performed to examine the percentage of honors grades given by each clerkship at schools included in the top 25 versus all others. A hierarchical regression analysis was then performed to evaluate the relationship between U.S. News & World Report ranking and percentage of honors grades given.

The grade distribution percentages were treated as continuous variables. Overall, the data were nonparametric. Wilcoxon rank sum tests were performed to identify differences between grade distributions within each clerkship rotation, as well as for the schools deemed to be in the top 25. The Steel-Dwass method of multiple comparisons was performed to examine differences in honors between each clerkship. We considered a P value less than .05 to be statistically significant for all comparisons. All statistical analyses were performed using JMP Pro 13.0.0 (SAS Institute Inc, Cary, NC). Parametric testing was performed to evaluate the correlation between numerical ranking for research and percentage of honors grades given, as well as the correlations between the percentage receiving honors in 1 clerkship and each of the other clerkships at that medical school.

Results

From 1 year of applications to 1 residency program, data from 86 medical schools were analyzed. For the surgery clerkship the median rate of honors grades was 32.5% (range 5%–67%, interquartile range 23.25; Table 1). Between 9% and 80% of students received a “high pass” across all medical schools analyzed; 2% to 84% received a grade of “pass.” This high rate of interinstitutional variability was present in all clerkships.

Table 1

Percent Honors Distribution by Specialty

Specialty	n	Mean	SD	Median	Interquartile Range	Range
Surgery	86	32.74	14.83	32.5	23.25	62 (5–67)
Pediatrics	86	34.29	13.95	33.0	21.0	67 (5–72)
Internal medicine	86	34.78	14.66	32.0	18.025	73 (7–80)
Psychiatry	85	42.91	17.70	41.0	27.0	78 (10–88)
Family medicine	82	37.65	19.32	33.5	22.95	91 (5–96)
Obstetrics and gynecology	85	36.92	14.93	35.0	22.5	70 (10–80)

At the individual clerkship level, the rate of honors grades was significantly higher in psychiatry clerkships (42.9%) than in surgery (32.7%, P = .002), pediatrics (34.3%, P = .016), and internal medicine (34.8%, P = .025). Schools in the top quartile of awarding honors grades did so across all clerkships when compared with schools in the remaining quartiles (P < .0001 for all clerkships). This was confirmed with correlation analysis (Table 2).

Table 2

Correlation Coefficientsa Between Proportion of Students Receiving Honors in Each Clerkship at a Program

Program	Surgery	Pediatrics	Internal Medicine	Psychiatry	Family Medicine	Obstetrics and Gynecology
Surgery		0.525	0.549	0.522	0.574	0.460
Pediatrics	0.525		0.632	0.719	0.591	0.617
Internal medicine	0.549	0.632		0.509	0.539	0.689
Psychiatry	0.522	0.719	0.509		0.576	0.615
Family medicine	0.574	0.591	0.539	0.576		0.619
Obstetrics and gynecology	0.460	0.617	0.689	0.615	0.619

Note: The correlation coefficient (r) measures the strength of the relationship between 2 variables (eg, a program with a high proportion of students receiving honors in psychiatry will also have a high proportion of students receiving honors in pediatrics; r = 0.719).

Subanalysis revealed that schools in the U.S. News & World Report top 25 (n = 16) gave out a significantly larger percentage of honors grades overall (41% versus 35%, P = .006) than other medical schools (Table 3), but this was not statistically significant when individual clerkships were evaluated or when research ranking for each medical school was used.

Table 3

Average Percent Honors Given by Schools Considered Top 25 by U.S. News & World Report Compared With All Other Schools Available for Review

Schools	n (Total = 86)	% Honors (SD)	% High Pass (SD)	% Pass (SD)
Top 25	16	41.01 (14.75)	39.59 (12.5)	18.65 (16.49)
All others	70	35.49 (16.4)	39.76 (15.73)	23.93 (20.24)
P valuea		.0006	.94	.028

Bold P values are significant.

Discussion

Our data demonstrate a high overall variability in the distribution of honors grades in third-year medical school clerkships, including the surgery clerkship. Thus, it seems that for specialties that prioritize an individual clerkship grade when evaluating applicants, such as orthopedic surgery does with the surgery clerkship, using an individual grade may not be a reliable criterion.

Prior studies5,10 have reported correlations between medical school grades (ie, number of honors grades) and objective outcome measures of orthopedic resident success, including OITE scores and ABOS Part I scores. However, our study demonstrates that there is a wide variation in the distribution of honors grades among all clerkships, with an average 30% of students receiving an honors grade for each clerkship across all medical schools analyzed. The high variability in grade distribution explains, at least in part, the conflicting evidence regarding the predictive potential of honors grades on those specific areas that are thought to measure resident success.2

Our data recapitulate the findings of Harris et al,6 yet our study uses more current data from a larger number of medical schools. In addition, our data show that interinstitutional variability persists, that no standardized grading format is present, and that schools that give out more honors grades in one clerkship tend to do so in all clerkships. Therefore, a student who is considered “good” to “great” at one medical school may have more honors grades than even some “great” or “excellent” students at another medical school that gives out a lower total percentage of honors scores. This raises concern that students from medical schools with more rigid grading criteria may be ranked unfairly, or worse, may be discouraged from applying to more competitive specialties since they may have fewer honors grades on their applications; yet they could potentially be very qualified and successful residents and surgeons. In these cases, it must be emphasized that Medical Student Performance Evaluation/class rank, which mitigates the effect of an individual school's grade inflation or deflation, should be evaluated closely. Although our study was done with a focus on surgery, our data are relevant to all specialties that use clerkship grades as criteria for interview and applicant selection.

Additionally, although our study demonstrates that medical schools ranked in the top 25 for research by U.S. News & World Report give out an overall larger percentage of honors grades, when each school was evaluated by its national research rank, a relevant correlation did not exist, and there was no association between research ranking and individual clerkship grades. Thus, students from the 25 highest-ranked medical schools may have a small advantage of an increased chance of receiving both a grade of honors for their core clerkship rotations as well as more honors grades overall, but this does not apply to all medical schools.

Overall, the range of students receiving honors in their surgery clerkships has narrowed slightly from 2% to 75% in 20054 to 5% to 67% in our more recent study. We propose that a more standardized grading system be implemented across different medical schools to increase the utility of clerkship grades in truly evaluating students. This would have to address both the number of honors given as well as the criteria for determining an honors grade. We acknowledge the difficulty of this but believe it is important to the future of health care. Our data might suggest that at schools with high rates of honors, failure to achieve this grade may provide more useful information about an applicant than actually achieving the grade of honors, but this would require further study.

Limitations to this study include that this is a survey of grading practices at only 86 of 142 medical schools; we do not have information about characteristics of the 39% of schools that either did not follow the honors/high pass/pass grading scheme or did not have students apply to our program, which may have created a selection bias. In addition, it was noted in the review of the Electronic Residency Application Service data that some schools provided different data for different students. For example, if a student performed poorly, some schools did not include grade distribution charts on the application, whereas grade distribution data were provided for higher-performing students. Therefore, there may be some bias in data from the schools that were included, and our averages, if compared to that of all schools combined, may not be entirely accurate. Lastly, our data are limited to one specific year of grade distributions from orthopedic surgery residency applicants, which may not fully or accurately represent every school's grading practices.

Conclusions

There is a high degree of interinstitutional variability in medical school clinical clerkship grades, as a standardized method for grading students does not exist. Surgery clerkship grades are an unreliable measure for comparing orthopedic surgery residency applicants from different medical schools. A more standardized grading system implemented across different medical schools to increase the utility of clerkship grades may be helpful when evaluating students.