Scoring of orthopaedic residency applicants: is a scoring system reliable?

The purpose of the current study was to examine the interobserver reliability of a scoring system designed to objectify the screening of orthopaedic residency applications. Forty residency applications were selected randomly from those received in 1998 and were scored independently by six observers. The scoring system used included objective and subjective elements. Weights were assigned to individual data elements based on the results of a previous investigation. Interobserver reliability was calculated using an intraclass correlation coefficient. The overall reliability of the scoring system revealed an intraclass correlation coefficient of 0.80. The intraclass correlation coefficients for individual elements in the scoring system ranged from 0.28 to 0.98. The intraclass correlation coefficient was high for elements that were numeric, but was low for elements that were more subjective. Observers thought the scoring form was easy to complete but found some of the subjective data elements difficult to score. No benchmarks exist by which to define an acceptable intraclass correlation coefficient of a scoring model for resident applicants. Even with the use of careful definitions, raters had poor reliability in scoring elements such as letters of recommendation and personal statements. The results of the current study indicate that one score of objective data elements is adequate to screen a residency application for these elements. Programs screening applicants based on subjective elements, however, should be aware that great interobserver variability exists in the interpretation of those elements.