Development of an objective rating system for residency applications.

A method was developed to include the opinions of all full-time surgical faculty in reproducible ratings of surgical residency applications according to five attributes: knowledge, judgment, technical skills, work habits, and interpersonal skills. Criteria for classifying each attribute on a scale of five intervals from deans' letters, recommendations, and interviews were developed and used to describe 25 theoretic applications. The applications were rated by each of 14 faculty members on a scale of 1 to 100. The value for each interval of each attribute was determined by conjoint measurement and verified with a second set of theoretic applications. The variation in the second set from faculty opinion was less than 1% (r2 greater than 0.99). The 56 applications for the four positions in our 5-year surgical training program were stripped of identifying information and rated objectively according to the criteria for classification and the values generated. The objective ratings correlated significantly (p less than 0.001) with the interview ratings of seven of the eight faculty interviewers. Variation of the objective ratings from the consensus of all full-time faculty was less than 1% (r2 greater than 0.99). Variation between the actual rank submitted to the National Resident Matching Program and that from the objective rating system was less than 1% (r2 greater than 0.99). As a result of our experience, we will be using this reproducible rating system for the preliminary ranking of our applicants next year.