PURPOSE: To elicit and analyze the strengths and weaknesses of resident operative performance (ROP) as identified by independent expert ratings. METHODS: Four groups of expert surgeons, blinded to resident training level, evaluated ROP on 2 each of laparoscopic cholecystectomy, thyroidectomy, open inguinal, and laparoscopic ventral hernia repair audio videos, and listed strengths and weaknesses. Comments were coded as technical skills, forward planning, self-direction, situation awareness, and patient safety, and as either procedure specific or general skills. RESULTS: Eighteen experts independently entered 1087 comments on 8 ROP strengths and weakness. In the post independent rating discussion, consensus was reached on 85 (28%) of 300 post rating comments with majority agreement on another 83 (28%). Overall, the dominant focus was on forward planning. With the exception of the laparoscopic cholecystectomy cases, raters focused more on general than on procedure-specific skills (P < .05). CONCLUSION: Fewer than 30% of expert rater comments focused on technical skills when considering ROP strengths and weaknesses. Although there was some variation in individual comments, majority agreement was reached on 56% of comments during the post independent rating discussion. These findings will improve rater training and further the implementation of a national assessment process to evaluate end of training surgical competence and operative proficiency.
PURPOSE: To elicit and analyze the strengths and weaknesses of resident operative performance (ROP) as identified by independent expert ratings. METHODS: Four groups of expert surgeons, blinded to resident training level, evaluated ROP on 2 each of laparoscopic cholecystectomy, thyroidectomy, open inguinal, and laparoscopic ventral hernia repair audio videos, and listed strengths and weaknesses. Comments were coded as technical skills, forward planning, self-direction, situation awareness, and patient safety, and as either procedure specific or general skills. RESULTS: Eighteen experts independently entered 1087 comments on 8 ROP strengths and weakness. In the post independent rating discussion, consensus was reached on 85 (28%) of 300 post rating comments with majority agreement on another 83 (28%). Overall, the dominant focus was on forward planning. With the exception of the laparoscopic cholecystectomy cases, raters focused more on general than on procedure-specific skills (P < .05). CONCLUSION: Fewer than 30% of expert rater comments focused on technical skills when considering ROP strengths and weaknesses. Although there was some variation in individual comments, majority agreement was reached on 56% of comments during the post independent rating discussion. These findings will improve rater training and further the implementation of a national assessment process to evaluate end of training surgical competence and operative proficiency.