PURPOSE: An operative performance rating system for urology residents was developed for 6 sentinel urological procedures. We tested the reliability, validity and feasibility of the operative performance rating system for urology residents. MATERIALS AND METHODS: The operative performance rating system of each procedure contained a 3-point case difficulty scale, 4 to 6 procedure specific items, 3 general items and an overall performance item. A Likert scale of 1 to 5 was used for each item. A single video/audio record of each procedure was evaluated by the faculty. Single item interrater agreement was measured by comparing the observed variance and random measurement error variance. Resident operative performance evaluations were completed on line. Internal consistency reliability was measured using Cronbach α. Overall scale scores by resident training postgraduate year level were compared using 1-way ANOVA. RESULTS: Faculty evaluation of video/audio records showed an interrater agreement range of 0.71 to 0.92. Faculty evaluations of resident operative performance demonstrated an internal consistency reliability range of 0.91 to 0.95. Significant differences in overall scale scores between postgraduate year levels were noted for 3 of the 6 procedures (p ≤0.0016). CONCLUSIONS: An operative performance rating system for urology residents is feasible using an Internet based resident management system. Interrater agreement and internal consistency reliability meet threshold limits for checklist evaluation instruments. The operative performance rating system can discriminate among postgraduate year levels of resident training. A validated operative performance rating system can offer residents immediate, objective feedback on surgical performance and enable program directors to monitor progress in resident operative performance.
PURPOSE: An operative performance rating system for urology residents was developed for 6 sentinel urological procedures. We tested the reliability, validity and feasibility of the operative performance rating system for urology residents. MATERIALS AND METHODS: The operative performance rating system of each procedure contained a 3-point case difficulty scale, 4 to 6 procedure specific items, 3 general items and an overall performance item. A Likert scale of 1 to 5 was used for each item. A single video/audio record of each procedure was evaluated by the faculty. Single item interrater agreement was measured by comparing the observed variance and random measurement error variance. Resident operative performance evaluations were completed on line. Internal consistency reliability was measured using Cronbach α. Overall scale scores by resident training postgraduate year level were compared using 1-way ANOVA. RESULTS: Faculty evaluation of video/audio records showed an interrater agreement range of 0.71 to 0.92. Faculty evaluations of resident operative performance demonstrated an internal consistency reliability range of 0.91 to 0.95. Significant differences in overall scale scores between postgraduate year levels were noted for 3 of the 6 procedures (p ≤0.0016). CONCLUSIONS: An operative performance rating system for urology residents is feasible using an Internet based resident management system. Interrater agreement and internal consistency reliability meet threshold limits for checklist evaluation instruments. The operative performance rating system can discriminate among postgraduate year levels of resident training. A validated operative performance rating system can offer residents immediate, objective feedback on surgical performance and enable program directors to monitor progress in resident operative performance.