BACKGROUND AND PURPOSE: Global assessment by direct observation has been validated for evaluating operative performance of surgery residents after formal skills training but is time-consuming. The purpose of this study was to compare global assessment performed from edited videotape with scores from direct observation. MATERIALS AND METHODS: Junior surgery residents (N = 22) were randomized to 2 weeks of formal videotrainer skills training or a control group. Laparoscopic cholecystectomy was performed at the beginning and end of the rotation, and global assessment scores were compared for the training and control groups. Laparoscopic videotapes were edited: initial (2 minutes), cystic duct/artery (6 minutes), and fossa dissection (2 minutes). Two independent raters performed both direct observation and videotape assessments, and scores were compared for each rater and for interrater reliability using a Spearman correlation. RESULTS: Correlation coefficients for videotape versus direct observation for five global assessment criteria were <0.33 for both raters (NS for all values). The correlation coefficient for interrater reliability for the overall score was 0.57 (P = 0.01) for direct observation v 0.28 (NS) for videotape. The trained group had significantly better overall performance than the control group according to the assessment by direct observation (P = 0.02) but not by videotape assessment (NS). CONCLUSIONS: Direct observation demonstrated improved overall performance of junior residents after formal skills training on a videotrainer. Global assessment from an edited 10-minute videotape did not correlate with direct observation and had poor interrater reliability. Efficient and valid methods of evaluating operative performance await development.
RCT Entities:
BACKGROUND AND PURPOSE: Global assessment by direct observation has been validated for evaluating operative performance of surgery residents after formal skills training but is time-consuming. The purpose of this study was to compare global assessment performed from edited videotape with scores from direct observation. MATERIALS AND METHODS: Junior surgery residents (N = 22) were randomized to 2 weeks of formal videotrainer skills training or a control group. Laparoscopic cholecystectomy was performed at the beginning and end of the rotation, and global assessment scores were compared for the training and control groups. Laparoscopic videotapes were edited: initial (2 minutes), cystic duct/artery (6 minutes), and fossa dissection (2 minutes). Two independent raters performed both direct observation and videotape assessments, and scores were compared for each rater and for interrater reliability using a Spearman correlation. RESULTS: Correlation coefficients for videotape versus direct observation for five global assessment criteria were <0.33 for both raters (NS for all values). The correlation coefficient for interrater reliability for the overall score was 0.57 (P = 0.01) for direct observation v 0.28 (NS) for videotape. The trained group had significantly better overall performance than the control group according to the assessment by direct observation (P = 0.02) but not by videotape assessment (NS). CONCLUSIONS: Direct observation demonstrated improved overall performance of junior residents after formal skills training on a videotrainer. Global assessment from an edited 10-minute videotape did not correlate with direct observation and had poor interrater reliability. Efficient and valid methods of evaluating operative performance await development.
Authors: Sayra M Cristancho; Antony J Hodgson; O N M Panton; Adam Meneghetti; Garth Warnock; Karim Qayumi Journal: Surg Endosc Date: 2008-12-31 Impact factor: 4.584