BACKGROUND: The Accreditation Council for Graduate Medical Education common program requirements mandate a competency-based assignment of duties. To accomplish this, valid and reliable assessment tools must be developed to evaluate competence. This study evaluated a rating tool to assess competence in basic surgical suturing skills. METHODS: A technical skills exercise consisting of the closure of three incisions, 3 cm long, was devised in 2006. The incisions were closed with simple stitches with two-handed knots, vertical mattress stitches with instrument knots, and a running stitch with one-handed knots. Fifteen min were allotted for completion. A rating instrument with 17 competency markers worth 1 point and a global 5-point Likert scale competency score was used to evaluate the performance. Twelve first-week post graduate year 1 surgical residents completed the exercise in 2006, and 16 final-month post graduate year 1 surgical residents completed it in 2011. All tasks were scored on video review by two independent raters. Statistical analysis included descriptive statistics, t-score analysis, rank sum analysis, Cohen's kappa coefficient, and Cronbach's alpha. RESULTS: The mean total score (11.8 versus 13.9, P = 0.002) and median global competency rating (1 versus 3, P < 0.001) were lower for the first-week cohort. Cohen's kappa coefficient of inter-rater reliability was 0.77. Cronbach's alpha measure of internal consistency was 0.87. CONCLUSION: This rating form is a valuable tool to evaluate technical skill competency. Construct validity was demonstrated with improvement in total score and global rating. Excellent internal consistency and inter-rater reliability were also demonstrated. This form may be used to assess technical skill competency in an efficient skills exercise.
BACKGROUND: The Accreditation Council for Graduate Medical Education common program requirements mandate a competency-based assignment of duties. To accomplish this, valid and reliable assessment tools must be developed to evaluate competence. This study evaluated a rating tool to assess competence in basic surgical suturing skills. METHODS: A technical skills exercise consisting of the closure of three incisions, 3 cm long, was devised in 2006. The incisions were closed with simple stitches with two-handed knots, vertical mattress stitches with instrument knots, and a running stitch with one-handed knots. Fifteen min were allotted for completion. A rating instrument with 17 competency markers worth 1 point and a global 5-point Likert scale competency score was used to evaluate the performance. Twelve first-week post graduate year 1 surgical residents completed the exercise in 2006, and 16 final-month post graduate year 1 surgical residents completed it in 2011. All tasks were scored on video review by two independent raters. Statistical analysis included descriptive statistics, t-score analysis, rank sum analysis, Cohen's kappa coefficient, and Cronbach's alpha. RESULTS: The mean total score (11.8 versus 13.9, P = 0.002) and median global competency rating (1 versus 3, P < 0.001) were lower for the first-week cohort. Cohen's kappa coefficient of inter-rater reliability was 0.77. Cronbach's alpha measure of internal consistency was 0.87. CONCLUSION: This rating form is a valuable tool to evaluate technical skill competency. Construct validity was demonstrated with improvement in total score and global rating. Excellent internal consistency and inter-rater reliability were also demonstrated. This form may be used to assess technical skill competency in an efficient skills exercise.