BACKGROUND: Despite its multifaceted importance, no validated or reliable tools assess the quality of the dictated operative note. This study determined the construct validity, interrater reliability, and internal consistency of a Structured Assessment Format for Evaluating Operative Reports (SAFE-OR) in general surgery. METHODS: SAFE-OR was developed by using consensus criteria set forth by the Canadian Association of General Surgeons. This instrument includes a structured assessment and a global quality rating scale. Residents divided into novice and experienced groups viewed and dictated a videotaped laparoscopic sigmoid colectomy. Blinded, independent faculty evaluators graded the transcribed reports using SAFE-OR. RESULTS: Twenty-one residents participated in the study. Mean structured assessment scores (out of 44) were significantly lower for novice versus experienced residents (23.3 +/- 5.2 vs 34.1 +/- 6.0, t = .001). Mean global quality scores (out of 45) were similarly lower for novice residents (25.6 +/- 4.7 vs 35.9 +/- 7.6, t = .006). Interclass correlation coefficients were .98 (95% confidence interval, .96-.99) for structured assessment and .93 (95% confidence interval, .83-.97) for global quality scales. Cronbach alpha coefficients for internal consistency were .85 for structured assessment and .96 for global quality assessment scales. CONCLUSIONS: SAFE-OR shows significant construct validity, excellent interrater reliability, and high internal consistency. This tool will allow educators to objectively evaluate the quality of trainee operative reports and provide a mechanism for implementing, monitoring, and refining curriculum for dictation skills.
BACKGROUND: Despite its multifaceted importance, no validated or reliable tools assess the quality of the dictated operative note. This study determined the construct validity, interrater reliability, and internal consistency of a Structured Assessment Format for Evaluating Operative Reports (SAFE-OR) in general surgery. METHODS: SAFE-OR was developed by using consensus criteria set forth by the Canadian Association of General Surgeons. This instrument includes a structured assessment and a global quality rating scale. Residents divided into novice and experienced groups viewed and dictated a videotaped laparoscopic sigmoid colectomy. Blinded, independent faculty evaluators graded the transcribed reports using SAFE-OR. RESULTS: Twenty-one residents participated in the study. Mean structured assessment scores (out of 44) were significantly lower for novice versus experienced residents (23.3 +/- 5.2 vs 34.1 +/- 6.0, t = .001). Mean global quality scores (out of 45) were similarly lower for novice residents (25.6 +/- 4.7 vs 35.9 +/- 7.6, t = .006). Interclass correlation coefficients were .98 (95% confidence interval, .96-.99) for structured assessment and .93 (95% confidence interval, .83-.97) for global quality scales. Cronbach alpha coefficients for internal consistency were .85 for structured assessment and .96 for global quality assessment scales. CONCLUSIONS: SAFE-OR shows significant construct validity, excellent interrater reliability, and high internal consistency. This tool will allow educators to objectively evaluate the quality of trainee operative reports and provide a mechanism for implementing, monitoring, and refining curriculum for dictation skills.
Authors: Shannon E Stogryn; Krista Hardy; Michael J Mullan; Jason Park; Christopher Andrew; Ashley Vergis Journal: Surg Endosc Date: 2017-09-15 Impact factor: 4.584