Outcome measures for surgical simulators: is the focus on technical skills the best approach?

BACKGROUND: Mastery of operative performance is based on technical skill and intra-operative judgment. However, previous simulation studies have largely focused on technical skills and measures. This study investigates changes in operative performance when assessment and feedback focus on decision making.
METHODS: Using a nonequivalent, pretest/post-test experimental design, 8 senior residents (PGY4-5) performed a laparoscopic ventral hernia repair using a newly developed box-trainer style simulator fabricated to induce surgical decision making. The pretest simulator had a 10 x 10-cm defect 5 cm above the umbilicus. The post-test simulator had a 10 x 10-cm defect in the right upper quadrant. After the pretest, faculty provided immediate feedback on operative decisions that lead to errors. In addition, residents were allowed to visually inspect their repair by removing the box trainer skins. Video-analysis using a 9-item decision making checklist was used to categorize pretest and post-test error differences.
RESULTS: Common errors made during the pretest included improper visualization of the suture passer and improper mesh preparation on the back table. These errors resulted in incomplete hernia repairs by 75% of residents on the pretest. In contrast, 100% of residents successfully completed the more difficult, nonequivalent post-test hernia. Checklist analysis showed residents committed more errors on the pretest resulting in lower performance scores (score = 48.12; SD = 19.26) compared with post-test performance (score = 75.00; SD = 14.39; P < .05).
CONCLUSION: Residents' decision-making skills seem to significantly affect operative performance. To facilitate mastery of operative performance, additional research is needed on simulation-based, operative skills measures that focus on intra-operative decision making. Copyright 2010 Mosby, Inc. All rights reserved.