BACKGROUND: Intraoperative anesthesia equipment failures are a cause of anesthetic morbidity. Our purpose in this study was 1) to design a set of simulated scenarios that measure skill in managing intraoperative equipment-related errors and 2) to evaluate the reliability and validity of the measures from this multiple scenario assessment. METHODS: Eight intraoperative scenarios were created to test anesthesia residents' skills in managing a number of equipment-related failures. Fifty-six resident physicians, divided into four groups based on their training year (Resident 1-Resident 4), participated in the individual simulation-based assessment of equipment-related failures. The score for each scenario was generated by a checklist of key actions relevant to each scenario and time to complete these actions. RESULTS: The residents' scores, on average, improved with increased level of training. The more senior residents (R3 and R4) performed better than more junior residents (R1 and R2). Despite similar training background, there was a wide range of skill among the residents within each training year. The summary score on the eight scenario assessments, measured by either the key actions or the time required to manage the events, yielded a reliable estimate of a resident's skill in managing these simulated equipment failures. DISCUSSION: Anesthesia residents' performances could be reliably evaluated using a set of simulated intraoperative equipment problems. This multiple scenario assessment was an effective method to evaluate individual performance. The summary results, by training year, could be used to determine how successful current instructional methods are for acquiring skill.
BACKGROUND: Intraoperative anesthesia equipment failures are a cause of anesthetic morbidity. Our purpose in this study was 1) to design a set of simulated scenarios that measure skill in managing intraoperative equipment-related errors and 2) to evaluate the reliability and validity of the measures from this multiple scenario assessment. METHODS: Eight intraoperative scenarios were created to test anesthesia residents' skills in managing a number of equipment-related failures. Fifty-six resident physicians, divided into four groups based on their training year (Resident 1-Resident 4), participated in the individual simulation-based assessment of equipment-related failures. The score for each scenario was generated by a checklist of key actions relevant to each scenario and time to complete these actions. RESULTS: The residents' scores, on average, improved with increased level of training. The more senior residents (R3 and R4) performed better than more junior residents (R1 and R2). Despite similar training background, there was a wide range of skill among the residents within each training year. The summary score on the eight scenario assessments, measured by either the key actions or the time required to manage the events, yielded a reliable estimate of a resident's skill in managing these simulated equipment failures. DISCUSSION: Anesthesia residents' performances could be reliably evaluated using a set of simulated intraoperative equipment problems. This multiple scenario assessment was an effective method to evaluate individual performance. The summary results, by training year, could be used to determine how successful current instructional methods are for acquiring skill.