Walter A Boyle1,2, David J Murray1,3, Mary Beth Beyatte1,2, Justin G Knittel1,2, Paul W Kerby1,2, Julie Woodhouse1,3, John R Boulet4. 1. Department of Anesthesiology, Washington University School of Medicine, St Louis, MO. 2. Division of Anesthesiology Critical Care Medicine, Department of Anesthesiology, Washington University School of Medicine, St. Louis, MO. 3. Wood Simulation Center, Washington University School of Medicine, St. Louis, MO. 4. Foundation for Advancement of International Medical Education and Research, Philadelphia, PA.
Abstract
OBJECTIVES: Develop a standardized simulation method to assess clinical skills of ICU providers. DESIGN: Simulation assessment. SETTING: Simulation laboratory. SUBJECTS: Residents, Critical Care Medicine fellows, acute care nurse practitioner students. INTERVENTIONS: Performance scoring in scenarios from multiple Critical Care Medicine competency domains. MEASUREMENTS AND MAIN RESULTS: Three-hundred eighty-four performances by 48 participants were scored using checklists (% correct) and holistic "global" ratings (1 [unprepared] to 9 [expert]). One-hundred eighty were scored by two raters. Mean checklist and global scores (± SD) ranged from 65.0% (± 16.3%) to 84.5% (± 17.3%) and 4.7 (± 1.4) to 7.2 (± 1.2). Checklist and global scores for Critical Care Medicine fellows and senior acute care nurse practitioner students (Experienced group, n = 26) were significantly higher than those for the Novice acute care nurse practitioner students (Novice group, n = 14) (75.6% ± 15.6% vs 68.8% ± 21.0% and 6.1 ± 1.6 vs 5.4 ± 1.5, respectively; p < 0.05). Residents (Intermediate group, n = 8) scored between the two (75.4% ± 18.3% and 5.7 ± 1.7). 38.5% of the Experienced group scored in the top quartile for mean global score, compared with 12.5% of the Intermediate and 7.1% of the Novice groups. Conversely, 50% of the Novice group scored in the lower quartile (< 5.3), compared with 37.5% of the Intermediate and 11.5% of the Experienced groups. Psychometric analyses yielded discrimination values greater than 0.3 for most scenarios and reliability for the eight-scenario assessments of 0.51 and 0.60, with interrater reliability of 0.71 and 0.75, for checklist and global scoring, respectively. CONCLUSIONS: The simulation assessments yielded reasonably reliable measures of Critical Care Medicine decision-making skills. Despite a wide range of performance, those with more ICU training and experience performed better, providing evidence to support the validity of the scores. Simulation-based assessments may ultimately prove useful to determine readiness to assume decision-making roles in the ICU.
OBJECTIVES: Develop a standardized simulation method to assess clinical skills of ICU providers. DESIGN: Simulation assessment. SETTING: Simulation laboratory. SUBJECTS: Residents, Critical Care Medicine fellows, acute care nurse practitioner students. INTERVENTIONS: Performance scoring in scenarios from multiple Critical Care Medicine competency domains. MEASUREMENTS AND MAIN RESULTS: Three-hundred eighty-four performances by 48 participants were scored using checklists (% correct) and holistic "global" ratings (1 [unprepared] to 9 [expert]). One-hundred eighty were scored by two raters. Mean checklist and global scores (± SD) ranged from 65.0% (± 16.3%) to 84.5% (± 17.3%) and 4.7 (± 1.4) to 7.2 (± 1.2). Checklist and global scores for Critical Care Medicine fellows and senior acute care nurse practitioner students (Experienced group, n = 26) were significantly higher than those for the Novice acute care nurse practitioner students (Novice group, n = 14) (75.6% ± 15.6% vs 68.8% ± 21.0% and 6.1 ± 1.6 vs 5.4 ± 1.5, respectively; p < 0.05). Residents (Intermediate group, n = 8) scored between the two (75.4% ± 18.3% and 5.7 ± 1.7). 38.5% of the Experienced group scored in the top quartile for mean global score, compared with 12.5% of the Intermediate and 7.1% of the Novice groups. Conversely, 50% of the Novice group scored in the lower quartile (< 5.3), compared with 37.5% of the Intermediate and 11.5% of the Experienced groups. Psychometric analyses yielded discrimination values greater than 0.3 for most scenarios and reliability for the eight-scenario assessments of 0.51 and 0.60, with interrater reliability of 0.71 and 0.75, for checklist and global scoring, respectively. CONCLUSIONS: The simulation assessments yielded reasonably reliable measures of Critical Care Medicine decision-making skills. Despite a wide range of performance, those with more ICU training and experience performed better, providing evidence to support the validity of the scores. Simulation-based assessments may ultimately prove useful to determine readiness to assume decision-making roles in the ICU.
Authors: Geoffrey K Lighthall; Juliana Barr; Steven K Howard; Eran Gellar; Yasser Sowb; Edward Bertacini; David Gaba Journal: Crit Care Med Date: 2003-10 Impact factor: 7.598
Authors: Derek C Angus; Andrew F Shorr; Alan White; Tony T Dremsizov; Robert J Schmitz; Mark A Kelley Journal: Crit Care Med Date: 2006-04 Impact factor: 7.598
Authors: John D Buckley; Doreen J Addrizzo-Harris; Alison S Clay; J Randall Curtis; Robert M Kotloff; Scott M Lorin; Susan Murin; Curtis N Sessler; Paul L Rogers; Mark J Rosen; Antoinette Spevetz; Talmadge E King; Atul Malhotra; Polly E Parsons Journal: Am J Respir Crit Care Med Date: 2009-08-15 Impact factor: 21.405
Authors: Diane B Wayne; Aashish Didwania; Joe Feinglass; Monica J Fudala; Jeffrey H Barsuk; William C McGaghie Journal: Chest Date: 2007-06-15 Impact factor: 9.410