Jan Schmutz1, Tanja Manser2, Julia Keil3, Ellen Heimberg4, Florian Hoffmann3. 1. Department of Management, Technology, and Economics, ETH Zurich, Zurich, Switzerland. Electronic address: jschmutz@ethz.ch. 2. Institute for Patient Safety, University Hospital Bonn, Bonn, Germany. 3. Dr. von Hauner University Children's Hospital, Munich, Germany. 4. Department of Pediatric Cardiology, Pulmology and Pediatric Intensive Care Medicine, University Children's Hospital, Tübingen, Germany.
Abstract
OBJECTIVE: To develop and validate 3 performance evaluation checklists (PECs) for systematic performance assessment in 3 clinical scenarios: cardiopulmonary arrest, dyspnea with oxygen desaturation after intubation, and respiratory syncytial virus (RSV). STUDY DESIGN: The 3 PECs were developed using an integrative approach and used to rate 50 training sessions in a simulator environment by different raters. Construct validity was tested by correlating the checklist scores with external constructs (ie, global rating, team experience level, and time to action). Further interrater reliability was tested for all 3 PECs. RESULTS: The PECs for the desaturation and cardiopulmonary arrest scenarios were valid and reliable, whereas the PEC for RSV had limited validity and reliability. CONCLUSION: For 2 pediatric emergencies, the PEC is a valid and reliable tool for systematic performance assessment. The unsatisfactory results for the PEC for RSV may be related to limitations of the simulation setting and require further investigation. Structured assessment of clinical performance can augment feedback on technical performance aspects and is essential for training purposes as well as for research. Only reliable and valid performance measures will allow medical educators to accurately evaluate the behavioral effects of training interventions and further enhance the quality of patient care.
OBJECTIVE: To develop and validate 3 performance evaluation checklists (PECs) for systematic performance assessment in 3 clinical scenarios: cardiopulmonary arrest, dyspnea with oxygen desaturation after intubation, and respiratory syncytial virus (RSV). STUDY DESIGN: The 3 PECs were developed using an integrative approach and used to rate 50 training sessions in a simulator environment by different raters. Construct validity was tested by correlating the checklist scores with external constructs (ie, global rating, team experience level, and time to action). Further interrater reliability was tested for all 3 PECs. RESULTS: The PECs for the desaturation and cardiopulmonary arrest scenarios were valid and reliable, whereas the PEC for RSV had limited validity and reliability. CONCLUSION: For 2 pediatric emergencies, the PEC is a valid and reliable tool for systematic performance assessment. The unsatisfactory results for the PEC for RSV may be related to limitations of the simulation setting and require further investigation. Structured assessment of clinical performance can augment feedback on technical performance aspects and is essential for training purposes as well as for research. Only reliable and valid performance measures will allow medical educators to accurately evaluate the behavioral effects of training interventions and further enhance the quality of patient care.