OBJECTIVE: The objective of this study was to seek validity evidence for simulation-based assessments (SBA) of paramedics by asking to what extent the measurements obtained in SBA of clinical competence are associated with measurements obtained in actual paramedic contexts, with real patients. METHODS: This prospective observational study involved analyzing the assessment of paramedic trainees at the entry-to-practice level in both simulation- and workplace-based settings. The SBA followed an OSCE structure involving full clinical cases from initial patient contact to transport or transfer of care. The workplace-based assessment (WBA) involved rating samples of clinical performance during real clinical encounters while assigned to an emergency medical service. For each candidate, both assessments were completed during a 3-week period at the end of their training. Raters in the SBA and WBA settings used the same paramedic-specific seven-dimension global rating scale. Reliability was calculated and decision studies were completed using generalizability theory. Associations between settings (overall and by dimension) were calculated using Pearson's correlation. RESULTS: A total of 49 paramedic trainees were assessed using both a SBA and WBA. The mean score in the SBA and WBA settings were 4.88 (SD = 0.68) and 5.39 (SD = 0.48), respectively, out of a possible 7. Reliability for the SBA and WBA settings reached 0.55 and 0.49, respectively. A decision study revealed 10 and 13 cases would be needed to reach a reliability of 0.7 for the SBA and WBA settings. Pearson correlation reached 0.37 (p = 0.01) between settings, which rose to 0.73 when controlling for imperfect reliability; five of seven dimensions (situation awareness, history gathering, patient assessment, decision making, and communication) reaching significance. Two dimensions (resource utilization and procedural skills) did not reach significance. CONCLUSION: For five of the seven dimensions believed to represent the construct of paramedic clinical performance, scores obtained in the SBA were associated with scores obtained in real clinical contexts with real patients. As SBAs are often used to infer clinical competence and predict future clinical performance, this study contributes validity evidence to support these claims as long as the importance of sampling performance broadly and extensively is appreciated and implemented.
OBJECTIVE: The objective of this study was to seek validity evidence for simulation-based assessments (SBA) of paramedics by asking to what extent the measurements obtained in SBA of clinical competence are associated with measurements obtained in actual paramedic contexts, with real patients. METHODS: This prospective observational study involved analyzing the assessment of paramedic trainees at the entry-to-practice level in both simulation- and workplace-based settings. The SBA followed an OSCE structure involving full clinical cases from initial patient contact to transport or transfer of care. The workplace-based assessment (WBA) involved rating samples of clinical performance during real clinical encounters while assigned to an emergency medical service. For each candidate, both assessments were completed during a 3-week period at the end of their training. Raters in the SBA and WBA settings used the same paramedic-specific seven-dimension global rating scale. Reliability was calculated and decision studies were completed using generalizability theory. Associations between settings (overall and by dimension) were calculated using Pearson's correlation. RESULTS: A total of 49 paramedic trainees were assessed using both a SBA and WBA. The mean score in the SBA and WBA settings were 4.88 (SD = 0.68) and 5.39 (SD = 0.48), respectively, out of a possible 7. Reliability for the SBA and WBA settings reached 0.55 and 0.49, respectively. A decision study revealed 10 and 13 cases would be needed to reach a reliability of 0.7 for the SBA and WBA settings. Pearson correlation reached 0.37 (p = 0.01) between settings, which rose to 0.73 when controlling for imperfect reliability; five of seven dimensions (situation awareness, history gathering, patient assessment, decision making, and communication) reaching significance. Two dimensions (resource utilization and procedural skills) did not reach significance. CONCLUSION: For five of the seven dimensions believed to represent the construct of paramedic clinical performance, scores obtained in the SBA were associated with scores obtained in real clinical contexts with real patients. As SBAs are often used to infer clinical competence and predict future clinical performance, this study contributes validity evidence to support these claims as long as the importance of sampling performance broadly and extensively is appreciated and implemented.