Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments.

OBJECTIVES: To determine feasibility and reliability of skills assessment in a multi-agency, triple-site major incident response exercise carried out in a virtual world environment.
METHODS: Skills assessment was carried out across three scenarios. The pre-hospital scenario required paramedics to triage and treat casualties at the site of an explosion. Technical skills assessment forms were developed using training syllabus competencies and national guidelines identified by pre-hospital response experts. Non-technical skills were assessed using a seven-point scale previously developed for use by pre-hospital paramedics. The two in-hospital scenarios, focusing on a trauma team leader and a silver/clinical major incident co-ordinator, utilised the validated Trauma-NOTECHS scale to assess five domains of performance. Technical competencies were assessed using an ATLS-style competency scale for the trauma scenario. For the silver scenario, the assessment document was developed using competencies described from a similar role description in a real-life hospital major incident plan. The technical and non-technical performance of all participants was assessed live by two experts in each of the three scenarios and inter-assessor reliability was computed. Participants also self-assessed their performance using identical proformas immediately after the scenarios were completed. Self and expert assessments were correlated (assessment cross-validation).
RESULTS: Twenty-three participants underwent all scenarios and assessments. Performance assessments were feasible for both experts as well as the participants. Non-technical performance was generally scored higher than technical performance. Very good inter-rater reliability was obtained between expert raters across all scenarios and both technical and non-technical aspects of performance (reliability range 0.59-0.90, Ps<0.01). Significant positive correlations were found between self and expert assessment in technical skills across all three scenarios (correlation range 0.52-0.84, Ps<0.05), although no such correlations were observed in non-technical skills.
CONCLUSIONS: This study establishes feasibility and reliability of virtual environment technical and non-technical skills assessment in major incident scenarios for the first time. The development for further scenarios and validated assessment scales will enable major incident planners to utilise virtual technologies for improved major incident preparation and training.