The use of trained observers as an evaluation tool for a multi-hospital bioterrorism exercise.

INTRODUCTION: Evidence suggests that regular disaster exercises have beneficial effects on subsequent mock and actual disaster responses. The purpose of this report is to describe a multiple hospital, bioterrorism exercise, evaluated by independent observers who used an evaluation template.
METHODS: The overall tabletop exercise design included participation from 23 Joint Commission Accreditation of Healthcare Organizations hospitals, four health departments, and a representative from one federal agency. The exercise was evaluated by trained exercise observers utilizing an independently prepared, evaluation protocol.
RESULTS: All exercise sites successfully identified the bio-agent involved and answered after-exercise debriefing questions without much difficulty. Evaluations, in the form of an after-action report by the independent observers, commented upon the many limitations to the construct of the exercise.
CONCLUSION: Having an independent observer group at the exercise appeared to provide a value-added benefit for capturing subjective information and data. However, these data were not in a form conducive to statistical analysis. Further work is needed to create an evaluation tool that would allow for statistical analysis so that exercises can be compared and improvements can be objective.