A qualitative content analysis for determining indexes and factors affecting for evaluation of disaster exercises immediate feedback stage.

BACKGROUND: Taking exercise in health sector is one of the important steps to implement the disaster risk management programs, especially preparedness phase. The present study aimed to identify indexes and factors affecting successful evaluation of disasters preparedness exercises in hot wash stage.
MATERIALS AND METHODS: This study was a qualitative content analysis. Data were collected by purposeful sampling through in-depth and semi-structured individual interviews with 25 health professionals in the field of disasters. The data were analyzed using directed content analysis method by which the initial codes were extracted after transcribing the recorded interviews and immersing them in the data analysis. The initial codes were reviewed, classified, and subdivided into several stages to determine the main classes.
RESULTS: The data analysis resulted in the production of 24 initial codes, 5 subcategories, 2 main categories of "evaluation and exercise debriefing" and "modification of programs and promotion of exercise operational functions" under the original theme of "exercise immediate feedback."
CONCLUSION: This study can be considered a suitable standard guide for health care organizations to evaluate successfully disasters exercises in hot wash stage, maintain and promote their preparedness, and properly respond to disasters. Copyright:

Introduction

Disaster risk management requires a systematic process, including executive, organizational decisions, other capacities to perform policies, strategies and society compatibility and capacity to reduce the negative effects and consequences of hazards.[12] Disaster risk management consists of four phases: Mitigation and prevention, preparedness, response, and recovery.[3] Preparedness is actions which take place beforehand to make sure of effective response.[45] Among several components of disaster management, the health centers, and affiliated units can reduce physical, financial, and social damage caused by disasters through providing the preparedness plans and appropriate strategies.[67] Revision and improvement of health centers' preparedness plans for the proper and timely response is a major role for reducing damages caused by disasters. Otherwise, carrying out disaster exercises is the most important way to create, maintain, and improve preparedness plans.[78]

Running exercise courses in different sectors of health system are one of the important steps to prepare and deploy disaster risk management programs, especially response phase.[9] Exercises simulate the realistic conditions, so that people improve their mental and physical skills in situations similar to real conditions and provide an appropriate response to disasters.[10] Disaster exercises can be used for testing and validating policies, programs, procedures, teaching personnel their roles and responsibilities, as well as improving the individual performance, and improving inter-organizational communication and coordination.[11] There are two types of disaster exercises: (a) discussion-based exercise and (b) operation-based exercise. Discussion-based exercise includes four types of exercises: Seminars, workshops, tabletop, and games; and operation-based exercise consists of three types of exercises, including drills, functional-scale, and full-scale exercises.[9]

Skryabina et al. argued that operational exercises are activities aimed at training and practicing empowerment, identifying core competencies in prevention and mitigation, reducing vulnerability, responding, and rehabilitating in a risk-free environment for participants.[12] Studies done on the evaluation of hospitals disaster preparedness suggest that the exercise evaluation indicators should be selected during the design phase of the exercise, and they should be based on the efficiency and performance expected from each of the different sections.[51314]

In after-action stage, evaluators should investigate exercises during a session with all key participants and delegates of involved units. This session aims to collect information on individual performance and apply them to revise and improve the preparedness program and response process.[12] It should be noted that identification of unskilled and unprofessional staff and finding their fault, error, and poor performance are not among the objectives of the session, and programs will be reviewed and modified based on the available information and results of the evaluation at the end of the session.[611]

Therefore, considering the importance of promoting health system preparedness through exercise, the present study aimed to identify indexes and factors affecting successful evaluation of disasters exercises in the after-action stage. The findings of this study can be considered a suitable standard guide for health care organizations to evaluate successfully disasters exercises in the hot wash stage, maintain and promote their preparedness, and properly respond to disasters.

Materials and Methods

Study design

This qualitative study was done in 2020, and content analysis approach[9] was employed for conducting the study.

Setting, participants, and data collection

This qualitative study was conducted in Iran, one of the most disaster-prone countries in the world. The study population included 25 experts in disasters who had practical experience or theoretical knowledge about designing, implementation, and evaluation of preparedness exercises in disasters and had been present at least once in operational-based or discussion-based exercises. Participants were chosen using a purposeful sampling method with maximum diversity. Sampling was carried out until data saturation occurred, i.e., when the researcher concluded that further interview would fail to provide new information. Participants included seven prehospital directors, four hospital directors, two nursing experts in hospital emergencies committee, four experts in Emergency Operation Centers (EOC) in the University of Medical Science, six health experts in disaster risk reduction, and two deputies of the logistic in the University of Medical Science.

The interviews conducted face-to-face and individually. The interviewees answered to a similar set of questions, which began with “have you ever evaluated preparedness exercises of the health system in the hot wash stage?,” “Describe the worst and best evaluations that you have experienced in the hot wash stage?” “Based on your experience, what components and features should be considered in the hot wash stage of discussion-based exercise or operational-based exercise? Based on the above guide, additional questions were raised during the interview and when authors found new concepts. Moreover, who, when, why, and how were used for concept saturation as well as “Could you please give an example” or “Please explain more” for data and concept saturation. The interviews were taped and lasted 25–90 min. The place and time of the interview were selected by agreement between the interviewer and the interviewee, field notes were written during interviews to describe and interpret the responses correctly.

Reliability and validity

This study employed strategies recommended by Lincoln and Guba[15] for reliability and validity tests. According to this recommendation, four criteria of creditability, dependency, conformability, and transferability are required to ensure reliability. Credibility was ensured through assignment of sufficient time for data collection and analysis, prolonged engagement with the participants, constant comparison of participants' expressions, understanding of their experiences by the researcher and maximum variety of the participants. Conformability was achieved by member check, peer check, and expert check. Member check was done by returning the text of interview and summary of results to four participants for confirmation of the findings. Two qualitative researchers in the research team (expert check and peer check) checked the validity of data collection and analysis process.

Ethical consideration and consent to participate

Informed consent was obtained orally and in writing before the interview after explaining the aim and process of the study. In addition, participants were informed about the purpose of the study, the interview method, confidentiality of their information, and the right to withdraw from the study at any time.

Data analysis

The data were analyzed using the conventional content analysis method. Systematic stages were followed, and simultaneous analysis was undertaken: First, recorded interviews were transcribed verbatim. Then, prior to coding, the transcribed text was read several times for familiarization. In the initial coding process, the participants' words were used and condense meaning units were formed; and then the codes were categorized into subcategories based on their similarities and differences. This process continued for all interviews until the formation of the main categories.

Results

Demographic information of participants

The participants included 4 females and 21 males with a mean age of 42.25 ± 4.8 years ranging from 25 to over 45 years. The mean duration of work experience was 15.5 ± 3.4 years, and all participants had more than 5 years of work experience in designing preparedness exercises of the health system in disasters.

Main results

An original theme of the exercise immediate feedback, 2 main categories, 5 subcategories and 24 codes were formed: Evaluation and exercise debriefing (with tree subcategories of formation of evaluation sessions, identification of strengths and weaknesses, and documentation and provision of exercise results), modification of programs and promotion of exercise operational functions (with two subcategories of planning for improvement, and employees' persuasion and encouragement) Table 1. The main theme, categories, and subcategories are described in the following sections.

Table 1

The Categories, sub-categories of components affecting for hot wash of health system preparedness exercises in disaster

Main theme	Category	Sub-category	Example of codes
Exercise immediate feedback	Evaluation and exercise debriefing	Formation of evaluation sessions	Holding the hot wash session within 30 min after the postexercise time in the operations management center of the executive organization
		Identify of strengths and weaknesses	Implementation of the debriefing program on time in accordance with the time sequence program of the scenario event with the presence of the members of the exercise evaluation group
		Documentation and providing exercise results	Preparing the after-action report including the reports of the managers, the participants, and partner and support organizations and the exercise planning group
			Reviewing the executive challenges of the incident operation plan by the managers of the units participating in the exercise
	Modifying of programs and promotion of exercise operational functions	Planning for improvement	Making required planning to hold the debriefing session one week after completing the exercise
			Employees persuasion and encouragement
			Identifying the strengths and weaknesses of the action response plan and prioritizing the
			solutions proposed for improvement
			Dividing the tasks between the units participating in the exercise to implement the
			proposed solutions
			Determining the funds required for implementing the corrective activities of the (IAP)
			Determining a number of professionals to monitor the correct implementation of corrective activities
			Making necessary planning for reviewing the IAP and correcting
			the incident response processes based on the results of the evaluations
			Appreciating all participants in the exercise after completion
			Appreciating all participants in the exercise after completion

IAP=Incident action plan

Main theme

Exercise immediate feedback

A hot wash as the ultimate goal of exercise aims to prepare the health sector for an effective, accurate, and timely response to real disasters. It should be noted that a hot wash program should not only be limited to the postexercise time, rather it can also contribute to developing the exercise plan and improving it in the future. Similar to any other operational program, for the hot wash program and its corrective measures to be developed and implemented, the person in charge of implementing each action, the time frame and facilities required for implementing the programs, the procedure taken, and the progress actions must be continuously assessed and monitored.

Main categories and subcategories

Evaluation and exercise debriefing

Disaster specialists in Iran believe that representatives of all units inside and outside the organization must participate in the after-action session at the EOC to critically review and discuss the Incident Action Plan (IAP), identify the strengths and weaknesses of the response plan, and to develop effective measures to eliminate the weaknesses of the plan. This category covers three subcategories, including “organizing evaluation meetings,” “identifying strengths and weaknesses,” and “documenting and providing exercise results.”

Formation of evaluation sessions

Health professionals consider the after-action evaluation sessions the core of operation-and discussion-based exercises and believe that failure to provide immediate feedback means the noncompletion of the exercise. They also suggest that the best time to control the exercise and identify its strengths and weaknesses is the hot wash phase.

“One of the things that is often absent in exercise sessions held in Iran is the debriefing session, which is usually not done according to a given standard. Therefore, this session, one of the main components of a hot wash session, must be organized in the form of a brainstorming meeting. The controller and the exercise evaluators are important people who should attend the debriefing session. The controller must encourage everyone attending the meeting to comment and discuss the performance of the exercise” (Participant#4).

“A hot wash session depends on the type of exercise and the organizations involved in the exercise. However, the session must be held when the exercise is still fresh in memory. Another important session is the cold debriefing session that has to be held by health organizations with the presence of the same people attending the hot wash session to assess the attainment of goals after the exercise, and preferably within one week after the exercise,” (Participant # 10).

Identifying strengths and weaknesses

The most important part of evaluating health exercises is the identification of strengths and weakness of the operational response plan in evaluation sessions.

“At a hot wash session (called also the brainstorming meeting), the exercise evaluators who were selected professionally in the design stage provide a detailed and highly precise report of the performance of organizational units. However, the most important thing is to select a sufficient number of evaluators by taking into account the level and type of exercise, so that we can properly evaluate all the functions of an organization. It is best to first report the positive functions of organizational units in the exercise and then identify and report their poor functions” (Participant #6).

According to one of the specialists:

“It is perhaps better to use newer techniques to identify the strengths and weaknesses of the exercise along with paper tools. I think we need to use new technologies such as using drones or installing cameras on the clothes of evaluators and all exercise staff, including hypothetical victims. This needs infrastructure and research in this field, which can be effective in real incidents for tracking victims and injured persons in disasters, I mean it is applicable in the triage zone for people who get red or green tags” (Participant #17).

Documenting and providing exercise results

One of the important functions of the hot wash session is to record all the exercise documents from the design stage to the immediate debriefing stage or the recovery phase of disaster exercises.

“In my opinion, what is not recorded is not done, so it is very important to record all the exercise documents, including the after-action report, the evaluators' reports, the reports of the immediate debriefing session both electronically and on paper as the learned lessons of the exercise and to send a copy to all the units participating in the exercise” (Participant #3).

“It is necessary to correctly document the exercise procedure, to record the learned lessons, determine our priorities for future exercises, and eliminate the weaknesses of the IAP of the incident” (Participant #16).

Modification of programs and promotion of exercise operational functions

Another important main category emerged in this study is modifying programs and improving the operational functions of the exercise. This category was subdivided into “planning for improvement” and “employees' encouragement.”

Planning for improvement

The improvement planning process includes methods that turn the recommendations and suggested points in the after-action report document into measurable and controllable stages so that the implementation of the recommended cases in the form of a continuous quality improvement cycle leads to the development and improvement of response capabilities.

Concerning the necessary measures for planning the improvement, the specialists stated:

“First, after identifying the strengths and weaknesses of the exercise, some solutions should be devised in the hot wash session by representatives of the organizational units participating in the exercise. Moreover, the organizational resources, including structural and human resources must be determined, so that we can revise the IAP accordingly” (Participant #9).

“It is very important to determine a person or unit to record solutions and design a time sequence program to implement the steps and improve the poor functions of the in IAP. In addition, an organizational unit must be assigned to follow up the proposed actions” (Participant #7).

Employee's persuasion and encouragement

According to health professionals, encouraging employees is one of the most important issues that is ignored in most health exercises. Therefore, it is necessary to consider financial or spiritual incentives in the organizational funding line to increase employees' motivation.

“In my opinion, we have to allocate financial incentives to employees participating in the exercises. Those employees who did their best and the organizational units that followed and implemented the after-action review strategies should receive special rewards. Furthermore, the units that did not follow these proposed solutions should be punished accordingly” (Participant #20).

“We don't see the motivation used for implementing the exercise phase in the after-action phase anymore, and employees are reluctant in this regard. I think we need to use a series of incentive and motivational tools and consider items for stick holders, so that they continue the work up to the final stage with the same motivation shown in the initial stage of exercise” (Participant #1).

Discussion

The present study aimed to identify indexes and factors affecting successful evaluation of disasters preparedness exercises in the hot wash stage. The main components affecting the hot wash of preparedness exercises in the health system are exercise evaluation and debriefing and modification of programs and promotion of operational exercise functions.

The evaluation and debriefing of the exercise were one of the main categories emerged for the analysis of the interviews with the health experts in the field of disasters. Various studies have also emphasized the implementation of the hot wash programs and reported that if this step is not taken, the exercise will not be completely performed.[1617] Health experts in the field of disasters believe that the best time to control the exercise and determine the strengths and weaknesses of the exercise session is the hot wash sessions. Various important issues should be considered at this stage of the exercise, the most important of which are as follows: A hot wash session in the EOC within 30 min after completing the exercise, the presence of representatives of all units inside and outside the organization, the reporting or debriefing program, preparation of the after-action report, provision of exercise critique forms to receive critiques and suggestions from exercise participants, documentation of all exercise stages, and review of the IAP of the disaster. This finding is consistent with the results of previous studies.[181920] Innis and Mack declare that disaster simulation exercises provide a unique opportunity for nurses to experience interprofessional collaboration.[21] These studies showed that evaluating the exercise was an important and extensive process and covered many different functions and activities, including hot and cold debriefing, after-action reports, the review of the exercise by the actors, and immediate documentation of the exercise.

The modification of programs and promotion of operational exercise functions was another main category emerged in this study. According to the interviewees, the most important actions that complete the first dimension of exercise evaluation include: (a) providing effective and constructive measures to address the weaknesses of the organizational IAP in both human resources and action programs, reinforcing positive functions of the exercise by creating mechanisms to positively reinforce the behavior of human resources (b) developing a time sequence program to implement the actions and solutions planned in the hot wash session, (c) assigning an organizational unit to control and follow the planned actions for improvable points of the operational response program in the organization, (d) making necessary planning to review the IAP of the disaster, and (e) documenting the exercise to record the lessons learned and consider priorities to start a new exercise program and to gain skills for removing the weaknesses discovered from previous exercises. These findings were consistent with results of previous studies.[22242526] These studies confirmed that improving one's abilities through exercise was a continuous process that began before exercise and continued until after the recovery phase and that the exercise planning team needed to continuously control and monitor the activities determined in the hot wash session to reach the final achievements and make effective changes to improve the organization performance. In addition, the suggestions offered in the exercise evaluation sessions must be taken into account for obtaining, maintaining, and promoting the organizational abilities.

The planning for improvement was another subcategory identified in this study. This finding is consistent with the results of previous studies.[26272829] Beerens et al. in a study with title How Can we make disaster management evaluations more useful? An empirical study of Dutch exercise evaluations declared the usefulness of an evaluation intended for learning purposes is improved when its analysis and conclusions are clearer. In contrast, evaluations used for accountability purposes are only improved by the clarity of the conclusion.[30] Therefore, the planning process for promoting the exercise functions include the techniques that turn the recommendations and suggested points in the after-action report document into measurable and controllable stages so that the implementation of the recommended cases in the form of a continuous quality improvement cycle leads to the development and improvement of response capabilities.

Limitations and recommendation

The strength of this study is the diversity of participants from different sections of the health system in a different university of medical sciences in Iran. One of the limitations of the qualitative study was the bias in the analysis and interpretation of the results that maximized the consistency and accuracy of the study by using strategies such as checking the qualitative data at various stages of the analysis with the selected participants and co-researchers. In addition, comparing quantitative studies and the low number of participants may be another limitation; however, rich and well-saturated information from participants could overcome this problem.

Conclusion

One of the main challenges of evaluating health preparedness exercises in Iran is that the officials and executors ignore the exercise hot wash and are satisfied with just doing the exercise. Moreover, they do not perform any exercise standard actions and requirements in accordance with national and international documents. Therefore, it is very valuable that representatives of all units inside and outside the organization participating in the exercise hold a hot wash session immediately after the exercise at the EOC of the organization to critically review and discuss the operational response plan, identify the strengths and weaknesses of the plan, and develop effective measures to eliminate the weaknesses of the plan because the exercise is a good way to test our abilities and shortcomings and it will be more effective if, in addition to designing and implementing the exercise, the hot wash phase as an important step following the exercise is implemented in accordance with scientific principles and upstream documents.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.