Empirical research on the utility of a preparation manual for a disaster medical response drill.

PURPOSE: It is difficult for emergency physicians to plan and execute a disaster medical response drill while conducting their daily work activities. Readily available drill preparation manuals are therefore essential, alongside assessment methods to ensure quality. Here, we propose email text analysis as a manual assessment method, and investigate its validity.
METHODS: The preparation status of two similar large-scale disaster medical response drills were compared. All email texts exchanged during the preparation stage were analyzed, and frequently appearing words (quality element) and word counts (quantity element) were compared between Drill 1, which was organized without a manual, and Drill 2, organized with a manual.
RESULTS: Word frequency analysis revealed that the key components of the manual (visualization of necessary work, preparation of documents in a certain format, and clarification of aims of the drill) contributed to the effectiveness of the preparation process for Drill 2. Furthermore, work volume during the preparation for Drill 2 was decreased by 41.9% from that during the preparation for Drill 1.
CONCLUSION: Preparation of a high-quality manual is crucial so that emergency physicians can plan and execute a disaster medical response drill. Email text analysis can serve as an objective method assessing the quality of manuals.

Introduction

Fostering experts with accumulated experience in disaster medicine is difficult1,2 and highlights the importance of disaster medical response training. However, emergency physicians cannot easily attend such drills in daily clinical practice. In particular, it is a great burden for emergency physicians to participate in the planning and operation of disaster medical response drills3–8 because the details of training phases and scenes are so varied and different types of training must be thoroughly understood, including actual drills, map-based drills, and tabletop drills. Nevertheless, because of the close relationship between emergency medicine and disaster medicine, emergency physicians have a central role to play in the planning and implementation of disaster drills. Thus, readily available drill preparation manuals are essential. If a system allowing emergency physicians to have and view a manual for a specific drill is available, a considerable volume of unnecessary work will be avoided while preparing for the drill. We previously organized a disaster medical response drill for our medical district in Tokyo Metropolis and realized that the preparation process was extremely cumbersome. Therefore, we developed a drill manual and passed it on to the next organizers.

The validity of manuals is generally assessed based on an analysis of users’ opinions obtained through interviews or questionnaires. However, these approaches are not highly objective. To provide high-quality manuals, development of a new method which assesses manual contents in a more objective manner is needed.

To this end, we propose email text analysis as a new objective method for the assessment of manuals. In this study, we employed email text analysis to assess a drill manual which we developed, and then evaluated the validity of email text analysis for the objective assessment of manuals.

Material and methods

Tokyo disaster medical response simulation drill

After the Great East Japan Earthquake, Tokyo Metropolis established a disaster medical response system in preparation for a large-scale disaster and has continued to improve its systems in order to ensure smooth medical operations in the event of a disaster. As part of the system, the Tokyo Disaster Medical Response Simulation Drill started in 2013. Twelve medical districts in Tokyo Metropolis participate in the drill in turns. As of November 2016, the drill has been implemented in ten of the 12 medical districts. This is a simulation drill aimed at evaluating and improving the functions of the medical information and communications system.

The Disaster Countermeasures Basic Act of Japan states that, at the time of a disaster, the Office of Disaster Response and Emergency Management (hereinafter, the Office) will be set up in prefectures and municipalities, with the head Office and local Offices forming a hierarchical structure, as shown in Figures 1 and 2. The general role of these Offices will be to collect disaster-related information and implement disaster emergency measures.9–11 Disaster medical services are also provided in line with this basic framework, and disaster-related information is transferred from the Offices supervising small local areas to Offices supervising wider areas and then to the head Office governing the entire area. In turn, the head Office disseminates information gathered from the entire area and specific instructions to local Offices. In this hierarchical system, the Offices at each level must provide the right information to each Office that needs it. For this, information must be collected, screened, and disseminated promptly. The functions of the Offices of Disaster Response and Emergency Management are shown in Figure 2.9,12–14

Figure 1

Hierarchical structure formed by the Offices of Disaster Response and Emergency Management.

Figure 2

Functions of the Offices of Disaster Response and Emergency Management.

In Tokyo Metropolis, the Offices form a three-layer structure composed of 1) those in individual municipalities, 2) those in the 12 medical districts, and 3) one in the Tokyo Metropolitan Government Building. Tokyo Metropolis sponsors and implements the Tokyo Disaster Medical Response Simulation Drill in each of the 12 medical districts to evaluate information exchange between the Offices.

Planning of the disaster medical response simulation drill

Tokyo Metropolis sponsors the Tokyo Disaster Medical Response Simulation Drill, but commissions the planning and implementation of the drill to regional hospitals designated as core disaster medicine centers and where regional disaster medical/health coordinators for the medical district work. The coordinators play a central role in planning and implementing the drill, in cooperation with hospital staff and Tokyo Metropolitan Government staff who are assigned to the drill. All regional disaster medical/health coordinators are emergency physicians who play an active role in the community and have been appointed to serve as coordinators by Tokyo Metropolis. The procedures followed were in accordance with the ethical standards of our institutional committee and with the Helsinki Declaration, as revised in 2004.

The first drill (Drill 1) conducted in the Minami-Tama medical district was planned and organized mostly by the authors, who were appointed as regional disaster medical/health coordinators. Many staff, such as drill instructors and simulation coordinators (collectively referred to as drill staff) were deployed to simulate actual disaster conditions and increase the realism of the disaster experience for the participants (hereinafter, players).15,16 On the day of the drill, Offices were set up for Tokyo Metropolis, each medical district, and each municipality in accordance with the Disaster Countermeasures Basic Act of Japan (Figure 3). Then, players were assigned a profession to play, such as administrator or health care professional. When the drill began, simulation coordinators provided players with a heavy, continuous stream of previously prepared information regarding the disaster and patients. Each Office processed information as it came in and arranged for information dissemination and patient transfer.

Figure 3

Deployment of different disaster drills.

The drill was retrospectively evaluated and verified. Specifically, the data obtained during the drill were objectively evaluated by government research staff at a meeting held on a different day from the drill. Based on the verification results, Tokyo Metropolis is currently improving the disaster medical response system.

Creating the disaster drill manual

We previously organized a disaster medical response drill for our medical district in Tokyo Metropolis and realized that the preparation process was extremely cumbersome, involving many complex and detailed activities, including planning the details of the drill, arranging the drill staff, venues, supplies and equipment, and budgeting, among various other administrative procedures. Also, a lack of previous experience meant there were many unclear points about individual preparation activities, making the entire preparation process ineffective. We therefore created a drill preparation manual and passed it on to the next organizers.

We learned from the numerous problems we encountered in Drill 1 (Table 1) that organizing a drill from scratch is unrealistic, so looking back at our experiences, we created a disaster drill manual to streamline preparation for future drills. This manual includes a time schedule of tasks to complete, necessary staff members, detailed supply and equipment lists, and formatted documents/forms. In this study, we assessed the effectiveness of this drill manual by comparing preparation procedures for Drill 1 (before creation of the drill manual) with those of Drill 2 (in which we used the drill manual for drill preparation). Drill 2 was our third drill, conducted in the western medical district.

Table 1

Problematic events

Problematic issues/events	Details
Administrative work	Order and timing of necessary operations
	Number and role of necessary drill staff
	Arrangement of venues for the drill, briefings, etc.
	List of supplies and equipment to prepare
	Budget to request
Formatted documents/forms	Template for drill manual
	Action cards for participants
	Format used in a form to communicate information
Verification of drill quality	Goal of drill
	Appropriate disaster simulation and scenario
	Methods to evaluate participants

Assessment of the drill manual content

For a drill of this scale, planners come from multiple institutions, which makes frequent face-to-face meetings difficult. Therefore, most of the drill preparation activities were handled by email, resulting in large numbers of email communications. The contents of these emails were discussions between drill planners about various preparation processes, such as developing drill scenarios, creating documents/forms and formats, communicating with Tokyo Metropolitan Government, and making requests to drill staff. We decided to use these emails exchanged during the preparation process as a quantitative and qualitative indicator of the drill preparation process, comparing the results for Drill 2 against those of the earlier Drill 1 as an indication of the effectiveness of the drill manual created.

Specifically, to elucidate the scale and details of Drills 1 and 2, we analyzed the number of drill staff, number of players, number of booths placed, and ratings of players. To quantify the preparation process for Drills 1 and 2, we next analyzed the preparation period, total number of emails, total number of characters in each email, and duration of face-to-face meetings (excluding time spent on email). In addition, to analyze the drill preparation processes qualitatively, frequently occurring words were extracted from emails, and the word frequencies were compared between the drills. We then evaluated words showing a large difference in frequency between the drills to show a change in the quality of the drill preparation process.

Note that we excluded words that do not have meaning by themselves, such as “this” and “and”, when extracting frequently occurring words. In addition, we manually examined words that were automatically extracted and counted those with a specific meaning when appearing in sequence as one term; for example, “action” directly followed by “card” was regarded as the single term “action card”.

Frequently occurring words were analyzed using the following equation to calculate the difference in the frequencies of appearance between Drills 1 and 2.

Number of occurrences in Drill 1 emails/total number of characters in Drill 1 emails – Number of occurrences in Drill 2 emails/total number of characters in Drill 2 emails.

To eliminate observer and respondent bias, text analysis was performed by research staff different from those involved in the organization/execution of drills or the development of the drill preparation manual.

Results

There were 45 drill staff in both Drill 1 and Drill 2, four of whom participated in both drills as drill instructors (Table 2). In Drills 1 and 2, there were 31 and 58 players and six and eight booths, respectively (with booths representing Offices and hospitals). The two drills did not differ significantly in any factors (Tables 2 and 3). In addition, drill results were very similar between the drills (Table 4).

Table 2

General data for each simulation drill

	Drill 1	Drill 2
Simulation time (h)	3	3
Total participants (persons)	76	103
Players (persons)	31	58
Simulators (persons)	23	22
Instructors (persons)	10	13
Other (persons)	12	10
Rehearsals (times)	3	3
Explanation meetings (times)	3	3
Assumptions for earthquake	Wind velocity 4 m/s, Tama Inland Earthquake, 12:00 pm during winter	Wind velocity 4 m/s, Northern Tokyo Bay Earthquake, 12:00 pm during winter
Red tag victims (persons)	75	105
Total booths	6	8
Headquarter booths	3	4
Hospital booths	3	4
Communications tools	Fax (simulated), telephone (real)	Fax (simulated), telephone (real)

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual.

Table 3

Diagnoses of red tag players

	Drill 1	Drill 2
Severe brain contusion	5	8
Intracranial lesion requiring surgery	8	10
Airway obstruction (for example, maxillofacial trauma)	5	8
Facial burn + airway injury	5	8
Thoracic trauma (hemothorax, pneumothorax, contusion)	10	12
Limb injury with massive bleeding	8	11
Burn > 50% TBSA	7	10
Blunt abdominal trauma/major pelvic disruption with clinical signs of shock	8	11
Penetrating abdominal trauma requiring surgery	8	11
Neck trauma, suspected spinal injury	5	7
Other	6	9
	75	105

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual. All red tag players were assigned a diagnosis randomly in each simulation.

Abbreviation: TBSA, total body surface area.

Table 4

Performance indicator results in the first and second simulation drills

			Drill 1		Drill 2
1)	Official announcement of the mode of disaster	Not announced in certain areas	3	Not announced in certain areas	4
2)	Assess and evaluate content of alarm	Fixed format was used	5	Fixed format was used	5
3)	Inform the Metropolitan Government of the estimated number and severity of patients	Designated form was filled in and sent back	5	Designated form was filled in and sent back	5
4)	Inform municipalities of overall information from the Metropolitan Government	Largely not executed	1	Executed, but took a long time	3
5)	Confirm appropriate transportation based on appropriate information	Judgment delayed due to the lack of information	3	Slight improvement with an increased amount of information	4
6)	Determine the need for remote transportation based on medical assessment	Difficulties in making judgments were observed	3	Difficulties in making judgments were observed	3
7)	Appropriate staff placement/directive order system at headquarters	Largely satisfactory, slight understaffing	4	Largely satisfactory	4
8)	Appropriate use of communications infrastructure	Difficulties encountered in fax-based communications	2	Difficulties encountered in fax-based communications	2
9)	Issue commands according to the rulebook	Largely satisfactory	4	Largely satisfactory	4
	Total score (maximum 45 points)		30		34

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual.

The number of emails exchanged for drill preparation was 797 for Drill 1 and 502 for Drill 2, a 37.0% reduction. The corresponding total number of email characters was 204,360 and 118,692, a 41.9% reduction. The approximate preparation period was 7 months and 2.5 months (Table 5), a 64.3% reduction. The total amount of time spent in meetings was approximately 6 h for both drills. Email text analysis identified 72 frequently occurring words, 50 of which appeared less frequently in Drill 2 emails than in Drill 1 emails (Table 6), whereas 23 words appeared more frequently in Drill 2 emails (Table 7).

Table 5

Quantitative comparison of drill preparation

	Drill 1	Drill 2	Reduction rate (%)
Number of emails	797	502	37.0
Number of characters	204,360	118,692	41.9
Preparation period (months)	7	2.5	64.3

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual.

Table 6

Words that appeared less frequently in Drill 2 than in Drill 1 emails

	Drill 1	Drill 2	Value (×10,000)
Medical care	900	401	10.26
Disaster	471	164	9.23
Medical association	291	90	6.66
EMERGO	136	9	5.90
Triage	128	5	5.84
Coordinator	282	109	4.62
Municipality	91	10	3.61
Request	105	19	3.54
Report	131	37	3.29
Plan	77	14	2.59
Maybe/may be	98	31	2.18
Action card	65	13	2.09
Medical institution	51	7	1.91
Schedule	57	11	1.86
Public health center	43	3	1.85
Telephone	119	48	1.78
Regional disaster medicine	37	3	1.56
Injured and diseased	186	91	1.43
Meeting	37	5	1.39
Aid station	31	2	1.35
Corporative hospital	52	15	1.28
Measures	79	31	1.25
Patient	94	40	1.23
Report	47	13	1.20
Manual	38	9	1.10
Admission request	28	4	1.03

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual.

Table 7

Words that appeared more frequently in Drill 2 than in Drill 1

	Drill 1	Drill 2	Value (×10,000)
Office	114	183	−9.84
Hospital	553	426	−8.83
Meeting	86	122	−6.07
Explanation	154	149	−5.02
Drill	735	486	−4.98
Clinical case	85	103	−4.52
Request	132	129	−4.41
Medical district	266	198	−3.67
Player	51	70	−3.40
Communication	132	112	−2.98
Guide	18	45	−2.91
Information	165	125	−2.46
Person-in-charge	51	56	−2.22
Preparation	60	60	−2.12
Cooperation	37	41	−1.64
Tabletop drill	146	103	−1.53
Hospitals providing disaster medical services	55	47	−1.27
Controller	124	86	−1.18

Notes: Drill 1, organized without a manual; Drill 2, organized with a manual.

Discussion

Less preparation was involved in Drill 2 than in Drill 1. Moreover, in Drill 2, the length of the drill preparation showed a larger reduction than the total number of emails or email characters, suggesting that many emails were exchanged over a shorter period of time when preparing for Drill 2. There were also fewer preparation tasks overall and each process was performed more efficiently when preparing for Drill 2, which we attribute to use of the drill preparation manual.

The analysis of frequently occurring words revealed that a wider variety of word types were used in preparation emails for Drill 1 than for Drill 2, suggesting that discussion was more tightly focused in the preparation for Drill 2. When preparing for Drill 1, we had to start off with a discussion of what to prepare because we had no drill manual to follow, whereas the discussion for Drill 2 was more focused because we were using the manual.

The most frequently occurring words were “medical care”, then “disaster”, and “Medical Association” in Drill 1 preparation emails, whereas these were “Office”, “hospital”, and “conference” in Drill 2 preparation emails. This suggests that players needed to start discussions from the most fundamental issue – the nature of disaster medicine – in Drill 1 emails, whereas their discussions in Drill 2 emails were about the specifics of the drill itself.

Interestingly, the words “EMERGO” and “triage”, both of which appeared frequently in Drill 1 emails, were not factors incorporated into either Drill 1 or Drill 2, suggesting that while we continuously debated what type of drill to perform in Drill 1, such discussion was unnecessary in Drill 2 because of the presence of the manual. Furthermore, because the Drill 1 planners needed to create various documents and forms from scratch, words such as “planning documents”, “paper reports”, and “action card” appeared often in Drill 1 emails.

On the other hand, words that frequently appeared in the Drill 2 emails were “cooperation”, “information”, “communication”, “request”, and “explanation”, all of which were associated with information sharing and thus matched with the theme of the drill, which is to verify the medical information and communications system between the Offices. In addition, words representing people and roles, such as “player”, “administrative staff ”, and “person in charge”, appeared frequently, suggesting that discussions on Drill 2 were more specific to the process.

Based on the above findings, we believe the drill preparation manual enabled the following:

Visualization of necessary work, making the preparation efficient.

Use of formatted documents/forms, making the preparation easy.

Easy decisions on the specifics of the drill because the goal of the drill was clear.

This also means that the problematic events in Drill 1 (Table 1) were adequately addressed in Drill 2 because of the manual, objectively showing the utility of the manual.

The planning of various types of drills can be made more effective by creating manuals, similar to ours, for different hospital scales, administrative levels (prefecture, city, smaller municipality, etc.), and drill patterns (tabletop, map-based, and full-scale drill). In addition, we plan to revise the manual over time, improving it based on the results of using it to plan future drills and making it more effective.

The formulation of high-quality preparation manuals is necessary to help emergency physicians plan and execute disaster medical response drills while also performing their routine duties in the clinical setting. Manuals can be assessed using subjective assessment methods, such as questionnaires for users, or objective text analysis methods developed for qualitative and quantitative investigation in various fields including cultural anthropology.17,18 In addition, text analysis and text mining of emails are performed routinely in research as well as business.19–21 Therefore, in this study, we proposed email text analysis as an objective alternative and then tested it for the assessment of an actual manual. We found that examining the attributes of frequently appearing words during the preparation of the drill, enabled quantitative assessment of the manual as well as qualitative assessment of reasons to demonstrate the validity of the manual. Repeat studies examining a larger number of drills would be valuable.

This study has some limitations. It is a retrospective study comparing preparation status between two drills only. Also, assessment of qualitative and quantitative effects of the manual on the preparation of a drill entirely relies on text analysis without taking other factors into account.

Conclusion

It takes enormous time and effort to plan disaster drills. However, the findings of this study suggest that the use of a drill manual helps those involved in the preparation of drills visualize the tasks needed for planning, and then streamline the preparation process to reduce wasted time and resources. This study suggests that email text analysis is a useful objective method for assessing the effectiveness of such drill manuals.