Development and Validation of a Scoring Assessment Tool for Hospital Safety: A Pilot Study Comparing Hospital Preparedness in Thailand.

INTRODUCTION: Safe hospitals are crucial in the management of major incidents and disasters. A hospital self-assessment tool was developed for Thailand to identify gaps and shortcomings in hospital preparedness. However, this tool lacks the ability to determine the level of preparedness and cannot be used to standardize hospital readiness and enable continuous quality control.
OBJECTIVE: The aim of this study was to test a developed scoring hospital assessment tool to evaluate the level of hospital preparedness and enable quality control and compare the results of various hospitals.
MATERIAL AND METHODS: Using the nominal group technique, three experts evaluated all sections of the previously developed hospital self-assessment tool and recognized that each element could be answered by one of the three options: Yes, Not Known, and No. A pilot study was conducted in 11 hospitals to evaluate the feasibility of the tool. The number of Yes responses was divided by the total number of elements to represent the level of hospital preparedness and reported as either low (0‒59), average (60‒79), or good (80‒100). The results identified areas for improvement.
RESULTS: Eleven out of 13 hospitals (85% response rate) in two provinces were enrolled in the study. The results showed various levels of preparedness in all the investigated hospitals. Two hospitals had low preparedness and needed great improvements. The remaining nine hospitals in the two provinces had average preparedness levels and needed improvements. One of the nine hospitals had a score very close to achieving good preparedness. No significant parameters were associated with the preparedness level.
CONCLUSION: The developed scoring assessment tool for hospital safety demonstrated high utilization feasibility and indicated preparedness levels. The scoring tool also provided assessment levels that could enable continuous quality evaluation and improvements.

Introduction

Hospital Safety Index

Hospital safety and functionality are of the utmost importance for the delivery of healthcare during major incidents and disasters. Safe hospitals have long been a priority of the World Health Organization (WHO), and together with the Pan American Health Organization developed the Hospital Safety Index (HSI), which is an international and widely used hospital assessment tool.1 During a major incident and disaster, hospitals should be ready, like other parts of the community, to protect the lives and well-being of the victims in all phases of disaster management.2,3 Consequently, the functionality of a hospital may rely on several structural and non-structural factors, such as robustness of its building, safety, functionality, and maintenance of its critical systems, equipment, and its management capacity.1,3 A hospital should be able to meet a surge in capacity by providing the needed staff, stuff, structure, and systems.4

The WHO HSI is a self-assessment tool developed by experts that aims to facilitate a rapid and reliable assessment of hospital preparedness. In addition, it allows comparisons between hospitals in regional, national, and international settings to improve their functionality and quality of care.1,5 However, the use of the index is time- and resource-consuming since it consists of 46 pages and 191 items in an evaluation form plus a 176-page guideline for the evaluator. Furthermore, the WHO HSI needs a multidisciplinary team that includes engineers, architects, healthcare personnel, and emergency and disaster management specialists.1,5 Results from the self-assessment identify risks and gaps for improvement as part of quality assurance.

Thailand and Hospital Self-Assessment Tool

Thailand is prone to major incidents and disasters, such as landslides in the north, terrorism in the south, flooding in the northern, southern, and middle regions, and fires.6–11 These events necessitate having safe and functional hospitals and consequently demands continuous hospital evaluations in terms of safety and functionality. To study hospital safety in Thailand, a variety of assessment tools, such as the one from WHO can be used.1,12

The WHO HSI1,13 is a tool designed to assess hospital safety in emergencies or disasters and is most suitable for tertiary, university, and major referral hospitals. The tool has two components: general information and a safe hospital checklist. General information includes descriptive information about the hospital in both actual and surge capacities. The checklist is divided into four modules: (i) hazards, (ii) structural, (iii) non-structural, and (iv) emergency and disaster management. The sum of the scores in the checklist represents the level of hospital preparedness dealing with emergencies and disasters.

However, the WHO HSI is not entirely appropriate to assess the safety of hospitals in Thailand. First, the checklist module of the HSI is different from the Ministry of Public Health (MOPH) System framework that is categorized into governance/leadership, financing, health workforce, medical products and technologies, service delivery, and participation, whereas the HSI is categorized into hazards, structural, non-structural, and emergency and disaster management. Second, hospitals in Thailand are categorized based on their capacity and categorized as First hospitals (10‒90 beds), Middle hospitals (90‒300 beds), Standard hospitals (300‒500 beds), and Advanced/University hospitals (more than 500 beds). However, the WHO HSI focuses on tertiary, university, and major referral hospitals. Lastly, the checklist has 191 items and many items need to be evaluated by a multidisciplinary team. Additional 176 pages of the guideline are for the evaluator, which results in consuming time and resources. Consequently, a concise and simple hospital self-assessment tool14 was developed based on Thailand’s MOPH System framework, which in turn was based on the WHO’s six building blocks (governance/leadership, financing, health workforce, health information system, medical products and technologies, service delivery and participation)14,15. The development of the tool was facilitated by implementing various methodologies, such as a systematic literature search enrolled 76 full texts from 5869 titles and abstracts, validity assessment by three recognized experts in emergency medicine and disaster management, and a stakeholders meeting composed of 46 emergency managers from hospitals and emergency medical service related organizations. The tool has four components: general information, preparedness checklist, comments and suggestions, and actual and surge capacity (Table 1). In the tool, the preparedness checklist was customized based on hospital capacities of Thailand and key elements were rearranged into the WHO six building blocks plus components. The tool facilitated hospitals to assess risk, hazards, structure, and functions in accordance with the MOPH health framework resulting in a gaps analysis and strategic planning, which proposed to serve as the national standard14,15 ().

Table 1

Components of a Thai Hospital Preparedness Assessment Tool for Mass Casualty Incident and Disaster (Advanced Level or University Level) Compared with the WHO Hospital Safety Index

Category	Components (Items)
	The Study Tool	WHO Hospital Safety Index 2015
General information	Location, Capacity, Annual ED visits, Personnel (clinical, non-clinical), Surge capacity, Accreditation, Level of trauma center, Burn capacity, Helipad (9 items)	Name, Address, Name of senior manager, contacts, average bed occupancy, Personnel (clinical, non-clinical), General description, Physical distribution, Treatment & operating capacity (specify each department/function), Surge capacity (14 items plus 45 sub-items)
Preparedness checklist	Governance/Leadership (24)Financing (11)Health workforce (14)Information system (17)Medical products and technologies (29)Service delivery (25)Participation (7)Total 127 items	Module 1: Hazards (40)Module 2: Structure (18)Module 3: Non-Structure (93)Module 4: Emergency and Disaster Management (40)Total 191 items
Suggestions	According to preparedness components	Evaluator comments
Hospital actual and surge capacity	Actual capacity, Surge capacity (within 12 hours), Vital signs monitors, Respirators, Negative pressure room, etc	Items 13 in General information.Not specified.

The Need to Develop a New Tool

Despite its good functionality, the self-assessment tool used in Thailand does not determine the level of preparedness. Thus, a tool that is less subjective and can assess the level of preparedness might be more feasible for standardizing hospital readiness assessment and providing continuous quality assurance. This study aimed to develop a scoring assessment tool for hospital safety by converting the current hospital self-assessment tool used in Thailand to a scoring tool to enable objective comparison of hospital preparedness levels.

Methods

The authors intensively employed the nominal group technique16 to study the previously developed Thai hospital self-assessment tool14,15 to assure the validity of the data and analysis. The nominal group technique is a structured method for group brainstorming that encourages all participants to contribute to a discussion about a subject and facilitates quick agreement on the relative importance of issues, problems, or solutions.

However, the second section (preparedness checklist) needs special attention since it contains special features and key elements that influence sections 3 and 4 and the grade of preparedness. Following adjustment of relevance, clarity, logic, and accuracy, the questions in this section were particularly studied for the feasibility of being answered as either Yes, Not Known, or No based on group consensus of the researchers according to the nominal group technique.16 In order to initiate a simple prototype of a scoring tool, the researchers decided to divide the number of Yes responses for each hospital by the total number of Yes of 127 items. The outcome presented the preparedness percentage. All items aimed at all hospital levels (ie, First, Middle, Standard, Advanced/University) were analyzed. Missing values were considered to be No or Not Known responses.

The tool was sent to the evaluators and/or hospital preparedness officers of 13 hospitals in two provinces of southern Thailand: Pang-Nga [PN] and Phuket [PU]. Therefore, a thorough review of the tool was done within each hospital group before researchers visited the hospitals. Two Swedish medical students, supervised by PW and AK, studied the hospital assessment tool including the preparedness checklist. The representatives of each hospital were then interviewed (eg previous experiences, hospital incident command system, etc.) and all the data from each hospital were collected as part of their projects.17,18 A Thai supervisor accompanied them during the hospital visits and interview. The evaluators reviewed and filled in the answers in consensus during the visits. The students documented all responses during the interviews in the presence of and supervised by their Thai supervisor. All answers were compared with pre-written hospital responses and, if needed, hospital staff personnel were asked to clarify any outstanding issues. The results were categorized into low preparedness (in need of great improvement) (0‒75 points or 0‒59%), average preparedness (in need of some improvement) (76‒100 points or 60‒79%), and good preparedness (in need of less improvement) (100‒127 points or 80‒100%).

Statistical Analysis

The means of the preparedness percentages were compared between hospitals to investigate potential differences between hospitals or provinces. The key elements in the preparedness checklist were tested for any correlation to the total preparedness percentage to determine whether any of the key elements in the preparedness checklist could predict good preparedness. The data were analyzed using Microsoft Excel (Microsoft Corporation) and SPSS Statistics 24 (IBM). Analysis of variance (ANOVA) was used to compare the means of preparedness percentage between the respective hospital levels and between the two provinces. All data on the variables in elements were collected assuming that the data were continuous variables, had a linear relationship, no significant outliers existed among the hospitals, and the data fulfilled other requirements for a regression analysis according to the SPSS Statistics manual. Spearman correlation coefficient was used to correlate key items and preparedness percentage. The level of statistical significance was a p-value ≤0.05. A statistician performed all the statistical analyses.

Ethics

The Ethics Committee of the Faculty of Medicine, Prince of Songkla University, Thailand, approved this study as part of the national survey in 2017 (REC number: 59-328-20-1). Permission to conduct the study was obtained from each hospital director of the participating hospitals.

Results

Eleven out of 13 hospitals were enrolled in the study (85% response rate), which included six First hospitals, two Medium hospitals, two Standard hospitals, and one Advanced/University hospital. The hospitals have various capacities and emergency room annual visits (Table 2).

Table 2

Investigated Hospitals According to Their Capacity Levels

Hospital	Level*	Number of Doctors	Number of Nurses	Level of Trauma Center**	ER Visits Yearly
PN1	First	5	21	4	<25,000
PN2	First	2	11	4	<25,000
PN3	First	5	38	4	<25,000
PN4	First	2	11	4	<25,000
PN5	First	2	24	5	<25,000
PN6	First	6	10	4	25,000‒50,000
PN7	Medium	34	179	4	25,000–50,000
PN8	Standard	32	204	2	25,000‒50,000
PU1	Medium	13	79	4	25,000‒50,000
PU2	Standard (private)	112	271	2	<25,000
PU3	Advanced/University	148	629	2	50,001‒75,000

Notes: *The hospital levels are defined as first (10‒90 beds); middle (90‒300 beds); standard (300‒500 beds); and Advanced/University (more than 500 beds). **The levels of trauma centers in Thailand are defined as: 1 = comprehensive and advanced tertiary care capability, regional and referral trauma center; 2 = definitive care capability but does not include advanced-surgical critical care, cardiac and great vessel injury capabilities; 3 = resuscitation and emergency care capability; 4 = basic trauma care capability provided by general practitioners.

Abbreviations: PN, Phang-Nga; PU, Phuket; ER, emergency room.

Total Scores

The total scores showed that two hospitals had low preparedness. Both hospitals were located in one province and needed great changes. The remaining nine hospitals had average preparedness scores. One of the nine hospitals had a score very close to achieving good preparedness. These hospitals were located in both provinces and were in need of various improvements. Results from the regression analysis showed no significant differences in preparedness scores associated with the province or capability level (Table 3).

Table 3

Results of Total Scores and Scores of All Elements in the Preparedness Checklist

Hospital (Level)	TS %	Preparedness Level	GOV %	FIN %	HWF %	INFO%	SD %	MPT %	PART %
PN1 (F)	77.4	Average	80.0	22.2	100.0	93.3	100	89	57
PN2 (F)	17.3	Low	0.0	0.0	85.7	0.0	0	22	14
PN3 (F)	63.6	Average	70.0	22.2	100.0	73.3	65	72	43
PN4 (F)	11.7	Low	0.0	0.0	0.0	0.0	15	67	14
PN5 (F)	68.8	Average	55.0	22.2	100.0	60.0	90	83	71
PN6 (F)	75.4	Average	55.0	77.8	71.4	93.3	95	78	57
PN7 (M)	68.6	Average	42.9	44.4	87.5	73.3	86	75	71
PN8 (S)	68.3	Average	52.2	54.5	76.9	70.6	100	81	43
PU1 (M)	78.5	Average	81.0	77.8	100.0	73.3	86	75	57
PU2 (S)	67.9	Average	78.3	54.5	69.2	70.6	76	56	71
PU3 (A)	63.6	Average	50.0	18.2	78.6	70.6	96	61	71

Abbreviations: TS, total score; GOV, governance; FIN, financing; HWF, health workforce; INFO, information; SD, service delivery; MPT, medical products and technologies; PART, participation; F, First (10‒90 beds); M, Middle (90‒300 beds); S, standard (300‒500 beds); A, Advanced/University (more than 500 beds); PN, Phang-Nga; PU, Phuket.

Scores in Each Component

Governance

Two hospitals with low total scores also had very low governance. Of the remaining nine hospitals, two hospitals had good governance scores, while the other seven had average levels.

Financing

All but two hospitals had low financing scores. One of the two with an average score also had good scores in governance, health workforce, and service delivery, while the other had good scores in service delivery and information system.

Health Workforce

Only one hospital failed to have an acceptable health workforce score. In general, hospitals with average to good governance and financing had better scores in health workforce.

Information System

Two hospitals with low total scores in governance and financing also had low scores in information system. The majority of the remaining hospitals had average to good information system scores.

Service Delivery

Seven hospitals had good service delivery scores, two had average scores, and two failed to achieve an acceptable level. The latter two hospitals also failed to score an acceptable total score.

Medical Products and Technologies

Three hospitals had good scores, six had average scores, and the remaining two scored low preparedness in this component.

Participation

Training and participation in different activities had the lowest scores in this study. None of the hospitals achieved a good score and only three had an average score.

The results from regression analysis using ANOVA (univariate analysis of variance) showed no significant differences in preparedness scores associated with the province (p = 0.617) or capacity level (p = 0.894). However, a significant correlation with the overall score was found for the following elements: 1) Mass casualty incident committee (‒0.710, p = 0.014), 2) Mass casualty incident plan (‒0.671, p = 0.024), 3) Training, incident command system (‒0.725, p = 0.12), and 4) Hospital teams and Emergency management team (‒0.671, p = 0.024).

Discussion

This study illustrated the feasibility of converting a hospital self-assessment tool into a scoring tool for evaluation of hospital safety. The tool may also facilitate comparative studies between hospitals and consequently enable continuous quality improvement.

Studies in Iran showed that a modified version of the WHO tool (Farsi Hospital Safety Index) could be used repeatedly to evaluate hospital safety and to create strategies and measures to improve hospital disaster preparedness.19 Other studies on hospital disaster preparedness in Europe showed that preparedness might be correlated to the number of emergency department visits rather than hospital size.20 Furthermore, an Italian pilot study reported a relative consistency between evaluated safety and preparedness scores and actual performance.21 Furthermore, the perception of hospital preparedness facilitated staff knowledge in disaster management, and consequently their willingness to participate in the management of public health emergencies and disasters.22

Although self-assessment tools are valuable instruments to evaluate hospital safety and preparedness, this type of assessment tool is subjective, consumes time and resources, and may be subject to bias by the evaluator.23 Nevertheless, the bias imposed by the evaluator can be improved by combining the evaluation with a post-evaluation meeting, a table-top exercise, or using grades instead of Yes, Not Known, or No as answering alternatives. As the latter sounds more feasible, such grading may be similar to the one in this study (ie, a percentage for each Yes answer) or using a Likert scale with 5 to 10 options, which requires clearer descriptions of what should be included to acquire a certain grade. Using a scoring tool by converting limited options to percentages in this study gave a rough estimation that was promising but imperfect since the answers had to be checked and corrected in a post-evaluation meeting with all evaluators. Nevertheless, this approach offered a valuable base for developing future global tools with more feasibility. Such a base for comparing different hospitals may not only give a better overview of what should be changed locally and regionally but may also increase the feasibility and the use of the tool.

This study dealt with 11 hospitals in southern Thailand that were hit by the 2004 tsunami and hypothetically should have higher degrees of preparedness. The 85% response rate (11/13 hospitals) was fully acceptable and the results showed various levels of preparedness. The outcomes of this study indicated that the majority of the hospitals had average preparedness and were in need of improvements. Although these results were obtained by a new method and tool, they were similar and fully comparable with the results presented in previously published reports.6,11,24 The higher degrees of preparedness in these hospitals were due to changes applied to the disaster management systems after the 2004 tsunami and other devastating events. In particular, the flooding in 2011 in Thailand revealed shortcomings in command and control, emergency plans, reliable external and internal information systems, and interagency collaboration.24–29 Psychological consequences (eg, post-traumatic stress disorder, depression, and insomnia) were experienced at 20/32 (63%) hospitals during the initial 6-month interval after the flooding in 2011 in Thailand.28 During the flooding of central Thailand in 2011, road network disruptions led to problems with supply chains and transport of personnel and equipment to hospitals. Logistic planning of personnel and equipment was found to be good, which also indicated improvement.24,28,29

The outcomes from this current study demonstrated the feasibility and reliability of a developed scoring assessment tool. The tool was practically easy to use, but having less than desirable information for each element led to uncertainty and different interpretations by the evaluators, which were sorted out during post-evaluation meetings. Development of an assessment tool manual and educational initiatives to guarantee the use of the tool should be planned and developed. The national use of the tool in this study would facilitate the ideal method from a national perspective since each component corresponds to an area of responsibility within the MOPH framework. Further studies and improvements of the tool should simplify its use and verify the feasibility of implementing other types of grading, such as the Likert scale, as the standard. A standardized tool for continuous evaluation of hospital safety might help improve the preparedness levels and the quality of hospitals.

Limitations

The main limitation of this study is its small sample size. A larger sample, such as a national implementation, is needed to obtain power that enables a more detailed statistical analysis. The limited number of items tested in this study may be a limitation. However, since this was a pilot study, it would be desirable in the future to recruit a higher number of participating hospitals to enable testing of all items for correlations, and the findings can then be compared with other reports in the literature. Additionally, a standardized tool can assess all hospitals according to the same criteria, apart from any differences in patient groups at the hospitals. Since the needs could differ between different patient groups, this factor needs to be considered in future tools. Finally, the study did not perform a psychometric analysis because the scoring was developed based on a previous hospital self-assessment tool in which a validity assessment was previously conducted.

Conclusion

The scoring method of the hospital preparedness assessment tool for mass casualty incident and disaster is feasible to determine the level of hospital preparedness. A mass casualty incident committee, mass casualty incident plan, training of an incident command system, hospital teams, and an emergency management team were significant elements correlated with the level of hospital preparedness. The tool can also be used for hospital safety evaluation and quality assurance.