Content validity of the newly developed risk assessment tool for religious mass gathering events in an Indian setting (Mass Gathering Risk Assessment Tool-MGRAT).

BACKGROUND: Risk assessment (RA) for mass gathering events is crucial to identify potential health hazards. It aids in planning and response activities specific to the event but is often overlooked by the event organizers. This paper reports the content validity process of a newly developed tool called Mass Gathering Risk Assessment Tool (MGRAT), which intends to assess the risks associated with religious mass gathering events in Indian settings.
METHODS: Qualitative approach was followed to identify the risks associated with mass gathering events and to identify the domains and items to be included in the RA tool. The draft tool was shared with six experts who were selected by the convenient method; selected experts were requested to assess the tool and give their comments about the domains, items, relevant responses, and overall presentation of the tool using content validity questionnaire. Content validity index and Fleiss kappa statistics were calculated to assess the agreement between multiple raters.
RESULTS: Agreement proportion expressed as scale-level content validity index (S-CVI) calculated by the averaging method is 0.92. S-CVI; calculated by universal agreement is 0.78. Fleiss kappa statistics to measure the agreement between multiple experts after adjusting the component of the chance agreement is 0.522 (95% CI: 0.417, 0.628, P value: 0.001).
CONCLUSION: MGRAT is a valid tool, which has an appropriate level of content validity. As the number of raters increases, there will be difficulty in achieving consensus among all the items, which is the reason for lower Content Validity Index/Universal Average (CVI/UA) when compared with Content Validity Index/Average (CVI/Ave). Fleiss kappa statistics also indicated moderate agreement among the raters beyond the chance agreement, which also supports the appropriate content validity of MGRAT.

Introduction

According to the World Health Organization, MG event or MG is a gathering of persons that is usually defined as “the congregation of more than a specified number of people (this may be as few as one thousand persons; although most of the literature available describes these as gatherings that exceed 25,000 people) at a specific location for a specific purpose (a social function, a large public event, and a sports competition) for a defined period of time.”[1]

Mass gatherings cause notable challenges in terms of communicable and non-communicable disease surveillance, emergency preparedness, environmental health, vaccination, food safety, crowd management, etc.[2] In spite of the fact that MGs are an undeniably regular element of our society attended by huge crowds, yet such occasions are not very well understood. Although such gatherings are accumulations of “well people,” vast number of people associated with MGs can put a serious strain on the entire health care system.[3] Along these lines, such MG events are more perilous and hazardous in terms of higher incidence of injury and illness compared to the population in general.[4]

Preplanning for MG events is crucial and identification of potential health risks can be a vital element in the pre-event planning phase.[1] Risk assessment (RA) is an integral component of risk governance and serves as an initial step in the process of planning RA for an MG is a process that determines the intent and implementation of risk reduction measures, response planning, and capacity building for health functions.[5] RA for MGs is undertaken to empower the public health authorities to identify and evaluate the generic characteristics of a MG that introduce or escalate specific threats. Systematic assessment of risks also helps to identify the potential health security risks that require the cooperation of other departments and government agencies.[2]

During religious gatherings in India, some special events and unforeseen events occur at the places of religious MGs besides fixed places of worshipping. Special events such as idol procession, chariot pulling, fire walking, and animal sacrificing happen pulling larger crowds within the MG events and causing more damage to human beings and property. History is replete with incidences when MGs at fairs and festivals of India have turned into the hotspots of various types of risks and disasters.[6] A ten-year analysis of public health safety in 27 traditional MG events of India indicated around 936 dead and 540 injured casualties.[7]

Although public health system in India efficiently manages most of the religious MG events, there is no systematic process of the RA conducted in the field. Most of the process of RA is either overlooked or depends upon intuition and previous experiences. There is a need to systemize the process of RA for MG events for further risk reduction.[689] Hence, to place the right measures in place to address the foreseeable and unforeseeable risks, the proposed RA tool (MGRAT-Mass Gathering Risk Assessment Tool) in this study will be fundamental in identifying potential public health risks and prioritize planning and response activities specific to the religious MG events in an Indian setting.

Validity and reliability assessment are an essential part of tool development. Content validity is the index of whether the items in a test, both individually and as a whole represent the construct that it is proposed to measure. Content validity is concerned with the comprehensiveness of the item pool and representativeness of the mass gathering RA items included in the tool. After the initial design of the tool, content validity assessment is the leading process in the development of the tool to assess the contents for appropriateness for further field testing the tool.[101112] This paper reports the content validity process of a newly developed tool called MGRAT, which intends to assess the risks associated with mass gathering events in the Indian settings.

Methods

Initial process involved in the development of MGRAT tool

A qualitative approach was followed to identify the risks associated with mass gathering events and to identify the domains and items to be included in the RA tool. First, an extensive review of literature about the risks associated with the mass gatherings, theoretical basis for RA and available RA tools for mass gathering events was done. Second, key informants (n = 15) involved in the planning and management of religious mass gathering events in the state of Tamil Nadu, India were purposively identified and interviewed using a semi-structured interview guide. Principle of redundancy was followed. Content/Thematic analysis was done. A detailed explanation of the steps/process involved in identifying the domains and items is not reported in this paper.

Domains and Items identified

A sum total of forty-eight unique health risks were identified. Stampedes, fire accidents, structural collapse, drowning, outbreak of communicable diseases, exacerbation of existing medical illnesses (such as cardiac diseases, asthma, etc), etc., are some of the health risks identified. Seven domains (characteristics related to event, participant, environment, food and water related, disaster preparedness, medical service preparedness, and pre-event planning activities) and twenty-three items were generated from the content analysis of key informant interviews and literature review.

Content validity process

For evaluating content validity, the draft tool was shared with six experts who were selected by the convenient method. Draft tool was circulated among six subject experts for review (four experts from the department of public health and preventive medicine and two academics/research consultants). Selected experts were requested to assess the tool and give their comments about the domains, items, relevant responses, and overall presentation of the tool.

They were also requested to assess the tool according to whether all the items refer to relevant aspects of the construct to be measured, whether all the items together comprehensively reflect the construct and whether all the items are relevant for the setting where it is going to be applied, are simple and understandable. Using a self-administered content validity questionnaire, the experts were requested to assess the relevance of each item generated on a four-point Likert scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, and 4 = highly relevant to avoid having a neutral and ambivalent midpoint).

Content validity questionnaire consisted of 28 questions. Out of these 23 questions addressed, the assessment of relevance of all the individual items (n = 23) under 7 domains in the developed tool. There were 5 questions addressed to assess the over-all relevance, comprehensiveness, usability, simplicity, understandability, etc.

Data collected from experts through content validity questionnaire were entered and analyzed with statistical software Epi info 7.1.5.2 version. Agreement proportions between the experts were calculated. Content validity index for overall scale (S-CVI) and item wise (I-CVI) were calculated. Fleiss kappa statistics to assess the agreement between multiple raters adjusting for chance agreement were calculated.

Ethical committee clearance

Approval for the study was obtained from the institutional ethics committee. Participants were informed about the purpose of the study by information sheet, which was provided in the English language and informed written consent for participation was obtained.

Results

Table 1 indicates that out of the twenty-three items, eighteen items were agreed upon as “Relevant” by all the six experts. The item “Solid waste management” was agreed upon as “Relevant” by only one rater and “Irrelevant” by the other five experts. Item-content validity index (I-CVI) of “Solid waste management” (item no 14) was calculated to be 0.17. The I-CVI of items “Participant origin (item no 7)” and “Psychosocial behavior of participants (item no 9)” was 0.67 and I-CVI of item “Area involved (item no 4)” and “Fire safety (item no 20)” was 0.83. The items 7, 9, 14, and 20 were retained in the tool with minor modifications, and “Solid waste management” (item no 14) was merged with “Sanitary and hygiene facilities.”

Table 1

Content Validity Index (CVI)

Item no	Domain	Item Description	Rater_1	Rater_2	Rater_3	Rater_4	Rater_5	Rater_6	Number agreement	I_CVI
1	1	Activity level	×	×	×	×	×	×	6	1
2	1	Topographical characteristics	×	×	×	×	×	×	6	1
3	1	Event duration	×	×	×	×	×	×	6	1
4	1	Area involved	×	×	-	×	×	×	5	0.83*
5	1	Expected no of participants	×	×	×	×	×	×	6	1
6	2	Special participant profile	×	×	×	×	×	×	6	1
7	2	Participants origin	×	×	-	×	-	×	4	0.67*
8	2	Predominant age group	×	×	×	×	×	×	6	1
9	2	Psychosocial behavior of participants	-	×	×	×	×	-	4	0.67*
10	3	Season	×	×	×	×	×	×	6	1
11	3	Type of accommodation	×	×	×	×	×	×	6	1
12	3	Access route to the event	×	×	×	×	×	×	6	1
13	3	Sanitation and hygiene facilities	×	×	×	×	×	×	6	1
14	3	Solid waste management	-	-	-	-	-	×	1	0.17**
15	3	Special rituals	×	×	×	×	×	×	6	1
16	4	Food safety	×	×	×	×	×	×	6	1
17	4	Water safety	×	×	×	×	×	×	6	1
18	5	Crowd management	×	×	×	×	×	×	6	1
19	5	Event access points	×	×	×	×	×	×	6	1
20	5	Fire safety	×	×	×	-	×	×	5	0.83*
21	5	Natural hazards management	×	×	×	×	×	×	6	1
22	6	Level of medical services at the venue	×	×	×	×	×	×	6	1
23	7	Preplanning activities	×	×	×	×	×	×	6	1

S-CVI/Ave: 0.92. Total agreement: 18; S-CVI/UA: 0.78. **Item no 14 merged with item no. 13; *item no 3, 7, 9, 20 retained with minor modifications

Agreement proportion expressed as scale level content validity index (S-CVI) calculated by averaging method (S-CVI/Ave = average of all I-CVI) was calculated to be 0.92. In addition, S-CVI calculated by universal agreement method (S-CVI/UA = Total number of items agreed relevant by all six experts/Total number of items) was calculated to be 0.78. Table 2 shows Fleiss kappa statistics to measure the agreement between multiple experts after adjusting the component of chance agreement was 0.522 (95% CI: 0.417, 0.628, P value: 0.001). As per the interpretation of Fleiss’ kappa (κ) (from Landis and Koch 1977), 0.522 indicates moderate agreement.

Table 2

Fleiss Kappa statistics to assess agreement between multiple experts (n=6)

Overall agreement between the experts
Fleiss Kappa Value	95% CI	P
0.522	0.417, 0.628	0.001

Discussion

The study reports the content validity process of the newly developed tool called MGRAT. The tool was shared with six subject experts, and they were requested to assess and rate the tool. Agreement between the experts expressed as CVI and Fleiss kappa statistics in assessing the tool as “Relevant” was calculated and reported.

Agreement proportion measured by averaging method showed 92% agreement and when measured by universal agreement method, it showed 78% agreement. These results support the excellent content validity of the developed MGRAT. However, the CVI of the instrument using universal agreement approach was found to be low compared to the averaging method. This is because of the fact that high number of content experts makes consensus difficult. The developed tool MGRAT is a pioneer attempt to develop a RA tool to assess health risks associated with mass gathering events in Indian settings. We could not find previously reported measures of agreement to compare with the agreement values of MGRAT found in this study. However, content validity studies reported that scale with excellent content validity should be composed of I-CVIs of 0.78 or higher value and S-CVI/UA and S-CVI/Ave of 0.7 and 0.9 or higher value, respectively.[11131415]

Although content validity index measures are widely used to estimate content validity by researchers, the problems with those measures are that they overlook the percentage of agreement between the experts owing to chance. Therefore, Wynd et al., propose that both CVI and multi-rater kappa statistic should be calculated in the content validity studies because unlike the CVI, kappa adjusts for chance agreement.[12] The chance agreement is an issue of concern while studying agreement indices among assessors, especially when we place four-point scoring within two “relevant” and “not relevant” classes. In other words, kappa statistic is a consensus index of inter-rater agreement that adjusts for chance agreement and is an important supplement to CVI because Kappa provides information about the degree of agreement beyond chance. Nevertheless, the CVI is mostly used by researchers because it is simple for calculation, easy to understand, and provide information about each item, which can be used for modification or deletion of instrument items.[1012]

In the present study, Fleiss kappa statistics to measure the agreement between multiple experts after adjusting the component of chance agreement showed a significant agreement value of 0.522 (95% CI: 0.417, 0.628, P value: 0.001). As per the interpretation of Fleiss’ kappa (κ) (from Landis and Koch 1977), 0.522 indicates moderate agreement. Overall, CVI and Fleiss’ kappa (κ) both the measures suggest that the content validity of the newly developed tool is appropriate. MGRAT was judged to be valid, appropriate, and feasible to assess the health risks associated with the MG events.

All seven domains were retained after the content validity process. Out of 23 items, 22 items were retained and 1 item was removed after the content validity process. The item “Solid waste management” was merged with “Sanitation and Hygiene facilities” after the content validity process as it was agreed upon as relevant by only one rater and irrelevant by the other five experts. Thus, in the refined MGRAT after content validity, there were 7 domains and 22 items. This validated MGRAT was developed further into a mobile web APP (web-based application) and field tested for its feasibility assessment in one of the religious mass gathering events in Tamil Nadu, India.

Limitations

Limitations of the study are that the experts’ feedback is subjective; thus, the study is subjected to bias that may exist among the experts. If content domain is not well-identified, this type of study does not necessarily identify content that might have been omitted from the instrument owing to the assessment. However, experts are asked to suggest other items for the instrument, which might help minimize this limitation.

This content validity study included a multidisciplinary team of experts. An effort was made to quantify the agreement and efforts to report kappa statistics to assess agreement beyond the chance agreement are some of the advantages of this study. Added to that, the idea of the development of systematic RA tool to assess health risks associated with mass gathering events is itself pioneer effort in India.

Content validity is a systematic process, which includes the judgment/quantification on instrument items by content experts. Such a process should be the leading study in the process of making an instrument to guarantee instrument reliability and prepare a valid instrument in terms of content for the preliminary test phase.[16] To conclude, the present study indicated that the MGRAT is a valid tool, which enjoys an appropriate level of content validity. As the number of rater's increases, there will be difficulty in achieving consensus among all the items, which is the reason for lower CVI/UA when compared with CVI/Ave. Fleiss kappa statistics also indicated moderate agreement among the raters beyond the chance agreement, which also support the appropriate content validity of MGRAT.

Financial support and sponsorship

Upasana Sharma is supported by Indian Council of Medical Research (ICMR) Senior Research Fellowship.

Conflicts of interest

There are no conflicts of interest.