Development and validation of a web-based questionnaire for surveying the health and working conditions of high-performance marine craft populations.

BACKGROUND: High-performance marine craft crews are susceptible to various adverse health conditions caused by multiple interactive factors. However, there are limited epidemiological data available for assessment of working conditions at sea. Although questionnaire surveys are widely used for identifying exposures, outcomes and associated risks with high accuracy levels, until now, no validated epidemiological tool exists for surveying occupational health and performance in these populations. AIM: To develop and validate a web-based questionnaire for epidemiological assessment of occupational and individual risk exposure pertinent to the musculoskeletal health conditions and performance in high-performance marine craft populations.
METHOD: A questionnaire for investigating the association between work-related exposure, performance and health was initially developed by a consensus panel under four subdomains, viz. demography, lifestyle, work exposure and health and systematically validated by expert raters for content relevance and simplicity in three consecutive stages, each iteratively followed by a consensus panel revision. The item content validity index (I-CVI) was determined as the proportion of experts giving a rating of 3 or 4. The scale content validity index (S-CVI/Ave) was computed by averaging the I-CVIs for the assessment of the questionnaire as a tool. Finally, the questionnaire was pilot tested.
RESULTS: The S-CVI/Ave increased from 0.89 to 0.96 for relevance and from 0.76 to 0.94 for simplicity, resulting in 36 items in the final questionnaire. The pilot test confirmed the feasibility of the questionnaire.
CONCLUSIONS: The present study shows that the web-based questionnaire fulfils previously published validity acceptance criteria and is therefore considered valid and feasible for the empirical surveying of epidemiological aspects among high-performance marine craft crews and similar populations. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

A dynamic web-based questionnaire for investigating occupational risk exposure associated with health and performance of high-performance marine craft populations was developed by a consensus panel, then systematically validated by expert raters for content relevance and simplicity by providing quantitative scores and qualitative comments iteratively followed by consensus panel revisions and finally pilot tested for its feasibility.

The validation process substantially improved the relevance and the simplicity of the questionnaire and the pilot test confirmed the feasibility.

The web-based questionnaire is valid and feasible for the empirical surveying of epidemiological aspects among high-performance marine craft crews and similar populations, and the entire process may be beneficial for similar questionnaire development and validation studies.

Experts individually did not represent all the dimensions of the content domain, but altogether covered the entire content domain including methodological expertise.

Introduction

The working conditions aboard high-performance marine craft, characterised by the stochastic nature of the waves, challenge the safety of life at sea. Detrimental health effects ranging from motion sickness to severe psychophysical acute and chronic conditions such as musculoskeletal disorders, fatigue and stress are common among seafarers.1–4 These conditions are believed to be the results of complex interactive factors related to environmental, human, social and other factors such as vessel type, speed and duration, among which the exposure to whole-body vibration has been identified as a distinctive factor elevating the risk for adverse health effects.5–7 Types of occupant debilitation related to whole-body vibration containing repeated shock have been identified as acute injuries from severe isolated shocks, chronic injuries from severe shocks or repeated moderate shocks and performance degradation produced by shock-related fatigue and discomfort.8 Epidemiological studies suggest that health effects are associated with prolonged exposure to vibration and repeated shock; however, no adequate epidemiological data are available to associate the human exposure to vibration and shock and the risk for adverse health effects.7 9 10 Nonetheless, the present European Union and Swedish legislation11–13 refers to the magnitude of vibration measures,10 despite the doubts on the relation between magnitude and risk. Studies by KTH Royal Institute of Technology and Umeå University in collaboration with the Swedish Coast Guard show that the statutory vibration exposure action and limit values11–13 are exceeded after a short time of exposure.14 This indicates that severe conditions are common and the fact that the exemption of statutory laws for seaborne and airborne occupations seems contradictory.11 Consequentially, the measures to assess the working conditions aboard marine vehicles related to risk for health impairments and performance deteriorations are in debate,14–20 together with the question on how to interpret the measures.

The performance depicts the influence of the working conditions on the overall performance of the man–machine system. The working conditions are largely influenced by the craft operators, which makes the delineation of performance more complicated.

Weak links in the chain: risk factors—measures—magnitudes—adverse health effects—performance, hinder designing and engineering balanced man–machine systems and delay further prevention of work-related disorders and development of an efficient and safe work environment at sea. This leads to a strong demand for thorough epidemiological studies.

Questionnaire surveys are adequate for accurately identifying exposures, outcomes and associated risks. A web-based questionnaire can be used for quickly and cost-effectively mapping large and geographically dispersed populations while preventing internal data losses.21–23 Nevertheless, a web-based survey might receive a low response rate, a drawback which could be overcome by applying an appropriately validated questionnaire. However, until now, there is no validated and feasibility tested survey tool available for assessment of work exposure in high-performance marine craft populations such as coastguard, navy, customs and maritime pilots. Therefore, the objective of this study is to develop and validate a web-based questionnaire for assessment of occupational risk exposure associated with health conditions and performance of high-performance marine craft crews and similar populations.

Methods

Study design and procedure

In this methodical study, a web-based questionnaire was developed, validated and pilot tested in collaboration between the Royal Institute of Technology, Karolinska Institutet and the Swedish Coast Guard. Initially, the domain of interest and its subdivisions (content domain) were identified by a consensus panel and the pool of items (questions) was generated for collecting sufficient data to measure each subdomain, thus the content domain.24 25 Then the validity of the items was assessed in consecutive stages by requesting individual experts to evaluate the content relevance and simplicity of individual items and the entire set of items (questionnaire) as a tool,26 27 followed by the iterative loops of consensus panel revisions. Finally, the validated questionnaire was pilot tested in a sample from the target population.

Consensus panel and expert raters

The consensus panel consisted of the four authors of this paper; two physiotherapists: one a PhD with experience in environmental physiology and validation processes with rating forms, and the other an MSc; and two naval architects: one a PhD with expertise in acceleration and loads on high-speed marine craft, and the other a PhD student with several years working experience in high-speed marine craft design and manufacturing. Initially, 11 expert raters were recruited (see table 1), of whom 8 were male and 3 female from different geographic locations of Sweden.25 27 They were selected on the basis of the methodology described in,24 considering the relevant training and experience, a history of publications in refereed journals, national presentations, clinical expertise and research on the phenomenon of interest. They were convenience chosen and were representative samples of the content domain. Although the mother tongue of the expert raters was Swedish, the language used in the questionnaire was English, with the purpose that the instrument can be used worldwide. Details of the expert raters are presented in table 1.

Table 1

Details of expert raters

Expert	Domain of expertise	Profession
1	HSC human factors engineering	Engineer
2	HSC operations, target group	Coastguard officer
3	HSC operations, target group	Coastguard officer
4	HSC operations, target group	Coastguard officer
5	HSC operations, target group	Armed forces officer
6	Impact and vibration associated health	Researcher
7	Impact and vibration associated health	Researcher
8	Marine medicine, armed forces	Physician
9	Physiotherapy, epidemiology, ergonomics	Researcher
10	Physiotherapy, epidemiology, questionnaire development	Researcher
11	Psychology, questionnaire development	Researcher

HSC, high-speed craft.

Written consent was provided by each expert prior to taking part in the validity assessment. All experts were informed about the non-tractability of their work and the ability to withdraw from the process anytime without providing reasons, conforming to the norms of the ethics committee.28

Content domain

The questionnaire was designed for collection of data on exposures and outcomes in order to investigate health and risk factors for adverse health and performance impairments in high-performance marine craft crews and similar populations. Coastguards were identified as a favourable population for the study since they use several types of vessels for various duties, ranging from patrolling to search and rescue missions in different sea conditions. Both male and female officers with a relatively large age span work at different geographic locations. Swedish Coast Guards were selected as the study population (subjects). Exposure items relate to land-based and sea-based activities and conditions, characterised by their nature, dose and time. Associated health-related outcomes were identified as the current and previous musculoskeletal pain episodes and perceived general, physical and mental health conditions. Owing to the fact that performance outcomes are difficult to measure only by a survey, it was decided to collect items with respect to fatigue symptoms, duration of work at sea, subjective severity of working conditions aboard different types of craft, reasons for reducing craft speed in rough sea conditions, availability of shock mitigation techniques and ergonomics of the craft under a separate subdomain as indirect measures of performance. Items were also collected on the factors that might affect either exposures or outcomes or both, for instance, anthropometrics, demographics, rest, work-demand, work-control and work-support.

Initial questionnaire development

The initial questionnaire was developed by the consensus panel, based on a literature review,29–33 and adjusted to fit the content domain. The questionnaire tool was developed under four subdomains, for example, demography, lifestyle, work exposure and health status. The development of the questionnaire was an iterative process focusing on the content, clarity and comprehensiveness of the final questionnaire.24 The content representation of the items was decided on the basis of their sensitivity to reflect, sample and measure the respective subdomain. This was inspired by the previous questionnaire development work related to musculoskeletal pain34 and fatigue.35 The comprehensiveness of the questionnaire was addressed by mapping the questions into the International Classification of Functioning, Disability and Health (ICF) framework inside the content domain.36 By mapping into the ICF framework, the density of the questions distributed among the subdomains was also maintained at a moderate level.

Validity assessment

The validation process was performed in three consecutive assessment stages. In each stage, experts assessed the questions individually and as a questionnaire tool, with respect to their content relevance and simplicity. Experts were requested to rate each item using two four-point ordinal Likert rating scales quantised as 1=not relevant, 2=somewhat relevant, 3=quite relevant and 4=highly relevant, and 1=not simple, 2=somewhat simple, 3=quite simple and 4=very simple. The scale was dichotomised into agreed (rating of 3 and 4) and not agreed (rating of 1 and 2) and item level content validity was determined as the number of experts giving a rating of 3 or 4 (agreed) divided by the total number of experts. The entire questionnaire tool was assessed using the S-CVI/Ave computed by averaging the I-CVIs (ie, sum of all the I-CVIs divided by the number of items). Items were accepted as content-valid if I-CVI≥0.78 and the questionnaire as a tool was considered to be valid if S-CVI/Ave≥0.90 as recommended by.26 The proportion of items that achieved a rating of 3 or 4 by all the experts, that is, scale content validity index with universal agreement (S-CVI/UA), was also calculated after each stage in order to get an indication about the universal agreement of the experts on the questionnaire as a tool. This measure (S-CVI/UA) was not used to evaluate the questionnaire but to obtain a better depiction of to what extent all the experts agree on all the items in it. The universal agreement was considered ‘good’ if the S-CVI/UA≥0.80 as claimed in.24 37 At the end of each stage, the questionnaire was revised by the consensus panel according to the expert ratings and comments as suggested by.24–26 Items with low I-CVI were either discarded or improved and retained for the next assessment stage. The revised questionnaire was then sent to the experts for the next stage of validity assessment. The language was thoroughly reviewed prior to the third assessment stage.

Active and conscious participation of the experts was checked using a control question added into the questionnaire in the first stage. Experts were considered to be excluded after the first and second stages if an expert had rated the relevance of the control question grade 3 or 4 in the first stage, as well as all the items at the same grade without reasoning and incongruently with the grades of the majority of the experts.24

In each stage, the questionnaire assessment document was emailed to the experts in MS Word format, incorporating all questions and response options with the specific instructions for completion. Each item was followed by separate scales for relevance and simplicity with text boxes for expert comments. Extra questions (open-ended) were included in the assessment document for the expert to provide any additional comments and suggestions on adding or removing questions, the length of the questionnaire and their overall views on the questionnaire as a tool, so as to find out whether all dimensions of the content domain had been addressed. Access to the online questionnaire was provided in the third assessment stage.

Pilot test

Eventually, a pilot test was conducted using 25 respondents randomly selected from the study population to verify the feasibility of the web-based questionnaire.38 The respondents were requested to offer their opinions on the overall questionnaire tool separately after completing the survey.

Results

Details of the validation process are shown in figure 1. The initially generated 38 items were assessed by 11 experts in the first assessment stage, including 33 items for relevance and 19 items for simplicity, obtaining an acceptable level of I-CVI.

Figure 1

Flow chart of the questionnaire development and validation process. I-CVI, item content validity index; S-CVI/Ave, scale content validity index, average; S-CVI/UA, scale content validity index, universal agreement.

Following the experts' assessment, items were revised by the consensus panel and one expert was excluded due to the incongruity of the ratings with other experts (all items had been rated grade four without appropriate reasoning). One expert dropped out of the process, leaving 9 experts in the second assessment stage in which 35 items were rated, revealing 1 item for relevance and 4 items for simplicity below the threshold I-CVI level. In the final stage, 36 items were assessed by the same 9 experts, resulting in 3 items with low I-CVIs, 1 for relevance and 2 for simplicity. In spite of the low I-CVI values, the consensus panel decided to include them in the final questionnaire for reasons discussed in the next section. Table 2 shows the I-CVIs and S-CVI/Ave for all the assessment stages.

Table 2

I-CVIs and S-CVIs for three assessment loops of the validation process

Item Content Validity Index (I-CVI) defined as the proportion of expert ratings higher than two. Cut-off for acceptable I-CVI was 0.78. Scale Content Validity Index, Average (S-CVI/Ave) defined as mean I-CVI.

*Discarded after the first stage.

†Split into four questions after the first stage.

‡Replaced by two standard Alcohol Use Disorders Identification Test (AUDIT) questions after the second stage.

§Merged into endurance exercise question.

¶Converted into a set of demand–control–support questions after the first stage and split into three separate questions (demand, control and support) after the second stage.39

() Control question.

As can be seen in table 2, the S-CVI/Ave increased from 0.89 to 0.96 for relevance and from 0.76 to 0.94 for simplicity, resulting in 36 items in the final questionnaire (see online supplementary material for the final questionnaire).

Discussion

The objective of this study was to develop a questionnaire for investigation of risk factors for adverse health effects and reduced performance among high-performance marine craft crews and similar populations. The validity of the questionnaire was determined with respect to its relevance and simplicity, achieving an S-CVI/Ave of 0.96 and 0.94, respectively.

Although the stepwise assessment procedure used in this study increases the external validity of the questionnaire, it is difficult to compare the obtained S-CVIs with the results of most similar studies, because either they have used S-CVI/UA as the assessment criterion or the methods of computing the S-CVI/Ave are not distinct.26 34 Despite the method, the S-CVI/Ave in this study is considerably greater than that of similar studies discussed in.26

Since work tasks of the target populations are strongly dependent on annual seasons, musculoskeletal pain occurrence was investigated over a time period of 6 months. This time period was considered to account for periodical changes and increase the likelihood of registering rare events. Pain occurrences were separated into single, recurring and constant pain; however, total pain duration was not examined because the relatively long time periods substantially mitigate the accuracy of the estimations due to recall bias. In contrast, average pain intensity was decided to be measured over the previous 7 days, as it was considered relatively stable. This is commonly practised in scientific investigations and was also recommended by many of the expert raters in this study. As suggested by several experts, a set of demand–control–support questions39 has been integrated into the questionnaire, in order to assess how health and well-being are affected by work demand, decision latitude and support, and to facilitate comparison with studies of other occupational groups.

A limitation of this process was that every expert did not represent all the dimensions of the content domain, which led to some experts having difficulties in interpreting the relevance of some items outside their respective expertise. The consensus panel concluded this to be the reason for the item (Item ‘Speed reduction’ in table 2) receiving low I-CVI (0.67) for relevance in the final stage (failed only by one expert). The high grades given by the experts having expertise in the specific domain supported the consensus panel's view on the item's relevance and motivated the decision to keep it in the final questionnaire. This implies that subdomain-wise I-CVI computation could have improved the validity of the individual items as well as the universal agreement level (S-CVI/UA) of the questionnaire.24 Still, the growth of the S-CVI/UA from 0.37 in the first stage to 0.75 in the final stage for relevance, and from 0.16 to 0.64 for simplicity, confirms that eventually the majority of expert raters were satisfied with the content relevance and simplicity of the questionnaire tool, which was a decent level compared with the similar studies discussed in.26 34

Although the power of the validity is limited by the expertise of the consensus panel and the expert raters, it was believed that the method used in this study was the best available for the purpose. Previous studies support the fact that the selection of assessment scales (ie, I-CVI≥0.78 and S-CVI/Ave≥0.90), the number of assessment stages (ie, 3) and the number of expert raters (ie, 9 and 11), as recommended in,24–27 have considerably improved the scrutiny level and diminished the effect of chance agreement on the results.

It was identified that the low simplicity rating (0.33) of the item ‘Duration aboard craft’ (in table 2) in the second stage was due to the complexity in the arrangement of the response options. Dynamic response options were introduced in the online version as a solution, which increased the I-CVI up to 0.67 in the third stage. In contrast, the lower I-CVI obtained by the item ‘Duration of work at sea’ (in table 2) was attributed to the operational difficulties experienced with its dynamic response options provided in the third stage. Significant gain observed in the S-CVI/Ave for simplicity from 0.86 in the second stage to 0.94 in the final stage indicates the cooperative contribution of the consensus panel revisions, online version and the language enhancement to the improvement of simplicity.

The web-based survey is self-administered, and thus the social desirability biases are minimal. It also has dynamic, responsive and active interactions with the respondent such as ability of skipping and routing by delivering questions based on the answers provided to the previous question(s) and indicating the missed questions. A missing data handling mechanism was incorporated in the web survey, which allows the respondent to express unwillingness in responding to any question, thus reducing internal missing data and improving the analysing power. The risk of urge for selecting this response option (ie, ‘I do not wish to answer this question’) has been addressed by making it appear only if a respondent tries to skip a question.

Even after receiving an acceptable level of validity in the third stage, the remaining expert concerns and suggestions with respect to relevance and simplicity and also to the appearance of the web-based version were addressed prior to the pilot test. From the pilot test, it was found that the survey completion time is about 30 min, which is considered to be a reasonable duration for a motivated study population such as coastguards or similar. The pilot test results also justified the decision of the consensus panel to keep the two questions with low I-CVIs for simplicity in the final questionnaire.

In this study, it was difficult to capture performance outcomes only by a questionnaire survey due to the complex attributes of man–machine working conditions interactions. This is also supported by the fact that the three lower-rated items are related to the performance. Moreover, items and test modules can be incorporated in future studies to measure performance outcomes such as cognitive performance, physical performance (man, machine and both together) and biomarkers.

Further studies to evaluate reliability aspects of the questionnaire tool have been initiated and are in progress.

Conclusion

A web-based questionnaire for investigation of health and performance in high-performance marine craft crews was developed, validated and pilot tested. The validation process results revealed an average scale content validity index of 0.96 for relevance and 0.94 for simplicity, and pilot testing suggested a good feasibility of the survey. These factors combined support that the questionnaire is valid for further epidemiological use in high-performance marine craft crews and similar populations.