Design and validation of a questionnaire to measure the attitudes of hospital staff concerning pandemic influenza.

BACKGROUND AND
OBJECTIVE: When pandemics lead to a higher workload in the healthcare sector, the attitude of healthcare staff and, more importantly, the ability to predict the rate of absence due to sickness are crucial factors in emergency preparedness and resource allocation. The aim of this study was to design and validate a questionnaire to measure the attitude of hospital staff toward work attendance during an influenza pandemic.
METHOD: An online questionnaire was designed and electronically distributed to the staff of a teaching medical institution in the United Kingdom. The questionnaire was designed de novo following discussions with colleagues at Imperial College and with reference to the literature on the severe acute respiratory syndrome (SARS) epidemic. The questionnaire included 15 independent fact variables and 33 dependent measure variables. A total of 367 responses were received in this survey.
RESULTS: The data from the measurement variables were not normally distributed. Three different methods (standardized residuals, Mahalanobis distance and Cook's distance) were used to identify the outliers. In all, 19 respondents (5.17%) were identified as outliers and were excluded. The responses to this questionnaire had a wide range of missing data, from 1 to 74 cases in the measured variables. To improve the quality of the data, missing value analysis, using Expectation Maximization Algorithm (EMA) with a non-normal distribution model, was applied to the responses. The collected data were checked for homoscedasticity and multicollinearity of the variables. These tests suggested that some of the questions should be merged. In the last step, the reliability of the questionnaire was evaluated. This process showed that three questions reduced the reliability of the questionnaire. Removing those questions helped to achieve the desired level of reliability.
CONCLUSION: With the changes proposed in this article, the questionnaire for measuring staff attitudes concerning pandemic influenza can be converted to a standardized and validated questionnaire to properly measure the expectations and attendance of healthcare staff in the event of pandemic flu. Copyright Â

Introduction

When pandemics lead to a higher workload in the healthcare sector, the attitude of healthcare staff and, more importantly, the ability to predict the rate of absence due to sickness are crucial factors in emergency preparedness and resource allocation. Pandemic flu is one of the most important pathogens that cause outbreaks of disease, and it has a high risk of spreading rapidly because of the airborne nature of its dissemination. To identify the attitudes of healthcare professionals regarding work attendance during such an outbreak, a questionnaire was designed de novo following discussions with colleagues at Imperial College and with reference to the literature on severe acute respiratory syndrome (SARS) [1], [2], [3], [4]. The choice of pandemic influenza was pragmatic because preparations were underway to prepare for an expected pandemic in the near future. To aid planning for business continuity, an understanding of staff attitudes toward pandemic flu is invaluable. Pandemic flu was compared with SARS, the most recent example of a global infectious disease that spread quickly around the world and affected patients and healthcare staff. Published literature was consulted to determine the lessons that had been learned. The questions for the questionnaire were empirically chosen based on the assumption that generic issues arising from the way the SARS outbreak was managed would be relevant to the management of an influenza pandemic. Subsequently, questions from a Department of Health (DH) questionnaire about pandemic flu that asked respondents about the distance of their residence from the hospital were combined with this questionnaire.

The individual questions were empirically formulated based on experience and consensus views. The questionnaire consists of 15 independent fact variables and 33 dependent measure variables. The fact variables focus on personal and work-related items and the respondents’ contact history with infectious diseases. The measured dependent variables focus on the prospect of work attendance in the case of a pandemic flu, factors that may affect work attendance and respondents’ expectations of the trust (i.e., public organization providing services on behalf of the National Health Service in England [5]) in the case of such events.

This study was conducted to ensure the reliability and validity of the questionnaire. The questionnaire was electronically advertised to the staff of Imperial College Healthcare hospitals in London. Senior managers were also asked to distribute the questionnaire to their staff. Because this study addresses a sensitive subject (attitudes about work absence), the responses were collected anonymously.

Material and methods

The questionnaire was evaluated through the four steps listed below:

Face validation: In this step, we ensured that the respondents’ understanding of the questions was aligned with our goals.

Pilot data preparation: In this step, we ensured that response bias was minimized, and we investigated the possibility of predicting missing data.

Content validation: In this step, we verified that the question items targeted the aim of the study.

Content reliability: In this step, we investigated the relevance of the question items.

In the face validation step, the questionnaire was evaluated from the design point of view. None of the respondents reported problems understanding the content of the questionnaire. The options for scaling questions consisted of 6 items that were compatible with the recommendation by Fowler [6]. The items were labeled from negative (strongly disagree) to positive (strongly agree). Labeling the items on a scale using minimal descriptive words helped to eliminate confusion about the value of the scale levels [7].

The next step was data preparation. First, we analyzed the pilot data for outlier responses. The risk of extravagant respondents is a possibility in any survey, especially when the survey is anonymous. To identify the outliers, we used three methods: standardized residuals, Mahalanobis distance and Cook's distance. After removing the outlier responses, we scanned the data for missing values. Missing values are a major factor in reaching valid conclusions. There are methods for calculating these values that remove the effect of hidden bias in the data. We used Expectation Maximization Algorithm (EMA) to calculate the missing values [8].

The next step in the evaluation focused on content validity. There are two methods for measuring the validity of a questionnaire. The first method involves comparing the result with a highly valid measure, such as the work attendance of the participants, to exclude the effect of confounding factors. The second method uses the level of correlation between the dependent and independent variables [9]. The first method was not possible because of the anonymous nature of the study; thus, we applied the second method. To ensure content validity in the second method, the questionnaire should be checked for two issues. The first issue ensures that the independent variables have a minimum level of correlation with the measured dependent variables. This procedure was used to identify irrelevant questions. The second issue involves the identification of highly correlated independent variables. If there is a high degree of correlation between these variables, the questionnaire should be checked again. If possible, it is recommended that these questions be merged, resulting in a shorter questionnaire with the same level of validity.

The last step involved checking the reliability of the questionnaire. There are two methods to measure reliability: test–retest and internal consistency [9]. Because of the anonymous nature of the study, it was impossible to utilize the test–retest method; therefore, the internal consistency of the questionnaire was calculated. This test can identify variables that may reduce the inter-correlation between the question variables. If possible, omitting these variables improves the consistency of the questionnaire.

Results

This questionnaire was published online, and 367 responses were recorded in the system. The actual response rate could not be calculated because the questionnaire was electronically advertised, and all employees of the hospital may have been exposed to the study. The responders covered a wide range of hospital professions. The relative frequencies of the occupations are presented in Table 1 .

Table 1

Relative frequency of job in participants.

Job	Percentage
Admin and clerical	0.27%
Allied to medicine	15.26%
Ancillary staff	19.35%
Consultant	66.67%
Training doctors	2.72%
Manager	1.91%
Midwife	11.17%
Nurse	26.16%
Pharmacist	0.27%
Scientist	7.36%
Technician	4.63%
Other	4.36%
Unspecified	0.27%

We received responses from different age groups and both genders, as shown below in Table 2, Table 3 .

Table 2

Relative frequency of age group in participants.

Age group	Percentage
18–25	4.90%
26–35	31.88%
36–45	28.61%
46–55	22.34%
56–65	11.44%
Unspecified	0.83%

Table 3

Relative frequency of gender in participants.

Gender	Percentage
Female	75.48%
Male	22.07%
Unspecified	2.45%

Regarding work and employment status, we received responses from a variety of groups, as presented in Table 4, Table 5 .

Table 4

Relative frequency of work status in participants.

Work status	Percentage
Full-time	86.92%
Part-time	12.81%
Unspecified	0.27%

Table 5

Relative frequency of employment status in participants.

Employment Status	Percentage
Bank/agency staff	1.36%
Contract staff	3.82%
Permanent staff	94.55%
Unspecified	0.27%

Answers to three negatively worded questions were mirrored with their positive values. These questions involved the adverse effect of dependency on public transportation and concerns about personal and family health on work attendance in the event of pandemic flu.

Regarding the outlier detection methods, the threshold level for the standardized residual was ±3.3. The degree of freedom for the Mahalanobis Distance test was 29, which resulted in a critical value of 58.3 for the alpha level of 0.001. Using these outlier detection methods, 19 cases (5.17%) were identified as outliers and were excluded from the final analysis.

Analysis of the responses showed a different range of missing data in the survey questions, as presented in Table 6 .

Table 6

Top 10 variables with the most missing values in the questionnaire.

Question Item	Number of missing values
My ability to come to work depends on effective elderly care arrangements	74
My ability to come to work depends on effective child care arrangements	58
If staff develop flu, they should be quaranteened and sent home	34
If staff develop flu, they should be quaranteened and treated in the hospital	29
If staff develop flu, the trust should provide anonymized updates on the numbers of staff affected by flu	26
My ability to come to work depends on use of public transport	25
If staff develop flu, there should be eating and social restrictions on all staff to prevent cross infection	24
If staff develop flu, there should be rapid access counseling for staff during the epidemic	23
If there were cases of pandemic influenza in the trust, I would like the provision of health monitoring	17
If there were cases of pandemic influenza in the trust, I would expect provision of personal protective equipment for all staff who have direct contact with all patients	12

By applying the EMA method to the data, the missing data were calculated. Because of the non-normal distribution of the data, Student's t distribution was used in this calculation instead of the normal distribution likelihood function. A maximum of 25 iterations was assigned for this algorithm. The missing values were predicted by this method.

The residual scatterplot shows a roughly rectangular shape, which supports the homoscedasticity of the results (Fig. 1 ).

Figure 1

Scatterplot for the comparison of standardized predicted values vs. standardized residuals to identify outliers.

The normal P–P plot of regression standardized residual (Fig. 2 ) shows a reasonably straight diagonal line from the bottom left to the top right, which supports the linearity of the pilot data obtained by this questionnaire. The above analysis shows the homogeneity of variance between the variables measured in the questionnaire. This homogeneity is required to measure the correlation between the variables.

Figure 2

Normal P–P plot of regression standardized residual in flu questionnaire to measure deviation from normality.

Evaluating the correlation between the independent (fact) question items and the dependent (measure) variables showed a correlation above 0.3. This correlation assures that there are no irrelevant independent variables in the questionnaire.

The effect of identifying flu cases in London or single or multiple cases in the trust on the probability of work absence showed a high multicollinearity. Furthermore, the items “Expect rapid access to diagnosis” and “Expect rapid access to treatment” were highly correlated. The other two correlated question items were “Expectation of personal protection equipment for traveling to work on public transport” and “Dependency on public transport for ability to come to work”. The existence of multicollinearity in the questionnaire variables can falsely overfit the regression models that are developed using the data collected with that questionnaire.

Cronbach's alpha test was used to measure the reliability of the questionnaire, and the initial result of this test was 0.507. This result is below the minimum acceptable value (0.7). Further evaluation of the questions showed that the items “Years of working in the current job”, “Years of working in National Health Service (NHS)” and “Working time if part-time” caused this low reliability. The result of the same test without these question items increased the reliability to 0.712.

Discussion

The data collected through this survey represented a wide range of professions and age groups from both genders and employment states. The missing value analysis using the EMA method helped to improve the data and to handle missing values. We aimed to generate a validated questionnaire that provides the required information with the lowest number of questions. The questionnaire was validated using various processes and statistical methods, as discussed in the “Results” section. Based on these results, some changes were suggested to improve this questionnaire.

We merged three questions on the effect of flu infection in London or single and multiple cases in the trust on the probability of work absence because of the high correlation between these questions. A high correlation between the expectation for rapid diagnosis and the expectation for rapid treatment also supported merging these two items as “Expectation for rapid diagnosis and treatment”.

To generalize this questionnaire, we recommended using the term “your city” instead of “London”, which was the pilot city.

Although the dependency on public transportation and the expectation of protection equipment for this service were highly correlated, these two question items point to separate concepts. We retained both of the questions in the questionnaire for this reason and for the purpose of controlling the responses.

Repeating the Cronbach's alpha test without the “Years of working in the current job”, “Years of working in National Health Service (NHS)” and “Working time if part-time” showed that the questionnaire had reached the required level of reliability. Although the removed items seemed to be related to the concept of the study in the design phase, the results from the pilot study showed that they should be omitted from the questionnaire. The final version of the questionnaire is included in Appendix 1.

Conclusion

This pilot study of a questionnaire designed to measure the attitude of healthcare staff about a flu outbreak resulted in some improvements in the questions used. The pilot study demonstrated that the items in the questionnaire are relevant to the subject, the questionnaire has an acceptable level of type I error and reliability and the questionnaire has been optimized to obtain the desired information with the fewest questions.

Overall, these measures show that the questionnaire can be used as a standardized and validated measurement of healthcare staff's expectations and attendance in the event of pandemic flu. However, there are limitations to this study. First, the questionnaire was electronically advertised, but it is not clear whether all employees had access to computers in the hospital, which could be a source of bias for the staff who completed the survey. Second, the actual response rate could not be calculated because the questionnaire was electronically advertised, and all of the hospital employees may have been exposed to the study. We are also unable to comment on the non-responders.

We recommend that this questionnaire be assessed in a multi-center study with a larger sample size to increase the reliability of the results and to establish a score model for the questionnaire. Furthermore, an investigation of other confounding factors using qualitative methods, such as soft system methodology (SSM) [9], could potentially improve this questionnaire.

Conflicts of interest

None.

Funding

None.

Competing interests

None declared.

Ethical approval

Not required.