Stress and burnout in health care workers during COVID-19 pandemic: validation of a questionnaire.

Aim: To validate a questionnaire to assess stress and burnout in healthcare workers during COVID-19 pandemic. Subjects and methods: In this study, content validity, Cronbach's alpha, and test-retest reliability method were utilized among 60 HCWs to evaluate the validity, internal consistency, and reliability of the questionnaire respectively. The final questionnaire was composed of four parts asking for the background information, questions about the stress caused by the COVID-19, the Depression, Anxiety and Stress Scale - 21 (DASS-21), and six questions from the Copenhagen Burnout Inventory (CBI).
Results: The CVR of 46 questions was equal to 1, making them acceptably valid (CVR > 0.78), so that the items could be arranged into a final questionnaire. Moreover, all items could successfully attain CVI values above 0.79, confirming the content validity of the questionnaire. The Cronbach's alpha was between 0.80-0.95 for different sections of questionaire, confirming the stable reliability and high repeatability of the questionnaire.
Conclusion: The results of this study showed that the DASS-21 offers adequate levels of validity and reliability for assessing the stress, anxiety, and depression among the HCWs engaged with the COVID-19 pandemic. Moreover, the six items adapted from the Copenhagen burnout inventory (CBI) were found to provide a good instrument for investigating the job burnout among the HCWs at Masih Daneshvari Hospital during the outbreak of the COVID-19 epidemic. © Springer-Verlag GmbH Germany, part of Springer Nature 2020.

Introduction

Since late December 2019, the outbreak of a new coronavirus (COVID-19, formerly known as nCoV-2019) was reported in Wuhan, China. As of April 4th, 2020, the disease had spread into more than 60 countries around the world, with more than 1 million cases of infected patients.1 Based on the data from 72,314 cases, 14% and 5% of the patients are undergoing serious and critical conditions respectively, with an overall mortality rate of 2.3% (Wu and McGoogan 2020). Table 1 contains the stats on the positive cases and the disease-caused mortalities in Iran by April 4th, 2020.

Table 1

Iran’s reported laboratory-confirmed COVID-19 cases and deaths. Data as of 4 April 2020

Country	Total confirmed cases	Total deaths	Transmission classification
Iran	53,183	3294	Local transmission

Local transmission indicates locations where the source of infection is within the reporting location.

In general, COVID-19 is an acute fatal disease that may end up with death due to progressive respiratory complications (Huang et al. 2020; Wu et al. 2020). This is the seventh coronavirus that has been proved to infect humans (Zhou et al. 2020; Zhu et al. 2020). However, there are only limited data on the clinical traits and natural evolution of the disease among the patients engaged with COVID-19 (Huang et al. 2020).

Healthcare workers (HCWs) are exposed to various infectious diseases, including those transmitted through the blood or other body fluids and/or airborne infectious agents (Shiao et al. 2007). Studies have shown that the group of HCWs who are in direct contact with the patients are exposed to highest levels of risk (World Health Organization 2003). Nurses are particularly vulnerable to many job-related hazards, and undergo a considerable amount of emotional pressures in relation to their jobs (Wheeler 1997). This is even more important during a pandemic outbreak of an infectious disease on a global scale, and can lead to stress and job burnout among the HCWs (Maunder et al. 2006).

A primary challenge in the COVID-19 pandemic has been to take care of infected patients, which defines a serious risk to the workers at intensive care units (ICUs) because of the need for constant contact with infected patients for extended periods of time. Widespread infection and fatalities among the HCWs have been reported previously for MERS and SARS and currently for the COVID-19 disease, imposing considerable amounts of social and mental pressures on the workers. During the outbreaks of SARS and MERS, the HCWs reported concerns about themselves and their families’ health and explained their painful experiences of fear, anxiety, and even social prejudice and stigma (Almutairi et al. 2018). Moreover, it has been shown that HCWs tend to continue to undergo the job burnout, mental pressure, anxiety, and depression symptoms even upon the mitigation of the outbreak (Lancee et al. 2008). Previous researches have also shown that post-traumatic stress disorder (PTSD) is pretty common among the survivors of infectious diseases (Hong et al. 2009).

By definition, burnout syndrome (BOS) refers to the experience of fatigue for extended periods of time and reduced levels of motivation and interest in the job, which lead to decreased job productivity. This is a result of too much effort in the workplace with limited opportunities for recovery (Embriaco et al. 2007; Goldberg et al. 1996). Studies have shown that highly stressful jobs are more likely to cause job burnout (Embriaco et al. 2007). When the focus is on the nurses, job burnout may lead to less willingess to undertake leadership, lower quality of medical care, reduced satisfaction level of hospitalized patients, increased level of healthcare-related infections, and higher mortality rates among the patients (Dyrbye et al. 2019). Previous studies have further indicated that nurses suffering job burnout are more likely to be dissatisfied with their jobs and to look for opportunities to leave their jobs (Dyrbye et al. 2019).

HCWs are very often susceptible to job burnout, with the highest levels of job burnout reported among HCWs working in the emergency and ICU environment where they are exposed to an overwhelming amount of job-related stress (Elshaer et al. 2018). Known factors contributing to the high risk of job burnout include intensive patient care, high mortality rate, and inappropriate job conditions in terms of high workload coupled with lack of time to adequately address the patient’s needs. Accordingly, the relevant HCWs experience stress levels beyond their capacities, which may lead to job burnout. Job burnout has been reportedly accompanied by reduced quality of care provided to patients and by a high rate of absence and turnover among the HCWs, all of which is known to cause irrecoverable consequences for the healthcare sector (Bakker and Le Blanc 2005; Embriaco et al. 2007). The nurses in ICUs are further exposed to high-risk, unpredictable, and highly variable conditions. Being engaged with a handful of diseases, highly traumatic incidents, and emergency conditions, such nurses are provided with limited time for recovery and rather experience job-related pressures constantly. Accordingly, reports indicate that the nurses in ICUs are likely to experience high levels of job burnout compared to their counterparts in other units of a hospital (Kim and Choi 2016).

Job burnout not only puts the individual’s health and wellbeing at risk, but also is associated with the frequency of medical errors and quality of the healthcare services. Therefore, it is of paramount importance to identify the factors contributing to the job-related stress and burnout among the HCWs. Such knowledge can be devised to protect and preserve the workforce while improving the quality of services provided to the patients. As of now, numerous studies have been performed to investigate the states of stress and job burnout among the HCWs at hospitals, including the nurses and doctors, with a handful of questionnaires, in original language, available in this respect. However, given the epidemic outbreak of the novel COVID-19 virus and the unpredictable conditions and extensive workloads experienced by HCWs in relation to this epidemic, the present research was performed to design a questionnaire, and further evaluate its validity and reliability, for investigating stress and burnout among the HCWs at Masih Daneshvari Hospital during the COVID-19 pandemic.

Methodology

In this study, content validity, Cronbach’s alpha, and the test–retest reliability method were utilized to evaluate the validity, internal consistency, and reliability of the questionnaire respectively. Various steps of this study are explained in the following subsections.

Questionnaire design and preparation

In order to design and prepare the questionnaire, firstly, the questions were designed based on existing references, experts’ ideas, and studies performed on job-related stress and burnout, and the implications for HCWs resulting from the novel COVID-19 virus. The final questionnaire was composed of four parts asking for background information, questions about the stress caused by the COVID-19, the Depression, Anxiety and Stress Scale - 21 (DASS-21), and six questions from the Copenhagen burnout inventory (CBI) that were used to evaluate the job burnout among nurses in previous studies (Estryn-Behar et al. 2008).

The DASS-21 was used to measure the stress. This questionnaire was developed by Lovibond in 1995 to measure the stress, anxiety, and depression through a total of 21 items (Lovibond and Lovibond 1995). The DASS-21 is composed of three sub-scales, each of which encompasses seven questions, with a final score for each scale obtained by summing up the scores of the relevant questions. For each item, the score is an integer number between 0 (it does not apply to me at all) and 3 (it applies to me exactly). Since the DASS-21 represents a summarized form of the original scale (with 42 items), the final score of each sub-scale must be doubled. Table 2 illustrates the severity ratings for different subscales of DASS-21.

Table 2

DASS-21 severity ratings for different subscales

Severity	Depression	Anxiety	Stress
Normal	0–9	0–7	0–14
Mild	10–13	8–9	15–18
Moderate	14–20	10–14	19–25
Severe	21–27	15–19	26–33
Extremely severe	28 +	20 +	33 +

The questionnaire was initially translated to Persian followed by asking English linguists to translate the Persian questions back to English. Subsequently, certified translators were consulted to address the gaps between the original questionnaire and the back-translated version. The validity of this questionnaire for the Iranian population was previously undertaken by Sahebi et al. in 2005 (Sahebi et al. 2005), but we preferred to revalidate it through the translation and back-translation processes to ensure its compliance with the conditions established by the COVID-19 virus pandemic.

In order to evaluate job burnout, six questions out of the CBI, whose adequacy, validity, and reliability were previously confirmed by the NEXT study, were utilized (Estryn-Behar et al. 2008). With regard to these questions, a score between 1 (never/almost never) and 5 (every day/almost every day) was assigned to each item. Eventually, a final score below 2.99 indicates no job burnout, while any score beyond 3.00 was interpreted as engagement with job burnout (Estryn-Behar et al. 2008). Similarly to the procedure used with the DASS-21, the selected items from the CBI were translated to Persian and then back-translated to English before having them sent to experts, together with the other parts of the final questionnaire, for investigating their validity and reliability. Table 3 presents examples of the items in different parts of the questionnaire.

Table 3

Examples of the items in the questionnaire on the stress and job burnout among the HCWs at Masih Daneshvari Hospital during the COVID-19 epidemic

The six-item CBI
	Almost never	Once or a few times per month	Once or twice a week	Four or five times a week	Almost every day
- How often are you physically exhausted? - How often are you emotionally exhausted?
DASS-21
Please read each statement and circle a number 0, 1, 2 or 3 which indicates how much the statement applied to you over the past week.	Not at all	Mildly	Moderately	Highly
- I found myself getting agitated - I felt scared without any good reason
COVID-19-related items
- How much do you worry about your family members and friends getting sick as a result of your exposure?	Not at all	Mildly	Moderately	Highly	Extremely
- How much do you worry about getting infected by Covid-19?

Content validity assessment

Qualitative and quantitative approaches were followed to evaluate the content validity of the questionnaire. Moreover, in order to assess the qualitative validity of the questionnaire, the items were provided to nine experts in relevant disciplines from the Shahid Beheshti University of Medical Science and the Masih Daneshvari Hospital, to submit their corrective ideas regarding the grammar and wording of the items. It is worth noting that the quantitative content validity assessment was performed based on the content validity ratio (CVR) and content validity index (CVI). In this respect, for the CVR, the group of experts were asked to evaluate each item as either “necessary”, “useful but unnecessary”, or “unnecessary”; finally, the answers were summarized to calculate the CVR (DeVon et al. 2007). Being dependent on the number of participating experts (here nine), the acceptable threshold was computed as 0.78 for this study (Lawshe 1975). In this work, the CVI was evaluated by averaging the CVR over all items. Any CVI above 0.79 would confirm the content validity of the respective scale.

Internal consistency and repeatability of the final questionnaire

In this study, in order to assess the external reliability of the questionnaire, it was completed by 60 HCWs at the Masih Daneshvari Hospital, and the test–retest methodology was used with a period of 14 days to measure the correlation coefficient. The test–retest method has been used to measure the repeatability in many research projects. In this methodology, the considered test is conducted twice on the same sample at two different times separated by a relatively short period; the results of the two tests are then analyzed to calculate a correlation coefficient. The internal reliability of the items of the questionnaire was assessed based on an internal consistency factor (Cronbach’s alpha) that varied between 0 and 1.

Findings

Once finished with preparing an initial version of the questionnaire with a total of 49 items and confirming the face validity of the questionnaire, the CVR of three questions from the researcher-made questionaire was less than 0.78, and they were excluded from the questionnaire. The CVR of 46 questions was equal to 1, making them acceptably valid (CVR > 0.78), so the items were arranged into a final questionnaire. Moreover, all items could successfully attain CVI values above 0.79, confirming the content validity of the questionnaire. Bartlett’s test and the Kaiser–Meyer–Olkin (KMO) index were respectively used to assess the appropriateness and adequacy of the sample. As shown in Table 4, the value of the KMO index was acceptable for all aspects of the questionnaire, while the Bartlett values were significant, indicating significant relations among the variables.

Table 4

Relevance and reliability of the questionnaire data

Section	Bartlett	KMO index	Sig.
DASS-21	915.657	0.903	0.000
Burnout	327.334	0.880	0.000
COVID-19 related items	88.069	0.811	0.000

In this project, the external reliability and repeatability of the questionnaire were assessed through the test–retest method and the inter-cluster correlation (ICC) coefficient respectively. To this end, the questionnaire was distributed among and completed by 60 HCWs at the Masih Daneshvari Hospital before being collected by the author. After 2 weeks, the questionnaires were distributed among and completed by the same 60 HCWs. Analyzed by the SPSS 25 software, the collected data showed a Cronbach’s alpha between 0.80–0.95 for different sections of the questionaire, confirming the stable reliability and high repeatability of the questionnaire. Table 5 provides the values of the Cronbach’s alpha and ICC for different parts of the questionnaire.

Table 5

Internal consistency and stability

Section		Interclass correlation coefficient		Cronbach’s Alpha
		95% Confidence Interval
		Lower bound	Upper bound
DASS-21	.953	.934	.969	0.953
Burnout	.942	.916	.962	0.942
Covid-19 related items	0.803	0.711	0.872	0.803

Discussion

The outbreak of the COVID-19 coronavirus in Iran and around the globe is probably the most challenging threat to human health at both national and international levels during the past few decades, greatly affecting the medical communities. In such a public health crisis, HCWs must not only put greater efforts into their activities for extended working hours, but also suffer from less-than-enough human knowledge about such an unforeseen emerging condition. In addition, the constant use of personal protection equipment (PPE) adds to the physical fatigue and mental pressures on the HCWs (Al-Rabiaah et al. 2020). In general, the outbreak of an emerging disease contributes to a general atmosphere of fear that needs to be psychologically studied through comprehensive research activity to understand its possible negative impacts on individuals’ mental health and productivity, so as to mitigate such impacts on the HCWs, in particular, who are in the front line of counteracting the disease. At this time, providing the HCWs with adequate support in terms of considering their conditions and presenting solutions, increasing their awareness, encouraging them, and acknowledging their importance, tends to reinforce them against the disease.

During the outbreak of the severe acute respiratory syndrome (SARS) epidemic in 2003 and the H1N1 pandemic in 2009, researchers evaluated the psychological stress on HCWs through various methods and observed high levels of stress among HCWs. Other emotional stress-inducing factors that were highly common among HCW patients included exaggerated concerns about the health of self or their families, fear, and a sense of distress for working at a hospital (Mobaraki and Ahmadzadeh 2019; Sim and Chua 2004). Studies on the outbreak of the MERS-CoV in 2014, which caused a great amount of anxiety among people in the infected countries, showed that the healthcare community were undergoing extreme levels of stress, as previously published data indicated that this group of people were likely to be exposed to the highest levels of infection risk. It was also found that 30% of the total cases of the epidemic were HCWs, with most of the infection transmitted within the hospital premises (Memish et al. 2014; Oboho et al. 2015).

Today, with the development of research in various sciences, many phenomena are studied through questionnaire-based surveys to quantify the behavior of the studied individuals or societies. Given the large number of studies performed on the basis of questionnaires, it is essential to study the validity and reliability of such instruments (Cook and Beckman 2006; Drost 2011). According to a review on the existing literature, no single questionnaire for investigating the conditions established upon the pandemic outbreak of COVID-19 and its impacts on the stress and job burnout among the HCWs was found, and this was why the present questionnaire was designed.

Conclusion

Respecting the necessity of the research on health care workers (HCWs) involved with the treatment of COVID-19 patients, it was crucial to have an appropriate yet short enough questionnaire for such a purpose. The presented questionnaire was a combination of two existing inventories and a couple of the researcher-constructed items for COVID-19, in particular, which was validated by the researchers for assessing the anxiety, stress, depression, and burnout among HCWs. The results of this study showed that the DASS-21 offers adequate levels of validity and reliability for assessing the stress, anxiety, and depression among HCWs engaged with the COVID-19 pandemic. Moreover, the six items adapted from the Copenhagen burnout inventory (CBI) were found to provide a good instrument for investigating job burnout among the HCWs at Masih Daneshvari Hospital during the outbreak of the COVID-19 epidemic.