Development of the Stress and Anxiety to Viral Epidemics-9 (SAVE-9) Scale for Assessing Work-related Stress and Anxiety in Healthcare Workers in Response to Viral Epidemics.

BACKGROUND: The coronavirus disease 2019, or COVID-19, has had a major psychological impact on healthcare workers. However, very few scales are available to specifically assess work-related stress and anxiety in healthcare workers responding to a viral epidemic. This study developed a new assessment tool, the Stress and Anxiety to Viral Epidemics-9 (SAVE-9) and aimed to validate it among healthcare workers directly affected by COVID-19 in Korea.
METHODS: A total of 1,019 healthcare workers responded through anonymous questionnaires during April 20-30, 2020. Exploratory factor analysis (EFA) was conducted to explore the construct validity, and the reliability was assessed using internal consistency measures of Cronbach's alpha coefficients. Receiver operating characteristic analysis was conducted to define the most appropriate cut-off point of SAVE-9 using the Generalized Anxiety Disorder-7 scale (GAD-7; ≥ 5). Second, Spearman's rank correlation coefficient was used to establish convergent validity for the SAVE-9 questionnaire with GAD-7 and the Patient Health Questionnaire-9.
RESULTS: The nine-item scale had satisfactory internal consistency (Cronbach's α = 0.795). It adopted a two-factor structure: 1) anxiety regarding viral epidemics and 2) work-related stress associated with viral epidemics. A cut-off score of 22 for the SAVE-9 ascertained levels of stress and anxiety in response to a viral epidemic in healthcare workers that warranted clinical attention. Correlations between the SAVE-9 and the other scales were statistically significant (P < 0.05).
CONCLUSION: The results suggest that the SAVE-9 is a useful, reliable, and valid tool to evaluate stress and anxiety responses in healthcare workers during viral epidemics.

INTRODUCTION

In December 2019, the first report of the coronavirus disease 2019, or COVID-19, appeared in Wuhan, Hubei, China.1 The outbreak spread to countries around the world within a few months, and the World Health Organization (WHO) declared it a global pandemic. In South Korea, the first patient was discovered on January 20, 2020, and the number of patients increased quickly, exceeding 5,000 within six weeks. Outbreaks are still occurring in clusters within churches, hospitals, and crowded facilities such as call centers, entertainment venues, and logistics centers. As of June 2021, there have been 149,700 confirmed cases of COVID-19 in Korea.2

Amid this outbreak, healthcare workers are experiencing high levels of psychological distress, as they are at high risk of being exposed to the infectious disease and are worried about infecting their family members.34 In addition, healthcare workers face anxiety about unprecedented and unfamiliar clinical roles resulting from the pandemic, including the perceived burdens of increased workloads, reluctance to work, perceived stigmatization, the need to avoid crowds and colleagues, and the feeling of being scrutinized.5 During previous pandemics, such as those of severe acute respiratory syndrome (SARS), influenza A/H1N1, and the Middle East respiratory syndrome (MERS), healthcare workers also experienced severe emotional stress, including anxiety, depressive symptoms, and insomnia.678 Moreover, during this COVID-19 pandemic, symptoms of acute stress disorder or posttraumatic stress disorder have also been observed.

During contagious epidemics like SARS and COVID-19, healthcare workers are adversely affected psychologically, often experiencing exhaustion and emotional distress that subsequently affects their ability to respond effectively in emergency situations.39 The COVID-19 epidemic has thus far lasted for almost a year since the WHO declared it a pandemic and has evolved into a prolonged stressor for many healthcare workers. While it is essential to care for both physical and psychological health during such a pandemic,10 it is rare that the psychological health of healthcare workers is routinely and systematically assessed during these emergencies. This may be due in part to a lack of validated screening tools that apply specifically to healthcare workers and their unique situation. In recent studies on psychological assessment of healthcare workers, researchers have used the typical rating scales, such as items from the Patients Health Questionnaire-9 (PHQ-9), The Depression, Anxiety and Stress Scale-21 Items (DASS-21), Hospital Anxiety and Depression Scale (HADS), or the Generalized Anxiety Disorder-7 (GAD-7) to assess the mental health of healthcare workers.11 Though those scales are reliable and validated rating scales, we believed that a rating scale specific to the viral epidemic rather than a general one for frontline healthcare workers was needed. To date, very few studies have examined psychological assessment tools assessing the stress and anxiety of healthcare workers specifically in response to viral epidemics. To the best of our knowledge, only one study developed a 72-item questionnaire for healthcare workers during the MERS outbreak. However, the length of the questionnaire was impractical to implement during an epidemic, thus making it difficult to compare with other scales and establish validity.12

In this study we aimed to develop a new rating scale called the Stress and Anxiety to Viral Epidemics-9 (SAVE-9) scale, which is specific to the viral epidemic and can be applied for healthcare workers, and explore its reliability and validity among healthcare workers sample.

METHODS

Participants and procedure

This study was conducted at Asan Medical Center, a large university hospital located in Seoul, Korea. This survey was conducted via on-line survey system. Employees of Asan Medical Center were recruited via the hospital's internal communication server through an advertisement, which employers logged into everyday to monitor fever or respiratory symptoms. We developed a survey form using Google Forms® (Google LLC, Mountain View, CA, USA) and presented the link to access the survey in the intranet. In addition, we announced that name, work unit or any personal or identifiable information will not be collected. The study was completed voluntarily and no specific reward for the participation was provided.

To ensure the quality of the survey, we used detailed explanations of the questionnaire to encourage participants to answer carefully. A total of 1,023 employees, including doctors, nurses, and other administrative employees, responded to the survey, and 1,019 consented to participate and were used in the final analyses. The study was conducted during April 20–30, 2020.

Development of the stress and anxiety to viral epidemics scale

We developed the scale under the following framework: Step 1 – Defining the intended purpose of the survey instrument; Step 2 – Item generation; Step 3 – Item reduction; Step 4 – Psychometric testing and further item reduction; and Step 5 – Final revision of the prototype into a useable survey instrument.13

During the first stage of scale development, we reviewed previous studies on the psychological impact on healthcare workers during the SARS, MERS, and Ebola outbreaks.81415161718192021 The main objective is to develop a viral epidemic-specific anxiety rating scale, and the main target was set as frontline healthcare workers. Items were generated based on literature review, item review of existing questionnaires,12 expert consultation of three psychiatrists, and focus group interviews with doctors and nurses. Based on the literature review and review of the existing questionnaires, researchers (S.C, YW.S, and MH.A) gathered the potential features constituting the scale; anxiety or fear of the virus, somatic or physical symptoms, depression, insomnia, contagion, stigmatization, work-related stress or burnout. After consulting experts in this field, we decided to exclude insomnia, one of important variables affecting anxiety symptoms, to prevent the direct influence of shift-working pattern of healthcare workers on this scale, depression to make this scale specific to anxiety response, and include concerns about their own health and the health of their family members. The 1 session of the focused group interview (FGI) with 3 doctors and 3 nurses who were working in the hospital was conducted to explore their work-related stress in COVID-19 pandemic. After one hour duration of FGI, we newly added the potential feature of their worry about colleagues' attitudes for their abstinence.

Strong consideration for brevity and accessibility of the scale was a main determining factor in final item retention, resulting in 10 items that assessed: A) Are you afraid the virus outbreak will continue indefinitely? B) Are you afraid your health will worsen because of the virus? C) Are you worried that you might get infected? D) Are you more sensitive toward minor physical symptoms than usual? E) Are you worried that others might avoid you even after the infection risk has been minimized? F) Do you feel skeptical about your job after going through this experience? G) After this experience, do you think you will avoid treating patients with viral illnesses? H) Do you think exposure to the virus is a result of not being careful? I) Do you worry that your family or friends may become infected because of you? J) Do you think that your colleagues would have more work to do due to your absence from a possible quarantine and that they might blame you? Respondents rated their agreement with each item on a five-point Likert scale ranging from 0 (never) to 4 (always).

Measures

Patient Health Questionnaire-9 (PHQ-9)

The PHQ-9, a self-administered, nine-item questionnaire, assesses depression symptoms. Each item is scored on a three-point Likert scale (0 = not at all to 3 = nearly every day). Scores can range from 0 to 27, with higher scores reflecting greater symptom severity (0–4 = minimal depression, 5–9 = mild depression, 10–14 = moderate depression, 15–19 = moderately severe depression, and ≥ 20 = severe depression).22

GAD-7

The GAD-7 is a self-administered, seven-item questionnaire assessing general anxiety. Each item is scored on a three-point Likert scale (0 = not at all to 3 = nearly every day). Scores can range from 0 to 21, with higher scores reflecting higher levels of anxiety. The questionnaire was originally developed by Spitzer and his colleagues,23 who recommended using a cut-off score of 10 for identifying cases of Generalized Anxiety Disorder. The cut-off intervals for anxiety were as follows: 0–4 = minimal, 5–9 = mild, 10–14 = moderate, and 15–21 = severe. For the current study, we used a cut-off of 5 to identify cases associated with anxiety. We used the Korean version of GAD-7 in this study.

Statistical analysis

Demographic information and clinical characteristics of the sample were summarized using descriptive statistics. Statistical analysis was conducted using SPSS version 21.0 for Windows (IBM Corp., Armonk, NY, USA) and JASP version 0.14.1. The level of significance for all of the analyses was defined as two-tailed, P < 0.05. First, to determine the factor structure of the scale, exploratory factor analysis (EFA) was conducted with principal axis factor extraction methods and promax rotation. Normality assumption for the factor analysis was checked using skewness and kurtosis with an acceptable range of ± 2.24 Sampling adequacy and data suitability were examined using the Kaiser-Meyer-Olkin (KMO) measure and Bartlett's test of sphericity. The retained number of factors was determined via a screen test with a rule of an eigenvalue greater than 1.0. The reliability was assessed using internal consistency measures of Cronbach's alpha coefficients. Second, Spearman's rank correlation coefficient was used to evaluate the correlation between the SAVE-9 and other scales (GAD-7 and PHQ-9). Wilcoxon's rank sum tests were conducted to explore SAVE-9 levels according to anxiety levels using GAD-7 and PHQ-9. Next, receiver operating characteristic (ROC) analysis was conducted to explore the appropriate cut-off point of the SAVE-9. Finally, the additional value (i.e., detection of those who were not screened through GAD-7) of the SAVE-9 was estimated through a McNemar test.

Ethics statement

The study protocol was approved by the Institutional Review Board (2020-0580) of the Asan Medical Center. The study was conducted according to the criteria set by the Declaration of Helsinki, and obtaining of informed consent was waived by the Board.

RESULTS

Demographic information

A total of 1,019 healthcare workers working at the Asan Medical Center consented to participate in the study. Table 1 shows the demographic characteristics of the participants: 598 (58.7%) were nursing professionals, 808 (79.3%) were female, 529 (51.9%) were single, and 129 (12.7%) had experienced or received treatment for either depression, anxiety, or insomnia. The proportion of participants in their 20s (n = 309, 30.3%), 30s (n = 387, 38.0%), and 40s and above (n = 253, 24.8%) were similar. The mean duration of employment was 9.9 ± 9.0 years.

Table 1

Demographic characteristics of participants (n = 1,019)

Variables		Values
Healthcare workers
	Medical doctors	192 (18.8%)
	Nursing professionals	598 (58.7%)
	Other healthcare workers	229 (22.5%)
Sex (female)		808 (79.3%)
Age, yr
	20–29	309 (30.3%)
	30–39	387 (38.0%)
	40–49	253 (24.8%)
	50–59	70 (6.9%)
Marital status
	Single	529 (51.9%)
	Married	482 (47.3%)
	No response	8 (0.8%)
Did you experience or receive treatment for depression, anxiety, or insomnia? (Yes)		129 (12.7%)
Years of employment, yr		9.9 ± 9.0
Symptom assessments
	Patient Health Questionnaire-9 (PHQ-9)	5.0 ± 4.6 (0–27)
	Generalized Anxiety Disorder-7 (GAD-7)	3.7 ± 4.0 (0–21)
	Stress and Anxiety to Viral Epidemics-9 (SAVE-9)	20.3 ± 5.7 (2–36)

Factor structure of the SAVE-9

EFA was conducted to identify the factor structure of the scale. The normality assumption was checked using skewness and kurtosis, and it revealed that the distribution of items was within normal limit (Table 2). Sampling adequacy and data suitability was checked using KMO measure (0.857) and Bartlett's test of sphericity (P < 0.001). Among the 10 items that were initially included, one item (“Do you think exposure to the virus is a result of not being careful?”) was excluded due to its low factor loading (< 0.40). Among 9 items retained, the eigenvalue plot and scree test showed evidence for two factors. Eigenvalues for the first three factors were 3.51, 1.07, and 0.88, respectively, and we adopted a two-factor structure based on eigenvalue greater than one rule. Factor I was labelled “Anxiety about the epidemic” (items 1, 2, 3, 4, 5, and 8), and Factor II was labeled “Work-related stress associated with the epidemic” (items 6, 7, and 9; Table 2). Cronbach's alpha for the final nine items was 0.795. Also it is very similar to Cronbach's alphas if each item was deleted (0.757–0.791).

Table 2

Score of items on the stress and anxiety to viral epidemics-9 (n = 1,019)

Items		Responses to items					Skewness	Kurtosis	CID	Mean ± SD	Factors
		Never	Rarely	Sometimes	Often	Always					I	II
Factor 1. Anxiety about the viral epidemic										14.22 ± 4.20
	Item 1. Are you afraid the virus outbreak will continue indefinitely?	21 (2.1%)	82 (8.0%)	166 (16.3%)	568 (55.7%)	182 (17.9%)	−0.934	0.899	0.773	2.79 ± 0.90	0.621	0.507
	Item 2. Are you afraid your health will worsen because of the virus?	42 (4.1%)	213 (20.9%)	286 (28.1%)	400 (39.3%)	78 (7.7%)	−0.315	−0.621	0.757	2.25 ± 1.00	0.808	0.546
	Item 3. Are you worried that you might get infected?	36 (3.5%)	199 (19.5%)	257 (25.2%)	438 (43.0%)	89 (8.7%)	−0.405	−0.569	0.763	2.34 ± 1.00	0.789	0.411
	Item 4. Are you more sensitive towards minor physical symptoms than usual?	47 (4.6%)	200 (19.6%)	178 (17.5%)	424 (41.6%)	170 (16.7%)	−0.461	−0.721	0.768	2.46 ± 1.12	0.708	0.418
	Item 5. Are you worried that others might avoid you even after the infection risk has been minimized?	210 (20.6%)	442 (43.4%)	176 (17.3%)	160 (15.7%)	31 (3.0%)	0.589	−0.454	0.774	1.37 ± 1.07	0.534	0.519
	Item 8. Do you worry your family or friends may become infected because of you?	17 (1.7%)	51 (5.0%)	122 (12.0%)	550 (54.0%)	279 (27.4%)	−1.101	1.625	0.770	3.00 ± 0.86	0.674	0.501
Factor II. Work-related stress associated with the viral epidemic										6.08 ± 2.28
	Item 6. Do you feel skeptical about your job after going through this experience?	252 (24.7%)	406 (39.8%)	178 (17.5%)	139 (13.6%)	44 (4.3%)	0.657	−0.382	0.788	1.33 ± 1.12	0.338	0.743
	Item 7. After this experience, do you think you will avoid treating patients with viral illnesses?	82 (8.0%)	288 (28.3%)	244 (23.9%)	345 (33.9%)	60 (5.9%)	−0.127	1.625	0.786	2.01 ± 1.09	0.371	0.589
	Item 9. Do you think that your colleagues would have more work to do due to your absence from a possible quarantine and might blame you?	28 (2.7%)	115 (11.3%)	161 (15.8%)	504 (49.5%)	211 (20.7%)	−0.808	0.182	0.791	2.74 ± 1.00	0.396	0.429

Cronbach's alpha for the final nine items was 0.795.

CID = Cronbach's alpha if item is deleted, SD = standard deviation.

Evidence based on relations to other variables

Spearman's correlation coefficient was used to explore convergent validity for the SAVE-9 against GAD-7 and PHQ-9 scales. The results indicated positive correlations between SAVE-9 scores and GAD-7 (rho = 0.51, P < 0.001) and PHQ-9 (rho = 0.41, P < 0.001) scores, respectively. The SAVE-9 scale score was significantly higher among healthcare workers who were rated as having depressive symptoms (PHQ-9 ≥ 10, [t(1,017) = 10.36, P < 0.001) and generalized anxiety symptoms [GAD-7 ≥ 5, t(1,017) = 14.22, P < 0.001]. The SAVE-9 scale total score was significantly higher among women [t(1,017) = 7.01, P < 0.001] compared to men and higher among nursing professionals [t(1,017) = 7.61, P < 0.001] compared to other healthcare workers.

Cut-off score for the SAVE-9

We performed ROC analysis to explore the appropriate cut-off score of the SAVE-9 total score and Factor I subscale score in accordance with “at least a mild degree of anxiety” by a GAD-7 score of 5. Based on the results from the ROC analysis, the appropriate cut-off point for the SAVE-9 was calculated as ≥ 22 (area under the curve, AUC = 0.748, sensitivity = 0.67, specificity = 0.68) in accordance with a GAD-7 score of 5 points. In addition, the appropriate cut-off for the first subscale (Factor 1), “Anxiety about the Epidemic” (GAD-7 ≥ 5) was ≥ 15 (AUC = 0.728, sensitivity = 0.72, specificity = 0.61). The SAVE-9 score was normally distributed, which contrasted with the distribution of GAD-7 (Fig. 1A). The GAD-7 had higher skewness (1.41) and kurtosis (1.94), compared to the SAVE-9 (skewness -.18, kurtosis .1). Participants were classified into three groups based on the GAD-7 (GAD-7 score of 0, 1–4, and ≥ 5) to compare subgroups of individuals based on severity scores on a traditional measure. The SAVE-9 total score (Fig. 1B) and Factor I (anxiety subscale) score (Fig. 1C) were normally distributed and well discriminated.

Fig. 1

The distribution of SAVE-9 scale score. (A) The difference in the distribution of GAD-7 and SAVE-9 scales scores, (B) The distribution of SAVE-9 total score among participants with each GAD-7 severity scores, (C) The distribution of SACE-9 anxiety subcategory scores (Factor I) among participants with each GAD-7 severity scores.

SAVE-9 = stress and anxiety to viral epidemics-9, GAD-7 = generalized anxiety disorder-7.

SAVE-9 scores based on anxiety and depression levels

The total scores for the SAVE-9 were significantly different among groups when participants were classified into three groups based on anxiety levels (GAD-7 score of 0, 1–4, and ≥ 5) (Table 3, Fig. 1B) or depression levels (PHQ-9 scores of 0, 1–9, and ≥ 10; Table 3). While the GAD-7 is widely used, the distribution of GAD-7 scores was negatively skewed in the general population (Fig. 1A). A closer look at the score distribution revealed that 24.1% of the study sample reported a zero on the total GAD-7 score, and 50.6% reported less than two. In contrast, the dispersion of SAVE-9 scores was similar to a normal distribution with a slightly long left tail (Fig. 1A). SAVE-9 scores increased with GAD-7 scores (Fig. 1B). A similar pattern was observed while using the SAVE-9 anxiety subscale score (Fig. 1C). Among cases that were classified as “high anxiety” individuals using the GAD-7 total score (GAD-7 > 5), 329 (32.3%) were screened for further evaluation. Results indicated that 232 (22.8%) individuals who were categorized as having significant stress and anxiety based on the SAVE-9 score ≥ 22 were not screened by the GAD-7 (kappa = 0.32, P < 0.001; Fig. 2A). Similarly, when we defined a Factor I score ≥ 15 as experiencing anxiety in response to the viral epidemic, 272 (26.7%) workers were not screened by the GAD-7.

Table 3

SAVE-9, Factor I, and Factor II scores among groups based on GAD-7 and PHQ-9 scores

Variables	GAD-7				PHQ-9
	0 (n = 246)	1–4 (n = 444)	≥ 5 (n = 329)	P value	0 (n = 172)	1–9 (n = 689)	≥ 10 (n = 158)	P value
SAVE-9 total score	16.6 ± 5.1	19.9 ± 4.9	23.6 ± 5.0	< 0.001	16.7 ± 5.7	20.3 ± 5.1	24.4 ± 5.3	< 0.001
Factor I score	11.7 ± 3.9	14.0 ± 3.8	16.4 ± 3.8	< 0.001	11.8 ± 4.4	14.3 ± 3.8	16.7 ± 4.1	< 0.001
Factor II score	4.9 ± 2.0	5.9 ± 2.0	7.2 ± 2.3	< 0.001	4.9 ± 2.1	6.0 ± 2.1	7.7 ± 2.4	< 0.001

SAVE-9 = stress and anxiety to viral epidemics-9, GAD-7 = generalized anxiety disorder-7, PHQ-9 = patient health questionnaire-9.

Fig. 2

Comparisons between SAVE-9 and GAD-7 scales. (A) SAVE-9 scale and GAD-7 scales, (B) SAVE-9 anxiety subscale and GAD-7 scale.

SAVE-9 = stress and anxiety to viral epidemics-9, GAD-7 = generalized anxiety disorder-7.

DISCUSSION

The aim of the current study was to develop and validate a questionnaire to measure stress and anxiety specifically in healthcare workers responding to a viral epidemic. The developed questionnaire, the SAVE-9, was confirmed to have good reliability and convergent validity with existing stress and anxiety measures. It was found to have two factor structures: Factor I - anxiety about the viral epidemic and Factor II - work-related stress associated with the viral epidemic. The SAVE-9 score and subscales were significantly correlated with GAD-7 and PHQ-9 scores. The ROC analyses showed that a score of 22 on the SAVE-9 and 15 on the anxiety subcategory were appropriate cut-off points. Additionally, the SAVE-9 was able to identify 232 (22.8%) healthcare workers with anxiety regarding the viral epidemic who were not identified by the GAD-7 cut-off, proving its incremental validity in measuring healthcare workers' stress and anxiety in response to viral epidemics.

For scale development, we reviewed previous studies on the psychological impact of other outbreaks8152526 on healthcare workers. Based on our clinical experience and literature review, we developed 10 items to reflect fear of the virus, contagion, stigmatization, anxious responses, job reluctance, and worry about colleagues' attitudes. Internal consistency was found to be satisfactory based on the Cronbach's alpha of the final SAVE-9 (r = 0.795). The SAVE-9 score was significantly correlated with typical anxiety (GAD-7) and depression (PHQ-9) rating scales.

Scores on both factors were significantly correlated with those on the GAD-7 and PHQ-9. The item 5 “Are you worried that others might avoid you even after the infection risk has been minimized?” loaded on both Factor I (loading = 0.534) and Factor II (loading = 0.519). Healthcare workers suffer from depression or anxiety during pandemics due to social isolation, loss of social support, and worry about infecting their friends or families.27 In some communities, neighbors who live adjacent to healthcare workers worry about them being contagious. Healthcare workers, who may be vulnerable to infection due to their workplace, may experience stress regarding the safety of their family members. Therefore, while item 5 could be classified into both Factor 1 and 2, we included it under Factor I because it loaded higher on Factor 1 and because it was identified as a stressor for healthcare workers through expert interviews.

The SAVE-9 was developed to assess the anxiety responses of healthcare workers in response to a viral epidemic. When developing a psychological support system for healthcare workers, we need to screen them for their psychological state to identify which individuals warrant clinical attention. The SAVE-9 scale, which was designed to be brief and practical, could be used to identify individuals who need additional psychological support. We observed that 329 (32.3%) participants among our sample were classified as having at least a mild degree of anxiety (GAD-7 score ≥ 5). In addition, we screened 232 (22.8%) healthcare workers as having scores above the threshold on the SAVE-9 indicating those who may need additional psychological support. In general, scales for depression and anxiety, including the PHQ-9 or GAD-7, are widely used to assess participants' psychological status. However, such scales do not reflect the psychological stress specifically caused by viral epidemics. Therefore, the developed scale has merit in that it can assess the psychological state of healthcare workers that is specific to a situation such as the COVID-19 pandemic and add incremental validity to already existing measures. Additionally, considering that psychological distress during an epidemic takes a toll on the productivity and effectiveness of a healthcare worker, the proportion of individuals identified through the developed scale is alarming and warrants further study and intervention.

Furthermore, we believe that Factor I (anxiety subcategory) of the SAVE-9 can be independently used for evaluating anxiety in non-healthcare workers related to viral epidemics. The appropriate cut-off score of Factor I was determined as 15 with good sensitivity and specificity, and the subscale had good convergent validity with already existing anxiety scales. Future studies should explore the utility of the six items of Factor I for the general population. In this study, Factor II was clustered as work-related stress related to the viral epidemic. Unfortunately, as we did not include a rating scale for the work-related stress of healthcare workers in the current study, we could not explore the validity of Factor II. Further studies are needed to assess the convergent validity of Factor II of the SAVE-9.

While this study has merit, there are limitations in interpreting its results. This study's first limitation is that it was carried out in just one hospital in South Korea. The SAVE-9 needs to be validated in other cultures, where the response to an epidemic may be vastly different based on location. The second limitation is that we did not account for biases related to an anonymous online survey because the sampling of our study was voluntary. Although face-to-face clinical interviews would have been helpful in validating the SAVE-9, there were significant barriers to interviewing workers during the pandemic situation. The online survey also disrupted the important clinical interview or further evaluation of participants. For example, we asked participants about their past psychiatric history with the question “Did you experience or receive treatment for depression, anxiety, or insomnia?” However, we could not evaluate their treatment history in further detail. This study was conducted when healthcare workers were working hard on the frontline of COVID-19, and it is not easy to evaluate them in detail face-to-face. Moreover, we could not assess the work units (for example, emergency department, intensive care unit, or general outpatient clinics) of healthcare workers in this study. In the beginning of the outbreak of COVID-19 in a hospital, all healthcare workers must support and help each other in caring for confirmed or suspicious patients. It is not easy to differentiate their duties and work unit clearly because all patients were a risk in regard to the viral epidemic.

There were also limitations in the process of scale development. Insomnia is an important symptom when assessing one's depression or anxiety symptoms. At first, we considered including insomnia symptoms during scale development, but we decided not to include insomnia because we needed to consider the shift-work patterns of healthcare workers. Insomnia symptoms are associated with shift-work patterns, and the scale could thus be directly influenced by shift work rather than the viral epidemic. The lack of a specific statement of the duration of anxiety symptoms in response to the viral epidemic (for example, within recent weeks or months) in the SAVE-9 scale may be another limitation. However, we plan to develop a rating scale that users can use freely for their purposes or the duration they want to assess. Though we assessed the mental health of healthcare workers within the most recent 2 weeks, users can define the duration for their purposes.

In conclusion, the SAVE-9 is a reliable and valid rating scale for assessing the anxiety and work-related stress of healthcare workers. The scale measures two dimensions-anxiety about the viral epidemic and work-related stress associated with the viral epidemic. The first subscale can be useful as an independent measure when used with individuals who are not healthcare workers but are impacted by a viral epidemic. We expect that the SAVE-9 will be useful for screening the psychological state of frontline healthcare workers and that it can be suitably applied for developing psychological support systems for healthcare workers during this pandemic era.