Development of the "COVID-19 psychological impact Scale": A validity and reliability study.

This scale development study aimed to identify the level of COVID-19 impact on the psychology of adults during or after the pandemic. The study group consisted of 1604 individuals for the first stage and 597 individuals for the second stage, who were reached online through convenience sampling method. After the exploratory factor analysis performed on the 50-item pilot survey, a three-factor structure was obtained explaining 60.96% of the total variance. The load values of the items in the "Dejection" sub-dimension were between 0.65 and 0.82, in the "Anxiety" sub-dimension between 0.59 and 0.77 and in the "Frustration" sub-dimension between 0.70 and 0.76. In the reliability analysis, the Cronbach Alpha value for the whole scale was found to be 0.93, and the Cronbach Alpha values for the subscales were found to be Dejection = 0.94, Anxiety = 0.81 and Frustration = 0.83. For similar criterion validity, the survey was applied to 597 new participants and the Pearson Moment Correlation Coefficient between the "COVID-19 Psychological Impact Scale" and the "Depression Anxiety Stress Scale" was found to be 0.71. A confirmatory factor analysis was carried out with the new data set from 597 participants and the goodness of fit index values obtained showed that the model was acceptable and/or perfect compatible. As a result of the research, it has been revealed that the "COVID-19 Psychological Impact Scale" is a valid and reliable scale applicable to adults aged 18 and over.

Introduction

COVID-19 emerged in Wuhan, China, as an unidentified acute respiratory infection. It was highly contagious and was declared a pandemic as a result of its transmission to most of the world through travels between cities and countries (Guan et al., 2020). It spread rapidly, significantly influencing the world and the lives of societies in all aspects (Zhu et al., 2020). The most common of these effects is the anxiety of getting sick seen in the individuals living in societies. The anxiety of contracting the disease creates uneasiness and fear of contracting the virus as the number of cases increases in the individual’s environment. This fear may lead the inability to fulfil daily functions and routines to be disrupted and cause to have a completely dysfunctional and helpless attitude (Ahorsu et al., 2020; Pakpour & Griffiths, 2020).

Fear of getting infected with the virus also leads to impaired cognitive function of individuals and to experiencing depression, stress, and anxiety. It can be said that individuals who experience these fears intensely cannot think and act in a healthy way to protect themselves against the coronavirus (Ahorsu et al., 2020). Moreover, individuals who fear contracting the virus may even attempt suicide (Goyal et al., 2020). Individuals, who do not know when the pandemic will end and when they will go back to their daily routines, have uncertainty about the future. This uncertainty may cause individuals to experience pessimism and avoid thinking about their future expectations. The COVID-19 pandemic has also profoundly affected the psychological health of people (Wang et al., 2020). Serious psychological disorders such as psychosis, mania, depression, anxiety, insomnia, fatigue, and catatonia (physical and mental withdrawal) are seen in this population (Lai et al., 2020). The most common psychological disturbance in this population is depressive disorders, which is a reflection of dejection (Qiu et al., 2020). As can be understood, COVID-19 appears to have a serious psychological impact on individuals.

Due to compulsory isolation and restrictions during COVID-19, people’s social relationships decreased, the feeling of loneliness and boredom increased, an intense loss of freedom was experienced, and sense of satisfaction from activities decreased (Holt-Lunstad, 2017). All these show that the physical and social limitations imposed during the pandemic has a psychological impact on individuals. Studies showed that quarantine processes had negative effects on people’s psychological well-being (Liu et al., 2012; Sprang & Silman, 2013). In a study on the effect of isolation duration on the psychological health of patients, it was observed that post-traumatic stress disorder (PTSD) significantly increased in patients who were hospitalized for a long period of time (Day et al., 2013). In another study, it was revealed that personnel who were quarantined had more anxiety, feeling of burnout, nervousness, attention problems, indecision, decrease in work performance and unwillingness, insomnia, and indecision compared to those who were not quarantined (Samantha et al., 2020). In another study in which the psychological impact of isolation was examined, it was found that individuals who were isolated felt imprisoned, limited, stigmatized, anger, anxiety, depression, low self-esteem, and impulsivity (Davies & Rees, 2000). In a survey study conducted in China, participants evaluated the psychological impact of the pandemic. 53.8% of the participants were revealed to be psychologically affected by the epidemic at a moderate or severe level (Wang et al., 2020).

While global attention is focused largely on the effects of coronavirus on physical health, the effects of coronavirus on psychological health cannot be ignored (Satici et al., 2020). Therefore, assessment tools are needed to reveal the psychological impact of COVID-19. When the studies in the literature are examined, development of fear of COVID-19 scale (Ahorsu et al., 2020) and COVID-19 anxiety scale (Lee, 2020) draw attention. Apart from the studies in which the aforementioned scales were adapted to different languages Ahmed et al., 2020; Bitan et al., 2020; Chi et al., 2021; Doshi et al., 2020; Evren et al., 2020; Haktanir et al., 2020; Magano et al., 2021; Martinez et al., 2020; Reznik et al., 2020; Soraci et al., 2020) there are also studies in which different scales have been developed on the subject (Arpaci et al., 2020; Nikčević & Spada, 2020; Riad et al., 2020). Examining the psychological impact of COVID-19 on individuals will contribute to the process of creating road maps for how they will experience psychological distress. This research is therefore important. In this study, it was aimed to develop a scale to identify the level of psychological impact that COVID-19 has on individuals. It is predicted that the developed scale will contribute to the scale development studies made in the literature, as well as helping individuals to improve their psychological health.

Method

This research was a scale development study aimed to identify the level of COVID-19 psychological impact on the adults during or after the pandemic. The stages of the process were explained below.

Study Group

The data was collected online in April 2020, when COVID-19 pandemic was intense. Utilizing convenience sampling method, one of the non-probability sampling methods, 1604 participants were reached from Turkey. 66.4% (n = 1065) of the participants were female and 33.6% (n = 539) were male. In terms of marital status, 53.7% (n = 861) of the participants were single while 46.3% (n = 743) were married. The mean age of the participants aged 18 or above was 29.7.

Scale Development Process

At first, an application was made to the Scientific Research and Publication Ethics Board of a University in Turkey for ethics committee report. Domestic and foreign literature Ahorsu et al., 2020; Arpaci et al., 2020; Lee, 2020; Nikčević & Spada, 2020; Repisti et al., 2020; Riad et al., 2020) was reviewed to create an item pool, and the scales that may be relevant were examined. According to the findings of the literature review, an item pool was created. Also, 10 adults representing the sample were asked the question of “What is the psychological impact of COVID-19?“ Based on the data obtained from this study and in line with the theoretical framework, an item pool consisting of 68 items was created. Item pool was presented to 10 experts (2 psychology faculty members, 3 psychological counselling and guidance faculty members, and 5 educational sciences faculty members) for expert opinion. The experts were asked to score the items between 1 and 10 points. After the expert review, 18 items were removed as they were not fit for purpose. The remaining item pool consisting of 50 items was examined by a language specialist in terms of language use, and suggested corrections were made. The 50-item pilot survey, which was ready for application, was applied to adults. As a result of the exploratory and confirmatory factor analysis performed on the data, the scale, which took its final form with 24 items, was applied to a different group of 597 people together with the Depression Anxiety Stress Scale to test for criterion related validity.

Data Collection Tools

Depression Anxiety Stress Scale: The scale was developed by Lovibond & Lovibond (1995), and the reliability and validity tests of the Turkish short form of the Depression Anxiety Stress Scale were carried out by Yilmaz et al., (2017). The reliability tests showed that the reliability coefficients of the items were between 0.75 and 0.82. The Cronbach’s Alpha coefficients of the scale dimensions were 0.81, 0.82, and 0.75, respectively. According to the result of the confirmatory factor analysis, the goodness of fit index values revealed that the model in the scale had an acceptable fitness. In this study, the reliability coefficients of the scale were found to be 0.87, 0.82, and 0.89, respectively.

Analysis of Data

SPSS 22 and Lisrel software packages were used for data analysis. When the sample size required to perform factor analysis on the data collected within the scope of the study was examined, it was seen that the sample size of 1604 was adequate (Tabachnick & Fidell, 2007). Outliers were examined in each of the scale items and convergent items, and no pattern was identified. For the normality assumption, the skewness was found to be 0.37, while the kurtosis was found to be -0.46. and it was seen that the condition of having skewness and kurtosis values between ± 3 was met, which is the normal distribution indicators of the data (Shao, 2002). After testing the suitability of the data for factor analysis, exploratory and confirmatory factor analysis was carried out. While applying EFA and CFA in the test development process, different versions are applied, but when the sample size is sufficient, it is recommended to apply EFA with half of the data and CFA with the remaining half. Since the sample size was sufficient in this study, it was decided to apply EFA to half of the data and CFA to the remaining half.

Exploratory factor analysis was performed to using the varimax method, the Cronbach’s Alpha values were examined for the reliability coefficient of the scale, the fit index values were examined for the confirmatory factor analysis, and the Pearson Moments Correlation Coefficient statistical technique was used for criterion-based validity. In the interpretation of the correlation coefficients obtained, the value ranges suggested by Evans (1996) were used (moderate relationship between 0.40 and 0.59; strong relationship between 0.60 and 0.79; very strong relationship between 0.80 and 1.00).

Results

Validity

Findings Regarding the Structural Validity of the Scale

In the exploratory factor analysis, the suitability of the data set for factor analysis was evaluated before determining the construct validity. In this direction, first Kiaser-Meyer-Olkin (KMO) and Barlett Sphericity Test were performed. In the exploratory factor analysis, the KMO value regarding the adequacy of the sample size takes a value between 0 and 1 and gives an idea about the adequacy of the sample size as it approaches 1. KMO value between 0.80 and 0.90 indicates that the sample size is good, and a higher KMO than 0.90 indicates that the sample size is adequate (Field, 2009). The KMO value obtained for compliance with factor analysis was found to be 0.98. The KMO value obtained showed that the sample size is adequate for factor analysis. Also, as a result of Bartlett’s Test of Sphericity [X2 = 18271.498; p <.00] the value obtained was found to be statistically significant (Tabachnick & Fidell, 2007).

Exploratory Factor Analysis

In order to determine the factor structure of the psychological impact scale, exploratory factor analysis (EFA) was carried out first. In determining the items to be included in the exploratory factor analysis, the eigenvalues should be at least 1.00 and the item factor loadings should be at least 0.30 and above, and the difference between the item load values in the items in two factors should be at least 0.10 and each factor should have at least three items (Buyukozturk, 2012; Kline, 1994). In line with the criteria listed, 26 items were removed from the scale and a 24-item survey was obtained. The scree plot of the factor structure of the scale was given in Fig. 1.

Fig. 1

Scree plot of psychological impact scale

When Fig. 1 was examined, it was seen that the curve became a straight line after the 3rd point, which indicated that the structure has three factors. In addition to these, the varimax vertical rotation method was used because the correlation between the factors of the scale was low, i.e. independent from each other (DeVellis, 2014).

The purpose of vertical rotation is to try to minimize the number of variables with high load value on the factor. What is expected as a result of the rotation process is to enable the factors to be represented strongly by ensuring that each of the variables in the factors is strongly loaded on a factor (Pallant, 2007). The results obtained using the varimax vertical rotation method in the study were given in Table 1.

Table 1

Exploratory factor analysis results of psychological impact scale

ItemsIn the process of the Covid-19 outbreak;	Anxiety	Frustration	Dejection
I’m afraid of getting sick	0.77
I’m worried about what will happen to me if I get sick	0.73
The thought of ‘what if I have the disease’ worries me.	0.77
I am investigating whether I have the symptoms of the disease.	0.73
I do compulsive things (excessive cleaning…) not to get infected.	0.60
I feel sorry for not being able to meet people.		0.76
I get bored of always doing the same things		0.76
It makes me sad that I’m not able to do the things I want to do.		0.71
It bothers me to stay away from many things due to worrying about getting sick		0.71
I feel very tired.			0.65
I don’t feel like doing anything.			0.67
I am having difficulty concentrating on what I have to do.			0.76
I feel restless, nervous, and overwhelmed.			0.74
I have difficulty understanding what I read, watch, or listen to.			0.76
I have problems in my relations with the people around me.			0.76
I have difficulty in starting something.			0.76
Whatever I do, I can’t relax.			0.67
I get sad and cry more easily than before.			0.72
I think that I cannot overcome the difficulties of life.			0.78
I even have difficulty doing my daily routine work.			0.81
It is difficult for me to bear the stress caused by this uncertainty.			0.70
My joy of living is decreasing day by day.			0.82
I think I don’t look after myself/care about myself very much compared to the past.			0.71
I feel that life is so pointless.			0.74
Eigenvalue	1.55	2.31	10.76
Variance %	6.46	9.64	44.85	60.96

As seen in Table 1, as a result of the exploratory factor analysis, the communalities ratio of each item in a common factor was examined and it was observed that it took strong values between 0.51 and 0.72. As a result of the exploratory factor analysis, 24 items and three sub-dimensions (anxiety, frustration, and dejection) were formed. The load values of the items in the “Anxiety” sub-dimension were found to be between 0.59 and 0.77, the items in the “Frustration” sub-dimension, between 0.70 and 0.76, and the items in the “Dejection” sub-dimension, between 0.65 and 0.82. As a result, a three-factor structure that explains 60.96% of the total variance was obtained.

In order to examine the item validity of the COVID-19 Psychological Impact Scale, item total correlation values were examined and it was found that the item total correlation values of the scale ranged between 0.51 and 0.72. In evaluating the item-total correlation, items that have a value above 0.30 are accepted as an indicator that the related items are sufficient to distinguish the feature to be measured, and they are compatible with the whole scale (Buyukozturk, 2012; Field 2009). In line with these values, it could be stated that there was a high level of relationship between the total score obtained from the scale and the scale items, and that the validity condition was met.

It is stated that the correlation value obtained in terms of identifying the relationship between the factors of the COVID-19 Psychological Impact Scale and the multicollinearity problem between the sub-dimensions should be over 0.90 (Field, 2009; Pallant, 2007). Correlation values between sub-dimensions were given in Table 2.

Table 2

Correlations between the sub-dimensions of the COVID-19 psychological impact scale

	1	2	3
1. Anxiety	1	0.31	0.39
2. Frustration		1	0.21
3. Dejection			1

Table 2 showed the correlation values between the sub-dimensions of the COVID-19 Psychological Impact Scale. In line with these data, it revealed that there was a statistically significant relationship between the sub-dimensions of the scale and there was no multiple correlation problem.

Confirmatory Factor Analysis

Data were collected from a different group of 597 individuals for the Confirmatory factor analysis of the COVID-19 Psychological Impact Scale. Of the 597 participants aged 18 or above, 69.3% (n=414) were female, 30.7% (n=183) were male; 73.2% (n=437) were single and 26.8% (n=160) were married. The mean age of the participants was 28.7. The model fit of the two-factor structure obtained as a result of the exploratory factor analysis of the COVID-19 Psychological Impact Scale was examined by first and second-level factor analysis; the first-level factor analysis was given in Fig. 2.

When Fig. 2 was examined, it was seen that the fit indexes of the COVID-19 Psychological Impact Scale, which consisted of 24 items and three sub-dimensions, were significant (χ2 =915.22 sd/df =247, p <.01, χ2/sd=3,705, ≤ 5= Moderate fit) (Kline, 2005; Sumer, 2000). The values of fit indices of the scale were found to be RMSEA =0.07, RMR=0.07, SRMR=0.06, NFI=0.96, NNFI= 0.97, PNFI=0.90, CFI=0.97, IFI=0.97, RFI=0.96, GFI=0.90 and AGFI=0.86. In line with the first level CFA, it can be said that the fit levels of the model are acceptable and/or perfect (Schumacker & Lomax, 2004). After the confirmatory factor analysis, a modification was made among the items in line with the suggestions provided. Between the 7th item (I feel sorry for not being able to meet with people) and the 9th item (It makes me sad that I’m not able to do the things I want to do) and between the 22nd item (My joy of living is decreasing day by day) and the 24th item (I feel that life is so pointless) modifications were made due to their proximity to each other. Both items (7th and 9th) explained from a different perspective that the participants could not do what they wanted to do during the covid-19 outbreak. Due to forced isolation and restrictions, people’s social relationships decreased, the feeling of loneliness and boredom increased, an intense loss of freedom was experienced, and sense of satisfaction from activities decreased (Holt-Lunstad, 2017). Both items (22th and 24th) were related to the feelings of meaninglessness and hopelessness that are often experienced during the covid-19 epidemic. During the Covid-19 epidemic, the most common psychological discomfort in the masses were depressive disorders, which included emotional states such as meaninglessness and hopelessness (Qiu et al., 2020).

Fig. 2

First level CFA result of the COVID-19 psychological impact scale

T values between items and factors were analysed after standard solutions. The significance level was accepted as the criterion at the level of 0.05 if T value exceeded 1.96, and at the level of 0.001 if it exceeded 2.56 (Simsek, 2007). It was found that the T values of all items had significance at the 0.001 level.

While applying confirmatory factor analysis, it is recommended to make second-level factor analysis of multi-factor models (Meydan & Sesen, 2011). The second level CFA results related to this two-dimensional model obtained were shown in Fig. 3.

Fig. 3

Second level CFA Result of the COVID-19 psychological impact scale 

When Fig. 3 was examined, it was seen that the fit indices of the structure consisting of 24 items and three sub-dimensions were significant (χ2 =915.22 sd/df =247, p <.01, χ2/sd=3.705, ≤ 5= Moderate compliance) (Sumer, 2000). The values of fit indices of the scale were found to be RMSEA =0.07, RMR=0.07, SRMR=0.06, NFI=0.97, NNFI= 0.97, PNFI=0.90, CFI=0.98, IFI=0.98, RFI=0.96, GFI=0.89 and AGFI=0.86. In line with the second level CFA, it can be said that the fit levels of the model are acceptable and/or perfect (Schumacker & Lomax, 2004).

Criterion-Related Validity

In order to examine the criterion-related validity of the COVID-19 Psychological Impact Scale, its relationship with the Depression Anxiety Stress Scale was examined and given in Table 3.

Table 3

The relationship between COVID-19 psychological impact and depression anxiety stress

		Overall DAS Scale	Depression Sub-Dimension	Anxiety Sub-Dimension	Stress Sub-Dimension
COVID-19 Psychological Impact Scale	Pearson’s r	0.71	0.65	0.54	0.66
	p-value	0.00	0.00	0.00	0.00
	N	597	597	597	597

According to Table 3, it could be stated that there was a positive and strong (r = .71) significant relationship between the scores of the COVID-19 Psychological Impact Scale and the general scores of the Depression Anxiety Stress Scale (p <.05). It could be stated that there was a positive and strong correlation (r = .65) between the scores of the COVID-19 Psychological Impact Scale and the Depression sub-dimension, a positive and moderate correlation (r = .54) with the Anxiety sub-dimension, and a positive and strong correlation (r = .66) with the Stress sub-dimension (p< .05) (Evans, 1996). As a result of the findings, it can be stated that the COVID-19 Psychological Impact Scale meets the criteria validity conditions.

Reliability Analysis

In order to examine the reliability of the COVID-19 Psychological Impact Scale, the Cronbach Alpha internal consistency coefficient was examined. Accordingly it was concluded that the Cronbach Alpha value for the overall scale was 0.94. Cronbach Alpha values for the sub-dimensions of the scale were found to be 0.81 for the anxiety sub-dimension; 0.83 for the frustration sub-dimension, and 0.94 for the dejection sub-dimension. In scale development and adaptation studies, a Cronbach Alpha value is acceptable above 0.70, good between 0.80 and 0.90, and perfect above 0.90 (DeVellis, 2014; Jain & Angural, 2017). In line with these values, it is possible to say that the internal consistency coefficients of the scale are sufficient. In addition, the corrected total correlation was calculated to determine the predictive power of the total score and to determine the item discrimination. In order to determine the discrimination levels of the items and to determine the predictive power of the total score, corrected item-total correlations were calculated. When the results of the analysis were examined, it was seen that the item-total score correlations vary between 0.53 and 0.70. Item-total correlation values above 0.30 indicate that the items are distinctive. The item-total correlation values of each item in the scale were given in Table 4.

Table 4

Results of item-total correlation

Item No	Item-total correlation	Item No	Item-total correlation
1	0.62	13	0.58
2	0.59	14	0.59
3	0.64	15	0.62
4	0.55	16	0.65
5	0.52	17	0.53
6	0.61	18	0.56
7	0.70	19	0.66
8	0.64	20	0.70
9	0.61	21	0.68
10	0.54	22	0.72
11	0.60	23	0.58
12	0.66	24	0.57

Discussions and Conclusions

This study was conducted to develop a valid and reliable tool to measure the degree of psychological impact that COVID 19 has on people during this period of global pandemic. In this direction, an exploratory factor analysis was performed, and a 3-factor structure with 24 items was obtained, which explains 60.96% of the total variance, with an eigenvalue above 1, and with an explained variance value of over 5%. The model fit of the 3-factor structure determined using exploratory factor analysis was tested using confirmatory factor analysis, and the model was confirmed in general. The scale showed a strong parallelism with the Depression Anxiety Stress Scale, which was previously proven to be valid and reliable, with a correlation coefficient of 0.710. Scale items were prepared in 5-point Likert-type as “never (1)”, “rarely (2)”, “occasionally (3)”, “often (4)” and “always (5)”. There were no reverse items in the scale. The highest score that could be obtained from the scale was 120 and the lowest score is 24. High scores on the scale indicated a high level of COVID-19 Psychological Impact.

During the scale development stages, first the literature was reviewed and after the item pool formed from the obtained items was presented to the expert opinion, the item fit indices were reviewed by performing a pilot application. After the pilot applications were carried out, EFA was applied to determine the factor structure of the scale and a three-factor structure that explains 60.96% of the total variance of the scale was obtained. Kline (1994) stated that the variance explained in the scale development process should be at least 40%. In this respect, it can be said that the variance rates explained as a result of the exploratory factor analysis in the study are sufficient to decide the factor structure of the scale.

For the model fit of the factors obtained as a result of EFA, the first and second level CFA were examined and it was found that the model fit indices of the scale were sufficient and the values of the fit indices could be interpreted as sufficient for the model fit of the scale (Kline, 2005; Schumacher & Lomax, 2004). After finding that the model fit indices of the COVID-19 Psychological Impact Scale were at a sufficient level, the sub-dimensions were named anxiety, frustration, and dejection within the framework of the literature review and theoretical information on the three-factor structure. As a result, it was concluded that the “COVID-19 Psychological Impact Scale” was a valid and reliable scale applicable to individuals aged 18 or above.

Witnessing cases or deaths from COVID-19, being constantly exposed to news about worldwide deaths or the infection rate of the pandemic cause individuals to experience anxiety, uneasiness, and depression (Li et al., 2020). The COVID-19 pandemic threatens the lives of people and particularly causes psychological problems in humans (Stankovska et al., 2020). The ability to overcome this crisis in a healthy way depends largely on conducting research on the psychological impact of the pandemic (Arden & Chilcot, 2020). Therefore, it is important to investigate the psychological impact of COVID-19, to identify individuals who are psychologically affected by COVID-19, and to develop prevention programs (Pakpour & Griffiths, 2020). However, it cannot be said that there has been enough focus on the psychological impact of the disease on mental health as much as physical health (Satici et al., 2020). Identifying the psychological or mental health needs of people affected by this pandemic has been relatively neglected (Xiang et al., 2020). Since COVID-19 is a disease that leaves deep psychological scars in individuals and in society, psychological well-being of people should be protected and supported against this disease. In order to prevent the psychological problems caused by COVID-19, it is vital to carry out serious studies as soon as possible on possible prevention strategies such as identifying the psychological impact of the pandemic process on humans and intervening for those who need urgent psychological support.

The limitations of the current study include collecting data online rather than face-to-face, not having a homogeneous distribution of the gender of the participants, and being between 18 and 65 years of age. Considering the aims of this study and future practical purposes, it is important to mention that the scale is valid in a specific country and with a specific language. Therefore, the validity of the scale should be testes in other countries with different languages. Another limitation is whether participants with a diagnosis of psychiatric disorder and/or psychopharmacological treatment were excluded from the study.

Despite these limitations, this scale is important for identifying individuals in need of psychological interventions or prevention programs during or after COVID-19 pandemic. The scale is one of the scales developed to measure the psychological impact level of COVID-19 on adult individuals. It is thought that scientific research on the psychological impact of COVID-19 pandemic will contribute to improving the psychological health of individuals and shed light on future studies in the relevant literature. In future studies, the psychological impact of COVID-19 and its relationship with variables such as life satisfaction, depression, anxiety, suicide, stress, and psychological resilience can be investigated.