Factorial Validity, Reliability, Measurement Invariance and the Graded Response Model for the COVID-19 Anxiety Scale in a Sample of Ecuadorians.

The aims of the research are to evaluate the factorial validity, internal consistency, measurement invariance, discrimination, and difficulty of the Covid-19 Anxiety Scale (CAS) applied to a sample of Ecuadorian adults (N = 451). The study is based on an instrumental design with Classical Test Theory (CTT) and Item Response Theory (IRT) technics. The results confirmed the validity of the CAS single-factor structure, with measurement invariance across gender and high internal consistency. Additionally, all CAS items displayed adequate discrimination indexes and proper ordering of the difficulty thresholds. In a conclusion, the CAS is a valid measurement scale for Ecuadorian adults.

Introduction

There is a robust and growing body of research exploring the impact the COVID-19 pandemic has had on mental health worldwide. Indeed, the pandemic has posed wide ranging social and economic difficulties (i.e., changes related to work, economic instability, mobility, education, and public health measures), along with the additional stress, fear and worry of those who become infected or whose family and close acquaintances become infected (Usher et al., 2020). Although preventative public health measures such as mass vaccination programs, mask mandates, and lockdowns, have helped lower mortality rates and the rate of serious health complications, the end of the pandemic is still not in sight. Indeed, lack of access to vaccines for the world’s most vulnerable population, vaccine hesitancy, resistance to public health measures, and the appearance of new COVID-19 variants (i.e., the omicron variant), have made it difficult to curb infection rates (Bajema et al., 2021; Karim & Karim, 2021). In this context, the pandemic has had significant effects on the short-term and medium-term mental health of people around the world (Cullen et al., 2020; Lieven, 2021; Magano et al., 2021) and in Ecuador in particular (Zumba-Tello & Moreta-Herrera, 2022). The psychological distress caused by the pandemic is exacerbated for individuals who have difficulties with cognitive emotion regulation strategies and with coping and conflict-resolution mechanisms (Rodas et al., 2021). This results in overall symptomatology such as excessive stress, fear, anxiety, and depression, amongst others (Chen et al., 2021; Moreta-Herrera et al., 2021; Vinaccia et al., 2021). As such, it is particularly important to conduct psychological research into the general and specific psychopathologies related to the pandemic, along with treatment for these conditions. In order to adequately diagnose and employ corrective individual and group treatment, it is important to undertake exploratory research to identify the psychological attributes related to the COVID-19 pandemic. Likewise, research to develop, adapt and validate instruments that can measure these attributes is necessary (Jonason et al., 2020; Li, 2016; Moreta-Herrera et al., 2020; Mueller & Hancock, 2018).

Evaluation of the Coronavirus Anxiety Scale

As COVID-19 related mental health problems become increasingly compounded, developing psychometric instruments to measure these difficulties is imperative. Measurements such as the Coronavirus Anxiety Scale (CAS; Lee, 2020a), the Fear of COVID-19 Scale (FCV-19S) (Ahorsu et al., 2020) and the Preventive COVID-19 Infection Behaviors Scale (PCIBS) (Chang et al., 2020), are particularly useful. Testing and validating these scales in different cultural, social and linguistic contexts is necessary in order to further evidence their validity and reliability. COVID-19 related anxiety is of particular interest since it has become one of the most prevalent psychological disorders in the context of the pandemic (Peteet, 2020; Schafer et al., 2022).

The Coronavirus Anxiety Scale (CAS; Lee, 2020) was developed as an instrument to evaluate physiological symptoms specific to COVID-19-related anxiety. Lee’s (2020) original study on the CAS showed that it had a unidimensional factorial structure. The study also found that the CAS had excellent reliability according to its internal consistency and that it showed evidence of validity as correlated to measures of distress, disability, and coping. Currently, the CAS is one of the most employed instruments used to measure COVID-19-related anxiety. Indeed, its psychometric properties have been tested in 25 countries around the world in Asia, Europe, and the Americas (Lieven, 2021). Amongst countries where the CAS has been applied are: Iran (Mohammadpour et al., 2020), China (Chen et al., 2021) South Korea (Choi et al., 2020), Portugal (Magano et al., 2021), Turkey (Evren et al., 2020), the United States (Serpas & Ignacio, 2021), Brazil (Padovan-Neto et al., 2021); as well as specifically Spanish-speaking countries such as Cuba (Broche-Pérez et al., 2020), Colombia (Vinaccia et al., 2021), Mexico (Mora-Magana et al., 2020), Peru (Caycho-Rodríguez et al., 2021a, 2021b) and Argentina (Eidman et al., 2021). All the above-mentioned studies have confirmed the CAS’s unidimensional model, along with its high internal consistency. These studies also offer evidence of the measurement invariance of the CAS across age and gender (Ahmed et al., 2020; Lee, 2020). As such, the body of evidence shows the CAS to be both a consistent and an adequate instrument for measuring COVID-19-related anxiety.

Most of the above-mentioned studies have evaluated the psychometric properties of CAS based on Classical Test Theory (CTT). CTT allows analysis of the validity and reliability of an instrument using a linear model. However, the CTT does have certain limitations such as the non-invariance of the instrument (DeVellis, 2006). As such, it remains important to explore other, less studied, psychometric properties of the CAS. For this reason, this study also applies Item Response Theory (IRT) that relates empirical scores with the latent trait variables being studied to offer a broader analysis of the CAS. While CTT presents an overall analysis of the CAS, IRT provides more detailed information about the items (Birnbaum, 1968; DeMars, 2010; Rasch, 1960; Toland, 2014). As such, the combined application of both CTT and IRT allows for a more complete interpretation of the instrument.

The present research constitutes the first study in Ecuador to analyze the psychometric properties of the CAS using both CTT and IRT. In fact, there are few studies that analyze the discrimination and difficulty of CAS items. Recently, Caycho-Rodríguez and colleagues measured these psychometric characteristics using the Graduated Response Model (GRM; Samejima, 1997). Their study used a Spanish version of the CAS with Peruvian adults (Caycho-Rodríguez et al., 2021a) and seniors (Caycho-Rodríguez et al., 2021b), and found that the CAS items had adequate discrimination and difficulty parameters.

Objectives and Hypothesis

The objectives of this study are (a) to confirm the unidimensionality of the CAS with a sample of Ecuadorian adults, (b) to establish the invariance of measurement of the CAS across gender, (c) to verify the internal consistency of the CAS, (d) to estimate the discrimination and difficulty parameters of the items. The hypotheses of this study are that the CAS has adequate internal or factorial validity, that it is invariant across gender (H1), that it has high internal consistency (H2), and that the discrimination and difficulty parameters of the CAS items are adequate (H3).

Method

Design and Procedure

This study used a psychometric quantitative design to analyze the validity of the Coronavirus Anxiety Scale (CAS) (Ato et al., 2013) in a non-probabilistic sample of Ecuadorians. In particular, the factor structure, dimensionality, equivalence of measurement and internal consistency of the CAS was analyzed. The Graded Response Model of Item Response Theory (IRT) was used to analyze the discriminatory power of each item and its difficulty.

The development of the study was conducted during the second semester of 2021, during the course of the Covid-19 pandemic. Given these circumstances, the data collection was carried out virtually. The evaluation link was made through social networks (Facebook, WhatsApp, Twitter, and others). The participants to be evaluated given their consent by clicking on the option “I agree to participate” in the consent letter. It contained the objective of the investigation, the activities to be carried out, the handling and treatment of the data that will be used only for investigative purposes.

It should be noted that in the course of this investigation the ethical criteria corresponding to the Helsinky Convention were handled. Finally, this study has the endorsement of the Ethics Committee of the Provincial College of Clinical Psychologists of [anonymised] in Ecuador.

Participants

Participants were selected through a non-probabilistic sampling. The inclusion criteria to participate in the study was that participants had to be at least 18 years old. They also had to participate voluntarily and sign a letter of consent. The sample included 451 individuals from the general population of Ecuador. 30.4% of participants were men and 69.6% were women. The ages of participants ranged from 18 to 67 years of age (M = 29.06; SD = 10.60). 68.5% were single, 20.4% were married, and the remaining 10.1% were either separated, divorced, or widowed. 79.4% of participants had pursued post-secondary studies, 17.1% had finished high school, and the remaining 3.3% had finished elementary school and had some level of technical training. 37.7% of participants had permanent employment, 15.5% had temporary employment and 46.8% were unemployed either because they were students or because they had been unable to find a job.

Instrument

The Coronavirus Anxiety Scale (CAS; Lee, 2020a). The CAS is a unidimensional instrument to assess physiological reactions of fear and anxiety related to the COVID-19 pandemic. It includes 5 items on a 5-point Likert-type scale that range from 0 (not at all) to 4 (nearly every day over the last 2 weeks). The final score for the CAS can range from 0 to 20 points. The CAS does not determine a cut score to distinguish between clinically significant impairment due to COVID-19 related anxiety and anxiety that does not cause disability. However, participants’ total score can reveal general levels of anxiety. This study used the Spanish translation of the CAS by Caycho-Rodríguez et al. (2020).

Analysis of the Data

The analysis of the data was divided into three stages. The first stage included a descriptive analysis of the items. The mean, standard deviation, skewness (g1) and kurtosis (g2) were then computed for each item (Table 1). The hypothesis of univariate normality was tested using the scores for g1 and g2, where these values should be between ±1.5 (Ferrando & Anguiano-Carrasco, 2010).The hypothesis of multivariate normality was tested using Mardia’s Test for multivariate skewness and kurtosis (Cain et al., 2017; Mardia, 1970). Finally, the matrix of polychoric correlations of the questionnaire items was computed to examine their relationships within the scale configuration (Table 1).

Table 1.

Preliminary Analysis of CAS.

Items	Descriptive analysis				Polychoric correlations
	M	SD	g1	g2	1	2	3	4	5
Item 1	0.91	1.13	1.19	0.57	1	.718	.700	.684	.675
Item 2	0.93	1.16	1.11	0.23		1	.795	.785	.745
Item 3	0.98	1.15	1.00	0.02			1	.810	.759
Item 4	0.68	1.01	1.69	2.12				1	.822
Item 5	0.74	1.09	1.49	1.42					1
Mardia			806.3**	36.73**

Note. **p < .05; M: mean, SD: Standard Deviation; g1: Skewness; g2: Kurtosis.

During the second stage of the data analysis a Confirmatory Factorial Analysis (CFA) was applied to test whether there is a single latent factor that can describe the CAS (Figure 1). Given that the data used in this study is ordinal and it doesn’t have a multivariate normal distribution, the method of weighted least squares, with mean and variance adjusted (WLSMV) (Li, 2016), was used to fit the model. To support the fit of a single factor structure this study took into account absolute fit indices (the chi-square statistic, χ2 with degrees of freedom [df] adjusted for the model specification, the standardized chi-squared statistics [χ2/df], and the standardized root mean squared residual [SRMR]), relative fit indices (the Comparative Fit Index [CFI], and the Tucker-Lewis Index [TLI]), and the residual mean square error approximation (RMSEA). The following thresholds were used to decide a good fit for the model: A non-significative χ2 at the 5% level of significance, a RMSEA and a SRMR that are less than .08, a CFI and a TLI that are greater than .9, and factor loadings (λ) which should be no greater than .50 (Brown, 2015; Byrne, 2008; Mueller & Hancock, 2018; Wolf et al., 2013). A Multi-group Confirmatory Factorial Analysis (MGCFA) using WLSMV was fitted to assess measurement invariance across gender (Table 2). The degree of invariance was assessed by the inclusion of restrictions (configural invariance model, metric model, scalar model) and by analyzing the changes between models in χ2, in the CFI, and in RMSEA values (Asparouhov & Muthén, 2014; Brown, 2015). The internal consistency reliability was calculated using McDonald’s Omega coefficient (ω) (McDonald, 1999) together with a 95% confidence interval (95% CI) (Table 2).

Figure 1.

Confirmatory Factorial Analysis of the CAS using WLSMV. Note. χ2: Chi-square; df: degrees of freedom; χ2/df: Standardized chi-square; CFI: Comparative Fit Index; TLI: Tucker-Lewis Index; RMSEA: Root Mean Square Error Approximation; SRMR: Standardized Root Mean Square Residual.

Table 2.

Analysis of Measurement Invariance Across Gender for the CAS.

Models	χ2 (df)	CFI	RMSEA	Δχ2	ΔCFI	ΔRMSEA
Masculine	19.43 (5)***	.991	.146	––	––	––
Feminine	12.39 (5)*	.998	.069	––	––	––
General	1.82 (10)	.989	.041	––	––	––
Configurational	4.08 (14)	.985	.041	5.04 (4)	.004	.000
Metric	5.08 (18)	.991	.028	2.30 (4)	.006	.013
Scalar	26.05 (19)	.925	.079	2.63 (1)	.066	.051

Note. *p < .05; ***p < .001; χ2: Chi-squared; df: degrees of freedom; CFI: Comparative Fit Index; RMSEA: Root Mean Square Error Approximation; Δ: Delta.

During the third stage, the discrimination and difficulty parameters of the CAS items were analyzed using the model of Graded Response Method (GRM) for ordered polytomous categories (Hambleton et al., 2010; Samejima, 1997) (Table 3). To determine adequate discrimination, the slope of the discrimination parameter as a function of the level of ability of the individual should be greater than 1 (Stone, 1992). The difficulty parameter is defined as the point where the probability of obtaining a correct answer is 50% or more––there are 4 difficulty estimations as there are 5 possible answers (1 per threshold). The Test Information Curve (TIC) and the Test Information per Items (TIIC) was used to obtain a more detailed view of the performance of the CAS. The R programming language (4.01) (R Core Team, 2019) was used to analyze the data together with the MNV, MBESS, Lavaan y ltm libraries.

Table 3.

Item Saturations for the CAS Questionnaire and Their Internal Consistency Globally and Across Genders.

Items	λ Total	λ Men	λ Women	Δ λ
Item 1	––	.77	.81	−.04
Item 2	––	.88	.87	.01
Item 3	––	.90	.87	.03
Item 4	––	.89	.96	−.07
Item 5	––	.89	.83	.06
Internal consistency reliability
	Total	Men	Women	Δ ω
ω	.90	.91	.91	.00
95%-CI	[.89–.93]	[.87–.95]	[.88–.93]	[-.04–.05]

Note. λ: factor loadings; Δ: Delta; ω: McDonald coefficient.

Results

Preliminary Analysis

Table 1 shows a preliminary analysis of the items. The mean and the standard deviation range between M(item 4) = 0.68 (SD = 1.01), and M(item 3) = 0.98 (SD = 1.15). These values display a low anxiety symptomatology related to Covid-19. The skewness (g1) and kurtosis (g2) are within ±1.50, except for item 4. Hence, the hypothesis of univariate normality of each item is weakly accepted. However, the Mardia’s test of multivariate normality rejects it (p > .05). The polychoric correlation matrix of the items shows correlations ranging from .675 to .822. Hence, they appear to be pertinent (greater than .2) and non-redundant (less than .9).

Confirmatory Factorial Analysis, Measurement Equivalence Across Gender, and Internal Consistency

Figure 1 shows the adjusted CFA for the CAS. The unidimensional structure of the measure is confirmed because the goodness-of-fit measures are within their thresholds (χ2/gl <1.14, SRMR = 0.016, RMSEA = 0.07, [0.038, 0.114], CFI = 0.998, TLI = 0.995). The factor loadings (λ) of each item are greater than 0.50. This indicates that the items are consistent and that they explain the variance of the measurement.

Table 2 shows the results of the analysis of measurement invariance across gender. When comparing the fit of the CFA for men and women using an anova analysis, there is a significant difference in the χ2, Δχ2 = −3.47; p < .05. Hence, when comparing across gender the CFA presents relevant differences (Since the analysis uses robust statistics, to compute the χ2, we used the Satorra–Bentler (2001) scaled χ2). As can be observed in Table 2, the measurement invariance across gender is achieved for the configural and metric invariance models. However, measurement invariance is not achieved for the scalar invariance model. This establishes certain limitations for the CAS across gender. The model fits better for woman than for men.

The upper part of Table 3 shows the saturations of the items for the CAS segmented across gender. The factor loadings for each of the groups are adequate as they are greater than 0.5. Also, the differences between the saturations amongst the groups is less that 10%. The lower part of Table 3 shows the McDonald’s Omega coefficients (ω) for the internal consistency reliability of the CAS questionnaire. These are presented without taking gender into account, and for either male or females. In all three instances, the McDonald Omega coefficient (ω) is greater than 0.7. As such, it can be concluded that the questionnaire is internally consistent without taking gender into account, and for each gender. Also, there is no statistical differences in the internal consistency of the questionnaire for men and women because the 95%-CI contains 0.

Graded Response Modelling

Table 4 shows the discrimination and difficulty parameters for CAS items. Column (a) presents the discrimination values for each item. Since all the values are greater than 1, all the items show an acceptable level of discrimination between individuals. Columns (b1) through (b4) present the estimated thresholds for each item. Since they increase monotonically the level of difficulty of the questionnaire is adequate.

Table 4.

Discrimination and Difficulty Parameters for the CAS Items.

Items	(a)	(b1)	(b2)	(b3)	(b4)
Item 1	2.105	−0.005	1.038	1.703	2.438
Item 2	3.044	0.038	0.807	1.444	2.204
Item 3	3.042	−0.080	0.769	1.440	2.269
Item 4	3.703	0.388	1.108	1.661	2.073
Item 5	3.146	0.297	1.036	1.695	2.212

Note. (a) Discrimination parameters; (b): Difficulty parameters.

The upper panel of Figure 2 shows the Item Information Curves (IIC). The lower panel shows the Test Information Scale (TIC) of the CAS. As can be seen in the IIC, item 4 shows the most discriminatory capacity. Hence, it can be considered the most relevant and precise item to evaluate the latent trait variable. Items 2, 3, and 5 are similar to each other in terms of discriminant capacities. The TIC indicates that the test is more reliable within −0.5 and 3.

Figure 2.

Information curves for the items and the questionnaire. Note. IIC: Item Information Curves; TIC: Test Information Curves.

Discussion

The purpose of this study was to evaluate the properties of the Coronavirus Anxiety Scale (CAS) as a mental health screener for COVID-19 related anxiety in a sample of Ecuadorian adults. As such, this study sought to confirm the unidimensional structure of the CAS, and to analyze its measurement invariance across gender, its internal consistence reliability, and the discrimination and difficulty parameters of CAS items.

Since the goodness of fit values (χ2/gl, CFI, TLI, SRMR y RMSEA) and the factor loadings (λ) computed for the CFA are all within acceptable ranges (Brown, 2015; Byrne, 2008; Mueller & Hancock, 2018; Wolf et al., 2013) (Figure 1) it can be concluded that the CAS has a unidimensional structure (Lee, 2020). This suggests that it is a sound measure for evaluating COVID-19 related anxiety. Additionally, this also means that it offers easy interpretation of the scores, and that it can be analyzed by IRT. Moreover, the unidimensional structure of the CAS resembles the findings of other studies assessing the global validation of the CAS (Lieven, 2021), as well as studies conducted in Europe (Evren et al., 2020; Magano et al., 2021), Asia (Chen et al., 2021; Choi et al., 2020; Mohammadpour et al., 2020) and in the Americas (Padovan-Neto et al., 2021; Serpas & Ignacio, 2021)––this includes Ecuador’s neighboring countries Peru and Colombia (Caycho-Rodríguez et al., 2021a, 2021b; Vinaccia et al., 2021), and other Spanish-speaking countries of the continent such as Cuba, Mexico and Argentina (Broche-Pérez et al., 2020; Eidman et al., 2021; Mora-Magana et al., 2020).

To test the measurement invariance across gender the Multi-group Confirmatory Factorial Analysis (MGCFA) was applied to the CAS. The study found metric, but not scalar, invariance. Also, there were significant differences (p < .05) between the χ2 for the two groups. The CFA fitted better for women than for men. These results differ from Lee’s (2020) original study which found scalar measure invariance. However, other studies, such as Ahmed et al. (2020) only found configural invariance. This suggests the necessity for further studies assessing the measurement invariance across gender for the CAS because gender seems to have incidence in the COVID-19 related anxiety construct.

Additionally, the CAS shows high internal consistency reliability both globally and when divided by gender. This allows for the inference that the Spanish-version of the CAS is a reliable and appropriate instrument for measuring COVID-19 related anxiety in the adult population of Ecuador. These results are similar to those in Lee’s (2020) original study. Likewise, the results resemble other validation studies for the CAS (Broche-Pérez et al., 2020; Chen et al., 2021; Choi et al., 2020; Eidman et al., 2021; Evren et al., 2020; Lieven, 2021; Magano et al., 2021; Mora-Magana et al., 2020; Mohammadpour et al., 2020; Padovan-Neto et al., 2021; Serpas & Ignacio, 2021; Vinaccia et al., 2021).

The psychometric analysis based on IRT using the Graduated Response Model (GRM), found that the item discrimination of the CAS is greater than 1 and thus adequate. This study also found that the estimations of the difficulty thresholds for each item increased monotonally. Hence, the difficulty of each item is also adequate. The ICC curves (Figure 2) show that item 4 has the best discrimination to evaluate the latent trait. This item is associated with eating behaviours that are affected by thinking about, or the idea of, COVID-19. This could be helpful to identify if a person is at risk of suffering from COVID-19 related anxiety. On the other hand, the Total Information Curve (TIC) (Figure 2) shows that the CAS has greater reliability and precision for scores between −0.5 and 3. These values are within the usual parameters accepted for both discrimination and difficulty of the items (Hambleton et al., 2010; Samejima, 1997). These results are similar to those in both of Caycho-Rodríguez et al. studies (2021a, 2021b) which applied the CAS to a Peruvian sample.

This study contributes to a growing body of research analyzing psychological attributes, such as dysfunctional anxiety, that arise in the context of the COVID-19 pandemic. Moreover, it offers validation for an instrument to identify anxiety as a phenomenon that continues to be latent during the pandemic. Furthermore, the validation of the CAS for the Ecuadorian population offers additional information regarding the psychometric properties of the CAS in Ecuador–an ethnic, social, and cultural context where it had not been applied previously.

This study also contributes at a methodological level by applying IRT (Toland, 2014) in addition to CTT. IRT is a methodology which has been applied less for analyzing the psychometric properties of measurement instruments. As such, this study offers additional evidence of the validity of the CAS. In fact, at the time of writing there are only two other studies applying IRT to the CAS (Caycho-Rodríguez et al., 2021a, 2021b). Finally, in providing evidence of the validity of the CAS in the Ecuadorian population, this study offers a useful instrument for evaluating COVID-19 related anxiety in Ecuadorian adults. In turn, this can help better detect the phenomenon and apply effective and efficient psychological intervention measures in the country.

Limitations

The greatest limitation of this study is that the sample was limited to the adult population of Ecuador older than 18 years of age. As such, the conclusions of this study are limited to this population group. Hence, further analysis of the instrument with teenagers and children is important. Another limitation of the study is that its relation to other instruments was not analyzed. Therefore, further analysis in this direction is recommended.