Using a bifactor exploratory structural equation modeling framework to examine the factor structure of the Depression Anxiety and Stress Scales-21.

This study proposed an improved representation of the DASS-21 factor structure developed by Lovibond and Lovibond in Behaviour Research and Therapy, 33, 335-342 (1995) using bifactor exploratory structural equation modeling (bifactor ESEM). This research was conducted by reference to 521 Turkish adults (45.3% females; M age = 27.86, SD = 8.23). The bifactor ESEM findings indicated a strong general factor of negative affect underlying responses to all DASS-21 items but also that despite the presence of three specific factors (depression, anxiety, and stress), the depression subscale explained a high degree of variance and could be considered to constitute a specific factor. The results obtained from this study show that there is a common factor associated with DASS-21 scales, the total score of DASS-21 can be identified as a measure of general negative affect, and the bifactor ESEM structure of DASS-21 ensures measurement invariance across genders.

Introduction

The relationship between the concepts of depression and anxiety, which causes negative emotional states in individuals, has attracted significant theoretical and clinical attention (Akiskal, 1985; Clark & Watson, 1991; Stavrakaki & Vargo, 1986; Watson et al., 1988). Although depression and anxiety are conceptually quite different structures, they share certain common features, and these structures overlap to a high degree (Clark & Watson, 1991). For this reason, these two concepts are mostly discussed together by relevant studies (Brown et al., 1997). S. H. Lovibond and P. F. Lovibond (Lovibond & Lovibond, 1995) developed a measurement tool to evaluate all the basic symptoms of anxiety and depression (Depression Anxiety Stress Scales, DASS-42). Although this scale is mainly intended to focus on two factors, i.e., anxiety and depression, the items pertaining to difficulty relaxing, irritability, and agitation constitute a third factor: stress (Lovibond & Lovibond, 1995). In a study conducted by reference to nonclinical samples, principal component analysis and oblique rotation were initially used. As a result, it was determined that one item related to the anxiety factor (Item 10) exhibited a low loading on its own factor (0.20). Second, a single-factor model, a two-factor model, and a three-factor model were tested via confirmatory factor analysis, and it was determined that the results of the three-factor model were superior. As a result, a three-factor solution was identified that explains 55% of the item variance in this scale, which consists of three factors, each containing 14 items. DASS-42 items are grouped into three scales: depression (DASS-D), anxiety (DASS-A), and stress (DASS-S). Analyses have shown that while DASS successfully distinguishes among these three negative emotional syndromes, these syndromes remain moderately highly correlated with each other. However, it has been reported that the correlations among the various scales of DASS are not only caused by items that load on more than one factor but also result from the fact that these correlations represent common underlying causes of depression, anxiety, and stress, such that “natural” correlations among factors can emerge (Lovibond & Lovibond, 1995).

The structure of DASS-42 was examined later by Brown et al. (1997), and following the exploratory factor analysis conducted by these authors, they reported a structure quite similar to the factor structure obtained by Lovibond & Lovibond (1995) by reference to a nonclinical sample. However, the same results were not reported by this study, and it was determined that some items exhibited high loadings on both factors (DASS-Stress Item 34 also loaded on the Anxiety factor, while DASS-Anxiety Item 9 also loaded on the Stress factor). It was also concluded that DASS-Anxiety Item 30 had an item loading of less than 0.40. Researchers have established a revised three-factor model in accordance with these results. Additionally, three other models have been created: a single-factor model, a two-factor model (combining the DASS-Anxiety and DASS-Stress scales into a single factor), and a three-factor model combining all three DASS scales presented by Lovibond and Lovibond (1995). These four models have been tested via confirmatory factor analysis, and it has been determined that the best model is a revised three-factor model.

This three-factor solution for the scale was later tested by Antony et al. (1998). Unlike other research, this study included several clinical groups as well as a comparison group of nonclinical controls. However, it was determined that the factor loadings did not suit the structure, especially regarding the items included in the DASS-Anxiety scale. Additionally, Crawford and Henry (2003) showed that the three-factor structure fits the data well but also that the most appropriate model is the one that allows the items determined by Brown and colleagues to be loaded on more than one factor. Similar findings were also reported by Page et al. (2007), and researchers have found that the model suggested by Brown et al. (1997) improves model fit.

The 42-item DASS, whose factor structure was determined to be nonstable, was abbreviated to include only seven items for each subscale, and the version known as DASS-21 was thus created by Lovibond and Lovibond (1995). Antony et al. (1998) determined that DASS-21, whose psychometric properties were examined, exhibited a more distinct factor structure and that correlations among factors were lower. The three separate but correlated three-factor structure of the scale has been confirmed by other studies. However, these researchers were able to obtain a three-factor solution by allowing (a) multiple associated errors in domain-specific scales or (b) cross-loading items to obtain optimal models (e.g., see Antony et al., 1998; Brown et al., 1997; Clara et al., 2001; Henry & Crawford, 2005).

In different studies throughout the literature, different models have been tested to identify the model that best represents the factor structure of DASS-21. Crawford and Henry (2003) employed single-factor analysis and found that although they obtained low values for model fit, all items were significantly loaded onto a single factor. Duffy et al. (2005) identified two-factor models including a generalized negativity factor and a factor comprising items indicating physiological arousal as the best. In later studies, the DASS-21 scale was also examined by reference to the bifactor model (which views all items as being influenced by a general factor - general distress as well as the specific factors of depression, anxiety, and stress). For instance, Henry and Crawford (2005) found that a four-factor (i.e., quadripartite) model consisting of the factors of three depression, anxiety, and stress alongside a general distress factor represented the ideal fit of all the structures they tested. Similarly, Osman et al. (2012) found that DASS-21 may measure a general dimension of distress rather than independent dimensions of depression, anxiety, and stress. Szabo (2010), in his research focusing on a young adolescent sample, noted that each of the three DASS-21 scales reflects a substantial common factor (i.e. general psychological distress, or Negative Affect). Tully et al. (2009) also suggested that a theoretical tripartite structure of depression and anxiety (the depression and anxiety factors plus a general negative affect factor) was robust and applicable among youth. Shaw et al. (2017), in their study focusing on adolescents and young participants, found that the general factor explained most of the common variance in DASS-21 scores.

Many studies have tested the factor structure and psychometric properties of DASS-21 across different countries, cultures, and languages (Apóstolo et al., 2006; Bottesi et al., 2015; Camacho et al., 2016; Daza et al., 2002; Musa et al., 2007; Szabo 2010; Taouk et al., 2001; Tonsing, 2014; Yıldırım et al., 2018). Studies have been conducted by reference to clinical (Clara et al., 2001; Ng et al., 2012; Page et al., 2007) or nonclinical samples (Henry & Crawford, 2005; Johnson et al., 2016; Norton, 2007; Osman et al., 2012; Yılmaz et al., 2017) as well as to a combination of both groups (Antony et al., 1998; Bottesi et al., 2015; Sarıçam, 2018; Yıldırım et al., 2018). Therefore, this measure is suitable not only to be utilized in any clinical or nonclinical setting but also for adaptation to any culture (Kyriazos et al., 2018). However, although findings regarding the validity and reliability of DASS-21 have been reported by many studies, it can be said that no complete agreement regarding the factor structure of DASS-21 (single factor, three-factor, or bifactor structure) has been reached. Therefore, different approaches to the factor structure of the scale remain possible. However, such divergent views can be addressed using exploratory structural equation modeling (ESEM).

The advantages of Exploratory Structural Equation Modeling (ESEM)

The independent cluster model of confirmatory factor analysis (ICM-CFA) models faces a constrained in situations in which items are associated only with specific dimensions and all loadings of unspecified dimensions (cross-loadings) are limited to zero (Joreskog, 1969). Thus, ICM-CFA models can produce biased parameter estimates as a result of this limitation (Morin et al., 2016). However, items may tend to present a valid relationship with more than one construct to some extent. These associations would be expressed via cross-loadings in exploratory factor analysis (EFA), and they would thus lead to inflated factor correlations in CFA (Fadda et al., 2017). Based on such situations, Asparouhov and Muthén (2009) developed exploratory structural equation modeling (ESEM) as a means of testing the factor structure of a measure. Therefore, as a combination of the EFA and CFA approaches, the ESEM featuring target rotation offers the advantages of both the EFA approach (allowing cross-loadings between items and nontarget factors) and the CFA approach (model-based and testing a structure that has been defined a priori) (Gomez et al., 2020). In ESEM analyses, including cross-loadings in the model could reduce the bias affecting parameter estimates. Thus, it is possible that ESEM can help produce a more precise and focused estimation of factor correlations (Asparouhov et al., 2015). ESEM allows for the use of “target” rotation. Target rotation implements a method similar to EFA (where items can load on any factor but cross-loadings of “targeting” items remain as close to 0 as possible) while simultaneously rotating the solution to achieve the best fit to a specified confirmatory model (Asparouhov & Muthén, 2009). However, a limitation of the model is the fact that the general factor is not considered in the ESEM. For this reason, the ESEM and the bifactor model approach are discussed together later.

Bifactor and bifactor ESEM models

According to the bifactor model, which was originally developed as a confirmatory model, there is a general factor (G-factor) that can describe the variance common to all items, and two or more specific factors (S-factors) are recommended to capture the remaining variance in each item (Reise et al., 2007). Including these factors orthogonally in the model enables us to eliminate the problems arising from high factor correlations. However, the bifactor model always fits better than the first-order factor model, even if such a fit is not actually the case, since it contains better nonsense response patterns in the dataset (Gomez et al., 2020, p. 4). In addition, an important limitation of the bifactor model is the fact that it does not take cross-loadings into account, which can be anticipated from DASS-21 due to the presence of partially overlapping components among depression, anxiety, and stress (Jovanović et al., 2019).

The bifactor ESEM featuring target rotation, which represents a version of the basic ESEM that has been expanded to include the target rotation model, also offers the advantages of a bifactor approach (allowing a general factor and specific factors that are not correlated). In such a case, the bifactor ESEM approach, including both cross-loadings among specific dimensions (i.e., ESEM) and a general factor (i.e., bifactor), appears to be particularly relevant (Morin et al., 2016).

Considering the positive aspects of the ESEM approach, some studies have conducted ESEM reviews of DASS-21 (Gomez et al., 2020; Johnson et al., 2016; Jovanović et al., 2019; Kyriazos et al., 2018; Vaughan et al., 2020) tested the factor structure of DASS-21 on Parkinson’s patients using different models. In this study, they determined that the ESEM featuring target rotation exhibits good fit values. However, when the factor loadings obtained using the three-factor ESEM were examined, these authors concluded that the items in the anxiety dimension were insufficient. In addition, it was found that a few items exhibited significant loadings on other factors. Kyriazos et al. (2018) examined the factor structure of DASS-21 using EFA, CFA, bifactor CFA, and ESEM by reference to a sample of 2,272 Greek adults. Despite reporting high indices of fit with respect to the three-factor ESEM, the researchers selected the three-factor CFA model because the factor loadings were unsatisfactory for the three-factor ESEM. Similarly, Gomez et al. (2020, p. 12), due to poorly defined specific factors, low reliability for two specific factors, and lack of support for external validity, preferred the ICM-CFA model over the ESEM featuring target rotation, bifactor CFA, and bifactor ESEM featuring target rotation. However, Vaughan et al. (2020) found that a bifactor representation for DASS-21 offered the best fit to the data with respect to an athlete population. These authors found that factor loadings indicated minimal misspecification (i.e., below 0.32) and higher loadings on the general factor. The inconsistent results of these studies require more bifactor ESEM reviews of DASS-21.

Scale reliability

In studies conducted using the original factor structure of DASS-21, it has been noted that the internal consistency reliability for subscales is sufficient and quite high (≥ 0.70) (e.g. Antony et al., 1998; Henry & Crawford, 2005; Lovibond & Lovibond, 1995). However, omega (ω) and omega hierarchical (ωh) coefficients provide better estimates of reliability for multidimensional constructs than the traditional use of Cronbach’s alpha because Cronbach’s alpha is typically underestimated (Dunn et al., 2014; Widhiarso & Ravand, 2014).

The ω coefficient is a factor-analytic “model-based” estimate of the proportion of variance in the unit-weighted total score that is attributable to all sources of common variance (Zinbarg et al., 2005). Coefficient 𝜔ℎ, which is part of the 𝜔 family of model-based estimates of reliability, can be used to quantify the strength of a general factor when controlling for specific factors. A high ω value indicates a highly reliable multidimensional composite construct, and a high ωh value (> 0.80) in the bifactor structure indicates that the general factor is the dominant source of systematic variance. Additionally, the coefficient omega hierarchical subscales (ωhs) estimates the strength of the influence of subdomain factors. Coefficient ωhs represents the proportion of reliable systematic variance in a subscale score after removing general factor variability (Reise et al., 2013).

In CFA studies conducted to investigate DASS-21, it has been determined that there are strong correlations among the factors. This close relationship may indicate that the structure of this scale should also be expressed by reference to a general factor. Additionally, the ICM-CFA model is too restrictive to account for most multidimensional measures used in psychology (Marsh et al., 2011) and is insufficient to explore the factor structure of multidimensional scales such as DASS-21 fully (Jovanović et al., 2019). With the help of newer models, such as the bifactor model, ESEM, and bifactor ESEM, the factor structure of DASS-21 can be made more intelligible. The main purpose of this study is to examine the three-factor structure of the scale proposed by Lovibond and Lovibond (1995) using the bifactor ESEM approach. Again, this three-factor structure of the scale was tested using ICM-CFA, ESEM, bifactor CFA, and bifactor ESEMs. All models include depression, anxiety, and stress factors there is as well as a general negative affect factor in the bifactor models. The base tested model is shown in Fig. 1.

Fig. 1

The 3-factor B-ESEM model

The present study has both theoretical and practical significance. Especially during the COVID-19 pandemic, the mental health status of adults has become a very important issue. In studies concerning this subject, the DASS-21 scale has been used widely across different cultures (e.g. Planchuelo-Gómez et al., 2020; Polat & Coşkun, 2020; Vanni et al., 2020). It is necessary to consider the factor structure of such a widely used scale from different perspectives and to test the measurement invariance of the structure. Another important aspect of this study is the fact that an approach that attempts to examine this scale using EFA and CFA models is insufficient to represent the multidimensional structure of the scale. The bifactor model, on the other hand, can create a bias, although it offers better fit values than other models (Murray & Johnson, 2013). Therefore, it is also necessary to conduct bifactor ESEM examinations of scales such as DASS-21.

Thus, we aimed to examine the generalizability of the structure of the scale by investigating whether the scale exhibited a similar structure to those reported by previous studies in different cultures. Additionally, considering the sources of multidimensionality related to the factor structure of DASS-21 in this study may help explain the inconsistent findings of previous studies regarding the factor structure of DASS-21. Furthermore, internal consistency and measurement invariance across genders were investigated regarding the tested structure.

Methodology

Participants

The sample comprised 521 Turkish adults from the general population, with ages ranging between 18 and 50 years (M = 26.58; SD = 7.77). The sample included 285 males (54.7%; M = 25.53, SD = 7.19) and 236 females (45.3%; M = 27.86, SD = 8.23). All participants were selected via a convenience sampling technique. The data of the study were collected during the COVID-19 pandemic period. For this reason, the convenience sampling technique had to be preferred, considering that it would be very difficult and costly to provide the necessary conditions to randomly select a sample from the adult population with a wide age range. The rule of thumb used in scale development/adaptation studies is the requirement of at least 10 participants for each scale item (Nunnally, 1978). Since the two scales used in this study had a total of 39 items, at least 390 participants were needed. In this case, it was thought that the number of participants of 521 participants was sufficient.

Data collection tools

The Depression, Anxiety, and Stress Scale-21 (DASS-21)

The scale is a general measure of the symptoms of depression, anxiety, and stress (i.e., over the past 7 days) (Lovibond & Lovibond, 1995). Initially, the scale was designed to feature 42 items, and it was subsequently abbreviated to include only 21 items. The short version of DASS, consisting of seven items per scale, is scored on a 4-point Likert-type scale with answers ranging from 0 (did not apply to me at all) to 3 (applied to me very much or most of the time). Higher scores represent higher levels of symptoms. This scale was adapted to Turkish by Sarıçam (2018). The original three-factor structure of the scale has been confirmed by reference to both clinical and nonclinical samples. Cronbach’s alpha internal consistency coefficient for all three scales has been reported to be high for both samples (ranging from 0.77 to 0.87).

The Anxiety Sensitivity Index-3 (ASI-3)

The ASI-3, which features 18 items, was developed by Taylor et al. (2007). In the ASI-3, participants are asked to indicate the extent to which they are concerned about the possible negative consequences of anxiety-related symptoms (e.g., “It scares me when my heart beats rapidly.”). Each item included in the ASI-3 is scored on a 5-point Likert-type scale with answers ranging from 0 (very little) to 4 (very much). The Turkish version of the ASI-3 was developed by Mantar et al. (2010). In that study, following exploratory factor analysis, the ASI-3 was found to exhibit a three-factor structure, including physical, cognitive, and social concerns. It was determined that the ASI-3 showed high internal consistency (α = 0.93). In the present study, the internal consistency coefficients were found to be 0.83, 0.80, and 0.76 for physical, cognitive, and social concerns, respectively.

Ethics statement

During the data collection process, permission was obtained from the Ethics Committee for the use of the scales, and the legal permission required for the implementation of the scales was obtained from the Board of Scientific Research and Publication Ethics of Nigde Omer Halisdemir University. The research procedure was conducted in accordance with the guidelines listed in the Declaration of Helsinki. All participants were informed of the study aims, voluntarily agreed to participate, and provided informed consent regarding the use of their anonymized data for research purposes. The data used in the study were collected over a period of approximately six months (October 2020-March 2021) via Google Forms. Participants were sent a link via e-mail and social media to answer the scales, and they completed these scales at their convenience and voluntarily. The data collected from participants included demographic information (e.g., age and gender) and their responses to two scales. The time required to complete the data collection was approximately 10 min for each participant.

Data analysis

Preliminary analyses and reliability

Data screening and descriptive analyses were performed using SPSS 24 software (Corp, 2016). The mean, standard deviation, and internal consistency coefficients of each subscale and the total scale were calculated.

Alternative models

Analyses were conducted using the weighted least square mean and variance adjusted (WLSMV) estimator available in Mplus Version 7.0 software (Muthen & Muthen, 2012). When fewer than five response categories are available, it is recommended to use WLSMV as the method of estimation (Beauducel & Herzberg, 2006).

In this study, ICM-CFA, bifactor CFA, ESEM, and bifactor ESEMs based on the DASS-21 three-factor structure were tested. Model fit was assessed using the comparative fit index (CFI), the Tucker & Lewis index (TLI), and the root mean square error of approximation (RMSEA). The cutoff values used for these indices are as follows: CFI and TLI ≥ 0.95 indicates excellent fit, while values between 0.90 and 0.95 indicate acceptable fit; RMSEA ≤ 0.06 indicate excellent fit, while values smaller than 0.08 indicate acceptable fit (Kline, 2016). Additionally, the WLSMVχ2 fit statistic is reported, but it is sample-dependent and has been shown to reject models with ordinal indicators excessively. Therefore, this value is not interpreted.

Subsequently, the correlations among the factors in the ICM-CFA and ESEM featuring target rotation were compared. For this purpose, the Pearson correlation coefficient was used, and results of 0.70 or above were interpreted as indicating a high correlation. In the next step, factor loadings obtained from different models were examined. In accordance with the suggestions of Tabachnik and Fidell (2007), 0.32 was used as a cutoff point for factor loadings. First, the factor loadings of the ICM-CFA and ESEM with target rotation were examined, followed by the factor loadings of the bifactor CFA and bifactor ESEMs, which indicated a hierarchically higher structure.

Omega coefficient and explained common variance

In the following step, the reliability and explained common variance (ECV) values of the tested models were examined. Omega (𝜔) and omega hierarchical (𝜔ℎ) coefficients were used, as these measures provide better estimates of reliability for multidimensional constructs than those offered by the traditional Cronbach’s alpha (Dunn et al., 2014). A high ωh value (> 0.80) in the bifactor structure indicates that the general factor is the dominant source of systematic variance. Additionally, a coefficient concerning the omega hierarchical subscales (ωhs) was estimated to indicate the strength of influence of the subscale factors. Coefficient ωhs represents the proportion of reliable systematic variance of a subscale score after removing general factor variability (Reise et al., 2013). ωhs < 0.50 indicates that the majority of that subscale score’s variance is due to the general factor and that negligible unique variance is due to that specific factor (Reise et al., 2010).

Explained common variance (ECV) is a statistical reliability index that provides a quantification of the importance of the general factor compared to that of the specific factors. In general, a high value of ECVgen indicates that the model has a strong general latent dimension rather than latent subdimensions. When ECVgen > 0.70, this finding indicates that the dataset can be one-dimensional (Rodrigez et al., 2016).

Measurement invariance

For measurement invariance across gender, multigroup confirmatory factor analyses (MGCFAs) were employed. In the present study, the following steps were used to test MI using MGCFA (Brown, 2015): (1) The model was tested separately for each group. (2) For a configural invariance step, it was determined whether the same pattern of fixed and free factor loadings was specified at each group variable. (3) A metric invariance model was estimated, according to which the factor loadings were constrained to be invariant across gender groups. (4) A scalar invariance model was estimated, according to the item thresholds and factor loadings were constrained to be invariant across gender groups. (5) A strict invariance model was estimated, according to which the items were invariant, as indicated by the constraints on the items’ uniqueness across genders.

At this stage, the CFI and RMSEA differences (ΔCFI ≤ 0.010, ΔRMSEA ≤ 0.015) proposed by Chen (2007) were employed alongside the previously mentioned CFI, TLI, and RMSEA criteria to evaluate model fit. Changes in CFI of less than 0.010 and changes in RMSEA of less than 0.015 indicate invariance across groups. Additionally, fit improvement was evaluated using the DIFFTEST function of Mplus (MDΔχ2; Asparouhov et al., 2006). However, MDΔχ2 also tends to be oversensitive to sample size. Therefore, additional indices must be used to complement chi-square difference tests when comparing nested models (Chen, 2007; Cheung & Rensvold, 2002).

Concurrent validity

The ASI-3 scores were used to examine the convergent validity of DASS-21. Relationships among the scales in general and among their subfactors were analyzed by reference to the Pearson correlation coefficient. Correlation coefficients above 0.40 were accepted as sufficient to indicate convergent validity (Kaasa et al., 1995).

Findings

Preliminary analyses and reliability

The mean scores, standard deviations, and internal consistencies of DASS-21 as calculated for the whole sample and for the specific groups of females, males, younger adults (under 30 years old) and older adults (30 and upper 30 years old) are presented in Table 1.

Table 1

Means, standard deviations, and reliabilities for the DASS–21

Scale	Gender			Age			Total Group	α
	Female	Male	t	Younger	Older	t
	M (SD)	M (SD)		M (SD)	M (SD)		M (SD)
Depression	7.16 (4.11)	6.11 (3.87)	2.994*	6.62 (4.14)	6.53 (3.76)	0.253	6.77 (4.19)	0.83
Anxiety	5.37 (3.23)	4.58 (3.01)	2.886*	4.93 (3.09)	4.96 (3.22)	0.086	5.21 (3.47)	0.76
Stress	7.54 (3.37)	7.22 (3.04)	1.128	7.40 (3.16)	7.29 (3.26)	0.376	7.57 (3.39)	0.72
Total	20.07 (8.83)	17.92 (8.40)	2.803*	18.96 (8.83)	18.78 (8.56)	0.221	19.55 (9.51)	0.89

*p < .05. α = Cronbach’s alpha coefficient

When Table 1 is examined, it can be seen that the scores of females are higher than those of males with respect to the subscales as well as the total of the scale. With the exception of the DASS-Stress scale, there are significant differences in the scores on the other scales according to gender. However, since the Cohen’s d coefficients calculated for all scales are less than 0.50, the effect size of the gender variable on the scores is low. With respect to the age variable, no significant differences are observed for any scale. The internal consistency coefficient of the subscales and the total scale is higher than the recommended value of 0.70 (Cortina, 1993); therefore, it can be said that the internal consistency of the scales is high.

Alternative models

The model fit of all alternative models is reported in Table 2. The ICM-CFA solution (χ2 = 686.982, df = 186, CFI = 0.924, TLI = 0.914, RMSEA = 0.072, 90% CI = 0.066-0.078), ESEM solution (χ2 = 514.135 df = 150, CFI = 0.945, TLI = 0.922, RMSEA = 0.068, 90% CI, 0.062-0.075), bifactor CFA solution (χ2 = 542.118, df = 168, CFI = 0.943, TLI = 0.929, RMSEA = 0.065, 90% CI, 0.059-0.072), and bifactor ESEM solution (χ2 = 366.069, df = 132, CFI = 0.964, TLI = 0.943, RMSEA = 0.058, 90% CI, 0.051-0.065) provide an acceptable degree of fit to the data. However, the results show that bifactor ESEM solutions exhibit a higher level of fit to the data than the ESEM and CFA solutions (higher CFI and TLI values and lower RMSEA). When the values obtained from all models are compared, the bifactor ESEM appears to offer the best representation of the data. However, in addition to this information, a more detailed review of the best model should be conducted by examining the factor loadings. For this detailed review, the ICM-CFA and ESEM results are compared first, followed by the results for bifactor CFA and bifactor ESEM.

Table 2

Summary of fit indices of the alternative measurement models

Model	χ2	df	CFI	TLI	RMSEA [90% CI]	Factor Correlation
						D – A	D – S	A – S
ICM-CFA	686.982*	186	0.924	0.914	0.072 [0.066, 0.078]	0.724	0.800	0.853
ESEM	514.135*	150	0.945	0.922	0.072 [0.066, 0.078]	0.591	0.308	0.303
B-CFA	542.118*	168	0.943	0.929	0.065 [0.059, 0.072]
B-ESEM	366.069*	132	0.964	0.943	0.058 [0.051, 0.065]

*p < .05. ICM-CFA = independent cluster model - confirmatory factor analysis, ESEM = exploratory structural equation model, B-CFA = bifactor confirmatory factor analysis model, B-ESEM = bifactor exploratory structural equation model, χ2 = WLSMV chi square, df = degrees of freedom, CFI = comparative fit index, TLI = Tucker–Lewis index, RMSEA = root mean square error of approximation, CI = 90% confidence interval, D = depression factor, A = anxiety factor, S = stress factor

ICM-CFA versus ESEM

Correlations among latent factors

Table 2 also includes the factor correlations in the ICM-CFA and the ESEM featuring target rotation. When these factor correlations are examined, the ESEM featuring target rotation indicates a much lower factor correlation (i.e., a clearer differentiation) than the ICM-CFA. In this case, it can be noted that the ESEM better reflects the variance in items (Marsh et al., 2009). The presence of such close relationships among factors in the ICM-CFA also indicates that these factors should be addressed using the bifactor model and supports the theoretical adequacy of ESEM.

Factor loadings

The standardized parameter estimates from all alternative model solutions are reported in Table 3. First, when the ICM-CFA model is examined, all factors are well defined by the presence of target loadings greater than 0.32 (ranging from |λ| = 0.410 to 0.825; M = 0.613). Second, when the ESEM featuring target rotation is examined, the ESEM factor loading estimates reveal generally well-defined factors due to substantial target factor loadings (ranging from |λ| = 0.051 to 0.837; M = 0.490). The value for target loadings is greater than 0.32, with the sole exception of Item 2 (λ = 0.305) for the Anxiety factor and Items 6 (λ = 0.173), 11 (λ = − 0.051), 12 (λ = 0.261) and 18 (λ = 0.065) for the Stress factor. In addition, it can be determined that two items related to the Stress factor do not exhibit a significant load on their dimension (Items 11 and 18).

Table 3

Standardized parameter estimates for all model solutions

	ICM-CFA		ESEM				B-CFA			B-ESEM
Items	Factor(λ)	δ	Factor D(λ)	Factor A(λ)	Factor S(λ)	δ	G-Factor(λ)	S-Factors(λ)	δ	G-Factor(λ)	Factor D(λ)	Factor A(λ)	Factor S(λ)	δ
Depression
Item 3	0.631	0.602	0.620	− 0.189	0.349	0.503	0.512	0.383	0.591	0.462	0.429	− 0.096	0.301	0.502
Item 5	0.573	0.671	0.519	0.184	− 0.156	0.627	0.494	0.262	0.687	0.519	0.272	0.013	− 0.205	0.615
Item 10	0.718	0.485	0.688	− 0.089	0.226	0.456	0.568	0.479	0.449	0.528	0.520	0.059	0.111	0.435
Item 13	0.825	0.320	0.619	0.244	0.012	0.372	0.725	0.300	0.384	0.748	0.262	− 0.079	− 0.009	0.366
Item 16	0.590	0.651	0.518	− 0.090	0.292	0.616	0.471	0.368	0.642	0.389	0.497	0.187	0.158	0.542
Item 17	0.721	0.481	0.734	0.119	− 0.167	0.404	0.594	0.405	0.483	0.652	0.343	− 0.143	− 0.174	0.406
Item 21	0.728	0.470	0.837	− 0.116	0.031	0.386	0.549	0.585	0.356	0.575	0.523	− 0.109	− 0.026	0.382
Anxiety
Item 2	0.480	0.770	.060	0.305	0.239	0.772	0.432	0.192	0.777	0.400	0.051	0.205	0.160	0.770
Item 4	0.492	0.758	− 0.282	0.664	0.214	0.606	0.354	0.611	0.501	0.414	− 0.140	0.464	0.082	0.587
Item 7	0.521	0.729	0.049	0.473	0.055	0.726	0.452	0.290	0.712	0.433	0.109	0.424	− 0.118	0.607
Item 9	0.635	0.596	0.234	0.478	− 0.062	0.608	0.595	0.116	0.633	0.586	0.088	0.206	− 0.154	0.583
Item 15	0.749	0.439	0.231	0.615	− 0.081	0.436	0.710	0.073	0.490	0.808	− 0.215	− 0.110	− 0.050	0.287
Item 19	0.650	0.577	− 0.246	0.766	0.298	0.393	0.532	0.614	0.340	0.576	− 0.143	0.520	0.148	0.356
Item 20	0.682	0.535	0.233	0.516	− 0.046	0.556	0.634	0.123	0.583	0.674	− 0.059	− 0.005	− 0.027	0.542
Stress
Item 1	0.547	0.701	0.119	0.123	0.579	0.533	0.521	0.164	0.702	0.445	0.050	0.034	0.583	0.458
Item 6	0.500	0.750	0.207	0.228	0.173	0.773	0.448	0.327	0.692	0.477	0.005	− 0.075	0.189	0.731
Item 8	0.636	0.595	0.175	0.257	0.415	0.568	0.578	0.652	0.241	0.549	0.060	0.055	0.377	0.550
Item 11	0.561	0.685	0.234	0.459	− 0.051	0.668	0.568	− 0.077	0.672	0.531	0.079	0.212	− 0.138	0.648
Item 12	0.737	0.457	0.189	0.343	0.261	0.510	0.733	− 0.029	0.462	0.678	0.047	− 0.012	0.254	0.474
Item 14	0.410	0.832	0.291	0.060	0.390	0.763	0.390	0.109	0.836	0.249	0.301	0.349	0.249	0.664
Item 18	0.487	0.763	0.298	0.187	0.065	0.786	0.456	0.214	0.746	0.469	.059	− 0.110	0.086	0.757

Factor loadings of items on their target factor(s) are bolded. For each factor, non-significant loadings are italicized, ICM-CFA = independent cluster model - confirmatory factor analysis, ESEM = exploratory structural equation model, B-CFA = bifactor confirmatory factor analysis model, B-ESEM = bifactor exploratory structural equation model, D = depression factor, A = anxiety factor, S = stress factor, λ = standardized factor loading, δ = standardized uniqueness, G-Factor = general factor from a bifactor model, S-Factor = specific factor from a bifactor model

Additionally, as expected, the ESEM solution reveals multiple cross-loadings. These cross-loadings remain relatively small and are generally lower than the main target loadings. However, although most cross-loadings remain small (|0.012–0.32|), some cross-loadings are sufficiently high (> 0.32 for Items 3, 11, and 12) to suggest the presence of another source of unmodeled multidimensionality. Therefore, multiple nontarget cross-loadings provide additional support for the ESEM solution. Morin et al. (2016) noted that high cross-loadings indicate hierarchically superior or conceptually related structures; therefore, bifactor ESEM analyses should be examined.

When the factor loadings obtained from the ESEM solution are examined, it can be seen that all items (seven items) related to the Depression factor load on their own factors at a significant and sufficient level (> 0.32). With the exceptions of Items 10 and 16, the other five depression items load significantly on the Anxiety factor, with the loadings for two of these items (Items 3 and 21) being negative. However, all of these factor loadings are smaller than 0.32. Similarly, five depression items (Items 3, 5, 10, 16, and 17) load significantly on the Stress factor, with the loadings for two of these items (Items 5 and 17) being negative. However, only one of these factor loadings (for Item 3) is greater than 0.32.

When the factor loadings in the anxiety factor are examined, it can be determined that all items load significantly (> 0.32) on their own factors (with the exception of Item 2). Five anxiety items (Items 4, 9, 15, 19, and 20) also exhibit significant level loadings on the Depression factor, with the loadings for two of these items (Items 4 and 19) being negative. However, all of these factor loadings are smaller than 0.32. In addition, three anxiety items (Items 2, 4, and 19) load significantly and positively on the Stress factor. However, all of these factor loadings are smaller than 0.32.

With the exceptions of two items (Items 11 and 18), the other five stress items load significantly on the Stress factor. However, two of these items are not salient (Items 16 and 12). Additionally, all seven items load significantly and positively on the Depression factor. However, all of these factor loadings are smaller than 0.32. Additionally, with the exception of one item (Item 14), the other six stress items load significantly and positively on the Anxiety factor. However, only two of these items are salient (Items 11 and 12).

When the findings are considered as a whole, the significant cross-loadings indicate that a good definition cannot be reached with respect to all three factors. Factor loadings of items on different factors may indicate that a higher-order factor model (such as bifactor CFA) is required to represent the factor structure of DASS-21 adequately (Gomez et al., 2020).

Bifactor CFA versus bifactor ESEM

The factor loadings obtained from the bifactor CFA and bifactor ESEMs are given in Table 3. With respect to item loadings on the bifactor CFA model, the G-factors are generally high and positive for items associated with the DASS S-factors (varying from |λ| = 0.471 to 0.725, M = 0.559 for the S-depression factor; |λ| = 0.354 to 0.710, M = 0.530 for the S-anxiety factor; and |λ| = 0.390 to 0.733, M = 0.528 for the S-stress factor). The S-depression factor is also generally well defined by relatively high loadings (|λ| = 0.262 to 0.585; M = 0.397). However, such a generalization cannot be made for the S-anxiety factor (varying from |λ| = 0.073 to 0.614; M = 0.288) and the S-stress factor items (varying from |λ| = 0.029 to 652; M = 0.225).

All items in the S-depression factor are positive and significant, and only two of these are not salient (Items 5 and 13). Five anxiety items (Items 2, 4, 7, 19, and 20) for the S-anxiety factor are significant, but three of these items (Items 2, 7, and 20) are not salient. Four stress items (Items 1, 6, 8, and 18) load significantly on the S-stress factor, and two of these items (Items 6 and 8) are salient. Additionally, two items (Items 11 and 12) load negatively on the S-stress factor. According to these results, it can be determined that the bifactor CFA model exhibits good results for the general factor and that only the Depression factor among the S-factors is defined sufficiently. It is seen that the Anxiety and Stress factors are not well defined according to this model.

Table 3 also includes the factor loadings for the bifactor ESEM solution. According to these values, the G-factor is generally high and positive for items associated with the DASS S-factors (|λ| = 0.462 to 0.748, M = 0.553 for the S-depression factor; |λ| = 0.400 to 0.808, M = 0.556 for the S-anxiety factor; and |λ| = 0.249 to 0.678, M = 0.485 for the S-stress factor). All seven depression items load significantly on the S-depression factor, but two of these items (Items 5 and 13) are not salient. Two anxiety items (Items 15 and 20) load negatively on the S-anxiety factor. The other five anxiety items load positively on the S-anxiety factor, but two of these items are not salient (Items 2 and 9). All stress items load significantly on the S-stress factor (with the exception of Item 18). Item 9 loads negatively, but other significant items load positively on the S-stress factor. Additionally, two of these items (Items 1 and 8) load saliently.

It can be seen that most of the items (particularly a few items in the S-depression factor) exhibit higher factor loadings on the G-factor rather than the S-factors. Therefore, it can be seen that the G-factor contributes to the factor structure of the scale and that bifactorial representation is beneficial. However, the bifactor ESEM featuring target rotation also indicates poorly defined S-anxiety and S-stress factors. In addition, while several items in the bifactor ESEM featuring target rotation do not exhibit a significant and positive load on the S-factors, they do exhibit significant and salient loadings on nontarget factors. It can be seen that there is no noteworthy item that falls into these categories in the S-depression factor. However, it can also be determined that several items associated with the S-anxiety factor exhibit nonsignificant and nonpositive loading (Items 15 and 20). Additionally, several items associated with the S-stress factor (Items 11 and 18) fall into one or more of these categories. Therefore, it can be seen that among the S-factors, only the Depression factor is well defined.

Reliability of the DASS-21 bifactor ESEM

Based on the bifactor ESEM, the general DASS-21 showed excellent reliability (ω = 0.93), and ωh indicated a predominant general factor (ωh = 0.82). A comparison of ωh with ω (0.82/0.93 = 0.89) showed that most of the reliable variance in total scores could be attributed to the G-factor. Therefore, omega hierarchical subscale coefficients were very small for depression, anxiety, and stress (0.31, 0.13, and 0.14, respectively), thus showing that little common variance remained after accounting for the general factor. According to this model, the ECVgen and the ECV for depression, anxiety, and stress were 0.70, 0.14, 0.09, and 0.08, respectively. ECVgen was 0.70, thus also indicating a quite strong general factor, which accounted for well over half the common variance and exceeded the 0.70 cutoff, thus indicating that part of the variance was explained by the general factor and that specific factors may not contribute value in this context.

Measurement invariance of the DASS-21 bifactor ESEM across genders

First, a single-group bifactor ESEM was investigated in terms of its application to the female and male groups. Subsequently, the measurement invariance of the bifactor ESEM was tested using MGCFA (Table 4).

Table 4

Results of measurement invariance across gender

Model	χ2(df)	CFI	TLI	RMSEA [90% CI]	Model Com.	ΔCFI	ΔRMSEA	χdiff2(df)
Single-Group
Female	203.837*(132)	0.975	0.960	048 [0.035, 0.061]
Male	295.349* (132)	0.955	0.929	0.066 [0.056, 0.076]
Multi-Group
1. Configural	493.000*(264)	0.965	0.944	0.058 [0.050, 0.066]
2. Weak	596.960*(332)	0.959	0.948	0.055 [0.048, 0.062]	2 vs. 1	− 0.006	− 0.003	141.813* (68)
3. Strong	604.771*(370)	0.964	0.959	0.049 [0.042, 0.056]	3 vs. 2	0.005	− 0.006	42.910* (38)
4. Strict	660.503*(391)	0.959	0.955	0.051 [0.045, 0.058]	4 vs. 3	− 0.005	0.002	66.329* (21)

*p < .05. CFI = comparative fit index, TLI = Tucker–Lewis index, RMSEA = root mean square error of approximation, CI = 90% confidence interval, ΔCFI = difference among CFIs, ∆RMSEA = difference among RMSEAs, = WLSMV χ2 DIFFTEST results

For DASS-21, gender invariance of the final bifactor ESEM was tested separately for each gender group as a baseline model (for females, N = 236, and for males, N = 285). As seen on Table 4, this model exhibited a good fit for females (χ = 203.837, df = 132, χ/df = 1.54, CFI = 0.975, TLI = 0.960, RMSEA = 0.048, 90% CI = 0.035–061] and for males (χ = 295.349, df = 132, χ/df = 2.38, CFI = 0.955, TLI = 0.929, RMSEA = 0.066, 90% CI = 0.056-0.076). Subsequently, this baseline model was tested for both gender groups concurrently. This model, which was a configural invariance model, provided an acceptable fit to the data across genders (CFI = 0.965, TLI = 0.944, RMSEA = 0.058, 90% CI = 0.050; 066). With respect to this model, invariance constraints across genders were progressively added to the factor loadings (weak invariance), items’ thresholds (strong invariance), and items’ uniqueness (strict invariance). In the measurement invariance test, the model fits of the increasingly constrained models were compared. The model fit of the metric invariance model did not display a decrease with respect to any of the model fit indices that would suggest noninvariance (ΔCFI = − 0.006; ΔRMSEA = − 0.003). A similar situation is valid for strong (ΔCFI = 0.005; ΔRMSEA = − 0.006) and strict invariances (ΔCFI = − 0.005; ΔRMSEA = 0.002). Therefore, none of these constraints resulted in a decrease in model fit that exceeded the recommended cutoff scores for the fit indices (ΔCFI < 0.01 and ΔRMSEA < 0.015). As a result, although all χ and some Δχ were significant, it can be said that the goodness-of-fit indices exhibited fully satisfactory model fit at each stage.

Concurrent validity

The concurrent validity of DASS-21 was compared to that of the ASI-3 via analysis of the Pearson correlation coefficient. The correlations presented in Table 5 indicate that the correlation analysis showed a correlation coefficient change ranging from 0.235 to 0.574 for all scales and that the subscales of DASS-21 and total DASS were significantly correlated with the ASI total score and subscale scores (p < .05).

Table 5

Correlations coefficients

Scale	ASI-3	Physical concerns	Social concerns	Cognitive concerns
DASS-21	0.546*	0.494*	0.319*	0.547*
DASS-21 depression	0.462*	0.423*	0.235*.	0.476*
DASS-21 anxiety	0.574*	0.517*	0.530*	0.552*
DASS-21 stress	0.553*	0.521*	0.473*	0.516*

* p < .05

Discussion

The present study aimed to assess the psychometric properties of DASS-21 and to provide evidence concerning its invariance across genders. For this purpose, first, the various alternative structural models of DASS-21 (ICM-CFA, ESEM, bifactor CFA, and bifactor ESEM) were tested by reference to a sample of Turkish adults. Only a limited number of studies have examined the factor structure of DASS-21, which is widely used in research, by reference to ESEM and/or bifactor ESEMs across different cultures (Gomez et al., 2020; Johnson et al., 2016; Jovanović et al., 2019; Kyriazos et al., 2018; Vaughan et al., 2020). No study has hitherto examined whether the scale exhibits measurement invariance in Turkish culture.

As a result of the model examinations, it was demonstrated that the original three-factor model exhibits an adequate fit to the data. However, high correlations among the factors indicate that there may be a general factor or cross-loadings among the factors (Jovanović et al., 2019). For this reason, the results of addressing the structure of the scale is through the use of bifactor and ESEM approaches have been examined. According to this examination, it was determined that the best model fit is exhibited by bifactor ESEM. Bifactor ESEM findings support a strong general factor underlying the responses to the items included in DASS-21. However, when the specific factors were examined, the best results were obtained for the Depression factor. Although partially suitable results were obtained for the Anxiety factor, negative and insufficient loadings were reported for the Stress factor.

The findings obtained by the study are in accordance with the conclusions that have been reported in the literature regarding the factor structure of DASS-21. For example, Jovanović et al. (2019) showed that the bifactor ESEM is the best representation of the DASS-21 structure in the context of adolescents. Similarly, Gomez et al. (2020) noted that among the four alternative models that they tested, both the versions of ESEMs (ESEM and bifactor ESEM) fit better than both the first-order models (CFA and bifactor CFA). In addition, in the literature, sufficient fit values ​​have been obtained with respect to the three-factor ICM-CFA model, as in this study (Gomez et al., 2020; Jovanović et al., 2019; Kyriazos et al., 2018). However, parallel to the findings of these studies, it was observed that the relationships among factors are quite high according to the ICM-CFA model, especially for the Anxiety and Stress factors (Gomez et al., 2020; Kyriazos et al., 2018; Moore et al., 2017). For this reason, it the fact that different models may need to be tested should be taken into consideration. As a result, lower correlations among factors were obtained in the ESEM. This finding is the same in the current study.

Although the bifactor ESEM is the model with the best fit, various problems pertaining to factor loadings could be seen. It was concluded that the Anxiety and Stress factors in particular were insufficiently defined as specific factors. Some items associated with these factors exhibited negative and/or insignificant loading values on their respective factors. Moreover, it was observed that some items exhibited significant cross-loadings to other specific factors. However, among the specific factors, the most problematic results were obtained regarding the Stress factor. Additionally, Jovanović et al. (2019) reported the existence of substances that are insufficient for the Stress factor or that exhibit negative loadings on their own factors rather than specific factors. Johnson et al. (2016) determined that the Anxiety factor was poorly defined according to the factor loadings obtained via the ESEM. It has been observed that some items in this factor load significantly on the Depression or Stress factors. Additionally, in their study, the bifactor ESEM showed a slightly better fit than the ESEM; however, these researchers noted that this improvement was not sufficient to justify the loss of parsimony resulting from the use of former model. Similarly, Kyriazos et al. (2018) found that both the bifactor models and the ESEMs were good but that the factor loadings were unacceptable. Gomez et al. (2020) found that there were no consistent results regarding the Anxiety and Stress factors. Therefore, both Kyriazos et al. (2018) and Gomez et al. (2020) preferred the three-factor ICM-CFA model in their studies. Vaughan et al. (2020) found that a bifactor representation exhibited the best fit to the data, and while some degree of misspecification was present, these errors were below predetermined cutoffs. Therefore, these authors continued their studies using this model. Similarly, Jovanović et al. (2019) conducted their study based on the bifactor ESEM, despite the misspecifications that occurred with respect to factor loadings.

According to the bifactor ESEM used in this study, the results regarding the Depression factor show that the items in this factor exhibit high factor loadings on both general and special factors. In addition, similar to Jovanović et al. (2019), it was observed that the factor loadings on the general factor were higher for all items in this factor. However, Item 5 exhibited a lower loading than the criterion regarding the S-depression factor, similar to the findings of other studies (Gomez et al., 2020; Jovanović et al., 2019). By examining the cross-loadings, no factor loading higher than the criterion value was identified. For the Depression factor, similar findings have been reported by Gomez et al. (2020), Jovanovic et al. (2019), and Vaughan et al. (2020).

In this study, according to the bifactor ESEM, it can be seen that some items associated with the Anxiety factor exhibit a high loading on the general factor but low loadings on their own factors. Similar results have been obtained in the literature for Items 9 and 15, which are included in this set of items (Jovanovic et al., 2019; Moore et al., 2017; Shaw et al., 2017). This situation may result from the fact that these two anxiety items have slightly different content than the remaining anxiety items in terms of reflecting the cognitive aspect of anxiety (Jovanović et al., 2019). In this study, the expected results for Items 2 and 20 were not observed. Although these items were positive and significant in the study of Jovanović et al. (2019), they are lower than 0.32. Similarly, Gomez et al. (2020) found that a satisfactory condition was not reached for Item 2 of the Anxiety factor. According to the results regarding the anxiety items, heterogeneity can be seen in this subscale.

When the results of the bifactor ESEM in the Stress subscale are examined, it can be seen that the items included in this subscale exhibit high and positive loadings on the general factor (except Item 12), while the S-stress factor exhibits very low loadings. In addition, the ωhs of this factor is quite low, thus indicating that most of the variance in the stress items can be accounted for by the general factor. This result by Tully et al. (2009) is similar with respect to the tripartite factor structure, and the items related to the Stress factor are defined in terms of the general factor. Similar results have been obtained from other studies that have shown that the S-stress factor is poorly defined (Gomez et al., 2020; Jovanovic et al., 2019).

When the omega coefficients obtained from the bifactor ESEM are examined, a high value can be obtained for the general factor. The omega coefficients for the subscales are quite low. When the ECV values are examined, it can be seen that the effect of the general factor is high. Jovanović et al. (2019) concluded that most of the variance in DASS-21 is explained by the general factor.

Another piece of evidence collected with respect to the validity of the bifactor ESEM in the study is the measurement invariance test. It has been reported that the model ensures strict invariance across genders. Jovanović et al. (2019) also concluded in their study that the DASS-21 bifactor ESEM is invariant between males and females. In that study, the concurrent validity results for DASS-21 with the ASI-3 were also found to be good.

Therefore, all results should also be taken into account, indicating a strong general dimension associated with DASS-21. This strong general dimension has also been highlighted by previous studies (Gomez et al., 2020; Jovanović et al., 2019, Osman et al., 2012; Vaughan et al., 2020). In addition, it is striking that the items included in the scale exhibit significant and high cross-loading on other factors in the ESEM studies. Therefore, it is suggested that the items included in the scale are insufficiently defined with respect to the Anxiety and Stress factors. The fact that the Stress factor was not predicted during the development of the scale (Lovibond & Lovibond, 1995) may be the reason for these results. Previous studies have generally shown that the Depression factor has a well-defined construct, but the results are less clear regarding the Anxiety and Stress factors (Gomez et al., 2020; Oei et al., 2013; Osman et al., 2012). These factors (especially the Stress factor) appear to be closely related to general distress (Henry & Crawford, 2005). Additionally, items pertaining to the Stress factor measure both the Anxiety and Depression factors (Antony et al., 1998). However, Vaughan et al. (2020) reported that the bifactor ESEM worked very well for DASS-21. The reason why the bifactor ESEM did not work as well as expected in this study is likely the fact that the specific factors are not well defined. However, bifactor ESEM examinations can help identify substances that are problematic in three ways: (i) they lack significant and salient positive loading on the targeted factors, (ii) they do not load significantly on the targeted factors, or (iii) they load significantly and saliently on nontarget factors (Gomez et al., 2020, p. 13). However, considering all the results, it can be determined that the bifactor ESEM is suitable for DASS-21.

Limitations

This study faces certain limitations. The first such limitation is the study’s sample size. A validation sample can also be referenced when larger sample sizes are attained. Thus, the findings of this study can be verified. Another such limitation is the use of models based on a 3-factor structure in this study. Other models (1-factor, 2-factor, etc.) for DASS-21 in the literature can also be tested. The cutoff points in the model-fit indices referenced in this study are associated with CFA, but these cutoff points were also used in the bifactor ESEMs. However, these cutoff points may not be suitable for bifactor ESEMs. Additionally, because the data were obtained online, we were unable to control for other variables that tend to affect responses to an assessment instrument negatively. Another limitation of this study is the fact that modern measurement theories such as Rasch and IRT are not consulted.

Implications for future research and practice

Very few studies have employed bifactor ESEM for adaptations of DASS-21 into other languages (Jovanović et al., 2019; Kyriazos et al., 2018). The structure of this scale, which has been adapted to many cultures, should be addressed by using bifactor ESEM in other cultures. A shorter version of the DASS scale featuring 9 items (DASS-9) is also available. The factor structure of this form of the scale can be revealed by a similar study focusing on Turkish culture. In addition, the measurement invariance of the scale can be tested with respect to other variables, such as age. Unlike in this study, the factor structure of the scale can be examined using similar psychometric analyses with respect to data collected from the clinical samples and/or other cultures. It is recommended that psychologists should consider the existence of a dominant general factor when interpreting the scores obtained through the use of this scale. For this reason, it is recommended that professionals who will use DASS-21 be more careful when scoring and interpreting the depression, anxiety, and stress sub-scales. Additionally, to explore the robustness of the findings of this study, these findings should be replicated by reference to other cross-cultural samples. In future studies, concurrent validity can be tested by including more positive and negative psychological variables.