Evaluation of the construct reliability and validity of the DSM-5 Self-Rated Level 1 Cross-Cutting Symptom Measure-Chinese version in maintenance hemodialysis patients.

OBJECTIVE: The study aim was to assess the construct reliability and validity of the internal structure of the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition Self-Rated Level 1 Cross-Cutting Symptom Measure (CCSM)-Chinese version in maintenance hemodialysis (MHD) patients, and to clarify whether the CCSM is suitable for patients with psychological issues.
METHODS: Participants were 190 MHD patients. Structural equation modelling was used to establish a measurement model to confirm the reliability and validity of the internal structure of the CCSM-Chinese version, and to compare it with the Chinese version of the Symptom Checklist-90 (SCL-90).
RESULTS: Of the 13 CCSM domains, 8 showed stability and validity in the initial screening of psychological symptoms. Although the Chinese version of the SCL-90 is widely used, the measurement model of the 8-domain CCSM was a better fit than the SCL-90 (CCSM: chi-square = 35.668, chi-square/degrees of freedom [CMIN/df] = 1.877; root mean square error of approximation [RMSEA]  = 0.061; adjusted goodness of fit index [AGFI] = 0.931 vs. SCL-90: chi-square = 89.159, CMIN/df = 2.972; RMSEA = 0.084; AGFI = 0.879).
CONCLUSION: The Chinese version of the CCSM is a useful rapid screening tool to detect psychological symptom risk in MHD patients.

Introduction

Psychological factors are sometimes involved in the development and progression of kidney diseases. Such factors can be major problems for patients on maintenance hemodialysis (MHD) because they affect quality of life.[1,2] With the development of psychosomatic medicine, there is evidence that stress-related chronic diseases are associated with comorbid psychiatric disorders, and that the combined effect of stress-related chronic disease and comorbid psychiatric disorders may be greater than the sole effect of comorbidities.[3] Moreover, comorbid psychiatric disorders interfere with the clinical management of MHD patients.[4] This often results in considerable confusion among clinicians with no psychological training in diagnosing comorbid psychiatric disorders.

Previous studies have assessed psychiatric disorders in MHD patients using one-dimensional psychological scales such as the Beck Depression Inventory or the Beck Anxiety Inventory. Such studies have identified a depression incidence of 20% to 42%[5-7] and an anxiety incidence of 27% to 45.7%,[1,8] with a substantial reduction in quality of life.[9,10] However, these studies focused only on whether MHD patients were depressed or anxious.[6,8,11,12] Some of the interventions and treatments derived from these study findings and developed to address psychological problems encountered by MHD patients have failed to produce good results. The poor performance of such interventions may result from a failure to consider the psychological comorbidity of patients.[13,14] Therefore, an appropriate psychological scale is needed to evaluate and understand comorbid psychiatric disorders in MHD patients.

Self-rating scales are useful to help clinicians assess and understand psychological comorbidity in Chinese MHD patients. The validation of scales that have not been previously validated can yield profound clinical benefits. Reliability and validity are important indicators of the quality of such scales, and it is essential to determine the goodness of fit of the measurement model and the validity evidence based on the internal structure.[15]

The Chinese version of the Symptom Checklist-90 (SCL-90) is a popular measure of psychiatric symptoms in patients with chronic diseases, and SCL-90 scores that have been validated by comparison with norms are widely used in hospitals across China.[16-19] This scale evaluates several dimensions of psychopathology and comprises nine psychological distress subscales (factors). Initially considered to represent a multidimensional structure,[19,20] the scale also reflects sensory experience, emotional experience, thought, consciousness, behavior, lifestyle, interpersonal relationships, diet and sleep. The SCL-90 has been used in China as a multidimensional self-report inventory to evaluate the psychological status of patients with MHD.[21] However, the SCL-90 contains 90 items, and is therefore time-consuming to complete and inconvenient for clinical use with outpatients. Patients on MHD often cannot fully complete the Chinese version of the SCL-90 owing to the effort required. Therefore, clinicians need a simple assessment tool to evaluate and understand psychological problems in MHD patients. The Diagnostic and Statistical Manual of Mental Disorders Fifth Edition (DSM-5) Self-Rated Level 1 Cross-Cutting Symptom Measure (CCSM) was developed by the US DSM-5 Task Force and Work Groups in 2013.[22] This new assessment tool helps clinicians to identify symptoms that substantially affect treatment and prognosis. It can also be used to monitor the progression of patients’ symptoms and as a clinical follow-up tool.[23] It contains only 23 questions for short-term screening of psychiatric disorders. These questions measure 13 psychiatric domains (subscales): depression, anger, mania, anxiety, somatic symptoms, suicidal ideation, psychosis, sleep problems, memory, repetitive thoughts and behaviors, dissociation, personality functioning and substance use. Some studies have shown that this scale is a viable initial screening tool to assess psychopathology.[24-25] Its short yet comprehensive structure may be useful for high risk patients with severe negative psychiatric symptoms who are unlikely to seek treatment.[26,27] However, the CCSM has not been studied in China.

The aim of this study was to investigate the goodness of fit of the measurement model, the construct reliability, and the validity (based on convergent and discriminant evidence) of the Chinese version of the DSM-5 Self-Rated Level 1 CCSM for psychiatric assessment of MHD patients. We aimed to determine whether the Chinese version of the CCSM is more suitable than the Chinese version of the SCL-90 for the diagnosis of negative psychiatric symptoms in Chinese MHD patients.

Methods

Participants

All data used in this study were collected from MHD patients at the Hemodialysis Center of Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, between August 2019 and November 2019. The inclusion criteria were subject age ≥18 years and ≥1 weekly MHD session. The exclusion criteria were 1) having an in-dwelling catheter; 2) untreated malignant tumors; 3) cognitive impairment; 4) inability to complete the scale; 5) incomplete clinical data.

Measures

The Chinese version of the CCSM and the Chinese version of the SCL-90 were used to assess patients’ mental health and psychiatric symptoms during the previous 2 weeks.

Chinese version of the CCSM

The adult Chinese version of the CCSM assesses 13 psychiatric domains: depression, anger, mania, anxiety, somatic symptoms, suicidal thoughts, psychosis, sleep problems, memory, repetitive thoughts and behaviors (obsessive–compulsive disorders), dissociation, personality functioning, and substance use.[28] These domains are assessed by between 1 and 3 items. Each item has five response categories: 0 = none or not at all; 1 = slight or rare, less than a day or two; 2 = mild or several days; 3 = moderate or more than half the days; and 4 = severe or nearly every day. The score on each item within a domain should be examined. The clinician records the highest score on each domain, and any additional inquiry is based on these highest scores. A rating of mild (i.e., 2) or greater on any item within a domain (except for substance use, suicidal thoughts, and psychosis) can serve as a guide for additional inquiry and follow-up to determine if a more detailed assessment for that domain is necessary. For substance use, suicidal thoughts, and psychosis, a rating of slight (i.e., 1) or greater on any item within the domain can guide additional inquiry and follow-up to determine if a more detailed assessment is needed.

Chinese version of the SCL-90

The Chinese version of the SCL-90 comprises 10 psychiatric factors: somatization, obsessive–compulsive, interpersonal sensitivity, depression, anxiety, hostility, phobic anxiety, paranoid ideation, psychoticism, and other factors (sleep and diet). The factor scores can be used to examine the symptom profile and the evolution of each symptom. The 10 psychiatric factors have different numbers of items, and each item has five response categories: 0 = none or not at all; 1 = slight; 2 = mild; 3 = moderate; 4 = severe. A factor score >2 can serve as a guide for further screening and examination. The standard score (normative score) can also be referred to for analysis and evaluation (see details in Supplementary materials).[16]

Data analysis

Data for demographic characteristics were analyzed using R Software version 3.6.1 (www.r-project.org) and were displayed according to frequency (%) or mean ± standard deviation (SD). SPSS Amos 23 (IBM Corp., Armonk, NY, USA) was used to conduct confirmatory factor analysis (CFA) and to generate a measurement model of the two self-rating scales. Structural equation modelling was used with several fitness indexes: (1) the absolute fit index, which tests whether the causal path map of the overall model fits the data well (chi-square [χ2] should be small, the root mean square error of approximation [RMSEA] should be <0.08,[29] the goodness of fit index [GFI] must be >0.9[30] and the adjusted GFI [AGFI] should be >0.8); (2) the value-added fit index can be used to reflect the degree of difference between a hypothetical model and an independent model that has no covariation between the hypothesized observed variables (the comparative fit index [CFI] and the Tucker–Lewis index [TLI] should be >0.90);[29] (3) the parsimony fit index (chi-square divided by degrees of freedom [CMIN/df] should be between 1 and 3).[29] The construct reliability and validity were tested using two indexes: composite reliability (CR) and average variance extracted (AVE). These two indexes can provide useful information about construct reliability and validity. The CR should be above the acceptable threshold level of 0.50, and the AVE should be higher than the acceptable threshold level of 0.50.[31,32] The reliability was evaluated using Cronbach’s α; the acceptable threshold of Cronbach’s α is >0.60. Harman’s single factor test was used to measure the common method bias of the two scales; the proportion of variance explained was <40%, indicating that the common method bias was not substantial.

Ethics approval and consent to participate

This cross-sectional, observational, single-center study received ethical approval from the medical ethics committee of Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine (Approval No.: SH9H-2019-T143-1). It was registered with the Chinese Clinical Trials Registry (No.: ChiCTR1900028275). All participating patients provided written informed consent after thoroughly reading and fully understanding the information about the study. A copy of each signed consent form was retained.

Results

A total of 200 patients participated in this study. Of these, 10 did not fully complete the SCL-90 because of the complexity and time-consuming nature of the scale, and the data of these participants were excluded. Finally, data for 190 participants were statistically analyzed. The investigation process is shown in Supplementary Figure 1. The mean age of the final study sample was 61.68 years (SD = 13.39); 61.6% (n = 117) were men and 38.4% (n = 73) were women. Of the 190 patients, 170 (89.5%) were married and 20 (10.5%) were widowed or single. MHD duration was <2 years for 82 (43.2%) patients and ≥2 years for 108 (56.8%) patients. Statistical analysis of education level showed that 15.8% of participants (n = 30) had a college degree, 8.4% (n = 16) had a high school degree, 52.6% (n = 100) had a junior high school degree, 15.3% (n = 29) had an elementary school degree and 7.9% (n = 15) were illiterate. The illiterate participants were assisted by the same doctor and completed the two scales by dictating their answers to each question.

Structural equation modelling of the CCSM and SCL-90

Measurement model of the CCSM

A measurement model was generated using the CCSM data (Figure 1). The standardized (normalized) factor loadings (λ) of the domains mania, memory, dissociation, psychosis and substance use were well below 0.5 (the loadings for anger [0.49] and sleep problems [0.48] were rounded up to 0.5). Therefore, the former five above-mentioned domains were removed and the measurement model was reconstructed using the remaining eight domains (i.e., depression, anger, anxiety, somatic symptoms, suicidal thoughts, sleep problems, personality functioning, and repetitive thoughts and behaviors) (Figure 2). Subsequent analysis used the 8-domain CCSM. In this 8-domain model, the standardized factor loadings (Figure 2) ranged from 0.441 to 0.605, indicating that each domain contributed to the overall rating. The measurement model showed satisfactory fit after removing the five domains mania, memory, dissociation, psychosis and substance use (χ2 = 35.668; CMIN/df = 1.877; RMSEA = 0.061; GFI = 0.963; AGFI = 0.931; CFI = 0.953; TLI = 0.930). The analysis of common method bias using Harman’s single factor test showed that the proportion of variance explained by the first component was 22.276%, indicating that the common method bias of the 8-domain CCSM was not substantial.

Figure 1.

Measurement model of the Chinese version of the CCSM.

CCSM, DSM-5 Self-Rated Level 1 Cross-Cutting Symptom Measure.

Figure 2.

Measurement model of the 8-domain Chinese version of the CCSM (after removal of five factors).

CCSM, DSM-5 Self-Rated Level 1 Cross-Cutting Symptom Measure.

Measurement model of the SCL-90

A measurement model was generated using the SCL-90 data (Figure 3). The standardized factor loadings (Figure 3) ranged from 0.522 to 0.894 (>0.5), indicating that each SCL-90 factor contributed to the overall rating. The measurement model showed acceptable fit (χ2 = 89.159; CMIN/df = 2.972; RMSEA = 0.084; GFI = 0.934; AGFI = 0.879; CFI = 0.959; TLI = 0.939). The analysis of the common method bias using Harman’s single factor test showed that the proportion of variance explained by the first component was 17.688%, indicating that the common method bias of the SCL-90 was not substantial.

Figure 3.

Measurement model of the Chinese version of the SCL-90.

SCL-90, Symptom Checklist-90.

Construct reliability of the CCSM and SCL-90

Construct reliability of the 8-domain CCSM

The CR of the scale was 0.757 and Cronbach’s α was 0.729. Cronbach’s α for each of the eight CCSM domains ranged from 0.429 to 0.825 and was generally higher than the threshold level of 0.60 (Table 1), except for the domains anger, suicidal ideation and sleep problems (these domains contain only one question and so Cronbach’s α could not be calculated). The repetitive thoughts and behaviors domain showed the highest consistency, and the somatic symptoms domain the lowest consistency. These results show that the Chinese version of the 8-domain CCSM is reliable.

Table 1.

Cronbach’s α for each domain on the 8-domain CCSM.

Domain	Cronbach’s α
Depression	0.724
Anger	N/A*
Anxiety	0.681
Somatic symptoms	0.429
Suicidal thoughts	N/A*
Sleep problems	N/A*
Personality functioning	0.703
Repetitive thoughts and behaviors	0.825

Data show composite reliability. *The domains of anger, suicidal thoughts, and sleep problems have only one question, so Cronbach’s α could not be calculated.

CCSM, DSM-5 Self-Rated Level 1 Cross-Cutting Symptom Measure.

Construct reliability of the SCL-90

The CR of the scale was 0.903 and Cronbach’s α was 0.939. Cronbach’s α for each factor was generally higher than the threshold level of 0.60, except for the factors psychoticism and paranoid ideation, for which Cronbach’s α was 0.542 and 0.545, respectively (although these values are close to 0.6) (Table 2). Depression showed the highest consistency and psychoticism the lowest. These results confirm the reliability of the Chinese version of the SCL-90.

Table 2.

Cronbach’s α for each factor on the SCL-90.

Factor	Cronbach’s α
Depression	0.791
Interpersonal sensitivity	0.732
Anxiety	0.639
Obsessive–compulsive	0.703
Psychoticism	0.542
Paranoid ideation	0.545
Hostility	0.708
Others	0.645
Phobic anxiety	0.745
Somatization	0.785

Data show composite reliability. SCL-90, Symptom Checklist-90.

Convergent and discriminant validity of the CCSM and SCL-90

Convergent and discriminant validity of the 8-domain CCSM

The evidence for convergent validity was assessed using the AVE index, which was 0.283 (<0.5), indicating that less variance was captured by the construct than could be attributed to measurement error.

Discriminant validity was evaluated using the CFA and measurement model. The square root of the AVE for each scale domain () (on the diagonals of Table 3) surpassed the factor correlations between each pair of domains, except that the (0.523) for personality functioning was slightly smaller than the correlation between personality functioning and repetitive thoughts and behaviors (0.556). Thus, the discriminant validity results suggest that most parts of the construct showed some validity. However, the evidence for discriminant validity regarding personality functioning and repetitive thoughts and behaviors was weak.

Table 3.

Construct correlation matrix for the 8-domain CCSM.

Domain	Depression	Anger	Anxiety	Somatic symptoms	Suicidal thoughts	Sleep problems	Repetitive thoughts and behaviors	Personality functioning
Depression	0.605
Anger	0.351	0.533
Anxiety	0.367	0.359	0.600
Somatic symptoms	0.351	0.213	0.281	0.441
Suicidal thoughts	0.267	0.200	0.208	0.230	0.461
Sleep problems	0.300	0.256	0.321	0.317	0.232	0.471
Repetitive thoughts and behaviors	0.239	0.284	0.213	0.192	0.341	0.174	0.590
Personality functioning	0.322	0.247	0.350	0.277	0.464	0.237	0.556	0.523

Domain correlations are shown below the diagonal. The square root of the average variance extracted for each domain is shown in bold on the diagonal.

CCSM, DSM-5 Self-Rated Level 1 Cross-Cutting Symptom Measure.

Convergent and discriminant validity of the SCL-90

The evidence for convergent validity of the SCL-90 was assessed using the AVE index, which was 0.487 (close to 0.5),[30,33] indicating that almost 50% of the variance was explained.

The of each SCL-90 domain (on the diagonals of Table 4) surpassed the factor correlations between each pair of domains (subscales). Thus, the results suggest that the construct has adequate discriminant validity.

Table 4.

Construct correlation matrix for the SCL-90.

Factor	Hostility	Anxiety	Psychoticism	Phobic anxiety	Paranoid ideation	Others	Obsessive–compulsive	Somatization	Interpersonal sensitivity	Depression
Hostility	0.615
Anxiety	0.459	0.745
Psychoticism	0.433	0.524	0.703
Phobic anxiety	0.213	0.317	0.395	0.562
Paranoid ideation	0.596	0.464	0.575	0.191	0.667
Others	0.282	0.449	0.423	0.338	0.402	0.602
Obsessive–compulsive	0.449	0.429	0.513	0.454	0.550	0.439	0.730
Somatization	0.344	0.498	0.392	0.448	0.373	0.477	0.407	0.558
Interpersonal sensitivity	0.507	0.614	0.648	0.397	0.658	0.496	0.601	0.460	0.824
Depression	0.551	0.667	0.629	0.503	0.597	0.538	0.653	0.499	0.737	0.894

Domain correlations are shown below the diagonal. The square root of the average variance extracted for each domain is shown in bold on the diagonal.

SCL-90, Symptom Checklist-90.

Discussion

Psychological issues represent one of the most common health problems affecting the quality of life of patients on MHD.[12,34,35] Nephrologists, therefore, need to identify psychological health issues in MHD patients in a timely manner. An appropriate self-rating psychiatric symptom scale could be used as a screening tool to identify psychiatric issues and comorbidity in such patients. Very few studies have used measurement models and CFA to investigate self-rating psychiatric symptom scales for MHD patients.

The measurement model and the CFA showed that the standardized factor loadings of the Chinese version of the SCL-90 were higher than the threshold level of 0.5, indicating that each SCL-90 factor contributed to the overall rating and the measurement model fit the data well. The CR results for the SCL-90 (CR = 0.903) indicated that the whole scale has significant reliability. Almost all factors showed good reliability; the exceptions were psychoticism and paranoid ideation, which had Cronbach’s α values of >0.6. However, the Chinese version of the CCSM showed low efficiency in assessing psychological state on the domains mania, memory, dissociation, psychosis, and substance use (these domains had λ loadings <0.5) in MHD patients. Therefore, these five domains were removed and a measurement model with eight domains (depression, anger, anxiety, somatic symptoms, suicidal thoughts, sleep problems, personality functioning, and repetitive thoughts and behaviors) was generated. This reconstructed model showed good fit and indicated that the common method bias was not substantial. However, almost all the eight domains in this reconstructed CCSM measurement model had less than three questions, making it difficult to calculate the CR of each domain. Moreover, the domains anger, suicidal ideation and sleep problems had only one question each, making it impossible to calculate Cronbach’s α. Therefore, the overall CR of the reconstructed scale was calculated, and showed adequate internal consistency (CR = 0.757). This suggests that the 8-domain CCSM (which includes the domains anger, suicidal ideation and sleep problems) is a more reliable psychiatric disorder symptom measure for MHD patients. In other words, the results suggest a significant correlation between the whole scale and the symptom items of the original versions of the scale. Of those domains for which Cronbach’s α could be calculated, depression, anxiety, personality functioning, and repetitive thoughts and behaviors showed particularly good reliability. Therefore, comparing the 8-domain CCSM with the SCL-90, both scales showed good reliability. However, an examination of the reliability of each domain suggested that the SCL-90 provided more comprehensive psychiatric symptom disorder information than the 8-domain CCSM. This may be because the SCL-90 scale contains many questions. The examination of convergent validity of the two scales confirmed this conclusion. The AVE of the 8-domain CCSM was 0.283 and <0.5, indicating that the CCSM is not superior to the SCL-90 (AVE = 0.487). The anxiety domain of the 8-domain CCSM has three questions and the remaining domains have less than three questions (three domains have only one question). Teasing out limited information from a single psychiatric symptom disorder domain often leads to measurement errors in assessing MHD patients.

Regarding evidence for discriminant validity, the SCL-90 performed better than the 8-domain CCSM; the latter showed limited discriminant validity. The 8-domain CCSM demonstrated some evidence of validity for each domain among MHD patients. However, the value for personality functioning (0.523) was close to the correlation between personality functioning and repetitive thoughts and behaviors (0.556), and the scale was poor in differentiating personality functioning from repetitive thoughts and behaviors. This indicates that when evaluating personality functioning, additional assessment methods should be used to differentiate this domain from repetitive thoughts and behaviors.

Although the SCL-90 is often used in China to assess the psychological status of MHD patients,[21,36] the present results indicated that the measurement model of the reconstructed 8-domain CCSM demonstrated better fit than the SCL-90. The 8-domain CCSM contains 14 questions; thus, this scale could facilitate the rapid screening of psychological problems and psychological comorbidities in Chinese MHD patients. In contrast, the SCL-90 is cumbersome and time-consuming, and patients with chronic diseases (such as MHD patients) are often physically unable to complete the whole scale. In this study, 10 participants did not have the energy to fully complete the SCL-90. Additionally, the CR and validity evidence (regarding discriminant validity) showed that the SCL-90 is superior to the 8-domain CCSM, but the reconstructed CCSM demonstrated good CR and evidence of validity for most domains, so could be used to assess Chinese MHD patients. However, this scale has some problems. For example, it showed measurement error owing to the small number of questions. Comparatively, most psychological factors on the SCL-90 are evaluated using 6 to 12 questions. The present findings showed that the AVE of the SCL-90 was better than the CCSM AVE; the SCL-90 AVE was close to 0.5 and indicated minimal measurement error. Another deficiency of the 8-domain CCSM is that the scale was poor in differentiating personality functioning from repetitive thoughts and behaviors, indicating weak discriminant validity for these two domains. To adequately discriminate between these two domains, standard diagnostic procedures and other assessment tools, such as the Structured Clinical Interview for DSM-5, may be useful. The present analysis showed mixed evidence of reliability and validity for both instruments, suggesting that neither measure is ideal. A good assessment tool is needed to help clinicians evaluate and understand psychopathy in MHD patients; such a tool should be simple to administer, possess a suitable measurement model based on the internal structure and demonstrate evidence of effectiveness.[15,37]

This study has several limitations. First, a single factor measurement model was used rather than a multifactor measurement model. This was because many domains/factors of the 8-domain CCSM have only one question, so the scale is not suitable for the fitting of a multifactor measurement model. Second, our analysis of the measurement models of the two scales did not include path analysis, as we lacked the null hypotheses required for path analysis (e.g., the hypothesis that the depression factor would affect the somatic factor). Third, the data were obtained from the hemodialysis center of one hospital. The sample size was therefore relatively small, which may have affected the performance of the fit indexes and the statistical results of the model. Specifically, the AVE of the two scales needs to be confirmed with a larger sample size. Finally, the quality of the results of this study may have been affected by patient psychological defense and self-protection mechanisms. During the self-rating process, patients may have concealed or underreported the severity of some psychiatric symptoms, so the results may not reflect their actual experiences.

Conclusion

The results of this study show that the 8-domain Chinese version of the CCSM is relatively simple and could be used as a rapid screening tool to identify MHD patients at high risk of depression, anger, anxiety, somatic symptoms, suicidal thoughts, sleep disorders, personality functioning, and repetitive thoughts and behaviors. This instrument could be used as a preliminary self-rating scale to help physicians who are not psychiatric specialists to identify mental health disorders in MHD patients in a timely manner.