Validation of the Korean Version of the 15-Item Community Assessment of Psychic Experiences in a College Population.

OBJECTIVE: To investigate the reliability and validity of the Korean version of the Community Assessment of Psychic Experiences-15 item positive scale (CAPE-15) in college students.
METHODS: This study had two stages: initial screening with self-report questionnaires including the CAPE-15, and semi-structured interviews to investigate the instrument's diagnostic validity. The initial screening involved 1,749 college students. The modified Korean version of Prodromal Questionnaire-16 item (mKPQ-16) was also administered. The criteria for ultra-high risk (UHR) of psychosis in the Comprehensive Assessment of At-Risk Mental States (CAARMS) were the gold standard for diagnosis.
RESULTS: Twelve of the interviewed subjects met the CAARMS criteria for UHR of psychosis. The area under the receiver operating characteristic curve was highest (0.936) for the CAPE-15 distress score (p<0.001). The use of 6 as the cutoff for the CAPE-15 distress score resulted in the best balance of sensitivity (91.7%) and specificity (85.2%), with a favorable positive predictive value of 32.4%. The coefficients of correlation between the CAPE-15 and mKPQ-16 were significant.
CONCLUSION: The Korean version of the CAPE-15 is a good instrument for screening for psychosis risk in collegiate settings. The validation of this scale could contribute to the early identification of psychosis in the Korean community.

INTRODUCTION

The retrospective concept of “prodrome” was replaced with the terms “clinical high risk” (CHR) or “ultra-high risk” (UHR) of psychosis to emphasize the prospective nature of psychotic disorders [1]. The Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) includes “attenuated psychosis syndrome” (APS) as a new diagnosis requiring further research [2]. This new diagnosis has been developed based on evidence that treatment and intervention in these UHR or CHR states can prevent or delay the onset of psychosis [3]. Timely screening for and identification of APS in young individuals in community and school settings are required [4].

To increase the efficiency of identifying individuals at risk of psychosis, a preferable strategy is a stepwise process, with initial screening via self-report and subsequent detailed assessment using a structured interview. Brief self-reported screening instruments for initial pre-diagnostic filtering of the risk of psychosis before entering into a structured in-depth diagnostic evaluation are needed [5,6]. In Korea, a few screening tools for UHR of psychosis have been validated, including the 16- item version of the Prodromal Questionnaire (PQ-16) [7,8], Prodromal Questionnaire-brief version (PQ-B) [9,10], and the Eppendorf Schizophrenia Inventory (ESI) [6,11]. However, calculation of the ESI score is somewhat complicated because there are many items and a mixture of validity items in the questionnaire. The positive predictive value (PPV) of the original PQ-16 was found to be low (13.8%). Compared with the number of items assessing perceptual abnormalities, the PQ-16 has relatively few items assessing thought disorders and cognitive disturbances. Therefore, to increase the PPV of the PQ-16, we modified Korean version of the PQ-16 (mKPQ-16) by adding three ESI items (ideas of reference, cognitive impairment, and persecutory ideas) [7].

The Community Assessment of Psychic Experiences-Positive 15-items scale (CAPE-15) is another brief, valid screening tool used to identify people at high risk of psychosis in the community [12]. This study examined the reliability and validity of the Korean version of the CAPE-15 in university students.

METHODS

Subjects and procedures

This study had two stages: initial screening with self-report questionnaires, including the CAPE-15, and semi-structured interviews to investigate the instrument’s diagnostic validity. The semi-structured interviews were conducted by phone with subjects who gave informed consent by trained mental health professionals (clinical psychologists, nurses, and social workers). In the second evaluation, the subjects were assessed with the Comprehensive Assessment of At-Risk Mental States (CAARMS) [13] to investigate whether they truly had UHR symptoms. When investigators strongly suspected that subjects did not meet the CAARMS UHR criteria, they made final diagnoses of no UHR of psychosis. Subjects whose diagnoses were not confirmed by phone interviews were then interviewed face-to-face by experienced psychiatrists. Final diagnoses were reviewed at a research team meeting with experienced psychiatrists.

All study subjects were college students aged between 18 and 30 years. In the classroom setting, professors and research associates provided an explanation of the purpose and procedure of this study to the prospective student participants (n=3,000), 2,010 (67.0%) of whom voluntarily completed the anonymous self-screening questions. Of these, 261 students older than 30 years of age were excluded and 1,749 students were the final subjects for the initial analysis. Of the initial study population, 1,225 (70.0%) consented to potentially being contacted a second time and thus provided their phone number. In line with the procedure of our previous validation studies [7], individuals with CAPE-15 scores ≥4 (n=224, 18.3%) and with depressive symptoms regardless of CAPE-15 score (n=76, 6.2%) were selected for semi-structured interviews. Of the subjects for whom the next investigation was indicated (n=300), 191 (63.7%) participated in a semi-structured interview. From these procedure, 20 participants were invited for face-to-face interviews, and 14 subjects underwent the face-to-face interviews. Seven more college students who visited an early intervention center for the evaluation of psychosis-like symptoms during the study period were also included in the final study population for diagnosis validation after completing the same face-to-face interview.

The Institutional Review Board of Chonnam National University Hospital approved this study (CNUH-2018-228), and all participants provided informed consent. The initial prescreening was performed in September and October 2018 and the diagnostic interviews were conducted between October 2018 and March 2018.

Measures

The initial screening questionnaires included questions regarding demographic characteristics, in addition to the Patient Health Questionnaire (PHQ-9), mKPQ-16, and CAPE-15. The PQ-16 is a self-report screening questionnaire that assesses the presence of attenuated psychotic symptoms using dichotomous questions (true/false). The level of distress associated with each symptom is rated on a four-point scale ranging from “no distress” to “much distress.” The modified Korean version of the PQ-16 includes three additional item drawn from the ESI, in addition to the 16 items of the original instrument [7,8].

The original CAPE is a self-report screening questionnaire that comprised 42 items [14]. The CAPE-15 is a shorter subscale that has been widely used as a screening tool to identify people at UHR in a clinical setting [15-17]. It assesses the presence of psychotic-like experiences with a three-factor structure: persecutory ideation (five items), bizarre experiences (seven items), and perceptual abnormalities (three items). It measures both the frequency of and distress associated with these experiences. Each item uses a 4-point Likert scale from 0, ‘never,’ through ‘sometimes’ and ‘often,’ to 3, ‘nearly always.’ For each endorsed item, distress was rated on a 4-point Likert scale from 0, ‘not distressed,’ through ‘a bit distressed’ and ‘quite distressed,’ to 3, ‘very distressed,’ which produced a total score of 0–45. Higher scores indicated a higher frequency of psychotic experiences and an increased level of distress due to these experiences. Three authors (SWK, YSK, and JSY) translated the CAPE-15 into Korean after obtaining permission from its original authors. Bilingual translators majoring in medicine back-translated the CAPE-15 into English, after which the instrument was finally edited. No significant discrepancies between the translated and original instruments were identified.

The CAARMS was used as the gold standard for UHR diagnosis [13]. The Korean version of the CAARMS, which was developed through a standard translation/back-translation procedure (followed by a final revision based on feedback from the original authors) and applied in various studies validating screening measures for UHR populations, was also used in this study [6,18]. The original CAARMS consists of seven subscales, but symptomatic criteria for psychosis risk are based exclusively on positive symptom items (disordered thought contents, perceptual abnormalities, and disorganized speech). In this study, four items from the positive symptom subscale were administered to subjects for the diagnosis of UHR of psychosis.

The Korean version of the nine-item Patient Health Questionnaire (PHQ-9), validated by Han et al. [19,20], was administered to measure the level of depressive symptoms.

Statistical analyses

Using data from the screening samples, the degree of correlation between the CAPE-15 and mKPQ-16 scores was assessed using Spearman’s correlation coefficient to examine convergent validity. Associations among these scales and PHQ-9 were investigated with Spearman’s correlation. The internal consistency was estimated with Cronbach’s alpha. Using data from the subjects who underwent structured evaluations, we conducted receiver operating characteristic (ROC) analyses for the diagnosis of UHR by the CAARMS interview. We determined the optimal cutoff values for the CAPE-15 frequency and distress scores based on the sensitivity, specificity, PPV, and negative predictive value (NPV). The accuracy of a test depends on how well it separates the group being tested into those with and without the disease in question, and is measured by the area under the curve (AUC). The validity of the mKPQ-16 item total score, which we previously proved to be the most valid measure, was also calculated for the study subjects to compare with the CAPE-15. The test-retest reliability of the CAPE-15 was estimated using the intraclass correlation coefficient (ICC) in the subjects who went through face-toface interviews. The data were analyzed using SPSS for Windows (ver. 21.0; IBM Corp., Armonk, NY, USA), and p-values< 0.05 were deemed statistically significant.

RESULTS

Diagnostic results

The mean age of the subjects who participated in the first screening (n=1,749) was 21.8±1.9 years. Female participants were more numerous than male participants (64.3%). For the screening dataset, Table 1 shows the Spearman’s coefficients of correlation between the mKPQ-16 and CAPE. Scores for the CAPE-15 frequency and distress scores were significantly associated with the mKPQ-16 item total and distress scores (all p-values<0.001). The PHQ-9 scores were significantly associated with the CAPE and mKPQ-16 scores. The correlation between frequency and distress scores of the CAPE-15 was strong (r=0.878, p<0.001).

Table 1.

Spearman’s coefficients of correlation among the CPAE-15, mKPQ-16, and PHQ-9

	CAPE-15 frequency	CAPE-15 distress	PHQ-9
CAPE-15 frequency		0.878[*]	0.465[*]
CAPE-15 distress	0.878[*]		0.419[*]
mKPQ-16 frequency	0.564[*]	0.596[*]	0.493[*]
mKPQ-16 distress	0.518[*]	0.578[*]	0.504[*]

all p-values<0.001.

CAPE-15: 15-item Community Assessment of Psychic Experiences, mKPQ-16: modified Korean version of the Prodromal Questionnaire-16 item, PHQ-9: nine-item Patient Health Questionnaire

Receiver operating characteristic analysis

The mean age of the subjects who underwent further investigations (n=198) was also 20.8±2.1 years, and females (74.3%) predominated. Among the subjects who were interviewed, 12 met the CAARMS criteria for UHR of psychosis. Two subjects met subthreshold frequency criteria and 10 subjects met subthreshold intensity criteria. To predict CAARMS-based diagnoses of UHR of psychosis, ROC curves were plotted (Figure 1). The AUCs were all significant for all measures, which were put into the model (all p-values<0.001). The AUC of the CAPE-15 distress score was the greatest (0.936) among the three measures.

Figure 1.

The receiver operating characteristic (ROC) curves of the CAPE-15 and mKPQ-16 with CAARMS criteria of ultra-high risk of psychosis as a gold standard. CAPE15: 15-item Community Assessment of Psychic Experiences, mKPQ-16: modified Korean version of the Prodromal Questionnaire-16.

Validity and reliability

Table 2 summarizes the diagnostic validity of the CAPE-15 based on the AUCs and optimal cutoff points for sensitivity and specificity. For the CAPE-15 distress score, a cutoff score of 6 resulted in the best balance of sensitivity (91.7%) and specificity (85.2%), with a favorable PPV of 32.4%. For the CAPE-15 frequency score, a cutoff score of 6 resulted in the greatest sensitivity (100%), despite a relatively low specificity (67.6%) and a PPV of 16.9%. A cutoff item total mKPQ-16 score of 7 showed favorable sensitivity (83.3%) and specificity (79.1%), and a PPV of 20.8%.

Table 2.

Diagnostic validity of the CPAE-15 and mKPQ-16 based on the area under receiver operating characteristics curve (AUROC) and the optimal cutoff points for sensitivity and specificity

Scale	Score	AUROC	95% CI	p-value	Cutoff	Sensitivity	Specificity	PPV	NPV
CAPE-15	Frequency	0.872	0.803–0.940	<0.001	6	100%	67.6%	16.9%	100%
	Distress	0.936	0.892–0.980	<0.001	6	91.7%	85.2%	32.4%	99.2%
mKPQ-16	Item total	0.839	0.724–0.953	<0.001	7	83.3%	79.1%	20.8%	98.6%

CAPE-15: 15-item Community Assessment of Psychic Experiences, mKPQ-16: modified Korean version of the Prodromal Questionnaire-16 item, PPV: positive predictive value, NPV: negative predictive value, CI: confidence interval

Cronbach’s alpha values for the CAPE-15 frequency and distress items were 0.867 and 0.835, respectively. The test-retest reliability was evaluated using data from subjects who underwent face-to-face evaluations, and the ICC for the CAPE-15 frequency and distress scores was 0.659 (p=0.010) and 0.859 (p<0.001), respectively.

DISCUSSION

This study validated the Korean version of the CAPE-15 as a screening measure for psychosis risk in a community setting. The Korean version of the CAPE-15 showed good internal consistency and diagnostic validity, successfully distinguishing subjects at UHR of psychosis from the general population. It showed favorable convergent validity with the mKPQ-16. In this study, the cutoff score of the mKPQ-16 for identifying psychosis risk was the same and its PPV was similar to those of our previous validation study for the mKPQ-16.7 This suggests that the process used in the current study was credible and showed the stability of the mKPQ-16 as a screening tool. The sensitivities and specificities of the PQ-16, PQ-B, and ESI were below 80% in previous studies [6,7,10], but those of the CAPE-15 in this study were 91.7% and 85.2%, respectively. The AUROC and PPV were higher for the CAPE-15 than for the mKPQ-16. It appears that the CAPE-15 is a more valid and useful screening tool compared with scales previously validated for use in Korean populations, and thus should facilitate early identification of psychosis in Koreans.

In this study, the PPV of the CAPE-15 distress score was improved (32.4%) compared with previous screening tools [6,7,10], but it was still relatively low for a screening tool with strong predictive power. Nevertheless, when a tool is used to screen for a serious, low-prevalence disease, such as psychosis, the chosen cutoff point should provide good sensitivity, even when the PPV is low or the false-positive rate is high [6,21]. However, clinicians should not use a screening tool for psychosis risk syndrome to give a final diagnosis, but simply for pre-testing the general population to identify those needing further clinical interviews, and should realize that screening the general population for psychosis risk using a self-report questionnaire could result in a high false-positive rate unless other factors are considered [21-23].

The validity of the distress score of the CAPE-15 was better than that of the frequency score of the CAPE-15. Functional decline and high subjective distress should be considered when an individual is diagnosed with UHR of psychosis [7,13]. Furthermore, given the problem of the high false-positive rate and the potential stigma in the non-help seeking population, distress level is important for detecting the true at-risk population [23-25].

In this study, the CAPE-15 and mKPQ-16 scores were significantly associated with the scores of the PHQ-9. This correlation between psychosis risk symptoms and depression was compatible with previous studies [17,26]. Comorbid depression is present in about 40% of those at UHR for psychosis [27,28]. Pathogenesis for comorbid depression and psychosis-like symptoms should be further investigated. Moreover, therapeutic strategy such as cognitive behavioral therapy for at-risk mental states should aim to manage both psychotic and depressive symptoms [29,30].

This study had several limitations. First, this population was mainly non-help seeking. In addition, the subjects in this study were young adults. Previous studies showed that children and adolescents were more likely to report symptoms indicative of UHR. Therefore, the CAPE-15 cutoff score identified in this study should be applied with caution in younger populations, and when used in the clinical setting [7,31]. Second, selection bias may have affected the results due to the loss of subjects from the study population. Finally, the number of individuals undergoing face-to-face clinical interviews was low. However, the results of the mKPQ-16 were similar to those in our previous study, suggesting that our current study procedure is reliable.

In conclusion, this study showed that the CAPE-15 is a good screening instrument for use in community and school settings. In particular, the better validity of the CAPE-15 suggests that it is a promising alternative to other brief self-report screening tools that are currently used to detect UHR. This validation of a questionnaire with a small number of items may make it feasible to screen large numbers of young adults in the community and shorten the duration of untreated psychosis through prompt early intervention.