The Community Assessment of Psychic Experiences: Optimal cut-off scores for detecting individuals with a psychotic disorder.

OBJECTIVES: The need for a brief screening tool for psychosis is widely recognized. The Community Assessment of Psychic Experiences (CAPE) is a popular self-report measure of psychosis, but a cut-off score that can detect those most likely to fulfill diagnostic criteria for psychotic disorder is not established.
METHODS: A case-control sample from the Genetic Risk and Outcome of Psychosis Project study (N = 1375, healthy individuals, n = 507, and individuals with a psychotic disorder, n = 868), was used to examine cut-off scores of the CAPE with receiver operating curve analyses. We examined 27 possible cut-off scores computed from a combination of scores from the frequency and distress scales of the various factors of the CAPE.
RESULTS: The weighted severity positive symptom dimension was most optimal in detecting individuals with a psychotic disorder (>1.75 cut-off; area under the curve = 0.88; sensitivity, 75%; specificity, 88%), which correctly identified 80% of the sample as cases or controls with a diagnostic odds ratio of 22.69.
CONCLUSIONS: The CAPE can be used as a first screening tool to detect individuals who are likely to fulfill criteria for a psychotic disorder. The >1.75 cut-off of the weighted severity positive symptom dimension provides a better prediction than all alternatives tested so far.

INTRODUCTION

The need for a brief instrument for the screening of psychotic disorder is widely recognized. There are comprehensive clinical diagnostic instruments, such as the Structured Clinical Interview for DSM Disorders, but these take a lot of time to administer. A brief screening instrument that can detect individuals with psychosis with a satisfactory reliability would be useful for researchers interested in the prevalence of psychotic disorder among the general population or in screening out individuals with a psychotic disorder in community samples. Furthermore, it could also be a relevant tool for clinicians to use during initial assessments.

One frequently used self‐report measure that may address this issue is the Community Assessment of Psychic Experiences (CAPE, Stefanis et al., 2002). The CAPE was developed to measure both the frequency and distress levels of psychotic experiences (PEs) within the positive (e.g., “Do you ever hear voices when you are alone?”), negative (e.g., “Do you ever feel that your emotions are blunted?”), and depressive symptom dimension in community settings (Stefanis et al., 2002), and has often been used in clinical samples (e.g. Hanssen et al., 2003; Pignon et al., 2019). The items are answered with a Likert scale ranging from 1 to 4 for both the frequency (“never” to “nearly always”) and the distress (“not distressed” to “very distressed”) scale. The CAPE has been shown to have good psychometric properties in many languages, including Dutch, French, English, German, Indonesian, Spanish, and Swedish (Brenner et al., 2007; Fonseca‐Pedrero et al., 2012; Jaya, 2017; Schlier et al., 2015; Ziermans, 2013). The original factorial structure of the CAPE includes three dimensions (positive symptoms, negative symptoms, and depressive symptoms (Stefanis et al., 2002)). However, the most recent psychometric examination of the CAPE shows that a nine‐cluster factorial structure has better model fit (Jaya et al., 2020; Schlier et al., 2015). This structure consists of a positive symptom dimension with five subdimensions (bizarre experiences, hallucinations, grandiosity, magical thinking, and paranoia), a negative symptom dimension with three subdimensions (affective flattening, avolition, and social withdrawal), and a depressive symptom dimension. The nine‐cluster factorial structure has been shown to have partial scalar measurement invariance between participants from low‐and‐middle‐income countries and high‐income countries (Jaya et al., 2020), and the binary version of the three‐dimensional factorial structure has been shown to have partial scalar measurement invariance too in Brazil, France, the Netherlands, Italy, Spain, and the UK participants (Pignon et al., 2019).

We identified three studies that investigated the usefulness of the CAPE as a screening tool, either for prodromal individuals in a help‐seeking sample (Bukenaite et al., 2017; Mossaheb et al., 2012) or for individuals with first‐episode psychosis in a help‐seeking sample (Boonstra et al., 2009). Bukenaite et al. (2017) examined a 15‐item version of the frequency scale of the positive symptom dimension of the CAPE and found a sensitivity of 77% and specificity of 58% with a cut‐off score of 1.47 (based on a scoring procedure of the CAPE that uses the average score, with a range from 1.00 to 4.00). This was an improvement on the previous study that used the complete 20‐item frequency scale of the positive symptom dimension in the same sample, finding a 67% sensitivity and 73% specificity with a cut‐off score of 3.20 (Mossaheb et al., 2012). In detecting individuals with first‐episode psychosis from a sample of referred patients, Boonstra et al. (2009) suggested a cut‐off score of 50 on the frequency and distress scale of the positive symptom dimension of the CAPE (based on using the sum scoring procedure with a possible range from 20 to 80) which showed a 77% sensitivity and 71% specificity. However, all of these studies used specific samples of help‐seeking prodromal or first‐episode patients. Thus, so far, no study has investigated the usefulness of the CAPE to distinguish individuals with psychotic disorders from healthy controls. However, this type of study is necessary to estimate the usefulness of the CAPE as a general screening tool and be able to compare its detection rates with those of other measures.

Furthermore, previous studies on cut‐off scores were conducted in 2009 and 2012 before the publication of factorial structure studies of the CAPE and have thus were not able to make use of the complete dimensions and subdimensions of the CAPE that are published later. Utilizing the subdimensions of the CAPE is important because a meta‐analysis of studies using the CAPE in 2016 recommended that the three‐dimensional factorial structure should be further divided into subdimensions to improve psychometric properties (Mark & Toulopoulou, 2016). Here is a demonstration on how improved psychometric properties result in better cut‐off score. In 2012, Mossaheb et al. (2012) published a study in which they examined cut‐off scores from four sets of scores by summing the items of the positive symptom dimension, negative symptom dimension, depressive symptom dimension and the CAPE total score. A year later, in 2013, Capra et al. published a study on the factorial structure of the CAPE and recommended the use of only 15 of the 20 items positive symptom dimension (CAPE‐P15, Capra et al., 2013). Four years later, using the newer CAPE‐P15 factorial structure of the CAPE, Bukenaite et al. (2017) re‐examined the sample from Mossaheb et al. (2012) and found improved specificity and sensitivity over the older study (Mossaheb et al., 2012). This demonstrates that incorporating the latest findings on the factorial structure of the CAPE can improve the detection accuracy of individuals with psychotic disorders.

In the present study, we used a holdout cross‐validation procedure to compare the discriminative ability of the CAPE cut‐off scores based on the dimensions from published factorial structures of the CAPE to select those that best differentiate between individuals with a diagnosis of a psychotic disorder and healthy controls.

METHOD

Participants and procedure

We analyzed data (database version 5.0) from the “Genetic Risk and Outcome of Psychosis Project” (GROUP) study (Korver et al., 2012). The GROUP study was designed to investigate risk and protective factors of non‐affective psychotic disorders. The baseline sample of the longitudinal GROUP study consisted of individuals with non‐affective psychotic disorders, their siblings, their parents, and a healthy control group. Participants were recruited from 36 mental healthcare institutions in the Netherlands and Belgium including four academic medical centers (Amsterdam, Groningen, Maastricht, and Utrecht). The diagnoses were made based on the DSM‐IV criteria and were obtained through a consensus of an independent psychiatrist and a structured interview delivered by a trained psychologist or psychiatrist. In case of incongruence, the site coordinators made a final decision on the diagnosis of the participants. The structured interview in three recruitment sites was the Comprehensive Assessment of Symptoms and History (CASH, Andreasen et al., 1992) and one recruitment site used the Schedules for Clinical Assessment for Neuropsychiatry (SCAN 2.1, Wing et al., 1990). Detailed information of the sample characteristics and recruitment methods has been previously published (Korver et al., 2012). Here, we used data from the sample of patients with a psychotic disorder (PD), including schizophrenia, schizoaffective disorder, schizophreniform disorder, delusional disorder, and brief psychotic disorder Not Otherwise Specified (NOS) and from healthy controls (HC) who did not fulfill criteria for any mental health diagnosis at the time of the assessment from the first wave (baseline) of the GROUP study who completed the CAPE. This data selection procedure resulted in 868 PD (78% of the original 1120 clinical sample) and 507 HC (86% of the original 590 controls), thus the combined sample used in this study was 1375 participants (80% of the original 1710 participants).

Measure: Community Assessment of Psychic Experience

The CAPE is a self‐report measure assessing lifetime PEs consisting of 42 items that include positive symptoms (20 items, e.g., “Have you ever felt that you were being persecuted in any way”, “Do you ever hear voices when you are alone?”), negative symptoms (14 items, e.g., “Do you ever feel that you experience few or no emotions at important events?”, “Do you ever feel that you have no interest to be with other people?”) and depressive symptoms (8 items, e.g., “Do you ever feel sad?”, “Do you ever feel pessimistic about everything?”; Stefanis et al., 2002). Each item's responses are recorded on 4‐point Likert scales from 1 to 4 indicating frequency (“never”, “sometimes”, “often”, “nearly always”) and distress (“not distressed”, “a bit distressed”, “quite distressed”, “very distressed”). The item's distress scale is not answered and recorded as “not distressed” if the item's corresponding frequency scale is answered “never”.

Based on the frequency and distress scales of the CAPE, four possible methods of score computations have been used in the literature. The most common computation is to use a single frequency or distress score, which creates an average score ranging between 1 and 4. A second option is to sum the frequency and distress scales to create an average score ranging from 2 to 8 (e.g., Mossaheb et al., 2012). The third option is to multiply each item's frequency score with its associated distress score. This creates a weighted severity score, with a mean total score ranging from 1 to 16 (e.g., Jaya et al., 2018). For example, an individual who reports to “sometimes” hear voices (frequency score of 2) and to feel “quite distressed” by this experience (distress score of 3) would receive a severity score of 6. The fourth option is to create a binary score of absence versus presence of each item (e.g., Pignon et al., 2021; Wigman et al., 2011; Yung et al., 2009). Absence was coded when participants answered “never” or “not distressed.” Presence was coded when participants answered “sometimes” to “nearly always” or “a bit distressed” to “very distressed.” However, we do not use this fourth option because we aimed to keep as much variance as possible.

We computed a total of 27 different scores of the CAPE measuring positive symptoms, negative symptoms, and depressive symptoms constructs. Six scores were computed from each of the frequency and distress scales of the original three‐dimensional factorial structure, which consists of the average of the frequency and distress scales of the 20 positive symptoms items, 14 negative symptoms items, and 8 depressive symptoms items (Stefanis et al., 2002). Sixteen scores were computed from the frequency and distress scales of the nine‐cluster factor structure (Schlier et al., 2015), which consists of the average of the frequency and distress scales of five subdimensions of positive symptoms (i.e. seven items were used to compute bizarre experiences subdimension, four items were used to compute hallucinations subdimension, two items were used to compute grandiosity subdimension, two items were used to compute magical thinking subdimension, and five items were used to compute paranoia subdimension) and three subdimensions of negative symptoms (i.e. four items were used to compute affective flattening subdimension, seven items were used to compute avolition subdimension, and four items were used to compute social withdrawal subdimension). Three scores were computed from the combination of frequency and distress scales for the positive symptom, negative symptom, and depression dimension as suggested by Mossaheb et al. (2012). In addition, Mossaheb et al. (2012) also suggest the creation of a score based on the sum of frequency and distress of three selected positive symptoms items. One score was computed from multiplying frequency and distress items from the positive symptom dimension to create a weighted severity score of positive symptoms (Jaya et al., 2018).

Analyses

Receiver‐operating characteristic (ROC) analysis was used to compute cut‐off scores of the dimension scores. The ROC curves along with area under the curve (AUC) statistic and its 95% confidence interval were reported. The AUC statistic is a measure of the diagnostic power of a test. A perfect test will have an AUC of 1.0 and an AUC of 0.5 means the test performs no better than chance. Cut‐off scores were selected based on the Youden index (Youden, 1950). Additionally, we computed the true positive rate (sensitivity), false negative rate (type II error rate), false positive rate (type I error rate), true negative rate (specificity), positive likelihood ratio, negative likelihood ratio, and diagnostic odds ratio for each dimensional cut‐off score. To aid interpretation, we also interpret our finding with a clinical significance guidance by Cicchetti et al. (1995) on diagnostic accuracy for sensitivity, specificity, and accuracy (i.e., below 70% is poor, 70%–79% is fair, 80%–89% is good, 90%–100% is excellent). ROC analyses were computed using the ROCIT package version 1.1.1 (Khan & Brandenburger, 2019).

In addition, to reduce the chance of overfitting, we used holdout cross‐validation (Koul et al., 2018). We randomly partitioned our dataset into a training dataset consisting of two‐thirds of the sample (n = 917, PD, n = 564, HC, n = 353) and a testing dataset consisting of one‐third of the sample (n = 458, PD, n = 304, HC, n = 154). Participants' characteristics and descriptive statistics of the CAPE scores of the training and testing dataset are reported in the supplementary. Furthermore, psychometric properties of the CAPE in the total, training, and testing dataset were examined by conducting confirmatory factor analyses on the original three‐dimensional and nine‐cluster factorial structure using the lavaan package ver. 0.6‐8 (Rosseel, 2012). They were estimated using maximum likelihood with robust standard errors. Model fit was assessed using the following criteria: Comparative Fit Index (CFI) > 0.95, Root Mean Square Error of Approximation (RMSEA) < 0.06, and Standardized Root Mean Square Residual (SRMR) < 0.08 (Hu & Bentler, 1999). The χ2 is reported but not used as a fit criterion because it tends to reject models that are based on a large sample size (Bentler & Bonett, 1980). Omega reliability estimates of each subdimension and dimension were computed with the semTools package ver. 0.5‐4. All analyses were conducted in R version 4.1.0 (R Core Team, 2021).

RESULTS

Participant characteristics

We present demographic characteristics of the total, PD and HC samples in Table 1. The participant characteristics of the testing and training dataset are available in the supplementary (Tables S1 and S2). The diagnostic information of the sample is presented in Table 2. The mean and standard deviations of all computed CAPE dimension scores of each sample group are presented in Table 3. Tables on the CAPE dimension scores of the testing and training dataset are available in the supplementary (Tables S3 and S4). The missing data of the total sample was minimal ranging from 88% (n = 1211) from the weighted severity positive symptoms score to 99% (n = 1374) from several variables such as frequency of positive symptoms, frequency of negative symptoms, and 10 other variables. Outliers were not removed because psychotic symptoms scores vary greatly in the field and this should be reflected in our data.

TABLE 1

Participants' characteristics

Characteristic	Total sample (N = 1375)		Individuals with a psychotic disorder (n = 868)		Healthy individuals (n = 507)
	n/M	%/SD	n/M	%/SD	n/M	%/SD
Age (M, SD)	28.67	9.03	27.98	7.99	29.87	10.48
Sex (n, %)
Male	900	65.45%	655	75.46%	245	48.32%
Female	475	34.55%	213	24.54%	262	51.68%
Highest educational degree (n, %)
No education (0)	7	0.51%	7	0.81%	0	0.00%
Primary school (1)	114	8.29%	101	11.64%	13	2.56%
Secondary school (2–3)	343	24.95%	264	30.41%	79	15.58%
High school (4–5)	381	27.71%	220	25.35%	161	31.76%
Vocational education (6–7)	429	31.20%	231	26.61%	198	39.05%
University	89	6.47%	34	3.92%	55	10.85%
Marital status (n, %)
Not married	1028	74.76%	741	85.37%	287	56.61%
Married/living together	265	19.27%	80	9.22%	185	36.49%
Divorced/Widowhood	40	2.91%	26	3.00%	14	2.76%

Note: Secondary school consists of the following Dutch system of secondary school, LBO/HH/LHNO/VBO and MAVO/VMBO; high school consists of HAVO and VWO; vocational education consists of MBO and HBO. The PD sample was younger than the HC sample (t[850.83] = 3.51, p < 0.001), had fewer females (χ2[1, N = 1375] = 103.04, p < 0.001), had lower education level (χ2[8, N = 1375] = 151.93, p < 0.001), and fewer were married (χ2[3, N = 1375] = 161.52, p < 0.001).

TABLE 2

Diagnostic information of the total sample (N = 1375)

Characteristic	Diagnostic code DSM‐IV‐TR	Individuals with a psychotic disorder (n = 868)		Healthy individuals (n = 507)
		n	%	n	%
Schizophrenia, disorganized type	295.1	35	4.03%	0	0%
Schizophrenia, catatonia type	295.2	1	0.12%	0	0%
Schizophrenia, paranoid type	295.3	487	56.11%	0	0%
Schizophreniform disorder	295.4	43	4.95%	0	0%
Schizophrenia, residual type	295.6	19	2.19%	0	0%
Schizoaffective disorder	295.7	102	11.75%	0	0%
Schizophrenia, undifferentiated type	295.9	52	5.99%	0	0%
Delusional disorder	297.1	17	1.96%	0	0%
Brief psychotic disorder	298.8	23	2.65%	0	0%
Psychotic disorder NOS	298.9	89	10.25%	0	0%
Diagnosis deferred	799.9	0	0.00%	3	0.59%
Bereavement	V62.82	0	0.00%	1	0.20%
No diagnosis	V71.09	0	0.00%	503	99.21%

Abbreviation: NOS, not otherwise specified.

TABLE 3

Means and standard deviations of the CAPE dimension scores

Measure	Total sample (N = 1375)			Individuals with a psychotic disorder (n = 868)			Healthy individuals (n = 507)
	n	M	SD	n	M	SD	n	M	SD	Possible score range
Three dimensions
F.POS	1374	1.50	0.47	867	1.68	0.50	507	1.19	0.17	1–4
F.NEG	1374	1.82	0.54	867	2.02	0.53	507	1.47	0.32	1–4
F.DEP	1374	1.84	0.54	867	2.00	0.58	507	1.56	0.33	1–4
D.POS	1374	1.43	0.52	867	1.64	0.55	507	1.07	0.12	1–4
D.NEG	1373	1.69	0.61	866	1.91	0.62	507	1.32	0.34	1–4
D.DEP	1371	1.85	0.70	864	2.06	0.72	507	1.49	0.48	1–4
Nine‐cluster
F.BIZ	1374	1.40	0.52	867	1.58	0.56	507	1.07	0.16	1–4
F.HAL	1369	1.30	0.49	864	1.47	0.55	505	1.02	0.09	1–4
F.PAR	1374	1.66	0.57	867	1.84	0.61	507	1.35	0.30	1–4
F.MAG	1373	1.74	0.81	866	1.92	0.86	507	1.43	0.59	1–4
F.GRA	1371	1.61	0.75	864	1.79	0.82	507	1.30	0.48	1–4
F.SOW	1373	1.91	0.62	866	2.12	0.62	507	1.57	0.43	1–4
F.AFF	1373	1.73	0.72	866	1.98	0.74	507	1.30	0.43	1–4
F.AVO	1373	1.80	0.55	866	1.99	0.57	507	1.49	0.33	1–4
D.BIZ	1374	1.40	0.59	867	1.61	0.65	507	1.03	0.11	1–4
D.HAL	1367	1.32	0.56	862	1.51	0.64	505	1.01	0.06	1–4
D.PAR	1373	1.62	0.69	866	1.87	0.74	507	1.20	0.28	1–4
D.MAG	1368	1.34	0.66	862	1.52	0.77	506	1.04	0.18	1–4
D.GRA	1364	1.34	0.71	857	1.52	0.84	507	1.06	0.22	1–4
D.SOW	1369	1.66	0.67	862	1.88	0.69	507	1.28	0.41	1–4
D.AFF	1370	1.63	0.76	863	1.87	0.81	507	1.22	0.41	1–4
D.AVO	1370	1.74	0.65	863	1.95	0.67	507	1.38	0.40	1–4
Mossaheb et al. (2012)
DF.POS	1374	1.46	0.48	867	1.66	0.50	507	1.13	0.14	1–8
DF.NEG	1374	1.76	0.55	867	1.97	0.55	507	1.39	0.31	1–8
DF.DEP	1374	1.84	0.60	867	2.03	0.62	507	1.52	0.39	1–8
3 items	1374	1.50	0.67	867	1.77	0.71	507	1.05	0.15	1–4
Jaya et al. (2018)
SEV.POS	1211	2.57	1.96	742	3.35	2.14	469	1.34	0.40	1–16

Note: Presented scores are average scores (i.e., the mean of item scores).

Abbreviations: AFF, affective flattening (3 items); AVO, avolition (7 items); BIZ, bizarre experiences dimension (7 items); CAPE, community assessment of psychic experiences; D, distress scale; DEP, depression dimension (8 items); DF, sum of frequency and distress scale following Mossaheb et al. (2012); F, frequency scale; GRA, grandiosity dimension (2 items); HAL, hallucinations dimension (4 items); MAG, magical thinking dimension (2 items); NEG, negative symptoms dimension (14 items); PAR, paranoia (5 items); POS, positive symptoms dimension (20 items); SSEV.POS, severity positive symptom dimension scale is computed by multiplying (weighting) each frequency answer with its corresponding distress score following Jaya et al. (2018)OW, social withdrawal dimension (4 items); Three items, sum of frequency and distress scale of Q7, Q31, and Q33 recommended in Mossaheb et al. (2012).

The average age of the total sample was 28.7 years (SD = 9.0), ranging from 15 to 61 years and 65% were male. The PD sample was younger than the HC sample (t (850.83) = 3.51, p < 0.001), had fewer females (χ2 [1, N = 1375] = 103.04, p < 0.001), had lower education level (χ2 [8, N = 1375] = 151.93, p < 0.001), and fewer were married (χ2 [3, N = 1375] = 161.52, p < 0.001).

Reliability and factorial structure of the CAPE

In the total dataset, the three‐dimensional factorial structure met the fit criteria for RMSEA and SRMR, but not CFI (χ[816] = 3263.278, p < 0.001, CFI = 0.857, RMSEA = 0.057 [90% CI 0.055, 0.059], SRMR = 0.052, AIC = 108,576). The Omega reliability estimates of the three‐dimensional factorial structure were good (positive symptom dimension, ω = 0.92; negative symptom dimension, ω = 0.91; depressive symptom dimension, ω = 0.87). Similarly, the nine‐cluster factorial structure met the fit criteria for RMSEA and SRMR, but not CFI (χ[808] = 2171.591, p < 0.001, CFI = 0.921, RMSEA = 0.043 [90% CI 0.041, 0.045], SRMR = 0.053, AIC = 106,932). The Omega reliability estimates of the nine‐cluster factorial structure were acceptable (positive symptom dimension, ω = 0.88; bizarre experiences subdimension, ω = 0.85, hallucinations subdimension, ω = 0.81, grandiosity subdimension, ω = 0.81, magical thinking subdimension, ω = 0.65, paranoia subdimension, ω = 0.79; negative symptom dimension, ω = 0.92; affective flattening subdimension, ω = 0.82, avolition subdimension, ω = 0.83, social withdrawal subdimension, ω = 0.78; depressive symptom dimension, ω = 0.88).

A similar pattern of results was found in the training and testing dataset. In the training dataset the three‐dimensional (χ[816] = 2278.401, p < 0.001, CFI = 0.860, RMSEA = 0.058 [90% CI 0.056, 0.061], SRMR = 0.058, AIC = 62,455) and nine‐cluster factorial structure (χ[808] = 1617.551, p < 0.001, CFI = 0.923, RMSEA = 0.043 [90% CI 0.040, 0.047], SRMR = 0.058, AIC = 61,424) met the fit criteria for RMSEA and SRMR, but not CFI. The Omega reliability estimates of the three‐dimensional factorial structure were good (positive symptom dimension, ω = 0.92; negative symptom dimension, ω = 0.91; depressive symptom dimension, ω = 0.88), and likewise for the nine‐cluster factorial structure (positive symptom dimension, ω = 0.89; bizarre experiences subdimension, ω = 0.86, hallucinations subdimension, ω = 0.81, grandiosity subdimension, ω = 0.82, magical thinking subdimension, ω = 0.69, paranoia subdimension, ω = 0.78; negative symptom dimension, ω = 0.93; affective flattening subdimension, ω = 0.85, avolition subdimension, ω = 0.84, social withdrawal subdimension, ω = 0.79; depressive symptom dimension, ω = 0.89).

Again, in the testing dataset, both factorial structures met the fit criteria for RMSEA and SRMR, but not CFI (three‐dimensional, χ[816] = 1583.627, p < 0.001, CFI = 0.840, RMSEA = 0.059 [90% CI 0.054, 0.063], SRMR = 0.059, AIC = 32,638; nine‐cluster, χ[808] = 1205.801, p < 0.001, CFI = 0.918, RMSEA = 0.042 [90% CI 0.037, 0.047], SRMR = 0.062, AIC = 32,076). The Omega reliability estimates in the testing dataset were similar to those in the training dataset (three‐dimensional: positive symptom dimension, ω = 0.91; negative symptom dimension, ω = 0.90; depressive symptom dimension, ω = 0.86; nine‐cluster: positive symptom dimension, ω = 0.87; bizarre experiences subdimension, ω = 0.83, hallucinations subdimension, ω = 0.82, grandiosity subdimension, ω = 0.80, magical thinking subdimension, ω = 0.54, paranoia subdimension, ω = 0.80; negative symptom dimension, ω = 0.92; affective flattening subdimension, ω = 0.82, avolition subdimension, ω = 0.82, social withdrawal subdimension, ω = 0.78; depressive symptom dimension, ω = 0.87).

The cut‐off scores of the CAPE

The ROC analyses on the cut‐off scores of the 27 dimensional scores from the total dataset (N = 1375) are presented in Table 4, and a similar table from the training (n = 917) and testing (n = 458) dataset are presented in Tables S5 and S6, respectively).

TABLE 4

Cut‐off points of various dimensional scores of the CAPE (N = 1375)

Measure	Cut‐off points	AUC	95% CI: Lower	95% CI: Upper	True positive rate (sensitivity)	False negative rate (type II error rate)	True negative rate (1‐specificity)	False positive rate (type I error rate)	Positive likelihood ratio	Negative likelihood ratio	Diagnostic odds ratio	Accuracy
Three dimensions
F.POS	1.44	0.836	0.815	0.856	64.24%	35.76%	91.32%	8.68%	7.40	0.39	18.91	74.24%
F.NEG	1.77	0.815	0.793	0.836	68.51%	31.49%	82.45%	17.55%	3.90	0.38	10.22	73.65%
F.DEP	1.80	0.747	0.722	0.773	57.55%	42.45%	82.64%	17.36%	3.32	0.51	6.46	66.81%
D.POS	1.25	0.866	0.847	0.884	70.47%	29.53%	92.11%	7.89%	8.93	0.32	27.86	78.46%
D.NEG	1.57	0.800	0.777	0.823	69.28%	30.72%	79.68%	20.32%	3.41	0.39	8.85	73.12%
D.DEP	1.71	0.745	0.719	0.770	64.70%	35.30%	75.15%	24.85%	2.60	0.47	5.54	68.56%
Nine cluster
F.BIZ	1.29	0.833	0.813	0.854	67.13%	32.87%	88.17%	11.83%	5.67	0.37	15.21	74.89%
F.HAL	1.25	0.800	0.777	0.823	62.31%	37.69%	94.67%	6.14%	10.41	0.38	25.49	74.95%
F.PAR	1.80	0.750	0.725	0.775	52.60%	47.40%	86.98%	13.02%	4.04	0.54	7.41	65.28%
F.MAG	2.00	0.669	0.641	0.698	51.96%	48.04%	74.75%	25.25%	2.06	0.64	3.20	60.38%
F.GRA	2.00	0.684	0.656	0.712	46.18%	53.82%	83.23%	16.77%	2.75	0.65	4.26	59.88%
F.SOW	2.00	0.765	0.741	0.790	63.97%	36.03%	77.71%	22.29%	2.87	0.46	6.19	69.05%
F.AFF	1.50	0.782	0.758	0.806	69.98%	30.02%	73.57%	26.43%	2.65	0.41	6.49	71.30%
F.AVO	1.83	0.772	0.748	0.797	60.39%	39.61%	83.83%	16.17%	3.73	0.47	7.90	69.05%
D.BIZ	1.14	0.833	0.812	0.854	73.13%	26.87%	87.97%	12.03%	6.08	0.31	19.89	78.60%
D.HAL	1.25	0.764	0.740	0.789	54.29%	45.71%	97.82%	2.18%	24.93	0.47	53.34	70.37%
D.PAR	1.50	0.784	0.761	0.808	61.43%	38.57%	86.19%	13.81%	4.45	0.45	9.94	70.58%
D.MAG	1.50	0.689	0.661	0.717	42.34%	57.66%	94.27%	5.73%	7.39	0.61	12.08	61.55%
D.GRA	1.50	0.658	0.629	0.687	37.69%	62.31%	92.70%	7.30%	5.16	0.67	7.68	58.14%
D.SOW	1.33	0.773	0.749	0.797	71.69%	28.31%	71.60%	28.40%	2.52	0.40	6.38	71.66%
D.AFF	1.67	0.754	0.729	0.779	59.44%	40.56%	82.45%	17.55%	3.39	0.49	6.88	67.96%
D.AVO	1.67	0.764	0.740	0.789	64.19%	35.81%	78.30%	21.70%	2.96	0.46	6.47	69.42%
Mossaheb et al. (2012)
DF.POS	1.78	0.865	0.847	0.884	36.68%	63.32%	99.80%	0.20%	185.96	0.63	293.09	59.97%
DF.NEG	2.13	0.817	0.795	0.839	36.79%	63.21%	96.65%	3.35%	10.97	0.65	16.78	58.88%
DF.DEP	1.86	0.753	0.728	0.779	57.44%	42.56%	83.83%	16.17%	3.55	0.51	6.99	67.18%
3 items	1.20	0.838	0.818	0.859	71.63%	28.37%	89.74%	10.26%	6.98	0.32	22.09	78.31%
Jaya et al. (2018)
SEV.POS	1.75	0.875	0.856	0.894	75.47%	24.53%	88.06%	11.94%	6.32	0.28	22.69	80.35%

Note: Cut‐off points are based on average scores.

Abbreviations: AUC, area under the curve; AVO, avolition (7 items); BIZ, bizarre experiences dimension (7 items); CAPE, community assessment of psychic experiences; D, distress scale; DEP, depression dimension (8 items); DF, sum of frequency and distress scale following Mossaheb et al. (2012); F, frequency scale; GRA, grandiosity dimension (2 items); HAL, hallucinations dimension (4 items); MAG, magical thinking dimension (2 items); NEG, negative symptoms dimension (14 items); PAR, paranoia (5 items); POS, positive symptoms dimension (20 items); SSEV.POS, severity positive symptom dimension scale is computed by multiplying (weighting) each frequency answer with its corresponding distress score following Jaya et al. (2018) SOW, social withdrawal dimension (4 items); Three items, sum of frequency and distress scale of Q7, Q31, and Q33 recommended in Mossaheb et al. (2012).

The ROC analyses from the training dataset revealed the top five highest Youden index cut‐off scores to be distress weighted severity positive symptom dimension (>1.75, J = 0.66), distress positive symptom dimension (>1.25, J = 0.64), sum of frequency and distress of the positive symptom dimension (>1.30, J = 0.63), distress of bizarre experience dimension (>1.14, J = 0.62), and sum of frequency and distress of three selected items from the positive symptom dimension (>1.20, J = 0.60). Of these cut‐off scores, the weighted severity positive symptom dimension had the highest accuracy (82%). Therefore, the weighted severity positive symptom dimension was selected for further examination. The ROC curve of the weighted severity positive dimension score from the total sample is presented in Figure 1.

FIGURE 1

Receiver‐operating characteristic curve, sensitivity versus 1‐specificity for severity positive symptom dimension. Severity positive symptom dimension scale is computed by multiplying (weighting) each frequency answer with its corresponding distress score following Jaya et al. (2018)

With the cut‐off point of >1.75 of the weighted severity positive symptom dimension in the training dataset, we found that 80% of the training sample was correctly identified as belonging to the PD or the HC. Specifically, 318 participants were correctly identified as HC, which was 90% of the total healthy sample (n = 345), and 370 participants were correctly identified as PD, which was 66% of the total PD sample (n = 566). We found 35 healthy individuals to be incorrectly identified as PD (10% of the total HC sample) and 194 individuals with a PD to be incorrectly identified as HC (34% of the total PD sample). The diagnostic odds ratio for the cut‐off point of >1.75 of the weighted severity positive symptom dimension was 26.84.

With the cut‐off point of >1.75 of the weighted severity positive symptom dimension in the testing dataset, we found that 78% of the total sample was correctly identified as belonging to the PD or the HC. Specifically, 133 participants were correctly identified as HC, which is 86% of the total healthy sample (n = 154), and 190 participants were correctly identified as PD, which was 63% of the total PD sample (n = 304). We found 21 healthy individuals to be incorrectly identified as PD (14% of the total HC sample) and 114 individuals with a PD to be incorrectly identified as HC (37% of the total PD sample). The diagnostic odds ratio for the cut‐off point of >1.75 of the weighted severity positive symptom dimension was 16.3.

DISCUSSION

The aim of the study was to discover which cut‐off score from the various dimensions of the CAPE can best differentiate between healthy individuals and individuals with a psychotic disorder. We found from the literature that there are 27 ways to produce scores from the various dimensions and subdimensions of the CAPE. Among these, the weighted severity positive symptom dimension with a cut‐off score of >1.75 out of 16.00 performed best in differentiating between healthy individuals and individuals with a psychotic disorder in our study. The weighted severity positive symptom dimension is computed by summing the multiplication of each positive symptom item's frequency score with its corresponding distress score. The cut‐off score had 75% sensitivity, 88% specificity, and 80% accuracy, which can be interpreted as fair sensitivity, good specificity, and good accuracy in clinical usage (Cicchetti et al., 1995).

Our proposed cut‐off score thus has better accuracy in comparison to previously suggested cut‐off scores that use the frequency or distress scale of the positive symptom dimension with a 77% sensitivity and 71% specificity (Boonstra et al., 2009) or the frequency scale of the CAPE‐P15 with a 77% sensitivity and 58% specificity (Bukenaite et al., 2017). However, it is important to note that our cut‐off score is appropriate to be used in non‐help‐seeker populations such as community samples to differentiate between healthy individuals and individuals with a psychotic disorder, while the previously published cut‐off score is more appropriate to be used in clinical settings to differentiate between help‐seekers without and with psychosis.

Even though our suggested cut‐off score from the weighted severity positive symptoms dimension performs better than previously published cut‐off scores and can be considered good in terms of clinical significance, the sensitivity and specificity statistics do not perform as well as commonly used screening tools for physical disorders, such as cancer, or for depression. Mammography, a well‐known screening tool for breast cancer can detect breast cancer with sensitivity of 87% and specificity of 97% in the Million Women Study (Banks et al., 2004). Furthermore, the Patient Health Questionnaire‐9 can detect major depressive disorder with a sensitivity of 88% and a specificity of 88% (Kroenke et al., 2001). More research into screening tools for psychosis is needed to improve the detection rate in order for it to be on par with these high precision tools.

Interestingly, we found a trend that the distress scale of positive symptoms dimension of the CAPE plays an important role in discriminating PD and HC sample whereas the frequency scale is less helpful. The emphasize on distress in evaluating the clinical relevance of psychotic symptoms is also found in our analysis of psychotic experiences in participants from 13 countries (Wüsten et al., 2018) and other studies looking at the relationship between religiosity and psychotic symptoms (e.g., Brito et al., 2021; Oh et al., 2018). However, there are exceptions in our results. The hallucination and grandiosity dimensions had higher AUC in the frequency than the distress scale. This may have methodological reasons, for example, the items in the hallucination dimensions appear to reflect stronger severity than in the other dimensions (e.g., hallucinations, “Do you ever hear voices when you are alone?”, compared to paranoia, “Do you ever feel as if you are being persecuted in some way?”). Although individuals with psychotic disorders have high level of depressive symptoms (Etchecopar‐Etchart et al., 2021), we found positive symptoms rather than depressive symptoms were the best indicator in discriminating individuals with a psychotic disorder from healthy individuals.

Strengths and limitations

The cut‐off score recommended in this study was based on the paper‐and‐pencil version of the CAPE using the Dutch version of the CAPE. Hence, how well the cut‐off is suited for different versions of the CAPE (e.g., the online version) or across different countries needs to be investigated in combination with issues related to measurement invariance of the CAPE. Furthermore, the CAPE is a self‐report scale and limitations to self‐report assessment such as social desirability and other response styles apply. Demographic characteristic of the sample such as the majority being male and single may have influenced the computation of the cut‐off. Additionally, because the case‐control sample was not drawn from a representative population sample, in which a base psychosis prevalence rate of 0.17% can be expected (Evensen et al., 2015), we were not able to compute the additional statistics (i.e., positive predictive value and negative predictive value) that would be helpful to further judge the accuracy of the CAPE in distinguishing between healthy individuals and individuals with a psychotic disorder. Future studies also need to examine the accuracy of the suggested cut‐off in a representative population sample consisting of healthy individuals, individuals with non‐psychotic mental disorders, and individuals with a psychotic disorder.

Conclusion

The distress weighted positive symptom dimension from the CAPE can be used as a preliminary screening tool to detect or screen out individuals with a high likelihood of having a psychotic disorder. It performs better than previously published cut‐off scores but does not reach the high precision of screening tools for depression. Thus, it cannot replace validated diagnostic assessment tools if the aim is to make a diagnosis.

CONFLICT OF INTERESTS

Edo S. Jaya, Therese van Amelsvoort, Agna A. Bartels‐Velthuis, Richard Bruggeman, Wiepke Cahn, Lieuwe de Haan, Rene S. Kahn, Jim van Os, Frederike Schirmbeck, Claudia J.P. Simons, and Tania M. Lincoln declared no conflict of interest.

Supporting Information S1

Click here for additional data file.