Validation of the Korean version of the 16-Item Prodromal Questionnaire in a Non-Help-Seeking College Population.

OBJECTIVE: To examine the reliability and validity of the Korean version of the 16-item Prodromal Questionnaire (KPQ-16) in non-help-seeking university students.
METHODS: Among 2,246 university students participated in the initial screening, 257 subjects with KPQ-16 scores ≥4 were interviewed. The criteria for ultra-high risk (UHR) of psychosis of the Comprehensive Assessment of At-Risk Mental States (CAARMS) were the gold standard for diagnosis. An exploratory modified version of the questionnaire (mKPQ-16), to which three items from the Eppendorf Schizophrenia Inventory were added, was also used to compensate for items on thought and cognitive problems.
RESULTS: Seventeen subjects met the CAARMS criteria for UHR of psychosis. The area under the receiver operating characteristic curve (AUROC) was highest for the mKPQ-16 total score (AUROC=0.831, p<0.001). The use of cutoff total scores of 7 for the mKPQ-16 and 6 for the KPQ-16 resulted in the best balance of sensitivity (76.5% and 64.7%, respectively) and specificity (75.4% and 71.2%, respectively).
CONCLUSION: The Korean versions of the PQ-16 are good instruments for screening for psychosis risk in university students. This validation of a questionnaire version with a small number of items may make it feasible to screen large numbers of young adults in the community.

INTRODUCTION

Schizophrenia is a severe mental illness that typically begins in adolescence or early adulthood. Intervention during the early stages of the disorder can reduce its ultimate severity [1]. However, individuals’ first contact with mental health professionals and psychiatric treatment tend to be delayed, even when they develop symptoms of psychosis [2,3]. Treatment delay and a long duration of untreated psychosis are associated strongly with a poor treatment response and outcomes [4,5]. The reported duration of untreated psychosis in Korea is longer than those in Western countries [6,7]. Therefore, a strategy for the early identification of psychosis in Korean young adults and adolescents is urgently needed. Recently, an effective strategy for the prevention of psychosis development from a prodromal state was reported [8,9].

The retrospective concept of “prodrome” was replaced with the terms “ultra-high risk” (UHR) and “clinical high risk” (CHR) of psychosis to emphasize the prospective nature of the evaluation and treatment of patients with early signs of illness [10]. Research findings regarding the pathogenesis and course of UHR of psychosis have also accumulated [11-13]. Three UHR/CHR criteria are in use: 1) attenuated psychotic symptoms (subthreshold, attenuated positive psychotic symptoms), 2) brief limited intermittent psychotic symptoms (short episodes of frank psychotic symptoms that resolve without treatment), and 3) trait vulnerability (schizotypal personality disorder or a first-degree relative with psychosis) in addition to a marked decline in psychosocial functioning or chronic low functioning [10,13]. The Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), includes “attenuated psychosis syndrome” (APS) as a new diagnosis in Section 3, which describes conditions requiring further research [14]. This new diagnosis has been developed based on evidence that treatment and intervention in these UHR or CHR states can prevent or delay the onset of psychosis [13]. However, many individuals with subthreshold symptoms do not seek mental health services early. Therefore, timely screening for and identification of APS in young individuals in community and school settings are necessary.

Two structured interview tools are used widely to identify individuals at UHR of psychosis: the Comprehensive Assessment of At-Risk Mental States (CAARMS) [15,16] and the Structured Interview for Prodromal Syndromes [17]. However, these structured face-to-face diagnostic interview tools are unsuitable for the screening and identification of UHR individuals in the general population; for those who do not seek specialized help; and for mental health gatekeepers who may not have sufficient time or expertise to complete structured interviews [18,19]. Therefore, brief self-reported screening instruments that are suitable for initial pre-diagnostic filtering of the risk of psychosis before entering into structured in-depth diagnostic evaluation are needed [18,19]. To increase the efficiency of identifying increased psychosis risk, a preferable strategy is a stepwise process, with initial screening via self-report and subsequent detailed assessment using a structured interview. Effective screening instruments should be reliable and valid, as well as brief, simple, and easily applicable [18].

The 16-item version of the Prodromal Questionnaire (PQ-16) is a self-report measure used widely to screen for UHR of psychosis in the general population [20]. The PQ-16 was derived from the original 92-item Prodromal Questionnaire [21], which takes approximately 20 minutes to complete, with the number of items reduced for easier administration in large populations. A European study demonstrated that the PQ-16 had good psychometric properties and validity in comparison with the original PQ [20]. In Korea, the Eppendorf Schizophrenia Inventory (ESI) [22] has been validated and used for the identification of individuals at UHR of psychosis [19]. The ESI was originally developed to assess subjective symptoms that are predominant in psychotic patients. It consists of 40 items, of which 34 are combined into four scales and 6 are used to evaluate frankness and general survey motivation. Although the ESI is a valid instrument for the screening of psychosis risk, a briefer, simpler measure is needed for mass screening. Therefore, this study examined the reliability and validity of the Korean version of the 16-item Prodromal Questionnaire (KPQ-16) in non-help-seeking university students.

METHODS

Subjects and procedures

This study had two stages: initial screening with the KPQ-16 and the performance of semi-structured interviews to investigate the instrument’s diagnostic validity. Freshman students aged ≥18 years completed the KPQ-16 as part of a packet of anonymous questionnaires at the beginning of the school year. After the study purpose and procedure were explained to first-year students (n=2,921), 2,246 (76.9%) students participated voluntarily in the initial screening. Based on the initial validation study conducted in Europe [20], individuals with scores ≥4 [97% sensitivity, 99% negative predictive value (NPV)] were selected for semi-structured interviews because we believed that the risk of missing true-positive subjects should be very low. In the screening test, 518 (23.1%) subjects had KPQ-16 scores ≥4. Our method was in line with those of previous validation studies, in which people with PQ-positive symptom scores in the top 20% of the distribution were selected for further investigation [23]. Exclusion criteria were age >25 years, a diagnosis of psychotic disorder, current or previous use of antipsychotic medication, and insufficient fluency in the Korean language.

Subjects with KPQ-16 scores ≥4 were assessed with the CAARMS to investigate whether they truly had UHR symptoms. Of the 518 subjects with such scores, 361 (70.0%) voluntarily gave us contact phone numbers. One subject was excluded from this analysis because he was older than 25 years. A trained clinical psychologist, psychiatric nurse, or psychiatric social worker first contacted each subject by telephone for an in-depth interview and interviewed him or her using the CAARMS. Of the 518 subjects, 73 (20.3%) subjects did not respond to more than three contact attempts and 30 (8.3%) subjects refused further evaluation, despite our recommendations. When investigators strongly suspected that subjects did not meet the CAARMS UHR criteria, they made final diagnoses of no UHR of psychosis (n=197, 54.7%). Further face-to-face interviews by psychiatrists who had been trained and had considerable experience with the CAARMS were arranged for 60 (16.7%) subjects. These subjects completed the ESI and KPQ-16 again to examine convergent validity and test-retest reliability. Final diagnoses were reviewed at a research team meeting with experienced psychiatrists.

The Institutional Review Board of Chonnam National University Hospital approved this study (CNUH 2016088), and all participants provided written informed consent. The initial pre-screening was administered in February 2016 and the diagnostic interviews were conducted between March and early April 2016. The interval between the first and second contacts was about 1 month.

Measures

PQ-16

The PQ-16 is a self-report screening questionnaire that assesses the presence of attenuated positive and negative psychotic symptoms on a 2-point scale (true/false) [20]. For each endorsed item, distress was rated on a 4-point scale (ranging from no distress to much distress) [21]. The initial total score, used to determine whether a further structured interview would take place, was calculated from the number of items with responses of true, and ranged from 0 to 16. The distress scale ranged from 0 to 96 and was also used for exploratory analyses. The PQ-16 consists of nine items on the perceptual abnormalities/hallucinations subscale, five items related to unusual thought content/delusional ideas/paranoia, and two negative symptoms. Compared with the number of items assessing perceptual abnormalities, relatively few items assess thought content and cognitive symptoms. The authors who validated the PQ-16 also stated that perceptual aberrations seem to be overrepresented [20]. Therefore, for exploratory investigation to increase the sensitivity of the questionnaire, we modified the KPQ-16 by adding three ESI items (4, 5, and 8; ideas of reference, cognitive impairment, and persecutory ideas); we called this instrument the modified Korean version of the 16-item Prodromal Questionnaire (mKPQ-16). The three added items were selected from those that showed high degrees of predictability in a validation study of the Korean version of the ESI [19], and based on our experience in screening with the ESI in community mental health service settings. The authors (SWK, JM, YSK) translated the PQ-16 into Korean after obtaining permission from the original authors of the PQ-16. Bilingual translators majoring in medicine back-translated the KPQ-16 into English. As no discrepancies between the translations and the original instrument, the final version of the KPQ-16 was completed.

CAARMS

The diagnosis of UHR of psychosis was assessed using the semi-structured CAARMS as the gold standard. The CAARMS is used to determine the intensity/severity and frequency/duration of psychotic like symptoms, with scores ranging from 0 to 6. The original CAARMS consists of seven subscales, but symptomatic criteria for psychosis risk are based exclusively on positive symptom items (disordered thought contents, perceptual abnormalities, and disorganized speech). In this study, four items from the positive symptom subscale were administered to subjects for the diagnosis of UHR of psychosis.

Measures for depression

When investigators conducted face-to-face evaluations, depressive disorder was diagnosed according to the DSM-IV criteria using the Mini-International Neuropsychiatric Interview (MINI) [24]. The nine-item Patient Health Questionnaire (PHQ-9) [25,26] was administered to investigate the level of depressive symptoms.

Statistical analyses

Using data from the 60 subjects who underwent face-to-face evaluations, the internal consistency, reliability, and test-retest reliability of the KPQ-16 were estimated using Cronbach’s alpha and the intraclass correlation coefficient (ICC). Scores >0.65 were considered to indicate sufficient internal consistency [20]. Using data from these subjects, the degree of correlation between KPQ-16 and ESI scores was assessed using Spearman’s correlation coefficient to examine convergent validity. We conducted receiver operating characteristic (ROC) analyses using data from the 257 subjects who finished secondary interviews. We determined optimal cutoff values for total and distress KPQ-16 and mKPQ-16 (19 items) scores based on the sensitivity, specificity, positive predictive value (PPV), and NPV. The PPV reflects the probability that an individual has a CAARMS diagnosis in the case of a positive PQ test result. The accuracy of the test depends on how well it separates the group being tested into those with and without the disease in question, and is measured by the area under the curve (AUC). An AUC value of 0.5 indicates the absence of a difference, and a value of 1 indicates perfect separation of the two groups. We compared the prevalence of depressive disorder according to the diagnosis of UHR of psychosis with Fisher’s exact test. For concurrent and discriminant validity, the Mann-Whitney U-test was performed to compare the KPQ-16, mKPQ-16, and PHQ-9 scores according to the diagnosis of UHR of psychosis based on CAARMS and major depressive disorder and depressive disorder [MINI, not otherwise specified (NOS)] criteria. The data were analyzed using SPSS for Windows (ver. 21.0; IBM Corp., Armonk, NY, USA), and p values <0.05 were considered to be statistically significant.

RESULTS

Diagnostic results

The mean age of the subjects who participated in both evaluations (n=257) was 19.2±0.7 (range 18–23) years. The male: female ratio was almost 1:1 [n=129 (50.2%) females]. The mean age of the subjects who underwent face-to-face investigations (n=60) was also 19.2±0.7 years, but males [n=35 (58.3%)] predominated compared with all study subjects. Among the subjects who were interviewed (n=257), 17 (6.6%) met the CAARMS criteria for UHR of psychosis. One subject met subthreshold frequency criteria and 16 subjects met subthreshold intensity criteria. Thought problems were more common than perceptual disturbance in the UHR diagnosis group. All but one subject (with a subthreshold frequency of hallucination) had subthreshold delusional symptoms. Only four subjects had subthreshold intensity hallucinations meeting the CAARMS criteria.

Receiver operating characteristic analysis

To predict CAARMS-based diagnoses of UHR of psychosis, ROC curves were plotted for the KPQ-16 and mKPQ-16 total scores (Figure 1). AUCs were significant for the KPQ-16 (AUC=0.702, p=0.005) and mKPQ-16 (AUC=0.831, p<0.001) total scores, indicating that these tools are useful for the screening identification of those at UHR of psychosis. The AUC of the mKPQ-16 was greater than that of the KPQ-16, with a difference of 0.129, although both were significant. Exploratory analyses of distress scores of the measures yielded AUC values that were satisfactory and similar to those of the total scores (KPQ-16, 0.716; mKPQ-16, 0.796).

Figure 1.

The receiver operating characteristic (ROC) curves of the Korean version of Prodromal Questionnaire-16 item (KPQ-16) and the modified Korean version of Prodromal Questionnaire-16 item (mKPQ-16) with Comprehensive Assessment of At-Risk Mental States (CAARMS) criteria of ultra-high risk of psychosis as a gold standard.

Validity and reliability

Table 1 summarizes the diagnostic validity of the two scales based on the AUCs and the optimal cutoff points for sensitivity and specificity. For the KPQ-16, cutoff total item scores of 6 and 7 resulted in the best balance of sensitivity (64.7% and 52.9%, respectively) and specificity (71.2% and 84.6%, respectively). For the mKPQ-16, a cutoff total score of 6 resulted in the greatest sensitivity (100%), despite a low specificity value (53.3%) and a PPV of 13.2%. A cutoff total mKPQ-16 score of 7 was more balanced, with favorable sensitivity (76.5%) and specificity (75.4%) and a PPV of 18.1%. Cutoff KPQ-16 and mKPQ-16 distress scores of 14 and 16, respectively, resulted in the greatest sensitivity (64.7% and 76.5%, respectively), with specificities of 74.6% and 74.2%, respectively, and PPVs of 15.2% and 17.3%, respectively.

Table 1.

Diagnostic validity of KPQ-16 and mKPQ-16 based on the area under receiver operating characteristics curve (AUROC) and the optimal cutoff points for sensitivity and specificity

Scale	Score	AUROC	95% CI	p-value	Cutoff	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
KPQ-16	Item total	0.702	0.564–0.840	0.005	5	82.4	37.5	8.5	96.8
					6	64.7	71.2	13.8	96.6
					7	52.9	84.6	19.6	96.2
mKPQ-16	Item total	0.831	0.764–0.898	<0.001	6	100	53.3	13.2	100
					7	76.5	75.4	18.1	97.8
					8	58.8	85.8	22.7	96.7
KPQ-16	Distress	0.716	0.567–0.866	0.003	12	82.4	56.7	11.9	97.8
					13	70.6	64.6	12.4	96.9
					14	64.7	74.6	15.2	96.8
					15	58.8	80.4	17.5	96.5
mKPQ-16	Distress	0.796	0.697–0.894	<0.001	14	88.2	60.4	13.6	98.6
					15	76.5	67.5	14.3	97.6
					16	76.5	74.2	17.3	97.8
					17	52.9	81.2	16.7	96.1

KPQ-16: Korean version of Prodromal Questionnaire-16 item, mKPQ-16: modified Korean version of Prodromal Questionnaire-16 item, 95% CI: 95% confidence interval, PPV: positive predictive value, NPV: negative predictive value

Cronbach’s alpha values for KPQ-16 and mKPQ-16 items were 0.680 and 0.685, respectively. Test-retest reliability was evaluated using data from subjects who underwent face-to-face evaluations, and the ICC for the KPQ-16 total item score was 0.478 (p=0.007). For this dataset, Spearman’s coefficient of correlation between the ESI and KPQ-16 item total scores was 0.644 (p<0.001).

Depressive disorder was diagnosed in 15 (25.0%) subjects (2 with major depressive disorder and 13 with depressive disorder, NOS). The prevalence of depressive disorder in subjects with UHR diagnoses was 29.4%, which did not differ significantly from the prevalence in those with no UHR diagnosis (p=0.743). Table 2 compares the KPQ-16, mKPQ-16, and PHQ-9 scores according to diagnoses of UHR of psychosis and depressive disorder. KPQ-16 and mKPQ-16 scores were significantly higher in subjects with UHR diagnoses, but not in those with depressive disorder, whereas the PHQ-9 score was significantly higher in subjects with depressive disorder, but not in those at UHR of psychosis.

Table 2.

Comparisons of scores on the KPQ-16, mKPQ-16, and Patient Health Questionnaire-9 according to diagnosis

	Ultra-high risk of psychosis				Depressive disorder
	Yes (N=17)	No (N=240)	Z	p-value	Yes (N=15)	No (N=45)	Z	p-value
KPQ-16, total	7 (5–7.5)	5 (4–6)	2.912	0.004	6 (4–8)	7 (5–8)	0.486	0.627
mKPQ-16, total	8 (6.5–8)	5 (5–6)	4.664	<0.001	7 (6–8)	8 (6–8)	0.096	0.924
KPQ-16, distress	16 (12–19.5)	11 (9–14)	2.994	0.003	14 (10–18)	16 (12–19)	1.010	0.313
mKPQ-16, distress	19 (15–22)	12 (10–16)	4.098	<0.001	16 (12–23)	17 (15–20.5)	0.660	0.509
Patient Health Questionnaire-9	3 (2–8)	5 (3–7)[*]	0.438	0.662	7 (5–11)	4 (2–6)	3.282	0.001

number of subjects were 43.

KPQ-16: Korean version of Prodromal Questionnaire-16 item, mKPQ-16: modified Korean version of Prodromal Questionnaire-16 item

DISCUSSION

This study validated Korean versions of the PQ-16 as a screening measure of psychosis risk in the college setting. Overall, the Korean versions of the PQ-16 showed good internal consistency and diagnostic validity, successfully distinguishing subjects at UHR of psychosis from the general population. Furthermore, they showed favorable convergent validity with the ESI and discriminant validity against depression. We believe that the validation of a short screening test for psychosis risk will promote screening and the early identification of psychosis risk in Korean schools and communities.

In the context of use as a screening instrument, a cutoff mKPQ-16 score of 7 had a satisfactory balance of sensitivity (76.5%) and specificity (75.4%), comparable to those of the Korean version of the ESI (77% sensitivity, 70% specificity), in differentiating subjects at UHR of psychosis from those with no CAARMS diagnosis. Based on mKPQ-16 scores, 8 of every 10 normal individuals were rejected accurately; using a positive test cutoff score of 7, the probability that an individual had a psychosis risk syndrome was approximately 18%. Investigators who do not want to lose false-negative subjects at UHR of psychosis could apply a cutoff mKPQ-16 score of 6 to improve the sensitivity to 100%, despite the relatively low specificity (53.3%) and PPV (13.2%). A cutoff KPQ-16 score of 6 yielded the same results as did the score in the original study, with generally favorable sensitivity (64.7%) and specificity (71.2%), although these were lower than in previous studies18 and with the mKPQ-16. The characteristics of the PQ-16 with a high number of perceptual disturbance items might contribute to the missing of subjects with no perceptual aberration. This speculation was supported by the finding that the addition of three ESI items on thought and cognitive problems to the KPQ-16 increased the instrument’s sensitivity. The AUC values and performance of the KPQ-16 and mKPQ-16 distress scores were generally similar to those of the measures’ respective total item scores. Considering simplicity and validity, the mKPQ-16 total score might be the most useful screening tool.

In this study, PPVs were relatively low for a predictable screening tool [27,28]. This result may be explained by the observation that PPVs were relatively high in studies of selected help-seeking samples visiting specialized prodrome clinics and community mental health clinics [27,28], and low in studies of subjects with no help-seeking behavior, similar to ours [19,29,30]. Therefore, investigators who wish to increase the PPV of the KPQ-16 may use a higher cutoff score, which would lower the false-positive rate and sensitivity. Nevertheless, when a tool is administered to screen for a serious, low-prevalence disease such as psychosis, the chosen cutoff point should provide good sensitivity, even when the PPV is low or the false-positive rate is high [19]. Moreover, because the PQ-16 is not used for final diagnosis, but simply for pre-testing of the general population to identify those needing further clinical interviews, it could be an appropriate primary mass-screening measure. Screening of the general population for psychosis risk using a self-report questionnaire could result in a high false-positive rate unless other factors are considered [30-32]. In particular, functional decline and high subjective distress should be considered when a diagnosis of UHR of psychosis is given to an individual who does not seek treatment.

Given the high false-positive rate of this measure, we should carefully consider the potential stigma of psychosis screening in the non-treatment-seeking population [31,33]. False identification of youth as prodromal to psychosis can lead to unnecessary concern and emotional harm [31], particularly considering the relatively low PPV of the tool. However, the psychotic behavior and poor clinical outcome of schizophrenia may be more damaging than evaluation in terms of stigma [34,35]. Furthermore, the amelioration of distressful symptoms and prevention of a major psychiatric illness would have much benefit compared with the risk of stigma associated with the screening process. If prevention is successful with screening, the potential stigma generated by a psychotic illness is also ultimately resolved.

Several limitations should be considered when interpreting our data. First, the participants in the study were from non-help-seeking general college population. In addition, the study subjects were young adults. Previous studies have shown that children and adolescents are more likely to report UHR symptoms [36-39]. Therefore, we should consider use of the KPQ-16 cutoff score identified in this study carefully because a different score may be more applicable in a clinical setting involving a treatment-seeking or younger population. Second, this study was cross sectional, rather than longitudinal. Consequently, further research investigating the predictive validity of the KPQ-16 for the longitudinal transition to psychosis is warranted. However, recent studies demonstrated that the longitudinal course of UHR of psychosis is pluripotential and dysfunctional, even when it does not lead to the development of psychosis [13,40]. Therefore, the identification of UHR of psychosis, regardless of psychotic conversion, is important. Finally, selection bias may have affected the results because subjects with KPQ-16 scores <4 were not enrolled in the second investigation and many candidates did not participate in face-to-face clinical interviews.

In conclusion, the results of this study suggest that the KPQ-16 and mKPQ-16 are good screening instruments for use in community and school settings. The small numbers of items make it feasible to screen large numbers of students and young adults. The validation of these measures could contribute to the early identification of psychosis and shorten the duration of untreated psychosis by prompt early intervention in the Korean community.