Validation of the Korean Version of the Prodromal Questionnaire-Brief Version in Non-Help-Seeking Individuals.

OBJECTIVE: In this study, we examined the reliability and validity of the Korean version of the Prodromal Questionnaire-Brief (PQ-B), a self-report screening instrument for clinical high risk for psychosis (CHR).
METHODS: A total of 3,400 middle, high school and college students participated, and 261 subjects with a PQ-B total score ≥3 completed both the PQ-B and the Structured Interview for Prodromal Syndromes (SIPS). Receiver operating characteristic (ROC) curves were used to examine the psychometric properties of the PQ-B.
RESULTS: A cut-off of 7 for the PQ-B total score and a cut-off of 22 for the PQ-B distress score showed the best balance of sensitivity (72.2%, 77.8%) and specificity (71.4%, 75.0%).
CONCLUSION: The Korean version of the PQ-B showed good reliability and validity for predicting prodromal risk symptoms in the community population.

INTRODUCTION

The duration of untreated psychosis (DUP) is defined as the period between the onset of psychosis and the first treatment, and it has been reported that patients with schizophrenia with a longer DUP showed a lower response to antipsychotic treatment and a higher rate of recurrence [1,2]. Thus, early detection of and early intervention for psychosis have become important treatment strategies for improving the outcomes of early psychosis [3,4]. These efforts have been directed toward treating help-seeking individuals before the onset of psychosis and detecting and investigating those at risk of developing psychosis; consequently, individuals referred to as at “ultra high risk” (UHR) or “clinical high risk for psychosis” (CHR) have attracted the attention of many clinicians and researchers [5-7]. However, recent evidence has shown that only 20% of CHR individuals develop psychosis despite being specifically at risk of psychosis and not nonpsychotic disorders [8,9]. This incidence is not low compared to other clinical diseases, but given their various clinical courses, CHR individuals are regarded as a heterogeneous group that shares attenuated psychotic symptoms [10-12]. Furthermore, only a small percentage of people in the community who meet the diagnostic criteria for CHR visit high-risk clinics [13,14]. This heterogeneity problem requires examining the characteristics of subgroups for a larger number of subjects. Therefore, extensive efforts are needed to find out more subjects in the general population than in those who ask for help at the hospital.

An epidemiological study reported that approximately 8% of the general population has psychotic-like experiences, but most do not visit a psychiatric clinic early [15]. Unlike high-risk groups for other major mental illnesses, the psychosis high-risk group is highly likely to develop psychosis specifically; therefore, it is necessary to classify these individuals correctly and provide appropriate education and intervention [16-18]. Various diagnostic tools for diagnosing CHR, such as the Structured Interview for Prodromal Syndromes (SIPS) and the Comprehensive Assessment of At-Risk Mental States (CAARMS), have been developed [19]. However, these diagnostic tools are limited in their use as evaluation tools for a large proportion of the general population because they require direct assessment by trained professionals over a long period.

Thus, various screening tools have been implemented to evaluate CHR in community or clinical settings [20]. Among them, the Prodromal Questionnaire (PQ) is a screening instrument comprising 92 items [21]. It has 90% sensitivity and 49% specificity and is useful for screening specific samples prior to fullfledged diagnosis. Furthermore, the shortened 21-item version [PQ-Brief (PQ-B)] and the 16-item version (PQ-16) were developed to increase the efficiency of evaluation by reducing the time required to complete the large number of items on the PQ [22-25]. Although they show equivalent sensitivity compared to the PQ, these shortened instruments are known to be more effective screening tools with higher specificity. Among them, the PQ-B has been adapted into English, Chinese and Italian versions, but a Korean version has not yet been developed [23,26]. Therefore, this study describes the development of the Korean version of the PQ-B for the general population aged 10 to 29 years and examines its reliability and validity.

METHODS

Subjects and procedures

To investigate the efficacy of the PQ-B as an early screening tool for the high risk of psychosis, this study performed a semistructured test divided into two stages.

First, to check the mental health status of healthy people in the community and to examine the internal consistency and concurrent validity of the PQ-B, we designed an Internet-based questionnaire of PQ-B, and asked the subjects to mark whether they were willing to participate in follow-up testing in the future. For the primary screening, we conducted a survey of unspecified people aged 12–38 years from September 2017 to February 2018, and a total of 3,400 subjects responded to the survey. However, when responses with insufficient data were excluded, a total of 2,443 subjects were analyzed. Among the 2,443 subjects, 991 (40.6%) had a total PQ-B score of 3 or more, and 400 of these subjects were willing to undergo follow-up. To determine whether the subjects with a total PQ-B score of 3 or more were in the CHR group, we contacted the 400 people by phone and text; 261 of these people were willing to participate in the second screening. In that screening, the SIPS was administered by trained counselors by telephone or face-to-face interview to determine whether the psychiatric high-risk group fulfilled the positive symptom criteria; the PQ-B was also re-administered at that time. A total of 74 people (PQ-B total score=6.1±3.2) were suspected of meeting the criteria for positive symptoms. For a more precise evaluation, a face-toface interview with a psychiatrist was carried out to determine the presence or absence of CHR.

Written informed consent was collected from each participant, and the study was approved by the SNUH Institutional Review Board (IRB 1110-009-380).

Measures

Prodromal Questionnaire-Brief Version

Loewy et al. [22] modified the original 92-item PQ to create 21-item and 16-item versions with improved accuracy and efficiency, and these shorter versions show higher specificity and equivalent sensitivity than the earlier version. This study used the PQ-B, an effective and efficient self-report rating scale comprising 21 items for screening individuals to identify those at high risk of psychosis [23]. The respondents were asked to answer “yes” if they had experienced the positive symptoms described in the PQ-B’s 21 questions and “no” if they did not. The respondents were required to indicate the degree of distress using a five-point Likert scale only for those questions to which they answered “yes.” The PQ-B was translated and back-translated by a clinical psychologist after obtaining permission from the original author and was proofread by a bilingual person proficient at Korean and English before the adaptation.

Structured Interview for Prodromal Syndromes

The SIPS is a semistructured interview tool comprising five questions; it is effective for distinguishing the prodromal phase of psychosis and is designed to be evaluated and managed by a clinician or specialist [27]. The evaluation contents of the Scale of Prodromal Symptoms (SOPS) include Unusual thoughts/Delusional ideas (P1); Suspiciousness/Persecutory ideas (P2); Grandiose ideas (P3); Perceptual abnormalities/Hallucinations (P4), and Disorganized communication (P5). The prodromal symptoms of psychosis are mainly identified according to these positive symptoms and can be categorized into three subtypes: brief intermittent psychotic syndrome (BIPS), attenuated positive symptom syndrome (APSS) and genetic risk and deterioration syndrome (GRDS).

Statistical analyses

All the data were analyzed using SPSS (Statistical Package for the Social Sciences) 20.0 (IBM Corp., Armonk, NY, USA). Cronbach’s alpha coefficient was used to verify the internal consistency and reliability of the PQ-B. The Pearson correlation coefficient was used to test the concurrent validity of the PQ-B. To estimate the optimal cut-off point of the PQ-B, data from the 261 subjects who completed the secondary screening were used to analyze the receiver operating characteristic (ROC) curve, and the optimal cut-off score was determined based on sensitivity, specificity, positive predictive value, and negative predictive value. We also calculated the area under the curve (AUC) to ensure that the psychosis high-risk group was correctly identified.

RESULTS

Demographic characteristics

The sample included a total of 2,443 people, among whom 261 subjects participated in the secondary screening for high risk of psychosis. In the primary screening, 823 participants (33.7%) were male and 1,620 (66.3%) were female. The mean age was 17.66±3.173 years old; the largest proportion of subjects was 16–20 years old (53.2%), followed by 11–15 years old (24.9%). Regarding education level, 1,053 participants were college students, comprising the majority (43.1%). Most of the participants (1,306; 53.4%) responded ‘None’ to religion, and most were unmarried (97.8%). All the participants answered ‘No’ to comorbidities and mental illness. For the second screening, 64 participants (24.5%) were male, and 197 (75.5%) were female (Table 1).

Table 1.

Demographic characteristics of subjects

Variables	Screening (#1)	Screening (#2)
	N (%)	N (%)
Gender
Male	823 (33.7)	64 (24.5)
Female	1,620 (66.3)	197 (75.5)
Age (mean±SD)
11–15 yrs	608 (24.9)	0 (0.0)
16–20 yrs	1,300 (53.2)	110 (42.1)
21–25 yrs	464 (19.0)	126 (48.3)
26–30 yrs	60 (23.5)	14 (5.4)
31–35 yrs	5 (0.2)	5 (1.9)
36+ yrs	6 (0.2)	6 (2.3)
Education
Middle school	400 (16.4)	0 (0.0)
High school	990 (40.5)	11 (4.2)
University	1,053 (43.1)	250 (95.8)
Religion
Christianity	819 (33.5)
Catholic	166 (6.8)
Buddhism	108 (4.4)
Other	44 (1.8)
None	1,306 (53.4)
Marital status
Unmarried	2,389 (97.8)
Married	25 (1.0)
Medical problem
Yes	0 (0.0)
No	2,443 (100.0)
Psychological problem
Yes	0 (0.0)
No	2,443 (100.0)
Total	2,443 (100.0)	261 (100.0)

Psychometric properties of the PQ-B

Reliability, validity, and correlation with the SIPS

To verify the internal consistency and reliability of the PQ-B, Cronbach’s alpha coefficient was confirmed; in this study, it was 0.830. The correlation coefficient between each item and the total PQ-B was 0.23–0.55. To observe the correlation between the diagnosis classified by the SIPS and the PQ-B total score and distress score, the categorical variable of dichotomous diagnosis was transformed into dummy variables (psychosis high-risk group=1, normal group=0), and Pearson correlation analysis was performed. The PQ-B total score (r=0.503, p<0.0001) and distress score (r=0.527, p<0.0001) were significantly correlated with the differentiated SIPS diagnosis.

Sensitivity, specificity, and the cut-off point

The ROC analysis was performed to calculate the cut-off point for the PQ-B (Figure 1), and the AUC was calculated to calculate the PQ-B total score and distress scores. The PQ-B total score was 75.2% AUC [95% confidence interval (CI): 0.613–0.892], the distress score was 79.9% AUC (95% CI: 0.677–0.920), and the PQ-B showed a significant value. The sensitivity and specificity of the cut-off score for the PQ-B total score and the distress score are presented in Table 2. When a sensitivity of 70% or more was expected at a moderate specificity, the cut-off scores of the PQ-B total score and the distress score were estimated to be 7 and 22, respectively, and the positive predictive values (PPVs) at this time were 11.7 and 14.1, respectively. In this case, the sensitivity and specificity of the distress score were slightly higher than those of the total score.

Figure 1.

ROC curve of the PQ-B scores for predicting the SIPS diagnosis of prodromal/psychotic syndrome versus no SIPS diagnosis.

Table 2.

Sensitivity, specificity and AUC at various cut-off values of PQ-B scores

PQ-B cut-off	Sensitivity (%)	Specificity (%)	PPV	NPV	LR+	AUC	95% CI	p
Total score
>5	72.2	51.8	7.3	97.3	1.50	0.752	0.613–0.892	0.001
>6	72.2	62.5	9.2	97.7	1.91	0.752	0.613–0.892	0.001
>7	72.2	71.4	11.7	98.0	2.53	0.752	0.613–0.892	0.001
>8	66.7	80.4	15.2	97.9	3.40	0.752	0.613–0.892	0.001
>9	50.0	85.7	15.5	97.0	3.50	0.752	0.613–0.892	0.001
Distress score
>20	77.8	67.9	11.3	98.3	2.42	0.799	0.677–0.920	0.000
>21	77.8	69.6	11.9	98.3	2.56	0.799	0.677–0.920	0.000
>22	77.8	75.0	14.1	98.5	3.11	0.799	0.677–0.920	0.000
>23	72.2	76.8	14.1	98.1	3.11	0.799	0.677–0.920	0.000
>24	66.7	78.6	14.1	97.8	3.11	0.799	0.677–0.920	0.000

Cut-off point for 80% or more sensitivity under the condition of moderate specificity. LR+: positive likelihood ratio, AUC: area under the curve

DISCUSSION

In this study, the authors evaluated the reliability and validity of the Korean version of the PQ-B, a self-report measure for screening for CHR in the community. Compared with the SIPS, the PQ-B showed generally good diagnostic validity and was effective for screening for CHR in the general population.

The total score and distress score of the PQ-B Korean version showed a significant correlation and high internal consistency with the SIPS; p<0.01, and Cronbach’s alpha was 0.830. In addition, the AUC for the total score and distress score for distinguishing subjects with early psychosis from the general population were 75.2% and 79.9%, respectively, and these scores were highly correlated with the SIPS. However, the cut-off scores of the Korean PQ-B total score and distress score were 7 and 22, respectively, which were much higher than the scores (3 and 6, respectively) recommended by previous study [22]. Additionally, the total score was the same as the scores for the Chinese version of the PQ-B (7 and 24, respectively), but the distress score was two points lower. In addition, the total score and distress score were the same as those of the Italian version of the PQ-B. The subject recruitment information indicated that Loewy’s study included a large number of subjects referred from the community and primary health care institutions for the evaluation of high-risk groups. Xu’s study, on the other hand, was carried out among general outpatients visiting clinics for various psychological and psychiatric problems, and Preti’s study included general college students [23,26]. It is known that in the diagnosis of groups at high risk for psychosis, enriched samples show a higher prevalence of psychosis high-risk groups compared with non-help-seeking general populations. Therefore, the difference between the total score of 3 in Loewy’s study and this study probably reflects the characteristics of the sample. In contrast, the total score in this study was the same as that of a previous study with a nonenriched sample. In addition, the sensitivity and specificity tended to be higher than 80% and as low as 60%, respectively, because previous studies targeted patients. In this study, the specificity increased to more than 70%, and the sensitivity remained at 70% because the study was aimed at healthy adolescents and college students in the general population. Thus, the use of the PQ-B in the first stage for the early detection of high-risk patients with psychosis in the general population in Korea is as valid as the results of other studies, but the cut-off score (7, 22 points) suggested in this study seems reasonable to prevent too many false positives [28]. However, the PPV of the PQ-B in our samples was 11.7%, which was much lower than the 93% proposed by Loewy et al. [22]; furthermore, the LR+ was 2.53, similar to the 2.12 and 2.09 proposed by Loewy et al. [22] The low PPV may suggest that the PQ-B’s function as a screening tool is weak; however, the tools used to screen diseases that are difficult to detect and have low prevalence, such as psychosis, may be useful if they have good sensitivity, even if the PPV is low. Moreover, the PQ-B does not necessarily have to have a high PPV because it can be used by nonspecialists for the early detection of a high risk of psychosis in the undiagnosed general population [29-31]. However, considering the low PPV, a high score on the test is not the same as a final diagnosis of high-risk, and thorough interviews with a medical specialist should be recommended [32-34].

Turning our attention to another version of PQ, the PQ-16 gives a more exciting result. PQ-16 is a self-reporting tool with fewer items than PQ-B, which is advantageous for large-scale samples [14]. In a recently published Korean version of PQ-16, Kim et al. [30] reported 6 and 7 cutoff total scores for KPQ and mKPQ-16, respectively. Taking into account that the AUC results are higher for the mKPQ-16 compared with KPQ-16, a score of 7 on the 16-item scale in mKPQ-16 is equivalent to our results suggesting that the mKPQ-16 may be a more efficient instrument. Further studies will be needed to consider the incidence rate.

The PQ-B is a screening tool for the early detection of those who are experiencing abnormal sensations or unusual accidents and for determining whether they should be referred to specialized medical institutions for early treatment. In this sense, this study is of significance because the survey targeted the population of ordinary adolescents and college students.

Several limitations should be considered when interpreting our results. First, during recruitment, participants who with a PQ-B total score of 3 points or higher were selected; the second screening followed, and then the clinical interview was conducted. However, there may have been a selection bias due to the presence of CHR subjects who did not meet the screening requirements. Therefore, this should be taken into account when applying to the general population. Second, the SIPS is designed for the evaluation of people who need help and requires a long and detailed assessment in face-toface interviews. However, in the present study, only a subset of SIPS was arbitrarily used in a wide range of subjects to enrich the samples in the screening process. The application of these measures can lead to problems of false positives or false negatives. Therefore, it is necessary to pay attention to interpretation in generalizing this result. Third, we targeted only individuals who were participating in normal school life in the community. Our results do not represent those who were not attending school for other reasons. Third, there were fewer middle school students (16.4%) than high school students (40.5%) or college students (43.1%) among the subjects. This was because the content of the questions was relatively difficult for middle school students, many of whom could not complete it. Therefore, random sampling or hierarchical sampling will be required, even at the secondary screening. In future research, it will be necessary to pay attention to prevent subject sampling bias when the research is designed.

This study showed that the Korean version of PQ-B was a useful self-report screening instrument for use as the first step in a screening process. In the future, the Korean version of PQ-B may be useful for CHR screening of the adolescents and young adults who are most likely to develop psychosis.