Psychosis Relapse Prediction Leveraging Electronic Health Records Data and Natural Language Processing Enrichment Methods.

Background: Identifying patients at a high risk of psychosis relapse is crucial for early interventions. A relevant psychiatric clinical context is often recorded in clinical notes; however, the utilization of unstructured data remains limited. This study aimed to develop psychosis-relapse prediction models using various types of clinical notes and structured data.
Methods: Clinical data were extracted from the electronic health records of the Ajou University Medical Center in South Korea. The study population included patients with psychotic disorders, and outcome was psychosis relapse within 1 year. Using only structured data, we developed an initial prediction model, then three natural language processing (NLP)-enriched models using three types of clinical notes (psychological tests, admission notes, and initial nursing assessment) and one complete model. Latent Dirichlet Allocation was used to cluster the clinical context into similar topics. All models applied the least absolute shrinkage and selection operator logistic regression algorithm. We also performed an external validation using another hospital database.
Results: A total of 330 patients were included, and 62 (18.8%) experienced psychosis relapse. Six predictors were used in the initial model and 10 additional topics from Latent Dirichlet Allocation processing were added in the enriched models. The model derived from all notes showed the highest value of the area under the receiver operating characteristic (AUROC = 0.946) in the internal validation, followed by models based on the psychological test notes, admission notes, initial nursing assessments, and structured data only (0.902, 0.855, 0.798, and 0.784, respectively). The external validation was performed using only the initial nursing assessment note, and the AUROC was 0.616. Conclusions: We developed prediction models for psychosis relapse using the NLP-enrichment method. Models using clinical notes were more effective than models using only structured data, suggesting the importance of unstructured data in psychosis prediction.

Introduction

The lifetime prevalence of psychotic-spectrum disorders, such as schizophrenia spectrum disorder and mood disorders with psychotic features, is ~3%; moreover, these disorders are accompanied by high levels of morbidity and mortality (1, 2). Psychotic experiences are associated with an increased risk of adverse health outcomes (3). Individuals with psychotic disorders have high relapse rates (4) and ~58% of affected individuals will experience a further episode within 5 years of remission from the initial episode (5).

In practice, early detection and intervention in psychosis have long been considered crucial because they could reduce the severity of relapse or prevent its occurrence (6, 7). Psychosis relapse is associated with poorer occupational and social functioning and more severe symptoms (8, 9). Several studies identified some predictors associated with relapse of psychosis such as poor adherence to treatment (including medication adherence), poor social support, and comorbidities with active psychiatric disorders (10–12). Also, Alvarez-Jimenez et al. (13) mentioned that structured clinical variables and general demographics might have a lower impact on the relapse rates than adherence or social functioning in their meta-analysis. However, it is difficult to utilize these predictors because they are not usually collected directly, and the recorded data format is heterogeneous (14).

A transdiagnostic approach using clinical predictors had been previously attempted to overcome this issue and exhibited a high accuracy (15). Various studies examined the application of natural language processing (NLP) to mental disorders even though privacy concerns limited the accessibility to data sources. Adequately expressing mental illness with only structured data is difficult, and the information on important psychiatric clinical context is usually recorded as free text in clinical notes (16); utilizing such unstructured data might be crucial in psychiatry. Therefore, various NLP techniques could be useful in detecting or identifying patients at risk of various psychiatric disorders. Researchers could use sentimental analysis to predict depression (17), document classification methods to predict suicide attempts (18), semantic relationships to predict anxiety (19), and NLP-derived predictors to predict psychosis (20). However, studies predicting psychosis relapse using the NLP technique are rare, despite these potential advantages, and have not been validated.

Various types of notes are used for prediction using NLP (21). Among them, the clinical rating scale is useful to classify psychiatric cases (22, 23). The admission notes are frequently used as a reference document for the patient's history in clinical settings (24), and nursing notes also record important signals about the patient's condition and clinical outcome. These records help predict outcomes and identify risk factors (25). Although predictive models using each note type or multiple notes simultaneously are being developed, the most valuable notes for NLP have not yet been established.

In this study, we aimed to develop prediction models for psychosis relapse with the NLP technique using various types of clinical notes in addition to structured data. We also compared the performance of models using both structured data and clinical notes and a model based on structured data only and externally validated the performance of both.

Methods

Data Source

The clinical data for the model derivation were extracted from the electronic health records (EHRs) of the Department of Psychiatry and Mental Health Center at the Ajou University Medical Center (AUMC) in South Korea between 2012 and 2020 (26). All patients in the database had at least one psychiatric diagnosis (ICD-10-CM codes F00-F99). Only patients diagnosed by psychiatrists and psychologists using assessment scales were selected from the database to clearly classify patients with mental illness, and a total of 1,986 patients were collected. A model was developed by extracting the data of patients who met the criteria from the records of 1,986 patients. The clinical data included socio-demographics, diagnoses, medications, procedures, laboratory tests, and clinical notes. In particular, the admission notes, initial nursing assessment notes, and psychological test notes were extracted. The documents were limited to the nearest record within 1 month before the index date. And the source texts of the document were 33% in English and 67% in Korean, similar by patient, document, and database. We used the AUMC database formatted according to the Observational Medical Outcomes Partnership–Common Data Model (OMOP-CDM) version 5.3.1, maintained by the Observational Health Data Sciences and Informatics (OHDSI) and de-identified (27).

Furthermore, another EHR database was used for the external validation of the predictive models developed. The Myongji Hospital (MJH) database in South Korea has data from 882,646 patients who visited hospital from 2003 to 2021. The MJH database was also included in the OMOP-CDM version 5.3.1. In contrast to the AUMC database, there was no separate classification process for psychiatric cases in the Myongji Hospital. Data of the same types as those in the AUMC database were extracted.

This study was approved by the Institutional Review Board of the Ajou University Hospital (AJIRB-MED-MDB-21-151), and informed consent was not required due to the use of de-identified data. Access to the MJH database during the external validation process was allowed under the IRB mutual recognition agreement (Research Free Zone agreement).

Clinical Notes

Psychological Tests

These notes contained clinical and cognitive function assessments conducted by psychiatrists and clinical psychologists. It also included the developmental history, brief past history, and diagnosis. The assessments included the Korean version of the Wechsler Adult Intelligence Scale-III (WAIS) for intelligence quotient (IQ), the Trail Making Test for processing speed, the Wisconsin Card Sorting Test (WCST) for perseverative errors, the Korean version of the verbal fluency task for semantic fluency, the Korean version of the California Verbal Learning Test (K-CVLT) for verbal memory, the Beck Depression Inventory (BDI) for depression, the State-Trait Anxiety Inventory (STAI) for anxiety, and the Minnesota Multiphasic Personality Inventory-2 (MMPI-2). The psychological test notes did not include the scores of each test, only the parts described in the psychologist's interpretation and the score evaluations, to increase comparability by note. For example, when the patient's IQ was ≤ 70, the only description included was “the patient's overall intelligence was below average.”

Admission Notes

These notes included various types of information, personal (age, sex, family composition, location, level of education, and personality), medical history (past medical history, family history, social history, past psychiatric history, and history of present illness and symptoms), and the psychiatric assessment (diagnosis, medication, and plan). These notes were recorded by a psychiatrist.

Initial Nursing Assessment

These notes mainly recorded social information, current status, and the transfer pathway. Social information comprised religion, alcohol intake, smoking, and interpersonal relationships; current status included prominent symptoms and symptoms that required attention, such as aggression or violence. The transfer pathway referred to information about the patient's path, such as whether they were hospitalized from an outpatient clinic, another hospital, or another ward. These notes were recorded by a psychiatric ward nurse.

Study Population and Outcome

The study population included patients with psychotic disorders, including schizophrenia spectrum disorders, mood disorders with psychotic symptoms, and other psychotic disorders, and health records covering more than 365 days. The index date was defined as the patient's first record of a diagnosis of psychotic disorder. Exclusion criteria were the absence of antipsychotic prescriptions and psychiatric procedures after the index date.

The definition of study outcome was relapse of psychosis within 1 year after the index date. Relapse was defined as emergency department visits or hospitalization due to exacerbation of a patient with psychosis. In addition, re-hospitalization after discharge for the first diagnosis was included (28). We binarized the outcomes into relapse and non-relapse based on the occurrences recorded only in the main database. A detailed code list of the clinical diagnoses, medications, procedures, laboratory tests, and visit concepts in this study is provided in the online Supplementary Material.

Model Development

We developed a model based on structured data alone (initial model) and other models with both structured data and unstructured free text (NLP-enriched models) (Figure 1). These models were separately developed according to the clinical notes used, and their performances compared. We used the patient-level prediction (PLP) framework of the OHDSI to develop and validate these predictive models. This framework consisted of standardized model development and validation processes that require defining predictable problems and selecting the study population, outcome, population settings, predictors, and statistical algorithms (29).

Figure 1

Overview of model development process. Initial model was developed using selected features only from structured data. NLP-enriched models were developed using selected features plus features from unstructured clinical notes.

Initial Model (Model 1)

The predictive variables for model training were extracted and dichotomized for existence within short-term (-30 days) and long-term (-365 days) intervals prior to the index date to capture the medical history temporality of the psychiatric cases. The variables included patient demographics (sex and age in 5-year groups), condition group (medical diagnosis, grouped using a SNOMED–CT hierarchy), drug group (based on the active ingredients), procedure (e.g., psychotherapy, electroconvulsive therapy.), measurement (e.g., assessment scale, laboratory test.) and observation (e.g., smoking status, alcohol intake). Predictors not recorded in our EMR system were considered non-occurring. Through this process, 6,069 candidate variables were generated. For feature selection, we conducted the least absolute shrinkage and selection operator method (LASSO) and selected the final predictors with variable importance and clinical relevance. The initial prediction model was developed using logistic regression with selected variables.

NLP-Enriched Models (Models 2, 3, 4, and 5)

We developed the NLP-enriched models using the final variables from the initial model and additional NLP-derived variables. Three models were developed, one for each type of clinical notes (admission note, initial nursing assessment note, and psychological test note), and one model using all types of clinical notes. NLP algorithms were used to extract the topics as predictive variables from each clinical document. In pre-processing, we performed a morphological analysis that automatically indexes morphological forms in the documents rather than the vocabulary itself (30). We filtered nouns in the Korean text and the entire text for English documents. Then, we converted the documents into a bag-of-words model of the corpus after pre-processing, including stemming, normalization, and stop word removal. Latent Dirichlet allocation (LDA), an unsupervised learning method, was used to cluster the topics from each document (31). With an LDA-based topic model, the topic probabilities were calculated for each note. For instance, if 10 topics were created by the LDA from the admission notes, the probability of being assigned to 10 topics for each admission note was generated. We developed models with the addition of covariates that were probability values obtained using the NLP algorithm. Before using the LDA, we also calculated the perplexity scores to determine the optimal number of topics in the LDA (32). Using the perplexity score can estimate relative quality of statistical models (33). The study population was randomly split into the training set (75%) and the test set (25%) to develop the initial and enriched models, and 3-fold cross-validation was conducted within the training data set.

External Validation

We conducted an external validation to confirm the validity of the model performance using a different dataset from the MJH database. All settings and evaluation processes were conducted using the same methods as in the model development. However, distinguishing the MJH database admission notes from the psychiatry department was not possible due to their formatting difference from the clinical notes in the AUMC database; furthermore, the psychological test results were not available. Therefore, only two models were validated–the initial model (model 1) and the enriched model developed using the initial nursing assessment notes (model 4), tagged as being from the Department of Psychiatry.

Statistical Analysis

All the variables were appropriately summarized. The baseline demographic and clinical characteristics (medical history, psychiatric history, and psychiatric medication use) are presented as counts (percentage) for categorical variables, and mean and standard deviation or median and interquartile range for continuous variables. The chi-square test was used to compare categorical variables between populations. In all analyses, p-values < 0.05 were considered statistically significant. For the predictive model evaluation, we calculated four metrics: accuracy, F1 score, area under the precision and recall curve (AUPRC), and area under the receiver operating characteristic curve (AUROC). We used the maximal Youden index to select the optimal cut-off value in each prediction model and calculated the accuracy using its cut-off value (34).

All analyses were performed using R software version 3.6 (R Foundation for Statistical Computing, Vienna, Austria), OHDSI's Health Analytics Data to Evidence Suite (HADES) packages, and open-source statistical R packages. All source codes are available in the GitHub repository (https://github.com/ABMI/PsychosisMultimodalValidation).

Results

Baseline Characteristics

Table 1 shows the baseline characteristics of the study population. A total of 330 patients were selected according to the inclusion and exclusion criteria. When the patients were grouped by age at 10-year intervals, we noted that patients in their 20's had a high incidence of relapse (39.2% and 53.2% of patients were in their 20's in the non-relapse and relapse groups, respectively; p = 0.047). There was no significant difference in sex and medical history between the groups. Moreover, the proportion of schizoaffective disorder in the relapse group was 0%, significantly lower than the 9.7% in the non-relapse group (p < 0.01). The proportion of mood and anxiety disorders was also significantly lower in the relapse group (p < 0.01 and p = 0.04, respectively). In terms of psychiatric medication use, antipsychotics and antidepressants were significantly less used in the relapse group (p < 0.01 and p < 0.01, respectively).

Table 1

Baseline characteristics for study population with or without relapse.

Variable	Non-relapse (n = 268)	Relapse (n = 62)	X(df)2	P-value
Age group, n (%)
<20	63 (23.5)	8 (12.9)	3.35 (1)	0.07
20–29	105 (39.2)	33 (53.2)	4.08 (1)	0.04*
30–39	60 (22.4)	11 (17.7)	0.64 (1)	0.42
v≥40	40 (14.9)	10 (16.1)	0.06 (1)	0.81
Sex, n (%)
Male	115 (42.9)	27 (43.5)	0.01 (1)	0.93
Medical history, n (%)
Diabetes mellitus	1 (0.3)	0 (0.0)	0.23 (1)	0.63
Heart disease	2 (0.7)	0 (0.0)	0.47 (1)	0.50
Hypertension	11 (4.1)	1 (1.6)	0.89 (1)	0.34
Psychiatric history, n (%)
Acute transient psychotic disorder	45 (16.8)	12 (19.4)	0.23 (1)	0.63
Anxiety disorder	21 (7.8)	1 (1.6)	3.13 (1)	0.08
Delusional disorder	14 (5.2)	2 (3.2)	0.44 (1)	0.51
Insomnia	13 (4.9)	1 (1.6)	1.30 (1)	0.25
Mood disorder	103 (38.4)	3 (4.8)	26.06 (1)	<0.01*
Neurodevelopmental disorder	12 (4.5)	2 (3.2)	0.19 (1)	0.66
Schizoaffective disorder	26 (9.7)	0 (0.0)	6.53 (1)	0.01*
Schizophrenia	129 (48.1)	38 (61.3)	3.49 (1)	0.06
Psychiatry medication use, n (%)
Anticholinergics	19 (7.1)	1 (1.6)	2.65 (1)	0.10
Antidepressants	214 (80.0)	22 (35.5)	48.65 (1)	<0.01*
Antiepileptics	12 (4.5)	2 (3.2)	0.19 (1)	0.66
Antipsychotics	219 (81.7)	17 (27.4)	62.98 (1)	<0.01*
Benzodiazepine	156 (58.2)	17 (27.4)	19.14 (1)	<0.01*
Beta blocking agents	19 (7.1)	2 (3.2)	1.26 (1)	0.26
Opioids	11 (4.1)	0 (0.0)	2.63 (1)	0.10

;

Indicates statistical significance (P-value <0.05).

;

Model Specification

As part of the variable selection, six predictors were selected through L1 regularization among a total of 6,069 candidate predictors and were used in the initial and NLP-enriched models. The characteristics selected were male sex, non-drinker, non-smoker, and exposure to antipsychotic drugs, individual psychotherapy, and diagnosis of depression within a year before the index date.

We selected 10 topics as the most reliable hyperparameters for LDA performance (Online Supplementary Figure 1) for each NLP-enriched model based on the perplexity scores. Each topic had the probability of being assigned to the topic as the variable value. These topics were added to the six predictors selected in the initial model. Each model was implemented by selecting only some of the topics in the final prediction process. Table 2 lists the topics finally used for each model. Five topics were selected from the psychological tests; three related to intellectual function (i.e., borderline disability, developmental, and normal), and two to mood symptoms (anxiety and bipolar I disorder). Only one topic was selected and included in the model from the admission notes, comprising delusion, persecutory disorder, and irritability. Three topics were selected from the initial nursing assessment notes; one related to alcohol intake, aggressive behavior, and psychosis; the second included depression and bipolar I disorder, and the last was related to marginal, withdrawal, and self-talk.

Table 2

Domains selected by LASSO model in admission note, a note of psychological tests, and a note of initial nursing assessment.

Note type	Main features of the domain	Topic examples of the domain
Psychological test	Intellectual function, borderline disability, moderate disability, poor	Psychotic, intellectual, disability, (intellectual function), intellectual functioning, moderate disability, borderline disability, borderline intellectual disability, (low level), poor
	Developmental, early onset, functioning	(overall), (high school), (developmental history), functioning, (level), (in high school), (friends)
	Normal level, possibility, potential intelligence	(normal level), (normal level of intellectual function), (academic background), (potential intelligence), possibility
	Mood, schizophreniform, anxiety, agitation	stress, anxiety, schizophreniform, auditory hallucination, mood, delusional, psychomotor agitation, panic attack, non-functioning, anger, personality, disorder
	Bipolar I disorder, manic episode, psychotic features	manic, manic episode, bipolar I disorder, current episode, mood-congruent psychotic features, depressive, severe, self-talking, (emergency room), acute episode, manic severe, aggressive behavior, sleep disturbance
Admission notes	Delusion, persecutory, disorder, irritable, parent history	(mother), (father), disorder, persecutory, personal, delusional, medication, (emergency room), acute, persecutory delusion, delusional disorder, irritable, auditory, talkative, panic, seroquel, (complication), (length of stay), problem, onset, illness
Initial nursing assessment	Admission, alcohol intake, first, aggressive, psychotic	(story), (alcohol intake), marginal, (first), (admission), (involuntary admission), (aggressive), (auditory), persecutory delusion, (inappropriate), (psychotic), anxiety, 90(voluntary admission)
	Depression, bipolar I disorder, persistent	(inpatient treatment), stress, (persistent), (self-talking), (emergency room), (depression), 81(persecutory delusion), Dystonia, marginal, major depressive disorder, bipolar I disorder
	Adjustment, marginal, withdrawal, self-talking, resistant	(outpatient), delusion, disturbance, adjustment disorder, Idea of reference, withdrawal, adjustment, marginal, withdrawal, ativan, diazepam, self-talking, stress, (resistant)

Model Performance

Among the 330 patients with psychotic disorders in the AUMC database, 62 (18.8%) experienced a psychosis relapse. The mean interval to relapse was 35 days. The initial model had an accuracy of 0.775, an F1 score of 0.595, an AUPRC of 0.362, and an AUROC of 0.784 in the internal cross-validation dataset (Table 3). Figure 2 shows the ROC curves of the initial model and the NLP-enriched models, obtained using logistic regression. In terms of accuracy, F1 score, and AUROC, all the enriched models with unstructured covariates had a higher performance than the initial model in the internal validation dataset (all note types: 0.900, 0.705, 0.946; psychological tests: 0.842, 0.675, 0.902; admission notes: 0.835, 0.686, 0.855; and initial nursing assessment: 0.835, 0.697, 0.798, respectively). Among the enriched models, the one using all note types had the highest performance. Among the enriched models for each type of clinical notes, the one based on the psychological tests had the highest accuracy, AUPRC, and AUROC (0.842, 0.625, and 0.902, respectively, internal validation). The calculated importance of the variables in each model is shown in Supplementary Table 1.

Table 3

Performance results of the initial model and NLP-enriched models using the clinical note.

Performance metrics	Initial model (Model 1)	Psychological test (Model 2)	Admission notes (Model 3)	Initial nursing assessment (Model 4)	All note types (Model 5)
ACC	0.775	0.842	0.835	0.835	0.900
AUPRC	0.362	0.625	0.407	0.340	0.883
AUROC	0.784	0.902	0.855	0.798	0.946
F1 score	0.595	0.675	0.686	0.697	0.705

ACC, accuracy; AUPRC, area under the precision recall curve; AUROC, area under the receiver operating characteristics curve.

Figure 2

Receiver Operating Characteristic (ROC) curve of models predicting relapse in psychosis. The ROC curve for initial model and NLP-enriched models is shown. Performance of models using area under the receiver operating characteristic curve (AUC) is compared.

External Model Validation

The external validation of the initial model and enriched model 4 was conducted using the MJH database. Among the 4,391 patients identified in the external validation dataset, 202 (4.6%) experienced psychosis relapses. The mean interval to relapse was 80 days. The external validation performance of the initial model had an accuracy of 0.114, an F1 score of 0.089, an AUPRC of 0.042, and an AUROC of 0.468. In contrast, the NLP-enriched model using the initial nursing assessment had an accuracy of 0.832, an F1 score of 0.209, an AUPRC of 0.097, and an AUROC of 0.616 (Online Supplementary Table 2).

Discussion

In this study, we developed prediction models for psychosis relapse using NLP enrichment methods; these models demonstrated higher performance than the initial model using only structured data, as the traditional approach. Furthermore, we compared the models according to the type of clinical notes used and found that the model based on psychological tests provided the highest performance compared to the other enriched models for each type of clinical notes. External validation was performed using a different database converted into the same data type, and showed that the enriched model was more effective than the structured-data model. It has been recently reported that models with NLP features have higher predictive performance (20), and this result was confirmed in our study. However, how the performance differs depending on the type of clinical notes used remains unclear, and the present study investigated which note types are more helpful to predictive performance.

We extracted the characteristic information of each note type using the LDA method. LDA can reportedly reflect and represent the semantic characteristics of the document through topic clustering of similar words (35). Moreover, LDA can treat all notes as if written in on common language, despite originally being in a mix of Korean and English in our data (36). Furthermore, the LDA method is relatively straightforward to understand since it reflects semantic characteristics compared to other machine learning techniques using black-box algorithms and insufficient transparency (37). Thanks to these advantages, prediction models using the LDA method have been used in several studies (38, 39).

We compared the performance of the prediction models based on the type of notes used. Previous studies on prediction models used admission notes, discharge notes, nursing notes, and notes with psychological scales (38, 40, 41). In addition, prediction models have been created through the indiscriminate use of several notes (42, 43). However, to the best of our knowledge, no study has compared prediction models based on the type of notes used. Although the model that used all note types showed the most reliable performance, limited data is commonly used for clinical prediction (44); therefore, it may be useful to determine which data type is more helpful for this purpose.

Among more than 6,000 candidate predictors from the structured data, we identified six predictors for the model development, consistent with the results of previous studies. In a systematic review of prediction models for psychosis relapse (45), a history of prior health services and symptoms of depression were used in the prediction model. Other studies have found that substance use, including alcohol consumption and smoking, was significantly associated with relapse (13, 46). Moreover, the relapse rate in relation to transfer is also reportedly greater for females than for males (47).

Since admission notes usually contain information on past history, family history, and current status (48), we extracted parent history and current status such as delusion and irritability from these notes. Similarly, from the psychological tests, we extracted the test-related developmental history, intelligence, and symptoms of depression and anxiety, and from the initial nursing assessment, the alcohol intake, aggressiveness, involuntary admission associated with the patient's condition, social information, and transfer pathway (49). After comparing the performance of the enriched models based on each type of clinical note, the model using the psychological tests emerged as the most promising. This result is consistent with previous studies reported high-performance predictive models using clinical assessment or developmental history, such as psychological tests (50, 51). Specifically, a lower IQ reportedly predicts worse outcomes in psychosis (52). In the model using the psychological tests, topics related to low IQ were used as variables that predict an increased likelihood of relapse. In addition, through symptoms measured by the BDI and STAI, topics related to severe anxiety and depression were also used as variables, including low IQ. Both depression and anxiety are associated with the severity of psychosis (53). Furthermore, early onset is reportedly associated with negative outcomes (54) and was also used as a variable in our models. However, it should be noted that the variable of early onset was used as a variable to predict a reduced chance of relapse, since most of the patients with such records in our study were diagnosed early. Ultimately, early detection improves the outcomes of psychotic disorders (55). Thus, our findings suggest that psychological tests are a useful note type for predicting psychosis relapse.

Interestingly, despite using already validated predictors in the initial model, the performance of the external validation was poor, and the predictability was lost. The external validation database appears to have lower performance because it does not distinguish psychiatric patients in advance. The data of 4,391 patients have been extracted from the Myongji Hospital, a markedly larger sample than from the AUMC database (330 patients); however, the Myongji Hospital had mixed cases, including non-psychiatric. In fact, it is difficult to extract psychiatric symptoms or records from data (56). For this reason, some psychiatric cases are registered separately from the EHR data for psychiatric research (57). In addition to the classification problem for these psychiatric cases, general difficulties in the external validation of prediction models have been previously reported. Other studies have shown that prediction models exhibited poor performance during external validation for some prediction tasks (58, 59). Due to poor reproducibility and generalizability, the implementation of prediction models in clinical settings is limited despite the development of various models. More diverse data sources could be the solution to this problem, and it has recently been reported that adding free text to data sources using NLP enhances the predictive power (20, 37, 60). In this regard, our study developed models using NLP features, confirming an improved performance compared to the initial model based only on structured data. Moreover, the performance of the NLP-based model in the external validation was higher than that of the initial model. Although the improved model performance during external validation was insufficient, we enhanced the result using only the initial nursing assessment, the least effective among the NLP models.

This study has several limitations. First, we could not include external data from different hospitals due to the limited hospital data. More complete claim data must be used to overcome this limitation. However, claim data does not include unstructured data such as admission note; therefore, our research design could not be performed as such. In this regard, as with other studies using EMR, those not recorded were considered to have not occurred. Second, according to the definition of relapse, relapse cases with an outpatient visit or visits to other hospitals were not included in this study. Defining relapse with an outpatient diagnosis and prescription has a practical limitation. Therefore, we tried to limit conditions with hospitalization or emergency room visit. Inpatient records from other hospitals could not be obtained. This can be overcome by linking with other hospital data and with health insurance claim data, but there are practical difficulties. Third, when defining relapse, we did not distinguish causes such as manic episode and drug-induced psychosis. However, individuals with psychotic symptoms in actual clinical practice have comorbid symptoms, such as substance misuse and anxiety (61). Fourth, external validation of the model was performed using only the data from one hospital, and performance comparison by note was not performed. Psychiatric records are often managed separately or are not disclosed (62), making it difficult to find hospital records that could be used for external validation; furthermore, it is more difficult to distinguish various types of records. Future research is therefore needed to investigate and validate our results. Fifth, the Myongji Hospital database used in external validation had different characteristics from Ajou University database. Unlike 18% of patients in the Ajou University database, 4.6% of patients experienced a relapse in the Myongji Hospital database. In the population of MJH database, the proportion of patients in their 40's or older was higher than that of the developing dataset. Also, unlike the developing dataset, there were significant differences between the groups with and without outcomes in medical history (diabetes mellitus and hypertension), psychiatric history (insomnia and neurodevelopmental disorder), and psychiatry medication (anticholinergics). These differences seem to have affected the difference in relapse rates (Online Supplementary Table 3). And there was a difference in the mean duration to relapse. In both the Ajou University Medical Center and the Myongji hospital, most of the patients relapsed within the first 3 months. Although the median was shorter at the AUMC than at the MJH, this can be explained by the fact that 6% of MJH's relapse was emergency room visits and AUMC's 22% were emergency room visits (Online Supplementary Table 5). It has been reported that there were differences according to institutions for psychiatric patients in South Korea (63). In addition, it is rather desirable to use data with different features to evaluate the generalizability of the model (64). Sixth, when defining patients with psychotic disorder, we included both schizophrenia spectrum and affective psychoses. Because of that, we could not use some specific criteria such as PANSS. However, there is diagnostic uncertainty of first episode psychosis, especially in electronic health data (65). In addition, the highest diagnostic conversion rate from unipolar depression to schizophrenia emerged during the first year (66). Actually, other studies have included both schizophrenia spectrum and affective psychoses in psychotic disorders (67, 68). In this regard, we tried to develop the general model for early relapse prediction of first episode psychosis. Seventh, due to the limitation of the observational database, we could not include the length of the disorder and the duration of the untreated disorder as covariates. Eighth, considering the topic correlation, we determined the appropriate topic number with CaoJuan2009 metric in the LDA model. Although the model was developed in consideration of the topic correlation, overlapping topics makes clinical interpretations unclear. Ninth, in Table 1, there was no statistically significant difference between two groups for acute transient psychotic disorder which had a tendency to relapse. It appears to be due to a problem with the limited number of patients, and further studies are needed in the future.

In summary, we utilized three types of clinical notes to predict clinical relapse in patients with psychotic disorders. Clinical relapse could be more effective predicted using NLP-based models than a model based only on structured data. Furthermore, we found that the predictive model based on the psychological tests provided the highest predictive performance. In clinical situations with large data heterogeneity for each patient, our findings suggest that which type of note would be more useful to use.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Materials, further inquiries can be directed to the corresponding author.

Ethics Statement

The studies involving human participants were reviewed and approved by the Institutional Review Board of the Ajou University Hospital (AJIRB-MED-MDB-21-151). Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author Contributions

DL and CK contributed to data analysis, and writing the manuscript. S-MC, YC, JL, and SS gave critical opinions on the study design, and the manuscript. SL and RP contributed to interpreting the results, and supervised the entire process. All authors contributed to the literature review, study design, data interpretation, and approved the final manuscript.

Funding

This research was funded by the Bio Industrial Strategic Technology Development Program (20003883 and 20005021), the Ministry of Trade, Industry and Energy (MOTIE, Korea) and a grant from the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), the Ministry of Health and Welfare, Republic of Korea (Grant Number: HR16C0001), and Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education [NRF-2020R1I1A1A01072208].

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.