An exploratory application of machine learning methods to optimize prediction of responsiveness to digital interventions for eating disorder symptoms.

OBJECTIVE: Digital interventions show promise to address eating disorder (ED) symptoms. However, response rates are variable, and the ability to predict responsiveness to digital interventions has been poor. We tested whether machine learning (ML) techniques can enhance outcome predictions from digital interventions for ED symptoms.
METHOD: Data were aggregated from three RCTs (n = 826) of self-guided digital interventions for EDs. Predictive models were developed for four key outcomes: uptake, adherence, drop-out, and symptom-level change. Seven ML techniques for classification were tested and compared against the generalized linear model (GLM).
RESULTS: The seven ML methods used to predict outcomes from 36 baseline variables were poor for the three engagement outcomes (AUCs = 0.48-0.52), but adequate for symptom-level change (R2  = .15-.40). ML did not offer an added benefit to the GLM. Incorporating intervention usage pattern data improved ML prediction accuracy for drop-out (AUC = 0.75-0.93) and adherence (AUC = 0.92-0.99). Age, motivation, symptom severity, and anxiety emerged as influential outcome predictors.
CONCLUSION: A limited set of routinely measured baseline variables was not sufficient to detect a performance benefit of ML over traditional approaches. The benefits of ML may emerge when numerous usage pattern variables are modeled, although this validation in larger datasets before stronger conclusions can be made.

INTRODUCTION

The enthusiasm for digital interventions to address eating disorders (ED) is mounting. Digital interventions can alleviate existing help‐seeking barriers (Weissman & Rosselli, 2017). However, response rates to digital interventions are variable, with symptom remission occurring in only 20%–50% of cases (Fitzsimmons‐Craft et al., 2020). Using commonly collected baseline and process data to develop accurate predictive models capable of identifying likelihood of success is necessary for improving patient outcomes (Chekroud et al., 2021).

Machine learning (ML) may help with developing accurate prediction models. ML involves data‐driven techniques that enable computer algorithms to identify and iteratively refine the optimal parameters to fit complex variable patterns (Jordan & Mitchell, 2015). ML is suited to situations where there are a large number of predictors to model and the best combination of predictors is uncertain a priori. ML enables prediction models to be cross‐validated by comparing precision and accuracy across training and test subsets of a single dataset, or with external validation datasets for greater out‐of‐sample generalizability (Shalev‐Shwartz & Ben‐David, 2014).

The value of ML may depend on several factors. When assumptions and sample size requirements are satisfied, the number of predictor variables is small, and nonlinear effects are weak, traditional techniques produce prediction models as accurate as ML (Christodoulou et al., 2019). However, when the number of predictors is large and there are unanticipated nonlinear interactions, the incremental benefits of ML become apparent (Pearson et al., 2019).

ML techniques have been applied in the context of digital interventions. Accurate ML prediction models for program engagement have been observed in online mindfulness programs for wellbeing (Lekkas et al., 2021), internet‐based cognitive therapy for depression (Chien et al., 2020), and web‐based behavioral programs for insomnia (Bremer et al., 2020). In digital interventions for body dysmorphic disorder, ML has resulted in superior prediction models to traditional techniques when predicting symptom‐level change from baseline and process variables (Flygare et al., 2020).

We applied ML to predict outcomes from baseline data collected in RCTs of digital interventions for ED symptoms. Our exploratory aims were to: (1) generate accurate ML‐based predictive models using mostly baseline data; (2) determine whether ML enhances prediction over traditional techniques; and (3) explore influential outcome predictors that will inform future confirmatory work.

METHOD

Study design

We aggregated data from 848 participants enrolled in three RCTs of self‐guided digital interventions for ED symptoms. Details of these RCTs have been published elsewhere (Linardon, Messer, Shatte, Greenwood, et al., 2021; Linardon, Messer, Shatte, Skvarc, et al., 2021; Linardon, Shatte, Rosato, & Fuller‐Tyszkiewicz, 2020). There were five participant groups with pre‐ and 4‐week post digital intervention exposure data. Ethics approval and informed consent were obtained.

Study population and recruitment

The population and recruitment method were nearly identical across the three trials. Participants were recruited via advertisements distributed throughout the first author's psychoeducational ED platform. This platform consists of an open‐access website and social media accounts that offer educational material related to EDs (for more detail about this platform, see Linardon, Rosato, & Messer, 2020).

There was one difference in study inclusion criteria between trials. Participants needed to report the presence of at least one objective binge eating episode in one trial, whereas this criterion was not employed in the other two. All participants were required to be aged 18 years or over.

The final sample used in this study included 826 individuals who provided data on all engagement outcomes and baseline predictors. Three‐hundred and fifty‐nine participants provided pre‐ and posttest data on symptom‐level change.

Digital interventions

There were two digital intervention programs delivered across the trials, Break Binge Eating (two groups; 1 trial) and Breaking the Diet Cycle (three groups; 2 trials). Both programs were structurally the same; that is, they both contained four learning modules/sessions, were self‐guided and based on CBT principles, offered similar homework exercises and functionality (symptom monitoring, quizzes, progress monitoring, reflection tasks, etc.), and only slightly differed in program length. There were two key differences between the programs. The first was that Break Binge Eating was delivered solely through a smartphone app, while Breaking the Diet Cycle was delivered through both a web platform and an app. The second was that Break Binge Eating targeted multiple maintaining mechanisms (restriction, body image, and mood dysregulation) while Breaking the Diet Cycle targeted dietary restriction only. Given the similarities between the two programs, we deemed it appropriate to amalgamate them in the analyses. Please refer to Table S1 for a description of these programs.

Measures

Outcomes

Four outcomes were selected for analyses: uptake; adherence, drop‐out, and adherence (see Supporting Information Materials S1 for their operationalization).

Baseline predictors

Thirty‐six predictors were used, divided into three clusters: demographic, psychiatric and treatment; and symptom severity. For the list of baseline variables, see the Supporting Information Materials S1.

Usage behavior predictors

In one trial (n = 340), 110 usage pattern variables were available for analysis (Table S2). These were modeled in sensitivity analyses.

Data analysis

Predictive models were generated using eight classification approaches: (1) traditional regression, (2) elastic‐net penalized regression, (3) support vector machine with linear kernel, (4) support vector machine with polynomial kernel, (5) support vector machine with radial basis function kernel, (6) k‐Nearest Neighbor, (7) Classification and Regression Tree (CART), and (8) random forest. Predictive models were implemented using the caret package (Kuhn, 2021). Predictors were all centered and standardized.

An iterative process was implemented to examine predictive performance and reduce data biases, whereby the full dataset was split into 100 different training (67%) and testing (33%) datasets. Each model was generated in the training data and then validated in the testing data. Models were trained using fivefold cross‐validation to select the optimal model. To quantify predictive performance from each validation set, accuracy (total correct predictions divided by all predictions), area under the curve (AUC), F1 score, true negatives, false positives, false negatives, and true positives were computed for binary outcomes (engagement indices), and R 2, Root‐Mean‐Square Error, and Mean Absolute Error were computed from the continuous outcome. Each predictive performance index was averaged across all iterations. Variable importance was extracted by running the most accurate model on the full dataset.

Sensitivity analyses were conducted using 10 iterations of training–testing splits. For binary outcomes (engagement indices), models were repeated using: (1) an artificially oversampled dataset (synthetic minority over‐sampling technique; Chawla et al., 2002) to balance outcomes; and (2) only those who accessed their digital intervention. For symptom‐level change, models were repeated excluding baseline levels of modeled outcome (objective binge eating). Additional sensitivity analyses were conducted using intervention usage pattern data available from one RCT to see whether predictive accuracy improves.

RESULTS

Sample characteristics

Sample characteristics are presented in Table S3. Most participants were well‐educated, White women who were highly symptomatic.

Outcomes

Rate of uptake, adherence, and drop‐out was 83%, 36%, and 57%, respectively. Objective binge eating decreased by an average of six episodes (SD = 14) from pre‐ to postintervention.

Predicting engagement

The mean predictive performance for engagement outcomes across 100 iterations of validation data sets is presented in Table 1. Predictive performance was poor for the three engagement outcomes across the nine models based on AUC values.

TABLE 1

Mean model predictive performance of engagement from baseline variables across 100 iterations of training and testing data splits

Modeling approach	Accuracy	AUC	F1 score	Negatives		Positives
				True negative	False positive	False negatives	True positives
Intervention uptake
Generalized linear model	0.81	0.50	0.90	1.5%	98.5%	1.7%	98.3%
Elastic‐net	0.82	0.50	0.90	0.2%	99.8%	0.3%	99.7%
SVM (Linear)	0.83	0.50	0.90	0.0%	100.0%	0.0%	100.0%
SVM (Polynomial)	0.82	0.50	0.90	0.0%	100.0%	0.0%	100.0%
SVM (Radial)	0.83	0.50	0.90	0.0%	100.0%	0.0%	100.0%
k‐Nearest Neighbors	0.82	0.50	0.90	0.9%	99.1%	1.2%	98.8%
Random Forest	0.82	0.50	0.90	1.3%	98.7%	0.9%	99.1%
Classification/Regression Tree	0.79	0.50	0.88	4.6%	95.4%	4.9%	95.1%
Intervention adherence
Generalized linear model	0.59	0.49	0.17	86.3%	13.7%	88.3%	11.7%
Elastic‐net	0.63	0.50	0.09	97.7%	2.3%	98.3%	1.7%
SVM (Linear)	0.64	0.50	0.02	99.9%	0.1%	100%	0.0%
SVM (Polynomial)	0.64	0.50	0.03	99.8%	0.3%	99.8%	0.2%
SVM (Radial)	0.64	0.50	0.03	99.8%	0.2%	99.8%	0.2%
k‐Nearest Neighbors	0.63	0.50	0.13	96.1%	3.9%	96.1%	3.9%
Random Forest	0.63	0.50	0.10	94.5%	5.5%	93.8%	6.2%
Classification/Regression Tree	0.59	0.50	0.30	80.1%	19.9%	79.8%	20.2%
Study dropout
Generalized linear model	0.55	0.52	0.64	33.8%	66.2%	29.1%	70.9%
Elastic‐net	0.57	0.51	0.71	7.1%	92.9%	4.9%	95.1%
SVM (Linear)	0.57	0.51	0.72	4.9%	95.1%	3.1%	96.9%
SVM (Polynomial)	0.57	0.51	0.71	7.2%	92.8%	5.6%	94.4%
SVM (Radial)	0.56	0.50	0.71	4.7%	95.3%	4.5%	95.5%
k‐Nearest Neighbors	0.56	0.50	0.71	4.7%	95.3%	4.8%	95.2%
Random Forest	0.55	0.52	0.65	29.5%	70.5%	25.6%	74.4%
Classification/Regression Tree	0.55	0.52	0.66	27.2%	72.8%	22.9%	77.1%

Abbreviations: AUC, area under the curve; SVM, support vector machine.

Predictor importance

Predictor importance was extracted from the SVM (linear) ML model. For uptake, these were motivation levels, age, overvaluation, eating concerns, and a current anxiety disorder. For adherence, these were age, depressive symptoms, current anxiety disorder, past anxiety disorder, and motivation levels. For dropout, these were age, education status, prior binge‐eating disorder diagnosis, objective binge eating, and driven exercise (Table S4).

Sensitivity analyses

When we balanced engagement outcomes by deriving an oversampled training data using SMOTE technique (Table S5), predictive performance remained poor. Models on adherence and dropout that excluded “nonuptake” participants (Table S6) also had poor predictive performance.

Predicting symptom change

Predictive performance based on amount of variance explained in symptom‐level change (Table 2) was acceptable across the eight models. The elastic‐net (R 2 = .37), SVM (linear) (R 2 = .40), SVM polynomial (R 2 = .37), and random forest models (R 2 = .36) explained slightly more variance in symptom change than the traditional linear model (R 2 = .35).

TABLE 2

Mean model predictive performance for objective binge eating change from baseline variables across 100 iterations of training and testing data splits

	Binge eating change (including baseline binge eating)			Binge eating change (excluding baseline binge eating)
Modeling approach	R 2	RMSE	MAE	R 2	RMSE	MAE
Generalized linear model	.35	11.64	8.31	.06	14.94	10.40
Elastic‐net	.37	11.37	8.04	.06	14.64	10.10
SVM (Linear)	.40	11.21	7.63	.06	14.06	9.38
SVM (Polynomial)	.37	11.51	7.80	.05	14.16	9.22
SVM (Radial)	.29	12.24	8.22	.06	14.09	9.28
k‐Nearest Neighbors	.15	13.35	8.92	.05	14.14	9.15
Random Forest	.36	11.51	7.88	.06	13.96	9.13
Classification and Regression Tree	.33	11.91	7.86	.09	13.97	9.21

Abbreviations: MAE, mean absolute error; RMSE, root‐mean‐square error; SVM, support vector machine.

The five most important variables predictive of symptom change were baseline objective binge eating, subjective binge eating, eating concerns, self‐induced vomiting, and overvaluation (Table S7). When removing baseline objective binge eating, predictive performance decreased. The CART model explained the most amount of variance in symptom change (R 2 = .09), slightly outperforming the traditional general linear model (R 2 = .06).

Incorporating usage variables

Analyses were re‐computed on two of participant groups for whom usage data were available. Analyses on this subsample were run with and without the addition of usage variables to enable comparison (Table S8). Incorporating usage variables produced strong predictive performance for adherence (AUC = 0.92–0.99) and dropout (AUC = 0.62–0.93) across all models except SVM (Radial). Predictive performance did not improve for symptom‐level change.

DISCUSSION

ML‐based predictive performance from 36 baseline variables for engagement outcomes was poor and not superior to traditional logistic regression. ML models also explained less than 5% more variance than traditional generalized linear models on symptom‐level change. Although some influential yet unanticipated predictors emerged, these are difficult to interpret given the poor model performance. Thus, a limited set of baseline predictors was not sufficient to detect a performance benefit of ML over traditional approaches, or that possible linear associations between modeled variables prevented interactive terms from enhancing predictive performance. Preliminary findings suggest that ML appears to not confer benefit to prediction modeling when the predictors chosen for analyses are small, unrelated or weakly related to outcomes. This finding is consistent with studies in depression (Lee et al., 2018), schizophrenia (Hettige et al., 2017), and anxiety (Wallert et al., 2018) reporting small to no incremental benefits of ML when restricting analyses to routinely collected baseline data, suggesting that the utility of ML may only emerge when multiple sources of data (sensor, neuroimaging, etc.) are modeled.

Study limitations must be considered. First, model validation was based on a subset of participants rather than an entirely new sample, which may contribute to overfitting concerns. While this has the advantage of holding methodological effects constant, testing on a new sample would enable stronger inferences. Moreover, model overfitting may be exasperated when small sample sizes are used in conjunction with a large number of predictors. Despite this, predictive performance was comparable across all ML models, including the elastic‐net approach, which is useful even when the number of predictors is greater than the number of participants (Friedman et al., 2010). Second, model performance is dependent on the predictors analyzed. Our choice of baseline variables was guided mostly by pragmatic choices (to characterize the sample, those used as key outcome measures, etc.). Fluctuations in these or other variables closer to the point at which a person disengages may yield more precise prediction models, potentially explaining why prediction improved when incorporating usage variables in supplementary analyses. Third, our sample was mostly educated, White women and that the inclusion of predictors related to race, gender, and age may contribute to ongoing concerns about algorithmic bias within ML (Hooker, 2021). Future work utilizing ML in diverse samples is necessary.

This exploratory work offers avenues for future research. As it appears that our chosen predictors and timing of assessment are not decisive outcome determinants, future research should consider modeling the predictive performance of other sources of data that can be easily collected during digital intervention trials. Information on design choices, user experience, perceived usability, and reasons for disengagement may prove useful for generating accurate predictive models. Furthermore, assessing predictors repeatedly throughout the intervention phase may help to determine whether poor prediction is a function of measuring these constructs in a static manner and at an inappropriate time. Heterogeneity of treatment efficacy is well known (Linardon, Shatte, Messer, et al., 2020), such that two individuals with similar baseline scores may have markedly different symptom trajectories across different time intervals. Thus, it should not be surprising to find that earlier measurements of these constructs offer poor predictive value. Overall, findings will ideally encourage researchers to consider assessing these factors in future trials so that treatment outcome predictions can be optimized.

Public significance statement

Being able to accurately predict patient outcomes is needed to accelerate the delivery of personalized eating disorder treatments. We sought to generate accurate predictive models from routinely collected baseline data from digital interventions using data‐driven machine learning techniques. Our findings highlight the pitfalls of relying solely on baseline information for predicting success from digital interventions, instead suggesting that other sources of big data may prove more useful.

AUTHOR CONTRIBUTIONS

Jake Linardon: Conceptualization; funding acquisition; investigation; methodology; writing – original draft; writing – review and editing. Matthew Fuller‐Tyszkiewicz: Conceptualization; data curation; supervision; writing – original draft; writing – review and editing. Adrian Shatte: Conceptualization; data curation; methodology; software; supervision; writing – review and editing. Christopher Greenwood: Conceptualization; data curation; formal analysis; investigation; methodology; writing – original draft; writing – review and editing.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

Appendix S1 Supporting Information.

Click here for additional data file.