Predictors of treatment response in a web-based intervention for cannabis users.

BACKGROUND: Trials demonstrate the effectiveness of web-based interventions for cannabis-related disorders. For further development of these interventions, it is of vital interest to identify user characteristics which predict treatment response.
METHODS: Data from a randomized factorial trial on a web-based intervention for cannabis-users (n = 534) was reanalyzed. As potential predictors for later treatment response, 31 variables from the following categories were tested: socio-demographics, substance use and cognitive processing. The association of predictors and treatment outcome was analyzed using unbiased recursive partitioning and represented as classification tree. Predictive performance of the tree was assessed by comparing its cross-validated results to models derived with all-subsets logistic regression and random forest.
RESULTS: Goal commitment (p < .001), the extent of self-reflection (p < .001), the preferred effect of cannabis (p = .005) and initial cannabis use (p = .015) significantly differentiate between successful and non-successful participants in all three analysis methods. The predictive accuracy of all three models is comparable and modest.
CONCLUSIONS: Participants who commit to quit using cannabis, who at least have moderate levels of self-reflection and who prefer mild intoxicating effects were most likely to respond to treatment. To predict treatment response on an individual level, the classification tree should only be used as one of several sources of information.Trial registration: http://www.isrctn.com/ISRCTN99818059.

Introduction

With a lifetime prevalence of 26% and 1% daily users, cannabis is the most widely used illegal drug in Europe. Cannabis-related problems have become the most important reason to begin drug treatment (European Monitoring Centre for Drugs and Drug Addiction [EMCDDA], 2017). In recent years, a range of Internet interventions targeting cannabis users have been developed, extending the reach of care to individuals who otherwise would not seek specialized treatment (EMCDDA, 2014).

The therapist-guided program “Quit the Shit” (QTS) is an evidence-based Internet intervention for cannabis use disorder (CUD) made available by the German Federal Centre for Health Education (BZgA). It provides several weeks of counselling for individuals who aim to reduce or quit using cannabis. QTS is currently one of only two evidence-based Internet interventions for cannabis users worldwide that are freely available to the public (Rogers et al., 2017; Tossmann et al., 2011).

To improve user experience and effectiveness, QTS is subject to continuous development. Therefore, identification of user characteristics which may influence treatment response is of vital interest. Such predictors could be used for the further development or to detect vulnerable subgroups of participants who need additional support. However, since only few predictor or moderator trials were conducted in this field of studies, these characteristics are largely unknown.

Regarding demographic variables, neither age nor gender is associated with the effectiveness of Internet interventions for cannabis users (Tait et al., 2013). In interventions targeting alcohol use, female gender predicted beneficial outcomes in two studies (Riper et al., 2008; Henson et al., 2015), while two other trials found no such association (Castro et al., 2017; Blankers et al., 2013). Participants with higher education yielded better results in the studies of Riper et al. (2008) and Castro et al. (2017), an association not replicated by Blankers et al. (2013). Of 46 possible candidate variables in the study of Blankers and colleagues, the most important predictor of a positive treatment outcome was living together with others.

In Internet interventions targeting CUD, the association between initial cannabis use and later treatment outcome has not yet been studied. In online interventions for alcohol abuse, evidence is heterogenous. In a study conducted by Doumas et al. (2016), largest effects were observed among those who were abstinent at baseline. In another trial, participants with low daily alcohol use at baseline had a better outcome than those with daily medium use, weekend only use or no use (Baumann et al., 2017). It is also not clear, how an early onset of substance use in life is associated with treatment outcome. While there is again no evidence from cannabis-related interventions, studies on alcohol-related interventions show diverging results. According to Doumas et al. (2016), participants with an onset of alcohol use at 11 years or earlier achieved the best results. In contrast, subjects in the study of Henson et al. (2015) were less likely to respond if they started drinking early in life. In two web-based brief interventions targeting CUD, beneficial outcomes were found among participants with a higher readiness to change (Lee et al., 2010; Palfai et al., 2016). In studies being conducted in a traditional face-to-face intervention setting, cannabis users with a higher degree of refusal self-efficacy (i.e. the belief to abstain from cannabis use in certain situations) and those who were legally coerced into treatment achieved better outcomes (Connor et al., 2014; Copeland and Maxwell, 2007).

To broaden evidence in this field of research and to use the results as potential basis for further development of QTS, we tested a wide range of possible predictors of treatment outcome in QTS participants. Our goal was to develop a parsimonious and efficient model, containing the most relevant predictors of treatment response.

Methods

Study design

We used data from a randomized factorial trial in which we tested whether shortening the program duration of QTS from 50 to 28 days or whether removing live counselling via chat had negative impact on the program effectiveness. None of these changes had meaningful impact on the effectiveness of QTS (Jonas et al., 2018).

In the study, participants either started into QTS via a one-to-one chat with a counsellor or via a self-guided tour, depending on the randomization result. All users of QTS, irrespective of being study participants or not, are to commit themselves to a use-related goal at the beginning of the intervention. On this occasion, they are to choose whether they aim to reduce, pause or quit using cannabis. Individuals who only want to reduce or pause cannabis consumption are requested to set a clearly defined and substantial goal such as e.g. “reduce from daily use to weekend-use with one joint on each evening”.

After starting into QTS, all participants gain access to the personal login area of the program. The login area consists of a cannabis use diary and several coping exercises. Once a week, participants receive detailed feedback by their counsellor on all their entries. More details on the program can be found elsewhere (Jonas et al., 2018).

Measures

Selection of potential predictors was mainly based on the quoted evidence and on considerations specific to QTS. In total, we included 31 potential predictors from the following categories: socio-demographic, variables related to substance use and one measure associated with cognitive processing.

As socio-demographic measures, we included age, gender, level of education and housing situation. Predictor candidates related to substance use included the cannabis use days, cannabis use quantity (grams) and cannabis use events, each referring to the 30 days prior to baseline and measured using the Timeline Followback method (TLFB; Sobell and Sobell, 1992). Other predictors were cannabis dependence measured by the Severity of Dependence Scale (SDS; Gossop and Darke, 1995; Steiner et al., 2008) and cannabis craving measured by the cannabis craving screening (CCS-7; Schnell et al., 2011). Coping self-efficacy, i.e. the confidence to control one's own cannabis use in high risk situations, was measured with the Drug-Taking Confidence Questionnaire (DTCQ-8; Sklar and Turner, 1999). Self-developed items were used to measure the age of first-time cannabis use, the cannabis use in the social environment, the perceived support by the social environment, the preferred effects when using cannabis, whether the pursued behavior change was motivated by a third party and if other drug treatment was currently attended. Moreover, the days participants consumed alcohol, cigarettes, amphetamines, cocaine, ecstasy, LSD and other illegal drugs within 30 days prior to study baseline were each included as potential predictors. We also included the goal commitment of each client. As described above, clients are to define whether they want to quit, reduce or pause using cannabis. The participants' degree of self-reflection was also included as a candidate predictor. Self-reflection is a metacognition which refers to the monitoring and evaluation of one's thoughts, feelings and behaviors, and can be regarded as an antecedent for purposeful behavior change (Roberts and Stark, 2008). Self-reflection was measured with the 20-item Self-reflection and Insight Scale (SRIS; Grant et al., 2002; Roberts and Stark, 2008). All potential predictors were collected at study baseline prior to randomization, except for the use-related goal, which is set on the first day of participation.

Treatment response was defined as binary outcome (“responder” vs. “non-responder”) and measured during the follow-up three months after randomization. To be categorized as responder, participants had to reduce cannabis use days, use quantity and use events by at least half between baseline and follow-up. Further criteria for being counted as responder were no increase in any other illegal substance or alcohol between baseline and follow-up, as well as cannabis use on a maximum of 8 days during the past 30 days measured at follow-up (proxy indicator for weekend-use). Participants not meeting all of these criteria were defined as non-responder of the intervention.

Participants and recruitment

Trial participants were recruited from the regular program website https://www.quit-the-shit.net, from all individuals who were interested in signing up for QTS. In total, 534 individuals met all study criteria and were included in the trial. Of those, 252 individuals (47.2%) provided data at the first follow-up three months later. Data on treatment response was available from 239 participants (44.8%). In the present study, we analyzed data from study baseline and from this follow-up survey. Participants and recruitment procedure are detailed in the prior publication (Jonas et al., 2018).

Statistical analysis

To analyze the association between potential predictors and treatment outcome we used unbiased recursive partitioning, a non-parametric regression and classification approach adopted from machine learning (Hothorn et al., 2006; Strobl et al., 2009). Recursive partitioning has gained popularity over the past years (Zhang and Singer, 2010). In psychology and medicine, recursive partitioning was applied to predict substance use-related, mental health and medical outcomes (e.g. Blankers et al., 2013; Berman and Hegel, 2014; Koskas et al., 2015). One reason for its gaining popularity is its straightforward interpretation, since its results are usually represented as classification tree. Recursive partitioning methods are not bound to assumptions of parametric regression methods and allow to analyze a high number of variables and their interactions simultaneously even in relatively small samples (Strobl et al., 2009). Predictor variables were included in the classification tree if they met or fell below a Bonferroni-corrected significance level of alpha = 0.05.

To assess the stability and performance of the classification tree, its results were compared to models developed with all-subsets logistic regression and random forest. In a random forest, not a single classification tree, but a whole ensemble of trees (e.g. 500) is combined using recursive partitioning (Strobl et al., 2009). In all-subsets logistic regression, every combination of predictor variables is tested in search for the best model (Field et al., 2012; Kabacoff, 2015). Predictive accuracy, sensitivity and specificity of the resulting three models were calculated using 10-folds cross-validation.

Missing data was estimated using an Expectation Maximization (EM) algorithm resulting in one imputation. EM (Dempster et al., 1977) is an iterative approach frequently used for single imputation and was shown to give reasonable estimates of missing data on alcohol use in an earlier simulation study (Blankers et al., 2010). We used both factor variables and all measures described above as imputation variables. Before conducting the imputation, we applied a square root transformation on any variable quantifying substance use, i.e. the use days of cannabis and of any other substance named above; the number of use events and the use quantity of cannabis.

Data processing and analysis was conducted with R 3.5.1 (R Core Team, 2018), utilizing the package Amelia II (Honaker et al., 2011) for the imputation and party (Hothorn et al., 2006), caret (Kuhn, 2018) and glmulti (Calcagno, 2013) for the predictor analyses.

Results

Sample description

The majority of the participants were male (65.7%) and had a high educational level with 64.7% attendance or successful completion of the highest German secondary school type (see Table 1). Cannabis use was high with only a few abstinent days last month. Randomization resulted in similar groups, except for a small age-related difference in factor 2 (Treatment length; OR 0.972, 95% CI 0.948–0.996, p = .026) and in participation goal in factor 1 (Chat-based communication; OR = 1.411, 95% CI 1.001–1.992, p = .050).

Table 1

Participant characteristics at baseline and goal commitment.

	Factor 1: chat-based communication		Factor 2: length		All participants (n = 534)
	No (n = 263)	Yes (n = 271)	28 days (n = 266)	50 days (n = 268)
Gender, n (%)
Female	85 (32.4%)	98 (36.2%)	91 (34.3%)	92 (34.3%)	183 (34.3%)
Male	178 (67.6%)	173 (63.8%)	175 (65.7%)	176 (65.7%)	351 (65.7%)
Age, mean (SD)	27.5 (7.3)	27.6 (6.7)	28.2 (7.1)	26.8 (6.8)	27.5 (7.0)
Educational level, n (%)
Basic school (Hauptschule)	25 (9.5%)	29 (10.7%)	30 (11.3%)	24 (9.0%)	54 (10.1%)
Middle school (Realschule)	64 (24.4%)	57 (21.0%)	61 (23.0%)	60 (22.4%)	121 (22.7%)
High school (Gymnasium)	165 (62.7%)	181 (66.8%)	167 (62.8%)	179 (66.8%)	346 (64.8%)
Other school	9 (3.4%)	4 (1.5%)	8 (3.0%)	5 (1.9%)	13 (2.4%)
Housing situation, n (%)
Alone	70 (26.6%)	56 (20.7%)	66 (24.8%)	60 (22.4%)	126 (23.6%)
With parents	41 (15.6%)	59 (21.8%)	45 (16.9%)	55 (20.5%)	100 (18.7%)
With partner	85 (32.3%)	90 (33.2%)	88 (33.1%)	87 (32.5%)	175 (32.8%)
Shared flat	38 (14.4%)	39 (14.4%)	40 (15.0%)	37 (13.8%)	77 (14.4%)
Other	29 (11.0%)	27 (10.0%)	27 (10.2%)	29 (10.8%)	56 (10.5%)
Cannabis
Use days, mean (SD)a	24.7 (7.3)	25.1 (6.5)	24.9 (7.0)	24.9 (6.8)	24.9 (6.9)
Use occasions, mean (SD)a	122.5 (111.6)	120.1 (104.1)	123.7 (108.9)	118.9 (106.8)	121.2 (107.7)
Amount (grams), mean (SD)a	23.2 (18.8)	21.3 (18.6)	23.2 (19.6)	21.3 (17.8)	22.2 (18.7)
SDS, mean (SD)	9.9 (2.8)	10.0 (2.7)	10.1 (2.5)	9.8 (2.9)	10.0 (2.7)
CCS-7, mean (SD)	4.4 (1.3)	4.2 (1.2)	4.4 (1.3)	4.2 (1.3)	4.3 (1.3)
DTCQ-8, mean (SD)	46.1 (18.8)	47.6 (18.0)	45.8 (17.6)	47.8 (19.1)	46.8 (18.4)
Preferred effect, n (%)
Soft/mild	100 (38.0%)	107 (39.5%)	109 (41.0%)	98 (36.6%)	207 (38.8%)
Strong/intense	163 (62.0%)	164 (60.5%)	157 (59.0%)	170 (63.4%)	327 (61.2%)
Other substancesa
Alcohol use days, mean (SD)	4.8 (6.1)	3.8 (4.8)	4.5 (5.8)	4.1 (5.2)	4.3 (5.5)
Use of illegal substances, n (%)	57 (21.7%)	48 (17.7%)	58 (21.8%)	47 (17.5%)	105 (19.7%)
Other measures
Self reflection (SRIS)	54.0 (6.3)	54.5 (6.3)	54.1 (6.6)	54.4 (6.0)	54.3 (6.3)
Goal commitment
To reduce/pause, n (%)	103 (39.2%)	129 (47.6%)	106 (39.8%)	126 (47.0%)	232 (43.4%)
To abstain, n (%)	160 (60.8%)	142 (52.4%)	160 (60.2%)	142 (53.0%)	302 (56.6%)

During the past 30 days.

Identification of predictors

According to the classification tree (Fig. 1), the goal commitment (p < .001), the extent of self-reflection (p < .001), the preferred effect of cannabis (p = .005) and the cannabis use days (p = .015) significantly differentiate between successful and non-successful participants. Participants regarded as treatment responders by the above definition make up a good half of the whole sample (52.8%).

Fig. 1

Classification tree.a

aScales indicate the proportion of responders in each group.

The most important predictor is the use-related goal, chosen at the beginning of the program. With 64.9% being treatment responders, individuals who aim to abstain from using cannabis (i.e., the right main tree branch; n = 302) are significantly more likely successful than those who only want to reduce or pause consumption (the left main branch; n = 232; with 37.1% responders; values not shown in Figure).

Participants who aim to cease using cannabis can further be differentiated by their score on the Self Reflection and Insight Scale (SRIS) and by their initial cannabis use. With 75.1% treatment responders, the large group of individuals with >44 points in the SRIS and >22 cannabis use days at baseline (n = 209) is particularly successful. In contrast, participants with exceptionally low self-reflection (n = 28) have a low rate of treatment response (25.0%).

Individuals, who aim to reduce or pause using cannabis are significantly differentiated by their preferred cannabis effect. Participants who favor getting mildly intoxicated (n = 103) have a success rate close to the overall mean (50.5%), whereas individuals aiming for strong psychoactive effects (n = 129) have a particularly low response rate (26.4%).

Comparative results

The logistic regression and random forest models likewise identify the goal commitment, the preferred effect and the degree of self-reflection as the three most relevant predictors (Table 2). In contrast to the classification tree, the amount of cannabis used emerged as fourth predictor. Like the cannabis use days, it is positively associated with treatment response. Another predictor is the educational level, according to which participants with a middle and high education have significantly higher response rates than individuals with low education.

Table 2

Predictors found in the all subsets logistic regression and random forest.

Logistic regressiona				Random forestb
Predictor	OR (95% CI)	T-value (absolute)	p-value	Predictor	Variable importance
1. Goal commitment: to abstain	5.06 (3.00; 8.72)	5.960	< .001	1. Goal commitment	0.050
2. Preferred effect: soft	2.77 (1.82; 4.24)	4.739	< .001	2. Preferred effect	0.014
3. Self-reflection	1.08 (1.05; 1.12)	4.590	< .001	3. Self-reflection	0.013
4. Cannabis use amount	1.02 (1.01; 1.04)	3.501	< .001	4. Cannabis use amount	0.011
5. Educational level: middle	3.13 (1.50; 6.62)	3.019	.003	5. Educational level	0.005
6. Educational level: high	2.60 (1.34; 5.14)	2.786	.005	6. Cannabis use days	0.005
7. Cannabis use days	1.04 (1.01; 1.07)	2.365	.018	7. Cannabis use events	0.004
8. Cannabis use in social environment	1.20 (1.01; 1.44)	1.998	.046	8. Alcohol use days	0.003
Other predictor candidates			n.s.		–

Sorted downwards by the absolute value of the t-statistic; only significant predictors shown.

Sorted downwards by the variable importance. Following the recommendation of Strobl et al. (2009), only the ranking of the predictors, and not their importance values, is interpreted.

The accuracy, sensitivity and specificity of the three models is shown in Table 3. With an accuracy of 0.64, the performance of the classification tree is slightly lower than the performance of the other two models. The overlapping confidence intervals however indicate no significant difference. Compared to the accuracy which would be achieved without a model (no-information rate; NIR = 0.53; Kuhn and Johnson, 2013), the classification tree significantly increases prediction by 11%. With 0.65 and 0.63, modest rates for sensitivity and specificity are achieved in the classification tree.

Table 3

Comparison of performance characteristics.

	Accuracy [95%-CI]	P [Accuracy vs. NIRa]	Sensitivity	Specificity
Classification tree	0.64[0.60–0.68]	<.001	0.65	0.63
Logistic regression	0.67[0.63–0.71]	<.001	0.70	0.64
Random forest	0.68[0.64–0.72]	<.001	0.71	0.65

NIR: No-information rate.

Discussion

In the present study, we aimed to develop a parsimonious and efficient prediction model of treatment response in QTS, a web-based intervention for CUD. By using a classification tree, we identified the commitment to abstain as the most important predictor for treatment response, followed by the degree of self-reflection, the preferred effect and the initial cannabis use. The relevance of these predictors was verified by two other statistical methods.

Compared to the mere wish to reduce, aiming to quit obviously implies a stronger desire to change and therefore leaves less room for doubts and ambivalence. Corresponding results were found in a recent study on adolescents receiving outpatient alcohol treatment (Kaminer et al., 2018). In that trial, participants who were committed to abstinence were less likely to continue problem drinking than those who only pursued harm reduction. Devotion to change also seems less susceptible to future circumstances and contingencies than the motivation to change (Kelly and Greene, 2014), which itself predicted treatment response in two web-based brief interventions for cannabis users (Lee et al., 2010; Palfai et al., 2016). This evidence can be seen as an impetus for a clearer focus on abstinence within QTS. The benefits of abstinence should probably be communicated more clearly, without however sacrificing the benefits of a low-threshold intervention, which explicitly targets individuals who are ambiguous about behavior change.

The degree of self-reflection also seems to play a relevant role in achieving behavior change in QTS. As the low cut-point in the respective tree node suggests, only a moderate level of self-reflection seems to be necessary to use the intervention efficiently, and to successfully implement behavior change strategies. According to the results, higher levels of self-reflection do not necessarily promote the chances of success any further. Though, not yet supported by other evidence in the field of study, these results point to the role of self-reflection as antecedent of self-regulation and behavior change.

In the large subgroup of participants who strive for abstinence, and who at least have a moderate degree of self-reflection, initial cannabis use further differentiates response rates. The high response rates among individuals with (almost) daily use may largely be explained by the close conceptual link between this predictor and the outcome. Since it makes little sense to view high initial cannabis use as a factor of success, this however points to the question whether this, and similar use-related baseline variables, should have been included in the analysis. We decided to do so, since we aimed to develop a model with the highest possible predictive accuracy.

Among participants who aim to reduce cannabis use, preference for strong intoxicating effects is associated with particularly low chances of success. This affinity presumably goes hand-in-hand with more and stronger use motives, which themselves might pose an obstacle to engage in behavior change . In an earlier study among individuals who utilized an online-self-screening for cannabis users, the preference for strong intoxicating effects was closely associated with intense cannabis use, a high degree of cannabis dependency and the presence of several cannabis use motives (Jonas et al., 2009).

In total, the classification tree increases prediction accuracy by merely 11% compared to chance. The inclusion of other possible predictors, like use-related norms or outcome expectancies (Henson et al., 2015; Connor et al., 2014), could have helped to increase the predictive accuracy by some degree. However, the limited accuracy of other prediction models (e.g. Blankers et al., 2013) and the partly inconsistent evidence in this field of study (e.g. Doumas et al., 2016, Baumann et al., 2017) point to the difficult task of identifying substantial and reliable, client-related early predictors for treatment response. Therefore, the predictive accuracy of the model probably also would be limited if we would add other baseline variables.

Strengths and limitations

The present analyses have the following strengths. We included a wide range of potential predictors which previously were found to be associated with treatment outcome. We also defined the response variable conservatively, including other criteria besides cannabis use. In contrast to most other predictor or moderator analyses, our results were cross-validated and compared with other analysis methods.

The shortcomings of the underlying trial include a limited follow-up rate, and the reliance on self-reported data (see Jonas et al., 2018, for details). The current analyses also have several limitations: Since validated instruments for some potential predictors were not available, these variables were measured with self-developed and previously untested items. Therefore, we cannot be sure whether the targeted constructs were measured reliably and in all their facets. Moreover, it may be questioned whether our classification tree could be replicated by independent data. Although we cross-validated the derived model on resampled data, it is well likely that the exact structure of the classification tree would not be replicated in later samples of QTS participants or in other interventions targeting CUD. The tree therefore should always be interpreted cautiously. The only alternative to get better information on the generalizability of our models would have been to cross-validate results by splitting our dataset in two independent parts, one to train the model and the other to test its predictive accuracy. Given the limited sample size of the study, such a data split however was no viable option, as it likely would have resulted in greater imprecision of estimates (Kuhn and Johnson, 2013).

Conclusions

In the present study, the commitment to quit using cannabis, moderate to high levels of self-reflection and an affinity for softer intoxicating effects were identified as the three most relevant predictors of a positive outcome in the web-based CUD intervention QTS. In contrast, individuals who only aim to reduce or pause cannabis use, those who prefer an intense intoxication, or those who have a low degree of self-reflection, were apparently less likely to be successful. They should be given special consideration in the counselling process and when developing or optimizing Internet interventions targeting CUD.

To anticipate treatment response on an individual level, the classification tree should only be used as one of several sources of information. Decisions, like the allocation of participants to intervention modules, or the recommendation for further treatment, should always be made using all available information on the specific case. To enhance classification accuracy, further potential predictors should be tested.

Role of funding source

The study was funded by the Federal Centre for Health Education (BZgA). The BZgA provided technical infrastructure for the data collection and approved the manuscript.

Contributors

Design of the study: BJ

Design of the intervention: MT, ES, PT

Collection of data: BJ

Analysis of data: BJ

Manuscript writing: BJ, MT, FL, ES, PT

Declaration of Competing Interest

Benjamin Jonas, Marc Tensil, Fabian Leuschner and Peter Tossmann are researchers at Delphi Gesellschaft which developed “Quit the Shit” on behalf of the BZgA. Evelin Strüber is project manager for www.drugcom.de and “Quit the Shit” at the BZgA. None of the authors derive personal or financial benefit from the results.