Evaluation of a brief unguided psychological online intervention for depression: A controlled trial including exploratory moderator analyses.

BACKGROUND: Psychological online interventions (POIs) reduce depression but we know little about factors influencing their effectiveness. We evaluated a new, brief POI for depression and conducted exploratory moderator analyses.
METHODS: In this online trial (German Clinical Trials Register; DRKS00011045), we allocated participants to treatment as usual (TAU; n = 67) or POI (n = 65). At first, we randomized participants; later we allocated participants based on depression severity in order to counter baseline differences. The unguided POI addressed behavioral activation and depressive thinking in a single module with 25 webpages (including a smartphone application). We did one assessment at baseline and a post-assessment four weeks later.
RESULTS: At post-assessment, depression (p = .586), behavioral activation (p = .332), and dysfunctional attitudes (p = .499) did not differ between groups. When concurrent treatments (medication/psychotherapy) remained constant/decreased, the POI outperformed TAU (p = .031). POI-participants with lower willingness to change (p = .030) or higher education (p = .017) were less likely to worsen (i.e., experience increased depressive symptoms) compared to TAU. DISCUSSION: The targeted sample size was not reached, measurements were self-reported, and randomization failed. The POI's content may have been too limited. Concurrent treatments, which were more often sought out by TAU participants, diminished group differences and should be considered in future studies. Brief POIs may protect against worsening of depressive symptoms among highly educated participants or those with low willingness to change.

Introduction

Although there are several effective treatment options for depression, only a subset of people receive empirically based treatment (Kazdin, 2017; Kohn et al., 2004). It has been proposed that psychological online interventions (POIs) could help to narrow this treatment gap (Kohn et al., 2004). Meta-analyses have confirmed that POIs are effective in reducing depressive symptoms, yielding small effects (d = 0.25–0.36, Hedges' g = 0.27) for unguided POIs, and medium to large effects (d = 0.58–0.78) for guided POIs (Karyotaki et al., 2017; Richards and Richardson, 2012; Saddichha et al., 2014). A recent meta-analysis found equivalent effects for unguided versus guided interventions for comorbid anxiety and depression (Pasarelu et al., 2017). Although their efficacy is well established, it remains largely unresolved for whom POIs are effective, for whom they are not, and for whom they may even be harmful. In this context, one can differentiate between outcome predictors (i.e., variables that predict a positive outcome irrespective of the received treatment) and moderators (i.e., variables that influence whether a treatment is superior to control conditions; see Kraemer et al., 2006). Regarding outcome predictors, better outcomes have been reported for women, as well as for individuals who have more education or fewer dysfunctional attitudes (Donker et al., 2013; Warmerdam et al., 2013). The effect of baseline depression severity as an outcome predictor remains unclear as high baseline depression was found to be a predictor of greater improvement in depression (Warmerdam et al., 2013), whereas Sunderland et al. (2012) found that high symptom severity was a predictor of low improvement. El Alaoui et al. (2016) examined a large cohort of depressive patients receiving internet-based cognitive behavioral therapy (iCBT) in a naturalistic setting without a control group. They found that adherence to treatment, working full time while undergoing treatment, and perceiving the treatment as credible were associated with a greater rate of improvement and lower post-treatment depression, as measured with the Montgomery Åsberg Depression Rating Scale Self-Rated (MADRS-S). Regarding moderators, Button et al. (2012), who conducted a secondary analysis on data from a randomized controlled trial, found that high baseline depression, as well as being widowed, divorced, or separated from one's partner predicted lower depression scores, measured with the Beck Depression Inventory (BDI), at 4-month follow-up. Terides et al. (2018), who conducted mediation rather than moderation analyses, found that increased usage of skills traditionally taught in cognitive behavioral therapy was associated with reduced symptoms after iCBT treatment (Patient Health Questionnaire, Generalized Anxiety Disorder-7 Item) and increased life satisfaction (Satisfaction With Life Scale). So far, findings on outcome predictors and moderators are preliminary as they stem from single and often underpowered studies. In contrast, the highly powered meta-analysis by Karyotaki et al. (2017), which assessed the efficacy of self-guided iCBT using individual participant data, found no moderators.

In comparison to outcome predictors and moderators of improved treatment outcomes, there are only a few studies that have examined negative outcomes of POIs (Boettcher et al., 2014; Rozental et al., 2014). Regarding moderators, a recent meta-analysis found that deterioration (i.e., a significant worsening of symptoms based on the reliable change index) is generally lower in POI groups compared to control groups (Ebert et al., 2016). When education is low, however, deterioration rates are no longer different between groups, indicating that low education might serve as a risk factor (Ebert et al., 2016). Regarding outcome predictors, El Alaoui et al. (2016) found that a history of psychotropic medication use is associated with slower improvement and higher post-treatment depression.

More research is needed to examine moderators of positive and negative outcomes of POIs (Ebert et al., 2016; Rozental et al., 2014; Schröder et al., 2016). Therefore, in this study, we not only evaluated a newly developed brief POI for depression but we also conducted moderator analyses to detect variables that influence its effectiveness. The POI evaluated in this study was newly developed by the authors. While more extensive POIs that contain several modules have already proven effective (Karyotaki et al., 2017), we decided to administer a very brief POI to answer the question if a focused approach could be similarly efficacious. Our POI addressed behavioral activation and cognitive restructuring, two core aspects of cognitive behavioral depression treatment that effectively reduce depression in face-to-face psychotherapy (Cuijpers et al., 2013; Ekers et al., 2014). Our aims were (1) to evaluate the effectiveness of our POI, (2) to find moderators of post-treatment depression scores, and (3) to identify moderators of the worsening of depressive symptoms from pre- to post-assessment.

We hypothesized that, compared to treatment as usual, allocation to the POI would result in lower depression scores at post-assessment, as well as less dysfunctional attitudes and higher behavioral activation scores. We made no a priori hypotheses regarding moderator analyses given their exploratory nature.

Methods

This pre-registered controlled online trial (German Clinical Trials Register; ID: DRKS00011045) was approved by the local Ethics Committee of the University of Hamburg, Germany. All participants gave electronic informed consent online at the start of the trial.

Trial design

Patients were allocated either to a brief 4-week POI or to TAU. For both groups we imposed no restrictions regarding concurrent treatments. Therefore, TAU could include participants who underwent concurrent treatments but also participants who did not receive any treatment. Participants were not blinded. Outcome measures were assessed at baseline and after 4 weeks, using the online survey program “EFS survey” developed by Questback (version EFS Fall 2016).

Initially, we planned the trial as a parallel group, individually randomized superiority trial with a randomization ratio of 1:1. The randomization plan, which was electronically generated, was provided by a statistician in the biometrics department at the University Medical Center Hamburg-Eppendorf. The study personnel were not blind to randomization. After each participant had completed the baseline assessment, we checked whether any of the exclusion criteria were met and then randomized the participant. This was usually done within few days of the participant's completion of the baseline assessment. We noticed baseline differences on the primary outcome after recruitment of 109 eligible participants (73% of the pre-defined target sample size). This check was conducted at that time because we had to manually inspect the data to exclude participants who reported a diagnosis of psychotic or bipolar disorder. It became apparent that randomization had failed; there was a significant group difference on baseline depression scores (p = .027, η2p = 0.043), with more severe depressive symptoms in the POI group. As depression served as the main outcome, we decided to change our allocation strategy thereafter from randomization to allocation based on depression scores. This was a deviation from the initial protocol. Whenever two persons had completed the baseline assessment, we inspected their depression scores (PHQ-9) and allocated the participant with higher scores to TAU and the one with lower scores to the POI. TAU participants received access to the POI after completing the post-assessment.

Recruitment

Recruitment was conducted exclusively online. Between September 1st, 2016, and June 24th, 2017, we contacted potential participants from a database, which consisted of participants who took part in former depression studies of the first author's research unit and patients from a nearby outpatient clinic who were undergoing or awaiting psychotherapy for depression. All potential participants from the database had given informed consent to be contacted via e-mail. Individuals were eligible to participate if they (1) were 18 to 80 years old, (2) reported at least mild depressive symptoms (Patient Health Questionnaire score > 4; Löwe et al., 2004), (3) had access to a PC with Internet connection, and (4) agreed to enter an e-mail address to receive information throughout the study. Owning a smartphone was not mandatory. Exclusion criteria were (1) a diagnosis of psychotic or bipolar disorder as well as (2) acute suicidal tendencies indicated by a score of 2 or higher on the BDI-2 suicide item (for the predictive validity of the item, see Green et al., 2015). If a participant had to be excluded due to suicidal thoughts, we provided them with emergency phone numbers, as well as webpages and contacted them to offer assistance. All measures were based on self-report.

Fig. 1 depicts the schedule of the study as a flow diagram.

Fig. 1

Flow diagram of recruitment, enrollment, and allocation.

Intervention

The POI aimed to convey behavioral activation skills, as well as the correction of depressive thinking styles (“cognitive restructuring”). Behavioral activation components were adopted from Schaub et al. (2006), while cognitive restructuring was based on the metacognitive training approach for depression (D-MCT; Jelinek et al., 2015a; Jelinek et al., 2015b). The POI consisted of psychoeducational information about depression, as well as worksheets with psychotherapeutic strategies. These worksheets contained check boxes, text boxes or drop down menus and were meant to help apply the psychoeducational information in practice. The POI consisted of one module that was comprised of 25 webpages. We provided a smartphone application for operating systems “Android” and “Apple iOS” that was designed to remind and support participants to use techniques from the POI in their everyday life. For example, participants could use the smartphone application to plan behavioral activation activities and to set reminders when these activities were due. The POI was unguided, meaning that we did not contact participants who used the intervention to offer guidance. We did, however, answer questions if participants contacted us via e-mail or telephone. Also, we sent one reminder e-mail two weeks after they had received access to the POI to all participants in the POI group, irrespective of whether they had already logged into it or not, in which we reminded them to use the POI. We did not provide instructions for the participants regarding how to use the POI or the application. We simply informed them that they could use the POI wherever and whenever they wanted, self-paced and anonymously within the next month. Participants in the POI group could continue to use the POI after the four weeks were done. Fig. 2 shows a web browser screenshot of the POI, and Fig. 3 shows three screenshots from the smartphone application.

Fig. 2

Screenshot of the POI, showing the second of 25 webpages of the POI accessed through an internet browser on a PC. On this webpage, the interplay of feelings, thoughts, and behavior is emphasized. Additionally, an example is provided of how feelings, thoughts, and behavior can negatively affect each other in depression in a vicious circle.

Fig. 3

Screenshots of the smartphone application (f.l.t.r.). Screenshot 1 shows how participants could set a timer to be reminded of a planned activity; on screenshot 2, participants could note possible obstacles that might discourage them and could find solutions for overcoming these obstacles; on screenshot 3, participants could indicate whether they had completed the activity and how they felt afterwards, and they also received an encouraging message (“Well done!”).

Measures

The Patient Health Questionnaire (PHQ-9; Löwe et al., 2004) served as the primary outcome. The scale consists of 9 items using 4-point Likert-scales and is valid and reliable (Löwe et al., 2004; Martin et al., 2006). For certain moderator analyses we dichotomized PHQ-9 change scores into worsening of depressive symptoms (if scores increased) vs. no worsening of depressive symptoms (if scores decreased or remained unchanged) from baseline to post-assessment. The Behavioral Activation for Depression Scale (BADS; Fuhr et al., 2016) measures behavioral activation levels in depression and served as a secondary outcome. The 25-item questionnaire is comprised of four subscales; “activation”, “avoidance/rumination”, “work/school impairment”, and “social impairment”. Internal consistency and test-retest reliability are acceptable (Teismann et al., 2016). An 18-item version of the Dysfunctional Attitudes Scale (DAS 18-B; Rojas et al., 2014) was implemented to measure dysfunctional attitudes on a 7-point Likert-scale. The higher the score, the more dysfunctional attitudes are endorsed. The DAS-18B shows good reliability and validity (Rojas et al., 2014). The “action”-subscale of the University of Rhode Island Change Assessment (URICA; Hasler et al., 2003) was used to measure participants' willingness to change. It assesses a person's efforts to actively change their own behavior or their environment (“I am doing something about the problems that had been bothering me”), and demonstrates very good internal consistency (Hasler et al., 2003). The Attitudes towards Psychological Online Interventions Questionnaire (APOI; Schröder et al., 2015) measures participants' acceptance towards POIs across four subscales (skepticism and perception of risks, confidence in effectiveness, technologization threat, and anonymity benefits) on a 5-point Likert-scale. The APOI shows good internal consistency (Schröder et al., 2015).

Statistical analyses

We used SPSS 24® for all analyses except the power analysis, which was conducted using G*Power version 3.1 (Faul et al., 2009). To compare groups at baseline, we conducted χ2- and t-tests.

Power analysis

A priori, we conducted a power analysis assuming a medium effect (f = 0.25) based on previous findings (Jelinek et al., 2015b; Richards and Richardson, 2012), a type 1 error rate of 0.05, a power of 0.80, and an attrition rate of 20%, resulting in a target sample size of 150, to reach 120 participants after dropouts. The power analysis was designed solely to meet aim 1, namely, to test the POIs effectiveness, and was unaffected by exploratory moderator analyses (aim 2).

Main outcomes: intention to treat, complete cases, and per protocol analyses

In intention to treat (ITT) analyses, we used multiple imputation (MI) with 20 imputations to account for dropouts. Based on MI, pooled p-values are reported. All ANCOVA models include the baseline score of the respective outcome as a covariate, the outcome at post-assessment as the dependent variable, and group as the independent variable. ITT analyses with imputed post scores included all 132 participants (TAU, n = 67; POI, n = 65), whereas complete case analyses included 103 participants (TAU, n = 54; POI, n = 49).

Moderator analyses

The term moderation indicates that the size or direction of an effect is contingent on another variable, called the ‘moderator variable’. In the regression equation, this dependency of effects is modelled by an interaction term (Hayes, 2013, p. 211).

We used the SPSS 24® plugin PROCESS (Hayes, 2013) to identify moderators of treatment outcomes. All moderator analyses included the baseline score of the respective outcome as a covariate, group as independent variable, and the moderator variable, as well as the interaction of group and the moderator variable as additional covariates. PROCESS estimates models with both continuous and binary outcomes but as it requires a complete dataset, we used complete cases rather than imputed data. When a moderator variable was binary, we followed up a significant interaction by conducting subgroup analyses in both groups (e.g., low vs. high education). We report bootstrap confidence intervals based on 5000 samples (ULCI = upper level of confidence interval, LLCI = lower level). Moderator analyses were exploratory without correcting for multiple comparisons. There were 18 moderators and two outcomes, depression and worsening of depressive symptoms, resulting in a total of 36 analyses.

Results

Retention and baseline characteristics

The final sample consisted of 132 participants. Baseline characteristics and group differences are displayed in Table 1.

Table 1

Sample characteristics at baseline.

Characteristics	TAU (n = 67)	POI (n = 65)	Statistics
Demographics
Age in years, mean (SD)	36.52 (13.39)	38.54 (11.82)	t(130) = 0.916, p = .361
Gender (female), proportion (%)	45/67 (67.16)	54/65 (83.08)	χ2(1) = 4.456, p = .035
Education (≥11 years), proportion (%)	49/67 (73.13)	43/65 (66.15)	χ2(1) = 0.761, p = .382
Employed or in education, proportion (%)	47/67 (70.15)	41/65 (63.07)	χ2(1) = 0.743, p = .389
Clinical variables
Proportion reporting depression diagnosis (%)	60/67 (89.55)	52/65 (80.00)	χ2(1) = 2.342, p = .126
Proportion taking psychotropic medication (%)	24/67 (35.82)	25/65 (38.46)	χ2(1) = 0.137, p = .934
Proportion undergoing psychotherapy (%)	30/67 (44.78)	33/65 (50.77)	χ2(1) = 0.475, p = .491
Proportion completing pre- & post-assessment (%)	54/67 (80.60)	49/65 (75.38)	χ2(1) = 0.523, p = .470
Outcome variables
PHQ-9, mean (SD)	13.58 (4.08)	14.11 (4.41)	t(130) = 0.712, p = .478
BADS, mean (SD)	68.04 (19.57)	64.98 (20.52)	t(130) = 0.877, p = .478
DAS, mean (SD)	76.10 (17.01)	74.68 (20.68)	t(130) = 0.434, p = .665

Note. “Psychotherapy” includes both inpatient and outpatient treatment; “reporting depression diagnosis”: diagnoses were self-reported and not verified.

Treatment adherence

We analyzed log files to derive a measure of utilization intensity. Whenever a participant clicked on “login”, “next page”, “save page”, etc., the POI saved these clicks in a log file. For two participants (3.1%), log file data was not available. Of those allocated to the POI, 12 (18.5%) did not use it. The mean number of clicks in the POI was 71.79 (SD = 123.72); the median was 44, which is approximately the amount of clicks needed to work through the POI once. The number of clicks ranged from 0 to 916. As a second measure of usage intensity, we counted the number of logins. The mean number of logins was 3.49 (SD = 4.14); the median was 2, and the number of logins ranged from 0 to 18.

Treatment efficacy: primary and secondary outcomes

To address aim 1 of our study (evaluating the effectiveness of the POI), we analyzed complete cases, intention to treat with imputed data, and per protocol (PP) data. In PP-analyses, we excluded participants who provided post-assessment data but did not log into the POI at least once (n = 8). Results of primary and secondary outcome analyses are presented in Table 2. There were no significant group differences at post-assessment for any of the outcomes.

Table 2

Adjusted means as well as effects of group (ANCOVAs) on primary and secondary outcomes.

Outcome measure	Adj. TAU post(n = 54)	Adj. POI post(n = 49)	Complete cases-analysis	ITT-analysis (MI)	PP-analysis (n = 95)
PHQ-9	12.40	12.03	F(1, 100) = 0.207, p = .650, η2p = 0.002	p = .586	F(1, 92) = 0.068,p = .795, η2p = 0.001
BADS	70.49	76.91	F(1, 100) = 2.838, p = .095, η2p = 0.028	p = .332	F(1, 92) = 2.613,p = .109, η2p = 0.028
DAS-18B	70.43	72.20	F(1, 100) = 0.661, p = .418, η2p = 0.007	p = .499	F(1, 92) = 2.183,p = .143, η2p = 0.023

Note. Adj. = adjusted for baseline scores; ITT = intention to treat; MI = multiple imputation, PP = per protocol; we report baseline-adjusted group means at post-assessment, therefore, no SDs were available; unadjusted post scores are as follows: TAU (n = 54): M = 11.89, SD = 5.64, POI (n = 49): M = 12.69, SD = 5.29; for multiply imputed data, we report the pooled significance provided by SPSS 24®.

As stated in the Methods section, we switched from randomization to allocation based on baseline depression severity over the course of the study. To rule out the possibility that the change of allocation biased the results, we repeated the main analysis using only participants that were randomized (N = 109; n(TAU) = 55, n(POI) = 54) rather than allocated based on depression severity. Analyzing only randomized participants had no effect on the direction or the significance of coefficients.

Post-hoc equivalence test of non-significant primary result

The group difference in PHQ-9 scores at post-assessment was non-significant which could mean that there was an existing group difference which was too small to reach significance or that the group difference was, in fact, zero. We used “two one-sided tests” (TOST) to examine this (Lakens, 2017). We entered complete cases PHQ-9 baseline-post-change scores and corresponding SDs for both groups (TAU: M = 1.44, SD = 4.39, n = 54; POI: M = 1.92, SD = 3.65, n = 49) because the TOST spread sheet requires t-test results rather than ANCOVA results. Equivalence bounds were calculated based on an effect size of 0.28 (Cuijpers et al., 2011). The equivalence test (t(100, 27) = 0.82, p = .206) indicated that the observed effect size (d = 0.012) was not significantly within the equivalent bounds. We therefore conclude that neither a difference nor equivalence between groups is supported by the data.

Exploratory moderator and subgroup analyses

We examined the following moderators to address aim 2 of the study, namely, to identify moderators of post-treatment depression scores: age (in years), sex (male vs. female), education (<11 years vs. ≥11 years), willingness to change, number of psychotherapies that participants underwent before taking part in this study, presently undergoing psychotherapy vs. not, number of self-help books that participants had read before, number of online-self-help programs that participants had used before, treatment expectation (single Likert-scale item: “How successful do you expect the self-help program to be?”), APOI total score and subscale scores, number of diagnoses, having a diagnosis of depression (yes vs. no), and dysfunctional attitudes total score and subscales. Additionally, we created a binary variable indicating whether participants started or increased concurrent therapy between baseline and post-assessment. If psychotherapy or medication was initiated, or if an increased dosage of medication was reported, the variable was coded 1. If therapy stayed the same, discontinued or decreased, the variable was coded 0. Of 103 participants providing baseline and post-assessment data, 23 (22.3%) started or increased concurrent treatment between baseline and post-assessment. At trend, more people increased concurrent therapy in the TAU group (16 out of 54; 29.6%) compared to the POI group (7 out of 49; 14.3%), χ2 (1) = 3.487, p = .062. All interactions are depicted in Table 3.

Table 3

Results of exploratory moderator analyses with depression at 4-week post-assessment as outcome (n = 103).

Moderator	Interaction	SE	t	p	LLCI	ULCI
Sex	1.09	2.06	0.53	0.60	−2.99	5.18
Education	−2.31	1.78	−1.30	0.20	−5.84	1.22
Willingness to change	0.25	0.15	1.66	0.10	−0.05	0.55
number of psychotherapies	−0.67	0.48	−1.39	0.17	−1.62	0.29
undergoing psychotherapy	2.44	1.61	1.52	0.13	−0.75	5.63
number of self-help books	0.13	0.07	1.74	0.09	−0.02	0.27
treatment expectation	0.49	0.56	0.88	0.38	−0.62	1.59
APOI total score	0.04	0.11	0.37	0.71	−0.17	0.25
APOI skepticism and risks	0.23	0.35	0.66	0.51	−0.46	0.92
APOI confidence in effectiveness	0.31	0.38	0.80	0.42	−0.45	1.07
APOI technologization threat	−0.13	0.30	−0.42	0.68	−0.72	0.47
APOI anonymity benefits	0.09	0.23	0.38	0.70	−0.37	0.55
Number of diagnoses	−0.44	0.68	−0.65	0.52	−1.79	0.91
Depression diagnosis	0.55	2.42	0.23	0.82	−4.25	5.36
DAS total score	−0.02	0.04	−0.41	0.69	−0.10	0.07
DAS performance evaluation	−0.03	0.08	−0.35	0.73	−0.19	0.14
DAS approval by others	−0.10	0.20	−0.48	0.63	−0.49	0.30
Concurrent therapy	4.47	2.04	2.19	0.03*	0.42	8.53

Notes. Willingness to change = University of Rhode Island Change Assessment, action subscale; APOI = Attitudes towards Psychological Online Interventions Questionnaire; DAS = Dysfunctional Attitudes Scale; models included baseline PHQ-9 as a covariate, depression at post assessment as the outcome, group as independent variable, and the moderator as well as the interaction of group and the moderator as additional covariates; LLCI = lower level bootstrap confidence interval, ULCI = upper level.

<0.05.

As can be seen in Table 3, the interaction of concurrent therapy and group allocation reached significance. Simple slopes were not significant. In subgroup-analyses, we separately analyzed participants who did not receive increased concurrent treatment (n = 80; n(TAU) = 38, n(POI) = 42) and those who received increased concurrent treatment (n = 23; n(TAU) = 16, n(POI) = 7). In the sample with constant/decreased concurrent treatment, the effect of group was significant in favor of the POI group (F(1) = 4.097, p = .046, η2p = 0.051). In the sample of participants receiving increased concurrent treatment, effect was not significant (F(1) = 1.200, p = .286, η2p = 0.057). Therefore, the data indicate that the POI might be beneficial if concurrent therapies apart from the POI stay constant or decrease. We did descriptive analyses to better understand how the groups with and without concurrent treatment differed. Those who received concurrent treatment were slightly younger (M = 35.43, SD = 9.95 vs. M = 38.28, SD = 13.54), were more often male (39% vs. 19%), and were more depressed at baseline (M = 14.52, SD = 3.65 vs. M = 13.71, SD = 4.14), but the level of education was the same (in both groups, 70% had received ≥11 years of education).

Results of moderator analyses with worsening of depressive symptoms as the outcome are presented in Table 4 (increase of depression scores from baseline to post-assessment). Across groups, 29 out of 103 participants worsened (28.2%), 10 out of 49 in the POI group (20.4%) and 19 out of 54 in the TAU group (35.2%), χ2(1) = 2.773, p = .096. For low willingness to change (1 SD below the mean), the probability of worsening of depressive symptoms was lower in the POI group compared to TAU (b = −2.023, SE = 0.804, p = .012). When willingness to change was high (1 SD above the mean), groups did not differ regarding the probability of worsening of depressive symptoms (b = 0.249, SE = 0.649, p = .702). Therefore, the POI was effective in reducing worsening of depressive symptoms only when willingness to change was low at baseline. Furthermore, the probability of worsening of depressive symptoms was lower in the POI group compared to TAU if education was high (b = −1.617, SE = 0.625, p = .010). If education was low, the effect was reversed, albeit not significant (b = 0.894, SE = 0.853, p = .294). This finding was supported by a subgroup analysis comparing the effect in the low education subgroup (n = 31; n(low, TAU) = 15, n(low, POI) = 16) to the high education subgroup (n = 72; n(high, TAU) = 39, n(high, POI) = 33). In the high education subgroup, the POI significantly reduced the risk for worsening of depressive symptoms (χ2(1) = 7.444, p = .006), while there was no effect in the low education subgroup (χ2(1) = 1.151, p = .283). Again, we analyzed subgroups descriptively. Participants with higher education had lower baseline depression (M = 13.47, SD = 4.07 vs. M = 14.40, SD = 4.54), were younger (M = 36.40, SD = 11.33 vs. M = 40.08, SD = 15.08), and were more often female (78% vs. 68%).

Table 4

Results of exploratory moderator analyses with worsening of depressive symptoms (binary) as outcome (n = 103).

Moderator	Interaction	SE	z	p	LLCI	ULCI
Sex	0.10	1.31	0.07	0.94	−2.47	2.66
Education	−2.51	1.05	−2.38	0.02*	−4.58	−0.44
Willingness to change	0.21	0.10	2.17	0.03*	0.02	0.40
Number of psychotherapies	−0.00	0.25	−0.01	0.99	−0.50	0.49
Undergoing psychotherapy	1.41	0.96	1.47	0.14	−0.47	3.29
Number of self-help books	0.05	0.05	1.05	0.29	−0.04	0.14
Treatment expectation	0.12	0.31	0.39	0.70	−0.49	0.73
APOI total score	−0.04	0.06	−0.60	0.55	−0.16	0.08
APOI skepticism and risks	−0.13	0.14	−0.94	0.35	−0.40	0.14
APOI confidence in effectiveness	0.10	0.22	0.47	0.64	−0.32	0.53
APOI technologization threat	0.12	0.17	0.72	0.47	−0.21	0.45
APOI anonymity benefits	−0.06	0.13	−0.47	0.64	−0.32	0.19
Number of diagnoses	−0.33	0.38	−0.87	0.38	−1.07	0.41
Depression diagnosis	20.37	20,915.98	0.00	1.00	−40,974.2	41,014.94
DAS total score	0.00	0.02	0.11	0.92	−0.05	0.05
DAS performance evaluation	0.00	0.05	0.08	0.93	−0.09	0.10
DAS approval by others	0.03	0.11	0.29	0.77	−0.18	0.25
Concurrent therapy	1.39	1.17	1.18	0.24	−0.91	3.68

Notes. Willingness to change = University of Rhode Island Change Assessment, action subscale; APOI = Attitudes towards Psychological Online Interventions Questionnaire; DAS = Dysfunctional Attitudes Scale; models included baseline PHQ-9 as a covariate, worsening of depressive symptoms (increase in PHQ-9 vs. equal/lower score) as the outcome, group as independent variable, and the moderator as well as the interaction of group and the moderator as additional covariates; LLCI = lower level bootstrap confidence interval, ULCI = upper level.

<0.05.

As mentioned before, all moderator analyses were conducted in the complete cases sample. Post hoc, we repeated the moderator analyses for the significant interactions in the imputed data set to examine if the effects were similar. We modelled interactions using the regression command in SPSS 24® as it provides pooled p-values over all imputed data sets. The effect of concurrent treatments on post treatment depression remained significant (p = .03) in the imputed data set; the effects of education (p = .07) and willingness to change (p = .09) on worsening of depressive symptoms only reached trend significance.

Participants' satisfaction with the POI and application

Table 5 depicts the participants' satisfaction with the POI and the smartphone application.

Table 5

Participants' evaluation of the POI and the smartphone application.

POI
“How do you evaluate the quality?”	“excellent” (3.1%), “good” (65.6%), “not so good” (28.1%), “bad” (3.1%)
“Did you receive the treatment you wanted?”	“definitely yes” (12.5%), “yes, mostly” (50.0%), “not really” (34.4%), “definitely not” (3.1%)
“Did the POI meet your needs?”	“all of my needs” (0%), “most of my needs” (34.4%), “only some of my needs” (43.8%), “did not meet my needs” (21.9%)
“Would you recommend it?”	“definitely yes” (31.3%), “probably yes” (46.9%), “probably not” (15.6%), “definitely not” (6.3%)
“How satisfied are you with the amount of help that it provided?”	“very satisfied” (3.4%), “mostly satisfied” (62.1%), “slightly dissatisfied” (34.5%), “very dissatisfied” (0%)
“Did it help you to cope with your problems in a more appropriate way?”	“yes, it helped a lot” (10.3%), “yes, it helped a bit” (48.3%), “no, not really” (34.5%), “no, it made things worse” (6.9%)
“Overall, how satisfied are you with the POI?”	“very satisfied” (0%), “mostly satisfied” (60.0%), “slightly dissatisfied” (36.7%), “very dissatisfied” (3.3%).
“Would you use the POI again?”	“definitely yes” (10.0%), “probably yes” (53.3%), “probably not” (33.3%), “definitely not” (3.3%)
App
“How do you evaluate the quality?”	“excellent” (5.6%), “good” (50.0%), “not so good” (22.2%), “bad” (22.2%)
“Did you receive the treatment you wanted?”	“definitely yes” (15.8%), “yes, mostly” (21.1%), “not really” (47.4%), “definitely not” (15.8%)
“Did the app meet your needs?”	“all of my needs” (5.6%), “most of my needs” (16.7%), “only some of my needs” (33.3%), “did not meet my needs” (44.4%)
“Would you recommend it?”	“definitely yes” (27.8%), “probably yes” (22.2%), “probably not” (44.4%), “definitely not” (5.6%)
“How satisfied are you with the amount of help that it provided?”	“very satisfied” (5.9%), “mostly satisfied” (41.2%), “slightly dissatisfied” (41.2%), “very dissatisfied” (11.8%)
“Did it help you to cope with your problems in a more appropriate way?”	“yes, it helped a lot” (12.5%), “yes, it helped a bit” (43.8%), “no, not really” (31.3%), “no, it made things worse” (12.5%)
“Overall, how satisfied are you with the app?”	“very satisfied” (5.9%), “mostly satisfied” (29.4%), “slightly dissatisfied” (41.2%), “very dissatisfied” (23.5%).
“Would you use the app again?”	“definitely yes” (11.8%), “probably yes” (35.3%), “probably not” (29.4%), “definitely not” (23.5%)

Discussion

We tested the efficacy of a brief cognitive behavior therapy based POI to reduce depression (PHQ-9, main outcome) and dysfunctional attitudes (DAS-18B, secondary outcome), and to increase behavioral activation (BADS, secondary outcome). The POI was not significantly superior to TAU in any of those outcomes at 4-week post-assessment. A post-hoc test demonstrated no statistical equivalence in depression change scores between POI and TAU conditions. Our results were somewhat unexpected in light of meta-analytic findings that unguided online interventions are superior to control conditions (Karyotaki et al., 2017). Compared to other online interventions; however, our POI was very brief and targeted, which might explain the lack of group differences. Our POI merely included behavioral activation and cognitive restructuring techniques. In contrast, the unguided POIs summarized in Karyotaki et al. (2017) are comprised of, for example, 7 (Clarke et al., 2005), 8 (Spek et al., 2007), or 10 (Meyer et al., 2009) modules/sessions addressing multiple depression-related topics. Perhaps the extent of the therapeutic strategies of an intervention determines its success. Interestingly; however, meta-analyses have been inconclusive regarding whether interventions comprised of many modules are superior to interventions containing fewer modules (Pasarelu et al., 2017; Richards and Richardson, 2012). Another explanation for our null results could be that the POI's quality was poor and that it did not meet the participants' needs. The participants' evaluations reveal disappointment with the quality of the POI. For example, the POI apparently only partly addressed relevant issues as not a single participant stated that it fully met their needs. In its current form, the POI does not adequately cover topics that participants expect from an online intervention for depression. The POI should be thoroughly revised, such as by adding further modules on “worrying”, “mindfulness”, “sleep”, etc., that other POIs include.

Another possible reason for the absence of effects could be our recruitment strategy: The “traditional” meta-analysis by Karyotaki et al. (2017) includes four negative (favoring the control condition) or very small nonsignificant study results (Hedges' g < 0.1). Of these four studies, two were conducted in a primary care setting (de Graaf et al., 2009; Gilbody et al., 2015). In contrast, only two of the remaining 12 significant studies recruited partly from primary care settings (Hedges' g > 0.25 for the remaining 12 studies). Possibly, our recruiting from an outpatient clinic might have resulted in a rather effective TAU control condition. In combination with a weak POI, this strong TAU could have resulted in nonexistent group differences. One might also argue that our second method of recruitment, namely contacting former study participants who were still reporting depressive symptoms, could have affected the results. Although speculative, it is possible that these participants represented a treatment-resistant group.

Lastly, the low adherence to the POI could be a reason for the null results. Almost 20% did not use the POI at all, and the median number of logins was two. Infrequent usage has been discussed as a possible reason for null results in other trials. For example, Clarke et al. (2002), who compared an internet-based cognitive restructuring training for depression to TAU, report that participants did not use their intervention frequently enough. The median number of sessions was two, which is comparable to the median of two logins that we found. Likewise, Gilbody et al. (2015), who evaluated two online interventions, report low uptake. The median number of completed sessions was one and two, respectively, and the authors suspect that this low engagement with the treatment is the main reason for the lack of treatment effects.

Based on our exploratory moderator analyses, future studies should examine the effect of concurrent treatments in POI trials. We found that the non-significant main result could in part stem from different additional help-seeking behavior in the POI and TAU group. At trend, patients in the TAU group sought more concurrent treatment (antidepressant medication, psychotherapy) than the POI group, and in subgroup analyses our POI was superior to TAU if concurrent treatments did not increase (i.e., untreated waitlist). We speculate that control group allocation resulted in TAU participants searching more actively for alternative treatment options. Post hoc, we examined whether this increased help-seeking behavior in the TAU group was associated to higher willingness to change at baseline but this was not the case (t (52) = 0.567, p = .573).

Regarding the worsening of depressive symptoms, our data suggest that when willingness to change was high, the risk was the same in both groups. That is, if participants were eager to act upon their problems, they were not very likely to worsen regardless of group allocation. If willingness to change was low; however, the risk of increased symptoms was higher in the TAU group compared to POI. Therefore, when it comes to preventing the worsening of depression, participants who suffer from avolition/poor drive might benefit from easily accessible POIs. Furthermore, we found that for highly educated participants, the POI lowered the probability of worsening of depressive symptoms compared to TAU. However, when education was low, this effect was reversed, albeit not significant. Low education has been shown to hinder progress in traditional psychotherapy (Melchior et al., 2016) and may be a factor which limits the effectiveness of any type of therapy. Our exploratory findings should be interpreted with caution but it seems plausible that low education may diminish the effectiveness of POIs specifically, as they are mostly text-based programs, which require good reading comprehension, conscientiousness, and the ability to learn autodidactically.

Limitations

First, we did not reach the targeted sample size. Second, we changed the allocation strategy from randomization to allocation based on depression severity during the recruiting period to prevent statistical problems because of large baseline differences in depression scores. Post-hoc analyses revealed that this change of allocation did not change the direction and magnitude of main effects. Third, a prevalent problem of studies examining moderator effects is insufficient power (Donker et al., 2013; Warmerdam et al., 2013). This study made no exception in this regard. As mentioned before, the results from moderator analyses are exploratory and not corrected for multiple comparisons so that they can only be used to generate hypotheses for future studies. In particular, this limitation affects subgroup analyses of the worsening of symptoms as subgroups were small. Another important limitation is that we used complete cases in moderator analyses in order to benefit from the advantages that the PROCESS macro offers. When we repeated the analyses in the imputed dataset, we found that the effect of concurrent therapies remained significant. However, the effects of education and willingness to change on worsening of depressive symptoms no longer reached significance in the imputed dataset. Fourth, our POI had not been piloted prior to this study. Piloting the POI might have led to the use of the POI at least once by all participants. The large range of the number of clicks reflects the variance in acceptance of the POI by the participants.

Conclusions

Our results indicate that our brief POI is not able to reduce depression; presumably, because it does not cover depression-related topics other than behavioral activation and cognitive restructuring. Exploratory analyses show that such a brief POI could, however, be effective when concurrent treatment stays constant. This is particularly interesting because, at a trend level, those allocated to TAU sought alternative treatments more actively. Another interesting exploratory finding is that worsening of depressive symptoms was reduced by the POI only if the participant's education was high. Additionally, for participants with low willingness to change, our POI reduced rates of worsened symptoms, while there was no effect in participants with high willingness to change. It is important to emphasize that these findings are exploratory in nature and future studies should further investigate those moderators in a confirmatory fashion.

Disclosure statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.