Outpatient and self-referred participants: Adherence to treatment components and outcome in an internet intervention targeting anxiety disorders.

OBJECTIVE: While adherence is an important factor influencing the effectiveness of internet interventions, many studies operationalize adherence only by the number of sessions and do not report adherence to specific treatment components. The goal of this study was to investigate adherence to treatment components as well as outcome in outpatients and self-referred participants who participated in an internet intervention targeting anxiety.
METHOD: Outpatients (N = 50) were compared to self-referred (N = 37) participants and a matched outpatient waitlist sample (based on nearest neighbor matching): Using t-test and χ2 tests adherence to treatment components based on the number of completed exercises was compared between participant groups. A 2 × 2 repeated measures ANOVA was used to compare pre-to post symptom change between participant groups. Primary measures included the Generalized Anxiety Disorder Scale-7 (GAD-7) and the Mini Social Phobia Inventory (Mini-SPIN). Using nonparametric bootstrap analyses number of sessions and adherence to treatment components were investigated as potential mediators of the relationship between participant group and outcome. Finally, predictors of adherence to treatment components in outpatient participants were investigated using LASSO and logistic regression.
RESULTS: Self-referred participants were more adherent than outpatient participants, however the groups did not differ significantly in outcome. Outpatient participants who adhered to relaxation showed greater improvement during the waiting period than the matched outpatient waitlist sample. The effect of participant group on outcome was mediated via adherence to exposure and number of sessions.
CONCLUSIONS: In internet interventions adherence to treatment components differs between participant groups and has a mediating effect on treatment outcome. Therefore, it should be fostered, especially when participants are not self-referred. In line with these findings more studies should investigate relevant participant characteristics in more depth.

Introduction

Internet interventions for anxiety disorders have been found to be effective in numerous studies (see Andrews et al., 2018). Yet not all participants of internet interventions receive the same dosage of treatment as adherence rates vary highly across studies (Beatty and Binnion, 2016). These differences in adherence rates are crucial as higher adherence to internet interventions has been found to be associated with higher outcome (Couper et al., 2010). Thus it remains a priority to investigate adherence in internet interventions (Hilvert-Bruce et al., 2012). To date, the identification of consistent predictors of adherence to internet interventions has shown to be difficult (e.g., Castro et al., 2018; El Alaoui et al., 2015; Lutz et al., 2017). Results from a study by Alfonsson et al. (2016) used varying measures of adherence and found that different predictors of adherence emerged. The authors conclude that it is necessary to carefully define treatment adherence in psychotherapy research.

However, many studies that have investigated adherence have only used general measures of adherence such as the number of times the website was accessed or the number of sessions completed (e.g., Castro et al., 2018; Couper et al., 2010). This follows research practices in face-to-face settings where adherence is often defined via session attendance with low adherence operationalized via premature treatment termination (Koffel et al., 2018). However, these general measures may not be the most relevant indicators of adherence, nor do they necessarily imply that the desired outcomes will be achieved (Sieverink et al., 2017). Instead of this broad definition of adherence, adherence to treatment components should be more closely investigated, as these components are thought to be responsible for treatment change (Domhardt et al., 2019; Wampold, 2015). In treatments targeting anxiety disorders, exposure, relaxation, and cognitive restructuring can be considered essential (Borza, 2017) as these components are assumed to break the cycle of physical arousal, dysfunctional, catastrophizing thoughts and avoidance. Several factors may impact adherence to these treatment components. In face-to-face treatments, higher rates of premature treatment termination have been reported in treatments targeting PTSD and anxiety symptoms using exposure (Cooper et al., 2018). This may indicate that certain treatment components, such as exposure may be perceived by patients as more difficult than others. If engagement of patients with such potentially challenging treatment components is not monitored or patients are not supported adequately, this may lead to varying levels of adherence and thus also to varying levels of treatment outcome. In internet interventions the risk that participants do not find adequate support or clarification when facing difficulties may be higher than in face-to-face settings. This could lead to more participants showing low engagement with challenging treatment components.

Interestingly, studies focusing on insomnia and pain also report low adherence (see Koffel et al., 2018; Matsuzawa et al., 2019). Thus, in addition to perceived difficulty, patients' perception of their problems as well as their perceived probability of treatment success may also represent crucial factors regarding adherence. While patients with anxiety disorders may be more prone than patients suffering from pain to perceive their problems internally, their believe that they may be able to profit from a specific treatment and specific treatment components may vary, e.g. depending on their level of impairment as well as their internal and external resources.

In the context of internet interventions, considering these patient factors is especially relevant, because some studies have focused on internet interventions available to the broader public (self-referred participants), while fewer studies have investigated adherence in internet interventions integrated into a routine care setting (Andersson and Hedman, 2013; El Alaoui et al., 2015; Kenter et al., 2013). While one study found promising results concerning adherence in primary care (Berger et al., 2017), it has been reported that adherence in routine care is less than half of that in research trials (Hilvert-Bruce et al., 2012). Therefore, it remains necessary to investigate if and how self-referred participants and patients in routine care differ regarding adherence and treatment outcome. This could have important implications for the implementation of internet interventions indicating for which kind of participants they may work best and how they need to be optimized to increase adherence to crucial treatment components. However a possible barrier of interpreting findings regarding adherence to treatment components is that it remains difficult to estimate a justified threshold of usage that is likely to lead to desired outcomes (Sieverink et al., 2017).

In summary the goal of this study was to investigate adherence to treatment components as well as outcome in outpatients and self-referred participants who participated in an internet intervention targeting anxiety. As we assumed self-referred participants to be more motivated to participate we expected adherence and outcome to be higher in self-referred participants. By including secondary measures such as the Patient Health Questionnaire-9 (PHQ-9) and the Hopkins Symptom Checklist-11 (HSCL-11), we attempted to consider differences between groups regarding depressive symptoms and overall impairment. In addition, we compared outpatient participants to a matched sample of outpatients without access to the internet intervention with regard to change during the waiting period. We assumed that outpatients who participated in the intervention would improve more during the waiting period than outpatients without access to the intervention.

Furthermore, we investigated adherence to treatment components as mediators of outcome. Specifically we assumed that participants would improve more if they adhered more to the crucial treatments components of the intervention and completed more sessions. Finally, we investigated patient variables such as demographic variables, treatment expectations and self-efficacy (see Matsuzawa et al., 2019) as potential predictors of adherence to treatment components in outpatient participants. Based on the findings from Alfonsson et al. (2016) we expected to identify slightly different predictors depending on the measure of adherence used (adherence to exposure, to relaxation or to cognitive restructuring).

Methods

Flow of participants

In this study, an internet intervention was offered to two groups: One consisted of outpatients that had registered for a face-to-face therapy in an outpatient clinic and were offered the internet intervention during the waiting period. The second group consisted of interested participants who were recruited by means of advertisements in regional newspapers and the university press. All participants were screened for suicidality via three items: “I have thoughts of ending my life”, “During the past seven days, how much were you distressed by thoughts of ending your life”, and “In the last week I had thoughts of ending my life”. Participants were excluded if they endorsed “at times” on one of the items or “seldom” on two of the three items. Furthermore, highly depressive symptoms as indicated by a Patient Health Questionnaire-9 (PHQ-9) score of over 21 were an exclusion criterion. As the internet intervention targeted anxiety disorders, only participants who obtained a Generalized Anxiety Disorder Scale-7 (GAD-7) score of 5 or higher were offered the intervention. Outpatients filled out the PHQ-9 and GAD-7 at registration at the clinic, while self-referred participants filled out a screening questionnaire when they registered for the study. All participants that fulfilled the initial inclusion criteria were screened using the Mini International Neuropsychiatric Interview (M.I.N.I, Sheehan et al., 1998). The interviews were conducted by two trained master-level students and seven psychologists in post-graduate clinical training.

In total, 1128 outpatients who registered in the outpatient clinic indicated being interested in taking part in an intervention during the waiting period (see Fig. 1). The routinely applied registration questionnaires were used to screen for inclusion and exclusion criteria. After screening for high levels of anxiety (here indicated by a GAD-7 score over 5), excluding outpatients who showed risk of suicidality, and high depressive symptoms (here indicated by a PHQ-9 score over 21) 537 outpatients were contacted and offered information on the study and the intervention. For 238 outpatients who gave informed consent, an interview appointment was scheduled. The inclusion criteria comprised a diagnosis of panic disorder, social phobia, or generalized anxiety disorder as well as an age of between 18 and 65 years. In addition, outpatients were excluded from the study if acute suicidality or a diagnosis of bipolar disorder or psychosis was reported. After screening for inclusion and exclusion criteria, 86 outpatients with one of the anxiety disorder diagnoses mentioned above were offered the internet intervention.

Fig. 1

Flowchart of outpatient participants and self-referred participants.

In total, 104 self-referred participants gave informed consent and filled out the screening questionnaire to participate in the study. After screening level of anxiety (GAD-7 score over 5), suicidality and high level of depressive symptoms (PHQ-9 score over 21), 85 participants were contacted and an appointment for a diagnostic interview was scheduled. Again only participants who fulfilled the criteria of one of the anxiety disorders mentioned above were included. If participants fulfilled reported acute suicidality or fulfilled the criteria of bipolar disorder or psychosis, they were excluded from the study. After screening for inclusion and exclusion criteria, 48 outpatients were allocated to the intervention.

After inclusion, participants filled out a pretreatment questionnaire and were then able to access the internet intervention.

Outpatient waitlist sample

151 outpatients did not have access to the internet intervention during the waiting period, fulfilled the described study criteria (PHQ-9 not over 21, GAD-7 over 5), and filled out pre-face-to-face treatment questionnaires. One important goal was to estimate the degree to which outpatient participants benefited more during the waiting time than nonparticipating outpatients. To rule out potential differences between participating and nonparticipating outpatients, a matching procedure was used to identify nonparticipating outpatients, who were the most similar to participating outpatients on relevant pretreatment variables. The waitlist sample of outpatients fulfilling inclusion criteria was used as a basis for the matching procedure. By using LASSO regression relevant variables for matching were identified. To evaluate the success of the matching procedure the standardized mean difference was used to examine the balance of covariate distribution between the matched groups.

Intervention

The intervention consisted of eight modules and was primarily based on a cognitive-behavioral approach developed for social anxiety disorder, panic disorder, and generalized anxiety disorder (see also Berger et al., 2014). The specific content was tailored with regard to the anxiety disorder (social phobia, general anxiety disorder, or panic disorder) that was diagnosed with the MINI. The following treatment elements were addressed in the modules with relevant exercise protocols introduced during the sessions: (1) introduction to the program and motivational enhancement, (2) psychoeducation and relaxation, (3) cognitive restructuring, (4) self-focused attention and detached mindfulness, (5) exposure and behavioral experiments, (6) summary and repetition, (7) lifestyle modification and problem solving, and (8) repetition and relapse prevention (Berger et al., 2014). In the first module, participants were introduced to the program and could set individual treatment goals. In the second module, participants were given information on the etiology of anxiety disorders and relaxation was introduced. Participants were asked to practice relaxation and keep protocol record of when they had practiced. In the following module, participants were informed on the role of dysfunctional thoughts in anxiety disorders. They were instructed to keep a record of their anxiety provoking thoughts and to question them in order to achieve a more realistic view. In module four, participants were instructed to do an exercise to demonstrate the effects of self-focused attention in comparison to mindfulness. No protocol was used here. Finally, in module five, participants were instructed to practice exposure and to keep track of their progress regarding exposure in a final protocol. The modules were accessed sequentially and after a module was completed the introduced protocols could be used throughout the rest of the treatment. The participants were instructed to work with the program for six weeks, with a workload of 1–2 modules per week. If after six weeks the participants wished to continue to use the intervention, they were provided with access to the program for up to another six weeks.

Secure Sockets Layer encryption was used to secure all internet-based communication and participants were identified using anonymous login names and passwords. The study was conducted in compliance with the Declaration of Helsinki and was approved by the local Ethics Committee of the University of Trier.

Participants were informed that they could contact the study coordinator via e-mail or phone. Once a week, therapists wrote a message with half-standardized supportive feedback to the participants. Three master-level psychology students provided weekly feedback. They received brief training with examples of feedback and were supervised by the first author, a psychologist in post-graduate clinical training. Within the feedback, participants were recognized for making important steps by working with the exercises and motivated to continue treatment. If the participant showed no activity during the past week, participants received a reminder to continue treatment.

Assessments

All participants filled out a questionnaire at registration: Participants that registered at the outpatient clinic received a standardized battery of clinical questionnaires, of which the GAD-7, PHQ-9, and three suicidality items were used for screening. Participants that registered directly for the internet intervention filled out a screening questionnaire (consisting of the PHQ-9, GAD-7, and sociodemographic variables) that was linked to the website containing information on the study and intervention. Following inclusion based on the screening questionnaires and diagnostic interview, all participants filled out a pre-treatment questionnaire consisting of the PHQ-9, the Hopkins Symptom Checklist-11 (HSCL-11), the GAD-7 and the Mini Social Phobia Inventory (Mini-SPIN). Subsequently, participants were asked to fill out in-treatment questionnaires each week during the internet intervention as well as one post-treatment questionnaire after the internet intervention. If they did not fill out the in-treatment or post-treatment questionnaires, they were reminded to do so up to three times.

18 participants did not fill out the pre-treatment questionnaire: NOutpatient participants (NO) = 14, NSelf-referred participants (NSR) = 4. A similar number of participants did not log in on the website (NO = 15, NSR = 3). Eleven participants did not fill out any questionnaires during or after the intervention. Of the remaining 87 participants (NO = 50, NSR = 37), 58 participants filled out a post-treatment questionnaire.

Diagnostic instruments

The M.I.N.I is a short structured diagnostic interview (Sheehan et al., 1998). It is based on the Diagnostic and Statistical Manual of Mental Disorders, 4th Edition (DSM-IV) and the International Classification of Diseases, Tenth Edition (ICD-10) and has showed good interrater reliability (Rossi et al., 2004).

Measures

Treatment expectations were assessed before the start of the internet intervention. All symptom measures (GAD-7, Mini-SPIN, HSCL-11, PHQ-9) were assessed before, during, and after the internet intervention. As described above, outpatient participants completed a routine battery of standardized questionnaires at registration at the outpatient clinic. For this study, a subset of these questionnaires was used: the PHQ-9 and the GAD-7, the Brief Symptom Inventory (BSI), the Questionnaire for the Evaluation of Psychotherapeutic Progress (FEP-2), the General Self-Efficacy Scale (GSE), and the Incongruence Questionnaire – short version (INC-S). Details on the four further questionnaires are provided below (see Routine measures at the outpatient clinic section).

Measures assessed for all participants in the internet intervention

Generalized Anxiety Disorder Scale (GAD-7)

The GAD-7 is an anxiety questionnaire (Löwe et al., 2008) that can be used to screen for generalized anxiety disorder, but can also be used to detect panic disorder or social anxiety disorder. It consists of seven items that reflect the seven core symptoms of generalized anxiety disorder and is rated on a scale from 0 to 3 (“not at all” to “nearly every day”). The total score ranges from 0 to 21 with scores higher than 5 representing mild levels of anxiety (scores >10 represent moderate, and scores >15 represent severe levels). Good internal consistency has been reported (Cronbach's α = 0.89, Löwe et al., 2008).

Hopkins Symptom Checklist-11 (HSCL-11)

The HSCL-11 (Lutz et al., 2006) is a modified 11-item version of the Symptom Checklist-90-R (Derogatis, 1994). Questions are answered on a four-point Likert scale ranging from “not at all” to “extremely”. The questions focus primarily on depressive and anxious symptoms. The HSCL-11 has been found to have adequate psychometric properties (e.g., Cronbach's α = 0.85; Lutz et al., 2006).

Mini Social Phobia Inventory (Mini-SPIN)

The Mini-SPIN is the short version of the Social Phobia Inventory (Connor et al., 2000), which measures fear, avoidance, and physiological symptoms. In contrast to the SPIN, which contains 17 items, the Mini-SPIN consists of three items assessing avoidance and fear of embarrassment experienced in the past week. Answers are provided on a five-point Likert scale (0 – “not at all”, 4 – “extremely”). Good internal consistency and good convergent and discriminant validity have been reported (Wiltink et al., 2017).

Patient Health Questionnaire-9 (PHQ-9)

The PHQ-9 (Kroenke et al., 2001) measures depressive symptoms based on the criteria of depression according to the DSM-IV, with higher scores indicating more severe depressive symptoms. Answers are provided on a four-point Likert scale (0 – “not at all” and 3 – “nearly every day”). The test re-test reliability has shown to be good (r = 0.84; (Kroenke et al., 2001).

Treatment expectations

Participants were able to indicate their expectations regarding treatment on three items: how important it was for them to use the internet intervention (1 – “my life is depending on it”, 5 – “it is not important at all”), how convinced they were that the interventions could help them (1 – “not convinced at all”, 4 – “very convinced”), and how much they believed they could cope in their daily life after the internet intervention (1 – “very poorly, I will not be able to cope at all”, 6 – “very well, as I wish”).

Adherence measures

The number of logins and the number of completed sessions were documented. Adherence to the treatment components (exposure, relaxation, and cognitive restructuring) was based on the number of reports in the exposure, relaxation, and ‘realistic thought’ protocol, respectively. The protocols used a combination of Likert scales and free text space to take notes according to instructions. Likert scales were used to report intensity of relaxation (relaxation protocol), intensity of fear and degree of conviction regarding specific thoughts (cognitive restructuring) and intensity of expected and observed fear (exposure). Free text space was used to report situations, thoughts, and behaviors. An exercise reported in the protocol was included if each space of the protocol was completed. We checked for nonsense entries, but no threshold regarding number of words was set.

Routine measures at the outpatient clinic

Brief Symptom Inventory (BSI)

The BSI measures self-reported psychological symptoms and was developed based on the SCL-90-R (Franke, 2000). Analogue to the SCL-90, the BSI consists of nine scales (somatization, obsessive-compulsive, interpersonal sensitivity, depression, anxiety, anger-hostility, phobic anxiety, paranoid ideation, and psychoticism). The 53 items are answered on a five-point Likert scale that indicates how strong the impact of symptoms was (1 – “not at all”, 5 – “very strong”). For the primary symptom dimensions of the BSI, internal consistencies range between 0.70 and 0.89 and their correlations with the comparable dimensions of the SCL-90-R are quite high (Geisheim et al., 2002).

Questionnaire for the Evaluation of Psychotherapeutic Progress (FEP-2)

The FEP-2 consists of 40 items and four scales: well-being, symptoms, interpersonal relationships, and incongruence with respect to approach and avoidance goals. It is used to measure therapeutic progress and has been shown to be reliable and change sensitive (Lutz et al., 2009).

Incongruence Questionnaire – short version (INC-S)

The INC-S assesses the degree of satisfaction with approach and avoidance goals that are particularly relevant for psychotherapy (Grosse Holtforth and Grawe, 2003). It consists of 23 items on two subscales: the approach motivational goals (14 items; e.g., “recently, I've been independent”) and the avoidance motivational goals (9 items; e.g., “recently, I've been criticized”). Answers are provided on a five-point Likert scale (1 – “not enough” to 5 – “entirely sufficient”). A high score for the sum of motivational goals means that both approach and avoidance motivational goals cannot be met. Cronbach's alpha ranges between 0.65 and 0.86 for the approach and avoidance scales and the sum of motivational goals (Grosse Holtforth and Grawe, 2003).

General Self-Efficacy Scale (GSE)

The GSE consists of 10 items that measure the broad and stable sense of personal competence to deal effectively with a variety of stressful situations (Schwarzer, 1999). The response format is a four-point Likert scale (1 – “not at all true”, 4 – “exactly true”). The GSE scale has been used in numerous studies, where it typically yielded internal consistencies between α = 0.75 and 0.91 (Scholz et al., 2002).

Data analytic strategy

In a first step, we investigated how much the participants adhered to the internet intervention and compared adherence between self-referred participants and outpatients using χ2-tests and t-tests. Then we compared outcome across participant groups. For participants for whom post scores were missing (N = 29) the missing value were imputed using the missForest function in R. A 2 × 2 repeated measures analysis of variance (ANOVA) was used to estimate outcome considering time (pre to post) and group effects (self-referred participants vs. outpatient participants). Additionally within-group effect sizes were calculated for each measure by subtracting the symptom score at post-treatment from the symptom score at pre-treatment and dividing the result by the SD of the pre-scores. Next we compared outpatient participants to outpatients who did not have access to the internet intervention. To control for sample differences, 151 outpatients were selected who did not have access to the internet intervention and fulfilled the inclusion GAD-7 (>5), but not the exclusion criteria PHQ-9 (>21), that were applied to outpatient participants. Then a tenfold cross-validated LASSO (least absolute shrinkage and selection operator) was used (Tibshirani, 2011) to identify and select the most important predictors of study participation during the waiting period. The VarImp function was used to rank predictors according to their importance. Based on the ten most important predictors, a sample of outpatients (outpatient control group) was matched to the sample of outpatient participants. Within the software R, the caret package and the Matchit package were used to implement LASSO and the matching procedure, respectively.

We used nonparametric bootstrapping analyses to test whether adherence to treatment components mediated the effect of participant group (self-referred vs. outpatient) on anxiety symptoms as measured by the GAD-7. Specifically, in the model it was assumed that higher adherence to each treatment component would result in a higher number of sessions, and both would result in a better treatment outcome. To control for the effect of initial impairment, the residualized post score was used as the dependent variable. When participant groups showed significant differences on other initial impairment measures, those measures were entered as covariates. Additionally, treatment expectations were entered as a covariate. Analyses were performed using the PROCESS function V.3.4 in SPSS V.25, we applied model 80 (model as a parameter in the PROCESS function, see Hayes, 2017).

Finally, as findings regarding predictors of adherence have remained largely inconsistent, we again used a tenfold cross-validated LASSO to identify predictors of adherence to active treatment components. The VarImp function was used to rank predictors according to their importance.

Self-referred and outpatient participants

Before the internet intervention began (pre), participants were highly impaired on all measures (see Table 1). On average, they exceeded the GAD-7 score of 15, which is considered to indicate very high anxiety-related impairment (Löwe et al., 2008). On the Mini-SPIN they exceeded the score of 6, making a diagnosis of social phobia probable (Wiltink et al., 2017). PHQ-9 scores were above 15, indicating high impairment in depressive symptoms (Kroenke et al., 2001). General impairment on the HSCL-11 was also high. Outpatient participants showed significantly higher scores on the PHQ-9 (p = .017) and Mini-SPIN (p = .019) compared to self-referred participants. There were no significant differences between self-referred and outpatient participants concerning the frequency of M.I.N.I diagnoses of agoraphobia, panic disorder, social phobia, and generalized anxiety disorder (see Table 1). 56% of participants were female with no significant difference between self-referred and outpatient-participants. Nearly 64% of participants had a university entrance diploma with a significant difference between groups: More self-referred participants (p = .006) had a university entrance diploma. On average, participants were approximately 36 years old (SD = 12.70) with no significant difference between groups. Treatment expectations ranged between 2.22 and 4.33 (M = 2.93, SD = 0.48), with higher scores indicating more positive treatment expectations. Self-referred and outpatient participants did not differ significantly regarding treatment expectations.

Table 1

Baseline variables of participants by group.

Baseline variables	Overall (N = 87)	Outpatient participants (NO = 50)	Self-referred participants (NSR = 37)	Test statistic (p)
M.I.N.I. diagnosis agoraphobiaFrequency N (%)	30 (34.5)	20 (40)	10 (27)	χ2 (1) = 1.58p = .208
M.I.N.I. diagnosis social phobiaFrequency N (%)	43 (49)	25 (50)	18 (48.6)	χ2 (1) = 0.02p = .901
M.I.N.I. diagnosis GADaFrequency N (%)	35 (40.2)	24 (48)	11 (29.7)	χ2 (1) = 2.95p = .086
M.I.N.I. diagnosis panic disorderFrequency N (%)	30 (34)	16 (32)	14 (37.8)	χ2 (1) = 0.59p = .443
SexFemale N (%)	56 (64.4)	32 (64)	24 (64.9)	χ2(1) = 0.01p = .934
Level of educationUniversity entrance diploma N (%)	55 (63.5)	26 (52)	29 (78.4)	χ2(1) = 7.63p = .006**
Pre- mean score GAD-7bM (SD)	2.79 (0.59)	2.81 (0.55)	2.75 (0.65)	t(85) = 0.45p = .652
Pre- mean score Mini-SPINcM (SD)	3.04 (1.09)	3.28 (1.08)	2.72 (1.04)	t(85) = 2.43p = .017*
Pre- mean score HSCL-11dM (SD)	2.27 (0.52)	2.32 (0.53)	2.19 (0.49)	t(85) = 1.14p = .257
Pre- mean score PHQ-9eM (SD)	2.30 (0.59)	2.43 (0.58)	2.13 (0.58)	t(85) = 2.39p = .019*
Treatment expectationsM (SD)	2.94 (0.48)	2.91 (0.47)	2.97 (0.49)	t(85) = −0.63p = .532
AgeM (SD)	35.91 (12.70)	34.30 (11.5)	38.08 (14.1)	t(85) = 1.38p = .171

GAD: Generalized Anxiety Disorder.

GAD-7: Generalized Anxiety Disorder Screener-7.

Mini-SPIN: Mini Social Phobia Inventory.

HSCL-11: Hopkins Symptom Checklist-11.

PHQ-9: Patient Health Questionnaire-9.

Matching procedure and results

A sample of 151 outpatients was used to identify outpatient characteristics that predicted study participation using LASSO: The matching procedure was then based on the ten most important predictors of study participation (FEP-2, PHQ-9, HSCL-11, incongruence, the BSI subscales phobic anxiety and anxiety, age, level of education, using medication as well as self-efficacy; GSE). Via nearest neighbor (NN) matching, an outpatient waitlist sample was selected (N = 40) that was the most similar to the outpatient participants who had accessed the internet intervention during the waiting period and had completed the pre-face-to-face treatment questionnaires (N = 40).

After the application of NN matching, nearly all baseline variables under consideration were sufficiently well balanced: standardized mean difference scores (smd) ranged from 0.006 for initial impairment on the INC-S to 0.162 for using medication with a higher standardized mean difference score for sex only (smd = 0.27). After NN matching, there was no significant difference in waiting period (p = .830) and the groups did not differ on any of the baseline variables (all p > .05) indicating similar levels of initial impairment and similar characteristics regarding demographic variables.

Results

Adherence of self-referred and outpatient participants

On average, participants completed five of the eight sessions (M = 5.01, SD = 2.72, see Table 2). Adherence to the treatment components (exposure, relaxation and cognitive restructuring) varied highly across participants. Participants showed relatively high average adherence to relaxation (M = 6.44, SD = 10.39). Compared to adherence to relaxation adherence to cognitive restructuring (M = 3.43, SD = 5.44) and exposure (M1.5, SD = 3.5 was rather low. Overall less than half of the participants completed any exercises at all (use of cognitive restructuring: yes (N = 41), use of exposure: yes (N = 27).

Table 2

Adherence to intervention by participant group.

Variable	Participant group	M (SD)/N	Test statistic	p
Number of sessions	Outpatient (NO = 50)	4.48 (2.73)	t(85) = −2.56	.012*
	Self-referred (NSR = 37)	5.95 (5.95)
	All (N = 87)	5.10 (2.72)
Adherence to relaxation (N of exercises)	Outpatient (NO = 50)	4.38 (7.82)	t(85) = −1.96	.054
	Self-referred (NSR = 37)	8.24 (9.89)
	All (N = 87)	6.02 (8.01)
Adherence to exposure (yes)	Outpatient (NO = 50)	11 (22.0%)	χ2(1) = 4.48	.034*
	Self-referred (NSR = 37)	16 (59.3%)
	All (N = 87)	27
Adherence to cognitive restructuring (yes)	Outpatient (NO = 50)	16(32%)	χ2(1) = 10.80	.001*
	Self-referred (NSR = 37)	25(50%)
	All (N = 87)	41(42%)

*p < 0.05.

Self-referred participants completed significantly more sessions than outpatient participants (t(85) = −2.56, p = .012). Before comparing adherence to relaxation between self-referred and outpatient participants, the presence of outliers was checked and winsorizing was used to reduce potential bias: Four extreme values on the number of reported relaxation exercises were replaced with the nearest value that was not an outlier (Field, 2013). On average, self-referred participants did more relaxation exercises, however this difference was not significant (t(85) = −1.96, p = .054). This result did not change whether or not winsorizing was used.

Also, adherence to cognitive restructuring was higher in self-referred participants, both on average (t(46.35) = −3.79, p < .001) and regarding the frequency of reporting use of cognitive restructuring (NO = 16 (32%); NSR = 25 (50%)).

As only 11 outpatient participants (22%) and 16 self-referred participants (43%) indicated any exposure in vivo at all, the frequency of reporting the use of exposure (yes/no exposure) was compared between groups. Self-referred participants were significantly more adherent than outpatient participants with regard to exposure in vivo (χ2(1) = 4.48, p = .034).

Outcome of self-referred and outpatient participants

Time had a significant effect on anxiety symptoms as measured by the GAD-7 (F(85) = 27.07, p < .001) and the SPIN (F(85) = 24.95, p < .001), as well as on depressive symptoms (F(85) = 13.32, p < .001) and overall impairment as measured by the HSCL (F(85) = 12.80, p < .001). Symptom scores were significantly lower post-treatment compared to pre-treatment. There was a significant interaction between time and group effects on HSCL-11 scores (F(85) = 4.04, p = .048) with self-referred participants showing more change in impairment scores than outpatient participants. Overall within-group effect sizes ranged from small to medium with smaller effect sizes for outpatient participants on average (see Table 3).

Table 3

Outcome measures at pre and post and effect sizes by participant group.

Measure	Group	Begin waiting time M (SD)	Pre treatment M (SD)	Post treatment M (SD)	End waiting time M (SD)	Effect-size waiting time Cohens d	Effect-size pre to post Cohens d
GAD-7a	Outpatient waitlist (N = 40)b	1.73 (0.82)			1.60 (0.87)	0.16
	Outpatient participants (NO = 50)	1.78 (0.52)	2.81 (0.55)	2.54 (0.72)	1.65 (0.55)	0.25	0.49
	Self-referred participants (NSR = 37)		2.75 (0.65)	2.32 (0.75)			0.66
	All participants (N = 87)		2.79 (0.59)	2.44 (0.73)			0.59
HSCL-11c	Outpatient waitlist (N = 40)	2.02 (1.06)			1.85 (1.08)	0.16
	Outpatient participants (NO = 50)	1.91 (0.82)	2.32 (0.53)	2.24 (0.59)	1.71 (0.78)	0.24	0.15
	Self-referred participants (NSR = 37)		2.19 (0.49)	1.90 (0.51)			0.59
	All participants (N = 87)		2.27 (0.52)	2.10 (0.58)			0.33
PHQ-9d	Outpatient waitlist (N = 40)	1.65 (0.64)			1.42 (0.68)	0.36
	Outpatient participants (NO = 50)	1.36 (0.53)	2.43 (0.58)	2.21 (0.58)	1.37 (0.56)	−0.01	0.38
	Self-referred participants (NSR = 37)		2.13 (0.58)	1.91 (0.51)			0.38
	All participants (N = 87)		2.30 (0.59)	2.09 (0.57)			0.36
Mini-SPINe	Outpatient participants (NO = 50)		3.28 (1.08)	2.83 (0.85)			0.41
	Self-referred participants (NSR = 37)		2.72 (1.04)	2.19 (0.64)			0.51
	All participants (N = 87)		3.04 (1.09)	2.56 (0.83)			0.44

GAD-7: Generalized Anxiety Disorder Screener-7.

40 outpatient participants stayed registered for face-to-face therapy after receiving internet treatment.

HSCL-11: Hopkins Symptom Checklist-11.

PHQ-9: Patient Health Questionnaire-9.

Mini-SPIN: Mini Social Phobia Inventory, only available for participants of the internet intervention.

In addition to this analysis, we compared the 40 outpatient participants who had access to the internet intervention and filled out the pre-face-to-face treatment questionnaire to 40 eligible outpatients who did not have access to the internet intervention and filled out the pre-face-to-face treatment questionnaire (outpatient waitlist sample). After matching, we used 2 × 2 repeated measures ANOVA to estimate time and group (outpatient participant vs. waitlist outpatients) effects, while controlling for initial impairment on the available measures.

No significant interaction effects, effects of time or group showed on the GAD-7, the Mini-SPIN, the PHQ-9 or the HSCL-11 (see Table 3). A significant effect showed only for anxiety symptoms as measured by the anxiety subscale of the BSI with a significant main effect of group (F(74, 1) = 5.35, p = .024) as well as a significant interaction effect of time and group (F(74,1) =5.96, p = .017). Outpatients who did not participate showed a higher impairment regarding anxiety symptoms and improved less than outpatient participants.

As adherence to treatment components was low in outpatient participants, in a second step, we compared those outpatient participants who had shown some adherence to relaxation (at least two relaxation exercises (N = 22) to the outpatient waitlist sample. Outcome differed significantly between these two groups as indicated by a significant interaction of time and group on the anxiety subscale of the BSI (F(56,1) =8.63, p = .005), on the BSI subscales phobic anxiety (F(56,1) = 4.79, p = .033) and on the HSCL-11 (F(57,1) =5.19, p = .027). On these measures outpatient participants who showed adherence to relaxation showed more improvement than outpatients from the outpatient waitlist sample. No significant effects showed on the PHQ-9 (p = .289) or on the GAD-7 (p = .366).

Mediation model predicting change in anxiety symptoms

Results based on 5000 bootstrapped samples indicated that the direct effect of participant group on outcome on the GAD-7 was not significant (βdirect = −0.078, SE =0.02, p = .725). However, there was a significant indirect effect (βindirect = −0.03, SE = 0.016, CI: LL = 0.040 to UL = 0.066) indicating mediation (see Fig. 2 for a path diagram). Specifically, there was a significant indirect effect of treatment group on outcome via adherence to exposure (number of reported exercises) and number of sessions (IEexposure and number of sessions = 0.056, 95%, CI: LL = 0.0006 to UL = 0.0136). This result indicates that self-referred participants were more likely to report a higher number of exposure exercises, which was linked to a higher number of sessions and associated with better outcome. None of the other mediators significantly contributed to the indirect overall effect, including number of sessions alone (IEnumber of sessions = 0.0087, CI: LL = -0.0079 to UL = 0.0276).

Fig. 2

Mediation model with parallel mediators (n = 87). Indirect effects of participant group through adherence to relaxation, cognitive restructuring, exposure and number of sessions. The direct path from participant group to outcome depicts the direct effect.

Participant group predicted adherence to cognitive restructuring (B = 0.77, p < .001) and adherence to exposure (B = 0.22, p = .026) with self-referred participants reporting a higher number of completed exercises for both treatment components. Initial impairment on the MINI-Spin and on the PHQ-9 was not a significant predictor of any of the adherence measures.

Overall, treatment outcome as measured by the GAD-7 was significantly predicted by initial impairment regarding social anxiety (B = −0.14, p = .022), adherence to relaxation (B = −0.02, p = .029), adherence to exposure (B = 0.08, p = .009), and number of sessions (B = 0.08, p = .002). Thus a higher impairment in social anxiety was negatively associated with outcome on the GAD-7 and this was also true for a higher number of relaxation exercises. Both, adherence to exposure and adherence regarding number of sessions were positively associated with outcome.

When using the MINI-Spin as dependent variable, there was neither a significant direct nor significant indirect effect of participant group on outcome in social anxiety symptoms.

Prediction of adherence in outpatient participants

As adherence was especially low in outpatient participants and additional baseline variables collected during registration at the outpatient clinic were available1 we tried to identify relevant predictor variables of adherence to treatment components for outpatients. To investigate whether any of these variables predicted adherence to active treatment components for outpatient participants, again, LASSO was used. As too few outpatient participants reported having done exposure, only predictors of adherence to relaxation were investigated. The most important predictors identified using LASSO were level of education, initial impairment on the FEP-2, incongruence as measured by the INC-S, self-efficacy as measured by the GSE, and treatment expectations (see Table 4). When entered into logistic regression, incongruence (b = −2.56, p = .016), self-efficacy (b = 2.49, p = .016), and level of education (b = 1.53, p = .022) significantly predicted adherence to relaxation (see Table 5 for more details).

Table 4

Importance by feature as estimated by the LASSO algorithm.

Adherence to relaxation in outpatient
Feature	Importance
Initial impairment FEP2a	3.06
Self-efficacy GSEb	3.01
Incongruence INC-Sc	2.98
Level of education(University entrance diploma)	2.25
Treatment expectations	1.97
Sex	1.00

FEP-2: Questionnaire for the Evaluation of Psychotherapeutic Progress (FEP-2).

GSE: General Self-Efficacy Scale.

INC-S: Incongruence Questionnaire – short version.

Table 5

Baseline variables as predictors of adherence to relaxation in outpatient participants estimated with logistic regression analysis.

Variables	B (SE)	p	Lower	OR	Upper
Intercept	−5.90(3.84)	.124		0.00
Initial impairment FEP-2a	0.90(0.93)	.333	0.40	2.47	15.42
Initial impairment INC-Sb	−2.56(1.06)	.016	0.01	0.08	0.62
Self-efficacy GSEc	2.49(1.03)	.016	1.58	12.03	91.43
Treatment expectations	1.31(0.75)	.078	0.86	3.72	16.93
Level of education(University entrance diploma)	1.53(0.66)	.022	1.25	4.60	16.02

Likelihood ratio test χ2= 11.85. p = .037; Nagelkerke R2 = 0.250.

FEP-2: Questionnaire for the Evaluation of Psychotherapeutic Progress (FEP-2).

INC-S: Incongruence Questionnaire – short version.

GSE: General Self-Efficacy Scale.

Discussion

In this study, we investigated differences in outcome and adherence to treatment components between self-referred and outpatient participants. In addition, we investigated adherence to exposure, relaxation, and cognitive restructuring as well as adherence regarding number of sessions as mediators of outcome. We found that adherence varied across treatment components with relatively high adherence to relaxation and low adherence to exposure and cognitive restructuring. This could fit to the idea that perceived difficulty of treatment components varies. In line with our expectations, we also found that adherence to treatment components and adherence regarding number of sessions differed depending on the setting, with higher levels found for self-referred than for outpatient participants.

However, results on the association between adherence and outcome were mixed: While self-referred participants showed higher adherence than outpatient participants, these two groups hardly differed in terms of outcome. At the same time, some results did point to an association between adherence to treatment components and outcome: Results indicated that adherence may mediate the relationship of participant group on treatment outcome. Specifically, participant group had an effect on adherence to exposure, which in turn impacted treatment outcome directly as well as via a higher number of sessions. This could indicate that especially for certain participant groups, it is important to not only track adherence to number of sessions, but also adherence to treatment components in internet interventions. A more thorough investigation of adherence to treatment components could allow us to pinpoint where participants experience difficulties and how to solve them to enable participants to continue treatment and achieve a good treatment outcome. Of course, it is also possible that persons who have already decided to enter face-to-face treatment and are already registered for treatment are generally less willing to participate in internet interventions. In that case, variables such as simple preference or negative attitudes towards internet interventions may be relevant (e.g., Schröder et al., 2017).

In addition to the indirect effect of exposure and number of sessions adherence to relaxation also had a direct effect on treatment outcome. Surprisingly, the number of reported relaxation exercises was negatively associated with outcome. With regard to anxiety symptoms, it has been noted that in some cases, relaxation can lead to an increase in symptoms (Newman et al., 2018). For example, too much relaxation may not be indicated when used as a means of avoidance. In our study, only 14 participants used relaxation more than ten times. Still, this finding may implicate that for some participants, adherence to relaxation should be more closely monitored in guided internet interventions targeting anxiety.

Furthermore, it should be noted that the results could not be replicated for outcome in anxiety symptoms as measured by the Mini-SPIN. It is possible that the short, three item measure was not sensitive enough to capture differences between participant groups.

In line with the findings from our mediation analysis, several studies have also reported an association between adherence and outcome (El Alaoui et al., 2015; Lutz et al., 2017), which underlines the importance of investigating adherence in internet interventions. In this study, we also investigated predictors of adherence in a potentially unique participant group, namely outpatient participants. Our results showed that level of education, incongruence, and self-efficacy predicted adherence to relaxation in outpatient participants. This finding indicates that in outpatient clinic settings, certain patient characteristics such as low self-efficacy may be relevant. Thus, more studies should examine patient characteristics in participants from different treatment settings to identify relevant predictors of adherence in internet interventions.

One reason for the in part divergent findings in this study regarding the relationship of adherence and outcome could be the small sample size that limited the power to detect differences between groups in our study. In addition, it is possible that outpatient participants were more prone to use additional mental health services compared to self-referred participants.

Overall, our findings suggest that adherence to treatment components is an important factor that has thus far been neglected in research, despite its potential to shed more light on the process of change during internet interventions. Its investigation could reveal potential mechanisms in internet interventions that may lead to poor treatment outcome for certain participant groups if important patient characteristics are not considered during treatment. Differences between participant groups could potentially include important patient variables such as comorbidity and level of self-efficacy, but also varying perceptions regarding the difficulty of certain treatment components. Identifying important patient characteristics and process variables for different participant groups could result in clinical implications regarding the implementation of internet interventions as well as the optimal allocation of participants.

Limitations

There are several limitations, which need to be considered. One limitation is the small sample size limiting the generalizability of the findings as well as power to detect differences between groups. Post-hoc sensitivity analysis using Gpower showed that the power to detect an interaction effect of group (self-referred and outpatient participants) and time with a 2 × 2 repeated measures ANOVA was very low for outcome on the MINI-Spin and PHQ-9 (below 0.15) and only higher for outcome on the GAD-7 (0.74) and the HSCL-11 (0.99). In addition only a small subgroup of outpatients were eligible for the study and agreed to participate. Thus, it remains uncertain to what degree self-referred and outpatient participants really do differ regarding adherence and outcome. Furthermore, the indicators of adherence used here are only proxies, as it is possible that participants used the treatment components without reporting so in their dairy. Also, it may be that important predictors of outcome and adherence were missed, so it is necessary to investigate relevant patient variables in more depth using larger samples before any clear implications for the optimization of internet interventions can be derived. Similarly, the rate of participants showing reliable change was quite low in this study. One explanation could be the overall high impairment in both outpatients and self-referred participants and the low adherence of participants to exposure.

Another limitation is the study design, which was naturalistic. A dismantling design would have been more appropriate to study the differential impact of adherence to treatment components. In addition, we did not control for ordering effects, which would allow the estimation of adherence to treatment components independent of number of sessions. Furthermore, no control group with repeated measurements was available, so no causal inference regarding treatment outcome can be drawn. The usage of additional interventions was not controlled, so it is possible that participants also completed other treatments that were available to them in routine care. Also, for economic reasons, not all measures were available for self-referred compared to outpatient participants. Furthermore, no follow-up was conducted, so inferences regarding long-term treatment effects cannot be made. The reported findings are preliminary.

To better understand how adherence to treatment components is an important mechanism in internet interventions and to identify predictors of adherence, further studies are needed. Future studies should include larger sample sizes and participants from various settings, applying repeated measurements to investigate differences in adherence between various participant populations. In addition, various measures of adherence and potentially important predictor variables such as self-efficacy should be investigated in more detail. Further research in this area has the potential to shed more light on important patient variables that may foster or hamper the adherence to and outcome of internet interventions. This knowledge would allow clinicians to consider important variables when optimizing internet interventions or allocating participants to internet interventions.

Authors' note

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of competing interest

None.