Targeting subjective engagement in experimental therapeutics for digital mental health interventions.

Engagement is a multifaceted construct and a likely mechanism by which digital interventions achieve clinical improvements. To date, clinical research on digital mental health interventions (DMHIs) has overwhelmingly defined engagement and assessed its association with clinical outcomes through the objective/behavioral metrics of use of or interactions with a DMHI, such as number of log-ins or time spent using the technology. However, engagement also entails users' subjective experience. Research is largely lacking that tests the relationship between subjective metrics of engagement and clinical outcomes. The purpose of this study is to present a proof-of-concept exploratory evaluation of the association between subjective engagement measures of a mobile DMHI with changes in depression and anxiety. Adult primary care patients (N = 146) who screened positive for depression or anxiety were randomized to receive a DMHI, IntelliCare, immediately or following an 8-week waitlist. Subjective engagement was measured via the Usefulness, Satisfaction, and Ease of Use (USE) Questionnaire. Across both conditions, results showed that individuals who perceived a mobile intervention as more useful, easy to use and learn, and satisfying had greater improvements in depression and anxiety over eight weeks. Findings support our proposed experimental therapeutics framework that hypothesizes objective/behavioral and subjective engagement metrics as mechanisms that lead to changes in clinical outcomes, as well as support directing intervention design efforts for DMHIs to target the user experience.

Introduction

The disruptive innovation of digital mental health interventions (DMHIs) over traditional psychotherapy is extending treatment beyond in-person sessions into users' daily lives. Thus, for a DMHI to improve clinical outcomes, it must be engaging so it is used in the moments and contexts when psychological support is needed most. Because engagement is a “precondition” for effectiveness (Yardley et al., 2016; Perski et al., 2017), it has been the focus of design efforts for DMHIs.

To date, clinical research on DMHIs has overwhelmingly defined engagement and assessed its association with clinical outcomes through the behavioral metrics of use of or interactions with a DMHI, such as number of log-ins or time spent using the technology (Perski et al., 2017). However, the relationship between use and outcomes is complex (Yardley et al., 2016). DMHI use tends to be lower when symptoms improve (Mohr et al., 2010), and randomized clinical trials (RCTs) of DMHIs show mixed results for whether more use leads to better outcomes (Donkin et al., 2011).

By contrast, the field of human-computer interaction conceptualizes engagement not just as quantity of use, but also quality of use (Doherty and Doherty, 2018). This type of engagement has been described as “cognitive/affective” (Kelders et al., 2020b; Kelders and Kip, 2019) and “subjective/experiential” engagement (Perski et al., 2017; Graham et al., 2019), and refers to whether users who interact with a technology perceive it to be, for example, useful, usable, and satisfying (e.g., Norman, 2013; Doherty and Doherty, 2018) as well as their level of interest, attention, and affect during that interaction (e.g., Short et al., 2018). Greater attention is being paid to designing digital behavioral interventions to improve users' subjective experience (e.g., Kelders, 2015; Yardley et al., 2016), and we have proposed that these metrics of “subjective engagement” be considered a target mechanism by which DMHIs may achieve changes in clinical outcomes (Graham et al., 2019).

More specifically, we proposed an adaptation to the National Institute of Mental Health's (NIMH) Research Domain Criteria (National Institute of Mental Health, 2015) experimental therapeutics framework for DMHIs that identifies “engagement” as a target mechanism by which DMHIs may yield changes in clinical outcomes (Graham et al., 2019). The NIMH experimental therapeutics framework indicates that interventions should be designed to engage and test a hypothesized mechanism to determine if changes in the hypothesized mechanism lead to changes in clinical outcomes (Insel, 2015). Our proposal indicates that “engagement” be measured by both the behavioral metrics of using a DMHI (which we grouped as “objective”) and the experiential or cognitive/affective metrics of interacting with a DMHI (which we grouped as “subjective”), as the combination of these measures is important for understanding user engagement (Doherty and Doherty, 2018). While our experimental therapeutics framework theorizes that designing a DMHI to target subjective engagement metrics will yield clinical improvements, research is largely lacking that tests the relationship between subjective engagement metrics and clinical outcomes (c.f. Kelders, 2015, Altman et al., 2018).

This paper presents a proof-of-concept exploratory evaluation of the association between subjective engagement measures of a mobile DMHI with changes in depression and anxiety. Consistent with other RCTs (Donkin et al., 2011), we previously found negligible associations between objective use metrics of this DMHI and outcomes in a RCT, which had high rates of intervention use and completed follow-up assessments (Graham et al., 2020).

Methods

Participants & procedure

This is an exploratory secondary analysis of a RCT testing the IntelliCare Platform, a suite of mobile apps for depression and anxiety delivered over 8 weeks with coaching, compared to treatment-as-usual waitlist control (Graham et al., 2020). Adult primary care patients (N = 146) who screened positive for depression (Patient Health Questionnaire-8 (PHQ-8) ≥10; (Kroenke et al., 2009)] or anxiety [Generalized Anxiety Disorder-7 (GAD-7) ≥8; (Spitzer et al., 2006)] were randomized to receive IntelliCare immediately or following an 8-week waitlist. All participants provided online informed consent. The study was approved by the University of Arkansas for Medical Sciences Institutional Review Board, and a Data Safety Monitoring Board provided oversight.

Measures

Outcomes [PHQ-9 (Kroenke et al., 2001) and GAD-7] were assessed at baseline (i.e., pre-treatment) and every four weeks for 16 weeks. Subjective engagement was measured via the commonly-administered 19-item Usefulness, Satisfaction, and Ease of Use (USE) Questionnaire (Lund, 2001) Short-Form, modified for DMHIs. The USE was administered at mid-treatment and post-treatment for each condition (i.e., relative to when each condition received the intervention). This means that among participants in the IntelliCare condition, the USE was completed at the 4-week and 8-week assessments; for those in the waitlist control condition, the USE was completed at the 12-week and 16-week assessments. Items on the USE were rated from 1 (strongly disagree) to 7 (strongly agree), aggregated into four subscales: Usefulness (6 items; e.g., “It helps me be more effective.”), Ease of use (5 items; e.g., “It is easy to use.”), Ease of learning (3 items; e.g., “I learned to use it quickly.”), and Satisfaction (5 items; e.g., “I am satisfied with it.”).

Analyses

Because participants in both conditions received the intervention, data were combined across study conditions to assess the association between USE subscales and changes in outcomes at post-treatment. Spearman's correlations were used to assess the association between USE subscales at post-treatment and changes in outcome (i.e., post-treatment minus baseline) for depression or anxiety (presented separately), combined across conditions. To assess the magnitude of change in depression or anxiety from baseline to post-treatment as predicted from the USE subscales, we also conducted regression analyses; primary models were unadjusted with secondary models adjusted for age, sex, and race (consistent with the primary outcome analyses in the parent trial (Graham et al., 2020)). Wilcoxon tests compared USE scores by treatment arms. Statistical significance was set at p < 0.05. Observed data are presented.

Results

Descriptive statistics for each USE subscale among individuals with depression or anxiety are presented in Table 1. In correlation analyses, higher scores on each USE subscale were associated with greater improvement at post-treatment in depression (correlation range across USE subscales = −0.24 to −0.25; ps ≤ 0.013) and anxiety (correlation range across USE subscales = −0.21 to −0.25; ps ≤ 0.025). Table 1 also presents results of the regression analyses. The adjusted regression parameters ranged from −0.16 to −0.32 for depression and − 0.17 to −0.37 for anxiety. This means that for an approximately 3 to 6-point difference in USE subscale score, we would expect to see a 1-unit decrease on the PHQ-9 and GAD-7 from pre- to post-treatment. Waitlist delay did not impact users' USE ratings compared to immediate intervention (Table 2).

Table 1

Regression parameters for change in clinical outcome per unit change in subjective use measures of a digital mental health intervention (all observed data).

USE subscale	Depression (n = 112)median (25th, 75th) [min, max]	Change in PHQ-9 unadjusted	Change in PHQ-9 adjusted	Anxiety (n = 121)median (25th, 75th) [min, max]	Change in GAD-7 unadjusted	Change in GAD-7 adjusted
Usefulness(6 items)	33 (28, 37) [12, 42]	−0.21 (SE = 0.09) [P = 0.014]	−0.25 (SE = 0.09) [P = 0.007]	33 (26, 38) [12, 42]	−0.16 (SE = 0.06) [P = 0.007]	−0.19 (SE = 0.06) [P = 0.002]
Ease of Use(5 items)	31 (28, 35) [5, 35]	−0.18 (SE = 0.11) [P = 0.090]	−0.16 (SE = 0.11) [P = 0.136]	30 (26, 35) [5, 35]	−0.19 (SE = 0.07) [P = 0.008]	−0.20 (SE = 0.07) [P = 0.008]
Ease of Learning(3 items)	21 (18, 21) [3,21]	−0.36 (SE = 0.18) [P = 0.056]	−0.32 (SE = 0.19) [P = 0.098]	21 (18, 21) [3, 21]	−0.36 (SE = 0.12) [P = 0.004]	−0.37 (SE = 0.13) [P = 0.004]
Satisfaction(5 items)	28 (22, 32) [5, 35]	−0.18 (SE = 0.08) [P = 0.040]	−0.20 (SE = 0.09) [P = 0.022]	28 (22, 33) [5, 35]	−0.15 (SE = 0.06) [P = 0.014]	−0.17 (SE = 0.06) [P = 0.005]

Note: PHQ-9 = Patient Health Questionnaire-9; GAD-7 = Generalized Anxiety Disorder-7. Change in PHQ-9 and GAD-7 was defined as post-treatment minus baseline scores. Adjusted analyses accounted for age, sex, and race.

Table 2

End of treatment USE scores by treatment arm.

USE subscale	IntelliCare (n = 70)median (25th, 75th) [min, max]	Waitlist control (n = 64)median (25th, 75th) [min, max]	Wilcoxon P
Usefulness (6 items)	31 (26, 38) [17, 42]	34 (28, 37) [12, 42]	0.437
Ease of Use (5 items)	32 (26, 34) [5, 35]	30 (28, 35) [7, 35]	0.982
Ease of Learning (3 items)	21 (18, 21) [4, 21]	20 (18, 21) [3, 21]	0.265
Satisfaction (5 items)	27 (22, 33) [5, 35]	29 (25, 32) [5, 35]	0.283

Note: Participants in the IntelliCare arm received the intervention after randomized; participants in the Waitlist Control arm received the intervention after an 8-week delay.

Discussion

Results support our experimental therapeutics framework for DMHIs. Participants who perceived IntelliCare to be more useful, easy to use and learn, and satisfying had greater improvements in depression and anxiety over the 8-week intervention period. Further, overall inferences were fairly consistent between the correlations and regressions. These findings extend the limited literature to date on the association between subjective engagement metrics and changes in clinical outcomes.

Notably, our findings are in contrast to the previously-published usage data (e.g., number of app sessions) which were not significantly related to outcomes in this trial (Graham et al., 2020). This suggests that how a person uses the DMHI may be as or more important to target and assess for eliciting changes in clinical outcomes than how often. Consequently, results substantiate the need for clinical research to move beyond only targeting and assessing objective use of DMHIs to also target the user experience (Yardley et al., 2016). New measures have been created to capture multiple components of engagement in digital interventions (Kelders and Kip, 2019; Kelders et al., 2020a; Perski et al., 2020), which can improve tests of the relation between engagement and outcomes.

As our proposed adaptation to the NIMH's experimental therapeutics framework suggests, inclusion of these subjective engagement metrics in an experimental therapeutics framework provides insight into how the usability of a DMHI contributes to clinical outcomes and, therefore, where design efforts can be directed to improve the quality and effectiveness of DMHIs.

CRediT authorship contribution statement

AKG, EGL, MR, & DCM designed the study. AKG, MJK, CJG, & DCM performed the trial and collected the data. MJK performed the statistical analyses. AKG and DCL drafted the manuscript, and MJK, EGL, CJG, NVG, & MR critically reviewed the manuscript for important intellectual content. All authors read and approved the final manuscript.

Declaration of competing interest

DCM has an ownership interest in Adaptive Health, Inc., which has a license from Northwestern University to commercialize IntelliCare. AKG, MKJ, & EGL have received consulting fees from Actualize Therapy, LLC. DCM also has received consulting fees from Apple Inc.