The peril of self-reported adherence in digital interventions: A brief example.

Adherence is an important predictor of intervention outcomes, but not all measures of adherence are created equally. Here, we analyzed whether there was a discrepancy between self-report adherence and objective adherence in a digital mindfulness meditation randomised, controlled trial. A sample of 174 young adult undergraduate university students trialled either an app-based or email-based mindfulness meditation program (or an app-based attention control). Participants' adherence (number of sessions completed) and mental health was self-reported. Objective adherence data were provided by the owners of the digital mindfulness programs. We found evidence of inflated self-reported adherence to the app-based intervention and argue that the inflation was not explained by social desirability biases because participants were aware we would have access to object data and no remuneration was tied to adherence. We also comment on the different conclusions we would have drawn about the effectiveness of the digital interventions on mental health, had we used the self-reported adherence data rather than the objective adherence data. We use this example to suggest that it may be perilous to rely on self-reported measures of adherence when assessing the effectiveness of digital interventions.

RCT Entities:   Population Interventions Outcomes

Introduction

Between 2009 and 2015, yearly publications on e-mental health interventions trebled (Firth et al., 2016), but meta-analytic reviews reveal that self-guided digital interventions often have only modest effects on mental health (Andersson and Cuijpers, 2009; Cuijpers et al., 2011; Spijkerman et al., 2016). One explanation for these modest effects might be that adherence to digital interventions is low (Cuijpers et al., 2011; Eysenbach, 2005). Adherence refers to whether individuals access the content and use it in the manner it was designed to be optimally effective (Christensen et al., 2009; Donkin et al., 2011).

To be optimally effective, regular practice is considered a key component of mindfulness-based interventions (Segal et al., 2013) and adherence to practice guidelines is correlated with intervention outcomes (meta-analysis: k = 28, r = 0.264, p < .001; Parsons et al., 2017). Likewise, adherence in self-guided iCBT is associated with lower depressive symptoms and stronger responsiveness to treatment (Karyotaki et al., 2017). Outside of digital interventions, adherence is a strong predictor of intervention outcomes, particularly when the health issue is less serious, chronic, non-medicated, in a pediatric population, or where outcomes are not disease specific (DiMatteo et al., 2002). Counterintuitively, self-reported adherence is also a strong predictor of intervention outcomes (DiMatteo et al., 2002). But, few digital interventions report adherence rates, and even fewer report how adherence relates to intervention outcomes (Brown et al., 2016; Donkin et al., 2011).

To date, the majority of research on adherence in digital interventions has focused on operationalizing adherence and identifying predictors of adherence (see: Christensen et al., 2009) but the complexity of adherence is often neglected (see: Sieverink et al., 2017 for a systematic review). Adherence has been operationalized in a number of ways (e.g., for practical reasons metrics such as sessions completed, days used, logins, or a combination of these are often used; Donkin et al., 2011; Donkin et al., 2013; Sieverink et al., 2017) but the measures often fail to capture the quality of the engagement with the intervention (e.g., were skills acquired), nor do they distinguish between observed adherence (how much the individual experienced the content of the intervention) and prescribed adherence (how much the individual experienced the intervention as recommended or intended; Kelders et al., 2012; Sieverink et al., 2017). Further, few digital interventions report or justify the level of adherence required to make the intervention work and instead rely on a ‘more-is-more’ approach (Sieverink et al., 2017) that presupposes that the dose-response relationship is linear. But, a linear dose-response relationship between adherence and outcomes is not always the case (e.g., Donkin et al., 2013; Blanck et al., 2018). Researchers have identified a host of additional factors that influence adherence including persuasive intervention design (Kelders et al., 2012), amount of support provided (Andersson and Cuijpers, 2009; Christensen et al., 2009), and participant characteristics (Christensen et al., 2009).

Another important but overlooked issue is the accuracy of self-reported adherence in digital interventions. Although self-reported adherence data are easily collected, they may be subject to biases that affect all self-report data (e.g., recall bias and response bias; Kimberlin and Winterstein, 2008; Schwarz, 1999), which may lead to inaccurate conclusions about the effectiveness of the intervention. Researchers can mitigate recall and response biases in self-report data by using research designs like experience sampling or daily diaries to reduce recall time (Schwarz, 2012) or by nonjudgmentally acknowledging normality of non-adherence2 and anonymizing online reports of sensitive topics to reduce socially desirable responding (Gnambs and Kaspar, 2015). A more direct approach would be to use objective measures of adherence. In contrast to some other interventions, objective measures of adherence are readily available in digital interventions in the form of number of logins, sessions completed, or minutes completed (Donkin et al., 2011) and can be used to measure adherence differences across intervention platforms (Morrison et al., 2018). In the current short report, we demonstrate the peril of relying on self-reported adherence by comparing the discrepancies between self-reported and objectively-gathered adherence data in web-based and app-based digital mindfulness meditation interventions.

Method

Design

This study was a protocol replication of an earlier study (Flett et al., 2018) with a few minor adaptations. The study was a 40-day randomised, controlled trial (RCT) comparing the use of one of two mindfulness meditation programs (or an attention control) on changes in mental health (University of Otago ethics committee #D15/063). A convenience sample of 174 undergraduate university students (M = 19.76 years, SD = 2.56 years, 79.9% female, 71.8% New Zealand European/Pākehā)3 were randomly assigned to use either an app-based mindfulness program (Headspace, n = 65), an email-based mindfulness program (10 Minute Mind, n = 51), or an app-based attention control program (Evernote, n = 58). We recommended that participants use their program for 10 min per day. This period was equivalent to one session of the intervention and was consistent with previous digital mindfulness research (e.g., Flett et al., 2018; Howells et al., 2016). We measured self-reported and objective adherence over two time periods: 1) Prescribed adherence: a 10-day period where adherence was requested each day and 2) Discretionary adherence: a 30-day period where adherence was at the discretion of the individual (mimicking more realistic or natural uptake). Both mindfulness interventions (Headspace and 10 Minute Mind) involved similar active therapeutic components (e.g., they introduced mindfulness through a series of brief formal mindfulness practices such as mindful breathing [using the breath as an attentional object of intense focus] and body scanning [systematically focusing on certain parts of the body]). Access to the interventions followed a hybrid structure whereby the interventions involved fixed core content with additional optional components (Sieverink et al., 2017); app-users had to complete the first 10 sessions consecutively in order to ‘unlock’ other intervention content, whereas email-users were emailed new sessions each day but had access to brief mindfulness “top up” sessions (optional 3-min meditations). For email-based participants, all intervention sessions were 10 min long, whereas, for app-based participants the first 10-sessions were 10 min long, but longer sessions (up to 45 min long) were available during the 30-day Discretionary adherence period.

Procedure and measures

All participants reported their mental health (depressive symptoms, anxiety, stress, flourishing, resilience, mindfulness, and adjustment to college: measures described in Flett et al., 2018) on Day 0 in the research lab (baseline; also, demographic and personality characteristics using the NEO-FFI 60, Costa and MacCrae, 1992), and online on approximately Day 10 and Day 40. Self-reported and objective adherence were operationalized as the number of intervention sessions completed; this was a pragmatic operationalisation based on the available objective use data. Self-reported adherence was measured daily during the first 10 days as well as retrospectively on Day 10 and Day 40 with a single item survey question (“how many times did you access the app in the preceding study period”). Objective adherence data were also provided by the owners of the mindfulness programs, which gave us the number of times each user completed a session using their mindfulness program. To reduce the likelihood of socially desirable responding, adherence was not tied to any form of remuneration and participants were aware that we would be provided with their objective adherence data. Participants used their participation to obtain a small portion of course credit tied to survey completion (not app use). Except where specified, we only present results for the mindfulness conditions because we did not have access to objective adherence data for the app-based Attention Control program.

Results

Discrepancies between self-reported and objective adherence were calculated by subtracting objective adherence from self-reported adherence (self-report – objective = discrepancy) during the 10 days of prescribed adherence (discrepancies were calculated for both retrospective and daily self-reported adherence) and for the 30 days of discretionary adherence (retrospective only). Positive values indicated over-reporting of adherence and negative values indicated under-reporting of adherence. Adherence discrepancies were assessed using two-way mixed ANOVA (self-report vs objective; app-based user vs email-based user) with Bonferroni adjustment. In Supplementary Tables 2–9 we present descriptive statistics and tests comparing all major outcomes over time within (S Table 2) and between conditions (S Table 3), moderation by adherence (S Table 4–7), and correlations between adherence measures and demographic, personality, and outcome measures by condition (S Table 8–9).

Discrepancies between self-reported and objective adherence

As shown in Table 1, the discrepancy between self-reported and objective adherence differed by digital platform (app vs. email), particularly for the longer time period. Only app-users showed significant discrepancies between self-report and objective measures of adherence. App users self-reported doing 1.54 more sessions than they objectively did during the 10-day prescribed use period and 9.13 more sessions than they objectively did during the discretionary 30-day time period (D11-40). By contrast, for email-based intervention users, the discrepancy was negligible during both the 10-day prescribed adherence period (over-reported by about 0.3 sessions for both retrospective and daily self-report, not significant) and the 30-day discretionary adherence period (over-reported by 3.00 sessions, not significant). In fact, app users over-reported their adherence by over three times as much as email users when adherence was discretionary across 30 days (e.g., App: M = 9.13, SD = 7.84 vs. Email: M = 2.58, SD = 11.40) and almost 35 times as much overall (e.g., App: M = 8.62, SD = 9.24 vs. Email: M = −0.25 SD = 13.91). Furthermore, during the 10-day prescribed adherence period, there were no differences between daily and retrospective self-report of adherence for the sample overall (t(171) = 0.13, p = 0.899) or within conditions (all ps > 0.581; S Table 1), suggesting that daily reports of adherence during the 10-day period were no more accurate than retrospectively recalling their adherence at the end of the 10 days.

Table 1

Means (M) and standard deviations (SD) of self-reported and objective adherence when adherence was prescribed (Days 0–10, daily and retrospective), discretionary (Days 11–40), and overall (Days 0–40) for all conditions (self-report descriptive statistics only for controls). F tests indicate the results of the mixed ANOVA.

Adherence		Self-report		Objective		Over-reporta			Self-report vs objective			App-based vs email-based over-reportingb
App- and Email-combined	n	M	SD	M	SD	%	M	SD	F	df	p	F	df	p
Prescribed D0-10 Retro	116	7.87	2.14	6.87	3.53	14.6	1.00	3.44	8.49	1114	0.004	3.71	1114	0.057
Prescribed D0-10 Daily	116	7.84	2.50	6.87	3.53	14.1	0.97	3.93	5.99	1114	0.016	2.70	1114	0.103
Discretionary D11-40	89	12.42	8.29	5.83	8.79	49.8	6.58	9.83	36.02	1, 87	<0.001	9.21	1, 87	0.003
Overall D0-40 Retro	116	17.48	9.55	12.73	10.87	37.3	4.75	11.81	16.78	1114	<0.001	18.71	1114	<0.001
Overall D0-40 dailyc	116	17.45	9.71	12.73	10.87	37.1	4.72	12.30	15.04	1114	<0.001	16.93	1114	<0.001
App-user
Prescribed D0-10 retro	65	8.45	2.08	6.91	4.29	22.3	1.54	3.69	13.32	1114	<0.001
Prescribed D0-10 daily	65	8.40	2.08	6.91	4.29	21.6	1.49	4.18	9.51	1114	0.003
Discretionary D11-40	52	12.23	8.31	3.10	6.77	294.5	9.13	7.84	49.10	1, 87	<0.001
Overall D0-40 retro	65	18.23	9.78	9.57	8.78	90.5	8.66	8.93	40.33	1114	<0.001
Overall D0-40 dailyc	65	18.18	9.76	9.57	8.78	90.0	8.62	9.24	36.33	1114	<0.001
Email-user
Prescribed D0-10 retro	51	7.14	2.61	6.82	2.26	4.7	0.31	2.98	0.43	1114	0.551
Prescribed D0-10 daily	51	7.12	2.81	6.82	2.26	4.4	0.29	3.51	0.29	1114	0.591
Discretionary D11-40	37	12.68	8.37	9.68	9.89	31.0	3.00	11.25	3.77	1, 87	0.050
Overall D0-40 retro	51	16.53	9.26	16.76	11.98	−1.4	−0.24	13.17	0.02	1114	0.879
Overall D0-40 dailyc	51	16.51	9.66	16.76	11.98	−1.5	−0.25	13.91	0.02	1114	0.875
Attention control
Prescribed D0-10 retro	58	8.62	2.15
Prescribed D0-10 daily	56	8.43	2.15
Discretionary D11-40	42	11.33	10.56
Overall D0-40 retro	58	17.02	11.29
Overall D0-40 dailyc	58	16.83	11.15

Over-reporting = Self-report – Objective, % = ((Self-report/Objective) ∗ 100) − 100; Positive values indicate number of sessions where adherence was over-reported in a time period.

Interaction from Two-Way Mixed ANOVA showing the difference between self-reported vs objective adherence varied by platform.

Overall using Prescribed D0–10 Daily and Discretionary D11-40.

Effect of intervention and adherence on outcomes

We found no consistent nor convincing evidence that mental health changed over time within conditions (S Table 2), nor that intervention condition (app-based or email-based) predicted change in mental health at Day 10 or Day 40 (controlling for Day 0 outcome; all condition model ps > 0.05; S Table 3). Likewise, we found no consistent nor convincing evidence that adherence (self-reported or objective) predicted change in mental health at Day 10 or Day 40 (controlling for Day 0 outcome; all adherence model ps > 0.05 following adjustment for multiple comparisons; S Tables 4–7). Finally, there were no consistent predictors of self-reported or objective adherence that help explain the app-based or email-based differences in over-reported adherence (S Tables 8–9).

Discussion

Self-reported adherence –whether reported daily, or retrospectively– was not an adequate representation of app-based intervention adherence. When adherence was at the discretion of the user and occurred over a month-long period, which is more representative of realistic mindfulness meditation platform usage, self-reported app adherence was even less reliable. In fact, the discrepancy between self-reported and adherence to the app during that longer period was staggering – 12 sessions self-reported versus only 3 sessions actually logged. This discrepancy occurred even though participants were aware that we would receive access to their objective adherence data and that remuneration (i.e., course credit) was not contingent on adherence. So, the over-reporting does not appear to be the result of social desirability response biases. Over-reporting app use was also not ameliorated by self-reporting app use each day, which suggests that even daily reporting of adherence is problematic.

Interestingly, adherence was more accurate for the email-based intervention than the app-based intervention. This could be due to the email intervention being more unusual and requiring more effort to enact, which would enhance memory for the session. The email-based mindfulness program was delivered to students' university email address on a platform that is not particularly mobile-friendly. As a result, participants would have likely accessed their intervention using a PC or laptop (indeed, several participants reported that they were unable to easily access the email-based intervention using their mobile phones). By contrast, the app intervention integrated seamlessly into participants' lives, requiring less effort to enact, which could reduce memory for the session. Given that young adults spend on average between 2 and 4 h per day on their mobile phones (Montag et al., 2015, sample primarily from Germany; Liao et al., In Review, sample from New Zealand) and more than two-thirds of global internet use in 2017 was completed on mobile devices rather than laptops (Enge, 2018), it may be that the app-based mindfulness intervention was less salient (and more subject to memory biases) than the relatively more unusual email-based mindfulness intervention. The meditation instructor was a New Zealander, so this may have been more salient to our New Zealand-based participants. Whether other plausible mechanisms explain this modality-based discrepancy in over-reporting requires further research; however, it might be a less relevant question in the future as digital interventions continue to increase in technological sophistication.

The objective adherence data may explain the null effects for this particular intervention. We found no evidence that the digital mindfulness interventions improved mental health over time (Day 10 or Day 40: see Supplementary Table 2 and Table 3 for detail). In the absence of adherence data, we naturally would have concluded that the interventions were not effective. However, objective adherence showed that intervention use was much too low to be effective. Given that face-to-face mindfulness programs typically recommend 45 min of home practice, six days per week (Segal et al., 2013), it is unlikely that three sessions of the mindfulness meditation app over the course of 30 days would qualify as a sufficient dose of a psychotherapeutic intervention to produce any lasting or meaningful benefits. This interpretation fits with previous literature that suggests increased adherence is associated with intervention outcomes (Karyotaki et al., 2017; Parsons et al., 2017). However, it could also be that any effects were present but short-lived (occurring only on days of use) (Schumer et al., 2018). This is also the case in other brief or ‘microinterventions’ (Elefant et al., 2017).

In conclusion, our results suggest that self-reported adherence to app-based intervention trials is suspect, particularly over longer time periods. We present this data as a brief cautionary tale about the peril of relying on self-reported adherence when assessing the effectiveness of digital interventions, and app-based interventions in particular. If self-reports of adherence are inflated, researchers and clinicians may both over-estimate the acceptability of a tool (i.e., thinking usership is higher) and under-estimate the effectiveness of a tool (although, this only holds provided there is a positive relationship between adherence and outcomes). Objective adherence data is recommended to determine whether people access the content and use their digital interventions in the intended manner of use.