Lyndsay A Nelson1,2, Andrew J Spieker3, Lindsay S Mayberry1,2,4,5, Candace McNaughton6,7, Robert A Greevy3. 1. Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA. 2. Center for Health Behavior and Health Education, Vanderbilt University Medical Center, Nashville, Tennessee, USA. 3. Department of Biostatistics, Vanderbilt University, Nashville, Tennessee, USA. 4. Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA. 5. Center for Diabetes Translation Research, Vanderbilt University Medical Center, Nashville, Tennessee, USA. 6. Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA. 7. Geriatric Research Education Clinical Center, Tennessee Valley Healthcare System VA Medical Center, Nashville, Tennessee, USA.
Abstract
OBJECTIVE: Guidance is needed on studying engagement and treatment effects in digital health interventions, including levels required for benefit. We evaluated multiple analytic approaches for understanding the association between engagement and clinical outcomes. MATERIALS AND METHODS: We defined engagement as intervention participants' response rate to interactive text messages, and considered moderation, standard regression, mediation, and a modified instrumental variable (IV) analysis to investigate the relationship between engagement and clinical outcomes. We applied each approach to two randomized controlled trials featuring text message content in the intervention: REACH (Rapid Encouragement/Education and Communications for Health), which targeted diabetes, and VERB (Vanderbilt Emergency Room Bundle), which targeted hypertension. RESULTS: In REACH, the treatment effect on hemoglobin A1c was estimated to be -0.73% (95% CI: [-1.29, -0.21]; P = 0.008), and in VERB, the treatment effect on systolic blood pressure was estimated to be -10.1 mmHg (95% CI: [-17.7, -2.8]; P = 0.007). Only the IV analyses suggested an effect of engagement on outcomes; the difference in treatment effects between engagers and non-engagers was -0.29% to -0.51% in the REACH study and -1.08 to -3.25 mmHg in the VERB study. DISCUSSION: Standard regression and mediation have less power than a modified IV analysis, but the IV approach requires specification of assumptions. This is the first review of the strengths and limitations of various approaches to evaluating the impact of engagement on outcomes. CONCLUSIONS: Understanding the role of engagement in digital health interventions can help reveal when and how these interventions achieve desired outcomes.
OBJECTIVE: Guidance is needed on studying engagement and treatment effects in digital health interventions, including levels required for benefit. We evaluated multiple analytic approaches for understanding the association between engagement and clinical outcomes. MATERIALS AND METHODS: We defined engagement as intervention participants' response rate to interactive text messages, and considered moderation, standard regression, mediation, and a modified instrumental variable (IV) analysis to investigate the relationship between engagement and clinical outcomes. We applied each approach to two randomized controlled trials featuring text message content in the intervention: REACH (Rapid Encouragement/Education and Communications for Health), which targeted diabetes, and VERB (Vanderbilt Emergency Room Bundle), which targeted hypertension. RESULTS: In REACH, the treatment effect on hemoglobin A1c was estimated to be -0.73% (95% CI: [-1.29, -0.21]; P = 0.008), and in VERB, the treatment effect on systolic blood pressure was estimated to be -10.1 mmHg (95% CI: [-17.7, -2.8]; P = 0.007). Only the IV analyses suggested an effect of engagement on outcomes; the difference in treatment effects between engagers and non-engagers was -0.29% to -0.51% in the REACH study and -1.08 to -3.25 mmHg in the VERB study. DISCUSSION: Standard regression and mediation have less power than a modified IV analysis, but the IV approach requires specification of assumptions. This is the first review of the strengths and limitations of various approaches to evaluating the impact of engagement on outcomes. CONCLUSIONS: Understanding the role of engagement in digital health interventions can help reveal when and how these interventions achieve desired outcomes.
Authors: Lindsay S Mayberry; Jeffrey S Gonzalez; Kenneth A Wallston; Sunil Kripalani; Chandra Y Osborn Journal: Diabetes Res Clin Pract Date: 2013-09-26 Impact factor: 5.602
Authors: Kingshuk Pal; Charlotte Dack; Jamie Ross; Susan Michie; Carl May; Fiona Stevenson; Andrew Farmer; Lucy Yardley; Maria Barnard; Elizabeth Murray Journal: J Med Internet Res Date: 2018-01-29 Impact factor: 5.428
Authors: Sascha Miller; Ben Ainsworth; Lucy Yardley; Alex Milton; Mark Weal; Peter Smith; Leanne Morrison Journal: J Med Internet Res Date: 2019-02-15 Impact factor: 5.428
Authors: Sarah Edney; Jillian C Ryan; Tim Olds; Courtney Monroe; François Fraysse; Corneel Vandelanotte; Ronald Plotnikoff; Rachel Curtis; Carol Maher Journal: J Med Internet Res Date: 2019-11-27 Impact factor: 5.428