A digital health psychological intervention (WebMAP Mobile) for children and adolescents with chronic pain: results of a hybrid effectiveness-implementation stepped-wedge cluster randomized trial.

Although psychological treatments benefit youth with chronic pain, treatment is not accessible in most communities. Digital health interventions offer promise for expanding access and reach to this population. Using a stepped-wedge cluster randomized trial design, we evaluated effectiveness and implementation of a digital health delivered psychological intervention for pediatric chronic pain. One hundred forty-three youth, aged 10 to 17 years, with chronic pain and a caregiver were recruited from 8 clinics in the United States. Active intervention included access to the Web-based Management of Adolescent Pain (WebMAP) Mobile app and the WebMAP parent web site to learn pain self-management skills. Effectiveness outcomes included pain intensity, disability, and patient global impression of change, while Reach, Adoption, Implementation, and Maintenance were implementation outcomes. Results showed that youth in both treatment conditions (WebMAP vs Usual Care) had similar changes over time in pain and disability. Youth in the WebMAP condition perceived greater improvement (patient global impression of change) at post-treatment and follow-up (d's = 0.54 and 0.44, P < 0.05) compared with youth receiving usual care. Use of the digital health intervention was modest and variable; approximately 30% of youth and parents completed treatment. Greater engagement (number of completed modules) was associated with significantly greater reductions in pain and disability from pre-treatment to follow-up (d's = -0.57 and -0.38, P < 0.05). Parents, youth, and providers found treatment acceptable; providers had positive attitudes and demonstrated referrals over a maintenance period. Further research is needed to understand how to enhance treatment engagement with digital health interventions and optimize implementation.

1. Introduction

In the United States alone, 2 to 5 million children experience chronic pain[21,26] and are at risk of impairments in physical and psychosocial functioning. A robust evidence base demonstrates that psychological treatments have small to moderate effects for reducing pain and disability in children with chronic pain.[15] Because children with chronic pain are at risk of continued pain,[23,41] worse educational, vocational, and social outcomes in adulthood,[31] effective intervention during childhood is a key priority to address the overall societal burden of chronic pain. Despite the promise of psychological interventions for chronic pain management, few children have access to them[39] because of numerous barriers including geographic distance[4,7] and long wait-lists.[35]

Digital health interventions targeting behavior change (ie, psychological interventions delivered through the Internet and smartphone applications) have the unique potential to bridge this critical gap in service delivery. According to a recent Pew Research Center Internet Survey, 95% of teens in the United States own a smartphone and use the Internet daily.[37] Digital health interventions for pediatric chronic pain are emerging.[13] Our own Internet-delivered program, Web-based Management of Adolescent Pain (WebMAP),[33] has shown excellent patient engagement and small to moderate treatment effects for improving pain-related outcomes in a large-scale randomized controlled trial (RCT).[33]

Although positioned to overcome major barriers to access, effectiveness of digital health interventions for pediatric chronic pain in real-world settings—outside the tightly controlled setting of a randomized clinical trial design—has not been widely studied.[15] Furthermore, factors that may hinder or facilitate real-world implementation of digital health interventions in this population are understudied.[3,30] Systematic reviews have shown that end users do not typically have access to tools developed during research studies; for example, only 13 of 53 tools were found to be publicly available in a review of eHealth tools for pediatric pain management,[19] reducing their real-world impact. Hybrid effectiveness-implementation clinical trials have the potential to address these gaps.[28] To the best of our knowledge, there have not been any previous clinical trials of psychological interventions for youth with chronic pain that have evaluated both treatment effectiveness and implementation in real-world clinical settings.

Building on our previous work with WebMAP,[32,33] the goals of this hybrid effectiveness-implementation clinical trial are to (1) evaluate the real-world usage and effectiveness of WebMAP for youth with chronic pain and (2) assess implementation of WebMAP in pediatric specialty care and pain clinic settings across the United States. Using a stepped-wedge cluster randomized trial (SW-CRT) design, we hypothesized that youth receiving the WebMAP program would show greater improvements in pain-related disability, pain intensity, and patient-reported global impression of change compared with youth receiving usual care. We hypothesized that youth and parents would use the intervention in real-world clinical settings, and that higher engagement in the WebMAP program would be associated with greater treatment benefit. Finally, we expected to find that WebMAP would have adequate reach, adoption, implementation, and maintenance (assessed by the Reach, Effectiveness, Adoption, Implementation, and Maintenance [RE-AIM] framework).

2. Methods

2.1. Participants and setting

The sample included 143 children and adolescents aged 10 to 17 years with chronic pain and a participating caregiver. The trial is registered (Clinicaltrials.gov Identifier NCT03332563), and the protocol is published,[34] which provides full details on the study design. Participants were recruited from 8 clinics (5 pain clinics and 3 specialty gastroenterology clinics) at 5 children's hospitals across the United States: Seattle Children's Hospital (gastroenterology clinic and pain clinic), Children's Mercy Medical Center (abdominal pain clinic and pain clinic), C.S. Mott Children's Hospital (pain clinic), Nationwide Children's Hospital (pain clinic), and Connecticut Children's Medical Center (gastroenterology clinic and pain clinic). This study was approved by the institutional review board at the primary site (Seattle Children's Research Institute), and approvals were obtained from collaborating study sites to refer patients to the study.

2.2. Participant inclusion criteria

To study effectiveness in real-world clinic settings, inclusion criteria for participants were purposefully broad to enhance external validity. All youth were new patients receiving evaluation at 1 of the 8 participating clinics and were (a) aged 10 to 17 years, (b) had chronic pain defined as pain present for at least 3 months, and (c) had access to a smartphone (iOS or Android) and/or web-enabled device (eg, laptop, computer, or iPad). Exclusion criteria were (a) non-English speaking, (b) presently in a psychiatric crisis (eg, recent inpatient psychiatric admission and suicide attempt), and (c) unable to read at the fifth grade level.

2.3. Recruitment

Providers at referring centers gave potential participants a flyer about the study and asked whether they would be willing to be contacted by phone by study staff to undergo additional screening. Providers then transferred referral information through a secure study web site or email. Potential participants could also contact study staff directly by calling a toll-free number provided on the study flyer. Study staff not involved in clinic randomization/treatment allocation screened potential participants through phone and, if eligible, obtained verbal parent consent and child assent for study participation. Recruitment took place from November 2017 until November 2018; final follow-up data collection was completed in June 2019.

2.4. Trial design

To evaluate intervention effectiveness, we used an SW-CRT design where clinic served as the unit of randomization.[18] Given the goals of this hybrid effectiveness-implementation trial, an SW-CRT design provided several advantages over alternatives such as a parallel-group RCT, including (1) enhanced external validity, and (2) the ability to simultaneously test intervention effectiveness and collect implementation data.[8] This design included unidirectional and random sequential crossover of the clinics from control (“nonexposure”) phase to intervention (“exposure”) phase until all clinics were exposed (November 2017-November 2018). There was an initial control period during which none of the clinics were exposed to the intervention and all participants who were referred to the study received usual care alone at their respective clinic. Subsequently, clinics were randomized to 1 of 4 waves (occurring at 2, 4, 6, and 8 months after study commencement) to cross from the control/nonexposure phase to the intervention/exposure phase, with 2 clinics switching at each wave.

This was an open cohort study in which patients were recruited throughout the study duration, but patients were not allowed to switch exposure status. Patients referred during their clinic's exposure phase received the active intervention, which included access to the WebMAP Mobile app for teens and WebMAP parent web site to learn pain self-management skills, in addition to receiving usual care at their referring clinic. Participants referred during their clinic's nonexposure phase received usual care at their referring clinic and were not given access to the WebMAP interventions even after the clinic switched to active intervention phase. Thus, all youth in the study received usual care, which was not altered by study participation. All the clinics provided interdisciplinary care and youth received at minimum an initial evaluation visit with treatment recommendations, which could include referrals for other services (psychology, physical therapy, and acupuncture) and clinic follow-ups as needed. Recruitment into the trial ended as planned per our trial protocol in November 2018.

2.4.1. Maintenance period

From November 2018 to August 2019, all clinics had access to the treatment and could refer patients who had not enrolled in the study to the active intervention (the WebMAP Mobile app). During this 8-month period, we collected 2 outcomes: a provider survey and the number of referrals made by providers to the WebMAP Mobile app. We created clinic-specific login codes, so that we could track the number of users from each clinic. At the end of the trial, the WebMAP Mobile app was then made freely available to the public (in English-speaking countries) through the iPhone and Android app stores in September 2019.

2.5. Randomization

A biostatistician not involved in any other study procedures generated the randomization schedule of the clinics assigned to cross from the usual care control/nonexposure period to the active/exposure period at each of the 4 waves. The randomization schedule was stored in an encrypted document that was only accessible to study staff responsible for intervention procedures; this document was not accessible to study staff involved in recruitment or outcomes assessment. Participants were informed of their assignment to usual care or active intervention after completing the pre-treatment assessment. Because we used a usual care/nonexposure period, blinding of participants to their intervention group assignment was not possible.

2.6. Participant flow

Figure 1 shows our CONSORT diagram depicting flow of participants through each phase of the study. A total of 249 referrals were received from the 8 clinics. Of those families who were referred to participate, 33 declined because of lack of time or interest and 30 were unable to be reached. This left 186 participants who were contacted and assessed for eligibility by study staff, of which an additional 43 participants were excluded (n = 23 declined, n = 15 did not complete pre-treatment assessments or were unable to be reached after screening, and n = 5 did not meet inclusion criteria). Thus, the final sample consisted of 143 patients and a participating caregiver.

Figure 1.

Consort diagram of participant flow. The primary analysis was intention to treat (ITT) and included all subjects.

Seventy parent–youth dyads were allocated to the Usual Care only condition because these patients entered the study during the “nonexposure” phase at their referring clinic; the remaining 73 parent–youth dyads were allocated to the WebMAP intervention condition (+Usual Care) because their referring clinic had crossed over to the “exposure” phase at the time of study entry. Post-treatment assessments were available for 92.3% (N = 132) of the youth sample (86.3% WebMAP and 98.6% Usual Care); and 3-month follow-up was available for 91.6% (N = 131) of the sample (86.3% WebMAP and 97.1% Usual Care) (Figure 1). The full sample was included (N = 73 WebMAP; N = 70 Usual Care) in intention-to-treat analyses.

2.7. Procedures

2.7.1. Outcome assessments

Study assessments were completed by youth and parents online through Research Electronic Data Capture,[17] a secure web-based data collection platform, at baseline before treatment allocation (T1), after completion of the 8-week intervention (immediately after treatment) (T2), and at 3-month follow-up (T3). As part of their assessments, youth completed online diaries through Research Electronic Data Capture for 7 days at each assessment period. Study staff contacted youth about completion of assessments only; both the app and web program include automated reminders to log in and complete intervention tasks. We provided gift card incentives after completion of each of the 3 study assessments.

2.7.2. Implementation strategy

The implementation method targeted provider referrals to the app. Several implementation strategies were used, including (1) preparing the intervention programs for wide-scale dissemination, (2) identifying site champions, (3) establishing site communication, and (4) engaging individual providers. The original, Internet-based version of the WebMAP program delivers treatment to youth and their parents through separate web sites.[32,33] To facilitate large-scale dissemination of the intervention, we developed a smartphone application for youth called WebMAP Mobile. Youth randomized to the treatment arm received access to WebMAP Mobile, while their parents received access to the WebMAP parent web site to learn pain self-management skills. One champion was identified at each site who served as the point person for training providers on the study procedures and how to refer individual patients. A PowerPoint presentation about the study was shared with each champion, and access to the WebMAP programs was provided to each site with test accounts. Instructions on how to complete and return patient referrals (ie, through email or a secure online referral web site) were sent regularly to each site. Site communication included contact with individual providers by study staff throughout the trial by sending a thank you email each time they made a referral and by sending monthly email newsletters. The newsletters included updates about study progress including recruitment and enrollment. As a site engagement strategy, we also conducted periodic referral contests (eg, number of referrals in a month) and provided clinics with small tokens of appreciation (eg, popcorn).

2.7.3. Treatment conditions

2.7.3.1. Usual care control condition (nonexposure phase)

As part of their routine care, all enrolled participants received an initial evaluation for chronic pain at their referring specialty or pain clinic and subsequently received recommendations for treatment (eg, physical therapy, psychology, and medication management). For this clinical trial, we did not alter usual care as recommended by each participant's referring clinic. Youth and parents who entered the study during their referring clinic's usual care/nonexposure phase continued with usual care and did not receive access to the WebMAP intervention.

2.7.3.2. WebMAP intervention condition (exposure phase)

In addition to usual care, youth who entered the study during their referring clinic's active intervention/exposure phase received access to the mobile app version of the program, called WebMAP Mobile, available on Android and iOS operating systems. Their parents also received access to the web-based version of the WebMAP parent program (evaluated and described in Ref. 33). For this trial, we chose to deliver treatment to youth through a mobile app version of the program rather than a web site to facilitate dissemination of the program to youth in real-world settings. The design and treatment content of both the mobile and web-based versions of WebMAP follow cognitive-behavioral, social learning, and family systems frameworks.

2.7.3.2.1. Youth intervention: WebMAP Mobile

The WebMAP Mobile app was adapted from the web site version of WebMAP, which we have previously described in the published protocol for this trial and contains screenshots of the app.[34] WebMAP Mobile is an interactive, self-guided smartphone application that includes 4 functional components: (1) treatment modules (places/lessons), (2) skills library (audio clips, videos of peers, and infographics for skills practice), (3) daily check-in (record and track pain, sleep, activity, and mood), and (4) skills tracker (record and track skills practice, eg, relaxation). Badge and reward systems were included to encourage engagement and skills practice.

Youth visit different “places” (such as the desert, an island paradise, or the northern lights) to learn skills for managing chronic pain. These “places” corresponded with 6 core treatment modules and 2 supplementary modules. Core modules included (1) pain education, (2) stress, emotions, and thoughts, (3) relaxation and imagery, (4) lifestyle (including sleep) and school interventions, (5) staying active, and (6) maintenance and relapse prevention. This is a tailored treatment, such that youth can receive 2 supplementary modules for problems with low mood and insomnia. Specifically, when setting up their initial profile within the app, youth completed the Patient Health Questionnaire-2[1] to screen for depressive symptoms; youth scoring 3 or higher were provided the supplementary module to address negative mood. Youth also completed 2 items from the Adolescent Sleep Wake Scale[11] to screen for (1) difficulties with falling asleep and (2) staying asleep, with endorsement of “quite often” or greater for either symptom triggering release of a supplementary module targeting insomnia symptoms. To facilitate engagement and skills acquisition, youth review their knowledge at the end of each module with brief quizzes and complete an assignment corresponding with the skills taught in each module (eg, relaxation practice). Upon completion of assignments, youth receive a badge and receive access to the subsequent module.

Content is metered according to a time schedule, requiring youth to spend 5 days on skills practice before an assignment can be completed. This was intended to space out treatment delivery and provide opportunity for skills acquisition. Components of the program can be used daily (eg, check in, skills practice). Total treatment duration is approximately 15 to 20 minutes per week over 6 to 8 weeks depending on the number of supplementary modules assigned. The app generated daily reminder notifications to encourage users to log in and complete intervention tasks. Study staff did not contact participants to encourage individual adherence to the treatment.

2.7.3.2.2. Parent intervention: WebMAP Internet Program

During the intervention phase, parents received access to the WebMAP Internet-based parent program, which was previously developed and evaluated by our team.[33] The 8 parent modules include (1) education about chronic pain, (2) recognizing stress and negative emotions, (3) operant strategies I (using attention and praise to increase positive coping), (4) operant strategies II (using rewards to increase positive coping; strategies to support school goals), (5) modeling, (6) sleep hygiene and lifestyle, (7) communication, and (8) maintenance and relapse prevention. Similar to the app, the WebMAP parent web site includes weekly behavioral assignments to facilitate skills practice, as well as vignettes, videos, illustrations, and reinforcing knowledge quizzes.

Parents were asked to complete 1 module per week (15-20 minutes each in duration) over the 8-week intervention period. Parents received automated weekly reminder notifications through email to use the program. Study staff did not use any strategies to encourage individual compliance with the treatment.

2.7.4. Measures

2.7.4.1. Pre-treatment participant characteristics

Parents reported on the youth's age, sex, and race, annual household income, and parent education status. Parents also reported on whether the youth had a chronic disease (eg, arthritis and sickle cell disease) and listed current prescription medications for pain. Youth reported on their pain location, duration of their pain condition (in months), and their pain frequency over the past 3 months. Youth completed the PROMIS Pediatric Emotional Distress scales[22] to assess depression and anxiety symptoms over the past 7 days. The 8-item scale of anxiety symptoms and 8-item scale of depressive symptoms are scored on a 4-point scale (1 = never and 5 = almost always). A score of 50 represents the mean of the US general population with a standard deviation of 10 points. T scores above 60 indicate elevated anxiety and depressive symptoms.

2.7.4.2. Effectiveness measures

For this primary report, we report on 3 clinical effectiveness outcomes: pain-related disability (primary outcome), pain intensity, and global impression of change. There were several other secondary youth and parent outcomes (eg, depression, anxiety, and insomnia) collected in the trial. Because of the complexity of the design and the aim to also evaluate implementation outcomes, we limit the effectiveness outcomes presented. However, we may present these in subsequent secondary data analysis reports.

2.7.4.2.1. Pain-related disability

Our primary effectiveness outcome measure was pain-related disability, assessed using the Child Activity Limitations Interview-9 (CALI-9 diary version), a prospective measure that evaluates perceived difficulty completing typical daily activities because of pain.[20] Youth rated the difficulty of 9 daily activities (eg, walking 1 or 2 blocks, doing schoolwork, and doing things with friends) on a 5-point scale (0 = “no difficulty” and 4 = “extreme difficulty”) once per day over 7 days. Consistent with established scoring procedures,[20] daily ratings for the 9 activity items were linearly transformed into a 0 to 100 scale (0 = 0, 1 = 25, 2 = 50, 3 = 75, and 4 = 100) and then averaged to create daily disability scores (0-100). We used mean daily disability scores across the 7-day period in analyses, with higher scores indicating greater pain-related disability.

2.7.4.2.2. Pain intensity

Youth also reported on daily pain intensity through the 7-day daily diary at each assessment period using an 11-point numerical rating scale (0 = “no pain” and 10 = “worst pain possible”).[40] We used mean pain intensity ratings across the 7-day diary period in analyses.

2.7.4.2.3. Patient global impression of change

At immediate post-treatment and 3-month follow-up, youth completed a patient global impression of change (PGIC)[12] consisting of the following, single-item, “Since the start of the study my overall status is…” with 7 response options ranging from 1 = “no change (or condition has gotten worse)” to 7 = “a great deal better, and a considerable improvement that has made all the difference.”

2.7.4.2.4. Adverse events

Study staff tracked participant spontaneous reports of unanticipated problems and adverse events at the post-treatment and follow-up assessments.

2.7.4.2.5. Treatment engagement

Exposure to (engagement with) the WebMAP interventions was computed using data automatically recorded by the app and web program: (1) number of youth who downloaded the app onto their phone, (2) number of parents who logged in to the web program, and (3) number of completed treatment modules by youth and parent dyads.

2.7.4.3. Implementation measures

Our implementation outcomes were guided by the RE-AIM public health impact framework.[9] The goal of RE-AIM is to help guide program planners, evaluators, funding agencies, and researchers to develop and assess interventions considering external validity and sustainable adoption and implementation to ultimately disseminate effective, generalizable, evidence-based interventions.

2.7.4.3.1. Reach

Reach was measured by (1) tracking the percentage of patients giving consent to participate in the study out of the eligible patients referred (to determine reach of this study) and (2) treatment acceptability rated on the Treatment Evaluation Inventory (TEI),[24] by children and parents who were randomized to receive the WebMAP program (to determine potential future reach of the intervention). The TEI is a treatment acceptability measure; scores above 27 indicate that patients perceive the treatment to be at least “moderately acceptable.[24] For our analyses, we computed the percentage of participants with TEI scores above 27.

2.7.4.3.2. Adoption

Adoption was computed as the number of clinics that referred patients to the study vs the number of clinics that agreed to participate in the study.

2.7.4.3.3. Implementation

We assessed implementation by evaluating provider attitudes about the intervention using a 6-item survey (eg, “I think my patients would benefit from this app”) with response options that ranged from “strongly disagree” to “strongly agree.” We also compared our original budgeted costs vs actual costs incurred to develop the WebMAP Mobile app and make it freely available to the public.

2.7.4.3.4. Maintenance

Maintenance was computed as (1) the percentage of clinics agreeing to continue referring patients to the intervention (ie, agreeing to continue to use the app in clinic) and (2) the percentage of clinics actually providing referrals of their patients to the app during the maintenance period, which was tracked by unique clinic-specific log in codes.

2.7.4.4. Sample size and power analysis

Sample size estimates were based on linear mixed effects (LME) models and adjusted for the clustering of patients within clinics, using the intraclass correlation coefficient (ICC) derived from the study team's previous multisite investigation,[33] ICC = 0.02 for this estimate. The effect size estimate was derived from the pain-related disability outcome in our previous multisite RCT of WebMAP (d = 0.25).[33] Minimal attrition was expected in the study sample based on previous clinical trials of digital health interventions conducted by our research group that have consistently achieved follow-up completion rates >90% at 6-month follow-up with similar populations.[32,33] With 8 clinic sites referring on average 16 patients per clinic into the study, using an ICC of 0.02, and assuming a 2-tailed alpha of 0.05, a total sample size of N = 128 subjects provides 80% power to detect an effect size of 0.25 on the primary outcome of pain-related disability in mixed linear models for examining differential changes between the 2 treatment conditions.

2.7.4.5. Data analysis plan

We performed intention-to-treat analysis of all participants according to the exposure status of the clinic from which a participant was referred. All data analyses were conducted using IBM SPSS (Version 25 for Windows). We calculated engagement and implementation outcomes using frequencies and descriptive statistics. We summarized demographic characteristics using descriptive statistics, including frequencies for categorical variables and mean and SD for continuous variables. Our main analysis was based on LME regression models, which allowed us to use all available data from participants. A participant may have missing outcomes values at 1 or more time points, but as long as the participant had at least 1 outcome value at any of the assessment time points, the data were retained and used in the analysis, which can ameliorate selection bias due to drop out. Overall, missing data were minimal on the primary and secondary outcomes, ranging from 0.7% to 1.4% at baseline, 11.2% to 16.1% at post-treatment, and 9.1% to 20.3% at 3-month follow-up.

To test effectiveness of the intervention, we applied LME regression models to examine the effect of the WebMAP intervention vs usual care only on pain-related disability and pain intensity. These models included a binary intervention group indicator (ie, whether their clinic was in the usual care control/nonexposure phase or active treatment/exposure phase at the time the participant enrolled in the study; 0 = control; 1 = active treatment), a 3-level time indicator representing the outcome assessment time point (baseline, immediate post-treatment, and 3-month follow-up), a 4-level period indicator representing wave at study entry (wave 1, wave 2, wave 3, and wave 4), and the interaction term between intervention group × time indicator. Our primary parameter of interest was the intervention group × time indicator interaction term, which evaluated whether change in the outcome over time differed between the 2 treatment groups. We report the beta coefficient, P-value, and effect size for each interaction term. Significance tests were based on restricted maximum likelihood estimates and Kenward–Roger degree-of-freedom approximation.[25] Because patient-reported global impression of change was measured at only 2 time points (immediate post-treatment and 3-month follow-up), we conducted 1-way analyses of covariance controlling for wave at study entry to determine group differences on this outcome at each time point. We calculated effect sizes (Cohen's d) to elucidate the magnitude of treatment effects. By convention, d = 0.20, d = 0.50, and d = 0.80 are interpreted as small, moderate, and large effects, respectively.[5]

Because engagement in the WebMAP program was expected to be variable in this effectiveness trial, we also examined the effects of treatment engagement on treatment response from pre-treatment to 3-month follow-up for families assigned to the WebMAP condition (n = 73). For these analyses, we applied LME regression models comprising treatment engagement (number of modules completed by parent–youth dyads), time (baseline, post-treatment, and 3-month follow-up), and slopes of improvement (treatment engagement × time). To display significant engagement × time interaction effects graphically, we calculated parameter estimates at high (+1 SD above the mean) and low (−1 SD below the mean) values of treatment engagement.[2] This technique uses all the data in the LME model to calculate model predicted parameters for different levels of engagement, allowing for an understanding of the trajectory of disability and pain intensity among individuals with high and low engagement.

3. Results

3.1. Participants

Participants were 143 youth aged 10 to 17 years (M = 14.5; SD = 1.9) and their parents. Participants were primarily female and Anglo-American (Table 1). About half (53%) reported a household annual income above $70,000. Most parents had completed a college degree. Youth were referred from pain clinics (78.3%) or gastroenterology clinics (21.7%) and had chronic pain that occurred daily (73.2%), that was moderate to severe in intensity (M = 5.9, SD = 1.8), and that had been present on average 4 years before study enrollment (M = 47.8 months; SD = 42.4). Pain-related disability was moderate (M = 37.5, SD = 22.0). A subset of youth (14.3%) had a comorbid chronic disease such as arthritis, chronic pancreatitis, or sickle cell disease. Medication use was common with 25.9% of youth using 3 or more prescription medications for pain; very few youth (n = 6, 4.2%) were prescribed opioid medication. A subset of youth had elevated anxiety (36%) and depression (35%) symptoms on the PROMIS Emotional Distress scales. Independent t tests and χ2 tests indicated that the 2 treatment groups were equivalent across demographic and clinical characteristics at baseline (p's > 0.05; Table 1).

Table 1

Baseline characteristics of the sample, overall and by arm.

	Total sample (N = 143)	Treatment (N = 73)	Control (N = 70)
Demographic characteristics
Child age (M/SD, y)	14.5 (1.9)	14.4 (2.0)	14.6 (1.8)
Child sex (%/n, female)	81.8 (117)	84.3 (58)	79.5 (59)
Child race (%/n)
White	79.3 (111)	80.3 (57)	78.3 (54)
Hispanic	10 (14)	8.5 (6)	11.6 (8)
Black or African American	2.1 (3)	2.8 (2)	1.4 (1)
Asian	2.1 (3)	2.8 (2)	1.4 (1)
American Indian/Alaska Native	0.7 (1)	1.4 (1)	0 (0)
Multiracial	5.7 (8)	4.2 (3)	7.2 (5)
Annual household income (%/n, $)
<10,000	2.1 (3)	0 (0)	4.3 (3)
10,000-29,999	11.9 (17)	12.3 (9)	11.4 (8)
30,000-49,999	14.1 (20)	16.4 (12)	11.4 (8)
50,000-69,999	19.0 (27)	19.2 (14)	18.6 (13)
>70,000	52.8 (75)	52.1 (38)	52.9 (37)
Parent education (%/n)
High school or less	9.9 (14)	11.1 (8)	8.7 (6)
Vocational school/some college	21.3 (30)	22.2 (16)	20.3 (14)
College	44.7 (63)	45.8 (33)	43.5 (30)
Graduate/professional school	24.1 (34)	20.8 (15)	27.5 (19)
Clinical characteristics
Referral site (%/n)
Pain clinic	78.3 (112)	82.2 (60)	74.3 (52)
Specialty clinic	21.7 (31)	17.8 (13)	25.7 (18)
Chronic disease (%/n, yes)	14.3 (20)	11.3 (8)	17.4 (12)
Anxiety symptoms (%/n, yes)	35.7 (51)	35.6 (26)	35.7 (25)
Depression symptoms (%/n, yes)	35.0 (50)	38.4 (28)	28.6 (20)
Pain duration (M/SD, mo)	47.8 (42.4)	48.2 (46.8)	47.3 (37.8)
Pain frequency (%/n)
1x per week or less	5.6 (8)	8.2 (6)	2.9 (2)
2-6x per week	21.1 (30)	24.7 (18)	17.4 (12)
Daily	73.2 (104)	67.1 (49)	79.7 (55)
Pain location (%/n)
Headache	53.1 (76)	53.4 (39)	52.9 (37)
Abdominal pain	52.4 (75)	54.8 (40)	50.0 (35)
Musculoskeletal pain	81.1 (116)	83.6 (61)	78.6 (55)
Multisite pain	77.6 (111)	80.8 (59)	74.3 (52)
Prescription medications
# of medications (M/SD, range 0-5)	1.6 (1.4)	1.4 (1.3)	1.7 (1.5)
3+ medications (%/n)	25.9 (36)	19.7 (14)	32.4 (22)
Opioid medication (%/n)	4.2 (6)	1.4 (1)	7.1 (5)

For pain location, total percentage is greater than 100% because participants were able to endorse more than 1 pain location. Multisite pain was coded as 1 = yes, 0 = no if teen endorsed more than 1 pain location. Depression and anxiety were coded as 1 = yes, 0 = no if teen's T score was above 60 on the PROMIS scale. Missing data: parent education (n = 2), household income (n = 1), and child race (n = 3).

3.2. Engagement with the WebMAP intervention

Of the 73 youth assigned to the WebMAP group, 68 youth (93%) downloaded the WebMAP Mobile app and 54 youth (74%) completed at least 1 module of the intervention. Of the 73 parents assigned to the WebMAP group, 60 parents (82%) logged in to the web program and completed at least 1 module. Youth completed an average of 3.1 modules (SD = 2.8; range = 0-8), and 27% of youth completed the entire intervention program. Similarly, parents completed an average of 4.2 modules (SD = 3.1; range 0-8), and 27% of parents completed all of the intervention. Across parent–teen dyads, an average of 7.3 modules (SD = 5.3, range = 0-16) were completed; 27% of dyads completed the full treatment, while 12% did not access any of the intervention.

3.3. Effectiveness

Mean and SD for pain-related disability and pain intensity at pre-treatment, immediate post-treatment, and 3-month follow-up are presented in Table 2. We found that pain-related disability and pain intensity improved over time in both treatment conditions. We did not detect a statistically significant effect of the treatment group (WebMAP vs Usual Care) on change over time in pain-related disability from baseline to post-treatment (b = −0.56, P = 0.87, 95% CI [−7.11 to 6.00], d = −0.08) or baseline to 3-month follow-up (b = 0.96, P = 0.78, 95% CI [−5.75 to 7.66], d = 0.03). The pattern of results was similar for pain intensity, where we also did not find a statistically significant effect of the treatment group on change over time from baseline to post-treatment (b = 0.31, P = 0.26, 95% CI [−0.23 to 0.85], d = 0.11) or baseline to follow-up (b = −0.24, P = 0.38, 95% CI [−0.79 to 0.30], d = −0.14).

Table 2

Unadjusted descriptive statistics for effectiveness outcomes by the treatment group.

	WebMAP program (n = 73), M (SD)			Usual care control (n = 70), M (SD)			Between-group difference over time
	Baseline	Post-treatment	3-month follow-up	Baseline	Post-treatment	3-month follow-up	Post-treatment		3-month follow-up
							P	d	P	d
Pain-related disability	35.7 (20.1)	34.9 (25.4)	34.1 (21.8)	39.3 (24.0)	37.8 (25.7)	35.1 (27.7)	0.87	−0.08	0.78	0.03
Pain intensity	5.6 (1.9)	5.8 (1.9)	5.3 (1.9)	6.4 (1.7)	6.1 (2.1)	6.2 (1.8)	0.26	0.11	0.38	−0.14
Global impression of change	—	3.9 (1.8)	4.2 (1.7)	—	2.9 (1.8)	3.4 (2.0)	<0.01	0.54	0.01	0.44

On the PGIC, we found a statistically significant moderate-sized intervention effect at immediate post-treatment and 3-month follow-up (F(1, 122) = 8.60, P = 0.004, d = 0.54; F(1, 127) = 6.32, P = 0.01, d = 0.44, respectively). At immediate post-treatment, greater improvement in overall status due to treatment was reported by youth in the WebMAP group (M = 3.9, SD = 1.8, 95% CI = 3.43-4.38) compared with youth in the Usual Care group (M = 2.9, SD = 1.8, 95% CI = 2.5-3.4). Youth in the WebMAP group continued to report greater treatment benefit at 3-month follow-up (M = 4.2, SD = 1.7, 95% CI = 3.8-4.7) compared with the Usual Care group (M = 3.4, SD = 2.0, 95% CI = 3.0-3.9).

Consistent with our hypothesis, greater treatment engagement (total number of modules completed by parent–youth dyads) was associated with greater improvement on our primary outcome of pain-related disability from pre-treatment to 3-month follow-up (b = −1.27, SE = 0.53, P = 0.02). In Figure 2, we graphically display the estimated marginal means of pain-related disability × treatment engagement from baseline to 3-month follow-up. Relative to families with lower engagement (M − 1 SD), those with higher engagement (M + 1 SD), demonstrated significantly greater decreases in pain-related disability (higher engagement: b = −8.39, SE = 3.50, P = 0.02, d = −0.38; lower engagement: b = 4.94, SE = 4.20, P = 0.24, d = 0.27), and this was a small effect size. Greater treatment engagement was also associated with greater improvement on our secondary outcome of pain intensity at 3-month follow-up (b = −0.10, SE = 0.04, P = 0.02). Specifically, families with higher treatment engagement (M + 1 SD) had significantly greater reduction in pain intensity than families with lower engagement (M − 1 SD) (higher engagement: b = −0.94, SE = 0.28, P < 0.01, d = −0.57; lower engagement: b = 0.10, SE = 0.33, P = 0.77, d = 0.05), and this was a moderate effect size.

Figure 2.

Higher dyadic treatment engagement is associated with greater reduction in pain-related disability compared to lower treatment engagement. Note. Treatment engagement was defined as the total number of modules completed by youth and their parents (range = 0-14 across the dyad). Figure 2 displays estimated marginal means of pain-related disability from baseline to 3-month follow-up when (A) dyadic engagement was higher (mean + 1 SD), and (B) dyadic engagement was lower (mean − 1 SD).

3.3.1. Adverse events

Participants did not report any treatment-related adverse events at post-treatment or follow-up.

3.4. Implementation outcomes using RE-AIM

Implementation results from our trial using the RE-AIM framework are summarized in Table 3.

Table 3

Summary of results using the RE-AIM framework.

RE-AIM domain	Operational definition	Result
Reach	Proportion and representativeness of individuals willing to receive the intervention.Treatment acceptability by teens and parents.	Sample characteristics were similar to other trials of youth with chronic pain. Enrollment goal was exceeded.Majority of families (86% youth; 89% parents) rated the WebMAP program as moderately to highly acceptable.
Effectiveness	Short- and long-term effects of the WebMAP program compared with Usual Care on pain-related disability (primary outcome), pain intensity, and patient global impression of change (secondary outcomes).Within the WebMAP group, examination of the effect of treatment engagement (parent–youth module completion) on treatment benefit (improvement in pain-related disability and pain intensity).	No statistically significant treatment group effect identified on pain-related disability or pain intensity at post-treatment or 3-month follow-up.Youth in the WebMAP group reported greater improvement in their overall status at post-treatment and follow-up compared with youth in the Usual Care groupIn the WebMAP group, youth in families with higher treatment engagement achieved greater reductions in pain-related disability and pain intensity at follow-up.No adverse events were reported.
Adoption	Percentage of clinics agreeing to participate in the study.Percentage of participating clinics referring patients to the study.	100% of invited clinics agreed to participate and made referrals during the trial period (range of referrals per clinic: 6-80)
Implementation	Percentage of providers reporting a positive experience with the intervention.Budgeted vs actual costs to develop and release WebMAP Mobile.	Most providers reported a positive experience with the intervention.Actual costs exceeded the original budget by 7%.
Maintenance	Percentage of clinics that agreed to continue referring patients to WebMAP Mobile.Percentage of clinics who referred patients to WebMAP Mobile and the number of patients who downloaded WebMAP Mobile during the 8-month maintenance period.	All clinics (8/8) agreed to continue to refer patients to the WebMAP Mobile app.During the maintenance period, most clinics (7/8) referred patients to the app and 56 new patients downloaded the app.

3.4.1. Reach

We received 181 referrals and 143 parent–teen dyads consented to participate in the trial (79% referral/enrollment rate, which is consistent with our previous multisite trial of WebMAP[19]). Our final sample size of 143 youth and their parents was 19% above our original planned enrollment goal of 120. Consistent with our previous trials of WebMAP,[32,33] we found 85.7% of youth and 88.5% of parents rated the WebMAP program as moderately to highly acceptable on the TEI, suggesting that the existing program has reasonable potential for future reach.

3.4.2. Adoption

We invited 8 clinics to participate in this study. All 8 clinics agreed to participate, and all the clinics referred patients to the trial. The range of referrals per clinic was 6 to 80, and the range of enrolled participants per clinic was 4 to 44.

3.4.3. Implementation

Twenty-seven of 58 providers (47%) completed the online provider survey about their experience with the WebMAP program. Approximately 100% of providers responded to the following 3 items with “agree” or “strongly agree”: “I believe pain management apps like WebMAP are a helpful tool to provide cognitive and behavioral skills for my patients,” “I believe my patients would benefit from using WebMAP, and “It improves the quality of care for patients.” For the remaining 3 items, 93% of providers responded with “agree” or “strongly agree” and 7% responded with “neutral”: “It helps our clinic better use resources”, “It fills an important need in my clinic”, and “I will encourage my patients to use WebMAP Mobile after the study is over.”

We also examined the actual costs to develop WebMAP Mobile and make it freely available to the public vs our original budgeted costs. We found that the actual costs exceeded the original budget by 7%; this overage was due to unanticipated costs incurred to perform the public release of the app.

3.4.4. Maintenance

After we closed enrollment into the trial, all the participating clinics (8/8) agreed to continue referring the WebMap Mobile app to their patients. Over the subsequent 8-month maintenance period, we found that 87.5% of the participating clinics (7/8) referred the WebMAP Mobile app to their patients (range 0-26 patients referred/clinic). Fifty-six new patients downloaded the app during the maintenance period.

4. Discussion

Building on our previous work delivering Internet-based pain self-management to youth with chronic pain,[32,33] we conducted a hybrid effectiveness-implementation trial to facilitate large-scale dissemination of the WebMAP intervention program. In our first phase of work, we devised an implementation approach to target provider referrals to the intervention and converted our web-based intervention into a smartphone application for youth (WebMAP Mobile) to allow it to be more readily disseminated. We trained providers and clinics in making study referrals and implemented procedures to increase provider and clinic engagement. In the second phase of work, we conducted a hybrid effectiveness-implementation trial using a stepped-wedge design to evaluate implementation and effectiveness outcomes and intervention use in real-world clinic settings, without tightly controlled study monitoring to influence patient compliance. In the third phase of work, we released the app to the clinics to use with their patients in a maintenance period, followed by a public release of the app.

Our first aim was to evaluate the real-world usage and clinical effectiveness of WebMAP Mobile for youth with chronic pain. Our findings demonstrated that the intervention could be delivered in real-world clinic settings through provider referral in interdisciplinary pain clinics and pediatric specialty clinics. Intervention usage by individual patients and families was variable; most youth downloaded and tried the WebMAP Mobile app and parents logged into the WebMAP parent web site, but only about 30% of parents and youth completed the treatment program. Across the treatment and usual care conditions, pain and disability decreased at a similar rate at post-treatment and follow-up. However, youth in the WebMAP treatment group indicated greater improvements on the PGIC compared with the control group, with small to moderate effect sizes at post-treatment and follow-up. Thus clinical effectiveness at the group level was only weakly demonstrated. However, within the treatment group, higher levels of intervention use/engagement were associated with significantly reduced pain and disability, and these were small to moderate effects, indicating that use of the treatment program was associated with clinical benefit.

We used the PGIC in our trial, which, although widely used in adult clinical care and clinical trials, has had very limited use in pediatric chronic pain clinical trials, despite identification of global impression of change as a key outcome domain.[29] For example, in a recent systematic review of design characteristics of trials of chronic and recurrent pain conditions in children and adolescents,[6] none of the 52 included studies reported on global impression of change as an outcome. Future studies are needed to understand the relationship between PGIC and other pain outcomes in pediatric patients with chronic pain and to understand how to integrate the patient's perception of treatment effectiveness.

Clinical effectiveness of digital health interventions is interwoven with user engagement since these are often self-administered interventions, which rely on patient motivation to be exposed to treatment and to then make behavioral changes.[38] In our previous tightly controlled RCTs of the WebMAP program, we achieved high levels of engagement with treatment completion around 75% and parent–teen dyads completing 14 of 16 treatment modules on average”; we also demonstrated efficacy on pain-related disability and multiple secondary outcomes.[33] By contrast, in this effectiveness study where we broadened the inclusion criteria and did not use staff to influence participant adherence to treatment, engagement with the WebMAP program was modest: Parent–teen dyads completed 7.3 of 16 treatment modules on average and about 30% completed treatment. Indeed, systematic reviews have demonstrated relatively low engagement in digital mental health interventions in teens, with most studies reporting completion of less than half of intervention components and high rates of study attrition.[16] The issue of engagement has received a great deal of attention and includes concerns about whether engagement with the digital intervention is too limited to support behavior change and whether there are demographic or clinical characteristics that can be used to prospectively identify individuals at risk of lower engagement with digital health technologies (eg, mental health comorbidity, computer literacy, social/physical environment, etc.).[36] More research is needed to define “effective engagement” for the WebMAP program, to develop and test strategies that may enhance engagement, as well as to understand whether the modality (app vs web site) plays any role in treatment effectiveness. Our findings demonstrated a relationship between engagement with treatment and clinical benefit (ie, reductions in pain and disability) supporting the obvious idea that exposure to treatment is needed and that a few periods of engagement may not be sufficient to produce effects that are greater than usual care.

Our second aim was to assess implementation of WebMAP in pediatric specialty care and pain clinic settings through provider referral. We evaluated our implementation approach using the RE-AIM framework[9] because it provides a broad perspective to understand potential public health impact of an intervention. Our findings demonstrated high levels of treatment acceptability as perceived by parents, youth, and providers; moreover, providers had positive attitudes about the helpfulness of a digital health intervention focused on pain self-management and intentions to use the intervention with their patients. During our maintenance period, we demonstrated that 7 of 8 clinics did uptake the intervention with their patients. Our overall costs were mostly on-target with a slight expenditure over budget to ready the WebMAP Mobile app for public release.

To the best of our knowledge, this is the first study using a hybrid effectiveness-implementation design to evaluate a pain management intervention and offers several advantages over a traditional RCT design. Several general concerns have been raised about the parallel-group RCT design including initial sample selection bias and unclear representativeness of samples.[27] Moreover, specific to the psychological treatment literature in chronic pain, significant concerns have been raised regarding the abundance of trials using small sample sizes with confirmatory designs and exclusion of patients with common medical and psychological comorbidities.[10,14] Our trial used broad inclusion criteria, resulting in a more heterogeneous, representative group of youth with medical and mental health comorbidities. Furthermore, there is a lack of studies that have examined organizational/system barriers and facilitators to implementing efficacious psychological interventions for pain management in pediatric settings. Hybrid effectiveness-implementation designs address the limitations inherent in tightly controlled, parallel-group RCTs by promoting external validity and increasing efficiency through a single trial that tests both intervention effectiveness and effectiveness of implementation strategies.[8]

We chose to use a stepped-wedge design, which offered several advantages over a more traditional parallel-group design. Because we already had evidence in support of our WebMAP intervention, we could offer a known effective treatment to clinics. Moreover, all clinics could end the recruitment period in an exposure phase to allow for studying maintenance after the close of enrollment. Most typically, stepped-wedge designs are used with routinely collected data (ie, through the electronic medical record) and eliminate the need for individual participant recruitment. However, because we wanted to use best practice for outcomes assessment, we included patient-reported outcome measures and an individual participant recruitment method.

There are several limitations of the study that should be considered in interpreting the findings. Although our sample reflects the general composition of youth presenting to pain and specialty clinics for chronic pain, it still represents a predominantly white and female population. Our design was limited to short-term follow-up at 3 months, which may not have been enough time to understand continued use of the WebMAP program, as well as maintenance of treatment effects. The amount or type of care received by the participants at their referring clinics is unknown. However, given that all the participants received usual care, irrespective of the study group, and that participants in both groups were referred by all the 8 clinics, significant differences between groups in care are unlikely. Certain implementation outcomes were difficult to evaluate within our study design. For example, we attempted to collect information on clinic patient volumes to understand the potential reach of the intervention, but these data were not interpretable across the clinics because of differences in who they serve (pain vs specialty clinic) and availability of data on patient characteristics (age and chronic pain diagnosis).

Because our objective was to disseminate our intervention to the public at the end of the study, we developed plans during the trial to facilitate a public release. Previous reviews have shown that very few evidence-based digital health tools become publicly available for pain management after research studies are concluded.[19] We achieved our aim of publicly releasing the app, WebMAP Mobile, which is available for Android and iPhone in English to the following countries: United States, Australia, Canada, Ireland, Israel, New Zealand, South Africa, Spain, and the United Kingdom.

Data from this trial provide critical information for future development of strategies to disseminate evidence-based digital health interventions in clinical settings serving youth with chronic pain. Future work is needed to enhance user engagement and to further develop and test implementation strategies including targeting direct to consumer uptake as well as provider referral for disseminating digital health interventions for pain management.

Conflict of interest statement

The authors have no conflicts of interest to declare.

A video abstract associated with this article can be found at http://links.lww.com/PAIN/B103.