In-person vs. web-based administration of a problem-solving skills intervention for parents of children with cancer: Report of a randomized noninferiority trial.

BACKGROUND: Bright IDEAS (BI) problem-solving skills training is an evidence-based intervention designed to help parents manage the demands of caring for a child with cancer. However, the resource intensiveness of this in-person intervention has limited its widespread delivery. We conducted a multicenter, randomized trial with a noninferiority design to evaluate whether a web-based version of BI requiring fewer resources is noninferior to in-person administration.
METHODS: 621 caregivers of children with newly diagnosed cancer were randomly assigned to standard BI delivered face-to-face or a web-based version delivered via mobile device. The primary outcome was caregiver-reported problem-solving skills. The noninferiority margin was defined as 0.2 standard deviation units of the change from baseline to end of intervention. Secondary outcomes included caregiver-reported mood disturbance, depression, and posttraumatic stress symptoms. The study was registered with ClinicalTrials.gov Identifier: NCT01711944.
FINDINGS: The effect of the standard treatment was preserved; parents in the standard BI arm improved their problem-solving (effect size = 0.53, t = 8.88, p < .001). Parents in the web-based BI group also improved their problem-solving (effect size = 0.32, t = 5.32, p < .001). Although the web-based intervention preserved 60% of the standard treatment effect, the test of noninferiority was non-significant (effect size = -0.21, p = 0.55). Similarly, the web-based intervention preserved > 60% of the standard intervention effect on all secondary outcomes; however, tests of noninferiority were non-significant.
INTERPRETATION: Noninferiority of web-based BI relative to standard face-to-face administration was not established. Further development of the web-based BI is needed before it can be recommended as a stand-alone intervention. However, the documented benefits of the web-based intervention as well as the advantages of low resource utilization and ease of delivery suggest that further development of web-based BI is indicated, and that it may play a valuable role in alleviating distress in caregivers of children with serious or chronic illness. FUNDING: National Institutes of Health (U.S.), R01 CA159013 (P.I. Sahler).

Evidence before this study

The Bright Ideas (BI) problem-solving skills intervention has strong empiric support for its benefits in helping parents manage the stresses of childhood cancer, but is resource intensive; thus we developed a web-based version of the intervention and tested this against the empirically supported in-person approach. We anticipated there might be some attenuation of efficacy of an online administration due to reduction in interpersonal attention and support, but reasoned that a minor reduction in efficacy would be acceptable given the capacity for wider dissemination of a web-based intervention requiring fewer resources. Thus, this was designed as a noninferiority trial.

Added value of this study

This randomized controlled trial failed to demonstrate the noninferiority of a web-based implementation in comparison to the standard in-person approach. However, benefits of the web-based intervention were documented, which encourages further tailoring and incorporation of newer technologies to enhance the efficacy of this approach.

Implications of all the available evidence

Considering all available evidence, further development of the web-based BI program is needed before it can be recommended as a stand-alone intervention. Web-based technologies are rapidly developing, and with utilization of new machine learning approaches to more closely mirror a personal therapist, the effects of web-based BI are likely to improve. Future research should investigate enhanced web-based BI, as well as multimodal approaches that combine in-person intervention supplemented with enhanced online materials.

Alt-text: Unlabelled box

Introduction

The initial diagnosis and early treatment of childhood cancer is highly stressful for parents, leaving them at risk for symptoms of depression, anxiety, and posttraumatic stress [1], [2], [3]. The coping and adjustment of parents as they struggle to manage the many challenges of their child's illness may also have a significant effect on child outcomes, not only for the ill child [4], [5], [6] but for healthy siblings as well [7]. Thus, interventions to facilitate parental coping and improve their well-being can have synergistic beneficial effects on all family members who must adapt to a new cancer diagnosis.

The Bright IDEAS (BI) paradigm of problem-solving skills training (PSST) has been demonstrated to be an effective intervention to improve problem-solving skills and decrease negative affectivity (symptoms of mood disturbance, depression, and posttraumatic stress) in mothers of children with recently diagnosed cancer [8], [9], [10], [11]. Data from three multisite randomized trials has demonstrated both the efficacy and specificity of the BI program, which has been recognized by the National Cancer Institute as a Research Tested Intervention Program (RTIP) [12]. Bright IDEAS received outstanding RTIP scores of 4.4 and 5.0 (out of 5) for Research Integrity and Dissemination Capability respectively, but a lower score of 2.0 for Intervention Impact. The primary factors for the low impact score were related to high resource intensiveness (standard BI requires 8 h of face-to-face (F2F) administration by specially trained staff) and the emphasis on mothers of recently diagnosed childhood cancer patients, which limited reach to a relatively small population [12]. In response, we developed an web-based version of the BI intervention appropriate for all caregivers, as a strategy to reduce resource utilization for administration, to extend reach, and to provide 24/7 availability to anyone with Internet access.

Use of internet and mobile technologies has grown rapidly across many areas of medicine, including pediatric oncology [13], [14], [15]. Internet or computer based psychological interventions have been found to be effective for various kinds of problems compared to usual care [16,17]. Such eHealth/mHealth interventions often reach a medium effect size, which approaches that of in-person treatment. Digital health programs are low cost relative to their potential reach, and they provide a variety of benefits to participants and researchers alike, including scale, fidelity, privacy, and ease-of-access. Software is, inherently, much more scale-able than human resources, improving the potential impact of digital interventions, especially in resource-constrained clinical contexts. Apps can be integrated seamlessly into a participant's daily life activities with minimal effort, allowing users to engage with the program on their terms, in their own time.

Despite the benefits of mobile technologies, the effect of internet-based behavioral health treatments appear to be moderated by therapist support, with studies that included access to the support of a therapist showing larger effects than those that do not [16,17]. Although such eHealth/mHealth approaches address many barriers to access of services, there is a potential cost for the loss of in-person support. The specificity of the problem-solving skills training component of the BI intervention, beyond the non-specific effects of providing empathic, supportive contact, was clearly documented in our prior trial [11]. Nevertheless, a proportion of the intervention effect remained attributable to the in-person effects of attention and therapeutic support. Thus we anticipated that some attenuation of efficacy was possible with online administration, but this was considered acceptable, as it would be counterbalanced by the ease of delivery and capacity for widespread dissemination. Therefore, rather than conducting a superiority trial, the current study was designed to test the noninferiority of a web-based version of BI as the candidate intervention in comparison to our standard in-person approach.

Methods

Participants

Caregivers of children with cancer were recruited at 5 sites across the U.S. (St. Jude Children's Research Hospital (SJCRH), University of Texas/MD Anderson Cancer Center (MDACC), Children's Hospital of Los Angeles (CHLA), Children's Hospital of Pittsburgh (CHP), and Texas Children's Hospital (TCH)). Eligibility criteria included: parent of a child diagnosed with any malignancy, and treated with chemotherapy, radiation, and/or surgery; child diagnosed 4–16 weeks prior to enrollment; parent ability to speak and read English or Spanish; child not in medical crisis at time of enrollment (e.g., admitted to ICU). At 3 sites (MDACC, CHLA, TCH), parents who were primarily Spanish-speaking were preferentially recruited, with a goal of 20% of total enrollment across all sites. At SJCRH and CHP, only English-speaking parents were enrolled. In prior trials, recruitment was limited to mothers, given the typically limited availability of fathers [8], [9], [10], [11]. In the current trial, fathers were eligible, and if both parents were equally available, fathers were preferentially recruited, with a goal of including a minimum of 10% fathers in the sample. Recruitment began in October 2013, and the trial was closed to accrual in December 2017.

Procedures

This multicenter RCT used a noninferiority design to compare the Bright IDEAS problem-solving skills training intervention (BI) administered in its standard format, which involves 8 1-hour F2F sessions, to a web-based version of BI requiring significantly less in-person contact. The trial was registered with ClinicalTrials.gov (NCT01711944), and Institutional Review Board approval was obtained at all participating sites. Participants provided written informed consent, and completed a baseline (T1) battery of assessment measures prior to random assignment to treatment arm (face-to-face [F2F] or web-based BI) in a 1:1 ratio, using a block design stratified by site and language. An identical battery of measures was completed at the end of intervention (T2) and 3 months post-intervention (T3). Assessments were completed using the web database, REDcap, with paper and pencil as backup. Participants received modest gift cards for their time ($25 at T1 and T2, $50 at T3).

Interventions

Standard (F2F) BI

This intervention was delivered as in previous trials [8], [9], [10], [11]. BI has been developed as an appealing, easily understood approach to the teaching and coaching of problem-solving skills. A comprehensive manual has been developed to guide the interventionist, along with worksheets and attractive graphic materials for the caregiver. The term “Bright” signifies optimism, and instilling the belief that problems can be solved, which is considered an essential component for successful implementation of the intervention. The acronym ‘IDEAS’ is used as a mnemonic for the 5 essential steps of our problem solving approach, with each letter signifying a step: I (Identify the problem), D (Determine the options), E (Evaluate/choose the best option), A (Act), and S (See if it worked). Problem-solving is presented as a general coping skill applicable to many life challenges, including those commonly faced in parenting a child during cancer treatment. The intervention is administered in 8 1-hour F2F individual sessions. Guided by the manual, the sessions are designed to follow a set sequence: session 1: Rapport building, understanding relevant personal and medical history, introduction of the BI program and worksheets; session 2: Review of the BI program and worksheets, and initial application to a real problem; sessions 3–7: continued application of BI to identified problems and promotion of problem-solving strategies and skills in vivo; and session 8: review of BI principles, relapse prevention, and termination.

Interventions were delivered by research assistants (RAs) with graduate education in clinical psychology or related fields, under the supervision of the site principal investigators (PI's) who were all licensed psychologists. RAs were trained in and delivered both the F2F and web-based formats to minimize differences in personal style. Bilingual RAs worked with Spanish-speaking participants using Spanish language materials. The interventionists were initially trained together as a group before the trial opened to accrual. Site PI's provided weekly supervision to ensure adequate therapeutic delivery and fidelity to the treatment manual. All sessions were digitally recorded for review of treatment integrity (discussed below).

Web-based BI

The web-based intervention was produced in collaboration with web developers from Radiant Digital Corporation. Development followed a user-centered design process, which included a series of formative focus groups to obtain parent perspectives. An initial website prototype was then reviewed by representative users from the study sites. Feedback from this review was used by the developers, and program changes were updated in a stepwise incremental process. Ultimately, the content of the web-based intervention included modeling videos, interactive activities and homework/worksheet tools designed to be similar to the F2F intervention. This included 7 videos presenting the steps of problem-solving via the interaction between a clinician and parent, and 4 fotovellas depicting parents working through typical problems encountered. The web-based program also featured a counselor interface for web-users; this permitted counselors to track progress through the lessons and view completion status/follow-up as necessary. The fotonovelas incorporated interactive skills practice—brief practice exercises for each step of the BI method. The skills practice delivered reinforcement or remediation based on user responses.

An initial meeting between participant and RA provided explanation of BI, and directions for navigating the website on the caregiver's preferred device (lap/desktop, tablet or cellphone). A caregiver without convenient access was loaned an inexpensive device for the duration of his/her participation in the project. Participants were asked to access the site at least weekly and to: (1) view the 7 instructional videos and 4 fotonovellas illustrating common problems; and (2) work through ≥1personal problem(s). Prompt logistical support was available throughout from the developer. Content or intervention questions were emailed to the site RA for response by the next business day. Two weeks after the initial meeting, the RA phoned the caregiver to review web-based usage and address barriers if adherence was suboptimal. After 8 weeks, a second face-to-face session reviewed principles learned, and encouraged ongoing use of BI. After the wrap-up session, the participant completed the T2 assessment. Three months later, participants completed the T3 assessment. Brief questionnaires about the usefulness of the intervention and suggestions for improvement of the site were included.

Treatment integrity

Treatment integrity (TI) assurance began with training of the interventionists as a group, in a meeting that included all study staff and site PI's, and with emphasis on the use of the standardized treatment manual. Ongoing monitoring of TI was conducted by both the site PI's and a TI team. The TI team included the study PI (OJZS) and 4 study staff (2 Spanish-speaking) who had experience with, but were not providing the intervention in this trial. All sessions conducted with those in the F2F BI group, as well as the first and last sessions of those in the web-based group, were digitally recorded and uploaded to a central password-protected server. The first two sessions conducted by each interventionist in each condition (F2F and online) were reviewed by both the site PI and the TI team. Thereafter, 10% of the standard arm sessions, and 20% of the first and last sessions in the online arm, were chosen at random for review by the TI team. Reviewers scored sessions using a rubric developed in prior trials and revised for the online arm. Sessions were scored both on non-specific factors (e.g., therapeutic alliance, support, empathy) and on BI specific factors (e.g., description of BI, review of worksheets, attention to homework). The PIs received reports monthly, and reviewed any noted deviations with the RA. Sessions were scored on a 10-point metric. Of sessions reviewed, 91% (446/492) received a score of ≥ 8 (very good-excellent).

Measures

Demographics

Demographic information obtained included child age, gender, diagnosis, and date of diagnosis; and caregiver age, gender, marital status, educational level, occupational status, and self-reported race/ethnicity.

Problem-solving skills

The Social Problem-Solving Inventory, Revised (SPSI-R) is a widely used, well validated 52-item measure of problem-solving skills, which assesses two dimensions of problem orientation (positive vs. negative) and three dimensions of approach to problems (rational; impulsive-careless; avoidance), and also yields a total score [18,19]. As in prior trials, change in total SPSI-R score served as the primary outcome. Internal consistency in the current trial was excellent (α = 0.95).

Negative affectivity

Parental emotional functioning was measured as in previous trials, with a triad of instruments assessing mood disturbance, symptoms of depression, and posttraumatic stress. The Profile of Mood States Scale (POMS) is a widely used measure of mood disturbance with excellent reliability and validity [20]. We used a 15-item short-form [21] to obtain a composite total mood disturbance scale (TMD) as the outcome. Reliability (α) in the current trial was 0.90). To measure symptoms of depression we used the Patient Health Questionnaire (PHQ-9) [22]. The diagnostic validity of the PHQ-9 for assessing the presence of major depression is well established, as is its sensitivity to change. Reliability in the current trial was 0.88. The Impact of Events Scale-Revised (IES-R) is a well-established measure of symptoms of posttraumatic stress in response to a specific event (here identified as the child's cancer diagnosis), with excellent reliability and validity [23]. Reliability (α) in the current trial was 0.93. This triad of measures was obtained at all timepoints, and serve as secondary outcomes. All measures are available in Spanish and have been validated in Spanish-speaking samples.

Statistical analysis

The study was designed as a non-inferiority trial with a one-sided alternative hypothesis [24,25]. A non-inferiority trial seeks to determine whether a new intervention is not worse than an established treatment by more than an acceptable amount, referred to as the margin of equivalence (E). When the null hypothesis is rejected (or the 95% CI excludes E), non-inferiority is established; failure to reject the null hypothesis is considered ‘indeterminate’. Estimates of the effect of F2F BI were obtained from prior trials, where moderate effects (0.4 S.D.) were observed in change in SPSI-R scores from baseline to end of intervention (T2) [11]. The margin of equivalence was therefore set at 0.2 times the S.D. of the change from T1 to T2 (or approximately 50% of the prior observed effect). Given these parameters, a sample size of 620 (310 in each group) was required to achieve 80% power to detect noninferiority using a one-sided, two-sample t-test, at α of 0.05.

Primary analyses were performed on an intent-to-treat basis (ITT), using maximum likelihood estimation (MLE) for incomplete repeated measures (SAS Proc Mixed), with the constraint that under the conditions of randomization, the baseline estimates are equal [26]. Missing data were handled under the assumption of Missing at Random (MAR) [27]. Estimated change from T1 to T2 in SPSI-R total scores was considered the primary endpoint. Change in SPSI-R from T1 to T3, and changes in negative affectivity measures from T1 to T2 and T3 were examined as secondary endpoints. Because the trial was designed with one-sided alternative hypotheses, the 95% CIs are also one-sided and only one boundary exists. Effect sizes are defined as the estimated mean change divided by the standard deviation of the change.

In addition to the ITT analysis, an objective ‘per protocol’ analysis was also conducted to account for participants who did not receive the full intervention in either arm due to non-compliance or other factors. This per protocol analysis excluded participants in the F2F arm who received fewer than 6 sessions; and in the online arm who did not complete all 7 instructional videos, 2 of 4 fotonovellas, and work on at least one independent problem.

Role of the funding source

The funder of the study had no role in the study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication, following review and approval by all authors.

Findings

Of 945 eligible caregivers approached, 621 (65.7%) consented, completed baseline measures, and were randomized (311 standard BI; 310 web-based BI; CONSORT diagram, Fig. 1). This participation rate was comparable to that of our prior trials and targeted accrual was met. There were no differences between participants and non-participants based on child diagnosis, language, or whether the target participant was a mother or father. There was a difference in time since diagnosis for recruitment, with participants recruited earlier than decliners (7.3 weeks, vs. 8.9 weeks). Baseline demographic and medical characteristics of the treatment arms were comparable (Table 1). In terms of per protocol analysis, in the F2F arm, of 222 who completed the study, 146 (66%) received ≥ 6 sessions, and were considered per protocol. In the web-based arm, of 220 who completed the study, there was a nearly identical 66% rate of per protocol adherence.

Fig. 1

CONSORT Diagram. *Due to a temporary server failure, complete data regarding online usage was not available for 64 participants in the web-based arm, thus they were not included in the per-protocol analyses.

Table 1

Demographic and Medical Characteristics of Intent-to-Treat Sample.

	FACE-TO-FACE		ONLINE
	M	SD	M	SD	P
Child Age	8.2	5.5	8.3	5.5	.712 NS
Time DX to T1 (weeks)	7.3	3.2	7.4	3.5	.968 NS
Parent Age	36.7	8.8	37.0	8.6	.541 NS
Parent Highest	13.8	3.5	13.6	3.6	.809 NS
Grade Completed	N	%	N	%	P
Child Gender
% Male	175	56.3	170	55.2	.788 NS
Parent Gender
% Female	269	86.5	280	90.3	.136 NS
Language
English	267	85.9	266	85.8	.987 NS
Spanish	44	14.1	44	14.2
Parent Race
White	195	64.4	193	64.3	.182 NS
Black	43	14.2	34	11.3
Other/Unknown	65	21.4	73	24.4
Child Diagnosis
ALL	118	36.7	111	35.8	.986 NS
Other Leukemia	33	10.6	37	11.9
HL/Non HL	27	8.6	27	8.7
Solid Tumor	76	24.4	78	25.2
Brain Tumor	26	8.4	27	8.7
Other	35	11.3	30	9.7

Primary outcome

Fig. 2a shows the change in problem-solving scores across the study period for the F2F and web-based groups. The effect of the standard F2F treatment was preserved compared to prior trials. Parents in the F2F BI arm increased their problem-solving score on the SPSI-R by a mean of 1.09 points (95% CI, 0.85 to 1.33, effect size d = 0.53, t = 8.75, p < .001). Parents in the web-based group also significantly improved their SPSI-R score by 0.66 points (95% CI, 0.41 to 0.90, effect size d = 0.32, t = 5.32, p <0.001). Thus, 60.3% of the standard treatment effect was preserved in the web-based group. The mixed model analysis of overall effect showed a between-group difference of −0.44 points favoring the F2F group. P-value for noninferiority = 0.55. This result is considered ‘indeterminate’, failing to establish noninferiority, but also failing to demonstrate that the web-based intervention is inferior (Table 2, Fig. 3) [26]. At T3, results on the SPSI-R were similar, with a between-group difference of −0.40 points, p value for noninferiority = 0.57 favoring the F2F intervention. In the per protocol analysis, results were similar. For both the F2F and web-based groups, a significant improvement on the SPSI-R was observed, with a between-group difference favoring the face-to-face group, but tests of noninferiority were non-significant and the overall results indeterminate.

Fig. 2

Longitudinal changes in study outcomes across study timepoints: 1) baselne, 2) end of intervention, 3) 3-months post-intervention Longitudinal changes in study outcomes across study timepoints: 1) baseline, 2) end of intervention; 3) 3-months post intervention. Figure 2a, the primary outcome, the Social Problem Solving Inventory-Revised, total score. Figures 2b-2d, secondary outcomes on b) mood disturbance, c) posttraumatic stress symptoms, d) depressive symptoms. Note, all y-axes have been truncated for ease of interpretation.

Table 2

Noninferiority analyses on study outcomes.

	Within group changes					Noninferiority Test
	Mean Difference t2 – t1	SE	t	p	ES	Margin	95% CI		z	p
SPSI-R Total
Face-to-face	1.09	0.12	8.9	<0.01	0.53
PWeb-based	0.66	0.12	4.9	<0.01	0.32
Difference	−0.44	0.17	—	—	−0.21	−0.41	−0.71	–	−0.13	0.55
POMS TMD
Face-to-face	−19.8	1.86	−10.6	<0.01	−0.58
Web-based	−13.0	1.86	−6.9	<0.01	−0.38
Difference	6.8	2.29	—	—	0.20	6.87	–	10.6	−0.03	0.49
PHQ-9
Face-to-face	−2.99	0.30	−9.9	<0.01	−0.56
Web-based	−2.36	0.30	−7.8	<0.01	−0.44
Difference	0.63	0.38	—	—	0.12	1.07	–	1.27	−1.15	0.13
IES-R
Face-to-face	−9.60	0.90	−10.7	<0.01	−0.60
Web-based	−7.08	0.90	−7.9	<0.01	−0.44
Difference	2.52	1.15	—	—	0.16	3.20	–	4.41	−0.59	0.28

ES = effect size (mean change/ standard deviation of the change)

95% CI = one-sided 95% confidence interval Boundary).

Fig. 3

Forest plot of noninferiority analyses on study outcomes.

Secondary outcomes

Longitudinal changes on the negative affectivity measures are illustrated in Fig. 2b-d. On the POMS, the F2F group showed a decline in mood disturbance from baseline to T2, with a mean change of −19.8 points (95% CI, −23.4 to −16.6, effect size d = −0.58, t = −10.6, p < .001). The web-based group showed a decline of −13.0 points on the POMS (95% CI, −16.6 to −9.30, effect size d = −0.38, t = −6.9, p < .001), representing preservation of 65.5% of the standard intervention effect. The estimated between-group difference was 6.8 points favoring the F2F group. P-value for noninferiority = 0.49. For depression, the F2F group showed a decline on the PHQ-9 from baseline to T2, with a mean change of −2.99 points (95% CI, −3.6 to −2.4, effect size d = −0.56, t = −9.9, p < .001). In the web-based group, there was a mean decline of −2.4 points on the PHQ-9 (95% CI, −2.95 to −1.77, effect size d = −0.44, t = −7.8, p < .001), representing preservation of 78.6% of the standard intervention effect. The estimated between-group difference was 0.63 points favoring the F2F intervention. P-value for noninferiority =0.13. For posttraumatic stress, scores for the F2F group showed decline on the IES-R from baseline to T2, with a mean change of −9.6 points (95% CI, −11.4 to −7.8, effect size d = −0.60, t = −9.6, p < .001). In the web-based group, there was a mean decline of −7.1 points on the IES-R (95% CI, −8.8 to −5.3 effect size d = −0.44, t = −7.9, p < .001), representing preservation of 73.3% of the standard intervention effect. The estimated between-group difference was 2.5 points favoring the F2F intervention. P-value for noninferiority = 0.28. Again, the pattern of results is indeterminate in relation to inferiority/noninferiority of the web-based intervention (Table 2, Fig. 3). Similar results were found on all secondary outcomes at T-3.

Per protocol analyses were conducted for all secondary outcomes and revealed results similar to the ITT analyses, demonstrating significant benefits of both intervention arms and between-group differences favoring the F2F group. Tests of noninferiority failed to reach significance, and results were indeterminate in relation to inferiority/noninferiority.

Discussion

This multicenter RCT, conducted in two languages, examined the benefits of a newly developed web-based version of an established, evidence-based problem-solving skill intervention, Bright IDEAS (BI), in comparison to the standard F2F mode of delivery. The trial used a noninferiority design, testing the hypothesis that the between group differences would not exceed a margin of equivalence set at 0.2 times the standard deviation of change from T1 to T2. The effects of the benchmark intervention (F2F BI) were preserved as in prior trials. Noninferiority of the web-based comparator intervention was not demonstrated on the primary outcome, nor on any of the secondary outcomes. The difference between treatment arms did not indicate that online BI is inferior to the standard intervention, rather the confidence intervals fell in a range considered indeterminate. Although noninferiority was not demonstrated, significant benefit was also found for the web-based condition, and of a magnitude that exceeded 60% of the benchmark intervention on the primary and all secondary outcomes (range 60% - 78%). These findings suggest that further development of the web-based BI is needed before it can be recommended as a stand-alone intervention. However, the documented benefits of the web-based intervention as well as the advantages of low resource utilization and ease of delivery suggest that further development of web-based BI is indicated.

In evaluating the validity of a noninferiority trial, a crucial consideration is whether the effect of the standard treatment was preserved [25]. In the current trial, not only was the effect of F2F BI on change in problem-solving skills preserved, it was descriptively larger than in prior trials [9,11]. In each successive trial of F2F BI, this effect size has increased, which likely reflects several factors, including fine-tuning of intervention implementation, improvements in the manual and intervention materials, and increased experience of the investigative team which has remained largely consistent over trials. From that perspective, it is informative to consider that the effect on problem-solving skills observed in the online arm in the current trial (0.32) exceeds what was found in our initial trial of F2F BI, where we demonstrated superiority against a standard of care comparison [9]. Likewise, the effects of the F2F intervention in the current trial on reduction of negative affectivity were descriptively larger than in our prior trials. Thus, although noninferiority of online BI was not demonstrated, it should be noted that it was being compared to an improved and more efficacious version of F2F BI than had been tested in prior trials. With increased experience, fine-tuning, and upgrading of our online materials, we might also expect similar increases in the potential benefits of a web-based intervention. Future upgrades to the web-based intervention would profit from emerging technologies such as machine learning and an artificial intelligence-based conversational agent with humanlike empathic responses. Our group currently has efforts underway to take advantage of these broad advances in digital technology to develop a more engaging, interactive, and responsive version of our electronic BI intervention.

Commonly, noninferiority designs are used to examine a less intensive intervention with an aim of reducing treatment side-effects or adverse events. Reduction in efficacy must be considered in light of a comparable reduction in adverse outcomes. In the current trial, no negative side-effects were anticipated in either treatment arm, and no adverse events were reported. In this setting, the advantage of the web-based intervention is reflected in reduced administrative burden: cost, time and participant inconvenience. Beyond burden, the advantage of an online approach comes from its potential for widespread dissemination and availability for consumers who could access the intervention from anywhere at their convenience. Current estimates of penetration of internet access in North America exceed 90%, and continue to increase, resulting in near universal access of a freely disseminated online intervention [28]. Our design did not include a formal economic analysis; however, labor costs for the F2F administration exceeded all costs for the development and maintenance of the online site, and cost savings would increase with wider dissemination.

The design of the current trial was intended to compare two different approaches to implementation of the same evidence-based intervention, and efforts were made to keep the arms as distinct as possible Therefore, we attempted to minimize the in-person contact provided in the web-based arm, while the F2F arm did not allow access to the web-based materials. We recognize that this does not reflect developing trends in eHealth where multi-modal approaches combining face-to-face and internet-based components are increasingly common. Future research should explore how these approaches can be combined to optimize outcomes and minimize costs.

Our primary analysis utilized an intent to treat (ITT) approach. For noninferiority trials, there has not been a clear consensus on the optimal sample for analysis, although consideration of analyses that exclude participants that did not receive the intervention as planned is often recommended [24]. Thus we also conducted a per protocol analysis that was specified a priori, defining a minimal exposure to the intervention in each arm in order to be included. The proportion of participants considered per protocol vs. not per protocol in each arm was nearly identical (66%). The per protocol analysis had substantially reduced power and should be interpreted cautiously, but the findings were consistent with the ITT analysis.

Other potential study limitations include the participation rate (65.7%) of those recruited to the study, which could suggest some bias in the study sample. However, this rate is comparable to that obtained in our prior trials 9,11 but substantially higher than in other trials aimed at parents of children with newly diagnosed cancer or for internet-based interventions for prevention of medical traumatic stress in parents [29,30]. Additionally, follow-up was limited to 3 months. Perhaps distinctions between F2F and web-based approaches would become more apparent over time. Finally, our reliance on the SPSI-R as the primary outcome measure, in the absence of an independent, ecological measure of actual problem-solving by participants has been a potential issue in all of our trials. Our prior findings demonstrating that improvement in negative affect following the intervention is mediated by change in problem-solving skills as measured by the SPSI-R serves to largely assuage this concern [9].

Across 4 large RCT's we have demonstrated that F2F BI is supported by considerable evidence. Noninferiority of an online version of BI relative to standard F2F administration was not established in the current trial. Thus, there is not sufficient evidence to warrant widespread implementation of web-based BI as a stand-alone intervention. However, significant benefits of the web-based intervention were documented on both the primary outcome of problem-solving skills, and the secondary outcomes of negative affectivity. These results suggest that further development and enhancement of a web-based approach are warranted. Given the advantages of low resource utilization and ease of delivery, it appears that a web-based implementation of BI can play a valuable role in alleviating distress in caregivers. However it is likely that web-based administration will be most effective in combination with in-person support. Future research should investigate the benefits of reduced frequency face-to-face delivery of BI, supplemented with enhanced online materials.

Declaration of Competing Interest

All authors indicate there are no conflicts of interest to report.