Navigating the development and dissemination of internet cognitive behavioral therapy (iCBT) for anxiety disorders in children and young people: A consensus statement with recommendations from the #iCBTLorentz Workshop Group.

Initial internet-based cognitive behavioral therapy (iCBT) programs for anxiety disorders in children and young people (CYP) have been developed and evaluated, however these have not yet been widely adopted in routine practice. The lack of guidance and formalized approaches to the development and dissemination of iCBT has arguably contributed to the difficulty in developing iCBT that is scalable and sustainable beyond academic evaluation and that can ultimately be adopted by healthcare providers. This paper presents a consensus statement and recommendations from a workshop of international experts in CYP anxiety and iCBT (#iCBTLorentz Workshop Group) on the development, evaluation, engagement and dissemination of iCBT for anxiety in CYP.

Introduction

Approximately 6.5% of children and young people (CYP) worldwide have an anxiety disorder (Polanczyk et al., 2015). Effective psychological treatments for anxiety disorders in CYP exist, however the vast majority of CYP who would benefit from them do not access them (e.g., Green et al., 2005; Lawrence et al., 2015; Merikangas et al., 2011). This situation is especially true for CYP in low and middle income countries, where the ratio between available mental health professionals to inhabitants is lower than in high income countries, the distance to mental health care is larger, the stigma of mental health difficulties can be greater, and there are fewer opportunities to get sessions reimbursed (World Health Organization, 2009). In the USA, although a high income country, mental health provision is limited by being privately funded and issues surrounding reimbursement. Internet-based cognitive behavioral therapy (iCBT) may provide opportunities to reach many individuals in need of mental health care in a potentially efficient, cost-effective and non-stigmatizing way.

There is now a substantial literature demonstrating that iCBT can be effective in reducing anxiety in CYP and is acceptable to clinicians, CYP, their families and other stakeholders (e.g. Pennant et al., 2015; Podina et al., 2016; Rooksby et al., 2015; Topooco et al., 2017; Vigerland et al., 2014). This is encouraging but there are a number of challenges for the field. For example, evidence-based iCBT needs to be disseminated to healthcare providers so that they can be available in routine clinical practice, however iCBT programs have often been developed by universities and mental health institutes which are not typically well-placed to act as providers to (other) healthcare organizations (Hill et al., 2017). A further challenge is how to ensure that iCBT has a sufficient and robust evidence base in a timely manner that keeps apace with technological advancements so that evidence-based iCBT is not technologically obsolete or outdated by the time it can be disseminated. Another key issue is how to maximize engagement in iCBT, to ensure that CYP and their parents access and use the intervention, and that therapists are sufficiently skilled to support online interventions.

The lack of guidance and formalized approaches to the development and dissemination of iCBT has arguably contributed to the difficulty in developing iCBT that is scalable and sustainable beyond academic evaluation and that can ultimately be adopted by healthcare providers. As such, a group of international experts in child and adolescent anxiety and iCBT was convened for a 3-day workshop that was funded following a peer-reviewed process at the Lorentz Centre at the University of Leiden, The Netherlands in October 2017 (#iCBTLorentz Workshop Group). The goal was to produce a consensus statement and recommendations for the development, evaluation, engagement and dissemination of iCBT for anxiety in CYP. We define iCBT as CBT which is delivered online without the need to attend a clinic to receive support. It does not include the delivery of CBT through virtual reality or augmented reality, or the use of technological tools such as artificial intelligence or wearables, although it is anticipated that the principles discussed here could be applicable to other digital innovations.

Development of iCBT

We begin by exploring different approaches to development and what to consider from the outset, and follow with discussions of important features of iCBT development: engagement, co-design, confidentiality, and data protection. A summary of recommendations relating to the development of iCBT are shown in Table 1.

Table 1

Recommendations for development of iCBT.

	Those developing iCBT (academia and industry)	Those using iCBT (Clinicians)	Those funding iCBT	Policy makers
Approaches to development	• Be clear on who your target audience is • Consider what your goals are • What skills do you have and what do you need to outsource? • What are the implications for dissemination?	• Ensure developers are from a credible source (e.g. look for the evidence base)	• Consider are the goals met? • Ask for the dissemination plan • Provide grants including plans for evaluation and dissemination	• Look for evidence of (cost-)effectiveness, sustainability and scalability • Demand empirically-supported content • Co-ordinate international collaborations (don't reinvent the wheel)
Engagement	• Consider engagement of CYP, parents, and clinicians when developing iCBT • Involve multiple disciplines • Report on theoretical models informing the iCBT program design	• Look for evidence that service users engage well with the iCBT program	• Support funding bids that examine impact of specific technological features on CYP engagement	• Consider macro-level factors that impact on clinician engagement that influence decision to commission iCBT
Co-design	• Involve intended end-users in co-designing your iCBT • Test out iterations in usability testing in your initial development stage	• User-friendly • Meets CYP and your needs • Feels suitable for CYP and you	• Demand it	• Inform the public about iCBT
Confidentiality and data protection & ethics	• Consider the parameters of what your system requires • Use privacy by design • Budget for this • “Do no harm”	• Ask is your data protected? Is privacy guaranteed? • Ask where does my data go?	• Be aware of & demand researchers disclose conflicts of interest	• Demand it • Make the general rules specific to iCBT - more concrete guidance

Approaches to development

ICBT development by academics and clinicians is typically funded via a grant which can be used to 1) develop ‘in-house’ within universities or mental health institutions (i.e. employ computer developers or use existing in-house computer programmers), 2) commission a technology company purely to do the build, or 3) enter a collaborative agreement with a technology company to build and commercialize the iCBT.

When deciding which development approach to take, developers should consider what skills are required and be clear about their goals. Delivering internet-based treatments requires a specific set of skills and management considerations such as 1) development of treatment content that translates well into online delivery, including strategies of engagement of clinicians, CYP and parents with iCBT, 2) website/app design, 3) technical programming, 4) technical support, 5) data storage and analysis, 6) safety and confidentiality, and 7) ethical concerns. Whilst researchers or clinicians may have the skills to design evidence-based content for a successful intervention, it is unlikely that they will possess advanced technological (e.g. coding) or design skills. Instead, outsourcing assistance from technology companies may be required in order to produce a sophisticated program.

Developers will need to consider the advantages and disadvantages of contracting out the technological work as a one-off commission versus engaging in a partnership with a technology company that includes future potential commercialization of the iCBT in order to maximize dissemination. A one-off commission may allow the developer to retain full ownership over any intellectual property (IP) but careful consideration will be needed of how the iCBT can be sustained and delivered at scale beyond this first iteration as well as how technological troubleshooting will be handled (e.g. how available and costly will technological troubleshooting be if/when required). It is a common misconception that digital delivery equals free delivery, but there are significant costs involved in hosting and maintenance, updating the iCBT to be consistent with users expectations for technological functions and design, and continued research and development in order to maximize treatment outcomes. Furthermore, dissemination of iCBT to healthcare providers requires time and resources (e.g. marketing and sales) that academics and clinicians typically do not possess. The advantage of entering a partnership with a technology company is that the costs and resources required for scaling, sustaining and disseminating the iCBT can be set out in the collaborative agreement. However, a partnership arrangement will require negotiations around ownership and IP, responsibilities and agreements on who will finance what, contingencies for a course of action should the program not prove to be effective and the risk of the technology company folding or taking a different commercial direction in the future.

Another important consideration is the different needs of different users of iCBT. These users may be CYP and/or their parents, but users may also be therapists who deliver the treatment and may require a secure case management system. Researchers may need a portal for collecting and analyzing data generated by the iCBT. It is crucial to ensure that the technology and any necessary software or hardware is available to all users and can be integrated within the intended clinical care pathway.

Finally, we recommend that plans for dissemination should be considered early in the development stage to prevent iCBT from struggling to make it into routine clinical practice. Funders and policymakers can play a role. For example, funders can commit to financial support for dissemination once efficacy has been shown, and policymakers can insist on empirical support before dissemination.

Engagement with iCBT

It is essential that user engagement in iCBT is considered right from the start of development. Although calls have been made to better consider engagement when designing iCBT, few have explicitly addressed this in their conceptual frameworks or defined levels/stages of engagement. To understand engagement in iCBT it may be helpful to distinguish between the “micro” level of moment-to-moment engagement with intervention components and the “macro” level of engagement at the clinician and health system level (Murdoch, 2016). To optimize engagement, developers need to balance an emphasis on micro level patient interactions with the treatment and macro level engagement with iCBT at critical touch points of the care delivery workflow (McLeod et al., 2016).

We propose that research stemming from multiple disciplines including human computer interaction (Poole, 2013), persuasive system design (Wozney et al., 2017a), learning theory (Aronson et al., 2012; Cheek et al., 2015), information science (Perski et al., 2016) and marketing (Singh, 2011) should be considered during design to better reflect knowledge about how users engage with online health programs (Riley et al., 2008). The adult-focused literature on patient engagement in online interventions (e.g., aesthetic and sensory appeal, perceived control and time, and affect) is growing but not well documented in relation to young people and therapists using iCBT (Cavanagh, 2010; Yardley et al., 2016). There is a similar gap in knowledge around macro-level factors that have an impact on clinician engagement (e.g. treatment ideologies, billing models) that may influence decisions to commission iCBT as part of health services offered to CYP (van der Vaart et al., 2014; Wozney et al., 2017b). Some literature reviews have sought to retrospectively identify which factors are associated with higher engagement in iCBT but the conclusions are limited by the correlational nature of the evidence and the authors' incomplete descriptions of the interventions and real-world experiences of providers (Michie et al., 2009). Best-available evidence suggests that superior engagement in iCBT for CYP is found when a program is guided (by a virtual or real therapist or coach), developmentally appropriate (Beidas et al., 2014; Sauter et al., 2009), and has interactive skills-training (Kobak et al., 2015; Stiles-Shields et al., 2016). The evidence base relating to the impact of specific technological features (e.g., automated prompts and reminders, gamification, multi-media, social networking) on CYP engagement is less well-developed and we support innovative research programs that can comparatively assess these features in iCBT programs.

Co-design

Co-design is a useful tool to maximize user engagement. Co-design actively involves all relevant stakeholder groups in the design process to help ensure that the technology that is developed meets their needs and is usable. Within co-design, intended users work with designers/developers and researchers during the innovation process. This involves end user input on both (a) problem definition and ideas for the digital solution and (b) the evaluation of the digital solution.

It has been suggested that co-design brings benefits including improved (a) design quality (Kujala, 2003; Yardley et al., 2015), (b) user adherence (Howe et al., 2014), (c) usability (Maguire, 2001), (d) efficacy and sustainability (Gibbs et al., 2008), and stakeholder acceptance and adoption at the system and organizational levels (Wölbling et al., 2012). As such recent guidelines for digital mental health intervention design emphasize co-design (e.g., Mohr et al., 2013; Mohr et al., 2014; Richards et al., 2016) and funding agencies increasingly require evidence of stakeholder involvement.

How does co-design work in practice? First, developers need to identify the intended iCBT end users (e.g. CYP, parents/caretakers, or therapists) and involve them in the design and development of the iCBT. We would recommend that this is carried out systematically through a research study that is integral to the iCBT development.

Co-design typically includes 3 stages. The first stage is a “Needs analysis and design specification.” Workshops are coordinated to identify user needs and expectations for the iCBT. The MOSCOW approach (Must have, Should have, Could have, Won't have) (Clegg and Barker, 1994) can help prioritize the possible functions of iCBT. This approach will identify what the users really want. The second stage is “Intervention design, prototyping and development.” Here, the end users are involved in the program design. Scenario-based design techniques (e.g., providing a storyboard; asking users to adopt a particular persona) and simply asking users for ideas about how to sketch out features of the iCBT (Beaudouin-Lafon and Mackay, 2003; Brandt et al., 2012; Kushniruk et al., 2013). The final stage is “Usability and pilot testing.” Once there is a working iCBT prototype, usability testing provides feedback on what works and doesn't work, and can also identify technical and aesthetic issues that may affect user satisfaction (e.g., Breakey et al., 2013). A single usability testing cycle with 5–10 users can lead to up to a 10-fold reduction in issues (Kushniruk, 2002). The user can be observed performing specified tasks within the iCBT program in order to identify what errors are made and classify them according to severity. User feedback can be gathered quantitatively (Bangor et al., 2009; Sauro and Lewis, 2012; Wozney et al., 2015), and semi-structured interviews can be used to gather qualitative data on user satisfaction. Snodgrass and Coyne (1996) endorse an iterative process of seeking further feedback from users based on the feedback gained at each stage. Importantly, exploring how the iCBT functions under different “real world” scenarios (e.g., accessed from different locations, devices, operating systems or web-browsers) is important to identify issues that might not emerge in a research lab (Rooksby et al., 2009). Finally, we encourage developers to publish their iCBT development process so that further knowledge on how best to use to co-design for iCBT (Owens et al., 2011) in particular in the child and adolescent anxiety field (Orlowski et al., 2015) can be shared.

Ethics, confidentiality, and data protection

The key ethical question for iCBT developers is whether the potential benefits of iCBT are justified against possible risks, in the context of the primary ethical principle to “Do No Harm”. To this end, developers of iCBT need to consider the risks involved to all stakeholders, minimize risks as much as possible, and fully inform potential users regarding risks and benefits to allow them to make an informed decision to use iCBT (Childress, 2000). Ethical challenges related to iCBT differ depending upon which internet channel is used (websites, email, chat or skype/video technology). Developers should ensure that the terms and conditions of use of the iCBT and the privacy policy is ethically and legally fit for purpose and are easily accessible to users.

Informed Consent is central to ethical practice. To avoid treating minors without the consent of their parents/guardians, developers of iCBT must address how the age of users will be verified. ICBT should include an explanation to users about the risks of breaches of confidentiality that can occur with online communication and the risk of miscommunication when using emails.

Breaches in confidentiality may occur during electronic transmission of information (e.g., hackers). To improve security in emailing with clients, developers can use encryption technology at the therapist's and the clients' end (although it is important to note that family members or employers might be able to read client's emails). In some countries, email records may be subject to subpoena and it remains unclear whether legal protection extends to therapists using iCBT (Childress, 2000). Errors in addressing emails to the wrong person often occur and can be particularly harmful in the context of therapy, so encryption is especially valuable but cannot be guaranteed when emailing outside of secure platforms. Given the lack of security of emails, we would recommend that any written communication that is identifiable or clinically sensitive is not emailed to service users, but is sent within a secure messaging system within the iCBT platform.

Other potential ethical issues relating to the use of written communication between the service user and therapist, include difficulties in the therapist's ability to understand and complete an attuned treatment plan due to the lack of nonverbal cues. Equally, words can sound more harsh than intended, and clients may experience emotional injury and withdraw from therapy. Add to this the fact that most therapists are not trained in providing therapy via a written medium, and the further challenge of understanding the text language of adolescents. Therapists delivering iCBT should receive training in how to effectively communicate with service users online and developers should consider how training could be incorporated within the therapist portal of the iCBT.

The facility for written online communications can also create service user expectations about immediate responses. Therapists need to be clear about when responses will be sent (for example, being clear if the portal will not be continuously manned and alternative ways to seek urgent help if needed) and consider how to respond when a user sends an email after the termination of therapy, or when a user sends messages that harass or stalk a current or former therapist. A related point is that the provision of care directly into home settings requires careful consideration of the safety of families and providers' ability to appropriately respond to and manage risk, both during and between appointments (Kramer et al., 2015). Before starting a therapeutic online relationship, therapists need to arrange and discuss with the user the plans for addressing potential crises. The plans can range from what to do when the therapist discovers that the client would benefit more from intensive therapy or hospitalization or what to do when there are concerns about suicide. Other suggested provisions include having a second contact for families and collaborating with local providers to respond to urgent technical or clinical emergencies to alleviate concerns regarding iCBT providers' abilities to respond to urgent situations. Finally, iCBT therapists need to consider issues associated with informing users about their professional affiliations and facilities for users to address grievances. These matters are especially complex because of the increased likelihood that the therapist and user live in different states or countries, with different legislation.

Data protection (handling of client data) must comply with current legislation. For instance, in May 2018 a new European legislation act will extend and strengthen privacy rights of clients. Developers need to employ encryption, firewalls and passwords to protect the security of computers and websites against high-tech invasions by hackers and low-tech invasions (e.g. therapist's office staff, family members reading emails). Backup systems for data and dossier storage, and duration of data storage, need to be in in accordance with the proper jurisdiction.

Evaluation of iCBT

With the rise in the number of iCBT programs for anxiety disorders in children and adolescents comes the need for standards of evaluation in determining empirical support. Here we discuss and make recommendations regarding the design, measurement, and level of reporting for iCBT studies. A summary of recommendations regarding evaluation of iCBT are shown in Table 2, Table 3.

Table 2

Essential and useful measures for future CYP anxiety iCBT research.

Construct	Measure/notes	Essential/useful
Anxiety diagnoses, severity and symptoms	• Parent and youth diagnostic interviews e.g., Anxiety Disorders Interview Schedule for Children (ADIS-C/P; Silverman and Albano, 1996) – telephone administration has been shown to be equally reliable (Lyneham and Rapee, 2005) • Both loss of primary anxiety diagnosis and loss of all anxiety diagnoses, should be reported	Essential
	• Anxiety symptom questionnaires including both parent report and child self-report e.g., Spence Child Anxiety Scale (SCAS; Spence, 1998)	Essential
Interference associated with anxiety	• Degree to which anxiety interferes with the young person's life e.g., Child Anxiety Interference Scale (CAIS-C/P; Langley et al., 2004; Langley et al., 2014), Child Anxiety Life Interference Scale (CALIS; Lyneham et al., 2013)	Essential
Coping with anxiety	• Coping Questionnaire (CQ; Kendall and Marrs-Garcia, 1999) assesses the child's perceived ability to manage anxiety-provoking situations.	Useful
Dropout/adherence	• The number of participants who began the study and completed each aspect of the iCBT	Essential
	• Average number of sessions completed	Essential
	• Number of participants completing the iCBT as intended	Essential
	• Percentage of participants completing each session	Useful
	• Usage data e.g., logins, page views, frequency of repeat visits, overall time spent on site/activities, periods of low activity or inactivity, length of time to complete the program, reengagement after a period of inactivity	Useful
	• Differences (e.g., in terms of baseline/demographic characteristics) between those who dropped out of treatment and those who were retained	Essential
	• User's (CYP/parent) understanding/implementation of iCBT (through knowledge quizzes or records of accurate skill rehearsal)	Useful
	• Reasons for drop out (qualitative) including adverse events and effects of treatment	Useful
	• How the program has affected the CYP's or parent's future help seeking	Useful
Satisfaction	• Participant satisfaction, credibility and expectancy	Essential
Working alliance	• E.g., Working Alliance Inventory (WAI; Horvath and Greenberg, 1989)	Useful
Attitudes towards iCBT	• E.g., preferences for iCBT versus face-to-face, perceived advantages and disadvantages of iCBT, etc.	Useful
Reach of iCBT	• E.g., has the participant received psychological assistance before? Does the participant have other options? Distance from nearest clinic?	Essential
Assessment for healthcare settings	• Effect on waitlists, non-attendances, length of time to first appointment, access by priority groups, number achieving reliable change, impact on clinicians and their compliance and adherence	Useful
Cost effectiveness	• Therapist time e.g., time spent responding to emails, technological issues, preparation etc.	Essential
	• Cost to family in terms of cost of program, time, travel, time spent away from work etc.	Useful

Table 3

Recommendations for reporting of iCBT programs.

Mechanism	What should be presented	Note
Duration	• Length of treatment (e.g., number of weeks) • Booster sessions	• Is treatment duration fixed or flexible?
Intensity	• Number of sessions offered in treatment • Length of sessions	• Are the sessions fixed or flexible?
Content	• Description of included CBT strategies • The start and amount of exposure training • Program type (e.g., game, bibliotherapy)	• How is information gained about the amount of exposure tasks conducted? • Provide a visual overview • Include screenshots
Therapist support	• Types of therapist communication included (e.g., via internet, telephone, face-to-face) • Whether synchronous/asynchronous • Frequency (i.e., fixed/flexible) • Content (e.g., manualized)	• What therapist competence is recommended?
Parental involvement	• Are parents involved? • What role do parents have during treatment?	• How is the parent instructed to support the CYP's exposure training?
Technological strategies	• How therapy content is presented (e.g., use of text, audio-visual methods, animations, interactive tasks, gaming techniques)	• Justification for use of these particular technological strategies

Study designs

To date, iCBT programs for CYP anxiety disorders have been evaluated in a variety of ways including case reports (e.g., Spence et al., 2008), uncontrolled trials (Nordh et al., 2017; Silfvernagel et al., 2015; Vigerland et al., 2013) and randomized controlled trials (e.g., Donovan and March, 2014; Khanna et al., 2017; March et al., 2009; Spence, 2011; Spence et al., 2017; Vigerland et al., 2016; Wuthrich et al., 2012). Within the RCTs, iCBT has compared favorably to conditions including waitlist (e.g., Khanna et al., 2017; March et al., 2009) and face-to-face treatment (e.g., Spence, 2011). The findings provide support for the efficacy of iCBT for anxiety disorders in youth.

Randomized Controlled Trials (RCTs) remain the gold standard for program evaluation. However, iCBT is rapidly growing and changing. Given that numerous RCTs have demonstrated the efficacy of iCBT, are future RCTs of iterations of programs necessary? RCTs are labor, time, and money intensive, and by the time a program, or an iteration of a program, is evaluated through an RCT, the field may have moved on and/or the technology changed.

RCTs are the best design to test causal relations between treatment and effect. However, they may not always be feasible and/or desired. For example, if blind conditions cannot be achieved the effect size estimates will be exaggerated (Schulz et al., 1995) and if patient groups are rare, a large RCT may not be feasible.

When RCTs are not feasible, one might employ less powerful approaches. For example, a quasi-experimental design such as propensity score matching where groups statistically similar on a set of relevant variables are created in lieu of random assignment (Rosenbaum and Rubin, 1983), a single case design (Hanin, 2017) or graphical models (Carey and Stiles, 2016; West et al., 2008). Common to quasi-experimental designs is the increased difficulty in making causal claims. When using quasi-experimental designs it is best to conduct multiple evaluations and demonstrate results that are in the same direction. For instance, if both a propensity score study and two single case evaluations suggest that a certain treatment was likely to be effective, then the combined evidence outweighs that of the individual studies (convergence of evidence; Carey and Stiles, 2016).

We would recommend that any new psychological treatment program, whether iCBT or face-to-face, should be evaluated via the gold standard (RCT). To implement a program without such evaluation would be both poor science and poor judgement in that it has the potential to weaken the reputation of iCBT within research and clinical communities. Depending on the research question, the RCT would compare iCBT to a waitlist control, to an active control, to a care as usual condition, or to a face-to-face therapy condition. In all cases, the decision is guided by the level of previously gained empirical support for the iCBT program and the number of participants likely to be recruited. Whether the RCT conducts tests to evaluate differences between conditions or uses a non-inferiority/equivalence trial will also depend on the research question.

Once efficacy of an iCBT program has been determined through an RCT, iterations or minor adaptations of the program may be tested through alternative, less stringent research designs such as the quasi-experimental designs listed above. For instance, a benchmark based on effect sizes from past RCTs of a particular program, may be set prior to conducting an uncontrolled trial of a briefer version (or updated version) of the program, to determine relative effectiveness of the original versus briefer (updated) versions. Another option could be to use hybrid designs (Curran et al., 2012) that use a flexible study protocol that can change during the study. Once an iCBT program is implemented within routine clinical practice, treatment outcomes should be systematically monitored, evaluated and the iCBT further refined to maximize treatment outcomes. We recommend that routine outcome measures (ROMs) are built into the iCBT to facilitate this process, ROMs from iCBT should be made transparent to service users and commissioners (Clark et al., 2017) and the cost for continued research and development of the iCBT should be factored into the dissemination plan.

Measurement

It is essential that researchers employ consistent measures to allow meaningful and accurate comparisons across studies evaluating iCBT programs for child and adolescent anxiety disorders. There is a fine line between overburdening families with too many assessments, and ensuring that sufficient information is obtained to adequately assess the efficacy of a program. Table 2 outlines recommendations for measures to be used in iCBT CYP anxiety studies, categorizing them into ‘essential’ and ‘useful’ measures.

First, we endorse including measures that enable comparisons with evaluations of face-to-face youth anxiety treatment research. These include parent and child reports of youth anxiety disorder diagnoses, symptoms, and severity, as well as interference associated with anxiety. We endorse assessments by Independent Evaluators (IEs), and we add that it is important to assess loss of principal diagnosis as well as loss of all anxiety diagnoses (Warwick et al., 2017).

Second, we endorse accurate measurement and reporting of program dropout and adherence. High dropout can reduce the dose of therapy and contribute to less accurate evaluations of outcome. We recommend that researchers report (a) dropout rates akin to CONSORT E-Health recommendations (Eysenbach, 2011), and (b) differences between those who dropped out and those retained. To guide future work, qualitative information that helps to explain why participants dropped out may be useful. For instance, a user may drop out at Session 4 out of 8 because they are not making gains or because they have made gains and no longer require treatment.

Researchers should also routinely report on adherence or compliance within iCBT programs. Two recent reviews noted the significant variation in the types of adherence reported (e.g. number of sessions completed, length of iCBT program, pages viewed), with a substantial proportion of studies failing to provide data or clear definitions of adherence at all (Rooksby et al., 2015; Vigerland et al., 2016). Thus, we endorse the establishment of clear definitions of iCBT adherence, detailed recording of micro-level data and consistent reporting within iCBT evaluation. We propose that adherence should specifically refer to the degree to which users complete the intervention as intended (that is comply with the program or protocol requirements). This requires researchers to conceptualize and report adherence beyond simply activity (duration or length of the iCBT program) which has previously been reported as a proxy for adherence. Rather, it is necessary to consider adherence in relation to the intended duration, length, intensity and completion of specific components of iCBT programs. The degree to which a user adheres to the clinical protocol will influence the nature, type and dose of the intervention received. Without accurate reporting of adherence, we are unable to understand how iCBT programs work and which components are most effective and for whom.

There are additional considerations for the reporting of adherence in iCBT for child and adolescent populations. First, iCBT programs for children often incorporate parents/caregivers into treatment, with many offering separate parent programs or involving the parent as a coach or support person in the treatment (March et al., 2009; Spence, 2011; Vigerland et al., 2016; Wuthrich et al., 2012). Thus, it is particularly important to report on the expected parent role in iCBT treatments as well as their adherence with parent components, both separate and in relation to their child's adherence. The same would be true for any other informants or users directly involved in the child's care, for example teachers or school guidance counsellors if iCBT programs are implemented in school settings. Second, previous research has demonstrated that children and parents may be slower to complete iCBT programs than in traditional face-to-face CBT (March et al., 2009; Spence, 2011). Thus, researchers and clinicians should consider the timing of outcome assessment and make decisions as to whether this should be conducted at the ‘desired’ or ‘actual’ completion date, which can have consequences for the study design chosen. Such decisions must be grounded in the purpose of the study and objectives in measuring adherence. Third, the importance of alternative methods for examining adherence may be more pronounced in child iCBT programs where the child is required to use self-directed learning and the therapist has reduced capacity to correct and reinforce learning. Thus, incorporating measures of knowledge acquisition or engagement with the material might be more useful than generic program completion data in determining whether a young person has ‘received’ the intended dose.

Therapist support is often assumed to be an important determinant of adherence, with some evidence showing similar adherence rates for clinic- and guided internet-delivered CBT for both children and adults (Spence, 2011; van Ballegooijen et al., 2014), but the results to date are inconclusive for CYP, with information regarding the level and type of support often poorly specified (Hollis et al., 2017). There is some indication that factors such as therapeutic alliance and treatment expectancy might be related to greater adherence (e.g., Anderson et al., 2012). However, there is a scarcity of research reporting on predictors of adherence across therapist-supported and self-directed forms of iCBT. Thus, we endorse the need for future research to also examine individual, program and therapist level factors that are associated with adherence in order to understand how iCBT works and for whom.

Finally, given that the genesis of iCBT was in part driven by the need to circumvent barriers to CYP accessing psychological treatment (e.g., stigma, access and cost), we endorse assessing program satisfaction, reach, and cost-effectiveness. Program satisfaction, expectancy and credibility data provide evidence of engagement, as well as the viability and suitability of iCBT. Assessing reach (i.e., distance to nearest clinic, has the CYP sought help previously; other treatment options) provides data on the ability of iCBT to address barriers. In terms of cost-effectiveness, iCBT has appeal due the perception that it requires less therapist time than traditional face to face CBT, but does it? Information regarding the amount of therapist-support (i.e., duration, frequency), how therapist support is delivered (e.g., internet, telephone, face-to-face, synchronous/asynchronous, fixed/flexible), and the level of training and supervision required by the therapists, all need to be presented. Further costs that are specific to iCBT (such as program hosting, program upgrades, etc.) should be included in order to accurately assess cost-effectiveness.

Precise reporting of iCBT programs

It is important to be clear and precise about the content and format of the iCBT program under evaluation to allow for comparisons across trials (see Table 3 for a summary of recommendations). A detailed description of the CBT strategies included should always be reported, and where possible a visual overview of the treatment content should be provided, as well as screenshots from the program. ICBT can differ in duration, intensity, content, type, amount of therapist support and parental involvement, and therefore these aspects should be clearly described. The duration of treatment refers to the number of weeks that treatment is conducted and whether or not there are booster sessions. The intensity of treatment refers to the number and length of sessions, as well as whether sessions are flexible or fixed (sessions are in a set order, and each session requires completion before the next session becomes available). Developers should also describe the form of the content (e.g., a game, bibliotherapy), the ways in which the program is interactive, and the technological strategies used to present therapy content, such as the use of text, audio-visual methods, animations, and gaming techniques.

We further recommend that especially detailed information be provided about exposure because it is considered a key strategy in CBT programs for CYP anxiety disorders (Kendall et al., 2006; Silverman and Kurtines, 1996). For example, information about when exposure is introduced in the treatment and how many sessions involve exposure is important. Although equally important to capture, the amount of exposure the youth engages in outside the program has been hard to quantify in face-to-face treatment trials and it may be that technology could be developed that reliably collects that information.

The amount of parental involvement also needs to be specified. If parents have their own parallel treatment program, then information pertaining to parent program duration, intensity, content, and therapist support should be provided. If parents do not have their own parallel program, then a description of how parents are involved in treatment should be specified since both parental behaviors and how they are involved in treatment have been found to be associated with treatment outcome (Breinholst et al., 2012; Manassis et al., 2014).

Research ethics and evaluating iCBT

As stated above, many different parties are involved in the development of iCBT, including academia, industry, CYP, parents, clinicians, funding agencies, and policy makers. When evaluating the effectiveness of iCBT programs, these different parties may have different interests in the results and a potential risk of bias arises in reporting on the results. We realize that this risk of bias is also present when evaluating regular “paper” CBT programs. However, publishing companies are typically not part of the development phase of a program, and are more likely to commercialize the program after its effectiveness has been established in academia. In contrast, a technology company is vital to the development of iCBT. Depending on the funding of the trial, the technology party often substantially invests in the new iCBT product, it carries financial risks, and anticipates of profit.

We recommend the research group to take measures to minimize the risk of bias in reporting on clinical trials. Preregistration of the effectiveness trial is essential, registering the most important aspects of the trial including primary and secondary outcome measures and endpoints. Involving an independent statistician to conduct the statistical analyses according to the preregistered plan is essential. If feasible, it may be useful to install an independent data and safety monitoring board. When reporting on results of a trial, authors should disclose their potential financial and personal (career) interest in the intervention, and the role of the technology company and the funding agency in the reporting of the results should be clarified.

Dissemination and adoption of iCBT

The ultimate objective for developers of iCBT is to produce a program that results in meaningful benefits for CYP with anxiety difficulties. For this to happen, developers need to consider how the program will reach, and ultimately be used by, CYP. A dissemination strategy is needed from the outset of iCBT development that can be implemented following proper and sufficient evaluation. Within this strategy, consider the desired breadth of the dissemination and which countries, sectors or ‘markets’ are most suitable for the iCBT program. Potential ‘markets’ for iCBT programs may include healthcare systems (public and/or private), education systems, and direct to consumer. Some developers may decide to make their program freely available as a website, app or downloadable program, although as noted before, it is important to recognize that digital delivery isn't free and it will be important to establish who will pick up any costs.

ICBT can move from research into real world adoption and use following the Commercial Approach (working collaboratively with industrial partners) or an In House Approach (where the developer takes responsibility for dissemination), or a combination of the two. We will now consider some of the benefits and challenges associated with different models for dissemination.

A partnership with a commercial company can bring expertise in sales, marketing, preparing contracts, IT/hosting and data protection. The partner may also provide an infrastructure and process for program maintenance, support and user inquiries. A commercial company may also have existing distribution channels into health services and/or insurance companies. Entrusting the dissemination to a separate company releases researchers to do research, which may be attractive. However, challenges with the commercial approach include a possible lack of control over how the iCBT will be used and the quality and rigor of implementation. It is important that rigorous due diligence is conducted in the process of selecting a commercial partner to increase the chance that the company is financially stable. The agreement between parties needs to address issues such as what will happen to the program if the company is sold, who retains the IP if the company fails financially, what happens if the company doesn't perform as expected, and whether the developers/researchers retain access to the program for research.

The degree to which these various factors present risks will vary according to the different ways in which commercial partners can be used. Considerations involving ‘Who will do what?’ and ‘How will it be financed?’ are core to any commercial partnership. For example, the dissemination of Child Anxiety Tales (www.copingcatparents.com), a parent training program for parents concerned about their child's anxiety (Khanna et al., 2017), involved the program developers having a service company make the program available for purchase on the internet. One benefit inherent in this approach is that the developer keeps control of the program. However, this approach may require the developer to acquire new skills (e.g. business models, marketing, legal issues), and therefore takes time and resources to both run and update the business, the technical platform, the program content, and the daily transactions with users.

Some developers might choose to work in partnership with a healthcare provider, and thereby tailor the development of the program to a specific context and plan for implementation (e.g., Vigerland et al., 2016). It is beneficial to identify and partner with a healthcare provider who has the knowledge and necessary organization to make dissemination possible. In some countries, it may be possible to integrate iCBT into an existing mental health clinic, such as the Internetpsychiatry unit for adults in Stockholm, Sweden (http://web.internetpsykiatri.se/en/). The benefits associated with this approach include the developers retaining control over how the program is used and how therapists are trained, and it potentially requires less time and effort to implement the program into different clinics. The disadvantage is that this approach is reliant on commissioners and health care services being willing to fund such an effort. Also, dissemination may be limited to the catchment area of the clinic where the program was developed.

Other things to consider when developing a partnership approach include technical matters. For example, what to do if the technical platform developed to deliver the iCBT program is no longer supported by the partner? Who will be responsible for the stability and security of the platform, and how will maintenance, support, bug fixes and updates be financed. If a program is initially used locally or at one site, this might not be an issue, but as the number of users and sites increase, there will be related issues that threaten sustainability.

It is also important to consider how training will be provided when disseminating iCBT. Researchers could provide online training in the program, for example as done by the developers of Cool Kids (https://openmq.com.au/course/CKAP01), and Coping Cat, the C.A.T. Project, and Camp Cope-A-Lot (through CBT4CBT) or through face-to-face or video conference training as done in the BiP project in Sweden (Jolstedt et al., 2017) and BRAVE Program in Australia. However, the scalability and sustainability of this approach to iCBT therapist training should be considered.

Future steps, summary and conclusions

Technological innovation continues to develop at a rapid pace and presents an exciting opportunity for new and enhanced ways to increase access to and potentially improve the effectiveness of psychological treatments for CYP anxiety. To harness the full potential of the evolving technology, we underscore the need for academic developers of iCBT and crucially their funding bodies to consider how best to keep pace with the tech world whilst retaining scientific credibility and ensure that iCBT outputs from research can be implemented in a timely manner within routine clinical practice. Going forwards, we advocate that those from the psychological field collaborate with experts from other disciplines and industries, such as e-learning and gaming, to maximize the potential for digital mental health innovations such as iCBT and newer technologies (e.g. virtual reality, augmented reality, artificial intelligence, and wearables). Whilst these principles apply to all service users, this is particularly salient for CYP who may be early adopters of new technological innovations and have particular potential relevance for exposure, an integral part of anxiety treatment.

To accommodate the challenges that working in the digital space brings, a paradigm shift is needed in the typical process of obtaining a grant to deliver a research project before thinking seriously about dissemination. For example, negotiating ownership of IP and dissemination is essentially a commercial venture (albeit with the possibility to provide digital services on a not-for-profit basis) that requires skills beyond those of academic grant holders. Early stage consideration of these factors are necessary before output from digital research projects can succeed in reaching routine clinical practice, and ultimately improve mental health service provision for CYP.

ICBT for CYP anxiety is a more developed field than iCBT for other child mental health problems (Pennant et al., 2015), so the conclusions drawn here are based on our extensive learning from iCBT work in the CYP anxiety field. However, they are likely to apply more broadly and hopefully can be used to support the development and dissemination of iCBT not just for anxiety disorders but also for other mental health problems in children and young people.

Declaration of interest

Claire Hill and Cathy Creswell are the developers of OSI Anxiety (Online Support & Intervention for Anxiety), an online therapist-supported, parent-guided CBT program for child anxiety disorders and child game application. Cathy Creswell is funded by a National Institute of Health Research (NIHR) Research Professorship (2014-04-018). Maaike Nauta developed and translated CBT treatment manuals for youths, including a blended internet-based treatment program and the Dutch Coping Cat manual, for which she receives no personal fees. Maaike Nauta has received grants from ZonMW (The Netherlands Organization for Health Research and Development) for evaluating effectiveness of youth interventions. Maaike Nauta receives travels expenses, some subsistence and sometimes an associated speaker honorarium for lectures or clinical training workshops. Philip C. Kendall receives royalties from the sale of materials related to the treatment of anxiety in youth. Sonja March, Susan Spence and Caroline Donovan may in the future receive income from the commercialization of the BRAVE program. Maral Jolstedt, Eva Serlachius and Sarah Vigerland have developed the BIP Anxiety program and Martina Nord is co-developer of the BIP SOFT program. They recieve no personal income from these programs.