Teens Engaged in Collaborative Health: The Feasibility and Acceptability of an Online Skill-Building Intervention for Adolescents at Risk for Depression.

BACKGROUND: There is an ongoing need for effective and accessible preventive interventions for adolescent depression and substance abuse. This paper reports on a field trial of an online indicated preventive intervention, ProjectTECH, which is based on cognitive-behavioral therapy (CBT) techniques. The study aims to gather information about the feasibility and acceptability of this program. Secondary aims of this study were to examine the impact of the program on depression symptoms, perceived stress, positive affect, and substance use and to compare differences between groups that were led by a peer versus those that were led by a licensed clinician.
METHODS: High school students (n = 39) were recruited primarily through social media advertisements, and assigned to four groups of 8-12 individuals to collaboratively participate in an 8 week peer network-based online preventive intervention which were led by a trained peer guide or a licensed clinician. Participants were provided with didactic lessons, CBT-based mood management tools, and peer networking features, and completed quantitative and qualitative feedback at baseline, midpoint, end of intervention, and 1 month follow up.
RESULTS: The program attracted and retained users primarily from social media and was used frequently by many of the participants (system login M = 25.62, SD = 16.58). Participants rated the program as usable, and offered several suggestions for improving the program, including allowing for further personalization by the individual user, and including more prompts to engage with the social network. From baseline to end of intervention, significant decreases were observed in depressive symptoms and perceived stress (p's < .05). Significant increases in positive affect were observed from baseline to midpoint (p < .05) and no changes were observed in substance use, although the rate of substance use was low in this sample. While this study had low power to detect group differences, no consistent differences were observed between participants in a peer-led group and those in a clinician-led group.
CONCLUSIONS: Results of this study indicates that ProjectTECH, an indicated preventive intervention for high school-aged adolescents, demonstrates both feasibility, acceptability, and short-term, longitudinal psychological benefits for participants. Future iterations of the program may benefit from close attention to user interface design and the continued use of trained peer support guides.

RCT Entities:   Population Interventions Outcomes

Introduction

Depression and substance use are common in adolescence, and frequently co-occur (Aseltine et al., 1998, Avenevoli et al., 2015, Nezami et al., 2005). A national survey of U.S. high school students found that 29.9% had felt so sad or hopeless almost every day for 2 or more weeks in a row that they had stopped doing some usual activities in the previous year (Kann et al., 2016). Other studies suggest that nearly 1 in 5 (17.3%) adolescent females meet criteria for a Major Depressive Episode (MDE) in a given year (Center for Behavioral Health Statistics and Quality, 2015), as do 5.7% of adolescent males. In the U.S., the onset for alcohol and drug abuse is typically during early adolescence (between 14 and 15 years old) (Swendsen et al., 2012). Depression during adolescence is significantly correlated with the development of other mental illness and psychosocial difficulties later in life, such as substance abuse (Thapar et al., 2012). Adolescents with a past year MDE are more than twice as likely to use illicit drugs in the same year compared to those with no MDE (Center for Behavioral Health Statistics and Quality, 2015), and have a three-fold increase risk of substance abuse disorder (Avenevoli et al., 2015). Further, we know that early alcohol use onset predicts a more chronic pattern of depressive symptoms (Cerda et al., 2013). Thus, prevention of both adolescent depression and substance abuse is vital to public health and can prevent individuals from initiating a trajectory of further suffering. A number of large trials have demonstrated that behavioral interventions aimed at adolescents and/or parents can prevent depressive episodes (Beardslee et al., 2003, Rohde et al., 2015), alcohol use (Brody et al., 2006), and both depressive symptoms and substance use (Pantin et al., 2009, Perrino et al., 2016). A synthesis of 19 trials found preventive effects on depressive symptoms lasted at least 2 years (Brown et al., 2016). However, these prevention programs often entail substantial costs and logistical demands that impede broad implementation (Brent et al., 2015, Young et al., 2015).

The internet has the potential to offer access to effective prevention programs broadly and cost-effectively. These interventions, which may be used both for treatment of individuals with depressive disorders and for prevention of future episodes among those with elevated symptoms, typically offer didactic lessons and interactive tools that can be used flexibly and often with more privacy than traditional face-to-face interventions. Typically, lessons can be completed any time of day or night, from wherever the user feels comfortable accessing the internet. Over the years, internet-based mental health interventions, particularly when accompanied by regular support via phone or messaging from a coach, have repeatedly demonstrated efficacy for treating a wide range of mental health concerns, including major depressive disorder and a variety of anxiety disorders (Andrews et al., 2010, Mohr et al., 2013). Coach supported, Internet-based mental health interventions have also demonstrated the ability to reduce the incidence of major depressive disorder among adults with subthreshold depressive symptoms (Buntrock et al., 2016), and similar results have been found in multiple studies with adolescents and college students (Clarke et al., 2015).

Internet-based mental health programs can be delivered through a variety of venues. While some internet-based mental health interventions have been deployed through primary care (Van Voorhees et al., 2009) or based on referral from mental health clinicians (Rice et al., 2016), most studies of internet-based mental health interventions focused on prevention with adolescents have been conducted through school systems (Calear et al., 2009, Calear et al., 2013, O'Kearney et al., 2006, O'Kearney et al., 2009, Attwood et al., 2012, Werner-Seidler et al., 2017). A recent review of internet-based prevention interventions for adolescents (Clarke et al., 2015) found that programs typically have moderate to high rates of program non-completion and that inclusion of face-to-face and/or web-based support was important for program completion and for clinical outcomes. Effect sizes for depressive symptoms in these trials have ranged from small to medium. For example, O'Kearney et al. (2009) found no significant effect at end of treatment, but medium effect size on depression at 20-week follow-up (d = 0.46) and Calear et al. (2009) found no significant effect on depression symptoms for females, d = 0.06, and a small to medium effect for males at post-intervention and 6 month follow-up (Cohen's d ranged from 0.27–0.45). While universal interventions (including those delivered in school settings) are often expected to have low rates of completion due to many people being offered the intervention who do not need it, we recognize the lack of access that many adolescents face in seeking mental health care (Gulliver et al., 2010), and that understanding ways to reach adolescents in need and ways to keep them engaged in interventions is vital to the success of these programs.

Human support, provided individually and in group formats, may help users stay engaged in these programs. The inclusion of human support in internet-based mental health interventions has varied, with many mirroring traditional face-to-face therapy sessions with weekly communication from a support person, who is typically a mental health professional, provided over the telephone or via messaging. However, little is definitively known about how much and what type of support is necessary for enhancing outcomes (Schueller et al., 2016), and there is evidence that support provided by a lay person (i.e., non-mental health professional) is as effective as support provided by a clinician (Titov et al., 2010). The question of who should lead or moderate these types of interventions has not yet been clearly answered and deserves further investigation.

A growing body of evidence suggests that collaborative peer support may increase young adults' program completion and engagement in internet-based mental health interventions (see Rice et al., 2016). Findings from a systematic review of peer support interventions for depression provided in any format (group, pairs, telephone, or in person) indicate that peer support interventions reduce symptoms of depression compared to usual care in a low-cost manner, and the reduction in depressive symptoms is comparable to group CBT (Pfeiffer et al., 2011). A combination of professional and peer support has shown clinical benefits, and appears to afford several advantages, including increased engagement, access to social support, and decreased costs in supporting the intervention (Duffecy et al., 2013, Ho et al., 2016). Existing trials of internet-based mental health interventions making use of peer support have mainly targeted young adults rather than adolescents, and have mimicked group therapy in real-time chat sessions (van der Zanden et al., 2012), mimicked one-to-one peer support through assignment of a peer mentor (Bohleber et al., 2016), been comprised of a peer moderated message board delivered separately from structured online modules (Ellis et al., 2011), or deployed in conjunction with a professional care team (Rice et al., 2016). To date, there has been a lack of internet-based depression prevention programs with peer or clinician support designed for broad implementation.

This paper reports on a feasibility and acceptability trial of our Teens Engaged in Collaborative Health (ProjectTECH) treatment program. ProjectTECH is an internet-based, indicated, preventive intervention, based on cognitive-behavioral therapy techniques, for adolescent depression and substance abuse. The primary aims of this study were to gather information about the usability and usage of this program. To develop programs that are engaging and capable of producing change in individuals, it is important to first test user experience with a minimum viable product (MVP; (Münch et al., 2013) and subsequently optimize the program based on user feedback. This version of ProjectTECH is an MVP, or early prototype intended to gather feedback on design requirements. Here, we expected that ProjectTECH would be generally usable, however in testing this MVP, we were exploring how to best design digital, peer-enabled interventions for youth and expected to gather information on areas for design improvements. In this trial, we hypothesized that participants would report reductions in depression symptoms, perceived stress and substance abuse, and increases in positive affect. We explored the feasibility of recruiting through community venues. While we were not powered to conduct non-inferiority analyses, we nevertheless aimed to see if there was any indication that our hypothesis that peer support would be equivalent to clinician support might be incorrect.

Methods

Recruitment and eligibility criteria

High school students were recruited from February to April of 2016. In order to reach a broad, diverse sample of adolescents, recruitment primarily took place through advertisements on social media (Instagram), as well as through schools and other community settings. Eligible candidates were between 14 and 19 years old and currently in high school, could read and speak English, owned a personal smartphone, and met symptom score criteria on either the Center for Epidemiological Studies-Depression Survey (CES-D; Radloff, 1977) indicating presence of depression symptoms (12–39 for males and 15–39 for females) or on the Youth Risk Behavior Survey (YRBS; Centers for Disease Control and Prevention, 2015) indicating use of cigarettes or alcohol in the past 30 days or any lifetime use of marijuana or other illegal drugs. Adolescents with a self-reported mental health diagnosis for which participation in the trial would be inappropriate (e.g. bipolar disorder, schizophrenia, severe substance abuse) were excluded. Adolescents' mental health histories were reviewed further by a clinical psychologist before being included in the study to ensure the diagnosis would not make the trial inappropriate for the individual and a higher level of care were needed.

Adolescents were first screened through a brief, web screener. They were then contacted by phone and provided further information on the study. For those interested, consent was obtained using an online consent form with e-signature. For adolescents under 18, parental consent was obtained along with child assent in a separate online form emailed to the parent, which also included the parent's e-signature. These procedures were approved by the Northwestern University Institutional Review Board. Those eligible at web screening were passed on to a baseline eligibility assessment, consisting of an online questionnaire and a 30-minute phone interview. During the baseline phone interview, further information was gathered regarding demographics, mental health history, suicidality, and smartphone use.

Study design

Eligible participants were assigned to groups of 8–12 participants who collaboratively moved through the 8-week intervention together. Participant group assignment was based upon time at which participants were recruited and deemed eligible for the study. A total of 4 groups were conducted, two led by a peer guide, who was a high school student close in age to the participants, and two led by a clinician guide. The needs of this pilot study did not warrant the significant expense and logistics required for random assignment into groups, which would require accumulation of sufficient participants meeting criteria at study start to be randomized to two groups.

Participants were asked to complete follow-up online questionnaires at midpoint (4 weeks into the intervention), at the end of the intervention (8 weeks), and 1 month after the intervention was over (12 weeks). Participants were also asked to complete user feedback interviews over the phone with research assistants at 4 weeks and 8 weeks after they started using ProjectTECH. During this call, they were asked questions regarding the lessons and tools on the site, the value of the peer network and guide, how the site had been helpful, and what elements needed improvement. Participants were compensated for completing follow up assessments via Amazon credit.

Intervention

The ProjectTECH program was built on the ThinkFeelDo intervention framework (Schueller and Mohr, 2015) which is a responsive webapp that can be accessed from mobile devices as well as computers. ProjectTECH includes didactic lessons and tools based on cognitive behavioral principles. Lessons were released five times per week and required about 5 min to read. Example stories of two fictional high school students (Brandon and Taylor) are used as examples of how each CBT concept and tools can be implemented. Each lesson concluded with a call to action to practice skills by using site tools or communicate with the peer group (see Table 1 for lesson topics by week). In total, participants received 40 new lessons and 9 lesson summaries during the 8-week program.

Table 1

Program lessons.

Week #	Lesson 1	Lesson 2	Lesson 3	Lesson 4	Lesson 5
1	Feel Tool	Introducing Brandon and Taylor	Thoughts, Feelings & Behaviors: The DO tool	Accomplishment and Pleasure	Activities to Boost Mood
2	Harmful Thoughts and Unhealthy Choices	“I already know all about drugs and drug use…”	Identifying Patterns of Harmful Thoughts	Challenging Harmful Thoughts	Acting AS IF You Believe Your RE-Thinking
3	SMART Goals	Long-term and Short-term Goals	Importance of Social Support	The Good, The Bad, The Neutral	Healthy Relationships Start with You
4	Assertive Communication	Nonverbal Communication	You've Been Here Almost 4 Weeks!	The Best Kept Secrets About Achieving Goals	Reviewing Your Activities
5	Healthy Communication: Active Listening	Healthy Communication: Setting Boundaries	Healthy Communication: Dealing with Anger and Irritability	Anxiety	Types of Anxiety
6	Anxious (Harmful) Thoughts	What Makes Us Anxious	How to Relax (a.k.a. Stress Management)	Muscle Relaxation	Visualization & Performance
7	Positive Psychology and Strengths Building	Strengths and Values: Courage	Strengths and Values: Restraint	Strengths and Values: Transcendence	Strengths and Values: Wisdom and Knowledge
8	Writing Your Own Prescription	Working with the Mind	Social Support and Asking for Help	Dealing with Difficult Emotions	Staying Positive About Yourself

As seen in Fig. 1, the main CBT based tools were found under the THINK, FEEL and DO tabs. THINK provided cognitive restructuring techniques such as challenging negative thoughts, FEEL encouraged users to track their mood, and DO included behavioral activation techniques like planning activities and reflecting on the amount of enjoyment and accomplishment certain activities brought users. Participants also had access to a goal-setting tool, named ACHIEVE, and a series of relaxation recordings (under RELAX). For examples of the THINK, FEEL and DO tools, see Fig. 2, Fig. 3, Fig. 4, respectively.

Fig. 1

ProjectTECH webapp main menu.

Fig. 2

DO tool.

Fig. 3

THINK tool.

Fig. 4

FEEL tool graph.

The peer group was reflected on ProjectTECH via a live activity Feed on the homepage, plus features similar to those found on Facebook, such as commenting, likes and nudges (See Fig. 5). Participant activities such as reading lessons, entering a goal, etc., are posted to the Feed. Participants could comment or like others' achievements or activities that were posted to the Feed, which is visible in Fig. 5. However, participants had the option to share or not share certain activities on the Feed. The peer or clinician guide, as well as participants, also had the option to nudge other users when they were not active on the program. Each participant also created a profile on the program where they answered brief questions about themselves and shared personal interests (Fig. 6).

Fig. 5

Sample participant feed.

Fig. 6

Sample participant profile.

Program guide

The peer and clinician guides offered similar levels and types of support to their groups. The clinician guide was a child clinical psychologist (J.H.) who developed the guide protocol for ProjectTECH, and the peer guide was a high school student with a background serving as peer mentor and mental health group facilitator at local high schools. We were interested in exploring how participants viewed support provided by a clinician guide (i.e., an adult with formal clinical training) compared to a trained peer guide, and in examining how these differences might influence use.

In each group, the guide conducted a videochat orientation with participants in the group and then remained available throughout their participation in the 8-week online intervention to facilitate discussions, encourage program use, and be the point of contact for any technical difficulties experienced by participants. During the 8-week trial period, the adult guide (J.H.) worked closely with the peer guide to develop ways of interacting that both guides used to facilitate program activity and to encourage collaborative learning between group members.

In addition to familiarizing herself with the guide protocol, the peer guide received didactic training on basics of the cognitive-behavioral approach to depression prevention in adolescents, training on using the coach dashboard to moderate groups, and also conducted 2 mock orientation video calls with study staff to prepare for her role. Such training was not necessary for the clinician guide given her role in intervention development and her clinical training.

Outcomes

Primary outcomes were the usage and usability of the program as led by a peer or a clinician guide. To measure these outcomes, program usage data were examined in addition to the qualitative user feedback interview, the system usability scale (SUS; (Brooke, 1996) questionnaire, and a modified version of the Usefulness, Satisfaction and Ease of use questionnaire (USE; Lund, 2001) particularly regarding the participants' relationships with the peer network. The SUS questionnaire is a 10 item measure that assesses usability, acceptability and satisfaction. The USE questionnaire is a 19 item measure of usability with 4 subscales: Usefulness, Ease of Learning, Ease of Use, and Satisfaction.

Clinical outcomes were secondary, as there was no control condition and insufficient time to observe preventive effects. The clinical outcomes examined included depressive symptoms, positive affect, perceived stress, and alcohol and drug use. Depressive symptoms were measured using the 20 item Center for Epidemiological Studies-Depression Survey (CES-D; Radloff, 1977). Scores ≥ 12 for males and ≥ 15 for females indicates the presence of elevated depressive symptoms. Positive affect was measured using the PANAS (Watson et al., 1988), a 10 item measure. Perceived stress was measured using the 10 item Perceived Stress Scale (PSS; Cohen et al., 1983). Scores ≥ 20 on the PSS indicate a high level of stress (Cohen and Williamson, 1988). Alcohol and drug use was measured using the Youth Risk Behavior Survey (YRBS; Centers for Disease Control and Prevention, 2015).

Data analyses

Descriptive statistics were computed to examine participant demographics, program usage and clinical outcomes. t-Tests were used to examine for differences in system usage between participants in the peer-led condition and participants in the clinician-led condition. Descriptive statistics are presented using raw data, and multiple imputation was used to handle missing follow-up assessment data for further analyses. Following guidelines by Graham et al. (2007), 20 imputed datasets were created using SPSS, and results were pooled. Analyses of variance (ANOVAs) were used to examine potential differences in system usability ratings between study conditions. Using intent-to-treat analyses, repeated measures ANOVAs were used to examine changes in depressive symptoms, positive affect, and perceived stress over time. Effect sizes were calculated and partial eta-squared values were interpreted using adopted conventions (i.e., 0.01 = small, 0.06 = medium, 0.14 = large). We also conducted similar repeated measures ANOVAs to explore differences in the course of these behavioral outcomes by peer or clinician supported conditions. Because there were only two groups in each condition, we ignored variation at the group level, something we would account for in a larger study (Murray, 1998).

Results

Study sample

As seen in Fig. 7, a total of 445 adolescents completed a brief online eligibility screen, a total of 67 adolescents consented to participate in an additional online questionnaire and telephone interview to determine eligibility, and 40 were eligible and enrolled in the study. Of the 40 eligible, 23 were eligible based on elevated CES-D scores alone, 3 were eligible based on substance use alone, and 14 were eligible based on both elevated CES-D scores and substance use.

Fig. 7

Participant flowchart.

Primary reasons for ineligibility included low CES-D scores, lack of substance use, comorbid mental health diagnoses that could interfere with participation in the program (e.g. bipolar disorder, dissociative disorder) and not being enrolled in high school. One participant withdrew prior to beginning the program, and was excluded from these analyses. One participant withdrew from the study after the program began, and was included in these analyses. As seen in Table 2, the sample was racially and ethnically diverse. The majority of participants had elevated symptoms of depression (CES-D M = 20.53, SD = 10.86), and endorsed high levels of perceived stress (PSS M = 23.89, SD = 3.31). Recruitment via social media was successful, with 28 of the 40 eligible participants having been referred to the study through Instagram advertisements, and an additional 9 eligible participants being referred through word of mouth.

Table 2

Participant demographics.

	M (SD)
Age in years	16.23 (0.99)
	n (%)
Gender
Male	9 (23.1%)
Female	29 (74.4%)
Other	1 (2.5%)
Race
White	23 (61.5%)
Black/African-American	3 (7.7%)
Asian	4 (10.3%)
More than one race	4 (10.3%)
Decline to report	4 (10.3%)
Ethnicity
Hispanic or Latino	10 (25.6%)
Not Hispanic or Latino	29 (74.4%)

There were no significant differences in baseline levels of depressive symptoms (CES-D), perceived stress (PSS), positive affect (PANAS) or use of substances (YRBS) between participants who were placed in a peer-led group compared to those who were placed in a clinician-led group (all p's > 0.05). There were no differences in rate of follow-up assessment between groups.

Program usage

Participants in the study tended to access the program multiple times and explored the program tools. The median number of system logins was 26 across the 8 week trial (M = 25.62, SD = 16.58, range = 0–82) and the frequency of logins decreased as the weeks progressed (see Table 3).

Table 3

Program logins per participant by week.

	Week 1	Week 2	Week 3	Week 4	Week 5	Week 6	Week 7	Week 8
Mean	8.81	4.86	3.14	2.65	2.49	2.24	1.43	1.65
SD	4.03	5.05	2.36	2.50	1.94	2.22	1.82	1.95
Median	8	4	3	2	3	2	1	1
Range	0–17	0–29	0–11	0–11	0–7	0–9	0–7	0–9

The median number of lessons read was 39 (M = 35.75, SD = 15.16, range = 0–49). Participant use of the mood rating tool was variable, with the median frequency of ratings being 6 (M = 9.41, SD = 11.59, range = 0–61). Most participants entered thoughts into the THINK tool at least a few times (median = 4, M = 4.08, SD = 3.10, range = 0–14) and entered several activities into the DO tool (median = 22, M = 27.95, SD = 25.15, range = 0–130). Most participants used the ACHIEVE tool infrequently (median = 2, M = 2.95, SD = 3.68, range = 0–16), and many did not use the RELAX tool (median = 0, M = 1.18, SD = 2.17, range = 0–12). There were no significant differences in any of the tool use metrics between participants who were placed in a peer-led group compared to those who were placed in a clinician-led group (all p's > 0.05).

Participants used the built-in social networking components of the program less frequently, with 16 participants (41%) never using the Like feature, 16 participants (41%) never using the Comments feature, and 25 participants (64%) never using the Nudge feature. Of the 23 participants who used the Like feature, number of uses ranged from 2 to 17 (median = 3, M = 5.52, SD = 4.63). Of the 23 participants who used the Comments feature, number of uses ranged from 1 to 17 (median = 3, M = 4.78, SD = 4.17). Of the 14 participants who used the Nudge feature, number of uses ranged from 1 to 38 (median = 3.5, M = 6.50, SD = 9.52). There were no differences between groups on use of the Comment or Like features (p's > 0.05), but close examination of Nudge feature use revealed that the majority of use took place in one of the peer-led groups. In the first peer-led group, the mean number of Nudges by participant was 9.13 (SD = 12.23), while the mean number of Nudges by participants in the other 3 groups ranged from 0.25 to 1. This difference was statistically significant, F(3, 35) = 5.006, p = 0.005, and demonstrates the creation of group-specific norms or trends in social network feature use.

Usability and user feedback

On the USE questionnaire (Lund, 2001), a score of 4 indicates the midpoint between “strongly disagree” and “strongly agree” and according to industry standards, a score of 68 on the SUS questionnaire (Brooke, 1996) indicates average usability. As seen in Table 4, results from the USE questionnaire and SUS questionnaire indicate that the ProjectTECH program was viewed as fairly average or neutral by participants, and could benefit from further improvement. There were no significant differences in these usability measures between participants who were enrolled in a peer-led group compared to those who were enrolled in a clinician-led group (all p's > 0.05). In this study, the SUS and USE questionnaires demonstrated internal consistency (SUS α = 0.95; USE Usefulness subscale α = 0.93, USE Ease of Learning subscale α = 0.98; USE Ease of Use subscale α = 0.95, USE Satisfaction α = 0.96).

Table 4

Usability questionnaire descriptive statistics.

	Midpoint	End of program
	M (SD)	M (SD)
USE
Usefulness	4.21 (1.42)	4.44 (1.32)
Ease of Learning	5.37 (1.77)	5.55 (1.38)
Ease of Use	4.96 (1.65)	4.99 (1.36)
Satisfaction	4.18 (1.75)	4.26 (1.67)
SUS	67.43 (18.02)	67.50 (18.10)

Results from midpoint and end of program user feedback interviews identified multiple positive aspects of the ProjectTECH program, in addition to several opportunities for growth and program optimization. On a 10-point scale with 1 meaning “not at all” and 10 meaning “the most,” participants tended to rate the lessons as easy to understand (M = 9.08, SD = 1.08), interesting (M = 7.42, SD = 1.50) and relatable (M = 7.87, SD = 1.08).

Program guide

Half of the participants (n = 19) voiced appreciation for their guide (both peer and clinician) as someone who was there if there were questions and provided support, while many (n = 13) commented they expected more support from the guide. There were not notable differences in how participants from the peer-led groups described their guide compared to participants from the clinician-led groups. Example quotes:

“I think she was nice. It wasn't intimidating—it was something official, but not cut throat. Very easy to talk to.”

“I really liked how there was an option where you could contact them and that was really helpful cuz you could, if I had any problems I could always talk to her, I never used it, but that was really reassuring, like if anything happened.”

“The interaction was infrequent. I thought they would lead us in discussion. They were just focusing on telling us to sign in. They didn't really facilitate discussions or help us learn the content.”

Program benefits

Multiple participants (n = 17) reported that they found it particularly useful to set and then track progress on their personal goals, and many participants (n = 14) noted that they learned new ways of coping with stress through participation in this program. Example quotes:

“I think it encourages me to actually pay attention to [my goals], before I didn't really have anything to say about when I was going to do the goals or not.”

“I'm usually stressed out especially because it's the end of the year. One thing I thought was really helpful was to learn how to relax and not stress myself out. It helped me be more positive about my experiences. It taught me that I'm not the only one feeling this way. Because of the program I was able to shift my mindset. That was pretty incredible.”

Requests for general system modification

While the program was designed to be mobile phone-accessible and available as a website and webapp, several participants (n = 8) noted difficulties in utilizing program components on the mobile (webapp) version of the program compared to the desktop (website) version. When prompted to provide suggestions to improve the ProjectTECH program, most participants requested further personalization of the program to better meet their needs. These suggestions included reducing the frequency of lessons and reminders, and allowing for the participant to choose their automated reminder times. Several participants (n = 7) noted that the text message reminder arrived a time that didn't work well for them, and that they would have liked the option to set a personalized time. Other suggestions included more personalized choice in what would be displayed on the program's feed, as well as the presence of a more detailed, automatic “to do” list on the main page which the participant could update as desired. Example quotes:

“I haven't seen any problems, it's just a little more glitch on the mobile version. It would be easier if you created an actual app for a phone than a mobile phone version.”

“I would get the reminders at school and by the time I go home, I'd be busy or I would just forget until I got the reminder the next day … If I could set it to a more convenient time, I would do that. It was always right in the middle of class for me, so it wasn't doing much.”

Requests for peer network modifications

The majority of participants who participated in the user feedback interview (n = 19) reported that they didn't turn to the peer group features often; however, this appeared to be due to problems with the design of the peer group features rather than a lack of interest in engaging with their peers. They appeared to want interaction, but were not interested in or motivated towards using the social networking features embedded in the program. For example, there were multiple requests for more prompted opportunities for interaction with group members, such as a chat room, or regularly scheduled collaborative discussions, while the features in their current form require self-initiation such that prompts for group discussion were located in the lessons rather than on the Feed. Many commented on the relatively poor usability of the Feed, and while they found it helpful to see examples of how others used the tools, they didn't find it helpful to see when others read lessons. Example quotes:

“I wanted a little more group interaction, since it was all through the feed, it was encouraged that people talk to each other but no one did. At the beginning, there were a lot of people saying their problems and I could read through and see everyone's comments on it. But by the end, everyone got tired of it and stopped trying.”

“Having prompts for people to talk about things in the peer network. We definitely could have gone and posted on the feed, it just depends on the people in the group.”

“I wish I could control what goes on there, I didn't really want to share some things. It would say when somebody completed a lesson, I didn't think that was really relevant cuz a lot of people wouldn't do a lesson every day, they'd do like 7 a day and it would clog it up. The only thing I would change is choosing what goes on the feed and taking out the completed lessons from the feed.”

Several participants (n = 5) suggested that the program could be improved by providing opportunities for more interaction on site other than reading, such as games or other gamification strategies to enhance general enjoyment of the site. Example quotes:

“It starts to feel like a chore. Maybe after [you] fill in a certain tab, there could be a game you could play after it. Some motivation for it after - try to beat your score. Make it more than just mental health check.”

“[I wanted] more activities and interactive pieces and more hands-on stuff instead of just reading the material.”

While most participants described areas for improvement, many remarked on the potential for this type of program to make a difference in the lives of teenagers. For example, at end of intervention, one participant commented:

“I think the program has a good idea and a nice target audience. The hard part is where I come from, we do most of the lessons in our health classes because our school is well-funded. We have speakers who come in and talk about drugs and dating violence, so keep that in mind when you're looking at my response. If it looks like I'm not learning anything in my survey, it's just because I'm already learning it in school. There are underprivileged kids who don't have these things. The internet is a good way to reach a lot of people. It's a great idea for kids who aren't getting those things in their classes.”

Psychosocial outcomes

Mean scores on the CES-D, PSS, and PANAS at each assessment time point can be seen in Table 5. There were no significant differences in baseline CES-D, PANAS, or PSS scores between intervention conditions (p's > 0.05), and all of the scales demonstrated adequate internal consistency (CES-D α = 0.85, PANAS α = 0.96, PSS α = 0.92). There were also no significant differences in these measures between those who completed end of program and one month follow-up assessments and those who did not complete these assessments (p's > 0.05).

Table 5

Descriptive statistics for psychosocial outcome measures.

	PSS	CES-D	PANAS
	M (SD)	M (SD)	M (SD)
Full sample
Baseline	23.89 (3.31)	20.53 (10.86)	31.32 (7.74)
Midpoint	23.30 (3.82)	19.50 (10.45)	34.38 (7.72)
End of intervention	22.94 (5.99)	19.34 (13.27)	34.22 (9.17)
One month follow-up	22.09 (3.97)	15.28 (10.30)	36.19 (8.51)
Peer-led
Baseline	25.00 (3.26)	17.69 (6.89)	32.06 (8.42)
Midpoint	24.38 (3.93)	17.19 (9.64)	38.50 (8.17)
End of intervention	24.00 (8.12)	18.94 (13.03)	34.07 (11.18)
One month follow-up	23.19 (4.92)	16.56 (10.74)	36.63 (9.22)
Clinician-led
Baseline	23.16 (3.06)	23.53 (12.88)	30.79 (7.47)
Midpoint	22.81 (2.90)	22.29 (10.84)	30.88 (5.28)
End of intervention	22.31 (3.00)	20.73 (13.83)	34.13 (7.49)
One month follow-up	21.00 (2.47)	14.00 (10.02)	35.75 (8.00)

We found a significant decrease in depressive symptoms (CES-D) with a medium effect size over time with a mean difference of − 6.48 (SD = 12.76), F(3,99) = 3.853, p = 0.01, ηp2 = 0.061. Comparing peer-involved with clinician-involved effects, we found no significant group by time differences. Additionally, we found a significant decrease in perceived stress (PSS) with a large effect size over time with a mean difference of − 2.42 (SD = 4.26), F(3,99) = 4.981, p = .003, ηp2 = 0.159, with no significant group by time differences.

Self-report assessments indicated a significant increase with a large effect size in positive affect as measured by the PANAS from baseline to midpoint assessment with a mean difference of 3.53 (SD = 5.29), F(1,36) = 17.89, p < 0.001, ηp2 = 0.321 with more significant gains made by those in the peer-led condition, F(1,36) = 7.39, p = 0.01, ηp2 = 0.165. As seen in Table 5, gains in positive affect among those in the peer-led condition decreased after midpoint assessment, whereas those in the clinician-led condition continued to report small increases in positive affect at subsequent time points which created a statistically significant cubic effect, F(1,36) = 7.09, p = 0.012, ηp2 = 0.167, with a large effect size.

As seen in Table 6, the number of participants who used cigarettes, electronic vapor products, alcohol, and marijuana was low and fairly consistent throughout the 12-week study period. Among the few participants who reported use of substances in the prior 30 days, frequency of use was typically low. For participants reporting use of alcohol, most reported consuming alcohol 1–2 days in the past 30 days, with the greatest frequency of alcohol use at 6–9 days in a 30 day period. Only two participants reported binge drinking (marked by the consumption of 5 or more servings of alcohol in a row), and both of these reports were single instances. It appears that use of marijuana fluctuated over the study period; however, most participants who used marijuana reported using 1–2 times in the past 30 days with just one participants reporting more habitual marijuana use (at the frequency of 10–19 times in the past month). There was no indication of differences in substance use by group condition.

Table 6

Number of participants who reported use of substances in the prior 30 days.

	Cigarettes	Electronic vapor product	Alcohol	Marijuana
Baseline	0	2	5	5
Midpoint	0	1	6	7
End of intervention	0	1	6	4
One month follow-up	0	1	6	3

Discussion

Results of this field trial indicate the feasibility and acceptability of ProjectTECH as an internet-based skill-building intervention for adolescents at risk for depression. The effort to increase adolescents' engagement in depression and health risk behavior prevention programs is inherently challenging, since treatment is not yet indicated and thus adolescents may not perceive the need for participating in such programs. Use of the program was higher than other online mental health interventions targeting adolescents that have primarily been conducted in school settings. The relatively high rate of program engagement may be due in part to our targeted sample (e.g. this was an indicated intervention rather than a universal intervention), in which adolescents responded primarily to social media advertisements, and in part due to ProjectTECH's design such as relatively short lessons, the availability of peer or clinician support and embedded peer networking features. The use of social media to recruit participants demonstrated feasibility for reaching adolescents (28 of the 40 eligible participants were referred to the study through Instagram advertisements) and retention throughout the 12-week study was high.

We observed high program completion when compared to similarly structured online programs for high school students. For example, an RCT evaluating an online depression prevention program with adolescents from Hong Kong secondary schools reported a median of 3 modules completed (out of 10), with only 26 out of 130 completing the intervention (Ip et al., 2016). A study evaluating the effectiveness of MoodGYM with adolescents in Norwegian senior high schools reported that only 8.5% of enrolled participants logged into this self-directed CBT based intervention website (Lillevoll et al., 2014). These are both unguided, individual interventions. However, in another study of depression prevention with MoodGYM in an Australian secondary school, even with in-classroom teacher supervision, only 32.7% of students completed all 5 modules (Calear et al., 2009), another Australian school study reported that only 29.8% of adolescents completed three or more of the five MoodGYM modules (O'Kearney et al., 2009). Differences observed between engagement with ProjectTECH and engagement in these trials may be partially explained by the setting through which participants were recruited and the universal nature of most of these programs. While those in the Ip et al. (2016) study were invited to participate in the program based the presence of mild to moderate depressive symptoms, the other school-based studies were universal rather than indicated prevention programs. All ProjectTECH users voluntarily presented for participation in the program and had elevated risk for depression at baseline, and thus were likely more motivated to participate. Indeed, Van Voorhees et al. (2009) found that the CATCH-IT program, an indicated module-based prevention program for adolescents in primary care (and thus participants had to have come in for an appointment and subsequently consent for further participation), had a higher rate of use with participants completing approximately two-thirds of the program exercises. These observed differences in program use may also be partially explained by program design. Modules within MoodGYM are somewhat long (approximately 30 min to complete), modules within CATCH-IT are slightly shorter than MoodGYM (approximately 15–20 min to complete), whereas modules in ProjectTECH were designed to be read in approximately 5 min.

Apart from some reports of difficulties with the mobile version of the program provided during user feedback, there were no significant usability issues discovered. While it is challenging to design programs that satisfy the needs of all potential end users, there were identified areas for improvement. The majority of participants reported that they hadn't used peer group features often, and objective program usage data supported those observations. However, participants appeared to want to interact with one another, but weren't satisfied with the existing peer-networking features. Some participants requested that there be more prompted opportunities to connect with their peer group, indicating that they wanted to use this feature, and the use of prompted opportunities may indeed set the norm for participants to interact with one another.

We observed the role of group norms being set, such that one peer group began to use the Nudge feature heavily, and use of that feature became more common among members of that group. In other groups, norms failed to be established - it appeared that some participants tested out the social networking features, and stopped using them when other participants did not readily engage. The role of group norms establishment in both network- and small-group based virtual communities and its potential for encouraging participation within the community has been previously established (Dholakia et al., 2004). For internet-based programs like ProjectTECH that target broad samples, fostering group norms appears to be important for maximizing social feature use. Program guides in this trial had not been directed to actively monitor and encourage the development of group norms within the network. Future iterations of this program may benefit from a more active protocol in which the program guide leads group participants through mutually establishing how they would like to use the social network tools, similar to the mutual establishment of group norms recommended in face to face therapy groups (Leszcz and Kobos, 2008).

As this trial did not include an individual arm with no peer networking features, it is unclear whether the availability of a collaborative peer group, and site features displaying peer use activity, may in itself have promoted site use, regardless of participant's active use of liking and commenting. In other words, just knowing that there are other same-age peers in the same intervention, and observing their activity and comments on the Feed, may have prompted teens to continue to log in. Still, the lack of social network use may be in part due to changing social networking preferences among high school students, and in part due to unmet expectations on behalf of the participants who simply wanted different features. This program was designed to mimic newsfeed features found on Facebook, and by the time of this trial, adolescents and young adults were leaving Facebook and increasingly using other programs, such as Instagram and Snapchat (Greenwood et al., 2016, Lenhart, 2015). While technology and social networking preferences change over time in the general population, this is especially true of adolescent for whom norms change very quickly. Several participants suggested the program could be made more enjoyable by incorporating more interactive methods, such as more gamification strategies. This feedback highlights the potential benefit of carefully assessing and iteratively designing technologies that will be not just usable, but pleasant to use so that users will continue to engage with them (Garrett, 2010). In line with more thoughtfully designing for the end user, several participants requested more capabilities to personalize the program to them – such as choosing when to receive text message reminders, and choosing what to view on their homepage. This preference toward personalization is frequently observed in research on information system design (Fan and Poole, 2006, McLoughlin and Lee, 2008), but should be more deliberately examined in further iterations of ProjectTECH in order to develop an increasingly usable, useful and enjoyable program.

While users identified areas for improvement and this was a relatively short program, we observed psychological benefits (reductions in depressive symptoms and perceived stress, and increases in positive affect) throughout the study period. Online programs for depression prevention in youth have shown mixed results, with published studies evaluating MoodGYM demonstrating inconsistent effects on depression between experimental groups (Calear et al., 2009 revealed a significant decrease for males and no change for females; Lillevoll et al., 2014 revealed no significant differences), while other studies demonstrated a significant decrease in depression scores (Ip et al., 2016, Van Voorhees et al., 2009). More recently, depression prevention programs for adolescents have also used other telecommunication technologies such as text messaging (Ranney et al., 2016, Whittaker et al., 2012). These feasibility studies showed that adolescents generally welcome interventions that are delivered using technology they already use. Therefore, it is important to continue evaluating internet-based depression prevention for youth so that programs that are engaging and effective can serve this public health need.

We observed a basement effect of relatively low substance use in this sample, with no apparent differences in substance use throughout the study period. While substance use was one of the potential criteria for inclusion in the study, we did not appropriately target a sample with frequent substance use and thus were unable to fully test the impact of the program on substance use. Given the limited follow-up, we are unable to ascertain if psychological benefits led to changes in substance use or prevented the development of substance abuse in the long-term. Cross-sectional relationships are often observed between poor mental health and substance abuse (Swendsen and Merikangas, 2000, Grant et al., 2004), and temporal relationships in which internalizing disorders increase risk for later substance use disorders have been broadly observed (O'Neil et al., 2011). However, research on the ability of mental health promotion or mental health treatment programs to prevent later substance abuse has been limited. For example, Brody et al. (2006) found a protective effect on alcohol use approximately 2 years after a preventive intervention and Kendall and colleagues (Kendall et al., 2004) previously observed that children who responded positively to anxiety treatment had lower rates of substance use involvement and related problems > 7 years later.

There was no evidence of any difference between clinician- and peer-led groups in program use, system usability or psychological outcomes. While we were not powered to test for differences, much less non-inferiority, this finding is in line with past work by Titov et al. (2010) indicating that individuals experienced similar clinical outcomes following iCBT whether led by a trained layperson or a mental health clinician. Given these similarities, the use of trained peer support can be considered for further implementation of this program as a method for reducing intervention costs while maintaining intervention effect.

Limitations

This study is not without limitations. The absence of control conditions precludes us from parsing the effects of the intervention technologies, the trained guide, the peer network, and the natural course of symptoms. Additionally, there was a lack of randomization to groups, which may have resulted in some slight cohort effects. For example, participants in the clinician-led groups began later in the school year and started summer vacation during their participation in the trial. With only two groups per condition, there was limited power to formally test hypotheses, such as differences between clinician-led and peer-led groups. The sample contained few male participants and thus did not provide adequate statistical power to test for gender differences in psychosocial outcomes observed in past research (Calear et al., 2013, Calear et al., 2009). Finally, there was a relatively low rate of substance use in the recruited sample, as the vast majority of participants qualified based on mood symptoms. A sample of adolescents with higher-risk substance use behaviors may have interacted differently with this program, and thus yielded different results.

Conclusions

The ProjectTECH intervention for teenagers at risk for depression demonstrated feasibility and acceptability, and was associated with short-term, longitudinal psychological benefits for participants. Future iterations of this program would benefit from close attention to program design in order to enhance the user experience of likely end users. This would include modifications to the design and functionality of the social network features, as well as broadening the extent to which users can personalize their experience, such turning on and off different features of the program. Future research is needed to determine the efficacy of this program relative to a control condition and over a longer period of time to more accurately measure if the program prevents onset of a depressive episode and/or subsequent substance abuse.

Conflict of interest

The authors have no conflicts of interest to declare.