A pilot evaluation of the SPARX-R gaming intervention for preventing depression and improving wellbeing among adolescents in alternative education.

AIM: The use of computerized mental health programs with vulnerable youth, such as early school leavers, remains relatively unstudied. This pilot study examined the feasibility of delivering a computerized cognitive behavioral therapy (cCBT) gaming intervention (SPARX-R) for young people (age 15-20 years) who have left school early and are attending Youthreach, an alternative education (AE) program in Ireland.
METHOD: Students (n = 146) from twenty-one Youthreach Centers were randomized to SPARX-R and no-intervention control. All students within the group were included in the study whether or not they were exhibiting heightened levels of depression. Program impact was examined on both negative and positive indicators of mental health, including depression (primary outcome), generalized anxiety, general mental wellbeing, coping and emotion regulation. Assessments were conducted at baseline and post-intervention (7 weeks). Participants that provided data at post-assessment (n = 66) were included in the analysis.
RESULTS: The participants completed on average 5.3 modules of SPARX-R with 30% (n = 9) completing the entire program. A significant improvement in emotion regulation strategies was detected, with expressive suppression decreasing significantly in the SPARX-R group in comparison to the control (- 2.97, 95% CI - 5.48 to - 0.46, p = 0.03).
CONCLUSIONS: Findings suggest that SPARX-R has a positive impact on emotion regulation. The lack of significant findings on other outcome measures may be attributed to inadequate sample size, and therefore, further research with larger samples are required to establish the effectiveness of the program in reducing depression and anxiety and improving psychological wellbeing among young people attending AE.

Introduction

There is increasing evidence supporting the delivery of computerized mental health programs to promote youth mental health and wellbeing (Clarke et al., 2015, Pennant et al., 2015). Computerized cognitive behavioral therapy (cCBT) programs particularly, have been shown to significantly reduce symptoms of anxiety and depression in youth (Calear et al., 2009, Fleming et al., 2011, Merry et al., 2012) as well as improve some aspects of positive mental health, such as self-esteem (O'Kearney et al., 2006), and mental health literacy (Lintvedt et al., 2013). However, adherence can be problematic with high dropout rates, with supported delivery, e.g., within education settings, being linked to increased adherence and better outcomes (Baumeister et al., 2014, Clarke et al., 2015). However, to improve the uptake and impact of computerized programs more also needs to be known about how young people use and engage with these programs and how adherence is linked to outcomes (Clarke et al., 2015, Fleming et al., 2016a, Van Gemert-Pijnen et al., 2011). Furthermore, the transferability of these findings to young people who are socially and economically disadvantaged, such as those who have left school early, remains relatively unstudied.

Approximately 11% of young people in the EU leave school before earning a school leaving qualification (European Commission, 2015). There is no one reason for early school leaving, but rather, it is a complex interplay of factors such as low socio-economic status, negative school experience, dysfunctional family backgrounds and personal problems (Farmer et al., 2003, Garnier et al., 1997, Harrington, 2008, Jimerson et al., 2000, Lamb et al., 2015, Stokes, 2003). Young people who are socially and economically disadvantaged are disproportionately affected by mental health problems (Reiss, 2013) and leaving school can further adversely affect the social and health outcomes of these already disadvantaged young people by bringing about feelings of inadequacy and worthlessness and limiting access to further education and the labour market (Freudenberg and Ruglis, 2007, Kaplan et al., 1994, McHugh, 2014).

In Ireland, the National Second-Chance Education Program, Youthreach, provides an alternative route for early school-leavers (age 15–20 years) to receive second level qualifications and vocational training. These students experience social, economic and educational disadvantage (WRC Social and Economic Consultants, 2007). Therefore, interventions that will enhance the mental health and resilience of these young people and equip them with the cognitive, social and emotional skills for life, school and work are needed to ensure healthy development and positive outcomes (OECD, 2015). Although early school leavers are at a high risk of experiencing mental health difficulties, previous studies suggest that an approach that focuses on promoting protective factors for positive mental health and wellbeing rather than treating mental health problems may be more acceptable to youth in AE (Fleming et al., 2016b, Fleming and Merry, 2012, Kuosmanen et al., 2017).

Teaching cognitive behavioral techniques to encourage healthier patterns of thinking and behavior decreases symptoms of anxiety and depression (James et al., 2015, Weisz et al., 2006). However, there is inconclusive evidence as to whether CBT delivered universally is effective in preventing depression in adolescents (Calear and Christensen, 2010). Nevertheless, computerized CBT delivered universally irrespective of risk has been shown to improve psychological wellbeing in healthy individuals (Powel et al., 2013). The delivery of computerized CBT within the AE setting has a number of advantages considering that resources are often limited and students may be reluctant to talk about mental health topics face-to-face due to perceived stigma (Fleming et al., 2016b). Because of stigma and social isolation, both students and staff also prefer universal delivery to a more targeted approach. Furthermore, many cCBT programs include lessons around building self-esteem, stress reduction and coping, which have relevance for all youth.

One program that appears promising is SPARX-R cCBT gaming program for promoting wellbeing and preventing low mood, stress and anger in youth. The program has a focus on managing strong emotions and teaching positive coping mechanisms, such as problem solving and help-seeking. SPARX-R is rich in multimedia and most text-based content is accompanied by audio, which is important considering common literacy difficulties among these students (WRC Social and Economic Consultants, 2007). Furthermore, elements of gaming utilized in SPARX-R, such as following a narrative, completing challenges and interacting with program characters, may improve engagement and facilitate learning (Wouters et al., 2013). Although the evidence on the effectiveness of serious games (“computerized interventions which utilize gaming for serious purposes” [Fleming et al., 2014, p. 1]) for improving mental health and wellbeing is still limited (Lau et al., 2017, Johnson et al., 2016), gaming interventions have shown promising results in the area of depression prevention and treatment (Fleming et al., 2014, Li et al., 2014).

SPARX-R is a revised version of the original SPARX cCBT for depression in youth, which has been shown to reduce symptoms of depression in educationally alienated teenagers in comparison to a wait-list control condition (Child Depression Rating Scale Revised (CDRS-R) baseline to 5-week mean change − 14.7 versus − 1.1, p < 0.001; Fleming et al., 2011). SPARX has also been found non-inferior to treatment-as-usual (face-to-face counselling) with adolescents (n = 187) at risk of developing depressive disorder (between group difference in CDRS-R improvement score 2.73 in favor of SPARX, 95% confidence interval − 0.31 to 5.77, p = 0.079; Merry et al., 2012). Furthermore, qualitative feedback from young people who participated in SPARX (Fleming et al., 2016b) indicates that the program may promote the use of positive emotion regulation strategies, which are critical to healthy social functioning and emotional wellbeing (Zeman et al., 2006), where as poor emotion regulation skills have been linked to increased depression and anxiety in adolescents (Silk et al., 2003, Suveg and Zeman, 2004).

This pilot study examines the feasibility and potential impact of SPARX-R cCBT program when delivered as a universal intervention in an AE setting in Ireland. We hypothesized that participating in SPARX-R would lead to decreases in the mean level of symptoms of depression and anxiety, increases in mental wellbeing, and improvements in coping and emotion regulation skills among the students, with increased participation being linked to better outcomes.

The specific objectives of this pilot study were:

To examine the impact of SPARX-R on symptoms of depression and anxiety among a universal AE student population.

To examine the impact of SPARX-R on psychological wellbeing, coping and emotion regulation among a universal AE student population.

To explore user satisfaction and acceptability.

To explore the relationship between program engagement and outcomes.

As the effectiveness of SPARX-R has not yet been assessed in a RCT, the study provides important preliminary information on the potential impact of the program when delivered universally in AE settings, which can be used to aid in the planning of future larger scale trials. The study builds on a qualitative analysis of the requirements of Youthreach students and staff in relation to computerized mental health promotion (Kuosmanen et al., in press), which was undertaken to inform program selection, implementation and evaluation.

Method

Study design

This study was conducted as a cluster RCT with each Youthreach Center randomized to an intervention condition (SPARX-R) or no-intervention control condition. The study design initially incorporated two intervention conditions, SPARX-R and MoodGYM (www.moodgym.anu.edu.au). However, the MoodGYM trial arm was stopped and excluded from the study because of students' inability to complete the program due to literacy difficulties.

Intervention

SPARX-R

SPARX-R is a revised version of the original SPARX CBT-based self-help intervention, designed to treat symptoms of mild to moderate depression in adolescents seeking help for depression using a serious gaming approach (Merry et al., 2012). The content is in essence the same, however, SPARX-R is framed as a preventative program; instead of focusing exclusively on depression, SPARX-R is aimed for young people who ‘feel down, stressed or angry’ (Merry et al., 2014). The skills taught in SPARX-R include psychoeducation, relaxation skills, activity scheduling, problem solving, cognitive restructuring, interpersonal skills, help seeking, and dealing with strong emotions (Merry et al., 2012). SPARX-R incorporates elements of gaming, such as completing goals and challenges, presentation of a narrative and interaction with program characters, to facilitate learning. The program has seven sequential levels, each taking approximately 20–30 min to complete.

Participants

All Youthreach Centers (n = 110) were invited to take part in the study via email and/or phone. The coordinator was asked to select one class of students (aged 15–20 years) to take part in the study. In some of the smaller Centers, all students were asked to take part in the study. As a universal exploratory study, strict inclusion/exclusion criteria were not used. All participants who provided consent were allowed to participate in the study. Informed consent was obtained from all individual participants via an online consent form. Parent/guardian assent was a requirement for students under the age of 18 years. Participants who scored above the predefined cut-off point (score of 11 or above on SMFQ and/or 10 or above on GAD-7) for moderate levels of depression or anxiety were allowed to carry on in the study but were also offered additional support according to the resources available at the Centers. Their parents were also informed. These students were not excluded from the analysis.

Ethics

The study was granted ethical approval by the National University of Ireland, Galway Research Ethics Committee (ref: 14/MAY/11).

Measures

Pre-assessment and post-assessment questionnaires were completed online through Survey Monkey.

Demographic variables

The participants were asked to state their gender, age and year of study.

Depression and anxiety

Depressive symptoms

Depressive symptoms were measured using the 13-item Short Moods and Feelings Questionnaire (SMFQ; Angold et al., 1995). The SMFQ asks how the respondent has been feeling or acting in the past two weeks (0 = not true, 1 = sometimes, 2 = true). Based on previous studies (McKenzie et al., 2011, Stallard et al., 2013), participants with an overall score ≥ 5 were categorized as being at risk and those scoring 11 or above were identified as having high levels of depression. The internal consistency of the measure in the current study was very high (Cronbach α = 0.91).

Generalized anxiety disorder

Generalized anxiety was measured using the seven-item General Anxiety Disorder Rating Scale (GAD-7; Spitzer et al., 2006). The respondents are asked to rate on a four-point scale (0 = not at all to 3 = nearly every day) how often they have been bothered by particular problems in the last two weeks. Overall scores range from 0 to 21 with higher scores indicating greater levels of anxiety symptoms. Very high levels of internal consistency (Cronbach α = 0.90) were associated with GAD-7 in the current study.

Wellbeing measures

General mental wellbeing

The Warwick-Edinburgh Mental Wellbeing Scale (WEMWBS; Tennant et al., 2007) consists of 14 positively phrased items asking the respondents to indicate how often they have felt or thought in a certain way in the last two weeks on a scale of 1 (none of the time) to 5 (all of the time). The total score ranges from 14 to 70, with higher scores indicating higher levels of emotional wellbeing. The WEMWBS exhibited very high internal consistency (Cronbach α = 0.91) in this study.

Coping styles

A 15-item short form of Amirkhan's (1995) Coping Strategy Indicator (CSI-S; Ellis, 2004) measures three types of coping: problem solving, seeking support and avoidance behaviors. The respondents are asked, on a scale of 1 (Never) to 6 (Always), how often they use specific ways to handle a stressful situation. Scores are calculated for each type of coping separately, with higher scores reflecting more frequent use of the corresponding coping strategy. The measure exhibited high internal consistency (Cronbach α = 0.84) in the current sample.

Emotion regulation

Emotion Regulation Questionnaire (ERQ; Gross and John, 2003) is a 10-item scale measuring cognitive reappraisal and expressive suppression. Cognitive reappraisal is an antecedent-focused strategy, which involves using cognitive strategies to redefine a situation so that its emotional impact is changed before the emotional response takes place. Expressive suppression is a maladaptive response-focused strategy, referring to consciously inhibiting behavioral responses to emotions. The ERQ asks the respondent to rate on a seven-point Likert-scale how much they agree with items such as ‘When I want to feel more positive emotion (such as joy or amusement), I change what I'm thinking about’. The internal consistency of the measure was high (Cronbach α = 0.85) in the current study.

Acceptability

Program acceptability was measured using a structured implementation questionnaire adapted from a range of eHealth evaluation studies and frameworks (Fleming et al., 2016b, Kawai et al., 2007, Ritterband et al., 2011). Findings in relation to four elements of acceptability are reported in this study: 1) number of levels completed (self-reported), 2) overall satisfaction score (on a scale of 1 to 10), 3) perceived helpfulness and 4) extent of practicing the skills taught in the program. An adapted version of the CBT Helpfulness Scale (Van Voorhees et al., 2009, Zabinski et al., 2001) was used to examine perceived helpfulness of the intervention. The participant is asked to rate on a scale of 1 (Very unhelpful) to 5 (Very helpful) the usefulness of the program in recognizing and changing behaviors, thoughts and emotions. Three items were added to the scale to reflect the contents of the program: ‘confront issues or problems that I struggle with’, ‘Feel better about myself’, and ‘Manage my feelings (for example anger, sadness, frustration)’. The frequency of practicing the techniques taught in the program was measured using a five-point scale (1 = never to 5 = almost every day) adapted from Kuyken et al. (2013). The scale was developed to match program content and included eight items; four items focusing on specific CBT techniques and one item each on, relaxation, activity scheduling, communication skills and problem solving.

Process

The intervention was delivered to groups of Youthreach students during scheduled class time. At each weekly class, the students completed one module of the program. The overall program completion time varied due to gaps in delivery because of mid-term breaks, other inconsistencies in the curriculum or student absenteeism. Although the program was delivered in class, each student progressed with the program at their own pace. If a student missed a class, he/she was able to complete the module another time in the Youthreach Center or just carry on where they left off at the next class. Assessment was taken at the beginning and when the students had completed the program. Students who did not complete the entire program, either because of running out of time or not wanting to continue taking part, were also asked to complete the post-assessment questionnaire. The majority, 80% of participants, completed the post-assessment questionnaire seven to eight weeks after completing the pre-assessment questionnaire. The control Centers were asked to complete the post-assessment questionnaire seven weeks after the pre-assessment.

The researcher was present at the start of the program after which a staff member moderated the program. The researcher visited some Centers several times due to technical issues. The staff members were provided with a Program Manual and Study Instructions with detailed information on the day-to-day roll out of the study and the completion of the online assessment questionnaires.

Each participant was assigned with a program login and password. The staff delivering the program was provided with a list of each student's login details in case the students would forget them. The login also served as the participant ID. Participants were asked for their login when they completed the pre-intervention and post-intervention assessments.

Sample size

Original sample size calculations (power = 0.80, α = 0.05) were conducted for a three-arm cluster RCT with two intervention conditions (SPARX-R and MoodGYM). The study was powered to detect an absolute difference of 30% (Merry et al., 2012) from the mean SMFQ value in the control arm. Based on a pilot study, we assumed the value in the control arm to be 7.80 (SD 6.36) and the difference in the means 2.34. Assuming an intraclass correlation of 0.02 (Calear et al., 2009), an average group size of nine students and an attrition rate of 30%, the calculations resulted in a desired sample size of 27 Centers (216 participants). The sample size was adjusted after excluding the MoodGYM arm to n = 144 or eight Centers with nine participants per condition.

Randomization

Randomization was conducted in clusters to avoid between condition contamination and for practical reasons. Each Center was randomly assigned a unique identification number after which they were randomized to intervention or control using the random number sequence function in Excel. The researcher was not blind to the randomization process.

Statistical analysis

Participants that provided data at post-assessment (n = 66) were included in the analysis. Missing items were not allowed in the online assessment for items of the outcome measures, however, they were allowed for items of user satisfaction. Intervention effects were analyzed using random-effects linear regression models, taking into account the correlation of multiple measurements within one participant and the clustering of the data. Univariate analyses on each outcome variable were adjusted for baseline score, gender and age. Center ID was included as a random factor in all models to reflect the clustered nature of the data. Estimates of treatment effect are presented as the adjusted difference in outcome between groups, along with 95% CI and p value.

Baseline differences between males and females were examined using independent samples t-tests. Random-effects linear regression models were fitted to compare the effect at treatment at post-intervention for each response, while adjusting for baseline, the correlation due to cluster membership and within-subject over time. A model was also fitted within the intervention condition to examine whether the participants who showed greatest use of the techniques taught in the program had better outcomes (Kuyken et al., 2013). Plots of the (standardized) residuals were used to assess underlying model assumptions. All analyses were conducted using R software (version 3.3.1).

Results

Centers were recruited to the study between June 2015 and October 2015. However, due to issues with scheduling, the last Center did not start the program until May 2016. One Center that participated in a qualitative stage of the study (Kuosmanen et al., in press) was automatically included in the SPARX-R condition and the remaining twenty Centers were randomized into control (n = 9) or SPARX-R (n = 12). An additional eleven Centers were originally randomized to deliver MoodGYM, however, this trial arm was stopped and thereafter received no intervention. In total, 146 participants from 16 Centers completed the pre-intervention assessment (SPARX-R n = 92, Control n = 54). The flow of participants is presented in Fig. 1. Attrition rates were high, with 45.2% of the participants assessed at post-intervention. Of the Centers that stayed in the study, 59% of students completed post-assessment. One Center did not complete the user satisfaction questionnaire (n = 5).

Fig. 1

Participant flowchart.

Baseline analysis

Nearly half of the participants who provided baseline data were male (Table 1). The participants were between the ages of 15 and 20 years old (mean age 17.60 years). Most of the participants were first year students (50.7%) or second year students (38.3%). There were no significant demographic differences between participants who dropped out and those who completed the post-intervention assessment.

Table 1

Baseline characteristics by group.

		SPARX-R(n = 92)	Control(n = 54)	Total(n = 146)
Female % (n)		51.1 (47)	57.4 (31)	53.4 (78)
Male % (n)		48.9 (45)	42.6 (23)	46.6 (68)
Mean (SD) age		17.76 (1.21)	17.31 (1.24)	17.60 (1.24)
Year of study	1st	52.5 (48)	48.1 (26)	50.7 (74)
	2nd	38.0 (35)	38.9 (21)	38.3 (56)
	3rd	9.8 (9)	13.0 (7)	11.0 (16)
WEMWBS		46.96 (10.14)	43.06 (11.63)	45.99 (10.38)
SMFQ		6.30 (5.66)	8.47 (6.47)	7.09 (6.04)
GAD-7		6.78 (5.66)	9.25 (6.29)	7.67 (6.01)
CSI	AvCo	15.24 (6.84)	16.42 (6.60)	15.69 (6.73)
	SuSe	10.23 (4.83)	11.50 (5.87)	10.58 (5.23)
	PrSo	13.59 (5.80)	13.12 (5.63)	13.52 (5.69)
ERQ	CR	24.24 (8.27)	23.10 (8.68)	23.92 (8.28)
	ES	15.82 (5.93)	16.50 (6.06)	16.06 (5.97)

(WEMBWBS = Warwick-Edinburgh Mental Wellbeing Scale; SMFQ = Short Moods and Feelings Questionnaire; GAD-7 = Generalized Anxiety Disorder Scale; CSI-AvCo = Coping Strategy Indicator Avoidant Coping Subscale; CSI-SuSe = Support Seeking Subscale; CSI-PrSo = Problem Solving Subscale; ERQ-CR = Emotion Regulation Questionnaire Cognitive Reappraisal; ERQ-ES = Emotional Suppression.)

Of the sample at baseline, 36% were identified as being at risk for depression (SMFQ 5–10) and 24% had high levels of depression (SMFQ ≥ 11). Furthermore, 34% scored above the cut off for generalized anxiety disorder (GAD-7 ≥ 10). There were statistically significant differences between males and females at baseline, with males scoring lower than females, on average, on symptoms of anxiety (mean difference − 2.74; p = 0.006) and depression (mean difference − 1.94; p = 0.05) and higher on general mental wellbeing (mean difference 1.70; p = 0.04). Males were also significantly less likely than females to report using support seeking coping strategies (mean difference − 2.53; p = 0.003) and avoidant coping strategies (mean difference − 2.86; p = 0.01).

Outcome analysis

Mean outcome scores at pre- and post-assessment and adjusted mean differences in outcomes are shown in Table 2. Both the intervention and control group had a higher SMFQ score at post-intervention than at baseline. The adjusted analysis at post-intervention showed a non-significant treatment effect (p = 0.34) of the magnitude of − 1.46 units (95% CI − 4.76 to 1.83). No intervention effects were detected for symptoms of anxiety (p = 0.88). No significant correlation between baseline levels and intervention effects was detected on any of the outcome measures.

Table 2

Primary and secondary outcomes by group at pre- and post-assessment and estimates of effect size.

Outcome measure		SPARX-R (n = 30)		Control (n = 36)		Adjusted differencea (95% CI, p-value) at post-assessment
		Baseline(mean (SD))	Post-assessment(mean (SD))	Baseline(mean (SD))	Post-assessment(mean (SD))
SMFQ		6.33 (5.01)	6.77 (6.01)	8.81 (6.62)	9.89 (6.33)	− 1.46 (− 4.76 to 1.83, p = 0.34)
GAD-7		7.00 (5.58)	7.20 (6.53)	9.97 (6.12)	9.31 (5.46)	− 0.18 (− 2.92 to 2.55, p = 0.88)
WEMWBS		46.17 (9.00)	45.97 (11.41)	42.75 (10.35)	42.17 (10.25)	0.78 (− 3.07 to 4.64, p = 0.66)
CSI	AvCo	15.87 (6.19)	14.30 (5.73)	17.58 (6.18)	18.44 (7.02)	− 3.42 (− 7.05 to 0.22, p = 0.06)
	SuSe	9.87 (4.88)	11.73 (5.76)	12.17 (5.90)	11.03 (5.11)	1.88 (− 2.07 to 5.83, p = 0.31)
	ProSo	13.13 (5.07)	13.70 (5.52)	13.61 (5.15)	12.31 (5.23)	0.97 (− 2.70 to 4.64, p = 0.56)
ERQ	CoRe	23.03 (8.19)	22.27 (8.19)	21.89 (7.00)	22.83 (7.02)	− 0.93 (− 4.78 to 2.92, p = 0.60)
	ExpSu	15.57 (6.05)	13.60 (5.79)	16.89 (5.42)	17.31 (4.56)	− 2.97 (− 5.48 to − 0.46, p = 0.03)

(SMFQ = Short Moods and Feelings Questionnaire; GAD-7 = Generalized Anxiety Disorder Scale; WEMBWBS = Warwick-Edinburgh Mental Wellbeing Scale; CSI-AvCo = Coping Strategy Indicator Avoidant Coping Subscale; CSI-SuSe = Support Seeking Subscale; CSI-ProSo = Problem Solving Subscale; ERQ-CoRe = Emotion Regulation Questionnaire Cognitive Reappraisal; ERQ-ExpSu = Expressive Suppression.)

Function score adjusted for baseline, gender and age.

General mental wellbeing, coping and emotional regulation

There was no significant intervention effect on general mental wellbeing as measured by the WEMWBS (adjusted difference 0.78, 95% CI − 3.07 to 4.64, p = 0.66). The adjusted analysis at post-assessment showed differences in avoidant coping (− 3.42, 95% CI − 7.05 to 0.22, p = 0.06), support seeking coping (1.88, 95% CI − 2.07 to 5.83, p = 0.31) and problem solving coping (0.97, 95% CI − 2.70 to 4.64, p = 0.56) in favor of the intervention condition, however, none of these changes reached statistical significance. The adjusted differences in expressive suppression scores between groups showed a significant difference in favor of the intervention condition (− 2.97, 95% CI − 5.48 to − 0.46, p = 0.03). A non-significant decrease in cognitive reappraisal in the intervention condition (adjusted difference − 0.93, 95% CI − 4.78 to 2.92, p = 0.60) was also detected.

Acceptability

Adherence and practice of skills

Participants completed on average 5.3 levels (76%) of SPARX-R with the majority (n = 26; 87%) completing four or more levels and 30% (n = 9) completing the entire program. Of the skills and strategies taught in SPARX-R, the most frequently practiced techniques were cognitive behavioral techniques; 65.2% reported having practiced recognizing personal negative thinking patterns, 65.2% tried stopping negative thoughts and 63.6% had tried to identify positive things about oneself and the future (Table 3). The majority of the participants had also practiced listening and negotiating skills (60.9%) or used the problem-solving strategies (56.5%) taught in SPARX-R. The least utilized techniques were activity scheduling and relaxation techniques. Of the participants, 16.7% stated that they had not practiced any of the techniques taught in SPARX-R and 25% had not practiced more than two of the techniques, and those only once or twice. Independent-samples t-tests suggested that at baseline students who had not practiced the techniques scored higher on depression (mean difference 1.1; p = 0.65) and anxiety (mean difference 2.7; p = 0.30) and had lower levels of psychological wellbeing (mean difference − 7.2; p = 0.08) than the rest of the students in the SPARX-R condition. However, these between group differences in baseline scores were statistically non-significant.

Table 3

Participant self-reported frequency of practicing techniques taught in SPARX-R at post-intervention.

	Participants endorsing each response category (%)
	n	Never	Once or twice	About once a week	Several times a week	Almost every day
Practiced relaxation techniques	24	50.0	25.0	16.7	8.3	–
Identified and made time for activities that make you feel better	23	52.2	21.7	17.4	8.3	–
Tried to stop negative thoughts	23	34.8	–	39.1	26.1	–
Practiced listening and negotiation skills	23	39.1	8.7	34.8	8.7	8.7
Used STEPS to solve problems	23	43.5	26.1	13.0	17.4	–
Identified positive things about yourself and your future	22	36.4	9.1	27.3	27.3	–
Recognized your own negative automatic thoughts	23	34.8	26.1	13.0	26.1	–
Tried to swap negative thoughts for more positive ones	22	40.9	4.5	27.3	18.2	9.1

Overall satisfaction and perceived helpfulness

SPARX-R received an overall satisfaction score of 6.0 (SD 2.76) out of 10. The program was rated most helpful in terms of confronting problems and recognizing negative thoughts followed by having more satisfying relationships and feeling better about oneself, with nearly half of the participants endorsing the helpfulness of SPARX-R on each of these items (Table 4). Approximately 40% reported SPARX-R being helpful in relation to managing and expressing emotions. A third (34.7%) of students thought SPARX-R was helpful in preventing negative thoughts affecting one's mood. The program was rated least helpful in terms of changing behavior.

Table 4

Perceived helpfulness of SPARX-R (CBT Helpfulness Scale).

	Participants endorsing each response category (%)
	Unhelpful/Very unhelpful	Unsure	Helpful/Very helpful
Confront issues or problems that I struggle with.	26.0	26.1	47.8
Recognize negative thoughts about myself.	29.2	25.0	45.9
Have more satisfying relationships (e.g. with friends, boyfriend/girlfriend or parents).	21.7	34.8	43.5
Feel better about myself.	30.4	26.1	43.4
Manage my feelings (e.g. anger, sadness, frustration).	31.8	27.3	40.9
Show my feelings and reactions to important people in my life.	26.1	34.8	39.1
Not let negative automatic thoughts (GNATS) about myself get me down.	30.4	34.8	34.7
Change my behavior in ways that have made me feel better.	20.8	50.0	29.2

Relationship between practice and outcomes

Linear regression models were used to determine whether the frequency of utilizing the techniques taught in the program was linked to better outcomes. The interaction between frequency of practice and treatment effect was non-significant for all outcome measures except for support seeking coping (r = 0.39, p = 0.034), with those reporting more frequent use of the techniques also showing greater improvement in support seeking. Furthermore, more frequent practice of the techniques was significantly correlated with increased perceived helpfulness of the program (r = 0.43, p = 0.04).

Discussion

This pilot study examined the use of SPARX-R cCBT as a universal intervention to promote mental health and wellbeing among adolescents attending AE in Ireland. Although effect sizes were non-significant for most outcome measures, the findings indicate that the program has an impact on decreasing maladaptive emotion regulation strategies. Additionally, the study highlights the benefits of using a gaming approach and not relying on text-based content, particularly when delivering computerized mental health programs for socially and educationally alienated adolescents. MoodGYM, which has been successfully delivered with adolescents attending mainstream secondary schools (Calear et al., 2009), could not be completed by adolescents of the same age group attending AE due to difficulties with literacy.

Reinforcing staff perceptions of a high prevalence of mental health difficulties among AE students (WRC Social and Economic Consultants, 2007), 60% of the participants at baseline (n = 146) were exhibiting heightened levels of depression (SMFQ ≥ 5), nearly a quarter (24%) met caseness for depression (SMFQ ≥ 11) and a third (34%) exhibited above moderate levels of anxiety (GAD-7 ≥ 10). Furthermore, the participants in this study reported lower levels of mental health and wellbeing than their peers in mainstream education as reported in large scale international studies; mean WEMWBS score of 45.99 in comparison to 48.8 in 13–16 year-old secondary school students in UK and Scotland (n = 1650; Clarke et al., 2011); mean SMFQ score 7.09 compared to 3.55–4.10 in secondary school students in the UK (n = 5030; Stallard et al., 2013); and mean GAD-7 score of 7.67 compared to 4.6 among Australian Year 9/10 students (n = 976; Wong et al., 2014). As in the above studies, female participants reported poorer mental health and wellbeing at baseline than males, which is also in line with a previous Irish national study on youth mental health and wellbeing (n = 6085; Dooley and Fitzgerald, 2012).

It is clear that there is a need for mental health promotion and prevention strategies among the AE students. However, the considerable variation in the level mental health difficulties among the sample population presents a challenge as to how to deliver programs that are relevant and helpful to all students, particularly, as AE students consider universal delivery less stigmatizing (Fleming et al., 2016b). Interventions that focus on protective factors for resilience and coping with challenges have relevance to all youth, enhance good mental health and psychological wellbeing, and also reduce risk factors for mental health problems and negative life outcomes (Clarke et al., 2015, Durlak et al., 2015). However, additional more targeted support is also needed for students exhibiting high levels of mental health problems, in which case, computerized programs could act as a bridge to seeking help from these sources.

The lack of significant findings (p < 0.05) in the current study can be attributed at least to the small sample size (n = 66) that, due to significant dropout, remained much below the original sample calculations. Comparing the adjusted differences at post-assessment, the intervention condition fared better on all but one of the outcome measures, with the reduction in expressive suppression being significant (adjusted difference − 2.97, 95% CI − 5.48 to − 0.46, p = 0.03). Although a non-significant increase in symptoms of depression at post-assessment was detected in both conditions, this increase was greater in the control condition, adjusted difference − 1.46 units (95% CI − 4.76 to 1.83, p = 0.34). When expressed relative to mean at baseline, this is a difference of approximately 18%. This indicates that SPARX-R may prevent the worsening of symptoms in the long term and that the measurement period in this study might have been too short to show significant impact. Merry et al. (2012) reported further improvement in symptoms of depression and anxiety between post-intervention to 3-month follow-up in adolescents taking part in SPARX.

It is also possible, that the changes detected on emotion regulation will have long-term positive effects on psychological wellbeing and symptoms of depression and anxiety that reach beyond the end of this trial. Expressive suppression has been linked to negative outcomes in youth, such as low social support and connectedness (Gross and John, 2003, Srivastava et al., 2009), poorer self-esteem and life-satisfaction (Gross and John, 2003), depression (Betts et al., 2009, Larsen et al., 2013) and increased suicidality in the face of adversity (Kaplow et al., 2014). Although decreases in maladaptive expressive suppression were detected, equivalent increases in the more adaptive emotion regulation strategy, cognitive reappraisal, were not detected in this study. Further research is warranted to examine program impact on other adaptive strategies, such as increased self-awareness (Chambers et al., 2015), as well as the possible links between changes in emotion regulation and mental health and wellbeing in the long term.

However, it is clear that the effects of the intervention should be further explored in a sufficiently powered study including a follow-up assessment. Sample calculations should be conducted to allow for examining the impact of age, gender and baseline scores on outcomes, to better understand which young people most benefit from the program. The small sample size in the study did not allow analysis of differences in outcomes by gender or age. Previous studies have examined the effectiveness of SPARX delivered as a targeted program for adolescents with heightened symptoms of depression. In this study, SPARX-R was delivered universally, regardless of baseline levels of depression, which may have also contributed to the lack of significant findings, even though no significant interaction between baseline scores and intervention effect was detected.

Finally, low engagement rates reported in this study may have influenced program impact. Of the students who completed post-intervention assessment, 70% did not complete the entire program and over 40% reported practicing little to none of the techniques taught in the program. A better understanding of the factors contributing to disengagement, whether related to the user or the context of delivery, is needed to improve program uptake and impact. The findings from this study also suggest that those who are not utilizing the techniques may have poorer mental health and wellbeing at baseline, are less likely to improve on support seeking coping and are less likely to perceive the program as helpful. Further research is warranted to establish the link between degree of practice and outcomes, and to explore ways to promote practice of skills, for example by supplementing the program with face-to-face sessions or by providing opportunities for peer support online (Ho et al., 2016).

Limitations

Some of the limitations in this study include the small number of participants due to significant dropout, the lack of follow-up, the participating class of students having been selected by the Center staff and the use of self-report measures, particularly as literacy issues were prevalent among the study population.

Conclusions

Findings from this pilot study indicate that the delivery of computerized CBT in AE may be feasible, however, further trials with adequate sample sizes are warranted. Considering the extent of dropout in this study and lack of full completion, implementation research is also needed to understand how to support more effective delivery of computerized programs in the AE setting.

Funding

This work was supported by the Hardiman PhD Research Scholarships, National University of Ireland, Galway.