Cluster randomised controlled trial of an online intervention to prevent ecstasy and new psychoactive substance use among adolescents: final results and implications for implementation.

OBJECTIVES: To evaluate the effectiveness of the online Climate Schools: Ecstasy and Emerging Drugs module over 2 years, and examine the impact of intervention dose on outcomes.
DESIGN: Cluster randomised controlled trial.
SETTING: Secondary schools in Australia. PARTICIPANTS: 1126 students (aged 14.9 years) from 11 schools. INTERVENTION: Five schools were randomly allocated to the four-lesson internet-based Climate Schools: Ecstasy and Emerging Drugs module. This universal intervention uses cartoon storylines to deliver harm-minimisation information about ecstasy and new psychoactive substances (NPS). It was delivered during health education classes over 4 weeks. Six schools were randomised to the control group (health education as usual). Participants were not blinded to intervention allocation. OUTCOMES MEASURES: Students completed self-report surveys at baseline, post-test, 6, 12 and 24 months post-baseline. Intentions to use ecstasy and NPS (including synthetic cannabis and synthetic stimulants), knowledge about ecstasy and NPS and lifetime use of ecstasy and NPS were assessed. This paper reports the results at 24 months post-baseline. ANALYSIS: Mixed effects regressions were conducted to analyse intervention effects from baseline to 24 months. Post hoc analyses using Inverse Probability of Treatment Weighting compared controls with students who: i) completed all four lessons ('full dose') and ii) partially completed the intervention (≤three lessons, 'incomplete dose').
RESULTS: Primary analyses found that controls were significantly more likely to intend on using synthetic cannabis compared with intervention group students (OR=3.56, p=0.01). Results from the weighted analyses indicated that controls reported significantly lower knowledge about ecstasy (p=0.001) and NPS (p=0.04) compared with the full-dose group. No significant differences were observed between the incomplete dose and control groups.
CONCLUSIONS: The online intervention was effective in modifying students' intentions to use synthetic cannabis up to 24 months; however, this study highlights the importance of delivering prevention programmes in full to maximise student outcomes. TRIAL REGISTRATION NUMBER: ACTRN12613000708752. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

RCT Entities:   Population Interventions Outcomes

A key strength of this study is its robust and longitudinal design. A cluster randomised controlled trial was conducted in 11 Australian schools (n=1126 students) over a 2-year period.

Sophisticated statistical analyses were employed. Multilevel regressions were estimated to analyse differential group change in outcomes from baseline to the 24-month follow-up.

This study used novel online intervention completion data to examine the effect of dosage on student outcomes.

Attrition at the 24-month follow-up was higher than anticipated, despite comprehensive efforts to assess students who were absent from school on the day of a survey.

Ecstasy use among young Australians is relatively low, with 3.2% of adolescents aged 14–19 years reporting past year use in 2016.1 However, for those adolescents who do use ecstasy, the potential for harm is considerable, with adverse effects ranging from confusion and dehydration to impaired brain functioning and mental health problems.2 3 In addition to concerns about established illicit drugs such as ecstasy, there are new and emerging issues in the global landscape of drug use that also need to be considered. One such challenge is the growth in the number and availability of new psychoactive substances (NPS) and the severity of consequences associated with their use.4

NPS are substances that are specifically designed to imitate the effects of established illicit drugs and to circumvent existing drug laws. Although NPS are typically sold online or in ‘headshops’ as legal alternatives to traditional illicit drugs,5 laws about NPS are complex and frequently changing and many countries have introduced legislation to ban them. According to the European Monitoring Centre for Drugs and Drug Addiction,6 NPS continue to pose a significant public health problem, with 66 new NPS detected in 2016. Two of the most common types of NPS are synthetic cannabinoids (synthetic cannabis), which are designed to mimic the effects of cannabis and are sold under brand names such as Spice and Kronic, and synthetic cathinones (synthetic stimulants, ‘bath salts’), stimulant-like substances intended to imitate the psychoactive effects of ecstasy and amphetamine.7 8 Approximately 1% of Australians aged 14–19 years have used synthetic cannabis in their lifetime and the prevalence is <1% for stimulant-type NPS use.1 Although these rates are not high, the severity of adverse consequences associated with their use is concerning. Very little is known about the long-term effects of synthetic cannabinoid use; however, short-term effects include extreme agitation, anxiety and psychosis9 10 and serious physical side effects, including seizures, cardiac arrests and even death have been documented.11 Similarly, commonly reported side effects of synthetic stimulant use include aggression, heart palpitations and hallucinations.12–14 NPS use is also associated with other adverse consequences including alcohol, tobacco and cannabis use, truancy and risky sexual behaviour.15–17 Furthermore, misleading marketing, including use of colourful packaging and inaccurate terms such as ‘legal highs’, is likely to influence young people’s perceptions of the risks associated with NPS use, leading them to incorrectly believe that NPS are low-risk and safe to use.18 Therefore, it is clear that NPS use and related harms among youth need to be prevented. In response to this, the universal Climate Schools: Ecstasy and Emerging Drugs module was developed as the first online school-based prevention programme to simultaneously target ecstasy and NPS use among adolescents.19

The Climate Schools: Ecstasy and Emerging Drugs module is a four-lesson online universal programme delivered to year 10 students (aged 15 years) during health education classes. It is based on the effective Climate Schools prevention framework, which has previously been shown to reduce alcohol consumption, binge drinking and cannabis use, increase knowledge and modify intentions.20–24 Underpinned by social influence theory and a harm-minimisation approach, the Climate Schools: Ecstasy and Emerging Drugs module uses interactive online cartoon storylines to provide information about ecstasy and NPS use, challenge students’ perceptions of peer drug use and build resistance skills. Full details of the intervention content has been published elsewhere.19 The short-term efficacy of the module has been demonstrated previously25 via a cluster randomised controlled trial (RCT) with Australian students (n=11 schools, 1126 students, mean age=14.9 years). Results indicated that the intervention was effective in reducing intentions to use any NPS and synthetic cannabis (up to 12 months postbaseline) and increasing knowledge about ecstasy (at post-test) and NPS (up to 6 months postbaseline). In the context of universal prevention for illicit substances, an indication of the efficacy of a programme is often reliant on measures of intentions and knowledge.26 This is especially true for low-prevalence illicit substances such as ecstasy and NPS. Informed by principles of social influence theory,27 28 a key component of the Climate Schools programme is the provision of information about the short-term negative consequences of ecstasy and NPS use. The assumption is that by providing students with accurate, immediately relevant and evidence-based information, they will be equipped with the knowledge to make informed decisions when they are exposed to drug use.29 Knowledge about substance use is a precursor to attitude and beliefs, which in turn are predictive of intentions to use substances.30 Furthermore, intentions to use alcohol,31 tobacco32 and illicit substances have been shown to predict subsequent initiation of use.33 Therefore, an examination of the long-term effectiveness of the Climate Schools intervention in increasing knowledge and modifying future intentions to use ecstasy and NPS is important. Long-term outcomes of the Climate Schools: Ecstasy and Emerging Drugs module have not yet been examined, nor has the impact of programme adherence on outcomes, an area of growing importance in the literature.

The importance of high-quality implementation on programme outcomes is now well recognised in the prevention science field.34–37 In school settings, achieving high-quality implementation is often challenging given the many obstacles that can impede programme delivery such as a lack of resources in terms of teachers, time and money,38 as well as the fact that teachers often make adaptations to programme content that undermine efficacy.39 40 Although several different elements of implementation have been proposed, in the context of school-based prevention key factors to consider are adherence to the intervention by teachers and/or facilitators, quality of delivery, dosage of the intervention and student engagement.36 39 The internet-based nature of the Climate Schools programmes affords several advantages in terms of the degree of fidelity that can potentially be achieved. For example, the core content is preprogrammed online and completion is self-directed by students, thus delivery is not dependent on teacher training or skills, ensuring faithful delivery of the core programme components.41 42 The portable nature of the internet also means that students can complete missed lessons at home, or in a follow-up lesson, increasing their potential to receive the intervention in full.43 Related to this is the ability of online interventions, such as Climate Schools, to track and collect objective dosage data in real-time rather than relying on teacher self-report.44 Although previous studies have examined the impact of implementation of school-based substance use prevention programmes on student outcomes,36 45 the influence of implementation quality of online drug education prevention programmes has not yet been explored. Taking this all into account, the aims of the current paper are twofold: to evaluate the effectiveness of the Climate Schools: Ecstasy and Emerging Drugs module over a 2-year period, and examine the impact of intervention dose (number of lessons received by students) on outcomes. We hypothesised that compared with the control group, the Climate Schools intervention group would:

Report significantly greater knowledge about ecstasy and NPS at 24 months;,

Be significantly less likely to intend on using ecstasy and NPS at 24 months;

Be significantly less like to have ever used ecstasy and NPS at 24 months.

Methods

Design and procedure

A cluster RCT was conducted in 11 secondary schools in Australia between 2014 and 2016. Schools were randomly allocated to the Climate Schools intervention or to a control group (health education as usual). Cluster randomisation was employed to avoid contamination of the control group by the intervention group through student communication. Blocked randomisation was conducted by an independent researcher using the RALLOC function in Stata. Participants were not blinded to intervention allocation. Ninety independent schools (not run by the government or Catholic Church) in Sydney were contacted in 2013 and invited to participate. Schools that had previously collaborated with the researchers or that had expressed an interest in participating in future research were approached initially. School principals were sent a letter outlining the aims of the study and seeking their permission to conduct research with their students. Twelve schools agreed to take part and were randomised; however, one school withdrew from the study prior to baseline, as most students did not speak English as their first language and could not complete the intervention. Consent forms were sent home to parents of year 10 students (aged 15–16 years) at participating schools. Students with parental consent (88%) completed a confidential online self-report survey during class time at baseline, postintervention and 6, 12 and 24 months postbaseline. The final sample consisted of 1126 students (n=490 control, n=636 intervention) at baseline. Full details of the study protocol have been reported previously46 and details of participant recruitment, randomisation and retention are provided in figure 1.

Figure 1

Consolidated Standards of Reporting Trials flow diagram of participant recruitment, allocation and assessment.

Patient and public involvement

Extensive consultation was conducted with students, teachers and health professionals to inform the development of the Climate Schools: Ecstasy and Emerging Drugs module. Feedback was sought about the relevance and acceptability of the intervention, including language and aesthetics. Full details of the co-development process have been published previously.19 47 A summary of findings from the present study have been disseminated to teachers and students at participating schools, and results have been presented at scientific meetings and published in peer-reviewed journals.25

Sample size calculations

To account for cluster randomisation, sample size calculations were based on sample size requirements developed by Heo and Leon48 to detect intervention by time interactions in longitudinal cluster RCTs. We estimated that 10 schools (5 in each group) were needed to detect a standardised between-group mean difference of 0.20 (equivalent to an OR of 1.44), with 80% power (p=0.05) in primary and secondary outcomes at the end of the trial (see protocol for full details).49

Intervention and control groups

The Climate Schools: Ecstasy and Emerging Drugs module

Schools allocated to the intervention group were asked to implement the Climate Schools: Ecstasy and Emerging Drugs module during health education classes in 2014. The programme consists of four 40 min lessons to be delivered 1 week apart, over a 4-week period. Teachers were instructed to complete all four lessons of the programme, and as the intervention is preprogrammed online, lessons were designed to be delivered in order (ie, starting with lesson 1, then lessons 2, 3 and 4) and in full. The intervention aims to impart evidence-based information about ecstasy and NPS, correct overestimates of peer ecstasy and NPS use, teach refusal skills and address misconceptions. Full details of the lesson content have been described in detail elsewhere.19 25 46 Each lesson consists of a 20 min online cartoon component completed individually by students in a classroom setting, under teacher supervision. The remaining 20 min consist of optional activities, including online worksheets and teacher-led discussions and role plays. Teachers were provided with four preplanned activities per lesson and were able to choose which activities to implement to best meet the needs of their class. The core component of the intervention is the online cartoon lessons, therefore it was not essential that teachers completed the same activities or same number of activities. Participants accessed the programme via the study website using secure login details. The researchers met with teachers at the start of the study to assist in the navigation of the website, reiterate the study aims and to distribute hard-copy teacher manuals. The teacher manuals contained implementation guidelines, links to the syllabus, lesson summaries (for teachers and students) and materials for implementation of the optional activities (eg, suggested discussion points, worksheets and homework tasks). All the activities required minimal preparation by teachers and no ongoing training was required. This information was also available to teachers via the study website.

The control group

Control schools implemented health education classes as usual (which covers drug education) to their year 10 students in 2014. Teachers at control schools were invited to provide information about the amount and format of any drug education they delivered to their students (see online supplementary file 1).

Measures

Demographic data collected at the individual level included gender, age, country of birth and self-reported grades, which were scored categorically using the following response options: ‘90%–100%', ‘80%–89%', ‘70%–79%', ‘60%–69%', ‘59% or below’. The primary outcomes of interest are self-reported intentions to use ecstasy and any NPS, and secondary outcomes are intentions to use synthetic cannabis and synthetic stimulants, knowledge about ecstasy and NPS and lifetime use of ecstasy and NPS (including synthetic cannabis and synthetic stimulants). These findings have been reported previously at 12 months postbaseline,25 with the current paper focusing on 24-month outcomes.

Intentions to use ecstasy and any NPS, synthetic cannabis and synthetic stimulants

Four items, adapted from a previous prevention trial,50 were used to assess intentions. Students rated how likely they were to use ecstasy/any NPS/synthetic cannabis/synthetic stimulants in the next 6 months and responses were coded on a dichotomous scale ‘very unlikely, unlikely, unsure’=0 or ‘likely, very likely’=1.

Knowledge about ecstasy and NPS

Ecstasy knowledge was measured using a 15-item scale developed to reflect the intended content of the Climate Schools intervention. Items assessed knowledge about the prevalence of ecstasy use, risks and harm-minimisation information. For each statement, students were required to answer ‘true’, ‘false’ or ‘do not know’. Items were summed to provide a total continuous score, with higher scores indicating greater knowledge. A 5-item scale was used to assess NPS knowledge, including knowledge about associated harms and prevalence of use. Total continuous scores ranged from 0 to 5, with higher scores indicating greater NPS knowledge. The ecstasy knowledge scale demonstrated acceptable reliability (α=0.72), and the NPS scale had questionable reliability (α=0.67).

Ecstasy and NPS use

Lifetime ecstasy use was measured using a single dichotomous item ("Have you ever used ecstasy?" yes/no), adapted from the Australian National Drug Strategy Household Survey.51 Three equivalent items were used to assess use of ‘any NPS’, ‘synthetic cannabis (marijuana), for example, Spice, Kronic, K2’ and ‘synthetic stimulants' (‘bath salts’).

Intervention implementation

In the present study, implementation fidelity was primarily assessed by the dose of the intervention received. Teachers at intervention schools were asked to complete a logbook indicating which lessons (eg, "Did your students complete the online Climate Schools cartoon component for lesson 1?”) and activities ("Which activities did your students complete?’) they completed with their class. Teachers were also asked to rate the level of engagement among students for each lesson (‘very disengaged’=0 to ‘very engaged’=5) and to complete a programme evaluation (fidelity and evaluation summary available online). The use of objective evidence to assess dosage, in addition to teacher self-report, is advocated in the literature.44 Thus, lesson completion data for the online cartoon component were collected at the student and school level, and extracted via the study website. For each lesson, data were coded on a binary scale (0=‘did not complete online lesson’, 1=‘lesson completed in full’). To ensure the reliability of the online dosage data, teachers at participating schools were contacted via email and asked to retrospectively describe details of the intervention implementation. One teacher indicated that students at their school had completed the intervention as a group, rather than individually, due to limited computer resources, thus all students at that school were coded as completing all four lessons. A single binary measure was generated to represent overall dose, with completion of either one, two or three lessons representing an ‘incomplete dose’ and completion of all four lessons, a ‘full dose’.

Statistical analyses

The present data were analysed in two phases. First, multilevel regressions were conducted to estimate the overall effectiveness of the Climate Schools intervention for primary and secondary outcomes (intentions, knowledge and lifetime use). Second, in light of the low adherence to the intervention (59% received a full dose), post hoc inverse probability of treatment weighting analyses were conducted to examine the effect of dose on outcomes. Full details of the analyses are described below.

Primary analyses

To analyse differential group change in outcomes at the 24-month follow-up, multilevel regressions including fixed and random effects were estimated in Stata V.14. Specification of a random intercept in these models take into account variation at the individual level, and random effects at the school level were estimated to account for clustering or potential lack of independence in data from students within the same school.52 Preliminary models were estimated to determine the most appropriate covariance structure for the linear regressions (knowledge), and the optimal number of integration points for each logistic regression (intentions and lifetime use).52 An unstructured covariance structure was selected and 15 integration points were used. A three-level regression model was estimated for each outcome, with repeated observations (level 1) nested within students (level 2), and students clustered within schools (level 3). As per our previous analyses of the 12-month data,25 for each outcome, gender and baseline academic grades were included in the model as covariates. Rates of synthetic stimulant use were too low for multilevel analyses to be conducted for this variable. Time at each assessment occasion was modelled using a categorical variable (coded 0–5, ie, baseline, post-test, 6, 12 and 24 months), and the intervention effect was assessed with a trial group×time interaction. These models use baseline measurements as the reference point to estimate participant-specific starting points and change over time, and allows us to examine change over time from baseline to 24 months. This approach is ideal for analysis of longitudinal and clustered data.53 Multilevel mixed effects models are recognised as an emerging, rigorous method of analysing trial data and has several advantages over conventional approaches, such as better treatment of missing data, accommodation of variance at the individual level, and flexible modelling of time effects and the within-subject covariance structure.52–54

Missing data

Missing data were handled using maximum likelihood (ML) estimation, based on the intention-to-treat principle, which includes all participants in the sample (n=1126). This approach is widely accepted and has been employed in previous studies applying multilevel models to substance use.55 ML is a highly efficient way of using all available information to estimate parameters rather than deleting cases with missing data.52 56 It is a superior method for handling missing data as all available data are used to compensate for missing data for a particular student.57 ML is valid under the assumption that data are missing completely at random or missing at random (MAR; meaning missingness does not depend on unobserved response, given the observed covariates and outcomes in the model).54 In the present study, the assumption of MAR is plausible given the variables in the analysis models, including outcome measurements at other time-points, which are used to estimate unobserved outcomes.

Examining the impact of intervention dose on outcomes

In view of the observed intervention adherence (59% of students received a full dose), post hoc inverse probability of treatment weighting analyses were conducted to examine the effect of dose on intervention outcomes. Inverse probability weightings are based on propensity scores, defined as the probability of treatment assignment (or in the present study, of full exposure to the Climate Schools intervention), given participants’ observed baseline characteristics.58 Intervention exposure, or dose, is likely to be associated with participant characteristics (eg, truant or high-risk students may be more likely to receive a lower intervention dose), and thus propensity scores provide a way of balancing out these differences in baseline characteristics. In order to attain unbiased estimates of intervention effects at incomplete and full dose, analyses are adjusted using inverse probability weights to balance the distribution on measured baseline covariates between adherent and non-adherent participants.59 That is, comparisons are made between the incomplete dose group and the control group, and the full dose group and the control group, with the three weighted groups similar in all respects at baseline except for intervention allocation. In the present analysis, the propensity score was estimated by regressing intervention dose (0=control group, 1=incomplete dose, 2=full dose) on a set of 27 baseline covariates (categorical and continuous) using multinominal regression. In line with previous research,60 covariates included in the model were: i) variables found to predict lesson completion at the bivariate level, ii) risk factors associated with substance use in the literature (eg, mental health outcomes), iii) baseline measures of all outcome variables and iv) socioeconomic variables (eg, gender). To increase the explained variation and improve the propensity score model, two-way interaction and quadratic terms were also included where these improved prediction.61 All included variables were measured at baseline and therefore were not influenced by exposure to the intervention (a full list of covariates are summarised in online supplementary file 2). The final multinominal logit model used to derive propensity scores was successful at modelling full dose of the intervention (likelihood ratio χ2=672.9 (df=58; p<0.001); R2=29.0%). Balance checks were conducted on baseline covariates comparing weighted means and proportions across the three weighted groups (control, incomplete, full dose) and results indicated that baseline equivalence was achieved on all covariates.

Inverse probability of treatment weights

Weights were generated and summary statistics were examined to determine whether stabilising weights were required. The use of stabilising weights is recommended when extreme weights are present due to inaccurate or unstable weights for students with a very low probability of receiving the intervention.59 62 Results indicated that there were extreme weights present (M=3.05, SD=6.99; range=1.00–158.53), thus stabilised weights were generated. As the stabilised weights also varied in range (M=1.00, SD=1.76, range=0.23–36.46), a series of sensitivity analyses using truncated weights were conducted to examine the influence of the extreme weights. To create truncated weights, the cut point of the 1st and 99th percentiles was calculated and any values outside the cut points were replaced with those values. Mixed effects regressions using the truncated weights were conducted and compared with models using the stabilised weights. No material differences were observed between the two models for any outcome, thus results from analyses using the full range of weights are reported. Consistent with the primary analyses, mixed effects regressions were estimated to examine change over time in outcomes across the three weighted groups. These models were based on a sample of 1088 participants, as 38 cases were excluded due to missing covariate data, and included a variable representing dose, time×group interaction and the stabilised weight.

Results

Sample characteristics

A total of 1126 students completed the baseline survey (Mage=14.92 years, SD=0.43; 57% male). Almost all students (96%) provided follow-up data on at least one occasion, and 79% provided data at two or more follow-ups. The descriptive statistics for outcomes over time are reported in table 1 and participant retention is presented in figure 1. At baseline, there was a significantly greater proportion of females in the control group (78%) compared with the intervention group (16%; χ2 (1, n=1126)=438.29, p<0.001). Students in the intervention group also reported significantly greater ecstasy knowledge (F(1, 1124)=76.44, p<0.001) and NPS (F(1, 1123)=56.79, p<0.001) compared with controls.

Table 1

Descriptive statistics for outcome variables over time and by group

Outcome	Group	Baseline n=1126	Post-test n=729	6 months, n=874	12 months, n=717	24 months, n=684
Ecstasy intentions, % (n)	CLIMATE	2.20 (14)	3.79 (10)	3.30 (15)	1.55 (6)	5.63 (22)
	CONTROL	0.82 (4)	2.80 (12)	3.51 (14)	2.15 (7)	3.48 (10)
NPS intentions, % (n)	CLIMATE	1.73 (11)	3.79 (10)	1.98 (9)	0.52 (2)	3.33 (13)
	CONTROL	0.8 (4)	3.27 (14)	3.01 (12)	1.84 (6)	2.79 (8)
Synthetic cannabis intentions, % (n)	CLIMATE	6.92 (44)	7.20 (19)	6.39 (29)	6.74 (26)	7.42 (29)
	CONTROL	4.49 (22)	6.78 (29)	8.27 (33)	7.06 (23)	8.71 (25)
Synthetic stimulants intentions, % (n)	CLIMATE	2.20 (14)	2.65 (7)	1.54 (7)	0.52 (2)	3.10 (12)
	CONTROL	0.82 (4)	2.34 (10)	2.26 (9)	1.23 (4)	1.05 (3)
Ecstasy knowledge (M, SD)	CLIMATE	10.59 (2.53)	11.57 (3.61)	11.65 (2.77)	11.90 (2.36)	11.20 (3.48)
	CONTROL	9.18 (2.87)	9.57 (3.31)	10.11 (3.25)	10.40 (3.13)	9.99 (3.57)
NPS knowledge (M, SD)	CLIMATE	3.33 (1.32)	3.85 (1.49)	3.93 (1.24)	3.99 (1.13)	3.71 (1.44)
	CONTROL	2.71 (1.48)	2.78 (1.48)	3.12 (1.51)	3.28 (1.45)	3.07 (1.50)
Ever used ecstasy, % (n)	CLIMATE	2.50 (16)	4.10 (11)	3.07 (14)	2.14 (7)	7.36 (29)
	CONTROL	1.23 (6)	1.63 (7)	1.25 (5)	2.07 (8)	2.79 (8)
Ever used NPS, % (n)	CLIMATE	2.98 (19)	3.02 (8)	3.08 (14)	1.30 (5)	4.58 (18)
	CONTROL	2.86 (14)	2.80 (12)	3.52 (14)	1.23 (4)	2.44 (7)
Ever used synthetic cannabis, % (n)	CLIMATE	2.35 (15)	2.63 (7)	1.57 (7)	1.30 (5)	4.31 (17)
	CONTROL	2.45 (12)	2.33 (10)	2.05 (8)	0.93 (3)	1.38 (4)
Ever used synthetic stimulants, % (n)	CLIMATE	0.63 (4)	1.51 (4)	0.88 (4)	0.00 (0)	3.00 (12)
	CONTROL	0.0 (0)	0.70 (3)	0.50 (2)	0.00 (0)	1.02 (3)

Total knowledge scores ranged from 0 to 5, with higher scores indicating greater knowledge.

NPS, new psychoactive substances.

Intervention effects

Table 2 presents the results from the mixed effects regressions of the 24-month follow-up data. Results indicate that a significantly greater proportion of students in the control group reported being likely to use synthetic cannabis in the future, compared with the intervention group (OR=3.56, 95% CI 1.32 to 9.54, p=0.01). There were no significant group differences for any other outcome variable; however, students in the control group were trending towards being less likely to use synthetic cannabis compared with the intervention group (p=0.05), when dose was not considered.

Table 2

Results from multilevel regression models at the 24-month follow-up

Outcome	OR	95% CI	P values
Ecstasy intentions	2.00	0.42 to 9.17	0.39
NPS intentions	2.26	0.48 to 10.75	0.31
Synthetic cannabis intentions	3.56	1.33 to 9.54	0.01
Synthetic stimulant intentions	1.78	0.50 to 6.34	0.37
Ever used ecstasy	1.08	0.19 to 6.15	0.93
Ever used NPS	0.57	0.14 to 2.30	0.43
Ever used synthetic cannabis	0.17	0.03 to 0.97	0.05

Reference group=Climate Schools intervention.

NPS, new psychoactive substances.

Impact of intervention dose on outcomes

Analysis of online dosage data indicated that 59% of students (n=378) completed the Climate Schools module in full. Analyses examining the baseline equivalence among the unweighted groups indicated that the full-dose group had significantly greater knowledge about ecstasy (p<0.005) and NPS (p<0.005) compared with controls at baseline. Although the incomplete dose group had significantly greater knowledge of harms associated with ecstasy (p<0.001) and NPS (p<0.001) than controls at baseline, they were significantly more likely to have used ecstasy (p=0.008), any NPS (p=0.03) and synthetic stimulants (p=0.04) and to intend on using ecstasy (p=0.003), synthetic cannabis (p=0.03) and synthetic stimulants (p=0.04) in the future (the complete baseline and 24-month data for the unweighted groups are available in online supplementary file 3). Mixed effects regressions using the weighted data revealed that at 24 months, control students reported significantly less knowledge about both ecstasy (p=0.001) and NPS compared with students who received a full dose of the intervention (p=0.04). There were no significant differences between the incomplete dose and controls on any outcome (table 3).

Table 3

Results from multilevel regression models using weighted data (n=1088)

Outcome	OR	95% CI	P values
Climate full dose vs control
Ecstasy intentions	1.44	0.17 to 12.19	0.74
NPS intentions	0.56	0.05 to 5.81	0.63
Synthetic cannabis intentions	1.01	0.29 to 3.50	0.98
Synthetic stimulant intentions	0.90	0.13 to 6.44	0.91
Ever used ecstasy	1.28	0.15 to 10.90	0.82
Ever used NPS	0.33	0.04 to 3.02	0.33
Ever used synthetic cannabis	0.98	0.15 to 6.66	0.99

Reference group=control group.

NPS,  new psychoactive substances.

Discussion

This study evaluated the effectiveness of the first online school-based prevention programme for ecstasy and NPS over a 2-year period. Results indicated that at the 24-month follow-up, students in the intervention group were significantly less likely to intend on using synthetic cannabis in the next 6 months, compared with controls. This finding indicates that there was a sustained effect over time for synthetic cannabis intentions, with analysis of the short-term data previously demonstrating a significant group difference up to 12 months. However, beneficial short-term effects previously observed in relation to NPS intentions and increased knowledge about ecstasy and NPS were not sustained at 24 months, and there was no effect in terms of reducing lifetime use of ecstasy or NPS. The ability of the intervention to modify students’ future intentions to use synthetic cannabis up to 24 months after the intervention is notable, as future intentions to use illicit substances, such as cannabis, have been shown to predict initiation of use.33 This finding is especially important in light of the severe side effects associated with synthetic cannabis use, including renal and pulmonary injuries and cardiac arrests,11 63 as well as the fact that synthetic cannabis use has been associated with poor mental health outcomes, risky sexual behaviour, binge drinking, illicit drug use and injury or violence.15–17 Future research should specifically examine the relationship between NPS attitudes, intentions and use.

This study makes an important contribution to the literature by using novel online lesson completion data to examine the effect of dosage on student outcomes. Overall, we found that lesson completion was poor, with just over half of students (59%) completing the Climate Schools: Ecstasy and Emerging Drugs module in full. Results from the weighted analyses indicated that at the 24-month follow-up, control students reported significantly lower knowledge about ecstasy and NPS, compared with students who completed all four lessons of the programme. Students who received an incomplete dose of the intervention did not report significantly greater knowledge than controls. This informs interpretation of the primary analyses, in which effects for ecstasy and NPS knowledge were only evident at post-test and 12 months, respectively, and not sustained at 24 months. The present findings emphasise the importance of delivering the Climate Schools: Ecstasy and Emerging Drugs module in full to maximise student outcomes.

Despite the potential of internet-based interventions to improve implementation fidelity, some students did not receive a full dose of the intervention in the present study. Although we cannot be certain of the reasons for poor adherence, it is likely attributable to the timing of intervention delivery and competing demands faced by teachers. Unlike previous Climate Schools modules which are typically delivered in years 8 or 9, this module was delivered in year 10 (aged 15 years) to align with the later age of initiation for ecstasy and NPS use. The low adherence to the intervention and modest follow-up rates achieved in this study highlight the difficulties of conducting research among year 10 students. This is consistent with previous research that has shown that school-based drug education often exists in competition with academic agendas,64 including exams, excursions and a lack of time in the curriculum. Indeed, teachers in this study indicated that it was hard to implement the surveys and/or intervention due to examinations, excursions and a lack of lesson time. Implications for programme developers include considering ways of improving student engagement with the intervention to increase lesson completion and thus improve programme outcomes,45 65 such as the use of smartphone applications and novel technology, email/SMS reminders to encourage students to complete missed lessons and automatic notifications for teachers about student lesson completion. In terms of real-world implementation, teachers should consider their existing timetables and lesson plans to determine when the full four-lesson Climate Schools programme can be successfully implemented among students.

Examination of the baseline descriptive data indicated several differences between students who did, versus did not, complete the programme (see online supplementary file 3). Students who did not complete the programme in full (incomplete dose) tended to be ‘riskier’ at baseline than students in the control group, reporting greater use of ecstasy and NPS and greater intentions to use these substances. This pattern of results suggests that students in the incomplete dose group were the students at greatest risk of substance use and harms at the beginning of the study, which could impact both their propensity to complete the intervention, as well as their substance use behaviour at 24 months. This suggests that this subgroup of students may require more intensive or targeted intervention, which addresses underlying risk factors such as personality traits or emotional/behavioural difficulties,66 rather than a universal approach. Targeted, or ‘selective’ prevention approaches are aimed at individuals who are at high risk of developing substance use problems or who are showing very early signs or symptoms.67 There were no significant differences between control students and those who partially completed the intervention, suggesting that the intervention needs to be implemented in full to achieve beneficial student outcomes. This may be particularly important when implementing prevention programmes for low prevalence, but high-risk substances such as ecstasy and NPS, where students need to receive comprehensive harm-minimisation information and skills to fully understand the harms and learn the prevention messages.

In addition, the intervention effect found for NPS intentions at the 12-month follow-up was not sustained at 24 months. This suggests that additional booster lessons may be needed in the year following the intervention to reinforce key messages among students who received the intervention in full, and to provide an opportunity for students who only received an incomplete dose to acquire new knowledge and skills. It is also possible that greater effects would be found if schools implemented the Climate Schools: Ecstasy and Emerging Drugs module in combination with the other existing modules in the Climate School suite, rather than as a stand-alone intervention. If feasible for schools, students could receive the effective 12-lesson Climate Schools: Alcohol and Cannabis module in years 8 or 9 (aged 13–14 years), and the four-lesson Ecstasy and Emerging Drugs module in years 9 or 10 (aged 15–16 years), providing them with comprehensive harm-minimisation knowledge about licit and illicit drugs, as well as reinforcing refusal and resistance skills.

Although the prevalence of ecstasy use appeared to increase from baseline (2.5%) to 24 months (7.3%) among the intervention group, importantly, there were no significant group differences. These rates are consistent with patterns of ecstasy use observed among Australian students of comparable ages in the most recent Australian Secondary Students’ Alcohol and Drug (ASSAD) survey, a large survey of just under 25 000 adolescents. The ASSAD reported rates of ecstasy use to be approximately 3% among those aged 15 years, and increasing to 7% among those aged 17 years.68 It should also be noted that although there were no significant differences between the intervention and control groups in terms of use of any NPS or synthetic cannabis, the intervention group appeared to be trending towards being more likely to have ever used synthetic cannabis group than controls. An inspection of the unweighted data by dose suggests that it is the incomplete dose group driving this pattern of results.

The present findings should be interpreted in light of several limitations. First, our measure of implementation fidelity focused on only one dimension, intervention dose,36 and we did not assess teacher-related factors, such as quality of delivery and adherence to programme content. However, since the Climate Schools programme is completed individually by students via the internet, these teacher-related factors are potentially less important for this study. Additionally, the propensity model to predict likelihood of a full dose was constructed primarily through student-level variables, such as mental health outcomes and truancy rates, and did not include school-specific or teacher-specific factors. It should be noted that it is possible that other factors may have influenced the likelihood that a student would complete the Climate Schools programme in full, such as teachers' attitudes and willingness to prioritise intervention delivery among timetabling issues and competing demands in the school environment.37 Nonetheless, the use of online dosage data extracted from the study website, rather than relying solely on teacher self-report, is novel and a strength of this study. Attrition was higher than anticipated, despite comprehensive efforts to follow-up students who were absent from school on the day of a survey. Importantly, missing data were handled in our analyses using ML estimation, a widely accepted approach and highly efficient way of using all available information to estimate parameters rather than deleting cases with missing data.52 56 Finally, the low numbers of students in our sample who had used any NPS, synthetic cannabis and synthetic stimulants meant that cell sizes were small and our results should be interpreted with some caution. The low cell counts also meant that many outcomes were dichotomised and were analysed via multilevel logistic regressions, which deviates from the planned Poisson analyses in our protocol.49

Conclusions

This study evaluated the effectiveness of the Climate Schools: Ecstasy and Emerging Drugs module at the final 24-month follow-up assessment, and examined the impact of intervention dose on outcomes. The present findings demonstrated a sustained impact of the programme in terms of modifying students’ intentions to use synthetic cannabis up to 24 months; however, no other intervention effects were observed. An examination of outcomes by dose revealed that the intervention was effective in increasing knowledge about both ecstasy and NPS up to 24 months for students who received the programme in full, compared with the control group. These findings highlight the importance of ensuring that online interventions, such as the Climate Schools: Ecstasy and Emerging Drugs module, are delivered in full to maximise student outcomes.