Pilot evaluation of the Sleep Ninja: a smartphone application for adolescent insomnia symptoms.

OBJECTIVES: The aim of this study was to test the feasibility, acceptability and preliminary effects of a recently developed smartphone application, Sleep Ninja, for adolescent sleep difficulties.
SETTING: The study was conducted online with Australian individuals recruited through the community. PARTICIPANTS: Participants were 50 young people aged 12-16 years with sleep difficulties.
DESIGN: A single-arm pre-post design was used to evaluate feasibility, acceptability and sleep and mental health variables at baseline and postintervention. INTERVENTION: Cognitive-behavioural therapy for insomnia informed the development of the Sleep Ninja. The core strategies covered by the app are psychoeducation, stimulus control, sleep hygiene and sleep-related cognitive therapy. It includes six training sessions (lessons), a sleep tracking function, recommended bedtimes based on sleep guidelines, reminders to start a wind-down routine each night, a series of sleep tips and general information about sleep. Users progress through each training session and conclude the 6-week programme with a black belt in sleep. OUTCOME MEASURES: Feasibility was evaluated based on consent rates, adherence and attrition, acceptability was assessed using questionnaires and a poststudy interview, and sleep, depression and anxiety variables were assessed at baseline and postintervention.
RESULTS: Data indicated that the Sleep Ninja is a feasible intervention and is acceptable to young people. Findings showed that there were significant improvements on sleep variables including insomnia (within-group effect size d=-0.90), sleep quality (d=-0.46), depression (d=-0.36) and anxiety (d=-0.41).
CONCLUSIONS: The Sleep Ninja is a promising intervention that could assist adolescents who experience sleep difficulties. A follow-up randomised controlled trial is now warranted. TRIAL REGISTRATION NUMBER: ACTRN12617000141347. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

RCT Entities:   Population Interventions Outcomes

This is the first study to evaluate app-delivered cognitive–behavioural therapy for insomnia in adolescents with sleep difficulties.

The intervention being tested, Sleep Ninja, was developed with input from young people, is fully automated and does not require internet coverage to function.

The evaluation included measures of feasibility and acceptability as well as detailed semistructured interviews about participants’ experience with the app.

As a preliminary study, this study did not include a control group.

Introduction

Clinical insomnia is a sleep disturbance characterised by difficulty falling asleep, staying asleep or waking up too early, with associated daytime impairment.1 It effects approximately 4% of adolescents,2 however, subthreshold symptoms are common, with approximately 25% of young people reporting some degree of sleep disturbance.2 3

Depression and insomnia are closely linked, with comorbidity levels as high as 73% in young people.4 Insomnia is not only a symptom of depression, but is a common precursor, with high-quality longitudinal data having established insomnia as an independent risk factor for depression onset.5–7 For example, a recent meta-analysis found that insomnia was associated with a greater than twofold increase in depression risk.5 Although depression has multiple causes and maintaining factors that go beyond the presence of sleep problems, the literature suggests that sleep plays an important role, and targeting sleep in the context of depression may have wide-reaching benefits.6

There is emerging evidence that addressing insomnia in individuals with concurrent insomnia and depression improves both sleep and depression outcomes.8–10 This suggests that there may be value in targeting sleep to improve insomnia symptoms, with potential downstream effects on depression. To our knowledge, there have been three studies testing the hypothesis that targeting insomnia can prevent depressive symptoms. In an adult study, insomnia treatment led to a reduction in depression following the intervention and at 6 and 18 month follow-up, relative to an active control group.11 12 In a youth study, a face-to-face insomnia intervention was delivered to secondary school students and results showed improvements on sleep and anxiety outcomes, but not symptoms of depression.13 Data from the 2-year follow-up from this study have not yet been published.14 In a second youth study, a sleep intervention delivered either in group format or digitally led to decreased depressive symptoms at both 2 and 12 months follow-up, an effect that was mediated by improvements in insomnia.15

The gold-standard treatment for insomnia is cognitive–behavioural therapy for insomnia (CBT-I; Australasian Sleep Association, American College of Physicians), and there is accumulating evidence to support the use of digitally delivered CBT-I in both adults and young people.16–19 Delivering sleep interventions via digital formats may be particularly well suited to young people, with adolescents showing a strong preference for digital delivery (97%) when given the choice between face-to-face and digital CBT-I.17 This preference may in part be explained by the fact that young people are reluctant to seek help for psychological issues, for reasons that include stigma and a preference to manage the problem themselves.20 Sleep is typically less stigmatised than disorders like depression, suggesting that targetting sleep it may be more appealing to adolescents. Currently, there are no digital CBT-I programmes that are commercially available for youth.21 The overall objective of this study was to evaluate a newly developed digital CBT-I programme for adolescents with sleep difficulties.

The aim of this pilot study was to examine the acceptability, feasibility and preliminary effects of an intervention (Sleep Ninja) delivered to adolescents via smartphones. In line with the guidelines on the development of behavioural interventions,22 23 the primary purpose of this study was to investigate recruitment rates, uptake, intervention completion, reasons for non-adherence and participant retention. The secondary aim was to use both quantitative and qualitative methods to determine the acceptability of the app among young people with sleep difficulties and allow for the refinement of the intervention prior to a formal randomised evaluation. A final aim was to examine the impact of the Sleep Ninja app on sleep outcomes and mental health symptoms. We used a single-arm, pre–post design to address these aims. It was hypothesised that the app would be a feasible modality in which to deliver the automated sleep intervention, as measured by uptake, completion and retention rates, that the app would be acceptable to young people, and that its use would be associated with improvement in sleep and mental health symptoms.

Method

Participants

Fifty participants were recruited via media and social media channels, including the Black Dog Institute’s website and paid Facebook advertisements that targeted potential participants and their parents between April and June 2017. A sample size of 50 was selected in order to successfully meet the study’s feasibility and acceptability aims. Inclusion criteria were: aged 12–16 years, presence of at least mild insomnia, operationalised by endorsement of at least one of the following symptoms over the preceding 2-week period: difficulty falling asleep, difficulty staying asleep or waking up too early. These items are the first three questions on the Insomnia Severity Index (ISI),24 and were chosen to include a participant group with at least mild levels of insomnia. For study inclusion, participants also needed to own a smartphone running iOS or Android, have a valid email address, access to the internet and be able to provide personal and parental consent.

Measures

Insomnia Severity Index

The ISI is a psychometrically sound, seven-item self-report measure of insomnia symptoms over the previous 2 weeks.24 25 Responses are reported on a Likert scale from 0 to 4, producing total scores of 0–28.24 25 Cut-off scores are as follows: 0–7 reflects no clinically significant insomnia, 8–14 indicates subthreshold insomnia, 15–21 suggests moderate severity insomnia and 22–28 indicates severe insomnia.24 The ISI was designed for use in adults but has been widely administered to, and validated in, adolescent samples.25–27 In one adolescent validation study, reliability was strong (Cronbach’s α=0.83), and test–retest reliability was acceptable, r=0.79.25

Pittsburgh Sleep Quality Index

The Pittsburgh Sleep Quality Index (PSQI) is a widely used self-report 19-item scale that assesses usual sleep habits and experiences over the preceding month and has been validated in adolescent samples, with strong internal consistency (α=0.72) and test–retest reliability over a 6-week period (r=0.81).28 There are seven subscales which are sleep quality, sleep latency, sleep duration, habitual sleep efficiency (SE), sleep disturbances, use of sleeping medications, and daytime dysfunction.29 Each component is scored from 0 (no difficulty) to 3 (severe difficulty), which are summed to obtain a Global PSQI score ranging from 0 to 21.30

Patient Health Questionnaire-Adolescent Version

The Patient Health Questionnaire-Adolescent Version (PHQ-A) assesses depressive symptoms in the preceding 2 weeks in adolescents, and has been adapted from the widely used PHQ-9 designed for adults.31 This measure has excellent psychometric properties, including internal consistency of α=0.89 and test–retest reliability of r=0.84.32 In this study, we used the 8-item version in which the questions are identical to those asked in the PHQ-9 with the exclusion of the last item which asks about suicide. It is comparable to the PHQ-9 in terms of diagnosing depressive disorders.32 33 Each item is scored on a 4-point scale and summed together to form a total depression score ranging from 0 to 24. Scores correspond to the following cut-offs: 0–9 indicates minimal symptoms, 10–14 indicates mild symptoms, 15–19 reflects moderate symptoms and 20–24 is indicative of severe depression.31 The PHQ-A has demonstrated good sensitivity (73%) and high specificity (94%) for major depressive disorder.31

Generalised Anxiety Disorder 7

The Generalised Anxiety Disorder 7-item scale (GAD-7) evaluates symptoms of GAD.34 All items are scored on a scale from 0 (not at all) to 3 (nearly every day). The scores on each item are summed together to derive a total score, ranging from 0 to 21 of which 0–4 indicates minimal anxiety, 5–9 mild anxiety, 10–14 moderate anxiety and 15–21 severe anxiety.34 The GAD-7 has good sensitivity (89%) and specificity (82%) for GAD scores >10.34 The measure has also been validated in adolescent populations with Cronbach’s α=0.90 and high convergent and discriminant validity.35

Expectations of Success

A four-item scale was developed for this study to assess participants’ motivation and expectations for improving their sleep with an app (eg, ’I am confident that people could learn skills for improving sleep from an app'). The four items assess perceived confidence, importance, usefulness and readiness to change. The Expectation of Success measure was scored on a five-point scale and total scores were computed by summing each item, ranging from 0 to 16. Higher scores on this scale indicate greater confidence and readiness to target sleep using a smartphone app.

Acceptability of the Intervention

The Acceptability of the Intervention scale is a seven-item measure that was developed by the research team to assess participants’ attitudes and behaviours associated with using the app (eg, ’How much did you learn from the app?' and ’Would you recommend this program to others?'). This measure was informed by similar acceptability measures commonly used in the field.36 Each item was designed to assess a different domain. The first four domains assessed were app completion, ease of use, amount learnt and usefulness, with each being scored on an ordinal scale from 0 to 3. The final three items assessed behaviour change, whether the participant would use an app like this in the future, and whether they would recommend it to a friend, and were scored dichotomously as either yes or no, with an option for participants to describe the nature of their behaviour change if yes. As each question assessed a different domain, item scores were considered separately.

Reasons for Non-Adherence

The Reasons for Non-Adherence measure is a 23-item scale that was adapted from a previous measure37 to assess the degree to which different reasons impacted on participants’ use of the app. There are four domains assessed: phone/internet/technical issues (eg, ’My phone wasn’t working or was having problems'); personal issues (eg, ’I didn’t think I deserved help'); intervention-general issues (eg, ’I wasn’t convinced the app would be helpful'); and intervention-specific issues (eg, ’There was too much text to read'). Participants responded on an ordinal scale indicating whether each item played ’no', ’a little' or ’major part' in why they stopped or had difficulty using the app as intended. The scores on each item were considered separately.

Sleep Diary

The 10-item Sleep Diary was developed by the research team incorporating the questions from the Consensus Sleep Diary,38 with the addition of two questions regarding daytime naps and use of sleep medication. Participants answered 10 questions which included bedtime, time taken to fall asleep (sleep-onset latency; SOL), number and duration of night-time awakenings (number of awakenings; NWAK), duration of wakefulness after sleep onset (wake after sleep onset; WASO), time of final awakening, time participants got out of bed for the day, subjective sleep quality, how refreshed participants felt on awakening, duration of any daytime naps and use of sleep medication. Sleep diaries were completed electronically with preset categories from which users selected responses from a drop-down menu. A clock scroller was used to enter the time and/or duration of all sleep-related activities and all times were entered in 12 hour format to minimise errors associated with 24 hour time. Restrictions were set to ensure participants could not enter a wake time earlier than bedtime and visa versa. All questions required answers for the sleep diary to be submitted. From the sleep diary, we calculated time between waking in the morning and getting out of bed, time in bed (TIB), total sleep time (TST; calculated by subtracting SOL, WASO and time between waking and getting up in the morning, from TIB) and SE, (calculated by taking the percentage of TST/TIB).

Poststudy interview

After study completion, participants were invited to attend a face-to-face or telephone interview to provide feedback on their experience. Interviews were semistructured and explored participants’ opinions about the study in general, and specifically in relation to the intervention. Questions were open ended, and flexible enough to explore ideas that were raised during each interview. Interviews were audio recorded and then transcribed verbatim by the interviewer. The interview content was pragmatically coded into relevant themes by the same researcher, with oversight and guidance provided by the research team.

Intervention: ‘Sleep Ninja’

The Sleep Ninja app was derived from CBT-I and developed by our team, as a fully automated smartphone app. A participatory design process was used whereby young people contributed to the content, functionality and accessibility/user experience of the app through a series of focus groups.39 The core strategies included in the app were: psychoeducation, stimulus control, sleep hygiene and sleep-focused cognitive therapy. Sleep restriction, which aims to increase SE by reducing the amount of time spent in bed, was deliberately omitted because some support (parental and/or professional) is likely to be required to successfully implement sleep restriction, particularly in young people. Although sleep restriction did not comprise part of the app, there was instead a focus on the importance of regular sleep-wake cycles. The app teaches users about the importance of consistent sleep and wake times, and recommended bedtimes are calculated based on the time they need to wake up (according to sleep guidelines). This strategy draws from transdiagnostic approaches to target sleep difficulties that go beyond insomnia (eg, delayed sleep phase and irregular sleep presentations40) and may, therefore, be useful to adolescents experiencing a broad range of sleep difficulties.

The structure of the Sleep Ninja app includes six training sessions (lessons), a sleep tracking function, recommended bedtimes based on sleep guidelines, reminders to start a wind-down routine each night, a series of sleep tips and general information about sleep. The home screen has three options: train, track, and more (see figure 1). Users complete training sessions which are delivered through a chat-bot format where the sleep ninja essentially acts as a sleep coach. Training sessions take approximately 5–10 min to complete, and cover: (1) psychoeducation, information about circadian rhythms and the importance of keeping regular sleep schedules; (2) stimulus control, the value of only going to sleep when tired, and strategies that can be used at night when having trouble sleeping; (3) basic sleep hygiene such as avoiding caffeine and stimulating activity in the evenings, suggestions for daytime activities to promote night-time sleep (eg, exercise, no napping); (4) identifying and planning for high-risk situations, how to get back on track after a late-night or sleep in; (5) cognitive therapy including how to deal with unhelpful thoughts that can prevent falling asleep as well as sleep-related cognitive distortions, and (6) a final review session which summarises all of the material contained in the app. The user interacts with the app through a forced choice chat-bot format which is responsive to the input of the user, meaning it personalises information and recommendations based on the selections and sleep profile recorded by the participant. Users level up and reach their next ‘belt’ by completing one training session and tracking their sleep for three nights (out of a 7-night period). As there were six training sessions to complete, the app was made available for 6 weeks (42 days) before it locked. Users finish the programme with a black belt in sleep.

Figure 1

Example screens from the Sleep Ninja app. From left: homescreen, training session access and progress record, tracking and bedtime setting, more information.

It is notable that evidence suggests that the use of screens at bedtime interferes with sleep.41 Drawing on data from large epidemiological studies suggesting that refraining from screen use for the hour prior to bedtime alleviates potential interference from screens,42 this app has been designed to be used during the day. Users receive a prompt 1 hour before bed (calculated according to sleep guidelines and their wake-up time) to commence their prebed time routine and are encouraged to stop using electronic devices after this time. In fact, part of the cognitive component of the app is to educate and challenge beliefs about the importance of night time phone use in order to promote healthy sleep habits. We expect these factors to mitigate the risk that smartphone use in this context will contribute to poor sleep.

Procedure

Participants were encouraged to download consent forms if they met the eligibility criteria listed on the study website and submit this directly to the research team, once completed. Those who provided written informed consent and that of a parent or guardian were then enrolled in the trial and invited to complete the screening questions to verify study eligibility, before completing baseline questionnaires which included: demographics, ISI, PSQI, PHQ-A, GAD-7 and Expectation of Success. Participants could then access the first day of the online 7-day sleep diary. Another diary entry became available each day for the following 6 days and participants were reminded to complete entries via text message. At the completion of seven consecutive entries in the diary, participants were given access to the Sleep Ninja app on their personal smartphone devices. Participants could use the app for 6 weeks before the poststudy questionnaire was made available, which included the same battery as baseline with the omission of the Expectations of Success questionnaire and with the addition of the Acceptability and Reasons for Non-Adherence questionnaires. Participants then completed another 7-day sleep diary, which was delivered in the same format and schedule as baseline. After the study had finished, participants were invited to participate in a face-to-face or telephone interview to provide feedback on their experience of participating in the study. Participants were reimbursed for their time with gift cards to the value of AUD$10 each for completing baseline and poststudy assessment schedules; AUD$20 for a telephone interview and AUD$30 for a face-to-face interview.

Participant involvement and consultation

Prior to this study, a separate group of young people were consulted in a series of focus groups to inform the design, features and structure of the app (for more information please see39). As a feasibility and acceptability study, participants were asked to report on their experiences with respect to both the app and the study procedures, via questionnaires and an in-depth semistructured interview. Given that a key objective of this study was to assess the acceptability of the Sleep Ninja app, participants’ perspectives were of critical importance. A one-page lay summary of the study results has been sent to all participants.

Statistical analyses

Statistical significance was set at α=0.05. Summary scores for sleep diary variables at baseline and poststudy were obtained by averaging sleep diary entries at baseline, and averaging sleep diary entries at poststudy, respectively. All questionnaire and sleep diary variables were initially screened for excessive skew (>3) or kurtosis (>843). Six sleep diary variables did not pass screening and were further scrutinised (baseline: WASO entry; poststudy: time fallen asleep, TIB after final morning wake up, SOL, TST, SE variables). Examination of these six variables revealed each included an entry that was of an extreme value (z-scores ranged from |4.11| to |6.05|) and a decision was made to remove these six values (hence, n=47 for baseline WASO; ns=28 for postintervention time fallen asleep, TIB after final morning wake up, SOL, TST, and SE). Subsequently, all variables had satisfactory skew and kurtosis.

Questionnaire and sleep diary variables were examined using multilevel modelling. This modelling approach handles missing data by incorporating all available data from each subject into the analysis. Given the aims of our study, our interest centred on the main effect of time (ie, change from baseline to post). Random effects were modelled for intercept and time. Models were respecified with a random effect for intercept only in cases where there was no variation in individual baseline to poststudy changes. Within-group effect sizes were computed as the modelled mean difference between baseline and poststudy divided by the sample SD at baseline.

Results

Baseline characteristics

See table 1 for characteristics of the study sample. Participants had a mean age of 13.71 (SD=1.35), were spread across school grade and nearly all were born in Australia and living in the city. Most participants reported difficulty falling asleep, with about half also reporting problems staying asleep or waking up too early, and about a quarter of the sample were receiving treatment for sleep or mental health problem.

Table 1

Demographic variables

Characteristics	Sample (n=50)
Age in years, mean (SD, range)	13.71 (1.35, 12–16)
Age in years, n (%)
12	10 (20.4)
13	15 (30.6)
14	7 (14.3)
15	12 (24.5)
16	5 (10.2)
Female, n (%)	33 (66)
Born in Australia, n (%)	47 (94)
Live in the city, n (%)	44 (88)
Sleep problems, n (%)
Difficulty falling asleep	47 (94)
Difficulty staying asleep	28 (56)
Problems waking up too early	28 (56)
Receiving treatment for sleep or mental health problem, n (%)	13 (26)

One participant did not indicate their age, so n=49 for age.

Recruitment rate

There were more than 300 enquires made to the research team about participation in this trial. Of these, 60 individuals indicated eligibility and returned consent forms. Ten of these participants were not enrolled in the trial; four did not meet inclusion criteria and six withdrew prior to the trial. Reasons for withdrawing were: a change of mind (n=1), a lack of time (n=2), considering participation a chore (n=1), confidentiality concerns (n=1). One participant did not provide a reason. Therefore, 89% (n=50) of the 56 young people who provided consent and met screening criteria continued to trial. See figure 2 Consolidated Standards of Reporting Trials diagram for details.

Figure 2

Participant Flow.

Expectation of success

Overall, participants were optimistic about using the app, with a mean score of 12.90 (SD=2.09) out of a possible 16 points. Every single participant agreed that in principle, people could learn skills for improving sleep from an app and indicated that they felt that study participation was important. All participants reported that improving their sleep habits were important, with 49% indicating it was ‘very important’. Finally, the sample demonstrated their readiness for change with 100% of the sample indicating that they were either moderately ready (16%), ready (43%) or completely ready (41%) to improve their sleep patterns using an app.

Retention

Of the 50 participants in the study sample, 47 (94%) completed the baseline questionnaire and sleep diaries and were invited to download the Sleep Ninja. At poststudy, 34 participants completed the poststudy battery (72% retention). Participants who had available data at both time points did not differ significantly from those who only had baseline data on any of the questionnaires or sleep diary measures (all Fs <2.58, ps >0.115).

Uptake and adherence

Forty-five participants (96%) who completed the baseline assessment downloaded the Sleep Ninja. Programme usage data indicated that of these, 82% completed the first lesson, 51% completed four of the six lessons and 33% completed all six. Participants were accurate in their reporting of app use, with approximately 80% of participants indicating that they completed ‘most’ or ‘almost all’ of the app.

Acceptability

Survey responses on the Acceptability of the Intervention questionnaire indicated that young people reported that the app was ‘easy’ or ‘very easy’ to use (97%). The majority of participants (59%) indicated that they learnt ‘a fair bit’ from the app, and 28% reported that they learnt ‘a great deal’, while 12% did not learn very much or almost nothing. Participants found the app to be either ‘useful’ or ‘very useful’ (78%), while the remainder (22%) did not find it useful. Most respondents (72%) reported changing their behaviour after using the app, and examples of behavioural change included changes to their prebedtime routine (22%), keeping more consistent sleep-wake cycles (65%), getting up earlier in the morning (22%) and restricting the use of their bed for sleep (30%). More than half of the participants reported that they would use this kind of app in the future (56%), and encouragingly, 91% would recommend the Sleep Ninja app to a friend. The degree to which participants found the app useful was positively correlated with module completion, (r=0.35, p=0.047), as was the amount participants reported learning from the app (r=0.49, p=0.004).

The interview mirrored the findings of the questionnaire in terms of acceptability and usefulness. However, there were some aspects of the app that users felt could be improved. Specifically, interviewees expressed a desire for improved explanation of the different app sections and what they needed to do each time they opened the app. Participants commonly reported wanting to be able to personalise their user experience more, including skipping information they knew, seeking more information around difficult or unfamiliar topics, accessing information in different formats (eg, video/audio), being able to speed up or slow down the Sleep Ninja’s text display, and being able to update their wind-down activity choices and the time the wind-down reminder appeared. Participants expressed a range of views about the tone of the Sleep Ninja, with nearly half of the interviewed participants commenting favourably on the Sleep Ninja’s jokes, with several participants commenting that the Sleep Ninja’s language was annoying and too childish. There was consensus that the Sleep Ninja’s language was repetitive and could be improved by cutting out superfluous dialogue that was not delivering core intervention strategies. Nearly half of the interviewed participants expressed some difficulty in implementing at least one of the Sleep Ninja’s recommended strategies due to conflicting parental bedtime rules. For instance, the strategies to delay bedtime until sleepy and leave the bed/bedroom if unable to get to sleep after more than 30 min most commonly encountered parental resistance or required modification. Numerous interviewees commented on the usefulness of receiving feedback and summaries of their logged sleep, however, several commented that this could be improved by displaying the information in graphs and over time, so that change and improvement is clearer. Overall, participants reported that the number of notifications in the app was acceptable, and that additional reminders should be sent to notify them of available lessons and after periods of inactivity and that these reminders should contain motivational and encouraging messages. While most interviewees considered three nights of sleep tracking per belt acceptable, there were others who felt fewer nights of tracking would have been better, and others who expressed willingness to track more than three nights before levelling up.

Reasons for Non-Adherence

Results from this questionnaire indicated that young people were very happy to use an app to receive help for their sleep issues (84%), did not have technical issues with its use (75%) and felt they had the technical skills to use the app (90%). Participants all reported that the material was relevant and conceptually easy to understand. The main reason participants reported not using the app was that they felt it took too long to work through (53%), there was too much text to read (47%) and that it was too repetitive (59%).

Preliminary effects

Sleep and Mental Health Questionnaire Outcomes: Table 2 shows the results for the questionnaire measures. As predicted, from baseline to poststudy, there was a significant decrease in insomnia severity measured by the ISI, β=−4.29, p<0.001, d=−0.90, and sleep quality on the PSQI, β=−1.88, p<0.001; d=−0.46. Similarly, mental health measures showed a decrease in both depression on the PHQ-A, β=−2.60, p<0.001, d=−0.36 and anxiety on the GAD-7, β=−2.56, p<0.001, d=−0.41.

Table 2

Questionnaire measures

		Preintervention		Postintervention	Modelled change from pre to postintervention
Outcome	n	M (SD)	n	M (SD)	β (95% CI)	d
ISI	50	14.12 (4.75)	34	9.62 (5.23)	−4.29 (−5.63 to 2.95)*	−0.90
PSQI	50	10.43 (4.12)	33	8.03 (4.08)	−1.88 (−2.85 to 0.90)*	−0.46
PHQ-A	49	13.04 (7.24)	32	9.88 (7.53)	−2.60 (−3.99 to 1.22)*	−0.36
GAD-7	49	9.92 (6.19)	32	7.09 (6.13)	−2.56 (−3.59 to 1.52)*	−0.41

Raw means (SDs) are presented. Cohen’s d values are time effects for preintervention to postintervention using the modelled mean difference divided by the sample preintervention.

GAD-7, Generalised Anxiety Disorder 7 Scale; ISI, Insomnia Severity Index; PHQ-A, Patient Health Questionnaire (Adolescent Version); PSQI, Pittsburg Sleep Quality Index.

*P<0.001.

Sleep diary outcomes: Results for the sleep diary entries are shown in table 3. At baseline, participants went to sleep, on average at 23:29 hours, reported taking an average of 1 hour and 12 min to fall asleep, spent an average of 9 hours and 39 min in bed, woke up an average of 1.47 times, slept for a total of 7 hours and 40 min, woke up at 7:27 hours, and spent approximately 29 min awake in bed before getting up. Overall SE was just above 80%. Results from the analysis at postintervention indicated that as predicted, participants went to bed 35 min earlier then at baseline (β=−0.58, p=0.003), there was a significant decrease of 21 min in how long participants took to fall asleep (SOL; β=−0.37, p=0.032), participants spent significantly less TIB after waking than they did at baseline (β=−0.27, p<0.001), and woke significantly less frequently during the night reducing to an average of 0.87 times (NWAK; β=−0.46, p=0.011). There were also improvements in TST of 33 min (TST; β=0.53, p=0.005), SE (β=5.25, p=0.016), how refreshing sleep was (β=0.43, p<0.001) and sleep quality (β=0.31, p=0.018). There were no significant differences in the time participants woke up in the morning (on average, at 7:20 hours), TIB, WASO or medication use (all ps>0.05). Within-group Cohen’s d effect sizes ranged from small to medium (0.31–0.68).

Table 3

Sleep diary measures

	Baseline		Postintervention		Modelled change from baseline to postintervention
Outcome	n	M (SD)	n	M (SD)	β (95% CI)	d
Time woken up	48	07:27 (0:46)	29	07:20 (0:54)	−0.10 (−0.41 to −0.21)	−0.13
Time in bed after final morning wake up (hour:min)	48	0:29 (0:23)	28	0:13 (0:10)	−0.27 (−0.41 to–0.12)***	−0.70
Time fallen asleep	48	23:29 (1:19)	28	22:54 (1:03)	−0.58 (−0.95 to –0.21)**	−0.44
TIB (hour:min)	48	9:39 (1:11)	29	9:38 (0:55)	−0.01 (−0.42 to 0.41)	−0.01
SOL (hour:min)	48	1:12 (1:01)	28	0:51 (0:47)	−0.37 (−0.70 to –0.03)*	−0.36
WASO (hour:min)	47	0:14 (0:14)	29	0:12 (0:14)	−0.04 (−0.12 to 0.05)	−0.17
TST (hour:min)	48	7:40 (1:09)	28	8:13 (1:06)	0.53 (0.17 to 0.90)**	0.46
NWAK	48	1.47 (1.50)	29	0.87 (1.35)	−0.46 (−0.81 to –0.11)*	−0.31
SE (%)	48	80.12 (12.01)	28	85.64 (11.30)	5.25 (1.03 to 9.47)*	0.44
Sleep refreshingness	48	2.37 (0.63)	29	2.78 (0.78)	0.43 (0.19 to 0.68)***	0.68
Sleep quality	48	2.84 (0.74)	29	3.10 (0.82)	0.31 (0.06 to 0.56)*	0.42
Use of medication	48	0.12 (0.30)	29	0.11 (0.30)	−0.01 (−0.02 to 0.01)	−0.03

Raw means (SDs) are presented. Cohen’s d values are time effects for preintervention to postintervention using the modelled mean difference divided by the sample preintervention SD. Time fallen asleep and time woken up are expressed as times in 24 hours time. Time variables (TIB, SOL, WASO, TST) are expressed in hours:min. Refreshingness of sleep is rated on a Likert scale from 1=exhausted to 5=very refreshed; Quality of sleep is rated on a Likert scale from 1=very poor to 5=very good. Use of medication is expressed as a proportion of days medication was used to help with sleep. SE is expressed as a percentage.

NWAK, number of awakenings; SE, sleep efficiency (expressed as a percentage); SOL, sleep onset latency; TIB, time in bed; TST, total sleep time; WASO, wake after sleep onset.

*P<0.05, **P<0.01, ***P<0.001. Bolded effect sizes reflect statistically significant change from pre to post.

Discussion

The purpose of this pilot study was to evaluate the feasibility, acceptability and preliminary effects of the Sleep Ninja app on sleep and mental health symptoms, for use among adolescents with sleep difficulties. A secondary objective of the study was to gather information in order to refine aspects of the app before evaluating it in a larger trial. Our findings confirmed that young people with sleep difficulties were optimistic about using the app and could complete baseline questionnaires and sleep diaries using an automated digital format without assistance. Feasibility was confirmed based on uptake, completion and retention rates with young people volunteering for the study, downloading the app and completing most of the lessons. This provides evidence that the Sleep Ninja app is a feasible intervention to deliver to young people experiencing sleep difficulties.

Intervention adherence levels suggested that while more than half of the participants completed more than half of the app, only one-third of participants completed all six lessons. While this is within the range of adherence rates reported in the literature for technology-mediated insomnia programmes,44 reasons for non-adherence require consideration as there is room for improvement. The main reason participants had difficulty or stopped using the app was reportedly due to the amount of text presented in the app and the repetitive nature of the material. Participants also requested a more tailored experience. Those who completed more of the app also reported it to be more acceptable. Therefore, it is likely that refining the app by taking these points into account is likely to increase engagement with the content and overall adherence to the intervention.

Efficacy outcomes showed that insomnia symptoms improved significantly from baseline to poststudy, effectively moving participants from the lower cut-off for clinical insomnia, firmly into the subthreshold symptom level. There was an improvement in self-reported sleep quality, with a medium effect size (d=−0.46) suggesting app use improves quality of sleep. The improvements detected on the two sleep questionnaire measures (ISI, PQSI) were corroborated by those found in the sleep diaries, with participants going to bed earlier, falling asleep more quickly at night, waking less frequency, sleeping for longer, spending less TIB in the morning after waking and reporting improved SE and quality.

The time that participants reported waking each morning did not shift. This finding needs to be considered in the context of young people having to get up at a specific time for school each morning. School start times are a modifiable contributing factor to insufficient sleep, with evidence suggesting that delaying school start times by even 30 min is associated with higher levels of school attendance, lower daytime sleepiness and improved attention and concentration.45 While there have been calls to delay school start times to improve sleep duration, health and functioning (eg, see ref 46), this is unlikely to change in the Australian context. Accordingly, a focus on bed times rather than rise times is likely to be most useful for treating sleep difficulties in this age group, at least for the time being.

Findings on these sleep outcomes are consistent with the two studies which have evaluated web-delivered CBT-I for adolescents with insomnia,15–17 and show for the first time that CBT-I, when delivered to adolescents by smartphone app, confer benefits. It is encouraging that the within-group effect size obtained in the current study for insomnia (d=−0.90) is comparable to within-group effect size found for digitally delivered CBT-I in a randomised trial (d=−0.9216). In this randomised study, the intervention group was found to be superior to the waitlist control, suggesting that a within-group effect size of this magnitude is likely to reflect improvement over and above what would be expected based on a standard effect of time.16

Beyond the sleep outcomes, we also found that there were decreases in depression and anxiety symptoms following the completion of the intervention suggesting that there may be value in using this app to address mental illness. The magnitude of the decrease in depression scores is notable, with individuals moving from the moderate range into the mild range, and a within-group effect size that is comparable to other adolescent depression prevention trials.47 These findings provide proof-of-principle evidence that the Sleep Ninja app may be useful in addressing depression. As a pilot study, we did not test the specific hypothesis that targeting insomnia will decrease depression risk. Given our encouraging findings, a follow-up randomised controlled trial which follows participants over time, which can determine causality between insomnia and depression, that can assess the mechanisms of change, as well as the impact of intervention on depression risk is now warranted.

This study makes a unique contribution to the literature by showing that smartphone delivery of CBT-I is a promising format in which to deliver this gold-standard intervention. This is the first study that we are aware of that has evaluated app-delivered CBT-I in young people,21 and only the second study that has tested mobile phone delivery of CBT-I, the first study being conducted in adults, with positive effects on sleep outcomes.48 Using smartphones to deliver interventions such as this offer a myriad of advantages, including immediate connectivity to automated interactive applications that can be accessed anytime, anywhere. Sleep Ninja has been developed so that it does not rely on internet access for use, which is likely to be important to young people who may have limited data plans, and individuals who do not have optimal internet coverage. Not requiring internet coverage to access digital programmes represents a new wave of flexibility in the delivery of health interventions and the automated nature of the intervention means it can be delivered without professional support. It is notable that we had a 72% retention rate, which is at the upper level of that detected for digital interventions which has been shown in a meta-analysis to range between 43% and 99%,49 with retention rates typically lower in non-supported interventions.50

There are several study limitations that need to be considered. First, participants in this trial were required to have relatively minor levels of insomnia symptoms for study entry. This decision was made by balancing inclusiveness against the requirement of sleep disturbance to ensure participants were motivated to use the app. Moreover, given the study focus was on feasibility and acceptability, we felt it would be prudent to establish these factors before targeting a more severe participant group. That said, while we set the threshold for entry relatively low, both the mean and the median converged on an ISI score of just above 14, indicating that participants were at the junction between having subthreshold and clinical levels of insomnia symptoms (cut-off score is 15). Therefore, it is relatively unlikely that there would have been a floor effect as the data showed there was sufficient room to detect symptom improvement. The high mean symptom level also suggests that the results may generalise to a group with clinical levels of insomnia. Second, we did not include a control group. Again, as the study goal was to establish feasibility and acceptability, it was not necessary to include a control group for this purpose. However, this design is not able to attribute causality to the intervention and a controlled study is now needed. Finally, this study relied exclusively on subjectively reported sleep outcomes. Although objective measures such as polysomnography provide the most accurate way to assess sleep, there is evidence that subjectively measured sleep variables could be more closely associated with functional outcomes.51 Moreover, subjectively experienced sleep quality and parameters have consistently shown to be strongly associated with psychological well-being,52 53 suggesting that perception of sleep is as important, if not more so, than objective measures. We are currently investigating how inbuilt smartphone sensors such as the accelerometer might be used to provide a more objective estimate of sleep.

This study provides preliminary evidence supporting the feasibility, acceptability and effects of a fully automated app that targets adolescent sleep difficulties. The Sleep Ninja intervention shows promise both as a sleep-focused intervention, but also potentially to reduce risk for depression.