Well-being app to support young people during the COVID-19 pandemic: randomised controlled trial.

OBJECTIVES: To evaluate the efficacy and acceptability of 'Whitu: seven ways in seven days', a well-being application (app) for young people.
DESIGN: Prospective randomised controlled trial of Whitu against waitlist control, with 45 participants in each arm. PARTICIPANTS: 90 New Zealand young people aged 16-30 recruited via a social media advertising campaign.
SETTING: Participants' homes.
INTERVENTIONS: Developed during the COVID-19 pandemic, and refined from a prototype version that was evaluated during a smaller qualitative study, 'Whitu: seven ways in seven days' is a well-being app that, as its name suggests, contains seven modules to help young people (1) recognise and rate emotions, (2) learn relaxation and mindfulness, (3) practice self-compassion and (4) gratitude, (5) connect with others, (6) care for their physical health and (7) engage in goal-setting. It can be completed within a week or as desired. MAIN OUTCOME MEASURES: Primary outcomes were changes in well-being on the WHO 5-item Well-Being Index and Short Warwick-Edinburgh Mental Well-Being Scale. Secondary outcomes were changes in depression on the Centre for Epidemiological Studies Depression Scale, anxiety on the Generalised Anxiety Disorder 7-item Scale, self-compassion on the Self Compassion Scale-Short Form, stress on the 10-item Perceived Stress Scale, sleep on the single-item Sleep Quality Scale and user engagement on the end-user version of the Mobile Application Rating Scale and via qualitative feedback during an online survey. Outcomes were evaluated at baseline, 4 weeks (primary study endpoint) and 3 months, and analysed using linear mixed models with group, time and a group-time interaction.
RESULTS: At 4 weeks, participants in the Whitu group experienced significantly higher emotional (Mean difference (md) 13.19 (3.96 to 22.42); p=0.005) and mental (md 2.44 (0.27 to 4.61); p=0.027) well-being, self-compassion (md 0.56 (0.28 to 0.83); p<0.001) and sleep (md 1.13 (0.24 to 2.02); p=0.018), and significantly lower stress (md -4.69 (-7.61 to -1.76); p=0.002) and depression (md -5.34 (-10.14 to -0.53); p=0.030), compared with the waitlist controls. Group differences remained statistically significant at 3 months for all outcomes. Symptoms of anxiety were also lower in the intervention group at 4 weeks (p=0.096), with statistically significant differences at 3 months (md -2.31 (-4.54 to -0.08); p=0.042). Usability of Whitu was high (subjective ratings of 4.45 (0.72) and 4.38 (0.79) out of 5 at 4 weeks and 3 months, respectively) and qualitative feedback indicated individual and cultural acceptability of the app.
CONCLUSIONS: Given the evolving psychological burden of the COVID-19 pandemic, Whitu could provide a clinically effective and scalable means of improving the well-being, mental health and resilience of young people. Replication of current findings with younger individuals and in other settings is planned. TRIAL REGISTRATION NUMBER: Australian New Zealand Clinical Trials Registry (ACTRN12620000516987). © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This randomised controlled trial was conducted with adequate power, a low drop-out rate and a small amount of missing data.

Key audiences of New Zealand Māori and Pacific young people were included.

Enrolment was limited to users over 16 years of age and there were fewer male participants.

Outcome measures were self-reported and there was no blinding of participants or researchers.

Introduction

The ‘invisible pandemic’ of psychological issues associated with COVID-19 is only beginning to be realised.1 2 Young people are particularly vulnerable to developing such issues due to pre-existing mental health challenges3 and lockdown-related disruption of their developmentally related needs.4 Within the past year, increased rates of mental distress,5 anxiety,6 depression7–9 and suicidal ideation10 have already been identified among young people in multiple countries. Additionally, those who have contracted COVID-19 have reported high rates of post-traumatic stress disorder.11 Long-term adverse health, academic and occupational consequences of these psychological issues are likely,3 7 12 13 especially in previously recognised subgroups with greater health needs.11 14 Despite increased demand for psychological support, access to face to face services has been significantly disrupted and delayed.15 16 Furthermore, evidence-based interventions for preventing and addressing psychological issues related to the pandemic are rare.17

Over the past decade, an increasing body of research has demonstrated the effectiveness of digital mental health interventions at improving the well-being and mental health of young people.18–20 This has led to some being recommended as first line treatments for conditions such as depression by the National Institute for Clinical Excellence in the UK.21 Given the frequency of smartphone use by young people,16 mobile health applications (apps) have particular appeal as a means of supporting young people to safely and conveniently learn and practice skills in the real world.15 16 18 19 However, out of over 20 000 available mobile health apps, very few have evidence of efficacy.22 Since the onset of the pandemic, the demand for mobile health apps has considerably increased23 and policymakers have recognised them as a widely disseminable means of improving immediate and longer-term well-being.24

Prior to the pandemic, New Zealand young people were experiencing high levels of mental distress, depression and the highest suicide rate among developed countries.25–28 Due to concerns about these issues becoming significantly worse in the context of mandated social distancing and repeated lockdowns, our research team rapidly developed an app to support the emotional well-being of this group, with special emphasis on the needs of young people of Māori and Pacific ethnicity who had always been disproportionately affected by mental health issues.15 16 ‘Whitu: seven ways in seven days’ (Whitu meaning seven in the NZ Māori language ‘Te Reo’) was based on a range of cognitive behavioural therapy (CBT), psychoeducation and positive psychology techniques previously shown to have efficacy in young people.15 16 18 The development of Whitu is discussed in more detail in our protocol paper.29 A small pilot trial (n=20) of the prototype app demonstrated statistically significant within-group improvements in well-being (p=0.021), anxiety (p=0.005), depression (p=0.031) and stress (p=0.004) between baseline and 6 weeks, but no significant changes in self-compassion, or sleep (in press, data available from the authors on request). User feedback led to improvements being made to the look and feel, cultural content and onboarding experience. This randomised controlled trial (RCT) was undertaken to evaluate the efficacy, usability and acceptability of the refined version of the app. We hypothesised that, compared with a waitlist control group, users of Whitu would experience improved well-being, self-compassion, sleep and reduced stress, anxiety and depression at 4 weeks and 3 months. Secondarily, we hypothesised that Whitu would be usable and acceptable to young people.

Methods

Study design

A mixed methods approach was used to determine the efficacy, usability and acceptability of ‘Whitu’. The study was prospectively registered with the Australian New Zealand Clinical Trials Registry.

Participants

New Zealand residents aged between 16 and 30 years who had reliable access to Wi-Fi, owned either an iPhone or Android mobile phone, were considered ‘healthy volunteers’ and not currently receiving mental health treatment, and could read and understand enough English to use the app via an online social media advertising campaign were recruited for the study. Participants were provided with a NZ $40 (Great British Pounds (GBP) 20) gift voucher on exit from the study as a thank you for their time.

Procedures

To optimise recruitment of New Zealand Māori and Pacific young people, the study was initially promoted to these groups via social media, and later opened up to individuals of any ethnicity. Participants (1) read study information, (2) completed informed consent procedures and baseline questionnaires and (3) were randomised to either the intervention group (Whitu app) or waitlist control group via REDCap, a secure web application designed to capture data for clinical research and projects that includes a randomisation module. At the point of recruitment, participants were asked not to use any well-being or mental health apps for the duration of the study. At the end of the study, they were also asked if they had done so, but none said that they had. Due to the nature of the study, neither participants nor researchers were blinded to treatment allocation. The intervention group was encouraged to download and use the app for 4 weeks. Both groups completed outcome measures via REDCap at 4 weeks and 3 months, following which control group participants were also provided with the app. No outcome measures were collected beyond this point. Further details are provided in our study protocol.29

Intervention

Whitu: seven ways in seven days is a free mobile application (app) that is currently available to New Zealand users via the App Store (https://apps.apple.com/nz/app/whitu/id1508135602?ign-mpt=uo%3D4) and Google Play Store (https://play.google.com/store/apps/details?id=com.carbonimagineering.whitu).

It contains seven positive psychology, CBT and psychoeducation-based modules that can be completed within a week. Users are encouraged to choose from a broad range of strategies and discover the ones that best work for them. Badge rewards and daily notifications encourage app completion and practice of preferred strategies. Further details of the app are provided in table 1 and figure 1. No user information or app analytic data are collected or stored over the Internet. Data entered by users are stored on their devices in an unencrypted SQLite database and can be safely removed at any time by deleting the app.

Table 1

The seven modules of Whitu

Module 1: feel	The first module acknowledges that young people may be feeling low and struggling with negative emotions due to the pandemic. The module introduces the concept of identifying and monitoring emotions, and identifying adaptive and maladaptive coping skills.
Module 2: relax	The second module addresses the uncertainty and stress that young people may be feeling due to the pandemic. Users are introduced to relaxation techniques such as deep breathing, progressive muscle relaxation and guided visualisation.
Module 3: be kind to yourself	The third module introduces the concept of self-compassion and users are guided through a short meditation and self-kindness writing exercise.
Module 4: be thankful	The fourth module introduces the concept of gratitude and how it is linked to positive well-being. Users are encouraged to create and use a diary or photographic record of things for which they are grateful.
Module 5: connect	The fifth module addresses the negative impact that lockdowns and physical distancing can have on relationships. Users are encouraged to identify important people in their lives and practice ways of staying connected with them.
Module 6: look after your body	The sixth module discusses how the pandemic makes it more difficult to stay active and look after our bodies. Users are encouraged to eat more healthily, identify and use available forms of exercise and practice good sleep hygiene.
Module 7: set goals	The final module acknowledges that the pandemic has probably interrupted routines and made it harder to set healthy goals. User are introduced SMART goals and encouraged to practice setting and achieving at least one such goal.

SMART, Specific, Measurable, Achievable, Realistic, and Timely.

Figure 1

Images of Whitu modules, including activities and badges.

Outcomes

Demographic data, including sex, age and ethnicity, were collected from all participants via REDCap at baseline. Outcome measures were assessed at baseline, 4 weeks and 3 months follow-up, with emotional and mental well-being outcomes at 4 weeks being the primary endpoints. Emotional well-being was measured using the 5-item WHO Well-Being Index (WHO-5).30 Mental well-being was measured by the 7-item Short Warwick-Edinburgh Mental Well-Being Scale.31 32 The scale has demonstrated good reliability (α=0.84) and validity in adolescent and young adult populations.33 34 Depression was measured by the 20-item Center for Epidemiological Studies Depression Scale (CES-D).35 The CES-D demonstrates high correlations with other depression measures and excellent internal consistency (α=0.85).35 Anxiety was measured by the Generalised Anxiety Disorder 7-item Scale.36 The scale has demonstrated excellent reliability (α=0.92) and validity in adults37 and adolescents.38 Self-compassion was measured by the Self-Compassion Scale-Short Form.39 The scale has demonstrated good reliability (α>0.86) in an adolescent sample.40 Stress was measured by the 10-item Perceived Stress Scale (PSS-10).41 42 The PSS-10 has demonstrated excellent psychometric properties compared with other stress measures, with good reliability and validity.43 Sleep quality was measured by the single-item Sleep Quality Scale (SQS).44 The SQS has been shown to have excellent concurrent and convergent validity with other lengthier sleep scales and has been demonstrated to be effective in determining clinically meaningful changes in sleep quality. User engagement was assessed by the app Subjective Quality subscale and the Perceived Impact subscale of the end-user version of the Mobile Application Rating Scale (uMARS) measure.45 The Subjective Quality subscale score consists of four items that determine user experience (eg, ‘Would you pay for this app?’). The Perceived Impact subscale score is derived from six items measuring the impact of using the app on knowledge, attitudes and intentions. The uMARS demonstrates good internal reliability (α=0.90), and the subscales demonstrate moderate reliability (α=0.71 and 0.80).45 In addition to the uMARS, participants also answered how many modules of the Whitu app they completed at each time point (1–7 modules) and provided brief qualitative feedback about their experience of using the app via an open-ended question in REDCap.

Data analysis

Using Gpower,46 we estimated a sample size of 90 participants (45 per treatment arm) would provide an effect size of f=0.155 for between group improvement in well-being using the WHO-5 index using a mixed analysis of variance (ANOVA) including within (three time points) and between (two groups) subject effects, with 90% power and at a two-sided significance level of 5%. This effect size relates to the between-group improvement in well-being found in a previous study of a web-based positive psychology intervention for mildly depressed adults.47 To ensure cultural acceptability of the app, we planned to recruit at least 36 (40%) young people of Māori and Pacific Island ethnicity. Baseline characteristics were summarised using means and SD or numbers and percentages. Repeated measures ANOVA was used with linear mixed models to include participants missing data at any of the three time points. The primary analysis aimed to determine whether changes in psychological outcomes were the result of the interaction between the intervention group and time, with post hoc tests to assess pairwise comparisons of groups at each time point and within-group changes over time. Cohen’s f2 was calculated as a measure of effect size for the group by time interaction.48 The primary comparisons of interest were between group differences at 4 weeks and 3 months, with results presented as marginal mean differences, 95% CIs and p values. Data were analysed using Stata software V.17, and statistical significance was set at p<0.05. Qualitative feedback was independently extracted and analysed by two authors (HT and ASS) using directed content analysis.49 Data was examined to the point of thematic saturation and any discrepancies in coding were resolved by consensus.

Patient and public involvement

Whitu was actively co-designed with New Zealand young people during the COVID-19 pandemic.29 However, no patients were involved in setting the research question or in developing plans for recruitment, design, implementation and dissemination of the results of the study.

Results

Participant characteristics

Of the 299 individuals who expressed interest, the first 90 eligible participants who met criteria were recruited to the study (45 per arm) between November 2020 and January 2021. One participant withdrew from the intervention arm without using the app due to technical difficulties or choice, four from the same arm were lost to follow-up at 4 weeks and another at 3 months. Only one participant was lost from the control arm at 4 weeks. Further details are presented in the Consolidated Standards for Reporting Trials (CONSORT) flow diagram (figure 2).

Figure 2

CONSORT flow diagram.

Participants ranged between 16 and 30 years, with a mean age of 23.8 years (SD 3.8). The majority of participants were female (n=79; 87.8%) and were students (n=59; 69.6%). Around a third reported having chronic health conditions including anorexia, anxiety, asthma, bipolar disorder, depression, eczema, epilepsy, hay-fever, hyperthyroidism, insomnia, migraines and polycystic ovarian syndrome. Participant demographics were similar between the intervention and control arm, apart from there being a greater proportion of participants reporting health conditions or medication use in the intervention arm and more participants of Pacific ethnicity in the waitlist arm. Further details are presented in table 2.

Table 2

Participant demographics

Characteristics	Whitu app (n=45)	Waitlist control (n=45)	Total (n=90)
Age (years); mean (SD)	22.71 (3.67)	24.64 (3.74)	23.68 (3.81)
Gender
Female	40 (88.9)	39 (86.7)	79 (87.8)
Male	3 (6.7)	6 (13.3)	9 (10.0)
Non-binary	2 (4.4)	0	2 (2.2)
Ethnicity*
New Zealand European	14 (31.1)	11 (24.4)	25 (27.8)
Māori	22 (48.9)	17 (37.8)	39 (43.3)
Pacific	2 (4.4)	9 (20.0)	11 (12.2)
Asian	5 (11.1)	4 (8.9)	9 (10.0)
Other ethnic groups	2 (4.4)	4 (8.9)	6 (6.7)
Occupation
Paid work	16 (35.6)	15 (33.3)	31 (34.4)
Student	29 (64.4)	30 (66.7)	59 (65.6)
Reported having a health condition	18 (40.0)	12 (26.7)	30 (33.3)
Reported taking medications	14 (31.1)	6 (13.3)	20 (22.2)
Reported previous related app use†	10 (22.2)	11 (24.4)	21 (23.3)

Data are displayed as n (%), unless otherwise stated.

*Pacific including: Samoan (n=6), Tongan (n=4), Fijian/Tuvaluan (n=1); and Asian including: Chinese (n=3), Indian (n=3), NZ Sri Lankan (n=1), Indonesian (n=1), Taiwanese (n=1).

†Apps previously used included Calm (n=7), Headspace (n=13) and Insight (n=1).

CONSORT, Consolidated Standards for Reporting Trials.

Changes in outcome measures over time

Results presented in table 3 demonstrate that the intervention had a significant effect, as observed by a significant time by group interaction, on emotional (p=0.04) and mental (p=0.008) well-being, stress (p=0.001) and self-compassion (p=0.003). Measures of well-being and self-compassion were significantly higher and stress was significantly lower in the intervention group at both the 4-week and 3-month follow-up. The interaction between group and time on depression, anxiety and sleep did not reach statistical significance. However, differences between groups indicated evidence of better outcomes for those in the intervention group, with lower levels of depression (significant at both follow-ups) and anxiety (significant at 3 months) and higher sleep scores (significant at both follow-ups) being observed, compared with the waitlist controls. All outcome measures significantly improved over time within the intervention group (p<0.05; online supplemental table 1). There were no significant differences in outcome measures over time in the waitlist control group, except for sleep scores, which were higher at both follow-ups compared with baselines, although the effects were smaller compared with the intervention group (online supplemental table 1). Further details are presented in table 3, figure 3 and online supplemental table 1.

Table 3

Comparisons between groups in outcome measures over the study period

Outcome	Whitu app (n=45), mean (SD)	Waitlist control (n=45), mean (SD)	Marginal mean differenceWhitu versus control (95% CI)	P value	Group by time interaction P value	Cohen’s f2 effect size
Emotional well-being (WHO-5)
Baseline	50.13 (20.42)	46.84 (23.78)	3.29 (−5.69 to 12.27)	0.473	0.043	f2=0.050
4 weeks	55.28 (23.03)	42.13 (21.02)	13.19 (3.96 to 22.42)	0.005*
3 months	60.51 (18.70)	47.09 (22.74)	13.77 (4.50 to 23.03)	0.004*
Mental well-being (SWEMWBS)
Baseline	22.36 (5.06)	22.24 (5.16)	0.11 (−2.00 to 2.23)	0.918	0.008	f2=0.077
4 weeks	24.69 (4.98)	22.27 (5.04)	2.44 (0.27 to 4.61)	0.027*
3 months	24.58 (4.95)	21.70 (5.47)	3.01 (0.82 to 5.20)	0.007*
Depression (CES-D)
Baseline	20.71 (12.56)	22.31 (11.51)	−1.60 (−6.30 to 3.10)	0.504	0.061	f2=0.049
4 weeks	15.72 (10.15)	21.56 (11.54)	−5.34 (−10.14 to 0.53)	0.030*
3 months	16.26 (9.42)	23.07 (12.15)	−6.62 (−11.43 to 1.82)	0.007*
Anxiety (GAD-7)
Baseline	9.38 (5.87)	9.42 (5.36)	−0.04 (−2.21 to 2.12)	0.968	0.060	f2=0.047
4 weeks	6.54 (4.76)	8.56 (5.74)	−1.89 (−4.11 to 0.33)	0.096
3 months	6.05 (4.22)	8.48 (5.15)	−2.31 (−4.54 to 0.08)	0.042*
Stress (PSS-10)
Baseline	21.84 (7.08)	21.62 (7.07)	0.22 (−2.63 to 3.07)	0.878	0.001	f2=0.108
4 weeks	16.62 (6.34)	21.42 (7.24)	−4.69 (−7.61 to 1.76)	0.002*
3 months	17.33 (6.32)	21.41 (7.29)	−3.85 (−6.77 to 0.91)	0.010*
Self-compassion (SCS-SF)
Baseline	2.77 (0.68)	2.69 (0.60)	0.08 (−0.19 to 0.35)	0.554	0.003	f2=0.094
4 weeks	3.21 (0.55)	2.68 (0.66)	0.56 (0.28 to 0.83)	<0.001*
3 months	3.11 (0.73)	2.82 (0.66)	0.31 (0.03 to 0.59)	0.028*
Sleep (SQS)
Baseline	5.20 (2.05)	4.84 (2.17)	0.36 (−0.51 to 1.23)	0.423	0.141	f2=0.084
4 weeks	6.90 (1.93)	5.82 (2.23)	1.13 (0.24 to 2.02)	0.013*
3 months	7.05 (1.85)	6.14 (2.31)	0.92 (0.03 to 1.82)	0.043*

*Statistically significant.

CES-D, Centre for Epidemiological Studies Depression Scale; GAD-7, Generalised Anxiety Disorder 7-item Scale; PSS-10, 10-item Perceived Stress Scale; SCS-SF, Self Compassion Scale-Short Form; SQS, Sleep Quality Scale; SWEMWBS, Short Warwick-Edinburgh Mental Well-Being Scale; WHO-5, WHO 5-item Well-Being Index.

Figure 3

Marginal mean outcomes by group and study time point. CES-D, Centre for Epidemiological Studies Depression Scale; GAD-7, Generalised Anxiety Disorder 7-item scale; PSS-10, 10-item Perceived Stress Scale; SCS-SF, Self Compassion Scale-Short Form; SWEMWBS, Short Warwick-Edinburgh Mental Well-Being Scale; SQS, Sleep Quality Scale; WHO-5, WHO 5-item Well-Being Index.

User feedback

Overall, feedback regarding the app was positive, with special mention made by Māori young people regarding features designed to increase cultural appeal such as the introductory ‘karanga’ (welcome song). Participants expressed diverse, and non-culturally related preferences regarding individual modules, with newly learnt content being most valued. Suggestions for improvement included the use of shorter videos, improved navigation and greater flexibility with reminders (currently set at once per day). Six users with older mobile phones experienced some technical difficulties, but were still able to use the app. Key themes and examples of participant feedback are provided in table 4. Usability scores for Whitu are also provided in table 5.

Table 4

Participant feedback

Theme	Examples
Most useful modules or features	‘I found the relax one most helpful. I just really enjoy the guided meditation aspect, the main thing that draws me to these apps. Lovely app, will definitely use again’ (Participant 346)‘I found the ‘be thankful’ module the most helpful. I liked this one as it made me stop and consciously focus on the positive aspects of my life’ (Participant 327)‘This is a well-thought out app and will go on to help many individuals like myself. I feel like I should make a special mention of the karanga at the beginning of the app when I first opened and downloaded it. As a young Māori woman, being called into the app and have it welcome all my problems and grief instantly sparked a spiritual connection for me and I instantly felt at ease and felt safe enough to embark on my healing and well-being journey. I also enjoyed the constant use of Te Reo Māori and the progress of watching my Puriri tree grow throughout the 4 weeks. It was a pleasant surprise and so culturally inclusive. The voice overs were pleasant to listen to, the videos, sounds and effects captivating. The best app after what was such a rollercoaster year! Thank you!’ (Participant 376)
Suggestions for improvement	‘Make the videos shorter somehow, I think young people nowadays have short attention spans… including me’ (Participant 308)‘I did find it was sometimes tricky to find the follow-up activities I was supposed to do—these could be better signposted/reminders could link to them directly’ (Participant 354)‘The daily reminder is good, but often came at a time when I was busy! Maybe a second reminder or setup as part of a daily routine’ (Participant 333)
Technical difficulties	‘On old phone, when completing modules there was graphical glitching (buttons and images being in the wrong place, the background video overlay being stuck in place between menus). There was also some issues with the video. Sometimes it just wouldn’t play until I restarted the app’ (Participant 335)‘Now that I check the app it has logged my progress with Module 2 but I did not find that right after I had completed it’ (Participant 337)

Table 5

Usability for n=38 participants in the intervention group using the Whitu app*

Measures	4 weeks (n=38†)	3 months (n=37†)
uMARS (score range 1–5)
Subjective app quality score	4.45 (0.72)	4.38 (0.79)
Perceived impact: awareness	3.89 (0.95)	4.00 (1.03)
Perceived impact: knowledge/understanding	3.76 (1.15)	3.86 (1.03)
Perceived impact: attitudes	3.58 (1.13)	3.46 (1.28)
Perceived impact: intention to change	3.71 (1.09)	3.57 (1.34)
Perceived impact: help seeking	3.66 (1.07)	3.57 (1.07)
Perceived impact: behaviour change	3.63 (1.10)	3.76 (1.19)

*Excluding n=2 participants who did not use the app.

†N=1 participant with no data for the 3-month follow-up.

uMARS, user version of the Mobile Application Rating Scale.

Discussion

Overall findings

To our knowledge, this is the first RCT of a well-being app for young people undertaken during the COVID-19 pandemic and it addresses the clear gap in the COVID-related literature (ie, the lack of studies to address anticipated psychological effects of the pandemic) highlighted by Gilbody et al.50 Our results indicate that Whitu is an effective, usable and acceptable composite digital health intervention with which to improve multiple aspects of young people’s health including well-being, self-compassion and sleep, and to reduce anxiety, depression and stress. Benefits were evident at 4 weeks and sustained at 3 months follow-up. The fact that well-being in the intervention group actually improved during a pandemic is also clinically significant. Based on uMARS scores (table 4), usability of Whitu was high, and greater than that of recently developed mental health apps and established norms.51 52

Comparison with previous research

Our findings are consistent with recent review evidence that mindfulness and multi-component interventions are most effective at improving the well-being of clinical and non-clinical populations.53 Despite the potential floor effect with a non-clinical population, users of Whitu reported significantly improved symptoms of anxiety and depression. Resulting effect sizes were similar to the small to moderate effect sizes of individually targeted digital interventions for treating these conditions,54 suggesting that Whitu may be beneficial for clinical populations. Since the onset of the pandemic, a rapid review of existing digital mental health interventions has ascertained they are usable, safe, acceptable and likely to be effective in ameliorating at least some of the psychological consequences of lockdown.54 However, only one other RCT of a 4-week mindfulness-based intervention delivered to Chinese university students via Zoom and asynchronous WeChat video and audio recordings has actually been undertaken and shown to improve symptoms of anxiety and depression compared with technology-based social support.55

Given reports that only 3.9% of individuals who download health apps use them for a median of 15 days more than 2 weeks56 and that only 0.5%–28.7% actually complete them,57 the relatively high efficacy and acceptability of Whitu may be related to its intentionally time-limited design. Encouraging young people to learn new self-management strategies via the app and then practice them in the real world should also help with generalisation of these skills.57 Although some may argue that an app designed to support young people during the pandemic may be of limited chronological relevance, previous evidence from earthquake survivors in New Zealand suggests that psychological effects of major events are likely to be delayed, with rates of problems increasing by between 25% and 40% even after 2 years.58 59 Given the protracted nature of the current pandemic, its true psychological cost will only be obvious in retrospect.

Strengths and limitations

Strengths of this study include the adequate power, overall low drop-out rate (less than the typical drop-out rate of 25% during studies of other mobile health interventions)60 and small amount of missing data. In addition, given our desire to develop a culturally safe and relevant app, the appeal of Whitu to Māori and Pacific young people and its efficacy with these groups is reassuring and likely to reduce existing health inequities, thereby honouring New Zealand’s commitment to the Treaty of Waitangi.61 62 Weaknesses of the study include the lack of blinding of participants, inclusion of fewer male participants and use of self-reported outcome measures. It is also possible that group differences may have been smaller if an active control had been used instead of a waitlist control. As Whitu was designed to preserve well-being in the general population (rather than treat existing mental health issues) and in order to limit confounding from concurrent psychological therapies, inclusion in the study was limited to individuals not currently receiving mental health treatment. As such, its applicability to those already experiencing mental health issues remains unproven and further research with this group would be worthwhile. Around a third of participants reported having an existing health condition and this is in keeping with previous evidence that around 18% of New Zealand high school students and up to 45% of adults live with chronic health conditions.63 64 Although it is possible that individuals with pre-existing health issues were more likely to enrol in a study involving the use a new health app, the studied population appears to be representative of young people in the community. A greater proportion of participants dropped out from the intervention group than the control group and, although characteristics of those who dropped out and those who continued within each group were similar (please see table 1 and figure 1), our primary analysis may be biased by this missing data. For example, if reasons for dropout (which were unavailable) were related to worse outcomes, this might have potentially overstated the positive effects of the intervention. Although none of these individuals who dropped out provided feedback on their experience at the end of the study, this difference may reflect challenges in using, or lack of appeal of, eHealth interventions for some young people. Our results need to be replicated in other settings (such as schools) and with young people below 16 years of age to ensure their generalisability. Evaluation of Whitu’s efficacy with higher-risk groups such as young people with long-term physical conditions16 and more objective measures of app use and clinical outcomes would be valuable. Finally, future research would benefit from formal economic analysis to bridge the gap between researcher interests and policymakers.65

Conclusions

For the moment, this study provides preliminary evidence that Whitu is a clinically effective and scalable means of improving the well-being and mental health of young people during the COVID-19 pandemic.