Pilot study of a well-being app to support New Zealand young people during the COVID-19 pandemic.

BACKGROUND: Well-being apps represent a promising and scalable approach for improving mental health outcomes in youth, especially during a global pandemic when access to face-to-face interventions may be limited. Whitu (meaning 7 in the New Zealand Māori language Te Reo) is a newly developed well-being app with 7 modules that support young people to learn and practice evidence-based coping skills, including relaxation, mindfulness, self-compassion, and goal-setting.
OBJECTIVE: During this pilot, we explored the acceptability, usability, and preliminary efficacy of Whitu before refining the app for a randomized controlled trial (RCT).
METHODS: We recruited 20 New Zealand young people aged 16-25 years via social media to trial the first prototype of the Whitu app over 6 weeks. Within-group differences from baseline to 2- and 6-weeks post intervention in self-reported well-being, depression, anxiety, stress, self-compassion, optimism and sleep quality were evaluated using repeated-measures ANOVA. A further 21 participants aged 16-30 years were recruited to participate in 4 focus groups to give feedback on the app's usability and cultural acceptability. Feedback was analysed using directed content analysis.
RESULTS: Statistically significant improvements in anxiety (p = 0.024) and stress (p = 0.017) were observed from baseline to 2-weeks post intervention. Improvements in well-being (p = 0.021), depression (p = 0.031), anxiety (p = 0.005), and stress (p = 0.004) were also observed from baseline to 6-weeks. No statistically significant changes were seen in self-compassion, optimism, or sleep quality. Effect sizes (partial η2s) ranged from 0.25 (depression) to 0.42 (stress). Qualitative feedback comprised of five key themes, namely: factors affecting engagement, issues with functionality, preferences regarding aesthetics, effectiveness and adverse effects, and cultural acceptability.
CONCLUSIONS: Our preliminary results suggest that Whitu may be an effective app for improving multiple dimensions of young people's well-being. Modifications to the look and feel, cultural content, and onboarding have been undertaken based on the qualitative feedback, and an RCT is currently underway.

Introduction

Prior to the COVID-19 pandemic, which arrived in New Zealand in early 2020, young people were already experiencing high rates of mental distress, depression, and suicide (Crengle et al., 2013; Gluckman, 2017; Fleming et al., 2007). Mental health issues are known to be heightened during times of increased anxiety, such as crisis events, including natural disasters and infectious disease outbreaks (Douglas et al., 2009; Norris, 2005). Furthermore, the current pandemic is likely to disproportionately affect young people due to prolonged school and university closures and mandated social distancing, disrupting peer interaction and socialization, a critical part of adolescent and young adult development (Orben et al., 2020). This impact is already reflected in emerging data from the United States, China, and Europe, where young people in particular are reporting increased rates of mental illness (Pierce et al., 2020; Wang et al., 2020b; O'Connor et al., 2020; Wang et al., 2020a).

Alarmingly, despite the increased need for mental health support during the COVID-19 pandemic, access to face-to-face mental health services is likely to be disrupted or delayed. Similarly, this pattern has already been documented for the provision of other types of healthcare, including physical and psychosocial support for vulnerable youth, as well as youth with chronic health conditions (Serlachius et al., 2020a; Green, 2020). Social distancing measures and national lockdowns mean that mental health services, like many other public health services, need to adapt to the new demands of the pandemic.

Digital mental health interventions offer a more feasible and scalable approach during the COVID-19 pandemic. Numerous reviews have supported the efficacy of digital mental health interventions in improving young people's mental well-being (Donovan and March, 2014; Ebert et al., 2015; Reyes-Portillo et al., 2014). Online interventions have also shown to be as effective as face-to-face therapies (Merry et al., 2012; Luo et al., 2020). More specifically, with the majority of young people reporting almost constant use of smartphones (Anderson and Jiang, 2018), mobile apps offer the unique advantages of increased accessibility, confidentiality, and applicability in real-life within-home environments (Liverpool et al., 2020). Additionally, there is growing evidence of their effectiveness at improving mental health and well-being (Firth et al., 2017; O'dea et al., 2020; Grist et al., 2017).

Due to the urgent need to address mental health during the COVID-19 pandemic, our research team rapidly developed a prototype well-being app between March and July 2020. The app was designed to support the mental health and emotional well-being of New Zealand young people, focusing on Māori and Pacific young people who prior to the pandemic have been disproportionately affected by mental health issues (Crengle et al., 2013; Clark, 2008). Whitu: 7 Ways in 7 Days (Whitu is 7 in the New Zealand Māori language Te Reo) was developed using evidence-based strategies including cognitive behavioral therapy and positive psychology techniques that have shown efficacy in adults and young people (Merry et al., 2012; Serlachius et al., 2016; Schache et al., 2019; van Agteren et al., 2021). The development and content of the Whitu app are discussed in more detail in our protocol paper (Serlachius et al., 2020b). Please see Fig. 1 for a screenshot of the 7 modules: (1) Feel, (2) Relax, (3) Be kind to yourself, (4) Be thankful, (5) Connect, (6) Look after your body, and (7) Set goals.

Fig. 1

Screenshots of the seven modules and associated badges of the Whitu: 7 Ways in 7 Days app.

This pilot study aimed to explore the preliminary efficacy of the Whitu prototype and examine changes from baseline to 6 weeks post intervention in well-being, depression, anxiety, stress, self-compassion, sleep, and optimism in a small sample of young people living in New Zealand during changing pandemic restrictions and lockdown periods. We also wanted to explore the usability and cultural acceptability of the app.

Materials and methods

Study design

A mixed-methods approach was adopted to determine the usability and cultural acceptability of the initial prototype of the app and examine any indicators of change in outcome measures, with the purpose of refining a second version of the app to test during a more formal RCT. This study was conducted in two parts: (1) a pre-post pilot study with quantitative outcomes evaluated at baseline, 2-weeks, and 6-weeks post-intervention, and (2) a qualitative focus group study. The study received ethics approval from the University of Auckland Human Participant Ethics Committee on 18th June 2020 (Ethics committee reference: 024542).

Pilot study

Participants

Participants were recruited using a flyer posted on online communities (e.g., Tuakana-teina/Māori student mentorship programs) and Facebook and Instagram advertising between 6th July 2020 to 10th July 2020. Participants were eligible if they were aged between 16 and 25 years old, living in New Zealand, had reliable access to Wi-Fi, owned either an iPhone or Android mobile phone, and could read and understand English. All participants received a $20 (NZD) gift voucher on completion of the final follow-up questionnaire.

Procedures

Once participants clicked on either the advertisement or flyer, they were directed to the Research Electronic Data Capture (REDCap) website and answered a series of questions screening for eligibility. If eligible, they were then provided with the Participant Information Sheet and an eConsent form. Once participants consented and completed the baseline questionnaires, they were then emailed the link to download the Whitu app and instructions to use the app over the next 2 weeks. Participants were emailed and texted a link to complete their follow-up assessments 2 and 6 weeks after completing their baseline questionnaires.

Measures

Demographic data, including sex, age, and ethnicity, was collected from all participants.

The following outcome measures were assessed at baseline and the 2- and 6-week follow-ups.

Mental well-being was measured by the Short Warwick-Edinburgh Mental Well-being Scale (SWEMWBS) (Fat et al., 2017; Tennant et al., 2007). Participants were asked to indicate their experience of a range of thoughts and feelings (e.g., “I've been able to make up my own mind about things”) over the past two weeks on a 5-point Likert scale (1 = none of the time to 5 = all of the time). To determine a total score (ranging between 7 and 35), items were summed and transformed. The scale has demonstrated good reliability (α = 0.84) and validity in adolescent and young adult populations (McKay and Andretta, 2017; Ringdal et al., 2018).

Depression was measured by the Center for Epidemiological Studies Depression Scale (CES—D) (Radloff, 1977). The CES-D is a 20-item measure designed to assess depressive symptoms in the general population. Participants were asked to indicate how often they have experienced a range of different behaviors or feelings (e.g., “I did not feel like eating; my appetite was poor”) on a 4-point Likert scale (0 = rarely or none of the time (less than 1 day) to 3 = most or all of the time (5-7 days)). To determine a total score (ranging between 0 and 60), positive items were reverse scored and items were summed. A cut-off score of 16 is generally used to indicate depressive symptoms. The scale demonstrates high correlations with other depression measures and excellent internal consistency (α = 0.85) (Radloff, 1977).

Anxiety was measured by the Generalized Anxiety Disorder 7-item Scale (GAD-7) (Spitzer et al., 2006). The GAD-7 is a scale designed to assess how often participants experienced symptoms of anxiety (e.g., “feeling nervous, anxious, or on edge”) over the past two weeks on a 4-point Likert scale (0 = not at all to 3 = nearly every day). Items were summed with scores of 5, 10, and 15 representing cut-offs for mild, moderate, and severe anxiety. The scale has demonstrated excellent reliability (α = 0.92) and validity in adults (Löwe et al., 2008) and adolescents (Mossman et al., 2017).

Optimism was measured by the Life Orientation Test-Revised (LOT-R)(Scheier et al., 1994). The LOT-R is a scale designed to measure dispositional levels of optimism (e.g., “In uncertain times, I usually expect the best”) on a 5-point Likert scale (0 = strongly disagree to 4 = strongly agree). To determine the total score, negative items were reverse scored and items were summed. The scale has demonstrated good reliability (α = 0.86) and validity in adolescents(Monzani et al., 2014).

Self-compassion was measured by the Self-Compassion Scale-Short Form (SCS-SF) (Raes et al., 2011). The SCS-SF is a scale designed to assess how participants typically act toward themselves in different situations (e.g., “when I fail at something important to me I become consumed by feelings of inadequacy”) on a 5-point Likert scale (1 = almost never to 5 = almost always). The SCS-SF contains 6 subscales comprising of 2 questions each, including: overidentification, mindfulness, self-kindness, self-judgement, isolation, and common humanity. To determine the total score, negative items were reverse-scored, then the means for each subscale were calculated and added together for the total mean, with higher scores indicating greater levels of self-compassion. The scale has demonstrated good reliability (α >0.86) in an adolescent sample (Bluth et al., 2016).

Stress was measured by the 10-item Perceived Stress Scale (PSS-10) (Cohen et al., 1983; Cohen et al., 1994). Participants were asked to indicate the extent to which they have felt a range of stressors over the last month (e.g., “In the last month, how often have you felt nervous or stressed?”) on a 5-point Likert scale (0 = never to 4 = very often). To determine a total score, items 4, 5, 7, and 8 were reverse coded and items were summed, with low stress represented by scores below 13, moderate stress between 14 and 26, and high stress between 27 and 40. The PSS-10 scale has demonstrated excellent psychometric properties compared to other stress measures, with good reliability and validity (Lee, 2012).

Sleep quality was measured by the Sleep Quality Scale (SQS) (Snyder et al., 2018). This measure is a single item questionnaire (“During the past 7 days, how would you rate your sleep quality overall?”) and was assessed on an 11-point visual analog scale (VAS) (0 = terrible, 1-3 = poor, 4-6 = fair, 7-9 = good, and 10 = excellent). The SQS has been shown to have excellent concurrent and convergent validity with other lengthier sleep scales and has been demonstrated to be effective in determining clinically meaningful changes in sleep quality (Snyder et al., 2018).

User engagement was assessed by the App Subjective Quality subscale and the Perceived Impact subscale of the end-user version of the uMARS measure (Stoyanov et al., 2016). The Subjective Quality subscale score consists of 4 items that determine user experience (e.g., “Would you pay for this app?”). The four items were initially scored on a 5-point Likert scale ranging from 1 to 5, with each having a different anchor. The 4 items were then averaged to get an overall mean score. The Perceived Impact subscale score is derived from 6 items measuring the impact of using the app on knowledge, attitudes, and intentions on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree). These items were reported as individual scores. Overall, the uMARS demonstrates good internal reliability (α = 0.90), and the subscales demonstrate moderate reliability (α = 0.71 and 0.80) (Stoyanov et al., 2016).

Statistical analyses

Repeated-measures ANOVA was used to explore differences in means between the 3 time points (baseline, 2-weeks, and 6-weeks) across all the outcome measures. Effect sizes were calculated using eta squared (η2). Normality was supported and the assumption of sphericity was not violated in any of the measures. Means, standard deviations, and 95% confidence intervals are presented with the analyses. All analyses were conducted using IBM SPSS Statistics software (26).

Focus groups

Similar to the pilot study, participants were recruited using a flyer posted on online communities (e.g., Tuakana-teina/Māori student mentorship programs) and Facebook and Instagram advertising from 22 June 2020 to 29 July 2020. Participants were eligible if they were aged between 16 and 30 years old, were currently living in New Zealand, had reliable access to Wi-Fi, owned either an iPhone or Android mobile phone, and could read and understand English. Exclusion criteria included having participated in the Whitu pilot study. All participants received $40 (NZD) for attending the focus group.

Once participants clicked on either the advertisement or flyer, they were directed to the REDCap website and first answered a series of answers screening for eligibility. If eligible, they were provided with the electronic participant information sheet and consent form. Following the provision of electronic consent, participants completed baseline questionnaires, including basic demographic details (age, sex, profession, prior well-being app use) and indicated their preference for focus group dates. They were then emailed the link to download the Whitu app and instructed to use the app for one week before their focus group date.

We conducted 5 focus groups of 2-h duration using Zoom video conferencing. Two groups, reserved solely for Māori and Pacific young people, were facilitated New Zealand /Samoan and New Zealand/Māori health psychologists (N.C. & E.M.). Each focus group followed a semi-structured interview schedule devised by the study team (A.S., H.T., D.L., A.B., N.C. & E.M). Questions covered the participants' first impressions of the app, usability, design, technical difficulties, performance, and specific feedback for each module. Examples from each module were shown as prompts to discuss features the participants liked and disliked. Audiotaped recordings were transcribed by a professional transcriber at the University of Auckland.

Facilitators

Each focus group was facilitated by at least two members of the following study team: A.S. a female, European Senior Lecturer in Health Psychology; H.T. a male, Asian Child Psychiatrist, Pediatrician and Senior Lecturer; N.C. a male, Pacific Island health psychologist; E.M. a female, Māori health psychologist; D.L. a male, Asian psychology student; and A.B. a female, European health psychology PhD candidate. All facilitators had experience in facilitating focus groups or group sessions with young adults. There was no established relationship with the participants prior to study commencement and participants were informed that the facilitators had developed the app and were completing this study to develop a new version.

Qualitative analysis

Audiotaped, transcribed feedback was independently extracted and analysed by two authors H.T. and D.L. using directed content analysis, a qualitative approach that is well suited for focus groups or interviews where predetermined concepts or categories are examined (e.g., usability and acceptability of the different functions and content of an app)(Hsieh and Shannon, 2005). Data was examined to the point of thematic saturation and any discrepancies in coding were resolved by consensus.

Results

Participant characteristics

Participant demographics were similar between the pilot and focus group cohorts, however, ethnicity was more diverse among focus group participants (see Table 1).

Table 1

Demographic characteristics of the pilot and focus group participants.

Characteristics	Pilot study (n = 20)	Focus Groups (n = 21)
Age (years)/mean (SD)	21.25 (3.26)	22.05 (3.25)
Sex (female) N (%)	15 (75%)	18 (85.7%)
Ethnicity N (%)
New Zealand European	14 (70%)	11 (52.4%)
Māori	0	2 (9.5%)
Pacific	1 (5%)	7 (33.4%)
Chinese	2 (10%)	0
Other ethnic groups	3 (15%)	1 (4.8%)
Occupation N (%)
Paid work	11 (55%)	8 (38.1%)
Student	9 (45%)	13 (61.9%)

The mean age for the participants in the pilot study was 21 years (SD 3.3) with an age range of 16-25 years. Of the 20 participants, 15 were female (75%) and 1 identified as non-binary. In terms of ethnicity, 70% (n = 14) of participants identified as New Zealand European, 1 participant identified as Samoan, 2 as Chinese, and 3 as other ethnic groups. Out of the 20 students, 9 participants (45%) were school or university students and the remainder were in paid employment.

Twenty-one young people attended the focus groups. The mean age of focus group participants was 22 years (SD 3.3), with an age range of 16-28 years. The majority of the participants were women (86%). Just over half of the participants were New Zealand European (N = 11, 52.4%), 2 participants (9.5%) were Māori, 7 were of Pacific descent (33.4%), and 1 self-reported as other ethnic groups (see Table 1).

Quantitative results: pilot study

The psychosocial outcomes reported at baseline for the pilot study participants reflected moderate to severe levels of depressive symptoms, anxiety and stress (Table 3). The group mean for depressive symptoms at baseline was 30.56 (SD = 9.75), 12.94 (SD = 4.52) for anxiety and 27.56 (SD = 5.72) for perceived stress. Out of the 20 participants recruited for the pilot study, 4 participants were lost to follow-up with 2 participants dropping out at 2-weeks and another 2 at 6-weeks.

Table 3

Mean differences across time in outcome measures (N = 16).

Outcome measures	Mean (SD)	Estimated mean difference from baseline (95% CI)	P
Well-being
Baseline	18.69 (4.81)	Reference	Reference
2-weeks	20.81 (3.73)	2.13 (−1.68 to 5.93)	0.461
6-weeks	22.88 (4.0)	4.19 (0.57 to 7.80)	0.021
Depression
Baseline	30.56 (9.75)	Reference	Reference
2-weeks	26.62 (12.13)	−3.94 (−8.72 to 0.85)	0.127
6-weeks	23.06 (11.00)	−7.50 (−14.41 to −0.59)	0.031
Anxiety
Baseline	12.94 (4.52)	Reference	Reference
2-weeks	8.94 (5.79)	−4.00 (−7.53 to −0.47)	0.024
6-weeks	7.25 (4.74)	−5.69 (−9.67 to −1.71)	0.005
Stress
Baseline	27.56 (5.72)	Reference	Reference
2-weeks	21.50 (7.13)	−6.06 (−11.11 to −1.02)	0.017
6-weeks	20.19 (7.26)	−7.37 (−12.40 to −2.35)	0.004
Self-compassion
Baseline	2.24 (0.55)	Reference	Reference
2-weeks	2.56 (0.53)	0.31 (−0.14 to 0.76)	0.240
6-weeks	2.82 (0.75)	0.57 (−0.04 to 1.19)	0.071
Optimism
Baseline	11.38 (3.72)	Reference	Reference
2-weeks	10.50 (3.18)	−0.88 (−3.37 to 1.62)	1.00
6-weeks	8.88 (3.44)	−2.5 (−5.63 to 0.63)	0.144
Sleep
Baseline	4.63 (2.16)	Reference	Reference
2-weeks	5.13 (2.28)	0.5 (−1.20 to 2.20)	1.00
6-weeks	6.31 (2.30)	1.68 (−0.32 to 3.70)	0.117

P-values <0.05 are indicated in bold. Mean differences between 2 and 6-weeks not tabulated.

User engagement to the Whitu app

Self-reported user engagement was assessed using the uMARS, which is scored out of a range of 1-5 (see Table 2). The subjective app quality score (the average of the 4 items that examine overall user experience) was >3 for both follow-ups. All 6 individual items that examined perceived impact of the app had average scores >3. For the awareness item (“This app has increased my awareness of the importance of addressing behaviors to help me manage my stress”) the mean scores were 3.83 (SD = 1.20) at 2-weeks and 4.00 (SD = 1.10) at 6-weeks. The next highest rated uMARS questionnaire item was behavior change (“Use of this app will help me improve how I manage my stress and anxiety”) which was rated 3.83 (SD = 1.04) at 2-weeks post intervention and 3.81 (SD = 1.05) at 6-weeks post intervention.

Table 2

Self-reported user engagement scores according to the uMARS.

Measures	2-weeks (N = 18)	6-weeks (N = 16)
uMARS (score range 1–5)
Subjective app quality score	3.07 (0.78)	3.23 (0.67)
Perceived impact: awareness	3.83 (1.20)	4.00 (1.10)
Perceived impact: knowledge/understanding	3.78 (1.06)	3.56 (0.89)
Perceived impact: attitudes	3.28 (1.02)	3.75 (1.13)
Perceived impact: intention to change	3.44 (1.04)	3.69 (1.01)
Perceived impact: help seeking	3.61 (0.98)	3.69 (1.01)
Perceived impact: behavior change	3.83 (1.04)	3.81 (1.05)

Data are means (SD).

Differences in outcome measures over time

A repeated-measures ANOVA was used to compare differences across the outcome measures from baseline to 2-weeks and 6-weeks follow-up. The ANOVA demonstrated that there was a significant difference in well-being across time [F(2,30) = 5.38, p = 0.010], with a partial η2 = 0.26. A significant difference across time was also demonstrated for depression [F(2,30) = 5.01, p = 0.013], anxiety [F(2,30) = 8.53, p = 0.001], self-compassion [F(2,30) = 4.84, p = 0.015] and stress [F(2,30) = 10.84, p < 0.001], with partial η2s ranging from 0.25 (depression) to 0.42 (stress).

Pairwise comparisons demonstrated that well-being improved from baseline (M = 18.69, SD = 4.81) to 6-weeks (M = 22.88, SD = 4.0, p = 0.021). Similar results were found for depression (p = 0.031), anxiety (p = 0.005) and stress (p = 0.004), which all demonstrated statistically significant improvements from baseline to 6-weeks (see Table 3). Anxiety (p = 0.024) and stress (p = 0.017) also demonstrated significant improvements from baseline to 2-weeks. There were no significant differences between any of the outcome measures between 2-weeks and 6-weeks (all p-values >0.05). Table 3 presents the means and standard deviations at the 3 time points.

Qualitative results: focus groups

Twenty-one participants took part in 5 focus groups (see Table 1 for demographic characteristics). We identified 5 main themes from the transcribed data, namely: (1) factors affecting engagement, (2) issues with functionality, (3) preferences regarding aesthetics, (4) effectiveness and adverse effects, and (5) cultural acceptability of the app. Themes, subthemes, and supporting examples are summarised in Table 4.

Table 4

Key themes, subthemes and supporting examples from focus group data.

Theme	Subtheme	Example(s)
Factors affecting engagement	Variable preferences regarding module content and duration	The fact that the sessions were short made it fairly easy to come back to it whenever, like, if I'm on the bus or right before bed or wherever. [male, 16, NZ European]I just reckon that some of the videos for me were just a bit too long. [female, 21, Tongan]Maybe if there was a module or somewhere in the modules, there was built in something to help you deal a little more with the uncertainty [of lockdowns]. [female, 25, NZ European]
	Interactivity of exercises	It's like someone talking to you, explaining more about it, yeah which makes it feel like someone else is involved with you – it's not just you writing down your goals or whatever, so I liked that. [female, 22, NZ European]I found it really helpful that all the modules had kind of an activity to do that was a very good because kind of instantly you have to do it for yourself and being talked through. [female, 25, NZ European]
	Connection with the outside world	I think a social platform would improve the app a lot. You know, having support forum where people could kind of express their, you know, how they're feeling or any advice they can give to other people. But also like, it's kind of hard to describe this but I felt comfortable, you know, writing whatever feelings I had because it felt like it was private in a way, like kept to myself. [male, 16, Chinese]I feel like maybe if it was like a prompt, like message someone that might be okay. But yeah, I probably wouldn't want to like post on social media, yeah. [female, 21, NZ European]
	Incentives to return	I was sort of relying on the notifications. [female, 21, NZ European]If you're going through a time say you are feeling particularly stressed or something you would be obviously seeking out resources to kind of assist with that, which obviously the app helps with. [female, 23, NZ European]I think the badges are a good idea…most people this kind of thing really does motivate them to do it and I think that's quite a smart idea. [female, 17, NZ European]
Issues with functionality	Navigation and layout	I really appreciated how well like each module is kind of like separated out. And it's really easy to find what kind of thing you want to refer back to which is like very helpful. [female, 25, NZ European]When I first opened the app, it was a little difficult to navigate. [male, 16, NZ European]I remember when I opened the app and I saw the phone down the bottom, I was like I don't want to press that and it take me straight through to something. [male, 16, NZ European]When you first like use the app, it would be useful to kind of have like a walkthrough because there's so much stuff from this evening that I've like not known is there, which would just be like, really helpful. [female, 25, NZ European]
	Customisation and reminders	I like being able to like customise the time when apps give me notifications so I usually set them at like when I know, okay this is a time when I will actually have time to do it. [female, 22, NZ European]It would almost be nice if it gave you a reminder, like if you hadn't been on the app in a couple days. [female, 21, NZ European]
	Technical issues	Yeah, I didn't get any notifications. I actually did have a bit of bugs with the, there's a lot of crashing there. I'm not too sure, maybe I'm on an older version of Android. [male, 16, Chinese]I just had a little issue with the first module… putting the different coping strategies into the backpacks. It was saying it like to check it, but it was correct…So just a little glitchy there [female, 25, NZ European]
Preferences regarding aesthetics	Preferences regarding design	It's simple, it's easy to use. Yeah, it's set out really nicely. [female, 25, NZ European]I liked the modern look of it but, you know, the colour, it was really easy on the eyes, it just overall had a really nice modern look to it so it was pretty welcoming. [male, 16, Chinese]
	Dislike of characters	I don't want to mean here, but I really cannot overstate how much I dislike the avatars. [male, 20, NZ European]The smooth patterns and stuff clashed a little bit with the design of the avatars that talked to you. I thought they felt like kind of like ClipArt's from Word, like they felt a little out of place. [male, 20, NZ European]But when you see something that's like a cartoon, I don't really know that older people would want to use it. It's more like that associated with like something for children. [female, 26, Samoan]
	Appeal of audio	I think the voices were good because there's some apps with the voices are just like annoying and yeah just annoying but I found these, yeah very calming. [female, 22, NZ European]Also helped they have New Zealand accents. [male, 20, NZ European]
Effectiveness and adverse effects	Acquisition of new knowledge	I liked that it gave me some like exercises and stuff that I'd never really received or saw online before. Like some of the breathing ones and the relaxing ones and thinking about what you eat and stuff. [female, 17, NZ European]I hadn't used the traffic light system before so I found it quite helpful. [female, 25, NZ European]I think it's pretty good because its kind of reminds you that there are people who care. Um, you know, which you don't really think about sometimes you kind get in that like, everyone kind of, I don't want to be around anyone kind of mood and seeing that helps you to recognize who really does care about you and that you should appreciate them. [female, 17, NZ European]I don't think there was anything like completely new to me, but I guess it was a good reminder that when you are feeling a certain way there are things you can do about it. [female, 23, NZ European]
	Resulting changes in behavior	I felt like it actually changed the way that I spoke to the people that I was trying to connect with…you're just talking more positively with those people. [female, 21, NZ European]I mean, for me personally, the fact that some of the modules had had stuff, which I enjoyed, and found myself actually using outside of the app…And obviously the one where you tense all your muscles and then relax—I found myself doing that quite a bit actually, especially before I was going to bed and that one helped. [male, 16, NZ European]
	Adverse effects	I don't really like the part where you're kind of rehashing like a bad experience and then talking about how would you be kind to yourself. [female, 25, NZ European]I saw the timeline and it was all a bit intimidated because I don't personally like thinking too deeply about myself or will my mental state. [male, 16, NZ European][Module 5: Connect] asks for three people, whereas, I would only say that I actively really care about keeping in touch with maybe two. So, the third person felt a little bit unnecessary. [male, 20, NZ European]
Cultural acceptability	Relatability to own culture	I like how it has a little bit of a Māori tinge like I don't know what to say to it, which is really cool to bring in New Zealand culture. [female, 25, NZ European]Given the name of the app. I think it's important to try and use as much Māori in it as possible. [female, 25, NZ European]
	Non-tokenism	I think that like it's great that the cultural aspects are in there and needs to be but if you're going to do it, I think you kind of need to look at all parts of it as well…If you're going to do that maybe incorporating karakia to kind of make the person feel safe and connected to the actual app itself. [female, 24, Samoan]I think if you do follow that sort of stream of using and grounding even in Te Reo Māori or like sort of Polynesian symbols, it's also important to have that content just as much, if not more important, so it doesn't get taken badly or with that intention. [female, 20, NZ European/Samoan]
	Concern for diversity	[Module 6: Look after your body] I guess some people do have different views on maybe like what's healthy or what's unhealthy and it could be different, like cultural use as well. [female, 23, NZ European]

Overall, participants rated the app positively. While participants expressed variable preferences for different modules within the app, all participants found at least some of the techniques and knowledge taught by the app to be useful in supporting their overall well-being. In addition, all participants responded affirmatively to the question of whether they would recommend the app to their friends and family.

Factors affecting engagement

Most participants found the modules to be of appropriate length. Key incentives to continue using the app included the flexibility to complete the app at the user's own pace, daily reminders via push notifications, and increased perceived need for mental and emotional support (for example, due to being in COVID-19 lockdown or experiencing stress in daily life). Response to the badges awarded for completing modules was largely positive. Although some participants stated the badges were not personally motivating for them, they admitted that they could see how it might be a useful feature for others.

Only 1 participant was universally supportive of social media integration in the app for the purposes of sharing progress and peer support. Of those surveyed on the issue, about half were opposed to social media integration, citing a preference to maintain privacy on issues of well-being, while the other half suggested limited forms of social media integration. For example, a leader-board function was suggested where the number of badges earned by peers would be visible without sharing any further information.

Issues with functionality

The majority of participants did not report any major technical or user experience–related difficulties that interfered with their ability to use the app successfully. Minor glitches that did not interfere with the main functionality of the app were reported by 2 users with older devices. However, about half of participants commented on the lack of clarity regarding some of the app features, such as the button to access immediate psychological assistance on the main page of the app. An improved in-app onboarding process was recommended to address this issue.

A number of participants reported mixed feelings with regards to daily reminders to use the app delivered via push notifications. While most participants admitted the reminders did prompt them to use the app, about half also found them annoying or felt they were being sent at inconvenient times. One participant admitted to disabling notifications entirely on their device. Three participants suggested being able to customise the time the reminder was given would be useful.

Preferences regarding aesthetics

While a couple of participants commented they found the design of the app clunky or excessive, the majority of participants consistently praised its simple, colourful, and modern look and feel. Participants also commented positively on the voice acting in the videos, with only 1 participant stating they would have preferred to have no voiceover. On the other hand, participants almost unanimously disliked the simplistic, cartoonlike, and relatively childish design of the avatars, which they felt was incongruous with the more sleek and contemporary design of the app.

Effectiveness and adverse effects

Many participants found the module content novel and helpful. For example, the traffic light system in Module 1 was cited by 5 participants as being a new tool to think about and categorise emotions. Two participants stated they found the language in some modules to be simplistic to the point of being condescending and a few said that they did not find the modules contained any new information for them. Participants who stated the app did not teach them anything new were able to acknowledge that it was still a helpful reminder of skills and information they already knew. Three participants reported feeling overwhelmed by some of the reflective content of the app, namely being intimidated by the ‘Goal Setting’ module, feeling worse by not being able to name 3 support people in the ‘Connect’ module, and feeling unpleasant about having to reflect on negative memories in the ‘Be Kind to Yourself’ module.

Cultural acceptability

Most participants appreciated the use of local voices and Māori language and symbols, explaining it made the app feel more relevant and relatable to their own situation. Five participants from the Māori and Pacific focus groups recommended integrating a more holistic view of health and well-being to reduce feelings of tokenism and improve the cultural congruity of the app. This included suggestions to include references to Māori spirituality, connection with nature, and more consideration for the health of whānau (family) and community. Five participants additionally raised concerns about sensitivity to cultural differences, with specific recommendations to make dietary advice in the ‘Look After Your Body’ module more relevant to people from different ethnic backgrounds or with dietary restrictions.

Discussion

This pilot study used a mixed-methods approach to assess the user and cultural acceptability of a new well-being app, as well as estimate its preliminary efficacy in improving the mental health and emotional well-being of a small sample of New Zealand young people. Piloting digital well-being interventions for young people is especially relevant in light of evidence showing young people's mental health has been disproportionately negatively affected by the COVID-19 pandemic (Pierce et al., 2020; Wang et al., 2020b; O'Connor et al., 2020; Wang et al., 2020a) and the lack of currently empirically validated, widely-available options for mental health support.

Summary of key findings

During this pilot study, participants who used the Whitu app experienced statistically significant improvements in well-being, depression, anxiety, and stress between baseline and 6-weeks. Findings regarding user engagement and acceptability were also promising, as evident from high uMARS scores, low rates of participant drop-out, and positive qualitative findings. Three participants from the focus groups also reported feeling overwhelmed when using the app, possibly reflecting the feelings of discomfort that may occur when learning new reflective coping skills. Importantly, the Whitu app also provides links to crisis support services in New Zealand for anyone who is in need of additional support.

Reported improvements in well-being, depression, anxiety, and stress at 6-weeks are especially promising as this coincided with a 2-week national lockdown during August 2020. Larger effect sizes at 6-weeks, rather than at the 2-week follow-up may reflect either a time lag for symptomatic change or cumulative benefit from ongoing app use. Self-reported optimism decreased across the study period, and although this change was not significant, it may be indicative of the increase in uncertainty brought on by the lockdown in August 2020. Despite not recruiting from a clinical population, the pilot study participants reported moderate to high levels of depressive symptoms, anxiety and stress at baseline, which is likely to reflect the significant ongoing negative impact of the pandemic on young people's mental health.

Comparison with other research

Due to the pilot/pre-post study design and the use of different therapeutic techniques, comparisons to other mHealth well-being interventions is challenging. However, reviews of RCTs evaluating online or eHealth interventions, in particular those based on CBT, have shown moderate to large effect sizes for improving depression and anxiety in young adults (Pennant et al., 2015; Grist and Cavanagh, 2013). Reviews assessing broader digital mental health interventions (incorporating different therapeutic techniques) for young people also suggest that eHealth interventions are promising for improving depression and anxiety between groups (Lattie et al., 2019). The findings from the forthcoming Whitu RCT will allow us to more easily compare effect sizes across studies for our key outcome variables.

Regarding our findings on user engagement as indicated by the uMARS, the scores were similar to a recent usability evaluation of a CBT-based mental health app called Mood Mission for adults aged 18-62 years in Australia (Bakker et al., 2018) but higher when compared to established norms (Hides et al., 2014). Despite these encouraging findings, the uMARS is a self-report measure, which according to a recent meta-analysis (Parry et al., 2021) may not be a reliable indicator of actual usage.

Implications for immediate app modifications and future research

As a result of qualitative feedback, a number of changes have been made to the Whitu app going forward. First, and most importantly, the avatars have been redesigned to align better with the overall aesthetics and style of the app. Second, with the assistance of our Māori investigator (EM), the use of Te Reo and Māori symbolism in the app has been augmented to provide greater cultural congruity and minimise cultural tokenism. For example, we added more Te Reo language throughout the app and a formal Māori welcome into the app (‘karanga’) during the onboarding process. Third, the onboarding process has been improved to make it easier for users to understand all of the features in the app. This has previously been demonstrated to be crucial to positive user engagement with digital interventions (Hightow-Weidman et al., 2020). Fourth, we have expanded gamification by including a native Puriri tree that grows with each badge collected on module or task completion. Fifth, we have expanded example food items in the ‘Look After Your Body’ module to reflect cultural diversity within the New Zealand population, such as including kumara (sweet potato) and rice. Lastly, we have also improved video formatting with clearer video controls to allow users to move between content more easily and to be able to watch the videos at their own pace.

There is mounting evidence of both short-term and long-term psychological effects from the continuing COVID-19 pandemic (Inchausti et al., 2020). In particular, evidence suggests measures to combat the disease such as national lockdowns and mandated social distancing may disproportionately affect young people's mental well-being (Boucher et al., 2021; Orben et al., 2020). While there have been a number of recent studies evaluating existing digital health interventions to improve mental health and well-being against the backdrop of COVID-19 (Hightow-Weidman et al., 2020; Boucher et al., 2021; Taylor et al., 2020; Alexopoulos et al., 2020), this project is the first to create and evaluate a custom-designed mobile intervention targeted specifically toward young people during COVID-19. In this regard, our results are encouraging as they support a body of growing literature suggesting that evidence-based digital health interventions are valuable tools to support youth in areas such as depression (Badawy and Radovic, 2020), anxiety (March et al., 2018), loneliness (Boucher et al., 2021), and general mental well-being (Seko et al., 2014; Dubad et al., 2018). In addition, our results suggest mobile apps delivering psychological tools are acceptable to a young New Zealand sample and effective, even during a lockdown period and time of changing restrictions.

Strengths and limitations

A key strength of our study is the incorporation of therapeutic tools and techniques with proven clinical effectiveness for young people, such as recognising and rating emotions, commonly used in psychoeducation (Taylor-Rodgers and Batterham, 2014), or using relaxation techniques and coping strategies, commonly used in CBT (Spence et al., 2011). The Whitu app was designed to not only offer young people a toolbox of evidence-based skills to use but was specifically developed to be used over the duration of 7 days in order to prevent the low rates of uptake and retention that are often seen in digital mental health interventions (Fleming et al., 2018). The pilot study had a high level of participant retention for a digital mental health intervention, with only 4 participants lost to follow-up.

The following limitations should also be noted. Due to the pre-post pilot design, our findings must be interpreted with caution. The sample size was small and not reflective of the ethnic diversity in New Zealand. All of the measures relied on self-report, which is prone to social desirability bias. A fully powered RCT of the updated Whitu app is currently underway, which will allow us to examine causality and explore the usability and efficacy of the improved second prototype of the Whitu app in a more ethnically diverse group of young New Zealanders.

Conclusions

For the moment, this study represents one of the crucial first steps toward developing an evidence-based and scalable digital intervention to help improve the well-being of young New Zealanders during the ongoing pandemic and beyond. As such, the second prototype of the Whitu app is currently being tested in a fully powered RCT.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. A.S. and H.T. came up with the concept for developing the Whitu well-being app. The app is owned by the University of Auckland and is not-for-profit.