Assessing the usability and user engagement of Thought Spot - A digital mental health help-seeking solution for transition-aged youth.

OBJECTIVE: To evaluate the perceived usability of and user engagement with a digital platform (Thought Spot) designed to enhance mental health and wellness help-seeking among transition-aged youth (TAY; 17-29-years old).
MATERIALS AND METHODS: Survey responses and usage patterns were collected as part of a randomized controlled trial evaluating the efficacy of Thought Spot. Participants given Thought Spot completed an adapted Usefulness, Satisfaction, and Ease of Use (USE) Questionnaire to measure perceived usability of the platform. User engagement patterns on Thought Spot were examined using analytics data collected throughout the study (March 2018-June 2019).
RESULTS: A total of 131 transition-aged participants completed the USE questionnaire and logged on to Thought Spot at least once. Ease of learning scored higher than ease of use, usefulness and satisfaction. Participants identified numerous strengths and challenges related to usability, visual appeal, functionality and usefulness of the content. In terms of user engagement, most participants stopped using the platform after 3 weeks. Participants searched and were interested in a variety of resources, including mental health, counselling and social services. DISCUSSION: Participants reported mixed experiences while using Thought Spot and exhibited low levels of long-term user engagement. User satisfaction, the willingness to recommend Thought Spot to others, and the willingness for future use appeared to be influenced by content relevance, ease of learning, available features, and other contextual factors. Analysis of the types of resources viewed and searches conducted by TAY end-users provided insight into their behaviour and needs.
CONCLUSION: Users had mixed perceptions about the usability of Thought Spot, which may have contributed to the high attrition rate. User satisfaction and engagement appears to be influenced by content relevance, ease of learning, and the types of features available. Further investigation to understand the contextual factors that affect TAYs' adoption and engagement with digital mental health tools is required.

Introduction

The transition between childhood and adulthood is critical, as the onset of most mental illnesses happens before youth turn 25-years, and the prevalence of suicidality is also the highest among this demographic (MacKinnon and Colman, 2016; Sumnall et al., 2010). Transition-aged youth (TAY), broadly defined as those between age 17- and 29-years, are a unique and vulnerable group for developing mental illness and suicidality (Mandarino, 2014). The challenges of this transition are exacerbated by poor help-seeking behaviour among TAY (Rickwood et al., 2007). Thus, various studies have stated the importance of helping TAY seek support because mental health issues that persist into adulthood can have negative health and socioeconomic consequences.

Given that TAY face different circumstances and exhibit different help-seeking behaviour compared to both children and adults, some studies have stated the need to create clinical services and solutions that are TAY-focused (MacKinnon and Colman, 2016). Among emerging strategies, there is a growing interest in using mobile health (mHealth) interventions to support TAY in finding and accessing mental health support. The current study looks at one such example, a co-created digital mHealth intervention called Thought Spot (VanHeerwaarden et al., 2018; Wiljer et al., 2016). Thought Spot supports transition-aged youth in seeking help from mental health and wellness resources (Centre for Addiction and Mental Health, 2019). The intervention allows users to locate and share resources through a map-based database on their devices. It also provides a private space for users to track their thoughts and moods (Fig. 1). Since 2014, TAY were involved throughout the entire development process of Thought Spot and played a crucial role in deciding the purpose, design, and functionalities of the intervention (Wiljer et al., 2017; VanHeerwaarden et al., 2018; Sennah et al., 2019).

Fig. 1

A compilation of screenshots depicting the various features of the Thought Spot mental health app.

From 2018 to 2019, a parallel-arm randomized controlled trial (RCT) was conducted to evaluate the efficacy of Thought Spot (Wiljer et al., 2020). When comparing users who received Thought Spot versus information pamphlets, there were no significant differences (i.e., group-by-time interaction) in help-seeking intentions, help-seeking behaviour, attitudes toward help-seeking, self-stigma, or youth empowerment (Wiljer et al., 2020).

Given the null results of the RCT (Wiljer et al., 2020), the research team examined how participants who were given Thought Spot engaged with the app. User engagement with mHealth technologies is a growing area of interest (Torous et al., 2020) because it can help researchers understand adherence and dropout during an RCT, which can potentially affect the efficacy of an intervention (Torous et al., 2018). In the literature, low adherence to internet interventions is a frequently reported observation (Christensen et al., 2009; Baumel and Kane, 2018; Eysenbach, 2005; Davis and Addis, 1999). Although there is a considerable amount of work that investigates this topic (Christensen et al., 2009; Baumel and Kane, 2018; Eysenbach, 2005), achieving optimal levels of engagement continues to be a challenge (Torous et al., 2018). Currently, there is lack of a gold-standard definition or established evaluation frameworks for user engagement of mHealth technologies (Lo et al., 2019; Perski et al., 2017; Pham et al., 2019a; Torous et al., 2020; Torous et al., 2018) which has led to a variability in how usage and engagement are measured. As outlined by Short et al. (2018), there are different methods for measuring user engagement, each offering different advantages and disadvantages in terms of validity and relevance. For example, back-end usage data (Pham et al., 2019b) is a popular data source to characterize user engagement through metrics such as number of logins and number of features used. However, it does not examine the experience (e.g., usability and user experience) aspects of user engagement that are critical for understanding effective engagement. Thus, to gain a deeper understanding of the impact of user engagement on the outcomes of an RCT, multiple approaches to measurement and analysis may be required.

Consequently, the aim of the current study is to look at perceived usability and TAYs' user engagement of Thought Spot throughout the RCT using quantitative usage data and qualitative survey data (Wiljer et al., 2016). Perceived usability is a core principle of user-centred design and can impact uptake and engagement with the intervention (Torous et al., 2018; McCurdie et al., 2012). For the purposes of this study, the definition of user engagement by Perski et al. (Perski et al., 2017) will be used: “engagement as using a digital innovation over time”. To our knowledge, few studies look at how TAY uses digital mental health interventions over a significant period of time (i.e., 6 months) (Lo et al., 2019). Thus, evaluating platform usage and engagement of TAY on Thought Spot can be helpful for understanding its potential impact on the outcomes of the Thought Spot RCT (Pham et al., 2019b; Pham et al., 2019a) while also informing how developers can optimize a mental health app for this demographic.

1.1 Objectives

This paper complements the objective of the broader RCT as a secondary analysis that focuses on the perceived usability and user engagement of Thought Spot by TAYs, with the following aims:

evaluate the perceived usability of Thought Spot by TAY at the end of the 6-month study period

characterize the usage and engagement patterns of TAY on Thought Spot using digital data collected throughout the RCT.

The results will contextualize the RCT findings (Wiljer et al., 2020) and inform usability and engagement-related factors that merit consideration in future mHealth development and evaluations.

Materials and methods

The methods used in the RCT have been published elsewhere and were approved by the research ethics boards at the Centre for Addiction and Mental Health (REB #023-2017) and the three participating post-secondary institutions (University of Toronto REB #34725, George Brown College REB #6004416, Ryerson University REB #2017-196) (Wiljer et al., 2016).

Participants

Individuals aged 17 to 29 years (median: 23, interquartile range: 20–25) enrolled in one of the participating post-secondary institutions were eligible for the study (Wiljer et al., 2016). Participants were recruited through class presentations, listservs and student groups. The current study focuses on participants who were randomized to the treatment arm (Thought Spot) and were instructed to use the intervention as needed.

Part 1: Usability survey

Instruments

After the RCT, participants were invited to complete a usability survey, which was an adaptation of the Usefulness, Satisfaction and Ease of Use (USE) questionnaire developed by Lund and colleagues (Lund, 2001). The questionnaire assesses ease of use, ease of learning, satisfaction and usefulness of a solution, and includes an open-answer section for feedback. The modified survey included 10 Likert-scale questions and two open-ended responses from the USE questionnaire (Lund, 2001) that were relevant to the features and functions of Thought Spot. It also included two questions that asked participants about their willingness to continue using Thought Spot and to recommend it to a friend. Demographic characteristics of participants who completed the survey were extracted from the broader RCT data (Wiljer et al., 2016).

Data analysis

The total score and individual components of the usability survey (Lund, 2001) were summed to calculate the mean score and standard deviation. Participants were excluded if they did not log into Thought Spot at least once because a fair evaluation of usability requires experience on the platform. Descriptive statistics were used to explore the distribution of usability measures. One-way ANOVAs determined the relationship between demographic characteristics and usability scores. Fisher's exact tests examined the impact of demographic characteristics on participants' willingness to continue using Thought Spot and to recommend it to a friend. Quantitative analyses were performed using R statistical package version 3.6.1.(R Core Team, 2018) Open-ended survey responses were assessed by two authors (BL, JS) using an inductive content analysis in NVivo 12.(Hsieh and Shannon, 2005; QSR International Pty Ltd, n.d.) These authors analyzed responses independently to create initial coding schemes, which were compared, discussed to address discrepancies and verified to produce approximately the same results (Elo et al., 2014).

Part 2: analysis of user engagement

The AMUsED framework was used to develop a post hoc analysis plan for the usage data (Miller et al., 2019). It involved three stages: 1) identifying possible measures of usage; 2) selecting relevant metrics and research questions; and 3) selecting analysis tools and developing a data analysis plan.

Stage 1: identifying possible measures of usage

This stage involved identifying relevant metrics from the features within Thought Spot (e.g. search, spots, thoughts and bookmarks) and from the overall RCT (Wiljer et al., 2016). Search and “spots” are the main Thought Spot features, which enable users to locate relevant mental health and wellness resources. Participants were asked to navigate the content and features of Thought Spot on an as-needed basis for six months. After the study, participants were free to continue using the platform. Back-end usage logs automatically recorded all interactions (e.g., clicks) and inputted data (e.g., spots accessed, search strings) across the core functions of the platform. Two authors (BL, JS) tested the validity of usage data through use cases to ensure that user activity was accurately tracked.

Stage 2: selecting relevant metrics and research questions

This stage highlighted contextual factors surrounding the usability data. Characterizing user engagement on the platform required evaluating the quantity, breadth and depth of platform usage. Given the post hoc nature of the current analysis, the research team could only use engagement metrics that were automatically stored to the back-end usage log during the RCT. Within the available usage data, the research team consulted a catalogue of metrics by Pham et.al (Pham et al., 2019a). The selected metrics were all associated with key Thought Spot features (e.g. login, search, clicks on mental health resources, etc.) because these features were most relevant to the youth mental health help-seeking process (Pretorius et al., 2019). After selecting the relevant user engagement metrics, the following research questions were developed (Miller et al., 2019):

Based on session duration and number of clicks, how long did users stay engaged and use Thought Spot throughout the six months of the study?

What types of resources (spots) did users choose to view on Thought Spot?

What searches were conducted on Thought Spot?

Did user engagement with Thought Spot differ across baseline demographic characteristics such as gender, college or university, and self-reported experience with mental health concerns and/or substance misuse?

Stage 3: selecting analytical tools and developing a data analysis plan

The final stage focused on creating a plan for data analysis (Miller et al., 2019), which was conducted using R statistical package version 3.6.1.(R Core Team, 2018) Descriptive statistics (mean, standard deviation) were used for overall user engagement. The relationship between engagement metrics and demographics was explored using Kruskal-Wallis ANOVAs. Mixed effects models were conducted using the lmer4 (Bates et al., 2015) and lmerTest (Kuznetsova et al., 2017) packages to evaluate the impact of usage over time on the duration and number of hits of each session. Fixed effects comprised of the trial week of each session, and participants were entered as random effects. Standardized beta for the fixed effects was calculated using sjstats (Lüdecke, 2020). Descriptive statistics (e.g., count) were used to determine the frequency of spots clicked and searches conducted. The threshold for significance was an alpha level of 0.05.

Results

Part 1: usability of Thought Spot

A total of 167 participants submitted the usability survey. Of these, 36 were excluded: one participant returned a blank survey and the others did not log on to the platform. This resulted in a sample of 131 surveys for analysis. Table 1 summarizes the demographics of participants who completed the survey. No significant demographic differences were found between participants who completed the survey and logged on to the platform at least once and those who did not.

Table 1

Demographic characteristics of participants who completed the usability survey.

Characteristic	Number of participants (n = 131) (%)	Excluded participants (n = 36) (%)	p-Value⁎
Gender
Female	106 (80.9%)	27 (75.0%)	0.21
Male	22 (16.8%)	8 (22.2%)
Non-binary	3 (2.3%)	1 (2.8%)
School status
Full-time	125 (95.4%)	35 (97.2%)	1.00
Part-time	6 (4.6%)	1 (2.8%)
Highest level of education
High-school diploma	80 (61.1%)	22 (61.1%)	0.73
College diploma	4 (3.1%)	1 (2.8%)
Bachelor's degree	38 (29.0%)	10 (27.8%)
Master's degree	4 (3.1%)	0 (0%)
Other	5 (3.8%)	3 (8.3%)
English as first language
Yes	87 (66.4%)	28 (77.8%)	0.23
No	44 (33.6%)	8 (22.2%)
Living location
City centre (downtown Toronto)	69 (52.7%)	19 (52.8%)	0.22
Outside city centre (downtown Toronto)	53 (40.6%)	11 (30.6%)
Other	8 (6.1%)	5 (13.9%)
Not reported	1 (0.8%)	1 (2.8%)

p-Values were calculated using Fisher's exact test in R.

The characteristics of the survey scores are shown in Fig. 2. The average total usability survey score was 53.04 (SD = 21.07; range 0–100). “Ease of learning” scored higher than other usability components. Of the participants, 29% (n = 38) indicated that they would use the platform in the future, and 58% (n = 75) indicated that they would recommend the platform to a friend. There were no differences in usability scores, willingness to use in the future or willingness to recommend to a friend between the demographic categories in Table 1 (Page 2-3 of the Supplement). A subgroup analysis of participants who identified as female yielded similar results (Page 4-5 of the Supplement).

Fig. 2

A violin plot with a box plot overlay depicting the distribution of scores across each usability component (n = 131).

Positive and negative aspects of the user experience

A content analysis of survey responses identified users' top three most positive and most negative aspects of Thought Spot. Three themes emerged: 1) usability and visual appeal; 2) functionality; and 3) usefulness of content.

Experiences with usability and visual appeal

Participants had mixed experiences and opinions on the usability and visual appeal of Thought Spot's user interface. Participants with positive experiences felt that the platform was intuitive and easy to use and found the user interface to be visually appealing.

“It does what it's supposed to. Easy to use. Clear interface.”

Participants with negative experiences stated that the platform was associated with a learning curve that made it unintuitive and not user-friendly. They thought the user interface was visually unappealing due to the colour, font size and layout.

“Not initially easy to figure out. Colour is not appealing.”

Experiences with functionality

Most participants liked the existing functionalities of Thought Spot, such as the ability to save spots, add spots, record thoughts and moods, add reviews and search for spots.

“I like that spots can be saved, so that's useful. I like that there are pre-made search terms to help refine the search more. I like the mood options.”

However, participants also stated that Thought Spot missed certain functionalities, which prevented them from using the platform to its full potential. These functionalities included the ability to make appointments, search by types of therapy, filter by language, log in with Gmail and create tags for categorizing newly added spots.

“There are a few times that I want to list the nice places I've just been, but can't find an adequate category for it, so I give up on sharing it.”

Participants also stated that the lack of prompts and notifications made Thought Spot easy to forget. This issue, along with the technical problems that most participants experienced with login-related functions, greatly reduced the frequency and duration of platform usage.

“I didn't like that it kept asking me to log in. I click ‘remember me’ but it doesn't [work]. That kind of deters me from using the app as often as I would like.”

“Not easy to learn, takes space on phone memory, no notifications to remind me.”

Experiences with usefulness of content

Most participants stated that Thought Spot's content (i.e., mental health resources) was useful and relevant.

“It provides good resources; there are lots of self-planners where you can reflect and journal.”

“It's helped me find help in downtown Toronto (mental health clinics, etc.). It also showed me things I didn't know about the city.”

However, the existing content within Thought Spot did not meet all the needs of participants. Some participants felt that some information, such as certain types of resources or the cost associated with services, was missing. There were also concerns that the information was outdated.

“No services like soup kitchens or food banks. Info is outdated at times.”

Usage data from 168 unique participants were included in the user engagement analysis, which represented 69% of the 241 participants randomized to the intervention arm of the RCT. Between March 2018 and June 2019, 535 unique sessions were recorded, resulting in 3696 clicks, 293 searches and 190 resources (spots) viewed. Eleven sessions were excluded in this analysis because the duration exceeded the expected typical (>2000 s) usage of the solution. The 15 months of back-end log data collected throughout the study (March 2018 and June 2019) revealed that the number of sessions per month ranged from 1 to 93 (Page 6 of the Supplement). About 50% of the sessions (n = 266) were conducted in the first five months, and most usage occurred between May 2018 and July 2018 and between September 2018 and December 2018. The spread of sessions on Thought Spot mostly reflects the distribution of participants randomized into the study (Fig. 3).

Fig. 3

A density plot depicting the distribution of sessions on Thought Spot and participants randomized to the trial throughout the duration of the study. The distribution of sessions is depicted in dotted lines and the spread of participants randomized into the study is depicted in solid lines.

The date of randomization was used as the starting point for analyzing each participant's user engagement pattern over the six-month study engagement period. Most users were active for one week (n = 103). Most sessions (n = 371) were completed within the first three weeks of enrollment in the trial. Similarly, most users (n = 151) were active for the first three weeks they were in the trial, followed by a steep decline to approximately 35 users logging in between four to nine weeks (Fig. 4). Usage stabilized at 1 to 6 users per week for the remainder of the trial. However, there was an increase in the number of users logging into Thought Spot after week 20, near the end of the study period. Users were allowed to continue using the platform after the study period, and 17 participants did so.

Fig. 4

A histogram depicting the number of unique users on Thought Spot over the number of weeks since randomization.

Session duration and number of clicks

Fig. 5 shows session duration and number of clicks. While most sessions occurred during the first three weeks after randomization (Fig. 4), the average duration per session increased as the trial progressed (Fig. 5A, top). A mixed effects model that specified participants as random effects was conducted to account for the fact that a single participant could be associated with more than one session. The number of weeks after receiving the app was significantly associated with session duration (estimate = 7.069, standard error = 1.669, 95% confidence interval (CI): 3.794 to 10.350, β = 0.19, p < 0.001). A similar trend was observed between the trial week and the number of hits on the platform (Fig. 5B, bottom). The trial week was significantly associated with the number of hits made during the session (estimate = 0.241, standard error = 0.043, 95% CI: 0.157 to 0.325, β = 0.25, p < 0.001). Similar findings were obtained for session duration (estimate = 4.675, standard error = 1.995, 95% CI: 0.752 to 8.632, β = 0.11, p = 0.020) and number of hits (estimate = 0.190, standard error = 0.050, 95% CI: 0.091 to 0.289, β = 0.17, p < 0.001) when sessions conducted beyond 30 weeks since randomization (potential outliers, n = 8) were excluded.

Fig. 5

Scatter plot depicting the relationship between analytics metrics (Fig. A: session duration; B: number of hits) and number of weeks since randomization. A linear regression line was added for each graph.

Types of resources (spots) identified

There were 1221 resources available on the platform when the study was completed, but only 15.6% (n = 190) of the unique mental health and wellness resources were accessed by participants among the 798 hits (e.g., clicking and opening resources to read). Of the resources that users viewed, 86% (n = 163) were physical locations. Other viewed resources included mobile apps (n = 15), websites (n = 11) and a phone line (n = 1). Accessed resources were most frequently tagged with the terms mental health (n = 95), counselling (n = 48), social services (n = 44), recreation (n = 38), sexuality and relationships (n = 37), school and academics (n = 27), group support (n = 26), physical health (n = 24), housing (n = 24) and substance use and addiction (n = 22).

The top 20 resources accessed by users represented 42% of all resource-specific hits (330 of 798). The two most popular resources were the mobile apps Happier (Happier, 2020) (n = 43) and Time Tune (TimeTune Studio, 2020) (n = 32). Happier allows users to share their moments with the app user community and Time Tune helps users plan their daily schedules.(Happier, 2020; TimeTune Studio, 2020) Users also frequently accessed four websites that provide peer support and education on topics such as mental health and sexual health, including Daily Strength (Daily Strength, 2020), 7 Cups of Tea (7 Cups of Tea, 2020), Open to Study (Open2Study, 2020) and Safe Sex.(State Government of Victoria Australia, 2020) The remaining resources (n = 13) were physical locations that provided mental health services for students (e.g., assessments and counselling) and spaces for studying and relaxing.

Types of searches conducted

There were 293 total searches conducted by 67/168 users (39.9%) on Thought Spot. Of these, 270 were included in the content analysis. Twenty-three were excluded because they were irretrievable from the back-end log data due to a technical glitch (n = 10) or were searches for specific addresses (n = 13). Almost all search strings leveraged the tags feature, which provided key terms for refining a search. There were 145 unique search terms, and the top 10 terms accounted for 43% (n = 207/478) of all searched terms. The most frequently used terms included mental health (n = 61), recreation (n = 25), relationships (n = 20). Other common terms included words related to diagnosis (e.g., anxiety, stress), location descriptors (e.g., garden) and topics (e.g., religion, sexuality).

Relationships between user engagement and demographic characteristics

Non-parametric Kruskal-Wallis ANOVAs were conducted to determine the impact of gender, type of post-secondary school (university vs. college) and self-reported experience with mental health concerns and/or substance misuse on engagement metrics. While gender had a significant relationship with the number of weeks active on the platform (H(2) = 8.25, p = 0.02), there were no relationships between gender and number of sessions (H(2) = 1.06, p = 0.59), total number of hits (H(2) = 0.09, p = 0.96) or total spots and searches made (H(2) = 0.73, p = 0.69). Participants who identified as female had more weeks active than those who identified as male or non-binary. There was no association of platform usage between university and college students, or those with experience with mental health concerns and/or substance misuse (Page 7 of the Supplement). A subgroup analysis was conducted with participants who identify as female, and similar results were obtained between platform usage and type of post-secondary school or experience with mental health concerns and/or substance misuse (Page 8 of the Supplement).

Discussion

Overall, TAY participants reported mixed views on the usability of Thought Spot, particularly around interface/visual appeal, functionality and usefulness. These factors likely translated to the observed user engagement patterns on the platform. While many end-users stopped using Thought Spot by the third week, the subset of participants who continued had significantly more clicks and a longer duration per session. Exploring the identified resources and searches conducted revealed the types of information that participants sought to support their mental health needs. These findings suggest several areas for further exploration and are described below.

Individual user engagement pattern analyses revealed that usage was concentrated at the beginning of the study, with a steep decline at week 3 of the 6-month trial (Fig. 4). Participants primarily used Thought Spot in the days right after randomization but less often afterwards. This is showcased in Fig. 3, where usage of the platform throughout the study reflected the distribution of participants randomized into the trial (Fig. 3). The research team saw an increase in the number of users on Thought Spot near the end of the trial (after week 20), suggesting a possible event during the trial that influenced users to revisit Thought Spot. However, the reason could not be accurately identified in the current post hoc analysis without further data collection from participants.

The usage patterns align with Eysenbach's law of attrition and the sigmoidal attrition curves reported by other mental health trials.(Christensen and Mackinnon, 2006; Eysenbach, 2005) Eysenbach applies Roger's diffusion of innovation model to explain the attrition curves using five factors associated with attrition: 1) relative advantage of the innovation over existing ideas; 2) compatibility with existing values, experiences and needs of adopters; 3) complexity of the idea; 4) trialability of the innovation; and 5) observability of results from the innovation to others (Eysenbach, 2005; Rogers, 2003). Some of these factors were observed in the usability survey responses and may have contributed to the drop in usage after 3 weeks. For example, the lack of relevant resources identified by some participants supports the idea that if the content on a digital mental health application is not compatible with TAYs' values, experiences or needs, then it may be hard for users to see an advantage over current strategies. The usability survey responses also reveal that participants' experience of using key features on the app, and the intuitiveness of features, may have impacted the use of Thought Spot (Nielsen, 1992; Nielsen, 1994). As mentioned by several participants, some features were complex and difficult to understand. Missing features were also another factor, as some users asked for additions such as push notifications and appointment booking functions to leverage the full potential of mobile apps. Although push notifications can encourage the use of platform features, more research with end-users is required to better understand the impact and value of these enhancements (Pham et al., 2019a; Pham et al., 2019b). However, in the specific context of Thought Spot, the usability survey responses suggest that satisfaction and usage were affected by content relevance, ease of learning, and the types of features available.

It is also important to mention that disengagement from mental health services is not unusual, and previous studies that have examined this behaviour may offer explanations for the usage drop-off observed in the current study. Suggestions for why young people disengage include mental health stigma and doubts about the usefulness of professional help. Although Thought Spot is not a true service but a tool that connects youth with relevant services, it may evoke the same thoughts and emotions that lead to dropout. Rates of disengagement with mental health services range from 4% to 46%, and one study reported that the median time to dropout for a prevention and early intervention program was five months (O'Brien et al., 2009; Anderson et al., 2013). Thus, a combination of Eysenbach's law of attrition and the tendency for youth to disengage with mental health services could explain the steep drop-off in usage.

Although TAY were engaged throughout Thought Spot's development process, just over half (58%) of participants who completed the adapted usability survey and used Thought Spot reported that they would recommend the platform to a friend. Yet when participants were asked if they would like to continue using Thought Spot, only 29% of participants indicated their intent to do so. The discrepancy between the willingness of future use and the willingness to recommend to others is an interesting finding that has at least two explanations. First, the research team did not require participants to have an active need for mental health support to participate in the RCT. Furthermore, some participants may not have needed to learn about mental health resources or services during the trial, and thus did not feel the need to use Thought Spot at all. Second, among participants who would recommend Thought Spot to others but would not use it themselves, there may be users who saw an early benefit without needing to use the app long-term (Davis and Addis, 1999). When contextualizing both explanations with the current study findings about usability and usage patterns, the importance of “quality over quantity” must be emphasized for developers of digital health solutions. Given how low adherence and low long-term usage is anticipated for many mHealth apps (Eysenbach, 2005), intervention developers must make sure that users' first experience of the app is useful, satisfying, and memorable. By doing so, an internet intervention is more likely to deliver a benefit to its users, even if the user does not adhere to an intervention. In the case of Thought Spot, participants' responses clearly indicate room for improvement because the usability survey results fall below the 82%–95% level of satisfaction reported by other internet interventions (Seko et al., 2014; Branson et al., 2013; Matthews et al., 2008).

The analysis of search strings and viewed resources revealed that TAY viewed a range of resources, including physical spots, websites and other mobile apps. Interestingly, given that 86% of resources viewed were physical locations, the two most viewed resources were mobile apps. This likely reflects students' desire for easily-accessible digital mental health resources (Punukollu and Marques, 2019; Kenny et al., 2016; Grist et al., 2017). This finding is particularly timely given the COVID-19 pandemic, where mental health services are shifting toward virtual delivery (Wind et al., 2020; Liu et al., 2020). The key terms used by participants could also reflect what transition-aged youth are seeking. Analyzing the search behaviours of youth using back-end logs may provide a data-informed way of identifying the emerging needs of this population (Pham et al., 2019b). It is also noteworthy that less than 20% of available resources on the database were viewed by end-users. Previous studies have identified the importance of tailoring content to end-users to achieve sustained engagement (Cheung et al., 2018). End-users also used a range of search strategies that included addresses and specific locations. These strategies were ineffective on Thought Spot because they were not anticipated during the design process and are beyond the capabilities of the search function. However, this finding demonstrates how reviewing analytics can reveal areas of improvement in design and user experience (McCurdie et al., 2012).

Gender, type of post-secondary school and presence of mental health disorders did not have a major impact on user engagement. Thus, the results suggest that the factors that contributed to the high attrition rate and low user engagement do not appear to be exclusive to a single group of users.

Limitations

The ratings and rationales provided on the usability survey were analyzed in aggregate and we did not examine whether there were differences between subgroups of participants. For example, participants with different mental health issues may have different expectations and attitudes toward using mental health apps (Baumel et al., 2019). As most participants only used the platform in the first few weeks of the study, it is unclear whether recall bias had an impact on usability survey scores. Back-end log data was also analyzed in aggregate and did not consider the influence of individual participants' motivations and needs around using Thought Spot. The research team made efforts to identify and remove usage patterns that were outside of expected usage (e.g., atypical session duration), but log data remain flawed and proximate in nature (Wolpert and Rutter, 2018; Pham et al., 2019b).

Due to the current study being a post hoc analysis, the research team could only explore usage data collected during the RCT. While the metrics that were selected revealed some interesting findings about activity on Thought Spot, there are some limits to the interpretability of these metrics. For example, session duration was a metric used as a proxy for measuring user engagement. However, the session duration metric does not account for whether the user was temporarily interrupted by other actions, tasks, or issues. Similarly, the increase in usage observed after week 20 may have artificially inflated the relationships observed between session duration, number of hits and number of weeks in the study. The research team could not determine the exact rationale for the increase and why users logged sessions after a prolonged break from usage. This must be taken into account when interpreting the relationships observed in the results section.

Similarly, it is unclear whether the number of resources viewed or searches on Thought Spot is a clear indicator of meeting the help-seeking needs of TAY (Perski et al., 2017). Both aforementioned examples demonstrate the challenges of usage data analysis because some metrics may not be precise enough to draw clear conclusions. Moreover, as indicated by the AMUsED Framework, developers of mHealth solutions must consider the interpretability of the usage data metrics they collect at the early stages of development (Miller et al., 2019). Doing so can enable more useful analysis but also improved decision-making during the design and optimization process.

Although TAY helped propose the main problem that Thought Spot was intended to solve, informed the design of the app, and supported usability testing, the current study findings point to gaps in Thought Spot's ability to address the needs and challenges of some TAY(Sennah et al., 2019; Wiljer et al., 2017; VanHeerwaarden et al., 2018). While over 140 different students participated in the co-design process(Wiljer et al., 2017), our results allude to possible volunteer bias, an important concern when conducting co-design studies. It is plausible that certain design decisions were skewed toward addressing the perspectives of participants who were most active during the development process. Thus, when a population of 481 TAY participated in the Thought Spot RCT, it is likely that the needs of certain TAY were not fully addressed and that may have contributed to mixed perceptions and usage observed.

The co-design process for Thought Spot was also primarily reliant on qualitative feedback received from participants when developing features on the app. Thus, the research team did not analyze preliminary usage data to gauge whether a particular feature of the Thought Spot prototype could affect the frequency of visits or number of resources viewed before the RCT. Furthermore, the findings of this study show that future co-design studies should consider analyzing usage analytics earlier in the development and optimization process. Analyzing usage data early on can supplement qualitative data by pinpointing potential issues or reveal disengaged users. The usage data can also serve as a preliminary indicator of whether design suggestions made by TAY lead to the expected behaviour changes.

Future directions

User engagement with Thought Spot may be improved by addressing the usability issues that participants identified in the current study. Exploring individual usage patterns and finding predictors of user engagement may facilitate the development of novel strategies to enhance implementation, engagement and user experience of mHealth solutions such as Thought Spot. It can also be valuable to include usage data analysis at multiple stages of the development and evaluation process to understand the impact of certain design decisions on user engagement. Finally, as few TAY-focused mHealth solutions exist, examining and comparing Thought Spot's user engagement with other TAY-focused mHealth solutions may help develop a framework that developers can consult to build interventions that this population will accept and benefit from (Pham et al., 2019a).

Conclusion

TAY users had mixed perceptions about the usability of Thought Spot and a high usage attrition rate was observed. Content relevance, ease of learning, and the types of features available were the key themes identified in TAYs' feedback about user satisfaction and engagement with Thought Spot. The post hoc analysis of qualitative responses and usage data highlight several considerations that can be useful for future developers of TAY-focused digital mental health tools. Foremost, the usage attrition and the user search patterns presented in the current study point to the importance of making key information on the app easily discoverable during participant's initial visits to the app. As many digital mental health solutions, including Thought Spot, face high attrition, it is important to make TAY's first experience on an app useful, satisfying, and memorable. Helping youth find tailored and relevant information faster can help make apps beneficial to them, even if they choose not to use the solution for more than a few times. Also, the current study showcases how a post hoc usage data analysis can be a valuable way of understanding TAY youths' behaviour on a mHealth solution. Data such as the types of clicks, resources viewed, and the searches TAY conduct, can help reveal the preferences, unmet mental health needs, and demands of this population. Rather than analyzing usage data following larger trials like RCTs, developers of TAY-focused digital interventions should consider analyzing usage data during the early development stages with smaller groups of individuals. Usage data may assist in identifying features with which the target user population may engage or disengage. Thus, in conjunction with using qualitative data, developers can become better informed when making design modifications which may lead to solutions that are more likely to meet the needs and preferences of TAY.