What criteria are young people using to select mobile mental health applications? A nominal group study.

Objective: The popularity of smartphone technology provides a unique opportunity to make mental health support widely accessible, especially among young people. Despite the promising results of some mobile mental health support applications, the overwhelming number of available applications (apps) on the market makes it difficult to make a choice that will be safe and effective. Currently, widely available tools are either developed by experts, without end user input or are solely based on usability rankings. Thus, it remains unclear what aspects of mental health apps are important for young people. The purpose of this study was to determine what criteria young adults use when they select mental health applications and what is the relative importance of these criteria to inform the development of a user-driven app-rating platform.
Methods: We conducted 4 group sessions with 47 youth and young adults aged 15-25 in British Columbia, Canada using a modified nominal group technique. This method allows for establishing the relative importance of criteria in a structured group discussion. We recorded, transcribed and analysed the resulting data using qualitative content analysis and quantitative methods.
Results: Criteria that are the most important to young adults when selecting mental health apps include accessibility, security and grounding in scientific evidence. We identified specific aspects of the discussed criteria which were ranked in the order of importance.
Conclusion: Consulting end users about their priorities when evaluating mental health apps ensures that their values and priorities are incorporated into future app-rating platforms, alongside expert opinions. The present study also outlines the common contexts in which apps are used as well as their desirable features to inform mental health app development.

Introduction

One in seven children and adolescents aged 10–19 experience mental illness worldwide. The American Centres for Disease Control and Prevention report an increase over time in the number of depression and anxiety diagnoses among children. Many children and youth are also immersed in a high-tech environment that contributes to their mental health in ways that are inadequately understood. New and emerging technologies such as mobile devices, virtual reality and robotics are being researched as promising avenues to support the mental well-being of children and young people that would complement in-person care.[3-5] It is crucial that the development of these new tools for mental health promotion and prevention be scientifically supported and centred around children's and young people's needs.

The wide use of smartphones among young people provides both challenges and opportunities for their mental health. According to a 2019 survey of over 1600 children and young people conducted in the United States, 81% of that population owns a smartphone by the age of 14 years, and 29% of young people aged 13–18 years spend over 8 h using their phone everyday. A systematic review on the relationship between social media use and mental health in young people suggests that time spent on social media is associated with the risk of psychological outcomes such as depression and anxiety. While there are various mental health concerns that stem from heavy smartphone usage among young people, the prevalence of smartphones also provides an opportunity for wide-reaching mental health support through interventions such as mobile applications (apps). Many developers and public health organizations have successfully implemented apps intended to diagnose or treat certain disorders, as well as improving mental well-being in general.

As an example of the number of available apps, a search of Google Play Store using the keyword ‘anxiety’ returns over 200 results. There is a wide variety of available apps ranging from cognitive behavioural therapy-based interventions to apps with motivational quotes or daily affirmations. However, a search of both Android and iOS app platforms revealed that only 2.6% of the apps provided any support for their effectiveness claims. Although evidence-based mental health support apps do exist, there is little to no regulatory oversight of the content of published apps and as a result, the vast majority of available applications are not grounded in empirical work.[11,12] When evidence-based apps have been clinically tested, results were promising. Based on a 2020 review, app interventions targeting anxiety and depression showed small to medium effect sizes based on overall good quality evidence. The authors, however, point out the lack of information on specific populations such as young people that are most likely to use smartphone apps.

Apps that make false or misleading claims may not only add unnecessarily to already high screen exposure, but may also discourage the user from further help-seeking by misrepresenting potential treatment. Untested apps are potentially harmful to users, given that they are relatively easy to access and typically presented by a ranking system that is based on user reviews and download frequency. In particular, previous qualitative research suggests that young people may be inclined to search for a sense of social connection with others through social support features of the apps, which, if unmoderated, could be harmful. A review of claims made by mental health apps revealed that over one-third of app descriptions that referred to scientific methods included in-app techniques unsupported by evidence, such as dream therapy for schizophrenia or binaural beats for depression. Additionally, real-world data supporting the use of mental health apps is limited. Such claims may mislead potential users and encourage them to select an app that makes unsubstantiated promises. Another concern related to using mental health apps is privacy. According to a study of mental health app privacy policies, the majority of apps collect extensive information on their users and allow for sharing the data with third parties, with only one-third of policies protecting users’ personal information from being shared without their consent.

Lack of regulations combined with a large number of available apps makes choosing between different apps difficult. Seeking mental health help or support is especially difficult for young people, due to barriers such as lower mental health literacy, lack of trusted adults and mental health stigma. Mobile apps make accessing mental health support easier for this demographic, therefore facilitating the selection of apps that are safe and effective is important. To ensure that young people in need of support can find what they are looking for in the age of an overwhelming amount of easily accessible information, there is a need to gain an in-depth understanding of the priorities and needs of young people when selecting mental health apps, to inform evaluation tools based on user values and priorities.[19,20]

The priorities of mobile health app users and developers, while generally overlapping, may diverge for certain key criteria. Thus, involving young people in the development process of the mental health application rating tools is crucial to ensure that the rankings reflect the priorities and values of the end users. While ranking tools developed by experts may aid in selecting evidence-based mental health apps, the lack of user perspectives may limit their adoption by young people. User-centred approaches have been successfully implemented, particularly in the application of innovative health solutions. Some of the identified benefits of user involvement at the design stage include greater sustainability, relevance and acceptance of the final product[22,23] In the present study, we harness the benefits of a user-centred approach by involving the target population in the research process to better understand their perspective and identify problems that only arise in the context of their experience.

The present study is part of the End-User Led Evaluation of Application Technology to Support Youth Mental Health (ELEVATE-MH) project. The first step to developing a sustainable, patient-centred app-rating platform that would benefit young people is to better understand what matters to the end users when they select apps to support their mental health. In this study, we aimed to:

identify the criteria that young people consider important when selecting mobile apps to support their mental health and

rank the relative importance of these criteria, as perceived by the target population.

Methods

This research has been reviewed and approved by the Behavioural Research Ethics Board at the University of British Columbia (H18-02405). Written informed consent was obtained from all individual participants included in the study aged 19 or older, legal guardians provided written informed consent for participants aged 18 and younger. Additionally, participants below the age of 19 years provided written assent.

Research team and reflexivity

K.K. and K.M. facilitated the nominal group meetings, J.M.R. observed three of the four meetings. All three researchers are female-identifying. At the time of the study, K.K. was a research assistant at the Neuroscience, Engagement and Smart Tech lab at the University of British Columbia, headed by J.M.R. who holds a PhD in neuroscience. K.M. at the time of the study was a Project Manager at BC Children’s Hospital in Vancouver and a Master’s student in Public Health program at the Lakehead University. K.K. and J.M.R. had prior experience in conducting qualitative and quantitative research. K.M. had experience managing and coordinating health literacy projects and closely collaborated with Foundry network of youth centres.

Potential participants contacted K.M. via email and she answered any participation-related questions. She coordinated with Foundry centres and participants to schedule the meetings. The facilitators introduced themselves at the beginning of each of the meetings and presented the rationale for conducting the present study.

Theoretical approach

This research is grounded in a realist ontological perspective, which posits that external reality exists. This study uses an interpretive, social constructionist epistemological lens, as the knowledge generated in this study is socially constructed through participant interactions in a nominal group setting.

Participant selection

We conducted a series of 4 nominal groups with a total of 47 participants in British Columbia, Canada. During the meeting, we asked participants about preferred pronouns. Thirty-five participants used she/her pronouns at the time of the study and 12 participants used he/him pronouns. Each nominal group was conducted in a different city (populations ranging from 33,000 to 675,000) to ensure a representation of young people from different settings. The number of participants varied between the meetings. There were 18 participants in Abbotsford (Abb), 13 in Vancouver (Van), 9 in Victoria (Vic) and 7 in Penticton (Pen). The nominal group meetings were carried out in 2019. We used convenience sampling to recruit participants. Young people aged 15−24, able to speak and read in English, were eligible to participate in the study. We recruited participants through posters, leaflets and social media advertising. Interested participants contacted one of the authors (K.M.) via email or phone. This author then provided more information about participation to interested individuals, scheduled the meetings and provided consent and assent forms in advance of the meetings. In total, seven prospective participants who contacted our research team did not participate. Out of seven, three young people did not come to the scheduled meeting and four were not able to attend due to either the location of the meeting or the date not working. This research was conducted in collaboration with Foundry, an integrated mental health service provider that operates with the guidance of a Provincial Youth Advisory group that meets quarterly and serves as a base for all projects.

We only collected the demographic information about the participants that was necessary for conducting the study, that is age range (15–18 or 19–25) and preferred pronouns. Since the participants were recruited via Foundry, it is reasonable to assume that they were a subsample of the population using Foundry services. Based on a 2021 report, two-thirds of Foundry users identify as White, one-third identifies as having sexual orientation other than heterosexual and one-in-ten have insecure living situations.

Setting

The nominal group meetings were conducted in person in the Foundry centres. Foundry is a network of centres throughout BC that provides mental health care, as well as primary care, social services and more. Each meeting took place in a private room. In all of the meetings, Foundry Peer Support Workers were present.

Data collection

The researchers developed a detailed nominal group guide, which was pilot tested with the target demographic. We utilized the modified nominal group technique-a method for eliciting relative importance and rankings in a group setting, which makes it well suited for the purposes of this research. The audio from the nominal groups was recorded using an audio recorder and final outcomes of participants’ ranking of criteria were documented. In each of the meetings, one of the facilitators took a leading role and the second facilitator took a supporting role. The detailed outline of the nominal group procedure is available in the Supplementary material. Each of the meetings lasted up to 2 h, including a break.

Qualitative data analysis

After verbatim transcription of the audio recordings, we analysed the data using a thematic analysis approach. Thematic analysis is a process through which the meaning of qualitative data is characterized by extracting patterns (themes). We used an inductive approach to developing the codebook. As the first step in the analysis, two coders familiarized themselves with the dataset. We then inductively developed a coding guide, based on the first group interview transcript. We looked at the data line by line and based on the content, we developed labels or names for fragments of data. The codebook which includes all the codes with definitions was continuously updated, as new codes were added during the coding process. The codes were then categorized under broader, emergent themes.

Using MAXQDA 18 software, the transcripts were coded by two coders in small text increments (of about half of each nominal group transcript), until the coders became comfortable with the codebook and all the code definitions were finalized and clarified. As a result, 2.5 transcripts were coded by two coders (K.K. and K.V.) with interrater agreement exceeding 85% and the rest of the dataset was divided between the two coders and coded independently, consulting on any issues. All disagreements were discussed between coders until an agreement was reached. When the two coders could not achieve agreement on a particular code, a third researcher familiar with the dataset (J.M.R.) was asked to weigh in.

Themes emergent from the dataset were then reviewed to combine any related themes and remove any codes that proved irrelevant. The resulting themes were then organized, and their definitions were finalized. The quotes included in this manuscript have been edited to improve readability. The full code system is available in Supplementary material.

Quantitative data analysis

To analyse the outcomes of final rankings across all four nominal groups, we calculated the relative importance score of each of the criteria based on their appearance and rank across nominal groups. This method was based on a study by Manera et al., who developed a formula that accounts for the frequency of appearance of an item, as well as individual rankings in each of the group interviews. The values were computed as an average of the reciprocals of rankings. The importance score ranges from 1 (most important) to 0 (least important).

Results

Quantitative results

The top five rankings identified in the criteria ranking exercise were affordability (importance score 0.425), whether the application works as advertised (0.375), accessibility (0.333), ability to use the application offline (0.306) and privacy (0.229). The full ranking of the top 15 items can be found in Table 1.

Table 1.

Criteria ranking summary.

Ranking	Criterion	Importance score
1	Free	0.425
2	Does it work	0.375
3	Accessible	0.333
4	Works offline	0.306
5	Privacy	0.229
6	Accurate information	0.225
7	Customizable	0.208
8	Easy to use	0.104
9	Attentive app owners	0.083
10	Transparency	0.063
11	No ads	0.063
12	Resources	0.050
13	Unique design	0.042
14	Tracking	0.036
15	Digestible content	0.036
16	Visually pleasing	0.036
17	Community	0.031
18	Self-care tips	0.028

Qualitative results

We identified the seven most salient themes that emerged from the nominal group discussions: security, accessibility, usability, customizability, modality of intervention, transparency and quality of intervention. Additionally, participants discussed the context of mental health app use and the app features they consider desirable. While discussion of all types of mental health support apps was encouraged, the conversations were mainly based on apps with on-demand content, rather than telehealth apps. A summary of emergent themes is shown in Figure 1.

Figure 1.

Summary of emergent themes.

Security

Participants across all four nominal groups emphasized the importance of security. The main concern recognized by young people was the fact that the apps can collect sensitive information that relates to their mental health and daily life:The possibility of the information stored in the app being accessed by other people was concerning:Password protection was mentioned as an effective strategy to ensure that the application data remains secure in case other people (such as parents) access the device. However, not all participants considered password beneficial, as depending on the context of use and the purpose of the app, having a quick and easy access may be crucial:

“I would not like someone to see my mood or my thoughts”(Van, P11).

“Apps can get hacked and personal information might be out” (Pen, P6).

“if somebody was using the app to (…) help themselves out of an anxiety attack” (Abb, P16).

Accessibility

Several aspects of accessibility were discussed, including the cost of an app, availability of different language settings, accommodations for persons with vision and hearing impairments and issues related to signing up for an app or creating an account. Accessibility was determined to be a necessary condition for using a mental health app. The participants expressed that everyone should have equal access to mental health apps, which included persons with disabilities:

“Equal access in terms, of disabilities” (Van, unidentified speaker),

and equal access across the gender spectrum:The cost of mental health apps was discussed as a barrier to accessibility. In all of the nominal group meetings, some participants expressed that they would only use an app, if it was free of cost. An aspect that was identified as a contributor to this issue was the fact that young people usually do not have an income and thus would need to rely on their caregivers to purchase the app for them:The cost was also raised in the context of choosing apps over other ways of accessing mental health help. Participants expressed that apps allow access to support or information for people who cannot afford therapy or other forms of mental health help.

“I noticed that there are a lot more apps made more for girls than directed towards the men. When men might not show as much because of society codes, they still have their own issues and need help” (Pen, P7).

“If your parents have a credit card, you do not necessarily want them to know that you downloaded this kind of app” (Van, unidentified speaker).

Among the remaining aspects of accessibility that emerged was availability of mental health apps in different languages:Additionally, some participants named mandatory account creation as an access barrier. Reasons for reluctance to create an account were related to security concerns:

“I would say language (…) because what is it good to have, my first language” (Van, P11).

“I also like apps that don't make you (…) make an account and put in any information because if you really don't want people knowing that you're using it you can just download it and then get rid of it” (Vic, P8).

Usability

Within the usability theme, participants discussed app performance and efficiency, ease of use and navigation as well as their opinions on the presence of advertisements within mental health apps. App performance was considered universally important, as multiple participants expressed that they would not use an app if it had bugs or was slow:Another aspect of performance that was raised in two of the groups was battery usage: Ease of use was also linked to being able to navigate the app easily and getting instructions on how to use the app:Having instructions was also considered important, because some applications may be complex:Regarding the presence of advertisements, generally, young people understood that advertising is usually the main source of income for app developers; however, they agreed the presence of a large number of ads negatively affects the usability of the apps. Pop-ups and ads that take over the entire screen were identified as especially undesirable:The participants had several ideas on how to make the experience better for the users while maintaining ads in the app. An example of a potential solution was having ads that are applicable to the young people’s mental health needs, such as ads for local yoga studios.

“If it's glitchy it's not easy to use” (Pen, P5).

“[the app being] battery intensive. That is a big no for me” (Van, unidentified speaker).

“you want to be able to find what you're looking for and not have to search through everything” (Vic, P1).

“[instructions are important] so you can understand what's going on because I can learn to use an advanced app but not all at once and [when] they don't really give you directions” (Pen, P2).

“an app that has ads that keep popping up every five minutes it's not going to be helpful” (Abb, unidentified speaker).

Customizability

Customizability was understood in two ways – as the ability to change settings within the app to personalize it:but also as being able to adapt the app to the user’s specific needs:

“that's kind of cool when you have your own avatar that you can personalize,” (Abb, P18)

Several participants agreed that personalizing colours is especially important in a mental health app, as some people may find certain colours unpleasant:Being able to customize notifications was another criterion that the participants agreed on. Unwanted notifications were described as overwhelming and bothersome. In one of the groups, participants also discussed being able to change who is able to access their information within the app as an important aspect of customizability. For instance, some participants would want their friends or family members to access some of the information that they are inputting in the app (e.g. mood diary), but not all of it.

“customizability to make it fit for you and not just a broad spectrum of things” (Pen, P3).

“many colors may be overwhelming” (Vic, P9).

Modality of intervention

Some of the nominal group discussions revolved around the types of interventions that young people prefer to see in mental health apps. The preferred modalities of delivering mental health support included tracking, positive affirmations, meditation, note making, calming music. The discourse relating to various modalities of intervention emphasized the importance of having multiple help strategies available within the app, as the users have varied needs:Participants also expressed the need for an ‘all-in-one’ solution that would address multiple problems at the same time, as they were frustrated with having too many apps.

“especially with mental health, everybody copes in different ways (…). [A]pps that work for other people don't work for everybody” (Vic, P4).

Transparency

The discussion around transparency was focused on the accessibility of information about an app and within the app. One of the young people described transparency as honest communication from the developers and sponsors about every aspect of using the app:The participants indicated that knowing who has access to data and where the information contained in an app comes from are important criteria for them. For example, participants described that they would like to know where their information is going:In the context of apps that collect personal data that can then get shared with a healthcare professional or other people, participants mentioned that they would like to be informed in case their data gets compromised:The participants would also like to know the source of the information presented within the app. They discussed the importance of claims being supported by evidence:Further, participants discussed the content and clarity of apps’ use terms and conditions. The young people admitted that often they do not read the terms of use before agreeing to them. The participants also declared that they would like to receive that information in a more accessible manner:Additionally, in one of the meetings, the participants pointed out that the app should contain a disclaimer stating that using it should not be a replacement for seeking professional mental healthcare.

“I think in general just like disclosure about everything and then the app just being upfront and honest about, (…) the claims and the limitations how things are stored and (…) if they're selling the information (…) just be really upfront about it” (Vic, P1).

“If it's a mental-health thing where you are putting information into it then it's good to know who you are giving your information to” (Pen, P5).

“Say there is a data breach; I want to hear about it” (Van, unidentified speaker).

“they will be like ‘X percent of people deal with this’ or ‘this has been found to help’. But okay, where is the reference, or how do I know that you didn't just make this up?” (Vic, P2).

“Perhaps just simplifying that [terms and conditions] and not having too much jargon or too much reading. Because we all know, we just scroll down to agreeing” (Van, unidentified speaker).

Quality of intervention

The quality of intervention criterion is related to the extent to which the content of the app is credible, safe, achieves what it sets out to do, up to date and within the law. Participants considered evidence-based app content to be crucial in determining apps’ credibility:They also discussed the importance of apps delivering promised results:Additionally, young people expressed that the information in the apps should be frequently updated and that the apps should operate according to local legal standards, regardless of where the app data is being stored.

“we don't want like fake information or information that's incorrect” (Vic, P2).

“if it doesn't work then it's not going to do anything for anyone” (Pen, P5).

Desirable app features

Part of the nominal group discussions was centred around the features that young people would like the apps to have. Four main themes relating to app features emerged: features of the mental health intervention, user interface, technological features and healthcare coordination. Features discussed are summarized in Table 2.

Table 2.

Desirable app features.

Item	Definition	Example
Intervention features
Data analysis	The app enables analysis of data.	"I like seeing a graph that shows the highs and lows and your mood spikes"
Track progress	The app allows for tracking user’s progress.	"[The app being] progress based, or track the progress."
Incentive for use	The app contains incentives for continued use.	"[S]omething sort of unique (…) that sets [the app] apart or it kind of gives you that incentive to use it is really sweet."
Interacting with other users	The app allows for interactions with other users.	"[I]nteracting with other people. Instead of using it just for yourself, (…) sort of more of a social network in a way."
Resource page	The app contains a page with relevant mental health resources.	"[R]esources out in the real world."
Daily check-in	The app checks in with the user daily.	"It feels comforting that someone is checking on you even if it's just like the app."
Messaging	The app allows for messaging between the users.	"An online community so you can talk to the other users and other people "
Contact page	The app has a contact page to contact creators.	"A contact page if the app has issues. Like an IT department."
Suggestions	The app contains suggestions aimed at improving mental health.	"Could there be self-care tips in there, say you have hard time going to bed at night, could there be little advice like things to take, or steps to help you sleep at night."
Feedback	The app allows user feedback.	“With attentive app owners, if you are having privacy issues and you comment about it they should fix it.”
Ability to share account data	Users can share the data from the app with other people.	"I like apps where you can share your account with people who help you with your mental help as well so that they can also see how you are doing."
User interface features
Search function	The app has a search function.	"I really like that idea of having even just a search bar in apps."
Organized	The app is well-organized.	"[Y]ou wanna be able to find what you're looking for and like not have to search through everything."
Visually pleasing	The visuals in the app are appealing.	"[V]isually pleasing to look [better] than the doctor's office."
Symbols	The app uses symbols that are easy to understand.	"[A]lso include icons and stuff like that."
Consistent across devices	The design is consistent on different devices.	"If it's on a different device it should look the same."
Consistency in design	Every page on the app feels like it belongs to the same app.	"I prefer things to be wild and crazy and completely different on every page."
Colours	The colours of the app are appealing.	"Picking the right colours may make a difference."
Simple	The design of the app is not overly complicated.	"Simply laid out."
Interactive	The app has interactive features.	"Maybe it prompts you with either like a notification on your phone or something like that, so interactive, I guess."
Technological features
Notifications	The app has notifications.	"[T]hey can send you notifications at the same time everyday and just to remind you, like, go on the app. I think that's a big one."
Transfer across devices	The app and accounts are available and sync on various devices.	"If the app was attached to a computer website so you can do the same things you could do on the app but just on a computer if you want to or need to."
Link with other apps	The app should connect with other apps.	"It would be cool if you could sync [the app] with your calendar."
Healthcare coordination
Information sharing with a doctor	The app allows information to be shared between users and their doctors.	"[If] you want, for example, your counsellor to know, you can maybe give them access to that information."

Contextual themes

The themes that relate to the context of app use do not directly address our research question; however, they provide background on the circumstances of mental health app use in the target population. The main goals of using the app for young people in our sample consist of help calming down, connecting with other resources and increasing long-term well-being. The participants tend to use the apps for their everyday mental health needs, as well as in the context of acute anxiety or panic attacks. The need to be able to use the apps without having Wi-Fi or data was emphasized as necessary to enable easy app access in any situation:

‘what if you go camping and you're really struggling, you need something with you. Then there is something you can have on your phone, already downloaded, that is really important’ (Vic, P4).

Discussion

Principal findings

The goal of this study was to identify the criteria that young people utilize to select and continually use mental health apps and to determine the relative importance of these criteria. Overall, we found that the most important criterion for mental health app end users is accessibility, followed by quality of intervention, security, customizability and usability. These results provide insight into what aspects of mental health apps young people take into consideration when deciding whether to use them. The new knowledge of end user priorities will contribute to the creation of a mental health app-rating tool to aid young people in selecting apps.

Criteria and their relative importance

The most frequently discussed criteria do not correspond exactly with the rankings of criteria; however, we did not expect there to be a perfect overlap. Some categories that ranked highly were not extensively discussed because of the group’s consensus on the relative importance of a particular criterion. Therefore, here we will discuss the qualitative and quantitative findings together and compare and contrast our findings with those of other work in this area.

Three criteria that were of high importance to participants: free, accessible and works offline are all related to the theme of accessibility. The participants highlighted the importance of having easy access to mental health apps, as this mode of intervention may be selected by individuals who do not have the ability to get mental health support through other means. The issue of app cost was controversial, as the participants understood that there is a trade-off between the cost of the app and quality or potential presence of intrusive advertisements. Other possible issues with apps that are free may include not being evidence-based, not coming from reputable developers, predatory data practices, infrequent updates, lack of technical support. Nonetheless, having to pay for an app was considered to be a significant barrier for many of the participants, especially minors who do not have their own income or credit cards. This finding is consistent with exploratory studies looking at adolescents’ perspectives of mental health apps and smoking cessation apps. Since accessibility is one of the major advantages of mobile mental health delivery, it is crucial that young people’s concerns about access are taken into account when developing mental health apps. Indeed, the American Psychiatric Association’s (APA) app evaluation model places accessibility, including cost and availability offline, at a fundamental level of their framework to ensure that people in various circumstances can benefit from mobile mental health support. Another reason for the significance of app availability offline is the unequal access to the Internet worldwide. UNICEF estimates that globally, two out of three young people (<25 years old) do not have Internet access at home. Digital exclusion, especially in the present context of a global pandemic, may not only prevent access to freely available mental health resources, but also directly negatively impact mental health, by increasing feelings of loneliness, distress and anxiety.

Criteria related to security were most extensively discussed and privacy was ranked highly in terms of importance. Young people’s concerns about the security of their data are well-grounded. A 2019 study on data security of app interventions for depression revealed that less than half of all 116 investigated apps had a privacy policy. Having a privacy statement does not ensure adherence to the described standard – an assessment of privacy practices among mental health and smoking cessation apps showed that only a third of top-ranked apps appropriately disclosed their data sharing practices. Additionally, some mental health apps available on the market require permissions to access or alter the device’s storage, read phone status, view Wi-Fi connections and others which are considered to be higher risk by guidelines for app developers. End users’ concerns about the protection of their sensitive information and how their data is being used reflect a prevalent problem in the mental health app market. Higher data security standards should be adopted by app developers and distributors. Some recommendations for better data security include using higher encryption standards and activating a remote data wipe option for users. Further investigation into how young people navigate the landscape of terms and conditions in health applications and how it affects their decision making is needed.

The high ranking of does it work criterion reflects the importance of mental health intervention accuracy and the quality of information that the young people receive through the app. Young people expressed that evidence-based applications that actually help users achieve promised results are the ones that they seek out. Despite the high ranking of this criterion, young people did not describe how it affects their choice of mental health apps in practice. Providing sources was cited as one of the strategies used to determine app information quality, without further details. This finding highlights the importance of developing a comprehensive tool for mental health app evaluation to assist users in making these decisions. This need is compounded by the limitations of existing app-rating platforms, such as contradictory app ratings and lack of focus on consumer preferences. Applying this criterion in an evaluation tool may be problematic due to the fact that not all mental health apps have been thoroughly tested for effectiveness ; however, it may be approximated by determining whether the app was developed based on a credible treatment or practice. It is worth noting that the apps themselves can be tested as interventions, they can include evidence-based interventions, or both. For instance, an app for anxiety might include evidence-based cognitive behavioural therapy elements, but not be tested for effectiveness.

While most of the present study’s high-level findings are congruent with the APA’s app evaluation model, this study provides additional contextualization of abstract ideas, such as accessibility or security and relative importance of the different aspects of criteria (e.g. free being more important than any other aspect of the app). Further, the APA model includes ‘Data Integration towards Therapeutic Goal’ which was briefly mentioned during some of the nominal group meetings; however, it did not appear in the final rankings in any of the sessions, suggesting that it is not as important for young people as other criteria.

Some of the app dimensions that were considered significant by the nominal group participants were not captured by the APA model. For instance, the importance of little to no sign up, added benefit of being able to conceal the app on the device for additional privacy or the importance of attentive app developers that are receptive to user feedback. This discrepancy highlights the benefit of involving end users in the evaluation process to aid in the selection of mental health apps that young people would be more willing to use and possibly adopt in a long term.

Despite the present study’s focus on end user perspectives, it is worth noting that some evaluation criteria that may not be important to young people are crucial for healthcare practitioners and therefore should not be seen as irrelevant. For example, APA’s ‘Data Integration towards Therapeutic Goal’, while not mentioned in end user discussions, is valuable to mental healthcare providers. Therefore, to be effective, an app-rating platform should combine end user and expert input.

Limitations

We only collected demographic information necessary to conduct the study from the participants; therefore, we could not establish an in-depth characterization of the sample. To offset this issue and to increase the credibility of our findings, we chose to conduct the nominal group meetings in four different cities, with varied population sizes, average incomes and accessibility of mental health services. As such, our participants were a diverse sample coming from varied backgrounds and thus likely representative of young people in various circumstances. Additionally, a nominal group setting is prone to being influenced by a few vocal participants. Measures were taken to encourage everyone’s participation, such as every person only contributing one idea at a time and quiet ranking time using stickers. Lastly, the pool of participants was recruited through Foundry which provides mental health support for young people. Therefore, it is likely that the nominal group participants have a higher than average mental health literacy, which could limit the transferability of our findings. We attempted to mitigate this limitation by reporting on additional contextual themes that help explain how and when the participants in the study use mental health apps.

Conclusion

Wide accessibility of mental health applications is promising in providing mental health support to young people. However, the large number of available applications and varied quality makes it difficult for young people to select apps that are safe and helpful. Our study uncovered the criteria that young people use to evaluate mental health apps and their relative importance. The results suggest that young people’s priorities largely overlap with those of professionals since criteria such as accessibility, security and grounding of the intervention in evidence were most highly ranked. Our research also identified specific concerns that young people have regarding the abstract criteria present in app evaluation models. Taken together, these findings will inform the creation of an app evaluation tool that is based on users’ values and priorities to facilitate mental health support app selection for young people. Moving forward, future research efforts should investigate how young people’s preferred criteria correlate with their decision-making behaviour, and characterize criteria of importance for other stakeholders such as mental health care providers and family members. As the mental health app landscape is ever evolving, in-depth engagement with users and those who support them will continue to be critical to maximize positive outcomes as a result of app usage.

Click here for additional data file.

Supplemental material, sj-docx-1-dhj-10.1177_20552076221102775 for What criteria are young people using to select mobile mental health applications? A nominal group study by Katarzyna Kabacińska, Kaleigh McLeod, Annika MacKenzie, Kim Vu, Michelle Cianfrone, Andrew Tugwell and Julie M Robillard in Digital Health