Quality evaluation of stress, anxiety and depression apps for COVID-19.

BACKGROUND: COVID-19 has caused increased stress, anxiety and depression with increased barriers to treatment. Mobile apps offer a potential solution, but there is no information on the quality of such apps recommended for COVID-19. This study aims to evaluate the quality of stress, anxiety and depression apps recommended for COVID-19.
METHODS: A search was conducted to identify relevant apps on the iOS and Android platforms. 44 apps were evaluated using the Mobile App Rating Scale (MARS), and the American Psychiatric Association's app evaluation model for data privacy and security.
RESULTS: Overall quality scores of iOS and Android apps were 3.69 ± 0.43 and 3.66 ± 0.47. Thirty percent had good/excellent overall scores. In general, the iOS and Android versions of the apps scored best for functionality (4.21 ± 0.48, 4.12 ± 0.53), followed by aesthetics (3.84 ± 0.50, 3.78 ± 0.56), information (3.39 ± 0.54, 3.40 ± 0.60), and engagement (3.31 ± 0.81, 3.34 ± 0.84). Over half (59%) shared personal information with third parties and 14% were compliant with data protection standards. LIMITATIONS: Only free apps available in Singapore were evaluated. Our results are time sensitive due to addition, removal, and update of apps in the app stores, thus our results should be extrapolated with caution to apps from other countries and paid apps.
CONCLUSION: Apps that addressed all three conditions had the highest overall quality scores. The top ranked apps (Sanvello, Woebot, Happify, Youper, Bloom) were of good quality, but majority were of acceptable quality and had room for improvement. App developers are encouraged to use our findings to improve and develop better quality apps.

Introduction

The coronavirus (COVID-19) pandemic is a global health crisis that has sent governments and citizens scrambling for resources and solutions to cope. The high rates of infection and fatality, sudden lockdowns, and the resultant economic crisis have caused high levels of stress (Hagger et al., 2020). These circumstances have resulted in an increased onset of mental health problems including anxiety and depression (Cullen et al., 2020; Figueroa and Aguilera, 2020; Galea et al., 2020; Ozamiz-Etxebarria et al., 2020; Rajkumar, 2020; Ueda et al., 2020; United Nations, 2020; World Health Organization, 2020c). It is estimated that the prevalence of stress, anxiety and depression among the general population during COVID-19 is approximately 29.6%, 31.9% and 33.7%, respectively (Salari et al., 2020). These psychological effects also predispose people to suicide (World Health Organization, 2019).

Mental health issues have been on an increasing trend prior to the pandemic, but access to treatment has remained poor (World Health Organization, 2020b). Anxiety and depression cost the world economy an approximate USD$1 trillion every year, while mean government expenditure on mental health remains low (World Health Organization, 2020b). The sudden emergence of stress caused by the COVID-19 pandemic has exacerbated the problems of rising mental health issues and the lack of appropriate mental health care (Figueroa and Aguilera, 2020). Additionally, the pandemic situation has increased the barriers to receiving necessary treatment with 93% of countries experiencing a partial, if not complete disruption, in mental health services. The ability to monitor the needs of patients and provide support have been impeded by social distancing measures (Pfefferbaum and North, 2020; World Health Organization, 2020a). Hence, there is a need to identify new and improved ways to cater to the growing mental health needs.

Stress, anxiety and depression apps provide a viable option for meeting mental health needs (Figueroa and Aguilera, 2020; Marshall et al., 2020). There has been an increase in demand for online mental health services and an increase in mental health app downloads in light of COVID-19 (Marshall et al., 2020). Stress, anxiety and depression apps have the potential to provide assessment, monitor symptoms and provide treatment, such as cognitive-behavioural therapy (Van Ameringen et al., 2017). They have been shown to be effective in reducing symptoms of stress, anxiety and depression (Coelhoso et al., 2019; Figueroa and Aguilera, 2020). Further advantages of turning to apps include lower costs, unlimited access to treatment, anonymity and convenience (Armontrout et al., 2018; Proudfoot et al., 2011). While stress, anxiety and depression apps can play an important role in helping people cope with the COVID-19 pandemic situation, it is important to ensure that they are of good quality. There are few stress, anxiety and depression apps that have been evaluated (Anthes, 2016) and are evidence-based (Figueroa and Aguilera, 2020). Although there are more than 10,000 apps available for mental health (Figueroa and Aguilera, 2020), from our knowledge, there is no information on the quality of apps that target COVID-induced stress, anxiety and depression. The hypothesis in this study is that stress, anxiety and depression apps that are recommended for COVID-19 are of adequate quality for users during the pandemic. The main objective is to evaluate the quality of stress, anxiety and depression apps that are recommended for COVID-19. A secondary objective is to provide a potential list of recommended apps that can be used during the pandemic and in the post-COVID era.

Methods

App search

Fifteen keywords were searched from 27 to 30 September 2020 on the Apple iOS and Google Android Play stores to identify apps that addressed stress, anxiety and depression (Fig. 1 ). The keywords searched included “stress”, “stress management”, “stress relief”, “anxiety”, “anxious”, “anxiety relief”, “depression”, “depressive”, “depressed”, “mental health” and “mental wellbeing”, and specific COVID-related terms like “COVID stress”, “COVID anxiety”, “COVID depression” and “COVID mental health”. The top 100 results returned for each keyword that belonged to the “Health and Fitness” or “Medical” categories were screened. In addition, the top 100 ranked free apps in the “Health and Fitness” and “Medical” categories of both app stores were included for screening. All apps were screened based on their titles and app store descriptions. Only those that were available in English and for stress, anxiety and/or depression were included. Apps were excluded if they were not in English, not for stress, anxiety or depression, did not provide symptom assessment or monitoring, did not provide an intervention, or were not downloadable. The remaining apps were then compared against a list of mental health apps that were recommended for COVID-19. This list was generated through a Google search conducted on 4 October 2020 for stress, anxiety and depression apps recommended for COVID-19. iOS and Android versions of the included apps were evaluated separately.

Fig. 1

Flowchart showing the app search and screening process.

App evaluation

The Mobile App Rating Scale (MARS), a validated quality assessment tool with excellent interrater reliability (intraclass correlation coefficient ICC = 0.79) and internal consistency (alpha = 0.90), was adopted to evaluate the 44 stress, anxiety and depression apps. It involved five sections of quality criteria – engagement, functionality, aesthetics, information quality, and subjective quality (Stoyanov et al., 2015). The engagement category evaluated apps on how entertaining, interesting, customizable, interactive, and well targeted they were to the audience. The functionality category assessed how well the app worked, ease of use, navigation within the app, and gestural design. The aesthetics section evaluated on the app layout, graphics and visual attractiveness; while the information section evaluated the quality and credibility of information provided by the app. The subjective quality category represented the evaluators' perceived value of the app, which in this case, was that of our raters. Each section consisted of questions that were scored on a 5-point scale (1-Inadequate, 2-Poor, 3-Acceptable, 4-Good, 5-Excellent). The mean scores of the first four objective sections were combined to give an overall quality score while the subjective quality section was scored separately.

In addition to the MARS tool, data privacy and security questions were adapted from a study based on the American Psychiatric Association's (APA) app evaluation model to evaluate the apps in this study (American Psychiatric Association, 2020; Lagan et al., 2020). These questions were rated on a dichotomous yes/no scale (Appendix A). All questions were compiled into a Google form and the apps were evaluated by two raters (LL and GG). The raters reviewed the MARS tool prior to beginning the evaluation to establish a common understanding of the questions and options. All apps were evaluated based on the descriptions found in the app stores and the raters’ walkthrough of the apps. All raters had no conflicts of interest with any of the evaluated apps or their companies in this study. Apps were downloaded onto an iPhone 6 s (iOS 13.5.1) or iPhone XS Max (iOS 13.6) and a Sony XPERIA (Android 5.1.1) or LG G6 (Android 8.0.0). The evaluation scores were tabulated for further analysis.

Data analysis

Descriptive statistics (means and standard deviations, SD) were used to report the findings. Interrater reliability was determined by two-way mixed intraclass correlation coefficient (ICC) where a higher value represented greater agreement among the raters. The Shapiro-Wilk test was performed to determine the normality of data. Data was considered to be normal for p-values > 0.05. The Spearman rank correlation test was used to determine if the overall app quality scores were correlated with the app store user ratings. Pearson correlation tests were used to determine the correlation between the subjective quality scores with the overall quality scores and the scores of the 4 objective sections of MARS. Independent t-tests were conducted to determine if there was a statistically significant difference between the mean scores of the iOS and Android versions of the apps. All analyses were conducted using the IBM Statistical Package for the Social Sciences (SPSS) software (version 26, IBM Corp, Armonk, NY).

Results

General app characteristics

Among all the apps evaluated, 10 apps (23%) were for a single condition (stress, anxiety, or depression), 13 (30%) addressed two conditions and 21 (48%) addressed all three conditions. Twenty-five apps (57%) provided a disclaimer that the apps should not replace medical care, while 21 apps (48%) provided crisis support within the app itself. All apps were designed as self-help apps, but 5 iOS and 5 Android apps also stated that they could be used in partnership with a mental health professional. Fourteen apps (32%) provided content related to COVID-19.

MARS quality scores

Overall quality

The interrater reliability between the two MARS raters was 0.78 (95% CI 0.53–0.89). The mean overall quality scores for iOS (n = 24) and Android apps (n = 20) were 3.69 ± 0.43 and 3.66 ± 0.47, respectively. Apps that targeted all three mental health conditions had the highest mean overall scores (n = 19; iOS 3.86 ± 0.43; Android 3.84 ± 0.49), followed by those that targeted only one condition (n = 10; iOS 3.73 ± 0.46; Android 3.64 ± 0.56). Apps that targeted two conditions had the lowest scores (n = 15; iOS 3.45 ± 0.32; Android 3.44 ± 0.34). Thirteen apps (30%) had a good or excellent overall score (4 or more), while 27 apps (62%) had an acceptable overall score (3 to < 4) and only 4 apps (9%) had a poor or inadequate overall score (< 3). In general, the apps scored best for functionality (iOS 4.21 ± 0.48, Android 4.12 ± 0.53), followed by aesthetics (iOS 3.84 ± 0.50, Android 3.78 ± 0.56), information (iOS 3.39 ± 0.54, Android 3.40 ± 0.60), and engagement (iOS 3.31 ± 0.81, Android 3.34 ± 0.84) (Table 1 ).

Table 1

MARS scores for all 44 apps presented as mean±SD.

Name of app (Mental Health Condition⁎⁎)	Platform	Engagement	Functionality	Aesthetics	Information	Overall Quality*	Subjective Quality
Sanvello (SAD)	iOS	4.10 ± 0.14	4.88 ± 0.18	4.00 ± 0.00	4.00 ± 0.07	4.31 ± 0.04	3.63 ± 0.18
	Android	4.10 ± 0.14	4.88 ± 0.18	4.15 ± 0.21	4.25 ± 0.07	4.34 ± 0.04	3.65 ± 0.21
Woebot (SAD)	iOS	4.40 ± 0.28	4.65 ± 0.21	4.30 ± 0.00	3.90 ± 0.28	4.31 ± 0.09	3.65 ± 0.21
	Android	4.40 ± 0.28	4.50 ± 0.00	4.50 ± 0.28	3.90 ± 0.28	4.33 ± 0.07	3.65 ± 0.21
Youper (SAD)	iOS	4.40 ± 0.00	4.40 ± 0.14	4.50 ± 0.28	3.25 ± 0.64	4.14 ± 0.19	3.15 ± 0.92
	Android	4.40 ± 0.00	4.40 ± 0.14	4.50 ± 0.28	3.25 ± 0.64	4.14 ± 0.19	3.15 ± 0.92
Happify (S)	iOS	4.60 ± 0.00	4.25 ± 0.00	4.15 ± 0.21	3.60 ± 0.14	4.15 ± 0.02	3.75 ± 0.00
	Android	4.60 ± 0.00	4.03 ± 0.32	3.85 ± 0.21	3.55 ± 0.21	4.01 ± 0.19	3.63 ± 0.18
Blooma (SAD)	iOS	3.80 ± 0.57	4.50 ± 0.71	4.50 ± 0.28	3.55 ± 0.35	4.09 ± 0.02	3.90 ± 0.14
Moodpath (SAD)	iOS	3.20 ± 0.00	5.00 ± 0.00	4.00 ± 0.42	4.00 ± 0.28	4.05 ± 0.04	3.15 ± 0.21
	Android	3.20 ± 0.00	5.00 ± 0.00	4.30 ± 0.28	3.80 ± 0.28	4.08 ± 0.14	3.13 ± 0.18
Mindshift CBT (A)	iOS	3.40 ± 0.57	4.65 ± 0.21	4.15 ± 0.21	3.80 ± 0.00	4.00 ± 0.14	3.65 ± 0.21
	Android	3.40 ± 0.57	4.65 ± 0.21	4.15 ± 0.21	3.90 ± 0.14	4.03 ± 0.11	3.65 ± 0.21
Wysa (SAD)	iOS	4.10 ± 0.14	4.40 ± 0.14	3.85 ± 0.21	3.50 ± 0.71	3.96 ± 0.30	4.40 ± 0.14
	Android	4.20 ± 0.28	4.40 ± 0.57	3.65 ± 0.49	3.65 ± 0.69	3.98 ± 0.46	4.40 ± 0.14
COVID Coachb (SA)	iOS	3.30 ± 1.27	4.40 ± 0.14	3.80 ± 0.71	4.25 ± 0.35	3.94 ± 0.62	3.28 ± 1.45
	Android	3.30 ± 1.27	4.40 ± 0.14	3.65 ± 0.49	4.25 ± 0.35	3.90 ± 0.57	3.28 ± 1.45
InnerHour (SAD)	iOS	4.10 ± 0.42	4.00 ± 0.00	3.65 ± 0.49	3.60 ± 0.28	3.84 ± 0.30	3.13 ± 0.88
	Android	4.10 ± 0.42	4.00 ± 0.00	3.65 ± 0.49	3.40 ± 0.28	3.79 ± 0.30	3.13 ± 0.88
Aura (SA)	iOS	3.30 ± 0.42	4.13 ± 0.18	4.00 ± 0.42	3.50 ± 0.28	3.73 ± 0.03	2.38 ± 0.18
	Android	3.30 ± 0.42	4.00 ± 0.00	4.15 ± 0.21	3.55 ± 0.21	3.75 ± 0.00	2.50 ± 0.35
Reflectly (SAD)	iOS	3.80 ± 0.85	4.15 ± 0.92	3.80 ± 0.71	2.90 ± 0.14	3.66 ± 0.65	3.15 ± 0.92
	Android	3.80 ± 0.85	4.40 ± 0.57	4.15 ± 0.21	2.90 ± 0.14	3.81 ± 0.44	3.15 ± 0.92
Rootd (A)	iOS	3.70 ± 0.14	3.90 ± 0.14	4.00 ± 0.00	3.25 ± 0.35	3.71 ± 0.16	2.50 ± 0.00
	Android	3.60 ± 0.28	3.90 ± 0.14	4.15 ± 0.21	3.05 ± 0.35	3.68 ± 0.25	2.40 ± 0.14
The Tapping Solution (SA)	iOS	2.80 ± 0.28	4.25 ± 0.35	4.15 ± 0.21	3.50 ± 0.00	3.68 ± 0.07	3.05 ± 1.06
	Android	2.80 ± 0.28	4.25 ± 0.35	4.00 ± 0.00	3.50 ± 0.00	3.64 ± 0.02	3.05 ± 1.06
Managing Your Stress & Anxietyb (SA)	iOS	1.90 ± 0.71	4.25 ± 1.06	3.35 ± 0.49	4.15 ± 0.21	3.41 ± 0.51	1.90 ± 0.57
	Android	1.90 ± 0.71	4.15 ± 1.20	3.35 ± 0.49	4.15 ± 0.21	3.39 ± 0.55	1.90 ± 0.57
Zen (SA)	iOS	2.80 ± 0.00	4.00 ± 0.00	4.30 ± 0.00	2.15 ± 0.21	3.31 ± 0.05	2.65 ± 0.49
	Android	2.80 ± 0.00	4.00 ± 0.00	4.30 ± 0.00	2.15 ± 0.21	3.31 ± 0.05	2.65 ± 0.49
MyLife Meditation (SA)	iOS	3.20 ± 0.57	3.13 ± 0.53	3.50 ± 0.28	3.10 ± 0.14	3.23 ± 0.38	2.30 ± 0.00
	Android	3.10 ± 0.71	2.88 ± 0.88	3.00 ± 0.42	3.10 ± 0.14	3.02 ± 0.54	2.13 ± 0.18
What's Up? (SAD)	iOS	3.20 ± 0.00	3.90 ± 0.14	2.80 ± 0.71	2.65 ± 0.21	3.14 ± 0.27	1.90 ± 0.14
	Android	3.20 ± 0.00	3.90 ± 0.14	2.80 ± 0.71	2.65 ± 0.21	3.14 ± 0.27	1.90 ± 0.14
Moodfit (SAD)	iOS	3.10 ± 0.42	3.40 ± 0.14	3.00 ± 0.00	3.00 ± 0.00	3.13 ± 0.14	2.13 ± 0.18
	Android	2.90 ± 0.71	3.30 ± 0.00	2.70 ± 0.00	3.00 ± 0.00	2.98 ± 0.18	2.13 ± 0.18
Moodmission (AD)	iOS	1.80 ± 0.57	3.15 ± 0.92	3.15 ± 0.21	3.65 ± 0.35	2.94 ± 0.51	1.80 ± 0.00
	Android	1.80 ± 0.57	3.15 ± 0.92	3.50 ± 0.71	3.70 ± 0.28	3.04 ± 0.62	1.90 ± 0.14
Self-help for Anxiety Management (A)	iOS	1.90 ± 0.14	4.15 ± 0.21	3.00 ± 0.00	2.30 ± 0.42	2.84 ± 0.09	1.65 ± 0.21
	Android	1.90 ± 0.14	4.15 ± 0.21	3.00 ± 0.00	2.30 ± 0.42	2.84 ± 0.09	1.65 ± 0.21
Balancea (S)	iOS	3.40 ± 0.00	4.25 ± 0.35	4.50 ± 0.28	3.45 ± 0.21	3.90 ± 0.21	3.90 ± 0.14
Moodnotesa (A)	iOS	3.20 ± 0.57	4.65 ± 0.21	4.15 ± 0.21	3.00 ± 0.00	3.75 ± 0.04	2.78 ± 0.04
ReachOut Breathea (SA)	iOS	2.00 ± 0.00	4.55 ± 0.35	3.50 ± 0.28	3.40 ± 0.14	3.36 ± 0.12	2.30 ± 0.71
	Platform	Engagement	Functionality	Aesthetics	Information	Overall Quality*	Subjective Quality
Mean scores of all apps in each platform	iOS (n = 24)	3.31 ± 0.81	4.21 ± 0.48	3.84 ± 0.50	3.39 ± 0.54	3.69 ± 0.43	2.92 ± 0.77
	iOS (n = 20) c	3.36 ± 0.83	4.15 ± 0.50	3.77 ± 0.49	3.40 ± 0.58	3.67 ± 0.45	2.86 ± 0.77
	Android (n = 20) c	3.34 ± 0.84	4.12 ± 0.53	3.78 ± 0.56	3.40 ± 0.60	3.66 ± 0.47	2.85 ± 0.76
	p-value c	0.96	0.83	0.99	0.99	0.92	0.97

Overall quality score was calculated by taking the mean of the scores from the four objective MARS sections: Engagement, Functionality, Aesthetics, and Information.

The condition(s) the app targets: S – Stress, A – Anxiety, D – Depression.

App was only available on iOS at the time of evaluation.

App was recommended for COVID-19.

P-values were determined by conducting independent t-tests on the 20 apps that were available in both iOS and Android platforms. Statistical significance was defined as p < 0.05. Shapiro-wilk tests were conducted to confirm normality of the data.

The top 3 ranked apps for stress, anxiety and depression on both iOS and Android platforms all had overall quality scores of above 4.00 (Table 2 ). Sanvello (iOS 4.31 ± 0.04, Android 4.34 ± 0.04), Woebot (iOS 4.31 ± 0.09, Android 4.33 ± 0.07) and Youper (iOS 4.14 ± 0.19, Android 4.14 ± 0.19) ranked among the top three positions on both platforms for all three mental health conditions. Additionally, Happify (4.15 ± 0.02) and Bloom (4.09 ± 0.02) were also among the top ranked apps on the iOS platform. The rankings of the top apps for the anxiety and depression categories were the same.

Table 2

Top 3 apps for stress, anxiety and depression as reflected by their overall MARS quality scores.

Mental Health Condition	Rank	Apps	MARS Overall Quality Score* (Mean ± SD)	Apps	MARS Overall Quality Score* (Mean ± SD)
		iOS platform		Android platform
Stress	1	SanvelloWoebot	4.31 ± 0.044.31 ± 0.09	Sanvello	4.34 ± 0.04
	2	Happify	4.15 ± 0.02	Woebot	4.33 ± 0.07
	3	Youper	4.14 ± 0.19	Youper	4.14 ± 0.19
Anxiety and Depressionb	1	SanvelloWoebot	4.31 ± 0.044.31 ± 0.09	Sanvello	4.34 ± 0.04
	2	Youper	4.14 ± 0.19	Woebot	4.33 ± 0.07
	3	Blooma	4.09 ± 0.02	Youper	4.14 ± 0.19

The overall quality score was calculated by taking the mean of the scores from the four objective MARS sections: Engagement, Functionality, Aesthetics, and Information.

App was only available on iOS at the time of evaluation.

Top 3 apps for the anxiety and depression categories are the same.

Among the top-ranked iOS apps, Sanvello scored the best in functionality (4.88 ± 0.18) and information (4.00 ± 0.07). Youper and Bloom scored the highest in aesthetics (4.50 ± 0.28 each), while Happify, which specifically targeted stress, scored the highest for engagement (4.60 ± 0.00), followed by Woebot (4.40 ± 0.28) and Youper (4.40 ± 0.00) – both of which targeted all three mental health conditions (Table 1). In contrast, the lowest scoring apps were Bloom for engagement (3.80 ± 0.57), Happify for functionality (4.25 ± 0.00), Sanvello for aesthetics (4.00 ± 0.00) and Youper (3.25 ± 0.64) for information. The trends for the top-ranking Android apps with the highest scores in the various categories were similar to their iOS counterparts, except for aesthetics, in which Youper and Bloom had the same scores (4.50 ± 0.28 each). In contrast, even though the Android version of Youper still scored the least for information (3.25 ± 0.64), the Android versions of Happify scored the least in terms of functionality (4.03 ± 0.32) and aesthetics (3.85 ± 0.21), and Sanvello in terms of engagement (4.10 ± 0.14). With the exception of Youper's information score, all scores attained by the top-ranking iOS and Android apps were above the mean quality scores for every other category.

Quality for engagement, functionality, aesthetics and information

The mean functionality scores of the evaluated apps were the highest among all the quality parameters – 4.21 ± 0.48 and 4.12 ± 0.53 for iOS and Android apps, respectively. Moodpath was the highest scoring app (iOS and Android 5.00 ± 0.00) for functionality (Table 3 ). The lowest scoring app in this category was MyLife Meditation, which had small differences between the platforms (iOS 3.13 ± 0.53, Android 2.88 ± 0.88). Among the 20 apps available on both platforms, the functionality scores for iOS versions scored better for 6 apps, while those of the Android versions scored better for one app (Table 1).

Table 3

Top scoring apps on the iOS and Android platforms according to the scores of each MARS category.

MARS Category	Rank	Apps	Mean Quality Score* (Mean ± SD)	Apps	Mean Quality Score* (Mean ± SD)
		iOS platform		Android platform
Engagement	1	Happify	4.60 ± 0.00	Happify	4.60 ± 0.00
	2	WoebotYouper	4.40 ± 0.284.40 ± 0.00	WoebotYouper	4.40 ± 0.284.40 ± 0.00
	3	SanvelloWysaInnerHour	4.10 ± 0.144.10 ± 0.144.10 ± 0.42	Wysa	4.20 ± 0.28
Functionality	1	Moodpath	5.00 ± 0.00	Moodpath	5.00 ± 0.00
	2	Sanvello	4.88 ± 0.18	Sanvello	4.88 ± 0.18
	3	WoebotMindshift CBTMoodnotesa	4.65 ± 0.214.65 ± 0.214.65 ± 0.21	Mindshift CBT	4.65 ± 0.21
Aesthetics	1	BloomaYouperBalancea	4.50 ± 0.284.50 ± 0.284.50 ± 0.28	WoebotYouper	4.50 ± 0.284.50 ± 0.28
	2	WoebotZen	4.30 ± 0.004.30 ± 0.00	MoodpathZen	4.30 ± 0.284.30 ± 0.00
	3	Mindshift CBTMoodnotesaHappifyThe Tapping Solution	4.15 ± 0.214.15 ± 0.214.15 ± 0.214.15 ± 0.21	SanvelloMindshift CBTReflectlyAuraRootd	4.15 ± 0.214.15 ± 0.214.15 ± 0.214.15 ± 0.214.15 ± 0.21
Information	1	COVID Coachb	4.25 ± 0.35	SanvelloCOVID Coachb	4.25 ± 0.074.25 ± 0.35
	2	Managing Your Stress & Anxietyb	4.15 ± 0.21	Managing Your Stress & Anxietyb	4.15 ± 0.21
	3	MoodpathSanvello	4.00 ± 0.284.00 ± 0.07	WoebotMindshift CBT	3.90 ± 0.283.90 ± 0.14
Subjective Quality	1	Wysa	4.40 ± 0.14	Wysa	4.40 ± 0.14
	2	BloomBalance	3.90 ± 0.143.90 ± 0.14	SanvelloWoebotMindshift CBT	3.65 ± 0.213.65 ± 0.213.65 ± 0.21
	3	Happify	3.75 ± 0.00	Happify	3.63 ± 0.18

Mean score of the two raters.

App was only available on iOS at the time of evaluation.

App was specifically for COVID-19.

The second highest quality parameter was aesthetics (iOS 3.84 ± 0.50, Android 3.78 ± 0.56). The highest scoring apps in this category (4.50 ± 0.28 each) were Youper (iOS and Android), Bloom and Balance (iOS only), and Woebot (Android only) (Table 3). In contrast, the lowest scoring apps were What's Up? (2.80 ± 0.71 both platforms) and Moodfit (iOS 3.00 ± 0.00, Android 2.70 ± 0.00). Within this category, the iOS and Android versions scored better for 6 and 7 apps, respectively among the 20 apps available on both platforms (Table 1).

The mean engagement (iOS 3.31 ± 0.81, Android 3.34 ± 0.84) and information scores (iOS 3.39 ± 0.54, Android 3.40 ± 0.60) were similar. Among all the apps, the iOS and Android versions of Happify (4.60 ± 0.00) and COVID Coach (4.25 ± 0.35) scored the highest in terms of engagement and information, respectively (Table 3). Furthermore, the Android version of Sanvello (4.25 ± 0.07) also scored high for information quality. In contrast, the lowest scoring apps were Moodmission (1.80 ± 0.57 both platforms) for engagement and Zen (2.15 ± 0.21 both platforms) for information. Among the 20 apps that were available on both platforms, 3 and 4 apps scored better in terms of engagement and information, respectively for their iOS versions, while the Android versions of 1 and 5 apps scored better in the same categories (Table 1).

Subjective quality

The mean subjective scores (iOS 2.92 ± 0.77, Android 2.85 ± 0.76) of the apps were the lowest in comparison to all the other objective quality parameters (Table 1). Both iOS and Android versions of Wysa (4.40 ± 0.14) scored the best, but both versions of Self-help for Anxiety and Management (1.65 ± 0.21) scored the worst in this category. The subjective scores of the iOS apps had the strongest correlations with the engagement and aesthetics scores (r = 0.75, p < 0.001 each), and weaker correlations with the functionality (r = 0.57, p = 0.004) and information scores (r = 0.42, p = 0.043). As with their iOS counterparts, the subjective scores of the Android apps were the strongest with engagement (r = 0.78, p < 0.001) and the weakest with information scores (r = 0.48, p = 0.032). The correlation of subjective scores with functionality (r = 0.64, p = 0.002) and aesthetic scores (r = 0.61, p = 0.005) of Android apps were similar. However, there was no correlation between the overall app quality scores and the app store user ratings for both iOS (rs = 0.038, p = 0.89) and Android platforms (rs = 0.300, p = 0.21).

Data privacy and security

Apart from the Self-help for Anxiety Management app where a privacy policy was available on the iOS but not on the Android version, the iOS and Android versions of the other apps did not differ in their data privacy and security practices. Majority of the apps (93%) had a privacy policy, while 91% declared their data collection and use, and 73% had security measures in place (Table 4 ). Only 11% of the apps had a privacy policy that was written at an acceptable readability level and 9% allowed users to opt out of data collection. Over three-quarters (86%) shared some form of data with third parties, with over half (59%) sharing personal information. Only 14% of apps were compliant with data protection standards.

Table 4

Data privacy and security characteristics of the apps.

Data privacy and security characteristics	Apps, n (%) (N = 44)
Has a privacy policy*	41 (93)
Has a readable (≤ grade 8) privacy policy	5 (11)
Declares data collection and use	40 (91)
Allows users to opt out of data collection	4 (9)
Allows users to delete their data	25 (57)
States where data is stored	18 (41)
Has security measures in place	32 (73)
Allows users to report suspected breaches	13 (30)
Shares data with third parties	38 (86)
Shares personal information with third parties	26 (59)
Compliant with data protection standards	6 (14)

For the Self-help for Anxiety Management app, a privacy policy was available on the Android app store but not on the iOS app store.

Discussion

This study aimed to evaluate stress, anxiety and depression apps recommended for COVID-19. Two of the identified apps, “COVID Coach” and “Managing Your Stress & Anxiety”, were specifically developed for COVID-19 by a healthcare institution and a governmental organization, respectively. These apps were ranked the top 2 for information quality. However, they did not score as well in the other quality parameters, probably because these were developed in rapid response during the pandemic. On the other hand, even though 14 other apps had incorporated content related to COVID-19 since the pandemic, only Happify managed to rank among the top 3 apps overall.

The apps in our evaluations were generally inconsistent across the different quality parameters. For example, the “Managing Your Stress & Anxiety” app had relatively high functionality, aesthetics, and information quality scores but received a low engagement score. Sanvello was the only app that consistently scored above 4 for all objective categories. The wide range of scores across the different quality parameters highlight the need for evaluation of apps and to exercise discernment when choosing such apps. We encourage developers to consider targeting all 4 categories of engagement, functionality, aesthetics and information in order to develop relevant, user-friendly and appropriate apps, particularly for public health crises, such as the COVID-19 pandemic.

In general, the apps scored the highest for functionality and the lowest for engagement and information. Our results were similar to another study that reported that their evaluated apps also scored badly in engagement (Shang et al., 2019). The low engagement scores of mental health apps could potentially correlate with poor user retention (Bauer et al., 2020). The top app for engagement in our study was Happify, which unlike the other apps in our evaluation, used gamification as a means of interaction with users. Previous studies had shown that the use of serious games and gamification could improve engagement, leading to better health-related outcomes (Fleming et al., 2016; Pine et al., 2020). Developers could consider the use of gamification to further engage and retain users of their apps.

In addition, the low information quality scores of our evaluated apps concurred with another study reporting that only 3% of mental health apps claiming effectiveness were backed by research, 30% had expert development input and only 20% were affiliated with either a government, academic or medical institution (Marshall et al., 2019). Due to the low entry cost for app developers, many of them do not have the necessary funds to test the efficacy of their apps in clinical trials (Bauer et al., 2020). Furthermore, in the current climate, timely solutions are needed to address the stress, anxiety and depression concerns brought about by the pandemic. Among all the objective quality criteria in the MARS tool (engagement, functionality, aesthetics and information categories), the fundamental parameters in the information quality category (i.e. the accuracy/ correctness, relevance/ appropriateness and comprehensiveness of the content provided; whether the app comes from a credible/ legitimate source; and the evidence base of the app in published literature) should be the most important when determining the quality of mobile health apps, such as the stress, anxiety and depression apps in this study. As a result of COVID-19, many organizations are beginning to realize the need for evidence-based models to assess mobile health apps, such as Health Technology Assessment frameworks (Haverinen et al., 2019; NHS Digital, 2020), in order to enhance the safety and efficacy of these applications. Developers should be cognizant of the fact that as the healthcare sector continues to embrace digital technologies post-COVID, the development and promotion of apps need to go hand-in-hand with an appropriate evidence base in order to substantiate their claims of effectiveness for health-related outcomes.

There was no correlation between the overall app quality scores and the app store ratings among the apps evaluated in our study. Our results could be due to the subjectivity and the lack of reliability of the star ratings in the app stores (Kuehnhausen and Frost, 2013). Furthermore, there were inconsistencies between the iOS and Android app versions, such as broken links and variations in app store descriptions of one version compared to the other. For example, the “Self-help for Anxiety Management” app had a privacy policy on the Android app store but not on the iOS app store. In addition, our findings indicated that the MARS subjective quality scores were strongly influenced by the app's quality of engagement and least influenced by the quality of app information. Since app user ratings are not meant to be an indicator of the safety, effectiveness or ability of the apps to provide acceptable medical care (Powell et al., 2016), patient and clinician users of such apps need to be aware that they should not solely rely on the app store ratings when selecting an app to support the stress, anxiety and/or depression experienced during and post-COVID. It is also important to be aware of the differences between the different app versions when recommending or selecting an app.

Data privacy and security is also a concern to users of mental health apps. Given the sensitive nature of personal mental health data, the lack of privacy and security can potentially be a barrier to adoption (Robillard et al., 2019; Thornicroft, 2008). The issues identified in our study are not new – these include the large proportion of apps sharing personal data with third parties despite only a minority being compliant with data protection standards. Moreover, a large majority of apps did not have privacy policies catered towards the average reading ability of adults – defined as a reading grade level of at least eight (Das et al., 2018). Although the onus is on users to read the privacy policy of apps before use, they may not be able to make well-informed choices regarding the handling of their personal data (Powell et al., 2018). Hence, developers are encouraged to comply with relevant data protection guidelines, such as the General Data Protection Regulation (Health On the Net, 2020), to improve the privacy and security of their apps.

Lastly, a large proportion of our evaluated apps did not provide a disclaimer that the apps should not replace the medical advice of healthcare professionals. In fact, mental health apps often conveyed that mental health problems could be easily managed with apps (Parker et al., 2018). The unfounded claims by some health apps, combined with the lack of such a disclaimer, could potentially mislead people with severe mental health issues to rely solely on apps and forego necessary professional treatment. The increased vulnerability to mental health problems during COVID-19 has led to an increased demand and supply of mental health apps to help people cope with stress, anxiety and depression during the pandemic. Developers are encouraged to adopt appropriate evidence-based recommendations, such as the ones proposed by Bakker and colleagues, to further enhance their apps so that better designed apps can be used by the public to combat the stress, anxiety and depression that they experience currently during the pandemic and in the “new normal” post-COVID.

Limitations and future research

The main limitation was that only apps that were available from the Singapore Apple and Google Play stores were assessed. Additionally, this study only evaluated apps that were free, and not paid apps. Furthermore, there is currently no quality evaluation tool that is designed specifically to evaluate COVID-related stress, anxiety and depression apps. The MARS tool was developed as a tool to assess the overall quality of health apps but did not account for the data privacy and security of apps and the quality of the stress-, anxiety- or depression-specific features. We circumvented the data privacy and security issue by adopting from the American Psychiatric Association app evaluation model. However, we could not find any validated quality assessment tool/framework for evaluating pandemic-related stress, anxiety and depression. Future quality evaluations could be extended to the app stores of other countries, paid apps, as well as assess specific features related to stress, anxiety and depression experienced during a pandemic. Furthermore, the results and recommendations provided by this study are time sensitive as apps are constantly added and removed from the app stores and existing apps undergo revisions. Therefore, apps should be evaluated regularly and by experienced evaluators to account for these changes.

Conclusion

This study evaluated apps that addressed the stress, anxiety and depression related to the COVID-19 pandemic. While the top ranked apps (Sanvello, Woebot, Happify, Youper, Bloom) were of good quality, majority were only of acceptable quality and had room for improvement. Apps that addressed all three mental health conditions were found to have the highest mean overall quality score. Healthcare professional and public users of such apps during the current COVID-19 pandemic can use the findings of this study to understand which areas the apps are currently lacking in. Developers are also encouraged to use these findings to improve and develop better quality apps.

CRediT authorship contribution statement

Lauren Su En Li: Formal analysis. Li Lian Wong: Conceptualization, Visualization. Kevin Yi-Lwern Yap: Conceptualization, Visualization, Writing – review & editing.

Declaration of Competing Interest

No competing financial interests exist. All authors have no conflicts of interest with any of the evaluated apps or their companies in this study.