Quality of Smartphone Apps Related to Panic Disorder.

Quality of smartphone apps related to panic: smartphone apps have a growing role in health care. This study assessed the quality of English-language apps for panic disorder (PD) and compared paid and free apps. Keywords related to PD were entered into the Google Play Store search engine. Apps were assessed using the following quality indicators: accountability, interactivity, self-help score (the potential of smartphone apps to help users in daily life), and evidence-based content quality. The Brief DISCERN score and the criteria of the "Health on the Net" label were also used as content quality indicators as well as the number of downloads. Of 247 apps identified, 52 met all inclusion criteria. The content quality and self-help scores of these PD apps were poor. None of the assessed indicators were associated with payment status or number of downloads. Multiple linear regressions showed that the Brief DISCERN score significantly predicted the content quality and self-help scores. Poor content quality and self-help scores of PD smartphone apps highlight the gap between their technological potential and the overall quality of available products.

Introduction

Panic disorder (PD) is a common anxiety disorder associated with an important social and economic burden (1, 2). Available treatments include pharmacotherapy and cognitive–behavioral therapy (CBT) (3). Such treatments have, however, been insufficiently disseminated in clinical settings (4).

Smartphones are widely used worldwide (5, 6) and have a growing role in health care (7–10). Concerns about the regulation of medical smartphone apps are, at the same time, rising among public administrations and the scientific community (9, 11, 12). The U.S. Food and Drug Administration regulates some health-related apps, as it does medical devices. Smartphone apps related to psychiatric conditions do not yet fall under this regulation (13).

A number of recent studies have assessed the quality of medically oriented apps in various fields, such as smoking cessation, weight management, sleep, cancer, and diabetes (14–36). While acknowledging the potential opportunity offered by apps-related technologies, these studies concluded that the apps available from different stores, with few exceptions, were of overall poor quality. A gap was furthermore found between the considerable number of apps related to medical conditions available in stores and the low number of peer-reviewed papers about them (37). In particular, despite their potential to improve health care, mental health apps currently available in stores lack scientific evidence about their efficacy (38). With few exceptions (39–41), preliminary findings reported for health apps were similar to previous findings on the poor quality of health information websites (42–46).

One may hypothesize that persons with PD could benefit from the development of well-conceived apps. Indeed, Internet-based CBT has previously been shown to offer some efficacy in PD (47–50), and several studies are under way to evaluate apps designed specifically for the treatment of PD (51, 52). Nonetheless, developers have not waited for scientific evidence to create apps for PD, as many are already available on smartphone stores. To our knowledge, no studies have yet been performed to rate these apps.

In the present study, we aimed to assess the quality of English-language PD-related apps available on the Google Play Store like any layperson searching for an app related to PD on the Google Play Store. It is a descriptive and exploratory study of what it is possible to find. Furthermore, the study aimed to compare free and paid apps. We furthermore assessed the factors associated with the main quality indicators, as well as the links between the quality indicators and users’ ratings (star ratings, as reported on the Google Play page) and downloads.

Materials and Methods

Selection of apps

A keyword search was performed between February and March 2014 to produce a comprehensive list of PD-related apps that were accessible in English on the Google Play Store. The Google play account was set to English United Kingdom language and linked to a mobile phone, which was registered on a Swiss mobile network.

Google is the developer of the Android operating system, the most widely used smartphone operating system in the world (53). The following queries were entered into the Google Play search engine: “stop panic,” “stop panic attack,” “panic attack,” “PD,” “anxiety attacks,” and “anxiety disorder.”

Studies of Internet users have shown that most people rarely search beyond the first 20 retrieved results (54). However, we extended the coverage of the present study to the first 50 free apps and to the first 30 paid apps for each tag to obtain the most comprehensive list of apps. Apps were included if they were related to PD. Exclusion criteria were as follows: the app could not be downloaded after more than three attempts, the app was not in English, or the app was a book or an article.

Evaluation of apps

Apps were reviewed on an HTC One Android 4.3. They were assessed by using tools reported in previous studies, tools adapted from quality evaluation studies of websites (55–59), and tools described in other studies on the quality of smartphone apps (15, 16, 23, 30, 31). The assessment instruments are described below.

Google Play’s Page and Functionalities of Apps

As reported by other investigators (15, 30), we extracted a number of items from the Google Play page, such as number of downloads and ratings of the apps.

Self-help Model

A self-help model assessment tool for PD was used (Table 1). The model was based on the potential of smartphone apps to help users in daily life and on the second edition of the Practice Guideline for the Treatment of Patients with Panic Disorder by the American Psychiatric Association (3).

Table 1

The self-help model.

General items, quoted as follows
Automated feedback	0 = absence of feedback	1 = non-tailored feedback	2 = tailored feedback
Biofeedback	0 = absent	1 = present
Personal statistics	0 = absent	1 = present
Promotion of non-evidence-based features	0 = present	1 = absent
Items usually present in panic-focused CBT, quoted as follows
	0 = absent	1 = present, only informative	2 = present with interactive help
Psychoeducation
Self-monitoring
Cognitive restructuring
Exposure to fear cues
Modification of anxiety-maintaining behaviors
Relapse prevention
Feature to help the user get through a panic attack

Content Quality

As in other studies on Internet websites related to mental health disorders (39–42), evidence-based content quality was assessed according to the availability of information related to the following questions that a patient could search for:

How do I know whether I have PD?

How do I know whether I have PD with or without agoraphobia?

Can I estimate the severity of my disease?

What are the effective treatments?

What are the various useful psychotropic drugs for PD, and what are their side effects?

What difficulties might I encounter during or after treatment?

What psychotherapies are effective in the treatment of PD?

Answers found on the apps were assessed on the basis of the American Psychiatric Association practice guideline (3). For every request, the coverage (the extent to which the question was addressed) and correctness (the extent to which the answer was right) of the answer were comprehensively scored on a 3-point scale (0 = absent; 1 = partially incorrect or incomplete; 2 = correct and complete). A total content quality score, ranging from 0 to 14, was calculated by combining the scores.

Interactivity

Interactivity (Table 2) was measured with an adaptation of the Abbott scale (58). Three items were added to the scale: presence of a gamification module, possibility of personalizing the user’s profile (avatar, color, sound), and tailoring of the app upon use.

Table 2

Assessment scales.

Scale		Coding
Interactivity, Abbott’s scale		0 = absent; 1 = present
1	Internal search engine	0 1
2	The presence of audio or video support	0 1
3	Questionnaire of satisfaction	0 1
4	Possibility of sending complaints and requests to the webmasters or to the authors?	0 1
5	Presence of forums, chat, or social networking?	0 1
6	Presence of a “game-like” module?	0 1
7	Tailoring of app	0 1
8	Personalization of app/user’s profile (avatar, color, sound)?	0 1
HON		0 = absent; 1 = present
1	Authoritative	0 1
2	Complementarity	0 1
3	Privacy	0 1
4	Attribution	0 1
5	Justifiability	0 1
6	Transparency	0 1
7	Financial disclosure	0 1
8	Advertising policy	0 1
Brief DISCERN		1 = no, yes = 5, in between = partially
1	Is it clear what sources of information were used to compile the publication?	1–5
2	Is it clear when the information used or reported in the publication was produced?	1–5
3	Does it describe how each treatment works?	1–5
4	Does the publication describe the benefits of each treatment?	1–5
5	Does it describe the risks of each treatment?	1–5
6	Does it describe how the treatment choices affect overall quality of life?	1–5
Silberg scale for accountability		0 = absent; 1 = present
1	Name of the author and qualification	0 1
2	Affiliation	0 1
3	Sources and references	0 1
4	Property of site	0 1
5	Sponsorship	0 1
6	Advertising	0 1
7	Date of creation	0 1
8	Date of last update	0 1
9	Updated within last 6 months	0 1

Health on the Net Code

The health on the net code (HON) label (55) was created for websites that focus on ethical standards in online publishing. Usually, a website requests evaluation, after which the label is awarded. In the absence of the common use of this label by the apps, we assessed whether they respected the HON criteria (Table 2).

Brief DISCERN

The Brief DISCERN (56) is a six-item (Table 2) assessment tool adapted from the DISCERN instrument (60). It is used as a potential indicator to estimate the quality of the information about the choice of treatment in websites. The Brief DISCERN includes six items on a five-point scale (1 = not at all; 5 = completely). The first two items identify the transparency of information sources; the other four items estimate the quality of the information regarding treatment. A cutoff score of ≥16 has previously been associated with good content quality scores of health-related websites (56).

Accountability

Accountability was estimated with the Silberg scale (57), which includes authorship (names of authors, affiliation, and references), attribution (sources and references), disclosure (property of site, sponsorship, and advertising), and currency (date of creation, modification of site, and updating in the last 6 months). A total score ranging from 0 to 9 (1 point for each item if present) was calculated for each app (Table 2).

Statistical analyses

Statistical analyses were performed with SPSS software (version 18.0, Chicago, IL, USA). An initial exploratory analysis involved the calculation of proportions, as well as means and SDs, of the above-mentioned outcome measures. Next, we compared paid apps with free apps in bivariate analyses by using parametric tests (t-test, chi-square, or Fisher’s exact test) or non-parametric tests (median test) when appropriate. Finally, we computed prediction models by using multiple linear regressions for two variables of interest.

Applying the principle of model parsimony and statistical relevance, the selection of the independent variables was driven by the objective to find the simplest model (i.e., contain a small number of variables):

that adequately fits the data and

that explains most of the variance in the dependent variable (the highest value of R2).

Beforehand, the relevance of the independent variables for the prediction of each dependent variable was discussed among the authors. Taking into account multicollinearity and using the «Enter»method, several subsets of these independent variables were regressed and the model with the least number of variables that still explain a percentage of variance in the dependent variable that is comparable to the percentage explained with all the variables in the equation was retained.

On the one hand, the prediction of content quality was fitted by using the Abbot interactivity scale, the Brief DISCERN score, the number of installs, and the link to paid content (yes vs. no) as the independent variables, and by controlling for payment status (free apps vs. paid apps). On the other hand, the prediction of self-help was assessed with the Brief DISCERN score, whether the app was recommended (yes vs. no), the link to paid content (yes vs. no), and the Silberg accountability scale as the independent variables, controlling for payment status (free apps vs. paid apps). For all analyses, a significance level of p ≤ 0.05 was used.

Payment status was hypothesized to matter as a potential confounder on the ground that it could moderate the impact of the independent variables on the dependent variable. Using the 10% change rule of thumb, confounding was considered present if the measures of association changed by 10% or more and absent otherwise. Measuring this change implied running the model twice: without and with the confounder.

Results

A total of 480 apps were found (50 free apps and 30 paid apps for each of the six keywords). The search with these keywords highlighted several duplicates among the apps identified. After their removal, of the 247 remaining unique apps, 52 were retained for analysis and 195 were excluded (Figure 1).

Figure 1

Study flow chart. PD, panic disorder.

One app offered only a self-assessment tool aiming to help user to screen for a possible PD. The other apps are designed to help user manage their symptoms (via information, assessments, and techniques to deal with PD). Most of the apps (58.1% of free apps and 61.9% of paid apps) recommend the user to consult a medical doctor if suffering from symptoms of PD.

There is furthermore a wide variability on the contents of the apps. For example, one of the most downloaded app offers features like psychoeducation, audio’s for relaxation, and mindfullness as well as a diary tool to record panic events. Some apps offer interactive modules to face panic attacks. For example, one app tells the user to control his breathing while giving instruction on the screen with pictograms. Other apps offer discussion forum. Some applications are selling various products, such as books or medicinal herbs.

The characteristics of paid and free unique apps are reported in Table 3.

Table 3

Selected characteristics of smartphone applications (apps) by status (free vs. paid).

Characteristics and scores	Free application (n = 31)	Paid application (n = 21)	p-Value
Play Store and app description
Category in Google Play Store
Book and reference	12.9	0	0.008
Health and fitness	67.7	42.9	0.008
Lifestyle	3.2	33.3	0.008
Medical	16.1	23.8	0.008
Mean rank (SD)	34.6 (10.0)	32.8 (12.5)	0.6
Mean star rating	2.2 (2.0)	2.2 (2.1)	1.0
Number of raters
None	38.7	47.6	0.3
Between 1 and 10	29.0	38.1
More than 10	32.3	14.3
App recommended?
No	29.0	38.1	0.5
Number of downloads
Less than 100	25.8	71.4	0.005
100–500	29.0	9.5
More than 500	45.2	19.0
Identity of the website developer?
No	38.7	14.3	0.06
Privacy policy?
No	83.9	85.7	1.0
Clear purpose of app?
No	12.9	4.8	0.6
Privacy authorization explanation?
No	90.3	100.0	0.3
Disclaimer?
No	87.1	100.0	0.1
Linked app?
No	90.3	90.3	–
Tutorial present?
No	93.5	90.5	1.0
User profiles?
No	87.1	90.5	1.0
Password protection?
No	80.6	90.5	0.4
Backup?
No	96.8	100.0	1.0
Bug report?
No	87.1	71.4	0.2
Publicity within the app?
No	38.7	100.0	<0.0005
Link to paid content from the play store app’s page?
No	38.7	61.9	0.1
Disclaimer?
No	80.6	81.0	1.0
Privacy policy?
No	83.9	85.7	1.0
Help system?
No	87.1	66.7	0.1
App designed as a “stand-alone”?
No	71.0	23.8	0.001
App designed to be used with external expert assistance/supervision?
No	90.3	76.2	0.2
Advice to consult doctor?
No	41.9	38.1	0.8
Content conformed to description in Google Play Store?
No	6.5	4.8	1.0
Self-help scores (0-21)	3.9 (3.1)	3.1 (2.7)	0.4
Content quality scores (0-14)	2.5 (3.0)	1.2 (2.5)	0.1
Abbott interactivity scale (0-8)	1.6 (1.5)	1.7 (1.1)	0.9
HON
0–2 criteria filled	80.6	38.1	0.002
3–7 criteria filled	19.4	61.7
All 8 criteria filled	0	0
Brief DISCERN
< 16	64.5	85.7	0.1
≥ 16	35.3	14.3
Silberg accountability scale (0-9)	2.4 (1.7)	2.6 (1.3)	0.6

Summary statistics report variable mean and standard deviation (SD), or percentage, as appropriate.

The two subgroups, paid and free apps, were similar, although we found several differences. For instance, the variable publicity appeared to be present in 61.3% of the free apps group but in none of the paid apps group, with a p-value of <0.0005. In addition, the HON criteria were more fulfilled in the paid apps group than in the free apps group (p-value 0.002). As shown in Table 3, the overall adherence to HON criteria was low: none of the assessed apps fulfilled the eight criteria. Significantly more apps that were designed to be stand-alone (no additional app content) were found in the paid apps group than in the free apps group (p-value 0.001). Both paid and free apps groups had low content quality and low self-help scores, with no differences between the two groups.

Regression results

Content quality was regressed on a set of independent variables, namely the Abbot interactivity scale, the Brief DISCERN score, the number of downloads, the link to paid content controlling for payment status. We found that the Abbot interactivity scale and the Brief DISCERN score significantly predicted the content quality (p = 0.01 and p < 0.0005, respectively), but not the number of downloads (p = 0.9), or the link to paid content (p = 0.1). After careful examination of the regression coefficients without and with payment status, no confounding effect could be imputed to this variable. The full model performed well with an adjusted R2 of about 70%.

Another regression model predicted the self-help score with the Silberg accountability scale, the Brief DISCERN score, whether the app was recommended, and the link to paid content as the independent variables, controlling for payment status. This model performed less well than the preceding model, as shown by an adjusted R2 of 54.4%. We found that the Silberg accountability scale (p < 0.0005) and the Brief DISCERN score (p = 0.03) significantly predicted the self-help score, but not whether the app was recommended (p = 0.4), or the link to paid content (p = 0.5). After due consideration of the regression coefficients without and with payment status, we did not detect a confounding effect of this variable.

Discussion

In this study, we aimed to assess the quality of English-language smartphone apps for PD. In particular, we evaluated the content quality and self-help scores with instruments that were adapted from other studies on the content quality of medical websites and from previous assessment studies on medical smartphone apps.

In consideration of the lack of specific studies with similar purpose to our study, and of the potentially high interactivity of apps in daily life, we adapted some assessment tools for the study herein. Abbott’s scale of interactivity (58) was adapted to match the specificity of smartphone apps, as described in the Section “Materials and Methods.” Another important adaptation was the self-help tool. It may be a helpful indicator for apps assessment and development, especially for those based on CBT treatment models. The low scores obtained with the tool may underline the gap between the clinical potential offered by apps technology and its rather low level of clinical development. The self-help model was based on CBT for two reasons: first, its easy translation into eHealth, as shown by important developments related to Internet-based therapy for mental health disorders (48, 61–64); second, its validity for the treatment of PD (3). The self-help model score is probably also useful for other CBT treatment apps. Further studies on apps for mental health and about the instruments proposed here are warranted.

Similar to the results reported in other studies on health-related apps, the mean content quality (14–16, 19, 23, 30, 31, 34, 35) and self-help scores were low in our study. Most apps were insufficiently evidence based; furthermore, the technological capabilities were underused in most of the available PD-related apps. As shown in website studies (40, 41, 56), the Brief DISCERN score (56) is linked to content quality scores, as well as to the self-help score specifically developed for CBT-based app assessment. In the present study, measures such as accountability and interactivity were associated with the main quality indicators, such as content quality and self-help scores, as was previously found in some (39, 41, 65), but not all (66), studies on health-related websites.

Factors related to the community success of a given app, such as the number of downloads and whether the app was recommended, as well as factors linked to the economic model, such as payment status or a link to paid content, were not associated with content quality or self-help scores. This is somewhat surprising, particularly in regard to the number of downloads. One might expect better quality for the most downloaded apps. The results are possibly limited by the assessments of apps found only on the Google Play Store as well as by the small number of apps with a high amount of downloads (only three apps with more than 5000 downloads).

The number of active users (unavailable on the Google Play Store) would, however, probably be more informative for the sustained success of a given app after the initial download.

Payment status was not associated with the quality indicators assessed. Further studies may assess in more details the commercial strategy linked to the development model related to health-related apps.

The link found between payment status and publicity reflects some differences in the commercial model. Other aspects should, however, be included in further assessments (i.e., marketing strategies, interaction with users.).

Our study contains several limitations. We assessed only apps from the Google Play Store and not the Apple Appstore or others. This aspect limits the generalization of the study findings.

In addition, the keywords for the search used in this study might be different from those used by people with PD, and we may have missed PD-related apps on the Google Play Store. Furthermore, the results may differ depending on the country and the language setting of the Google Play store.

Nonetheless, the study suggests possible modifications to medical eHealth assessments through the proposal of adaptations to apps. Despite expectations about the potential of PD apps to improve treatments (51, 52), the apps available to users from stores to date need to be improved and to include more patterns of evidence-based information, more interactive assessments, such as ecological momentary assessments (67), and more self-help options.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.