Observational study of UK mobile health apps for COVID-19.

The COVID-19 pandemic has caused global disruption to society and their health-care systems. In the setting of COVID-19, organisations in the UK such as National Health Service (NHS) Digital, NHSX, and NHS Business Services Authority have emphasised the need for mobile technology in managing the situation. This technology focus has led to an increase in the mobile phone apps developed for COVID-19.2, 3 However, despite the increased enthusiasm for mobile health technologies during the COVID-19 pandemic, data for their wider adoption is currently scarce because of potential hurdles related to their design, usability, functionality, and security features. It is difficult to evaluate the effectiveness of COVID-19 apps because they have been implemented quickly to ensure they have a timely effect. We have carried out an observational study to evaluate the features of mobile phone apps released in response to the COVID-19 pandemic.

We searched the App Store (Apple, California, USA) and Google Play (Google, California, USA) platforms on April 16, 2020 using the search terms “coronavirus”, “COVID 19”, “COVID-19”, and “novel coronavirus”. We identified 82 apps from 35 different countries. Of these 82 apps, 32 were within our geographical range (UK) and in the English language (appendix 1).

Apps were evaluated using the Systems Wide Analysis of mobile health-related technologies (SWAT) tool in line with the NHS Digital Assessment Questionnaire;4, 5 and were given a score for each category (usability, functionality, ethical values, security and privacy, user-perceived value, design, and content) by two independent assessors (table ). Scores are given as medians (interquartile ranges). Intraclass correlation coefficients were calculated to assess reliability between assessors using a two-way mixed effects model for total scale and subscale scores. These scores were interpreted within the definitions in Koo et al. All analyses were done using Microsoft Excel (version 16·0) and SPSS (version 26) (appendix 2).

Table

Critical appraisal of COVID-19 mobile phone apps

	Median (IQR)
Usability
Ease of use	1 (1–1)
Operability	1 (1–0·75)
Visibility of system status	0 (0–0)
User control and freedom	0 (1–0)
Consistency and standards	1 (1–1)
Error prevention	1 (1–1)
Completeness	0·5 (1–0)
Information needs	1 (1–0)
Flexibility and customisability	0 (0·25–0)
Competency	1 (1–0)
Style	1 (1–0)
Behaviour	0·5 (1–0)
Structure	1 (1–0)
Total	7 (9–6)
Functionality
Performance	1 (1–1)
Health warnings	1 (1–0)
Feedback	0 (1–0)
Connectivity and interoperability	1 (1–0)
Record	0 (1–0)
Display	1 (1–0)
Guide	1 (1–0)
Reminder or alert	0 (1–0)
Communication	0 (1–0)
Total	5 (6·25–4)
Ethical issues
Beneficence	1 (1–1)
Non-maleficence	1 (1–1)
Autonomy	0 (1–0)
Justice	0 (1–0)
Legal obligation	1 (1–0)
Total	3 (4–2)
Security and privacy
Security	1 (1–1)
Privacy	1 (1–0·75)
Total	2 (2–1·75)
User perception
Overall value	1 (1–0)
Design
Suitability of design	1 (1–1)
Aesthetic	1 (1–1)
Appearance	1 (1–1)
Design consistency	1 (1–0·75)
Total	4 (4–3)
Information and content
Credibility	1 (1–0)
Accuracy	1 (1–1)
Quality of information	1 (1–0)
Quantity of information	0 (1–0)
Total	2·5 (4–2)

Apps evaluated using SWAT scores (0–1) for each criterion. The totals are the overall median for the category.

Critical appraisal showed a total median SWAT score of 25·5 (range 9–32, interquartile range 18·75–28·25). The highest scores were achieved by apps from health-care organisations, such as the NHS and WHO. Specifically, the highest scores were achieved by the NHS24: COVID-19 app (scored 33 out of a possible 38) and the COVID Symptom Study app (scored 32 out of a possible 38). Scores for design were relatively high across most apps (median 4, IQR 3–4), with 78% of apps achieving a score of 3 or more. Median score for usability was 7 (IQR 6–9) and for functionality was 5 (IQR 4–6·25). Highest scores were attained by 75% of the apps (median 2, IQR 1·75–2) for data security and privacy. Statistical analyses using intra-class correlation revealed a high degree of agreement between raters (Cronbach's α=0·955, p<0·01).

Mobile phone apps can be used for information provision, contact tracing, and diagnostic purposes. In our analysis, 38 (50%) of the apps were aimed at providing information; 10 (13·2%) for contact tracing and seven (9·2%) as diagnostic tools. We have critically appraised mobile phone apps for COVID-19 and found that there are many high-quality apps developed for the features highlighted in this study, but their effect is limited by a low number of people using the app. We identified that the use of mobile apps for health care and public policy raises ethical issues surrounding data protection, and health-care regulators need to implement robust assessments to reassure the public of their safety and privacy. The balance between the effectiveness and the data privacy of a fully anonymised contact-tracing app raises practical challenges in implementing the technology.

To our knowledge, this is the first time that COVID-19 mobile phone apps have been evaluated using tools that follow the NHS Digital Assessment Questionnaire. Most apps had high scores for usability, functionality, design, and information provision. We also noted that apps verified by government organisations (eg, the Government Technological Agency in Singapore) had higher user uptake than unverified apps. For example, the NHS24: COVID-19 app, which scored highest in our assessment, had approximately 2 million users (3% of the UK population), but the NHS app, which scores low on our quality assessment, had a two-fold increase in new registrations from 56 655 to 119 512 between February and March, 2020. Similarly, the TraceTogether App, developed by the Singapore Government, had over 1 million users (20% of the population). The NHS has launched a programme to develop contact tracing apps and are targeting recruitment of 80% of the UK population. Given that the effectiveness of a contact tracing app relies on a high level of user uptake, public policy incorporate government effort into the fast-tracking assessment and approval of COVID-19 related apps, and in increasing the public awareness of NHS approved apps. Apps developed to serve different purposes (including contact tracing, risk stratification, and information provision) would not require similar levels of uptake; however, large sample sizes would increase the representativeness of any sample included within any app that is designed for research purposes.

The practical challenges of using mobile apps for contact tracing include the possibility of false data inputting and the practicality of location tracking to map epidemiology. An app can overcome these challenges if aligned with the SWAT scoring system that we have proposed. The app should have a navigable user interface and strict commitment to data security. Instead of continuously tracking user location, apps could rely on Bluetooth technology that the user can activate if tested positive through a peer–peer sharing function. Positive test results could also be inputted by health-care professionals to prevent erroneous data. The app should be able to communicate with other similar apps and collate data that can be shared with governments (after providing informed consent) to guide public health policies. Furthermore, when apps were used for COVID-19 contact tracing in other countries (eg, in China and Singapore), the main deterrents to app uptake were issues related to data protection, so a successful mobile phone app has to ensure users that data is handled securely and ethically. We reviewed the privacy statements of apps based in the UK, confirmed appropriate handling of personal information, and incorporated these into the SWAT scoring system. Specifically, apps based in the UK clearly asked for permission to collect, store, and share anonymised data without personal identifiable information. Users were also given details for withdrawing any of their information from the app at any point. These personal data features showed a high level of compliance with General Data Protection Regulation legislation that meet the standards set out by NHS Digital.

Limitations of our study include a bias towards UK-based apps in the English language. Furthermore, apps might rely on passively acquired data, which can be of variable quality, and our study did not specifically factor in the differences in data collection methods used by the apps. The quality of the app's data collection would not be assessible before instigating a post-hoc analysis of the data outputs. Our scoring system also uses broad criteria for apps of diverse purposes, which introduces an element of heterogeneity—eg, some of the apps we have included are aimed at contact tracing, whereas other apps are sources of information. The technology required for developing a contact tracing app that needs location tracking is more complicated and creates complex ethical dilemmas. Contract tracing apps need stricter policing and monitoring compared with apps that rely on daily updating of information and present less challenging ethical issues. Using the same tool to assess each type of app would reduce the generalisability of our results and can be overcome by introducing different criterion under each category of our scoring system.

Future studies should focus on developing formal assessment criteria that examines the quality, effectiveness, and validity of apps. Public health policies should aim to develop a more organised infrastructure and cooperation between local institutions (eg, NHS digital), international organisations (eg, WHO), and platform manager. Currently, digital solutions lag behind social distancing, mass testing, randomised clinical trials on treatment and vaccine research, and care delivery in research priorities. The utility of mobile phone apps is speculative without much clinical evidence at present. Further effort is required from governmental organisations to promote the development of research and increase public awareness of digital solutions. Therefore, although mobile technologies have huge potential to influence the course of the COVID-19 pandemic, major challenges such as data governance, quality, and efficacy must be addressed, and this will require a substantial international collaborative effort.