The function and quality of individual epidemic prevention and control apps during the COVID-19 pandemic: A systematic review of Chinese apps.

BACKGROUND: Coronavirus disease 2019 (COVID-19) has caused a global pandemic that has driven the widespread use of applications (apps) for outbreak management in China, but the characteristics and quality of these apps are currently unknown.
OBJECTIVE: The first objective of this study was to investigate the functional characteristics of individual epidemic prevention and control apps in China, and the second objective was to evaluate the quality of these apps.
METHODS: We searched the QimaiTM mobile application data analysis platform and the AladdinTM WeChat applet data analysis platform with keywords and quantified the search results based on the search index, relevance, and the Aladdin index to identify apps with high public popularity. The quality of the apps was rated by 2 independent raters using the Mobile App Rating Scale (MARS). The intraclass correlation coefficient (ICC) between raters was used as a measure of interrater reliability.
RESULTS: All 20 of the included apps had acceptable quality. Functionality had the highest score, followed by information quality, aesthetics, and engagement. There were no significant differences between the independent apps and WeChat applets in app quality (t = 1.907, p = 0.073) and subjective quality (t = 0.899, p = 0.381). These apps were related to COVID-19 individual prevention and control, and the functional features that contributed to the quality of the apps were grouped into six categories, i.e., health self-checking and reporting, news about COVID-19, scientific publicity and education, telemedicine services, personal travel inquiries, and digital contact tracing.
CONCLUSIONS: Individual COVID-19 prevention and control apps in China were developed by adding epidemic prevention and control functions to existing social apps rather than independently developing apps. The overall quality of such apps was acceptable, but scores in the engagement section were generally low, especially for WeChat applets.

Introduction

The worldwide spread of coronavirus disease 2019 (COVID-19) has resulted in significant morbidity and mortality. The COVID-19 pandemic has become one of the deadliest pandemics in the history of the world. As of March 18, 2021, more than 121 million people had been diagnosed with COVID-19 worldwide, and more than 2.68 million people had died [1]. The traditional medical system has had a substantial impact, and mobile health (mHealth) technology has some advantages in the prevention and control of the epidemic. mHealth refers to the monitoring and sharing of health information through mobile technologies, such as wearable devices and health-tracking applications (apps) [2]. Statistics have shown that as of January 2021, there were 4.66 billion active internet users worldwide, 92.6% (4.32 billion) of whom accessed the internet through mobile devices (e.g., smartphones, tablets), and global app downloads amounted to 218 billion [3], [4]. This suggests that the user base for mHealth technology during this major public safety emergency is broad.

Currently, more than 40 countries have adopted mHealth apps based on Bluetooth, global positioning system (GPS) tracking, quick response (QR) codes, and other technologies for epidemic prevention and control [5], [6], [7]. In Singapore, the government initially used a mobile app called Trace Together, which supported contact tracing and facilitated the quick identification and isolation of close contacts of COVID-19 patients in the early stage of the domestic outbreak [8]. Several German-speaking countries have developed a series of apps to facilitate government prevention and control of COVID-19 (Stopp Corona app by Austria's Red Cross, SwissCovid app by the Swiss government) [9]. The COVID Symptom Tracker app, jointly developed by the USA and the U.K., helps to monitor the COVID-19 transmission path, transmission speed, and susceptible populations in real time [10]. In China, apps with health QR codes are widely used and placed at entrances to public places, such as stations, airports, supermarkets, and communities, to record the locations visited by users [11]. Compared with traditional methods, apps can accelerate the tracking process because they do not rely on the memory of the infected person, enabling more contacts to be tracked and contacted. This automated digital approach provides the government with a convenient, low-cost, and easy-to-implement method [12].

However, general users could not judge whether the content provided by the mHealth app was correct [13]. The sudden outbreak and rapid spread of the epidemic, coupled with the lack of experience in epidemic control, resulted in a lack of sufficient analysis of demand for relevant prevention and control apps in the early stages of app development [14]. There is evidence during development that some mHealth apps never consult medical professionals for guidance or to verify that the information and services provided meet medical guidelines; consequently, trusting them may be harmful to people’s health [15], [16]. Therefore, it is necessary to conduct a systematic and comprehensive assessment of the quality of mHealth apps using appropriate rating tools. In 2013, Leanne's team developed the Mobile App Rating Scale (MARS), the first quality assessment tool for mHealth apps, which has been proven to have good internal consistency and interrater reliability [17], [18], [19]. The tool has been widely used to assess the quality of apps for chronic disease management [13], [20], [21], food allergies [22], sleep management [23], pain management [24], and blood pressure monitoring [25]. In contrast, relatively few studies have utilized the MARS to assess currently available COVID-19 apps. For example, the overall quality of these apps related to COVID-19 prevention and control in India, the USA, and the UK was inconsistent, with high functionality scores, low engagement scores, and app downloads unrelated to quality [19]. When evaluating COVID-19 contact tracking apps in Europe, it was found that these apps scored highly in terms of functionality, aesthetics, and information and that there was little difference among them. In addition, there is a direct link between app quality and adoption rates, which in turn were associated with lower infection rates [26].

During the COVID-19 outbreak, many emergency apps for prevention and control were developed in China. However, there is still a lack of empirical research to evaluate the quality of these apps. Therefore, the first objective of this study was to investigate the functional characteristics of individual epidemic prevention and control apps in China, and the second objective was to evaluate the quality of these apps.

Methods

Search strategy

Given the current situation of epidemic prevention and control app usage in China, a systematic search of the QimaiTM app data platform and the AladdinTM WeChat applet data platform from 21 to 25 January 2021 was conducted, following the PRISMA systematic literature review guidelines, and a checklist is provided in Multimedia Appendix A [27]. The apps were obtained as follows:

a. App Data Analysis Platform: The well-known domestic mobile app data analysis platform QimaiTM was used to retrieve apps related to COVID-19. The platform provides data related to all apps in the App Store, Google Play, and 9 major domestic Android markets (Huawei, Xiaomi, OPPO, Vivo, Meizu, Baidu, 360, App Store, and Pea Pod); the platform also provides tools for keyword expansion, app store optimization (ASO), and intelligent matching of association terms [28], which assisted in obtaining data in this study. The keyword expansion tool uses big data analysis to find additional keywords with relevance based on the keywords entered, and the platform also provides relevance ranking and search index values. The greater the relevance, the better the results of the keyword selection. Regarding the keyword search index, also known as search popularity, the larger the search index was, the more people searched for that particular term, and the greater the search popularity of a certain topic. In this study, we selected apps that met the requirements by selecting keywords with more than 50% relevance to the keyword “xin guan” (new crown) and combining them with a search index ≥ 4605 (usually, keywords with a search index ≥ 4605 have a large search volume). The search date was January 25, 2021.

b. WeChat Applet Data Analysis Platform: The WeChat applet is a mobile application developed by Tencent based on the WeChat platform that does not require downloading, installation, or registration; the applet is unique because of its development simplicity, ease of use, and direct accessibility to the public, becoming the “new favourite” of people affected by fast-paced, large public emergencies and security events [29]. We retrieved WeChat applets based on the Aladdin Index, which is an index developed by Beijing Aladdin Future Technology Co., Ltd., that specifically evaluates the influence of WeChat applets. This influence is calculated by daily activity, some shares, length of stay, depth of visit, and other WeChat applet indicators [30]. In this study, we applied the keywords “Yiqing” (epidemic), “Fangyi” (epidemic prevention), “Kangyi” (anti-epidemic), “Zhanyi” (war on epidemic), “Yikuang” (epidemic status), “Yifang” (epidemic prevention), “health punch card”, “isolation and prevention and control”, “joint prevention and control”, “new crown”, “new pneumonia”, “new coronavirus”, and “new prevention and control”. We searched all WeChat applets related to COVID-19 in the Aladdin applet list. The search date was January 21, 2021.

c. Supplementary Approach: Using snowball sampling, 20 undergraduates from a university and 10 employees from a public institution were selected as seeds, and their school or work unit asked them what kind of COVID-19 apps they had used in the past year. They were invited to recruit others through WeChat or QQ workgroups and their interpersonal network to identify COVID-19 apps that were widely used in their organization but were not identified by the above two search strategies. Feedback information was provided through WeChat and SMS messaging. The feedback period was from December 24, 2020, to January 24, 2021.

The app exclusion criteria were as follows: ① no health self-checking and reporting or tracking function; ② non-simplified Chinese language; ③ availability to only internal personnel of the organization (the internal account must be provided to assess the relevant function); and ④ advertising and product sales as the main purpose. The app screening process is shown in Fig. 1 (see Multimedia Appendix B for screening details).

Fig. 1

PRISMA flow diagram of the app search process.

App quality rating tool

This study was conducted to rate the quality of apps using the MARS, which consists of four parts. The first part (app classification) is used to collect descriptive and technical information in the App Store. The second part (app quality ratings) measures app quality considering four dimensions: section A (engagement), section B (functionality), section C (aesthetics), and section D (information), with a total of 19 items. The third part (subjective app quality) has 4 items in total. Finally, the app-specific portion includes a total of 6 items. These items are rated on a Likert scale (1 = inadequate, 2 = poor, 3 = acceptable, 4 = good, 5 = excellent), and an average MARS score ≥ 3 points (out of 5) is considered to be of “acceptable” quality [31].

Review process

The review process consisted of three steps. First, descriptive and technical information on the 20 apps was collected from the app stores (Apple Store & Android Market) and the WeChat applet platform (see Multimedia Appendix C). Second, a pre-assessment was conducted by three reviewers. Before conducting the assessment, three reviewers watched an instructional training video provided by the MARS developers and discussed and reached a consensus on the content of the questions [32]. Then, three apps and three WeChat applets were randomly selected for a review exercise and score discussion; the apps were reassessed by a third reviewer if any disagreements arose. Third, the quality of the 20 included apps was evaluated using MARS. All eligible apps were installed on smartphones (Android device: HUAWEI P30 Pro model VOG-AL00, version 10; iPhone device: iPhone 6S model A1700, version: 11.41). Then, to ensure consistency of the evaluation results, the evaluation of the sub-dimensions was completed simultaneously for all apps. The sub-dimensions of Section A for all apps were assessed first, followed by the sub-dimensions of Section B. The evaluation process was conducted in parallel, which means that the two reviewers independently rated and evaluated the same apps at the same time, and interrater reliability was measured.

Statistical analysis

Data were analysed using descriptive and analytical statistics. Quantitative variables are described using means and standard deviations (SDs). Categorical variables are described using frequencies and percentages. The intraclass correlation coefficient (ICC) between raters was used as a measure of interrater reliability for the MARS. The two-way mixed, average measures model with absolute agreement was used to estimate the reliability of the average measure between two raters [17]. Data collection and collation were completed using Excel 2016 (Microsoft Inc., Washington DC, USA), and data analysis was completed using IBM SPSS Statistics version 26 (IBM Inc., New York, USA).

Results

Overview

Among the 20 eligible apps, 11 independent apps were obtained from the App Store, and 9 applets were obtained from the WeChat platform. The characteristics of the apps are shown in Table 1 . Eleven independent apps were designed for both the Android and iOS platforms and could be downloaded for free, with the number of downloads and ratings provided by users. Nine of these independent apps had been downloaded more than 10 million times. However, 54.54% of these independent apps had a user rating of less than 3 stars, and only 3 apps in the Apple Store had a user rating of 4 stars or more. In addition, 11 of the included apps were developed by Chinese government departments, and the remaining 9 were developed by commercial organizations. The main target groups for these apps were the general population (70%), the student population (20%), and employees of enterprises and institutions (10%). Nineteen of the apps had been updated within the past year.

Table 1

Summary characteristics of the COVID-19 management apps (n = 20).

Characteristics of the apps	n (%)
Platform
Apple Store & Android Market	11 (55)
WeChat applet	9 (45)
Affiliations
Commercial	9 (45)
Government	11 (55)
Age group
General	14 (70)
Students	4 (20)
Enterprise and institution workers	2 (10)
Updated within the past year
Yes	19 (95)
No	1 (5)
Number of app downloadsa	(n = 11)
0–99,999	2 (18.18)
10 million-99,999,999	5 (45.45)
greater than1 billion	4 (36.37)
User star ratingb	Android(n = 11)	iOS (n = 11)
1–2.9	6 (54.54)	6 (54.54)
3–3.9	5 (45.46)	2 (18.18)
4–5	0 (0)	3 (27.28)

Number of app downloads: Information on the number of app downloads is available only for apps in the Android Market (n = 11).

User star rating: Information on the user star rating of the app was analysed separately for Android Market and Apple Store apps (n = 11).

APP focus/target

In this study, we comprehensively reviewed the features of these 20 apps related to COVID-19 personal prevention and control, and the functional features that contributed to the quality of the apps were grouped into six categories, namely, health self-checking and reporting, news about COVID-19, scientific publicity and education, telemedicine services, personal travel inquiries, and digital contact tracing (Table 2 ).

Table 2

Descriptions of personal COVID-19 prevention and control management functions of mobile apps.

Function type	Acronym	Detailed description	Corresponding app	n (%)
Health self-checking and reporting	HSR	Users are required to report health information, such as body temperature, relevant symptoms, exposure history, and main activity locations, online daily so authorities can accurately identify areas that may be affected in certain circumstances.	QFc, NGe, DTf, TCg, YSh, WEi, ACj, WCl, ALm, HBn, XAo, GZp, SIq, WHr, SKs, SCt, WTu.	17(85)
News about COVID-19	NC	Information on the current situation of COVID-19 is disseminated to the public in various forms, such as texts, sound bites, images, pictures, or data.	NGe, DTf, WSk, WCl, ALm, XAo, SIq, WHr.	8(40)
Scientific publicity and education	SPE	The app supports and provides educational information regarding scientific knowledge related to the prevention and control of COVID-19.	NGe, WSk, WCl, ALm, SIq, WHr, SCt, WTu.	8(40)
Telemedicine services	TS	A variety of health care services are available to users, including online medical consultations, virus detection result queries, vaccination information, and nucleic acid testing queries.	NGe, DTf, WSk, WCl, ALm, XAo, GZp, SIq, WHr, SCt.	10(50)
Personal travel inquiries	PTI	Users can inquire about their travel records within 14 days (including the day of visit) provided by the national authoritative administrative department (China Information and Communication Research Institute) and domestic telecom operators.	QFc, TXd, NGe, DTf, WSk, WCl, ALm, HBn, XAo, SIq, WHr, SKs, SCt, CSw.	14(70)
Digital contact tracing (DCT)	HQRC	Health QR Code (HQRC): The risk level of the epidemic is evaluated based on Big Data back-end analysis by analysing the information provided by the individual, which can be used for health personnel identification and as a “special pass” in applicable areas.	NGe, DTf, WSk, WCl, ALm, HBn, XAo, GZp, SIq, WHr, SKs, SCt, CSw.	13
	BLE	Bluetooth Low Energy (BLE): Bluetooth technology is less invasive regarding privacy than other positioning technologies because it only records which devices are in contact with each other and not their actual location [33]. Users are promptly notified and alerted when a stranger close to them has been diagnosed with confirmed/suspected COVID-19, resulting in timely isolation for observation and treatment.	QFc, TXd, NGe, DTf, WSk, WCl, ALm, HBn, XAo, SIq, WHr, SKs, SCt.	13
	GPS	Global Positioning System (GPS): tracking can accurately identify the user's location, which can help the user and authorities identify close contacts; however, it is invasive regarding user privacy and can be rejected by the user.	QFc.	1

QF: Quanmin Fangyi (TWS) app

TX: Tongxin Xingcheng Ka app

NG: National Government Service Platform app

DT: DingTalk app

TC: TODAY Campus (teacher edition) app

YS: YiSchool app

WE: Wecom app

AC: Ai Class app

WS: Wanshitong app

WC: WeChat app

AL: Alipay app

HB: Hebei Health Code applet

XA: Xi'an Citizen One-Code Pass applet

GZ: Guizhou Health Code applet

SI: Shen I Nin-Zizhu Shenbao applet

WH: Wuhan Zhanyi applet

SK: Sui Kang applet

SC: Sichuan Tianfu Health Code applet

WT: Wanzhong Yixin Tongzhou Gongji applet

CS: Churu Suji applet.

App quality assessment

All 20 apps included in this study had acceptable quality (defined by a MARS score ≥ 3) (described in Table 3, Table 4 ). In the four dimensions of app quality ratings (Section A, Section B, Section C, and Section D), the highest scores for all 3 platforms (iOS, Android, and WeChat applets) were in the functional dimension, followed by information quality, aesthetics, and engagement. Only 2 independent apps (WeChat and Alipay) achieved a score of 4 or more on both the overall quality and subjective scales, indicating high or excellent quality [31]. There were no significant differences in the 4 dimensions (Section A, B, C, D) of the app quality portion of the MARS and subjective quality scores between the iOS and Android platforms of the independent apps. In addition, there was no significant difference between the independent apps and WeChat applets regarding app quality scores (t = 1.240, p = 0.231) or subjective quality scores (t = 0.636, p = 0.533) (Table 5 ). The interrater reliability of the engagement section of the subscales, the app quality total scores, and the subjective quality scores were all excellent (Table 6 ). Detailed MARS scoring data are available in Multimedia Appendix D.

Table 3

Scores of each MARS section for the included apps.

	App quality ratings										Subjective quality
App name	Section A: Engagement		Section B: Functionality		Section C: Aesthetics		Section D: Information		App quality total score		Section E
	iOS	Android	iOS	Android	iOS	Android	iOS	Android	iOS	Android	iOS	Android
WeChat	4.60	4.60	4.50	4.50	4.00	4.00	4.33	4.33	4.36	4.36	5.00	5.00
Alipay	4.60	4.60	4.38	4.38	4.00	4.00	4.17	4.17	4.29	4.29	5.00	5.00
DingTalk	4.40	4.40	4.00	4.13	3.83	3.83	4.33	4.33	4.14	4.17	3.75	3.75
Wanshitong	3.80	3.80	4.13	4.13	3.83	3.83	4.20	4.20	3.99	3.99	4.13	4.13
Wecom	3.50	3.50	4.25	4.25	3.83	3.83	3.58	3.50	3.79	3.77	3.38	3.38
National Government Service Platform	2.60	2.60	3.50	3.50	4.00	4.00	4.20	4.20	3.58	3.58	4.50	4.50
TODAY Campus (teacher edition)	3.80	3.80	3.88	3.88	2.67	2.67	2.90	3.00	3.31	3.34	3.88	3.88
Ai Class	2.40	2.40	4.38	4.38	3.17	3.17	3.25	3.25	3.30	3.30	2.50	2.50
Quanmin Fangyi (TWS)	2.40	2.40	4.38	4.38	3.00	3.00	3.40	3.80	3.29	3.39	3.63	3.63
YiSchool	3.40	3.40	3.88	3.88	2.67	2.67	3.20	3.20	3.29	3.29	2.75	2.75
Tongxin Xingcheng Ka	1.70	1.70	4.50	4.50	2.50	2.50	4.17	4.17	3.22	3.22	4.63	4.63
t	0.000		−0.088		0.000		−0.173		0.289		0.000
p	1.000		0.930		1.000		0.865		0.775		1.000

p: Comparison between iOS and Android systems.

Table 4

Scores of each MARS section for the included WeChat applets.

Name	App quality ratings					Subjective quality
	Section A: Engagement	Section B: Functionality	Section C: Aesthetics	Section D: Information	App quality total score	Section E
Wuhan Zhanyi	3.70	4.50	3.67	4.00	3.97	4.75
Sichuan Tianfu Health Code	2.90	4.38	3.83	4.25	3.84	4.50
Shen I Nin-Zizhu Shenbao	2.90	4.38	4.00	4.00	3.82	4.00
Sui Kang	2.80	4.50	3.33	3.71	3.59	3.75
Hebei Health Code	2.30	4.25	3.50	3.71	3.44	4.00
Xi'an Citizen One-Code Pass	2.20	3.88	3.17	3.67	3.23	4.50
Guizhou Health Code	1.90	4.25	3.33	3.29	3.19	3.50
Wanzhong Yixin Tongzhou Gongji	3.60	4.25	2.00	2.92	3.19	1.00
Churu Suji	1.60	4.75	2.83	3.00	3.05	2.75

Table 5

Single-factor analysis of MARS quality.

MARS Quality	Mean	S.D.	t（F）	p
App quality			1.240	0.231
WeChat applets	3.48	0.34
Independent apps (Android)	3.70	0.43
App subjective quality			0.636	0.533
WeChat applets	3.64	1.16
Independent apps (Android)	3.92	0.84

Table 6

Descriptive results of MARS scores (N = 20).

Quality	Rater 1, mean	Rater 2, mean	ICC a (95% CI)
Engagement	3.226	3.116	0.906 (0.816–0.954)
Functionality	4.226	4.218	0.792 (0.610–0.894)
Aesthetics	3.43	3.323	0.782 (0.594–0.889)
Information	3.769	3.743	0.854 (0.719–0.927)
App quality	3.663	3.594	0.897 (0.787–0.950)
Subjective quality	3.927	3.75	0.916 (0.800–0.962)

ICC: Intraclass correlation coefficient.

Discussion

In this study, the functions and features of 20 personal prevention and management apps related to COVID-19 in China were classified and described in detail, and their quality was rated by the MARS. Health self-checking and reporting (17, 85%) and personal travel inquiry functions (14, 70%) were the main purposes of developing these apps to prevent the rapid spread of the novel coronavirus.

Application function overview

Mobile apps that support contact tracking are now considered an assistive technological tool that many governments around the world are utilizing to manage COVID-19, as these apps turn smartphones into tracking tools. In this study, 13 (65%) of the apps allowed the ability to access and apply health QR codes. During the early stages of the epidemic, the Chinese government used health QR codes as “electronic health cards” for Chinese residents for daily travel and as an effective epidemic prevention and control tool [34], [35], [36]. In China, different regional (provinces/municipalities) governments have already launched various apps with similar functions to prevent and control epidemics, enabling dynamic authentication of local residents' health status, with different colours (red, yellow, or green) indicating different levels of restrictions or regulations. This largely eliminates cumbersome reporting procedures, improves the efficiency of epidemic surveillance, and minimizes the possibility of transmission. However, there are some problems, as many provinces and cities have their own online health QR codes, and the health QR codes among the provinces/cities are not standardized; therefore, cross-provincial travel usually requires users to repeatedly register and enter their information on multiple platforms and reapply to obtain a regional health QR code. Some users who register to receive a local health QR code still need to isolate for 14 days, causing great hindrance to normal travel by the public.

In this study, there were 10 (50%) apps, including 5 independent apps (“National Government Service Platform”, “Wanshitong”, “DingTalk”, “WeChat”, “Alipay”) and 5 WeChat applets (“Xi'an Citizen One-Code Pass”, “Guizhou health code”, “Shen I Nin-Zizhu Shenbao”, “Wuhan Zhanyi”, “Sichuan Tianfu health code”) that provided various forms of healthcare services, such as online disease consultation, virus test result reporting, vaccination information, and nucleic acid testing facility information. There were 8 apps (40%), including the 5 independent apps and 3 applets (“Xi'an Citizen One-Code Pass”, “Shen I Nin-Zizhu Shenbao”, and “Wuhan Zhanyi”) mentioned above, that also provided news about COVID-19. These “noncontact” online services help users maintain social distancing and home isolation, relieving the pressure on a large number of offline hospitals and alleviating people's fear of uncertainty. In addition, it is worth noting that 4 of the apps and 4 applets had multiple functions, including consultation, education (epidemic status information and science education), contact tracking, and local epidemic risk information. Through the quality review of MARS, it was found that the above independent apps/WeChat applets, which can provide a wide range of functional services and links to external extensions, tended to be excellent in app quality and subjective quality, with 80% (8/10) scoring 4 and above. Therefore, we suggest that when developing and designing COVID-19-related apps, the above-mentioned features should be considered to help improve the quality of the app, thereby increasing the user acceptance and adoption rate.

Privacy protection is another key topic concerning COVID-19 prevention and control apps. The widespread use of digital contact tracking technology has aroused public concern about travel restrictions and the privacy and security of personal data. China has a top-down policy whereby managers of public places, such as communities or supermarkets, have the right to deny access to individuals who cannot provide a valid health QR code. In other words, individuals have little freedom of choice, even if they are unwilling to participate in digital contact tracking. A similar problem exists in some East Asian countries that use digital tracking technology [37]. Consequently, due to the severe situation of the epidemic, it should be a vital task for governments to protect the privacy and security of personal data and to explain the implementation of travel restrictions to improve public understanding and support. The Chinese government's approach was to widely publicize the importance of epidemic prevention and control, including the national efforts and personal sacrifices made by medical personnel in the early stages of the epidemic. This induced an emotional response among all citizens to come together to fight the epidemic and prompted all citizens to consciously prevent the spread of the epidemic via their actions. This strong ideological resonance and behavioural consensus are the fundamental reasons why the spread of the epidemic in China has been quickly and effectively curbed.

App quality rating

Currently, several tools have been developed to evaluate an app’s quality, such as the tool for evaluating drug use management apps [38] and the tool for evaluating the quality of healthcare-related apps [39]. In this study, we chose MARS as the app quality assessment tool, mainly for the following reasons. First, in the process of app screening, we found that most Chinese epidemic prevention and control apps were developed by adding epidemic prevention and control functions to existing social apps, such as Alipay or WeChat, rather than independently developing apps. MARS evaluation items have wider applicability [17]; thus, it is appropriate to use MARS to evaluate an app’s quality. Second, the MARS has been widely used in many studies with good reliability and validity [18], [19], [26]. In addition, video instructions are provided for the MARS on how to use the tool, which provided technical support for our study.

Based on the MARS rating results, in the 4 dimensions of app quality, 3 platforms (iOS, Android and WeChat applet) had the worst engagement scores. In particular, the engagement score of WeChat applet platforms was the lowest among them (mean = 2.66, SD = 0.72). The total quality (mean = 3.48, SD = 0.34), engagement, aesthetics (mean = 3.30, SD = 0.60) and subjective quality (mean = 3.64, SD = 1.16) of WeChat applets were all lower than those of independent apps. Additionally, the 4 independent apps and 7 WeChat applets developed by the Chinese government scored only 2.65 (SD = 0.67) in terms of engagement, indicating that even the government-developed COVID-19 apps had low user engagement. The majority of prevention-focused WeChat applets (77.78%) in this study were developed by local governments and built on a digital platform with high “traffic” (WeChat). Studies have shown that Chinese users encounter few operational barriers when using WeChat or Alipay, yet the installed epidemic prevention applets have become the main online tools due to their small size, low cost, ease of sharing, accessibility, and direct reach [40]. Through analysis of the scores of each item on the MARS subscale, we found that independent apps lacked some additional components or interesting videos or images to make them more attractive, which was the main reason for the low engagement score. WeChat applets cannot be customized according to user preferences (e.g., basic settings, such as sounds, notifications, and content), and the allowance of user input and provision of feedback functions are the main reasons for the low engagement scores. A study has shown that user engagement is very important, and there is a significant correlation between user engagement and increased app adoption [26]. Therefore, we suggest that developers should not only focus on meeting users' needs in all aspects of functionality, multi-dimensionality, and depth, as well as provide high-quality information on epidemic prevention and control, but also pay attention to app design in terms of entertainment, interest, interactivity, customization and other engagement features, which will help enhance user retention and achieve wider promotion.

Limitations

This study has certain limitations. First, the data collection period for this study was only from December 24, 2020, to January 25, 2021, which means that the results of this study provide only a cross-sectional view of a rapidly evolving app market. Second, some apps were excluded for reasons such as the requirement of an organization's internal registration code or inability to be opened due to the relatively old iOS platform app installation device (iphone 6 s). Third, Chinese is a profound, vast, and complicated language, and a word or phrase often has multiple meanings. An identical phrase can represent opposite meanings in different contexts. There are many Chinese words related to COVID-19, so the search terms we used were not exhaustive. In addition, our searches were limited to the Apple Store, Android Market, and WeChat app in China, which may have caused selection bias in the study sample.

Conclusions

By reviewing the existing apps on the Chinese App Market and WeChat platform, we found that apps with high “traffic”, such as WeChat, were utilized in COVID-19 prevention and control efforts; therefore, the Chinese government also moved its epidemic prevention apps to the WeChat platform, which has a large number of users. The overall quality of the 20 apps for COVID-19 personal control and management was acceptable. However, the engagement section scores were generally low, with the WeChat applets having the lowest scores. Most of these apps aim to provide comprehensive services to the public (e.g., disease consultations, information on the current status of epidemics, science promotion, travel trace inquiries, and epidemic risk warnings throughout the country). However, the growing usage of digital tracking tools raises the potential risk of personal data leaks. Therefore, we recommend that relevant departments not only urge software developers to standardize privacy protection policies for current epidemic prevention and control software, but also strictly manage the large amount of personal health data collected in the software, and regulate the trace management of the data use process to ensure the traceability of any data leakage incidents and risks.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.