Evaluation of mobile health applications for cervical cancer in the digital marketplace.

OBJECTIVE: To assess the quality of mobile health (mHealth) applications (apps) for cervical cancer using the mobile app rating scale (MARS), APPLICATIONS scoring system, and app rating using specific statements.
METHODS: We searched for cervical cancer apps on two major mobile operating systems (Google Play Store and Apple iTunes Store) in March 2021. Eligible apps were downloaded and assessed for quality by two independent reviewers using multimodal assessment tools.
RESULTS: The overall quality of the MARS score was 2.61±0.795. The highest scoring app was "The American Society for Colposcopy and Cervical Pathology (ASCCP) Management Guidelines" (3.98). Overall, apps scored highest in the functionality domain, followed by information, engagement, and aesthetics domains. The mean±standard deviation of the APPLICATIONS scoring system was 8.50±1.712. The highest-rated apps were "ASCCP Management Guidelines," "The British Society for Colposcopy and Cervical Pathology (BSCCP)," and "Cervical Cancer Guide." Apps scored the highest in the paid subscription and price domains. By contrast, apps scored poorly in the text search, literature, and subjective presentation domains. Concerning app content, many apps infrequently provided misconceptions regarding cervical cancer. The apps' rating using specific statements was 7.81±4.562.
CONCLUSION: Overall, the apps analyzed using the MARS and APPLICATIONS scoring systems demonstrated above-average quality. However, there is a need to improve the essential information conveyed by these applications. Moreover, the assessment tools have influenced different app quality rating results, confirming the lack of standardized quality assessment tools for mHealth apps.

Introduction

Cervical cancer remains a significant public health problem in low-resource settings [1]. It is the 2nd leading cause of cancer in women in Thailand, ranking after breast cancer [2]. The level of knowledge of cervical cancer is associated with the success of cervical cancer programs [3-5]. A recent nationwide social media survey in Thailand found that nearly half of Thai women have insufficient knowledge regarding cervical cancer, indicating the need for further health education intervention [6].

Mobile health (mHealth) applications (apps) have been acknowledged as effective tools for improving population-level health outcomes. In a recent systematic review undertaken to assess the use of mHealth apps in low- and middle-income countries, this health technology intervention improved various health outcomes. The most prominent health outcomes that improved with mHealth apps were communicable diseases and maternal health. These findings thus support the use of mHealth apps for health promotion in developing countries [7].

mHealth apps for cancer information are increasingly used as health education interventions for individuals. Many apps for cancer information provide general knowledge about the incidence, risk factors, signs and symptoms, diagnosis, treatment, and screening methods. Thus, they could facilitate cancer awareness in women and encourage participation in prevention and treatment programs [8-11]. Most healthcare providers have a positive attitude toward using oncological apps [11].

Significant numbers of gynecologic cancer apps, however, are not up-to-date and some of the contents are unreliable. In addition, almost all apps have no stakeholders involved in their development [12-15]. A previous study noted that only 1.5% (11 of 748) of gynecologic cancer apps were found to be both potentially helpful and accurate [16]. Owing to concerns about the quality of mHealth apps, the US Food and Drug Administration (FDA) has launched policies regarding the development of mHealth apps [17]. However, many apps are neither reviewed nor certified before being released to the public.

High-quality mHealth apps can serve as novel interventions to combat barriers that limit effective cervical cancer prevention and treatment by improving cancer awareness and changing behavior in healthy women and cancer survivors, such as enhancing the uptake of cervical cancer screening and increasing adherence to follow-up [18-21]. Several studies have described tools for assessing the quality of mHealth apps in cancer, such as the mobile app rating scale (MARS) [22-24], APPLICATIONS scoring system [16,25], and digital health scorecard approach [26]. A recent systematic review identified the standard domains of quality assessment criteria for mHealth apps and summarized 15 domains for researchers, developers, and users to assess the quality of the apps themselves [27]. Our study was conducted to assess the quality of cervical cancer apps using multimodal assessment tools (MARS, APPLICATIONS scoring system, and app rating by specific statements).

Materials and methods

1. Study design and setting

We performed a cross-sectional study to assess the quality of mHealth apps providing cervical cancer information available on two major mobile operating systems: Google Play Store (Android) and iTunes Store (iOS) [28]. Our study was conducted based on the five recommended steps for the quality assessment of mHealth apps [27].

2. Search strategy

Two authors searched mobile applications using the search bar on Google Play Store and iOS to identify cervical cancer-related apps. One author searched the Google Play Store, and the other searched iOS between March 1 and March 22, 2021. The following Medical Subject Heading (MeSH) terms appearing in MEDLINE/PubMed databases were used for the search: “cervical cancer” (mareng pak mdluk; cervical cancer), “cervix cancer,” “cervical tumor,” “cervix tumor,” “cervical mass,” “cervix mass,” “cervical malignancy,” “cervix malignancy,” “cervical neoplasia,” and “cervix neoplasia.” Any new apps identified by these MeSH term searches were added to the list of apps for each mobile operating system.

3. Selection criteria

After duplicate apps were manually removed, a complete list of apps from each mobile operating system was initially screened based on their title and description. Only apps related to cervical cancer available in English or Thai, free apps (no cost), and apps for a fee were included in the study. Apps were excluded if 1) they had no scientific content, 2) they were not specific to cervical cancer, or 3) they were non-oncologic apps, for example, horoscope apps. Apps that met the selection criteria were downloaded using either an ASUS ZenPhone Max Pro M1 mobile phone for the Google Play Store or an iPhone 6S mobile phone for the iTunes store for further assessment. Authors JK and NL assessed the accuracy of eligible apps to ensure that they were appropriate. Disagreements between the two reviewers were resolved through consensus-based discussion. The selection process is illustrated in Fig. 1.

Fig. 1

Flow diagram of the process utilized to identify cervical cancer apps for evaluation. Apps, applications; iOS, an operation system used for mobile devices manufactured by Apple Inc.

4. Data collection and app characteristics

Basic information on the apps, including the name of the app, name of the developer or seller, platform (Android or iOS), price, number of downloads (Android only), date of the last update, user rating, app store category, and language, was recorded using a Microsoft Excel spreadsheet (Office 365) for further analysis.

5. Apps evaluation

The included apps were independently reviewed by two authors (JK and NL) between April 5, 2021 and May 24, 2021. Each app was initially analyzed offline to determine function without internet connectivity. Apps that required payment for premium services were paid for to evaluate the full functionality of the app. Both reviewers used the apps for 7 days before performing quality assessment according to the recommended steps for research on the quality assessment of mHealth apps [27]. Each app was assessed using MARS [24], followed by the APPLICATIONS scoring system [16] and app ratings by specific questions. Before conducting the evaluation, the reviewers underwent MARS training using a MARS training video available on YouTube [29]. Shared understanding of MARS, APPLICATIONS scoring system items, and app ratings by specific statements was followed by an independent review of a few apps and discussion of scores to assess consensus. Any disagreements were discussed with a third reviewer (CK or AT).

6. MARS evaluation

The MARS evaluation tool consists of 19 items and four questions, as shown in Table 1 [24,29]. The first 19 items were designed to analyze the four domains of the objective quality section: 1) engagement, 2) functionality, 3) esthetics, and 4) information. The score for each item ranges from 1 to 5 (1=inadequate; 2=poor; 3=acceptable; 4=good; 5=excellent). The mean scores for each domain were calculated, and the total sum of the objective domains was divided by 4 to develop the average objective quality score of the apps. The fifth domain of the MARS consists of four questions and is designed to analyze the subjective quality section. The total MARS score was calculated using the objective and subjective quality scores to describe the overall quality of the app. The final score was calculated by two authors, as shown in Table 1.

Table 1

Mobile app rating scale of cervical cancer applications

App name	Objective quality section				Objective quality score (5 points)	Subjective quality score (5 points)	Overall quality MARS score (5 points)
	Engagement (5 points)	Functionality (5 points)	Aesthetics (5 points)	Information (5 points)
1. Mareng Pak Mdluk[a)]	2.20	4.00	2.33	3.29	2.95	2.75	2.85
2. Cervical cancer guide	2.20	4.25	2.00	3.14	2.90	2.25	2.58
3. Cervical Cancer (Bedieman)	1.60	3.75	1.00	3.29	2.41	2.25	2.33
4. Cervical Cancer (Nature Healthy Care)	1.80	3.75	2.67	3.14	2.84	1.75	2.30
5. Cervical Cancer (Fumo)	1.80	4.25	1.67	3.29	2.75	2.25	2.50
6. MyPap	3.40	4.25	2.67	3.29	3.40	3.75	3.58
7. Scanvix	2.40	3.00	2.33	2.85	2.65	1.25	1.95
8. Cervical Cancer Symptoms	2.20	2.75	2.00	2.71	2.42	1.50	1.96
9. Cervical cancer (Anastore)	2.40	2.00	1.00	2.71	2.03	1.00	1.52
10. TNM Cancer Staging Calculator	3.00	4.25	3.67	3.43	3.59	3.25	3.42
11. BSCCP	3.00	3.00	3.33	3.33	3.08	1.25	2.17
12. Cancer Genetics	2.40	3.50	2.33	3.43	2.92	1.00	1.96
13. FIGO Gyn Cancer Management	4.20	3.50	3.00	4.57	3.82	3.25	3.54
14. NCCN Guidelines®	1.80	3.00	2.67	4.86	3.08	4.00	3.54
15. Cervical Cancer (Personal Remedies LLC)	1.80	2.00	2.00	2.14	1.99	1.00	1.50
16. ASCCP Management Guidelines	3.00	4.00	4.00	4.86	3.96	4.00	3.98
Mean±standard deviation	2.45±0.71	3.45±0.76	2.42±0.84	3.40±0.76	2.92±0.57	2.28±1.10	2.61±0.79

App, application; MARS, mobile app rating scale; TNM, tumor, nodes, and metastases; BSCCP, The British Society for Colposcopy and Cervical Pathology; FIGO, The International Federation of Gynecology and Obstetrics; NCCN, The National Comprehensive Cancer Network; LLC, Limited Liability Company; ASCCP, The American Society for Colposcopy and Cervical Pathology.

Mareng Pak Mdluk” means cervical cancer in Thai language.

7. APPLICATIONS scoring system evaluation

The APPLICATIONS scoring system was developed based on existing literature [16,25]. This assessment tool includes 10 objective components and two subjective components, as shown in Table 2. The comprehensiveness score was dependent on the completeness of the topic in the app content. The app received 1 point for each criterion, contributing to a total of 3 points. Subjective components, including ease of navigation and presentation, were evaluated. The score for each item ranges from 1 to 5 (1=inadequate; 2=poor; 3=acceptable; 4=good; 5=excellent). An average score was calculated by two authors; less than three received no points, and three or higher received 1 point with an average score. The total score is calculated as the sum of the objective and subjective component scores.

Table 2

APPLICATIONS score of cervical cancer apps

App name	APPLICATION scoring system												Total score (16 points)
	A	P	P	L	I	C	A	T	I	O	N	S
1. Mareng Pak Mdluk[a)]	3	1	1	0	1	0	0	0	0	1	1	1	9
2. Cervical cancer guide	3	1	1	0	1	1	1	0	0	1	1	1	11
3. Cervical Cancer (Bedieman)	3	1	1	0	1	1	1	0	0	0	1	0	9
4. Cervical Cancer (Nature Healthy Care)	2	1	1	0	1	0	0	0	0	1	1	0	7
5. Cervical Cancer (Fumo)	3	1	1	0	1	1	0	0	0	0	1	0	8
6. MyPap	1	1	1	0	0	1	1	0	0	1	1	0	7
7. Scanvix	2	1	1	0	1	1	1	0	0	0	0	0	7
8. Cervical Cancer Symptoms	3	1	1	0	1	1	0	0	0	1	1	1	10
9. Cervical cancer (Anastore)	1	1	1	1	0	0	0	0	0	1	1	0	6
10. TNM Cancer Staging Calculator	1	1	1	0	1	1	0	0	1	0	1	1	8
11. BSCCP	2	1	0	0	1	1	1	1	1	1	1	1	11
12. Cancer Genetics	0	1	1	1	1	0	1	0	1	0	0	0	6
13. FIGO Gyn Cancer Management	1	1	1	1	0	0	1	0	1	1	1	0	8
14. NCCN Guidelines®	3	1	1	1	1	0	1	1	1	0	0	0	10
15. Cervical Cancer (Personal Remedies LLC)	1	0	1	1	1	0	1	0	1	2	0	0	8
16. ASCCP Management Guidelines	2	0	1	1	1	0	1	0	1	2	1	1	11
Mean±standard deviation	1.94±1.00	0.88±0.34	0.94±0.25	0.38±0.50	0.81±0.40	0.50±0.52	0.63±0.50	0.13±0.34	0.44±0.51	0.75±0.68	0.75±0.45	0.38±0.50	8.50±1.71

App, application; A, app comprehensiveness (3 points); P, price (1 point); P, paid subscription (1 point); L, literature used (1 point); I, in-app purchase (1 point); C, connectivity (1 point); A, advertisements (1 point); T, text search field (1 point); I, interdevice compatibility (1 point); O, other component (1 point); N, navigation ease (1 point); S, subjective presentation (1 point); TNM, tumor, nodes, and metastases; BSCCP, The British Society for Colposcopy and Cervical Pathology; FIGO, The International Federation of Gynecology and Obstetrics; NCCN, The National Comprehensive Cancer Network; LLC, Limited Liability Company; ASCCP, The American Society for Colposcopy and Cervical Pathology.

Mareng Pak Mdluk” means cervical cancer in Thai language.

8. App rating using specific statements

To provide a more specific assessment, we developed 16 statements regarding misconceptions about cervical cancer based on a previous study, as shown in Table 3 [6,30]. The app with content that matches a specific statement receives 1 point for each content that matches the specific statement. The maximum score for the app rating was 16.

Table 3

App rating using specific statements of cervical cancer apps

Statement	App name																Mean score
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
1. Cervical cancer is caused by HPV infection	1	1	1	1	1	0	1	1	1	0	0	1	0	1	1	1	0.75
2. HPV infection is contracted by sexual contact	1	1	1	1	1	0	1	0	1	0	0	0	0	1	1	1	0.63
3. HPV infection cannot be prevented by vaginal douching after intercourse	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0.00
4. HPV infection cannot be entirely prevented by using a condom	0	1	0	1	0	0	0	1	1	0	0	0	0	1	1	0	0.38
5. Smoking increases the risk of cervical cancer	1	1	1	1	1	0	1	1	1	0	0	1	0	1	1	1	0.75
6. Using combined contraceptive pills increases the risk of cervical cancer	1	0	1	1	1	0	1	1	1	0	0	1	0	1	1	1	0.69
7. Early stage of cervical cancer is asymptomatic	1	1	1	1	1	0	0	1	1	0	0	0	0	1	0	0	0.50
8. Cervical cancer can screen with a Pap smear	1	1	1	1	1	1	1	1	1	0	1	0	0	1	0	1	0.75
9. Pap smear is not solely indicated in women with vaginal discharge or bleeding	0	1	0	1	0	0	0	1	1	0	0	0	0	1	0	0	0.31
10. The HPV infection cannot be detected by conventional cervical cancer screening methods or Pap smear	0	0	1	1	0	0	0	1	1	0	1	0	0	1	0	1	0.44
11. An abnormal Pap smear result is not indicative of cervical cancer	0	0	1	1	1	0	0	1	1	0	1	0	0	1	0	1	0.50
12. A Pap smear result cannot be known immediately after the vaginal examination	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0.06
13. Most HPV infections can be healed by themselves	0	1	0	0	1	0	0	0	1	0	0	1	0	0	0	0	0.25
14. The HPV vaccine cannot provide 100% protection against HPV	1	1	1	1	1	0	1	1	1	0	0	1	0	1	1	0	0.69
15. Having multiple sexual partners increases the risk of cervical cancer	1	1	1	1	1	0	1	0	1	0	0	1	0	1	1	1	0.69
16. Sex at an early age increases the risk of cervical cancer	0	1	0	0	1	0	1	0	1	0	0	1	0	1	0	1	0.44
Total score (16 points)	8	11	11	12	11	1	8	10	14	0	3	7	0	13	7	9	7.81

1, Mareng Pak Mdluk; 2, Cervical cancer guide; 3, Cervical Cancer (Bedieman); 4, Cervical Cancer (Nature Healthy Care); 5, Cervical Cancer (Fumo); 6, MyPap; 7, Scanvix; 8, Cervical Cancer Symptoms; 9, Cervical cancer (Anastore); 10, TNM Cancer Staging Calculator; 11, BSCCP; 12, Cancer Genetics; 13, FIGO Gyn Cancer Management; 14, NCCN Guidelines®; 15, Cervical Cancer (Personal Remedies LLC); 16, ASCCP Management Guidelines.

App, application; HPV, human papillomavirus; TNM, tumor, nodes, and metastases; BSCCP, The British Society for Colposcopy and Cervical Pathology; FIGO, The International Federation of Gynecology and Obstetrics; NCCN, The National Comprehensive Cancer Network; LLC, Limited Liability Company; ASCCP, The American Society for Colposcopy and Cervical Pathology.

9. Statistical methods

Descriptive statistics, including numbers and percentages, were used to categorize the characteristics of apps, such as platforms (Android or iOS), free or paid, user rating, app store category, and language. App evaluations were calculated and presented as raw and mean scores, with a standard deviation (SD) for each assessment tool. All statistical analyses were performed using Microsoft Excel (Office 365).

Results

1. App identification

Seven hundred sixty-three apps were identified (727 from Google Play Store and 36 from the iTunes Store). Four hundred and fifty-nine applications were excluded owing to duplication. After screening the app names and descriptions, 265 were excluded. The remaining 39 apps were downloaded and assessed for their eligibility. After a detailed evaluation, 23 apps were excluded, leaving 16 included in our study (Fig. 1).

2. App characteristics

Table 4 summarizes the characteristics of the apps included. Of all the apps, 16 (100%) were available on the Android platform, and seven (43.8%) were available on both the Android and iOS platforms. The most significant number of apps were classified as medicine, followed by health, fitness, and entertainment. All apps were in English. Only one app was in both English and Thai. Most apps (87.5%) had been updated over a 3-year period (2019–2021), and 14 (87.5%) were free to download. Only two apps required payment, with costs ranging from $3.49 to $9.99. The number of app downloads ranged from 1 to 100,000. The National Comprehensive Cancer Network (NCCN) Guidelines®, produced by this scientific organization, was the only high-use app, with >100,000 downloads. The average star ratings on the Android and iOS platforms range from 0 to 4.8.

Table 4

Characteristic of cervical cancer apps

App name (developer)	Platform	Category	Language	Update dates[a)]	Price (USD)	Download	Average star rating
1. Mareng Pak Mdluk[b)] (Kimmydroid)	Android	Health and fitness	TH/EN	04/11/2020	0	>1,000	-
2. Cervical cancer guide (free apps for everyone)	Android	Health and fitness	EN	28/05/2020	0	>5,000	3.8
3. Cervical Cancer (Bedieman)	Android	Health and fitness	EN	11/04/2020	0	>100	-
4. Cervical Cancer (Nature Healthy Care)	Android	Entertainment	EN	20/06/2020	0	>1,000	-
5. Cervical Cancer (Fumo)	Android	Medical	EN	12/04/2020	0	>1,000	-
6. MyPap (Ovidiu Miu)	Android	Health and fitness	EN	06/02/2021	0	>50	-
7. Scanvix (Charles Ingorot)	Android	Health and fitness	EN	06/10/2019	0	>100	4.8
8. Cervical Cancer Symptoms (Revolxa Inc.)	Android	Medical	EN	01/03/2021	0	>1,000	4.8
9. Cervical cancer (Anastore)	Android	Medical	EN	22/03/2017	0	>100	4.1
10. TNM Cancer Staging Calculator (Integrated Cancer Research)	Android and iOS	Medical	EN	22/03/2018	0	>10,000	4.5
11. BSCCP (Thursday Studio)	Android and iOS	Medical	EN	22/07/2020	0	>500	-
12. Cancer Genetics (UBQO Limited)	Android and iOS	Medical	EN	18/02/2016	0	>1,000	-
13. FIGO Gyn Cancer Management (International atomic energy agency)	Android and iOS	Health and fitness	EN	09/04/2019	0	>10,000	4.7
14. NCCN Guidelines® (NCCN)	Android and iOS	Medical	EN	24/06/2020	0	>100,000	3.0
15. Cervical Cancer (Personal Remedies LLC)	Android and iOS	Medical	EN	02/12/2019	3.49	>1	-
16. ASCCP Management Guidelines (ASCCP)	Android and iOS	Medical	EN	14/10/2020	9.99	>5,000	2.8

App, application; EN, English language; TH, Thai language; TNM, tumor, nodes, and metastases; BSCCP, The British Society for Colposcopy and Cervical Pathology; UBQO, The Specific Name of Private limited Company; FIGO, The International Federation of Gynecology and Obstetrics; NCCN, The National Comprehensive Cancer Network; LLC, Limited Liability Company; ASCCP, The American Society for Colposcopy and Cervical Pathology.

Update dates are shown as (DD/MM/YYYY);

Mareng Pak Mdluk” means cervical cancer in Thai language.

3. App evaluation using MARS

Table 1 shows the MARS scores of the 16 apps stratified by quality. In terms of objective quality, the mean objective quality score was 2.92±0.57 out of 5, with the highest scoring app being “The American Society for Colposcopy and Cervical Pathology (ASCCP) Management Guidelines” (3.96). Overall, apps scored highest in the functionality domain (3.45±0.76), followed by information (3.40±0.76), engagement (2.45±0.71), and aesthetic (2.42±0.84) domains. In terms of the subjective quality section, the mean subjective quality score was 2.28±1.10 out of 5, with the highest scoring apps being “ASCCP Management Guidelines” (4.00) and “NCCN Guidelines®” (4.00). The overall quality MARS score was 2.61±0.79 out of 5. The highest scoring app was “ASCCP Management Guidelines” (3.98), followed by “MyPap” (3.58), “FIGO Gyn Cancer Management” (3.54), and “NCCN Guidelines®” (3.54).

4. App evaluation using the APPLICATIONS scoring system

The APPLICATION scores of the 16 apps are listed in Table 2. The mean±SD was 8.50±1.71 of 16. The highest scoring apps were “ASCCP Management Guidelines,” “The British Society for Colposcopy and Cervical Pathology (BSCCP),” and “Cervical Cancer Guide,” each receiving 11 points, closely followed by “NCCN Guidelines®” and “Cervical Cancer Symptoms,” each receiving 10 points. The lowest app received 6 points from a total of 16 points. Overall, apps scored highest in the paid subscription and price domains, because almost all apps were free to download. By contrast, apps scored poorly in the text search, literature, and subjective presentation domains. The app comprehensiveness mean score±SD was 1.94±1.00 out of 3.

5. App rating using specific statements

The app “Cervical Cancer,” developed by Anastore, has content that most highly matches misconceptions about cervical cancer (14 of 16 statements), followed by “NCCN Guidelines®” (13 of 16 statements). The mean score of all apps was 7.81±4.56 out of 16. Most apps (12 of 16) provide information that human papillomavirus (HPV) causes cervical cancer, and that smoking increases the risk of cervical cancer. However, none of the apps mentioned that HPV infection could not be prevented by vaginal douching after intercourse. Only one app provided information indicating that HPV smear results cannot be obtained immediately after vaginal examination (Table 3).

6. Comparison of app scores using different assessment tools

The top three apps identified by each assessment tool differed, as shown in Table 5. However, some apps, such as “NCCN Guidelines®,” were rated as high-quality by all three assessment tools, followed by “ASCCP Management Guidelines,” which were rated as high-quality apps by two assessment tools (MARS and APPLICATIONS scoring system). Some apps were rated as high-quality apps by one assessment tool but were not ranked as high-quality apps by other assessment tools, such as “MyPap,” “FIGO Gyn Cancer Management,” “Cervical Cancer Symptoms,” “BSCCP,” “Cervical Cancer” developed by Fumo, and “Cervical Cancer” developed by Anastore. An app’s average star rating did not seem to correlate with the ratings of other assessment tools.

Table 5

Multimodal assessment tools comparison of cervical cancer apps

App name	Assessment tools			Average star rating
	Total MARS score (5 points)	APPLICATIONS scoring system (16 points)	Apps rating using specific statements (16 points)
1. Mareng Pak Mdluk[a)]	2.85	9	8	0
2. Cervical cancer guide	2.58	11	11	3.8
3. Cervical Cancer (Bedieman)	2.33	9	11	0
4. Cervical Cancer (Nature Healthy Care)	2.30	7	12	0
5. Cervical Cancer (Fumo)	2.50	8	11	0
6. MyPap	3.58	7	1	0
7. Scanvix	1.95	7	8	4.8
8. Cervical Cancer Symptoms	1.96	10	10	4.8
9. Cervical cancer (Anastore)	1.52	6	14	4.1
10. TNM Cancer Staging Calculator	3.42	8	0	4.5
11. BSCCP	2.17	11	3	0
12. Cancer Genetics	1.96	6	7	0
13. FIGO Gyn Cancer Management	3.54	8	0	4.7
14. NCCN Guidelines®	3.54	10	13	3.0
15. Cervical Cancer (Personal Remedies LLC)	1.50	8	7	0
16. ASCCP Management Guidelines	3.98	11	9	2.8
Mean±standard deviation	2.60±0.79	8.50±1.71	7.81±4.56	-

App, application; MARS, mobile app rating scale; TNM, tumor, nodes, and metastases; BSCCP, The British Society for Colposcopy and Cervical Pathology; FIGO, The International Federation of Gynecology and Obstetrics; NCCN, The National Comprehensive Cancer Network; LLC, Limited Liability Company; ASCCP, The American Society for Colposcopy and Cervical Pathology.

Mareng Pak Mdluk” means cervical cancer in Thai language.

Discussion

Our study noted that cervical cancer apps available in the digital marketplace demonstrate above-average quality. However, the apps scored poorly in app-specific ratings using specific statements, indicating a need for improvement in terms of the essential information conveyed by these apps. Moreover, the type of assessment tool used influenced the different app quality rating results. Many assessment tools can be used to determine the quality of mHealth apps for cancer. Therefore, our study applied the MARS and APPLICATIONS scoring systems to evaluate the quality of the apps. The MARS and APPLICATIONS scoring systems have been frequently used to assess mHealth app quality over the past 6 years [16,22-25]. However, no studies have used both scoring systems and compared their results. The tools we used had domain criteria for evaluation that matched a recent systematic review of the quality assessment criteria for mHealth apps, which suggested criteria for future research on the quality of mHealth apps [27].

All apps in our study were available on the Google Play Store, with a more significant percentage available than the iTunes Store. Our findings were consistent with the results of previous studies, which found that more apps were available on the Google Play Store than on the iTunes Store [9,10,13]. A recent study found that up-to-date apps are uncommon [13,22]. Contrary to our findings, most cervical cancer apps (89.5%) were up-to-date over the last 3 years.

Our study showed a mean score of 2.61±0.79 out of 5 for the MARS overall quality score of the apps, which is similar to a previous study that showed a mean score of 2.98 out of 5 for the mHealth app for genitourinary tumors [22]. However, these scores were lower than for the evaluation of mobile apps to track patient-reported outcomes for cancer patients (3.50 of 5) [31], and the evaluation of apps to support medication adherence and symptom management in cancer patients showed a mean score ranging from 2.8 to 4.3 out of 5 [23]. One study reported a significantly lower mean quality score (1.96/5) for prostate cancer, breast cancer, and colorectal cancer apps [12].

We found that apps scored poorly in the MARS engagement domain, in that most apps are unable to entertain and have no interactive function with the app’s user. In contrast, apps scored highest in the functionality domain compared to other domains. These scores are consistent with the results of 2 previous studies [22,23].

One study omitted the subjective MARS quality score from evaluation because this domain criterion might impose a risk for bias [23]. However, our study included all MARS domains for evaluation. The subjective quality domain is displayed separately in Table 1, before the averaging of the overall MARS quality score in the final column.

Our study showed a significantly lower mean APPLICATIONS score (8.50±1.71 of 16) compared with a study of gynecologic cancer apps [16], which reported a mean score of 10±2.30 out of 16. However, the previous study was not specific to evaluating cervical cancer apps, thus limiting direct comparison of the results.

Cervical cancer is a significant health problem in developing countries [32]. However, some app content may not contain essential information that focuses on users in developing countries. Therefore, we developed a specific tool using misconceptions about cervical cancer from previous studies as a tool for assessment to ensure that apps provide this information [6,30].

The number of app downloads and average star ratings in digital marketplaces may not guarantee the quality of the apps. Only one app in our study (NCCN Guidelines®) was rated as high quality by all three assessment tools. However, the top three apps as determined by each assessment tool were different, which confirms the lack of standardized quality assessment tools for mHealth apps.

Most developers are commercial companies, and scientific organizations or healthcare providers have developed only a few apps (NCCN Guidelines and ASCCP Management Guidelines). According to our results, apps developed by scientific organizations are associated with high-quality scores. Incorporating healthcare professionals as partners or stakeholders during the development process may strengthen the quality of the apps.

Our study is the first to perform a comprehensive assessment of cervical cancer apps using multimodality assessment tools and compare the results from each assessment tool. However, this study has some limitations. First, this study did not include apps in languages other than English and Thai, which limits the generalizability of our findings. Second, new apps are being developed and updated all the time in the digital marketplace. There is a possibility that some apps may have been launched after our research was completed and thus were not included in this study. Finally, this study focused on mHealth apps and we did not therefore include websites or social media content, which have become common sources of consumer information.

Cervical cancer apps that are now available in the digital marketplace are of above-average quality. However, there is a need for improvement in terms of the information conveyed by these applications. Moreover, the assessment tools influenced different app quality rating results, confirming the lack of standardized quality assessment tools for mHealth apps.