Designing evaluation framework for the empirical assessment of COVID-19 mobile apps in Pakistan.

The significant proliferation in the mobile health applications (Apps) amidst Coronaviruses disease 2019 (COVID-19) resulted in decision making problems for healthcare professionals, decision makers and mobile users in Pakistan. This decision making process is also hampered by mobile app trade-offs, multiple features support, evolving healthcare needs and varying vendors. In this regard, evaluation model for mobile apps is presented which completes in three different phases. In first phase, features-based criteria is designed by leveraging Delphi method, and twenty (20) mobile apps are selected from app stores. In second stage, empirical evaluation is performed by using hybrid multi criteria decision approaches like CRiteria Importance Through Inter-criteria Correlation (CRITIC) method has been used for assigning weights to criteria features; and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) method has been used for assessment of mobile app alternatives. In last step, decision making is performed to select the best mobile app for COVID-19 situations. The results suggest that proposed model can be adopted as a guideline by mobile app subscribers, patients and healthcare professionals.

Introduction

The current COVID-19 disease started from Wuhan city, China and spread all over the world. It affected all countries around the globe due to its highly contagious nature. In the South Asian countries, Pakistan was also hit hard by COVID-19 pandemic due to its limited health infrastructure and scarcity of resources. Like other countries, Pakistan fought the pandemic through conventional approaches such as social distancing, washing and sanitizing hands, quarantining/self-isolation and face masking. However, the emerging role of digital technologies such as machine learning, Big data, Internet of Things (IoT), Immersive technologies etc. has been phenomenal to control the spread of the COVID-19 outbreak in different parts of the world. In view of its importance, Pakistan also altered its strategy from manual to digital technology during the rise of the second and third wave of the pandemic. In the current pandemic, these digital technologies have been applied for three major purposes such as prevention, clinical problems, diagnosis, planning and response and surveillance. Smartphone apps are also one specific class among the cutting-edge digital technologies. Mobile apps collect huge amount of data from many healthcare IoT and mobile devices and subsequently it is sent to the remotely secure server [1]. The role of mobile applications with respect to combating COVID-19 is remarkable because they have enormous applications in the current pandemic such as information sharing, home monitoring, self-symptom reporting, contact tracing, risk assessment and training [2,3]. Pakistan also opted to use mobile phone apps for a variety of tasks such as contact tracing, telemedicine, self-assessment, home monitoring and diagnosis, awareness and prevention purposes. Providing the extensive list of features and functionalities, mobile apps also bring certain kinds of problems, where the mobile adoption community is impeded by its usage due to some concerns. These concerns include privacy, design flexibility, exposure notifications, equity, type of architecture and accuracy etc. In Pakistan, these concerns become more significant due to an array of mobile apps presented to deal with the COVID-19 pandemic. The government of Pakistan in collaboration with private setups developed many apps during the pandemic. An interesting observation has been made that all apps are related to medical, telemedicine, online treatment, healthcare education and physical fitness. According to our literature study, over fifty (50) mobile apps have been updated to include COVID-19 related features and more than fifteen (15) apps have been newly launched during the pandemic. But, unfortunately, there does not exist a single specific mobile health app that is furnishing all the required features and functionalities. Similarly, trade-off exists among the features provided by the smartphone apps. This rapid rise in mobile apps and feature trade-off created decision making issues for the smartphone community regarding the selection of right mobile app technology. Decision making about the installation of right mobile app in the huge list is also tricky, time-consuming and tedious job. As, the decision making has significant impact on our daily life activities. It also has a good impact on the selection of quality of mobile app that can be used for different tasks in the current pandemic.

The decision making about choosing the most rational ad suitable mobile app in the current pandemic can be challenging job in the pandemic. Therefore, the proposed assessment model is presented to address the selection issues related to COVID-19-centered smartphone apps during the current COVID-19 scenario. The proposed evaluation framework evaluates the mobile apps with respect to designed criteria. It completes in three different steps such as in the first step, a benchmark/criteria is designed after collecting features from literature and consulting the experts group based on survey. In this step, mobile apps related to COVD-19 are selected from play stores such as Apple app store, Google plays store, Microsoft's store and web-based sources. During this process, Delphi method has been applied in an iterative fashion for feature classification and collecting data related to the smartphone features. After finalizing the mobile apps for assessment and designing criteria, the second step is to leverage hybrid multi criteria decision making (MCDM) methods for assessment and ranking of mobile phone apps. In this context, CRITIC method is applied to assign weights to the criteria features and TOPSIS method has been used for the assessment and ranking of mobile app alternatives. Last step of the proposed framework is the output, where decision making and ranking of mobile apps is done based on scoring values produced through the TOPSIS method. According to our empirical assessment, COVID-19 Gov PK mobile app alternative achieved the highest score among the mobile apps selected in this study. Hence, it is ranked as the best and most apposite choice for dealing with current COVID-19 pandemic in Pakistan. The COVID-19 Gov PK mobile health app provides a more appropriate and reasonable solution based on the identified set of features related to COVID-19 situations in Pakistan. The proposed framework is tested and evaluated by external experts for features by conducting a survey because we did not find any similar work that is adopting MCDM approaches for mobile app ranking and decision making. Therefore, the consistency and accuracy of results are also validated by using another MCDM technique known as Simple Additive Weighting (SAW).

Contribution

The major contributions of the proposed evaluation model are as give below.

Earlier works presented in this domain are comparing mobile apps based on theoretical ground such as without sufficient mathematical and empirical analysis procedures. However, the proposed evaluation model of mobile apps is more quantitative and empirical in nature.

According to a literature study, the existing evaluation methods for mobile apps are primarily focused on the usability and security features of COVID-19 mobile phone apps. The proposed evaluation model covers the overall features such as COVID-19 specific features and general features.

The heuristic and A/B testing approaches for evaluating mobile apps are solely intended to focus on the interface aspect of mobile apps. The proposed research study also considers different dimensions related to COVID-19 features and other important functions. Similarly, the number of experts in heuristic approaches is limited to three or less but the proposed mobile app evaluation framework is created by consulting ten (10) experts in this domain.

This is the first attempt to use the hybrid MCDM approach for the assessment and ranking of mobile health applications of COVID-19. The CRITIC method is applied to assign weights and TOPSIS is employed for evaluation and decision making related to COVID-19 mobile apps.

The existing evaluation methods or approaches are not fully tested and validated with respect to performance. The proposed evaluation model of mobile apps is tested for features and results are also verified by using SAW method.

This paper is consisted of five major remaining sections: Section (2) presents all the related works, section (3) describes the research methodology, section (4) presents the evaluation and testing of proposed framework, section (5) discusses the results and finally section (6) presents the conclusion of this paper.

Related work

According to a literature study, the existing studies related to evaluating COVID-19 apps are based on theoretical analysis or direct comparison of selected set of features. In the existing literature, there are very few studies that are based on using quantitative and empirical approaches to analyze COVID-19 applications. The majority of previous works are based on surveys, systematic reviews and literature reviews. In this section, our major focus is to compare the proposed evaluation approach with the existing studies to highlight the research gaps in comparison to our study. The systematic study presented by Kondylakis et al. [2] discusses the various features of mobile apps targeted towards dealing with the coronavirus. Chidambaram et al. [3] collected eighty two (82) apps from thirty five (35) countries in the UK. They applied a tool known as system-wide analysis of mobile health-related technologies (SWAT) for the analysis and evaluation of corona-centred mobile apps. Main features included in their analysis were usability, functionality, design, user perception, information and content and ethical issues. The work presented by Kouliaridis et al. [4] is focused on twenty six (26) mobile apps collected from different European countries from Google play store. They performed both static and dynamic analyses of mobile apps by studying the permission, access level, weakness, misconfiguration, vulnerabilities and privacy issues. Similarly, Sun et al. [5] work is also related to presenting the security based empirical assessment tool known as COVIDGUARDIAN for contact tracing apps related to COVID-19. They performed an analysis of forty (40) apps worldwide for security analysis. The proposed security-based assessment tool is primarily intended to test the security, information leakage, malware attacks and embedded trackers. Islam et al. [6] performed an exploratory analysis of apps related to the COVID-19 pandemic by conducting a review. They performed analysis using nine (9) features against 25 COVID-19-centered apps. The features included in their study were remote assistance, patient monitoring, current status, COVID-19 prevention, control, treatment services, communication support, raising awareness and augmenting mental health. Amann et al. [7] covered contact tracing apps reported in newspapers in Switzerland, Germany and Austria by using six (6) main themes related to developing and adopting contact tracing applications. The main agenda for discussing the different apps with respect to app role, scientific rigor, voluntariness, functional efficacy, data governance and the role of IT giants. Azad et al. [8] analysed twenty six (26) smartphone-based contact tracing apps from seventeen (17) different countries. They performed analysis based on three major analysis factors such as privacy, installation permissions and user reviews/comments from the play stores. Salehinejad et al. [9] evaluated thirteen (13) apps using the mobile application rating scale (MARS) technique. They used a five (5) point scale with the help of two evaluator in which COVID-19 GOV PK app achieved highest rank. This study used six (6) evaluation parameters: engagement, functionality, aesthetics, information provision, subjective and app-specific sore. Similarly, Ming et al. [10] evaluated mobile apps based on both basic and advanced features by assigning scores to the apps against the features. They categorized mobile health apps based on features and functionalities. The advanced features consist of knowledge, home monitoring, contact tracing, and consultation with health authorities online and are officially maintained by health authorities. Raman et al. [11] presented an evaluation framework for assessing the contact tracing apps. The framework is known as COVIDTAS. The complete detail of comparing our proposed work with existing works in terms of evaluation approach, parameters and limitations is given in Table 1 .

Table 1

Comparison of proposed work with existing evaluation approaches

Ref.	Evaluation method	Parameters	Limitations/Research gaps
[2]	Effective Public Health Practice Project (EPHPP) tool	Risk assessment	(-) Theoretical comparison(-) Lack of empirical analysis(-) Only risks are identified
[3]	Systems Wide Analysis Tool (SWAT)	Usability, functionality, Ethical issues, Security, Design, Perception, information and content	(-) Scoring system can be biased(-) Validation of the proposed study is missing(-) Focusing only on the contact tracing mobile apps(-) Requires more empirical analysis
[4]	Theoretical approach for app analysis	APIs, Network security, Permission level, Network traffic. Location etc.	(-) Focusing only on the contact tracing mobile apps(-) Only security and privacy are the major concern(-) Empirical analysis is required
[5]	COVIDGUARDIAN	Manifest weakness, Personally Identifiable Information (PII)leaks, Vulnerabilities, Malware detection	(-) Focusing only on the contact tracing mobile apps(-) Security aspect of apps are highlighted
[8]	Literature based study	Privacy, Installation Permissions and User reviews	(-) Restricted to study only contact tracing apps(-) Empirical analysis is missing(-) Theoretical study
[11]	Covid Tracing App Scale (COVIDTAS) framework	Security, usability, accessibility, data management, public ownership, Transparency rights, play store rating etc	(-) Focusing only on the contact tracing mobile apps(-) Criteria is not adequate enough(-) Important mobile apps are skipped
[12]	Literature based systematic review	Knowledge, Tracing, Home monitoring, Online consultation,Maintained by health authority	(-) Some important criteria features are missing(-) Empirical analysis is required(-) Subjectivity is also a concern in assigning values
[13]	Mobile Application Rating Scale (MARS)	Engagement, Functionality Aesthetics Information Subjective App- Specific score	(-) Empirical analysis is required(-) Subjectivity can be an issue(-) Heterogeneity of mobile apps(-) Information scoring requires advanced method
[14]	COVIGILANT Taxonomy	Availability, Subjective Satisfaction, Universality, Design effectiveness, User interaction	(-) Only usability aspects of apps are included(-) Only contact tracing apps are included(-) Study is based theoretical background
Proposed Work	COVID-19 Empirical Assessment Model	Privacy, Information provision, Design, Usability, TelemedicineContact tracing, Home monitoring, Self-assessment, Performance	(+) Based on mathematical and statistical approach(+) Free from biasness and subjectivity(+) All important features are considered(+) First evaluation model of its kind(+) Static analysis of mobile apps is conducted(+) Security & usability features are also included(+) Validated and tested by expert's group

Research methodology

In this research work, we present mobile app evaluation framework focused on evaluating and making decision related to the selection of best mobile app in the current COVID-19 pandemic in Pakistan. The main objective of proposed framework is to evaluate the existing mobile apps based on the features. This framework is designed to evaluate apps in light of the already defined criteria. A complete picture of the proposed evaluation framework is shown in Fig. 1 .

Fig. 1

Structure of evaluation framework

. There are three major steps of the proposed methodology such that in the first phase, the criteria and mobile apps are identified which are provided as input to the framework. In the second phase, the framework uses hybrid MCDM approach for the evaluation and ranking of mobile health apps against the designed criteria. In this phase, the proposed framework uses two methods for evaluation such as CRITIC and TOPSIS methods. The last step of the proposed evaluation model is output, where ranking is done according to the quantified score values. The complete step-wise procedure of the proposed evaluation model has been explained in detail as below.

Input: mobile app's selection procedure and criteria designing

In Pakistan, mobile health apps that were released before the pandemic were updated to add corona related features during the COVID-19 pandemic. Mobile apps of different healthcare service categories were presented to deal with COVID-19 scenario in Pakistan. The categorical division of mobile apps is shown in Fig. 2 . Among the mobile apps, the highest percentage category is of health and fitness followed by medical apps. App analysis was performed to better understand the research trends during the COVID-19 pandemic in Pakistan. In this section, we discuss the approach adopted for the collection of different apps from iOS stores, Google Play Store and web-based applications. This is the first phase of the proposed research method. In this step, initially suitable keywords are defined for the purpose of searching apps in different play stores such as iOS, Google Play, Microsoft stores and web sources.

Fig. 2

Categories of mobile apps in Pakistan

The defined keywords are: “Corona apps,” “COVID-19,” “SARS-CoV-2”, “COVID-19 and Corona contact tracing,” “COVID-19 and Corona Self-assessment,” “COVID-19 and home monitoring,” “COVID-19 and telemedicine app”. The search results were collected after applying all the keywords. During this procedure more than fifty (50) mobile apps were initially identified by searching through the keywords. The search was performed on 01.06.2021 on all mobile app stores. We collected many apps from the play stores based on the similarity of features mentioned in the play stores. During the first step of the app selection procedure thirty four (34) apps were collected from the Android app store, sixteen (16) from the iOS app store and three (03) applications are web-based applications. Finally, thirty eight (38) mobile apps were roughly collected.

The main focus of the proposed research work to include the most essential features in the evaluation criteria. In this section, our primary focus is to design a benchmark for the selection and evaluation of mobile apps launched in Pakistan related to health during COVID-19 pandemic. In this study, we set two criteria as benchmark for the evaluation and ranking of mobile apps.

The first criterion also known as eligibility criteria, which is only designed to filter the mobile applications from the paly stores. The second criterion is assessment criteria which is related to the assessment of apps selected after applying the first criteria. The first criteria is consisted of seven major evaluation parameters for selecting mobile health apps from the respective app stores. The main parameters included in this criteria are: app size, language, number of installations, COVID-19 specific, geographical location, application category and user's rating. For example, mobile apps with a size of less than 5 MB, user rating below 3, number of installations less than 100, languages other than English or Urdu, geographical regions other than Pakistan, apps not related to medical, health and fitness category and not intended for COVID-19 specific situations are excluded. After applying the eligibility criteria, the number of mobile apps included for the assessment were restricted to twenty (20). The complete procedure of selecting the mobile health apps is given in Fig. 3 . Finally, Twenty (20) apps were selected for final assessment after applying the eligibility criteria. These mobile apps were evaluated by using assessment criteria. The assessment criteria is the major criteria of the proposed evaluation framework for mobile health apps. This criteria is composed of nine (9) evaluation parameters to assess the final selected mobile health applications. This criteria has two subsets of parameters such as the first set of parameters defines the general features related to the mobile app while the second set defines the COVID-19 specific features. The major elements of the general criteria are usability, information provision, performance, design, privacy and security. The COVID-19 related features include contact tracing, telemedicine, self-assessment and home monitoring. The main objective assessment criteria is also to assess the mobile health apps in all dimensions such as privacy, look, performance, design and COVID-19 related functions and features. The list of complete features involved in both criteria are shown in Fig. 4 . In this figure, we can see the complete classification of features and sub-features involved in both criteria. The main features of assessment criteria are discussed below.

Fig. 3

Mobile apps selection procedure

Fig. 4

Taxonomy of criteria features

Home monitoring

Home monitoring became pivotal during the pandemic due to lock-down restrictions, burden on hospitals and provision of remote healthcare services to the coronavirus patients suffering from early mild symptoms.

This feature is included in almost every mobile health app designed for providing controlling and preventative measures against the coronavirus. It is an important feature as it keeps the patient with high risk at home to provide treatment without visiting hospitals.

Contact tracing

It is the process of identifying and isolating COVID-19 positive persons with the help of using smartphone technology. Apps enabled with contract tracing options have played a key role in mitigating COVID-19 risks. It helps in keeping people safe and quarantined at home without exposure to the infected people. Contract tracing is based on two types of working models such as centralized and decentralized. Decentralized contact tracing apps in the context of privacy and security concerns are deemed as better because of their significant advantages over centralized app architecture.

Telemedicine

It is related to provide healthcare services at remote locations to the patients by using technology and communication infrastructure. This features is included in our criteria as there are many apps that have already been used for telemedicine purposes in Pakistan. It is a vital feature of any COVID-19 mobile health app as it creates a bridge between patients and physicians to interact with each other without physical contact by saving time, energy, cost and limiting the hospital visits.

Self-assessment

This feature enables COVID-19 patients to check, assess and report health conditions on smartphone by using a suitable mobile health app. It allows the patients to conduct self-assessment and evaluate symptoms easily and safely at home and to take the next decision related to their health based on the initial report of self-assessment.

Information provision

Information provision of mobile app can be defined by the current status of COVID-19 information, raising awareness and preventive and controlling measures. The information provided by corona apps can be further defined by the quality and quantity of information, credibility and accuracy [3]. The reason for including this feature in app assessment criteria is that the majority of the apps are targeted to provide COVID-19 related information for the general public related to controlling and spreading of this deadly virus.

Privacy

As, the COVID-19 mobile apps collect sensitive data related to the users and ask for the level of access and permission to the user's phone. It must be ensured that app is using data without disclosure of COVID-19 information and should use it securely and safely even during the transmission or cryptographic activities; thus the duo privacy and security are considered as significant design considerations in previous literature [6].

Design

Design is also an important criterion of mobile apps that is why it has been considered to be included in our final evaluation criteria. The mobile app design has a significant impact on the overall app assessment report. The design of the app can be fully described by sub-attributes such as appearance, suitability of design, consistency of design, attractiveness and visual appearance [3,15]. Visual material helps in delivering complicated information regarding COVID-19 in an easy way. The design characteristics of apps are concerned with the easy and convenient way to access information, use of intuitive navigation buttons and visual representations and proper organization of the app contents [6].

Performance

Performance is an important indicator for assessing COVID-19 specific mobile apps. The performance of mobile health apps can be described by sub-attributes such as communication, alerts, health warnings, guids, connectivity and feedback mechanism.

Usability

The usability feature is taken into account as it covers many aspects of apps such as style, user-friendliness, ease of use, flexibility, accessibility, availability and interaction. Accessibility is further defined by sub-parameters such as functional requirements and user interface elements. The interaction is further designed by considering the user measures. The reason for including this feature in app assessment criteria is that the majority of apps, are targeted to provide COVID-19 related information for the general public and are related to controlling and spreading of this deadly virus in Pakistan.

Process: applying CRITIC and TOPSIS methods

In the second phase, the proposed framework uses hybrid MCDM approach for the evaluation of mobile health apps. For this purpose, the study applies two methods such as CRITIC and TOPSIS, which are discussed as below.

Assigning criteria weights using CRITIC

CRITIC is an acronym for “CRiteria Importance Through Inter-criteria Correlation” initially presented by Diakoulaki et al. [16]. The weights of features/attributes are the important subject of consideration in MCDM situations. Therefore, we applied the most suitable method such as CRITIC for assigning weights. The major purpose of applying CRITIC method in comparison to other MCDM methods are given as:

It is one of the top four weighting methods focused on revealing the subjectivity and eliminating the uncertainty during the decision making problems. CRITIC is the most objective method for assigning weights to the criteria. Objectivity is the main advantage of using this method as it reduces subjectivity and uses a pairwise comparison approach. This method becomes more effective, when it is combined with other objective or subjective methods. It eliminates the effect of decision makers in criteria weights. It has wide range of real world applications as well. The source of information for this method is the decision matrix which is built based on the relative importance of criteria [17]. The CRITIC technique is also a type of correlation approach.

CRITIC method assigns weights by using standard deviation for measuring the contrast intensity level of each criterion. Thus, it makes it sure that criteria feature having contrast intensity is given higher weights. It supports both conflicting relationships and contrasts intensity.

Following are the major steps involved in the CRITIC method [18,19].

Step-1. Creating a decision matrix

Dij indicates the performance value of ith alternative on jth criterion.

Step-2. Normalizing decision matrix

The decision matrix is normalized in this step by using the following equation.

“D ij ⁎” shows normalized performance value of ith alternative over jth criterion.

Step-3. Calculating weights by using standard deviation and co-relation

The weight (Wj) of jth criterion is calculated by using following equation.

In the above equation, Cj is the amount of information contained in the jth criterion. This is calculated as follows:

“” symbolizes the standard deviation of the jth criterion, and rjj ’ denotes the correlation coefficient between the two criteria.

In the proposed research work, the role of the CRITIC method is leveraged to assign weights to the criteria elements. Initially, weights are assigned to the mobile apps with respect to the criteria designed with the expert consulting. The experts assigned weights based on using ten (10) points scale ranging from 1 to 10 numeric values. The minimum value is 1, which means that the existence of feature is missing and the maximum value is 10 which means that the existing feature of the mobile app is providing full functionality. The values were assigned to the apps against the criteria within this scale. The features and mobile apps are interdependent upon each other such that every feature has impact on the mobile apps. The hierarchical structure and interdependencies of apps with criteria are shown in Fig. 5 .

Fig. 5

Features and alternatives interdependency and hierarchy

In this study, we selected twenty (20) mobile apps for final assessment and then ranked according to scoring values. All the selected mobile apps are used as alternatives. The list of mobile apps selected for final evaluation are: Instacare-Pro(A1), Pakistan's National Action Plan for COVID-19(A2), Health Assessment(A3), CoronaCheck(A4), Family Hifazat(A5), AKUH Sehat Check(A6), PurUmeed Aaghaz(A7), Pass Track (A8), HealthSolutions(A9), Sehat Kahani App(A10), MedIQ: Smart Healthcare(A11), MyPractice(A12), Health monitoring PDMA(A13), COVID-19 Gov PK(A14), COVID-19 Care for Media (A15), Healthwire(A16), AKUH Patient Care(A17), Pakistan Red Crescent Sindh(A18), Shifaam(A19) and Sehat Kahani corporate (A20). Similarly, the criteria features as previously mentioned are selected as Privacy (C1), Information provision (C2), Design (C3), Usability (C4), Telemedicine (C5), Contact tracing (C6), Home monitoring (C7), Self-assessment (C8) and Performance (C9). Delphi method was applied to collect data and creating the taxonomy of features. According to the Delphi method, a questionnaire is presented to the experts to provide data related to the selected apps against the criteria. This method works in iterative fashion and follows a round of steps. Data related to features is collected in the first round. After completing first round the intermediate results are again sent to experts for evaluation to get the most appropriate information and provide further detail in case any information is missing in the previous round. The procedure of application of Delphi method in context of data collection in our study is depicted in Fig. 6 .

Fig. 6

Data collection using Delphi method

Among all the criteria features, privacy is the major concern for the mobile apps, therefore a proper methodology of “static ananlysis” procedure is conducted to assess the mobile apps for different privacy issues and vulnerabilites. In this regard, data about mobile apps is collected from three different sources: (i) Privacy policy (ii) Users review and (iii) AndroidmanifestXML. Static analysis of mobile apps is performed with the special assistance of expert's panel. They examined the source codes of every mobile apps through the AndroidmanifestXML file to understand the permission levels and description of permissions. This file has vital information to understand the mobile apps from different security perspectives. Checking the privacy policy of mobile apps in respective play stores can be somehow confusing and misleading.

This analysis helps in understanding the security level of cryptographic functions, excessive permissions, architecture of mobile app, hidden features etc. We also checked the privacy policy of every app to understand about what kind of data is collected, how it is used and where it is shared with 3rd parties. It is also helpful in understanding the security practices like data encryption and request for the removal of data. The user's reviews about the mobile apps were scrutinized to understand about the working and behavior of mobile apps. For this purpose, sentiment analysis is performed on the reviews or the comments of the mobile users. Our procedure for security investigation inspired by the study presented in [20]. Studying the thousands of reviews can be time consuming and require huge effort. Therefore, we searched the user's reviews by defining different keywords related to the privacy and security of apps. The review sentences of every app were checked by searching through these keywords in respective play stores. It helped in collecting all apps that were suffering from vulnerabilities. The complete list of keywords is given in Table 2 .

Table 2

Keyword description

Keywords	Categories	Definition
“Leak”, “Permission”, “Breach”, “Confidentiality”, “Malware” “Phish”, “virus”, “Malware”, “Hack”,	Privacy & security	Privacy issues related to invasion of personal information of users
“Adware”, “Spam”, “Add”, “Advertisement”	Spam	Issues related to unwanted and unpleasant ads on
“Crash”, “Slow”, “Restart”, “Battery”, “ Hang”, “Bug”	System	Issues about creating negative effects on mobile devices

The outcome of our review analysis indicated that several of mobile apps suffer from vulnerability and security issues. For example, French hacker Elliot Alderson, issued a statement related to the security and privacy mobile app COVID-19 GOV PK. According to his statement, this app has many security flaws like its vulnerability to Distributed Denial of Service (DDoS) attacks, sending data on unsecure protocols such as Hyper Text Transfer Protocol (HTTP) and storing the physical location of patients on the map. Similarly, other mobile apps were investigated and score values were assigned based on results obtained from the experts groups and review analysis. The privacy score values (Psv) is obtained by using the following formula.

Where “T” shows the total number of vulnerabilities of individual mobile app. The Psv can also be obtained by adding all the risk score values of mobile app installed on the device. The value of Psv values fluctuates between 0 and 1. The value of privacy is provided as input to the decision matrix. The privacy values of all mobile apps along with number of permissions are given in Fig. 7 .

Fig. 7

Privacy score values and number of mobile app permissions

This procedure is also carried out for all non-security features, and the required values are provided as input in the decision matrix. Finally, the decision matrix built after consulting the experts by using Eq. (1) as given below.

Eq. (2) has been applied to normalize the contents of the decision matrix and results are given in the normalized matrix .

The correlation coefficient among all the criteria elements is calculated for final criteria weights by using (3), (4) and the details are listed in Table 3 .

Table 3

Criteria weights

Criteria	Sum	σj	Cj
C1	6.054	0.411	2.488
C2	2.020	0.556	1.123
C3	3.177	0.529	1.680
C4	2.623	0.586	1.538
C5	3.324	0.677	2.250
C6	3.914	0.550	2.153
C7	2.500	0.598	1.496
C8	2.521	0.620	1.562
C9	3.811	0.965	3.677

The weights assigned by CRITIC method to the criteria features are given in Fig. 8 . Telemedicine, performance, privacy and contact tracing have been given high weightage as calculated by CRITIC method.

Fig. 8

Criteria weights

Evaluation and ranking by TOPSIS

TOPSIS is acronym for “Technique for Order Preference by Similarity to Ideal Solution” was coined by Hwang and Yoon [21]. The reason behind using this method for the decision making and prioritization of mobile app is significantly clear. It has the multiple advantages, which makes it a best choice for assessment and ranking of alternatives. Some of the main reasons for the application this method are given below.

TOPSIS is based on an ideal solution; for example, if a particular alternative is nearer or has a minimum distance to the positive ideal solution and if it is farther or has a maximum distance from the negative ideal solution then it is considered as the best solution. It avoids the situation of same similarity index to negative and positive ideal solutions.

This method has the characteristics of well-established, simplicity in calculation and reliability [21].

It follows easy steps in computation procedure with efficiency and presents the results in more simple mathematical form[22]. It is very flexible approach can be adjusted and extended to any specific environment. This is the reason that it has range of applications in both real-world and theoretical decision making problems.

TOPSIS method uses the following steps in its mathematical calculation [21,22].

Step-1. Building the decision matrix

The decision matrix for “n” number of alternatives denoted by A1, A2, An, and “m” number of criteria such C1, C2, C3…Cm is constructed. The decsion matrix is constructed by using Eq. (6).

Step-2. Normalizing the decsion matrix

After building the decision matrix, the first step of the TOPSIS method is to normalize the decision matrix using the following equations.

Step-3. Computing weighted decision matrix

Step-4. Computing ideal positive and ideal negative solutions

Positive ideal solutions are denoted by A+ and negative ideal solutions are represented by A−. These are determined by using a weighted decision matrix using the following two equations.

The values of all alternatives must be between A+ and A−. The selected alterative must be having smallest distance from A+ and longest distance from the A−.

Step-5. Computing separation measures

The ideal and non-ideal separation measures are denoted by S+ and S− respectively. Both ideal and non-ideal separation measures were computed using the following two equations, respectively.

Step-6. Calculating relative closeness to the ideal solutions

The relative closness with respect to ideal solutions can be obtained by applying the following equation.

Step-7. Ranking of alternatives

The ranking is done by using the Ci value, where the maximum value of Ci means that the higher the ranking order and alternative can be considered as better in the performance. The ranking of preferences can be performed in ascending or descending order.

In context of our research, TOPSIS method is applied to evaluate and make decision related to selection of mobile app alternatives with respect to the selected features. The decision matrix previously defined by the expert panel is subjected as input to the TOPSIS method. The decision matrix is normalized again by using Eq. (7). The results obtained after the normalization process are given in matrix (Rij).

After defining the decision matrix, the weighted normalized decision matrix is computed using Eq. (8). Then. Ideal positive and negative solutions are computed using Eqs. (10) and (11), respectively and results are given in Table 4 .

Table 4

Ideal positive (A+) and ideal negative (A−) solutions

	C1	C2	C3	C4	C5	C6	C7	C8	C9
A+	0.042	0.038	0.037	0.033	0.070	0.102	0.048	0.061	0.080
A−	0.016	0.003	0.014	0.012	0.005	0.015	0.005	0.006	0.029

Finally, ideal separation points and relative closeness are computed by using Eqs. (12), (13) and (14) respectively. The complete numeric values are given in Table 5 .

Table 5

Ideal separation measures and relative closeness (Ci)

Alt	S+	S−	S+ + S−	Relative closeness
A1	0.100	0.072	0.172	0.4164
A2	0.100	0.058	0.158	0.3664
A3	0.075	0.090	0.165	0.5455
A4	0.093	0.093	0.185	0.5000
A5	0.102	0.080	0.182	0.4411
A6	0.099	0.069	0.168	0.4101
A7	0.101	0.067	0.168	0.4002
A8	0.106	0.066	0.172	0.3840
A9	0.117	0.065	0.182	0.3564
A10	0.113	0.061	0.174	0.3511
A11	0.127	0.045	0.172	0.2635
A12	0.133	0.038	0.171	0.2195
A13	0.126	0.045	0.171	0.2622
A14	0.082	0.109	0.191	0.5690
A15	0.123	0.050	0.172	0.2882
A16	0.136	0.025	0.161	0.1526
A17	0.133	0.029	0.163	0.1812
A18	0.128	0.043	0.171	0.2532
A19	0.121	0.050	0.171	0.2935
A20	0.131	0.036	0.167	0.2134

Output: decision making and ranking

The output of proposed evaluation framework of mobile apps is to obtain the relative closeness (Ci) score values for each mobile app alternative. The ranking and decision making are done based on the Ci value. The higher value of Ci indicates that alternative is showing better performance in comparison to the alternatives with low scores. The ranking is performed after applying the TOPSIS method. According to the results of proposed valuation framework, A14 alternative has the highest score of relative closeness among the mobile app alternatives. So, we can conclude that COVID-19 Gov PK is considered as best mobile app in the current pandemic in Pakistan. The complete performance of every mobile app after the application of CRITIC and TOPSIS approaches is given in Fig. 9 . According to the results of proposed evaluation framework, Health assessment, Corona Check and COVID-19 Gov PK are the top three mobile apps in the list of selected mobile apps. Hence, they are reckoned as best choice to deal with COVID-19 scenario in Pakistan.

Fig. 9

Mobile apps performance ranking

Proposed model evaluation and validation

It is very important to determine the most relevant, irrelevant, recommended and not recommended features related to smartphone applications. As, there is no previous similar works to compare the current proposed evaluation framework in terms of features. Therefore, the proposed evaluation framework is evaluated by the experts group. The accuracy, precision and recall are used as metrics for evaluation by surveying the experts of mobile application developers and experts. Our method of evaluation of features is based on surveying experts group in this area, which most effective for the assessment of features-based systems contextual systems [23,24]. The developers were requested to suggest the number of features of their choice for the proposed evaluation framework. The main focus is to analyze the results based on the number of recommended/not-recommended and relevant and irrelevant features as listed in Table 6 .

Table 6

Recommendations classification

	Relevant	Irrelevant
Recommended	a	c
Not recommended	b	d

The number of features suggested by experts and the proposed evaluation framework is represented by “a.” Similarly, the features recommended by our proposed framework are denoted by “b”. Features suggested by only expert panel are represented by “c.” Similarly, “d” denotes the features that were not recommended by the proposed evaluation framework nor by the expert panel. The accuracy, precision and recall are calculated using the following equations.

The details of the evaluation metrics/parameters related to criteria features by every expert panel are given in Table 7 .

Table 7

Calculating evaluation metrics

Expert	a	b	c	d	Accuracy	Recall	Precision
1	11	1	1	9	91%	92%	91%
2	13	0	1	12	96%	93%	100%
3	19	2	0	9	93%	100%	90%
4	12	1	1	10	92%	92%	92%
5	8	0	1	5	93%	89%	100%
6	21	2	3	17	88%	87.5%	91%
7	14	0	2	9	92%	88%	100%
8	22	1	0	25	98%	100%	96%
9	31	0	3	9	93%	91%	100%
10	10	0	0	03	100%	100%	100%
Average					94%	93%	96%

According to the features evaluation metric values it is concluded that criteria features are the most relevant and important for the evaluation of mobile apps during the COVID-19 scenario. The evaluation metrics describe the overall feature's selection procedure and importance of features. According to the values of precision, accuracy and recall, it is concluded that we selected the most relevant and up-to-date features in this study.

According to literature study, there does not exist any evaluation framework that is leveraging MCDM approaches. It is imperative to validate the accuracy and consistency of the proposed model. Therefore, this model is validated by using another well-known MCDM approach known as Simple Additive Weighting (SAW). This method uses the following procedure [25].

Step-1. Normalization is done by using the following equation.

Step-2. Assigning weights to criterion

W = [w 1, w 2……w] (19)

Step-3. Ranking score calculation

Where:

Ranking score value for ith alternative

Denotes the weight of jth criteria

Number of alternatives

Number of criteria

It is normalized rating, which is computed by Eq. (18)

Criteria weights already computed by the CRITIC method have been used in this approach therefore, the first two steps in the SAW method have been skipped. Eq. (20) of SAW method has only been used for calculating ranking score and decision is made according to the values of ranking score. The results of SAW approach in comparison to our proposed hybrid MCDM approach for all the selected alternatives is given in Table 8 .

Table 8

Comparison of proposed work with SAW technique

SAW approach			Proposed work
Alt(s)	Ranking score(Si)	Ranking	Relative closeness	Ranking
A1	4.393	13	0.4164	5
A2	3.947	18	0.3664	9
A3	5.695	2	0.5455	2
A4	5.667	3	0.5000	3
A5	4.654	8	0.4411	4
A6	4.663	11	0.4101	6
A7	4.121	16	0.4002	7
A8	4.329	10	0.3840	8
A9	4.363	12	0.3564	10
A10	4.153	15	0.3511	11
A11	4.676	7	0.2635	14
A12	4.214	14	0.2195	18
A13	4.829	6	0.2622	15
A14	6.182	1	0.5690	1
A15	4.985	5	0.2882	13
A16	3.58	20	0.1526	20
A17	3.802	19	0.1812	19
A18	4.609	9	0.2532	16
A19	5.096	4	0.2935	12
A20	4.035	17	0.2134	17

It validates the framework by ranking and selecting the same alternative. Hence, our proposed framework has produced the most accurate, consistent and error-free quantitative results. As, SAW method is a very simple MCDM techniques but it clearly verifies the ranking scores of the proposed model by using different mathematical procedure.

Results and discussion

After the mathematical procedure, it was concluded that among the selected mobile apps, the COVID-19 Gov PK provides the best features, and can be considered as the best mobile app option to deal with pandemic. This app was launched by the government of Pakistan with the support of the National Information Technology Board. It provides prevention and awareness of citizens related to COVID-19. This app has a dashboard that provides current information about COVID-19 with the alarming feature for hand washing. It has Chatbot to raise awareness about COVID-19 and World Health Organization (WHO) videos furnishing awareness and prevention of coronavirus. The most vital feature of this app is “Radius Alert.”, which helps in maintaining social distance from the infected person. This app is not only rank by our study but the research work conducted by Salehinejad et al. [9], rated the COVID-19 GOV PK with the highest score among thirteen (13) apps by using mobile application rating scale (MARS) technique. They used five (5) point scale with the help of two evaluators. In this study, the top score of (4.7) is achieved by the COVID-19 GOV PK app. Their assessment is based on six (6) evaluation parameters: engagement, functionality, aesthetics, information, subjective and app-specific score. This mobile app is also ranked first by other studies as well. According to literature study, we failed to identify studies that are utilizing MCDM approach for the mobile apps evaluation but still we are highlighting some of the studies in which same mobile app is selected based on different features and evaluation methods. The comparison of the proposed work and existing methods is given in Table 9 .

Table 9

Comparison with similar studies

Ref	Scoring Method	Comparison with other approaches
Ming et al. [12]	Score based assessment	○ Some important criteria features are missing ○ Empirical analysis is required ○ Subjectivity is also a concern in assigning weightage values
Salehinejad et al. [13]	MARS Scale	○ Lack of empirical analysis ○ Subjectivity problem ○ Heterogeneity of mobile apps ○ Information scoring requires advanced method ○ COVID-19 specific features are not included
Proposed Work	CRITIC AND TOPSIS	○ A full-pledged criteria with general and COVID-19 specific features ○ Supported by empirical and quantitative procedure ○ Subjectivity is removed by using CRITIC ○ Hybrid multi criteria decision making support

Thus, the proposed model can be used as guideline for the purposes of evaluation and decision making related to mobile health apps for COVID-19 disease. However, still this study has some limitations described below as.

The features selected cannot be considered as final benchmark or standard because they change from one criteria to other. For example, some authors may include one class of features while others may not be interested in the same set of features.

Security and privacy characteristics of mobile apps require more technical considerations about other security sub-factors such as authentication, authorization, poor encryption, cryptography, less or insecure data storage and data leakage etc.

Conclusion

During the early days of pandemic, many mobile health apps have been developed as a digital healthcare solution in the battle against the COVID-19 outbreak in Pakistan. It created a lot of decision making problems for the mobile users and healthcare professionals. In this regard, many evaluation models have been proposed but majority of them were suffering from certain limitations such as lack of empirical assessment, insufficient assessment criteria and theoretical nature. Therefore, we present evaluation model focused on empirical assessment of COVID-19 mobile apps in Pakistan. The proposed framework works in three steps such as in first step, features-based criteria is defined and twenty (20) mobile apps are scratched from the play stores. In second step, the proposed model leverages the MCDM approaches i.e. CRITIC and TOPSIS for criteria weightage and assessment of mobile health apps. In last stage, ranking is performed based on the results obtained through the TOPSIS approach. According to the results COVID-19 Gov PK app is reckoned as the best mobile apps solution to tackle the current pandemic. As, there does not exist evaluation framework of similar kind so therefore the proposed framework is tested by experts panel based survey; and SAW method has been applied to check the accuracy and consistency of the results. This paper will provide guideline for practitioners and mobile subscribers in practical and real world scenarios. The proposed framework will also allow the mobile developers to enhance the existing features or add new features by designing user-friendly, easy to use, multi-feature support and single point solution.

Our Future work is to enhance framework by including more important mobile apps features and introducing the concept of fuzzy logic in the existing decision making procedure. The security and privacy evaluation of selected COVID-19 mobile apps is also future research work.

Funding

This research is supported by Qatar National Library and Qatar University, Internal Grant IRCC.2021-010.

Declaration of Competing Interest

None.