Uncertain Multiplicative Language Decision Method Based on Group Compromise Framework for Evaluation of Mobile Medical APPs in China.

The mobile medical application (M-medical APP) can optimize medical service process and reduce health management costs for users, which has become an important complementary form of traditional medical services. To assist users including patients choose the ideal M-medical APP, we proposed a novel multiple attribute group decision making algorithm based on group compromise framework, which does not need to determine the weight of decision-maker. The algorithm utilizes an uncertain multiplicative linguistic variable to measure the individual original preference to express the real evaluation information as much as possible. The attribute weight was calculated by maximizing the differences among alternatives. It determined the individual alternatives ranking according to the net flow of each alternative. By solving the 0-1 model with the objective of minimizing the differences between individual ranking, the ultimate group compromise ranking is obtained. Then we took 10 well-known M-medical APPs in Chinese as an example, we summarized service categories provided for users and constructed the assessment system consisting of 8 indexes considering the service quality users are concerned with. Finally, the effectiveness and superiority of the proposed method and the consistency of ranking results were verified, through comparing the group ranking results of 3 similar algorithms. The experiments show that group compromise ranking is sensitive to attribute weight.

1. Introduction

Medical is the basic industry of the country, which is closely related to the health and development of the whole country. The prominent characteristics of the medical industry are strong professionalism and asymmetric information. In recent years, residents have valued healthcare and quality of life due to the disposable income of residents increasing. The degree of population aging has been further deepened in many countries. More convenient and quality medical services are urgently needed for the older generation, who are more vulnerable to disease and likely to contact with the doctor directly. Simultaneously, serious public health incidents occur from time to time, which put great pressure on the normal operation of the medical system, even bring catastrophic damage. Under the influence of these factors, there are still a series of problems in China’s medical care system, such as the imbalance of regional medical resources, the tension between doctors and patients and the low efficiency of medical services. To deal with these problems, the Chinese government has continued to deepen reform and issued programs in the medical sector, for example, medical big data, medical informatics, “Internet + medical” and healthy China 2020, which have made achievements in the transformation and upgrading of the medical industry.

In the past six years, medical resources in urban and rural areas have been expanding. However, the expansion speed of urban medical resources is faster, and the imbalance between urban and rural medical resources is further aggravated. Taking the number of licensed doctors or assistant doctors per 10k people as an example, Figure 1 shows the change of quantity from 2011 to 2017. The gap between urban and rural areas was 17 in 2011 and 23 in 2017. Consequently, it’s necessary to further reform the existing supply structure and business model in China’s medical market, especially in the defense of public health emergency, most of suspected patients cannot get timely diagnosis and treatment in the countryside. Furthermore, the problem of unequal medical resources as a global issue is not only in China but also in other countries. In the United States, with only 14% of practicing primary care physicians providing services to 25% of the rural residents [1].

Figure 1

Number of licensed doctors in urban and rural areas of China from 2011 to 2017. Data from China Bureau of Statistics.

Mobile Internet highlights the advantages of Internet anytime, anywhere, convenient and fast. With the development and maturity of mobile information technology and popularization of intelligent mobile terminal, the scale of mobile Internet users has increased year by year in China, which is presented in Figure 2. It shows that by the end of 2017, the number of mobile Internet users was 752.65 million, and increased by 8.25% compared with 2016 in China.

Figure 2

The change of mobile Internet users from 2011 to 2017 in China. Data from China Internet Network Information Center.

The elderly (age 60+) ask for medical services more frequently and tend to have high-quality medical resources in daily life. Due to the aging of body function, the older population often suffer from one or more chronic diseases and face a health emergency. As Figure 3 shows, there is a growth trend on the number of elderly from 2011 to 2017 in China, in which the number of older Internet users is increasing as well. Fortunately, the growth rate of older Internet users is higher than that of older population. Correspondingly, the popularity rate of Internet in Chinese older population was 16.66% in 2017 and greater than 12.67% in 2016. It indicates that the medical service that is delivered by mobile Internet is gradually known and accepted by the Chinese older generation and has great potential for development in the future.

Figure 3

The popularity rate of Internet in Chinese older population from 2011 to 2017. Data from China Bureau of Statistics and Internet Network Information Center.

Practically, medical resources are tight and a large number of patients gather together when public health emergencies occur, which leads to a vicious circle of the medical system. In order to maintain the normal operation of the whole medical system, medical institutions move some dangerous medical services online. At the same time, they trust the simple and mechanical medical process in the mobile medical applications (M-medical APPs), so as to improve the efficiency of medical services. The advantages of remote diagnosis and Internet hospital have been proved in practice. These Internet-based applications not only reduce cross infection between medical staff and patients, but also balance medical resources.

With the penetration of mobile Internet in rural areas of China, mobile information technology empowers the medical industry, which may be able to effectively solve the contradictions and problems in the medical field [2], and then improve the medical service level of the whole country. For instance, Alihealth realized cross-regional sharing and allocation of existing social medical and health resources. In this climate, practitioners and researchers pay more and more attention to the changes and upgrades of mobile information technologies in the area of medical care. The practice of using mobile technology infrastructure in medical care, especially 5G and smartphone, is termed as Mobile Medical [3,4]. Mobile Medical can improve the quality, safety and efficiency in current healthcare, and provides several new medical services with ubiquitous and mobile devices [2]. Additionally, it has greatly improved the convenience of patients’ medical treatment and drug purchases to meet consumers’ pursuit of a healthy lifestyle. The mobile devices include smartphones, tablets, PDA and laptop computers in common, which are installed Android, Apple or Windows operating systems. Many types M-medical APPs are developed to assist various target groups accessing high-quality medical services anywhere and anytime, since the features of mobile devices and technologies.

M-medical APPs are increasingly important for the health industry [5]. The existence of mobile integrated medical care and community paramedicine programs is reasonable and beneficial to the improvement of local medical and health environments [1]. Mobile medical care has a broad application prospect [6]. Although China’s mobile medical industry is still in its infancy [2], medical providers, technology companies, Internet enterprises, information platforms, etc. have developed mobile medical business and launched unique M-medical APPs. An increasing number of M-medical APPs are developed benefiting health service delivery [2]. As of December 2019, 20 M-medical APPs are active in Apple store in China market. Nevertheless, users encounter difficulty in choosing the appropriate APP when facing the abundant number of applications, because they could not invest an amount of time in determining APPs, and the market is full of commercial information to promote APPs.

However, nowadays people’s daily life has been flooded with numerous mobile applications [7]. In regards to M-medical APP, which APP should ordinary users choose to gain better medical experience? Which performance of mobile medical application should be designed or improved so as to be accepted by the public, for mobile medical companies? Unfortunately, there are few published studies on the evaluation of M-medical APPs, let alone that of China. The previous research around M-medical APP focuses on the following areas: (a) the development or improvement of M-medical APP [2,4,8,9,10]; (b) the attitudes of different groups to mobile medical technology or APP, including technical and medical students, working staff and healthcare professor [2]; (c) classification of M-medical APP and macro analysis of mobile medical industry [11,12]; (d) design or development of specific M-medical APP, such as in oncology [13], in Neurosurgery [14] and in chronic diseases [15]. Therefore, there is a pressing need for devising ways to evaluate the available M-medical APPs effectively.

To fill the gaps, the current study develops the uncertain multiplicative linguistic variable decision-making algorithm based on the existing group decision-making framework. After analyzing the medical services provided by app, the algorithm is utilized to investigate the expert group’s evaluation of M-medical APPs in China. The purpose of the paper includes providing mobile medical service providers and researchers with macro-understanding, and help users select the most appropriate APP. The structure of the paper is arranged as follows. In Section 2, works of literature related to M-medical APPs and evaluation of mobile medical healthcare application are reviewed. Section 3 summarizes the services provided by M-medical APPs. In Section 4, some basic concepts about linguistic terms are introduced. Section 5 designs the algorithm that uncertain multiplicative linguistic decision method based on group compromise ranking framework and analyze the characteristic of the algorithm. Section 6 constructs the assessment system and ranks 10 M-medical APPs using the proposed method. In Section 7, the comparative analysis of 3 similar algorithms is carried out. In Section 8, some conclusions are presented and further studies are discussed.

2. Related Literature

At present, the unified concept of mobile medical application has not been formed yet. Few studies carried out into the aforementioned four areas are as follows: Mark et al. developed the mobile traditional Chinese medicine application of the Android system using mobile devices and cloud services, which improved service efficiency and patient safety [16]. Katz and Rice investigated the attitudes of American adults toward mobile medical technology [6]. Vishnu et al. compared the diagnostic accuracy of the APP against neurology residents in movement disorders and asserted M-medical APP can effectively help doctors diagnose illness [17]. Oluwagbemi et al. developed a mobile application for some hereditary diseases and disorders, and compared the performance of similar APPs through analysis online and offline platform questionnaires [18]. Gabor et al. built a mobile APP that can diagnose users’ diseases. The diagnosis process is divided into two stages: forming a preliminary medical diagnosis by analyzing a series of questions answered by users; providing users with the opportunity to communicate with medical experts for accurate medical diagnosis [19].

On the topic of evaluating/assessing/ranking/ordering M-medical APPs, very limited studies were searched in archived literature. Thus, we reviewed similar research related to mobile healthcare application, due to mobile healthcare contains mobile medical, which has always been concerned with the wave of emerging technology. Mobile Application Rating Scale is often used to evaluate mobile health APP in some healthcare subdivision domains. The evaluation system consists of five dimensions: engagement, functionality, aesthetics, information and subjective quality. Furthermore, each dimension is subdivided into several sub-indicators such as interactivity, performance, ease of use, aesthetics, accuracy, quality, quantity, credibility and evidence based [20]. Some researchers have integrated the Fuzzy Analytic Hierarchy Process (FAHP) and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) into a decision-making approach for ranking alternatives from multiple attributes. In the approach, fuzzy AHP and TOPSIS were utilized to calculate the weight of evaluation attributes and ultimately ranking respectively [21]. Rajak and Shaw suggested a Fuzzy Multiple Attributes Decision Making (FMADM) methodologies, consisting of AHP and fuzzy TOPSIS, for accessing and selecting M-medical APPs. An evaluation structure was constructed considering nine main attributes, i.e., user satisfaction, compatibility, functionality, security, accessibility, easy to learn and use, empathy, information quality, responsiveness [7]. Chen developed a model, namely fuzzy geometric mean-FAHP, to assess the importance of critical factors influencing the application of smart technology to mobile health care [22]. Through interviews, observation, surveys and exams, Hao et al. collected the data of mobile English vocabulary learning APP, and students’ evaluation information [23].

Evaluation is one of the essential parts for decision-making. The purpose of the evaluation is to make scientific decisions. In prior studies, decision-makers were forced to evaluate with real numbers, which limited their cognitive thinking process to a specific range, and more likely to have inaccurate scores [24]. Nevertheless, in practical situations, decision makers tend to express their preferences in discrete language terms, and managers are not very confident in subjective decision-making [25]. To design a more practical evaluation method, it is necessary to integrate the experience, knowledge or opinions from multiple decision-makers [26]. Fuzzy Multiple Attribute Group Decision Making (FMAGDM) method can coordinate the different opinions expressed by the decision-makers in fuzzy evaluation language, and then find a most acceptable alternative for the whole group [27]. The determination of evaluation attribute weight and decision-maker weight plays an important role in the FMAGDM framework. The method of determination directly affects the final evaluation result [28]. It is difficult in eliciting appropriate attribute weights from human experts [29]. In extant literature, the most prevailing methods to calculate attributes’ weights are: Delphi method [30], Analytic Hierarchy Process [31], Network Analysis [32], Entropy Method [33], Simos Procedure [34], Single Objective Optimization Model [35,36,37], Multi-Objective Optimization Model [38]. Govindan et al. have designed a decision-making framework for determining compromise group ranking according to the alternatives’ ranked by each decision maker participated in evaluation activity. The significant advantage of this framework is that it does not need the weight information of decision-makers [34]. Thus, the framework is used as a reference by the designed methods for ranking M-medical APPs in this paper.

3. Mobile Medical Service for Public

We use Internet search, questionnaire interview and other ways to collect 20 public-oriented mobile medical services in the Chinese market and focus on 10 well-known M-medical APPs. Their name and affiliated company information are shown in Table 1.

Table 1

10 well-known M-medical APPs in the Chinese market.

NO.	Name	Affiliated Company
1	PingAn Good Doctor	Ping An Healthcare And Technology Company Limited
2	Alibab Health	Alibaba Group
3	DingXiang Doctor	Yinchuan Dingxiang Internet hospital Co., Ltd.
4	Good Doctor Online	Beijing Interactive Peak Technology Co., Ltd.
5	We Doctor	Guahao (Hangzhou) Technology Co., Ltd.
6	ChunYu Doctor	Beijing Spring Rain Software CO., Ltd.
7	Health 160	Shenzhen Ningyuan Technology Co., Ltd.
8	Micro-relationship	Hangzhou Choice Technology CO., Ltd.
9	Medical Consultation Rapidly	Hainan health cloud Internet hospital Co., Ltd.
10	Access to Medical	Daoyitong.com, Inc.

It is not always the patients who download and use the APP. To some extent, it is the person who care about their health or have health needs, usually called users. These mobile medical companies help countless users access to full-time and full-range medical and health services conveniently. Through the comprehensive analysis of the function modules of these 10 APPs, we summarized the user-oriented medical and health services of China mobile medical applications. M-medical APP provides users with one-stop health and medical services including 7*24-h online inquiry, consultation, interrogation, registration, drug purchase and other medical services, where consultation is the most important service for users.

3.1. Information Inquiry

Using M-mobile APPs, users can browse or understand health information independently, such as how to develop healthy living habits. Especially, during public health emergency, users can get timely information on the progress and protection of the epidemic. At the same time, users with medical needs can quickly query the schedule, charging and user evaluation of the corresponding hospital or doctor.

3.2. Consult with Doctor

For some mild or chronic diseases, patients need not to go to hospital immediately. They use M-mobile APPs to communicate with experts in the form of graphics, voice or video, so as to obtain professional advice or treatment programs. Some patients with common diseases can be diagnosed online. Different ways of communication correspond to different costs. Usually, video consultation is more expensive.

3.3. Registration and Appointment

After consulting doctors online, patients with more serious illnesses may need to be treated in offline hospitals. In this case, based on the M-mobile APPs, users can directly register online and appoint experts. Most of APPs registration fees are similar to offline registration fees, while the registration fees of few are significantly higher than that of offline. There is a significant difference in the cost of appointing experts of different levels or popularity, but it is generally more expensive.

3.4. Electronic Prescription

After fully mastering the patient’s condition, the consulted doctor issues an electronic prescription for the patient on the APP. According to the e-prescription, patients can not only buy drugs in the APP’s health mall, but also buy drugs in offline pharmacies or hospitals, which may be owned by mobile medical enterprises. The drugs can be directly delivered to the patients. It is no doubt that APP that can form electronic prescriptions or sell drugs to patients must be qualified.

3.5. Other Services

Other services of M-mobile APPs include two parts: one is free services such as health monitoring and health management; the other is charging services such as private doctors and disease management.

4. Preliminaries

In this part, a few fundamental concepts of the multiplicative linguistic evaluation scale, uncertain multiplicative linguistic variable and the superiority are introduced. We give the definition that the distance between any two UMLVs.

The scale of language evaluation is the reference scale for decision-makers to carry out qualitative evaluation. In the real situation, the decision-makers usually adopt the qualitative evaluation alternatives such as “good”, “not bad” and “general”, which are called as linguistic terms where

Supposed

Let

For any two UMLVs

;

.

Supposing

;

;

.

Let

;

If and only if

If and only if

If and only if

If

Let

5. Uncertain Multiplicative Linguistic Decision Method

In this section, the process of individual ranking and group ranking is described in detail, and a general algorithm for group evaluation is formed.

The general fuzzy multi-attribute group decision-making problem is described as: there are n alternatives, m evaluation attributes and h decision-makers. The alternative set, evaluation attribute set, decision-maker set and decision group set are expressed as , , . Simultaneously, represents the weight corresponding to the j-th attribute under preferences of k-th decision maker. The attribute weight vector of under k-th decision maker is recorded as . Generally, the individual preference is implied in individual language evaluation information.

5.1. Individual Alternatives Ranking

5.1.1. Individual Decision Matrix

Referring to a given MLES, the k-th decision-maker’s uncertainty language information of the j-th attribute on the i-th alternative is noted as . Then, the k-th decision-maker’s evaluation matrix is constructed as follows:

The larger the evaluation value of some attributes is, the more favorable for the superiority of alternatives is. They are called positive attributes or benefit attributes. On the contrary, they are called negative attributes or cost attributes. For the cost attributes, we use the reciprocal operator to normalize the language evaluation information. For example, is normalized as . We defined that is the result of normalized .

5.1.2. Individual Attributes’ Weights

From the perspective of information theory, if the evaluation value of alternatives under a certain attribute is similar, then the attribute is difficult to distinguish alternatives significantly, so the attribute should be given a smaller weight value [37].

According to the definition of , the total distance between and other alternatives under is obtained as follows:

Furtherly, the total distance among alternatives under each evaluation attribute is represented by:

Therefore, the optimal attribute weight should meet the following single objective optimization model that maximize the differences among alternatives.

We introduce Lagrange parameter to transform the optimization model into Lagrange function .

According to the extremum theorem of continuous function, the best attribute weight must satisfy the following necessary conditions.

The solution is as follows:

Therefore, is the extremum point and also the optimal point, because the is a continuous differentiable convex function in feasible region. We normalize w as follows:

The whole superiority between alternatives require a comprehensive comparison of the superiority of each attribute. Let

Let

Let

if

if

5.2. Group Compromise Ranking

Individual ranking only represents own preference for alternatives and is an important basis for final group decision-making. Group compromise ranking refers to the alternatives ranking with the smaller total distance from the ranking given by each decision maker [34], which is represented by . It can reflect the tendency of group selection.

Let

Then, the optimization model of minimizing the total distance among rankings is constructed. The decision variable is the priority relationship between any two alternatives and in group ranking, which is 0–1 variable. If is true in optimal group ranking, ; otherwise, . If is true in optimal group ranking, ; otherwise, .

In the optimal model, condition 1 limits and cannot occur at the same time. Condition 2 limits that only one case among , and can be true. Condition 3 makes strict priority relation and strict equivalence relation satisfy transitivity.

5.3. UMLDM Algorithm

Firstly, the alternatives, evaluation attributes, decision-makers and MLES are given. Secondly, the evaluation information of each decision-maker is collected to form the corresponding evaluation matrix. Thirdly, an optimization model is constructed to maximize the difference between alternatives, which is transformed into an unconstrained Lagrange function, and the preference attribute weights of each decision-maker is calculated. Then, alternatives’ positive flow, negative flow and net flow under decision makers’ evaluate are determined. At the same time, according to the value of net flow, the individual ranking is calculated. Finally, we minimize the total distance between individual rankings so as to obtain the group compromise ranking. The algorithm is named Uncertain Multiplicative Linguistic Decision Method based on Group Compromise Ranking Framework, abbreviated as UMLDM. The pseudocode of UMLDM is shown in Table 3.

Table 3

The pseudocode of uncertain multiplicative linguistic decision method.

Input: individual evaluate matrix, and Table 2.

1. Using reciprocal operator to normalize A˜_k and gaining matrix D¯_k, where k=1,2,⋯,h;2. let dd_1=zeros(1,h) and dd_2=zeros(m,h). They are used to record the total and j-th attribute corresponding to distance between alternatives under k-th decision maker respectively;3. for k, from 1 to h for j, from 1 to m  for i, from 1 to n   for v, from 1 to n and v≠i    dd_1(1,k)=dd_1(1,k)+dd(s˜_i,j,k,s˜_v,j,k), dd_2(j,k)=dd_2(j,k)+dd(s˜_i,j,k,s˜_v,j,k);   end for v  end for i end for jend for k4. According to Equation (7), w¯ is calculated by dd_2/(dd_1)^T;5. for k, from 1 to h According to Equation (9), we get H_k(X_v) of any alternative X_v and give rankingT^k;end for k6. for v, from 2 to n for i, from 1 to v-1  for k, from 1 to h   taking Table 1 and T^k to Equation (10) and calculating x_iv  end for k end for iend for v

Output: group compromise ranking T

From Table 2, it can be seen that the algorithm is composed of three loops, and correspondingly basic operation execution times are , and , respectively. Therefore, the time complexity of the algorithm is .

Table 2

Distance between different alternatives’ ranking.

	X_i≻X_v	X_i∼X_v	X_v≻X_i
X_i≻X_v	0	2	4
X_i∼X_v	2	0	4
X_v≻X_i	4	2	0

6. Empirical Analysis

The proposed group evaluation algorithm is used to rank 10 well-known M-medical APPs, according to the experts’ uncertain preference information in the form of UMLVs collected by questionnaire.

6.1. The Assessment Indicator System

Before evaluating the alternatives, it is necessary to conduct a scientific and effective assessment indicator system. In the application of FMAGDM, the assessment criteria are usually determined by literature analysis and expert group discussion [7,21,40].

Users concerns various attributes of M-medical APPs. We discussed with 100 users of M-medical APPs to construct an applicable evaluation system. According to common indicators of published studies on APP evaluation [7,20,21,41], and users’ special needs for mobile medical services, we finally determined the assessment indicator system. The system consists of 8 evaluation indicators, where response time and price are cost attributes.

Professionalism (): It means the professional level of mainstream applications such as online consultation, chronic disease management and other medical services. The operating company behind the M-medical APP has a mature and professional team, including medical experts, technical personnel, especially its own medical team, offline clinics, pharmacies and other physical facilities.

Response time (): Response time indirectly affects user satisfaction. It reflects the average waiting time of users from consulting experts to getting responses. Some M-medical APPs lack of expert resources, low professional quality of doctors and imperfect supervision system, which leads to long application response time.

User-friendly (): The interface of M-medical APP shall be simple, beautiful and the functional modules shall be clear. It includes simplifying the operation process of the target user, providing multiple consultation methods and other friendly settings.

Price (): Users need to pay for medical services on the App. On the one hand, users must pay a fixed fee when purchasing drugs and devices on the App. on the other hand, users should pay different fees according to the professional level, consulting time and consulting methods when online consulting.

Additional services (). Additional services refer to free non-medical services provided by APP for users, such as drug distribution and free physical examination.

Security (): The personal information, disease information, consultation record and health monitoring data uploaded by users through M-medical APP are all private data. The application platform should adopt technical approaches and improve the confidentiality of operating system to protect the user’s privacy data.

User scale (): The user scale refers to the number of active users on the APP, which shows the comprehensive performance and popularity of the APP to a certain extent.

Reliability (): The capital, potential and reputation of the company to which the mobile medical APP belongs affect the reliability of the APP. In the early stage of mobile medical business, APP revenue is very limited, and the company has enough capital flow to maintain the normal operation of APP.

6.2. Questionnaire Design

Before evaluating activity, researcher ought to combine the characteristics of specific decision-making problems and the appropriate scale of language evaluation to design questionnaire and collect assessment information. Experts use evaluation language to express their intuitive perception of APP. These languages are ambiguous and uncertain. Considering that MLs can retain the original evaluation information to the greatest extent. Non-uniform MLES to is employed, the MLES is noted as in this paper.

Based on the given evaluation system and language measure scale, we design an evaluation questionnaire for 10 M-medical APPs as Table 1 shown, noted by correspondingly. They large number of users in the current market and have their own characteristics and advantages, for example, has friendly interface and performance. Then, we push the questionnaire to the mobile phones of 5 decision-makers. Some topics of the designed questionnaire are shown in Figure 4. By collecting the questionnaire, we can get the language evaluation information of 5 experts, see Appendix A.

Figure 4

Some topics in the designed questionnaire.

6.3. Assessment Process

Step1: we normalized the original language evaluation matrix of experts to form a normalization individual evaluation matrix. just is taken as an example in the paper, it is shown as Table 4.

Table 4

The normalized evaluation information matrix of .

D_1	Y_1	Y_2	Y_3	Y_4	Y_5	Y_6	Y_7	Y_8
X_1	[s_2,s_4]	[s_1/2,s_1/3]	[s_1,s_2]	[s_1/3,s_1/2]	[s_2,s_2]	[s_3,s_4]	[s_3,s_4]	[s_3,s_3]
X_2	[s_1/3,s_1/2]	[s_5,s_5]	[s_1,s_1]	[s_1/3,s_1/3]	[s_1,s_1]	[s_4,s_4]	[s_1/2,s_1]	[s_5,s_5]
X_3	[s_4,s_4]	[s_3,s_5]	[s_1/2,s_1]	[s_1/5,s_1/3]	[s_3,s_4]	[s_4,s_5]	[s_1,s_1]	[s_1,s_2]
X_4	[s_4,s_5]	[s_1/2,s_1/2]	[s_4,s_4]	[s_1/5,s_1/5]	[s_2,s_3]	[s_3,s_3]	[s_1,s_2]	[s_2,s_3]
X_5	[s_3,s_5]	[s_1,s_2]	[s_2,s_4]	[s_1/5,s_1/4]	[s_2,s_4]	[s_2,s_2]	[s_3,s_5]	[s_1,s_1]
X_6	[s_2,s_3]	[s_2,s_3]	[s_1/3,s_1]	[s_1/4,s_1/3]	[s_1/3,s_1/3]	[s_4,s_5]	[s_1,s_3]	[s_1/2,s_1]
X_7	[s_3,s_4]	[s_1/2,s_1]	[s_3,s_4]	[s_1/2,s_1]	[s_1,s_2]	[s_1,s_1]	[s_4,s_5]	[s_2,s_2]
X_8	[s_2,s_2]	[s_1/4,s_1/3]	[s_1/4,s_1]	[s_3,s_3]	[s_1/3,s_1/2]	[s_1/2,s_1/2]	[s_2,s_3]	[s_3,s_5]
X_9	[s_1,s_2]	[s_1,s_1]	[s_2,s_3]	[s_1/4,s_1/4]	[s_1/2,s_1]	[s_1,s_2]	[s_3,s_3]	[s_1/2,s_1/2]
X_10	[s_1,s_1]	[s_1/4,s_1/4]	[s_2,s_2]	[s_1,s_2]	[s_1/5,s_1/4]	[s_1/2,s_1]	[s_1/4,s_1/3]	[s_1/3,s_1/2]

Step 2: According to the Pseudocodes 3–4, the attributes of the weights under 5 experts’ assessment are calculated in the Table 5. It can be seen from the Table that experts 1 and 5 both attach importance to the professionalism of M-medical APPs, and the remaining three decision-makers are more inclined to the user-friendly, additional services and user scale of APP respectively.

Table 5

The attribute’s weight of 5 experts.

	D_1	D_2	D_3	D_4	D_5
Y_1	0.1826	0.1342	0.1076	0.1336	0.1551
Y_2	0.0853	0.1361	0.1205	0.1056	0.1060
Y_3	0.1525	0.1410	0.1135	0.0895	0.1354
Y_4	0.1279	0.1314	0.1115	0.0929	0.0949
Y_5	0.1170	0.1130	0.1493	0.1668	0.1050
Y_6	0.0905	0.1186	0.1346	0.1178	0.1488
Y_7	0.1415	0.1056	0.1410	0.1773	0.1169
Y_8	0.1026	0.1202	0.1220	0.1164	0.1379

Remarks: the bold indicates the maximum value in the column.

Step 3: for 5 participants, the calculation result of positive flow , negative flow and net flow of 5 alternatives are shown in the Table 6 and Table 7.

Table 6

The positive, negative and net flow of for .

	D_1			D_2			D_3
	H1+	H1−	H_1	H2+	H2−	H_2	H3+	H3−	H_3
X_1	0.5936	0.3674	0.2261	0.5930	0.3314	0.2616	0.5963	0.3641	0.2322
X_2	0.3823	0.5586	−0.1763	0.5245	0.4531	0.0714	0.4806	0.4847	−0.0042
X_3	0.5540	0.4136	0.1404	0.5121	0.4139	0.0982	0.5875	0.3643	0.2232
X_4	0.6003	0.3455	0.2548	0.5240	0.4288	0.0952	0.6172	0.3455	0.2718
X_5	0.6403	0.3404	0.2999	0.6403	0.3337	0.3066	0.5224	0.4012	0.1212
X_6	0.4593	0.5138	−0.0545	0.3388	0.6351	−0.2964	0.3717	0.6161	−0.2444
X_7	0.6480	0.3327	0.3153	0.7132	0.2262	0.4870	0.6602	0.2617	0.3985
X_8	0.3054	0.6673	−0.3619	0.3332	0.6032	−0.2700	0.2983	0.6798	−0.3815
X_9	0.4037	0.5505	−0.1468	0.3138	0.6358	−0.3221	0.4244	0.5392	−0.1148
X_10	0.2304	0.7275	−0.4971	0.2658	0.6974	−0.4316	0.2057	0.7078	−0.5021

Table 7

The positive, negative and net flow of for .

	D_4			D_5
	H4+	H4−	H_4	H5+	H5−	H_5
X_1	0.5155	0.4589	0.0566	0.5034	0.4453	0.0581
X_2	0.4983	0.4613	0.0370	0.5049	0.4623	0.0426
X_3	0.5449	0.4145	0.1304	0.6609	0.3055	0.3555
X_4	0.5955	0.3639	0.2316	0.6100	0.3239	0.2861
X_5	0.6205	0.3561	0.2645	0.5643	0.4156	0.1488
X_6	0.4167	0.5133	−0.0967	0.3384	0.6393	−0.3008
X_7	0.6286	0.3488	0.2798	0.6826	0.2941	0.3886
X_8	0.3832	0.5852	−0.2020	0.3490	0.6142	−0.2652
X_9	0.4579	0.5098	−0.0519	0.4213	0.5399	−0.1185
X_10	0.1482	0.7975	−0.6493	0.1630	0.7581	−0.5951

Step 4: Based on net flow of each mobile medical APP, 10 APPs Rankings of 5 decision makers are given in Table 8.

Table 8

APPs rankings of 5 exports.

Position	D_1	D_2	D_3	D_4	D_5
1	X_7	X_7	X_7	X_7	X_7
2	X_5	X_5	X_4	X_5	X_3
3	X_4	X_1	X_1	X_4	X_4
4	X_1	X_3	X_3	X_3	X_5
5	X_3	X_4	X_5	X_1	X_1
6	X_6	X_2	X_2	X_2	X_2
7	X_9	X_8	X_9	X_9	X_9
8	X_2	X_6	X_6	X_6	X_8
9	X_8	X_9	X_8	X_8	X_6
10	X_10	X_10	X_10	X_10	X_10

Step 5: we only consider the case where there is a priority relationship between APPs. By calculating the 0-1 optimal model, we can obtain in Table 9.

Table 9

The priority relationship between APPs.

X_iv	X_1	X_2	X_3	X_4	X_5	X_6	X_7	X_8	X_9	X_10
X_1	-	1	0	0	0	1	0	1	1	1
X_2	0	-	0	0	0	1	0	1	1	1
X_3	1	1	-	0	0	1	0	1	1	1
X_4	1	1	1	-	0	1	0	1	1	1
X_5	1	1	1	1	-	1	0	1	1	1
X_6	0	0	0	0	0	-	0	1	0	1
X_7	1	1	1	1	1	1	-	1	1	1
X_8	0	0	0	0	0	0	0	-	0	1
X_9	0	0	0	0	0	1	0	1	-	1
X_10	0	0	0	0	0	0	0	0	0	-

Finally, the groups compromise ranking of 10 APPs is

The total distance between group and individual ranking is 144. Therefore, the expert group is more inclined to the health 160 mobile medical application that has been highly recognized by the National Health Planning Commission, the Health Administration and other health administrative departments. 5 experts’ evolution and groups compromise assessment on 10 M-medical APPs of China are represented in Figure 5. It indicates that there are significant differences among experts’ preference for 10 given APPs, and the ultimate APP ranking coincides with the personal ranking of decision-maker 4.

Figure 5

The individual ranking and group ranking for 10 M-medical APPs.

7. Comparison Analysis

Entropy Weight (EW) is also a commonly used method to determine the weight from the perspective of information theory [33]. To verify the effectiveness and superiority of proposed method, the ranking of the alternatives calculated by UML-TOPSIS, UML-TOPSIS-EW and UMLDM-EW are compared with that of UMLDM (see Figure 6a). Similar to UMLDM, TOPSIS [7,21] is utilized for individual alternatives ranking, while other processes remain unchanged, which is named ULM-TOPSIS, see Figure 6b. Both ULM-TOPSIS-EW (see Figure 6c) and UMLDM-EW (see Figure 6d) are group decision-making algorithms, in which the corresponding attribute weight of each decision-maker is determined through the entropy weight method, according to ULM-TOPSIS and UMLDM, respectively. Other than that, we analyze the sensitivity of evaluation results to attribute weights calculated by two methods (Single objective optimization and entropy weight method).

Figure 6

The procedure of 4 similar algorithms based on group compromise framework.

Information integration operator can integrate personal evaluation information into group decision preference. Let where [a,b] is interval number, and

Let here,

7.1. UML-TOPSIS

7.1.1. Deterministic ML Evaluation Matrix

The individual assessment information matrix based on UML is transformed into the deterministic ML evaluation matrix by operator. Five experts’ deterministic ML evaluation matrix, noted as, , respectively, are calculated in Appendix B.

Let

If

Let where

Let where

7.1.2. Individual Alternatives Ranking Based on TOPSIS

In this section, we calculate the ranking of 10 M-medical APPs for each expert based on from definitions 14 to 16, shown in Figure 7. The comprehensive deviation from attribute vector of each APPs to positive ideal point and negative ideal point, and alternative’s closeness to positive ideal point are given in Appendix C. Finally, according to Equation (10) and Table 2, the group compromise ranking of 10 M-medical APPs is obtained: and the total distance among group is calculated as 160. Additionally, the group is just same as the individual ranking of . The time complexity of the UML-TOPSIS algorithm also is .

Figure 7

The individual ranking and group ranking for 10 mobile medical APPs based on UML-TOPSIS.

Let where

7.2. UML-TOPSIS-EW

7.2.1. Entropy Weight Method

Based on Definition 18, the is determined with the following expression:

Table 10 shows the attribute’s weight of 5 experts calculated by entropy method, which is significantly different from Table 5. In regard to individual preference, response time is most valued by , additional services are emphasized by , and price is most important for others.

Table 10

The attribute’s weight of 5 experts calculated by entropy method.

	D_1	D_2	D_3	D_4	D_5
Y_1	0.0701	0.1181	0.1073	0.1281	0.1254
Y_2	0.1606	0.1259	0.1144	0.1076	0.1026
Y_3	0.1236	0.1144	0.1169	0.1187	0.0804
Y_4	0.1595	0.1546	0.1237	0.1876	0.2057
Y_5	0.1401	0.1429	0.1453	0.1203	0.1362
Y_6	0.1012	0.1292	0.1429	0.1452	0.1572
Y_7	0.1086	0.0974	0.1304	0.1128	0.0999
Y_8	0.1364	0.1175	0.1191	0.0797	0.0926

Remarks: The bold indicates the maximum value in the column.

7.2.2. The Group Ranking of Mobile Medical APPs

We use the individual attribute weight in Table 9 to replace the that in UML-TOPSIS. The rules of individual ranking and group decision are the same as UML-TOPSIS. The comprehensive deviation from to and , and each alternative closing to positive ideal APP are calculated in Appendix D. The individual ranking and for 10 M-medical APPs based on UML-TOPSIS-EW are presented in Figure 8. We obtain the group compromise ranking:

Figure 8

The individual ranking and group ranking for 10 mobile medical APPs based on UML-TOPSIS-EW.

The tendency of the group is exactly the same as that of . The corresponding total distance is 176. The time complexity of the algorithm is .

7.3. UMLDM-EW

Except for the calculation approach of alternative’s weight, the determination procedure of group ranking is completely consistent with UMLDM. Under expert’s language assessment information, we calculate the positive flow, the negative flow and the net flow of 10 M-medical APPs, as shown in Appendix E. The group compromise ranking is obtain:

It is consistent with the ranking of . The total distance among rankings is 192. Figure 9 presents 5 expert’s rankings based on UMLDM-EW. The time complexity of the algorithm is

Figure 9

The individual ranking and group ranking for 10 mobile medical APPs based on UMLDM-EW.

7.4. Discussion

The group rankings of 10 M-medical APPs determine by 4 FMAGDM based on group compromise ranking framework are shown in Figure 10. Through comparison analysis, it concludes that (a) in general, the M-medical APPs ranking obtained by UMLDM, UML-TOPSIS, UML-TOPSIS-EW and UMLDM-EW are utterly similar provably, with Health 160 APP at the first, Micro-relationship APP at ninth and Access to Medical APP at the tenth; (b) the total distance among individual APP rankings increase in turn(144, 160, 176, 192) corresponding to 4 algorithms, where the algorithm has a larger total distance with applying entropy method to determine attribute weight, which means the group evaluation result is sensitive to the attribute weight; (c) the time complexity of UMLDM and UML-TOPSIS are , while that of UML-TOPSIS-EW and UMLDM-EW .

Figure 10

The group rankings of 10 mobile medical APPs determined by 4 FMAGDM based on group compromise ranking framework.

After carried out comparison analysis, it indicates that the APP ranking calculated by UMLDM algorithm is highly consistent with that calculated by other 3 algorithms. The proposed algorithm can effectively solve FMAGDM problems based on individual language evaluation information. Although UMLDM algorithm is with higher time complexity, it can reduce the loss of evaluation information to some extent and has a better group compromise ranking.

8. Conclusions

With the improvement of living standards, residents pay more and more attention to their own health in China. Typically, they are eager to obtain high-quality medical resources conveniently and quickly and manage some chronic diseases by themselves. Mobile medicine developed with the advantages of mobile technology can not only meet people’s medical needs, but also effectively alleviate some medical dilemmas in China. In this environment, mobile medical applications with different characteristics are full of people’s daily life. M-medical APP can help patients reduce the decision-making cost, time cost and economic cost of consulting doctor and maintain health anywhere and anytime. Thus, it is vital for both users and relevant companies that how to choose the most suitable M-medical application for the masses in China’s existing market.

However, decision-makers are more inclined to use language terms to express themselves preference for alternatives in reality. Few studies have been carried out on evaluating mobile APPs from the perspective of UMLV, especially mobile medical APPs in China. To fill the gap, we firstly summarized the types of service provided by mobile medical APP for users. Then, a new multiple attributes group decision making algorithm considering uncertain multiplicative linguistic variable was designed, which is based on the group compromise ranking. The algorithm utilized UMLV to measure the decision-maker’s original preference information and the distance between any two UMLVs was defined, which contains three phases: the attribute weight calculated through maximizing the difference among alternatives; the APP ranking of each decision-maker determined based on its net flow; the group compromise ranking given by minimizing the total distance between individual ranking. It was used to rank 10 relatively well-known APPs based on the user-oriented assessment system including 8 indicators, and the calculation result was that Health 160 We Doctor Good Doctor Online DingXiang Doctor PingAn Good Doctor Alibab Health Medical Consultation Rapidly ChunYu Doctor Micro-relationship Access to Medical. Finally, the proposed algorithm is more effective and superior in evaluating mobile medical APPs and sensitive to the attribute weight, through the comparison and analysis of the ranking results determined by other similar 3 algorithms.

There are still some limitations in this study: (a) Uncertain multiplicative linguistic variables cannot measure the decision maker’s evaluation information completely. For this, intuitionistic fuzzy sets [42] and hesitant fuzzy sets [43] are usually utilized to collect real assessment information as much as possible. (b) The constructed evaluation system does not consider the information overlap effect between indicators. It is a better way that principal component analysis used to simplify the index system. (c) The designed evaluation framework can be further developed into a part of knowledge management system of smart hospital [44]. (d) The business model of mobile medical needs further development and innovation, so as to promote its widespread adoption among the public. These limitations are also the further research direction of this study.

Table A1

Linguistic evaluation information matrix of .

D_1	Y_1	Y_2	Y_3	Y_4	Y_5	Y_6	Y_7	Y_8
X_1	[s_2,s_4]	[s_2,s_3]	[s_1,s_2]	[s_2,s_3]	[s_2,s_2]	[s_3,s_4]	[s_3,s_4]	[s_3,s_3]
X_2	[s_1/3,s_1/2]	[s_1/5,s_1/5]	[s_1,s_1]	[s_3,s_3]	[s_1,s_1]	[s_4,s_4]	[s_1/2,s_1]	[s_5,s_5]
X_3	[s_4,s_4]	[s_1/5,s_1/3]	[s_1/2,s_1]	[s_3,s_5]	[s_3,s_4]	[s_4,s_5]	[s_1,s_1]	[s_1,s_2]
X_4	[s_4,s_5]	[s_2,s_2]	[s_4,s_4]	[s_5,s_5]	[s_2,s_3]	[s_3,s_3]	[s_1,s_2]	[s_2,s_3]
X_5	[s_3,s_5]	[s_1/2,s_1]	[s_2,s_4]	[s_4,s_5]	[s_2,s_4]	[s_2,s_2]	[s_3,s_5]	[s_1,s_1]
X_6	[s_2,s_3]	[s_1/3,s_1/2]	[s_1/3,s_1]	[s_3,s_4]	[s_1/3,s_1/3]	[s_4,s_5]	[s_1,s_3]	[s_1/2,s_1]
X_7	[s_3,s_4]	[s_1,s_2]	[s_3,s_4]	[s_1,s_2]	[s_1,s_2]	[s_1,s_1]	[s_4,s_5]	[s_2,s_2]
X_8	[s_2,s_2]	[s_3,s_4]	[s_1/4,s_1]	[s_1/3,s_1/3]	[s_1/3,s_1/2]	[s_1/2,s_1/2]	[s_2,s_3]	[s_3,s_5]
X_9	[s_1,s_2]	[s_1,s_1]	[s_2,s_3]	[s_4,s_4]	[s_1/2,s_1]	[s_1,s_2]	[s_3,s_3]	[s_1/2,s_1/2]
X_10	[s_1,s_1]	[s_4,s_4]	[s_2,s_2]	[s_1/2,s_1]	[s_1/5,s_1/4]	[s_1/2,s_1]	[s_1/4,s_1/3]	[s_1/3,s_1/2]

Table A2

Linguistic evaluation information matrix of .

D_2	Y_1	Y_2	Y_3	Y_4	Y_5	Y_6	Y_7	Y_8
X_1	[s_1,s_1]	[s_1,s_1]	[s_1,s_1]	[s_1/4,s_1/2]	[s_2,s_3]	[s_4,s_4]	[s_3,s_3]	[s_3,s_3]
X_2	[s_1/3,s_1/2]	[s_1/5,s_1/3]	[s_1/2,s_1]	[s_1/2,s_1]	[s_1/2,s_1]	[s_5,s_5]	[s_1/3,s_1/3]	[s_3,s_4]
X_3	[s_2,s_2]	[s_1/5,s_1/5]	[s_1/3,s_1/2]	[s_3,s_3]	[s_3,s_3]	[s_2,s_3]	[s_1/2,s_1/2]	[s_1,s_1]
X_4	[s_4,s_4]	[s_1/2,s_1]	[s_1,s_3]	[s_4,s_5]	[s_1/3,s_1/3]	[s_2,s_2]	[s_1,s_1]	[s_4,s_5]
X_5	[s_3,s_4]	[s_1/3,s_1/2]	[s_3,s_3]	[s_3,s_4]	[s_3,s_4]	[s_1,s_1]	[s_4,s_5]	[s_1/2,s_1]
X_6	[s_1,s_2]	[s_1,s_2]	[s_1/2,s_1/2]	[s_2,s_3]	[s_1/4,s_1/3]	[s_1/2,s_1]	[s_1,s_2]	[s_1/2,s_1/2]
X_7	[s_2,s_4]	[s_1/2,s_1/2]	[s_4,s_5]	[s_1/2,s_1/2]	[s_1,s_1]	[s_1,s_2]	[s_4,s_4]	[s_4,s_4]
X_8	[s_1/2,s_1/2]	[s_2,s_2]	[s_1/3,s_1/3]	[s_1/4,s_1/3]	[s_1/3,s_1/2]	[s_1/4,s_1/4]	[s_2,s_2]	[s_1,s_2]
X_9	[s_1/4,s_1/2]	[s_1/5,s_1/4]	[s_4,s_4]	[s_4,s_4]	[s_1/2,s_1/2]	[s_1/2,s_1/2]	[s_2,s_3]	[s_1/3,s_1/2]
X_10	[s_1/4,s_1/4]	[s_2,s_3]	[s_2,s_3]	[s_1/5,s_1/4]	[s_1/4,s_1/4]	[s_1/3,s_1/2]	[s_1/3,s_1/2]	[s_1/3,s_1/3]

Table A3

Linguistic evaluation information matrix of .

D_3	Y_1	Y_2	Y_3	Y_4	Y_5	Y_6	Y_7	Y_8
X_1	[s_2,s_3]	[s_1/3,s_1/2]	[s_1/3,s_1/2]	[s_1,s_1]	[s_2,s_2]	[s_3,s_3]	[s_2,s_3]	[s_4,s_5]
X_2	[s_1/3,s_1/2]	[s_1/2,s_1]	[s_1/3,s_1/3]	[s_1/2,s_1/2]	[s_1,s_2]	[s_3,s_5]	[s_1/4,s_1/3]	[s_5,s_5]
X_3	[s_2,s_2]	[s_1/4,s_1/4]	[s_1,s_1]	[s_1,s_2]	[s_4,s_5]	[s_4,s_5]	[s_1,s_1]	[s_1/2,s_1]
X_4	[s_4,s_5]	[s_1/2,s_1/2]	[s_4,s_5]	[s_3,s_3]	[s_3,s_4]	[s_1,s_2]	[s_2,s_2]	[s_2,s_3]
X_5	[s_3,s_3]	[s_2,s_2]	[s_2,s_3]	[s_2,s_2]	[s_4,s_4]	[s_1,s_1]	[s_3,s_3]	[s_1,s_2]
X_6	[s_1,s_2]	[s_1,s_2]	[s_1,s_2]	[s_1/2,s_1]	[s_1/3,s_1]	[s_1/2,s_1]	[s_1/3,s_1]	[s_1,s_1]
X_7	[s_3,s_4]	[s_1/3,s_1/3]	[s_3,s_3]	[s_1/3,s_1/2]	[s_1,s_1]	[s_2,s_2]	[s_4,s_4]	[s_2,s_2]
X_8	[s_1,s_1]	[s_2,s_3]	[s_1/4,s_1/3]	[s_1/3,s_1/3]	[s_1/4,s_1/3]	[s_1/5,s_1/3]	[s_1/2,s_1]	[s_3,s_5]
X_9	[s_1/2,s_1/2]	[s_1/4,s_1/3]	[s_3,s_4]	[s_3,s_4]	[s_1/2,s_1]	[s_1/3,s_1/3]	[s_3,s_5]	[s_1/2,s_1/2]
X_10	[s_1/3,s_1/3]	[s_3,s_3]	[s_2,s_2]	[s_1/4,s_1/4]	[s_1/3,s_1/3]	[s_1/3,s_1/2]	[s_1/4,s_1/4]	[s_1/3,s_1/3]

Table A4

Linguistic evaluation information matrix of .

D_4	Y_1	Y_2	Y_3	Y_4	Y_5	Y_6	Y_7	Y_8
X_1	[s_1,s_2]	[s_1,s_2]	[s_1/2,s_1/2]	[s_1/2,s_1]	[s_1,s_1]	[s_1,s_2]	[s_2,s_3]	[s_2,s_3]
X_2	[s_1/4,s_1/2]	[s_1/5,s_1/5]	[s_1/3,s_1/3]	[s_2,s_2]	[s_1,s_2]	[s_3,s_4]	[s_1/3,s_1]	[s_5,s_5]
X_3	[s_2,s_4]	[s_1/3,s_1/3]	[s_2,s_2]	[s_3,s_3]	[s_3,s_5]	[s_4,s_4]	[s_1/3,s_1/2]	[s_1,s_2]
X_4	[s_4,s_5]	[s_1,s_1]	[s_4,s_4]	[s_5,s_5]	[s_2,s_5]	[s_2,s_2]	[s_1/2,s_1]	[s_3,s_4]
X_5	[s_3,s_4]	[s_1/2,s_1]	[s_3,s_4]	[s_4,s_5]	[s_4,s_5]	[s_1,s_1]	[s_2,s_5]	[s_2,s_2]
X_6	[s_1,s_1]	[s_1/2,s_1/2]	[s_1,s_2]	[s_1,s_2]	[s_2,s_2]	[s_1/3,s_1/2]	[s_1,s_1]	[s_1,s_1]
X_7	[s_4,s_4]	[s_1/5,s_1/3]	[s_2,s_3]	[s_1,s_1]	[s_1/2,s_1]	[s_1/2,s_1]	[s_3,s_5]	[s_2,s_4]
X_8	[s_1/2,s_1/2]	[s_2,s_3]	[s_1/4,s_1/4]	[s_1/4,s_1/4]	[s_1/3,s_1/2]	[s_1/4,s_1/2]	[s_2,s_4]	[s_4,s_5]
X_9	[s_1/2,s_1]	[s_1/5,s_1/4]	[s_3,s_3]	[s_3,s_4]	[s_1/3,s_1]	[s_1/2,s_1/2]	[s_4,s_5]	[s_1/2,s_1/2]
X_10	[s_1/3,s_1/2]	[s_2,s_2]	[s_1/3,s_1/2]	[s_1/3,s_1/3]	[s_1/3,s_1/3]	[s_1/4,s_1/3]	[s_1/3,s_1/3]	[s_1/2,s_1]

Table A5

Linguistic evaluation information matrix of .

D_5	Y_1	Y_2	Y_3	Y_4	Y_5	Y_6	Y_7	Y_8
X_1	[s_1,s_1]	[s_1/3,s_1/2]	[s_1/3,s_1/3]	[s_1/2,s_1]	[s_2,s_2]	[s_3,s_4]	[s_3,s_3]	[s_1,s_2]
X_2	[s_1/2,s_1]	[s_1/3,s_1/3]	[s_1,s_2]	[s_1,s_2]	[s_1,s_1]	[s_4,s_5]	[s_1/2,s_1/2]	[s_2,s_4]
X_3	[s_1,s_3]	[s_1/4,s_1/4]	[s_2,s_2]	[s_1/2,s_1/2]	[s_3,s_5]	[s_2,s_4]	[s_1,s_2]	[s_2,s_3]
X_4	[s_3,s_3]	[s_1/2,s_1]	[s_3,s_3]	[s_4,s_5]	[s_4,s_4]	[s_2,s_2]	[s_2,s_2]	[s_3,s_4]
X_5	[s_3,s_4]	[s_1/2,s_1/2]	[s_2,s_3]	[s_3,s_3]	[s_4,s_5]	[s_1/2,s_1]	[s_3,s_4]	[s_1/2,s_1]
X_6	[s_1,s_2]	[s_2,s_2]	[s_2,s_4]	[s_2,s_3]	[s_1,s_2]	[s_1/4,s_1/3]	[s_1,s_1]	[s_1/3,s_1]
X_7	[s_2,s_3]	[s_1/4,s_1/2]	[s_3,s_4]	[s_1,s_1]	[s_3,s_4]	[s_1,s_2]	[s_2,s_3]	[s_3,s_3]
X_8	[s_1/3,s_1]	[s_1,s_2]	[s_1,s_1]	[s_1/4,s_1/4]	[s_1/3,s_1/2]	[s_1/3,s_1/2]	[s_1,s_3]	[s_2,s_2]
X_9	[s_1/2,s_1/2]	[s_1/4,s_1/3]	[s_1,s_4]	[s_4,s_4]	[s_1/2,s_1]	[s_1/3,s_1]	[s_4,s_5]	[s_1,s_1]
X_10	[s_1/3,s_1/3]	[s_3,s_3]	[s_1/2,s_1/2]	[s_1/4,s_1/3]	[s_1/3,s_1/3]	[s_1,s_1]	[s_1/3,s_1/2]	[s_1/2,s_1/2]

Table A6

Deterministic linguistic evaluation information matrix of .

D_1	Y_1	Y_2	Y_3	Y_4	Y_5	Y_6	Y_7	Y_8
X_1	s_2.3784	s_0.5000	s_1.1892	s_0.3689	s_2.0000	s_3.2237	s_3.2237	s_3.0000
X_2	s_0.5217	s_2.2361	s_1.0000	s_0.3333	s_1.0000	s_4.0000	s_0.5946	s_5.0000
X_3	s_4.0000	s_3.4087	s_0.5946	s_0.2272	s_3.2237	s_4.2295	s_1.0000	s_1.1892
X_4	s_4.2295	s_0.5000	s_4.0000	s_0.2000	s_2.2134	s_3.0000	s_1.1892	s_2.2134
X_5	s_3.4087	s_1.1892	s_2.3784	s_0.2115	s_2.3784	s_2.0000	s_3.4087	s_1.0000
X_6	s_2.2134	s_2.2134	s_0.3333	s_1.4142	s_0.3333	s_4.2295	s_1.3161	s_0.5946
X_7	s_3.2237	s_0.5946	s_3.2237	s_0.5946	s_1.1892	s_1.0000	s_4.2295	s_2.0000
X_8	s_2.0000	s_0.2686	s_0.3536	s_1.7321	s_0.3689	s_0.5000	s_2.2134	s_3.4087
X_9	s_1.1892	s_1.0000	s_2.2134	s_0.2500	s_0.5946	s_1.1892	s_3.0000	s_0.5000
X_10	s_1.0000	s_0.2500	s_2.0000	s_1.1892	s_0.2115	s_0.5946	s_0.2686	s_0.3689

Table A7

Deterministic linguistic evaluation information matrix of .

D_2	Y_1	Y_2	Y_3	Y_4	Y_5	Y_6	Y_7	Y_8
X_1	s_1.0000	s_1.0000	s_1.0000	s_2.3784	s_2.2134	s_4.0000	s_3.0000	s_3.0000
X_2	s_0.3689	s_3.4087	s_0.5946	s_1.1892	s_0.5946	s_5.0000	s_0.3333	s_3.2237
X_3	s_2.0000	s_5.0000	s_0.3689	s_0.3333	s_3.0000	s_2.2134	s_0.5000	s_1.0000
X_4	s_4.0000	s_1.1892	s_1.3161	s_0.2115	s_0.3333	s_2.0000	s_1.0000	s_4.2295
X_5	s_3.2237	s_2.2134	s_3.0000	s_0.2686	s_3.2237	s_1.0000	s_4.2295	s_0.5946
X_6	s_1.1892	s_0.5946	s_0.5000	s_0.3689	s_0.2686	s_0.5946	s_1.1892	s_0.5000
X_7	s_2.3784	s_2.0000	s_4.2295	s_2.0000	s_1.0000	s_1.1892	s_4.0000	s_4.0000
X_8	s_0.5000	s_0.5000	s_0.3333	s_3.2237	s_0.3689	s_0.2500	s_2.0000	s_1.1892
X_9	s_0.2973	s_0.2115	s_4.0000	s_0.2500	s_0.5000	s_0.5000	s_2.2134	s_0.3689
X_10	s_0.2500	s_0.3689	s_2.2134	s_4.2295	s_0.2500	s_0.3689	s_0.3689	s_0.3333

Table A8

Deterministic linguistic evaluation information matrix of .

D_3	Y_1	Y_2	Y_3	Y_4	Y_5	Y_6	Y_7	Y_8
X_1	s_2.2134	s_2.2134	s_0.3689	s_1.0000	s_2.0000	s_3.0000	s_2.2134	s_4.2295
X_2	s_0.3689	s_1.1892	s_0.3333	s_2.0000	s_1.1892	s_3.4087	s_0.2686	s_5.0000
X_3	s_2.0000	s_4.0000	s_1.0000	s_0.5946	s_4.2295	s_4.2295	s_1.0000	s_0.5946
X_4	s_4.2295	s_2.0000	s_4.2295	s_0.3333	s_3.2237	s_1.1892	s_2.0000	s_2.2134
X_5	s_3.0000	s_0.5000	s_2.2134	s_0.5000	s_4.0000	s_1.0000	s_3.0000	s_1.1892
X_6	s_1.1892	s_0.5946	s_1.1892	s_1.1892	s_0.4387	s_0.5946	s_0.4387	s_1.0000
X_7	s_3.2237	s_3.0000	s_3.0000	s_2.2134	s_1.0000	s_2.0000	s_4.0000	s_2.0000
X_8	s_1.0000	s_0.3689	s_0.2686	s_3.0000	s_0.2686	s_0.2272	s_0.5946	s_3.4087
X_9	s_0.5000	s_3.2237	s_3.2237	s_0.2686	s_0.5946	s_0.3333	s_3.4087	s_0.5000
X_10	s_0.3333	s_0.3333	s_2.0000	s_4.0000	s_0.3333	s_0.3689	s_0.2500	s_0.3333

Table A9

Deterministic linguistic evaluation information matrix of .

D_4	Y_1	Y_2	Y_3	Y_4	Y_5	Y_6	Y_7	Y_8
X_1	s_1.1892	s_0.5946	s_0.5000	s_1.1892	s_1.0000	s_1.1892	s_2.2134	s_2.2134
X_2	s_0.2973	s_5.0000	s_0.3333	s_0.5000	s_1.1892	s_3.2237	s_0.4387	s_5.0000
X_3	s_2.3784	s_3.0000	s_2.0000	s_0.3333	s_3.4087	s_4.0000	s_0.3689	s_1.1892
X_4	s_4.2295	s_1.0000	s_4.0000	s_0.2000	s_2.5149	s_2.0000	s_0.5946	s_3.2237
X_5	s_3.2237	s_1.1892	s_3.2237	s_0.2115	s_4.2295	s_1.0000	s_2.5149	s_2.0000
X_6	s_1.0000	s_2.0000	s_1.1892	s_0.5946	s_2.0000	s_0.3689	s_1.0000	s_1.0000
X_7	s_4.0000	s_3.4087	s_2.2134	s_1.0000	s_0.5946	s_0.5946	s_3.4087	s_2.3784
X_8	s_0.5000	s_0.3689	s_0.2500	s_4.0000	s_0.3689	s_0.2973	s_2.3784	s_4.2295
X_9	s_0.5946	s_4.2295	s_3.0000	s_0.2686	s_0.4387	s_0.5000	s_4.2295	s_0.5000
X_10	s_0.3689	s_0.5000	s_0.3689	s_3.0000	s_0.3333	s_0.2686	s_0.3333	s_0.5946

Table A10

Deterministic linguistic evaluation information matrix of .

D_5	Y_1	Y_2	Y_3	Y_4	Y_5	Y_6	Y_7	Y_8
X_1	s_1.0000	s_2.2134	s_0.3333	s_1.1892	s_2.0000	s_3.2237	s_3.0000	s_1.1892
X_2	s_0.5946	s_3.0000	s_1.1892	s_0.5946	s_1.0000	s_4.2295	s_0.5000	s_2.3784
X_3	s_1.3161	s_4.0000	s_2.0000	s_2.0000	s_3.4087	s_2.3784	s_1.1892	s_2.2134
X_4	s_3.0000	s_1.1892	s_3.0000	s_0.2115	s_2.0000	s_2.0000	s_2.0000	s_3.2237
X_5	s_3.2237	s_2.0000	s_2.2134	s_0.3333	s_4.2295	s_0.5946	s_3.2237	s_0.5946
X_6	s_1.1892	s_0.5000	s_2.3784	s_0.3689	s_1.1892	s_0.2686	s_1.0000	s_0.4387
X_7	s_2.2134	s_2.3784	s_3.2237	s_1.0000	s_3.2237	s_1.1892	s_2.2134	s_3.0000
X_8	s_0.4387	s_0.5946	s_1.0000	s_4.0000	s_0.3689	s_0.3689	s_1.3161	s_2.0000
X_9	s_0.5000	s_3.2237	s_1.4142	s_0.2500	s_0.5946	s_0.4387	s_4.2295	s_1.0000
X_10	s_0.3333	s_0.3333	s_1.0000	s_3.2237	s_0.3333	s_1.0000	s_0.3689	s_0.5000

Table A11

, and for based on UML-TOPSIS.

	D_1			D_2			D_3
	To P	To N	CD	To P	To N	CD	To P	To N	CD
X_1	4.4357	9.9766	−0.3652	3.1125	10.6268	−0.4486	3.6712	10.9434	−0.6792
X_2	6.6128	7.7878	−1.1408	6.5947	9.1814	−1.9732	7.5176	8.6398	−2.4828
X_3	5.8778	10.8230	−0.6857	6.4667	9.0956	−1.9278	3.9912	11.4568	−0.7733
X_4	5.7000	11.8820	−0.5483	7.0438	9.0031	−2.1692	3.6610	12.0332	−0.5885
X_5	5.1648	10.7894	−0.4949	5.1003	10.9639	−1.2298	4.3635	9.9497	−1.0497
X_6	6.6077	7.4690	−1.1663	9.6453	3.2298	−3.6682	7.4625	3.9827	−2.8288
X_7	3.6815	11.2019	−0.0572	2.4631	13.0378	0.0000	2.3731	12.6107	0.0000
X_8	7.5319	7.0754	−1.4504	9.6933	5.1560	−3.5400	11.5742	5.4372	−4.4461
X_9	6.7242	6.6248	−1.2689	12.4629	5.2963	−4.6537	8.2155	7.5668	−2.8619
X_10	11.0916	4.0772	−2.6697	12.1021	5.5135	−4.4905	12.5587	4.6220	−4.9256

Table A12

, and for based on UML-TOPSIS.

	D_4			D_5
	To P	To N	CD	To P	To N	CD
X_1	4.7284	6.6700	−0.6528	4.7134	8.7596	−1.1297
X_2	7.6454	9.4113	−1.1716	5.0081	8.7419	−1.2471
X_3	5.3585	10.4424	−0.4962	2.5674	11.2234	−0.0807
X_4	5.4175	10.9160	−0.4713	4.1837	12.0676	−0.6470
X_5	4.4157	11.8336	−0.1360	4.9517	10.1729	−1.1064
X_6	5.5720	5.8871	−0.9360	8.6503	4.6838	−3.0172
X_7	3.8869	11.1806	−0.0552	2.5402	11.5732	−0.0410
X_8	9.4268	7.5514	−1.7871	7.6831	5.8474	−2.5401
X_9	7.5796	8.7996	−1.2064	7.4993	6.9717	−2.3746
X_10	12.5693	3.1465	−2.9679	10.6996	3.7832	−3.8987

Remarks: To P means the comprehensive deviation from to ; To N means the comprehensive deviation from to ; CD means the closeness to for .

Table A13

, and for based on UML-TOPSIS-EW.

	D_1			D_2			D_3
	To P	To N	CD	To P	To N	CD	To P	To N	CD
X_1	5.3377	9.0922	−0.3004	2.9595	11.0195	−0.2455	3.7410	10.8451	−0.7087
X_2	5.9179	9.3192	−0.4060	6.3708	9.3350	−1.6877	7.3652	8.7613	−2.4006
X_3	6.3259	10.4385	−0.3883	6.3822	9.2017	−1.7028	4.0303	11.4134	−0.7851
X_4	7.0814	9.7372	−0.6213	7.8588	8.5290	−2.3226	3.8283	11.9050	−0.6608
X_5	6.1986	8.9975	−0.4985	5.5065	10.5785	−1.2565	4.4384	9.7826	−1.0871
X_6	6.7184	7.3807	−0.7674	10.1255	3.0965	−3.6284	7.3638	4.0340	−2.7775
X_7	4.5564	9.0719	−0.1309	2.6111	12.4098	0.0000	2.3757	12.5230	0.0000
X_8	8.7274	6.5543	−1.2875	9.6544	5.3901	−3.2632	11.5886	5.5351	−4.4360
X_9	7.7157	5.7744	−1.1402	12.7315	4.7651	−4.4920	8.4820	7.3532	−2.9831
X_10	12.5738	3.5817	−2.4164	12.2012	5.7076	−4.2130	12.2823	4.8807	−4.7802

Table A14

, and for based on UML-TOPSIS-EW.

	D_4			D_5
	To P	To N	CD	To P	To N	CD
X_1	5.0660	6.0891	−0.6528	4.0703	9.0638	−0.8569
X_2	8.0144	8.8434	−1.1087	5.3538	8.3572	−1.4357
X_3	5.7812	10.2735	−0.4251	2.4793	11.5480	0.0000
X_4	7.2021	10.3930	−0.7610	6.5954	10.7021	−1.7334
X_5	6.5623	10.3018	−0.6134	6.1349	9.5130	−1.6507
X_6	6.2171	5.4136	−0.9993	9.5635	4.0616	−3.5056
X_7	4.0897	10.3072	−0.0083	2.9044	10.5571	−0.2573
X_8	9.7516	7.8837	−1.6259	7.5388	7.2212	−2.4154
X_9	8.6474	7.9003	−1.3543	9.3302	6.1174	−3.2335
X_10	11.6029	4.3587	−2.4177	9.6517	5.3259	−3.4317

Table A15

The positive, negative and net flow of for based on UMLDM-EW.

	D_1			D_2			D_3
	H1+	H1−	H_1	H2+	H2−	H_2	H3+	H3−	H_3
X_1	0.5771	0.3631	0.2140	0.6137	0.3161	0.2976	0.5920	0.3668	0.2252
X_2	0.4343	0.4996	−0.0653	0.5399	0.4374	0.1025	0.4863	0.4777	0.0087
X_3	0.5790	0.4031	0.1759	0.5254	0.3990	0.1265	0.5859	0.3672	0.2187
X_4	0.5291	0.4003	0.1288	0.4970	0.4543	0.0427	0.6127	0.3504	0.2623
X_5	0.6010	0.3752	0.2258	0.6307	0.3450	0.2857	0.5170	0.4066	0.1103
X_6	0.4998	0.4739	0.0259	0.3354	0.6414	−0.3059	0.3755	0.6126	−0.2372
X_7	0.6171	0.3592	0.2579	0.6982	0.2380	0.4603	0.6602	0.2629	0.3972
X_8	0.3057	0.6771	−0.3714	0.3502	0.5913	−0.2411	0.3019	0.6750	−0.3731
X_9	0.4101	0.5334	−0.1233	0.2996	0.6462	−0.3466	0.4124	0.5506	−0.1382
X_10	0.2481	0.7165	−0.4683	0.2703	0.6919	−0.4216	0.2201	0.6942	−0.4741

Table A16

The positive, negative and net flow of for based on UMLDM-EW.

	D_4			D_5
	H4+	H4−	H_4	H5+	H5−	H_5
X_1	0.5313	0.4448	0.0864	0.5518	0.4040	0.1477
X_2	0.4874	0.4632	0.0241	0.5022	0.4640	0.0382
X_3	0.5469	0.3972	0.1498	0.6864	0.2748	0.4116
X_4	0.5522	0.3918	0.1604	0.5304	0.4097	0.1208
X_5	0.5656	0.4119	0.1537	0.5295	0.4397	0.0897
X_6	0.4254	0.5198	−0.0944	0.3212	0.6585	−0.3373
X_7	0.6229	0.3456	0.2773	0.6468	0.3233	0.3235
X_8	0.3889	0.5677	−0.1788	0.4039	0.5583	−0.1544
X_9	0.4397	0.5260	−0.0863	0.3711	0.5865	−0.2154
X_10	0.2274	0.7197	−0.4923	0.2531	0.6775	−0.4244