An integrated multi-criteria decision-making approach for identifying the risk level of musculoskeletal disorders among handheld device users.

In work-from-home (WFH) situation due to coronavirus (COVID-19) pandemic, the handheld device (HHD) users work in awkward postures for longer hours because of unavailability of ergonomically designed workstations. This problem results in different type of musculoskeletal disorders (MSDs) among the HHD users. An integrated multi-criteria decision-making approach was offered for identifying the risk level of MSDs among HHD users. A case example implemented the proposed approach in which, firstly, the best-worst method (BWM) technique was used to prioritize and determine the relative importance (weightage) of the risk factors. The weightages of the risk factors further used to rank the seven alternatives (HHD users) using Vlse Kriterijumska Optimizacija Kompromisno Resenje (VIKOR) technique. The outcomes of the BWM investigation showed that the three most significant risk factors responsible for MSDs are duration of working, poor working posture and un-ergonomic design. The outcome of the VIKOR technique exhibited that computer professionals were at the highest risk among all users. The risk factor priority must be used for designing a working strategy for the WFH situation which will help to mitigate the risks of MSDs.

Introduction

The coronavirus (COVID-19) has unfolded very swiftly throughout India and many other countries inflicting acute infectious pneumonia to break out (Bao et al. 2020). Staying at home is only the solution that can restrict the spreading of this disease. However, long stay at home can increase the sedentary activities (Owen et al. 2010) that lead to inactiveness. This inactiveness leads to anxiousness and unhappiness, and negative consequences on the fitness of human beings. Also, workers are subjected to a high level of musculoskeletal disorder (MSD) risk due to awkward postures in working for more extended hours during homestay. The different types of MSDs are most responsible reasons for losses in productive working time (Occupational Safety and Health Administration 2013).

The association between MSDs and computer use has been made a public health concern since the mid-1980s when computer usage in working environments increased dramatically (Hopkins 1990). According to the U.S. Census Bureau report, 120 million American households (75% of population) had personal computers with internet in 2012, which increased 35% from 2001 (U. S. Census Bureau 2012). The use of handheld devices (HHDs) and internet users is also multiplying, as increase of 5.3% in the internet users was observed from 2018 to 2019, with a total user of 4.1 billion worldwide (International Telecommunication Union 2019). Also, it is clear from the report that the numbers of personal computers with internet access have been decreased in recent years. It shows the popularity of portable HHDs (i.e. smart phones, tablets, etc.), these devices enable the users to work anywhere and anytime (Saito et al. 1997; Moffet et al. 2002), which generates various work-related disorders specially MSDs. The generation of these disorders initiates due to various work-related risk factors, which are characterized in the following categories: i.e. physical factor (PF), psychosocial factor (PSF), and individual factor (IF) with their subfactors (Janwantanakul et al. 2012). Hence, it appears imperative to explore the literature related to MSDs in HHD users and earlier soft computing tools used for decision-making.

The particulars of the remaining sections of this paper are described as: Sect. 2 discusses the relevant literature in this field. Section 3 represents the listing of primary factors and subfactors of risks which might result in the inception of MSDs indicated by the previous researches and decision-makers. Section 4 comprised of methodology to perform the current research. It includes the description related to implementation of integrated multi-criteria decision-making (MCDM) approach (BWM and VIKOR techniques). Section 4 describes the outcomes of the proposed integrated approach, and a comparison of current research outcomes with available literature. Finally, the last section exhibits conclusion, limitations, and future research directions based on the outcomes of current research.

Relevant literature

This section is classified into two parts: (1) MSDs among HHD users and (2) soft computing tools used for decision-making.

MSDs among HHD users

The use of HHDs has become vital in the various work environments. Several epidemiology studies demonstrate that MSDs are prevalent among the users working with HHDs (Chiang and Liu 2016; Woo et al. 2016; Taib et al. 2016; Xie et al. 2017; Soria-Oliver et al. 2019). Significant associations of MSDs with physical (Chiang and Liu 2016; Woo et al. 2016; Xie et al. 2017) and psychosocial (Janwantanakul et al. 2012; Taib et al. 2016; Soria-Oliver et al. 2019) factors have been found in previous studies. Previous studies also observed that work-related factor (PF or PSF) is not a single factor that can develop MSDs. IFs such as gender, age, obesity, and smoking behaviour are also significant reason of MSDs development (Taib et al. 2016; Xie et al. 2017).

Depending upon the severity of the pain, either MSDs can be at the initial level, or it converts into disability when not appropriately diagnosed. MSD at initial level is curable without difficulty, and it takes a month or less time to recover for a suffering person (Laisné et al. 2012; Kuijer et al. 2012). However, the treatment of disability is somewhat complicated, and it can take a long time for the individual to improve. Investigators have identified many factors that could account for a change from acute to chronic MSDs (Keefe et al. 2018). Identifying the responsible risk factors for the development of MSDs can assist to identify the risk level which will further help in deciding the preventive measures. Primary prevention helps in reducing the risk of the initial onset of a problem (Waongenngarm et al. 2018; Williams et al. 2018). To reduce the MSDs among HHD users, it is vital to think about the priority and relative importance of the risk factors. Previous researches reported in the literature have used various soft computing tools for identifying the risk level by appropriate decision-making strategies among various work environments.

Soft computing tools used for decision-making

In the reported literature, various soft computing techniques or approaches have been used by previous researchers to identify the risk level among various work environments.

Castillo and Melin (2020) proposed a hybrid intelligent approach for forecasting the future trends of pandemic situations based on the COVID-19 time series of confirmed cases and deaths. Dansana et al. (2020) used deep learning algorithms to map the computed tomography and X-rays reports of COVID-19 patients for providing better and faster treatment. Melin et al. (2020a) done a spatial evolution of different country maps for exploring COVID-19 pandemic situations by using an unsupervised neural network. Melin et al. (2020b) implemented the concepts of neural network and fuzzy logic for predicting the COVID-19 time series in Mexico. These evolutionary approaches provided the efficient predictions of collected data for larger sample sizes. However, the pairwise comparison of multiple factors rating data was done previously by MCDM approaches mostly.

Maldonado-Macías et al. (2014) evaluated the ergonomic compatibility of advanced technology used in manufacturing industries. Chiu and Hsieh (2016) applied fuzzy TOPSIS for improving the maintenance tasks in the aviation industry. Ahmadi et al. (2017) developed a scoring model for estimating the ergonomic risks to determine risky situations by using a mixture of MCDM approaches. Khandan et al. (2017) used fuzzy TOPSIS for evaluating the occupational disorders risk and ergonomic problems in workplaces. Khan et al. (2019) used the BWM method for prioritizing the risk factors of lower back pain in the industrial workers and advised that there is a need of MCDM techniques usage for solving the health problems faced by workers in various environments. There are various MCDM approaches used for evaluation of multiple criteria (risk categories) like AHP, ANP, DEMATEL, fuzzy TOPSIS, etc. (Ahmadi et al. 2017; Khandan et al. 2017). However, BWM was an efficient approach due to the merits of this approach as compared to other approaches with smaller amount comparisons of rating data from the experts and higher consistency in results (Rezaei 2015, 2016; Khan et al. 2019). Also, VIKOR was used for ranking of alternatives because of its capability to precisely optimize the multiple factors using co-operation precedence methodology (Mohanty and Mahapatra 2014; Mohanty et al. 2018). Despite these strengths, the MCDM approaches suffer from certain restrictions also; both BWM and VIKOR techniques governed by the decision-maker choices, therefore, it is essential to select decision-maker wisely depending on the expertise in relevant areas.

Most of the earlier investigations dedicated for finding risk factors of MSDs among various occupational groups. The longer duration usage of HHDs in awkward posture causes discomfort in HHD users. However, currently most of the office users and students use HHDs (user friendly) for extensive times than other type of technology devices used in the past due to COVID-19 pandemic. Till date, there is no such research available which used MCDM approach for the evaluation of priority and relative importance of MSD risk factors among HHD users. This research gap is filled by using the integrated approach (BWM and VIKOR techniques) for identifying the risk level of MSDs among HHD users.

Risk factors of MSDs

Previous studies (Janwantanakul et al. 2012; Xie et al. 2017) have described three categories of risk factors for MSDs, i.e. PF, PSF, and IF, and some subfactors among each category. The different categories of MSD risk factors among the HHD users have not been explained in detail. Therefore, firstly relevant literature related to different type of HHD users was (Chiang and Liu 2016; Woo et al. 2016; Taib et al. 2016; Xie et al. 2017; Soria-Oliver et al. 2019) explored and discovered the prevalent MSD risk factors among the HHD users. Figure 1 depicts a listing of the various categories of MSD risk factors (primary factors and subfactors). This listing was also evaluated by the expert team which includes four decision-makers and two experts from the institute project committee before finalization.

Fig. 1

Different categories of MSD risk factors (primary and subfactors) among the HHD users

Methodology

A three-stage methodology has been used in the current research (Fig. 2). The objective of using this three-stage procedure is to identify the risk level of MSDs among HHD users. The first stage includes the risk factors identification based on a previously published literature and suggestions of expert team. The second stage involves the priority building and relative importance (weightage) calculation of these risk factors by using the BWM technique. The third stage uses VIKOR technique for ranking the best alternative among seven types of HHD users with respect to priority and relative importance identified using BWM technique.

Fig. 2

Methodology of current research

BWM technique

Rezaei (2015) established an efficient MCDM approach named BWM. Herein process, the decision-maker selects the best and worst factor/subfactor from the developed list of factors/subfactors of risks. The most significant and least favourable factors/subfactors are termed as the best and worst factors/subfactors, respectively. Decision-makers then do a relative comparison of the best factor with other factors, and other factors with the worst factor. This comparative analysis generates two pairs of comparison in vector form, which helps to find out the weights of the factors. Determination of the optimal weight of the factors/subfactors of risk is solved by a linear programming model for optimized outcomes. Implementation methodologies of the BWM are explained very well in the previous works (Rezaei 2015, 2016; Gupta 2018; Khan et al. 2019). However, the BWM steps used in the current research are described below:

Step I Discovery of all the decision criterion/factors and subfactors of risks is essential for decision matrix preparation. On the basis of decision-makers choice, the best and the worst factors/subfactors of risk are designated.

Step II A pairwise evaluation matrix among the best risk factor/subfactor and all other factors/subfactors of risks is developed using a scale of 1 to 9 as given in Table 1. This process is used for obtaining a preference of the best risk factor/subfactor over the others.

Table 1

Rating scale used for BWM and VIKOR techniques

Scale for BWM technique (adapted from Rezaei et al. 2014)
1	2	3	4		5	6	7	8	9
Equal importance	Somewhat between equal and moderate	Moderately more important	Somewhat between Moderate and Strong		Strongly more important	Somewhat between strong and very strong	Very strongly important	Somewhat between very strong and absolute	Absolutely more important
Scale for VIKOR technique
Least important				1
Moderately important				2
Strongly important				3
Very strongly important				4
Extremely important				5

The best from others (BFO) vector is defined as given in Eq. (1):where vector represents the preference of the best risk factor/subfactor (B) over the other factors/subfactors of risks () and the self-preference defined as: .

Step III The factors/subfactors of risks are now compared with the worst risk factor/subfactor in the form of pairs, which helps to generate the others to worst (OTW) evaluation matrix in the similar way as defined in step II. The vector matrix of OTW is represented as below:where vector represents the preference of the other factors/subfactors of risks () over the worst factor/subfactor (W) and the self-preference is .

Step IV. For the calculation of the optimum weights (), the absolute maximum modifications for all j are minimalized. The mathematical likeness (objective function) of this modification is written as:

Objective function: .

subject to:

Equation (3) can be converted to the linear programming problem for determining the optimal weights.subject to:

Equation (4) is solved for best result for finding the optimum weights (). Similarly, results of Eq. (4) delivers the optimum value of the consistency/reliability ratio . The assessed value of specifies the reliability of the evaluations made. The close to zero value of represents the high level of reliability in outcomes.

VIKOR technique

VIKOR technique is a passive ranking method (Opricovic 1998) and is used frequently where dissimilar contradictory measures are present. It generates a passive result based on “closeness to ideal solution and mutual agreement through concessions”. The VIKOR technique was extensively used by many investigators to get ranking of alternatives in the ergonomics design researches (Mohanty et al. 2018; Alsalem et al. 2019). The necessary steps of VIKOR technique employed in the current research are presented underneath:

Step I Pairwise decision matrix generated for every alternative with respect to each subfactors of risks using linguistic scale of 1 to 5 is given in Table 1.

Step II The average decision matrix is processed using Eq. (5).

Step III This step is used to calculate best values and the worst values from the average decision matrix of all alternatives.where denotes the positive best outcome and signifies the negative best outcome for the bth characteristic () in the average decision matrix.

Step IV The weighted and normalized Manhattan distance or Utility measure (S) and weighted and normalized Chebyshev distance or Regret measure (R) values are calculated using Eqs. (6) and (7).where S denotes the distance of ath alternative () from positive best outcome and R denotes the distance of ath alternative from negative best outcome and W represents the weights of subfactors of risks gained from BWM technique.

Step V In this step, the scores for closeness coefficients or VIKOR index (Q) are calculated using Eq. (8) and (9).where represents the weightage of supreme effectiveness and is assumed to 0.5 in the current research.

Step VI In the last step, minimum score of Q is ranked first among the all alternatives based on the two situations below:

Situation 1: is selected if where is the alternative that has achieved second rank in the investigation and n is the total number of alternatives.

Situation 2: also attains first rank conferring to both and values.

Implementation of proposed integrated approach

The current research was carried out by taking the help of four decision-makers with research specialization in Human Factors and Ergonomics, occupational health and safety. The instructions and purpose of the research were briefed to all decision-makers before taking their responses. The decision-makers gave their informed consents and provided the requested data. On the basis of the input/data provided by the decision-makers, the further processing was done as per the steps of BWM and VIKOR techniques.

Risk priority determination using BWM technique

The probable factors/subfactors of MSD risks listing given in Fig. 1 were ranked using BWM technique. This technique was used for weight calculation of the primary factors, i.e. work-related individual factors (IF), psychosocial factors (PSF), and physical factors (PF). Similarly, the calculations for relative and global weights of the all subfactors of risks were also done and ranked the factors/subfactors on the basis of obtained weights.

Relative weight calculation/risk priority for primary risk factors

In the current research, decision-makers rated the PF as the best risk factor among other risk factors and provided ratings of other risk factors compared to the best risk factor using a scale of 1 to 9. A similar process was also carried out for providing the ratings to worst risk factor (IF). Equations (1) and (2) helped to generate the BFO and OTW vectors for the pairwise evaluation among the best and worst risk factor. The pairwise evaluation matrix is given in Table 2.

Table 2

BFO and OTW for primary risk factors

BFO	IF	PSF	PF
Best factor: PF	9	4	1
OTW			Worst Factor: IF
IF			1
PSF			3
PF			9

The optimum weights () and consistency coefficient () of primary risk factors were computed using Eqs. (3) and (4). The value of is very close to zero, which shows higher reliability in the evaluations and outcomes.

Relative weights calculation/ priority of the subfactors of risks

The relative weights of the subfactors of risks were computed using Eqs. (1) and (2). The pairwise evaluation matrix for subfactor comparisons is presented in Table 3.

Table 3

Pairwise evaluation matrix for subfactors of risks for three primary factor categories

BFO and OTW subfactors for IF
BFO	AG	GE	OB	SM	PA
Best factor: PA	3	5	4	6	1
OTW					Worst factor: GE
AG					8
GE					1
OB					5
SM					4
PA					9
BFO and OTW subfactors for PSF
BFO	JS	JSA	RW	TAT
Best factor: JS	1	3	2	5
OTW				Worst factor: TAT
JS				8
JSA				7
RW				6
TAT				1
BFO and OTW subfactors for PF
BFO	DW	PO	FE	PD
Best factor: DW	1	2	5	4
OTW				Worst factor: FE
DW				7
PO				3
FE				1
PD				4

The relative ranking of the primary risk factors and ranks of the subfactors of risks are presented in Table 4. It shows that out of three primary risk factors, PF is the best risk factor pursued by PSF, and IF, i.e. PF > PSF > IF. Additionally, Table 4 also exposes that among all subfactors of IF, the order of priority is as PA > AG > OB > SM > GE. Similarly, among subfactors of PSF, the order of priority is as JS > RW > JSA > TAT. Lastly, the order of priority is as DW > PO > PD > FE among the subfactors of PF.

Table 4

Weightage/relative importance and ranking of primary and subfactors computed using BWM technique

Primary factors	Weightage/relative importance	Relative ranking of primary factors	Subfactors	Weightage/relative importance	Relative ranking of subfactors	Global weightage/relative importance	Global ranking
IF	0.0769	3	AG	0.2182	2	0.0168	9
			GE	0.0519	5	0.0040	13
			OB	0.1636	3	0.0126	10
			SM	0.1091	4	0.0084	12
			PA	0.4571	1	0.0352	8
PSF	0.1923	2	JS	0.4385	1	0.0843	4
			JSA	0.2032	3	0.0391	7
			RW	0.3048	2	0.0586	5
			TAT	0.0535	4	0.0103	11
PF	0.7308	1	DW	0.4767	1	0.3484	1
			PO	0.3023	2	0.2209	2
			FE	0.0698	4	0.0510	6
			PD	0.1512	3	0.1105	3

Table 4 also exhibits that duration of working, posture and poor design are the topmost three MSD risk subfactors, since their rank is top three among all other risk subfactors whereas smoking, task/activity type and obesity are the last three risk subfactors. The outcomes of our research are comparable to the prior investigations (Janwantanakul et al. 2012; Moom et al. 2015; Kaliniene et al. 2016; Abaraogu et al. 2018; Sasikumar and Binoosh 2020), who reported that heavy duration of working, posture, poor design, and job strain are the subfactors of risks that causes MSD among the HHD users.

Evaluating alternatives using VIKOR

After acquiring optimum weights of risk subfactors, the alternatives are ranked based on weights of subfactors using VIKOR technique. By using linguistic scale provided in Table 1, the decision-makers were rated the all HHD alternatives with respect to the subfactor of risks. The rating given by the decision-makers is presented in “Appendix”. These average ratings from decision-maker were computed for each alternatives using Eq. (5). The average decision matrix is revealed in Table 5.

Table 5

Average ratings matrix derived from the rating provided by four decision–makers for seven handheld device alternatives with respect to subfactors of risks

Alternatives	Subfactors of risks
	AG		GE	OB	SM	PA	JS	JSA	RW	TAT	DW	PO	FE	PD
	Weights calculated from BWM technique
	0.0168		0.0040	0.0126	0.0084	0.0352	0.0843	0.0391	0.0586	0.0103	0.3484	0.2209	0.0510	0.1105
A1: University students		3.00	2.25	2.50	3.00	3.50	2.50	2.75	3.00	3.00	4.25	3.75	2.50	4.25
A2: Computer professionals		2.75	3.00	3.00	3.75	2.75	4.00	3.25	3.00	4.00	3.75	4.25	3.50	3.50
A3: Computer operators		2.75	3.00	3.00	4.00	3.50	3.50	4.00	3.50	4.00	3.25	3.50	3.00	4.00
A4: University faculty		3.00	3.25	4.25	3.50	2.75	3.25	3.00	3.00	3.00	4.00	3.50	3.00	3.50
A5: University staff		3.25	3.50	3.00	3.00	3.00	3.00	3.00	3.75	2.50	3.00	3.50	2.50	3.25
A6: School teachers		3.00	3.75	3.50	2.50	2.50	3.00	3.00	2.50	3.75	3.50	3.50	3.25	2.50
A7: School students		2.50	2.75	2.75	3.25	2.75	3.50	2.75	3.25	3.50	2.50	3.75	3.25	3.00

The maximum and minimum values of risk factors were computed using Eqs. (6) and (7). Equations (8)–(10) were used for computing the values of Sa, Ra and Qa (Table 6). The computer professional (A2) attains first rank, as it has lowermost Qa value and also fulfils both situations ( and attains first rank according to both Ra and Sa values as presented in Table 6). The obtained results of alternative ranking are in line with previous researches of numerous occupational groups (Widanarko et al. 2011, 2013; Silva et al. 2016) and computer professionals, causes higher MSDs due to long duration of working.

Table 6

Ranking of seven alternatives using VIKOR technique

Alternatives			Sa		Rank	Ra	Rank	Qa	Rank
A1	University students		0.3915		2	0.1473	2	0.2357	2
A2	Computer professionals		0.2555		1	0.0995	1	0.0000	1
A3	Computer operators		0.5233		4	0.2209	3	0.5190	4
A4	University faculty		0.4951		3	0.2209	3	0.4901	3
A5	University staff		0.7145		6	0.2488	6	0.7716	6
A6	School teachers		0.6958		5	0.2209	3	0.6963	5
A7	School students		0.7422		7	0.3484	7	1.0000	7
S*			0.2555			R*	0.0995
S−			0.7422			R−	0.3484

Conclusion, limitations and future scope

Conclusion

MSD is prominent health issue which forces workers to away from the work. Organizations are forced to disburse a larger amount of money to the workers experiencing MSDs in contradiction of their compensation claims. MSDs harmfully disturb workers’ health and also affect the work steering to significant loss of productivity and efficiency. Numerous risk factors have been testified for MSD generation, out of which some are prominent risk factors and few are risky. In the current research, an approach was proposed to decide priority and optimal weightages of the MSD risk factors using BWM. Also, the alternatives with respect to risk factors were ranked for selection of risky device users among all HHD users. The outcomes of the current research provided the following conclusions:

Out of three primary types of MSD risk factors, physical factors (PF) are the prominent followed by psychosocial factors (PSF) and individual factors (IF).

The rank of prominence of the different subfactors of the physical factors (PF) for MSDs is found as DW > PO > PD > FE.

The various subfactors of the psychosocial factors (PSF) follow the order as JS > RW > JSA > TAT in increasing MSDs.

The ranking of the subfactors in the individual factor (IF) category is found in the order of PA > AG > OB > SM > GE.

The ranking of seven type of HHD users is as A2 > A1 > A4 > A3 > A6 > A5 > A7, and the computer professionals are at higher risk among seven type of HHD users.

Limitations of the present research

Similar to previous investigations, the current research has also particular limitations which are described as:

The linear or combined interacting influences of the different risk factors are not prioritized in our research, though such connections have been conveyed to source of MSDs.

Our research uses a small group of four decision-makers for all judgements taken in both technique and does not include any decision-maker form industry.

Future research directions

The possibility for future work is always present in a research. In view of above explained restrictions, it is recommended that upcoming works might be performed to prioritize linear or combined interactions of the risk factors using suitable soft computing approaches with aggregation of factors as utilized in previous researches (Jana et al. 2019, 2020). Additionally, a comparatively bigger crowd of decision-makers from both industry and academia may be absorbed for gathering rating data. This collected data may be further analysed by using various evolutionary algorithms (Monte Carlo simulation, stochastic modelling, neural network, etc.) for building the prediction models based on various risk factors. These type of approaches were provided the effective results in the previous researches (Yi et al. 2018; Castillo and Melin 2020; Dansana et al. 2020; Kannan et al. 2020; Melin et al. 2020a, b). The compulsory input (on the linguistic scale) from all decision-makers may be gathered independently for optimum weight computation of the risk factors using BWM.

Table 7

Decision-makers ratings for seven alternatives with respect to subfactors of risk

Alternatives	Criteria
	AG	GE	OB	SM	PA	JS	JSA	RW	TAT	DW	PO	FE	PD
Ratings given by decision-maker 1
A1: University students	4	2	3	4	4	3	3	4	4	5	4	3	5
A2: Computer professionals	3	2	3	4	3	4	4	3	4	4	5	4	5
A3: Computer operators	2	3	4	4	3	4	4	4	5	4	4	3	4
A4: University faculty	3	4	4	3	3	4	4	3	4	4	4	3	4
A5: University staff	3	4	3	4	4	3	3	3	3	3	3	2	3
A6: School teachers	3	4	4	3	3	4	3	3	4	4	3	3	3
A7: School students	3	3	2	2	3	3	2	4	3	3	4	3	4
Ratings given by decision-maker 2
A1: University students	3	2	2	3	3	2	2	3	2	4	4	2	4
A2: Computer professionals	2	3	2	4	3	4	3	2	4	4	4	3	4
A3: Computer operators	3	2	3	4	2	3	4	3	4	3	4	2	3
A4: University faculty	2	4	4	2	2	4	3	2	3	3	3	2	3
A5: University staff	3	3	2	3	3	2	2	4	2	2	2	3	2
A6: School teachers	2	3	3	2	2	3	2	2	3	3	2	3	2
A7: School students	2	3	2	3	3	3	3	2	3	2	3	2	3
Ratings given by decision-maker 3
A1: University students	3	2	2	2	3	2	3	2	3	4	4	2	4
A2: Computer professionals	3	4	3	4	2	4	3	3	4	3	4	3	2
A3: Computer operators	3	3	2	4	5	4	4	4	3	2	3	3	4
A4: University faculty	3	2	4	4	3	2	2	3	2	4	3	3	3
A5: University staff	4	3	3	2	2	3	3	4	2	3	4	2	4
A6: School teachers	3	4	4	2	2	2	3	2	4	3	4	3	2
A7: School students	2	2	3	4	2	4	3	3	4	2	3	4	2
Ratings given by decision-maker 4
A1: University students	2	3	3	3	4	3	3	3	3	4	3	3	4
A2: Computer professionals	3	3	4	3	3	4	3	4	4	4	4	4	3
A3: Computer operators	3	4	3	4	4	3	4	3	4	4	3	4	5
A4: University faculty	4	3	5	5	3	3	3	4	3	5	4	4	4
A5: University staff	3	4	4	3	3	4	4	4	3	4	5	3	4
A6: School teachers	4	4	3	3	3	3	4	3	4	4	5	4	3
A7: School students	3	3	4	4	3	4	3	4	4	3	5	4	3