A comprehensive decision framework with interval valued type-2 fuzzy AHP for evaluating all critical success factors of e-learning platforms.

The COVID 19 pandemic not only affected our health and social life in many aspects, but it also changed the classical way of training in classrooms and education preferences of society. As a solution various e-learning platforms were developed and preferred by many educational institutions where the individuals had the opportunity to try the advantages of e-learning platforms. Since the COVID-19 pandemic is neither the first nor the last epidemic, e-learning attracts more attention than ever before and the need for e-learning platforms is expected to be more in the near future. Thus it is necessary to define all critical success factors determining the efficiency of e-learning systems. E-learning platforms have advantages as well as disadvantages and comparisons involve uncertainties and qualitative assessments. A systematic approach should be used to determine the platforms' dimensions, features and weights of critical criteria. The motivation of our study is to determine the weights of all critical success criteria and offer a reliable method for evaluating e-learning platforms. In this study, the interval type-2 fuzzy Analytical Hierarchy Process was utilized to compare critical success factors of e-learning platforms. This is the most comprehensive study considering all critical success factors of e-learning platforms as an Multi Criteria decision Making problem, where 11 criteria and 106 sub-criteria were defined, evaluated and prioritized. This study provides an acceptable rationale for evaluations of e-learning platforms and the results of this study can be used in real-world performance evaluations.

Introduction

E-learning is a teaching model that supports access to information via digital platforms or media which is mainly based on internet technologies to share academic contents. E-learning systems not only allow to create and share content but also to have control over other aspects of teaching (tuition fees, grades) and sources of exchange between students (chats, forums, etc.) (Muilenburg & Berge, 2005). E-learning platforms are defined as systems where students access online resources and curriculum, communicate with instructors and receive the instructor’s assessments (Bhuasiri et al., 2012). Today, e-learning platforms are preferred because they allow students to continue their education while fulfilling their other responsibilities. E-learning platforms have advantages as well as disadvantages. The flexibility provided by these platforms in terms of time and space also confronts us as an obstacle between teacher and student (Lara et al., 2014).

The market share of e-learning platforms is increasing day by day. Many factors affect the success of the platform and customer satisfaction. Instructor, courses, technology, design and environment are factors affecting satisfaction (Sun et al., 2008). The COVID-19 epidemic, which affects social life in many aspects, has also profoundly affected the education preferences of society, and higher education institutions needed to reevaluate their teaching techniques. In this context, the assessment of e-learning portals has become a necessity (Ouajdouni et al., 2021). Evaluation of e-learning systems is multidisciplinary and involves its difficulties (Roffe, 2002). A systematic approach should be used to determine the platforms' dimensions, features, and critical criteria (Ouadoud et al., 2016). E-learning portal selection or comparison is a complex process and, at the same time, a multi-criteria decision making (MCDM) problem (Gong et al., 2021).

The features of e-learning platforms or learning management systems (LMS) have been superficially studied in the literature and it has been reported that interfaces and models should be improved for comparisons (Buendia & Hervas, 2006). COVID 19 is neither the first nor the last pandemic. The interest in e-learning platforms will increase in the coming years. The possible risk of disease and the advantages offered by distance education increase, and the quality of education offered are equal to face-to-face education. The motivation of our study is to determine the weights of all critical success criteria and offer a reliable method for evaluating e-learning platforms. This study consists of five sections. The literature review is given in the second section, the evaluation criteria, Type-2 Fuzzy Set and F-AHP in the third section, the findings and obtained weights in the fourth section, the results and future works are presented in the fifth section. This is the most comprehensive study considering all critical success factors of e-learning platforms as an MCDM problem, where 11 criteria and 106 sub-criteria in Table 1 were defined, evaluated and prioritized. All these criteria and sub-criteria were determined and defined by conducting a deep literature survey. Originality and value of this paper are defining all critical factors for e-learning platforms, ranking these factors, revealing the most important ones and using MCDM methods for the evaluations of effectiveness factors for the first time.

Table 1

Criteria and Sub-criteria

C1	Adaptation	C5	Content	C8	Learning
C11	Compatibility	C51	Material	C81	Organization
C12	Extendibility	C52	Content	C82	Access
C13	Customization	C53	Presentation	C83	Source
C14	Adaptability	C54	Engaging content	C84	Threshing
		C55	Practice/test	C85	Re-access
C2	Framework	C56	Interactive mode	C86	Process control
C21	Warning/Message System	C57	Functional content	C87	Feedback
C22	Ease of understanding	C58	Current content	C88	Encourage
C23	Clear navigation	C59	The right content
C24	Attractive interface	C510	Applications	C9	Assessment and evaluation
C25	Graphic layout	C511	Guide Material	C91	Different exam mode
C26	Easy participation	C512	Ease of Use	C92	Result record
C27	Offline resource	C513	Perspective	C93	Rating
C28	Interface customization	C514	Instructional design	C94	Progress follow-up
C29	Computer knowledge	C515	Pedagogical content	C95	Experience observation
C210	User friendly	C516	Transnational curriculum	C96	Exam level
C211	Learning objectives	C517	Approved curriculum	C97	Progress Control
C212	Usability	C518	Material Quantity	C98	Performance recording
C213	Easy data access			C99	Process control
C214	Color match	C6	Collaboration Communication	C910	Transfer
C215	Ergonomics	C61	Information sharing
		C62	Interoperability	C10	Technical Specifications
C3	Function	C63	Discussion	C101	Payment
C31	Evaluation Architecture	C64	Announcement	C102	Style
C32	User control	C65	Dialogue	C103	Connection
C33	Search	C66	Forum	C104	Update
C34	Learning history	C67	Attendance	C105	Hierarchical structure
C35	Progress Control	C68	Collaboration	C106	Language support
C36	Application Architecture	C69	Mail / Messages	C107	Adaptation
C37	Counselor	C610	Conversation	C108	Access speed
C38	Discussion	C611	Question sharing	C109	Compatibility
C39	Additive content			C1010	Mobile support
		C7	Quality	C1011	Verification
C4	Security	C71	Integrity
C41	Logging	C72	Education Quality	C11	Support
C42	Authorization based access	C73	Instructor Quality	C111	Budget support
C43	Password	C74	satisfaction measurement	C112	Profile
C44	Assessment & Evaluation	C75	Material and Content	C113	Institutionalism
		C76	Reliability	C114	Reward system
		C77	Download Resources	C115	Planning
		C78	Documentation	C116	Admin
		C79	Standard	C117	Equipment support

Literature review

Today, e-learning platforms attract more attention than ever before. The issue that determines the quality of education in these platforms is the technical features of the platforms and the pedagogical method used. For this reason, the educational strategies of the platforms should be considered (Begičević et al., 2007). E-learning should be based on pedagogy. The Behaviouristic, Cognitive and Constructivistic Approach were used to compare some e-learning platforms (Moedritscher, 2006). The increase in the usage rate of e-learning platforms depends on the richness of the portal's contents. The creation and reusability of learning content are ensured by effective content libraries (Yigit et al., 2014). Increased multimedia use, high media update speed and short waiting times increase students' attentions (Lin et al., 2011). The framework (portal) does also increase learning effectiveness for all learning branches (Sahasrabudhe & Kanungo, 2014). Different multimedia content is used to support learning in e-learning platforms. A quality model has been proposed to determine the importance of factors such as media loss, video stream quality, download time, file size, user control, timeliness, user-friendliness that affect learning (Jeong & Yeo, 2014). It was shown that developing an e-learning platform according to students' needs using a student-oriented approach increases efficiency (Dominici & Palumbo, 2013).

Three important features of e-learning platforms have been defined as structure, media and communication capabilities. The factors affecting the design of e-learning platforms are theoretical orientation, learning objectives, content, student characteristics and technological ability (Susan & Kenneth, 2000). Furthermore there is a significant relationship between individual differences, academic achievement and usability of the e-learning portal (Karahoca & Karahoca, 2009). The purpose of educational platforms is to ensure that students acquire the skills they need and increase their knowledge level. However, each student’s capability and needs are different. Thus the e-learning systems should be adaptable for the various student needs and present information in different formats accordingly (Leka et al., 2016).

Multi-Criteria Decision-Making (MCDM) methods have been used in the literature to compare and evaluate e-learning platforms. The main components of the platforms were used as criteria for comparisons. These components are digital libraries of educational resources and services, learning objects and virtual learning environments (Kurilovas & Dagiene, 2009). Availability of online discussions is one of the essential factors for benchmarking e-learning platforms (Ng & Murphy, 2005). Furthermore, learning effectiveness evaluation is vital in comparing learning platforms. The hybrid MCDM evaluation method, which combines the Analytic Hierarchy Process (AHP) and fuzzy integral method, is employed to simultaneously considers the interactive relationship between the criteria and the blurriness of subjective perception. The relationship between the criteria was examined by factor analysis and Decision Making Trial And Evaluation Laboratory (DEMATEL) method (Tzeng et al., 2007).

Hwang et al. employed integrated group decision approach including fuzzy theory with the grey system theory to evaluate e-learning platforms (Hwang et al., 2004). In addition e-learning platforms were compared using consistent fuzzy preference relations (Chao & Chen, 2009). The Proximity Indexed Value (PIV), which is a multi-criteria decision-making method, has been developed to compare these platforms. Vise Kriterijumska Optimizacija I Kompromisno Resenje (VIKOR) and the Complex Proportional ASsessment (COPRAS) methods were also employed for platform evaluations (Khan et al., 2019). Moreover, a combination of AHP and Quality Function Deployment (QFD), was used to evaluate the e-learning systems (Xu et al., 2009). A combination of fuzzy logic-based Kirkpatrick and the layered evaluation framework PeRSIVA model is proposed to evaluate e-learning methods (Chrysafiadi & Virvou, 2013). Other parameters used to compare e-learning platforms include adaptability, customization, extensibility and customization. Adaptation is defined as the student's adaptation to the flow of the course (Graf & List, 2005). In addition, self-learning evaluation is recommended to compare e-learning platforms using AHP. Learning behavior, cooperation and communication, resource use and learning effect were considered as criteria in the AHP method (Chen & Yang, 2010; Mingli & Yihui, 2010). Comparison of educational platforms should focus on education issues as well as technical issues (Martin et al., 2008). AHP and artificial neural networks were used to compare the qualities and learning efficiency of the e-learning platform (Chen & Fu, 2010). A primitive cognitive network process, which considers multiple criteria and alternatives, is proposed to select the e-learning platform and this method was compared with the AHP (Yuen, 2012).

Implementation of e-learning platforms have also been examined to compare their performances. Success factors at the implementation stage were investigated using AHP with group decision-making and F-AHP (Naveed et al., 2020). AHP diagrams were used to evaluate the learning effectiveness of the web-based e-learning platforms (Murakoshi et al., 2001). Key factors affecting e-learning were examined with the help of AHP (Qin & Zhang, 2008). AHP was used for students' adoption of e-learning platforms, and 33 different factors were examined. Factors such as cost, quality, agility, timing control, degree certification and personal demands have been reported to have a significant impact on individuals' adoption of e-learning platforms (Zhang et al., 2010). A methodology has been developed to determine the difficulty levels of the e-learning platform. The Linear Program (LP) was used to evaluate the difficulty level of the questions (Matsatsinis & Fortsas, 2005). The association rule with F-AHP was used to evaluate the application score and interactive learning process in e-learning platforms (Wang & Lin, 2012). Quality function deployment (QFD) and fuzzy linear regression were used for e-learning platform selection (Alptekin & Karsak, 2011). MCDM was used to determine the quality of learning material in e-learning platforms (Kurilovas & Dagienė, 2009). Factors affecting the successful implementation of e-learning platforms were identified by AHP (Lo et al., 2011). Fuzzy mathematics was used to determine the factors affecting the effectiveness of e-learning platforms (Bo et al., 2009). AHP and F-AHP have been widely used to set criteria priorities in comparing e-learning platforms (Alptekin & Karsak, 2011; Bo et al., 2009; Chen & Fu, 2010; Lo et al., 2011; Martin et al., 2008; Murakoshi et al., 2001; Naveed et al., 2020; Qin & Zhang, 2008; Wang & Lin, 2012; Yuen, 2012). Other methods used for prioritizing the e-learning criteria can be expressed as DEMATEL, PERSIVA, LP, QFD, PIV and VIKOR [24,(Chrysafiadi & Virvou, 2013; Kurilovas & Dagienė, 2009; Matsatsinis & Fortsas, 2005). In previous studies, the adaptation of the platform student, the success factors for the implementation of e-learning platforms, and the determination of the weights of the education-related criteria was examined. (Graf & List, 2005; Naveed et al., 2020; Zhang et al., 2010).

In this study, 11 criteria (Adaptation, Framework, Function properties, Security, Content, Collaboration & Communication, Quality, Learning, Assessment and evaluation, Technical Specifications, Support) and 106 sub-criteria were defined to evaluate e-learning platforms. Main criteria and sub-criteria weights were determined by using Type-2 Fuzzy Sets AHP. This is the most comprehensive study performed on this issue with 11 criteria and 106 sub-criteria for evaluating e-learning platforms.

Material and method

The motivation of our study is to determine the weights of all critical success criteria and offer a reliable method for evaluating e-learning platforms. The decision model was structured hierarchically in Fig. 1 to prioritize all critical success factors for e-learning platforms. In the first level the goal of this study is explained as determining the priorities of critical success factors in e-learning systems. The criteria handled in this study are C1-adaptation, C2-framework, C3-function, C4-security, C5-content, C6-collabration & communication, C7-quality, C8-learning, C9-assesment and evaluation, C10-technicial specifications, C11-management support. The vagueness and subjectivity of trainee and trainer are taken into account by linguistic parameters of interval valued trapezoidal fuzzy numbers.

Fig. 1

AHP Model

Evaluation Criteria

In this study, 11 criteria and 106 sub-criteria in Table 1 for e-learning platforms comparison were defined, evaluated and prioritized. All these criteria and sub-criteria were determined and defined by conducting a deep literature survey. The criteria are determined as adaptation (C1), framework (C2), function (C3), security (C4), content (C5), cooperation and communication (C6), quality (C7), learning (C8), assessment and evaluation (C9), technical specifications (C10), and support (C11).

The C1-adaptation is based on the 4 sub-criteria, e.g C11-compatibility, C12-extendibility, C13-customization and C14-adaptability. C11-compatibility means high correlation between user needs and system features in terms of learning material and framework. C12-extendibility is architecture and content designed according to subsequent needs. C13-customization means user can customize the platform according to his needs. C14-adaptability means that platform can be customized according to user needs.

The C2-framework is based on the 15 sub-criteria. These are C21-warning/message system, C22-ease of understanding, C23-clear navigation, C24-attractive interface, C25-graphics layout, C26-easy participation, C27-offline resource, C28-interface customization, C29-computer knowledge, C210-user friendly, C211-learning objectives, C212-usability, C213-easy data access, C214-color match, C215-ergonomic. C21-warning/message system means that the warning and error messages guide the user with clear and understandable precaution. C22-ease of understanding is that the label and routing in the interface are clear and understandable. C23-clear navigation is that the menus and subpage hierarchy within the portal are clear and understandable. C24-attractive interface is a platform design that is engaging, simple and focused on learning material. C25-graphics layout refers to the placement of figures, tables, videos on the screen. C26-easy participation donates that participation in the training is straightforward, and there are no complicated links. C27-offline resource is that students can have offline access to pre-downloaded resources. C28-interface customization refers that the interface can be customized according to the needs of the user. C29-computer knowledge is the level of computer usage knowledge that the user will need to use the portal. C210-user friendly means that portal has an easy-to-understand and easy-to-use interface and a nice look, as well as a good user experience. C211-learning objectives is the ability of users to achieve the intended learning goal with the course and training. C212-usability is implied that the portal does not have design-related errors or omissions that have been overlooked. C213-easy data access points that educators can share additional data, documents with course content, and students can easily access the content provided. C214-color match is portal design includes harmonious and combined colors. C215-ergonomic betoken designing portal according to ergonomic criteria.

The C3-function criteria is based on 9 sub-criteria. These are C31-evaluation architecture, C32-user control, C33-search, C34-learning history, C35-progress control, C36-application architecture, C37-counselor, C38-discussion, and C39-additive content. C31-evaluation architecture is that a new evaluation process can be designed for training. C32-user control is the student's ability to organize learning activities. C33-search imply that the ability of searching for training/content on the portal with different parameters. C34-learning history is that the student can see the completed training and share the system certificates. C35-progress control donates that the student can see his progress in the training he has received. C36-application architecture is the student/trainer's ability to define a predecessor or successor relationship between training specifically for the student. C37-counsellor is that the student can access the advisor online 24/7. C38-discussion indicates that technical tools allow students to open a new topic within the portal and share their ideas. C39-additive content shows that trainers can add teaching materials (media, pictures, documents) to the portal.

The C4-security are based on 4 sub-criteria. The sub-criteria of C4-security are C41-logging, C42-authorization based access, C43-password and, C44- assessment & evaluation. C41-logging is the registration of user access information. C42-authorization based access is means that users have different menu groups and data authorization according to the authority they have. C43-password shows that the portal has strong password management. C44- assessment & evaluation refers to ensure assessment & evaluation (exam, test) security.

The C5-content is based 18 sub-criteria. These are C51-material, C52-content, C53-presentation, C54-engaging content, C55-practice/test, C56-interactive mode, C57-functional content, C58-current content, C59-right content, C510-applications, C511-guide material, C512-ease of use, C513-perspective, C514-instructional design, C515-pedagogical content, C516-transnational curriculum, C517-approved curriculum, and C518-material quantity. C51-material means that the portal has active and lively multimedia training material. C52-content means controlling the information contained on the portal and the assurance of the information quality through standards. C53-presentation refer to supervise the presentation of information and to meet certain standards requirements. C54-engaging content means attracting the attention of the student attention via rich multimedia content and making a positive contribution to learning. C55-practice/test points that the portal has good practice and test material. C56-interactive mode donates that the portal has a learning-based interactive mode. C57-functional content means using of the same educational content by different educational programs. C58-current content refers to follow the training material up-to-date. C59-right content means checking the accuracy of the training material. C510-applications refers to include practical educational applications. C511-guide material point that there are guide materials on the platform. C512-ease of use point that fit to be used of learning material. C513-perspective means learning activities have a systematic perspective. C514-instructional design shows that instructional management has a particular methodology and design. C515-pedagogical content is that pedagogical factors are taken into account in the teaching method. C516-transnational curriculum is to include transnational curriculum topics. C517-approved curriculum platform has an assessment tool and curriculum approved by the country's education authority. C518-material quantity has acceptable content.

C6-collaboration & communication criteria is based on 11 sub-criteria. These sub criteria are C61-information sharing, C62-interoperability, C63-discussion, C64-announcement, C65-dialogue, C66-forum, C67-attendance, C68-collaboration, C69-mail/messages, C610-conversation, and C611-question sharing. C61-information sharing is to provide mutual communication opportunity for the platform. C62-interoperability is to offer the possibility of working together with the means of remote access. C63-discussion is the ability to open a discussion topic and share ideas within the platform. C64-announcement is the ability to provide one-sided communication with trainers and managers. C65-dialogue has the opportunity to have a conversation with the trainer during the training. C66-forum is the ability of trainees to express their views on a topic that concerns the learning community. C67-attendance is the ability to list meeting participants. C68-collaboration is to provide the opportunity for students to work together on a particular learning topic. C69-mail/messages is the possibility to receive personal mail and messages within the platform. C610-conversation is the possibility of text chat during training. C611-question sharing is the opportunity to share questions between trainers and trainees.

C7-quality criteria is based on 9 sub-criteria. These criteria are C71-integrity, C72-education quality, C73-instructor quality, C74-satisfaction measurement, C75-material and content, C76-reliability, C77-download resources, C78-documentation, C79-standard. C71-integrity is training materials that complement each other. C72-education quality is that platform has a quality internal control system. C73-instructor quality is that trainers are subject to standardization training at certain periods. C74-satisfaction measurement is that measuring student/learning satisfaction. C75-material and content is the control of content quality. C76-reliability is that the number of unplanned failures of the system is at an acceptable level. C77-download resources documentation content count and streaming speed are sufficient. C78-documentation is the system usage document of the platform. C79-standard is that the platform has sufficient national and international education standards because it has a standard.

The C8-learning criteria is based on 8 sub-criteria. These sub-criteria are C81-organization, C82-access, C83-source, C84-threshing, C85-re-access, C86-process control, C87-feedback, C88-encourage. C81-organization is that the course objects are organized. C82-access is that the training content is accessible at different times and places. C83-source is that platform that provides access to libraries related to the subject. C84-threshing is that the training method is enriched with online training materials. C85-re-access is that there is no possibility of regaining access to the training material studied/used. C86-process control is the portals ability of the student development. C87-feedback is a timely response and feedback from the learning counsellor. C88-encourage is that students are encouraged by the trainer for discussion and feedback.

The C9-assessment and evaluation criteria is based on 10 sub-criteria. These sub criteria are C91-different exam mode, C92-result record, C93-rating, C94-progress follow-up, C95- experience observation, C96-exam level, C97-progress control, C98-recording, C99-process control, C910-transfer. C91-different exam mode is that measurement can be made with different techniques and methods. C92-result record is the regulation of recording the assessment score. C93-rating is the ability to rate trainees. C94-progress follow-up is the student's ability to follow their development. C95- experience observation is that the trainer cannot observe the learner experiences. C96-exam level is a form of exams that assess the learning level. C97-progress control is the trainee’s ability to control learning progress. C98-recording is the portal's ability to record learning performance. C99-process control is the learner's ability to control the learning process. C910-transfer is the transfer of knowledge gained in the process between the student and the teacher.

C10-technical specifications criteria is explained depend on 11 sub-criteria. These are C101-payment, C102-style, C103-connection, C104-update, C105-hierarchical structure, C106-language support, C107-adaptation, C108-access speed, C109-compatibility, C1010-mobile support, C1011-verification. C101-payment is that alternative methods can pay fees. C102-style is that the interface style is selectable. C103-connection is that the platform internet connection is stable. C104-update is to be maintained and updated regularly. C105-hierarchical structure is that the structure is that the sections and subsections of the tutorials and pages are clearly defined. C106-language support is that the platform has multi-language support. C107-adaptation is that feedback and adaptation are possible during training. C108-access speed is that the speed of accessing the content is sufficient. C109-compatibility is that the matching of metadata to content. C1010-mobile support is that it has mobile system support. C1011-verification is that the platform has data transmission verification.

The C11-support criteria is based on 7 sub-criteria. These sub criteria are C111-budget support, C112-profile, C113-institutionalism, C114-reward system, C115-planning, C116-admin, C117-equipment support. C111-budget support is the provision of necessary financial support by senior management or access to financial resources. C112-profile is personal user profile and account management. C113-institutionalism is that the unit that organizes the training is institutional. C114-reward system is that it is a certificate etc., reward mechanism for learning activities. C115-planning is the alignment of platform plans and activities. C116-admin is that the system has a maintenance operation and maintenance application manager. C117-equipment support is information technology equipment is sufficient.

Interval type-2 fuzzy sets

In this section the interval type-2 fuzzy sets is briefly explained and some definitions are given (Çalık and Paksoy, 2017; Kahraman et al, 2014).

Definition 1

A type- 2 fuzzy sets in the universe of discourse X can be represented by a type-2 membership function shown as follows:

where J denotes an interval [0,1]. The type-2 fuzzy set may also be expressed as follows:

where and represents the combination of all reasonable (acceptable) values of x and u.

Definition 2

If all , the set is called interval type-2 fuzzy set. A specific instance of a type-2 fuzzy set is an interval type-2 fuzzy set, which may be described as:

where .

Definition 3

The lower and upper membership functions of the interval type-2 fuzzy set are type-1 membership functions, respectively. Chen and Lee (2010) presented a new approach for employing interval type-2 fuzzy sets in solving fuzzy multi-criteria group decision-making issues in their investigations. The heights of the upper and lower membership functions, as well as the reference points of the interval type-2 fuzzy sets, were utilized to describe the type-2 fuzzy sets using this technique. The following item represents a trapezoidal interval type-2 fuzzy set.

where, and are type-1 fuzzy sets. , are the reference point of interval type-2 fuzzy set . denotes the membership value of the element in the upper trapezoidal membership function , . denotes the membership value of the element in the lower trapezoidal membership function , . , ,, and .

Definition 4

The following equation shows the summation operator of the trapezoidal interval type-2 fuzzy clusters:

Definition 5

The following is the procedure for subtracting the trapezoidal interval type-2 fuzzy sets:

Definition 6

The following is the multiplication between trapezoidal interval type-2 fuzzy clusters:

Definition 7

The followings are the arithmetic operations between trapezoidal interval type-2 fuzzy sets and scalar k:

where k>0.

Interval Type-2 Fuzzy AHP

We have employed Interval Type-2 Fuzzy AHP for evaluating our criteria. Kahraman et al. (2014) extended Buckley's (1985) fuzzy AHP approach based on type-1 fuzzy clusters into interval type-2 fuzzy clusters (Buckley, 1985; Çalık & Paksoy, 2017; Kahraman et al., 2014). The AHP technique was developed by Saaty (Saaty, 1980) and it has been used to find solutions to or help solve decision-making problems in various fields onward the day it was developed (risk assessment (Adem et al., 2018), green ergonomics (Adem et al., 2021), occupational health and safety (Adem et al., 2020), machine selection (Özceylan et al., 2016); site selection (Paul et al., 2021); vendor selection Gernowo & Surarso, 2021; data intelligence implementation (Merhi, 2021); green energy (Asadi and Pourhossein, 2021)).

E-learning platforms factors evaluation process with Interval Type-2 Fuzzy Sets is presented in Fig. 2. This method's steps are outlined as follows (Kahraman et al., 2014; Çalık & Paksoy, 2017):

Fig. 2

E-learning platforms factors evaluation process with interval type-2 fuzzy sets

Step 1: Determine the criteria, sub-criteria and alternatives of the determined decision-making problem. (Decision-making problems may contain all or some of the elements listed here.)

Step 2: Table 2 lists linguistic variables and associated interval type-2 fuzzy scales. Fuzzy pairwise comparison matrices are generated using linguistic variables, as shown in Eq (1).

Table 2

Definition and interval type 2 fuzzy scale of the linguistic variables

Linguistic Variables	Trapezoidal interval type-2 fuzzy scales
Absolutely Strong (AS)	(7,8,9,9;1,1) (7.2,8.2,8.8,9;0.8,.08)
Very Strong (VS)	(5,6,8,9;1,1) (5.2,6.2,7.8,8.8;0.8,0.8)
Fairley Strong (FS)	(3,4,6,7;1,1) (3.2,4.2,5.8,6.8;0.8,0.8)
Slightly Strong (SS)	(1,2,4,5;1,1) (1.2,2.2,3.8,4.8;0.8,0.8)
Exactly Equal (EE)	(1,1,1,1;1,1) (1,1,1,1;1,1)

Where ,

Step 3: If there is more than one expert in the decision-making process, the experts' judgements need to be aggregated with the help of the geometric mean. The calculation details of the geometric mean are shown as follows:

where,

Step 4: The fuzzy weights of each criterion are calculated. First of all, , the geometric mean of each row is computed. After that, the fuzzy weight of criterion () is calculated as follows:

where

Step 5: The computed weights must be defuzzied because they are in the form of interval type-2 fuzzy. The defuzzification procedure is based on the following formula:

Results and discussion

In this part of the paper, the criteria that affect the success of e-learning systems were prioritized by utilizing interval valued type-2 fuzzy AHP technique. 11 criteria and 106 sub-criteria were determined by conducting a deep literature survey, and the determined criteria were shown in Table 1. The joint fuzzy evaluations of a team of experts and calculation details of 11 criteria weights are shown in Tables 3, 4, 5 and 6.

Table 3

The linguistic evaluations of the main criteria

	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11
C1	EE	1/SS	FS	SS	SS	VS	SS	1/SS	1/SS	VS	VS
C2	SS	EE	VS	SS	VS	AS	1/SS	1/SS	1/VS	SS	AS
C3	1/FS	1/VS	EE	1/SS	SS	FS	1/SS	1/FS	SS	SS	FS
C4	1/SS	1/SS	SS	EE	SS	VS	SS	VS	1/SS	SS	FS
C5	1/SS	1/VS	1/SS	1/SS	EE	SS	SS	1/SS	1/VS	1/SS	1/VS
C6	1/VS	1/AS	1/FS	1/VS	1/SS	EE	1/SS	1/SS	1/VS	1/VS	1/SS
C7	1/SS	SS	SS	1/SS	1/SS	SS	EE	SS	1/SS	SS	VS
C8	SS	SS	FS	1/VS	SS	SS	1/SS	EE	1/SS	VS	VS
C9	SS	VS	1/SS	SS	VS	VS	SS	SS	EE	VS	AS
C10	1/VS	1/SS	1/SS	1/SS	SS	VS	1/SS	1/VS	1/VS	EE	FS
C11	1/VS	1/AS	1/FS	1/FS	VS	SS	1/VS	1/VS	1/AS	1/FS	EE

Table 4

Geometric means of main criteria

Main Criteria	Geometric means of criteria
C1		(1.11,1.53,2.51,3.37;1,1) (1.18,1.61,2.38,3.14;0.8,0.8)
C2		(1.17,1.57,2.37,2.98;1,1) (1.24,1.64,2.27,2.82;0.8,0.8)
C3		(0.52,0.71,1.17,1.56;1,1) (0.56,0.75,1.11,1.46;0.8,0.8)
C4		(0.95,1.39,2.35,3.20;1,1) (1.03,1.46,2.23,2.97;0.8,0.8)
C5		(0.26,0.34,0.57,0.86;1,1) (0.27,0.35,0.54,0.78;0.8,0.8)
C6		(0.17,0.19,0.29,042;1,1) (0.17,0.20,0.28,0.38;0.8,0.8)
C7		(0.64,0.97,1.76,2.54;1,1) (0.70,1.04,1.65,2.32;0.8,0.8)
C8		(0.90,1.30,2.12,2.76;1,1) (0.98,1.37,2.03,2.60;0.8,0.8)
C9		(1.85,2.63,4.04,4.87;1,1) (2.01,2.77,3.89,4.68;0.8,0.8)
C10		(0.39,0.48,0.76,1.09;1,1) (0.40,0.50,0.72,0.99;0.8,0.8)
C11		(0.25,0.29,0.39,0.47;1,1)(0.26,0.30,0.38,0.45;0.8,0.8)

Table 5

Fuzzy weights of criteria

Main Criteria	Fuzzy weights of criteria
C1	(0.05,0.08,0.22,0.41;1,1) (0.05,0.09,0.20,0.36;0.8,0.8)
C2	(0.05,0.09,0.21,0.36;1,1) (0.06,0.09,0.19,0.32;0.8,0.8)
C3	(0.02,0.04,0.10,0.19;1,1) (0.02,0.04,0.09,0.17;0.8,0.8)
C4	(0.04,0.08,0.21,0.39;1,1) (0.05,0.08,0.19,0.34;0.8,0.8)
C5	(0.01,0.02,0.05,0.10;1,1) (0.01,0.02,0.04,0.09;0.8,0.8)
C6	(0.01,0.01,0.03,0.05;1,1) (0.01,0.01,0.02,0.04;0.8,0.8)
C7	(0.03,0.05,0.15,0.31,1,1) (0.03,0.06,0.14,0.26;0.8,0.8)
C8	(0.04,0.07,0.19,0.34;1,1) (0.04,0.08,0.17,0.29;0.8,0.8)
C9	(0.08,0.14,0.35,0.59,1,1) (0.09,0.16,0.32,0.53;0.8,0.8)
C10	(0.02,0.03,0.07,0.13;1,1) (0.02,0.03,0.06,0.11;0.8,0.8)
C11	(0.01,0.02,0.03,0.06;1,1) (0.01,0.02,0.03,0.05;0.8,0.8)

Table 6

Defuzzied and Normalized weights of main criteria

Main Criteria	Defuzzied Weights	Normalized Weights
C1	0,17	0,138
C2	0,16	0,128
C3	0,08	0,064
C4	0,16	0,129
C5	0,04	0,033
C6	0,02	0,017
C7	0,12	0,098
C8	0,15	0,115
C9	0,27	0,214
C10	0,06	0,044
C11	0,03	0,022

Equation (2) was utilized to compute the geometric mean of each criterion. To illustrate this calculation, how to calculate the geometric mean of the first criterion is shown as follows:

After all the main criteria's geometric mean, now, the fuzzy weights of each criterion can be computed. Table 4 shows the geometric mean of each criterion.

Equation (3) was employed to compute fuzzy weights of criteria. To illustrate this calculation, how to calculate the fuzzy weights of the first criterion is shown as follows:

Table 5 shows the fuzzy weights of criteria. The computed weights must be defuzzied because they are in the form of interval type-2 fuzzy.

By utilizing Eq. (4), the computed fuzzy weights are defuzzied. Table 6 shows the defuzzied and normalized weights of the main criteria.

The same steps were repeated for all sub-criteria, and the local weights of the sub-criteria were calculated. Table 7 shows the global weights of the sub-criteria obtained by multiplying the weights of the main and sub-criteria.

Table 7

Weights and global weights of sub-criteria

Criteria	Weights	Global Weights	Criteria	Weights	Global Weights	Criteria	Weights	Global Weights
C1	0,138		C5	0,033		C8	0,115
C11	0,356	0,049	C51	0,085	0,003	C81	0,075	0,009
C12	0,201	0,028	C52	0,123	0,004	C82	0,173	0,020
C13	0,183	0,025	C53	0,112	0,004	C83	0,195	0,022
C14	0,259	0,036	C54	0,064	0,002	C84	0,262	0,030
			C55	0,035	0,001	C85	0,088	0,010
C2	0,128		C56	0,035	0,001	C86	0,117	0,013
C21	0,036	0,005	C57	0,079	0,003	C87	0,050	0,006
C22	0,146	0,019	C58	0,062	0,002	C88	0,040	0,005
C23	0,062	0,008	C59	0,059	0,002
C24	0,052	0,007	C510	0,045	0,002	C9	0,214
C25	0,026	0,003	C511	0,030	0,001	C91	0,182	0,039
C26	0,117	0,015	C512	0,141	0,005	C92	0,165	0,035
C27	0,024	0,003	C513	0,030	0,001	C93	0,048	0,010
C28	0,094	0,012	C514	0,039	0,001	C94	0,059	0,013
C29	0,030	0,004	C515	0,019	0,001	C95	0,030	0,006
C210	0,090	0,012	C516	0,013	0,000	C96	0,105	0,022
C211	0,037	0,005	C517	0,015	0,000	C97	0,062	0,013
C212	0,072	0,009	C518	0,014	0,000	C98	0,041	0,009
C213	0,029	0,004				C99	0,045	0,010
C214	0,017	0,002	C6	0,017		C910	0,264	0,056
C215	0,167	0,021	C61	0,233	0,004
			C62	0,175	0,003	C10	0,044
C3	0,064		C63	0,145	0,002	C101	0,022	0,001
C31	0,221	0,014	C64	0,103	0,002	C102	0,043	0,002
C32	0,250	0,016	C65	0,034	0,001	C103	0,161	0,007
C33	0,031	0,002	C66	0,022	0,000	C104	0,236	0,010
C34	0,086	0,006	C67	0,075	0,001	C105	0,028	0,001
C35	0,045	0,003	C68	0,059	0,001	C106	0,050	0,002
C36	0,155	0,010	C69	0,058	0,001	C107	0,142	0,006
C37	0,066	0,004	C610	0,042	0,001	C108	0,121	0,005
C38	0,097	0,006	C611	0,054	0,001	C109	0,077	0,003
C39	0,049	0,003				C1010	0,043	0,002
			C7	0,098		C1011	0,079	0,003
C4	0,129		C71	0,154	0,015
C41	0,225	0,029	C72	0,260	0,025	C11	0,022
C42	0,175	0,022	C73	0,030	0,003	C111	0,380	0,008
C43	0,111	0,014	C74	0,163	0,016	C112	0,071	0,002
C44	0,490	0,063	C75	0,196	0,019	C113	0,200	0,004
			C76	0,054	0,005	C114	0,041	0,001
			C77	0,063	0,006	C115	0,191	0,004
			C78	0,042	0,004	C116	0,048	0,001
			C79	0,039	0,004	C117	0,070	0,002

According to the weight values, the first three main criteria were C9-Assessment and Evaluation, C1-Adaptation and C4-Security with 0.214, 0.138, 0.129 weight values respectively. The ranked weights of 11 criteria are presented in Table 8. According to the global weight values first ten sub-criteria are C44- assessment & evaluation security, C910-transfer, C11-compatibility, C91-different exam mode, C14-adaptability, C92-result record, C84-threshing, C41-logging, C12-extendibility and C13-customization. The ranked global weights of 106 sub-criteria are presented in Table 9.

Table 8

Ranked weights of criteria

Criteria		Weights
C9	Assessment and evaluation	0.214
C1	Adaptation	0.138
C4	Security	0.129
C2	Framework	0.128
C8	Learning	0.115
C7	Quality	0.098
C3	Function	0.064
C10	Technical Specifications	0.044
C5	Content	0.033
C11	Management Support	0.022
C6	Collaboration & Communication	0.017

Table 9

The ranked global weights of sub-criteria

Rank	Sub-criteria		Global Weights	Rank	Sub-criteria		Global Weights
1	C44	Assessment & evaluation	0.063	54	C29	Computer knowledge	0.004
2	C910	Transfer	0.056	55	C213	Easy data access	0.004
3	C11	Compatibility	0.049	56	C37	Counselor	0.004
4	C91	Different exam mode	0.039	57	C52	Content	0.004
5	C14	Adaptability	0.036	58	C53	Presentation	0.004
6	C92	Result record	0.035	59	C61	Information sharing	0.004
7	C84	Threshing	0.03	60	C78	Documentation	0.004
8	C41	Logging	0.029	61	C79	Standard	0.004
9	C12	Extendibility	0.028	62	C113	Institutionalism	0.004
10	C13	Customization	0.025	63	C115	Planning	0.004
11	C72	Education Quality	0.025	64	C25	Graphic layout	0.003
12	C42	Authorization based access	0.022	65	C27	Offline resource	0.003
13	C83	Source	0.022	66	C35	Progress Control	0.003
14	C96	Exam level	0.022	67	C39	Additive content	0.003
15	C215	Ergonomic	0.021	68	C51	Material	0.003
16	C82	Access	0.02	69	C57	Functional content	0.003
17	C22	Ease of understanding	0.019	70	C62	Interoperability	0.003
18	C75	Material and content	0.019	71	C73	Instructor Quality	0.003
19	C32	User control	0.016	72	C109	Compatibility	0.003
20	C74	Satisfaction measurement	0.016	73	C1011	Verification	0.003
21	C26	Easy participation	0.015	74	C214	Color match	0.002
22	C71	Integrity	0.015	75	C33	Search	0.002
23	C31	Evaluation Architecture	0.014	76	C54	Engaging content	0.002
24	C43	Password	0.014	77	C58	Current content	0.002
25	C86	Process control	0.013	78	C59	The right content	0.002
26	C97	Progress control	0.013	79	C510	Applications	0.002
27	C94	Progress follow-up	0.013	80	C63	Discussion	0.002
28	C28	Interface customization	0.012	81	C64	Announcement	0.002
29	C210	User friendly	0.012	82	C102	Style	0.002
30	C36	Application architecture	0.01	83	C106	Language support	0.002
31	C85	Re-access	0.01	84	C1010	Mobile support	0.002
32	C93	Rating	0.01	85	C112	Profile	0.002
33	C99	Process control	0.01	86	C117	Equipment support	0.002
34	C104	Update	0.01	87	C55	Practice/test	0.001
35	C212	Usability	0.009	88	C56	Interactive mode	0.001
36	C81	Organization	0.009	89	C511	Guide Material	0.001
37	C98	Performance recording	0.009	90	C513	Perspective	0.001
38	C23	Clear navigation	0.008	91	C514	Instructional design	0.001
39	C111	Budget support	0.008	92	C515	Pedagogical content	0.001
40	C24	Attractive interface	0.007	93	C65	Dialogue	0.001
41	C103	Connection	0.007	94	C67	attendance	0.001
42	C34	Learning history	0.006	95	C68	Collaboration	0.001
43	C38	Discussion	0.006	96	C69	Mail / Messages	0.001
44	C77	Download resources	0.006	97	C610	Conversation	0.001
45	C87	Feedback	0.006	98	C611	Question sharing	0.001
46	C95	Experience observation	0.006	99	C101	Payment	0.001
47	C107	Adaptation	0.006	100	C105	Hierarchical structure	0.001
48	C21	Warning/message system	0.005	101	C114	Reward system	0.001
49	C211	Learning objectives	0.005	102	C116	Admin	0.001
50	C512	Ease of use	0.005	103	C516	Transnational curriculum	0.0005
51	C76	Reliability	0.005	104	C517	Approved curriculum	0.0005
52	C88	Encourage	0.005	105	C518	Material quantity	0.0005
53	C108	Access speed	0.005	106	C66	Forum	0.0005

The most important criterion is the “C9-assessment and evaluation” means that the level of knowledge and ability of the student varies so the achievements of students in the e-learning should be measured with sufficient accurate measurement techniques at different stages of the learning progress. The second important criterion is the “C1-adaptation” implies the harmony between users’ needs, architecture, and framework should be considered simultaneously. The third important criterion is the “C4-security” indicates that user information, authorization-based access, ensuring the security of measurement and evaluation system should be considered.

The most important sub-criterion according to global weight is “C44-sssessment & evaluation” emphasizes performing complete and inclusive assessment & evaluation processes, and taking anti-cheating precaution. The second important sub-criterion is “C910-transfer” means transferring knowledge gained in the process between students and teacher. The third important sub-criterion is “C11-compatibility” which shows correlation between user needs and system features in terms of learning material and framework. The fourth important sub-criterion is “C91-different exam mode” stands for measurement the level of learning which can be made with different techniques and methods. The fifth important sub-criterion is “C14-adaptability” means that platforms should be customized according to user needs.

Conclusion

E-learning is a teaching model that supports access to information via digital platforms or media. Distance education is learning, and knowledge management activities carried out through internet technologies. Today, e-learning platforms attract more attention due to COVID 19 pandemic. The individuals have had the opportunity to try the advantages of e-learning platforms for themselves throughout the COVID-19 pandemic. The need for e-learning platforms has increased by users and educational institutions. Different e-learning systems have been developed. The main problem in this area is assessment process includes uncertainty and qualitative assessment. Another problem is the need to examine the factors to be used in the evaluation of changing and developing platforms.

In this study, the criteria to be used to evaluate e-learning platforms were determined by a comprehensive literature review. The fuzzy analytic hierarchy method was used to compare the e-learning platforms. 11 criteria and 106 sub-criteria were determined to evaluate e-learning platforms. According to the weight values, the first three main criteria were C9-Assessment and Evaluation, C1-Adaptation and C4-Security with 0.214, 0.138, 0.129 weight values respectively. According to the global weight values first ten sub-criteria are C44- assessment & evaluation security, C910-transfer, C11-compatibility, C91-different exam mode, C14-adaptability, C92-result record, C84-threshing, C41-logging, C12-extendibility and C13-customization.

This study provides an acceptable rationale for evaluations of e-learning platform. The results of this study can be used in real-world performance evaluations of various e-learning platforms. The effectiveness of e-learning platform with respect to our factor weights can be compared.