Probabilistic Approach to Multi-Stage Supplier Evaluation: Confidence Level Measurement in Ordinal Priority Approach.

A popular framework of the supplier selection process is usually characterized by problem definition, criteria formulation, supplier screening, and supplier selection. The literature review suggested limitations of this framework as it ignores the screening of criteria (beyond criteria weighing) and evaluators (buyers) and its inability to guide the supplier selection problems where a measure of confidence or trust is needed to confirm the reliability of the selected supplier. While extending de Boer's influential supplier selection framework, the current study argues that the supplier selection problem is not merely about ranking suppliers based on given criteria; instead, it involves evaluating criteria and evaluators as well. Guided by the theory of statistics and the Ordinal Priority Approach (OPA), the study pioneers a probabilistic approach of supplier evaluation and selection under incomplete information using a novel Confidence Level measure. The study suggests, the probability that a supplier shortlisted for selection is actually the optimum choice or not can be explained through a probability distribution, called W-distribution, therefore, confidently preventing the decision-makers from selecting the sub-optimum suppliers. The study presents a novel contribution to the theory of multiple-attribute decision-making through the OPA. The proposed approach can help build intelligent decision support systems to aid managers while providing them with early warning tools and suggestions to improve confidence in their selection.

Introduction

As the fourth industrial revolution solidifies its gains, fundamental changes in our way of living and doing business have been observed. Further, one frequently witness technology at the heart of the natural (e.g., the COVID-19 pandemic) and man-made problems (e.g., trade disputes, pollution) either as a solution to a problem or a cause of new problems or conflicts. Scholars have begun arguing for structural changes in supply chains to adapt to these events that have been taking place around us recently (Kaur & Prakash Singh, 2021). For instance, the focus on supply chain resilience, multi-sourcing, and digitalization that we are witnessing today was hardly ever seen before (Colback, 2020; Fu, 2020; Khurana et al., 2021; Shih, 2020). Not only the supply chains are required to undergo structural changes; instead, it seems the technologies and decision support systems the people had been using to manage their supply and suppliers are also undergoing structural changes. A trend toward hybrid models is increasingly revealing the limitations of individual models and systems. For instance, Kaur and Prakash Singh (2021) used three models – Data Envelopment Analysis (DEA), Fuzzy Analytical Hierarchical Process (F-AHP), and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) – for supplier selection and evaluation. Each model had a different role in their multi-stage framework. Earlier, Falsini et al. (2012) had combined AHP, DEA, and linear programming to evaluate third-party logistics service providers. Li et al. (2012) had used fuzzy AHP to determine criteria weights and DEA to determine alternative suppliers, then the axiomatic fuzzy sets method to make the final selection and evaluation. Also, if these studies had evaluated criteria and buyers as well – an emerging practice – their methodologies could have been more complicated. Clearly, this trend toward hybridization brings some benefits but at the expense of increased computational cost and complexities, thus rarely finding their use beyond academic discussions. Also, this trend guides us to a gap in literature and scholarship – a need for disruptive technology that holds within it the benefits of multiple individual techniques and models while providing convenience to the business decision-makers.

Ordinal Priority Approach (OPA) is a promising new development in the multi-attribute decision-making discipline. Ataei et al. (2020) proposed the OPA and compared it with six other models; AHP, BWM, TOPSIS, VIKOR, PROMETHEE, and QUALIFLEX (see the last row of Table 1 for full names). Their results, and those of succeeding studies (Abdel-Basset et al., 2022; Pamucar et al., 2022; Quartey-Papafio et al., 2021), revealed that the methodology of OPA enjoys several benefits compared with other models. For instance, it has been shown that the OPA does not require a pairwise comparison matrix, normalization of input, and averaging methods for aggregating experts' opinions. Also, unlike other techniques, it allows the experts not to give an opinion when they lack sufficient information, thus paving the way for handling supplier selection problems under incomplete information. Meanwhile, de Boer’s four-stage supplier selection framework (de Boer et al., 2001) is a popular framework of the supplier selection process that has inspired many works since its introduction in the 1990s. However, recent developments in supply chain management have prompted that not only de Boer’s supplier selection framework has outlived its usefulness, but when one extends this framework to meet today’s needs better, new problems emerge that no single existing supplier selection technique is capable of handling. The framework involves four stages; problem definition, criteria formulation, supplier qualification analysis, and choosing a supplier. As the literature demonstrates the significance of screening the decision-makers (Bali et al., 2013; Boran et al., 2009) and estimating the reliability of criteria (Fu, 2020) for supplier selection problems, one finds that mere screening of the suppliers and limiting a methodology to merely criteria weighing no longer serves the needs of the modern-day supply chains, which have evolved a lot since de Boer proposed his four-stage framework. Also, as the importance of defining optimum suppliers based on some measure of trust or confidence is coming to attention (Mohammadi & Rezaei, 2020; Zha et al., 2019), one recognizes not only the limitations of de Boer’s supplier selection framework but also of existing supplier selection techniques. Thus, extending de Boer’s framework and proposing a reliable technique that can make the new framework applicable to modern-day real-world supplier selection problems is a challenging task before us.

Table 1

A comparison of the proposed framework with other studies

Literature	Core approach	Criteria weights	Evaluators’ weights	Suppliers’ weights	Suppliers’ ranking	Statistical testing of final ranking’s reliability	Can suggest corrective action?	Consistent approach of weighting?	Can handle missing values in input data?
Timmerman (1986)	Linear averaging (weighted-point averaging)	√	-	-	√	-	-	-	-
Nydick and Hill (1992)	AHP	√	-	√	√	-	-	-	-
Golmohammadi et al. (2009)	GA based NN model	√	-	-	√	-	-	-	-
Bai and Sarkis (2010)	RS and GST based integrated approach	-	-	-	√	-	-	-	-
Kuo et al. (2010)	ANN, ANP, DEA	√	-	-	√	-	-	-	√
Büyüközkan and Çifçi, (2011)	An integrated fuzzy approach based on the theories of FPR and ANP	√	-	√	√	-	-	√	√
Shemshadi et al. (2011)	Entropy, Fuzzy VIKOR	√	-	-	√	-	-	-	-
Bali et al. (2013)	IFS based GRA	√	√	-	√	-	-	-	-
Rajesh and Ravi (2015)	GRA	√	-	-	√	-	-	-	-
Liou et al. (2016)	Hybrid COPRAS-G MADM Model	√	-	-	√	-	-	-	-
Su et al. (2016)	Grey DEMATEL	√	-	-	√	-	-	-	-
Zhang et al. (2022)	GRA with fuzzy sets	√	-	-	√	-	-	-
Asadabadi (2017)	ANP, QFD and Markov chain	√	-	√	√	-	-	-	-
Govindan et al. (2017)	An integrated PROMETHEE-based framework	√	-	-	√	-	-	-	-
Luthra et al. (2017)	AHP, VIKOR	√	-	-	√	-	-	-	-
Awasthi et al. (2018)	Fuzzy AHP, fuzzy VIKOR	√	-	-	√	-	-	-	-
Banaeian et al. (2018)	Fuzzy VIKOR, fuzzy TOPSIS and fuzzy GRA	√	-	-	√	-	-	-
Kannan (2018)	Fuzzy Delphi, COPRAS-G, ANP	√	-	-	√	-	-	-	-
dos Santos et al. (2019)	The Entropy method, Fuzzy TOPSIS	√	-	-	√	-	-	-	-
Haeri and Rezaei (2019)	Grey-BWM	√	-	-	√	-	-	-	-
Zeng et al. (2019)	Pythagorean Fuzzy Group MCDM	√	-	-	√	-	-	-	-
Kannan et al. (2020)	Fuzzy BWM, Interval VIKOR	√	-	-	√	-	-	-	-
The current study	Improved OPA with CL measurement	√	√	√	√	√	√	√	√

AHP: Analytic Hierarchy Process; ANP: Analytic Network Process; BWM: Best Worst Method; COPRAS-G: Grey Complex Proportional Assessment; CL: Confidence Level; DEA: Date Envelopment Analysis; DEMATEL: Decision Making Trial and Evaluation Laboratory; FPR: Fuzzy preferential relations; GA: Genetic Algorithm; GRA: Grey Relational Analysis; GST: Grey system theory; IFS: Intuitionistic fuzzy sets; NN: Neural Network; PROMETHEE: Preference Ranking Organization Method for Enrichment Evaluations; QFD: Quality function deployment; RS: Rough sets; TOPSIS: Technique for Order of Preference by Similarity to Ideal Solution; VIKOR: VlseKriterijumska Optimizacija I Kompromisno Resenje; QUALIFLEX: QUALItative FLEXible multiple criteria method

The current study intends to fill these gaps in the literature by proposing a novel measure of confidence based on the OPA and Kendall’s W, identifying the position of the shortlisted supplier on a probability distribution curve of optimum suppliers, extending de Boer’s four-stage framework to seven-stage framework, and demonstrating the application of the proposed system on a real-world case. The contributions made by the current study can be utilized to develop a statistically significant, intelligent, and robust supply chain decision support system. The rest of the paper has been organized as follows: the second section reviews the relevant literature, the OPA model, and extends de Boer’s framework. The third section presents the development of the proposed Confidence Level Measurement system from scratch. The fourth section applies the proposed framework to two cases and discusses the significance of the results. In the last section, the study is concluded with important implications.

Literature Review

The Supplier Selection Techniques

Since the supplier selection problem aims to find the most suitable supplier, the problem is frequently studied in supply chain management, production management, and decision-making literature. The scholars employed various decision-making techniques and tools to overwhelm supplier selection problems. Such techniques mainly include mathematical programming, artificial intelligence, and multiple attributes decision-making (MADM).

Amin and Zhang (2013) used a mathematical programming approach for supplier selection and evaluation. They employed a mixed-integer nonlinear programming approach and proposed a three-stage model for supplier selection and assessment. Moreover, they used the fuzzy set to consider the uncertainties of input data. Talluri et al. (2013) utilized DEA to calculate the suppliers' efficiency. They stated that their model could determine performance diversity, an important concern for supply chain risk management. Babić and Perić (2014) employed fuzzy multi-objective and mixed-integer programming approaches to deal with the multiproduct vendor selection problem. They believed that suppliers suggest quantity discounts in most cases. Hence, formulating multiproduct can facilitate considering supplier discounts during the supplier selection problem. Purohit et al. (2016) proposed integer linear programming covering two important assumptions, including inventory lot-sizing and non-stationary stochastic demand for the suppliers. Amorim et al. (2016) proposed a mixed-integer programming model that contains two stages for supplier selection problems under uncertainty. Their model followed two important goals include maximizing profit and minimizing the risks. The performance of the model was validated through a case study in the food industry. Hu and Yu (2016) presented a new model with the aid of goal programming and voting method to address the electronic contract manufacturer selection problem. Their model contained two stages while they used the voting method to calculate the weights of the criteria in the first step, and then utilized weights from the first step into the second step as weights of the object functions in the goal programming model. Sawik (2016) presented a multi-objective stochastic mixed-integer programming for selection, production schedule, and shipment in supply chain management. He compared three shipment systems in mixed-integer programming, which could balance the expected service level and cost. Adeinat and Ventura (2018) presented a mixed-integer nonlinear programming model for selection, pricing, and inventory matters in supply chain management. Their model could calculate the number of orders placed, inventory lot size, among others. Arampantzi et al. (2019) presented a framework based on mixed-integer linear programming for strategic decisions related to supply chain management. These decisions included supplier selection, selecting suitable routes for transportation, designing and redesigning production facilities, etc. Manerba et al. (2018) proposed a stochastic programming model for capacitated supplier selection problem. They stated that their model was established based on activation costs and total quantity discount strategy, suitable for companies with different products. It should be noted that in the group of mathematical programming, most studies used linear programming for supplier selection problems.

Some researchers utilized artificial intelligence and applied mathematics to solve the supplier selection problem. Chai and Liu (2014) proposed a new believable rough set method for the supplier selection problem. They declared that their model exhibits high performance for problem-solving as it provides a complete procedure, including a scheme for rule application, criteria analysis, decision rule induction, and rough approximation. Memon et al. (2015) employed the grey system theory for supplier selection problems while decreasing the purchasing risks. They stated that the grey system theory does not need membership function or probability distribution and is more practical in a real-world situation. Diba and Xie (2019) also used the grey system theory for a sustainable supplier selection problem. Both of these studies stated that the grey system theory can produce reliable results for supplier selection when the sample size is small and the probability distribution and membership function are unknown. Nepal and Yadav (2015) presented a novel framework to consider the risk and cost factors for global supplier selection problems. Their study has provided a visual technique to illustrate the effect of cost and risk in the supplier selection process. Hosseini and Barker (2016) presented a Bayesian Network, a useful method to address the relationship among variables for supplier selection and evaluation variables. Their study was guided by the supplier resilience approach. Jabbarzadeh et al. (2018) proposed a hybrid model for sustainable supplier selection by considering random disruptions. They used the fuzzy c-means clustering method to evaluate the suppliers' performance in sustainable criteria. It is worth mentioning that in the artificial intelligence category, most studies employed K-means clustering and Bayesian Networks methods.

Many studies utilized MADM methods to solve supplier selection problems. Abdollahi et al. (2015) used the analytical network process (ANP) for supplier portfolio selection. They stated that the ANP method is suitable for calculating the criteria' weights, followed by the DEA method to rank the supplier. The main reason for employing the ANP was its ability to consider the interaction among the criteria. However, it is out of use when there are many suppliers, which was the main reason for using DEA. Hague et al. (2015) utilized TOPSIS with interval values for the supplier selection problem. They have considered the suppliers' strengths to provide the required components and illustrated the performance of the proposed framework in the aircraft industry. Zhang and Xu (2015) proposed a hesitant fuzzy QUALIFLEX method to solve the green supplier selection problem. They believed that their approach did not need a high volume of calculations to achieve good results. Govindan et al. (2018) used PROMETHEE to find the best suppliers based on corporate social responsibility (CSR) criteria. They collected the experts' opinions through the fuzzy Delphi method and then used the ANP method to determine the relationship and weights of the CSR criteria. Awasthi et al. (2018) proposed a new framework based on the AHP and VIKOR methods for sustainable supplier selection. They focused on risk concerns in the sustainability approach and used the fuzzy set theory to handle the uncertainties. Dong et al. (2017) employed the group AHP method to propose a dynamic model for selecting a suitable supplier for the companies. This model was established according to a feedback mechanism, which may lead to modifying the experts' opinions based on the situation. Govindan et al. (2017) adopted green supply chain management for the food industry. They presented a hybrid approach, including the revised Simos procedure and PROMETHEE method for supplier selection, using group decision-making. Interestingly, in the MADM literature, most scholars utilized the AHP and ANP methods for the supplier selection problem.

Chai and Ngai (2020) executed one of the most comprehensive and excellent surveys of supplier selection approaches to date. Their survey surpassed three major surveys on the topic (i.e., Fahimnia et al., 2015; Wetzstein et al., 2016; Zimmer et al., 2016) in-depth and breath. They reported that a decision approach is a [static] process from “inputs” to “outputs” where the inputs are the multiple sources of decisions (e.g., information) and the outputs are the “recommendations that conform with several predetermined and mostly fixed goals of decision.” If one compares the approach proposed in the current study with the approaches identified by them, then one finds that the proposed approach is a dynamic process from “inputs” to “outputs” where the inputs refer to the source of decisions that may come from the respondents or the earlier execution of the approach, and the outputs refer to recommendations that conforms with several predetermined and fixed goals of decision and warnings if the goals of decision are inconclusive to make a decision confidently. This early warning system allows for the re-execution of the decision approach unless a recommendation that conforms with the goals of the decision can be made confidently. Therefore, the decision approach proposed in the current study is novel and powerful compared to the most decision approaches in use today.

Further, Chai and Ngai (2020) reported that complexity in the decision environment, arising from uncertain information, supplier selection under different risk scenarios, non-quantitative or subjective judgments of the experts, is the “dominant issue” that the supplier selection literature is struggling to solve since last decade. If one looks at their survey, one finds hardly any single decision approach has succeeded in handling most of these issues simultaneously. With the aim to solve most of these issues simultaneously, the current study decided to extend the promising technology of the OPA, proposed by Ataei et al. (2020), in a way that the impact of uncertainty, incompleteness, and subjectivity in information and experts’ judgments on the risks arising from the selection of sub-optimal suppliers can be effectively and confidently minimized. Here, the theory of statistics and some novel procedures have greatly facilitated the task. Later, the contribution of the current study will be summarized along two dimensions (theory and application).

The Supplier Selection Process

The current study goes beyond simple modeling of supplier selection and quests to have a new look at the supplier selection paradigm. If one looks at Table 1, one finds that most of the studies restricted them to evaluating criteria and suppliers, and evaluation of evaluators (buyers) received relatively much less attention. One of the most influential frameworks of supplier selection problem was proposed by de Boer in the 1990s that has influenced many studies since then (see, e.g., Hadian et al., 2020; Lima Junior et al., 2014). According to the four-stage framework of de Boer et al. (2001), the supplier selection process includes; problem definition, criteria formulation, supplier qualification analysis, and choosing a supplier(s). In real-life environments where group decision-making is frequent, supplier qualification is not enough to guarantee the soundness of the supplier selection framework. A selection made by unreliable or unqualified evaluators can be questioned in a later stage, prompting conflicts within the stakeholders involved. Bali et al. (2013) and Boran et al. (2009) stressed the importance of considering the weight of decision-makers for supplier selection problems. Also, the exercise of supplier and buyer qualifications can prove futile if the criteria on which buyers evaluated the suppliers were unqualified. Weighing criteria is not enough if criteria are not reliable enough to produce high-quality solutions (Fu, 2020). A survey conducted by Sonmez (2006) identified that “Most papers attempted to identify and determine the relative importance of criteria for supplier selection in various industries.” It is observed that in literature, generally, the criteria identified by the decision-makers are considered absolute and unquestionable once chosen. There should be a measure of confidence or trust that demonstrates a selected set of criteria is reliable to a certain extent to generate reliable results. Recently, some scholars have shed light on the significance of this issue. Zha et al. (2019) proposed a model to enhance the consensus in group decision-making using bounded confidences. Mohammadi and Rezaei (2020) offered a trust level to compare the final rank obtained through their method with those from other MADM methods. Therefore, in this era of complex decision-making problems, one cannot rely on de Boer’s four-stage supplier selection framework without modifying it to meet today’s needs. Considering the gaps in the literature, the current study extends de Boer’s four-stage supplier selection framework to the seven-stage supplier selection framework (see Table 2):

Table 2

The supplier selection frameworks

de Boer’s four-stage supplier selection framework	The proposed seven-stage framework of supplier selection
Problem definition Formulation of criteria Supplier qualification Choice	Problem definition Formulation of criteria Supplier qualification Buyer qualification Criteria qualification Qualification testing (Confidence Level measurement) and corrective actions Choice

Problem definition (need supplier or not? how many?),

Criteria formulation (on which criteria the suppliers are to be evaluated?),

Supplier qualification (which suppliers to be in the pool of suppliers?),

Buyer qualification (which buyers to evaluate the suppliers?)

Criteria (attributes) qualification (which criteria qualified to be in the criteria pool?)

Qualification testing and corrective actions, if needed (can all criteria and buyers contribute to supplier selection confidently?)

Ranking and choosing the right supplier(s)

Once the suppliers are chosen, another stage of order allocation and post-qualification monitoring of suppliers usually begins. In this framework, which is both rationale and comprehensive, supplier qualification () is a function of criteria qualification (), which in turn is a function of buyer (evaluator) qualification (). This function can be represented as

where,

In other words, if a criterion fails to achieve a threshold of confidence, it implies that the existing set of criteria lacks the qualification to produce a reliable supplier ranking. In contrast, lack of criteria qualification results from a lack of buyer qualification. Therefore, corrective action is required, such as the criterion should be removed, data should be recollected, or new expert(s) should be added.

Ordinal Priority Approach

The OPA is a new technique of MADM, proposed by (Ataei et al., 2020). However, within a short period, it has seen several successful applications. Mahmoudi and Javed (2022) utilized the OPA and proposed a relative performance index to determine the performance of the sub-contractors in the construction industry. Sadeghi et al. (2022) employed the OPA to rank the barriers to implementing the blockchain in the construction industry. Wang et al. (2021) recognized the rationality of its ordinal information-driven layout. Unlike some other methods in MADM, the OPA does not require a pairwise comparison matrix, input's normalization, consideration for positive and negative aspects of the attributes, aggregation operator like geometric mean, among others. Moreover, this method can support incomplete data and group decision-making as well and can be used for both objective and subjective weighting. These advantages make the OPA an instrumental technique of handling supplier selection problems. Mahmoudi et al. (2021) demonstrated this fact by proposing the Grey Ordinal Priority Approach (OPA-G) for sustainable supplier selection in megaprojects under uncertainty. Later studies (e.g., Shajedul, 2021) confirmed the strengths of this approach. In another study, Mahmoudi et al. (2022) proposed the Robust Ordinal Priority Approach (OPA-R) for project portfolio selection problems. However, since the OPA is a new technology, there is no procedure to check its input data's reliability under different conditions. Therefore, the current study aims to develop a Confidence Level test to check the input data's reliability in the OPA for supplier selection problems. The form of the inputs in the OPA is ordinal data. Providing ordinal information is not difficult in a real-world situation, unlike the pairwise comparison and decision matrices. The most important assumption in the OPA model is the ordinality of the input data. In this section, the steps of the OPA are explained. First, variables and parameters of the OPA model are defined as follows:

With considering the aforementioned variables and parameters, the computing steps of the OPA are as follows (Ataei et al., 2020):

Step 1: In this stage, the essential attributes should be identified. Attributes (discrete criteria) play a vital role in the MADM problems and should be defined in light of the objective of the decision-making.

Step 2: Experts should be identified and ranked based on their knowledge and/or experience in the related field. Experts can be ranked based on one or more distinguishing characteristics.

Step 3: In this stage, the experts should rank the attributes based on their perceived importance.

Step 4: In this stage, alternatives should be ranked by the expert (s) in each attribute.

Step 5: Using the information from steps 1 to 4, Model (1) should be formed and solved.

After solving Model (1), the alternatives' weight can be calculated using Eq. (2).

To calculate the weights of the attributes, Eq. (3) can be utilized.

In case of need, the experts' weights can be determined employing Eq. (4).

Based on the aforementioned steps, the OPA requires a reasonable volume of computational cost and time. However, the input data should be checked carefully before solving the OPA model to confirm the reliability of the experts' opinions. In fact, we cannot accept every subjective view from the experts, and there is an essential need to ensure data reliability.

Designing the Confidence Level Measurement

As shown in Sect. 2.2, the measurement of confidence or trust is a relatively overlooked area in supplier selection literature, and only a couple of studies (Mohammadi & Rezaei, 2020; Zha et al., 2019) have yet recognized its real significance. The current section proposes a framework to evaluate the consistency of ordinal data in group decision-making. The framework effectively facilitates the application of the seven-stage Supplier Selection Process. In this regard, a literature review on ordinal rank assessment has been provided, and then, a system to measure the Confidence Level for the OPA has been proposed.

Ordinal Rank Assessment

In the statistics, there are primarily two streams of data analysis that include parametric and nonparametric methods. Before using parametric methods, some assumptions need to be addressed, such as the certain distribution of data and the law of large numbers. If the data passes the requirements, then a parametric method can be employed. On the other hand, nonparametric methods require fewer constraints than parametric methods and are also called distribution-free methods (Siegel, 1957). In light of the objectives of the current study, where data can be distribution-free, a nonparametric statistical approach is applied.

Spearman's rank correlation coefficient is one of the most well-known nonparametric methods for comparing the rank between two data series. Its value varies between − 1 to 1, where 1 implies the highest positive correlation, − 1 indicates the highest negative correction, and 0 shows no correlation (Spearman, 1904). Another alternative for comparing the ranks is the Somers’ D method. It varies between 1 and − 1 and determines the ordinal association. It is worth mentioning that Somers’ D method is an appropriate method for binary and independent variables (Somers, 1962). Goodman and Kruskal's gamma is another coefficient for determining the correlation between the ordinal data. It has been established based on counting the number of concordant and discordant pairs in the ordinal data. It should be noted that when the data dimension is 2 × 2 matrices, it becomes the Yule coefficient of association (Goodman & Kruskal, 1979). Therefore, the Yule coefficient of association is a particular case of Goodman and Kruskal's gamma. Also, Kendall's τ coefficient is another nonparametric method for comparing the rank between two data series. It ranges between 1 and -1, while 1 represents the perfect agreement, -1 represents perfect disagreement, and 0 represents independent variables (Kendall, 1938). Both Kendall's τ coefficient and Spearman's rank correlation coefficient have an excellent performance for determining the correlation between the two series ranks, yet Kendall's τ coefficient can estimate the parameter of the population better than Spearman's rank correlation coefficient and the standard error in Kendall's τ coefficient is known (Howell, 2012). Once there are more than two data series, Kendall's W method can be employed instead of Kendall's τ coefficient. Unlike other methods, Kendall's W only varies between 0 to 1, while 0 implies no agreement among the data, and 1 implies the complete agreement. Since Kendall's W is a nonparametric metric, it can be employed on different types of data without checking the nature of the probability distribution (Siegel, 1956). Based on the aforementioned features, the current study deploys Kendall's W for the reliability estimation of the OPA input data. In the next section, we aim to propose the Confidence Level measurement framework for the OPA in the MADM context.

Confidence Level Measurement for the OPA

Data reliability is one of the most crucial topics in the decision-making context. Since inappropriate input data can lead to erroneous results, data reliability should be addressed carefully during the whole process of decision making. In the MADM techniques, input data is usually experts' subjective opinions. Decision-making with one expert can involve a higher degree of risk (He et al., 2012). Employing a group of experts instead of one expert can decrease decision-making risk because of the aggregation of multiple perspectives (He et al., 2012). The OPA in multiple attributes decision-making has been established based on this viewpoint. It can handle single-expert decision-making as well, but it encourages group decision-making for problems involving uncertainty. In the current study, the Confidence Level has been proposed as an early warning tool to check the quality of input data before solving the group MADM problem.

As mentioned earlier, Kendall's W (Salkind, 2010) has been adopted to determine the Confidence Level in the current study. Guided by Kendall's W, the total rank for alternative in attribute can be determined using Eq. (5). It should be noted that index implies the expert's ID (identification number).

After that, the average of the total rank in attribute should be calculated utilizing Eq. (6).

The sum of squared deviations for attribute ought to be determined using Eq. (7).

Kendall's W can be determined using Eq. (8), while represents the number of alternatives, and implies the number of experts.

It is worth mentioning that Eq. (8) cannot be used to calculate Kendall's W where there are tie ranks. In this case, the number of ties should be calculated using Eq. (9).

is the number of tie ranks based on the viewpoint of expert . The corrected Kendall's W to consider the tie ranks has been shown in Eq. (10).

It should be mentioned that is the summation of tie ranks based on all experts' opinions in attribute .

When there are incomplete blocks in the input data, Kendall's W should be calculated for various situations. First, we should identify the possible interval range for each block. The general formula to fill out the incomplete block of expert in attribute for alternative has been defined in Eq. (11)

After that, should be calculated for all possible combinations of the incomplete blocks to find the minimum and maximum possible values where is state number. If we consider as the number of incomplete blocks, there are values for the (). Therefore, the interval range of can be calculated using Eq. (12) when we face incomplete input data.

After determining the , chi-squared distribution can be utilized to measure the Confidence Level. However, chi-squared distribution has some drawbacks when the number of alternatives is less than 20 (wrong level of Type I error) (Salkind, 2010). In MADM problems, this situation ( < 20) is common and it should be taken into account. To overcome this barrier, F-distribution can be utilized, which is a continuous probability distribution (Salkind, 2010). It is useful for situations involving more than one variable and small samples, and when the sample size is very large, its shape is very similar to the chi-squared distribution (Brereton, 2015). Thus, the F-distribution can be utilized for any values of and . In this regard, the random variable , also called x statistic, should be calculated with the degrees of freedom and using Eq. (13). It should be noted that the x statistic is a continuous positive real random variable. In Eq. (13), The value of represents the Kendall's W, represents the number of alternatives, and implies the number of experts. For estimating Kendall’s W, in the presence of ties, mean of ranks should be used to represent tied ranks.

Guided by the F-distribution, the probability density function of the -distribution (see Appendix A) for is illustrated in Eq. (14).where Euler integral was defined as follows:

Figure 1 illustrates a probability density function with the degrees of freedom and . It should be noted that the values of and can change its current form significantly.

Fig. 1

Fisher probability density function with the degrees freedom and and any

gives information about the probability of , yet we need to calculate the probability of all random variables equal and less than to demonstrate the Confidence Level. Since , the Local Confidence Level (LCL) in attribute can be calculated through Eq. (16) (see Appendix B).

Indeed, the cumulative [probability] distribution function (CDF) indicates confidence in a statistic (Madachy, 2007), and thus can be employed as a measure of Confidence Level. It should be noted that there are two types of Local Confidence Level (LCL) in the current study (see Appendix B). The calculation steps of both of them are the same, but one of them measures the Confidence Level of experts' opinions among the attributes (), while the other determines the Confidence Level of experts' opinions associated with the alternatives in each attribute (). Also, the value of local confidence level of the alternatives in all attributes () can be calculated using the and weight of attributes (see Eq. (17)). The LCL and its calculation have been shown in Fig. 2 for more clarification. The LCL reflects the trust in each attribute and among attributes which is critical for the decision-makers. The LCL values can be between 0 and 1, while 0 implies the lowest possible confidence, and 1 implies the highest possible confidence. The cumulative probability distribution has these characteristics as well. When there is no agreement among the experts' opinions, the value of Kendall's W approaches zero, and the LCL tends to be 0. Further, when there is very strong agreement among experts' opinions, the value of Kendall's W approaches 1, and the LCL tends to be one as well. Kendall's W directly relates to the LCL, which seems reasonable based on their range and characteristics.

Fig. 2

Local Confidence Level based on the cumulative distribution function

Also, the Confidence Level can be calculated for the MADM problem (Global Confidence Level). After solving the OPA and calculating the weight of attributes () using Eq. (3), the Global Confidence Level () for the MADM problem can be calculated using Eq. (17) (see Appendix B).

It is worth mentioning that we defined two types of confidence levels in this study; local confidence level and global confidence level. The local confidence level can be used for measuring the confidence level in each attribute and the global confidence level can be utilized to determine the confidence level of the whole MADM problem.

To sum up, the framework for calculating the Confidence Level has been depicted in Fig. 3. As shown in Fig. 3, the decision-maker should define (Subjective) or select the Global Confidence Level Threshold () in the first step (Objective, see Appendix C). Then, the data should be collected from the experts regarding the attributes and alternatives. When the experts lack sufficient knowledge of the issue, they should leave it empty instead of providing a false opinion. In such cases, the incomplete blocks should be filled using Eq. (11). Therefore, the value is an interval range in all subsequent steps. Next, it should be checked whether there is a tie or not among the ranks. When there are tie ranks in experts' opinions, Eq. (10) should be utilized to determine the value of . If there is not any tie rank, Eq. (8) can be employed. After that, the random variable and the degrees of freedom and should be calculated using Eq. (13). Then, the should be calculated through Eqs. (14) and (16). After that, the Local Confidence Level Threshold () for the alternatives can be determined using Eq. (18).

Fig. 3

The proposed framework to determine the Confidence Level

The value of should be determined in each attribute through Eqs. (14) and (16). Then, If these values are less than , the decision-maker should take corrective action. Corrective actions can be removing an expert, removing the attribute, recollecting the data from the expert (s), and/or adding more experts to the problem. The default corrective action for the proposed system is adding new expert(s). These corrective actions may increase the Confidence Level, thus facilitating the attainment of . If it does not increase the Confidence Level, the decision-maker should investigate the disagreements among the experts' opinions. Therefore, the proposed approach can provide an early warning tool to increase the reliability of the input data in the OPA. After meeting in each attribute, the problem should be solved using the OPA method. Based on the obtained weights of the attributes and Eq. (17), the can be determined for the MADM problem, which implies the Confidence Level of the decision-maker's decision. Moreover, if the threshold of the problem is subjective, the probability of a wrong decision () should be calculated.

In this section, the Confidence Level measurement for the OPA method is proposed. The next section aims to deploy the proposed approach on a numerical example and a case study to investigate its performance under different conditions.

Application and Discussion

In this section, the proposed methodology will be applied to two cases. The first one is a numerical case that will facilitate the readers to visualize each step of the process. The second one is a case from the real world.

Numerical Example

To evaluate the proposed approach's performance, we assume a hypothetical case involving a construction company that needs to select a supplier from potential suppliers of cement. There are three attributes: cost, quality, and timeliness for the supplier selection problem. The decision-maker categorized it as a sensitive problem, and thus, the objective should be higher than 90% (see Appendix C). In this example, we assume that all experts agreed that Attribute 1 > Attribute 2 > Attribute 3, and the decision-maker believes all experts are equally important. Hence, the value of is almost 100% based on Eqs. (14) and (16), and is 90% using Eq. (18). Moreover, there are five suppliers (S1, S2,…, S5) to compete. In this regard, the company has selected four experts whose opinions have been presented in Table 3. In the table, 1 represents the best rank, and 5 represents the worst rank.

Table 3

Experts' opinions for the numerical example

Suppliers	Attribute 1: Cost				Attribute 2: Quality				Attribute 3: Timeliness
	Expert 1	Expert 2	Expert 3	Expert 4	Expert 1	Expert 2	Expert 3	Expert 4	Expert 1	Expert 2	Expert 3	Expert 4
S1	2	3	2	4	1	1	1	2	5	4	3	3
S2	4	2	3	1	2	4	3	2	1	-	5	4
S3	1	1	1	5	3	2	2	1	3	2	2	2
S4	3	5	4	2	5	3	4	3	4	3	4	5
S5	5	4	5	3	4	5	3	4	2	1	1	1

As can be seen from Table 3, there are some tie ranks in Attribute 2 (see the bold numbers in the table) and one incomplete block under attribute 3. Therefore, we can see different kinds of situations in this numerical example. Based on the proposed steps in Fig. 3 and Eq. (11), the value for the incomplete block in Attribute 3 should be a number within the interval range [1,5]. Afterward, Kendall's W has been calculated using Eqs. (8) and (10). Kendall's W for the attributes has been reported as follows: Attribute 1 = 0.3125, Attribute 2 = 0.7276, Attribute 3 = [0.5114, 0.5750]. Next, the random variable has been determined by employing Eq. (13). The value of for the attributes has been reported as follows: Attribute 1 = 1.36, Attribute 2 = 8.01, Attribute 3 = [3.14, 4.06]. Moreover, the value of for all attributes have been illustrated in Fig. 4. As can be seen from Fig. 4, the value (s) of less than 2.8 or so cannot be accepted for because the decision-maker needs a Confidence Level that meets the threshold of 90%. Therefore, Attribute 1 does not meet the Local Confidence Level Threshold of the alternatives but Attributes 2 and 3 passed the threshold. It should be noted that we roughly estimated the minimum value of (2.80) through the drawing presented in Fig. 4. Hence, it is essential to calculate the Confidence Level to determine which attribute cannot pass the threshold.

Fig. 4

Demonstration of the random variable for the attributes

By utilizing Eqs. (14) and (16), each attribute's Confidence Level has been calculated in MS Excel and illustrated in Fig. 5. As can be seen from Fig. 5, Attribute 1 could not meet .

Fig. 5

The Local Confidence Level for the alternatives in the numerical example

Based on Fig. 4, the decision-maker should take corrective action because the data is not sufficiently reliable to arrive at a conclusion. In this situation, different options are available to increase the input data's reliability, such as adding a new expert, removing an attribute, removing an expert, and so on. However, in the current study, the default action is adding new experts. The corrected input data have been shown in Table 4, where Expert 5 has been added to the problem.

Table 4

Experts' opinions for the numerical example after corrective action

Suppliers	Attribute 1: Cost					Attribute 2: Quality					Attribute 3: Timeliness
	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 1		Expert 2	Expert 3	Expert 4	Expert 5
S1	2	3	2	4	2	1	1	1	2	1	5		4	3	3	3
S2	4	2	3	1	3	2	4	3	2	2	1		-	5	4	5
S3	1	1	1	5	1	3	2	2	1	1	3		2	2	2	2
S4	3	5	4	2	4	5	3	4**	3	4	4		3	4	5***	4
S5	5*	4	5	3	5	4	5	3	4	3	2		1	1	1	1

Note: The bold values imply the data that will be used for sensitivity analysis in Sect. 4.3

After executing the same steps, as executed earlier, the has been calculated in each attribute. The values are shown in Fig. 6, where the Confidence Level value before and after taking corrective action is also visible.

Fig. 6

Comparing the Confidence Level of the alternatives after and before the corrective action

As shown in Fig. 6, the has experienced an increasing trend in all attributes. Also, Attribute 1 met the value of . Now, the problem can be solved using the OPA because it assures that the total will not be less than 90%. After solving the problem using the OPA, the weights of the experts thus obtained are shown in Table 5.

Table 5

Experts' weights for the numerical example

Expert ID	Weight	Rank
Expert 1	0.1998183	3
Expert 2	0.1925522	5
Expert 3	0.1952770	4
Expert 4	0.2007266	2
Expert 5	0.2116258	1

The experts' weights have been compared in Fig. 7. Expert 5 achieved the highest weights among the experts because Expert 5 provided the opinion without incomplete data. Also, based on his/her opinion, he/she played the most important role in selecting the best supplier. On the other hand, Expert 2 achieved the lowest weight because Expert 2 did not opine supplier 2 in attribute 3. Hence, Expert 2 lost some weight compared to other experts.

Fig. 7

Comparing the experts' weights

The weights of the attributes have resulted in Table 6.

Table 6

Weight of the attributes for the numerical example

Attributes	Weight	Rank
Attribute 1	0.5449591	1
Attribute 2	0.2806540	2
Attribute 3	0.1743869	3

The weights and ranks of the attributes are compared in Fig. 8. Based on the input data, these results are realistic. However, the incomplete block in Attribute 3 has decreased its weight.

Fig. 8

Comparing the weights of the attributes

Finally, the weights of the suppliers are reported in Table 7. The weights and rank of the suppliers have been compared in Fig. 9. As can be seen from Fig. 9, the supplier S3 achieved the top position based on the experts' opinion in all attributes, and followed by the supplier S1.

Table 7

The weights and ranks of the suppliers

Suppliers	Weight	Rank
S1	0.2381472	2
S2	0.1933394	3
S3	0.3247351	1
S4	0.1070542	5
S5	0.1367242	4

Fig. 9

Comparing the weights of the alternatives

As mentioned earlier, the weights of experts, attributes, and suppliers have been obtained after solving the problem using the OPA. The Local Confidence Level in each attribute has been calculated before solving the problem using the OPA, shown in Fig. 6. Now, with the aid of the obtained weights in Table 6, the Confidence Level in Fig. 6, and Eq. (17), the for this numerical example problem can be as follows: gives valuable information to the decision-maker. It implies the decision-maker can select the best alternative, supplier S3 in our case, with a Confidence Level of around 95%. In the case of incompleteness, as was our case, is an interval grey number, an unknown crisp number lying within the known interval. Here, there is an interval range for the due to incomplete information in Attribute 3. In the case of the complete data, the value of is a crisp value. Meanwhile, one can see that, unlike the well-known Consistency Ratio (Asadabadi, 2017; Büyüközkan & Çifçi, 2011; Haeri & Rezaei, 2019; Kannan et al., 2020; Olson, 1988) and Self-Confidence Level (Zeng et al., 2019), the proposed Confidence Level neither focuses on the inconsistency of a judgment of an individual expert when compared with those of others nor the confidence one expert has on his/her judgment. Thus, it is more suitable for group decision-making requiring objective statistical evaluation.

Here a point needs discussion. It should be noted that even though we initially prioritized all experts equally, we did not get equal weights for each expert. Therefore, the priority of experts has been changed. It is a very interesting characteristic of the OPA that distinguishes it among all MADM techniques. In most MADM approaches, the priority of objects (experts or attributes) is assumed before ranking the alternatives, and it does not change during the process of achieving the alternatives' ranking (we call it static weighting), while in the OPA, one assumes the priorities of objects first and their impact on the alternatives' ranking later (we call it dynamic weighting). In short, in static weighting, the relative priorities (and thus weights) of the objects (experts or attributes) are pre-defined, and these priorities do not evolve even when one object turns out to be more impactful than the others (e.g., an expert with complete information is more impactful than the expert that revealed incomplete observations). On the other hand, in dynamic weighting, the objects' initial priorities are subjective and thus are not considered absolute truth. Here, the role these objects play in defining the alternatives' ranking helps the decision-maker distinguish them. There are two advantages of the dynamic weighting in the OPA, which are governed by the following two axioms:

Axiom I:

Priority of ‘all knower’ over ‘partial knower.’

This axiom enables the decision-maker to distinguish between ‘all knower’ and ‘partial knower,’ i.e., the object (expert or attribute) against which one received complete information and the one against which one received incomplete information cannot have the same weight. Therefore, why an expert who knows completely and one who does not know completely should have an equal role in terms of their influence on the final results?

Axiom II:

Priority of ‘top knower’ over ‘bottom knower.’

This axiom states that within ‘all knowers’ (complete objects), one that is more familiar with top priority alternative plays a significant role in the final decision (because we select top ranks).

As a result, the weights of experts and attributes of the OPA represent their role in the final decision (here, final selection), which is a fundamental concept behind the OPA theory. Therefore, the rank of experts is not definite; the experts’ opinions regarding the attributes and alternatives are important as well.

Comparative Analysis

For the readers not familiar with the approach's strengths, a comparative analysis of the OPA against other techniques is shown in this section. Table 8 presents the rankings of the suppliers obtained through the OPA, TOPSIS, GRA, and QUALIFLEX. Since the problem is group decision-making, the average of experts' opinions is considered as input data in TOPSIS, GRA, and QUALIFLEX, and the weight of the attributes is regarded as Table 6. The difference among the results of the MCDM problems can be clearly visualized in Fig. 10. The ranks in all MCDM methods are comparable, while the rankings of the OPA and GRA are precisely the same. On the other hand, the same results are obtained from the TOPSIS and QUALIFLEX methods. The only difference among all methods is the rank of the suppliers S4 and S5.

Table 8

The comparative analysis

Suppliers	QUALIFLEX	GRA	TOPSIS	OPA
S1	2	2	2	2
S2	3	3	3	3
S3	1	1	1	1
S4	4	5	4	5
S5	5	4	5	4

Fig. 10

The supplier ranking through the OPA, TOPSIS, GRA, and QUALIFLEX

Spearman’s rho is a valuable measure to compare ranks among MCDM methods. It confirmed that the obtained ranks of the methods are comparable, though not entirely (see Table 9). As can be seen from Table 9, Spearman’s rho value is significant in all comparisons. It shows that the results of the methods are significantly correlated in this example.

Table 9

Correlational analysis of the results among methods

			QUALIFLEX	GRA	TOPSIS	OPA
Spearman's rho	QUALIFLEX	Correlation Coefficient	1.000	0.900*	1.000**	0.900*
		Sig. (1-tailed)		0.019		0.019
		N	5	5	5	5
	GRA	Correlation Coefficient	0.900*	1.000	0.900*	1.000**
		Sig. (1-tailed)	0.019		0.019
		N	5	5	5	5
	TOPSIS	Correlation Coefficient	1.000**	0.900*	1.000	0.900*
		Sig. (1-tailed)		0.019		0.019
		N	5	5	5	5
	OPA	Correlation Coefficient	0.900*	1.000**	0.900*	1.000
		Sig. (1-tailed)	0.019		0.019
		N	5	5	5	5

* Correlation is significant at the 0.05 level (1-tailed)

** Correlation is significant at the 0.01 level (1-tailed)

Sensitive Analysis w.r.t. Incompleteness in Data

As mentioned earlier, the OPA is better equipped to handle incompleteness in data. In the current section, a sensitivity analysis will be presented to substantiate the claim. In this regard, many scenarios can be considered as to which observation is missing depending on the actual circumstances. However, for the sake of convenience, we will consider three scenarios, as shown in Table 10.

Table 10

Three scenarios for sensitivity analysis

Scenario	Description
Scenario 1	The number 5* is removed from Table 4 in this scenario
Scenario 2	The number 4** is removed from Table 4 in this scenario
Scenario 3	The number 5*** is removed from Table 4 in this scenario

The Local Confidence Level for the scenarios is presented in Fig. 11. Based on Fig. 11, when there is a missing value in the input values, the Confidence Level value is an interval instead of a crisp number. For example, the Local Confidence Level in Attribute 1 was 0.9185 (see Fig. 6) before applying Scenario 1, yet it became [0.80901, 0.91845] in Scenario 1. It implies that the Confidence Level value will be uncertain as well when there is uncertainty (missing value) in the input data. The range of confidence shows the level of uncertainty.

Fig. 11

The Local Confidence Level in each attribute for various scenarios

Figure 12 illustrates the impact of missing values on the weights of the experts. Based on Tables 4 and 10, removing the value related to Expert 1 in Attribute 1 for the supplier S5 implies Scenario 1. As can be seen from Fig. 12, the weight of Expert 1 in Scenario 1 is decreased. In other words, when the expert lacks sufficient information, its weight (its impact on the solution) should be reduced (see Axiom – I and Axiom – II). Likewise, the weight of Experts 3 and 4 is decreased in Scenarios 2 and 3, respectively.

Fig. 12

The sensitivity of the experts' weights against missing values

For the same reason, the weights of the attributes and alternatives with missing values will be decreased compared with the basic scenario as well. Figures 13 and 14 can be checked for more investigations, which show the impact of missing values.

Fig. 13

The sensitivity of the attributes' weights against missing values

Fig. 14

The sensitivity of the suppliers' weights against missing values

A Practical Case Study

A case study has been selected from a Chinese company with the head office in Jiangsu and a factory in Jiangxi, China, to demonstrate the proposed methodology's feasibility. This company, established in 2010, develops and produces chemical products such as modified silicone materials, mold release materials, and metal processing cutting fluids. These products can be used in different industries like rubber, tires, polyurethane, composite materials, and high-pressure casting. Staff working in its procurement department was approached, and with their input, a list of five suppliers of raw material was prepared. Later, the suppliers were arranged in alphabetical order and named S1, S2, S3, S4, and S5. Four important attributes were identified to evaluate these suppliers: Price, Safe and Timely Delivery, Operational Capacity, and E-commerce Capability. was subjectively set to be 85% by the decision-maker, the end-user of the information. Moreover, four experts have been selected to evaluate the suppliers against the given attributes. The experts' opinions regarding the attributes and alternatives have been summarized in Tables 11 and 12.

Table 11

The experts' opinions for the attributes

Attributes	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5
Attribute 1	2	2	1	2	1
Attribute 2	1	1	1	1	1
Attribute 3	2	2	1	3	2
Attribute 4	3	3	2	4	2

Table 12

Experts' opinions for the case study

Suppliers	Attribute 1: Price				Attribute 2: Safe and Timely Delivery				Attribute 3: Operational Capacity				Attribute 4: E-commerce Capability
	Expert 1	Expert 2	Expert 3	Expert 4	Expert 1	Expert 2	Expert 3	Expert 4	Expert 1	Expert 2	Expert 3	Expert 4	Expert 1	Expert 2	Expert 3	Expert 4
S1	2	1	1	1	5	4	3	4	3	2	2	3	3	3	3	4
S2	3	2	2	5	1	2	1	1	1	3	2	2	1	1	1	3
S3	1	1	2	2	3	3	3	4	2	4	1	1	2	2	2	2
S4	4	3	3	3	2	1	1	2	2	1	1	2	2	3	1	1
S5	5	4	3	4	2	1	2	3	3	2	3	4	3	3	3	4

Based on Table 11, the is 99.98% which is higher than 85%. Therefore, the value of is 85.02% (estimated through Eq. (18)). Guided by the proposed methodology and Table 12, has been calculated and illustrated in Fig. 15.

Fig. 15

The Local Confidence Level for the attributes in the case study

As shown in Fig. 15, the Local Confidence Level in Attributes 3 is not enough to meet the Confidence Level Threshold. Since the data’s reliability is not up to the mark, there is a need to take corrective action. As mentioned earlier, by default, the best corrective action is adding a new expert (s). One may choose to add one expert after another to see whether the Local Confidence Level improves or not, else benefit from the technology and computing power we have today at our disposal. Each strategy has its cost and time. In this regard, we choose to employ the Monte Carlo simulation with the addition of a new expert whose opinion was randomly generated thousands of times in the presence of four real experts from Table 12. The Local Confidence Level within each attribute was calculated 20,000 times, and the results are shown in Fig. 16.

Fig. 16

The results of the Monte Carlo simulation for the case study

As shown in Fig. 16, the Local Confidence Level is improved significantly (based on the maximum number of counts) within all attributes. Thus, the simulation suggests that adding one expert is likely to boost our confidence. In short, to achieve a Local Confidence Level of more than 85.02%, new experts can be consulted. Therefore, a new expert (Expert 5) was approached in the same company, and the revised input data is shown in Table 13.

Table 13

Experts' opinions for the case study after corrective action

Suppliers	Attribute 1					Attribute 2					Attribute 3					Attribute 4
	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5
S1	2	1	1	1	1	4	4	3	4	4	3	2	2	3	2	3	3	3	4	3
S2	3	2	2	5	2	1	2	1	1	1	1	3	2	2	1	1	1	1	3	1
S3	1	1	2	2	3	3	3	3	4	3	2	4	1	1	1	2	2	2	2	2
S4	4	3	3	3	5	2	1	1	2	2	2	1	1	2	1	2	3	1	1	3
S5	5	4	3	4	4	2	1	2	3	4	3	2	3	4	3	3	3	3	4	3

Based on the information available in Table 13, the Local Confidence Level within each attribute is recalculated, and the results are shown in Fig. 17. Also, the Local Confidence Level for before and after taking corrective actions has been compared together.

Fig. 17

Comparing the Confidence Level of the attributes after and before the corrective action

As can be seen from Fig. 17, all attributes have a value of Local Confidence Level of more than 85.02%. Therefore, the input data are reliable enough to use the OPA for the evaluation of the suppliers. It should be noted that the experts are considered equally important in the input data. After solving the problem using the OPA, the weights and the ranks of the experts, attributes, and suppliers thus obtained are shown in Table 14.

Table 14

The weights and ranks of the experts, attributes, suppliers

Experts	Weight	Rank
Expert 1	0.1754641	4
Expert 2	0.2047081	3
Expert 3	0.2625739	1
Expert 4	0.1419683	5
Expert 5	0.2152857	2
Attributes
Attribute 1	0.2569740	2
Attribute 2	0.3708390	1
Attribute 3	0.2318780	3
Attribute 4	0.1403091	4
Suppliers
Supplier 1	0.3245048	4
Supplier 2	0.5945453	1
Supplier 3	0.3979260	3
Supplier 4	0.4501918	2
Supplier 5	0.2328322	5

As can be seen from Table 14, Supplier 2 (S2) is the best supplier for the company based on the defined attributes. Also, Attribute 2 is the most important attribute—Safe and Timely Delivery of the material. Moreover, Expert 4 has the highest weight among experts because his/her suggestion plays the most important role in selecting the best supplier. Given that the Local Confidence Levels for attributes and alternatives are 99.98% and 98.93%, respectively, the for this problem is 98.91% (estimated through Eq. (17)), which is far above the (85%) and thus is very acceptable to the decision-maker. Moreover, the probability of a wrong decision can be 1.09% for this problem.

The overall contribution of the current study is summarized along two dimensions (theory and application) below.

In terms of supplier selection problem, the current work differs from the previous studies in four key ways: (1) The proposed model is appropriate when the final decision-maker (e.g., the head of the procurement department) uses a three-stage process for selecting an optimum supplier from a set of potential suppliers. Here, in the first stage, suppliers are ranked using the OPA. In the second stage, the evaluators (the members of buying group; the buyers) are themselves evaluated, and the confidence of whether the selected supplier is optimum or not is gauged using Kendall’s coefficient-based novel methodology. If the Confidence Level is below the acceptable threshold, the third stage is executed, where corrective actions are taken. Therefore, the study has expanded the scope of the traditional supplier selection problem by linking the selection of optimum suppliers to the selection of optimum evaluators; (2) The study introduces a system of early warning tool (at the second stage), and a corrective action plan (at the third stage), if needed, in the supplier selection problem. This characteristic effectively minimizes the cost of a poor decision and makes the organization’s supply base more resilient and reliable; (3) The study allows buyers not to give an opinion when they lack sufficient knowledge about a supplier’s attribute/characteristic. Therefore, buyers are given a space to stay flexible and honest in terms of their judgments about suppliers; (4) While filling the gap in the literature, de Boer's four-stage supplier selection framework has been extended to a seven-stage framework. The three points mentioned above aid this extension; (5) Also, the study paves the way for the journey towards statistically significant multiple-attribute decision-making.

In terms of supplier evaluation methodology, the current work differs from the previous studies in four key ways: (1) Guided by the theories of nonparametric statistics and the OPA, the study pioneers a probabilistic approach of supplier evaluation and selection under the informational uncertainty and incompleteness using a novel Confidence Level measure. Thus, it makes the results more realistic and probable; (2) The study suggests that the probability that a supplier shortlisted for selection is actually the optimum choice or not can be explained through a probability distribution, called W-distribution, therefore, confidently preventing the decision-makers from selecting the sub-optimum suppliers; (3) The study fixed a major limitation in the original OPA and enabled it to handle uncertainty (ties) in information as well; (4) The current study also identifies three properties of the OPA relevant for the group decision-making problems: (a) The scale depends on the size of the problem; (b) Ties in priorities/ranks is allowed; (c) The relative difference between the weights manifests the relative difference between the impacts of an object (attribute, expert, alternative) on the final results, the ranking of alternatives. Therefore, the OPA weights imply the degree of impact an object enjoys over the final results.

Conclusion

Supplier evaluation and selection is a well-known problem in the industry. To date, various MADM methods have been employed to solve this problem. However, few of them, if any, recognized the need to provide an index to evaluate suppliers with quantifiable confidence. Selecting a supplier is one thing and selecting the right supplier with confidence is another thing. The OPA is a new technology in the MADM theory with many benefits compared with other MADM methods commonly used for supplier selection problems. Since supplier selection is an important decision for the organizations, it should contain an adequate level of reliability. In other words, the reliability of the input data plays a vital role in the quality of the decision. In this regard, a novel methodology has been proposed to calculate the Confidence Level in the suppliers using the OPA model. Also, the solutions for enhancing the Confidence Level, and thus achieving the Confidence Level Threshold are suggested. The performance of the proposed methodology has been gauged through a numerical example and a real-life case study from a chemical company. Based on the results, the proposed method can determine the Confidence Level of the input data before solving the problem using the OPA as an early warning tool. After solving the problem, it can provide a Global Confidence Level for the supplier selection process, which is very useful for the managers. The Confidence Level can inform them how much they can trust the data. It can detect low-quality data and subjective information from the experts' opinions.

In most cases, after adding new experts, the reliability and cognition about the problem may be increased. In some situations, the decision-maker may require to remove an attribute when adding the experts is impossible, and the reliability of data in the attribute is low. Different corrective actions have different costs and processing time. Therefore, the choice of corrective action depends on the decision-makers, who take corrective actions in light of their priorities. Also, the study extended de Boer’s four-stage supplier selection framework to a seven-stage framework that is more comprehensive and realistic in light of the needs of today’s businesses.

Since the proposed Confidence Level system is designed for the OPA method, it only can handle ordinal data. In future studies, the proposed Confidence Level can be extended to other MADM methods like the QUALIFLEX method because its input data is ordinal. Also, it can be extended for different types of data using appropriate statistical methods. An in-depth analysis of the -distribution and its properties will also be studied in the future.

Sets:
I	Set of experts ∀i∈I
J	Set of attributes ∀j∈J
K	Set of alternatives ∀k∈K
Indexes:
i	Index of the experts (1,⋯,p)
j	Index of preference of the attributes (1,⋯,n)
k	Index of the alternatives (1,⋯,m)
Variables:
Z	Objective function
w_ijk^r_k	Weight (importance) of k th alternative based on j th attribute by i th expert at rkth rank
Parameters:
r_i	The rank of expert i
r_j	The rank of attribute j
r_k	The rank of alternative k