Study of consistency of expert evaluations of wine sensory characteristics by positional analysis.

Individual characteristics inherent in the expert, as well as their physical and psycho-emotional state subject to the influence of random, uncontrollable factors, contribute to subjectivity in the sensory evaluation of wines. With great variability of opinions, the final results of sensory evaluation may become doubtful. The presence of a random component in the sensory evaluation justifies the use of statistical methods for analyzing the consistency of expert evaluations. Along with Spearman's correlation coefficients and Kendall matching, Cronbach alpha criterion was used to assess the consistency of expert opinions. The advantages of positional analysis have been discussed - Cronbach's alpha criterion is calculated not by the rank of expert points, but by the initial point scale considering its variability; it allows to evaluate the contribution of each expert to the consistency of expert evaluations, as well as the reliability of the total scale of points set for each wine sample. Based on the data analysis from sensory evaluation of the quality of dry red and white wines of Russian production, the results of the consistency study of expert evaluations as well as the reliability of the total score scale have been presented. What is more, analysis of the "loyalty" of experts in evaluating the quality of wines has been performed.

Introduction

Despite the high level of development of tools for laboratory analysis of wine quality, the main way to comparatively evaluate their organoleptic properties is sensory evaluation conducted by specially trained professional experts. Statistical analysis methods have a wide range of applications in wine quality research. They are most often used in the sensory assessment of the quality of wines (Baker and Ross, 2014; Cortez et al., 2009; Etaio et al., 2008; Piclin et al., 2008; Rinaldi and Moio, 2018), differentiation of wines by grape growing regions (Khalafyan et al., 2019; Urvieta et al., 2018), analysis of the influence of the chemical composition of wines and grapes on their quality (Bindon et al., 2014; Kapusta et al., 2017; Pejina et al., 2016; Perestrelo et al., 2018; Vujovic et al., 2016), etc. When processing the results of sensory evaluation, various statistical methods are used – analysis of variance (ANOVA) (Etaio et al., 2008; Rinaldi and Moio, 2018; Taladrid et al., 2019), principal component analysis (PCA) (Piclin et al., 2008), discriminant analysis (Khalafyan et al., 2016), Cartesian coordinates mapping (Hopfer and Heymann, 2013), regression analysis (Baker and Ross, 2014; Cortez et al., 2009), statistical text analysis with Alceste (Rodrigues and Parr, 2019), etc.

Considering the publications describing sensory evaluation of wines (Jackson, 2015, 2017; Parker, 2003; Rodrigues and Parr, 2019; Spence, 2019; Spence and Wang, 2019; Taladrid et al., 2019; Wang and Spence, 2018), expert methods have certain disadvantages. The results of sensory evaluation of wines are influenced by the composition of experts, their qualification level and quantity as well as imbalance of wines. Individual characteristics inherent in each expert, along with their physical and psycho-emotional state subject to the influence of a large number of random and uncontrollable factors contribute to subjectivity in expert evaluations.

Processing expert evaluations includes checking the consistency (unanimity) of expert opinions and averaging expert opinions within an agreed group. Apparently, if expert opinions regarding the quality of wines are contradictory, the final result of sensory evaluation – wine ranking by the degree of preference by the experts, will be at least doubtful.

For an integrated assessment of the consistency of expert evaluations (Kuznetsov and Strijov, 2014), analysis of expert opinions on reaching consensus (Kozierkiewicz-Hetmanska, 2017) and conducting expert opinion procedures (Perez-Rodrıguez and Rojo-Alboreca, 2016), a matrix approach as well as theorems based on the relationship between expert opinions and their number have been proposed. Multicriteria decision-making tasks based on hierarchy analysis have been discussed to improve the consistency of experts (Khatwani and Kar, 2016) and establish the accuracy of expert evaluation of the object state by pairwise comparison with a quantitative assessment of preferences (Radaev, 2007). The consistency of expert evaluations is established by mapping on a Cartesian plane (Abramova and Telitsyna, 2013; Wilson et al., 2018; Risvik et al., 1994; Hopfer and Heymann, 2013).

Expert evaluations are empirical data in which a random component is present, therefore, to analyze the consistency of expert opinions, the statistical apparatus is widely used: correlation analysis, two-dimensional normal distribution, multiple factor analysis, Spearman's rank correlation coefficient, Kendall's concordance, etc (Cain and McLeay, 2016; Etaio et al., 2008; Khalaphyan et al., 2017; Le Dien and Pages, 2003; Radaev, 2007; Vaamonde et al., 2000; Wilson, 2017). The above statistical methods are not without drawbacks and do not fully allow to evaluate the contribution of each expert to their unanimity.

In this paper, some of the problems associated with establishing the consistency of expert evaluations of the wine quality have been considered. As an alternative statistical method for assessing the unanimity of experts, Cronbach's criterion has been proposed which is used in positional analysis to assess the reliability of the total scale of respondents, its advantages have been discussed as well. In contrast to Spearman's rank correlation coefficients and Kendall's correlation widely used for these purposes, Cronbach's alpha is calculated not according to the rank of expert points, but according to the initial point scale using the variability of expert evaluations. The criterion allows to determine the contribution of each expert to the overall consistency, to evaluate the reliability of the total scale of points set by each expert for wine samples.

Material and methods

Research objects

60 samples of natural dry red and white grape wines produced in 2010–2015 in the territory of main wineries of Krasnodar region (Russia) were analyzed: “Myskhako”, “Fanagoria Number Reserve”, “Kuban-Vino”, “Southern wine company (SWK)”, “Villa Victoria”, “Chateau Tamagne”, “Chateau le Grand Vostock”. All the wine samples were produced according to traditional technologies from European (Cabernet, Merlot, Aligote, Riesling, Saperavi, etc.) and hybrid grape varieties (Bianca, Viorica, Moldova, Pervenets Magaracha, etc.) and were kindly provided for research by their manufacturers. The wines were poured into dark green glass bottles with screw caps and stored until use at 10 °C. All wine samples were dry, alcohol content varied from 9 to 13 % (v/v) and pH values ranged from 3.61 to 3.79. Dissolved oxygen in wines was measured by the immersion of the probe before bottling in barrels, which was less than 1 mg/dm3.

Wines from European grape varieties obtained by traditional technologies without the use of sulfur dioxide were not considered, since this category significantly differs in taste from wines for which sulfiting was used.

Research methods

Sensory analysis

All experimental studies related to sensory analysis were carried out at the Federal Research Center of Horticulture, Viticulture, Wine-making (FSC HVW, Krasnodar, Russia). 11 specialists participated in the sensory evaluation procedure aged between 32 and 66 years (the average age was 50 years; 4 female and 7 male). The informed consent was obtained from the participants of the sensory test. All participants are considered experts in the field of wine, work in the wine industry and have professional experience in the field of sensory analysis. Questionnaire table for experts is presented in Table 1.

Table 1

Questionnaire table for experts.

Parameter	Value (0–100)
Typicality
Transparency
Color
Aroma
Taste

The wine sample (50 cm3) was poured into each glass and covered with a Petri dish with diameter of 5.7 cm 30 min before sensory evaluation. The tests were carried out in a well-lit tasting room with controlled temperature conditions. All samples were fed at 16–22°С at tables with white napkins. Experts were prohibited to communicate during the sensory assessment procedure. The wines were served in transparent tulip-shaped glasses with a volume of 220 dm3. After evaluating each sample, participants were asked to wait at least 30 s, clear their taste with water and crackers. The intervals between tasting of each sample were 2 min. During each interval, experts rinsed their mouths with water. Experts evaluated each sample in triplicate during the working week.

The sensory assessment results of wine quality were expressed on a scale from 50 to 100 according to the well-known rating system (Parker, 2003). For a consolidated assessment of the organoleptic characteristics of wines, the average values of sensory evaluations were used according to the results of tasting by a group of experts.

Table 2 shows the results of sensory evaluation for the first 10 wine samples. A table with the results of sensory evaluation of all 60 wine samples is given in the electronic supplement to the article. The top row shows the experts, their gender, age, while the first column shows the sample number.

Table 2

Results of sensory evaluation of tested wine samples.

Sample Number	Expert 1, f, 66	Expert 2, m,32	Expert 3, m, 59	Expert 4, f, 57	Expert 5, m, 42	Expert 6, f, 38	Expert 7, m, 58	Expert 8, m, 55	Expert 9, f, 59	Expert 10. m, 43	Expert 11, m, 41	Sum
1	77	80	80	80	81	82	77	85	85	83	81	891
2	83	83	79	79	79	63	82	79	83	78	78	866
3	89	79	81	83	82	76	83	79	86	77	85	900
4	90	85	82	78	82	78	85	76	85	84	78	903
5	90	87	79	85	85	82	84	83	85	85	85	930
6	90	84	80	84	83	76	77	86	83	81	84	908
7	85	87	82	85	84	78	83	86	84	86	87	927
8	90	86	80	83	84	84	83	87	82	82	79	920
9	95	92	85	84	86	80	88	88	84	84	80	946
10	88	86	82	79	81	64	85	80	83	79	86	893

Data analysis

All calculations were implemented using the STATISTICA software (v. 10) (Hill and Lewicki, 2007). The pairwise consistency of experts was determined using Spearman's rank correlation coefficient, the “individual” consistency was established by the multiple correlation coefficient, group consistency – by means of Kendall's concordance coefficient and Cronbach's alpha criterion (Reliability and Item Analysis). However, the listed statistical criteria for the consistency of expert evaluations – Spearman's rank correlation coefficients, Kendall's and Kronbach's alpha correlations do not have generally accepted ranges of variation for their interpretation in the nominal scale, therefore, we focused on the degree of their proximity to 0 and 1. The closer the value of the criteria is to 0, the lower the consistency is; following this trend, if the value is closer to 1 – the consistency is higher. Scatter plots for experts and wine samples were built using the Multidimensional Scaling module.

Results and discussion

A study of the consistency of expert evaluations through Spearman's rank correlation coefficients and Kendall's concordance

Traditionally, for a statistical assessment of the overall consistency of 2 experts, Spearman's rank correlation coefficient is used. When the tasting results coincide for 2 arbitrary experts, they turn out to be “completely” consistent and the correlation coefficient of the points they set is equal to 1. The complete inconsistency of the two experts means that their evaluations are random independent variables and, in accordance with the provisions of probability theory, the correlation coefficient will be 0. Therefore, the correlation coefficient can also serve as an assessment of the “pairwise” consistency of experts (Table 3). In the first 11 columns of Table 3, Spearman's rank correlation coefficients are shown (statistically significant coefficients are shown in bold). The highest r value of 0.530 was observed between experts 2 and 5; the consistency of evaluations between them was higher than between other pairs. Less consistency of evaluations was between experts 4 and 8 (r = 0.522); 3 and 4 (r = 0.489), etc.

Table 3

Correlation coefficients between the experts.

Expert	Expert 2	Expert 3	Expert 4	Expert 5	Expert 6	Expert 7	Expert 8	Expert 9	Expert 10	Expert 11	R
Expert 1	0.289	0.235	0.354	0.302	-0.02	0.198	0.192	0.240	0.089	0.254	0.650
Expert 2		0.467	0.314	0.530	0.304	0.258	0.411	0,097	0.400	0.393	0.689
Expert 3			0.489	0.406	0.411	0.399	0.387	0.294	0.470	0.276	0.842
Expert 4				0.487	0.310	0.213	0.522	0.121	0.164	0.295	0.783
Expert 5					0.486	0.261	0.461	0.134	0.198	0.157	0.784
Expert 6						0.162	0.465	-0.080	0.324	0.066	0.680
Expert 7							0.147	-0.117	0.321	0.001	0.574
Expert 8								0.071	0.474	0.177	0.803
Expert 9									0.129	0.224	0.741
Expert 10										0.202	0.689
Expert 11											0.713

For a more objective evaluation of the “individual” consistency of experts, we used the multiple correlation coefficient R which is a generalization of pairwise correlation coefficient in cases where the number of random variables is more than 2. R values are given in the last column of Table 2. Obviously, when multiple correlation coefficient increased, the individual consistency of expert evaluations also increased. The highest consistency with the evaluations of other experts was observed for expert 3 (R = 0.842), while the lowest was for expert 7 (R = 0.574). The predominantly close to 1 R values indicate a supposedly high overall consistency of expert evaluations which is of undoubted practical interest for researchers and practitioners. In the case where there are more than 2 experts, Kendall's concordance coefficient is used for the statistical consistency evaluation, the calculated value of which was 0.335 showing the low consistency of expert evaluations and discrepancy with the results obtained by correlation analysis.

Positional analysis of the consistency of expert evaluations

As an alternative to Kendall's concordance coefficient, positional analysis should be considered which was developed to build reliable profiles (scales) in psychology. Cronbach's alpha () statistics is used as the criterion for the reliability of the questionnaire (Eq. (1)):where is sample variance of the i-th statement (position); is sample variance of the total scale (sum of respondent ratings for all questionnaire positions); n – number of positions. The closer is to 1, the higher the reliability of the total scale is observed; similarly, the closer is to 0, the lower the reliability is. In relation to the results of sensory evaluation of wine quality given in Table 1, respondents are wine samples, positions are experts, the total scale is the sum of points set by experts for each sample (Sum column). In fact, the reliability of the total scale of expert evaluations also means the general consistency of expert opinions. If the expert evaluations coincide (the highest consistency), then the scores in the columns of the table are the same, consequently, by the variance property , where s = (i = 1,…, n). Substituting the obtained values in (1), the following Eq. (2) is obtained:

If there is no correlation between expert evaluations (the least consistency), then they are independent random variables. Therefore, using the dispersion property, i.e., the variance of the sum of independent random variables is equal to the sum of the variances, the equation is obtained (Eq. (3)), then

In some cases, in the presence of negative covariance relationships between expert evaluations, may be less than , then will be a negative value, which also means expert opinions inconsistency.

The final results of experimental data positional analysis are presented in Table 4. Cronbach's alpha criterion of 0.843 indicated in the information part of the table characterizes the consistency of expert evaluations as high and does not contradict the results of correlation analysis. From the above data it follows that Kendall's concordance coefficient does not fully take into account the consistency of expert evaluations in comparison to positional analysis. We believe that the reason is in the undoubted advantage of positional analysis. The calculation of Kendall's concordance coefficient is associated with the transition from a “strong” presumably numerical point scale to a “weak” rank scale accompanied by a significant loss of information regarding the variability of expert evaluations. So, if an expert set 85, 77 and 95 points for three wine samples, then when moving to the rank scale they would correspond to ranks 2, 3 and 1, respectively, leveling the degree of difference between samples in 8, 18 and 10 points, i.e. their variability sets only the order – more or less. When calculating Cronbach's alpha criterion, all calculations are carried out in the original, more differentiated point scale, considering the variability of expert evaluations. Therefore, the conclusions drawn on the basis of Cronbach's alpha criterion regarding the consistency of expert evaluations are more informative and mathematically justified.

Table 4

The results of positional analysis of unanimity among experts.

Variable	Summary for scale: Mean = 905,900; Standard Deviation = 34,139; Valid N:60. Cronbach alpha: 0,843, standardized alpha: 0,872.
	Mean if deleted	Variable if deleted	Standard Deviation if deleted	Item-TotalCorrelation	Alpha if deleted	Consistency rating
Expert 1	820.983	952.216	30.858	0.513	0.831	8
Expert 2	821.017	978.983	31.289	0.587	0.826	5
Expert 3	824.733	982.696	31.348	0.806	0.819	2
Expert 4	824.300	976.310	31.246	0.686	0.822	3
Expert 5	822.750	1027.188	32.050	0.680	0.829	7
Expert 6	824.650	934.928	30.577	0.391	0.849	11
Expert 7	822.867	1074.582	32.781	0.255	0.847	10
Expert 8	824.767	700.812	26.473	0.759	0.816	1
Expert 9	824.150	1008.494	31.757	0.458	0.835	9
Expert 10	824.950	961.347	31.006	0.601	0.824	4
Expert 11	823.833	984.706	31.380	0.576	0.827	6

Experts 6 and 7 reduce the overall consistency.

An important advantage of positional analysis is also that the indicators given in the columns of Table 4 allow to observe changes in the main statistics of the total scale with the sequential removal of the sensory evaluation of each expert. Obviously, if value in the column Alpha if deleted exceeds 0,843 in the information part of the table, then the expert reduces the overall consistency of expert evaluations; otherwise, the expert makes a positive contribution to the overall consistency by increasing it. As follows from the table, only experts 6 and 7 reduce the overall consistency, the remaining 9 experts increase it. The last column of Table 4 shows the rating of experts in descending order of their contribution to the overall consistency of expert evaluations which does not contradict the results of correlation analysis. Naturally, the removal of experts who reduce the consistency should lead to an increase in the overall consistency of expert evaluations. So, if we remove experts 6 and 7 from the consideration, will increase and assume a value equal to 0.857.

Graphical analysis of the consistency of expert evaluations

Analysis of a matrix file of pairwise correlation coefficients by the Multidimensional Scaling method, created by positional analysis using the original data table, allows to graphically represent experts as points on a plane according to their contribution to the overall consistency. With the decrease in the geometric distance between experts, the consistency of expert evaluations increases. The smaller the geometric distance between experts is observed, the higher the consistency of their expert evaluations is. It can be seen from the scattering diagram (Figure 1) that experts 3 and 8 who have the greatest contribution to the overall consistency of expert evaluations are located at the closest distance to each other. At an insignificant distance from them are experts 4, 10, 2, 11, 5, etc., following them in the ranking. Experts who make a positive contribution to the consistency of experts are located in relative proximity, forming a group of unanimous experts indicated by blue rhombi in the diagram. At the same time, experts 6 and 7 are located at the periphery of the chart at some distance from the other experts and from each other – they are indicated by red triangles.

Figure 1

Expert scattering diagram constructed by multidimensional scaling.

Analysis of the loyalty of experts using Cronbach's alpha criterion

A comparative analysis of the average points given by experts for all 60 wine samples shows that some of the experts are less demanding (loyal) and assess the wines with a higher average score than the more demanding experts with a lower average score. The level of expert requirements was expressed as a loyalty scale and defined as the aggregate of average scores given by them in the sensory evaluation of wines. For the credibility of the interpretation of the obtained loyalty scale, its reliability was assessed by Cronbach's alpha criterion. For this purpose, the original data file presented in columns 2–12 of Table 1 was analyzed by means of the STATISTICA software. Cronbach's alpha criterion was found to be 0.869, thus showing high reliability of the loyalty scale. Loyalty “rating”, i.e., number of experts after their ranking in descending order according to the loyalty scale, is presented in the column Rating for all wines in Table 5. The most loyal was expert 1 with 84.917 points. The least loyal was expert 10 with 80.95 points. Student's t-test for independent variables showed the statistical significance of the difference between the indicated mean values (p = 0.00002 < 0.05), which shows the practical feasibility of analyzing the loyalty of experts. Males and females turned out to be of the same loyalty on average, since their average point amounts were approximately the same: – 82.379 и 82.340, respectively.

Table 5

Expert loyalty rating for sensory evaluation of wines.

Expert number	Dry wines
	Gender expert	Expert loyalty scale	Rating for all wines	Expert loyalty scale for white wines	Rating for white wines	Expert loyalty scale for red wines	Rating for red wines
1	female	84.917	1	87.200	1	83.775	3
2	male	84.883	2	84.850	2	84.900	1
3	male	81.167	9	80.900	9	81.300	9
4	female	81.600	7	81.350	7	81.725	7
5	male	83.150	3	82.200	6	83.625	4
6	female	81.250	8	75.600	11	84.075	2
7	male	83.033	4	82.700	4	83.200	5
8	male	81.133	10	80.750	10	81.325	8
9	female	81.750	6	84.200	3	80.525	11
10	male	80.950	11	81.200	8	80.825	10
11	male	82.067	5	82.700	5	81.750	6

The use of positional analysis made it possible to assess the contribution of all 60 wine samples to the reliability of the loyalty scale. Cronbach's alpha values after a successive removal of wine samples from positional analysis allowed to identify samples reducing/increasing the reliability of the scale. Cronbach's alpha values after a successive removal of wine samples from positional analysis made it possible to isolate samples that reduce/increase the reliability of the loyalty scale. The reliability of the loyalty scale is reduced by samples 1, 12, 22, 25, 29, 32, 33, 34, 38, 42, 46, 47, 48, 49, 52, 53, 54, 55, 56, 60, the rest increase it. From the scattering diagram (Figure 2), it can be seen that wine samples that reduce the reliability of the scale (indicated by red triangles) are located on the periphery of the graph — in the right and lower parts of the scattering diagram. Samples that increase the reliability of the scale (indicated by blue rhombs) are localized mainly in the left and central parts of the scattering diagram.

Figure 2

Scattering diagram of wine samples constructed by multidimensional scaling.

Analysis of expert loyalty separately for white wines (Rating for white wines column) showed that expert 1 retained the position of a loyal expert with 87.2 points, the least loyal was expert 6 with 75.6 points. The position of a disloyal expert was also retained by expert 10 with 81.2 points. An analysis of expert loyalty for red wines (Rating for red wines column) showed that experts 2 and 1 with 84.9 and 83.775 points, respectively, were the most loyal. Experts 9 and 10 with 80.525 and 80.825 points, respectively, were the least loyal. The average scores set by experts for white and red wines were 82.146 and 82.455, respectively. A slight difference in average values suggests that expert loyalty is resistant to wine types.

Conclusions

Cronbach's alpha criterion for positional analysis allowed a deeper look at the issues of the consistency and variability of expert evaluations. The advantages of Cronbach's alpha criterion over the traditionally used statistical criteria – Spearman's rank correlation coefficients and Kendall's concordance are that the criterion is calculated not according to the rank of expert points, but according to the initial point scale. The criterion allows to evaluate the contribution of each expert to the overall consistency highlighting those experts who increase or decrease it. Using the criterion, the reliability of the total score scale set by each expert can be evaluated and their loyalty can be analyzed.

In relation to the sensory quality evaluation data of 60 natural dry red and white wines of Russian origin, the pairwise and individual consistency of experts was investigated using pairwise and multiple correlation coefficients. The analysis of the general consistency of expert evaluations using Cronbach's alpha criterion and Kendall's concordance coefficient has been carried out. Cronbach's alpha criterion characterizes the overall consistency of expert evaluations as high and does not contradict the results of correlation analysis. Kendall's concordance coefficient does not fully take into account the consistency of expert evaluations compared to positional analysis, as its calculation is associated with a transition from a “strong” numerical point scale to a “weak” rank scale accompanied by a significant loss of information regarding the variability of expert evaluations. The positional analysis allowed to identify experts increasing/decreasing the overall consistency. The reliability of the loyalty scale has been shown – the scale of the average points set by each expert. The absence of the influence of expert gender and wine type on the loyalty of experts has been established.

Declarations

Author contribution statement

A. A. Khalafyan, V. A. Akin'shina: Analyzed and interpreted the data.

Z. A. Temerdashev: Conceived and designed the experiments; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Yu. F. Yakuba: Performed the experiments.

Funding statement

This work was supported by the (18-03-00059).

Data availability statement

Data included in article/supplementary material/referenced in article.

Declaration of interests statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.