A generalized multi-skill aggregation method for cognitive diagnosis.

Online education brings more possibilities for personalized learning, in which identifying the cognitive state of learners is conducive to better providing learning services. Cognitive diagnosis is an effective measurement to assess the cognitive state of students through response data of answering the problems(e.g., right or wrong). Generally, the cognitive diagnosis framework includes the mastery of skills required by a specified problem and the aggregation of skills. The current multi-skill aggregation methods are mainly divided into conjunctive and compensatory methods and generally considered that each skill has the same effect on the correct response. However, in practical learning situations, there may be more complex interactions between skills, in which each skill has different weight impacting the final result. To this end, this paper proposes a generalized multi-skill aggregation method based on the Sugeno integral (SI-GAM) and introduces fuzzy measures to characterize the complex interactions between skills. We also provide a new idea for modeling multi-strategy problems. The cognitive diagnosis process is implemented by a more general and interpretable aggregation method. Finally, the feasibility and effectiveness of the model are verified on synthetic and real-world datasets.

Introduction

Under accelerated globalization and the rapid development of information technology, online education has gradually developed. To restrain the spread of COVID- 19, most schools around the world have been closed. Online learning on a large scale started, which further promoted the development of online education [35]. Online education brings more possibilities for the personalized learning. However, facing many learners, how to better provide learning services for learners is a challenging issue. The premise is to identify learners’ current cognitive state. However, learners’ cognitive states can not be directly observed, which needs to be reflected through learning behavior. Cognitive diagnosis models (CDMs) can be used to identify students’ potential cognitive states and alleviate this problem to a certain extent. It has been viewed as a psychometric tool in educational assessments to estimate students’ proficiency [33]. CDMs provide specific information in response patterns to aid in acquiring the knowledge state that cannot be observed directly. In terms of students’ strengths and weaknesses, better instruction and possible interventions to address individual and group needs are implemented.

Several CDMs of various formulations have been proposed in the literature. Most CDMs consist of three major components:(1) -matrix [42], which is the prior knowledge assigned by education experts to denote which skills are needed; (2) the diagnosis vector of students’ proficiency of required skills; and (3) the condensation rules [34], which can be represented by the aggregate function. A general CDM Framework is shown in Figure 1. The -matrix can build a link between problems and the corresponding skills. As shown in the table on the right, Problem 1 only needs a single skill, and Problem 2 requires two skills to be answered correctly. More skills are involved in Problem 3. The first layer of the CDM framework consists of several related skill nodes. Students’ proficiency in various skills is described as vector , which can be implied by the response to the problem (e.g., correct or wrong). The third layer reflects the aggregation when multiple skills are involved in a problem. Then, the latent response () is acquired by combining the mastery of skills () and specific aggregate functions. Finally, the success probability is computed by the latent response with the slip and guess factors.

Fig. 1

A general cognitive diagnosis model framework

From Figure 1, we can see the aggregation method of cognitive diagnosis should consider the weight of skills and the determination of aggregate function in the multi-skill learning scenario. Although skill weight have been mentioned in a few studies, skill vectors are often assumed to have the same probability of answering a problem correctly. Tatsuoka [41] claimed that certain skills would be more critical than others in solving a problem or completing a task. The G-DINA model [13] decomposed the sum of the effects due to the presence of specific skills and their interactions. Lei et al. [21] developed a polytomous cognitive diagnostic model that considers that different skills indeed have different weight. When a student with certain required skills or the subset of the skills’ coalition has a higher probability of answering the problem correctly, it can be considered that the skills are more important for solving the problem. For instance, a problem requires four skills to be answered correctly. If the probability of correct response is similar to possessing all skills when a student only masters a partition of skills (e.g., as shown in Table 1). The reason for this may be that each skill vector varies in its degrees of effect. Mastering the coalition of skills and is more important than knowing the skills and , and their probabilities of success are not identical. Here, the importance of each skill affects the response. Hence, we need to describe the weight of a single skill and, more importantly, represent the weight of the subset of the skills’ coalition.

Table 1

Example of multi-skill interactions

x_1	x_2	x_3	x_4	P(Correct)
1	0	0	0	0.3
0	1	0	0	0.4
1	0	0	1	0.95
0	1	1	0	0.5
1	1	1	1	1

On the other hand , the functions employed to aggregate multiple skills are divided into two categories [36]: conjunctive and compensatory approaches. Especially, the conjunctive approach is widely used in CDM models, such as the deterministic inputs, noisy, ’and’ gate model (e.g., DINA model) [11, 22, 27] and the generalized DINA (G-DINA) model [13]. In these models, the interaction of multiple skills is conjunctive, which means that only a student who has all skills can answer the problem correctly. In the compensatory approach, some skills may make up for the lack of other skills, including the deterministic input, noisy-or-gate model (DINO) [28, 43], and the compensatory reparametrized unified model [23]. Based on these two approaches, Wu et al. [16, 49] further distinguished objective and subjective problems under this assumption: the skills’ interaction on objective (or subjective) problems is conjunctive (or compensatory). Moreover, Zhang et al. [51] modeled the structure of knowledge components and capture the correlations between submissions to evaluate student knowledge states. The difference between the conjunctive and the compensatory approach is the selection of the multi-skill aggregate functions. The more skills involved in a problem, the more complicated it is to determine which aggregation function should be used, and this is not a trivial task. Neither the conjunctive nor the compensatory approach can effectively express the fusion between multiple skills. Generally, regardless of which cognitive diagnosis model is selected, the multi-skills aggregation method is considered. Thus, this paper focuses on a generalized multi-skill aggregation method to reflect the skill weight and aggregate functions.

The multi-strategy problem also increases the difficulty of aggregation method modeling. Most CDMs ignore the diversity of strategies and assume that a single strategy that corresponds to a -matrix may be used to solve a problem. As shown in Table 2, a student may achieve success with Strategy A (using the combination of and ) or Strategy B (using the combination of and ). To address the issue of multiple strategies, De la Torre and Douglas [10] introduced a model that extends the DINA model to allow for multiple strategies of problem-solving. The multi-strategy DINA model may be coded by constructing M different matrices. Based on this idea, Tu et al. [46] took into account polytomous scoring items, and then Ma et al. [32] proposed a generalized multi-strategy CDM. These studies adopt multiple matrices to represent multiple strategies, while skill aggregation methods still only consider a single strategy [10]. In addition, in practice, the -matrix is typically assigned by domain experts, which is certainly affected by the subjective tendency of experts and, to a large extent, may consist of some misspecifications [47]. Considering multiple strategies in the multi-skill aggregation method, the model will become more general.

Table 2

Example of multiple strategies

Strategy	x_1	x_2	x_3	x_4	P(Correct)
A	1	1	0	0	1
B	0	0	1	1	1

To further explore the above challenges, We introduce a general multi-skill aggregation method to reflect skill weight, accommodate various aggregate functions (e.g., conjunction, compensation methods), and different strategy selections. The main contributions of this paper are outlined as follows: The rest of the paper is as follows: Section 2 introduces the relevant background knowledge, including the cognitive diagnosis model, fuzzy measure, and Sugeno integral. Section 3 describes the modeling process of the multi-skill aggregation method in cognitive diagnosis. Firstly, we discuss the representation of skill weight based on the interactions. Secondly, the Sugeno integral is introduced to realize the aggregation of multi-skill in cognitive diagnosis. Section 4 is compared with the representative models to illustrate the generality of our proposed method. In Section 5, the effectiveness of the aggregation method in solving multi-strategy problems is discussed and proved. Section 6 presents the experiment and results. Finally, the conclusions and discussion of future research are given in Section 7.

Illustrate the interactions between multiple skills and model the skills’ weight.

Develop a general multi-skill aggregation method that is independent of any specific cognitive diagnosis model.

Put forward a new approach to explore alternative strategies in cognitive diagnosis.

Background

Cognitive diagnosis models

Cognitive diagnostic models (CDMs) aim to identify cognitive processing and to evaluate whether a student has mastered or possesses specific cognitive skills or knowledge [16]. CDMs assume a relationship between the students’ knowledge mastery and the skills required to solve a problem. A massive effort has been made about cognitive diagnosis models, such as the DINA model, Item Response Theory (IRT) [6], G-DINA model [13], Polytomous Hierarchical DINA model (PH-DINA model) [4]. Specially, IRT is one of the most basic and classic psychological and educational theories which roots in psychological measurement [37]. With the student’s latent trait , problem discrimination a, and difficulty b as parameters, IRT can predict the probability that the student answers a specific problem correctly with a logistic function. The function is defined as follows:where is the correct probability, D is a constant which often as 1.7. However, the latent traits of students (such as mathematics comprehensive ability) in the IRT model do not consider the ability of students in each skill, it is unnecessary to discuss the multi-skill aggregation issue.

In this section, the details of cognitive diagnosis models and involved aggregation methods are discussed. DINA is considered to be one of the most widely used and representative cognitive diagnostic models. A series of extended models such as HO-DINA (Higher-Order DINA) [8], NIDA (the noisy inputs, deterministic “and” gate model)[27], and P-DINA (Polytomous deterministic inputs, noisy “and” gate[14] have been developed based on the DINA model, which greatly enriched the application of the DINA model. Firstly, we take the DINA model as a prototype to understand the process of cognitive diagnosis.

The DINA model

DINA is a dichotomous cognitive diagnosis model in which the skill’s mastery is expressed by . At this time, the calculated answer probability is either 1 or 0. Under DINA model [9], each item has a corresponding -matrix, which represents the skills involved. The -matrix specifies the skills required for each item and is a matrix of zeros and ones; the element in the jth row and kth column of the matrix is named .Let denote the proficiency of K skills that are needed to solve the items for the student. For a specific student, can be simplified as . The aggregate function between the student’s knowledge mastery and the item specification defines the latent response variable:Here, without the slipping and guessing factors, the variable can be regarded as an ideal response or latent response, which is also known as the mastery of the student on the problem [40]. In this paper, can be simplified as , and the two symbols are not distinguished in the following description and can be written alternately. From (1), only a student who has all the skills from the -matrix may answer a problem correctly. The skill weight are viewed as the same. Moreover, noise is introduced in the cognitive diagnostic process due to slip and guess parameters. To account for the probabilistic nature of the observed response, slip and guessing parameters are considered on the latent response [12], and are defined as and respectively. Here, means that the answer is correct, otherwise it is wrong. The factor denotes the probability that a student fails to answer the problem correctly when he possesses all the required skills. The factor is the probability of correctly answering without one or more required skills. Therefore, the probability of the student with skills mastery vector answering the problem correctly is given byNo matter the type of cognitive diagnosis model, the specified skill proficiency, skill weight, aggregate functions, and slip and guessing factors need to be considered. In the cognitive diagnosis process, the latent response is determined by the skills proficiency , skill weight, and aggregate function( in Figure 2). The aggregation method can realize the generation of the latent response. In the DINA model, the weight of each skill is the same, and the latent response is obtained by multiplying the proficiency of multiple skills, which can be regarded as a conjunctive aggregate function. As long as one skill’s mastery equals 0, the latent answering response is also 0.

Fig. 2

The process of cognitive diagnosis

The fuzzy cognitive diagnosis framework

Based on the general cognitive diagnosis process, the fuzzy cognitive diagnosis framework (FuzzyCDF) considered both subjective and objective exercise types of balancing precision and interpretability of the diagnosis results [31]. Without considering the weight difference, FuzzyCDF models two multiple skills aggregate functions according to the characteristics of subjective and objective problems. As shown in Figure 3 [30], the framework is a generic process that starts with the student’s latent traits (e.g., a general ability in math) and then get the student’s skill proficiency as follows:Here, we can see that the proficiency of a student on a specific skill() depends on the difference between the student’s high-order latent trait () and the properties of the skill: the difficulty () and discrimination () of skill k for student j [45]. The coefficient 1.7 is an empirical scaling constant in logistic cognitive models [26]. By (3), we can determine a student’s proficiency in specific skills from his latent trait. Next, the student’s problem mastery (i.e., the latent response) is computed by specific aggregation methods, and the observable scores are generated by considering slip and guess factors.

Fig. 3

The fuzzy cognitive diagnosis framework

In FuzzyCDF, each skill is deemed as equally important. The skills’ aggregation on objective problems is conjunctive, and the minimum operator has been used for the aggregate function. The latent response is defined as the following formulation under a conjunctive assumption:For subjective problems, the skills’ aggregation is under the compensatory assumption. This assumption means that a student could solve the problem by mastering at least one required skill. The maximum operator forms the aggregate function. Similarly, the latent response can be written as follows:In DINA and FuzzyCDF, skills share the same weight, and different functions are used for skills fusion to form a multi-skill aggregation method. However, in the practical learning scenario, the weight between skills are often unequal. We also expect to establish a unified expression of conjunctive and complementary aggregate functions. Therefore, the determination of the aggregation method should consider the weight of multiple skills and the selection of the aggregate function.

Fuzzy measure

The classic measure theory stems from measuring the size of sets, such as the length of objects, the area of regions, or the weighted average method; these are additive measures. However, in many cases, additivity cannot be satisfied. For example, the efficiency of two workers’ cooperation cannot be simply equal to the sum of their efficiencies. Sugeno [38] first proposed weak monotonicity instead of additivity for set functions and called fuzzy measures. Thus, fuzzy measures can be used to handle non-additive issues. The fuzzy measures are an extension of the classic measure theory, enabling one to represent the non-additive property efficiently [2]. The definition is as follows:

Definition 1

A set function is a fuzzy measure if it satisfies the following axioms:

, (normalized)

whenever for (monotonicity)

Here, X is denoted as a finite set of elements, such as players in a cooperative game, criteria in a decision problem, attributes, experts, or voters in an opinion pooling problem, etc. Specifically, if we have , , the fuzzy measure is called a probability measure [17], which is an additive measure. The non-additive feature makes fuzzy measures reflect the interdependence among the elements in detail by measuring importance. Fuzzy measures usually denote importance, reliability, satisfaction, or similar concepts [44].

Sugeno integral

The Sugeno integral is one of the most important fuzzy integrals, and it has been proved useful in several applications [20], notably in some game theory problems and multi-criteria decision-making [18]. A decision-making problem is to find a desirable solution from a finite number of feasible alternatives assessed on multiple skills [48]. Here, the answer response depends on the proficiency of multiple skills, which is regarded as a decision-making problem. The properties of the Sugeno integral are well known, as well as its relationship with classical aggregation operators.

Definition 2

Given a finite set , a fuzzy measure and a function , also written as , the Sugeno integral with respect to fuzzy measures is defined by

Here, for all it holds . Without losing generality, we assume that is monotonically increasing with respect to k. Here, is treated as a function that is ordered by . The Sugeno integral is solely defined based on aggregate operators and as opposed to other integrals, it can be applied when set elements are in a complex relationship. Note that , if and only if , and , if and only if .

The generalized multi-skill aggregation method based on the Sugeno integral

This section models the generalized multi-skill aggregation method based on the Sugeno integral (hereinafter referred to as SI-GAM) independent of any specific cognitive diagnosis model. SI-GAM combines the mastery supplied by required skills according to weight of multi-skill to obtain the latent response. Therefore, the modeling of SI-GAM consists of two parts: the representation of skill weight and the selection of aggregate function.

The representation of skill weight

Firstly, we discuss the interactions between multiple skills. The interactions exist not only between single skills but also between skill sets. Then, we introduce fuzzy measures for modeling the interactions quantitatively.

The interactions between multiple skills

Most entities in the real world are interconnected, which form complex networks [50]. In the field of learning, there are complex interactions between skills. More often than not, mastering certain skills increases the probability of success than getting other skills. As mentioned in Jimmy’s research [13], mastering three of the four required skills for the problem, namely, performing a basic fraction subtraction operation (), separating the whole number from the fraction (), and borrowing one from the whole number (), provided a markedly higher probability of success compared to any other subsets of skills. This means that different skills may have different weight for solving a problem due to the influence of the skill or interactions with multiple skills. Thus, the importance of correctly identifying the skills cannot be overstated. The aggregation method should reflect the skill weight for correct answers.

The first step in making inferences with the Attribute Hierarchy Method depends on accurately identifying the psychological ordering of cognitive competencies required to solve test problems. Identifying skill hierarchies serves a critical function. In this paper, we discuss the possible interactions between multiple skills under the hierarchical structures (including linear, convergent, divergent, and independent hierarchies [29, 39]).

Redundancy If the contribution of a pair of skills , is not greater than the sum of their individual supports, there is a negative synergistic relationship between and . That is to say, there is no enhancement by combining the skills and . The general formula is defined as: Here, denotes a certain weight, that is, the degree of support or contribution of the skill to the correct answer. Redundancy can be regarded as a particular case of negative synergies, such as the prerequisite relationship between skills. In terms of the hierarchical structure between skills, the prerequisite can be viewed as a linear structure. For example,the skill is a prerequisite to skill (shown in Figure 4); this implies that a student is not expected to possess skill unless they already possess skill . Thus, skill is more important than for answering the problem correctly, and thus skill has a larger weight: . Then, we define the interaction of skills and , , that means that the support strength of the combination of and is equal to the skill weight with more support. The skill may not make any contribution to improving the correct answer rate. The redundancy relationship between and is defined by: Hence, the supports are not additive because of the existing hierarchy relationship. Other skills follow a similar interpretation. Mastering skill is a prerequisite for obtaining . Therefore, it can be considered that there is redundancy between and , that is, . Other skill pairs ( and , and , etc.) also follow this relationship.

Fig. 4

Linear structure between skills

Positive synergy The performance of the pair of skills is better than the sum of their individual performances. That is to say, the combination of and directly impacts the probability of a correct response as opposed to using each skill separately. The skills interact through compensation, and the expression is defined by: Combined with the hierarchical structure between skills, the divergent structure[29] (Figure 5) can be considered as a manifestation of positive synergy. If a student has mastered both the skills and , the probability of answering the problem correctly is higher than the sum of the success probabilities of mastering skill and skill alone. The coalition of and may produce more chemical effects, such as creating a new latent skill, which is important for solving the problem. This can be denoted as . The contribution of the combination of and is greater than the sum of the two skills separately. Mastering both and can achieve a dramatic increase in the probability of success on a problem. The interaction of positive synergy is also non-additive.

Fig. 5

Divergent structure between skills

Independency Intermediate case, where each skill contributes to the probability of a correct response: As shown in Figure 6 [29], the skills under the structure are independent. There are additive relationships since the multiple skills do not interfere with each other. The expression can be written as: .

Fig. 6

Independent structure with skills

From the assumption that skills are hierarchically organized, it is essential to note that the structures in Figures 4, 5 and 6 can be combined to form increasingly complex networks of hierarchies where the complexity varies with the cognitive task. That is, there may be non-additive interactions between skills in the problem. Commonly used weight calculation methods (such as the weighted average) cannot reflect this interactive relationship between skills. Thus, our paper tries to characterize non-additive weight, which involves the effect of singletons and the interactions of multiple skills.

Using fuzzy measures for the non-additivity weight of skills

In this paper, skills’ weight can be interpreted as certain importance, i.e., the support strength of skills for the correct response. Fuzzy measures are introduced to construct the non-additive weight for skills, which are powerful in characterizing the interactions of multiple skills [35]. For instance, we have a set of skills that solve a problem, and we assume the fuzzy measures and . Here, the skill has more support than for the correct response. That is to say, the skill is more critical than in solving the problem. As noted earlier, we cannot apply the weighted calculation directly to add the support measures. If we suppose , joining with does not yield a markedly higher probability of success than having a grasp of alone. Obviously, . The weak monotonicity is in line with the characteristics of skills. We can use a non-negative monotone set function defined on its set to describe the importance of each information source (i.e., the support strength of the skill) and their varied combinations. Thus, the non-additivity of fuzzy measures can satisfy the non-additive weight for multi-skill aggregation. Fuzzy measures enable us to express the interactions better.

It is not comprehensive to determine the global importance of each skill only by the measure of a single skill; we must also consider the measured value of the subset involved in the skill. Therefore, the importance of the skills coalition should be discussed. Let us denote X as a set of skills containing all possible skills needed to solve the problem. Then , represents the importance or strength of the coalition A for the particular skills group under consideration. To have a flexible representation of complex interactions between skills (e.g., positive synergy or redundancy between skills), it is necessary to consider fuzzy measures in their full generality and not restrict oneself. In multi-skill aggregation problems, X denotes the set of all involved skills.

Let be a set of skills. The number of fuzzy measures is : . We suppose that there are four skills required for a specific problem, the fuzzy measures are shown in Table 3.

Table 3

A sample of fuzzy measures

Fuzzy measure set	Corresponding value	Fuzzy measure sets	Corresponding value
∅	0	μ({x_2,x_3})	0.55
μ({x_1})	0.2	μ({x_2,x_4})	0.75
μ({x_2})	0.3	μ({x_3,x_4})	0.78
μ({x_3})	0.1	μ({x_2,x_3,x_4})	0.85
μ({x_4})	0.6	μ({x_1,x_2,x_3})	0.87
μ({x_1,x_2})	0.32	μ({x_1,x_2,x_4})	0.9
μ({x_1,x_3})	0.52	μ({x_1,x_3,x_4})	0.92
μ({x_1,x_4})	0.72	μ({x_1,x_2,x_3,x_4})	1

According to Definition 1 in Section 2, whenever for (monotonicity). It is easy to see that these corresponding values satisfy the properties of fuzzy measures. Table 3 shows that the interaction of the skill and is redundant, that is , and the combination of and enhances the support of success, that is . Moreover, we infer that the skill plays an important role in this problem. Using fuzzy measures instead of the equally weighted status can more effectively represent the interactions between skills.

The aggregation of multiple skills in cognitive diagnosis

Based on the importance of aggregate function selection, our idea is to propose a multi-skill aggregation method. This method combines the representation of skill weight and expresses the function of conjunctive and compensatory in a uniform way.

The importance of aggregate function selection

In practical learning scenarios, the aggregation of multiple skills is a complex issue. For an arithmetic problem, such as (Figure 7), four skills are required based on the -matrix.

Fig. 7

-matrix for the fraction subtraction data

Assuming the skill proficiency of a student is given in Table 4, the value is a binary variable in (Dichotomous value), such as in the DINA model and DINO model. If the aggregation method is conjunctive, the latent response equals zero; the latent response equals one under the compensatory approach.

Table 4

Different aggregation for dichotomous

Aggregation Method	Skill Proficiency (dichotomous)
	α_1=1	α_2=0	α_3=0	α_4=0
Conjunctive	η=0
Compensatory	η=1

The selection of an aggregate function also has a dramatic impact on the polytomous situation. In FuzzyCDF [49], the skill proficiency is a fuzzy variable in [0, 1] (polytomous value). From Table 5, if the method is regarded as the conjunctive, using the min operator can compute the latent response based on (4). Then the approach is the compensatory type, and we use the max operator to calculate based on (5).

Table 5

Different aggregation for polytomous

Aggregation Method	Skill Proficiency (polytomous)
	α_1=0.9	α_2=0.3	α_3=0.4	α_4=0.6
Conjunctive	η=min(α_1,α_2,α_3,α_4)=0.3
Compensatory	η=max(α_1,α_2,α_3,α_4)=0.9

Different aggregate functions lead to different results. In other words, the selection of aggregate functions may influence the latent response. Specifically, if the lack of any skill means the student will not answer correctly, we should choose the conjunctive approach. Conversely, if any one of these skills can make up for the lack of other skills so that the student can provide the correct answer, the compensatory approach is a more suitable choice. However, using only the conjunctive or compensation function may not reflect the fusion of practical problems.

Using Sugeno integral for multi-skill aggregation

In cognitive diagnosis, multiple skills may be independent or interact with each other. In this paper, the multi-skill aggregation should consider non-additive interaction. Furthermore, we should define an appropriate function for skills fusion. Sugeno integrals are known to be one of the most influential and flexible functions. They permit the fusion of the information under different assumptions on the independence of the information sources. In particular, they can be used to model situations in which sources are independent and in cases where such independence cannot be assured. Thus, we introduce the Sugeno integral into the multi-skill aggregation method (SI-GAM) to represent the interactions and fusion of skills. The Sugeno integral models the aggregation method among globally important criteria. Here, the criteria denote skill proficiency, which affects problem-solving performance. The global importance of criteria depends on the non-additive weight, i.e., the fuzzy measures. From (2), we explore an alternative way to calculate using Sugeno integral:In SI-GAM, the latent response is defined over an aggregate function on the reference set of skills X. Then, given in X, the variable corresponds to the mastery of the skills on X. Next, sort , and the sorted result refers to the ordered inputs generated by , that isHere, skill weight is expressed by the fuzzy measures , which are considered the support strengths for the correct response, and thus have some importance to the corresponding problem. The fuzzy measure :Corresponding with the chain nested subsets induced by the ordering permutation of . There are two operators involved in (10), which generalizes max-min operators. The SI-GAM can also be defined as:where , and is the skill corresponding to the ranked skill’s mastery .

Example 1

There are four skills required for solving the problem in Figure 7. The latent response can be computed by the mastery of skills and corresponding fuzzy measures. A student’s proficiency in the four skills is , , , and . is the vector of skill proficiency expressed on some ordinal scale. After reordering in ascending order, this is expressed as , , , .

Assuming the fuzzy measures are known (seen in Table 3), it is sufficient to define the mapping of the proficiency of skills to the latent response :

The solution of multi-skill aggregation was inspired by the Sugeno integral, one of the typical representatives for fuzzy integrals that are very useful for fusing information from various sources. The SI-GAM can take partial support for the result from the standpoint of each information source and fuse it with the worth of each subset of X is a non-additive fashion. This worth is encoded into a fuzzy measure [1]. In our proposed SI-GAM, the interactions between skills are expressed by fuzzy measures, and the min-max operator expresses the aggregate function to realize the modeling of the multi-skill aggregation method.

Comparison with other cognitive diagnosis models

In this section, SI-GAM will be compared to other aggregation methods in existing models. Through derivation, it can be proved that SI-GAM is also applicable to current models. Firstly, we discuss the non-compensatory /conjunctive model (DINA) and compensatory model (DINO )for dichotomous response. Secondly, we compare with FuzzyCDF, which can express the skill fusion of conjunctive and compensatory in polytomous response simultaneously. Since most cognitive diagnosis models consider the equal weight of each skill, without loss of generality, we simplify the value of fuzzy measures to , that is, if the skill sets () has support for correct answering, the , otherwise .

Comparison with DINA model

In the DINA model, the aggregate function is conjunctive. The fuzzy measures can be defined by:Using SI-GAM, we can well describe the skills’ fusion by conjunctive approach as follows (14):With SI-GAM, we can infer that the skill with the lowest mastery determines the value of the latent response . Thus, if , the skill is not mastered by the student. The latent responses obtained from both approaches are equal to 0. Our SI-GAM can be viewed as a variant of the conjunctive approach .

Example 2

Assuming the skill proficiency from Table 4, the response can be computed. The condensation rule in the DINA model typically assumes that a student that has mastered all required skills may solve the problem. Hence, the latent response (1). is the skill of the worst mastery among the four skills. Since the DINA model is dichotomous, we can define . Combined with the value of fuzzy measures from (14), the latent response is under SI-GAM. Table 6 shows that the two approaches obtain the same results. In other words, the DINA model’s conjunctive approach can be considered a special case of SI-GAM.

Table 6

Different aggregation methods for dichotomous

Aggregation method	Skill proficiency (dichotomous)
	α_(1)=0	α_(2)=1	α_(3)=1	α_(4)=1
Conjunctive approach	η=0×1×1×1=0
SI-GAM	η=max(min(0,1),min(1,0),min(1,0),min(1,0))=0

Comparison with DINO model

Moreover, the DINO model is the compensation version of the DINA model. It is assumed that a student can answer the problem correctly as long as they master any skill required by the problem, that is, the skills can replace or compensate each other. Thus, in the DINO model, the latent response can be defined as follows:where specifies whether skill is measured by item j. takes the binary values of 0 or 1. In case that skill k is measured by item j, would take a value of 1, otherwise, equals 0. If the students master the skill , then , and thereby would be 0. However, if the student does not master the skill , . Thus, the latent response of the DINO model is rewritten as:Here, is the representation of sorting the skills’ mastery in ascending order, that means . Hence, we can also illustrate the latent response variable using SI-GAM, which is written as:Under SI-GAM, we can deduce that the skill with the highest mastery determines the value of the latent response . The results are consistent with (15). Thus, it can be seen that the compensation approach in the DINO model can also be regarded as a special case of SI-GAM.

Comparison with FuzzyCDF

Similarly, SI-GAM can realize the unified modeling of aggregation under conjunctive and compensatory functions. Two types of fuzzy interactions express the problem mastery in (4) and (5). Suppose the skills’ interaction is conjunctive (or compensatory). In that case, a student’s mastery of this problem is the degree of membership of this student in the intersection (or union) set of the fuzzy sets related to the skills required by the problem. This framework fuzzifies the skill proficiency, which redefines the original binary variable (i.e., mastered or non-mastered) to a fuzzy one valued in [0, 1] [30].

Conjunctive aggregation For objective problems, the conjunctive aggregate function is defined as: In our SI-GAM, we first determine the fuzzy measures by the skill’s interactions. If the interaction for the skills has a conjunctive feature, we can argue that there is sufficient support to answer correctly if the fuzzy measure set includes all skills. We can infer the fuzzy measures using (14), and then, the fusion of with respect to for each skill will give the success probability for a specified problem. For objective problems, the lowest mastery of skills has a marked impact on the outcome.

Example 3

If the problem requires four skills, we make the following assumption: , , , from Table 5. In FuzzyCDF, the latent response under the fuzzy interaction equals . In SI-GAM, the value of fuzzy measures is given from (14), the computation of the latent response (from 13) is then expressed as:Now, two methods obtain consistent results. The aggregation method for objective problems in FuzzyCDF is a special case of SI-GAM as well.

Compensatory aggregation In this case, a student could obtain a higher probability of success for the problem when she has mastered at least one of the required skills . In FuzzyCDF, the latent response from (5) and (12). According to Definition 1, for any , the fuzzy measures as , if . Therefore, the latent response is It is easy to conclude that the highest proficiency of skills is the most critical influence.

Example 4

Given the four skills’ proficiency as in Table 5, the latent response is with the compensatory interaction in FuzzyCDF. We can also compute the result in SI-GAM. Here, SI-GAM achieves the same values with the aggregation method for subjective problems. Therefore, the conjunctive and compensatory aggregation for objective and subjective problems in FuzzyCDF can be regarded as special cases of SI-GAM.

In the above description, we have shown more of a generalized aggregation method to obtain a more precise representation of responses for each student. The fuzzy measures in SI-GAM can model the complex interactions between skills, reflect the support strength of skills and skill sets for the correct response, and combine the min-max operator to generate a more general aggregation method.

Using SI-GAM for multiple strategies problems

In practical applications, two or more strategies may provide similar support strength to solve the same problem. At present, the main idea behind solving multi-strategy problems suppose that each item has as many as distinct strategies that would suffice. In the multi-strategy DINA model [10], each problem has M distinct strategies. Multiple strategies may be coded by constructing M different matrices, . Let , for , where denotes the element in the row and column of . The variable indicates whether student has the skills to apply the strategy to the item. Here, the latent response is denoted as:Suppose there are two strategies for a problem. For Strategy A, there are n skills required to slove the problem, the set of involved skills is , where . The -matrix can be expressed as . Similar, the skills set , that is, for Strategy B, m skills are needed for the problem, where The corresponding matrix is . Here, , and , and we haveBased on this assumption, following Eqs. 17 and 18, the latent response can be written as:Since the skill proficiency of DINA model only takes 0 or 1, the result of multiplication operation depends on the lowest value of .

Next, we deal with multi-strategy problem using our SI-GAM. Obviously, and . For the aggregate function under the DINA model is conjunctive, we can determine fuzzy measures as , for satisfying or , and for other subsets .

Let , we haveFrom (13), the latent response is given bywith , , and . To compute the latent response, we need to determine the value of fuzzy measures and discuss the different cases that satisfy these constraints.For readability, the derivation of the latent response under three assumptions is presented in the Appendix 1. Here, it is not hard to notice that two methods can get the same result. The SI-GAM can also solve the multi-strategy problem in cognitive diagnosis. In addition, the SI-GAM can also deal with the polytomous scoring for the multiple strategies by changing the value range of the proficiency of skills to the interval of [0, 1] expands the existing multiple-strategy DINA Model.

Case 1. If , then .

Case 2. If , then .

Case 3. If , then .

Example 5

Two alternative strategies both support solving the problem; e.g., . Let X be a finite set . One strategy, Strategy A, requires students to perform fraction subtraction with mixed numbers and involves skills . The other strategy, Strategy B, requires subtraction of fractions where mixed numbers are first changed to improper fractions and involves skills . When using Strategy A, which requires skills , for the problem, arriving at the correct answer would involve the steps given in Figure 8. Alternatively, Strategy B, which requires skills , and for this problem, involves the steps outlined in Figure 9. Substituting the expression for the multi-strategy DINA model (19), we get:Suppose that the proficiency of each skill involved is , then for Strategy A, and for Strategy B, . The latent response is calculated as ,

Fig. 8

Solving a fraction subtraction problem using Strategy A

Fig. 9

Solving a fraction subtraction problem using Strategy B

Next, we use the SI-GAM to solve the multi-strategy problem. According to the given -matrix from Figures 8 and 9, the fuzzy measures can be determined. The fuzzy measures meet the following constraints: , for satisfying or . In addition, for , if and . From (20), SI-GAM can be used to calculate the latent response as follows:where and . Since , and , it is clear that . We can directly use the conclusion from Case 1, that is, . Clearly, the result demonstrates the effectiveness of SI-GAM in solving multi-strategy problems. By the specific example, it can be seen that SI-GAM can solve the multi-strategy problem instead of the multi-strategy DINA model.

Experiment

Since multi-skill aggregation is a part of cognitive diagnosis, it needs to be implemented in a specific cognitive diagnosis model. In the framework of FuzzyCDF, the generation from skill proficiency to problem mastery is the process of multi-skill aggregation. Only the skill aggregation process in the model is modified to more clearly observe the performance brought by the aggregation method. To better understand our proposed aggregation method, we put our SI-GAM into FuzzyCDF as an aggregation method to form an improved model ( FuzzyCDF-SI-GAM ) and represent FuzzyCDF-SI-GAM using a graphic model, as shown in Figure 10.

Fig. 10

The graphic model of FuzzyCDF-SI-GAM

Here, what we can observe are the score matrix R, and the -matrix with K skills (if the problem requires skill k, then ). A student is related to skill proficiency , , which depends on latent trait and skill parameters , , (3). The latent response is determined by required skill proficiency and the skill weight (13), and a problem score R is influenced by and problem parameters s, g. The generation of the scores is simulated as follows:In FuzzyCDF[30], the fuzzy intersection and fuzzy union are introduced to calculate the subjective and objective problems, respectively. However, SI-GAM is a more generalized aggregation method that does not distinguish between objective problems and subjective problems. Regardless of objective or subjective problems, SI-GAM adopts a unified formulation to calculate the latent response.

DataSet description

To verify the effectiveness of the proposed SI-GAM in dealing with skills aggregation and multi-strategy problems, we set up experiments to predict students’ learning performance by synthetic and real-world datasets1. The specific work involves mapping relations from skill mastery to the latent response with different aggregation methods. First, the synthetic datasets (D1 and D2)are presented in this paper. In dataset D1, we design 10 problems that include three interactions between skills, i.e., redundant, enhanced, and independent. The interactive relationship between skills is reflected by the value of fuzzy measures. For example, Problem 3 involves skills , where , which means that the support strength of skill for the problem is 0.2. From the Table 7, it is shown that . As can be seen from the (9), there is an independent relationship between skills and . In addition, . According to the (8), the interaction between skills and is positive synergy. In D1, which contains complex interactions, we got the scores of 5980 students on 10 problems. While considering the possibility of multiple strategies for specified problem, D2 generates by 10 problems as well. In D2 dataset, there are only two strategies for each problem. Similarly, we got the scores of 6209 students on 10 problems. Taking Problem 6 as an example, a student who has mastered the coalition of skills , , or only mastered skill and can solve the problem. Hence, the first strategy can be represented by fuzzy measure , another strategy is expressed as .

Table 7

A sample of fuzzy measures

Fuzzy measure sets	Corresponding value	Fuzzy measure sets	Corresponding value
∅	0	μ({x_2,x_3})	0.2
μ({x_1})	0.2	μ({x_2,x_4})	0.3
μ({x_2})	0	μ({x_3,x_4})	0.7
μ({x_3})	0.2	μ({x_2,x_3,x_4})	0.7
μ({x_4})	0.3	μ{x_1,x_2,x_3}	0.5
μ({x_1,x_2})	0.2	μ({x_1,x_2,x_4})	0.5
μ({x_1,x_3})	0.5	μ({x_1,x_3,x_4})	1
μ({x_1,x_4})	0.5	μ({x_1,x_2,x_3,x_4})	1

Table 8 describes the details of the datasets. Data sets D1 and D2 respectively contain the response data on 10 problems, of which Problem are objective problems and the rest are subjective problems. Then, the fuzzy measures with respect to specified skills are constructed, which follow the properties of the fuzzy measures described in Section 3. The real-world dataset (Math1) is collected from two final mathematical exams from high school students. There are twenty problems in Math1, including 16 objective and 4 subjective items. Each problem in the Math1 dataset involves more than two skills by the -matrix. The FuzzyCDF-SI-GAM model obtains students’ mastery on each skill as the input, and the output is the response score of students. From the datasets, input-output label data can be obtained to form a training set, which is used to train the parameters in the FuzzyCDF-SI-GAM model.

Table 8

Dataset details

Dataset	Students	Skills	Problem
			Obj.	Sub.
D1 ( Complex Interaction )	5980	4	6	4
D2 ( Multiple Strategies )	6209	4	6	4
Math1 ( Real-world Dataset)	4209	11	16	4

Experimental setup

This experiment introduces an effective training algorithm using MCMC method to estimate model parameters [15]. The parameters involved are randomly generated by a certain distribution, so as to calculate and generate students’ probability of correct answer. Specifically, following the settings adopted in the HO-DINA model [45], we assume the prior distributions of the parameters based on the as:where is a four-parameter Beta distribution that has two shape parameters v and w and is supported on the range . In the training algorithm for FuzzyCDF-SI-GAM, the joint posterior distribution of , and given the score matrix R is:where L is the joint likelihood function of FuzzyCDF-SI-GAM:where Lo and Ls denote the joint likelihood functions of objective and subjective problems, respectively, and they can be defined as follows:where . Note that , i.e., the latent response can be calculated given the -matrix by using (13). Then, the full conditional distributions of the parameters (a, b) given the observed score matrix R and the rest of the parameters are as follows:Similarly, the the full conditional distributions of the parameters, e.g., , s, g, are given by this way.

In our FuzzyCDF-SI-GAM, we first randomize the parameters ( a, b, , s, g, ) as the initial values. In this framework of FuzzyCDF-SI-GAM, fuzzy measures are known as prior knowledge or learned from data by other model. For each iteration, each parameter is uniformly and randomly sampled from a predefined interval. Then, given the observable R and the -matrix, we compute the full conditional probability of skill discrimination a, skill difficulty b, student’ trait , problem slip factor s, and guess factor g and the variance of normalized scores of subjective problems by using (23), (24), (26), (25) and (27). Next, the acceptance probability of samples can also be calculated using a Metropolis-Hastings-(M-H) based MCMC algorithm. In this way, we could estimate the parameters after T iterations of sampling.

Experimental results and analysis

In this section, we conduct experiments on the task of predicting the scores of students over each subjective or objective problem. To observe how the methods behave at different sparsity levels, we construct different sizes of training sets, with , , of the response data of each student and the rest for testing, respectively. We use the root mean square error (RMSE) and mean absolute error (MAE) as the evaluation metrics. Then, we consider baseline approaches as follows:

IRT: a cognitive diagnosis method modeling students’ latent traits and the parameters of problems like difficulty and discrimination [5, 37].

DINA: a cognitive diagnosis method modeling students’ skill proficiency and the slip and guess factors of problems with a -matrix. [27].

FuzzyCDF: a fuzzy cognitive diagnosis framework for students’ cognitive modeling with both objective and subjective problems [30].

In order to ensure fairness of comparison, we record the best performance of each algorithm by tuning their parameters. Figure 11 shows the performance of our SI-GAM model and other baseline approaches on two datasets when dealing with complex interactions and multi-strategy problems. The results prove the effectiveness of our SI-GAM. In particular, compared with the FuzzyCDF model, the performance is improved only by using SI- GAM under the same parameter training model. More importantly, as the sparsity of training data increases (the training data ratio decreases from to ), our SI- GAM still has advantages over other baselines. In D1 and D2, the performance of SI-GAM surpasses each baseline method on evaluation metrics (RMSE and MAE). In addition, it should be noted that SI-GAM is more explainable, it can more accurately understand the cognitive state of students, and better identify the support strength of multiple skills in problem-solving.

Fig. 11

The performance of SI-GAM model and baselines

To enhance the evaluation of our SI-GAM, we also conduct the same experiment on the Math1(Real-World Dataset) dataset [3]. Because the dataset is not labeled with fuzzy measures, we learn the fuzzy measures by a neural network. In this experiment, the fuzzy measures are known by the Fuzzy Integral Neural Network. According to the definition 1, the fuzzy measures can be expressed as:Here, the monotonicity of the fuzzy measures can be ensured. According to the monotonicity constraint of fuzzy measures, the Fuzzy Integral Neural Network designs an optimization method based on random gradient descent to realize the training of fuzzy measures [25]. In the experiment, AUC (Area Under the Curve) and MAE were used to compare the prediction of cognitive diagnostic models. Firstly, of the data were randomly selected for training and for testing. In order to further observe the effect under the different sparsity, of data were randomly selected for training, and the rest was test data. Table 6 shows that compared with the FuzzyCDF model, the MAE error of the SI-GAM decreases after introducing the fuzzy measures and has obvious advantages compared with the IRT and DINA model. On the other hand, we calculate the AUC metric for objective questions in the Math1 data set. From Table 9, the accuracy of SI-GAM is higher than all other baselines. In addition, with the increase of training data sparsity, the advantages of SI-GAM still exist. As shown from the results, our multi-skill aggregation method (SI-GAM) can achieve a better performance in predicting students’ performance .

Table 9

Comparisons with baselines in Math1 ( Real-World Dataset )

Dataset	Test Ratio	Model	MAE	AUC
Math1	20%	SI-GAM	0.285	0.687
		FuzzyCDF	0.322	0.678
		IRT	0.330	0.648
		DINA	0.375	0.633
	80%	SI-GAM	0.311	0.658
		FuzzyCDF	0.337	0.649
		IRT	0.361	0.623
		DINA	0.416	0.501

Furthermore, we set the sensitivity experiment of the parameters. We observe that fuzzy measure is an important parameter in SI-GAM, reflecting the weight of skills. However, it is unknown how the effectiveness of our SI-GAM varies with the precision of fuzzy measures. Therefore, we tune up the known fuzzy measures within a certain range and observe the corresponding latent response. Take Problem 1 in dataset D1 as an example, which involves three skills, , and , where . We choose skill , with large weight for this experiment. Firstly, the value of the fuzzy measure is set to fluctuate within the range of [0.38.0.42], and other fuzzy measures of supersets containing are randomly generated under the monotonicity assumption. Then, we calculate the average absolute error between the latent result of adjusted by and the real result when . It can be seen from Figure 12 that the average absolute error is less than , when changes within the range of [0.38, 0.42]. Similarly, we set the value of within the range of [0.28, 0.32], and the average absolute error is less than . In dataset D2, We randomly selected a problem involving four skills, where and . We set up fluctuation values of up and down for the fuzzy measures of skills and , respectively. As can be seen from the Figure 12, the average absolute error is still less than . With the decline of the accuracy of fuzzy measure( within ), the performance degradation begins a gradual drop, but the error remains at a low value, thus the SI-GAM has good stability in cognitive diagnosis.

Fig. 12

Sensitivity of Fuzzy measure

Case study

Here we present an example of a student’s diagnostic result of SI-GAM on D2 for multi-strategy problems. The Figure 13 shows the proficiency of the student on four skills. According to the -matrix, the strategy used by the FuzzyCDF includes skills 3 and 4. Problem 6 is an objective problem, involving skills and . From (4), we can calculate the latent response based on FuzzyCDF . Further, considering the slip and guess, the prediction probability is 0.42. Then, we can discretize the result for prediction on objective problems by the predefined threshold (usually set to be 0.5). Thus, the prediction equals 0. Using SI-GAM, the latent response . With the slipping and guess factors, the prediction probability is 0.66, the result equals 1. The result of SI-GAM model is consistent with the observed data. From the fuzzy measures, we can better explain the diagnostic result. Since there are two strategies for solving the Problem 6, that is and . Therefore, we can explain why the student answer correctly. Although he has a low level of skill , he has a better mastery of skills , and . Therefore, he has a higher probability of success when he uses another strategies (using skills , and ).

Fig. 13

The -matrix of Problem 6 and corresponding skills’ mastery and response of a student in D2 for multiple strategies problems

From the above experiments, we can observe that the aggregation method (SI-GAM) proposed in this paper provides more accurate prediction results for complex skill interactions and multi-strategy problems. From the perspective of the aggregation method itself, the fuzzy measures construct a more explanatory way of modeling the interactions between skills to represent the weight. In summary, SI-GAM captures the characteristics of students more precisely, and it is more suitable for the real-world scenarios where complex interactions and multiple strategies exist.

Conclusion

This paper proposes a generalized multi-skill aggregation method based on the Sugeno integral (SI-GAM). The proposed method incorporates non-additive weight, that is, fuzzy measure. Through the representation of fuzzy measures, the interactions among multiple skills are considered. In addition, the max-min operator of Sugeno integral can also describe different aggregate functions, including conjunctive and compensatory approaches. At the same time, it also proves that our SI-GAM is suitable for multi-strategy issues, which provides a new idea for solving problems of multiple strategies. Our future work is to consider the learning of fuzzy measures efficiently in the cognitive diagnosis model. We can also view the following two aspects for future research:

The application of the fuzzy measures can be further expanded. The global importance of skill is not only determined by its own fuzzy measure, but also needs to consider the fuzzy measure value that includes all subsets of the skill. The shapely value and interaction index[19] can be used to measure each skill’s importance and mutual influence. The calculation of these two indices depends on the determination of the fuzzy measure value.

How to learn fuzzy measures is an important issue. In this paper, the prerequisite of SI-GAM is that the fuzzy measure is known as prior knowledge or learned from data by a neural network separately. Constructing an effective cognitive diagnosis model that can learn fuzzy measures simultaneously is a focus for future work. At the same time, we also need to consider the dimensional space of the fuzzy measure. By definition, if n skills are involved, fuzzy measures are required. If there are multiple skills, such as , the number of possible fuzzy measures is 1024. As the number of skills increases, the number of fuzzy measures will increase exponentially, bringing dimensional disaster. How to balance dimensional space and accuracy is also a question worth considering. Next, we will establish a unified expression framework for cognitive diagnosis based on neural networks and realize fuzzy measures learning. However, they are often regarded as a black box, i.e., their predictions cannot be explained [7], which makes our future work face more challenges.