Social norms in indirect reciprocity with ternary reputations.

Indirect reciprocity is a key mechanism that promotes cooperation in social dilemmas by means of reputation. Although it has been a common practice to represent reputations by binary values, either 'good' or 'bad', such a dichotomy is a crude approximation considering the complexity of reality. In this work, we studied norms with three different reputations, i.e., 'good', 'neutral', and 'bad'. Through massive supercomputing for handling more than thirty billion possibilities, we fully identified which norms achieve cooperation and possess evolutionary stability against behavioural mutants. By systematically categorizing all these norms according to their behaviours, we found similarities and dissimilarities to their binary-reputation counterpart, the leading eight. We obtained four rules that should be satisfied by the successful norms, and the behaviour of the leading eight can be understood as a special case of these rules. A couple of norms that show counter-intuitive behaviours are also presented. We believe the findings are also useful for designing successful norms with more general reputation systems.

Introduction

The ability to cooperate with others that are genetically unrelated is a remarkable trait of humans. Reputation is formed by evaluating each other after observing who does what against whom, which in turn incentivizes an action that is costly but beneficial to others because those with good reputations are likely to receive benefits subsequently in society. This is known as indirect reciprocity, one of the most fundamental mechanisms for maintaining cooperation[1]. Whether a given action is perceived as good depends on the action itself, the context, and the social norm used by the observer. A central question is thus what are the requirements of social norms to achieve social cooperation.

The leading eight is a set of highly successful social norms for maintaining cooperation at a high level[2]. By comprehensive enumeration of possible social norms, it has been found that the leading eight are the only ones that can sustain evolutionarily stable cooperation for a broad range of the benefit-to-cost ratios of cooperation. Because of its simplicity and effectiveness, the leading eight have served as a baseline in a wide range of theoretical studies of indirect reciprocity[3-10].

Most of the previous studies on indirect reciprocity, including the leading eight, assume that the reputation of a player is represented by either ‘good’(G) or ‘bad’(B)[11-14]. Whereas the assumption of binary reputation has been widely adopted as a common practice for its simplicity and theoretical tractability, such dichotomy is not always realistic given experimental evidence and our daily experience. Furthermore, it is not always clear how much we can generalize the conclusions obtained from the binary-reputation models to more realistic and complex reputation models because the conclusions may be consequences of the oversimplification.

What would be the universal characteristics that every successful norm shares irrespective of the form of reputation? How should we revise the conclusion learned from the binary-reputation system when the binarity assumption is relaxed? Answering these questions has been a serious challenge because the strategy space expands super-exponentially with the number of possible values of reputations: The number of third-order social norms with k reputations is about as we will discuss in the next section. Although there are several studies that go beyond the binary assumption, only a small subset of the norms were studied in these studies by assuming ordinal relationships between reputations[15-22]. In particular, a continuum formulation of indirect reciprocity allows a perturbative analysis, from which one can derive a condition for linear stability against erroneous disagreement, but it is applicable only to mutants that are sufficiently close to the resident norm[22].

In this study, to bridge the gap between the binary and more general models of reputation, we comprehensively study the norms with a ternary-reputation model under public reputation, in which players are labelled by three types of reputations. The ternary counterparts of the leading eight will be fully identified by comprehensive enumeration of the third-order social norms through state-of-the-art supercomputing. Such a large-scale enumerative approach has also proved useful in studies of direct reciprocity[23-25], and this study is an application of the method to the study of indirect reciprocity. As we will see in the following, both similarity and dissimilarity are found between binary- and the ternary-reputation models, indicating potentially universal features in indirect reciprocity as well as limitations of the binary reputation. We also obtained some counter-intuitive results that would not have been discovered without computational methods.

This paper is organized as follows: In the next section, we present the model definition and an outline of the algorithm to find successful social norms. In the result section, the resulting ternary norms are presented after we briefly review the characteristics of the leading eight. To better comprehend the working mechanisms, we classify the norms by observing their differences. Some counter-intuitive examples are also presented. Finally, in the last section, we discuss the similarities and dissimilarities with respect to the leading eight and summarize this paper.

Model

Description

In this study, we closely follow the previous settings for the leading eight[2] but with ternary reputations. We consider an infinitely large population. In each round, two players are randomly picked to form a donor-recipient pair and play the one-shot donation game. The donor decides to either cooperate (C) or defect (D) considering the recipient’s social reputation as well as his or her own. Cooperation costs the donor a payoff loss , whereas the recipient receives a benefit . On the other hand, defection means that the donor does nothing to the recipient. The donor is always better off by choosing D. Hence the game represents a social dilemma. This process is repeated sufficiently long to reach a stationary state, until which players have engaged in the donation game many times with different opponents.

We assume public reputation, by which we mean that everyone in a population shares the same assignment rule and thus assigns the same reputation to each player. In other words, any reputation assigned to an individual is shared by all members of the population in complete agreement. We have three labels for representing reputations, i.e., G (good), N (neutral), and B (bad). However, note that we do not assume any a priori ordinal relationships among them so that N may be interpreted as worse than B, for instance.

A social norm is comprised of an assessment rule and an action rule, and both the rules are assumed to be deterministic: Concerning the assignment part, it means that a donor’s new reputation is a function of the donor’s reputation, the recipient’s reputation, and the donor’s action. In other words, an assignment rule is represented by a map , where are reputations of the donor and the recipient, respectively, and is the donor’s action. Likewise, an action rule is represented by a map , where X and Y are reputations of the donor and the recipient, respectively, and A is the prescribed action. In the following, the combination of assignment and action rules is denoted as a norm.

When k reputations are available, the number of norms is approximately : To define an assignment rule , Z must be determined out of k possibilities for each combination of (X, Y, A). Thus the number of assignment rules is . Similarly, to define an action rule , one has to determine for each combination of (X, Y), yielding action rules. Therefore, the total number of social norms is . Note that we have counted all the norms even if some of them are equivalent with respect to the permutation of k types of reputation. For example, if we swap G and B everywhere, all essential predictions derived from the model will remain unchanged because reputations are mere labels to distinguish k social states. Taking this permutation into account, the number of social norms is described by . However, this is a rough estimate because the norms that are symmetric under permutation would not be counted k! times[2].

Both implementation error and assignment error are included in our calculation: With probability , a player defects when the prescribed action is cooperation. The opposite error does not occur, i.e., a player does not cooperate by mistake when the prescribed action is defection. An assignment error occurs with probability . When an assignment error occurs, the donor is assigned a different reputation from the prescribed one. For instance, either B or N will be assigned to a donor with equal probability () when the assignment rule prescribes G. Following the indirect observation model[2], a misperception will also be shared by the population.

Calculation

We will identify all the norms that form strict Nash equilibria with sufficiently high cooperation rates. Because a strict Nash equilibrium implies an evolutionarily stable strategy (ESS)[26], we call each of them a cooperative ESS or a CESS for short. Because of 18 possible combination of (X, Y, A), we have different assignment rules in total. Taking the permutation symmetry of reputations into account[2], we computationally found that the number of independent assignment rules reduces to 64, 573, 605. For each assignment rule, possible action rules exist. Thus, the number of independent pairs of assignment and action rules amounts to 33, 061, 685, 760. We judge whether a norm S is a CESS by using the following algorithm (see “Methods” for more details): Here, we define a CESS as a norm whose defection level scales as as , that is, the probability of prescribing defection is of the same order as the error rates. Some norms show slower convergence, such as , both in the binary- and the ternary-reputation cases, and we do not include such norms that are fragile against noise. This requirement is consistent with the criteria for finding the leading eight in the binary-reputation case. In the following calculation, we use , where , and . We choose these values such that is sufficiently larger than but smaller than . For the CESS’s found in the following, we numerically confirmed that by calculating the cooperation levels for different values of .

Let , , and denote respective fractions of B, N, and G. Calculate their values in a stationary state, denoted by , , and , respectively, under the assumption that the entire population uses S.

Calculate the cooperation level , which means the probability that a donor cooperates towards a recipient when both are randomly picked from the resident population.

Reject the norm if , where is a threshold for the cooperation level.

Otherwise, calculate the payoff of a mutant with a different action rule from the resident one under the assumption that mutants occupy a sufficiently small fraction.

Repeat the above step for all possible action rules. If the payoff of a resident is higher than that of any possible mutants, S is a CESS.

As mentioned above, the norms generated by permuting G, N, and B are equivalent because we assume no ordinal relations among them. To remove trivial multiplicity, we use the following protocol: First, the reputation with the largest fraction is labelled as G. In other words, we always have and . To be a CESS, therefore, the action rule must achieve high by prescribing C when both the players have reputation G. If this rule is violated by mistake, the donor must get a reputation other than G because, otherwise, the donor would not find any incentive to cooperate. We define the reputation resulting from such defection as B, and the last remaining one as N. The labels assigned by these guidelines are overall consistent with our common sense of ‘good’, ‘neutral’, and ‘bad’, as we will see in the following.

Results

Leading eight in the binary-reputation model

Before showing our results for the ternary-reputation model, let us review the characteristics of the leading eight. We will check whether these are universally shared with successful norms in the ternary-reputation model. By the leading eight, we mean eight norms that qualify as CESS’s in the binary-reputation model, and they are characterized by the following four properties[12]: With the leading eight, the community is mostly occupied by a single type of players (G) who form mutual cooperation, whereas the fraction of B-players is of . When someone defected from cooperation, the population assigns reputation B to the defector to distinguish him or her from cooperators. G-players punish such a B-player by refusing cooperation, and the punishment is justified in the sense that the defection does not hurt their G-reputation. A B-player can obtain G-reputation by donating to a G-player as an apology. The prescriptions of the leading eight are summarized in Table 1.

Table 1

Prescriptions that are commonly shared by the leading eight. The asterisk () is a wildcard, meaning that it can be any of G and B.

donor	recipient	prescribed action A	reputation for A	reputation for ¬A
G	G	C	G	B
G	B	D	G	∗
B	G	C	G	B
B	B	†	∗	∗

The left two columns show reputations, and the third column is the action A prescribed by the action rule. The fourth column indicates the reputation assigned to the donor who executed the action A, and the last column shows the reputation resulting from the other action . The dagger () means that the action is either C or D depending on the assignment rule, so it is C if and only if and .

Maintenance of cooperation: and .

Identification of defectors: .

Punishment and justification of punishment: and .

Apology and forgiveness: and .

To describe a norm concisely, we hereafter use a notation composed of five characters separated by semi-colons such as . The first two characters denote the reputations of a donor and a recipient, respectively. In this example, the donor’s reputation is G, and the recipient’s reputation is B. The third character denotes the prescribed action, and the fourth character means the reputation that the donor obtains by following the prescription. Finally, the last character denotes the donor’s new reputation when choosing the opposite action. The above example is thus interpreted as follows: “When a G-donor meets a B-recipient, the donor should defect. He or she gets G if following the prescription, and N otherwise.” In addition, square brackets and a wildcard are used to indicate a set of prescriptions. For instance,  means a set of prescriptions, according to which a G-donor should defect against a B-recipient. By defecting, the donor earns either N or G. Otherwise, the donor’s reputation becomes B. By this notation, the leading eight can be characterized by the following prescriptions: In a stationary state, the fraction of B-players and the defection level scale as . This is because players get B-reputation only by implementation or assignment errors, whereas a B-player can almost always recover reputation by meeting a G-player. One can obtain the scaling of from the master equation of near the stationary state () in the following form:where the first term on the right-hand side means the rate of change from B to G represented by Eq. (1c), and the last term represents the opposite caused by error. We note that all the wildcards in Table 1 are prescribed for the events that happen with probability smaller than . For instance, a B-donor meets another B-recipient with probability . However, such an event is so rare that the prescriptions for these events remain arbitrary within the leading eight. In other words, we can understand the working mechanism of the leading eight by investigating the events that occur with probability .

Here, it is worth pointing out that the leading eight are the only CESS’s in the binary-reputation model. If any of the prescriptions in Eq. (1) is missing, the norm is no longer a CESS. For instance, without the justification of punishment [Eq. (1b)], the norm is essentially equivalent to Image Scoring, which cannot sustain stable cooperation because those who punish a B-player also lose good reputation, making greater than .

CESS’s in the ternary-reputation model

We exhaustively enumerated all the norms to find CESS’s in the ternary model. The number of CESS’s is shown as a function of b/c in Fig. 1. As shown in this figure, the number tends to increase in a stepwise fashion with b/c, indicating the existence of norms that qualify as CESS’s only for a certain range of b/c. Let us define the “core” set as norms that are evolutionarily stable within a reasonable range of b/c, say, [1.1, 10]. In other words, the core set is the common subset of the discovered CESS’s. Of course, even the norms in the core set may be evolutionarily unstable if b/c is extremely large or close to unity, but such edge cases were excluded from consideration. Our core set contained 2, 067, 861 CESS’s in total, and we examined this set. Note that the size of the core set is smaller than the number of CESS’s for the lowest b/c. Furthermore, the number of CESS’s does not increase monotonically as b/c grows in Fig. 1. Such behaviour implies the existence of nontrivial social norms that are CESS’s for a certain value of b/c but not when b/c takes a higher value. For example, if a player may lose good reputation by punishing an ill-reputed player, it could be better to overlook him or her than to inflict costly punishment as long as b/c is high enough.

Figure 1

Number of the CESS’s for various values of b/c when and . CESS’s are calculated for . The horizontal dashed line indicates the number of the core set, which is defined as the common subset of CESS’s for these values of b/c.

Table 2

Examples of C1 strategies.

Type	Prescriptions			(hB∗,hN∗,hG∗)	p_c
C1-P1-R1	BB : DG : B	BN : DN : B	BG : CG : N	(0.0015, 0.0005, 0.9980)	0.9980
	NB : DB : N	NN : DN : B	NG : CG : B
	GB : DG : N	GN : DG : N	GG : CG : B
C1-P1-R2	BB : DN : N	BN : DG : G	BG : CN : B	(0.0015, 0.0020, 0.9965)	0.9945
	NB : DG : G	NN : CN : B	NG : DG : N
	GB : DG : N	GN : DG : B	GG : CG : B
C1-P2-R1	BB : DG : G	BN : DB : B	BG : CG : B	(0.0015, 0.0020, 0.9965)	0.9945
	NB : DG : N	NN : DB : B	NG : DG : N
	GB : DN : N	GN : DG : G	GG : CG : B
C1-P2-R2	BB : DG : B	BN : DG : N	BG : DN : N	(0.0015, 0.0035, 0.9950)	0.9936
	NB : DN : N	NN : DG : G	NG : CG : N
	GB : DN : N	GN : DG : B	GG : CG : B

The stationary-state fraction of players and the cooperation level for are shown together with their prescriptions.

Table 4

Examples of C3 strategies.

Type	Prescriptions			(hB∗,hN∗,hG∗)	p_c
C3-P1-R1	BB : DN : N	BN : CG : B	BG : CG : B	(0.0015, 0.4985, 0.5000)	0.9985
	NB : DN : G	NN : CN : B	NG : CG : B
	GB : DG : G	GN : CG : B	GG : CN : B
C3-P1-R2	BB : DG : N	BN : CG : B	BG : DB : B	(0.0030, 0.4978, 0.4993)	0.9955
	NB : DG : B	NN : CG : B	NG : CN : B
	GB : DN : B	GN : CN : B	GG : CG : B
C3-P2-R1	BB : DG : B	BN : CG : B	BG : CN : B	(0.0030, 0.4981, 0.4989)	0.9970
	NB : DG : B	NN : CG : B	NG : CN : B
	GB : DB : B	GN : CG : B	GG : CN : B
C3-P2-R2	BB : DG : B	BN : DB : B	BG : CG : B	(0.0061, 0.3784, 0.6155)	0.9916
	NB : DB : B	NN : CG : B	NG : CG : B
	GB : DG : G	GN : CG : B	GG : CN : B

The stationary-state fraction of players and the cooperation level for are shown together with their prescriptions.

Figure 2a shows the distributions of and for the norms in the core set. The plot of has been omitted because . The figure shows for all the cases, and we numerically verified that for . However, whereas most norms have similarly to the leading eight, we found a small but non-negligible amount of norms for which is significantly smaller than unity, indicating the existence of CESS’s having different working mechanisms from those of the leading eight.

Figure 2

(a) Frequency of the stationary-state fractions and , respectively, for the core set. The vertical axis is on a logarithmic scale. (b) Scaling relations between and . The norms used in this plot are taken from the first rows of Tables 2–4. The dashed lines have power-law exponents 0, 1/2, and 1, respectively.

To understand CESS’s systematically, we first classified them according to how much G-players exist in the stationary state. For the leading eight, the majority of players have reputation G, i.e. and , and mutual cooperation is formed by these G-players. For some of the ternary strategies, on the other hand, not only G but N may occupy a significant fraction of the population as shown in Fig. 2a. Depending on the scaling behaviours of as , we found that the CESS’s in the core set are classified into the following three types:Figure 2b shows examples of the scaling relations between and for three norms, one in each class. In this way, despite the considerable differences at the prescription level, the vast number of CESS’s can be categorized into three well-defined classes unambiguously.

Type C1: Norms with .

Type C2: Norms with .

Type C3: Norms with .

We confirmed that all the core CESS’s, as in the leading eight, commonly have mechanisms to punish defectors and to recover their reputations from erroneous actions. However, we also observed a couple of variants in their ways of punishment and recovery. After classifying the norms into the above three types, we further classified them according to how players conducted punishment and recovery, which we call punishment and recovery patterns.

What do we mean by punishment patterns? With the leading eight, the majority G-players punish B-players by defecting against them, and those who inflicted punishment keep G-reputation after their punishment. Namely, the punishment is justified. However, this is not always the case for some of the ternary CESS’s, under which a punishing player’s reputation does change. We thus classify the norms into those with full justification (P1) and those with partial justification (P2).

Likewise, a recovery pattern means the way that B-players recover their reputations. With the CESS’s, a B-player can restore reputation by making an apology. In the leading-eight community, a B-player can immediately return to G after cooperating with a G-player, and such norms that allow instantaneous recovery are labeled as R1. However, R1 is not the unique recovery pattern in the case of the ternary reputation because some norms allow B-players to only gradually recover their reputation, which we call R2.

In summary, the CESS’s are grouped into three classes and 12 subclasses according to the taxonomy shown in Fig. 3. We will see the details in the following.

Figure 3

The taxonomy of the CESS’s. They are classified into C1, C2, and C3 types according to the scaling of . Each class is further categorized into four subclasses. Norms that fully (partially) justify punishment are labeled as P1 (P2). They are also categorized according to whether B-players are allowed to recover their reputation instantaneously (R1) or gradually (R2). The leading eight correspond to C1P1R1 in which N is totally irrelevant, or to C3P1R1 when G and N merge into a single reputation.

Details of each type

Type C1

The first class (C1) is the most common type. It contains more than two million norms, which comprises about of the core CESS’s. With this class of norms, the majority of the players have reputation G and form mutual cooperation. In other words, the norms prescribe   in common. The master equations near the stationary state are approximated in the following forms:which mean that players with N or B quickly change reputation by meeting G-players, the majority of the population (). If , we can see from Eqs. (3) and (4) that both and will decrease exponentially as time goes by. In the stationary state, the population will thus end up with , sharing a high degree of similarity to the leading eight. Nevertheless, we find some distinctions from the leading eight in the punishment and recovery patterns. Example norms in each subclass are shown in Table 2.

Let us first look at two different patterns, depending on which reputation is assigned to a punishing player:As we have seen in the binary-reputation model, a punishing G-player must not get B-reputation to keep the cooperation level high. Thus, the above two are the only possibilities. Class P1 works similarly to the leading eight. Namely, the punishing behaviour is fully justified, and a G-player can maintain the reputation. On the other hand, P2 is a novel class that has not been reported before. Under a P2 norm, a punishing G-player cannot maintain his or her original reputation but gets N. Their punishment is not always justified because the resulting N-players are also punished by G-players under some of the P2 norms. Nevertheless, the fractions of B- and N-players remain because of the prescription which is commonly found in P2. In other words, actions against N-players are always justified in P2.

Type P1: Norms with ,

Type P2: Norms with .

C1 norms are also classified according to recovery patterns as well:R1 is the most basic type, similar to the leading eight. R2 is unique to the ternary model: It takes two steps for a B-player to improve reputation to G. During the recovery process, a player needs to cooperate with G-players at least once to ensure that defection does not pay.

Type R1: Norms with ,

Type R2: Norms with and .

Figure 4 represents state transition for the C1 norms in Table 2. Transitions in the stationary state [Eq. (11)] are depicted as weighted edges. The corresponding graph for the leading eight is also presented for comparison: The thick blue self-loop at G indicates that the majority of the players have G with a high level of cooperation. When an implementation error happens, the state moves to B as indicated by the dashed edge . The other self-loop is , which means justified punishment inflicted by G-players. The remaining edge, BG : CG, corresponds to the recovery of reputation.

Figure 4

Graphical representation of C1 norms. In each node, the label of the corresponding reputation and its stationary-state fraction are shown. An edge  means transition from X to Z, where X and Y are respective reputations of the donor and the recipient, A is the action, and Z is the new reputation assigned to the donor. Only edges with weights are shown for the sake of visibility. The edges are depicted in blue (red) when the action is C (D), and their widths indicate the amounts of the probability flow. The dashed edges indicate the moves caused by implementation error. The graph for the leading eight is also shown for comparison.

The topology of the graph for the C1P1R1 norm is identical to that for the leading eight except for the unused node N, indicating that their working mechanisms are essentially the same. P2 norms have a directed edge from G to N in common, which corresponds to  instead of the self-loop GB : DG in P1 norms, and it implies reputation change caused by punishment. Whereas punishment against B-players is not justified, the action against N-players is justified as seen in the self-loop GN : DG or GN : CG, which is required to keep high. We can also find difference between R1 and R2 in that R2 norms require two steps to reach G from B, as seen from a path instead of a direct edge .

The numbers of CESS’s in each class and their percentages to the entire core set are the following: C1P1R1 has 1, 057, 956 norms (), C1P1R2 has 395, 829 norms (), C1P2R1 has 281, 586 norms (), and C1P2R2 has 269, 790 norms ().

Type C2

The second class contains 51, 363 norms, and this number is much smaller than C1. In this class, we have , which is small but significantly greater than . Near the stationary state, the leading terms of the master equations are written as follows:The dynamics of are the same as C1: It decreases exponentially fast, and its stationary value is of . As for , if is negligibly small, is solved by . Therefore, although approaches , this process is slow with a diverging time scale. In the stationary state, Eq. (5) implies , which is significantly greater than . Differently from C1, Eq. (5) shows that a G-player’s encounter with an N-player does not decrease the total number of N-players because their reputations are either preserved or swapped by this event. The fraction decreases mainly when two N-players meet, making the difference from C1.

Cooperation is prescribed not only between two G-players but between a G-player and a N-player, i.e., . When two N-players meet, however, they may defect under some of the C2 norms because such events occur with probability of and do not decrease the cooperation level significantly.

As in C1, C2 norms can be further classified according to punishment patterns as follows:where P1 means full justification, whereas P2 does partial one. We can also classify C2 norms according to recovery patterns:where R1 and R2 correspond to the instantaneous and gradual recovery processes, respectively.

Type P1: Norms with

Type P2: Norms with ,

Type R1: Norms with

Type R2: Norms with ,

Examples of C2 norms are shown in Table 3, and their state-transition graphs are depicted in Fig. 5. The difference between these subclasses is clear: The graphs for P1 have self-edges   in common, whereas those for P2 have edges from G to N (). Similarly, the graphs for R1 have edges from B to G (BG : CG) whereas those for R2 have a path .

Table 3

Examples of C2 norms. The stationary-state fraction of players and the cooperation level for are shown together with their prescriptions.

Type	Prescriptions			(hB∗,hN∗,hG∗)	p_c
C2-P1-R1	BB : DN : N	BN : CN : B	BG : CG : B	(0.0015, 0.0212, 0.9773)	0.9985
	NB : DG : G	NN : CG : B	NG : CG : B
	GB : DG : N	GN : CN : B	GG : CG : B
C2-P1-R2	BB : DN : B	BN : DG : N	BG : CN : B	(0.0015, 0.0420, 0.9565)	0.9967
	NB : CG : B	NN : DG : G	NG : CN : B
	GB : DG : N	GN : CG : B	GG : CG : B
C2-P2-R1	BB : DG : N	BN : DN : N	BG : CG : B	(0.0015, 0.0427, 0.9558)	0.9966
	NB : DG : B	NN : DG : N	NG : CG : B
	GB : DN : B	GN : CN : B	GG : CG : B
C2-P2-R2	BB : CG : B	BN : DN : B	BG : CN : B	(0.0016, 0.0579, 0.9406)	0.9983
	NB : DB : B	NN : CG : B	NG : CN : B
	GB : DN : G	GN : CG : B	GG : CG : B

Figure 5

The state transitions for the C2 norms shown in Table 3. The notations of the graphs are the same as those in Fig. 4.

The numbers of CESS’s in each class and their percentages to the entire core set are the following: C2P1R1 has 13, 275 norms (), C2P1R2 has 15, 543 norms (), C2P2R1 has 10, 659 norms (), and C2P2R2 has 11, 886 norms ().

Type C3

Finally, we categorize 11, 337 norms into the third class (C3). Differently from C1 and C2, the stationary fraction remains finite even when . As a result, most players end up with either G or N. Mutual cooperation is formed between G- and N-players, whereas those who defect receive reputation B. The fraction of B-players is a small quantity of because one can easily escape from B-reputation by meeting G- or N-players. We also obtained asymptotic dynamics of and as shown in “Methods”.

Similarly to the other classes, a finer classification of C3 norms according to their punishment patterns can be defined as follows:As before, punishment is fully justified in P1. P2 means partial justification in the sense that only punishment inflicted by G-players is justified, but not the one by N-players, or the other way around. C3 norms are classified according to recovery patterns as well:R1 is similar to the leading eight, and R2 is unique to the ternary-reputation model because B-players cooperate only with either G- or N-players and recover reputation. Thus, on average, more than one step is required to return to the original reputation state if it is lost by mistake.

Type P1: Norms with and ,

Type P2: Norms with () or ().

Type R1: Norms with   and ,

Type R2: Norms with () or ().

Examples of C3 norms as well as their state-transition graphs are shown in Table 3 and in Fig. 6. The graph for C3P1R1 is equivalent to that for the leading eight if G and N merge into one. Whereas P1 norms have no solid edges to B, P2 norms have a solid edge to B either from G or N, indicating that punishment is not always justified in P2 norms. No self-loop exists around B in the R1 graphs whereas those for the R2 graphs have a self-loop BN : DB or BG : DB, indicating that B-players cannot always escape from B.

Figure 6

The state transitions for the C3 norms shown in Table 4. The notations of the graphs are the same as those in Fig. 4.

The numbers of CESS’s in each class and their percentages to the entire core set are the following: C3P1R1 has 5, 199 norms (), C3P1R2 has 4, 413 norms (), C3P2R1 has 1, 593 norms (), and C3P2R2 has 132 norms ().

Counter-intuitive examples

Here we show a couple of norms with interesting differences from the leading eight and discuss why they nevertheless qualify as CESS’s with the ternary reputation.

The first example is “unfair” punishment, which is observed from the C1P2R1 norm in Table 2 and Fig. 4. Under this norm, a punishing player does not maintain G but gets N, and the player is punished by a G-player in the subsequent round. In other words, a player who initially had G-reputation gets unfairly punished after the unfortunate encounter with a B-player although the player has accurately followed the prescription. This unfairness is never observed in the leading eight because B cannot be assigned to a punishing player to keep . However, when the reputation is not binary, a norm without the full justification can be a CESS because both and can be of , justifying the action against N-players (.

In the second example, we see peculiar behaviour of “making an apology by defecting.” Such behaviour is observed from the C1P1R2 example in Table 2 and Fig. 4. A player needs two steps () to return to G once he or she gets B. First, a B-player must cooperate with a G-player to become N. Then, the N-player must defect, not cooperate, against a G-player to become G, which goes against our common sense. This counter-intuitive apology is not possible with the binary reputation because such a norm would allow a constantly defecting player to be better off than the rest of the population. However, when more than two kinds of reputation are available, this is not the case. A norm can be a CESS as long as cooperation is prescribed at least once in the course of apology, but not necessarily twice or more, because a single move of cooperation is enough to compensate for the defection.

Another interesting behaviour is found in C2P2R1 norm in Table 3 and in Fig. 5, where “inequality” among cooperators spontaneously emerges. As shown above, C2 norms are characterized by the fact that most players have G except a small fraction of N players in the stationary state. Those players form mutual cooperation, but N-players defect against each other under some C2 norms. Such an event occurs with probability , thus negligible at the societal level, but it makes a significant difference from a player’s perspective. That is, although G-players almost surely receive cooperation from the community, N-players do not benefit from N-players, yielding the drop of the individual cooperation level by . This inequality becomes significant as grows. For instance, when , we see that , and more than of the population suffers such a loss. Again, the drop of the cooperation level is still acceptable at the society level, and this C2 norm qualifies as a CESS.

Summary and discussion

Although reputation in our real life is not always distinguished between good and bad, most previous works have accepted an idealized assumption of binary reputation. Some researchers have attempted to go beyond the binary reputation[15-21], and the motivation behind the ternary reputation in Tanabe et al.[15] is close to ours. However, they studied only second-order norms and assumed an ordinal relationship among reputations to limit the number of norms to 512. By considering the third-order assessment rules, this study naturally takes into account the Self strategy[27] as well as all the second-order norms (see “Methods”). One may extend the binary system by representing reputation as integer values[16,19,20]. The other extreme is to regard reputation as a continuous variable. This approach makes it possible to use analytic tools, but it has its own limitations because we can only examine a small neighborhood of the existing cooperative norm[22]. Little is known about the consequences of these simplifying assumptions: For example, some lessons from indirect reciprocity might be due to the oversimplification. We should ask what are the fundamental properties that are preserved irrespective of the complexity of the reputation system to sustain cooperation. To address this question, we comprehensively searched for the CESS pairs of assignment and action rules with ternary reputations to compare with the leading eight. From more than thirty billion possibilities, we filtered out “core” CESS norms that constitute the counterpart of the well-known leading eight.

The result shows that the previous conclusions drawn from the binary- and continuous-reputation models do not fully capture the various possibilities of CESS’s. For example, under a certain norm, a player may lose G-reputation even when he or she has obeyed all the prescriptions of the norm, something unimaginable in the leading eight. Another norm requires an ill-reputed player to defect against the community to gain a better reputation. This observation suggests that a population may well achieve cooperation by actively making use of reputations far from G, a possibility that has been ignored by the linear stability analysis for the continuous model. Put differently, the viewpoint of this work is that N does not necessarily mean ‘less bad’ but functions in its own way. Based on this idea, we explored the strategy space in full without imposing any ordinal relationship a priori. Indeed, we have observed cases where N cannot be interpreted as the middle reputation between G and B. For instance, in some of C1P2R1 norms, a G-player who punished a B-player gets N and the N-player can defect against G-players while receiving cooperation from G-players, which implies that N is deemed better than the majority’s reputation G. Such a CESS would not be found if we assumed an ordinal relationships among the three reputations.

It is still instructive to compare our findings with the leading eight. As we have seen above, the leading eight have the following characteristics: (i) Maintenance of cooperation, (ii) Identification of defectors, (iii) Punishment and justification of punishment, (iv) Apology and forgiveness. Overall, these characteristics are shared with the ternary CESS’s, indicating key features universally required for general reputation systems. However, some of the above characteristics are relaxed when we go beyond the binary assumption. First, cooperation is not always maintained among a single type of players but among multiple types, as seen in C2 and C3. Second, partial justification of punishment is allowed: A player who inflicted punishment does not always keep the original reputation, differently from the leading eight. Third, forgiveness may be non-instantaneous: Instead of being forgiven right after cooperating with G as in the leading eight, it may take some steps for a B-player to recover his or her reputation.

In summary, based on the results for the ternary-reputation system, we conjecture that the CESS norms for general reputation models will share the common characteristics in a more relaxed form: We believe these rules serve as useful guiding principles when designing a norm with even more intricate reputations.

Maintenance of cooperation by the majority (but not necessarily all) of the population.

Identification of defectors.

Punishment, followed by partial or full justification.

Apology accompanied by gradual or instantaneous forgiveness.

Finally, we would like to stress that the ternary-reputation model is an interesting system in its own right because the third reputation may provide additional historical information for players. An important future direction is the study of the private reputation in a noisy environment[4,8-10,28-31]. Investigation into a polymorphic population in this context remains an open problem at large, and the same remark applies to stochastic reputation dynamics[15,21]. Because some of the ternary CESS’s have the diverging time scales in the dynamics of (C2 and some C3 norms, see Table 6), these norms could show behaviours that are significantly different from the binary ones.

Table 6

Summary of stationary values of and the asymptotic time evolution near stationarity.

Types	Stationary values	Time dependence
	h_N(μ,t→∞)	h_N(μ→0,t)
C1	O(μ)	exp(-t/τ_N)
C2	O(μ^1/2)	t^-1
C3	1/5+O(μ)	exp(-μt/τ_N)
	1/2+O(μ)	exp(-t/τ_N)
	φ^-2+O(μ)	exp(-t/τ_N)

In every case, the common findings are and . If necessary, a certain time scale of O(1) is denoted by or , which may differ norm by norm, and is the golden ratio.

Methods

Calculation of the stationary-state population

Let be the fraction of players having reputation . By construction, we always have . When an RP pair is given, one can calculate the time evolution of for a short time interval in the error-free limit as follows: Within this interval, we randomly choose a small fraction of players as donors, and we denote their fraction as . The fraction of those players having reputation X is . A recipient is assigned to each donor through random sampling, so the donor meets a recipient with reputation Y with probability . If everyone abides by the given RP pair, we may rewrite the assignment rule as . The inflow of is thus equal to , where is the Kronecker delta, because the donor with X has to interact with the recipient with Y and obtain new reputation Z according to the rule R(X, Y). On the other hand, the outflow is because the donors will have updated reputations other than Z in general. Thus, the time evolution of is given byTaking the limit of , we have the following differential equation:where and after rescaling the unit of time.

In the presence of implementation error, a donor fails to cooperate with probability . In other words, the prescribed action is correctly executed with probability , and the player must defects otherwise. By taking into account the implementation error, is thus redefined asIn the presence of assignment error, the donor does action A and receives correct reputation Z with probability , but Z may be assigned by mistake with probability although the assignment rule does not prescribe Z. Therefore, the probability that the donor obtains Z is given asThus, when both implementation and assignment errors may occur, is redefined asNote that the dynamics preserves becauseAccording to our numerical check, converges to a unique stationary state, , irrespective of the initial condition for most cases. However, in some cases where multiple stationary states coexist, we adopted the one obtained from the initial condition (1/3, 1/3, 1/3), regarding it as the most representative one. For each social norm, we obtain by using the fourth-order Runge-Kutta algorithm, normalizing by each time step.

Calculation of the cooperation level and the payoffs

Cooperation level for a resident species is defined asThe payoff of a resident player is calculated asThen, we calculate the dynamics of the fraction of mutant players for each reputation, , when a small number of mutant players exist in the community. The fraction of mutants having Z reputation is updated aswherewhere and is the action rule of the mutant. We numerically confirmed that converges to a stationary value after an initial transient period.

Using these stationary values, the probability that a mutant cooperate with a resident iswhereas its counterpart isUsing these, the payoff of the mutant is given as

Enumeration of norms

We enumerated all possible combinations of assignment and action rules to find every CESS’s. A supercomputer was used to deal with a large number of possibilities that amounts to . To speed up the calculation, we removed some of the norms that cannot be CESS’s as follows: When an assignment rule R contains a case where the assigned reputation is the same for both actions, D must be prescribed to be an ESS. For instance, when , a G player would have no incentive to cooperate with another G player. In such a case, an action rule prescribing C at (G, G) cannot be an ESS because the defector gains a strictly higher payoff than the resident. We exclude these cases to speed up the computation.

Second-order norms

Although we have focused on the third-order norms, second-order norms are included in the third-order CESS’s as a subset. Under a second-order norm, a new reputation is assigned to a donor, and the prescribed action is independent of the donor’s reputation, and the assignment and action rules are functions of the recipient’s reputation and the conducted action. These norms are thus simpler than third-order ones.

The full list of the second-order CESS’s is shown in Table 5. We represent the norms by using the same notation as those in the main text, but with the first character unspecified (denoted as ) because the second-order norms are independent of the donor’s reputation.

Table 5

List of the second-order norms that are included in the CESS’s.

Prescriptions			Type	Remark
_B:DG:∗	_N:DG:∗	_G:CG:B	C1-P1-R1	♠
_B:DG:∗	_N:CG:B	_G:CG:B	C1-P1-R1	♣
_B:DG:∗	_N:DB:[BN]	_G:CG:B	C1-P1-R1
_B:DN:[BN]	_N:DG:∗	_G:CG:B	C1-P2-R1
_B:DN:∗	_N:CG:B	_G:CG:B	C1-P2-R1	♣
_B:DG:∗	_N:CG:B	_G:CN:B	C3-P1-R1	♣
_B:DG:∗	_N:CN:B	_G:CG:B	C3-P1-R1	♣

The asterisk represents a wildcard, and the square bracket [BN] represents either B or N. Those denoted by are equivalent to SS or SJ when B and N are merged into a single reputation. Those denoted by are equivalent to SS or SJ when N and G are merged into a single reputation.

As shown in the table, there are 18, 9, and 6 second-order norms in C1-P1-R1, C1-P2-R1, and C3-P1-R1 classes, respectively. Most of these are relevant to norms with the binary reputation. The only two second-order norms in the leading eight are Simple Standing (SS) and Stern Judging (SJ), which are denoted as (). As seen in Table 5, a large fraction of the norms, those denoted by or , are equivalent to SS or SJ when two of the reputations are merged into one.

Dynamics of C3 norms

There are different kinds of dynamics of within C3 norms. The fraction of B-players is a small quantity of because one can easily escape from B-reputation by meeting G- or N-players. If we merge G and N into a single reputation, the merged reputation corresponds to G for the leading eight. A radical example is a mechanism found in 2, 139 norms, for which and are almost non-interacting: Just as G-players preserve their reputations through , the same is true for N-players with , and the interaction between G- and N-players cannot change their numbers because the reputations are either preserved or swapped. Their interaction is basically mediated by B-players, originating from error . Therefore, for each of these 2139 norms, the dynamics towards a fixed point becomes frozen as .

For the rest, the coupling between and is more explicit, and the convergence rate is a finite constant independent of . Let us give two representative examples: A common pattern among 2754 norms in C3 is an oscillation between G and N due to  and . In the absence of error, two G-players will change their reputations to N, and vice versa. The master equation for is thus written aswhere we have plugged , considering . The above equation clearly shows in the limit of small .

For other 3, 003 of the C3 norms, the interaction between G and N is more delicate. In addition to the above prescriptions for Eq. (20), they also have  and  in common. Therefore, when an N-player meets a G-player, both will earn G-reputation by choosing C. As a whole, these features lead to the following master equation:whose stationary value is obtained as , where is the golden ratio. The dynamical behaviours of the other C3 norms can be explained in similar ways.

Table 6 summarizes our dynamical characterization of the three classes.