Negotiation and appeasement can be more effective drivers of sociality than kin selection.

Two alternative frameworks explain the evolution of cooperation in the face of conflicting interests. Conflicts can be alleviated by kinship, the alignment of interests by virtue of shared genes, or by negotiation strategies, allowing mutually beneficial trading of services or commodities. Although negotiation often occurs in kin-structured populations, the interplay of kin- and negotiation-based mechanisms in the evolution of cooperation remains an unresolved issue. Inspired by the biology of a cooperatively breeding fish, we developed an individual-based simulation model to study the evolution of negotiation-based cooperation in relation to different levels of genetic relatedness. We show that the evolution of negotiation strategies leads to an equilibrium where subordinates appease dominants by conditional cooperation, resulting in high levels of help and low levels of aggression. This negotiation-based equilibrium can be reached both in the absence of relatedness and in a kin-structured population. However, when relatedness is high, evolution often ends up in an alternative equilibrium where subordinates help their kin unconditionally. The level of help at this kin-selected equilibrium is considerably lower than at the negotiation-based equilibrium, and it corresponds to a level reached when responsiveness is prevented from evolving in the simulations. A mathematical invasion analysis reveals that, quite generally, the alignment of payoffs due to the relatedness of interaction partners tends to impede selection for harsh but effective punishment of defectors. Hence kin structure will often hamper rather than facilitate the evolution of productive cooperation.