Evolution of social behavior in finite populations: a payoff transformation in general n-player games and its implications.

The evolution of social behavior has been the focus of many theoretical investigations, which typically have assumed infinite populations and specific payoff structures. This paper explores the evolution of social behavior in a finite population using a general n-player game. First, we classify social behaviors in a group of n individuals based on their effects on the actor's and the social partner's payoffs, showing that in general such classification is possible only for a given composition of strategies in the group. Second, we introduce a novel transformation of payoffs in the general n-player game to formulate explicitly the effects of a social behavior on the actor's and the social partners' payoffs. Third, using the transformed payoffs, we derive the conditions for a social behavior to be favored by natural selection in a well-mixed population and in the presence of multilevel selection.