Mathematical theory of group selection: Structure of group selection in founder populations determined from convexity of the extinction operator.

Genetic analysis for group selection is developed for the case of a biallelic locus (A, a) undergoing group selection of founder populations only. By contrast to R. Levins"E = E(x) models, extinction now depends on genetics at the propagule stage but acts uniformly on larger populations. Biological evidence supports this hypothesis, which also allows mathematical treatment at once simpler and biologically more general than the Fokker-Planck partial differential equation formalism adopted by Levins. It is presently possible to handle cytogenetics of both diploid and haplodiploid type. The model is set up as a quasideterministic recursion in the 5-simplex Sigma(5), collapsing both drift and mendelian selection effects into a single parameter u, which is a Fisher-Kimura-Ohta fixation probability. In the analysis, it is shown that the stability of the fixed points is determined by the convexity of the extinction operator acting on propagules, assumed to be of size 2.THUS, [FORMULA: see text] in which E(1) and E(3) are extinction probabilities for phenotypically uniform founder populations and E(2) is the corresponding probability for founder populations of mixed phenotype. Further parameter regions are defined where fixation of the group-selected gene is globally stable, and this is still possible even when extinction pressure acting on carrying capacity populations becomes weak relative to a fixed mendelian selection strength.