Effect of mutation on genetic differentiation among nonequilibrium populations.

The usefulness of GST and similar measures of genetic differentiation has been questioned repeatedly because of their dependence on the amount of heterozygosity within populations, creating problems when comparing degrees of divergence at loci with different mutation rates. Although the effect of mutation on GST is expected to be small in the early phases of divergence, it is unclear for how long after separation from a common ancestral population that GST is largely unaffected by mutation and by the resulting effect on heterozygosity. We address this question through analysis of the recursion equations for gene identity under the infinite allele model of mutation, and derive conditions describing when the effect of mutation on GST can be ignored under mutation-migration-drift equilibrium conditions and during the preceding transition phase. An important result is that during the transition phase GST is not only affected by mutation, but also by the heterozygosity in the base population from which the subpopulations diverged. The effect of mutation on GST is significant from the very start of the divergence process when initial heterozygosity is low, whereas GST is only weakly affected by mutation in the early phases of differentiation when initial heterozygosity is high. Thus, differentiation following a severe bottleneck is strongly dependent on mutation. The standardized measure of differentiation, G'ST, suggested by Hedrick (2005), may be helpful when comparing amounts of divergence at loci with different mutation rates under steady-state conditions, provided that migration is very low. In many other situations the use of G'ST might be misleading, however, and its application should be exercised with caution.