Mutation rate evolution in replicator dynamics.

The mutation rate of an organism is itself evolvable. In stable environments, if faithful replication is costless, theory predicts that mutation rates will evolve to zero. However, positive mutation rates can evolve in novel or fluctuating environments, as analytical and empirical studies have shown. Previous work on this question has focused on environments that fluctuate independently of the evolving population. Here we consider fluctuations that arise from frequency-dependent selection in the evolving population itself. We investigate how the dynamics of competing traits can induce selective pressure on the rates of mutation between these traits. To address this question, we introduce a theoretical framework combining replicator dynamics and adaptive dynamics. We suppose that changes in mutation rates are rare, compared to changes in the traits under direct selection, so that the expected evolutionary trajectories of mutation rates can be obtained from analysis of pairwise competition between strains of different rates. Depending on the nature of frequency-dependent trait dynamics, we demonstrate three possible outcomes of this competition. First, if trait frequencies are at a mutation-selection equilibrium, lower mutation rates can displace higher ones. Second, if trait dynamics converge to a heteroclinic cycle-arising, for example, from "rock-paper-scissors" interactions-mutator strains succeed against non-mutators. Third, in cases where selection alone maintains all traits at positive frequencies, zero and nonzero mutation rates can coexist indefinitely. Our second result suggests that relatively high mutation rates may be observed for traits subject to cyclical frequency-dependent dynamics.