Evolution of vertical and oblique transmission under fluctuating selection.

The evolution and maintenance of social learning, in competition with individual learning, under fluctuating selection have been well-studied in the theory of cultural evolution. Here, we study competition between vertical and oblique cultural transmission of a dichotomous phenotype under constant, periodically cycling, and randomly fluctuating selection. Conditions are derived for the existence of a stable polymorphism in a periodically cycling selection regime. Under such a selection regime, the fate of a genetic modifier of the rate of vertical transmission depends on the length of the cycle and the strength of selection. In general, the evolutionarily stable rate of vertical transmission differs markedly from the rate that maximizes the geometric mean fitness of the population. The evolution of rules of transmission has dramatically different dynamics from the more frequently studied modifiers of recombination, mutation, or migration.

Cavalli-Sforza and Feldman (1) distinguished two forms of nonparental phenotypic transmission in the context of cultural evolution. Horizontal transmission occurs when a trait is passed between members of the same generation and is analogous to transmission of an infectious agent. Oblique transmission to offspring is from nonparental members of the parental generation. Evolution under either of these is expected to be more rapid than under purely vertical (i.e., parent-to-offspring) transmission (2, 3).

Oblique transmission occurs via some mechanism of social learning, which may include imitation or active teaching. There has been an interesting debate over the past 30 y concerning the conditions under which social learning would have an advantage over individual learning or vertical (including genetic) transmission. This debate is usually couched in terms of the mode and tempo of environmental fluctuations that would affect fitness and hence, evolution (4–11). Mathematical analyses of models of competition between individual and social learning have generally shown that social learning has an advantage when the environment does not fluctuate too frequently. However, when environmental changes are very frequent, individual learning is favored, while innate (genetic) determination of the trait does best when periods between environmental change are long on average.

In some situations, oblique transmission of biological material is possible. In bacteria, phenotypes might be determined by heritable mobile genetic elements, such as phages (12), plasmids (13), integrons (14), and transposons (15). Similarly, some phenotypes are determined by genes that are commonly converted by uptake of foreign DNA (i.e., transformation) (16). In these cases, inheritance of a phenotype may combine vertical transmission from the parent cell and oblique transmission from other cells.

In some animals, transmission of microbes may occur during sharing or manipulation of food or other consumable resources during a social interaction. Although transmission of the microbiome in humans is likely to be mostly vertical (17), in other organisms, there is multigenerational food sharing, during which symbionts from the parental cohort may be transmitted obliquely to younger individuals (18). In such cases, fluctuations in the resource type or availability may have fitness effects that depend on features of the transmitted microbiome. This ecological perspective on community transmission is stressed by van Opstal and Bordenstein (19), who emphasize the “need to consider the relative roles of vertical and horizontal transmission of microbial communities.”

Another perspective on the evolutionary consequences of fluctuating environments (and as a result, fluctuating selection) derives from the phenomenon of phenotypic switching (20–25). In these studies, mutation causes the organism to switch phenotypes (usually treated as haploid genotypes), and the problem has usually been couched in terms of the optimal rate of mutation in models where the phenotypic fitness fluctuates over time. These models did not include social learning, and the evolution was regarded as a mode of bet-hedging against future environmental change. Optimal (that is, evolutionarily stable) mutation rates depend on many features of the fluctuations (for example, degree of fitness symmetry, strength of selection, and variance in the period of fluctuation) (23).

In a recent analysis of evolution under fluctuating selection, Xue and Leibler (26) allowed an organism to absorb information about the distribution of possible environments by learning the phenotypes of members of its parental lineage from previous generations. They describe this as “positive feedback that enhances the probability that the offspring expresses the same phenotype as the parent” (26). In this formulation, there was “reinforcement of the parent phenotype” in an offspring, such as might occur through epigenetic inheritance. Although their analysis was not couched in terms of oblique and vertical transmission, as defined by Cavalli-Sforza and Feldman (1), we have been stimulated by their analysis to develop a model in which oblique transmission, at a rate dependent on the trait frequency in the parental generation, occurs in addition to classical vertical transmission. We then ask how fluctuations in selection interact with the rate of oblique transmission to affect evolutionary dynamics and how the rate of oblique transmission itself might evolve.

In our formulation, both the parental phenotype and the distribution of phenotypes in the whole population contribute to an offspring’s phenotype. Using conventional modifier theory (27), we show that, in a symmetric cyclic selection regime with cycles of periods 1 or 2, an allele reducing the rate of vertical transmission is expected to increase in frequency when rare and in so doing, to increase the mean fitness of the population. However, for cycles of greater length or period asymmetry, interesting nonmonotonicities emerge both in the uninvadable rate of vertical transmission and in the rate that maximizes the geometric time average of the population mean fitness, which we will refer to as the “geometric mean fitness.” We develop the models in very large populations with cyclic selection and with random fitness and also in the case where drift occurs via sampling from generation to generation in a finite population.

Model

Consider an infinite population whose members are characterized by their phenotype , which can be of two types, or , with associated frequencies and , respectively. We follow the evolution of over discrete nonoverlapping generations. In each generation, individuals are subject to selection, where the fitnesses of and are and , respectively.

An offspring inherits its phenotype from its parent via vertical transmission with probability and from a random individual in the parental population via oblique transmission with probability . Therefore, given that the parent phenotype is and assuming uniparental inheritance (28), the conditional probability that the phenotype of the offspring is iswhere in the parent’s generation before selection.

Therefore, the frequency of phenotype after one generation is given by the recursion equationwhere is the mean fitness, namelyEq. can be rewritten as

In what follows, we explore the evolution of the recursion Eq. , namely the equilibria and their stability properties, in the cases of constant environments and changing environments.

Constant Environment.

When the environment is constant, the fitness parameters and do not change between generations, and we have the following result.

Result 1.

If and both and are positive with , then fixation in the phenotype is globally stable when .

Proof:

If we rewrite Eq. as , it can be seen that , and for and ,Hence, as and , both fixations in or in ( for fixation in and for fixation in ) are equilibrium points of Eq. . Moreover, if is the value of at the th generation , from Eqs. and , we have, for any and all ,and since or is the only equilibrium point, we haveTherefore, fixation of the favored phenotype is globally stable.

Periodically Changing Environment.

Suppose the environment changes periodically, such that the favored phenotype changes after a fixed number of generations. Simple examples are , in which the favored phenotype switches every generation, or , where every two generations, in which selection favors , are followed by a single generation, in which selection favors . In general, denotes a selection regime, in which the period is of generations, with generations favoring phenotype followed by generations favoring .

Let be the fitness of the favored phenotype and be that of the other phenotype, where . Rewrite Eq. as when is favored and when is favored. Then,If denotes the frequency of the phenotype at generation starting with initially, then as we are interested in the values of for with at the end of complete periods, we can writewhere is the composed functionClearly, since and , both fixations in or in are equilibrium points. An interesting question is when these fixations are locally stable. We concentrate on , the fixation of the phenotype . As for generations and for generations, the linear approximation of “near” isHence, the local stability of is determined by the product ; is locally stable if this product is less than one and unstable if it is larger than one.

From Eq. , we haveWe start with the case .

Result 2.

If and with , fixation of is unstable.

The local stability of , the fixation of , is determined by the productObserve thatSince and , fixation on is unstable.

Since and the above result also holds when , there is total symmetry between the two fixations in and , and fixation in is also unstable. Thus, neither phenotype can be lost, and there is a protected polymorphism (29).

For general , , the condition for local stability of fixation in isand that of B is Therefore, following ,and it is impossible that both fixations are stable. Furthermore, since by Eq. 12, > and 0>f(0)>1 when 0>w >W, by choosing k and l appropriately, fixation on A or fixation on B (but not both) can be stable. In addition, we can have both fixations unstable giving the following result.

Result 3.

With and in the case of periodically changing environments, both fixations may be unstable, producing a protected polymorphism.

Let and , and our assumption entails and . Following Eq. , fixation in is unstable if , and similarly, fixation in is unstable if . Therefore, both fixations are unstable ifor equivalently, ifNow the inequalities of Eq. hold if and only if

These inequalities are consistent if and only if (i.e., ), which is true by Eq. .

The linear approximation of near (Eq. ) does not depend on the order in which phenotypes and are favored within a cycle of generations. Therefore, the local stability properties of the two fixations depend only on the fact that, in a cycle of generations, is favored times and is favored times and not their order in the cycle. When neither fixation in nor that in are stable, there is a protected polymorphism, and we expect to have one or more polymorphic equilibria. Fig. S1 illustrates the relationship between and that gives polymorphism of and , or fixation, for different values of .

For the simple case of periodically changing environment, we have the following.

Result 4.

In the case with and , the two fixations are unstable, and there exists a unique stable polymorphism.

Let be the initial frequency of and be its frequency after one cycle of selection. Then, , where, by Eq. ,The equilibrium equation is , which reduces to a fourth degree polynomial equation in . Since the fixations in and are equilibria corresponding to the solutions and , the other equilibria correspond to solutions of a quadratic equation , with andAs and , we haveAlso, as and , the quadratic equation has two real roots, one negative and one positive , satisfying . The latter determines a unique polymorphism. Let . Then,Also,andFrom our assumptions on , , and , we have for . Observe that the numerator of is linear in ; its value when is , and when , it isHence, for all , and is positive when . Thus, is monotone increasing for ; is monotone increasing for , and is monotone increasing for . Starting from any initial value , we have as . Fig. S2 illustrates how the frequency of changes over time in the regime of cycling selection.

For more general cyclic fitness regimes, the polynomial that gives the equilibria is of higher order, and it is conceivable that more than one stable polymorphism could exist for given values of , , and . We have been able to show that, when neither fixation in nor fixation in are stable, in the case, this cannot occur. In fact, we have the following.

Result 5.

In the selection regimes, if the fixations in and are locally unstable, a single stable polymorphic equilibrium exists.

The proof of is in . Fig. S3 shows the stable equilibrium frequencies as a function of , , and in the regime. For selection regimes from to , Fig. S4 illustrates the convergence to a single stable polymorphism.

We have not been able to prove that, for selection regimes with , there is a single stable polymorphic equilibrium when the two fixations are unstable. However, the numerical examples in Fig. S1 for and in Fig. 1 and Fig. S5 for the special case all exhibit a single stable polymorphic equilibrium when fixations in and are unstable. These numerical results suggest that, for and , the high-order equilibrium polynomial has only a single root corresponding to a globally stable polymorphism. Fig. S6 shows that this is the case for the regime.

Fig. 1.

Stable frequency of phenotype A and geometric mean fitness in selection regime as a function of the vertical transmission rate and the fitness of the disfavored phenotype . (A) Stable frequency of phenotype at the end of each three-generation cycle. (B) Geometric average of the stable population mean fitness over the three-generation cycle: . Gray contour lines join and combinations that result in the same stable value. In all cases, W = 1.

Randomly Changing Environment.

We now consider the case where the environment changes according to a stochastic process. Without loss of generality, assume that the fitness parameters at generation () are for phenotype and for phenotype , where the random variables for are independent and identically distributed. Also assume that there are positive constants and , such that .

Corresponding to Eq. , with and , the recursion equation isAs for is a sequence of random variables, the notion of stability of the two fixation states needs clarification. Following Karlin and Lieberman (30) and Karlin and Liberman (31), we make the following definition.

Definition:

“Stochastic local stability” is defined as follows. A constant equilibrium state is said to be stochastically locally stable if, for any , there exists a , such that impliesThus, is stochastically locally stable if for any initial sufficiently near the process converges to with high probability.

In our case, there are two constant equilibria and corresponding to fixation in and , respectively. We can characterize the stochastic local stability of these fixations with the following results, and proofs are in .

Result 6.

Suppose . Then, , the fixation of phenotype , is not stochastically locally stable. In fact, .

Result 7.

Suppose . Then, , the fixation of phenotype , is stochastically locally stable. In particular, if , is stochastically locally stable.

Using the general notation for the fitness parameters and , the stochastic local stability of fixation in is determined by the sign of , and that of fixation in is determined by the sign of . For example, if the sign of the first is negative, fixation in is stochastically locally stable, and when it is positive, with probability of one, convergence to fixation in does not occur. It is also true that, if , then fixation of is stochastically locally stable. Following Eq. , for all realizations of and ,Therefore, as in the case of periodically changing environments , it is impossible that both fixations are simultaneously stochastically locally stable. It is possible, however, that neither fixation is stochastically locally stable, in which case, we expect the population to converge to a polymorphic distribution. Fig. 2 illustrates how the properties of in Eq. affect the frequency of phenotype and in particular, the stochastic local stability of fixation in phenotype . Fig. 3 shows the dynamics of the frequency of in a case where and are identically distributed and independent; in this case, the expectation of the stationary distribution is , and its variance increases as increases.

Fig. 2.

Stochastic local stability. The figure shows the frequency of phenotype A after generations in a very large population evolving in a stochastic environment (Eq. ). The fitnesses of phenotypes A and B are and , respectively, where is s with probability p and with probability . The gray lines mark combinations of p and s for which and . According to , between these lines, fixation of either phenotype is not stochastically locally stable, and we expect a stationary polymorphism between the lines. Here, initial frequency of is , and the vertical transmission rate is .

Fig. 3.

Effect of vertical transmission rate on phenotype polymorphism in a randomly changing environment. Dynamics of the frequency of phenotype over time starting at when the fitnesses of phenotypes and are identically and independently distributed random variables. As the vertical transmission rate increases from 0.001 to 0.5, the frequency reaches a polymorphic distribution with faster, but the variance also increases. The fitnesses of phenotypes and , and , respectively, are both exponential random variables with expected values of two.

Evolutionary Stability of Oblique Transmission

An interesting question concerns the evolution of oblique transmission itself. For example, is there an evolutionarily stable rate of oblique transmission? To answer this question, we use a modifier model, in which we suppose that the vertical transmission rate is controlled by a genetic locus with two possible alleles and . Let the vertical transmission rates determined by and be and , respectively. Thus, there are four pheno-genotypes: , , , and , with frequencies that, at a given generation, are denoted by , , , and , respectively. As the fitnesses are determined by the two phenotypes and and the modifier locus is selectively neutral, we have the following table:Following the rationale leading to Eq. , the next generation pheno-genotype frequencies , , , and arewith , the mean fitness, given byNote that, under these assumptions, the locus and the phenotypic dichotomy do not undergo anything analogous to recombination, which might be introduced if phenotypes were viewed as haploid genetic variants.

Starting with a stable equilibrium, where only the allele is present, we check its external stability (27, 32) to invasion by allele . A constant environment always leads to fixation of the favored type, independent of . We, therefore, assume changing environments and in particular, the simple case of the cycling environment, where a unique stable polymorphism exists and depends on ( has a computational analysis of the general case). Specifically, from Eq. with , in the first generation and , in the second generation, after two generations, we havewhere the nonlinear transformation is given by Eq. with , and the nonlinear transformation is given by Eq. with , . Here, , , and are the frequency vectors.

For the case, when only the allele is present with associated rate , , and , a unique stable equilibrium exists. is the only positive root of the quadratic equation , with specified in Eq. . Solving givesand it can be seen that

The external stability of to the introduction of the modifier allele with rate is determined by the linear approximation matrix . We prove the following result in .

Result 8.

has two positive eigenvalues, and

when , the two eigenvalues are less than one;

when , the largest eigenvalue is larger than one; and

when , the largest eigenvalue is one.

We conclude that, in the selection regime, an allele producing vertical transmission rate is stable to the introduction of a modifier allele with associated rate if , and it is unstable if . Thus, in this case, evolution tends to reduce vertical transmission and hence, increase the rate of oblique transmission, and there is a reduction principle for the rate of vertical transmission (27, 32). The evolutionary dynamics of the reduction in under the cycling regime are shown in Fig. 4, which also illustrates the change in phenotype frequencies over time.

Fig. 4.

Consecutive fixation of modifiers that reduce the vertical transmission rate in selection regime A1B1. The figure shows results of numerical simulations of evolution with two modifier alleles (Eq. ). When a modifier allele fixes (frequency 99.9%), a new modifier allele is introduced with a vertical transmission rate one order of magnitude lower (vertical dashed lines). (A) The frequency of phenotype A in the population over time. (B) The frequency of the invading modifier allele over time. (C) The population geometric mean fitness over time; Inset zooms in to show that the mean fitness increases slightly with each invasion. Invading alleles are introduced at frequency 0.01%; whenever their frequency drops below 0.01%, they are reintroduced. Parameters: vertical transmission rate of the initial resident modifier allele, ; fitness values: and . The x axis is on a log scale, as each sequential invasion takes an order of magnitude longer to complete. Fig. S12 shows and .

In the case of identically distributed random fitnesses and , Fig. 5 shows an example of the success of modifiers that reduce . We have not, however, been able to prove that there is a reduction principle for this class of fluctuating fitnesses.

Fig. 5.

Consecutive fixation of modifiers that reduce the vertical transmission rate under symmetric randomly changing selection. The figure shows results of numerical simulations of evolution with two modifier alleles (Eq. ). When a modifier allele fixes (frequency > 99.9%), a new modifier allele is introduced with a vertical transmission rate one order of magnitude lower (vertical dashed lines). (A) The frequency of phenotype A in the population over time. (B) The frequency of the invading modifier allele over time. Invading alleles are introduced at frequency 0.01%; whenever their frequency drops below 0.01%, they are reintroduced. Parameters: vertical transmission rate of the initial resident modifier allele is , and the ratio of fitness values is with probability 0.5 and also with probability 0.5. The x axis is on a log scale, as each sequential invasion takes an order of magnitude longer to complete.

Values of the evolutionarily stable vertical transmission rate, , for some examples ( and Fig. S10 have analytical details) are recorded in Table 1 for different values of relative to . Interestingly, with , the evolutionarily stable value of is zero for the regime but not for the and regimes, in which the only stable values are those that lead to fixation of phenotype (e.g., and , respectively); these are, therefore, neutrally stable (Fig. S10). results are plotted in Fig. 6. In the regime, , and there is reduction of vertical transmission for all selection values tested. However, for regimes with , we find , and depending on , can be as high as 0.95. In Table 1, blank values for indicate that our method was numerically unstable and that a precise value for could not be obtained. This is why, in Fig. 6, no points are shown for with . In Table 1, the word “fixation” indicates that fixation of occurs, at which point there can be no effect of modification of ; cannot be calculated in such cases.

Table 1.

Values of (stable ) and (optimal )

k	l	{w} †	ρ*	ρ^
1	1	0.1	0.000000	0.000000
1	1	0.5	0.000000	0.000000
1	1	0.9	0.000000	0.000000
1	2	0.1	0.000000	0.00065
1	2	0.5	Fixation	Fixation
1	2	0.9	Fixation	Fixation
2	2	0.1	0.000000	0.000000
2	2	0.5	0.000000	0.000000
2	2	0.9	0.000000	0.000000
3	10	0.1	>0.4265	0.00031
3	10	0.5	Fixation	Fixation
3	10	0.9	Fixation	Fixation
5	30	0.1	>0.1489	0.00027
5	30	0.5	Fixation	Fixation
5	30	0.9	Fixation	Fixation
12	12	0.1	0.84924	0.24347
12	12	0.5	0.91209	0.000000
12	12	0.9	0.95686	0.000000
20	20	0.1		0.223925
20	20	0.5	0.94643	0.000000
20	20	0.9	0.98304	0.000000
30	30	0.1		0.193280
30	30	0.5	0.96331	0.000000
30	30	0.9	0.99136	0.000000
50	50	0.1		0.15419
50	50	0.5	0.9768	0.22107
50	50	0.9	0.99581	0.000000

ρ* is the uninvadable value of the vertical transmission rate. is the rate that maximizes the geometric mean fitness at the stable equilibrium of the AkBl cycle

Note that .

Fig. 6.

Fitness “optimal” and evolutionary stable vertical transmission rate in selection regime. (A) The vertical transmission rate that maximized the geometric average of the population mean fitness is zero (complete oblique transmission) when selection cycles quickly between favoring phenotype and and then abruptly transitions to followed by a slow decrease (Figs. S5 and S6 have details on the abrupt transition). (B) The evolutionary stable rate , which cannot be invaded by modifiers with either higher or lower vertical transmission rate , rapidly increases from zero when selection cycles are short ( or ) to roughly one when selection cycles are longer. The dashed line shows , which fits the values for (41). The values for (blue) could not be calculated for due to numerical instability when selection is strong and the duration between selection fluctuations is long. In all cases, . has details on how we calculated the stable rate.

Geometric Mean Fitness and Rate of Vertical Transmission

Under fluctuating selection, the geometric mean fitness of genotypes has been shown to determine their evolutionary dynamics (8, 30, 33). For the evolution of mutation rates that are controlled by genetic modifiers, the stable mutation rate and the mutation rate that maximizes the geometric mean fitness of the population seem to be the same when the period of environmental fluctuation is low enough (24). We can ask the same question here: is the stable rate the same as the rate that maximizes the equilibrium value of the geometric mean fitness under fluctuating selection? For the selection regime, we have the following result.

Result 9.

If and , then the mean fitness at the stable equilibrium in the environment is a decreasing function of .

In , the stable frequency of phenotype is, by Eq. ,where . The geometric mean fitness at the stable equilibrium is , where is the mean fitness at the middle of the cycle; in this case due to the symmetry between the two phenotypes and , which allows us to reduce the problem to properties of . Now, because , is an increasing linear function of :Thus, is decreasing in if is negative. Using Eq. ,From Eq. , , and therefore, , which completes the proof.

Fig. 4 illustrates the increase over time of the geometric mean fitness with decreasing at a polymorphic equilibrium in the regime. The values of and are the same in and regimes, namely both are zero. Fig. 1 shows the geometric mean fitness in the regime, and we see that, for small values of , this mean fitness increases as decreases. At , Table 1 shows that and are roughly zero. In all regimes that we tested with , the value of was also zero, substantially different from the values of , as shown in Fig. 6. Also in Fig. 6, we see that, with , changes from zero to positive in the regimes with , while with , the change occurs at . In Fig. 6, with , is between 0.15 and 0.24 for , while with , exceeds 0.2 for . More details on the mismatch between , which cannot be invaded, and , which maximizes geometric mean fitness, are given in Table 1, Figs. S7 and S8, and .

Finite Population Size

To include the effect of random drift due to finite population size in the above deterministic model, we use the Wright–Fisher model. Let denote the number of individuals with phenotype in a population of fixed size at the th generation, and suppose that . Also, let represent the frequency of the phenotype in the infinite population model in the next generation, namely (Eq. )Then, according to the Wright–Fisher model (34), , the number of individuals of phenotype at generation , is determined by the probabilityfor . Thus, the fluctuations in the numbers of phenotypes and in the population of size are generated by the Wright–Fisher Markov chain, where, given that , has a binomial distribution with parameters .

This Markov chain has two absorbing states, and , corresponding to the two fixations in and , respectively, and we are interested in the fixation probabilities and the time to fixation of these two absorbing states as functions of the initial frequency and also of , , and . To these ends, we use a diffusion approximation to the process , which allows us to compute , the probability that phenotype goes to fixation when its initial frequency is , namelyThe expected time to fixation in starting from an initial frequency of is given bywhere is given in Eq. , and in generations, is multiplied by (the derivation is in ). Unfortunately, the integrals in Eq. cannot be done in closed form unless , in which case and (ref. 34, p. 160), and only numerical computation of is possible for specified values of , , and .

For the fixation probability , we have the following result.

Result 10.

When , so that the phenotype is favored, the fixation probability is monotone increasing in .

The proof of is in . Fig. S9 compares the fixation probability and time to fixation derived numerically from simulating the Wright–Fisher Markov chain with the diffusion-derived values of and . The fit is seen to be very good. Note that, when is large, the Wright–Fisher model exhibits persistent fluctuation around the deterministic expectation, as shown by the orange traces in Fig. S2.

We can also develop a diffusion approximation for the case of a cycling environment. Suppose that selection changes in cycles of length , such that, within the cycle, the fitness parameters are , for . Also, letFollowing Karlin and Levikson (35), we have the following result.

Result 11.

The mean and variance of the change in the frequency of in one generation for the diffusion approximation in the case of a cycling environment , where , are The proof of Result 11, based on induction on .

Using the moments in Eq. , the fixation probability and the expected time to fixation from an initial frequency of can be computed, where is replaced by . We findand can be computed similarly.

In the case of the cycling environment, we write , and if , for generations and , for generations, we haveFig. 7 shows an example of how , , and in Eq. for interact to affect fixation probabilities. More examples are illustrated in Fig. S11.

Fig. 7.

Fixation in a finite population with different ratios of selection periods . Fixation probability of phenotype when starting with a single copy in a population of size : (Eqs. and ). k and l are the numbers of generations in which phenotypes A and B, respectively, are favored by selection. Here, fitness of the favored phenotype is , fitness of the disfavored phenotype is , and the population size is . Fig. S11 shows additional examples.

Discussion

Nonchromosomal modes of phenotypic transmission are receiving increasing attention (36–38), especially with respect to their potential role in adaptation and maintenance of diversity (39). Here, we have focused on a dichotomous phenotype transmitted through a combination of parental and nonparental transmission. In addition to the roles that these transmission modes play in the dynamics of phenotypic diversity in large and small populations, we have also investigated a genetic model for the evolution of the transmission mode itself.

Our model differs markedly from that of Xue and Leibler (26), who took the individual phenotypic distribution (i.e., the probability that an individual develops one of a set of phenotypes) to be the inherited trait. In our model, the transmitted trait is the phenotype itself. Thus, with two phenotypic states and , we track the frequency of , whereas Xue and Leibler (26) focus on the dynamics of the per-individual probability of learning the phenotype . One interpretation of our model is as a mean value approximation to the model of Xue and Leibler (26), where , the state in our model, is the average of the population distribution of individual phenotype probabilities.

In a constant environment, the higher the vertical transmission rate , the faster the approach to fixation of the favored phenotype: if or if . Here, , the oblique transmission rate, represents the added chance that an offspring becomes by learning from the parent’s population after learning from the parents who have undergone selection (Eq. ). This simple phenotypic model does not allow a polymorphism to be achieved in a constant environment, but with more oblique transmission, approach to fixation is retarded.

With fluctuating environments, the dynamics of the phenotype frequencies are, in general, much more complicated. In particular, with deterministically cycling symmetric fitness values (the model), it is impossible for fixation in and to both be stable. If , for example, neither fixation is stable, and there is a single stable polymorphic equilibrium (with phenotypes and present) (). In the case, this polymorphism is globally stable. In the case, bounds on that determine the instability of both fixations and hence, the protection of polymorphism are given by the inequalities of Eq. , which depend on both the fitness differences and the rate of vertical transmission. We conjecture that, with , there is a unique stable polymorphism if both fixations are unstable. This result is similar to the storage effect (40), in which protection from selection maintains species coexistence: consider two species, and , with overlapping generations, an equal death rate , and different growth rates and ; then, Eq. describes the change in frequency of species . In our model, oblique transmission can be said to protect the disfavored phenotype from selection, because it allows transmission without reproduction.

In deterministic one-locus, two-allele diploid population genetic models with cycling fitness regimes, Haldane and Jayakar (33) first showed the relevance of the geometric mean of genotypic fitness (compare Eqs. and ) for the maintenance (or loss) of polymorphism. However, with equal homozygote fitness, alternating in strength as a two-generation cycle (compare with ), Karlin and Liberman (31) extended the results of Haldane and Jayakar (33) and found conditions under which both allelic fixations and polymorphic equilibrium could all be stable, with the evolution depending on initial allele frequencies as well as the homozygote fitness differences between alternate generations. Our haploid model does not seem to produce such dependence on the initial conditions.

When the fitnesses and are treated as random variables rather than varying cyclically, stochastic local stability is the appropriate analog to local stability in the case of cyclic fitness variation. While fixations in phenotypes and cannot both be stochastically stable in this case, both may be unstable, and a polymorphic distribution may result. The variance of this distribution is greater for larger values of . This is because the stochastic local stability conditions involve , and the effect of the variance of will clearly increase as increases. In the finite population case, a greater level of vertical transmission makes selection more effective, increasing the probability of fixation and reducing the expected time to fixation.

We have shown that, in the case, the rate of vertical transmission tends to decrease when it is under the control of a genetic modifier. From numerical iteration, it seems that this is also true in the random selection case when the fitnesses of and are identically distributed and independent between generations. However, for selection regimes more complicated than , evolution of a modifier of vertical transmission is not straightforward. While reduction of occurs in the regime, the uninvadable value is not zero for all of the fitness values explored in regimes with (Fig. 6 and Table 1). In fact, increases sharply as increases beyond . This is an unusual scenario for genetic modifiers, although it must be noted that a modifier of is not neutral; it affects primary selection, while neutral modifiers of recombination, mutation, and migration affect induced or secondary selection.

The dependence of the modifier dynamics on the strength of selection (that is, when ) is complicated by the approach of the system to fixation. When the phenotype frequencies become exceedingly small, dependence of the dynamics of the modifier of becomes extremely difficult to detect due to numerical instability; this is especially true for larger values of in regimes when is small (Fig. 6 and Table 1) (with ).

Fig. 6 (Table 1) shows that the value that maximizes the geometric mean fitness is the same as the evolutionarily stable value in the and selection regimes. For regimes with , our numerical analysis shows that depends strongly on the strength of selection (i.e., the value of relative to ). For regimes with , the difference between and is seen even with the environment. For regimes and , we find , while is close to . For larger values of , is between 0.15 and 0.25, while remains above 0.8 and can reach 0.99 for very large . Comparing Fig. 6 with the asymptotic growth rate (AGR) of Xue and Leibler (26), whose parameter is the rate at which an individual learns from its parental lineage, there is a similarity of our curves for and with their curve in the environment. They show the AGR decreasing with in the regime for small , but larger values of entail that the AGR has a maximum for an intermediate value of .

Although the models of Xue and Leibler (26) and that analyzed here both incorporate parental and nonparental transmission, they do so in qualitatively different ways. The model treated in this paper is squarely in the tradition of gene–culture coevolutionary theory together with modifier theory from population genetics. The different findings from the two classes of models are interesting and suggest that additional exploration of the overlaps and discrepancies between the two approaches would be worthwhile.