Assortative mating and self-fertilization differ in their contributions to reinforcement, cascade speciation, and diversification.

Cascade speciation and reinforcement can evolve rapidly when traits are pleiotropic and act as both signal/cue in nonrandom mating. Here, we examine the contribution of two key traits-assortative mating and self-fertilization-to reinforcement and (by extension) cascade speciation. First, using a population genetic model of reinforcement we find that both assortative mating and self-fertilization can make independent contributions to increased reproductive isolation, consistent with reinforcement. Self-fertilization primarily evolves due to its 2-fold transmission advantage when inbreeding depression (d) is lower (d < 0.45) but evolves as a function of the cost of hybridization under higher inbreeding depression (0.45 < d < 0.48). When both traits can evolve simultaneously, increased self-fertilization often prohibits the evolution of assortative mating. We infer that, under specific conditions, mating system transitions are more likely to lead to increased reproductive isolation and initiate cascade speciation, than assortative mating. Based on the results of our simulations, we hypothesized that transitions to self-fertilization could contribute to clade-wide diversification if reinforcement or cascade speciation is common. We tested this hypothesis with comparative data from two different groups. Consistent with our hypothesis, there was a trend towards uniparental reproduction being associated with increased diversification rate in the Nematode phylum. For the plant genus Mimulus, however, self-fertilization was associated with reduced diversification. Reinforcement driving speciation via transitions to self-fertilization might be short lived or unsustainable across macroevolutionary scales in some systems (some plants), but not others (such as nematodes), potentially due to differences in susceptibility to inbreeding depression and/or the ability to transition between reproductive modes.

When isolated lineages come into contact, selection may favor divergence in mating signals and preferences–a pattern that is termed reproductive character displacement (Brown and Wilson 1956; Pfennig and Pfennig 2009). When the source of increased sexual isolation is due to selection against hybridization between incompletely isolated lineages, this is referred to as reinforcement (Dobzhansky 1951; Howard 1993; Servedio and Noor 2003). Recently, a potential secondary outcome of selection against hybridization in sympatry has been described, where local selection in sympatry can lead to incidental reproductive isolation between sympatric and allopatric populations of the same species. This phenomenon has been called “cascade speciation” or “reproductive character displacement (RCD) speciation” (Ortiz-Barrientos et al. 2009; Hoskin and Higgie 2010). For cascade speciation to occur, there are typically three requirements: 1) reinforcement must act between two species, often due to selection for increased prezygotic isolation in response to the costs of maladaptive hybridization; 2) there must be variation among populations in their propensities to mate with individuals from other populations (both heterospecific and conspecific); and, 3) there is a correlated response to the increase in prezygotic isolation between species, that generates increased prezygotic isolation between populations within species. Of these, most theoretical and empirical attention to date has focused on the conditions that favor the first requirement—reinforcement. Our treatment here similarly focuses on reinforcement to explore the conditions and traits that could ultimately contribute to cascade speciation.

Empirically, cascade speciation has only been documented in a few animal systems, notably amphibians (Hoskin et al. 2005; Pfennig and Rice 2014), insects (Nosil et al. 2003; Jaenike et al. 2006; Higgie and Blows 2007, 2008; Porretta and Urbanelli 2012; Dyer et al. 2014), and one fish (Kozak et al. 2015). A common characteristic in these systems is that they fit into standard preference/trait models of sexual selection: male traits are used as signals that are favored/rejected by females (Kirkpatrick and Ravigne 2002). Therefore, these particular taxa might be especially prone to cascade effects because these preferences and/or traits differ among populations, potentially due to divergent sexual selection. However, many other traits could also be targets of reinforcing selection (and by extension cascade speciation). In particular, reinforcing selection can act on any trait that reduces the production of hybrid offspring, including traits that affect premating barriers (i.e., nonrandom mating via assortative mating, shifts in mating system, increased self-fertilization) or postmating prezygotic barriers (i.e., gametic isolation, conspecific gamete precedence). This suggests that the classes of traits potentially contributing to cascade speciation could be much broader than those captured in empirical studies.

Our interest here is in examining the potential contribution of two such traits—assortative mating and shifts in mating system—to elevated reproductive isolation due to reinforcing selection. The ways in which assortative mating and self-fertilization contribute to reproductive isolation via reinforcing selection, and the consequences for cascade speciation, are not necessarily the same. Assortative mating—nonrandom mating in which an individual chooses a mate based on variation in a distinguishing trait—can evolve quickly and reinforce reproductive isolation when the signal on which mate choice is based experiences divergent selection from the environment (Gavrilets 2004; Servedio et al. 2011). Reinforcing selection on assortative mating alleles can in turn lead to cascade speciation because alleles favored in sympatry may be disadvantageous in allopatry, for example—by promoting costly mating discrimination against suitable conspecifics in allopatry (Nosil et al. 2003; Jaenike et al. 2006; Higgie and Blows 2007, 2008; Ortiz-Barrientos et al. 2009; but see Yukilevich this issue). Assortative mating traits that appear to be the targets of reinforcing selection and cascade speciation include cuticular hydrocarbons in insects (Higgie and Blows 2007, 2008; Dyer et al. 2014), flower color and other traits that mediate pollinator visitation in plants (Hopkins 2013), and reproductive phenology (McNeilly and Antonovics 1968; Filchak et al. 2000; Silvertown et al. 2005; Powell et al. 2014).

Like assortative mating, selfing is a second type of nonrandom mating but one in which an individual preferentially self-fertilizes rather than outcrosses. Self-fertilization can experience reinforcing selection depending upon levels of inbreeding and outbreeding depression (Epinat and Lenormand 2009; Hu 2015). Increasing self-fertilization in response to reinforcing selection could also contribute to cascade speciation by reducing gene flow between sympatric and allopatric populations of the species responding to selection. However, unlike the case of assortative mating—where export of the trait in sympatry to allopatry would preclude cascade speciation—the spread of the selfing allele into allopatry could still result in cascade speciation; the spread of selfing alleles would then resemble a one-allele model of speciation (Felsenstein 1981), where different populations fix the same allele which can cause prezygotic isolation between them. Even if allopatric populations remain outcrossing, however, strong self-fertilization by sympatric populations might preclude gene flow, also resulting in cascade speciation. Evidence for self-fertilization contributing to reinforcement has not directly been evaluated in animals, but it has been observed several times in plant systems (Fishman and Wyatt 1999; Brys et al. 2014; Buide et al. 2015; Palma-Silva et al. 2015) and could contribute to cascade speciation via the mechanisms described above.

Given their potentially different consequences for cascade effects, it is also interesting to evaluate how these traits might interact to affect responses to reinforcing selection. Although many of the ecological and genetic conditions expected to promote increased prezygotic reproductive isolation due to reinforcing selection have been examined individually, there has been much less attention to how such factors might interact or interfere with each other. An exception is Lorch and Servedio (2007), who found that both gamete precedence and assortative mating can be targets of reinforcing selection, but that these traits negatively affect one another when allowed to coevolve; when gamete precedence evolves first, it inhibits the evolution of assortative mating and vice versa. Apart from this case, it remains unclear whether the joint action of two potential targets of reinforcing selection routinely results in antagonism between their contributions to reinforcement, even though such interactions are potentially common in nature. In particular, mixed mating systems are common in both plants and animals (Goodwillie et al. 2005; Jarne and Auld 2006); such groups provide an opportunity to study how different mechanisms for non-random mating (including assortative mating and selfing) interact to influence cascade speciation.

Here we examine whether assortative mating or self-fertilization has the greater potential to contribute to reinforcement. We focus on modeling the interaction of these two traits on reinforcement because cascade speciation is generally discussed in the framework of reinforcement (Hoskin and Higgie 2010). Requiring that reinforcement initiate divergence in mating traits likely represents the most challenging case for cascade speciation, because reproductive character displacement might be caused by many other factors than reinforcing selection (Pfennig and Pfennig 2009, 2010). Any results of this model should be generalizable to reproductive character displacement as a whole. In addition to examining individual effects, we also examine the interactions of these two potential mechanisms during their evolution, using a population genetic model that can be generalized to any species with the potential for mixed mating. We test if assortative mating and self-fertilization facilitate or inhibit the evolution of one another, and we evaluate the outcome of their interaction on the strength of isolation.

Results from our model suggest that shifts to self-fertilization in response to reinforcing selection might pre-empt responses based on assortative mating. This would also indicate that selfing could be a rapid route to cascade speciation, if sympatric and allopatric populations are now isolated by differences in mating system. If selfing is the more rapid route to both reinforcement and/or cascade speciation, and these processes happen frequently, then we expect to see that transitions to self-fertilization lead to higher diversification rates (more speciation events rather than extinction or hybrid swarms) across timescales of species formation, all else being equal. We evaluated evidence for this general association between transitions to selfing and diversification by contrasting selfing versus outcrossing in one animal (Nematoda) and one plant (Mimulus) group, to test if uniparental reproduction (Nematoda) or “selfing syndrome” (Mimulus) is associated with changed diversification rates. Note that these tests focus on contrasting selfing versus outcrossing, unlike our population genetic model, which contrasts selfing and assortative mating.

Materials and Methods

Reinforcement model

We employ a two-island population genetic model that examines a one-allele model of reinforcement between two populations exchanging migrants. The model is an individual-based stochastic model that contains five haploid diallelic loci. The first locus, L, has population specific alleles and describes local adaptation. If an individual has an allele of L that does not match the local environment, there is a probability l that it is removed from the population (relative fitness of that allele is 1−l). The second and third loci, M and N, describe post-zygotic isolation between the two populations based on a negative epistatic interaction (Dobzhansky–Muller incompatibility) and also have population-specific alleles. Hybrid genotypes at these loci (M1N2 or M2N1, where subscripts represent population of origin) have a probability h of being removed from the population. The L locus functions to establish linkage disequilibrium between itself and the two incompatibility loci (M and N), maintaining these alleles in separate populations before the introduction of assortative mating or selfing (selection against hybrids without local adaptation is not sufficient to maintain frequencies of M and N in each population). The fourth locus, A, determines if an individual will mate assortatively based on its MN genotype with probability a (individuals carrying allele A0 mate randomly, whereas individuals carrying allele A1 mate assortatively). In some models of reinforcement (Servedio 2004; Lorch and Servedio 2007), only females mate assortatively based on the males MN genotype; these models often assume separate sexes and no phenotype matching based on the A allele. We instead chose to require both individuals to express the same AMN genotype under assortative mating, mainly because we were interested in systems where selfing could occur; this requires simultaneous hermaphrodites (as in plants and nematodes) and therefore there cannot be just male/female genders in the population. In addition, the traits that we wished to consider might be most appropriately modeled via phenotype matching at the A locus. For example, if assortative mating is based on trait matching (which would be seen in phenology shifts, host shifts, floral color) and there is an association between signal and species identity, then assortative mating will occur because of all three loci.

Lastly, the fifth locus, S, determines if an individual will self-fertilize with probability s (individuals carrying allele S0 cannot self-fertilize, whereas individuals carrying allele S1 have the potential to self-fertilize). Progeny that result from self-fertilization have a probability d of being removed from the population, simulating inbreeding depression when d > 0. In each life cycle (described below), whether an individual self-fertilizes is determined prior to any assortative mating via outcrossing. This reflects a “prior selfing” mating system rather than “delayed selfing” (Lloyd 1979). We chose to evaluate a model of prior selfing because delayed selfing is typically thought to evolve due to reproductive assurance (when mating opportunities are rare) rather than evolving in response to the identity of local mates (i.e., presence of heterospecifics), which is the effect of interest here. In addition, the selfing locus has a 2-fold transmission advantage in terms of progeny produced per individual, but this is solely due to a transmission advantage via maternal (selfed) fitness; that is, the additional contribution of male gametes to outcross progeny is treated as negligible. This assumption is a reasonable approximation for several broad classes of organisms. For example, in many plant systems selfing is associated with a suite of traits that decrease flower size, and decrease the probability of exporting pollen (Sicard and Lenhard 2011). Similarly, in the Nematode phylum hermaphrodites are able to self-fertilize but have no mechanism to cross-fertilize (Hill et al. 2006; Baldi et al. 2009). The simulation code for the model is available through the IU Scholar works repository (http://hdl.handle.net/2022/20335).

Life cycle

Each generation both populations go through the following life cycle stages:

Migration occurs symmetrically between the two populations. The migration rate is constant and low at 1% of the population migrating per generation.

Individuals that carry the S allele self-fertilize with probability s. The resulting progeny are the same genotype as the parent (no mutation and no recombination). If individuals do not self-fertilize (1−s), then they either assortatively mate or go on to mate randomly with the rest of the population. Individuals that self-fertilize have a 2-fold advantage in progeny production.

Individuals that carry the A allele will mate with other individuals that also carry the A allele (and matching MN genotype) with probability a. Free recombination occurs in these matings. If individuals do not mate assortatively (1−a), then they mate randomly with the rest of the population. Hybrids (individuals with a nonmatching MN genotype) mate randomly regardless of the A locus.

Random mating and free recombination occurs between all individuals that did not self-fertilize or mate assortatively.

Natural selection occurs with probability l against individuals with the “incorrect” L allele. Selection against hybrids at the MN loci occurs with probability h and selection against progeny from self-fertilization occurs with probability d.

Simulations

This model leads to a large number of recursions that are intractable analytically. Instead, we used stochastic simulations iterating through the life cycle, using R v2.15. Each simulation was run three times and average values were used to make inferences (there was little variation between runs). Simulations were first run with just the L, M, and N loci. The L, M, and N allele frequencies started at 0.99 for each population and the simulation was run until equilibrium was reached. Each population had a finite size of 10,000 individuals (20,000 in total). Reproduction was density dependent and the carrying capacity for each population was K = 10,000. For individuals that self-fertilized they produced two times as many progeny as an individual that did not self-fertilize which gives this allele a transmission advantage.

At equilibrium in our simulations (determined when the change in allele frequency was <10e−6), prior to introducing either the A1 or S1 alleles, the proportion of pure species matings (Θ, see below) was 0.94. This occurred when selection due to local adaptation (l) was 0.3 and selection against hybrids (h) was 0.3. These initial values were chosen because preliminary simulations demonstrated that these were the smallest values that could maintain two distinct populations (in terms of MN genotypes) but still potentially detect increases in reproductive isolation after the inclusion of the A1 and S1 alleles. After equilibrium had been established, we introduced the A1 and S1 alleles at low initial frequency. Simulations were run until they reached equilibrium or until 1,000 generations had passed (whichever came first). We ran three sets of simulations, the first two examining the effect of assortative mating and selfing individually, and then the last set examining these two mating traits together.

Assortative mating individually

In this set of simulations, we introduced only the A1 allele and varied the probability of assortative mating a. We maintained a constant level of selection due to local adaptation (l = 0.3) and selection against hybrids (h = 0.3) as the main goal was to evaluate the relationship between levels of assortative mating and the strength of selection for assortative mating and level of reproductive isolation.

Selfing individually

In set two, we introduced only the S1 allele and varied the probability of selfing s, selection against hybrids (h), and the selection due to inbreeding depression d, while keeping selection due to local adaptation constant. The goal of these simulations was to determine whether reinforcing selection could act on the selfing locus, and whether we could detect these effects over and above the 2-fold advantage in progeny production (by varying h and d simultaneously).

These two sets of simulations allowed us to establish the strength of selection acting on these loci (measured as generation until fixation) and the strength of resulting reproductive isolation due to reinforcing selection (Θ, or the proportion of pure-species matings, measured as the number of pure-species matings L1M1N1 × L1M1N1 and L2M2N2 × L2M2N2 divided by the total number of matings).

Assortative mating and selfing simultaneously

Lastly, the third set of simulations introduced A1 and S1 simultaneously. We varied a, s, h, and d based on values we know facilitated increased reproductive isolation due to reinforcing selection when these alleles were considered separately. The goal of these simulations was to determine if one locus was able to preclude the other, or if they could jointly contribute to reproductive isolation, and under what conditions did these outcomes occur. The two values of a (a = 0.7, a = 0.9) were chosen as representative of moderate and strong assortative mating, both which greatly increased reproductive isolation (see “Results” section), and three values of s (s = 0.4, s = 0.6, s = 0.8) were chosen as values of selfing that increased reproductive isolation (see “Results” section). We varied d based on values of inbreeding depression where selfing advantage could not be distinguished from reinforcing selection (i.e., d = 0 .4, “low” inbreeding depression), where we could distinguish the effects of reinforcing selection from the 2-fold transmission advantage of selfing (i.e., d = 0.45, “moderate” inbreeding depression) and where the selfing allele was prevented from fixing under several simulation conditions (i.e., d = 0.48, “high” inbreeding depression).

Association between diversification rates and increased selfing

Since the results of our model suggest that shifts to self-fertilization in response to reinforcing selection might pre-empt responses based on assortative mating (see below), we examined whether transitions to greater rates of self-fertilization were associated with higher diversification (speciation) rates across timescales of species formation. Most datasets available for assessing the relationship between diversification and variation in self-fertilization are from plant systems (Goldberg et al. 2010; Ferrer and Good 2012), though at least one example exists for animals (Ross et al. 2013). We analyzed new data from one animal and one plant group.

Diversification rates and uniparental reproduction in the Nematoda

In the Nematode phylum (roundworms), reproductive modes include obligate outcrossing, obligate asexuality (parthenogenesis), and androdioecy (in which hermaphrodites self-fertilize or outcross with rare males). There have been multiple transitions from outcrossing to uniparental reproductive modes (asexuality and androdioecy) (Kiontke et al. 2004; Cutter et al. 2008; Denver et al. 2011). We evaluated if uniparental reproduction is associated with lower or higher diversification rates. Taxa were classified following Gibson and Fuentes (2015), in which male frequency was used as a proxy for mating system. Because the absence of males cannot distinguish parthenogenesis (asexuality) from androdioecy (mostly selfing), these two modes are treated as a single category of uniparental reproduction.

We used the phylogeny and character data originally reported in Gibson and Fuentes (2015), which included a phylogenetic reconstruction of 2,700 postburn-in trees modified from Meldal et al. (2007). Of the 162 taxa, 28% have a uniparental mode of reproduction. Note that in our sample of nematode taxa here, a large fraction of outcrossing taxa (59%) are parasitic, while the majority of uniparental taxa (67%) are free-living. Therefore, any differences detected between these groups might also be influenced by their distinct ecology.

Individual postburn-in trees were made ultrametric using the Grafen method (Grafen 1989) as implemented in ape package for R (Paradis et al. 2004). We estimated and compared diversification rates of outcrossing and uniparental nematode taxa using the Binary state speciation and extinction (Maddison et al. 2007) method (BiSEE) in the diversitree package (FitzJohn 2012) implemented in R v.3.20. Given the phylogenetic uncertainty in this clade (unresolved internal nodes), we performed two analyses. In the first analysis, we estimated the likelihood of three models for each postburn-in tree independently (2,700 trees total): a full model in which speciation and extinction rates of outcrossing and uniparental taxa were free to vary; a constrained model in which speciation rates of outcrossing and uniparental taxa were set equal to one another; and a second constrained model in which extinction rates were set equal. We compared the likelihood of the full model to that of each of the constrained models. A significant likelihood ratio test indicates asymmetric speciation or extinction rates caused by differences in reproductive mode. For trees in which the full model was a significantly better fit than a constrained model, we further asked if diversification of uniparental taxa is greater or less than that of outcrossing taxa. To do so, we compared the speciation and extinction rates of the two reproductive modes obtained in the full model. In the second analysis, we used a consensus tree and MCMC analysis to estimate all parameters simultaneously while allowing a clade to represent several taxa with known character states for clades with polytomies (FitzJohn et al. 2009).

The vast majority of nematode taxa are undescribed, and the Nematode phylum is vastly under-sampled in this phylogeny. BiSEE is sensitive to incomplete taxon sampling (FitzJohn et al. 2009; FitzJohn 2012) and we acknowledge that under-sampling might affect our results. However, such datasets for animals are very rare (though see Ross et al. 2013), so we present these analyses here with this caveat in mind.

Diversification rates and selfing syndrome in Mimulus

The plant genus Mimulus is a florally diverse plant clade of ∼120 species, with most species concentrated in Western North America (Beardsley et al. 2004). All Mimulus species are self-compatible, but outcrossing rates vary widely between species depending largely on floral morphology: outcrossing species tend to have large showy flowers, whereas inbreeders have traits consistent with “selfing syndrome” (small flowers often with enclosed reproductive parts that limit the access of pollinators and/or pollen from other flowers). In addition, patterns consistent with sympatric character displacement have been observed in Mimulus: compared to allopatric species pairs, sympatric species exhibit greater changes in floral morphology including size differences, reflecting mating system transitions (Grossenbacher and Whittall 2011), and there is evidence for reproductive character displacement due to pollinator differences (Schemske and Bradshaw 1999; Grossenbacher and Stanton 2014). To examine the relationship between selfing syndrome (smaller flowers compared to medium or large flowers) and diversification rates, we analyzed a comparative dataset in Mimulus with the majority of the data drawn from Grossenbacher and Whittall (2011). Using their phylogeny, we added floral measurements for any species that lacked these measurements, using publicly available electronic herbarium specimens found in the Global Plants Database (plants.jstor.org). For several specimens, we measured the length and width of each flower [corolla tube length and corolla tube width in Grossenbacher and Whittall (2011)]. We validated our measurements by comparing measurements to Grossenbacher and Whittall (2011) for one species that was highly represented (M. guttatus) and one species with few representatives (M. cuisickii). Our measurements for these species did not exceed the range of measurements collected by Grossenbacher and Whittall (2011). In the final analysis, we did not include species that had ambiguous phylogenetic placement in the Grossenbacher and Whittall (2011) topology. The final tree and measurement data are available through the IU Scholar works repository (http://hdl.handle.net/2022/20335).

To analyze diversification rates, we used the Quantitative state speciation and extinction (QuaSSE) model as presented in FitzJohn (2010). This model allowed us to determine if differences in diversification are a function of floral morphology, specifically the log of floral size (length × width), which we used as a proxy for selfing rate. In this framework, it can be difficult to simultaneously estimate speciation and extinction rates if they are both allowed to vary with trait value, so we chose to use a constant extinction rate for all lineages. Although previous studies have focused on self-fertilization effects on extinction probabilities (Goldberg et al. 2010; Ferrer and Good 2012), we chose to focus on differences in speciation rates because increased speciation was predicted from our population genetic model (models using constant speciation and allowing extinction to vary yielded similar results; data not shown). Several models were then fit where speciation rate was either independent of the trait value, a linear function of the trait value, a sigmoidal function of the trait value, or a modal function of the trait value. We then fit these models allowing for directional tendency in the evolution of floral size in lineages (FitzJohn 2010). The best fitting model was determined by likelihood ratio tests comparing the null model (speciation rate is independent of the trait value) to the alternative models.

Results

We found that both assortative mating and self-fertilization contributed to reproductive isolation (the reduction of gene flow between populations) when analyzed individually. After the A1 allele was introduced, we saw increases in Θ indicating that this allele contributed to increasing isolation as expected (Figure 1B). Our results for the assortative mating allele (A1) are also consistent with previous results (Servedio 2004; Lorch and Servedio 2007) specifically that the efficacy of selection initially increases with the probability of assortative mating but then decreases. This is most easily seen in the number of generations until the A1 allele fixed in these simulations (Figure 1A).

Figure 1.

The ability for assortative mating and self-fertilization alleles to contribute to reinforcement. (A) The time (in generations) until the A1 allele becomes fixed in simulatons where only the A allele is present. (B) The increase in reproductive isolation for the same simulations.

We initially observed that for a given level of selection against hybrids (h) the magnitude of reproductive isolation (Θ) increased with the probability of selfing (similar to the assortative mating alleles), but was not influenced by the level of inbreeding depression. Thus, it was difficult to disentangle the effects of the transmission advantage of the selfing allele from the effects of reinforcing selection. To disentangle these effects, we evaluated the relationship between time until fixation and selection against hybrids, under varying levels of selfing (s) and inbreeding depression (d) (Figure 2). If reinforcing selection is contributing to selection for selfing, we expect to see fewer generations until fixation for the selfing alleles as selection against hybrids (h) increases (i.e., an overall a negative relationship between the two factors). We observed this pattern for moderate levels of inbreeding depression (d = 0.45) but not for lower inbreeding depression (d = 0.4). We also observed a pattern consistent with reinforcing selection for high inbreeding depression (d = 0.48), but this was in terms of allele frequency (rather than time until fixation) because under some parameters the selfing allele was not able to fix (note that when d > 0.5, the S1 allele cannot fix—Fisher 1930; Nagylaki 1976).

Figure 2.

The strength of reinforcing selection on the selfing allele demonstrated by the relatipnship between the strength of selection against hybrids and time until the selfing allele fixes when selfing occurs at (A) s = 0.2, (B) s = 0.5, and (C) s = 0.8. Each point represents an individual simulation. The stronger the reinforcing selection, the fewer generations it takes until the allele becomes fixed. Signifiance was determined from a linear model.

When both loci, assortative mating and selfing, were allowed to co-evolve, we observed that increasing the probability of selfing, the probability of assortative mating, and the amount of selection against hybrids, all increased the level of reproductive isolation between populations. Since reproductive isolation was generally high after each simulation, we focused on the time until fixation as a metric of response to reinforcement conditions. For the time until fixation, there were three main outcomes of our simulations, depending upon specific parameter conditions: 1) The assortative mating allele excluded the selfing allele (i.e., went to fixation before the selfing allele could appreciably increase in frequency), 2) the selfing allele excluded the assortative mating allele, and 3) the two alleles both fixed, but with some difference in the time to fixation (Figure 3). The first case (assortative mating excludes selfing) occurred when inbreeding depression was high and the selfing probability was low (Figure 3A and Figure 3D). Under lower levels of inbreeding depression and higher probabilities of selfing, we observed that the selfing allele excluded the assortative mating allele (exemplified by Figure 3C and Figure 3F). We also observed that increased selection against hybrids typically favored the selfing allele over the assortative mating allele (Figure 3B and Figure 3E).

Figure 3.

The time until fixation of the selfing and assortative mating allele in simulations where both alleles were introduced simultaneously under several conditions of a, s, and h where: a is the probability that individuals mate assortatively, s is the probability that individuals self, and h is the selection against hybrid progeny. The y-axis is the log ratio of the time it took the selfing versus the assortative mating allele to fix. In some simulations, the selfing or assortative mating allele did not fix. Since simulations were terminated at 1000 generations, alleles that did not fix were given a value of 999 generations. Values greater than 2 indicate that the selfing allele excluded the assortative mating allele. The converse is true for values less that −2. A value of 0 indicates that both alleles fixed in the same number of generations.

The prior selfing assumed in our model could potentially effect whether selection favors self-fertilization over assortative mating. This would only occur when both alleles are at intermediate frequencies and individuals carry both alleles. However, the outcomes of our simulations were mostly determined by the early increase of the selfing allele in the first few generations (i.e., in the absence of individuals carrying both alleles). Interestingly, when there was high inbreeding depression, the A1 allele had the effect of facilitating the evolution of a higher selfing rate, as the average allele frequency reached by S1 was greater than when this allele evolved alone under this condition.

In the Nematoda, speciation rates either did not differ between reproductive modes or were greater in uniparental lineages compared to biparental lineages, consistent with the expectation inferred from our model. In our analysis of each reconstruction of phylogenetic relationships (2,700 postburn-in trees), results from 59% of the trees indicated there was no evidence for asymmetrical speciation rates. In the 41% of trees for which there was significant evidence for asymmetrical speciation rates, the speciation rate of uniparental lineages always exceeded that of outcrossing lineages (greater by 18.54 ± 4.22; mean ± standard deviation). Extinction rates similarly did not differ between reproductive modes or were lower in uniparental lineages. In 59% of trees there was no evidence for asymmetrical extinction rates. In the 41% of trees that exhibited significant differences in extinction rates, the extinction rate of uniparental lineages was lower than that of outcrossing lineages in the majority of cases (97% lower by 44.83 ± 27.80). Our second MCMC analysis on the consensus tree yielded no support for asymmetrical speciation or extinction rates (Table 1).

Table 1.

Summaries of the parameter estimates from the maximum likelihood (ML) and MCMC analysis aimed at determining if diversification rates differed between unpiarental and biparental modes of reproduction

	ML estimation 95% range		MCMC 95% HPD
Parameter	Lower bound	Upper bound	Lower bound	Upper bound
Speciation—biparental	25.129	38.059	5.355	27.165
Speciation—uniparental	36.303	50.772	4.642	29.376
Extinction—biparental	19.636	55.411	4.248	29.276
Extinction—uniparental	0.000	75.390	0.020	29.013

The ML estimates were summarized with a percentile interval to describe the entire distribution of estimates. The MCMC was summarized with 95% Highest posterior density (HPD).

In Mimulus, there was strong evidence supporting variation in diversification rate as a function of floral size (our proxy for reproductive mode) (Table 2). We were able to reject the null model of no variation in diversification rate in favor of the alternative model, except when speciation was a linear function of floral size and directional trends in character evolution were included. For all models, however, the speciation rate was lower for smaller sized flowers (our proxy for species with higher selfing rates). The model with the best fit was a modal model that included directional character evolution (Table 2); this model fit significantly better than the modal model without directionality. In our analysis, the directionality parameter was negative (see “Materials and Methods” section), indicating that the mean trait value was decreasing over time (i.e., flower sizes were decreasing over time). In this model, small flowers had the lowest speciation rate, approaching zero (Appendix Figure A1, inset). This produced negative diversification rates (the constant extinction rate was estimated to be 10.6). Consistent with this, species with smaller flowers are often nested within clades of more moderately sized flowers (Appendix Figure A1). Finally, in this type of analysis speciation rates can often directly mirror the phenotype distribution and thus be noninformative. Two observations, however, indicate that our rate estimates did not strictly reflect the distribution of the phenotypic data (which is roughly bell-shaped; Appendix Figure A1). First, in our analysis, the maximum speciation rate did not occur at the trait value most often seen in the data, but instead was right-shifted. Second, the function relating speciation rate to floral size was asymmetrical, as large flowered species (that had fewer lineages represented in the trait distribution) still had positive diversification rates.

Table 2.

Summaries of models used to estimate differences in diversification rates correlated with selfing syndrome morphology

Model	Degrees of freedom	ln(Likelihood)	AIC
Constant speciation	3	−71.737	149.47
Linear speciation	4	−69.563	147.13*
Sigmoidal speciation	6	−65.949	143.90*
Modal speciation	6	−62.886	137.77*
Linear+directionatlity	5	−69.514	149.03
Sigmoidal+directionality	7	−64.359	142.72*
Modal+directionality	7	−60.912	135.82*†

The models are ranked in order of complexity (number of parameters). Asterisks represents significance at P < 0.05 using a likelihood ratio test comparing the null model with constant speciation to a given alternative model. Dagger represents significance at P < 0.05 using a likelihood ratio test comparing models with and without a directionality parameter.

Discussion

Cascade speciation is one predicted collateral outcome when reproductive character displacement occurs in some populations of a species but not others. It is often proposed that cascade speciation results from reinforcing selection acting on populations sympatric with a closely related congener. Thus, some conditions that favor cascade speciation will be shared with those that favor reinforcement. In our analyses earlier, we focused on factors influencing the likelihood and timing of reinforcement for this reason. Below we discuss the results of our model and comparative analyses, including what these say about the conditions favoring the likelihood of reinforcement and thereby cascade speciation. We conclude by addressing how these might differ between reinforcement and cascade speciation, and highlight the circumstances under which cascade speciation is most likely to be an influential contributor to speciation and diversification rates.

Individual and interaction effects of traits controlling nonrandom mating

In our model, we aimed to understand conditions that can favor reinforcement when considering two traits that affect nonrandom mating: assortative mating and selfing. We focused on these traits because they are potentially pleiotropic in their effects on local phenotypic divergence in response to reinforcing selection. Using simulations, we assessed how assortative mating and self-fertilization can contribute to reinforcement (increased reproductive isolation in sympatry resulting from exposure to deleterious hybridization) when they are allowed to evolve simultaneously. Our results are similar to those of Lorch and Servedio (2007), in that we found that alternative trait responses to reinforcing selection can interfere with each other when allowed to evolve simultaneously. However, interference in our simulations was asymmetric, with selfing more often prohibiting the evolution of assortative mating. This effect depended on the strength of inbreeding depression, which determined if selfing evolved primarily due to transmission advantage or a function of the cost of hybridization. In particular, when inbreeding depression is weak self-fertilization evolved quickly due to its 2-fold transmission advantage; as a result, selfing contributed more strongly to reproductive isolation, and prohibited the evolution of assortative mating. In scenarios with increased inbreeding depression, where selfing evolved in response to the cost of hybridization, we observed a more variable pattern (Figure 3): although selfing could still exclude assortative mating under some conditions (especially where probabilities of selfing were high and/or there was stronger selection against hybrids), under other scenarios both alleles would fix in roughly the same amount of time, or assortative mating could fix and exclude the selfing allele.

How reasonable is our model in simulating actual trait transitions that could change assortative mating or selfing phenotypes? Data suggest that the genetic basis of both traits can be relatively simple. Transitions to self-fertilization can arise through few large effect loci (L'Hernault et al. 1988; Schedl et al. 1988; Slotte et al. 2012; Baldi et al. 2009). Similarly, in some cases, genetic changes that can drastically alter assortative mating have a simple genetic basis: in insects, changes in single genes can produce changes in cuticular hydrocarbons (pheromones) that lead to assortative mating (Greenburg et al. 2003; Chung et al. 2014). In plants, simple changes can alter flower color and pollinator preference leading to reproductive isolation (Hopkins and Rausher 2011). These cases suggest that trait responses to reinforcing selection could be rapid.

Even more rapid responses might be expected for mating traits in which there is direct pleiotropic effects between signal and preference (i.e., where mating is determined by phenotype matching), which are the kinds of traits we have considered here. In our model, both assortative mating and self-fertilization had a simple genetic basis, were pleiotropic, and evolved rapidly. There are other classes of traits, including reproductive phenology and host species identity, for which single trait changes can also simultaneously influence both signal and preference, and it is these traits, which might be expected to be most responsive to selection for reinforcement and thereby cascade speciation. Interestingly, in most known animal cases, signal divergence must be followed by subsequent changes in preference, unless changes in signal are exploiting a pre-existing sensory bias in females (Endler and Basolo 1998). That is, changes in signal and preference often involve at least two independent changes and cannot occur simultaneously. This suggests that the genetic basis of assortative mating transitions might be more genetically complex in some circumstances, compared to selfing transitions. Moreover, when assortative mating has a complex genetic basis, including when signal/cue is not linked with mating preference, our observed pattern of self-fertilization precluding the evolution of assortative mating might be expected to be even stronger.

Regardless, self-fertilization has many characteristics that make it a candidate for rapidly promoting reinforcement and cascade speciation over “short term” evolutionary time scales, provided that inbreeding depression is not strong. Consistent with our population genetic model, we know of one example where there is suggestive evidence that inbreeding depression influences whether assortative mating versus selfing transitions evolve in response to selection for character displacement in sympatry. In particular, closely related species of Mimulus with different mating systems and floral size are observed to differ in their magnitude of inbreeding depression such that the more inbreeding species have lower genetic load (Carr and Dudash 1996), suggesting that the lack of inbreeding depression could have facilitated mating system transitions to selfing in these species.

Effects of selfing transitions on speciation and diversification rates

Our model suggested transitions to selfing had generally more rapid and stronger effects on reproductive isolation from reinforcing selection, especially under conditions of low to moderate inbreeding depression. If this is the case, one implication is that transitions to self-fertilization in response to reinforcement could increase speciation rates and diversification rates in a clade. To indirectly address this prediction, we conducted comparative analyses to assess the long-term consequences of transitions to self-fertilization for evolutionary diversification. Previous phylogenetic analyses have asked whether self-fertilization causes reduced diversification because selfing lineages could experience increased extinction due to inbreeding depression and/or a reduced potential for adaptation due to lower genetic variation (Goldberg et al. 2010; Ferrer and Good 2012; Ross et al. 2013). Because we were interested in possible dynamics when inbreeding depression was not as strong, we specifically asked: is reproductive mode associated with increased diversification (esp. increased speciation rates)? We focus on mating system transitions primarily because there is little equivalent comparative data for trait transitions that mediate assortative mating. We also do not more finely consider the biogeographic context of any specific species comparisons (i.e., evaluate differences among species pairs in allopatry versus sympatry). This is because, while reinforcement might produce a pattern of increased divergence in mating system between sympatric species pairs compared to allopatric species (Grossenbacher and Whittall 2011), this pattern would not necessarily be expected for cascade speciation because the products of the latter process would appear as mating system shifts between closely related allopatric species. Our empirical comparison is therefore limited to simply evaluating whether we see evidence for the longer term macroevolutionary consequences of mating system transitions on diversification rates. Evidence that these are positively associated could indicate that selfing transitions play a role in accelerating diversification in part because of their labile response to reinforcing selection and their possible contributions to cascade speciation.

In our sample of nematode taxa, analyses of a fraction of individual trees did yield significant differences in diversification rates attributed to differences in reproductive mode. In these cases, we observed increased speciation rates in uniparental lineages, consistent with the prediction from our population genetic model: if transitions to self-fertilization in response to reinforcement are important for overall diversification in a group, then these should be associated with increased speciation rates. These results are preliminary, as the phylogeny used here greatly undersamples the Nematode phylum and the majority of trees gave no support for asymmetrical speciation rates. They nonetheless suggest that shifts to self-fertilization in this group might promote diversification via reinforcement and cascade speciation. In contrast to the Nematoda, we found evidence that increased self-fertilization is associated with reduced diversification in Mimulus (consistent with other examples of long-term fate of self-fertilizing taxa in plants: Goldberg et al. 2010; Ferrer and Good 2012)—the opposite of the expectation that emerges from our population genetic model.

Can self-fertilization be a driver of speciation?

Taken together, the results from our population genetic model and comparative data analysis raise the question: How important are selfing transitions to speciation via reinforcement and/or cascade speciation? The answer depends on three factors that may differ between taxonomic groups, and the groups from our comparative data analysis (Nematodes and Mimulus) can be used as examples to explain how these factors affect differences in the association between mating system and diversification. The first factor is the frequency and importance of reinforcement and/or cascade effects to speciation, which is still unclear even though empirical examples of reinforcement and cascade speciation exist (Marshall et al. 2002; Coyne and Orr 2004). In groups where reinforcement and cascade effects are not important for speciation, there is no reason to expect that transitions to self-fertilization will be an important contributor to speciation. While data do not exist for sympatric/allopatric relationships for the Nematodes, previous studies in Mimulus indicate that sympatric character displacement of mating traits between closely related species is quite common. Reproductive character displacement has caused divergence in both floral size (a proxy for mating system; Grossenbacher and Whittall 2011) and pollinator attraction (Schemske and Bradshaw 1999; Grossenbacher and Stanton 2014)—suggesting that our failure to detect an association between reproductive mode and diversification in this group was not limited by the lack of reinforcement and/or cascade effects.

The second factor that might limit the importance of selfing on speciation is the negative effects of selfing, including increased likelihood of extinction, which might periodically or frequently outweigh the positive influence of selfing transitions on speciation. In particular, increases in self-fertilization have been hypothesized to reduce diversification via increased susceptibility to extinction (Fisher 1930; Muller 1932; Maynard Smith 1978; Nunney 1989; Goldberg et al. 2010), and phylogenetic analyses in plant systems have generally supported this prediction (Goldberg et al. 2010; Ferrer and Good 2012; but see Johnson et al. 2011). Differences in the propensity for self-fertilizing lineages to go extinct might differ between the Nematode phylum and Mimulus, especially given that most self-fertilizing nematodes are androdioecious (Denver et al. 2011; Gibson and Fuentes 2015). Rare outcrossing evnts could mitigate the negative effects of the largely uniparental reproductive mode of these taxa (Charlesworth 2009). There may also be differences in the distribution and/or magnitude of inbreeding depression in the Nematode phylum and Mimulus. According to our model results, selfing only contributed to reinforcement under low to moderate levels of inbreeding depression. Transitions to self-fertilization might occur more often in Nematoda if the typical genetic load (and therefore levels of inbreeding depression) is conducive to these transitions.

The third factor limiting the potentially positive impact of selfing on long-term diversification could be limitations on repeated transitions to self-fertilization which would in turn preclude repeated speciation events. In order for transitions to self-fertilization to repeatedly respond to reinforcing selection, lineages must have the ability to transition between different rates of self-fertilization. If transitions to higher self-fertilization were irreversible, additional instances of reinforcing selection would not act on daughter lineages because this trait would no longer be able to respond to additional bouts of reinforcing selection (i.e., selfing rates would remain permanently high, and nonresponsive to selection). This does not preclude speciation of self-fertilizing lineages via other means, but not in terms of increased selfing in response to reinforcing selection. Importantly, transitions to self-fertilization in plants are widely thought to be genetically irreversible (Igic et al. 2006, 2008; Tsuchimatsu 2010). In contrast, transitions in nematodes seem to arise from simple genetic changes, and transitions between reproductive modes appear to be relatively common in this group (Hill et al. 2006; Baldi et al. 2009; Denver et al. 2011). This implies that selfing lineages may have the potential to regain outcrossing: indeed, this may have occurred once in the well-studied Rhabditid group (Kiontke et al. 2004). Overall then, for nematodes, selfing could be a recurring response to reinforcing selection, whereas this is thought not to be the case in many plants, including Mimulus. Finally, we note that this limitation on the response of selfing to recurring reinforcing selection is not necessarily true of assortative mating, as this merely requires changes in reproductive signals and preferences. These are, in principle, infinite in their dimensionality, although in practice they are likely limited by physiological limitations on signal modalities and perception systems (Hohenlohe and Arnold 2010; Oh and Shaw 2013). In other words, there are many ways for signals/preferences to diverge, but only a single dimension in which selfing can change.

Implications for cascade speciation

In our study, we have focused on factors influencing the likelihood and timing of reinforcement, because understanding these can also reveal conditions conducive to cascade effects. First, just as for reinforcement, traits that are pleiotropic should disproportionately contribute to cascade speciation. Pleiotropy can occur if traits contributing to assortative mating are also under ecological divergence (“magic traits”; Gavrilets 2004; Servedio et al. 2011) or if a single assortative mating cue and preference are linked. When only a single genetic change is responsible for assortative mating, cascade speciation is a more likely outcome because allopatric and sympatric populations can rapidly diverge with subsequent limited gene flow between them. Rapid evolution clearly is important in sympatry (i.e., for reinforcement) because it reduces the time that local populations are vulnerable to ongoing (deleterious) gene flow. Indeed, models of speciation by sexual selection and models of reinforcement place significant emphasis on the linkage disequilibrium that builds between signals/cues and preference because this is what enables rapid evolution of assortative mating (Lande 1981; Kirkpatrick 1982; Servedio 2000; Kirkpatrick and Ravigne 2002). However, rapid evolution is also expected to facilitate the rapid divergence of allopatric and sympatric intraspecific populations, thereby also increasing the probability of cascade speciation in the face of potentially homogenizing gene flow within species (Hoskin and Higgie 2010; Yukilevich this issue).

Overall, then, these considerations imply that for traits to strongly contribute to cascade speciation they must evolve rapidly and will likely have a simple genetic basis, be pleiotropic (either linking mating cue/preference or ecological function) or be under strong sexual selection to build-up linkage disequilibrium between mating cue and signal. Indeed, the traits we focused on in our population genetic model fall under a particular class of magic traits, whereby mating cue and preference are directly associated via pleiotropy. Interestingly, almost none of the current empirical examples of cascade speciation focus on magic traits. Two studies focus on nonrandom mating traits that are unlikely to have pleiotropic ecological phenotypes or do not directly link mating cue and preference (amphibian song; Hoskin et al. 2005; Pfennig and Rice 2014), and potential ecolgical selection has been ruled out in two other systems (Higgie et al. 2000; Kozak et al. 2015). The remaining studies do not address whether assortative mating and cascade effects involve magic traits (Nosil et al. 2003; Jaenike et al. 2006; Porretta and Urbanelli 2012; Dyer et al. 2014), although the specific mechanisms involved (cuticular hydrocarbons in insects) are known to respond to both ecological selection and sexual selection (Greenburg et al. 2003; Chung et al. 2014). In contrast, several recent studies of reinforcement likely involve magic traits but cascade effects have not been examined in these cases. These include plant examples where traits mediating responses to reinforcement involve shifts to new habitat niches, including phenology, pollinator, and mating system shifts (Hopkins 2013). These observations suggest that it would be valuable to directly examine the potential impact of magic traits for current examples of cascade speciation, as well as to examine cascade effects for cases of reinforcement that potentially involve magic traits, including assortative mating via trait matching and/or transitions to selfing.