Establishment of a new sex-determining allele driven by sexually antagonistic selection.

The turnover of sex-determining loci has repeatedly occurred in a number of species, rather than having a diverged pair of sex chromosomes. We model the turnover process by considering a linked locus under sexually antagonistic selection. The entire process of a turnover may be divided into two phases, which are referred to as the stochastic and deterministic phases. The stochastic phase is when a new sex-determining allele just arises and is still rare and random genetic drift plays an important role. In the deterministic phase, the new allele further increases in frequency by positive selection. The theoretical results currently available are for the deterministic phase, which demonstrated that a turnover of a newly arisen sex-determining locus could benefit from selection at a linked locus under sexually antagonistic selection, by assuming that sexually antagonistic selection works in a form of balancing selection. In this work, we provide a comprehensive theoretical description of the entire process from the stochastic phase to the deterministic phase. In addition to balancing selection, we explore several other modes of selection on the linked locus. Our theory allows us make a quantitative argument on the rate of turnover and the effect of the mode of selection at the linked locus. We also performed simulations to explore the pattern of polymorphism around the new sex-determining locus. We find that the pattern of polymorphism is informative to infer how selection worked through the turnover process.

Introduction

Recent genome analyses have demonstrated that genetic systems that determine sex are more labile than previously thought and that the turnover of sex-determining loci has repeatedly occurred. In some clades such as teleost fish and amphibians, sex-determining loci differ among closely related species or even within a species (Bachtrog et al. 2014; Beukeboom and Perrin 2014). Frequent turnover should occur in species especially when sex is determined by a single sex-determining locus, rather than a pair of highly diverged sex chromosomes. Turnover should be initiated by mutation at another potentially sex-determining locus, which could become a new sex-determining locus while the polymorphism at the original sex-determining locus disappears.

Many theoretical studies have investigated the evolutionary process of such turnover (Bull and Charnov 1977; van Doorn and Kirkpatrick 2007, 2010; Kozielska ; Blaser , 2014; Veller ; Scott ; Saunders , 2019). A consensus has been established that, if a new sex-determination system has higher fitness than the old one, the new system could potentially override the old one. This explains why turnover hardly occurs in species with a diverged pair of sex chromosomes, such as the X/Y system in mammals and the W/Z system in birds. Theoretical examinations of turnover usually involves a two-locus system, under which turnover has to pass through a phase in which dimorphic sex-determining alleles segregate at both of the two sex-determining loci. A deterministic theory assuming a population with an infinite size (Bull and Charnov 1977) demonstrated that the system with higher fitness can be stably maintained by selection. It is indicated that the fitness advantage of the new locus over the existing one would be an important factor for the turnover of sex-determining loci.

A possible scenario that a new sex-determining locus confers a fitness advantage is that the new locus arises in close linkage to a locus under sexually antagonistic selection (van Doorn and Kirkpatrick 2007, 2010). This process is explained in Figure 1A. Initially, sex is determined by the original X/Y locus, while the A/a locus is another potential sex-determining locus on a different chromosome (autosome). That is, the A/a locus is monomorphic (fixed for allele a) so that it does not play a role in sex determination, but when allele A arises from allele a by mutation, it creates a new sex-determination system. We refer to allele A as a sex-determining allele. There is another polymorphic locus (B/b) that is located close to the A/a locus. The B/b locus is assumed to be subject to sexually antagonistic selection; for example, allele B is beneficial in males and allele b is beneficial in females. At this point, the B/b locus does not confer any advantage or disadvantage because there is no physical linkage to the sex-determining locus X/Y. If a sex-determining allele (A) arises by mutation at the A/a locus, then one of the possible outcomes is that this new dimorphic locus takes over the role of sex determination and the original X/Y locus becomes monomorphic.

Figure 1

(A) The three loci model used in this work. (B) Relationship between sexes and genotypes when allele A has a masculinizing effect. (C) Relationship between sexes and genotypes when allele A has a feminizing effect. The red and blue stars are given to the genotypes that determine sex under the ancestral and new systems, respectively. The genotypes with no stars arise in the phase of transition from the ancestral to the new system.

We are here interested in how often such turnover of sex-determining loci occurs. To understand this, it is crucial to theoretically describe the entire process, from the birth of a new sex-determining allele to its stable establishment. However, previous studies on this topic mainly by van Doorn and Kirkpatrick (2007, 2010) focused on the second half of the process (as explained below). The purpose of this work is to provide an analytical description for the first half, which largely determines how often turnover occurs under what conditions.

The entire process may be divided into two phases, which are referred to as the stochastic and deterministic phases. The stochastic phase starts when a new sex-determining allele A arises and continues until the frequency becomes high enough to escape from extinction due to random genetic drift. Then, the deterministic phase follows, in which allele A further increases in frequency by positive selection and genotype Aa becomes fixed in the heterogametic sex; in this way, the new A/a locus takes over the role in sex determination. The theoretical results currently available are for the deterministic phase, in which analytical treatment is quite straightforward because random genetic drift may be ignored. van Doorn and Kirkpatrick (2007, 2010) used a deterministic approach to describe the rate of increase in the frequency of allele A during the early stages of the deterministic phase. In contrast, the behavior of allele A in the initial stochastic phase is much more complicated and many factors are involved in it, which is the scope of this work. One such factor is the effect of random genetic drift, which is the major evolutionary force acting when the allele frequency is low. The linkage to B/b is also a very important factor. If allele A arises in linkage with allele B, allele A immediately benefits from positive selection because the haplotype A–B confers a selective advantage. On the other hand, if allele A arises in linkage with allele b, selection works against allele A because the haplotype A–b is deleterious. Recombination plays a crucial role in determining the rate at which the advantageous haplotype A–B is created and broken up. Another important factor is how selection works on the B/b locus. Depending on the parameter setting for the effect of the B/b locus on fitness, selection operates in various modes (see Figure 2): selection works for or against allele B, or even in some parameter space, balancing selection works to maintain the two alleles at intermediate frequencies. It is easy to imagine that this parameter setting largely affects the fate of the sex-determining mutation.

Figure 2

Various modes of selection depending on dominance and selection coefficient at the B/b locus when locus X/Y determines sex. The net effect of selection at locus B/b is given by the average of the two sexes. If the dominance and strength of selection are similar in males and females but the directions of selection are opposite, the fitness of allele B equals the fitness of allele b (A). Positive selection works when the average fitness of the three genotypes genotype (shown in black) is in the order of b/b < B/b < B/B (B, C), while negative selection works when the order is b/b > B/b > B/B (D, E). Balancing selection works such that the average fitness is in the following order: b/b < B/b > B/B (F).

The purpose of this work is to theoretically understand how often turnover of a new sex-determining A/a locus occurs. We here mathematically describe the probability that a newly arisen sex-determining locus turns over the old one, which is referred to as the establishment probability. We found that the establishment probability is markedly high in the case of balancing selection, while it is very low in other modes of selection. It is indicated that the mode of selection at the B/b locus is critical in determining the fate of a newly arisen sex-determining allele. Therefore, to understand the rate of turnover, it is necessary to know how many sexually antagonistic loci are under balancing selection. Our simulations provide insight into how to distinguish the mode of selection from the pattern of polymorphism around the sex-determining locus.

Model

We use a discrete-generation model of a diploid species with population size N. Three loci are considered in the model (Figure 1A). One is the ancestral sex-determining locus, which is located on the sex chromosome where males have genotype XY and females have genotype XX. Although male heterogamety (XY system) is assumed here, our model can also handle female heterogamety (ZW system) by swapping males and females. The model includes another sex-determining locus A/a on the autosome, at which the initial state is that allele a is fixed, so that the locus does not play a role in sex determination. Then, we consider that masculinizing or feminizing allele A just arises by mutation at locus A/a. We are interested in how this new sex-determining allele A behaves in a finite population and how often it spreads and eventually enables the A/a system to override the old X/Y system.

In our model, if allele A has a masculinizing effect, its turnover does not change the heterogamety sex and involves four genotypes (case 1, Figure 1B). If allele A has a strongly feminizing effect (i.e. strong enough to make XYAa a female) (van Doorn and Kirkpatrick 2010), the turnover changes the heterogamety system and involves six genotypes (case 2, Figure 1C). In either case, to evaluate the effect of the B/b locus, we assume that the levels of fitness of the new and old systems are identical. Therefore, the fate of allele A is mainly determined by the selection effect of another linked locus B/b. It is assumed that the recombination rate is r between the A/a and the B/b loci. We ignore mutation at the A/a locus to trace the fate of a single mutation, whereas at the B/b locus, recurrent mutation is allowed between alleles B and b such that this locus should be under selection-mutation balance. The mutation rate from allele B to allele b is denoted by u and v denotes the reverse mutation rate from allele b to allele B. The frequency of allele B is denoted by p. The fitness of genotypes BB, Bb, and bb is given by , and 1 in males and , and 1 in females, respectively. As we assume allele B is beneficial for allele A, we set and when allele A has a masculinizing effect (case 1), and and is set when allele A has a feminizing effect (case 2).

Data availability

The authors state that all data necessary for confirming the conclusions presented in the manuscript are represented fully within the manuscript. Codes used for numerical analyses and simulation experiments are available at https://github.com/TSakamoto-evo/sex_codes_2020.

Supplementary material is available at figshare DOI: https://doi.org/10.25387/g3.13300967.

Results

We derive the probability that a single sex-determining mutation at the A/a locus spreads in the population so that the A/a locus becomes the new sex-determining locus and the old X/Y locus becomes monomorphic under the three-locus model illustrated in Figure 1A. This probability is essentially identical to the probability that the new sex-determining mutation successfully increases in frequency by avoiding extinction immediately after its introduction, as pointed out by van Doorn and Kirkpatrick (2007, 2010). Once the frequency of the new sex-determining mutation increases, it is very likely that it further increases to an intermediate frequency so that the sex-determining locus transitions from the X/Y locus to the A/a locus. This is because our model assumes that the presence of the B/b sex antagonistic locus provides a benefit only for the new A/a system. In the following, we derive this probability of the successful spread of allele A, which is referred to as the “establishment probability”. The “establishment” means that the new system is used by all individuals in the population (i.e. the new system is fixed but alleles A and a coexist stably), rather than the old and new systems coexisting in the population. The establishment probability is derived separately for the two cases (cases 1 and 2).

Case 1: turnover without changing the heterogametic sex

We first consider case 1, where allele A has a masculinizing effect (Figure 1B). For allele B to be beneficial for allele A, we assume and in this section. We derive the probability of allele A escaping immediate extinction by using the branching process. Let be the establishment probability of allele A that arises in linkage with allele i at the B/b locus when the frequency of allele B is p0. If allele A links with male-beneficial allele B, allele A is favored by linked selection. On the other hand, if allele A links with female-beneficial allele b, allele A is disfavored by linked selection. By denoting the frequencies of haplotypes A–B and A–b by x and x, respectively, and ignoring second-order terms in and v, the expected changes of the frequencies of A–B and A–b in one generation are given by where and are functions defined such as and (for details, see Appendix B). It is interesting to point out that Equation (1) involves only selection parameters in males, not those in females. This means that directly depends on selection among males but does not necessarily mean that selection does not work in females. Selection in females is involved because it affects p, the frequency of allele B.

In the following, we first derive the establishment probability for a special case where selection at the B/b locus does not provide any systematic pressure on p. In this case, p does not change rapidly, so we can treat it as a constant, at least in the timescale of a newly arisen allele escaping from initial extinction (see Appendix C). With this assumption, we can obtain the establishment probability as a solution of a cubic equation, which is given by a function of p0, the frequency of allele B when allele A arises. Next, we consider a more general case where p changes, from the initial value p0 to the equilibrium value . In this case, by contrast, we show that the establishment probability is given by a function of both p0 and , of which the effect of is quite large. In either case, we first obtain the establishment probability conditional on p0, and then we derive the unconditional establishment probability by incorporating the stationary distribution of p0.

Establishment probability when a constant p can be assumed

We consider a case where we can assume p does not change significantly in the timescale in which we are interested in. In other words, the expected change of p in one generation, , is assumed to be as small as , where M is given by: if the second-order terms of , and v are ignored. This assumption holds when and , so that the levels of selection in the two sexes are of the same strength, but work in opposite directions. This mode of selection is referred to as “equalizing selection”. We assume that , and r are so small that their second-order terms can be ignored. Then, following the branching process approximation (Barton 1987), it is straightforward to show that the two establishment probabilities satisfy:

(for details, see Appendix C). Notably, similar models were used to analyze the effect of a linked allele on the establishment of locally beneficial alleles (Aeschbacher and Bürger 2014) or the effect of population structure on the establishment of a beneficial mutation (Pollak 1966; Barton 1987; Tomasini and Peischl 2018; Sakamoto and Innan 2019). Equation (3) can be reduced to a cubic equation and be solved analytically (for details, see Appendix D). It is important to note that establishment is promoted by selection if is positive; otherwise, allele A is likely selected against and goes extinct. Whether depends on the leading eigenvalue of the matrix in Equation (1). When the leading eigenvalue is positive, and are given by: and where

When r is small, they can be approximated in quite simple equations:

We performed simple forward simulations in a Wright–Fisher population to check the accuracy of our derivation (for details of the simulations, see Appendix A). We confirmed that Equation (3) is in excellent agreement with the simulations for a wide range of the parameters. Some of the results are shown in Figure 3, where the two establishment probabilities, and , are plotted by assuming . Through this work, the mutation rates at locus B/b are fixed to be quite small values, , unless otherwise mentioned. Note that the effect of mutation rate is small in the establishment process unless the mutation rate is very large (but the mutation matters when the stationary distribution of allele B is considered, as demonstrated below).

Figure 3

Establishment probability of allele A for different dominance and recombination rates. Three dominance coefficients are assumed: (A) , (B) and (C) . The other parameters are as follows: . Error bars on the red and blue circles represent the 95% confidence interval, but they are too small to be seen.

We first focus on the case of r = 0, that is the A/a and B/b loci are completely linked, to investigate the effect of dominance (h). Suppose that allele B is recessive (h = 0, left panel in Figure 3A). Then, a newly arisen allele A can benefit from the B/b locus only when it arises in a BB homozygote. In such a case, increases as p0 increases to (plotted in red), where the effect of sexually antagonistic selection is maximized. decreases as p0 decreases from 0.5 to 1, making a symmetric function. With the assumption of no recombination, it is obvious that for any p0 (plotted in blue). The weighted average of and (i.e. ) is plotted in black.

In contrast, in the dominant case (h = 1, left panel in Figure 3C), for a small p0 is quite high because a newly arisen allele A in linkage with allele B is immediately selected for, regardless of the genotype at the B/b locus. This selection works particularly efficiently when B is so rare that the selective advantage of A–B haplotypes is large in comparison with the population fitness. Therefore, is given by a monotonically increasing function with decreasing p0, but as p0 decreases the probability that allele A arises in linkage with allele B decreases; therefore, the weighted average has a peak in the middle. An intermediate pattern is observed in a case of partial dominance (h = 0.5, left panel in Figure 3B). Similar results were obtained for other values of selection intensity as long as (not shown).

We next consider the effect of recombination. The approximations given by Equation (6) agree overall with the simulation results as long as r is small. As the recombination rate increases, decreases because the association with allele B becomes weaker. When r is relatively small, increases as the recombination rate increases because recombination gives a chance to link with allele B and the A–B association may be preserved if further frequent recombination does not break the association. However, this benefit does not hold for a large r, and decreases as the recombination rate increases because further recombination prevents the stable linkage of allele A and allele B and reduces the benefit of linkage.

Establishment probability when p changes

We next consider the case where p can change during the establishment process due to selection and mutation (i.e. ). To derive and , we use a continuous time approximation of the branching process (Barton 1995). With a deterministic approximation for M (), the two establishment probabilities satisfy:

(for details, see Appendix C). This differential equation can be solved numerically by setting an initial condition, for which we use the establishment probability of allele A that arises when p is at stable equilibrium . and can be numerically computed by Equation (3) because we can assume p does not change significantly around . Technically, we cannot use the exact values of and as an initial condition because they violate the assumption of . To avoid this problem, assuming a very small ε, we use and as an initial condition, which does not markedly affect the numerical solutions as long as ε is small.

When p is not constant, both p0 and play an important role in determining and . The relative contribution of can be large when selection is strong and p0 is not far from , so that p quickly approaches . In such a case, and cannot be large when and are very small. This argument can be explained as follows. Let us consider the fate of the descendants of an allele A that arises. After the mutation arises, p changes and finally reaches around . Denote the numbers of haplotypes A–B and A–b when p reaches as X and X, respectively. Then, the establishment probability is approximately given by: unless the population is very small.

Because largely depends on the mode of selection on the B/b locus (see Figure 2), we here consider and under three modes of selection separately (i.e. balancing, negative, and positive selection on allele B). For each mode of selection, we performed extensive forward simulations to check the performance of Equation (7). It was found that and computed by Equation (7) well agreed with the simulation results for a wide range of parameter space, and representative cases are shown in Figures 4–6 (see also Supplementary Figures S1 and S2).

Figure 4

Establishment probability for the case of balancing selection on allele B. Parameters are assumed to be . Error bars on the red and blue circles represent the 95% confidence interval, but they are too small to be seen. For full version, see Supplementary Figure S1.

First, we consider the case of balancing selection. To make a realization of balancing selection, it is necessary to set h large enough to secure the average fitness of B/b heterozygote higher than 1. When h is large, a newly arisen allele A would be immediately selected for together with allele B. Therefore, the overall behavior of and could be quite similar to that in the case of a high h when p does not change (e.g.Figure 3C where h = 1 is assumed). This is demonstrated in Figure 4, where we assume (as a consequence, ), while all other parameters are the same as those used in Figure 3. To emphasize the difference from the case of a constant p, this figure also shows and computed by Equation (3) for comparison. The major difference from Equation (3) is that, when is lower than that in the case of constant p (the red broken lines in Figure 4) because the selective advantage of haplotype A–B would be reduced by the rapid increase in the frequency of allele B by selection (i.e. both haplotypes A–B and a–B increase). is overall very low, similar to the case of a constant p (Figure 3C). On the other hand, when is larger than that in the case of constant p because the number of males having allele B decreases, with which haplotype A–B has to compete. Supplementary Figure S1, B and C, is for weaker selection coefficients ( and 0.002), where and are overall reduced and the difference from Equation (3) is small perhaps because p does not change rapidly with weak selection.

We next consider the case of negative selection, where p would move from p0 to , which is usually very low. If a very low is assumed, from Equation (3), we can approximate and as:

These equations mean that the establishment of allele A is very unlikely when . Thus, the major difference from the case of a constant p is that, as the recombination rate increases, the establishment rate decreases to around the threshold (see Appendix E for the behavior for large r). This is demonstrated in Figure 5, where are assumed such that the comparable result for the case of a constant p is Figure 3B with the same selection parameters for males (all other parameters are identical). In Figure 5, holds at r = 0.01, which works as the threshold. When r is smaller than this threshold (r = 0.0 and 0.005 in Figure 5), and are roughly in agreement with those in Figure 3B, although and are slightly higher than the case of a constant p, especially when p0 is not small. The situation changes dramatically as the recombination rate exceeds the threshold (i.e. r = 0.02 in Figure 5): and decrease to as low as (i.e. the neutral expectation for a sex-determining allele) where there is no benefit of linked selection and only drift-driven establishment occurs in a nearly neutral fashion. In contrast, in Figure 3B, unless p0 is close to 0 or 1. Such strong reduction of when r exceeds (Figure 5) can be explained as follows. A newly arisen allele A benefits from linked selection when it arises in association with allele B. Once the linkage is broken by recombination, allele A has almost no chance to recombine back to link to allele B because the frequency of allele B is very low. Therefore, after allele A loses linkage with allele B, there would be no selection for allele A so that the establishment of allele A has to rely on random genetic drift. Supplementary Figure S2, B and C, shows the results for weaker selection ( and 0.008), where the general pattern is similar to that in Figure 5, while and are overall reduced.

Figure 5

Establishment probability for the case of negative selection against allele B. Parameters are assumed to be , N = 10,000, and . Error bars on the red and blue circles represent the 95% confidence interval, but they are too small to be seen. For full version, see Supplementary Figure S2.

We finally consider the case of positive selection, where is generally very large (i.e. ). Unless p0 is small, p increases very quickly to , that is the B/b locus is almost fixed for allele B. Therefore, even when allele A arises in association with allele B, allele A does not benefit from linked selection on locus B, resulting in a very low . Particularly when allele A at , random genetic drift plays a role in the establishment process. The theoretical treatment for this situation is shown in Appendix E. Figure 6 shows and for the case of positive selection assuming . It is demonstrated that allele A can spread efficiently with the help of linked selection for allele B only when p0 is small and r is so small that the initial association between A and B can be maintained for a while. The performance of Equation (7) is not as good as that in the cases of balancing selection and negative selection. It appears that Equation (7) underestimates the establishment probability because our derivation based on the branching process ignores establishment events occurring in a nearly neutral fashion.

Figure 6

Establishment probability for the case of positive selection for allele B. Parameters are assumed to be , N = 10,000, and . Error bars on the red and blue circles represent the 95% confidence interval, but they are too small to be seen.

Unconditional establishment probability

In the above, we consider the establishment probability as a function of the initial frequency of allele B, p0. We are here interested in the unconditional establishment probability, which is the weighted average over the stationary distribution of p0. Following Wright’s formula (Wright 1931), the stationary distribution of p0, g(p), is given by: where and C is determined such that (Connallon and Clark 2012). It is well recognized that this formula works very well when the selection intensities, s and s, are relatively small so that their second-order terms are negligible (Crow and Kimura 1970; Ewens 2004). An intriguing exceptional case is when selection is weak but the absolute values of the selection intensities are not small. In the previous section, we considered a case where selection in males and that in females are well balanced when and , irrespective of the absolute values of s and s. In such a special case of equalizing selection, the stationary distribution may be better given by:

Then, the unconditional establishment probability of allele A, , is given by: where and are given by Equation (3) for and , and otherwise by Equation (7).

We can obtain an approximation of the establishment probability in a simple form for several special cases of r = 0. When sexually antagonistic selection works as balancing selection, the establishment probability for r = 0 is approximated by:

It is implied that is on the same order of magnitude as s.

In the case of negative selection, assuming that selection is much stronger than mutation (u and v) and h is not very small, we have and . Furthermore, because and , Equation (7) for r = 0 is roughly simplified:

Under these approximations, is given by indicating that is on the order of . For selection to be dominant over random genetic drift, is required.

If sexually antagonistic selection works as positive selection and the mutation rate is low, linked selection no longer works and establishment is driven by random genetic drift, as discussed above. Then, the establishment probability is roughly given by . In such cases, sexually antagonistic selection does not markedly increase the establishment probability.

The unconditional establishment probability computed by Equation (12) is plotted for the four modes of sexually antagonistic selection in Figure 7. The approximations for r = 0 [Equations (13) and (14)] are also presented by triangles on the y-axis, which show excellent agreement with the exact formula and simulation results. Three different values of the mutation rate are considered ().

Figure 7

Establishment probability of a masculinizing allele for different modes of sexually antagonistic selection. N = 100,000 and u = v are assumed. Other parameters are (A) , (B) , (C) , and (D) . Similar results were obtained for N = 10,000. Error bars for circles represent the 95% confidence interval.

The establishment probability is in general highest when balancing selection works (Figure 7A). This is because an intermediate p provides both the benefit of linkage and a higher chance to link with allele B to allele A. Because the mutation rate at locus B/b does not markedly influence the stationary distribution of p, the establishment probability does not depend on the mutation rate either.

When negative selection works at locus B, the establishment probability is quite low (Figure 7B) because negative selection keeps allele B very rare. Mutation from allele b to B contributes to the establishment of allele A in two ways. First, it increases the frequency of beneficial allele B, resulting in a higher chance that allele A acquires linkage with allele B by recombination. Second, it increases the probability that haplotype A–b mutates to A–B. As a consequence, as the mutation rate v increases, the establishment probability becomes larger.

When positive selection works, the establishment probability is very low for all three mutation rates (Figure 7C). This is because allele B is already fixed and linkage with allele B is no longer beneficial. Allele A may benefit from linked selection only when u is extremely high.

Thus, if selection is directional, is very low or high (under negative or positive selection, respectively), which does not markedly help the establishment of allele A. On the other hand, if balancing selection maintains alleles B and b in an intermediate frequency, allele A can take advantage of it. An intermediate situation is when equalizing selection works (Figure 7D, where the process is nearly neutral because and ). Because p fluctuates between 0 and 1 by random genetic drift, allele A can likely benefit from locus B/b if it arises when p is intermediate. This is why the establishment probability is largely affected by the mutation rate at locus B/b, which determines how often mutation is produced.

It is interesting to note that very tight linkage does not necessarily increase the unconditional establishment probability (e.g.Figure 7, B and D). With an increasing the recombination rate, in general, increases while decreases because recombination enhances exchange between haplotype A–B and haplotype A–b. Because the unconditional probability is given by the weighted average of and , there appears to be an optimal recombination rate to maximize .

The pattern of neutral polymorphism after turnover

To investigate the effect of the mode of selection at locus B/b on the pattern of neutral polymorphism in a surrounding region, we performed forward simulations under the infinite-sites model. The spatial distributions of the nucleotide diversity after turnover in a typical run are plotted for each of the four modes of selection in Figure 8. The amount of nucleotide variation is measured by the average pairwise differences (π) standardized by the neutral expectation ( where μ is the neutral mutation rate). Haplotype i ({A, a}) is denoted as the haplotype with allele i, and Figure 8 plots three π values: for π within haplotype A (red line); for π within haplotype a (blue line); and for π between haplotypes A and a (black line). Let us first focus on the case of t = 0 (i.e. just after the turnover has completed). When balancing selection works, has a striking peak around the locus B/b (Figure 8A) and a weaker peak is observed for and due to recombination between the two loci. This is because the B/b polymorphism had been maintained for a very long time by balancing selection before the turnover occurred, which is not observed in the other modes of selection (Figure 8, B–D).

Figure 8

The temporal dynamics of nucleotide diversity after turnover of the sex-determining locus. Results for a single simulation run with N = 10,000 are shown for each mode of selection. The spacial distributions of the normalized , and are shown in red, blue, and black, respectively, along the relative position in the (0,1) interval (also rescaled in units of 4Nr in parentheses). In the simulation, locus A/a is located at relative position 0.25 (vertical orange lines) in the (0,1) simulated region, while locus B/b is located at relative position 0.75 (vertical green lines). t is denoted as the number of generations since the new sex-determination system was fixed (i.e. since the old sex-determining locus X/Y became monomorphic). The recombination rate of the entire region was assumed to be 0.0002 such that the recombination rate between loci A/a and B/b 0.0001. The selection parameters used for each mode of selection were (A) , (B) , (C) , and (D) .

As time passes, in all four cases, neutral mutations start to accumulate around the A/a locus, making a novel peak of divergence between haplotypes A and a (i.e. ). The growth of the peak at the A/a locus proceeds while maintaining the peak at the B/b locus in the case of balancing selection (Figure 8A). The pattern is as predicted by Kirkpatrick and Guerrero (2014). In contrast, in the other three cases (Figure 8, B–D), a peak newly arises at the B/b locus and grows along with the peak at the A/a locus. Thus, the patterns are similar in the cases of negative, positive and equalizing selection, whereas balancing selection is unique in that the peak at the B/b locus already exists, which is much higher than that at the A/a locus shortly after the turnover.

Case 2: turnover with changing heterogametic sex

We next consider case 2, where allele A has a feminizing effect so that the heterogametic sex changes from male to female. That is, case 2 assumes for allele B to be beneficial for allele A. As in case 1, allele A can spread if allele A successfully avoids extinction shortly after it arises (van Doorn and Kirkpatrick 2010). Therefore, we again use the approximation of the branching process, following case 1. By ignoring the second-order terms in , and v, the expected change of the frequency in one generation is given by: where and (see Appendix B for details).

By comparing Equation (15) with Equation (1), we notice that the two equations are identical if we replace h by h and s by s. It is indicated that the above arguments for case 1 are also applicable to case 2 by changing these variables. When equalizing selection works and p does not change rapidly (i.e.  and ), the establishment probability is given by Equation (3). On the other hand, when p changes when allele A is rare, the establishment probability depends on both p0 and stable equilibrium . If balancing selection works, the establishment probability is given by Equation (7). When allele B is subject to negative selection, the establishment probability depends on whether r is larger or smaller than the threshold that is given by . When , the establishment probability is given by Equation (7). Whereas, when so that the establishment of allele A would be drift-driven once p reaches : therefore, could be as small as . When allele B is subject to positive selection, we have . Consequently, the establishment process is driven by random genetic drift after p reaches . Therefore, as long as p0 is near and positive selection is strong, the establishment probability is as low as (for more details, see Appendix E). Thus, the process in case 2 can be well described by the equations developed for case 1, as is demonstrated in Supplementary Figures S3–S6. The unconditional establishment probability is also given by Equation (12) (see Supplementary Figure S7).

DISCUSSION

In some clades such as teleost fish and amphibians, sex is often determined by a single locus rather than heteromorphic sex chromosomes. In such species, turnover of the sex-determining locus occurs so frequently that genetic divergence around the locus does not proceed. There are many factors that potentially promote the turnover of sex-determining loci, such as random genetic drift (Bull and Charnov 1977; Veller ; Saunders ), deleterious mutation load (Blaser , 2014), sexually antagonistic selection (van Doorn and Kirkpatrick 2007, 2010), sex ratio bias (Kozielska ), and haploid selection (Scott ). Recently, van Doorn and Kirkpatrick (2007, 2010) pointed out that turnover of sex-determining loci could be enhanced by linked selection. That is, a new sex-determining allele (allele A in our model) can be beneficial when it arises near a locus under sexually antagonistic selection (locus B/b). This article is aimed at understanding how often turnover of sex-determining loci occurs with the help of linked selection at a nearby locus. Previous studies mainly by van Doorn and Kirkpatrick (2007, 2010) focused on the deterministic phase to obtain the rate of increase in the frequency of allele A, which is not sufficient to address the question on the rate of turnover, as mentioned in the Introduction. We here provide a full theoretical description of the behavior of allele A from when it newly arises to its establishment, where both the initial stochastic and following deterministic phases are taken into account. We provide some technical comments in Appendix F to provide intuitive insights on how to understand the results of van Doorn and Kirkpatrick (2007, 2010) in our framework.

Our theory shows that the establishment probability is given by a function of the initial frequency of allele B, p0, and the equilibrium frequency, . It is demonstrated that the establishment probability largely depends on the mode of selection on allele B, which determines (Figure 7). The establishment probability is relatively high when balancing or equalizing selection works because polymorphism at locus B/b increases the benefit of linkage. In contrast, when directional selection works (either positive or negative), linked selection does not significantly help establishment. When negative selection works, the establishment probability is low because the frequency of allele B is so low that allele A has difficulty linking with it. When positive selection works, the establishment probability is also low because beneficial allele B should be almost fixed. The effect of p0 appears to be smaller unless p0 is very different from .

Our results demonstrate that the fate of newly arisen sex-determining mutation is mainly determined in the early phases (i.e. stochastic phase), where random genetic drift is the major evolutionary force. It is indicated that the stochastic phase plays an important role in understanding the evolutionary process of turnover of sex-determining loci. In the stochastic phase, p0 is a very important factor affecting the initial behavior of the new mutation, and the density distribution of p0 largely depends on the mode of selection at the B/b locus. The establishment probability of allele A is very low when directional selection works at the B/b locus, indicating that the presence of loci under sexually antagonistic selection could significantly enhance turnover of sex-determining loci only when sexually antagonistic selection works in the form of balancing selection.

Our theory has great implications for the rate of turnover of sex-determining loci in natural populations. An important biological question is how sexually antagonistic loci help the turnover of sex-determining loci on the genomic scale. One might think that most turnover occurs with the help of sexually antagonistic loci under balancing selection, because the establishment probability is markedly high when balancing selection works at the B/b locus (see also van Doorn and Kirkpatrick 2007, 2010). On one hand, one might consider that there may not be a large number of sexually antagonistic loci under balancing selection in a genome, so that their relative contribution might be small. Alternatively, there might be a negligible contribution of sexually antagonistic loci under other modes of selection (i.e. equalizing and directional selection). Even if the establishment probability is not high for each, on a genomic scale, their cumulative effect may not be small. To answer the question on the relative contribution of linked selection, we need to know how many sexually antagonistic loci exist in the genome, and what mode of selection is working. While empirical studies may be powerful to address this, it is also interesting to look at polymorphism data surrounding the sex-determining locus. Our simulations (Figure 8) provide insight into how to distinguish the mode of selection at a linked locus. Shortly after turnover, if divergence between male and female haplotypes is restricted in a very narrow region around the sex-determining locus, it may be likely that the sex-determining allele has become established with no help from linked selection (i.e.Myosho ; Kamiya ; Koyama ). On the other hand, if a highly diverged region spreads surrounding the sex-determining locus, then either the sex-determining allele has become established together with a linked allele at a sexually antagonistic locus, or sexually antagonistic loci arose after the establishment of the sex-determining locus.

Funding

This work was partly supported by grants from the Graduate University for Advanced Studies, SOKENDAI, and Japan Society for the Promotion of Science (JSPS) to HI and TS (19H03207, 20J21760).

Conflicts of interest: None declared.