How does male-male competition generate negative frequency-dependent selection and disruptive selection during speciation?

Natural selection has been shown to drive population differentiation and speciation. The role of sexual selection in this process is controversial; however, most of the work has centered on mate choice while the role of male-male competition in speciation is relatively understudied. Here, we outline how male-male competition can be a source of diversifying selection on male competitive phenotypes, and how this can contribute to the evolution of reproductive isolation. We highlight how negative frequency-dependent selection (advantage of rare phenotype arising from stronger male-male competition between similar male phenotypes compared with dissimilar male phenotypes) and disruptive selection (advantage of extreme phenotypes) drives the evolution of diversity in competitive traits such as weapon size, nuptial coloration, or aggressiveness. We underscore that male-male competition interacts with other life-history functions and that variable male competitive phenotypes may represent alternative adaptive options. In addition to competition for mates, aggressive interference competition for ecological resources can exert selection on competitor signals. We call for a better integration of male-male competition with ecological interference competition since both can influence the process of speciation via comparable but distinct mechanisms. Altogether, we present a more comprehensive framework for studying the role of male-male competition in speciation, and emphasize the need for better integration of insights gained from other fields studying the evolutionary, behavioral, and physiological consequences of agonistic interactions.

Introduction

The process of speciation is fundamental to understanding the origin of biodiversity (Schluter 2001; Weissing et al. 2011). During ecological speciation, divergent natural selection can favor the evolution of reproductive isolation between populations by adaptation to different environments or ecological niches (Dieckmann et al. 2004; Schluter 2009). Ecological speciation can either be facilitated or constrained by sexual selection via mate choice, an area that has recently received more attention (Safran et al. 2013). An important component of sexual selection is male–male competition for mates, which often involves direct combat between rival males (Andersson 1994). Compared with mate choice, the role of male–male competition in population differentiation and speciation is relatively understudied. This is surprising because it is well known that agonistic interactions can lead to rapid evolutionary change in traits that are used in competitive signaling and male contest (e.g., body color, body size, and weapons) (Berglund et al. 1996; Gross 1996; Wong and Candolin 2005).

Adaptive speciation requires the following mechanisms (Weissing et al. 2011): (1) diversifying selection that can fuel the origin of phenotypic variation; (2) a mechanism for maintaining this variation; (3) a sustained disruptive selection regime against hybrid phenotypes to promote the evolution of assortative mating and reproductive isolation; and (4) a mechanism that can facilitate the stable coexistence of daughter species. It is now increasingly appreciated that male–male competition for reproductive opportunities plays a key role at various stages of the speciation process: (1) male–male competition can generate diversifying selection on male traits, which could spark the formation of new species with novel traits; (2) negative frequency-dependent selection arising from male–male competition can stabilize a disruptive selection regime; (3) male competitive traits can be under disruptive selection, enhancing the evolution of reproductive isolation; and (4) male–male competition can facilitate the coexistence of daughter species (Seehausen and Schluter 2004; van Doorn et al. 2004; Dijkstra and Groothuis 2011; Qvarnström et al. 2012). Negative frequency dependence is generated when there is disproportionately strong contest competition for potential mates and other resources between phenotypically similar individuals. This can occur when males bias aggression toward similar phenotypes with whom they are expected to compete for the same pool of females. The competitive advantage of rare males is expected to enable rare and potentially novel male competitive phenotypes to invade, and stabilize the coexistence of divergent phenotypes during and after speciation. Male–male competition can also be a source of disruptive selection when, for example, intermediate phenotypes resemble both parental phenotypes and thus receive twice the aggression compared with the parental phenotypes (Seehausen and Schluter 2004). Assuming that receiving and initiating more aggression is costly, this form of selection against intermediate phenotypes (postzygotic isolation) can cause agonistic character displacement between coexisting phenotypes and facilitate the evolution of prezygotic isolation between divergent phenotypes (van Doorn et al. 2004; Dijkstra et al. 2011; Qvarnström et al. 2012). Male–male competition can also drive divergence in allopatry (Tinghitella et al. forthcoming; Mendelson et al. 2014), but here we mostly focus on how it influences the process of speciation with gene flow.

Individuals, morphs, or species often vary in competitive traits (morphological, behavioral), and these differences often lead to asymmetries in competitive ability (Peiman and Robinson 2007; Dijkstra and Groothuis 2011; Qvarnström et al. 2012; Grether et al. 2013). Such differences in competitive ability may lead to competitive exclusion or displacement of the less competitive phenotype to a non-preferred habitat. However, the relative fitness benefits of elevated competitive ability are most likely negatively frequency-dependent. For example, the more competitive phenotype can be socially dominant over the less competitive phenotype but only when the more competitive phenotype is rare. How negative frequency dependence arises from social interactions is often unclear.

Although superior fighting performance often translates into increased social dominance and reproductive success, the relationship between competitive ability and fitness is complex. Individuals often face a trade-off between behaving aggressively to attain high dominance status and other aspects of fitness such as immune function (Pryke et al. 2007; Dijkstra et al. 2010). Fitness trade-offs are essential to models of diversification but the role of male–male competition in these trade-offs needs more attention.

Our goals are 3-fold. First, we underscore that in order to understand the role of male–male competition in speciation, we need to consider how male–male competition can result in negative frequency-dependent and disruptive selection. Secondly, we discuss how fitness trade-offs involving male competitive traits can cause disruptive selection and thereby curtail gene-flow between divergent populations. Thirdly, ecological competition is an important driver of speciation (Dieckmann and Doebeli 1999; Weissing et al. 2011), and we highlight how aggressive competition for ecological resources or mates influences speciation via distinct but comparable mechanisms. While we are aware of the fact that disruptive selection occurs because competition is negative frequency-dependent (but see Keagy et al. 2015), we discuss the 2 processes as separate issues. This is because negative frequency-dependent selection is (more) important for stabilization of polymorphisms, while disruptive selection is more important for the evolution of diversity and reproductive isolation.

Our review is focused on male–male competition, but it is important to note that females may also compete aggressively for mates (Rosvall 2011) and female–female competition could facilitate sympatric speciation by sexual selection (van Doorn et al. 2004).

Male–Male Competition Can Generate Negative Frequency-Dependent Selection

Speciation driven by natural and/or sexual selection requires heritable variation in ecological and mating traits, a stable regime of disruptive selection favoring the evolution of assortative mating, and finally, the stable coexistence of newly formed daughter species (Weissing et al. 2011). The process can be driven by resource competition acting on both sexes leading to disruptive selection on ecological traits such as beak morphology, habitat preference, or body shape (Schluter 2001; Pfennig and Pfennig 2010; Weissing et al. 2011; Nosil 2012). Divergent ecotypes or niche specialization can lessen competition among organisms (Dieckmann and Doebeli 1999) and assortative mating can evolve as a by-product of a host shift or temporal variation in breeding. Assortative mating can also be favored by selection when individuals benefit from mating with their own ecotype in the case of selection against intermediate ecotypes (Weissing et al. 2011; Maan and Seehausen 2011). Competition-driven speciation is ultimately driven by negative frequency-dependent selection arising from ecological competition favoring “rare ecotypes” (Weissing et al. 2011). Similarly, male–male competition for breeding or display sites, or access to mates can generate negative frequency-dependent selection favoring males with “rare competitive traits” such as fighting behavior, color display, and combat weapons (West-Eberhard 1983; Wong and Candolin 2005; Hunt et al. 2009). A rare male advantage may arise when males bias aggression toward their own type (discussed below). The resulting negative frequency-dependent selection can facilitate speciation in several ways (Tinghitella et al. forthcoming; van Doorn et al. 2004; Seehausen and Schluter 2004; Qvarnström et al. 2012). First, it can facilitate the invasion of novel male phenotypes that are selected against by female mating preferences. For example, assuming that the novel male phenotype is able to acquire an above-average quality mating territory and that females use territory characteristics as a factor in their mating decisions (Coleman et al. 2004), females may mate with the novel male against the prevailing mating preference. Second, a rare male advantage arising from male–male competition can stabilize a disruptive selection regime because as the rare phenotype becomes more common, its advantage decreases. Negative frequency-dependent selection arising from male–male competition can also facilitate the coexistence of daughter species by favoring rare species in contest competition or creating “space” between competing species in situations where males tolerate heterospecifics as territorial neighbors (Kohda 1998). Finally, competitive traits can be directly under disruptive selection when intermediate male phenotypes are less successful in male–male competition than more extreme phenotypes, for example by receiving double the amount of aggression. The resulting disruptive selection could enhance the evolution of reproductive isolation.

Here, we consider how aggression biases can generate negative frequency-dependent selection favoring males with rare competitive traits. We then discuss how variation in male competitive traits can evolve by negative frequency-dependent selection.

Aggression biases can lead to negative frequency-dependent selection

Disproportionately strong contest competition between phenotypically similar organisms will lead to a fitness advantage to rarer phenotypes (Seehausen and Schluter 2004; Qvarnström et al. 2012). This can occur when males bias aggression toward phenotypically similar rival males. Such aggression biases are adaptive since conspecific males likely compete for the same set of mates and other resources (Grether et al. 2009; Dijkstra and Groothuis 2011). Although the evidence is somewhat mixed, there is support for males directing increased aggression toward similar males (Pauers et al. 2008; Ord and Stamps 2009; Sheehan and Tibbetts 2009; Peiman and Robinson 2010; Dijkstra and Groothuis 2011; Grether et al. 2013; Lehtonen 2014; Tinghitella et al. 2015). Aggression biases may enable rare males to maintain above-average body condition, maintain a high-quality breeding site, attain higher dominance rank, and allocate more time and effort toward courting females and other activities (Seehausen and Schluter 2004; Qvarnström et al. 2012).

Male aggression biases could cause a disadvantage to hybrid phenotypes as they are expected to receive a disproportionate amount of aggression if they resemble males of both parental phenotypes (Seehausen and Schluter 2004). The resulting disruptive selection could favor assortative mating and enhance the evolution of reproductive isolation between divergent phenotypes. However, evidence for disproportionate aggression toward hybrid phenotypes is rare.

Since biasing aggression toward your own phenotype is generally adaptive, selection should favor alleles or allelic combinations causing the expression of own-type aggression biases. Pleiotropy or physical genetic linkage between competitor recognition templates and traits used in male–male competition could facilitate the expression of aggression biases (Sluijs et al. 2013). Alternatively, social learning could create a link between aggression bias and male competitive trait (“rival imprinting”; Hansen and Slagsvold 2003).

It is often assumed that aggression biases favor rare males (Tinghitella et al. forthcoming). Consistent with this idea, numerous studies indicate that intense agonistic effort can be costly (Flack et al. 2005; Garratt et al. 2012; Beaulieu et al. 2014). Since agonistic interactions always involve at least 2 individuals, future studies should more clearly separate the effect of aggression relative to who is initiating or who is receiving aggression. The fitness consequences of aggression biases are dependent upon the relative rank of interacting individuals, the behaviors used in the agonistic interaction, and the social context. While it is generally accepted that enduring high rates of aggression is costly, being challenged by rival males during territorial defense may in fact increase a male’s attractiveness if females find males that are frequently challenged more attractive (Wong and Candolin 2005). This process would disadvantage rare males that are not recognized as competitors by the common phenotype. There are many opportunities here for future research to examine the long-term fitness consequences of aggression biases for rare versus common competitive phenotypes.

Variation in competitive traits can evolve by negative frequency-dependent selection

Many populations exhibit variation in male competitive traits (often referred to as competitive ability) that influence the outcome of competitive interactions. Such variation is often present within species in the form of discrete phenotypes suggesting that competitive traits could be under disruptive selection and that variation in competitive ability could play a key role during speciation (West-Eberhard 1979; Brockmann 2001; Gray and McKinnon 2007). Examples of polymorphisms in competitive traits include variability in aggression, nuptial coloration, body morphology, and weapon size (Tinghitella et al. forthcoming; McCullough et al. 2016).

One trait that is frequently variable both within and between species is intrinsic aggression level (i.e., aggression measured under standardized conditions, such as in resident–intruder paradigms). A number of empirical studies have found that the success of a behavioral phenotype is negatively frequency-dependent (Bleay et al. 2007; Pryke et al. 2007; Dijkstra et al. 2010; Mokkonen et al. 2011; Lichtenstein and Pruitt 2015). For example, work in bank voles Myodes glareolus showed that highly competitive males that are dominant in dyadic interactions experienced reduced reproductive success when common within the population (Mokkonen et al. 2011). These findings are consistent with models on animal aggression and boldness showing that alternative behavioral strategies can evolve and be stably maintained in a population by negative frequency-dependent selection (Maynard Smith 1982; Wolf et al. 2007).

More aggressive phenotypes are typically socially dominant over other phenotypes (Dijkstra et al. 2005; Pryke et al. 2007; but see Fitzgerald and Kedney 1987), giving the former increased access to reproductive opportunities (West-Eberhard 1979; but see Qvarnström and Forsgren 1998; Moore et al. 2001). However, intense conflict in environments with a high proportion of highly aggressive phenotypes can cause a lower average dominance rank for those same aggressive phenotypes, as has been shown in primates (Hemelrijk et al. 2008) and cichlids (Dijkstra et al. 2010). This is because increased conflict between highly aggressive phenotypes can decrease their average dominance rank in the population and increase the cost of maintaining high social dominance. Future studies should examine how male competitive traits influence the outcome of competition in terms of social dominance as well long-term fitness consequences.

Aggression biases and variation in male competitive traits can interact to influence the strength of negative frequency-dependent selection. For example, in replicate communities of red and blue cichlids, the more aggressive red males were more dominant than blue males but only when red was rare. A simulation model indicated that aggression biases enhanced the effect of negative frequency-dependent selection on the social dominance rank of the more aggressive types, perhaps by increasing their fight rate (Dijkstra et al. 2010).

Male–male competition can facilitate the invasion of novel male mating phenotypes if novel forms of fighting, coloration, or weapons confer a competitive advantage in fights. The advantage of novelty may arise because of a surprise effect or opponents lacking an effective counterattack (McCullough et al. 2016). This could aid in invasion of novel phenotypes against the prevailing mate preference (Dijkstra et al. 2007). Theoretical models indicate that developing and maintaining the variation necessary for speciation to occur can be problematic (Weissing et al. 2011); however, male–male competition could act to establish and maintain the needed variation.

Trade-offs Facilitate Disruptive Selection on Male Competitive Traits

Disruptive selection is selection favoring extreme phenotypes relative to intermediate phenotypes. It can arise from adaptation to unique ecological niches or, in the case of disruptive sexual selection, divergent selection on mating preferences and male mating traits. When disruptive selection is strong and sustained over a long period of time, it could favor reproductive isolation under certain circumstances. In this section, we discuss various examples that could favor alternative male competitive phenotypes under disruptive selection.

Trade-offs play an important role in the evolution of diversity because they can create different ways to optimize fitness (Futuyma and Moreno 1988; Stearns 1992; Roff and Fairbairn 2007). Trade-offs can arise when investment in one fitness component (e.g., territorial defense) occurs at the expense of another component (e.g., survival). This could favor divergent life-history strategies, consisting of animals that invest differently in these fitness traits (Stearns 1992; Gustafsson et al. 1995; Roff and Fairbairn 2007). Trade-offs [sometimes referred to as costs (Futuyma and Moreno 1988)] may lead to the evolution of niche differentiation and speciation (Schluter 1996; Nosil et al. 2002; Bolnick et al. 2003), the maintenance of alternative mating tactics (Gross 1996; Sinervo and Lively 1996; Alonzo and Warner 1999), or adaptation to alternative environments (Manceau et al. 2010). Such alternative fitness optima could then favor the evolution of reproductive isolation. Male–male competition can be an important part of life-history trade-offs because the expression of male competitive phenotypes (e.g., adopting highly aggressive tactics or the development of large weapons) is costly, imposing potential trade-offs with other important fitness traits.

Here, we consider how fitness trade-offs involving male–male competition influence the evolution of variation in male competitive traits and promote the evolution of reproductive isolation by disruptive selection. Two types of trade-offs can be distinguished based on the number of traits involved (Agrawal et al. 2010). A one-trait trade-off can be caused by opposing selection in different environments leading to local adaptation that makes a trait better-suited for reproduction and survival in one environment and less-suited in another environment. A multiple-trait trade-off results from 2 or more traits that share a limiting resource such that allocation to one trait has negative consequences for the other trait(s) requiring the same resource (Zera and Harshman 2001). A multiple trait trade-off can also result from the performance of one activity (e.g., reproduction, territorial defense) generating negative consequences for other fitness components, such as survival (e.g., intensive territorial defense leads to increased predation risk).

Variation in the environment can favor the evolution of divergent male competitive traits

The environment is often heterogeneous and the idea that trade-offs across environments can lead to local adaptation and speciation has been around for some time (Schluter 2009; Weissing et al. 2011; Nosil 2012; Servedio 2015). Local adaptation to different environments, which also includes temporal shifts in behavior, can lead to divergent selection on various traits such as body size and foraging behavior. Assortative mating evolves as a by-product when the trait under disruptive natural selection also influences mating because of positive assortative mating by an ecological characteristic, such as diet (Ward et al. 2004) or habitat/sexual imprinting (Servedio 2015). Assortative mating can also evolve under strong disruptive selection when migrants perform less well than resident individuals (Nosil et al. 2005) and/or females that mate randomly risk producing intermediate offspring with reduced fitness (Bolnick and Fitzpatrick 2007). In geographically separated populations, assortative mating is imposed by geographic barriers, and reproductive isolation can evolve by divergent selection, drift, or other evolutionary processes (Mendelson et al. 2014).

Variation in the local environment can also drive population differentiation via sexual selection (Maan and Seehausen 2011; Scordato et al. 2014; Boughman and Svanbäck 2017; Servedio and Boughman 2017). Two popular models, divergent sensory drive and “good genes,” describe the conditions for adaptive mate choice driving speciation. In sensory drive speciation, natural selection arising from different visual habitats can shift the visual system, which may cause diversifying selection on male display (Boughman 2002; Seehausen et al. 2008). The conspicuousness and perception of male ornaments can be influenced by the environment (van der Sluijs et al. 2011), and variation in signal efficiency across visual environments can promote differentiation in sexually selected traits. In “good genes” or “indicator” models, sexual selection by female mate choice reinforces disruptive ecological selection, eventually leading to ecologically specialized species that are reproductively isolated. In this situation, it is assumed that trade-offs across environments (meaning that individuals that perform well in one environment lose their ability to perform well in another environment) promote assortative mating by habitat. Assortative mating can evolve even in the absence of divergent mating preferences or male ornaments when females benefit from choosing mates based on a single condition-dependent ornament that reflects adaptation (good genes) to the local environment (van Doorn et al. 2009).

Local adaptation in relation to male–male competition can influence population differentiation and speciation in 2 distinct ways. First, male competitive traits may function better in one habitat compared with another because of sensory drive and/or “good genes” (condition-dependent) effects. Therefore, different male competitive traits may reflect superior performance in the local “competitive” environmental conditions. There is ample evidence that competitive phenotypes are shaped by abiotic and biotic environmental conditions. For example, conspicuousness or detectability is necessary for male displays to signal dominance (“badges of status”; Berglund et al. 1996), and therefore male–male competition could play an underappreciated role in sensory drive speciation (Tinghitella et al. forthcoming). In addition, habitat characteristics can profoundly influence the optimal fighting strategies or trait combinations that determine success in competitive social interactions, perhaps due to the fact that habitat complexity influences maneuverability and detectability of agonistic signals (Danley 2011; Lackey and Boughman 2013; Myhre et al. 2013). For example, in sticklebacks, habitats with open or dense vegetation density favored different combinations of body size and nuptial coloration during male–male competition (Lackey and Boughman 2013). Background color or ambient light conditions can enhance or diminish the detectability of competitive traits. Hence, the environment can influence competitor recognition and the effectiveness of certain competitive signals in communicating social status or competitive ability. Finally, the cost of expressing certain condition-dependent display traits, such as weapons (McCullough et al. 2016) or carotenoid-based coloration (Grether 2000; Craig et al. 2005), may be environment dependent. Local adaptation and habitat-driven disruptive selection on male competitive phenotypes in heterogeneous environments may favor the evolution of assortative mating and reproductive isolation (Tinghitella et al. forthcoming; Qvarnström et al. 2012).

The second way male–male competition can influence speciation through local adaptation is when the cost of male–male competition enforces honesty of a condition-dependent trait that signals local adaptation to both males and females. Enforcement of honesty could facilitate adaptive mate choice for locally adapted males (Candolin 1999). Male–male competition (and agonistic interactions in general) can be metabolically costly. Hence, male–male competition can influence the expression of a condition-dependent trait, making the trait a better indicator of the quality of the bearer. When the condition-dependent trait is also used as a communication cue in male–male interactions (e.g., a badge of status), it prevents males from misrepresenting their quality as a competitor.

Divergent selection by ecological context on male competitive traits could increase the total strength of diversifying selection, thereby facilitating the process of speciation (Nosil et al. 2009). Alternatively, male–male competition can also hinder speciation, for example when strong directional selection for a certain male competitive phenotype opposes diversifying ecological selection on the same trait(s).

Life-history trade-offs between different fitness components facilitate disruptive selection on male competitive phenotypes

Life-history theory predicts that trade-offs between different components of fitness (e.g., breeding effort, future fecundity, or survival) may favor the evolution of polymorphisms in male traits such as behavior or the form and quality of secondary sexual traits (Stearns 1992). Fitness trade-offs have been used to explain consistent individual differences in “personality types,” which correspond to alternative competitive phenotypes. When some individuals put more emphasis on future reproduction than others (Wolf et al. 2007), selection will favor systematic differences in risk aversion associated with different life-history strategies. Disruptive selection can favor behavioral diversification or specialization (Bergmüller and Taborsky 2010; Wolf and Weissing 2012). This process is similar to how adaptive speciation can be driven by ecological resource specialization under frequency-dependent disruptive selection selecting against intermediate or generalist ecotypes (Dieckmann and Doebeli 1999). Specialized life styles can evolve when divergent behavioral types enjoy higher fitness than intermediate phenotypes (Bergmüller and Taborsky 2010; Wolf and Weissing 2012) and when alternative behavioral types enjoy increased fitness when rare (Dijkstra et al. 2010; Lichtenstein and Pruitt 2015).

The evolution of variation in competitive phenotypes in response to trade-offs in reproduction or life-history optima has been extensively studied in the context of alternative reproductive tactics, where sexual selection (through condition-, density-, and negative frequency-dependent selection) favors different tactics, for example, competitive versus sneaker tactic. These reproductive phenotypes display differences in behavioral, morphological, and neurophysiological traits that influence competitive ability (Gross 1996; Brockmann 2001; Nugent et al. 2016). While these alternate competitive phenotypes are often stably maintained in the same population without speciation (e.g., through disassortative mating; Seehausen et al. 1999; Tuttle 2003), these polymorphisms may provide the starting material for new species (West-Eberhard 1979; Gray and McKinnon 2007). In geographically isolated populations, selection may lead to the loss or fixation of certain competitive phenotypes in one population but not another, with potential buildup of reproductive incompatibilities between populations. For example, the polymorphisms in throat color in the side-blotched lizards Uta stansburiana varied among geographically separated populations with several populations harboring 1 or 2 instead of 3 color morphs (Corl et al. 2010).

In addition to promoting variation in competitive traits, alternative life-history trade-offs may also promote the evolution of reproductive isolation through disruptive and negative frequency-dependent selection. For example, disruptive selection on male competitive traits can suppress gene-flow between divergent phenotypes. Male–male competition is influenced by a number of behavioral, morphological, and neurophysiological traits, and the expression of these traits influences other life-history traits. It is likely that males can only be successful competitors when they have certain trait combinations, resulting in strong selection against hybrids with poorly adapted trait combinations (Qvarnström et al. 2012; Wolf and Weissing 2012). Comparing different male competitive phenotypes in terms of success in male–male competition, and ultimately fitness, is a fruitful approach to test this hypothesis.

The expression and development of male competitive traits is influenced by physiological processes, for instance hormones and their regulatory pathways that control the allocation of resources to fighting behavior versus self-maintenance processes (Ketterson et al. 2009). Physiological traits might be primary targets of sexual selection (Sinervo and Svensson 1998; Briffa and Sneddon 2007; Irschick et al. 2007) and phenotypes with alternative physiological setups might be stably maintained by disruptive selection, as discussed in the previous paragraph, as well as negative frequency-dependent selection. Studies on male–male competition often assume that superior fighting behavior and “competitive ability” are fixed traits (see Box 1) and confer fitness benefits. However, extreme aggressiveness can be costly, and these costs are not merely a consequence of increased injury or metabolic rate but involve important physiological functions such as immunity (Briffa and Sneddon 2007) and antioxidant defense (Garratt et al. 2012; Beaulieu et al. 2014). Since the potential cost of aggressiveness is higher when the highly aggressive strategy is more common, life-history trade-offs linking male competitive traits to other traits may also be under negative frequency-dependent selection. For example, in the Gouldian finch Erythrura gouldiae, aggressive red-headed birds dominate the more common black-headed morph. This competitive advantage, however, was counterbalanced at a higher frequency of red-headed morphs by reduced immunity resulting from the social stress of frequent aggressive interactions (Pryke et al. 2007). This study elegantly demonstrates negative frequency-dependent fitness trade-offs between 2 competing phenotypes, which could maintain the polymorphism in the wild.

Box 1. Male competitive traits, competitive interactions, and its ecological and evolutionary consequences

In order to understand the role of male–male competition in speciation, we need to make careful distinctions among traits that mediate interactions among individuals and the outcomes of those interactions (see Figure 1). Below is a list of terms with some examples (modified from Peiman and Robinson, 2010).
Term	Definition and examples
1. Competitive traits	Traits within an individual that are used in competitive interactions, such as visual and chemical cues; behavior (fighting strategy, an individual’s motivation to fight); physiological traits that influence the development or expression of competitive traits (e.g., hormone levels).
2. Competitive interactions	Interactions among individuals, such as combats, threats or displays, agonism, contest, interference.
3. Immediate consequences	The direct behavioral and physiological consequences associated with the outcome of a competitive interaction, such as dominance/subordinance, social rank, territory holders versus floater, future competitive ability (e.g., resulting from winner or loser effects). It also includes physiological and life-history outcomes, such as access to mates and/or ecological resources, risk of injury, time taken from other activities, energetic expenses, up or down regulation of the hypothalamic–pituitary–adrenal axis.
4. Ecological consequences	Patterns of phenotype/species distribution, displacement or competitive exclusion, spatial and temporal resource use.
5. Evolutionary consequences	Agonistic character displacement, gene flow, various forms of selection (disruptive, negative frequency-dependent, directional selection), niche partitioning.
Dominance is an outcome that is influenced by a competitive trait such as fighting strategy or weapon size. For example, some studies confuse some of these traits, especially aggression level (a trait which is determined by an individual’s willingness to acquire or defend a resource) with dominance (which is a consequence of an interaction among individuals). Although dominance relationships may be “fixed” in some species, we advocate caution when using terms such as “dominant phenotype.”
Most studies on male competition focus on terms 1–3 (how does a particular competitive trait influence the likelihood of becoming dominant), or study 1 and 4 and/or 5 (e.g., different competitive phenotypes and long-term outcomes such as reproductive success, displacement, or gene flow without detailed information about the behavioral interactions that mediated these outcomes). We propose that for a comprehensive understanding of the role of male–male competition in speciation, we need to study male–male competition at every level, from competitive traits to evolutionary consequences.

Shown here are the different components of male–male competition and how it impacts evolutionary change.

Studies that examine how competitive traits influence the immediate outcome of agonistic interactions and how these outcomes influence life time reproductive success are generally lacking, in part because there are few model systems where behavioral interactions, physiology, and fitness outcomes can be effectively monitored. We need more studies that carefully quantify social relationships and structure, because the fitness consequences of male competitive traits are mediated by social interactions leading to, for example, differences in rank or social stability in a community. Social network analysis can be a powerful tool to quantify social relationships and rank, the amount of aggression an individual receives, and how this changes over time (Pinter-Wollman et al. 2014). The link between male–male competition and maintaining health and body condition is not a black box, but mediated by physiological processes (Sapolsky 2005; Beaulieu et al. 2014).

Male–Male Competition and Ecological Competition Influence Speciation in Comparable but Unique Ways

Sexual selection by female mate choice can influence mating patterns within species and lead to male mating traits that can be recruited for assortative mating between diverging populations. However, sexual selection has a controversial role in speciation (Servedio and Kopp 2012; Servedio and Boughman 2017), especially under sympatric conditions (van Doorn et al. 2004). For example, female mate choice may eliminate heritable variation required for speciation (Servedio 2015), or it may oppose disruptive selection on an ecological trait (Maan and Seehausen 2011). Furthermore, the cost of female choosiness may exceed the (ecological) fitness benefits of assortative mating (Bolnick and Fitzpatrick 2007). Less attention has been given to the role of male–male competition in speciation, in spite of the fact that intrasexual selection is an important component of sexual selection (Andersson 1994). In theory, male–male competition may aid the process of speciation by sexual selection. First, male–male competition can promote variation in male mating traits and favor divergence in both male traits and the corresponding female mating preferences (van Doorn et al. 2004). Second, rare male advantages in competition may also decrease the cost of female choosiness by reducing the search time for rare males, for example, when rare males have better display sites with higher mate encounter rates (Wong and Candolin 2005). In sum, diversifying selection arising from male–male competition likely aids the process of speciation by sexual selection, especially when sexual selection is influenced by divergent ecological selection and local adaptation as discussed in the section “Trade-offs Facilitate Disruptive Selection on Male Competitive Traits”.

It is well-known that ecological competition is a major driver of speciation (Weissing et al. 2011). However, there has been little exchange of ideas between researchers who study male–male competition versus ecological competition in relation to speciation (Weissing et al. 2011). Some of this is caused by a strong dichotomy in the speciation literature between divergence driven by sexual selection or by natural selection. We note that both forms of selection adhere to the same logical framework of competition for limiting resources (sexual selection arises from competition for mates while natural selection typically arises from competition for ecological resources, though all could be considered natural selection, Shuker 2010). While the distinction between competition for mates versus competition for other resources is helpful, competition for mates may often coincide with competition for ecological resources. Below, we first discuss the role of ecological competition and speciation, then the relationship between male–male competition and ecological competition in speciation (Box 2 defines some of the terms that we use).

Box 2. Ecological competition and male–male competition

Given the well-understood role of ecological competition in evolutionary change and speciation (for references see text), it is helpful to compare the role of ecological competition and male–male competition in speciation (see Figure 2).

Figure 2.

Shown here are the various categories of traits that are shaped by selection arising from different types of competition (aggressive versus exploitative; ecological competition versus male–male competition). Note that in the text, male–male competition is mostly considered in the context of male contest competition for females exerting diversifying selection on agonistic traits. The same trait(s) can be shaped by different sources of selection. See Grether et al. (2017) for a similar conceptual diagram concerning interspecific competition.

Both may involve direct aggressive competition as individuals are directly competing through agonistic interactions for priority access to a shared and limited resource (mates or ecological resources). Agonistic interactions involve fighting behavior, threat displays, or appeasement behaviors. Interference competition occurs when individuals compete aggressively for ecological resources. Male–male competition (also referred to as male contest competition) occurs when males compete aggressively for mates or breeding/display sites.

Both may involve exploitative competition for a shared, limited resource. Exploitative ecological competition occurs when individuals are utilizing the same resource limiting the resource availability to other individuals. It selects for traits that improve resource utilization such as beak morphology or body shape. The term exploitative competition is rarely used in relation to male–male competition but it would select for mate choice traits, such as body coloration, courtship display, that improve “utilization” of a limited pool of females. This process is identical to intersexual selection (Weissing et al. 2011).

Ecological competition and speciation: the roles of exploitative and interference competition

Studies on ecological speciation have focused on exploitative competition (indirect competition) for shared ecological resources and the resulting frequency-dependent disruptive selection on resource utilization traits in the consumer population(s) (Dieckmann and Doebeli 1999; Weissing et al. 2011). This focus on exploitative competition may be due to the fact that exploitative competition is stronger than interference competition especially in less aggressive taxa competing for ecological resources that are spatially dispersed and therefore less defendable (Dubois and Giraldeau 2005). However, numerous studies indicate that direct aggressive interactions between individuals competing for ecological resources can shape interference traits, patterns of gene flow, and distribution of phenotypes both between species (Grether et al. 2013, 2017) and within species (Bolnick et al. 2003). This suggests that interference competition could play a key role in ecological speciation (Zhang et al. 2015). Unfortunately, the effect of interference competition is usually not explicitly incorporated in studies on ecological speciation (Dieckmann and Doebeli 1999; Weissing et al. 2011), perhaps due to the fact that the outcome of interference competition is traditionally thought of as competitive exclusion or even extinction instead of speciation. At the same time, interference competition can force individuals to exploit (less) profitable resources (which could initiate speciation by ecological resource competition; Nosil 2012) and it has been implicated in several forms of evolutionary diversification (Kisdi 1999; Gröning and Hochkirch 2008; Pfennig and Pfennig 2009, 2010).

Interference competition could contribute to negative frequency-dependent competition for ecological resources driving sympatric speciation. This is because interference competition is expected to be stronger between ecologically similar individuals for 2 reasons. First, similar ecotypes are more likely to encounter each other, for example due to similar foraging strategies (Bolnick et al. 2003). Second, it is generally considered adaptive for individuals to preferentially attack similar ecotypes with whom they are competing exploitatively for food or space. Stronger interference competition between ecologically similar types than between dissimilar types could generate negative frequency-dependent selection on competitive traits, akin to how male–male competition can exert diversifying selection on male competitive traits. In addition, ecotypes may also differ in an ecological trait that promotes resource utilization (e.g., body shape or size that increases the effectiveness to deplete a particular resource) that also influences success in interference competition. Such differences in a resource utilization trait that also determines success in interference interactions between ecotypes could be directional, causing the competitively weaker ecotype to be driven to extinction or adopt an alternative ecological niche. At the same time, it is important to note that success of the generally more competitive ecotype could be negatively frequency-dependent due to more costly fights among the competitively superior ecotype when they are locally more common.

Interference competition can have immediate consequences for competing individuals and could modulate the negative frequency-dependent or disruptive selection effects arising from resource depletion and exploitative competition. For example, while exploitative competition may favor adaptation to new habitats or niche specialization (Dieckmann and Doebeli 1999), interference interaction rates (and associated fitness costs) can be density and negative frequency-dependent when the common ecotype has more frequent aggressive encounters over the same resources compared with the rare ecotype (Goss-Custard et al. 1984). Interference competition could force individuals to use underexploited resources (Bolnick et al. 2003), promoting divergent resource use under negative frequency-dependent selection. In addition, direct social interactions during interference competition can have dramatic fitness effects, such as social subordination or in extreme cases, the killing of competitors (Sapolsky 2005). Finally, interference competition and exploitative competition may favor different sets of traits. For example, phenotypes that are inferior in exploitative competition could dominate their competitors in interference competition as has been suggested for interspecific competition (Amarasekare 2002). It is also possible that exploitative competition is reduced by interference competition when territory owners forage on their territory or when territories differ in their resource base (Bolnick et al. 2003). Clearly, we need more studies investigating the relative contributions of aggressive interference competition and exploitative competition for ecological resources, and their interaction in facilitating or hindering competition-driven speciation (Winkelmann et al. 2014). Studies manipulating competitor density and resource availability combined with behavioral observations and measures of fitness of competing phenotypes are an important first step (Smallegange et al. 2006; Martin and Pfennig 2010).

Competition for mates versus competition for ecological resources

Male–male competition for mates or breeding sites and interference competition for ecological resources may often become intertwined (“all purpose territories”; Grether et al. 2013); for example, when a breeding territory is also used as a foraging site (Kohda 1998). Consequently, individuals that are successful competitors accrue not only reproductive benefits but may also monopolize vital ecological resources that can increase their fitness. The interaction between sexual selection and ecology in speciation has recently received a lot of attention (Maan and Seehausen 2011; Scordato et al. 2014). However, few studies consider the dual role of competitive traits in serving both sexual and nonsexual functions (Gröning and Hochkirch 2008). The rare male advantage arising from male conflict for access to mates may be amplified if those rare males also accrue ecological benefits through priority access to resources such as food or shelter.

Although speculative, it is also possible that disruptive selection arising from interference competition for ecological resources causes rare male advantages in male–male competition for mates, or vice versa. For example, disruptive ecological selection on a competitive trait (e.g., body size, body coloration) could cause rare male advantages if the same traits also influence success in competition for mates. Although speciation is not the only possible outcome of disruptive selection (Rueffler et al. 2006), speciation appears to be more likely when disruptive selection on the same trait is caused by multiple factors, increasing the total strength of selection and promoting the evolution of assortative mating (Nosil et al. 2009). If ecological competition exerts diversifying selection on an ecological interference trait, it could also influence patterns of mating as a by-product by shaping the outcome of aggressive competition for mates. Such interference traits would be akin to “magic traits,” which are ecological characters under disruptive selection that also lead to assortative mating (Servedio et al. 2011).

Interference competition may lead to convergence between species in signals mediating competitor recognition when there is considerable interspecific overlap in ecological resources (Grether et al. 2009; Tobias et al. 2014). However, in the presence of ecological divergence, it is generally adaptive for individuals to preferentially attack conspecific competitors with whom ecological resources are shared. Future studies should examine in more detail the roles of male–male competition and ecological competition and their interaction during population differentiation and speciation. Specifically, we need more studies testing how rare male advantages in male–male competition, as discussed in the section “Trade-offs Facilitate Disruptive Selection on Male Competitive Traits” (e.g., are rare males more likely to attain high social rank?), are influenced by potential ecological benefits (how does priority access to ecological resources influence reproductive success?). Ecological competition affects both sexes, and studying negative frequency-dependent selection arising from aggressive interactions within both males and females (and between males and females) is an interesting avenue for future research. Finally, we note that competition-driven character evolution has been extensively studied in the context of between-species interactions (Grether et al. 2017). While some processes, such as agonistic character displacement, has been mostly studied between species, we underscore the fact that these processes can act in similar ways within species (Pfennig and Pfennig 2009).

Conclusion

The goal of this paper was to provide a clearer conceptual framework to study the role of male–male competition in speciation. We may gain a better understanding of the role of male–male competition as a driver or inhibitor of adaptive speciation when future studies center around life history trade-offs and on how these trade-offs are mediated by physiological and social mechanisms in an ecological context. We highlight 2 considerations that may help future efforts in this area. First, future studies should not only focus on how male competitive traits influence the outcome of competition in terms of social dominance, but also include the long-term fitness consequences. Negative frequency-dependent survival and reproductive success has been demonstrated in a number of polymorphic species with divergent competitive phenotypes (Sinervo and Lively 1996; Sinervo and Calsbeek 2006; Bleay et al. 2007; Mappes et al. 2008; Mokkonen et al. 2011; Hughes et al. 2013), but we need more details of the social interactions that mediate these fitness effects. Social network analysis can be used to describe patterns in social structure and relationships (Pinter-Wollman et al. 2014; Williamson et al. 2016). The field can benefit from insights in neuroendocrinology on the short- and long-term physiological effects of social subordination and dominance (Sapolsky 2005). We also need more studies investigating the link between male–male competition and mate choice. Second, male–male competition has been almost exclusively studied in the context of sexual selection and speciation. Future studies should adopt a broader view, recognizing the fact that competition for mates often overlaps with competition for ecological resources, or that the same competitive traits can be used in different competitive situations (Pasch et al. 2013). The emergent picture is that in addition to specialization to different ecological niches, male–male competition can promote the development of discrete social or life-history niches, providing alternative options to be successful competitors for mates and ecological resources. Clearly, male–male competition matters for speciation and we encourage further empirical, theoretical, and comparative studies on the role of aggressive competition in the evolution of reproductive isolation.