Evolutionarily stable strategies in stable and periodically fluctuating populations: The Rosenzweig-MacArthur predator-prey model.

An evolutionarily stable strategy (ESS) is an evolutionary strategy that, if adapted by a population, cannot be invaded by any deviating (mutant) strategy. The concept of ESS has been extensively studied and widely applied in ecology and evolutionary biology [M. Smith, On Evolution (1972)] but typically on the assumption that the system is ecologically stable. With reference to a Rosenzweig-MacArthur predator-prey model [M. Rosenzweig, R. MacArthur, Am. Nat. 97, 209-223 (1963)], we derive the mathematical conditions for the existence of an ESS when the ecological dynamics have asymptotically stable limit points as well as limit cycles. By extending the framework of Reed and Stenseth [J. Reed, N. C. Stenseth, J. Theoret. Biol. 108, 491-508 (1984)], we find that ESSs occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. These functions are identified and shown to have a similar form for both stable and fluctuating populations. We illustrate these results with a concrete example.

1. Introduction

Natural systems exhibit both ecological and evolutionary dynamics. Nevertheless, many studies focus only on either the ecological or the evolutionary dynamics, assuming the other nonchanging and fixed. There are many excellent exceptions (see, e.g., refs. 1 and 2). Here, we study the ecological and evolutionary dynamics in both ecologically stable as well as periodically fluctuating populations of predator and prey, using the Rosenzweig–MacArthur predator–prey model (3) (see also refs. 4–9). Our overall aim is to understand under what conditions evolutionarily stable strategies (ESSs) will exist. Specifically, we focus on evolutionary stability in stable as well as periodically fluctuating ecological systems.

The concept ESS was originally coined by Maynard Smith (10) (see also Smith and Price) (11), as an extension of Hamilton’s (12) “unbeatable strategy.” Maynard Smith (10) defined it as a strategy that, once adopted by a majority of the members of a population, cannot be overturned by any alternative strategy that is initially rare. It is essentially a Nash equilibrium that is evolutionary stable (13). The concept of ESS was used initially in behavioral ecology and sociobiology but Lawlor and Smith (14)—see also Roughgarden (15) and Nowak (16)—extended its use to ecological dynamics of competing populations.

Many ecological systems are characterized by fluctuating dynamics, some of which have periodic fluctuations: the famous lemming and small rodent cycles (see, e.g., refs. 17–19) and the North American hare–lynx cycle (see, e.g., ref. 20). The predator–prey models of the kind presented by Rosenzweig and MacArthur (17) are therefore of broad ecological interest. Several studies have analyzed these—and related—models from an ecoevolutionary perspective (5, 21–26). However, no one has addressed this from an ESS perspective, particularly for the limit-cycle case. This we do in the present contribution.

Models for predatory–prey interactions have played a major role within the field of ecology since the pioneering contribution by Lotka (27, 28) and Volterra (29) (see, e.g., ref. 30). Depending on the choice of parameters, such models are known to accommodate equilibrium populations that are constant or fluctuating (including limit-cycle dynamics).

The biological contribution of this paper will be the theoretical question regarding the existence of ESS in periodically fluctuating populations exemplified by Rosenzweig–MacArthur predator–prey systems. Specifically, we establish sufficient conditions for the existence of ESSs in such models. We consider populations that become stable (constant) over time and periodically fluctuating populations or systems exhibiting stable limit-cycle dynamics. Extending the approach of Reed and Stenseth in ref. 31, we show that ESSs occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. We identify these functions and show that they have a similar form for both stable and fluctuating populations. These results are illustrated with a concrete example.

The main mathematical contribution is that we prove Lyapunov stability through an adaptation of classical linearization arguments, variation of parameter formulas, and Floquet-type analysis. The main finding is the case of limit cycles, where we establish stability of two-dimensional limit cycles in a four-dimensional system of ordinary differential equations. Within the approach of Reed and Stenseth (31) to evolutionary games and ESS, we then show that these stability results lead to sufficient conditions for ESSs for both types of equilibrium solutions. These sufficient conditions could be of independent interest in game theory.

In section 2, we state the model and results about equilibrium solutions. Then, in section 3, we combine ecology and evolution: evolutionary strategies, an extended ecological model, and the concept of ESS are introduced. The strategies are model parameters that can be changed over time by the populations to ensure a more beneficial evolution, and the extended model includes mutant populations for predators and prey satisfying the same model equations at different (i.e., mutant) values of the strategies. An ESS is introduced as an extension of the definition of Reed and Stenseth (31) to include periodically fluctuating populations—it is a choice of evolutionary strategies for which mutant populations cannot survive in the long run. Evolution can be seen as a competition between different species and be modeled as a game (13). An ESS is then a Nash equilibrium (32), in the sense that no population can improve their own survivability acting alone. A game can only move away from an ESS when the original populations change their strategies. An ESS is therefore a type of locally optimal evolutionary strategy for the original populations.

When time becomes large, the populations will approach limit populations, a (stable) equilibrium solution of the model. Examples of equilibrium solutions are constants (limit points) and periodic solutions (limit cycles). In sections 4 and 5, we study such solutions for the extended ecological model. We first observe that equilibrium solutions of the two-population model remain equilibrium solutions in the extended model when the mutant populations are zero. Then, we establish conditions to guarantee the (Lyapunov) stability of these solutions. Stability here means that solutions starting near an equilibrium solution will over time converge to the equilibrium solution. These results extend earlier results for models with two populations. By taking into account the dependence on the strategies in our extended model, we use the stability results and our definition of ESS to give conditions that guarantee ESS. In section 4, we give the results for equilibrium points (constant populations in the limit), and in section 5, we give the results for limit cycles (periodic limit populations). ESSs are shown to occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. These results are the main contribution of our paper. In section 4, we also discuss examples of functional dependencies that lead to an ESS: loosely speaking, if we take the predation rate to be an increasing bilinear function of the evolutionary strategies, then we find that the rate constants of prey growth and predator mortality have to be decreasing convex functions. We give a concrete example where the latter two rate constants are quadratic functions of the evolutionary strategies.

2. The Ecology of Predator–Prey Models

Let and denote the population sizes of prey and predator, respectively. We will assume that the dynamics follow a logistic Gause-type model:Here, the rate of predation is , where is given byfor positive constants and . The quantity is referred to in ecological literature as the functional response curve, with being the saturation point and the half-saturation constant in the sense that (33, 34). Furthermore, the predator mortality rate is given by for , and the prey growth rate by the commonly used logistic model, where is the rate constant and the carrying capacity of the prey population. Note that is decreasing, with and .

This model goes into a long tradition of predator–prey models of the formfirst identified by Kolmogorov (35) in 1936 (see also ref. 36). The system in Eq. has been intensively studied in the literature (see, e.g., refs. 37–39 and the references therein).

For this system, the first quadrant is an invariant region (i.e., the populations can never become negative). Under some reasonable assumptions, the system has three limit points and up to one limit cycle. These results along with stability results are given in the next theorem (also see Fig. 1).

Fig. 1.

The phase diagram of Eq. with , , , , , and , which corresponds to case (iii) in . (A) The full phase diagram. Dashed curves as well as the coordinate axes are isoclines. The brown points give the stationary points. The red curve is the limit cycle. (B) A close-up of the full diagram in A. If , then is unstable and there exists exactly one limit cycle in the first quadrant in the plane, which is an (asymptotically) stable limit cycle.

Theorem 2.1 (37)

Assume that and .

Solutions of are positive and bounded.

Define and if . Then:

If or , then the critical point is asymptotically stable and

If , then the critical point , with , is asymptotically stable and

If , then is unstable and there exists exactly one limit cycle in the first quadrant in the plane, which is an (asymptotically) stable limit cycle.

Further properties can be found in .

3. Evolution: The Extended Ecological Model

Evolutionarily, there is a “conflict” or “arms race” between the predator and the prey regarding predation: the prey “wants” to evolve to avoid being caught by the predator, whereas the predator “wants” to be able to catch prey as efficiently as possible, even when the prey is at low abundance. However, this will come at some costs: we assume that for the prey, a decreased predation rate will lead to a reduced growth rate, whereas for the predator, an increased predation rate may lead to an increased mortality rate. A natural way to analyze this situation is through (differential) game theory and concepts like Nash equilibriums and ESS; we refer to refs. 13, 14, 40, and 41 for more information. We will follow the simplified approach of Reed and Stenseth (31), an approach explicitly emphasizing that a new mutant will have to establish itself through competition, even though rare initially.

Let and represent the strategies of prey and predator, respectively. All model parameters are assumed to depend on and , i.e., different mutations correspond to different values of the various constants in Eqs. –. The idea is that over time, the populations can modify their own strategies in order to increase their chances of survival.

We now make some assumptions about how the different model parameters depend on and . We know from, e.g., studies of the hare–lynx cycle (42) that the predator and the prey mutually “disagree” on the value of the half-saturation constant; for instance, large and hence low predation is beneficial for the hare, while the opposite is the case for the lynx. We therefore takeTo simplify, we then assume that and are constants, whileWe assume the following reasonable constraints/tradeoffs among the parameters under evolution:The interpretation is the following: an increase in (which means smaller ) should result in both a decrease in and . This means that a lower predation rate, which is beneficial for the prey, comes at the cost of lower prey growth rates. Lower predation rates are bad for the predators and are compensated by lower mortality rates. Thus with , Eq. can be written asIn this paper, the strategies (i.e., the ecological parameters under evolution) do not depend on or time .

In the next step, we extend the model to include mutants. Let and denote the original prey and predator populations, respectively, and and denote the corresponding mutant populations. The extended mode then takes the formwhere for ,and are positive constants. Note that the rate of predation of upon is . Also note that and when , and .

We now introduce evolutionary strategies , for , , respectively. As before, we assume for thatand require thatNote that the dependence in is through and only and that the mutants and original populations satisfy the same model equations, only for different values of the strategies. With , Eq. 9 can then be rewritten as follows:where denotes the right-hand side of Eq. 9.

If is an equilibrium point (stable or not) for Eq. , then is an equilibrium point for Eq. and

If is a periodic solution (a limit cycle) for Eq. , then is a periodic solution for Eq. and

If , then by Eq. , , , and for , and hence is a solution to (the nonlinear) Eq. . This is consistent since now there are no mutants, and the populations should then be determined by Eq. alone.

Remark 3.1:

An ESS corresponds to the situation where the original population cannot be invaded by mutants when only small evolutionary deviations are allowed.

Definition 3.2:

Assume we are given an asymptotically stable equilibrium point (limit cycle) to Eq. with corresponding constant strategy . Then is called an ESS if is an asymptotically stable equilibrium point (limit cycle) to Eq. for all constant strategies sufficiently close to .

An ESS has the Nash equilibrium-like property that if does not change, mutants with strategy will never survive in the long run.

Remark 3.3:

The ESS, as defined in Definition 3.2, is stable against mutations in one or both populations at the same time. Single population mutations are realized simply by letting the other mutant population be zero over time.

The issue of time scales for the ecological and evolutionary processes is important here (see, e.g., Carroll et al.) (43). The approach of Reed and Stenseth (31) emphasizes the ecological interaction between the common wild-type strategy (with its corresponding phenotype) and the rare mutant strategy (with its corresponding phenotype). If the mutant strategy cannot (ecologically) invade the wild-type ecological system, there will be evolutionary stability (here in the form of ESS). If the mutant strategy can invade the wild-type ecological system, it will eventually change into a new one. The result will depend upon many features of the different populations—issues that are beyond our current discussion but are essentially addressed by Dercole et al. (44), Dercole and Rinaldi (45), Doebeli (46), and Cortez and Weitz (26). This all refers to ecological processes and time scales. The overall question being addressed in this contribution is under which conditions will we have evolutionary stability in ecologically varying (periodically fluctuating) systems.

The evolutionary processes and time scales enter in relation to how frequent new mutant strategies appear in the two populations (through the process of mutation and/or invasions)—issues outside of our current discussion. The discussion by Khibnik and Kondrashov (47) are relevant here.

4. Equilibrium Points, Stability, and ESS

has three equilibrium points: , , and , where states that the equilibrium point is asymptotically stable provided and , making the eigenvalues for Eq. negative. These eigenvalues are also eigenvalues of the full system Eq. 9.

We show in that the eigenvalues and for the Jacobi matrix of the full system Eq. 9 areWhen are negative at , then this point is an asymptotically stable equilibrium point for the full system Eq. . We summarize this discussion in the following result.

Lemma 4.1

The equilibrium point is asymptotically stable if , ,See and below in for a full proof. From this result and our definition of an ESS, we find the following conditions that guarantee an ESS when the system has a stable equilibrium point as the global attractor or omega limit point. In other words, at this ESS the original populations are nearly constant after some time.

Theorem 4.2 (Conditions for ESS—Equilibrium Point Case)

Assume . A constant evolutionary strategy is an ESS for if

,

,

the function has a strict local maximum at the point ,

the function has a strict local minimum at the point .

Proof:

Note that by the definition of , the second inequality in can be written as when . The result then follows from Eq. , the definition of an ESS, and .

Now, we consider an example where , , and are given as bilinear, quadratic, and rational functions, respectively, of and . In this case, we use to find the values of the ESS. We also check that Eq. is satisfied near these ESSs.

Example 4.3:

Assume , , and for positive constants . Then, the conditions of Theorem 4.2 are satisfied for , whereunder the conditionsNote that and . We conclude that is an ESS for Eq. . From a direct calculation, it now follows thatIn other words, the natural relations Eq. hold at the ESS, and the functions and are convex decreasing at the ESS (Figs. 2 and 3).

Fig. 2.

The function .

Fig. 3.

The function . Note that implies .

Note that we need . If , then , , and . However, this is unrealistic since the predator mortality rate constant must be strictly positive. Eq. is also violated. See for further computations on this example.

Biologically, this example may be interpreted as follows. An ESS will exist when the evolutionary strategies and are linked, respectively, to the logistic growth rate constant for the prey and the mortality rate constant for the predator, in a decreasing convex fashion, while at the same time, the predation half-loading constant is bilinearly increasing.

5. Limit Cycles, Stability, and ESS

We now find conditions for when there exist (asymptotically) stable limit-cycle solutions of Eq. 9 with the two last components and equal to 0. Thereafter, we will look at the parameter dependent system Eq. and identify ESSs. When and are identically 0, our model reduces to Eq. . To prove the existence of a limit cycle inside the first quadrant, both the position and the stability of the three equilibrium points , , and play an essential role. If there exists a limit cycle inside the first quadrant, it must surround at least one equilibrium point. In our case it must surround , which lies in the first quadrant by if we assume that . By the same theorem, it follows that is a stable node or spiral if and all phase paths inside the first quadrant end up in . In this case there cannot exist a limit cycle. If, on the other hand, , then there exists at least one limit cycle surrounding . To establish the existence of an asymptotically stable limit cycle is much harder than establishing the existence of an asymptotically stable equilibrium point, but the main ideas behind are quite similar.

In ref. 37, it has been shown that every limit cycle surrounding is asymptotically stable. As an immediate consequence it then follows that there is at most one limit cycle in the first quadrant, since an asymptotically stable limit cycle cannot be surrounded by another asymptotically stable limit cycle. The key ingredient in the argument is the following theorem, which is taken from ref. 48 (Corollary, p. 216) and adjusted to our notation and assumptions.

Theorem 5.1

Let be a periodic solution of of period , and assume that the right-hand side of , , is continuously differentiable. Then, is an (asymptotically) stable limit cycle if

Remark 5.2:

The condition in Eq. in Theorem 5.1 plays a similar role for limit cycles in as checking the eigenvalues to determine the stability of equilibrium points. It is derived by a topological argument that makes it possible to compute the sign of the generalized eigenvalues for the limit cycle of Eq. .

In our case, this condition, which has been checked in ref. 37, reads,Here, we used that and . Moreover, the last integral in the second line equals for a satisfying and hence is 0 by the periodicity of .

Based on the discussion in this section for Eq. , we are going to identify conditions under which the solution is an asymptotically stable limit cycle for Eq. 9. To be more precise, we show strong Lyapunov stability for by looking at it as a perturbation of the asymptotically stable limit cycle for Eq. . We follow the same strategy as is used for proving the stability of equilibrium points for a nonlinear system, i.e., one studies first the stability of the linearized system and then uses the variation of parameter formula and Floquet-type arguments to conclude for the nonlinear system.

We have the following result.

Lemma 5.3

Assume and . Then, the limit cycle will be asymptotically stable ifandThe proof is given in . We then show for which choices of parameters we can have an ESS. By and the discussion in this section, we have the following conditions.

Theorem 5.4 (Conditions for ESS—The Limit-Cycle Case)

Assume . A constant strategy is an ESS for ifhas a strict local maximum at the point , and at the same timehas a strict local maximum at the point .

,

,

The proof follows from and the fact that by definition.

With this result, we have given conditions that will ensure the existence of an ESS in a periodically fluctuating population (exhibiting a limit cycle). This is an important contribution of this paper. With empirically derived functions , we can now give the ESS values of and . It is, however, beyond the current contribution to do so.

6. Red Queen Type of Continued Evolution or Stasis

Our analysis is related to the issue of Red Queen type of continued evolutionary evolution of stasis (cf. Van Valen) (49); see also Schaffer and Rosenzweig (50) and Rosenzweig et al. (51). In , we have assumed that both the predator and the prey have equal influence on the determination of the overall half-saturation parameter . For this system, we find that evolution will lead to an ESS without evolutionarily fluctuating dynamics. If the two interacting species (the predator and prey) had asymmetric influence on the overall half-saturation constant, this might not be the case [see, e.g., Nordbotten and Stenseth (52)]. A further analysis of this would indeed be worthwhile. The contributions by Dercole and coworkers (44, 45), Doebeli (46), and Cortez and Weitz (26) are all important stepping stones in such further analysis.

7. Discussion

We establish conditions that guarantee stability of equilibrium solutions of the extended ecological model. This model has four equations, and, for periodic solutions, such a result appears to be previously undescribed. We prove full Lyapunov stability of the periodic solutions. The proof relies on an adaptation of classical linearization arguments, variation of parameter formulas, as well as Floquet-type analysis. The approach of Reed and Stenseth in ref. 31 to evolutionary games and ESS is extended to include fluctuating populations. ESSs are then shown to occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. We identify these functions and express them in a similar way for both stable and fluctuating populations. A concrete example to illustrate our results is given. Mathematically, these results are obtained from the abovementioned stability results, and our characterizations of ESSs seem to be previously undescribed and could be of independent interest in game theory.

The concept of ESSs was developed with a stable ecological setting in mind. However, typically ecological systems vary in time, often with more or less periodic fluctuations. Predator–prey systems are such examples. If we are to link ecology and evolution, we must allow for varying ecological population fluctuations. With this contribution, we have extended the ESS concept to be applicable for periodically fluctuating ecological systems.

The application of ESS to ecology is similar to the “adaptive dynamics” approach in that it models the time evolution of populations. See Dieckmann et al. (53); see also Kang and Fewell (54). The “adaptive dynamics” approach focuses on how strategies evolved under changing ecological conditions. The ESS approach, on the other hand, focuses on finding the unbeatable fixed strategy within a population. To our knowledge, it has not been shown before that such unbeatable ESS strategies do exist when the ecological system exhibits a periodically fluctuating dynamics. With this contribution, we derive the mathematical conditions for such fixed ESSs to exist when the ecological system exhibits limit cycle-type dynamics.

By extending the approach presented by Reed and Stenseth (31), it furthermore becomes clear that evolution occurs through the ecological process of a variant type being able to invade an ecological system (or population) when it is at its stable state—be this a stable equilibrium or a limit cycle. Whenever such ecological invasion occurs (i.e., the wild type is not an ESS), evolution will occur—and the ecology of the system changes. This emphasizes the interlinkages between ecology and evolution: it is a matter of realizing both that “nothing in biology makes sense except in the light of evolution” (55) as well as “very little in evolution makes sense except in the light of ecology” (56).