Canonical Structure and Orthogonality of Forces and Currents in Irreversible Markov Chains.

We discuss a canonical structure that provides a unifying description of dynamical large deviations for irreversible finite state Markov chains (continuous time), Onsager theory, and Macroscopic Fluctuation Theory (MFT). For Markov chains, this theory involves a non-linear relation between probability currents and their conjugate forces. Within this framework, we show how the forces can be split into two components, which are orthogonal to each other, in a generalised sense. This splitting allows a decomposition of the pathwise rate function into three terms, which have physical interpretations in terms of dissipation and convergence to equilibrium. Similar decompositions hold for rate functions at level 2 and level 2.5. These results clarify how bounds on entropy production and fluctuation theorems emerge from the underlying dynamical rules. We discuss how these results for Markov chains are related to similar structures within MFT, which describes hydrodynamic limits of such microscopic models.

Introduction

We consider dynamical fluctuations in systems described by Markov chains. The nature of such fluctuations in physical systems constrains the mathematical models that can be used to describe them. For example, there are well-known relationships between equilibrium physical systems and detailed balance in Markov models [20, Sect. 5.3.4]. Away from equilibrium, fluctuation theorems [12, 19, 25, 32, 37] and associated ideas of local detailed balance [32, 39] have shown how the entropy production of a system must be accounted for correctly when modelling physical systems. However, the mathematical structures that determine the probabilities of non-equilibrium fluctuations are still only partially understood.

We characterise dynamical fluctuations using an approach based on the Onsager–Machlup (OM) theory [36], which is concerned with fluctuations of macroscopic properties of physical systems (for example, density or energy). Associated to these fluctuations is a large-deviation principle (LDP), which encodes the probability of rare dynamical trajectories. The classical ideas of OM theory have been extended in recent years, through the Macroscopic Fluctuation Theory (MFT) of Bertini et al. [7]. This theory uses an LDP to describe path probabilities for the density and current in diffusive systems, on the hydrodynamic scale. At the centre of MFT is a decomposition of the current into two orthogonal terms, one of which is symmetric under time-reversal, and another which is anti-symmetric. The resulting theory is a general framework for the analysis of dynamical fluctuations in a large class of non-equilibrium systems. It also connects dynamical fluctuations with thermodynamic quantities like free energy and entropy production, and with associated non-equilibrium objects like the quasi-potential (which extends the thermodynamic free energy to non-equilibrium settings).

Here, we show how several features that appear in MFT can be attributed to a general structure that characterises dynamical fluctuations in microscopic Markov models. That is, the properties of the hydrodynamic (MFT) theory can be traced back to the properties of the underlying stochastic processes. Our approach builds on recent work by Mielke, Renger and M. A. Peletier, in which the analogue of the OM theory for reversible Markov chains has been described in terms of a generalised gradient-flow structure [43]. To describe non-equilibrium processes, that theory must be generalised to include irreversible Markov chains. This can be achieved using the canonical structure of fluctuations discovered by Maes and Netočný [38]. Extending their approach, we decompose currents in the system into two parts, and we identify a kind of orthogonality relationship associated with this decomposition. However, in contrast to the classical OM theory and to MFT, the large deviation principles that appear in our approach have non-quadratic rate functions, which means that fluxes have non-linear dependence on their conjugate forces. Thus, the idea of orthogonality between currents needs to be generalised, just as the notion of gradient flows in macroscopic equilibrium systems can be extended to generalised gradient flows.

The central players in our analysis are the probability density and the probability current j. For a given Markov chain, the relation between these quantities is fully encoded in the master equation, which also fully specifies the dynamical fluctuations in that model. However, thermodynamic aspects of the system—the roles of heat, free energy, and entropy production—are not apparent in the master equation. Within the Onsager–Machlup theory, these thermodynamic quantities appear in the action functional for paths, and solutions of the master equation appear as paths of minimal action. Hence, the structure that we discuss here, and particularly the decomposition of the current into two components, links the dynamical properties of the system to thermodynamic concepts, both for equilibrium and non-equilibrium systems.

Summary

We now sketch the setting considered in this article (precise definitions of the systems of interest and the relevant currents, densities and forces will be given in Sect. 2).

We introduce a large parameter , which might be the size of the system (as in MFT) or a large number of copies of the system (an ensemble), as considered for Markov chains in [39]. Then let be the (random) path followed by the system’s density and current, in the time interval [0, T]. Consider a random initial condition such that , asymptotically as , for some rate functional . Paths that in addition satisfy a continuity equation have the asymptotic probabilitywith the rate functionalhere is a force (see (12) below for the precise definition) and is what we call the generalised OM functional, which has the general formwhere is a dual pairing between a current j and a force f, while and are a pair of functions which satisfyas well as and . Note that (4) means that the two functions satisfy a Legendre duality. Moreover, these two functions and are strictly convex in their second arguments. Here and throughout, f indicates a force, while F is a function whose (density-dependent) value is a force.

The large deviation principle stated in (1) is somewhat abstract: for example, might be defined as a density on a discrete space or on , depending on the system of interest. Specific examples will be given below. In addition, all microscopic parameters of the system (particle hopping rates, diffusion constants, etc.) will enter the (system-dependent) functions , and F.

As a preliminary example, we recall the classical Onsager theory [36], in which one considers n currents and a set of conjugate applied forces . Examples of currents might be particle flow or heat flow, and the relevant forces might be pressure or temperature gradients. The large parameter corresponds to the size of a macroscopic system. The theory aims to to describe the typical (average) response of the current j to the force F, and also the fluctuations of j. In this (simplest) case, the density plays no role, so the force F has a fixed value in . The dual pairing is simply and is given by , where R is a symmetric matrix with elements . The Legendre dual of is , where is the Onsager matrix, whose elements are the linear response coefficients of the system. One sees that and can be interpreted as squared norms for currents and forces respectively. Denoting this norm by , one hasOn applying an external force F, the response of the current j is obtained as the minimum of , so (that is, ). One sees that measures the deviation of the current j from its expected value LF, within an appropriate norm. From the LDP (1), one sees that the size of this deviation determines the probability of observing a current fluctuation of this size.

In this article, we show in Sect. 2 that finite Markov chains have an LDP rate functional of the form (3), where (and thus ) are not quadratic. In that case, and j correspond to probability densities and probability currents, while the transition rates of the Markov chain determine the functions F, and . Since and measure respectively the sizes of the currents and forces, we interpret them as generalisations of the squared norms that appear in the classical case. The resulting is not a squared norm, but it is still a non-negative function that measures the deviation of j from its most likely value. This leads to nonlinear relations between forces and currents. The MFT theory [7] also fits in this framework, as we show in Sect. 4: in that case are a particle density and a particle current. However, there are relationships between the functions for MFT and for general Markov chains, as we discuss in Sect. 4.5.

Hence, the general structure of Eqs. (1)–(4) describes classical OM theory [36], MFT, and finite Markov chains. A benefit is that the terms have a physical interpretation. For a path , the time-reversed path is . Since both and are symmetric in their second argument and thus invariant under time reversal, it holds that . This allows us to identify as a rate of entropy production. In contrast, the term is symmetric under time reversal and encodes the frenesy (see [3]). Thus, within this general structure, the physical significance of Eqs. (1)–(4) is that they connect path probabilities to physical notions such as force, current, entropy production and breaking of time-reversal symmetry. Furthermore, we introduce in Sect. 3 decompositions of forces and the (path-wise) rate functional. Sect. 4 shows that some results of MFT originate from generalised orthogonalities of the underlying Markov chains derived in Sect. 3. Similar results hold for time-average large deviation principles, as shown in Sect. 5. In Sect. 6, we show how some properties of MFT can be derived directly from the canonical structure (1)–(4), independent of the specific models of interest. Hence these results of MFT have analogues in Markov chains. Finally we briefly summarise our conclusions in Sect. 7.

Onsager–Machlup Theory for Markov Chains

In this section, we collect results on forces and currents in Markov chains and on associated LDPs. In particular, we recall the setting of [38, 39]; other references for this section are for example [49] (for the definition of forces and currents in Markov chains) and [43] for LDPs.

Setting

We consider an irreducible continuous time Markov chain on a finite state space V with a unique stationary distribution that satisfies for all . The transition rate from state x to state y is denoted with . We assume that if and only if .

We restrict to finite Markov chains for simplicity: the theory can be extended to countable state Markov chains, but this requires some additional assumptions. Briefly, one requires that the Markov chain should be positively recurrent and ergodic (see for instance [9]), for which it is sufficient that (i) the transition rates are not degenerate: for all , and (ii) for each , the Markov chain started in x almost all trajectories of the Markov chain do not exhibit infinitely many jumps in finite time (“no explosion”). Second, one has to invoke a summability condition for the currents considered below (see, e.g., Eqs. 9 and 10), such that in particular the discrete integration by parts (or summation by parts) formula (15) holds. Finally, note that the cited result for existence and uniqueness of the optimal control potential (the solution to (70)) is only valid for finite state Markov chains.

As usual, we can interpret the state space of the Markov chain as a directed graph with vertices V and edges , such that if and only if . Let be a probability measure on V. We define rescaled transition rates with respect to asso that . With this notation, the detailed balance condition reads , so this equality holds precisely if the Markov chain is reversible (i.e. satisfies detailed balance). In general (not assuming reversibility), since is the invariant measure for the Markov chain, one has (for all x) thatWe further define the free energy on V to be the relative entropy (or Kullback–Leibler divergence) with respect to ,The probability current is defined as [49, Eq. (7.4)]Moreover, for a general current j such that , we define the divergence asWe say that j is divergence free if for every . The time evolution of the probability density is then given by the master equation(which is often stated as , with the (forward) generator ).

Non-linear Flux–Force Relation and the Associated Functionals and

To apply the theory outlined in Sect. 1.1, the next step is to identify the appropriate forces and also a set of mobilities . In this section we define these forces, following [38, 39, 49]. This amounts to a reparameterisation of the rates of the Markov process in terms of physically-relevant variables: an example is given in Sect. 3.5.

To each edge in E we assign a force F and a mobility a, asNote that , while : forces have a direction but the mobility is a symmetric property of each edge. The fact that depends on the density means that these forces act in the space of probability distributions. This definition of the force is sometimes also called affinity [49, Eq. (7.5)]; see also [1]. With this definition, the probability current (9) iswhich may be verified directly from the definition . In contrast to the classical OM theory, this is a non-linear relation between forces and fluxes, although one recovers a linear structure for small forces (recall the classical theory in Sect. 1.1, for which ).

Now consider a current j defined on E, with , and a general force f that satisfies (which is not in general given by (12)). Define a dual pair on E aswhere the summation is over all (the normalisation 1 / 2 appears because each connected pair of states should be counted only once, but E is a set of directed edges, so it contains both xy and yx, which have the same contribution to ).

We define the discrete gradient by . The discrete gradient and the divergence defined in (10) satisfy a discrete integration by parts formula: for any function , since , we have We will show in Sect. 2.3 that there is an OM functional associated with these forces and currents, which is of the form (3). Since and are convex and related by a Legendre transformation, it is sufficient to specify only one of them. The appropriate choice turns out to beThis means that defined in (3) is uniquely minimised for the current withas required for consistency with (13). From (4) and (14), one has alsowhereis the force required to induce the current j.

Physically, is a measure of the strength of the force f and is a measure of the magnitude of the current j. Consistent with this interpretation, note that and are symmetric in their second arguments. Moreover, for small forces and currents, and are quadratic in their second arguments, and can be interpreted as generalisations of squared norms of the force and current respectively. Note that Eqs. (16) and (18) can alternatively be represented asand

Large Deviations and the Onsager–Machlup Functional

As anticipated in Sect. 1.1, the motivation for the definitions of , , and F is that there is a large deviation principle for these Markov chains, whose rate function is of the form given in (2). This large deviation principle appears when one considers independent copies of the Markov chain.

We denote the ith copy of the Markov chain by and define the empirical density for this copy as , where is a Kronecker delta function. Let the times at which the Markov chain has jumps in [0, T] be . Further denote the state just before the kth jump with (such that the state after the kth jump is ). With this, the empirical current is given bywhere denotes a Dirac delta. Note that and the total probability is conserved, as (which holds for any discrete vector field with ). With a slight abuse of notation we define a similar empirical density and current for the full set of copies asNext, we state the large deviation principle where the OM functional appears. For this, we fix a time interval [0, T] and consider the large limit. We assume that the copies at time have initial conditions drawn from the invariant measure of the process (the generalisation to other initial conditions is straightforward). Then, the probability to observe a joint density and current over the time interval [0, T] is in the limit as given by (1). That is,withHere, is the force defined in (12) and the condition has to hold for almost all . Moreover, is of the form stated in (3), and the relevant functions , and are those of (16), (18) and (8). This LDP was formally derived in [38, 39]. Since the quantities defined in (22) are simple averages over independent copies of the same Markov chain, this LDP may also be proven by direct application of Sanov’s theorem, which provides an interpretation of as a relative entropy between path measures; we sketch the derivation in Appendix A. For finite-state Markov chains, (23) and (24) also follow (by contraction) from [48, Theorem 4.2], which provides a rigorous proof.

We emphasise that the arguments and j of the function correspond to the random variables that appear in the LDP, while the functions F, and that appear in encapsulate the transition rates of the Markov chain. Thus, by reparameterising the rates in terms of forces F and mobilities a, we arrive at a representation of the rate function which helps to make its properties transparent (convexity, positivity, symmetries such as (25)).

We note that for reversible Markov chains, the force is a pure gradient for some potential G (see Sect. 3), in which case one may write , which follows from an integration by parts and application of the continuity equation. In this case, Mielke, M. A. Peletier, and Renger [43] also identified a slightly different canonical structure to the one presented here, in which the dual pairing is , for a velocity and a potential G. The analogues of and in that setting depend on v and G respectively, instead of j and F. The setting of (3) and (4) is more general, in that the functions for the velocity/potential setting are fully determined by those for the current/force setting. Also, focusing on the velocity v prevents any analysis of the divergence-free part of the current, and restricting to potential forces does not generalise in a simple way to irreversible Markov chains. For this reason, we use the current/force setting in this work.

In a separate development, Maas [35] identified a quadratic cost function for paths (in fact a metric structure) for which the master equation (11) is the minimiser in the case of reversible dynamics. This metric corresponds to the solution of an optimal mass transfer problem which seems to have no straightforward extension to irreversible systems. Of course, in the reversible case, the pathwise rate function (24) has the same minimiser, but is non-quadratic and therefore does not correspond to a metric structure, so there is no simple geometrical interpretation of (24). It seems that the non-quadratic structure in the rate function is essential in order capture the large deviations encoded by (23).

Time-Reversal Symmetry, Entropy Production, and the Gallavotti–Cohen Theorem

The rate function for the large-deviation principle (23) is given by (24), which has been written in terms of forces F, currents j, and densities . To explain why it is useful to write the rate function in this way, we compare the probability of a path with that of its time-reversed counterpart , where as before.

In this case, the fact that and are both even in their second argument means thatThis formula is a (finite-time) statement of the Gallavotti–Cohen fluctuation theorem [19, 32]: see also [12, 37]. It also provides a connection to physical properties of the system being modelled, via the theory of stochastic thermodynamics [50]. The terms involving the free energy come from the initial conditions of the forward and reverse paths, while the integral of corresponds to the heat transferred from the system to its environment during the trajectory [50, Eqs. (18), (20)]. This latter quantity—which is the time-reversal antisymmetric part of the pathwise rate function—is related (by a factor of the environmental temperature) to the entropy production in the environment [37]. The definition of the force F in (12) has been chosen so that the dual pairing is equal to this rate of heat flow: this means that the forces and currents are conjugate variables, just as (for example) pressure and volume are conjugate in equilibrium thermodynamics. See also the example in Sect. 3.5.

Decomposition of Forces and Rate Functional

We now introduce a splitting of the force into two parts and , which are related to the behaviour of the system under time-reversal, as well as to the splitting of the heat current into “excess” and “housekeeping” contributions [50]. We use this splitting to decompose the function into three pieces, which allows us to compare (for example) the behaviour of reversible and irreversible Markov chains. This splitting also mirrors a similar construction within MFT [7], and this link will be discussed in Sect. 4. Related splittings have been introduced elsewhere; see [30] and [47] for decompositions of forces in stochastic differential equations, and [13] for decompositions of the instantaneous current in interacting particle systems.

Splitting of the Force According to Time-Reversal Symmetry

We define the adjoint process associated with the original Markov chain of interest. The transition rates of the adjoint process are . It is easily verified that the adjoint process has invariant measure , so . Under the assumption that the initial distribution is sampled from the steady state, the probability to observe a trajectory for the adjoint process coincides with the probability to observe the time-reversed trajectory for the original process.

From the definition of in (12), we can decompose this force aswithWith this choice, we note that the equivalent force for the adjoint processsatisfies . So taking the adjoint inverts the sign of (the “antisymmetric” force) but leaves unchanged (the “symmetric” force). For a reversible Markov chain, the adjoint process coincides with the original one, and .

Lemma 1

Given , with the mobility of (12), the forces and satisfy

Proof

From the definitions of , , and , one hasand . Hencewhere the last equality uses (7). This establishes (28).

In Sect. 4.4, we will reformulate the so-called Hamilton–Jacobi relation of MFT in terms of forces, and show that this yields an equation analogous to (28).

Physical Interpretation of and

In stochastic thermodynamics, one may identify as the housekeeping heat (or adiabatic entropy production) associated with a single transition from state x to state y, see [16, 50]. (Within the Markov chain formalism, there is some mixing of the notions of force and energy: usually an energy would be a product of a force and a distance but there is no notion of a distance between states of the Markov chain, so forces and energies have the same units in our analysis.) Hence is the rate of flow of housekeeping heat into the environment. The meaning of the housekeeping heat is that for irreversible systems, transitions between states involve unavoidable dissipated heat which cannot be transformed into work (this dissipation is required in order to “do the housekeeping”).

To obtain the physical interpretation of , we also defineFor a general path that satisfies , we also identifywhere we used (8), (15). That is, is the change in free energy induced by the current j. Moreover it is easy to see thatwhere denotes the functional derivative of the free energy given in (8). (Note that the functional derivative is simply in this case, since is defined on a discrete space. We retain the functional notation to emphasise the connection to the general setting of Sect. 1.1.) Also, the last identity in (30) can be phrased asThe same identity, with an integration by parts, shows thatEquation (31) shows that the symmetric force is minus the gradient of the free energy, so the heat flow associated with the dual pairing of j and is equal to (the negative of) the rate of change of the free energy. It follows that the right hand side of (25) can alternatively be written as .

We also recall from Sect. 2.2 that the force F acts in the space of probability densities: depends not only on the states x, y but also on the density . (Physical forces acting on individual copies of the system should not depend on since each copy evolves independently, but F includes entropic terms associated with the ensemble of copies.) To understand this dependence, it is useful to write . We also write the invariant measure in a Gibbs-Boltzmann form: , where U(x) is the internal energy of state x and is a normalisation constant. Then depends on the mean energy of the system, while is (the negative of) the mixing entropy, which comes from the many possible permutations of the copies of the system among the states of the Markov chain. From (31) one then sees that has two contributions: one term (independent of ) that comes from the gradient of the energy U and the other (which depends on ) comes from the gradient of the entropy. These entropic forces account for the fact that a given empirical density can be achieved in many different ways, since individual copies of the system can be permuted among the different states of the system.

Generalised Orthogonality for Forces

Recalling the definitions of Sect. 3.1, one sees that the current in the adjoint process satisfies an analogue of (13):Comparing with (27), one sees that the adjoint process may also be obtained by inverting (while keeping as it is). For the symmetric current is defined aswhich satisfies . It is the same for the process and the adjoint process, and also coincides with the current for reversible processes (where , or equivalently ). An analogous formula can also be obtained for the anti-symmetric current. With , the anti-symmetric current is defined asIt satisfies .

Let be the symmetric version of obtained from (16) with replaced by . (The Legendre transform of is similarly denoted ). This leads to a separation of in a term corresponding to and a term corresponding to .

Lemma 2

The two forces and defined in (27) satisfy

Using , Lemma 1 and the definition of , we obtain that the left hand side of (37) is given bywhich coincides with the right hand side of (37).

The physical interpretation of Lemma 2 is that the strength of the force can be written as separate contributions from and . The following corollary allows us to think of a generalised orthogonality of the forces and .

Proposition 3

(Generalised orthogonality) The forces and satisfy

This follows directly from Lemma 2 and the symmetry of .

We refer to Proposition 3 as a generalised orthogonality between and because is acting as generalisation of a squared norm (see Sect. 1.1), so (39) can be viewed as a nonlinear generalisation of , which would be a standard orthogonality between forces.

Moreover, Lemma 2 can be used to decompose the OM functional as a sum of three terms.

Corollary 4

Let be defined as in (3) with replaced by , and as defined in (29). Then

We use the definition of in (3) and (32) together with Lemma 2 to decompose aswhich proves the claim.

Recall from Sect. 1.1 that measures how much the current j deviates from the typical (or most likely) current . One sees from (40) that it can be large for three reasons. The first term is large if the current is pushing the system up in free energy (because D is the rate of change of free energy induced by the current j). The second term comes from the time-reversal symmetric (gradient) force , which is pushing the system towards equilibrium. The third term comes from the time-reversal anti-symmetric force ; namely, it measures how far the current j is from the value induced by the force .

Corollary 4 also makes it apparent that the free energy is monotonically decreasing for solutions of (11), which are minimisers of .

Corollary 5

The free energy is monotonically decreasing along minimisers of the rate function . Its rate of change is given by

For minimisers of the rate function one has . Hence (30) and Corollary 4 imply thatBoth and are non-negative, so is indeed monotonically decreasing.

Hamilton–Jacobi Like Equation for Markov Chains

It is also useful to note at this point an additional aspect of the orthogonality relationships presented here, which has connections to MFT (see Sect. 4). We formulate an analogue of the Hamilton–Jacobi equation of MFT, as follows. Definewhich we refer to as an extended Hamiltonian, for reasons discussed in Sect. 6.3 (see also Sect. IV.G of [7]).

The extended Hamilton–Jacobi equation for a functional is then (cf. equation (100) in Sect. 6.3) given byNote that the free energy defined in (8) solves (45), which follows from Proposition 3 (using (31) and that is symmetric in its second argument). In fact (see Proposition 13), the free energy is the maximal solution to this equation. In MFT, the analogous variational principle can be useful, as a characterisation of the invariant measure of the process. Here, one has a similar characterisation of the (non-equilibrium) free energy.

Since (45) with provides a characterisation of the free energy , which is uniquely determined by the invariant measure of the process, it follows that (45) must be equivalent to the condition that satisfies : recall (11). Writing everything in terms of the rates of the Markov chain and its adjoint, (45) becomeswhich must hold for all : from the definition of one then has , which is indeed satisfied if and only if is invariant (cf. Eq. (7)).

Example: Simple Ring Network

To illustrate these abstract ideas, we consider a very simple Markov chain, in which n states are arranged in a circle, see Fig. 1. So and the only allowed transitions take place between state x and states (to incorporate the circular geometry we interpret and ). In physics, such Markov chains arise (for example) as simple models of nano-machines or motors, where an external energy source might be used to drive circular motion [29, 53]. Alternatively, such a Markov chain might describe a protein molecule that goes through a cyclic sequence of conformations, as it catalyses a chemical reaction [31]. In both cases, the systems evolve stochastically because the relevant objects have sizes on the nano-scale, so thermal fluctuations play an important role.

Fig. 1

Illustration of a simple Markov chain with states arranged in a circle. The transition rates between states are . If the Markov chain is not reversible, there will be a steady-state probability current corresponding to a net drift of the system around the circle

To apply the analysis presented here, the first step is to identify forces and mobilities, as in (12). Let . The invariant measure may be identified by solving subject to . Finally, one computes the steady state current , where the right hand side is independent of x (this follows from the steady-state condition on ). The original Markov process has 2n parameters, which are the rates : these are completely determined by the independent elements of , the n mobilities and the current . The idea is that this reparameterisation allows access to the physically important quantities in the system.

From the definitions of and R, it may be verified thatand similarly Then writeIn this case, we can identify the three terms aswhich allows us to read off the mobility a and the forces and . The physical meaning of these quantities may not be obvious from these definitions, but we show in the following that reparameterising the transition rates in this way reveals structure in the dynamical fluctuations.

For example, equilibrium models (with detailed balance) can be identified via (for all x). In general is the (steady-state) entropy production associated with a transition from x to , see Sect. 3.2. The steady state entropy production associated with going once round the circuit is , as it must be [1].

Now consider the LDP in (23). We consider a large number () of identical nano-scale devices, each of which is described by an independent copy of the Markov chain. Typically, each device goes around the circle at random, and the average current is (so each object performs cycles per unit time). The LDP describes properties of the ensemble of devices. If is large and the distribution of devices over states is , then the (overwhelmingly likely) time evolution of this distribution is , where the current J obeys the simple formulawhich is (13), applied to this system. The simplicity of this expression motivates the parametrisation of the transition rates in terms of forces and mobilities. In addition, if one observes some current j [not necessarily equal to ] then the rate of change of free energy of the ensemble can be written compactly as , from (32). The quantity is the rate of dissipation via housekeeping heat (see Sect. 3.2). This (physically-motivated) splitting of motivates our introduction of the two forces and . Note that is the rate of heat flow from the system to its environment, and appears in the fluctuation theorem (25).

Finally we turn to the large deviations of this ensemble of nano-scale objects. There is an LDP (23), whose rate function can be decomposed into three pieces (Corollary 4), because of the generalised orthogonality of the forces and (Lemma 2). This splitting of the rate function is useful because the symmetry properties of the various terms yields bounds on rate functions for some other LDPs obtained from by contraction, see Sect. 5.

Connections to MFT

MFT is a field theory which describes the mass evolution of particle systems in the drift-diffusive regime, on the level of hydrodynamics. In this setting, it can be seen as generalisation of Onsager–Machlup theory [36]. For a comprehensive review, we refer to [7]. This section gives an overview of the theory, focussing on the connections to the results presented in Sects. 2 and 3.

We seek to emphasise two points: first, while the particle currents in MFT and the probability current in Markov chains are very different objects, they both obey large-deviation principles of the form presented in Sect. 1.1. This illustrates the broad applicability of this general setting. Second, we note that many of the particle models for which MFT gives a macroscopic description are Markov chains on discrete spaces. Starting from this observation, we argue in Sect. 4.5 that some results that are well-known in MFT originate from properties of these underlying Markov chains, particularly Proposition 3 and Corollary 4.

We consider a large number N of indistinguishable particles, moving on a lattice (indexed by , such that the number of sites is strictly increasing with L). These particles are described by a Markov chain, so the relevant forces and currents satisfy the equations derived in Sects. 2 and 3. The hydrodynamic limit is obtained by letting such that the total density converges to a fixed number . In this limit, the lattice is rescaled into a domain and one can characterise the system by a local (mass) density together with a local current , which evolve deterministically as a function of time [7, 28]. This time evolution depends on some (density-dependent) applied forces . The force at can be written aswhere the gradient denotes a spatial derivative, the function is a free energy density and is a drift. (The free energy is conventionally denoted by f [7]; here we use a different notation since f indicates a force in this work.) With these definitions, the deterministic currents satisfy the linear relation [41]which is the hydrodynamic analogue of (13). Here, is a (density-dependent) mobility matrix.

Onsager–Machlup Functional

Within MFT, the system is fully specified once the functions are given. These three quantities are sufficient to specify both the deterministic evolution of the most likely path , and the fluctuations away from it. We can again define an OM functional given byTo cast this functional in the form (3), we define the dual pair , together with the Legendre dualsGiven and f, we have that is uniquely minimised (and equal to zero) for the current .

Large Deviation Principle

Within MFT, one considers an empirical density and an empirical current. We emphasise that these refer to particles, which are interacting and move on the lattice ; this is in contrast to the case of Markov chains, where the copies of the system were non-interacting and one considers a density and current of probability. The averaged number of particles at site is denoted with , where is the image in the rescaled domain of site , and the corresponding particle current is given by (cf. Sect. VIII.F in [7] for details). Note that both the particle density and the particle current are random quantities (see also Sect. 4.5).

In keeping with the setting of Sect. 1.1, we focus on paths in the limit as , where the probability is, analogous to (1), given byNote that the parameter in (1), which is the speed of the LDP, corresponds to the lattice size . For the force defined in (49), the rate functional in (53) is given byHere is the quasipotential, which plays the role of a non-equilibrium free energy. We may think of as the macroscopic analogue of the free energy defined in (8). It is the rate functional for the process sampled from the invariant measure, which is consistent with the case for Markov chains in (24). We assume that has a unique minimiser , which is the steady-state density profile (so ).

An important difference between the Markov chain setting and MFT is that the OM functional for Markov chains is non-quadratic, which is equivalent to a non-linear flux force relation, whereas MFT is restricted to quadratic OM functionals.

Equation (53) is the basic assumption in MFT [7], in the sense that all systems considered by MFT are assumed to satisfy this pathwise LDP. In fact, both the process and its adjoint are assumed to satisfy such LDPs (with similar rate functionals, but different forces) [7].

Decomposition of the Force F

The force F in (49) can be written as the sum of a symmetric and an anti-symmetric part, , just as in Sect. 3.1. The force for the adjoint process is given by . Note that, unlike in the case of Markov chains, can here depend on . More precisely, and is given implicitly by .

The symmetric and anti-symmetric currents are defined in terms of the forces and as and . An important result in MFT is the so-called Hamilton–Jacobi orthogonality, which states thatIn terms of the forces and , we can restate (55) asThe latter is the quadratic version of the orthogonality (28) of Lemma 1; it is equivalent toor in other words, from (52),which is the result of Proposition 3 in the context of MFT. One can see (39), and hence Proposition 3, as the natural generalisation to the Hamilton–Jacobi orthogonality (55). Again, the MFT describes systems on the macroscopic scale, but the result (58) originates from the result (39), on the microscopic level.

Relating Markov Chains to MFT: Hydrodynamic Limits

We have discussed a formal analogy between current/density fluctuations in Markov chains and in MFT: the large deviation principles (23) and (53) refer to different objects and different limits, but they both fall within the general setting described in Sect. 1.1. We argue here that the similarities between these two large deviation principles are not coincidental—they arise naturally when MFT is interpreted as a theory for hydrodynamic limits of interacting particle systems.

To avoid confusion between particle densities and probability densities, we introduce (only for this section) a different notation for some properties of discrete Markov chains, which is standard for interacting particle systems. Let represent a state of the Markov chain (in place of the notation x of Sect. 2), and let be a probability distribution over these states (in place of the notation of Sect. 2). Let be the probability current.

We illustrate our argument using the weakly asymmetric simple exclusion process (WASEP) in one dimension, so the lattice is , and each lattice site contains at most one particle, so . The lattice has periodic boundary conditions and the occupancy of site i is . Particles hop to the right with rate and to the left with rate , but in either case only if the destination site is empty. Here E is a fixed parameter (an external field); the dependence of the hop rates on L is chosen to ensure a diffusive hydrodynamic limit (as required for MFT).

The spatial domain relevant for MFT is : site corresponds to position . For any probability measure on V, one can write a corresponding smoothed particle density on , aswhere is a smoothed delta function (for example a Gaussian with unit weight and width , or—more classically—a top-hat function of width , cf. [28]). Similarly if there is a probability current in the Markov chain, one can write a smoothed particle current aswhere is the configuration obtained from by moving a particle from site i to site ; if there is no particle on site i then define so that . Physically, is the average particle density associated to , and is the particle current associated to .

As noted above, MFT is concerned with the limit . The LDP (23) is not relevant for that limit (it applies when one considers many () independent copies of the Markov chain, with L being finite for each copy). However, the rate function that appears in (23) has an alternative physical interpretation, as the relative entropy between two path measures: see Appendix A. This relative entropy can be seen as a property of the WASEP; there is no requirement to invoke many copies of the system. Physically, the relative entropy measures how different is the WASEP from an alternative Markov process with a given probability and current .

The key point is that in cases where MFT applies, one expects that the rate function can be related to this relative entropy. In fact, there is a deeper relation between relative entropies and rate functionals: it can be shown that Large Deviation Principles are equivalent to -convergence of relative entropy functionals (see [42] for details).

Returning to the WASEP, we consider a particle density that satisfies . One then can find (for each L) a time-dependent probability and current , with , such on taking the limit after , the associated particle densities and moreoverIn order to find , one defines a “controlled” WASEP (similar to (69) in Sect. 5.3), in which the particle hop rates depend on position and time, such that the particle density in the hydrodynamic limit obeys .

For interacting particle systems, this “controlled” process is usually obtained by adding a time dependent external field to the system that acts on the individual particles. This was first derived for the symmetric SEP in [27] (see also [4] for a treatment of the zero-range process). For the WASEP (in a slightly different situation with open boundaries) a proof of (61) can e.g. be found in [6], Lemma 3.7.

Moreover, on decomposing and as in (3), the separate functions and obey formulae analogous to (61): this is the sense in which the structure of the MFT rate function is inherited from the relative entropy of the Markov chains. The quadratic functions and in MFT arise because the forces that appear in the underlying Markov chains are small (compared to unity), so second order Taylor expansions of and give in the limit the accurate description, similar to [2]. We will return to this discussion in a later publication.

LDPs for Time-Averaged Quantities

So far we have considered large deviation principles for hydrodynamic limits, and for systems consisting of many independent copies of a single Markov chain. We now show how some of the results derived in Sects. 2 and 3 also have analogues for large deviations for a single Markov chain, in the large-time limit.

Large Deviations at Level 2.5

Analogous to (22), we define the time averaged empirical measure of a single copy of the Markov chain and the time averaged empirical current as(where we choose and for the empirical density and current of the single Markov chain, as defined above in Sect. 2.3). For countable state Markov chains, the quantity satisfies a LDP as :We refer to such principles as level 2.5 LDPs. For countable state Markov chains the rate functional was derived in [39], and was proven rigorously in [8, 9] for Markov chains in the setting of Sect. 2.1 under some additional conditions (see [8, 9] for the details). We can recast the rate functional (see [8, Theorem 6.1]) aswith again given by (3), together with (14), (16) and (18).

We have stated this LDP for joint fluctuations of the density and the current. For Markov chains, the LDP for the density and the flow is also known as a level-2.5 LDP [9], so our general use of the name level-2.5 for (63) may be non-standard, but it seems reasonable. The rate functional for the density and the current in (63) can be obtained by contraction from the rate functional for the density and the flow (see Theorem 6.1 in [8]).

Using the splitting obtained in Sect. 3.3, we obtain the following representation for the rate functional on level-2.5.

Proposition 6

Let j be divergence free. Then the level-2.5 rate functional (64) is given by

We note from (33) that vanishes for divergence free currents j. The result then directly follows from Corollary 4.

Large Deviations for Currents

Proposition 6 is connected to recently-derived bounds on rate functions for currents, see [22, 23, 45, 46]. Indeed, the rate function for current fluctuations can be obtained by contraction from level-2.5, asThen, following [23, 46], it may be shown that for any one has for as in (3) with (14), (16)–(18) thatwith if ; otherwise is continuously extended by taking . Hence one has the result of [22], that the curvature of the rate function is controlled by the housekeeping heat , aswhere is the steady state current (recall (9)), and the ratio must again be interpreted as in the case where (and hence ) vanish. The first step in (68) comes from (66), the second step uses (65) as well as , and the third uses (67).

The significance of the splitting (65) for this result is that is the rate of flow of housekeeping heat associated with edge xy: the appearance of the housekeeping heat is natural since the bound comes from the second term in (65), which is independent of and depends only on .

Optimal Control Theory

It will be useful to introduce ideas of optimal control theory, whose relationship with large deviation theory is discussed in [10, 11, 18, 24]. In parallel with our given transition rates we introduce a new process, the controlled process, where the rates are modified by a control potential , asFor a given probability distribution , we seek a potential such that the controlled process has invariant measure . For this we needor equivalentlyWe stress that, for any fixed , (70) is equivalent to solving the minimisation problemwhich is also equivalent to maximisation of the Donsker–Varadhan functional, see for example Chapter IV.4 in [15]. A proof for the existence and uniqueness of can, e.g., be found in [40]. Now assume that solves (70). The resulting controlled process depends on and has rates given by (69). Throughout this section, we use tildes to indicate properties of the controlled process: all these quantities depend implicitly on the fixed probability . Hence the (time-dependent) measure of the controlled process is .

Repeating the analysis of Sect. 2.1 and noting that , we find that . Also, the force for the controlled process iswhich may be decomposed asThus, the symmetric force in the controlled process vanishes when . The antisymmetric force represents the force observed in the new non-equilibrium steady state . If the original process is reversible, then so .

It is useful to define and to identify the steady-state current for the controlled process as

Decomposition of Rate Functions

The ideas of optimal control theory are useful since they facilitate the further decomposition of the level-2.5 rate function into several contributions.

Lemma 7

Suppose that and j are given and that . Thenwhere is given by (74), evaluated in the optimally controlled process whose steady state is .

We writewhere the first line is (3) and (64); the second line is simple rewriting; and the third uses the definition of in (3) and also (72) with .

The current satisfies so one has (by definition of ) that . Using this relation together with (72) and (76), one hasFinally we note that (since is the invariant measure for the controlled process) and (by assumption), so integration by parts yields ; using once more the definition of yields (82).

The physical interpretation of (75) is as follows. The contribution is a rate functional for observing an empirical current j in the controlled process, while is the rate functional for observing an empirical current in the original process. Since is the (deterministic) probability current for the controlled process, one has that the more the controlled process differs from the original one, the larger will be . Hence the level-2.5 rate functional is large if the controlled process is very different from the original one, as one might expect. The rate functional also takes larger values if the empirical current j is very different from the probability current of the controlled process.

We obtain our final representation for the level-2.5 rate functional, consisting of the sum of three different OM functionals.

Proposition 8

Let j be divergence free. We can represent the level-2.5 rate functional (64) as

This follows immediately from Lemma 7 followed by an application of Corollary 4 to and that , from (33).

The three terms in (78) also appear in Lemma 7 and Corollary 4, and their interpretations have been discussed in the context of those results. Briefly, we recall that sets the probability of fluctuations in which a non-typical density and current j are sustained over a long time period. The first term in (78) reflects the fact that the free-energy gradient tends to push towards the steady state , so maintaining any non-typical density is unlikely if is large. Similarly, the second term in (78) reflects the fact that large non-gradient forces also tend to suppress the probability that maintains its non-typical value. The final term is the only place in which the (divergence-free) current j appears: it vanishes if the current j is typical within the controlled process (see Corollary 9); otherwise it reflects the probability cost of maintaining a non-typical circulating current.

Large Deviations at Level 2

As well the LDP (63), we also consider an (apparently) simpler object, called a level-2 LDP, where one considers the density only. It is formally given byThe contraction principle for LDPs [52, Sect. 3.6] states thatEquation (75) is uniquely minimised in its second argument for the divergence free current , such that the contraction over all divergence-free vector fields j yields the level-2 rate functionalThe same splitting as above finally allows us to write the level 2 rate functional as follows.

Corollary 9

The level-2 rate functional can be written as the sum

This follows from (80) and (78), since has a minimal value of zero.

This last identity extends the results obtained in [26] on the accelerated convergence to equilibrium for irreversible processes using LDPs from the macroscopic scale (i.e. in the regime of MFT) to Markov chains. The level-2 rate function in (82) can be interpreted as a rate of convergence to the steady state. It was shown in [26] that the rate is higher for irreversible processes, as opposed to reversible ones (as the second term for reversible processes). We remark that splitting techniques for irreversible jump processes have been used to devise efficient MCMC samplers; see for example [5, 34].

Connection to MFT

Under the assumption that no dynamical phase transition takes place, the time averaged density and current in MFT (recall Sect. 4.3 for definitions) also satisfy a joint LDP in the limit : one takes first and then , see [26, Eq. (36)]. The LDP is similar to (63):where the rate function is, for a density profile and a current j with , given byAs for Markov chains (see Sect. 5.1) if j is not divergence free. If then the rate function can be written in the form [26]such that a contraction to to the density only yieldsThe function in (85) and (86) is obtained by solvingClearly the solution depends on . In essence, we have reduced the minimisation problem (80) to the solution of this PDE. Comparing with (78), we identify the terms in the MFT setting, and also , so . We obtain the following representations for (85) and (86) reminiscent of Proposition 8 and Corollary 9.

Proposition 10

The rate functional for the joint density and current in MFT, which is given by (85), can be written in terms of the OM functional (51) asand (86), the rate functional for the density in MFT, is given by

This proposition is equivalent to Proposition 5 of [26], but has now been rewritten in the language of optimal control theory. As discussed in [26], Eq. (89) quantifies the extent to which breaking detailed balance accelerates convergence of systems to equilibrium, at the hydrodynamic level. For this work, the key point is that this result originates from Corollary 9, which is the equivalent statement for Markov chains (without taking any hydrodynamic limit).

Consequences of the Structure of the OM Functional

We have shown that the rate functions for several LDPs in several different contexts depend on functionals with the general structure presented in (3) and (4). In this section, we show how this structure alone is sufficient to establish some features that are well-known in MFT. This means that these results within MFT have analogues for Markov chains. Our derivations mostly follow the standard MFT routes [7], but we use a more abstract notation to emphasise the minimal assumptions that are required.

Assumptions

The following minimal assumptions are easily verified for Markov chains; they are also either assumed or easily proven for MFT. The results of this section are therefore valid in both settings.

We consider a process described by a time-dependent density and current j, with an associated continuity equation and unique steady state . We are given a set of (-dependent) forces denoted by , a dual pairing between forces and currents, and a function which is convex in j and satisfies . With these choices, the functions and are fully specified via (3) and (4). We assume that for initial conditions chosen from the invariant measure, the system satisfies an LDP of the form (1) with rate function of the form (2).

We define an adjoint process for which the probability of a path is equal to the probability of the time-reversed path in the original process. As above, we define . We assume that the adjoint process also satisfies an LDP of the form (1), with rate function . Hence we must haveMoreover, we assume that may be obtained from I by replacing the force with some adjoint force . That is,Here, is the rate function associated with fluctuations of the density , for a system in its steady state. That is, within the steady state, . For Markov chains, , the free energy; for MFT we have , the quasipotential. In the following we refer to as the free energy.

Symmetric and Anti-symmetric Forces

DefineAs the following proposition shows, is connected to the gradient of the free energy (or quasipotential) , and the forces and satisfy a generalised orthogonality (in the sense of Proposition 3). The proof follows Section II.C of [7], but uses only the assumptions of Sect. 6.1, showing that the result applies also to Markov chains.

Proposition 11

The forces and satisfyand

Combining (90) and (91), we obtain (for any path that obeys the continuity equation )Differentiating with respect to T and using (3) together with and (92), one hasUsing the continuity equation and an integration by parts, one finds , so thatThis equation must hold for all , which means that the two terms in square parentheses both vanish separately. Combining the last equation with (92), we obtain (93) and (94).

Proposition 11 also yields a variational characterisation of . The following corollary is analogous to Eq. (4.8) of [7], as is its proof.

Corollary 12

The free energy satisfieswhere the infimum is taken over all paths that satisfy , as well as and . Moreover, the optimal path is given by the time reversal of the solution of the adjoint dynamics .

We obtain from (95) (together with (2) and (90)) thatTaking the infimum on both sides yields (96); indeed the infimum of is 0, and this infimum is attained uniquely for the optimal path for (96). To see this, we note that is uniquely minimised for , and satisfies the conditions above, so the optimal path is indeed the time-reversal of the solution of the adjoint dynamics.

Hamilton–Jacobi Like Equation for the Extended Hamiltonian

Another important relationship within MFT is the Hamilton–Jacobi equation [7, Eq. (4.13)]. This provides a characterisation of the quasipotential, as its maximal non-negative solution. The following formulation of that result uses only the assumptions of Sect. 6.1 and therefore applies also to Markov chains. The functionalcan be interpreted as an extended Lagrangian. (Note that should not be interpreted as a Lagrangian in the classical sense, as it depends on density and current , rather than the pair consisting of density and associated velocity ). We follow Sect. IV.G of [7]: given a sample path ), define a vector field . The initial condition is chosen so that there is a bijection between the paths and . For example, in finite Markov chains, define as a constant density, normalised to unity, and let , where h solves , see [13] for the relevant properties of these vector fields. With this choice, and using , one has for all t, and one may also write (formally) . Comparing with [7, Sect. IV.G], we write instead of since for Markov chains one has (for any discrete vector field A) that , so it is not possible to solve if is normalised to unity (recall that discrete vector fields have by definition  [13]).

The fluctuations of A are therefore determined by the fluctuations of , so the LDP (1) implies a similar LDP for A, whose rate function is , where is a Lagrangian that depends on A and its time derivative (which we again refer to as extended Lagrangian, cf. [7]). The function in (97) is then related to via the bijection between and A. Considering again the case of Markov chains, the time evolution of the system depends only on (which is ) and not on A itself, one sees that depends only on and (which is j). Hence we write, formally, , and we recover (97).

Hence is nothing but the extended Lagrangian , written in different variables: for this reason we refer to as an (extended) Lagrangian.

To arrive at the corresponding (extended) Hamiltonian, one should write , or equivalentlywhere is a conjugate field for the current j. We identify as the scaled cumulant generating function associated with the rate function  [52, Sect. 3.1]. Analysis of rare fluctuations in terms of the field is often more convenient than direct analysis of the rate function [32, 33] and is the basis of the “s-ensemble” method that has recently been exploited in a number of physical applications (for example [21, 24]). Using (3) and (4), we obtain(This generalises the definition (44), which was restricted to Markov chains.)

To relate this extended Hamiltonian to the free energy (quasipotential), one can define an extended Hamilton–Jacobi equation, which is for a functional given byThe relation of this equation to the free energy is given by the following proposition, which mirrors equation (4.18) of [7], but now in our generalised setting, so that it applies also to Markov chains.

Proposition 13

The free energy is the maximal non-negative solution to (100) which vanishes at the steady state . In other words, any functional that solves (100) and has also satisfies .

From (92), (93), (94) and , one hasThus (99) yields , so does indeed solve (100). In addition, (101) is valid also with replaced by any that solves (100); combining this result with (3) yieldswhere the second step uses . Moreover, for any path with and , we have from (102) thatwhere the final equality uses an integration by parts, together with the continuity equation. Finally, taking the infimum over all paths and using Corollary 12, one obtains , as claimed.

Generalisation of Lemma 2

Before ending, we note that (94) is analogous to Proposition 3 in the general setting of this section, but we have not yet proved any analogue of Lemma 2. Hence we have not obtained a generalisation of Corollary 4, nor any of its further consequences. To achieve this, one requires a further assumption within the general framework considered here, which amounts to a splitting of the Hamiltonian. This assumption holds for MFT and for Markov chains, and is a sufficient condition for a generalised Lemma 2.

To state the assumption, we consider a reversible process in which the forces are . (For Markov chains we should consider the process with rates ; for MFT it is the process with and the same mobility as the original process.) We assume that such a process exists and that its Hamiltonian can be written as for some function (compare (99) and see Sect. 3.4 for the case of Markov chains). Also let the Hamiltonian for the adjoint process be , which is constructed by replacing F by in (99). Then, one assumes further thatwhich may be verified to hold for Markov chains and for MFT. Writing and using (99) with (94) and , one then obtainswhich is the promised generalisation of Lemma 2.

Conclusion

In this article, we have presented several results for dynamical fluctuations in Markov chains. The central object in our discussion has been the function , which plays a number of different roles—it is the rate function for large deviations at level 2.5 (Eq. 64), and it also appears in the rate function for pathwise large deviation functions (Eq. 2). These results—derived originally by Maes et al. [38, 39]—originate from the relationship between and the relative entropy between path measures (Appendix A). The canonical (Legendre transform) structure of (Eq. 4) and its relation to time reversal (Eq. 25) have also been discussed before [38].

The function depends on probability currents j and their conjugate forces f. Our Proposition 3 and Corollary 4 show how the rate functions in which appears have another level of structure, based on the decomposition of the forces F in two pieces , according to its behaviour under time-reversal. A similar decomposition is applied in MFT [7]: the discussion of Sects. 5 and 6 show how several results of that theory—which applies on macroscopic (hydrodynamic) scales—already have analogues for Markov chains, which provide microscopic descriptions of interacting particle systems. These results—which concern symmetries, gradient structures and (generalised) orthogonality relationships—show how properties of the rate functions are directly connected to physical ideas of free energy, dissipation, and time-reversal.

Looking forward, we hope that these structures can be exploited both in mathematics and physics. From a mathematical viewpoint, the canonical structure and generalised orthogonality relationships may provide new routes for scale-bridging calculations, just as the geometrical structure identified by Maas [35] has been used to develop new proofs of hydrodynamic limits [17]. In physics, a common technique is to propose macroscopic descriptions of physical systems based on symmetries and general principles—examples in non-equilibrium (active) systems include [51, 54]. However, this level of description leaves some ambiguity as to the best definitions of some physical quantities, such as the local entropy production [44]. We hope that the structures identified here can be useful in relating such macroscopic theories to underlying microscopic behaviour.