The Impact of Sparse Coding on Memory Lifetimes in Simple and Complex Models of Synaptic Plasticity.

Models of associative memory with discrete state synapses learn new memories by forgetting old ones. In the simplest models, memories are forgotten exponentially quickly. Sparse population coding ameliorates this problem, as do complex models of synaptic plasticity that posit internal synaptic states, giving rise to synaptic metaplasticity. We examine memory lifetimes in both simple and complex models of synaptic plasticity with sparse coding. We consider our own integrative, filter-based model of synaptic plasticity, and examine the cascade and serial synapse models for comparison. We explore memory lifetimes at both the single-neuron and the population level, allowing for spontaneous activity. Memory lifetimes are defined using either a signal-to-noise ratio (SNR) approach or a first passage time (FPT) method, although we use the latter only for simple models at the single-neuron level. All studied models exhibit a decrease in the optimal single-neuron SNR memory lifetime, optimised with respect to sparseness, as the probability of synaptic updates decreases or, equivalently, as synaptic complexity increases. This holds regardless of spontaneous activity levels. In contrast, at the population level, even a low but nonzero level of spontaneous activity is critical in facilitating an increase in optimal SNR memory lifetimes with increasing synaptic complexity, but only in filter and serial models. However, SNR memory lifetimes are valid only in an asymptotic regime in which a mean field approximation is valid. By considering FPT memory lifetimes, we find that this asymptotic regime is not satisfied for very sparse coding, violating the conditions for the optimisation of single-perceptron SNR memory lifetimes with respect to sparseness. Similar violations are also expected for complex models of synaptic plasticity.

Introduction

One line of experimental evidence suggests that synapses may occupy only a very limited number of discrete states of synaptic strength (Petersen et al. 1998; Montgomery and Madison 2002, 2004; O’Connor et al. 2005a, b; Bartol et al. 2015), or may change their strengths via discrete, jump-like processes (Yasuda et al. 2003; Bagal et al. 2005; Sobczyk and Svoboda 2007). Discrete state synapses overcome the catastrophic forgetting of the Hopfield model (Hopfield 1982) in associative memory tasks, turning memory systems into so-called palimpsests, which learn new memories by forgetting old ones (Nadal et al. 1986; Parisi 1986). Unfortunately, memory lifetimes in the simplest such models are rather limited, growing only logarithmically with the number of synapses (Tsodyks 1990; Amit and Fusi 1994; see also Leibold and Kempter 2006; Barrett and van Rossum 2008; Huang and Amit 2010). Memory lifetimes may be extended by considering either sparse coding at the population level (Tsodyks and Feigel’man 1988) or complex models of synaptic plasticity in which synapses can express metaplasticity (changes in internal states) without necessarily expressing plasticity (changes in strength) (Fusi et al. 2005; Leibold and Kempter 2008; Elliott and Lagogiannis 2012; Lahiri and Ganguli 2013). Two previous studies have examined complex models of synaptic plasticity operating in concert with sparse coding (Leibold and Kempter 2008; Rubin and Fusi 2007). For a discussion of the possible roles of the persistence and transience of memories and the synaptic mechanisms underlying synaptic stability, see, for example, Richards and Frankland (2017), and Rao-Ruiz et al. (2021).

We have proposed integrate-and-express models of synaptic plasticity in which synapses act as low-pass filters in order to control fluctuations in developmental patterns of synaptic connectivity (Elliott 2008; Elliott and Lagogiannis 2009). We have also applied these complex models of synaptic plasticity to memory formation, retention and longevity with discrete synapses (Elliott and Lagogiannis 2012), finding that they outperform cascade models (Fusi et al. 2005) in most biologically relevant regions of parameter space (Elliott 2016b). In this paper, we consider the role of sparse coding in the memory dynamics of a filter-based model. For comparison, we also consider the cascade model (Fusi et al. 2005), the serial synapse model (Leibold and Kempter 2008; Rubin and Fusi 2007) and a model of simple synapses (Tsodyks 1990) using our protocols.

Our paper is organised as follows. In Sect. 2, we present our general approach by describing the two memory storage protocols that we study, considering two different definitions of memory lifetimes, and obtaining general, model-independent results. Then, in Sect. 3, we consider both simple and complex models of synaptic plasticity, obtaining the analytical results required to study memory lifetimes in detail. We compare and contrast results for memory lifetimes in simple and complex models in Sect. 4. Finally, in Sect. 5, we briefly discuss our results.

General approach and formulation

We provide a convenient list of the most commonly used mathematical symbols and their meanings, excluding those that appear in the appendices, in Table 1.

Table 1

List of frequently used mathematical symbols and their meanings (excluding those in Appendix B)

Symbol	Meaning
P	Number of neurons in memory system
N	Number of synaptic inputs received by each neuron
N=NP	Total number of synapses in memory system
r	Rate of Poisson process for memory storage
S_i(t)	Strength of synapse i, i=1,…,N, at time t≥0 s for a typical perceptron in the population
ξ_^α	Memory α, α=0,1,2,…, to be stored by a single perceptron at Poisson storage step α
h(t)=h_ξ_^0(t)	Memory signal for the tracked memory stored at t=0 s
h_α	Memory signal for the tracked memory immediately after the storage of memory α
h_0	h_0=h(0), the initial perceptron activation induced by the tracked memory ξ_^0
μ(t), σ(t)^2, SNR(t)	Mean, variance and signal-to-noise ratio of h(t), the single-perceptron memory signal
τ_snr	SNR memory lifetime of a typical memory, for a single perceptron
τ_mfpt(h_0)	MFPT memory lifetime conditioned on a definite activation h_0 induced by a definite, tracked memory
ϑ	Perceptron’s firing threshold
τ_mfpt	Conditional MFPT τ_mfpt(h_0) averaged over all tracked memories with h_0>ϑ
σfpt2	Variance in the FPT
A(h), B(h)	Jump moments in Fokker–Planck equation
ζ	Level of spontaneous electrical activity, with 0≤ζ<1
f, g	Probabilities of evoked (rather than spontaneous) pre- and postsynaptic (respectively) activities; f≡g
s	Number of internal synaptic states for each of the two possible states of synaptic strength
M_+, M_-; M	Potentiating and depressing matrices describing transitions in a single synapse’s state; M=12M_++M_-
K_±; K	K_±=(1-f)I+fM_±, I being the identity matrix; K=12K_++K_-
T_n	Operator describing simultaneous changes in n synapses’ states at each (average) non-tracked memory storage step
A__n	Normalised unit eigenstate of T_n, giving the joint equilibrium probability distribution of n synapses’ states
Ω_	Ω_^T=-1_^T|+1_^T, the vector by which to weight a synapse’s internal states by their strengths
P_eff	Number of neurons in the population of P neurons that experience evoked activity during tracked memory storage
μ_p(t), σ_p(t)^2, SNR_p(t)	Mean, variance and signal-to-noise ratio of the population memory signal
τ_pop	SNR memory lifetime of a typical memory for the population of neurons
p, ψ	Probability p of synaptic updates for a simple stochastic updater (SU) synapse; ψ=fp is a convenient shorthand
κ_2	κ_2=ψ/(2-ψ), the equilibrium correlation between pairs of SU synapses’ strengths in the Hebb protocol
B_0	B_0=B(0)/(ψg), so that part of B(h) that is independent of h, up to scaling factors
N_eff	Number of a single perceptron’s synapses that experience evoked presynaptic activity during tracked memory storage
Θ	Synaptic filter threshold
X^opt	For any parameter X, the label “opt” indicates that value of X that maximises τ_snr or τ_pop
τsnropt, τpopopt	Maximum values of τsnropt or τpopopt, optimised with respect to some parameter X

Memories and memory lifetimes

We consider a population of P neurons forming a memory system, perhaps performing association or auto-association tasks. Let each neuron receive N synaptic connections from N other neurons that are randomly selected from the entire population. Fully recurrent connectivity would imply that (excluding self-connections) but in general . Other than the requirement that , N may be regarded as mathematically independent of P. Memories are stored sequentially, one after the other, by this memory system. We take them to be stored at times  s governed by a Poisson process of rate r Hz. This continuous time approach is more realistic than a discrete time approach in which memories are stored at uniformly spaced time steps. Due to ongoing synaptic plasticity driven by the storage of later memories, the synaptic patterns that embody earlier memories may be degraded, so that the fidelity of recall of earlier memories may fall over time, ultimately falling to an equilibrium or background level of complete amnesia. It is typical in these scenarios to track the fidelity of recall of the first memory as subsequent memories are stored. This first memory is taken to be stored at time  s on the background equilibrium probability distribution of synaptic strengths.

In previous work, we have focused on a single neuron, or perceptron, in such a system and have examined its recall of stored memories. Here, in a sparse population coding context, we must consider the collective dynamics of the entire population of neurons, but these collective dynamics are nevertheless driven by synaptic processes occurring at the level of single perceptrons in the system. Considering, then, a single perceptron in this population, let its N synapses have strengths , , at time  s. These two strength states should be thought of as low and high rather than inhibitory and excitatory. As memories are presented to the system for storage, the perceptron is exposed to synaptic inputs characterised by the N-dimensional vectors , , where indexes the memories. The component represents the input through synapse i during the presentation of memory , and for simplicity we assume that these components are independent between synapses and across memories.

In response to each of these memory vectors, the perceptron must generate the correct activation or output. With inputs through its N synapses, the perceptron’s activation is defined as usual byThe perceptron’s output is some possibly nonlinear function of its activation, where this output can correspond to spontaneous activity under conditions of no (or spontaneous) input. We track the fidelity of recall of the first memory by examining the perceptron’s activation upon re-presentation (but not re-storage) of this memory at later times  s. We refer to as the tracked memory signal or just the memory signal. The dynamics of h(t) will determine the lifetime of memory , at least as far as this single perceptron’s capacity to generate the correct output upon re-presentation of is concerned. Of course, we are not interested in the lifetime of any particular tracked memory stored on any particular pattern of synaptic connectivity and subject to any particular sequence of subsequent, non-tracked memories , , stored at any particular set of Poisson-distributed times . Rather, we are interested only in the lifetime of a typical tracked memory subject to a typical sequence of later memories. Thus, we consider only the statistical properties of h(t) when suitably averaged over all memories.

Memory lifetimes may be defined in a variety of ways using these statistical properties. The simplest definition is to consider the mean and variance of h(t) define the signal-to-noise ratio (SNR) as:and then define the memory lifetime as that value of t, call it , that is the (largest) finite, non-negative solution of the equation when this solution exists; otherwise, we set  s (Tsodyks 1990). This is the last time at which is distinguishable from its equilibrium value at the level of one standard deviation. Although an “ideal observer” approach to defining memory lifetimes has also been considered (Fusi et al. 2005; Lahiri and Ganguli 2013), it is essentially equivalent to the SNR approach (Elliott 2016b).

The activation h(t) provides a direct read-out of the perceptron’s response to the re-presentation of at later times, and would correspond to a neuron’s membrane potential in a more realistic, integrate-and-fire model. By focusing on this read-out of the perceptron’s state, we are naturally led to consider the first passage time (FPT) for the perceptron’s activation to fall below firing threshold, and thus to consider the mean first passage time (MFPT) for this process, which is the mean taken over all tracked and non-tracked memories (Elliott 2014). We may then define an alternative memory lifetime, call it , as this MFPT for the perceptron’s activation in response to re-presentation of a typical tracked memory to fall below firing threshold. We have extensively discussed and contrasted the SNR and FPT approaches to defining memory lifetimes elsewhere (Elliott 2014, 2016a, 2017a). In essence, SNR memory lifetimes are only valid in asymptotic, typically large N regimes, while FPT memory lifetimes are valid in all regimes. SNR lifetimes must therefore be interpreted with caution.

To compute FPT lifetimes, we require , the transition probability describing the probability that the perceptron’s activation (in response to re-presentation of the tracked memory) is immediately after the storage of average non-tracked memory , given that its activation is immediately before the memory’s storage. This transition probability is most easily computed in simple models of synaptic plasticity, for which it is independent of the memory storage step (Elliott 2014, 2017a, 2019). This independence arises because simple synapses with only two strength states are “stateless” (Elliott 2020), having no internal states and not enough strength states to carry information between consecutive memory storage steps. In this case, all the probabilities over all the possible, discrete values of and define the elements of a transition matrix in the perceptron’s activation between memory storage steps that is independent of the non-tracked memory storage step (). We can then drop the index and consider general elements between any two possible values of the perceptron activation, h and . We will therefore examine FPT lifetimes only for simple synapses, and SNR memory lifetimes for both simple and complex synapses, but with the understanding that SNR results must be interpreted cautiously. With the transition probabilities being independent of the memory storage step, and with the storage of the definite tracked memory inducing the definite activation immediately after its storage, the FPT lifetime of the memory is the solution of the equationwhere transitions to activations below the firing threshold are disallowed (Elliott 2014; see van Kampen 1992, for a general discussion). Equation (4) generalises in the obvious way to an integral equation when h can be regarded as a continuous rather than discrete variable. Solving Eq. (4) for for all values of entails solving a linear system involving the perceptron activation transition matrix. We therefore refer to Eq. (4) for simplicity but somewhat inaccurately as a matrix equation, to distinguish it from its integral equation equivalent in the continuum limit. This matrix or integral equation (MIE) approach to FPTs is exact. We may also consider an approximation involving the Fokker–Planck equation (FPE) approach by computing the jump moments induced by . Because memories are stored as a Poisson process, the jump moments are simply where the expectation values are calculated with respect to . Then, standard methods (Elliott 2014; see van Kampen 1992, in general) give the MFPT as the solution of the equation:subject to the boundary condition . Equations similar to Eqs. (4) and (6) give the higher-order FPT moments (Elliott 2019). Given , we obtain by averaging over the distribution of (for values of ), corresponding to averaging over the tracked memory , i.e. .

Hebb protocol

We adopt and adapt the memory storage protocol employed by Leibold and Kempter (2008). Their memory system performs an association task. Within the population of P neurons, a sub-population of “cue” neurons is required to activate a sub-population of “target” neurons. Synapses from cue to target neurons experience potentiating induction signals during memory storage, while those from target to cue experience depressing induction signals; all other synapses do not experience plasticity induction signals. Although Leibold and Kempter (2008) do allow for the possibility of some overlap between cue and target sub-populations, this will not be relevant here. The storage of different memories involves different cue and target sub-populations, so that the entire population of P neurons will be involved in storing many memories over time. If cue and target sub-populations are of equal size, as we assume, then potentiation and depression processes are equally balanced on average. This assumption stands in lieu of realistic neuron models, in which we expect (Elliott 2016a) synaptic plasticity to be dynamically regulated to move to stable dynamical fixed points in which such balancing is achieved automatically (Bienenstock et al. 1982; Burkitt et al. 2004; Appleby and Elliott 2006).

While Leibold and Kempter (2008) consider activities , corresponding to inactive () and active () input neurons, we will consider the more general case of , with , where represents a spontaneous, non-evoked, or background level of activity for an input neuron that is in neither cue nor target sub-populations, while represents evoked activity from a cue or target input. We often refer below to active and inactive inputs or neurons, with the understanding that we mean evoked activity and spontaneous activity, respectively. Because synaptic plasticity occurs only between cue and target neurons, synapses between a pre- or postsynaptic neuron that is only spontaneously active do not undergo synaptic plasticity. This accords with our expectations from known physiology: protocols for long-term potentiation (LTP; Bliss and Lømo 1973) and long-term depression (LTD; Lynch et al. 1977) require sustained bouts of evoked electrical activity rather than just spontaneous levels of activity. On a broadly BCM view of synaptic plasticity (Bienenstock et al. 1982), we would expect two thresholds for synaptic plasticity: as activity levels ramp up from spontaneous to weak to strong tetanisation, plasticity switches from none to LTD to LTP. Since synaptic plasticity can only occur between pairs of active, synaptically coupled neurons in this scenario, we refer to it as the Hebb protocol: Hebbian synaptic plasticity is typically understood to mean activity-dependent, bidirectional synaptic plasticity between active pre- and postsynaptic neurons. Although spontaneous activity has by assumption no impact on synaptic plasticity here, it nevertheless has a direct impact on h(t).

For a particular perceptron, let the probability that it is active during the storage of any particular memory be g. Since the perceptron could be part of either cue or target sub-populations, the probability that it is either cue or target during the storage of a memory is just . The probability that any one of its synaptic inputs is active during memory storage is also just g. However, for the purposes of clarity it is convenient to distinguish between these two probabilities, so we denote the probability that an input is active as f (). In this way, the appearance of a factor of g indicates a global, postsynaptic factor due to the perceptron, or postsynaptic cell, being in the cue or target population, while a factor of f indicates a local, presynaptic factor due to an input being in the cue or target population. The probability g, or f, controls the sparseness of the memory representation in this memory system. Considering just a single perceptron, if it is neither cue nor target, then none of its synapses can experience plasticity induction signals. If it is a cue, then only those inputs that correspond to target cells (if any) experience plasticity induction signals, and specifically depressing signals. If it is a target, then similarly only cue inputs experience induction signals, and so only potentiating signals. Without loss of generality, we may therefore just assume that during memory storage, an active perceptron’s active inputs are either all cue or all target neurons. This simplifying assumption effectively doubles on average the rate of plasticity induction signals experienced by synapses compared to the scenario in which the perceptron’s active inputs could represent a combination of cue and target neurons. We could therefore just scale f accordingly.

We summarise the Hebb protocol in Fig. 1, which schematically illustrates a sample of the population of pairs of pre- and postsynaptic neurons, showing all possible combinations of presynaptic activities and postsynaptic roles with their respective probabilities, together with the direction of synaptic plasticity induced by them.

Fig. 1

Schematic illustration of the Hebb protocol for memory storage. Six pairs of synaptically coupled neurons are shown. Each cell body is represented by a triangle, with the value ( or 1) inside the triangle indicating the neuron’s activity during memory storage. A neuron’s axon is denoted by a directed line, while two of its dendrites are denoted by the dashed lines. Synaptic coupling is indicated by a small black blob where an axon terminates on a dendrite, with the symbol to the right of the blob indicating the direction of induced synaptic plasticity during memory storage (“” indicates potentiation, “” depression, and “” no change). The labels “C”, “N” or “T” attached to a postsynaptic cell body indicate that the neuron is a cue cell, neither a cue cell nor a target cell, or a target cell, respectively, in the population. Probabilities of presynaptic activity (f or ) are indicated, as are the joint probabilities of postsynaptic activity and specific role ( or ). The fact that an active presynaptic neuron synapsing on a cue or target cell always experiences the induction of depression or potentiation, respectively, reflects the simplifying assumption discussed in the main text

To assess memory lifetimes under this protocol, we may track the ability of the cue sub-population to successfully evoke activity in the target sub-population. Considering a single perceptron in the target sub-population, we may obtain general expressions for and in Eq. (2), where these expressions are independent of any particular model of synaptic plasticity. Because and similarly , their expectation values lead to where we could pick any synapse i in Eq. (7a) and any distinct pair of synapses i and j in Eq. (7b) but we restrict without loss of generality to and . In these equations, we condition on whether a synapse has experienced a potentiating induction signal (“”) with probability f or not (“”) with probability , during the storage of . For the models of synaptic plasticity that we consider below, the (marginal) equilibrium probability distribution of any single synapse’s strength is uniform, or , so that if a synapse does not experience a plasticity induction signal during the storage of , then at  s and this remains true for all times  s when potentiation and depression processes are treated symmetrically, as indicated in Eq. (7a). However, for the pairwise correlations in Eq. (7b) that condition on one or both synapses not having experienced an induction signal, the expectation values do not vanish under the Hebb protocol. This is because of the higher-order equilibrium correlational structure induced by the fact that it is impossible for some of the synapses of an active neuron to experience potentiating induction signals while others experience depressing induction signals during the storage of the same memory under the Hebb protocol.

We may obtain general expressions for the expectation values in Eq. (7) by writing down the transition processes that govern changes in a single synapse’s strength or simultaneous changes in a pair of synapses’ strengths. Let each synapse have s possible internal states for each of its two possible strengths , so that the possible state of a synapse is described by a 2s-dimensional vector, with the internal states for strength (respectively, ) corresponding to the first (respectively, last) s components. Given the stochastic nature of the plasticity induction signals, this vector defines a joint probability distribution for a synapse’s combined strength and internal state. Let the transition matrix implement the definite change in a synapse’s state in response to a potentiating induction signal, and that for a depressing induction signal. We then determine the transition matrix governing the change in a single synapse’s state in response to the storage of a typical non-tracked memory by conditioning on all possible combinations of presynaptic activity and postsynaptic role. Defining , where is the identity matrix, this transition matrix is or just , where with . The three terms in Eq. (8a) arise from conditioning on the three possible perceptron roles in memory storage (determined by the global factor g), while the two terms in each of arise from conditioning on the two possible levels of presynaptic activity (determined by the local factor f). Similarly, the transition operator that governs simultaneous changes in pairs of synapses’ states during typical non-tracked memory storage is with the generalisation to for any number of synapses n being clear. The (marginal) equilibrium probability distribution of a single synapse’s state, denoted by , is the (normalised) eigenvector of with unit eigenvalue, which is also just the unit eigenvector of . That for any pair of synapses, , corresponds to the unit eigenvector of . However, because , then . Rather, must be explicitly computed as the unit eigenstate of . It is this failure of factorisation that induces the non-trivial pairwise correlational structure in the equilibrium state.

Using and , we may write down the conditional expectation values in Eq. (7). We define the vector , where denotes the transpose and the s-dimensional vector is a vector all of whose components are unity. This vector weights synaptic states according to their two possible strengths. Then, and and for the other two pairwise expectation values in Eq. (7b), we replace in Eq. (9b) by for and for . Since , we have , so that sparse coding just introduces a multiplicative factor of f and scales the rate r by the product fg in . In the equilibrium limit, by definition for any state corresponding to a probability distribution, as . Hence, , which always follows when potentiation and depression processes are treated symmetrically. For the equilibrium variance, we obtainThe second, covariance term does not in general vanish because of the equilibrium synaptic pairwise correlations.

These general results allow us to obtain SNR lifetimes when the matrices are specified for any particular model of synaptic plasticity. We defer the derivation of the transition matrix elements that are required for FPT lifetimes until we explicitly discuss simple models of synaptic plasticity in Sect. 3.1.

Hopfield protocol

Although the Hebb protocol is intuitive as a means of exploring memory lifetimes in an associative memory system, its non-trivial equilibrium distribution of synaptic states is awkward. To avoid this awkwardness, we may consider an alternative protocol that is nevertheless equivalent to the Hebb protocol in the limit of small fgN. We first define the protocol and then demonstrate the equivalence.

During memory storage, instead of defining cue and target sub-populations, we now specify the entire activity pattern, representing a memory, across the whole population of neurons. We allow these activities to take values from the set with probabilities . Here, the values represent evoked activity (the neuron is involved in memory storage), with (respectively, ) representing a strongly (respectively, weakly) tetanising stimulus in the usual LTP (respectively, LTD) sense. In contrast, the values represent spontaneous activity (the neuron is not involved in memory storage). For a single perceptron, this amounts to specifying memory vectors with components taking one of these four values, and also specifying the perceptron’s required output in response to an input vector, where this output is drawn from the same set with the same probabilities, but with f replaced by g as usual. We can track the perceptron’s activation when its required output is either or , but by symmetry its activation would differ only by a sign between these two cases, so for concreteness we just take the required output to be during the storage of . As with the Hebb protocol, a synapse does not experience a plasticity induction signal if its presynaptic input or the postsynaptic perceptron itself is only spontaneously active. However, if both input and perceptron are active, then the synapse experiences a plasticity induction signal, either potentiating if both activities are the same or depressing if different. This is just the standard Hopfield rule (Hopfield 1982), so we refer to this protocol as the Hopfield protocol: we obtain a pattern of synaptic plasticity induction signals in response to evoked activity that is identical to the standard Hopfield rule, but supplemented by the presence of spontaneous activity that does not induce synaptic plasticity.

Figure 2 summarises the Hopfield protocol, showing all allowed combinations of pre- and postsynaptic activities during memory storage, together with their associated probabilities and induced plasticity induction signals. Depressing and potentiating induction signals both occur with the same overall probability as in the Hebb protocol.

Fig. 2

Schematic illustration of the Hopfield protocol for memory storage. The format of this figure is essentially identical to that for the Hebb protocol in Fig. 1, except that labels indicating postsynaptic roles are not required. To avoid duplication, spontaneously active neurons are shown with both possible spontaneous activity levels, ; the corresponding probability is for each of these levels rather than for both

Computing and for the Hopfield protocol and using the various symmetries , , etc., we obtain These are structurally identical to the expressions in Eq. (7) for the Hebb protocol, except that the linear terms in are absent because of cancellation. Had we instead used a single level of spontaneous activity rather than the two levels , we would have obtained identical linear terms, too. Writing down the transition operators and in the Hopfield protocol, we obtain with immediate generalisation to . The (marginal) equilibrium distribution of a pair of synapses’ states is therefore determined by the unit eigenstate of and thus of , and so is just ; again, generalisation to is immediate. The result is that all conditional expectation values involving at least one synapse that does not experience a plasticity induction event during the storage of vanish, when potentiation and depression processes are treated symmetrically. So, whether we use four-level or three-level activities in the Hopfield protocol, the -dependent contributions to the covariance term in Eq. (11b) drop out, as indicated, so that the variance is affected only by the term in the first term on the right-hand side (RHS) of Eq. (11b). Moreover, the covariance term vanishes entirely in the large t, equilibrium limit, since , so that .

The equivalence of the Hebb and Hopfield protocols in the limit of small fgN is now clear. The corresponding transition matrices are in any case identical for both protocols, and hence so are the means. For , in both protocols we have thatand for general the term on the RHS contains N terms, each of which contains factors of and just one factor of . This structure reflects the fact that in the limit of small fgN, at most one of the perceptron’s synapses experiences a plasticity induction signal, regardless of the protocol. The corresponding unit eigenstate of in this limit is just , regardless of the protocol. Therefore, in the small fgN limit, the equilibrium distribution of synaptic states in the Hebb protocol reduces to that in the Hopfield protocol, and all statistical properties of h(t) must therefore also reduce in the same way. The Hopfield protocol therefore offers a way of extrapolating the small fgN behaviour of the Hebb protocol to larger f without the awkwardness of the Hebb protocol’s equilibrium structure in this regime. Furthermore, the simpler form of the results in the Hopfield protocol allow us to use it to extract the scaling properties of memory lifetimes as a function of small f (or g) in both protocols.

For the non-sparse-coding case of , spontaneous activity does not contribute to the Hopfield protocol’s dynamics, and we recover precisely the Hopfield model with discrete-state synapses. For , we expand the possible activities of neurons to allow for spontaneously active neurons that are not involved in memory storage. Thus, although the Hopfield protocol provides a convenient tool for examining the small fgN limit of the Hebb protocol, we also regard the Hopfield protocol as a fully fledged protocol in its own right, because it constitutes a very natural way of examining sparse coding with a Hopfield plasticity rule.

Population memory lifetimes

So far we have focused on the memory dynamics of a single perceptron. We now consider the memory dynamics of the entire population of P neurons. We do this only for the Hopfield protocol for simplicity. The tracked memory will evoke activity in a sub-population of on average gP neurons. In an experimental protocol, during the storage of the tracked memory we can at least in principle explicitly identify all those neurons that are active, and then subsequently track all their activities during later re-presentations of the tracked memory. Because of synaptic coupling between these tracked neurons and the other on average neurons, spontaneous activity in the other neurons will affect and potentially degrade the activation of the tracked neurons upon re-presentation of the tracked memory, affecting the tracked neurons’ ability to read out the tracked memory. But, as we are only concerned with the tracked neurons’ read-out of the tracked memory, we do not need to explicitly track the activities of all these other neurons: their activities do not directly form part of the memory signal from the tracked neurons.

In the Hopfield protocol, a tracked neuron will by definition have an output of or during memory storage. For a single perceptron, we focused on an output of without loss of generality. A perceptron with an initial output of will have identical dynamics to one with an initial output of , except that the activation will be reversed in sign. Therefore, we can just define the memory signal for any active perceptron to be , depending on this sign. Denoting the moment generating function (MGF) of for a tracked neuron with an initial output of by , the MGF for for a tracked neuron with an initial output of will also be just . All tracked neurons therefore have the same MGF for their memory signals.

Suppose that neurons form the sub-population that stores the tracked memory, where is binomially distributed with parameter P and probability g. Although these neurons’ activations will not in general evolve independently, as an extremely coarse approximation we assume that their activations do evolve independently during subsequent memory storage (cf. Rubin and Fusi 2007). Population memory lifetimes obtained from this simplifying assumption will therefore only be theoretical, and perhaps very loose, upper bounds on exact memory lifetimes. With this simplification, the MGF for the memory signal from the tracked sub-population is then , by independence. Averaging over , the MGF of the population memory signal is then just . The mean, , and variance, , of this population signal follow directly. Ignoring covariance terms (or considering the limit for the variance), we have and , where and are the single-perceptron mean and variance above.1 Hence, the population SNR, , is just scaled by the factor relative to the single-perceptron SNR, soThe population SNR memory lifetime, which we denote by , is then the solution of . With in the Hopfield protocol, depends on , the total number of synapses in the memory system, but it also contains the additional factor of compared to , which modifies scaling behaviour compared to single-perceptron results.

Models of synaptic plasticity

Simple synapses: the stochastic updater

The simplest model of synaptic plasticity to consider is one in which synapses lack any internal states so that , and given a plasticity induction signal, they change strength (if possible) with some fixed probability p (Tsodyks 1990). Because a synapse just changes its strength stochastically in this model, we have called such a synapse a “stochastic updater” (SU; Elliott and Lagogiannis 2012). The underlying strength transition matrices are thenand sowhere we define for convenience.

The equilibrium distribution of a single synapse’s strength in both protocols is just the normalised unit eigenvector of , or . For the Hopfield protocol, any pair of synapses’ strengths has the equilibrium distribution . For the Hebb protocol, we require the unit eigenstate of , which giveswhere . The quantity determines the pairwise correlations present in this state, since . For , and . With these equilibrium distributions, we may explicitly compute and in both protocols using Eqs. (7) and (11). For the common mean, we obtainand for the two variances, we need the various correlation functions in Eqs. (7b) and (11b). For the Hebb protocol, these are and for the Hopfield protocol, we just set in these equations. These results allow us to determine SNR lifetimes for simple, SU synapses. Approximating the Hopfield variance by its asymptotic form , the single-perceptron SNR memory lifetime in the Hopfield protocol for a stochastic updater is thenand for the population SNR lifetime , we replace by , where the subscript in indicates either N or .

To determine FPT lifetimes, we require for the MIE approach, or the induced jump moments and for the FPE method. We relegate the derivation of to Appendix A, where we also indicate our numerical methods for obtaining FPTs. From Appendix A, we obtain the jump moments in Eq. (5) for the FPE approach to FPTs. For both protocols we get the same first jump momentand for the second jump moment, we get for the Hebb and Hopfield protocols, respectively. We have explicitly indicated the dependence of on , where is the number of a perceptron’s synapses that are active during the storage of . We write , where we separate out the quadratic dependence on and it is convenient to remove an overall factor of from the definition of . Dropping the quadratic term from is equivalent to considering dynamics based on the Ornstein–Uhlenbeck process (Uhlenbeck and Ornstein 1930), which we have found to be a very good approximation (Elliott 2014, 2017a, 2019), so we work with just the constant term.

For the MIE approach to FPTs, a technical difficulty as discussed in Appendix A requires us to restrict to the specific case of only. We use numerical methods to obtain FPT lifetimes from the MIE approach, but for small fN, the dynamics are dominated by . For and , Eq. (4) is trivial because the only contribution to the sum involves no transition, occurring with probability regardless of the protocol. Writing as the variance in the FPT, we obtainat leading order, for small f () in both protocols. We see that scales as 1/f in this regime, but that scales as . Although swamps for small f, is nevertheless robustly positive. We may use our earlier results to obtain the corresponding forms for the FPE approach to FPT lifetimes for small f (see Eqs. (3.29) and (3.30) in Elliott 2017a). We obtainIn contrast to Eq. (23), now scales as and not 1/f, and scales as and not . Moreover, in the FPE approach, the FPT moments have lost their overall scaling with N. Although the forms in Eq. (24) are obtained using mean field approximations that are expected to be invalid when fN is small, in fact we obtain the same scaling behaviour when the expectation values are obtained by averaging properly over and . Our simulation results, discussed in Sect. 4, agree with the behaviour in Eq. (23). Therefore, the failure of the FPE approach for small fN in Eq. (24) is due to the approximations intrinsic to the FPE approach itself. These include the diffusion and especially the continuum limit. For small fN, the system is nowhere near the continuum limit, so the scaling behaviour must be incorrect there.

Complex synapses

We now turn to models of complex synapses that have internal states, so that . In such models, synapses can undergo metaplastic changes in their internal states without expressing changes in synaptic strength. We will only consider SNR lifetimes in relation to complex synapses. We have studied FPT lifetimes for filter-based synaptic plasticity for both bistate (Elliott 2017b) and multistate (Elliott 2020) synapses in a non-sparse coding context, but we have yet to consider other models of complex synapses. We therefore restrict to SNR lifetimes, but with the caveat that they are valid only in an asymptotic regime.

We have discussed filter-based models of synaptic plasticity at length elsewhere (Elliott 2008; Elliott and Lagogiannis 2009, 2012; Elliott 2016b), so we only briefly summarise them here. Synapses are proposed to implement a form of low-pass filtering by integrating plasticity induction signals in an internal filter state. Synapses then filter out high-frequency noise in their induction signals and pass only low-frequency trends, rendering them less susceptible to changes in strength due to fluctuations in their inputs. Potentiating (respectively, depressing) induction signals increment (respectively, decrement) the filter state, with synaptic plasticity being expressed (if possible) only when the filter reaches an upper (respectively, lower) threshold. For symmetric potentiation and depression processes, we may take these thresholds to be . The filter can occupy the states , with the thresholds not being occupiable states. Several variant filter models are distinguishable by their different dynamics upon reaching threshold (Elliott 2016b), but we consider only the simplest of them here. In the simplest model, the filter always resets to the zero filter state upon reaching threshold, regardless of its strength state and regardless of the type of plasticity induction signal. This filter generalises to any multistate synapse. If the synapse is saturated at its upper (respectively, lower) strength state and reaches its upper (respectively, lower) filter threshold upon receipt of a potentiating (respectively, depressing) induction signal, the filter resets to zero despite the fact that it cannot increment (respectively, decrement) its strength. The transitions for this filter for the case of are illustrated in Fig. 3A. Although for clarity we have shown all permitted transitions between all filter and strength states, we stress that each synapse possesses only a single synaptic filter: the filter is not duplicated for each strength state. Transitions in filter state occur independently of strength state. Nevertheless, to describe transitions in the joint strength and filter state, we require matrices, so , although the number of required physical states for filter-based synapses is just for the filter states themselves, and an additional, binary-valued variable for the bistate strength, so a total of states.

Fig. 3

Strength and internal state transitions for various models of complex synapses. Coloured circles indicate synaptic states, with red (respectively, blue) circles corresponding to strength (respectively, ), and the labelled numbers inside the circles identifying the particular internal states (indexed by I for filter states and i for serial and cascade states). Different internal states of the same strength state are organised in the same vertical column, while different strength states correspond to different columns. Solid (respectively, dashed) lines between states show transitions caused by potentiating (respectively, depressing) induction signals, with arrows indicating the direction of the transition. Loops to and from the same state indicate no transition. Three different models are shown, as labelled, corresponding to a filter model (A), and serial (B) and cascade (C) synapse models. For the filter and serial synapse models, given the presence of an induction signal of the correct type, the transition probabilities are unity. For the cascade model, the transition probabilities are as discussed in the main text

We state without derivation the result for in this filter model:where denotes the floor function. This expression is obtained from Eq. (4.24) in Elliott (2016b) just by multiplying by f and inserting a factor of fg into the exponents. This result is required for obtaining SNR lifetimes. The pairwise correlation functions required for are computed via numerical matrix methods using the matrices for the filter model (given in Elliott (2016b) or implied by the transitions in Fig. 3A), and we also obtain the Hebb equilibrium distribution by numerical methods.

To estimate SNR lifetimes in the filter model for the Hopfield protocol, we consider the slowest decaying mode in the first and second terms of Eq. (25). For non-sparse coding, it is usually enough to consider just the slowest mode in the first term, but with sparseness, both terms must be considered for a better approximation. For large enough, we then haveApproximating the Hopfield variance by its asymptotic form , the single-perceptron SNR memory lifetime for the filter model is thenwhere in deriving this expression, we have regarded the second term as a correction to the first term, with the first term arising purely from the first term in Eq. (26). To obtain the population SNR memory lifetime , we again just replace by in Eq. (27).

We also consider the serial synapse model (Leibold and Kempter 2008; Rubin and Fusi 2007). In this model, a synapse performs a symmetric, unbiased, one-step random walk on a set of 2s states between reflecting boundaries. The first (respectively, second) group of s states are identified as corresponding to strength (respectively, ). For each strength state, there are thus s metastates. If a synapse has strength (respectively, ) and experiences a sequence of depressing (respectively, potentiating) induction signals, then it is pushed into progressively higher metastates. However, the synapse can only change strength when in the lowest, metastate. The transitions are illustrated in Fig. 3b. The transition matrices are just where and denote the upper and lower diagonals, respectively. The eigen-decomposition of is standard (cf. Elliott 2016a, for the eigen-decomposition of the similar matrix there), so we can directly evaluate , where . We obtainFor the Hebb protocol, we again use numerical matrix methods to obtain . To estimate SNR lifetimes for the Hopfield protocol, it is sufficient to consider just the slowest decaying term in Eq. (29), givingfor s large enough, and henceas the required approximation, with obtained in the usual way.

In the cascade model of synaptic plasticity (Fusi et al. 2005), there are also 2s metalevels, s for each bistate strength state, but unlike the serial synapse model, a potentiating (respectively, depressing) induction signal for a synapse with strength (respectively, ) in metastate i can with probability (or for ) cause the synapse to change strength and return to metastate . The same probabilities govern transitions to higher metastates. The transitions are illustrated in Fig. 3C. The cascade model essentially constitutes a tower of stochastic updaters that progressively render the synapse less labile. We have extensively analysed the cascade model elsewhere (Elliott and Lagogiannis 2012) and compared its memory performance to filter-based synapses, which outperform the cascade model in almost all biologically relevant regions of parameter space (Elliott 2016b). It is possible to obtain analytical results for the Laplace transform of the mean dynamics in the cascade model (Elliott and Lagogiannis 2012), but here we use numerical matrix methods. Rubin and Fusi (2007) give a formula for the SNR based on finding a fit to numerical results. The implied formula for the mean isTaking the asymptotic variance in the Hopfield protocol, we can then use the expression for the SNR. This still cannot be solved analytically for the SNR lifetime (or the population form ), but we can use it to obtain numerical solutions that can be compared to results obtained from exact matrix methods.

A serial or cascade synapse possesses 2s states, with each set of s metalevels duplicated for each strength. Metalevel i for strength cannot be identified with metalevel i for strength because the transitions induced by plasticity induction signals are in opposite directions. This is in contrast to the filter model, in which the filter transitions are independent of the strength state. Serial and cascade synapses therefore possess fully 2s physical states characterising the state of a synapse, while a filter synapse possesses physical states and not states. Hence, we may directly compare the performance of a filter synapse with threshold to a serial or cascade synapse with a total of 2s metastates, or s metastates per strength state.

Results

We now turn to a discussion of our results, comparing and contrasting the various models of synaptic plasticity considered above, for the Hebb and Hopfield protocols. For simplicity we consider simulation results only for SU synapses, to confirm and validate our analytical results. Simulations are run according to protocols discussed extensively elsewhere (see, for example, Elliott and Lagogiannis 2012; Elliott 2014), but modified to allow for sparse coding. We first consider single-perceptron memory lifetimes and then population memory lifetimes.

Single-perceptron memory lifetimes

In Fig. 4, we show results for memory lifetimes for SU synapses with no spontaneous activity, , comparing the Hopfield and Hebb protocols. We consider both FPT and SNR lifetimes, and for FPT lifetimes, we show results for both the FPE and MIE approaches. Simulation results are also shown, although only for : for smaller values it becomes increasingly difficult to obtain enough statistics for decent averaging due to the longer simulation run times. We select an update probability of , which is our standard choice of p in earlier work (see, for example, Elliott 2014). From Eq. (24), and are expected to scale as for small f for the FPE approach, so we remove this scaling by multiplying by , which in this figure affords greater clarity and resolution.

Fig. 4

Convergence of Hebb and Hopfield protocol results for stochastic updater synapses in the limit of sparse coding. Scaled single-perceptron memory lifetimes are shown as a function of sparseness, f. Results in red (respectively, blue) correspond to the Hopfield (respectively, Hebb) protocol. Shaded regions indicate (with the central solid line showing ) computed using the FPE approach to FPTs, so that we show the (scaled) MFPT surrounded by the one standard deviation region around it, governed by . Short-dashed lines show obtained using the exact, MIE approach to FPTs. Circular data points correspond to results from simulation, for . Long-dashed lines show results for ; for the Hebb protocol over the whole range of f in panel A. The value of N is indicated in each panel. In all panels, , and

Above we showed that the Hopfield and Hebb protocols must coincide for . For the various choices of N used in Fig. 4, we see this convergence of both protocols’ results, becoming indistinguishable for f below , for all forms of memory lifetime. Focusing first on from the FPE approach, for smaller N we clearly see that asymptotes to a common, N-independent constant as f becomes small; we would see the same behaviour for larger N too, but would need to take smaller values of f than those used in this figure. We also see that from the MIE approach tracks that from the FPE approach quite closely, and indeed for intermediate values of f and smaller choices of N, it plateaus, so that scales as in this regime. However, for and , we clearly see the MIE turn downwards and approach zero as f decreases. This behaviour is consistent with the derived form of the exact scaling behaviour in Eq. (23), in which for small f. We also just see this change for for f close to , but for larger N we would need to take f smaller to see the 1/f scaling of the exact form of . Our simulation results agree with the results from the MIE approach, validating both. Although we do not take f small enough to see the switch to 1/f scaling for in Fig. 4A, we nevertheless do clearly see the start of the down-turn at .

For in Fig. 4, we see very significant differences between the Hebb and Hopfield protocols. While for the Hopfield protocol grows like for fN large enough, this is not the case for the Hebb protocol. For f in the region of unity, is roughly speaking independent of N. This means that the dynamics are dominated by the correlations between pairs of synapses’ strength in the Hebb protocol. For , we obtain and 5.35 for and , respectively, from the FPE approach. (The corresponding values from the MIE approach are 6.64 and 6.79, respectively.) In the regime of f not too far from unity, memory lifetimes in the Hebb protocol are therefore significantly reduced by the synaptic correlations induced by this protocol, where the influence of these correlations cannot be removed by increasing N.

We see that is robustly positive in Fig. 4 for all choices of N over the whole range of displayed f, and it remains so for small f because of the discussed scaling behaviour. However, looking at the one standard deviation region around , it is clear that in some regimes for f, there can be high variability in FPT memory lifetimes. For the Hopfield protocol, this regime of high variability occurs for small f (where what is “small” f depends on N), while in the Hebb protocol, there is an additional regime for f close to unity. High variability does not mean that memories cannot be stored: is always robustly positive. Rather, high variability simply means that some memories are stored strongly while others are stored weakly or not at all.

Turning to a consideration of , we see from Fig. 4 that exists (i.e. ) in precisely those regions of low variability in FPT lifetimes. Indeed, the results for track quite closely those for over some range of f, and deviate from it elsewhere. We have shown in a non-sparse coding context that FPT and SNR lifetimes for simple synapses essentially coincide (up to additive constants) in the regime in which the distribution of is tightly concentrated around its supra-threshold mean (Elliott 2017a). For the specific case of , as here, we showed that if we can write the initial variance in the form , then the parameter must be large enough, which means (Elliott 2017a). We then have that , which is just a condition on the initial SNR.2 Using the pre-averaged form (see Appendix A), this condition reduces to in the Hopfield protocol for . For the Hebb protocol, the limit of large N with p not too close to unity additionally satisfies the requirement on , giving the upper bound for . In the Hebb protocol, we therefore have the interval for equivalence of SNR and FPT memory lifetimes, for . (We must have for this interval to exist.) With in Fig. 4, these conditions are and for the Hopfield and Hebb protocols, respectively. For in both protocols (except for the Hebb protocol for , where the bounding range of f is invalid), we do indeed see that the FPE results for and those for run essentially parallel to each other, but that for , peels away from . The same is true for the Hebb protocol for : as f increases above 0.05, also peels away from . Thus, these two estimates for the two protocols appear to capture well the region of f for which is a reliable indicator of memory longevity. SNR lifetimes are therefore acceptable surrogates for FPT lifetimes when the latter are subject to low variability, but outside these regions SNR lifetimes fail to capture the possibility of memory storage, albeit with high variability. Importantly, the requirement that in both protocols means that the SNR approach cannot be extended to very small or just small f, because such values violate the asymptotic regime. Essentially, then, the SNR approach cannot probe the very sparse coding regime in either protocol.

For the Hopfield protocol, Eq. (20) is justWith , we require for , and we see precisely these threshold values for the different choices of N in Fig. 4. Alternatively, we require for memories to be stored according to the SNR criterion. However, these conditions do not carry over to FPT memory lifetimes: we need neither a minimum N nor a minimum f for , because it is always positive. This failure of SNR conditions to carry over to the FPT case also applies to any optimality conditions derived from . From Eq. (33) with , we may find that value of f, , that maximises , giving rise to , with the result that . The same value essentially applies to the Hebb protocol, albeit with complicated corrections. However, for the validity of the SNR results, both protocols require . If the SNR optimality condition is valid, then it must satisfy , or . This is clearly false, and hence the SNR optimality condition for f is spurious, because at , the asymptotic validity condition is violated. In fact, we may essentially take f as small as we like and will continue to grow, albeit with increasing variability in the FPT lifetimes. Thus, although we will shortly consider optimality conditions for SNR memory lifetimes with complex synapses, these conditions must be viewed with extreme caution.

Figure 4 considers only the case of exactly zero spontaneous activity, . In Fig. 5, we examine the impact of spontaneous activity on SU memory lifetimes. We show only the case of to avoid unnecessary clutter, but the results are qualitatively similar for other choices of N. In the Hopfield protocol, appears only through a quadratic term in or , while in the Hebb protocol, also appears through a linear term. This difference makes the Hebb protocol much more sensitive to spontaneous activity than the Hopfield protocol, and we see this explicitly in Fig. 5. In the Hopfield protocol, the asymptotic variance takes the form , so exerts a significant influence on memory lifetimes only for . We therefore only start to see a divergence of memory lifetimes from those for at around , and this is confirmed in the figure. However, as f is taken small, the dependence of (from the FPE) on is lost (just as its dependence on N is lost), so that for very small f, does not affect (FPE) FPT lifetimes, neither their means nor their variances. This is because for small f, the scaling results in Eq. (24) depend only on the A and not the B jump moment, so they depend only on drift and not diffusion. However, appears only through the diffusion term. In contrast to the Hopfield protocol, even a choice of induces a large reduction in memory lifetimes in the Hebb protocol, at least away from the small f regime. For small f, the Hebb and Hopfield protocols coincide, so we observe the same loss of dependence on in (FPE) FPT lifetimes in the Hebb protocol. However, away from the small f regime, the linear term in in B or significantly impacts memory lifetimes.

Fig. 5

Impact of spontaneous activity on stochastic updater single-perceptron memory lifetimes. Results are shown for (from the FPE approach) and for both the Hopfield and Hebb protocols, as indicated in the different panels. Different line styles correspond to different levels of spontaneous activity, , as indicated in the common legend in panel D. Some lines style are absent in panel D because there is no corresponding . In all panels we take , with and in all cases

Examining Eq. (33) for the Hopfield protocol, for , we have just in the logarithm, while for , we have . Roughly speaking, for intermediate values of , the effective power of f switches rapidly from one to two in the vicinity of . This switching can be seen clearly in Fig. 5, where as f decreases, (and also ) tracks closely the form for , until it rapidly peels away, following a different power. Although it is still clearly the case that optimality conditions obtained from are invalid, it is nevertheless worth examining . For , we again obtain , but for , we instead obtain , so that the N-dependence changes. The corresponding optimal lifetimes are for and for . Of course, we see explicitly in Fig. 5 that these SNR-derived optimal values of f and thus maximum possible SNR lifetimes are invalid, but SNR lifetimes do at least indicate when FPT lifetimes are subject to lower variability and when they are subject to higher variability.

Considering is of course biologically meaningless, as then there is no distinction between spontaneous and evoked electrical activity levels. However, taking either or allows explicit optimality results to be obtained for these two cases, while such results are not available for intermediate values of . As just indicated, empirically we observe a very rapid switching in dynamics in the vicinity of , with the explicit results for and therefore indicating the general behaviour prior to and after, respectively, this switching. When we give results for , we therefore do so with this understanding: that the limit is biologically meaningless, but that it nevertheless indicates the general behaviour for in excess of around .

We now turn to complex models of synaptic plasticity, considering only SNR lifetimes. In Figs. 6 and 7, we plot SNR lifetimes against sparseness, f, for the three complex models discussed above, for both zero and nonzero spontaneous firing rates, and for the Hopfield (Fig. 6) and Hebb (Fig. 7) protocols. All results are obtained by numerical matrix methods to solve the SNR equation , where the standard deviation is computed fully rather than via just its asymptotic form .

Fig. 6

Spontaneous activity reduces single-perceptron memory lifetimes and limits sparseness in complex synapse models: Hopfield protocol. Single-perceptron SNR memory lifetimes are shown for different complex models of synaptic plasticity under the Hopfield protocol, as a function of sparseness, f. Each panel shows results for the indicated model and choice of spontaneous activity, either or . Results are shown for or s ranging from 2 to 12 in increments of 2, with the particular choice identified by the line colour described by the common legend in panel B. In all cases,

Fig. 7

Spontaneous activity reduces single-perceptron memory lifetimes and limits sparseness in complex synapse models: Hebb protocol. The format of this figure is identical to Fig. 6, except that it shows results for the Hebb protocol, and in the right-hand panels we use a smaller value . Some lines of specific colour are absent in some graphs because there is no corresponding . In all cases,

For the Hopfield protocol in Fig. 6 we see in all cases and for all choices of parameters an onset of SNR lifetimes for a minimum, threshold value of f, the rapid attainment of a peak or optimal value of , followed by a steady fall in lifetimes as f increases further. For all complex models, this onset of SNR lifetimes occurs for increasingly large values of f as or s increases. At least for the parameter ranges in this figure, in the filter and serial models, for a given choice of f, increasing or s increases , although as the number of internal states continues to increase, ultimately will start to fall. In the case of the cascade model, however, the dependence of on s for fixed f is not as simple as for the other complex models. We note that for all models in this figure, the optimal values of decrease for increasing or s, at least for . However, when we increase the spontaneous activity to , the optimal values lose most of their dependence on or s in the filter and serial models, although not in the cascade model. This loss of dependence on or s is strongly N- and -dependent. For , we must take close to unity before this loss of dependence is noticeable, while for , even is sufficient.

For the Hebb protocol, in Fig. 7, for smaller f we obtain essentially the same results as for the Hopfield protocol because these two protocols must coincide for , regardless of the model of synaptic plasticity. However, for larger f, the synaptic correlation terms induced by the Hebb protocol again significantly impact SNR memory lifetimes, with the impact being greater for larger or s in the filter and serial models. Thus, as with SU synapses under the Hebb protocol, SNR lifetimes exist only in some interval of f (below ), with this interval shrinking and disappearing as the number of internal states increases (or as p decreases for SU synapses). These dynamics dramatically limit the number of internal states that give rise to positive SNR lifetimes. Nevertheless, as or s increases, in general increases, at least until the permissible range of f becomes very small and then disappears entirely. For the cascade model, however, the upper limit on f is roughly speaking independent of s, but we also see that in general, as s increases, decreases for fixed f. This relative insensitivity of the upper limit of the permissible range of f to s in the cascade model occurs because the cascade model has different metastates with different update probabilities, with some synapses residing in the lower metastates and so having larger update probabilities than those residing in higher metastates.

In the presence of spontaneous activity, we see a dramatic change in the memory lifetimes. Indeed, such is the sensitivity of the Hebb protocol to , especially for complex synapse models, that in contrast to Fig. 6 for the Hopfield protocol, for which we took , in Fig. 7 we take . Even with just 1% spontaneous activity, the filter and serial models’ number of internal states becomes severely restricted, in terms of giving rise to positive SNR lifetimes. The cascade model under the Hebb protocol is not quite so sensitive, again because of its different metastates, but a 10% level of spontaneous activity would still dramatically restrict the permissible ranges of f and s, compared to the Hopfield protocol.

We quantify these observations by explicitly considering the optimal choices of the parameters f and either or s, so and either or , that maximise , giving rise to . In Figs. 8 and 9, we plot and against or s, for different levels of spontaneous activity, , for the particular choice of . Results are obtained both by numerical matrix methods and by using the approximations for and given in Sect. 3.2. For the latter, we maximise as a function of f for fixed or s.

Fig. 8

Optimal sparseness in complex synapse models for single perceptrons in the Hopfield protocol. The left-hand panels (A, C, E) show the optimal single-perceptron memory lifetimes obtained at the corresponding optimal levels of sparseness shown in the right-hand panels (B, D, F), for the indicated complex models. Lines show numerical matrix results while the corresponding data points show approximate analytical results obtained as discussed in the main text. Results are shown for different values of , with identifying line styles corresponding to those in Fig. 5. We have set in all panels

Fig. 9

Optimal sparseness in complex synapse models for single perceptrons in the Hebb protocol. The format of this figure is essentially identical to Fig. 8, except that it shows results for the Hebb protocol. Approximate analytical results are not available for the Hebb protocol and so are not present. The termination of a line at a threshold value of or s indicates that above that value, no choice of f generates . We have set in all panels

For the Hopfield protocol in Fig. 8, we see that for , falls as a function of or s. However, in the filter and serial models, as increases, the fall in with or s reduces and disappears; indeed, the exact results in fact show a very slight increase in with or s for , although this behaviour is not noticeable in Fig. 8. For the displayed choice of , we need only take for the filter and serial models’ to be relatively insensitive to or s. This is N-dependent: for , even is sufficient; for , needs to be quite close to unity. In contrast, for the cascade model, always falls with s for any choice of , including . The behaviour of the filter and serial models’ is easy to extract from the approximate results in Sect. 3.2. Ignoring for simplicity the correction terms in Eq. (27), both filter and serial models’ can be written in the form(cf. Eq. (33)), where a and b are numerical constants and q denotes or s. For we obtain and , while for we obtain and . Therefore, scales differently with q and with N in these two cases, and for , falls as q increases, but for , is completely independent of q. Intermediate choices of result in intermediate behaviours between these two extremes, and the corrections terms in Eq. (27) provide only corrections to, rather than fundamentally alter, this behaviour. We see in Fig. 8 that the numerical and approximate analytical results agree well for the filter and serial models, and that, moreover, both these models’ optimal values are very similar. Unfortunately, in the case of the cascade model, no such simple analysis, even using the fitted form for in Eq. (32), is available to explain the fact that falls with s for all values of , including . The numerical and fitted results for agree well in the cascade model, although there are quite large discrepancies in obtained from the (exact) numerical methods and the fitted expression, particularly for larger values of s and for closer to zero than to unity. Fitting our numerical matrix results for in the cascade model to power laws in s and N for large enough s, we find that for , and , while for , and . While the scaling behaviour of is the same as that in the filter and serial models, the dependence of on differs in the cascade model compared to the filter and serial models.

For the Hebb protocol in Fig. 9, again the pairwise correlation structure present in , and the Hebb protocol’s extreme sensitivity to even very small levels of spontaneous activity , have a significant impact on optimality conditions. In the filter and serial models, the permissible range of or s is considerably reduced, so that even for , s cannot exceed 6 in the serial synapse model, or 5 in the filter model. As N is reduced from the displayed value of , the permissible ranges of and s reduce. The cascade model in the Hebb protocol is also extremely sensitive to noise, but as discussed, the different metastates’ different update probabilities somewhat ameliorate this sensitivity. Nevertheless, increasing from to just reduces by several orders of magnitude.

In Figs. 10 and 11 we instead examine or as a function of f, rather than vice versa, so that we maximise with respect to or s while holding f fixed. For the Hopfield protocol in Fig. 10, subject to a minimum, threshold requirement, or increases as a function of f, for any level of spontaneous activity , in all three complex models considered here. However, as moves from to , the functional dependence of or on f changes. We can derive this explicitly by again using the simple expression for in Eq. (34) for the filter and serial models. The optimal value of q (either or s) isThus, as f increases, essentially switches from linear growth in f to slower, growth, at around . This behaviour is clearer for the smaller nonzero choices of used in Fig. 10. The corrections due to the additional terms in Eq. (27) do not fundamentally change this behaviour for the filter model. The corresponding optimal SNR memory lifetime isFor , decreases as f increases, but for , is independent of f. As f increases, the transition from being independent of f to falling as 1/f is again sharp, occurring around . This transition is clear for the filter and serial models in Fig. 10. In the case of the cascade model, however, although increases with f, albeit according to clearly different power laws than for the filter and serial models, the corresponding value of always decreases as a function of f, regardless of .

Fig. 10

Optimal synaptic complexity in complex synapse models for single perceptrons in the Hopfield protocol. The format of this figure is very similar to Fig. 8, except that we have optimised with respect to or s rather than f. In panels A and C the lines switch from numerical matrix to approximate analytical results when the corresponding values of or exceed 20 in the right-hand panels; before this transition, the lines correspond to numerical matrix results and the discrete points to approximate analytical results. We have set in all panels

Fig. 11

Optimal synaptic complexity in complex synapse models for single perceptrons in the Hebb protocol. The format of this figure is essentially identical to Fig. 10, except that approximate analytical results are not available for the Hebb protocol. We have set in all panels

For the Hebb protocol in Fig. 11, again the small f behaviour must be identical to that for the Hopfield protocol in Fig. 10. However, the increase in or with increasing f is halted and then reversed as f increases further, as the effects of the pairwise synaptic correlations induced by the Hebb protocol are felt. These correlations not only pull down the optimal value of or , but they also have a deleterious effect on , changing the 1/f behaviour of the filter and serial models in the Hopfield protocol to approximately behaviour in the Hebb protocol (obtained by fitting), for . Furthermore, while spontaneous activity can make independent of f before the switch to 1/f behaviour in the Hopfield protocol for the filter and serial models, in the Hebb protocol, always decreases with increasing f, for all three complex models considered here.

We now turn to population SNR memory lifetimes. Because , optimisation of with respect to or s is not affected by the additional factor of . When we instead optimise population SNR lifetimes with respect to , however, the additional factor of in compared to functionally changes the optima compared to those for a single perceptron, and so we focus on this case. Because of the independence approximation involved in estimating in Sect. 2.4, is only an upper bound on population SNR lifetimes, and this will be implicit below.

For simple, SU synapses, Eq. (20) indicates that and differ in the logarithmic term, with the former having argument and the latter . We therefore see immediately that single-perceptron SNR lifetimes with and population SNR lifetimes with have identical f-dependence. For single-perceptron lifetimes with and and population lifetimes with and , the f-dependence under the logarithm is , , and , respectively. The effective power of f switches rapidly in the vicinity of , in an N- or -dependent way. Because , we expect very rapid switching in the population case, and with only very small, even negligible levels of spontaneous activity being required to induce the change in effective power. Above we found for single-perceptron optimal lifetimes that at for , and at for . For optimal population lifetimes these become at for , and at . Spontaneous activity changes the N-dependence of from to N, and the -dependence of from to , which latter is a smaller overall reduction, although in all cases the dependence on p involves a positive power. Because the dominant behaviour of SNR lifetimes in the filter and serial models is governed by a similar, single logarithmic term, many of these scaling observations for simple synapses carry over unchanged to these complex synapses.

We examine the behaviour of optimal population SNR lifetimes for complex synapses in Fig. 12. Compared to the single-perceptron optimal SNR lifetimes in Fig. 8, the population results in Fig. 12 are markedly different, particularly for the filter and serial models. For these models, with , is now approximately independent of or s, while with , grows as a function of or s. Even with , this growth is present and of almost the same profile as that for , while at the single-perceptron level, for smaller choices of N it is necessary to take close to unity to halt the decrease in with increasing or s. This sensitivity to small, nonzero values of at the population level is -dependent, but even with (e.g. and ), we only require for to adopt the same profile as that for . For the cascade model, however, optimal population SNR lifetimes fall with s just as they do for single perceptrons. Nonzero does render nearly independent of s for small s (), but for larger s, falls with s. We may quantify the filter and serial models’ population SNR lifetimes as before, using the slowest decaying modes. We obtainwhere now we have rather than in the numerator of the logarithm, just as for SU synapses. The optimal values of f are now (cf. ) for and (cf. ) for . The corresponding optimal memory lifetimes are (cf. ) and (cf. abN/e), respectively. The corrections due to the additional terms in the filter models’ results again modify but do not fundamentally alter this behaviour. Thus, at the population level, for , the filter and serial models’ optimal SNR lifetimes are independent of q, while at the single perceptron level, they fall as . However, for , the population lifetimes grow as , while for a single perceptron, they are constant. We cannot obtain similar analytical results for the cascade model, so we fit the numerical results for the cascade model in Fig. 12 to power laws in s and . We find that for larger values of s, at the population level with for (cf.  and , respectively, for a single perceptron), and with for (cf.  and , respectively, for a single perceptron), with the same rapid switching behaviour for intermediate as for the filter and serial models. The population dynamics soften the fall of with s, but not enough to turn the dependence into growth with s.

Fig. 12

Optimal sparseness in complex synapse models for neuronal populations in the Hopfield protocol. The format of this figure is essentially identical to that in Fig. 8, which shows results for the single-perceptron case. Lines show numerical solutions of the equation maximised with respect to f, so at , while data points show approximate analytical results. We have set and , or , in all panels

In Table 2 we summarise the scaling behaviour of and as functions of either p or q and of either N or , for simple and complex synapses, for both single-perceptron and population results, for and . In each column, regardless of the model, scales identically as a function of q or and of N or . This is not surprising given that the SU, filter and serial model results for come from the same dominating logarithmic behaviour, but the cascade results are obtained by fitting numerical matrix data to power laws to extract the behaviour. For , we also obtain the same scaling behaviour as a function of N or within each column, again regardless of the model. However, the scaling of with q (or ) within each column does depend on the particular model of plasticity. Across a row, moving from single-perceptron and results to population and results, the dependence of on q (or ) changes in such a way that increasing q (or decreasing p) has an increasingly less deleterious effect on memory lifetimes for SU and cascade synapses. For SU synapses, the power of p reduces from 3 to 1 to , while for cascade synapses the power of q (or s) changes from to to . For both SU and cascade synapses, optimal memory lifetimes therefore always decrease as p decreases or s (the number of metastates) increases, regardless of the level of spontaneous activity, and regardless of whether at a single-perceptron or population level. For filter and serial synapses, however, the power of q changes from to 0 (i.e. no dependence) to . Increasing the number of serial metastates or filter states available to a filter or serial synapse therefore increases optimal population SNR lifetimes, but only in the presence of spontaneous activity. As Fig. 12 indicates, we need only have very low levels of spontaneous activity to induce this growth of optimal population SNR lifetimes with the number of internal states available to filter or serial synapses.

Table 2

Overall dependence of optimal single-perceptron and population SNR memory lifetimes and the corresponding optimal sparseness on model parameters. Here, q represents or s, depending on the complex model, and is assumed large

		Single-perceptron		Population
		ζ=0	ζ=1	ζ=0	ζ=1
Stochastic updater	f^opt	(p^-2/N)^1	(p^-2/N)^1/2	(p^-2/N)^1/2	(p^-2/N)^1/3
	τ^opt	p^3N^2	pN	pN	p^1/3N^2/3
Non-cascade	f^opt	(q^2/N)^1	(q^2/N)^1/2	(q^2/N)^1/2	(q^2/N)^1/3
	τ^opt	(N/q)^2	N	N	(qN)^2/3
Cascade	f^opt	(q^2/N)^1	(q^2/N)^1/2	(q^2/N)^1/2	(q^2/N)^1/3
	τ^opt	(N/q^2)^2	(N/q^2)^1	(N/q^2)^1	(N/q^2)^2/3

Discussion

Memory is a complex, multi-level, system-wide phenomenon involving processes occurring over many time scales and across different brain regions, with integrated and orchestrated control processes coordinating, for example, the transition from short- to long-term memory (Eichenbaum and Cohen 2001). Palimpsest models of memory, in which older memories are forgotten as newer ones are stored (Nadal et al. 1986; Parisi 1986), focus on the dynamics of memory storage and retrieval within a single memory system, such as the hippocampal CA3 recurrent network (Andersen et al. 2007). Sparse population coding (see, for example, Csicsvari et al. 2000; Olshausen and Field 2004) enhances memory lifetimes in these memory models by reducing the overall rate of synaptic plasticity at single synapses, so effectively dilating time, and by decorrelating synaptic updates induced by overlapping memories (Tsodyks and Feigel’man 1988). Complex synapse models, involving metaplastic changes in synapses’ internal states without associated changes in synaptic strength, have also been proposed as a way in which to enhance memory lifetimes in palimpsest models (Fusi et al. 2005), whereas we introduced models of integrate-and-express, filter-based synapses as a means of enhancing the stability of developmental patterns of synaptic connectivity in stochastic models of synaptic plasticity (Elliott 2008; Elliott and Lagogiannis 2009).

Understanding the interaction between sparseness and synaptic complexity in palimpsest memory models is therefore crucial (Leibold and Kempter 2008; Rubin and Fusi 2007). Taken at face-value, our results for SNR single-perceptron memory lifetimes support two conclusions. First, optimal single-perceptron SNR lifetimes, optimised with respect to sparseness, require lower synaptic complexity for longer optimal lifetimes. Second, optimal single-perceptron SNR lifetimes, optimised instead with respect to synaptic complexity, again require lower synaptic complexity as sparseness increases for longer optimal lifetimes. These conclusions hold regardless of the level of spontaneous activity, although spontaneous activity can prevent the decrease in optimal single-perceptron memory lifetimes. These conclusions appear to argue in favour of reduced synaptic complexity in real neurons in the presence of sparse population coding, at least at the single-neuron level. However, at the population level, the first of these conclusions is overturned, at least for filter and serial synapses. Critically, even in the presence of low but nonzero levels of spontaneous activity, optimal population SNR lifetimes, optimised with respect to sparseness, increase rather than decrease with synaptic complexity, for filter and serial synapses but not for cascade synapses. At a population level, sparseness, synaptic complexity and, crucially, nonzero spontaneous activity interact to promote increased optimal population SNR memory lifetimes. It is remarkable that non-cascade complex synapse models therefore appear to require the existence of spontaneous activity in a population setting with sparse population coding.

In reaching these conclusions, we have employed two superficially rather different memory storage protocols. First, the Hebb protocol uses two-level inputs , while the Hopfield protocol uses four-level inputs , although in stripping out spontaneous activity, the latter reduces to the standard conventions of the Hopfield model with its two-level inputs . However, because we have considered binary strength synapses with , the effective contributions of these two sets of inputs to a perceptron’s activation are identical: in both protocols, takes the same four possible values. Second, the Hebb protocol uses the cue and target sub-populations approach to determine the direction of synaptic plasticity, while the Hopfield protocol uses the standard Hopfield rule governed by the product of evoked pre- and postsynaptic activity. However, in both protocols, synapses experience identical potentiating and depressing plasticity induction signals at the same separate rates, rfg/2. Furthermore, in both protocols, these induction signals are produced by imposing a pattern of electrical activity on the sub-population of active neurons during memory storage, rather than by allowing neurons’ activities to be generated via direct, afferent synaptic drive. Both protocols therefore implicitly assume executive control of memory storage by other brain regions (see, for example, Eichenbaum and Cohen 2001). These two differences are indeed therefore just superficial, and this is reflected in the fact that the mean activation evolves identically under both protocols. The real difference between the Hebb and Hopfield protocols does not reside in these matters of convention and definition. Rather, it resides in the fact that an active perceptron’s synapses with active inputs experience either only potentiating or only depressing induction signals during memory storage under the Hebb protocol, while in the Hopfield protocol some experience potentiating and others depressing induction signals. This difference gives rise to the Hebb protocol’s complicated equilibrium structure, with its nonzero pairwise and higher-order synaptic correlation functions. Remove this higher-order structure, and the two protocols would have identical statistics for perceptron activation. Indeed, in the limit of small fgN in which at most one of a perceptron’s synapses experiences a plasticity induction signal during memory storage, the dynamical difference between the two protocols vanishes and their statistical structures become identical.

Two earlier studies have considered memory lifetimes in complex models of synaptic plasticity in the presence of sparse population coding (Leibold and Kempter 2008; Rubin and Fusi 2007). Leibold and Kempter (2008) used the cue-target protocol that we have adapted and referred to as the Hebb protocol. They employed synaptic strengths rather than our , although this difference is unimportant because it just amounts to an effective re-definition of the firing threshold (Elliott and Lagogiannis 2012). They also employed two-level activities, but with , so without considering the possible influence of nonzero spontaneous activity, , on memory lifetimes. Rubin and Fusi (2007) used the Hopfield protocol with two-level activities, , interpreting as spontaneous activity and as evoked activity, and stressed the importance of considering the impact of spontaneous activity on memory lifetimes. We have modelled spontaneous activity in the Hopfield protocol by moving to four-level inputs, but as indicated, this approach is essentially equivalent to two-level inputs for synapses with in terms of the overall statistical structure of perceptron activation. However, by using four activity levels, we are able to consider varying over its allowed range in order to explore the impact of different degrees of spontaneous activity on memory lifetimes. A significant difference between our approach and that of Rubin and Fusi (2007) is that we do not allow spontaneous activity to induce synaptic plasticity, a position that we consider to be mandated by a broadly BCM (Bienenstock et al. 1982) view of synaptic plasticity, as discussed earlier. Finally, our respective definitions of the memory signal, from which SNR memory lifetimes are obtained, differ in a population setting. Rubin and Fusi (2007) define this signal over the entire population of neurons, while we define it over only that sub-population of neurons that are directly involved in memory storage (or the equivalent of Leibold and Kempter (2008)’s target sub-population). This difference leads to different scaling behaviours of optimal population SNR memory lifetimes as a function of the sparseness of the population coding.

The difference between the scaling behaviours of optimal SNR memory lifetimes (optimised with respect to sparseness) in the single-perceptron and population cases is intriguing. Furthermore, the role of even very small levels of spontaneous activity in enhancing optimal population SNR lifetimes with increasing synaptic complexity in non-cascade models is fascinating. However, we have cautioned against over-interpreting results from an SNR analysis of memory lifetimes. This analysis depends on the distribution of being tightly concentrated around its supra-threshold mean. We have shown in earlier work that this requirement is often not satisfied, and that a FPT approach is required to examine memory lifetimes away from this regime (Elliott 2016a, 2017a, 2020). Here, for simple synapses, we have explicitly seen that the single-perceptron SNR analysis breaks down in the limit of small f, and so it cannot probe the very sparse coding regime. The explanation for this failure is straightforward: as f is reduced, the initial SNR reduces, and below some threshold value of f the SNR validity condition fails. For a single perceptron, we saw that this condition is (for either protocol). Plugging in for SU synapses with and , this condition becomes and , respectively, where we saw the former case earlier. Both conditions are violated, although with spontaneous activity the violation is not so great. Although we have not extended our FPT analysis of filter-based synapses (Elliott 2017a, 2020) to the sparse coding regime considered here, the same issues arise with complex synapses. Therefore, we fully expect single-perceptron SNR optimality conditions to be violated for complex synapses, too.

Whether population SNR optimality conditions are violated, in either simple or complex models, is unclear. We would need to extend our single-perceptron FPT analysis to a population setting. Furthermore, this extended analysis would need to be reducible to the population SNR analysis with its rather coarse approximation that neurons’ activities evolve independently, despite synaptic coupling. However, it is extremely tempting to speculate that the simple synapse condition for population SNR validity is just the obvious generalisation, namely . Using the population results for for simple synapses, this condition becomes the false for and the true for . It is thus quite remarkable that if this speculation is borne out by a more careful analysis, then optimal population SNR memory lifetimes for simple synapses are valid in the presence of spontaneous activity.

Table 3

List of main mathematical symbols and their meanings appearing in the appendices

Symbol	Meaning
Φmn(p)	Φmn(p)=^nC_mp^m(1-p)^n-m, the binomial distribution’s mass function
e_(N)	Normalised unit eigenvector of (N+1)×(N+1) matrix with transition elements given in Eq. (42)
F_N(z)	F_N(z)=∑i=0Ne_i(N)z^i, probability generation function of the components of e_(N)
G(z, w)	G(z,w)=∑N=0∞F_N(z)w^N, ordinary generating function for the PGFs F_N(z)
ε_(N), F_N(z), G(z,w)	Equivalents of e_(N), F_N(z), G(z, w) for model-independent arguments
H(x)	G(z, w) re-expressed in terms of the single dependent variable x=w(1-z)/[2-w(1+z)]
κ_n	Coefficient in power series for H(x), or n synapses’ equilibrium strength correlation coefficient
K(z)	Exponential generating function for the correlation coefficients κ_n
ψ_j	ψ_j=ψ1-ψ^j