Timing is not Everything: Neuromodulation Opens the STDP Gate.

Spike timing dependent plasticity (STDP) is a temporally specific extension of Hebbian associative plasticity that has tied together the timing of presynaptic inputs relative to the postsynaptic single spike. However, it is difficult to translate this mechanism to in vivo conditions where there is an abundance of presynaptic activity constantly impinging upon the dendritic tree as well as ongoing postsynaptic spiking activity that backpropagates along the dendrite. Theoretical studies have proposed that, in addition to this pre- and postsynaptic activity, a "third factor" would enable the association of specific inputs to specific outputs. Experimentally, the picture that is beginning to emerge, is that in addition to the precise timing of pre- and postsynaptic spikes, this third factor involves neuromodulators that have a distinctive influence on STDP rules. Specifically, neuromodulatory systems can influence STDP rules by acting via dopaminergic, noradrenergic, muscarinic, and nicotinic receptors. Neuromodulator actions can enable STDP induction or - by increasing or decreasing the threshold - can change the conditions for plasticity induction. Because some of the neuromodulators are also involved in reward, a link between STDP and reward-mediated learning is emerging. However, many outstanding questions concerning the relationship between neuromodulatory systems and STDP rules remain, that once solved, will help make the crucial link from timing-based synaptic plasticity rules to behaviorally based learning.

Introduction

The first groundbreaking in vitro STDP studies seemed to paint a very clear picture: The near-coincidence of presynaptic input and postsynaptic spiking enables neurons to enhance or decrease their synaptic weights depending on the exact timing of these two events (Magee and Johnston, 1997; Markram et al., 1997; Bi and Poo, 1998; Debanne et al., 1998). This finding was a giant step forward in our view of synaptic plasticity rules: It tied together the idea that both single spikes and their precise timing matter; with the implication that neurons have a means to associate arriving inputs with the outgoing spikes and adapt the synaptic weights accordingly. It was therefore very intuitive to postulate STDP as a more temporally specific extension of Hebbian associative learning and experience driven plasticity (for definition, see below; and for modeling approaches see: Gerstner et al., 1993, 1996; Abbott and Blum, 1996; Blum and Abbott, 1996; Mehta et al., 2000).

However, for this idea to be relevant for behavioral learning as first formally proposed by Hebb (1949), it must hold true in vivo. This is where the STDP concept faces two conundrums, the first based on ongoing spiking activity, and the second based on the timing of spikes in relation to behavioral outcome.

Unlike in vitro, in vivo there are continuous barrages of ongoing presynaptic activity impinging on the dendritic tree that can generate postsynaptic spikes (Figures 1A,B) (Wilson and Groves, 1981; Cowan et al., 1994; Wilson and Kawaguchi, 1996; Stern et al., 1997). These spikes backpropagate throughout the neuron (Waters et al., 2003; Waters and Helmchen, 2004), potentially interacting with the vast number of dendritically located synapses as these synapses continue to receive barrages of excitatory inputs (Figure 1C2). Under these conditions, the implications of STDP rules on individual synapses would be that a synapse active just prior to a spike event will increase in efficacy, whereas a synapse that is active just after the spike, will decrease its efficacy. The question arises, whether the mere association of presynaptic input and postsynaptic spiking activity would be enough to alter synaptic efficacy, and whether individual synapses in turn continuously scale up and down as inputs and backpropagating spikes constantly interact? Moreover, do spontaneously occurring spikes (Figure 1C2) and stimulus-evoked spikes (Figure 1C3) equally change synaptic weights as they interact with presynaptic input (Figure 1D)? When a spike is fired, whether it is spontaneous or evoked, how are active synaptic inputs that are driven by a stimulus separated from those that are due to the ongoing activity? One possible solution to these selectivity problems was originally proposed in relation to reward mediated learning (Miller, 1981; Wickens, 1990). These theoretical studies proposed that, in addition to the associated pre- and postsynaptic activity, a “third factor” was available to the network that enabled both the temporal and the spatial selection of specific inputs. To apply this to the in vivo situation, raises several further questions: could a neuromodulator represent such a third factor for selecting specific active inputs to a neuron that is embedded in a continuously active network? Given that many repetitions of timed pre–post pairings are typically necessary for STDP induction, could a third factor modify the number of repetitions needed for plasticity induction? In addition to the third factor requirement, other possible solutions for plasticity induction in vivo have been proposed that are not covered in the present review (Gerstner et al., 1996; Kempter et al., 1999; Beggs, 2001; Seung, 2003; Xie and Seung, 2004).

Figure 1

Sub- and suprathreshold neuronal activity . (A) Whole-cell recording from a pyramidal neuron in primary sensory cortex in vivo. Membrane potential trace contains (1) upstates, generated by presynaptic input, with no APs (action potentials: subthreshold events; subthr), (2) upstates with spontaneous APs (spont), and (3) upstates with APs evoked by sensory stimulation (sensory stim, indicated by bar). Hyperpolarizing current steps (I) were applied to determine input resistance. (B) Examples of spontaneous and stimulus-evoked activity in vivo. Events marked 1–3 in A (gray boxes) are depicted here in higher magnification. APs are truncated.(C) Dendritic interactions of presynaptic inputs during both subthreshold upstates and suprathreshold upstates with a spontaneous or a stimulus-evoked backpropagating AP. Left: Biocytine-stained pyramidal neuron showing soma, dendritic and axonal arborization. Part of the dendrite is shown schematically in the three panels at the right: (C During subthreshold events, upstate related synaptic input (up) arrives at dendritic spines. (C Spontaneous backpropagating APs (bAP spont) putatively interact with upstate related synaptic input arriving at plasticity-relevant timings. (C During sensory stimulation, stimulus-evoked backpropagating APs (bAP stim) can putatively interact with upstate related or with stimulus-evoked synaptic input (stim).(D) Putative changes in synaptic strength based on the timing of the AP with respect to incoming synaptic input (for both, upstate-related input [red] and/or stimulus-evoked input [green]). The question arises, if spontaneous bAPs as well as stimulus-evoked bAPs induce plasticity, when they interact with upstate related inputs (D2 vs. D3). In addition, the question arises, if both, stimulus-evoked and upstate-related input – when timed to coincide with bAPs – induce changes in synaptic strength (D. Alternatively, in addition to timing, factors may exist that enable the spatial and temporal selection of activated synapses for plasticity.

If spike timing dependent synaptic plasticity rules are the basis for the modification of behavior, and neuromodulatory systems are critical for this process, then a second conundrum of temporal credit assignment is faced. Both the behavioral signals and the behavioral outcome must be taken into account temporally. This likely also includes the activation of subcortical modulatory nuclei that can mediate for example alerting or rewarding signals to target structures (Schultz, 2000). How does such behavioral activation of a modulatory center influence the interaction of near-coincident pre- and postsynaptic activity with spatial and temporal specificity? Near-coincident pre- and postsynaptic spiking activity and neuromodulators most likely act on different timescales, ranging from tens of milliseconds for pre- and postsynaptic spikes to seconds or longer for some neuromodulators (for review see: Schultz, 2007). How will these different temporal activation schemes work together during behavioral learning? This temporal credit assignment problem is not new in neuroscience, as for example in reward mediated learning the “distal-reward problem” has been recognized years ago: How can the reward relate to specific events that happened earlier in time than the reward (Hull, 1943; Blum and Abbott, 1996; Schultz, 1998, 2006; Sutton and Barto, 1998; Izhikevich, 2007; Vasilaki et al., 2009)? Specific subcortical “reward systems” have been implicated in such learning with the neuromodulator dopamine being the most characterized (for review see: Schultz, 2000, 2002).

Although the rules associated with STDP have started to be addressed in vivo (Meliza and Dan, 2006; Jacob et al., 2007), to date, all the data about the involvement of neuromodulators in STDP have come from in vitro studies. In vitro, the dopaminergic system, amongst a number of other neuromodulatory systems, has been found to influence timing-dependent plasticity (Bissiere et al., 2003; Couey et al., 2007; Seol et al., 2007; Lin et al., 2008; Pawlak and Kerr, 2008; Shen et al., 2008; Zhang et al., 2009). The following sections will present the existing in vitro experimental evidence concerning how neuromodulators are involved in timing-based plasticity. This review will be restricted to neuromodulator-actions on timing-dependent plasticity in the mammalian central nervous system. Furthermore, we concentrate on long-range neuromodulatory systems (that are thought to become activated by distinct behavioral states in vivo), although locally acting systems and retrograde messengers undoubtedly play an important role in STDP. Such locally acting systems important for STDP are for example endocannabinoids (Sjostrom et al., 2003; Bender et al., 2006; Nevian and Sakmann, 2006; Tzounopoulos et al., 2007), metabotropic glutamate receptors (mGluRs; Egger et al., 1999), and brain-derived neurotrophic factor (Mu and Poo, 2006; Sakata et al., 2009; Sivakumaran et al., 2009).

Experimental Evidence for Involvement of Neuromodulators in STDP

Neuromodulators are involved in most forms of synaptic plasticity ranging from short-term plasticity (ms) (for review see: Lovinger, 2010) to long-term plasticity (hours) (Neuman and Harley, 1983; Frey et al., 1990; Huerta and Lisman, 1993; Thomas et al., 1996), to experience-dependent plasticity (Bear and Singer, 1986; Kilgard and Merzenich, 1998) as well as structural plasticity (Ingham et al., 1998; Day et al., 2006; Gerfen, 2006) (for definitions, see below). Although over the past few decades the role of neuromodulation in certain forms of synaptic plasticity that mainly used high frequency stimulation induction protocols has been well established, it is not clear how these results relate to STDP (for reviews see: Jay, 2003; Hasselmo, 2006; Sara, 2009; Wickens, 2009).

To identify common neuromodulatory rules across the existing STDP studies is potentially difficult as not all studies have used the same induction protocols. The induction protocols used range from pairing of single spikes with a single synaptic input (Lin et al., 2003; Couey et al., 2007; Pawlak and Kerr, 2008; Zhang et al., 2009), and spike bursts with a single synaptic input (Seol et al., 2007; Shen et al., 2008) to “theta” burst paradigms, in which multiple excitatory postsynaptic potentials (EPSPs) are interleaved with multiple spikes (Bissiere et al., 2003; Shen et al., 2008) (see also Table 1). The use of multiple spikes with multiple EPSPs during theta burst protocols, usually evoked at around 30–50 Hz, does not allow true timing to be investigated as the preceding spike is always temporally close to the following evoked EPSP (for more details see: Froemke et al., 2010) but this may, in some cell types, be closer to what occurs during behavior. Studies into such complex pairing protocols indicate that the complex EPSP–spike interactions affect downstream signaling cascades differently to seemingly more simple EPSP–single spike interactions (Wang et al., 2005; Froemke et al., 2006). Also, other alterations to STDP recording conditions, for example the presence of GABAergic transmission or pre–post repetition rate may change the STDP window (for further reading see: Wickens, 2009). This opens the possibility that neuromodulators can activate different second messenger pathways depending upon the STDP induction protocol and recording conditions that were used. The identification of common neuromodulatory rules is further complicated by the use of different tools to manipulate neuromodulatory systems amongst studies, for example application of receptor agonists, receptor antagonists, or of the neuromodulator itself, often with different application times.

Table 1

Comparison of studies investigating the effect of neuromodulators on STDP.

Study	Brain region	Cell type investigated	Neuromodulator involved (via receptor subtype)	STDP induction protocol	Neuromodulator effect on STDP	Main method of neuromodulatory system manipulation	Mechanism mediating neuromodulator effect on STDP
Bissiere et al. (2003)	Lateral amygdala (mouse)	Projection neurons	Dopamine via D2 Rs	t-LTP: 3 EPSPs timed to 3 APs	Permitted t-LTP	Application of dopamine (100 μM) and receptor agonists	Suppression of feedforward inhibition
Pawlak and Kerr (2008)	Dorsal striatum (rat)	Spiny projection neurons (SPNs)	Dopamine via D1/D5 Rs	t-LTP: 1 EPSP – 1 AP; t-LTD: 1 AP – 1 EPSP	Permitted t-LTP and t-LTD	Application of dopamine receptor antagonists	?
Shen et al. (2008)	Dorsal striatum (mouse)	Spiny projection neurons	Dopamine via D1/D5 and D2 Rs	t-LTP: 3 EPSPs timed to 3 APs; t-LTD: 3 APs timed to 1 EPSP	Permitted t-LTP and t-LTD in specific SPN subgroups	Application of dopamine receptor antagonists	?
Couey et al. (2007)	Prefrontal cortex (mouse)	Layer 5 pyramidal neurons	Nicotine via nAChRs	t-LTP: 1 EPSP – 1 AP	Block of t-LTP; instead, induction of small amount of LTD (only 10 μM)	Application of nicotine (300 nM; 10 μM)	Increase in inhibition; note: stronger protocol (1 EPSP – 2 or 3 APs) still induces t-LTP in nicotine
Zhang et al. (2009)	Hippocampus (rat, dissociated culture)	Glutamatergic (presumably pyramidal) neurons	Dopamine via D1/D5 Rs	t-LTP: 1 pre-AP – 1 post-AP; t-LTD: 1 post-AP – 1 pre-AP	“Wider” range of spike timings induces t-LTP, less spike pairings required to induce t-LTP	Application of dopamine (20 μM)	?
Lin et al. (2003)	Hippocampus (rat)	CA1 pyramidal neurons	Noradrenaline via β-adrenergic Rs	t-LTP: 1 EPSP – 1 AP	“Wider” range of spike timings induces t-LTP	Application of agonists	Modulation of PKA or ERK/MAPK signaling??
Seol et al. (2007)	Visual cortex (rat)	Layer 2/3 pyramidal neurons	Acetylcholine via M1 muscarinic Rs; noradrenaline via β-adrenergic Rs	t-LTP: 1 EPSP timed to 4 APs; t-LTD: 4 APs timed to 1 EPSP	Cooperation between cholinergic and adrenergic systems allows for bidirectional STDP	Application of agonists	Promotion of AMPA receptor phosphorylation at sites implicated in plasticity expression

Rs, receptors; ERK, extracellular signal-regulated kinase; MAPK, mitogen-activated protein kinase; pre-AP, post-AP, connected pairs of neurons, in which an AP in the presynaptic neuron was timed with an AP in the postsynaptic neuron.

Therefore, while we attempt to summarize neuromodulatory actions during timing-based plasticity in the following paragraph, one should be aware that differences in STDP induction protocols as well as method of neuromodulatory manipulation might impede finding common principles of neuromodulatory actions across studies.

Permission to change: neuromodulators and STDP

Studies investigating the effect of neuromodulators on STDP have either used a fixed relative pre–post timing to induce timing-dependent long-term potentiation and depression (t-LTP or t-LTD) (Bissiere et al., 2003; Couey et al., 2007; Pawlak and Kerr, 2008; Shen et al., 2008), or used a whole range of pre–post stimulation timings to investigate the effects of neuromodulation on the STDP timing window shape (Lin et al., 2003; Seol et al., 2007; Zhang et al., 2009). As will be discussed in detail in the following section, the studies using a fixed pre–post timing have often identified neuromodulatory signaling as a requirement for STDP to occur.

Of the neuromodulators investigated, dopamine is the most widely studied and has been shown to influence timing-dependent plasticity across several brain regions. In amygdala, t-LTP was only induced by a protocol consisting of short bursts of afferent stimulation timed to action potential (AP) bursts, when either dopamine was applied or GABAergic inhibition was blocked (Bissiere et al., 2003). Here, dopamine acted by activating dopamine D2 receptors, thereby suppressing feedforward inhibition from local interneurons, which permitted t-LTP induction by burst-pairing. The effect of dopamine depended on intact GABAergic transmission, since no potentiation occurred when dopamine was applied during pairing when GABAergic transmission was blocked, suggesting that the pairing protocol triggers different processes depending on the absence or presence of synaptic inhibition (Bissiere et al., 2003). In dorsal striatum under GABAA block, timing-dependent LTP was induced when a single AP closely followed an EPSP, while timing-dependent LTD was induced when the order was reversed (Pawlak and Kerr, 2008). Here, blocking dopamine D1/D5 receptors prevented t-LTP as well as t-LTD (Figures 2A,B), while blocking dopamine D2 receptors altered the onset, but not the final peak change in plasticity.

Figure 2

Timing-dependent LTP and LTD are under the control of dopamine D1/D5 receptors in striatal principal neurons. Anatomy of neuromodulatory fibers and the respective receptors as exemplified for striatal dopamine. (A) t-LTP was induced under control conditions (black circles) with a STDP protocol, where the AP followed the EPSP by 10 ms (Δt = 10 ms). (B) t-LTD was induced under control conditions with a protocol, where the EPSP followed the AP by 30 ms (Δt=−30 ms). No plasticity was observed with these two protocols, when dopamine D1/D5 receptors were blocked (green circles). (A,B modified from: Pawlak and Kerr, 2008). (C)Excitatory (glutamatergic) synapses arising from the cortex (Cx) or the thalamus (Th) onto spines of a striatal principal neuron. Only some of these spines also receive innervation from nigrostriatal (SN) dopaminergic fibers. Dopamine receptors (D1 and D2 subgroups) are distributed across distinct pre- and postsynaptic sites. For simplicity, the dopaminergic receptors, which are located on several of the striatal interneuron classes, are omitted from this cartoon.

Hence, the activation of dopamine D1/D5 receptors allowed for two events, the presynaptic input and the postsynaptic spike that occurred on a timescale of a few tens of milliseconds, to induce a lasting change in synaptic efficiency. Because dopamine receptors were blocked throughout the experiment with specific antagonists, the issue still remains whether dopamine alone acts in the same way when applied during the induction period (see “Changes to the Shape of the STDP Window”). In the subpopulation of striatal principal neurons that do not express dopamine D1 receptors, other neuromodulatory receptor systems were required for STDP. Here, adenosine A2 receptors, which are coupled to the similar second messenger cascades as D1 receptors (Premont et al., 1977; Schwarzschild et al., 2006) had to be activated for t-LTP induction. Both, endocannabinoid CB1 receptor as well as dopamine D2 receptor activation was required for t-LTD induction (Shen et al., 2008). In contrast to the amygdala, in the striatum the effect of dopamine on STDP seems to operate independently of GABAergic transmission (Pawlak and Kerr, 2008; Shen et al., 2008).

Nicotine was shown to be involved in STDP by acting on nicotinic acetylcholine receptors (nAChRs) in prefrontal cortex (Couey et al., 2007). Here, nicotine application caused normally t-LTP-inducing pre–post pairings, consisting of EPSP and single APs to induce a small amount of t-LTD in layer 5 pyramidal neurons. As an underlying mechanism, nicotine was found to strongly increase inhibition of layer 5 pyramidal neurons. Accordingly, the blocking effect of nicotine on t-LTP was partly overcome when inhibition was also blocked or a stronger t-LTP-inducing stimulus, consisting of pairing EPSPs with AP bursts, was applied. Since calcium is thought to be a crucial second messenger in synaptic plasticity induced by using spike-timing or other plasticity inducing protocols (for reviews see: Artola and Singer, 1993; Malenka and Bear, 2004), and since pre–post timing protocols produce characteristic spatiotemporal calcium signals (Koester and Sakmann, 1998), Couey et al. (2007) also investigated dendritic calcium dynamics during AP–EPSP pairing. Under the influence of nicotine, calcium changes were reduced during a pairing protocol that normally induced t-LTP in control condition (single-AP pairings). In contrast, during a stronger t-LTP-inducing protocol (AP-burst pairings), changes in dendritic calcium were comparable between control groups and nicotine groups. Both, pre- and postsynaptic nAChRs, distributed across several classes of interneurons, were suggested as potential targets of nicotine when reversing prefrontal t-LTP into t-LTD (Couey et al., 2007).

Not only manipulations of nicotinergic signaling, but also manipulations of the balance between dopaminergic and adenosinergic signaling are capable of reversing the sign of plasticity (i.e., converting t-LTP into t-LTD or vice versa); when dopaminergic signaling via D2 receptors was blocked and adenosine signaling was “boosted” by application of adenosine A2 receptor agonists, t-LTP was reversed into t-LTD upon a pre–post timing protocol (Shen et al., 2008).

In summary, dopamine receptor activation is often a prerequisite for timing-dependent plasticity to occur, and nicotine's action is to increase the threshold for t-LTP induction. While the investigation of neuromodulatory influences using a few selective EPSP–AP timing protocols has led to important insights into specifically dopamine's and nicotine's actions during STDP, a more complete picture emerges when neuromodulatory influences are studied across the entire STDP window.

Changes to the shape of the STDP window

In hippocampus, unlike in striatum and amygdala, dopamine receptor activation was not a critical requirement for STDP induction. Here, t-LTP was induced by a pre–post protocol in the presence of dopamine receptor blockers (Zhang et al., 2009). However, dopamine application resulted in a modest, albeit not significant, increase in the amount of t-LTP observed when single postsynaptic APs closely followed the presynaptic activation with a delay of 10 ms. The much more dramatic effect observed with dopamine application was a change in the shape of the STDP window, allowing for longer pre–post timing delays to increase synaptic efficiency (Zhang et al., 2009). This widening effect was attributed to dopamine D1/D5 receptor activation, and was estimated to expand the t-LTP window by at least 25 ms. Surprisingly, t-LTD, as normally induced by a post–pre protocol, was converted into t-LTP by dopamine (Figure 3A). A similar broadening effect on the t-LTP window during the pairing of single APs with presynaptic activation was reported for the neuromodulator noradrenaline in hippocampal CA1 neurons (Lin et al., 2003). While the activation of β-adrenergic receptors widened the t-LTP window by about 15 ms, the overall amount of plasticity induced by the close pre–post pairings was not affected. Post–pre pairings were not tested. An unexpected similarity between both, β-adrenergic and dopaminergic actions on the “widening” of the t-LTP window is that the effect was expressed slowly, meaning that synaptic efficiency was unchanged directly after the pairing protocol and gradually started to increase from around 15 min post pairing protocol. The implications of window widening are that the activation of dopamine or noradrenaline receptors reduces the threshold for t-LTP induction by allowing for a wider range of pre–post timings to induce synaptic potentiation. The next question is whether the number of spikes needed to induce plasticity is altered with neuromodulation?

Figure 3

(A) Dopamine changes the shape of the STDP window in hippocampal neurons. STDP window in control conditions (black circles) and when dopamine was present during the STDP induction protocol (green circles). On the “t-LTP side” of the window (positive pre–post timings), dopamine allowed for longer intervals between spike and synaptic activation to induce potentiation of synaptic strength. On the “t-LTD side” of the window (negative pre–post timings), dopamine enabled t-LTP induction with a protocol that induced t-LTD under control conditions. (B) Dopamine reduces the number of spike pairs required to induce t-LTP. In control conditions, about 60 repetitions of timed pre–post spike pairings were required to induce robust t-LTP. In presence of dopamine, already 5–10 such pairings were sufficient to induce significant t-LTP. (A,B modified from: Zhang et al., 2009).

Changes to the number of trials required to achieve plasticity

In addition to having an effect on the shape of the STDP window, dopamine also affected the number of pre–post pairing episodes required to induce plasticity in hippocampal neurons (Zhang et al., 2009). Specifically, when dopamine D1/D5 receptors were activated, successful t-LTP induction required a strongly reduced number of timed pre–post pairings, namely instead of the typically required 60 pairing trials, less than 10 pairings were required (Figure 3B). Thus by decreasing the required number of spike pairings, dopamine decreases the threshold for t-LTP induction.

Cooperation between neuromodulators

As suggested by the anatomy of converging neuromodulatory fibers as well as direct physiological evidence, one neuromodulator often does not act in isolation, but several neuromodulatory systems interact (for example, see Bear and Singer, 1986; Zhou et al., 2001). In visual cortex layer 2/3 pyramidal neurons, pairing stimulation of layer 4 afferents with AP bursts did not result in plasticity, neither for pre–post protocols, nor for post–pre protocols (Seol et al., 2007; but compare Feldman, 2000; Froemke et al., 2005). For plasticity to occur, neuromodulatory receptors had to be activated during the pre–post timing protocols. Specifically under stimulation with β-adrenergic agonists, pre–post pairings, with timings between −50 and +50 ms, always induced t-LTP. Conversely, activation of M1 muscarinic receptors always resulted in t-LTD within the same range of timings. Finally, the “normal” standard STDP window displaying bidirectional plasticity, with causal pre–post timings leading to t-LTP and anticausal post–pre timings leading to t-LTD, was achieved with the combined application of β-adrenergic and M1 muscarinic agonists (Figure 4).

Figure 4

Coapplication of β-adrenergic and M1 muscarinic agonists is required for “standard” bidirectional STDP in visual cortex. In the presence of a β-adrenergic agonist alone, close positive as well as negative pre–post timings induced t-LTP (green circles). When a M1 muscarinic agonist was present, close positive as well as negative pre–post timings induced t-LTD (red circles). Only the combined application of β-adrenergic and M1 muscarinic agonists resulted in the “standard” STDP window with close pre–post timings leading to t-LTP, and post–pre timings leading to t-LTD (black circles). (Modified from: Seol et al., 2007).

The activation of β-adrenergic and M1 muscarinic receptors resulted in temporary phosphorylation of distinct sites at AMPA receptors that have been suggested to be crucial for t-LTP and t-LTD, respectively. This led the authors to conclude that neuromodulators supply AMPA receptors with distinct “tags” that allow during certain pre–post spiking timings the induction of t-LTP and t-LTD, respectively. In summary, the activation of noradrenergic and muscarinic acetylcholine receptors (mAChRs) is required for STDP, and more specifically, only the concurrent activation of the two neuromodulatory systems is required to achieve a “standard” STDP window with a t-LTP and a t-LTD side.

Conclusion of this section

The observed neuromodulatory actions so far can be divided into two categories (see also Table 1): In the first category, neuromodulator receptor activation is necessary for plasticity (Bissiere et al., 2003; Seol et al., 2007; Pawlak and Kerr, 2008; Shen et al., 2008), thereby representing in addition to the precise timing of pre- and postsynaptic activity, a third factor essentially required for plasticity induction. Notably, in one study, two neuromodulators acted in concert to enable bidirectional STDP (Seol et al., 2007).

In the second category, the neuromodulator changes the conditions for plasticity by either increasing (Couey et al., 2007) or decreasing the threshold for plasticity induction (Lin et al., 2003; Zhang et al., 2009; but see also Bienenstock et al., 1982). An effect observed in studies from both categories is that specific manipulations of one or several neuromodulator systems, result in sign reversal of plasticity, meaning that a normally t-LTP-inducing stimulus induced t-LTD, or vice versa (Bissiere et al., 2003; Couey et al., 2007; Seol et al., 2007; Shen et al., 2008; Zhang et al., 2009). Although the mechanisms underlying such sign reversal are not clear, activity patterns that “boost” backpropagating APs in remote dendrites have been shown to “switch” t-LTD to t-LTP (Sjostrom and Hausser, 2006). Since some neuromodulators can exert a short-term effect on dendritic excitability and backpropagation (for review see: Waters et al., 2005), neuromodulators could also modulate backpropagating APs during STDP protocols, although this might not occur with all neuromodulators (Gulledge and Stuart, 2003).

The effect of neuromodulators on dendritic excitability is not restricted to short-term effects, since for example, the combination of mACh receptor activation with weak dendritic spikes in a distinct dendritic compartment resulted in a long-lasting excitability increase restricted to the involved dendritic compartment (Losonczy et al., 2008). This excitability increase transformed the weak dendritic spikes into strong dendritic spikes. Strong dendritic spikes have been implicated in drastic trial reduction to induce plasticity (Remy and Spruston, 2007). Such dendritic spikes are usually evoked with rather focused massive or convergent activity. Alternatively, a neuromodulator could reduce the trial number required for plasticity induction by making a dendritic compartment receptive for strong dendritic spike initiation (Losonczy et al., 2008).

From a temporal point of view, neuromodulators have been found to influence STDP on at least three timescales: on the scale of tens of milliseconds, neuromodulators influenced the interaction of pre- and postsynaptic spikes to induce plasticity; on the scale of seconds, neuromodulators influenced the number of repetitions of pre–post activity needed to evoke plasticity; and on the order of minutes, neuromodulators influenced the time course of plasticity.

Does the Third Factor Have a Timing Issue, Too?

The studies listed in Section “Experimental Evidence for Involvement of Neuromodulators in STDP” have either constantly manipulated neuromodulator receptors during the entire experimental period (Lin et al., 2003; Pawlak and Kerr, 2008; Shen et al., 2008) or only during the induction period (Bissiere et al., 2003; Couey et al., 2007; Seol et al., 2007; Zhang et al., 2009). An important question that is very difficult to address experimentally, is how the outcome, in terms of plasticity, depends on the exact point in time of activation of neuromodulator-receptors, relative to pre- and postsynaptic spiking?

Since neuromodulator release sites and the receptors for the neuromodulator are not necessarily located close together on either side of the synaptic cleft, as in the classical concept of a synapse, the time required for diffusion of the released molecules has to be taken into account (see also Figure 2C). The modulator molecules have first to be released, then “travel” and bind to the respective receptors and initiate some G-protein coupled signaling cascade, which is a very different scenario from fast glutamate transmission. The time course of neuromodulator action was traditionally thought to be slow (on the scale of minutes), but recent evidence suggests that the time course is on the order of a few seconds (for review see: Sarter et al., 2009). Despite this recent change in thinking the question still arises how do these different timescales of spikes (1–2 ms) and neuromodulation (seconds) fit together in a working mechanism?

Three possible scenarios can be devised of how such a mechanism could work. The first two are at the single neuron level involving an “eligibility trace”, and the third is at the network level and relies on reverberating activity.

In the first scenario, the coordinated pre–post activity occurs before the neuromodulator release, as would be the case during unexpected reward. Here, spike and synaptic activation could leave a time decaying eligibility trace (Wang et al., 2000; Sarkisov and Wang, 2008) that subsequent neuromodulator receptor activation may then interact with to modulate plasticity.

In the second scenario, the neuromodulatory receptors are activated before the coordinated pre–post activity occurs, as would be expected during the learning of attention-based tasks. Here, the signaling mechanisms activated by neuromodulatory receptors themselves may create a slowly decaying eligibility trace, with which the coordinated pre–post activity can then interact with to modulate plasticity.

Experimental evidence for either scenario or the underlying molecular mechanisms is mostly lacking. However, the two presented scenarios resemble problems faced in the field of metaplasticity (Abraham, 2008) in which the concept of an eligibility trace has also been proposed. During metaplasticity, synapses will more easily undergo plasticity after a “priming” stimulus has changed the state of specific molecular signaling cascades; this change may for example “kick” plasticity-relevant enzymes into a more receptive state or it may result in enhanced phosphorylation of intracellular or extrasynaptic AMPARs that allows them to be inserted into postsynaptic membranes when an appropriate stimulus arrives (Sun et al., 2005; Abraham, 2008). For STDP, evidence for a similar gating mechanism in accordance with scenario two (neuromodulator receptor activation occurs first, followed by near-coincident pre–post spiking) was found by Seol and colleagues (see Sections “Experimental Evidence for Involvement of Neuromodulators in STDP and “Cellular and Molecular Targets of Neuromodulators During Timing-Dependent Plasticity”). When M1 muscarinic and β-adrenergic agonists were applied and washed out, a subsequent episode of timed pre–post pairings still initiated t-LTP or t-LTD, respectively. In addition, because neuromodulators have been shown to have a direct effect on glutamatergic receptor (AMPA and NMDA) location within the synapse and activated current efficacy (Seol et al., 2007; for review see: Cepeda and Levine, 2006), this is a possible mechanism that could create an eligibility trace to interact with subsequent pre–post pairing.

A third scenario how the three factors may interact in vivo, may be that the respective pre–post-activity patterns “reverberate” in the local circuit for some time (Hebb, 1949), and that such a memory trace can be transduced into a lasting modification if a third-factor success signal is present during the reverberation (Histed et al., 2009). A problem with such a mechanism is that the reverberating activity should not produce overt action, however if different circuits are involved, it is difficult to connect the success signal to the activity that did cause overt action.

How Can Neuromodulators Influence the Interaction of Pre- and Postsynaptic Spikes: Anatomical and Physiological Considerations

Increasing evidence for the critical involvement of neuromodulator systems in STDP raises the question of how the physical location of neuromodulator release sites relates to the pre- and postsynaptic complex, which is thought to be the locus of STDP induction. Typically, neuromodulatory centers are located quite distally from the brain regions they influence (see more in Section “Activation of Neuromodulatory Systems ). In their distal target areas, generally only a subgroup of neuromodulatory fibers makes direct contact with dendritic spines that receive excitatory inputs (Freund et al., 1984; Groves et al., 1994; Smith et al., 1994), whereas other neuromodulatory fibers target dendritic shafts and somata or form varicosities that lack synaptic specializations (Seguela et al., 1989, 1990). Therefore the question arises if only this subset of directly targeted synapses is influenced by neuromodulators when pre- and postsynaptic spikes collide? This is unlikely, because the receptors for the respective neurotransmitters are widely distributed across pre- and postsynaptic sites of principal neurons and interneurons (Gerfen et al., 1990, 1995; Sesack et al., 1994; Bergson et al., 1995; Caille et al., 1996) (Figure 2C), and receptor density and location can change (Paspalas et al., 2006). A certain degree of spatial specificity of neuromodulator-actions is probably achieved during behavior, when phasic release events occur, which temporally increases the concentration of neuromodulator, locally (reviewed in: Arbuthnott and Wickens, 2007; Sarter et al., 2009, see below for definition of phasic release and also Section “Activation of Neuromodulatory Systems ). Finally, both AMPA and NMDA receptors are located presynaptically on neuromodulatory release terminals, and the activation of these receptors by overspill from neighboring active glutamatergic synapses is thought to convey further spatial specificity to the neuromodulator signal (Roberts and Sharif, 1978; Desce et al., 1992, 1994; Jin and Fredholm, 1994). Because the actions of neuromodulators are through receptors, the specific receptor subtype involved in STDP has important implications for the interpretation of neuromodulatory actions at the single neuron level. Indeed, during STDP, neuromodulators acted through specific receptor subtypes (Bissiere et al., 2003; Lin et al., 2003; Seol et al., 2007; Pawlak and Kerr, 2008; Shen et al., 2008; Zhang et al., 2009). This is a complex issue as, for example for the neuromodulator dopamine, low concentrations are thought to activate dopamine D2-like receptors in their high-affinity state, whereas high concentrations are thought to activate dopamine D1-like receptors (Richfield et al., 1989). These two receptor-subgroups are differentially expressed across neuronal populations, and can activate opposing downstream target enzymes (Girault and Greengard, 2004). Hence, during phasic release, the heterogeneous structural arrangement of release sites, different receptors subtypes, and regulated degrading/reuptake mechanisms in combination with diffusional processes are likely to generate further spatiotemporal heterogeneity of the neuromodulator signal. In addition, there is strong evidence that local spines within a dendritic region are topographically organized functionally (Jia et al., 2010) and that activity-initiated signaling cascades within the postsynaptic spines and dendrites interact locally with other spines (Harvey and Svoboda, 2007). This implies that in addition to the spatial specificity of neuromodulator release, there is a postsynaptic organization that can potentially provide very spatially defined neuromodulator action without the need for individual fibers to innervate each and every postsynaptic spine. The implication for timing-dependent plasticity in vivo would be that the timing of neuromodulator release in relation to correlated pre- and postsynaptic activity can enable the spatiotemporal selection of specific synapses for plasticity.

Cellular and Molecular Targets of Neuromodulators During Timing-Dependent Plasticity

Only few studies so far have addressed the issue of the exact cellular and molecular targets of neuromodulators when they “gate” STDP. In general, their receptors are (often) coupled to G-proteins and hereby influence intracellular second messenger cascades; (for example dopamine D1 receptors are coupled directly to adenylyl cyclase (AC) and indirectly to protein kinase A (PKA) and protein phosphatase 1 (Hemmings et al., 1984),M1 muscarinic ACh receptors are coupled to phospholipase C (PLC), β-adrenergic receptors are coupled to PKA (exception: nicotinic ACh receptors are ligand-gated channels). As a result, many different voltage-gated and calcium-dependent ion channels are influenced, which can affect membrane potential, neuronal spiking and excitatory transmission as well as inhibitory transmission (for review, see: Hasselmo, 1995; Nicola et al., 2000; Magee and Johnston, 2005; Sara, 2009). By their ability to affect dendritic ion channels, neuromodulators are certainly empowered to influence how the backpropagating AP will interact with incoming synaptic input during spike-timing paradigms (Hoffman and Johnston, 1999; Zhou et al., 2005; Sjostrom and Hausser, 2006; for review see: Tsubokawa, 2000), although specific studies investigating neuromodulatory influences on such interactions are required (but see: Couey et al., 2007).

Particularly in older animals, a preventing effect of inhibition on STDP has been described (Meredith et al., 2003). The influence that some neuromodulators have on inhibitory tone is certainly a means to affect STDP rules (D2, A2, mGluR5; Bissiere et al., 2003; Schwarzschild et al., 2006; Couey et al., 2007). However, several studies describe an effect of neuromodulators on STDP while inhibition is blocked, indicating at least one alternative mode of action of neuromodulators (Pawlak and Kerr, 2008; Shen et al., 2008).

Such an alternative mode of action is putatively an influence on the postsynaptic anchoring of glutamate receptors. For example, dopamine D1/D5 receptors and β-adrenergic receptors can increase surface expression of AMPA receptors (Chao et al., 2002; Sun et al., 2005; Oh et al., 2006) promoting synaptic insertion (for reviews see: Derkach et al., 2007; Lee and Huganir, 2008). The trafficking of AMPA receptors in and out of the synapse depends on phosphorylation of AMPA receptors at distinct sites (Lee et al., 2000, 2003; Boehm et al., 2006; He et al., 2009). In agreement with this, acetylcholine (coupled to PLC via M1 muscarinic receptors) and noradrenaline (coupled to AC via β-adrenergic receptors) gate phosphorylation at AMPA receptor sites implicated in t-LTP and t-LTD (Seol et al., 2007). In addition, β-adrenergic receptors were recently found to be anchored postsynaptically, forming a signaling complex with PKA and AMPA receptors (Joiner et al., 2010). Also, for dopamine, a complex interaction between D1 receptors and NMDA receptor channels has been reported (Cepeda et al., 1992; O'Donnell and Grace, 1994; Levine et al., 1996; Gao et al., 2001; Cepeda and Levine, 2006). Given that both dopamine and NMDA receptor activation were required for STDP (Pawlak and Kerr, 2008; Shen et al., 2008; Zhang et al., 2009), an interaction of these two receptor systems during the correlated pre- and postsynaptic spiking is possible.

Finally, it is worth considering the possibility that neuromodulators might also alter the dynamic balance of the phosphatases and kinases that control the induction of t-LTP and t-LTD (for review see: Lisman and McIntyre, 2001). For example, it is well established that PKA can reduce the activation of the phosphatases subserving LTD (Blitzer et al., 1998). This could be a plausible mechanism to account for the observation made in some studies that receptors coupled to AC, like D1 dopaminergic and β-adrenergic receptors, not only promote t-LTP but prevent t-LTD (Seol et al., 2007; Lin et al., 2008; Zhang et al., 2009).

In summary, the neuromodulatory systems can potentially affect STDP through a variety of mechanisms like changing the recruitment of inhibition at the network level, or changing excitability, kinase/phosphatase balance and priming AMPA receptor trafficking at the subcellular level. An important question that remains largely open is how those multiple actions are integrated in the different behavioral states defined by the neuromodulatory systems.

Activation of Neuromodulatory Systems In Vivo

If one attempts a synthesis of STDP and neuromodulation, the question arises at which points during behavior neuromodulatory nuclei become activated? Due to tonic background activity of these nuclei, their innervated areas experience a constant low tone of release resulting in neuromodulator concentrations in the low nanomolar range. Salient behavioral events serve to drastically increase and decrease the activity of the respective nuclei (see below). The exact spatiotemporal profile of neuromodulator concentrations achieved during behavior is mostly unknown. Perhaps the best studied neuromodulator in this respect is dopamine, and some information about dopamine's in vivo concentration is available (see below), whereas for noradrenaline and acetylcholine the concentration reached during behavior is not well studied.

Dopamine

Dopaminergic fibers arise from the ventral tegmental area and the substantia nigra. Dopaminergic neurons are activated by primary-rewarding stimuli: Unexpected rewards, but also the attentional and rewarding aspects of novel stimuli cause midbrain dopaminergic neurons to increase their firing rate (Ljungberg et al., 1992; Mirenowicz and Schultz, 1996). As a certain task is being learned, dopamine neurons shift their firing temporally toward the stimulus that indicates reward is to follow (Schultz et al., 1993). Hereby, the success-predicting stimulus has become rewarding. After a certain task has been learned, the primary-rewarding stimulus does not activate dopaminergic signals anymore; a dopaminergic signal is only initiated when a reward is unexpected or better than predicted. If a predicted reward is omitted, dopaminergic cells respond by decreasing their firing (Hollerman and Schultz, 1998). Together, this is consistent with theories of reinforcement learning stating that reinforcers only contribute to learning when they are not entirely predictable (Sutton and Barto, 1981). Recently, it has been found that a subpopulation of dopamine neurons also fire in response to aversive stimuli or associated cues (Joshua et al., 2008; Matsumoto and Hikosaka, 2009) suggesting that dopamine can code for multiple external events (Redgrave and Gurney, 2006).

The timescale of the phasic increase in firing rate of dopaminergic neurons is 50–110 ms (latency) and <200 ms (duration) with dopamine concentrations at target structures remaining elevated (150–400 nM) for up to 400 ms (Chergui et al., 1994; Dugast et al., 1994; Schultz, 2002). It is less clear how pauses in dopamine cell firing would affect local concentration levels, since the time course of clearance is relatively slow. However subtle changes in the degree of synchrony of firing have significant effects (Joshua et al., 2009).

Noradrenaline

Noradrenaline neurons located in locus coeruleus seem to play a role in vigilance, since these neurons show low firing rates during drowsiness and slow-wave sleep, regular firing at quiet wakefulness, and burst-firing in response to arousing stimuli (Aston-Jones and Bloom, 1981). A large variety of arousing and attention-demanding stimuli cause a response in these noradrenergic neurons, this also includes primary-rewarding stimuli and aversive stimuli (Foote et al., 1980; Rasmussen et al., 1986; Sara and Segal, 1991; Aston-Jones et al., 1994). In detail, this response consists of a very brief increase in AP firing (15–70 ms latency, 2–3 APs) followed by a longer suppression of AP firing (300–700 ms duration) (Berridge and Waterhouse, 2003). The noradrenergic response disappears with repeated stimulus presentations, but reappears when the stimulus is followed by reinforcement (Sara and Segal, 1991). In general, the noradrenaline signal is thought to be involved in sensory processing, decision-making, working memory, and memory formation (Cahill and McGaugh, 1996; Robbins and Roberts, 2007).

Acetylcholine

Cholinergic neurons in the basal forebrain nuclei are activated during arousal and attention (Paxinos, 2004; Sarter et al., 2005), they respond to unfamiliar stimuli (Wilson and Rolls, 1990), but also to unpredicted and predicted rewards (Richardson and DeLong, 1986, 1990). Also in striatum, tonically active striatal interneurons (TANs), which are cholinergic (Wilson et al., 1990; Zhou et al., 2002) respond to primary rewards and reward-predicting stimuli with a pause and sometimes a subsequent increase in firing (Aosaki et al., 1995; Apicella et al., 1997; Sardo et al., 2000; Zhou et al., 2002). The firing of TANs mainly encodes outcome delivery and omission at termination of the behavioral trial episode (Joshua et al., 2008). Within the cortex, acetylcholine has been suggested to enhance the response to sensory stimuli, and on more broad terms, to be important for attention and working memory (Hasselmo and Giocomo, 2006).

This section shows that release of neuromodulators occurs in a wide range of behavioral situations. Hence, the combination of theoretical and experimental work suggests that neuromodulatory influence on STDP might be linked to an equally wide range of behavioral learning processes, namely fear-conditioning (Bissiere et al., 2003), rapid learning (Zhang et al., 2009), reward-based learning (Pawlak and Kerr, 2008; Zhang et al., 2009), cognitive performance (Couey et al., 2007), but also pathological states (Shen et al., 2008) (for modeling approaches see: Baras and Meir, 2007; Florian, 2007; Legenstein et al., 2008; Vasilaki et al., 2009; Fremaux et al., 2010; Potjans et al., 2010).

Future Directions

Our knowledge about STDP and its regulation by neuromodulators has substantially increased during the last years, although the overall number of published studies concerning this topic remains low. Current experimental evidence suggests that neuromodulators shape the interaction between presynaptic and postsynaptic spike activity across many brain areas, and the predominating effect of neuromodulators is to allow plasticity or to make plasticity induction easier. Although there is amassing data from many different brain regions, it needs to be clarified how universal this additional modulatory factor is in regulating STDP. In addition, many brain areas are targeted and influenced by not only one, but by several neuromodulators (Bear and Singer, 1986; Zhou et al., 2001; Wang et al., 2006; Sara, 2009), and accordingly, there is experimental evidence that several neuromodulatory systems can interact to influence STDP rules (Seol et al., 2007). How the interaction of multiple neuromodulator systems exactly occurs during STDP, and if this interaction is a universal principle across many brain areas, will be interesting targets for future studies. Perhaps the most important outstanding question regarding STDP and neuromodulation concerns the exact time, when neuromodulatory receptors need to be activated to exert an influence on synaptic efficacy during causal and anticausal pre- and postsynaptic spiking. Along these lines, it will be important to directly test whether neuromodulators are capable of spatial or temporal selection of specific synapses for plasticity (see Sections “How Can Neuromodulators Influence the Interaction of Pre- and Postsynaptic Spikes: Anatomical and Physiological Considerations” and “Conclusion”).

Finally, not only will both experimentalists and theorists need to translate the effect of neuromodulators on STDP rules from in vitro to in vivo conditions (see: Meliza and Dan, 2006; Jacob et al., 2007), but also to the behaving animal. In addition, since it is almost certain that specific memories are stored across neuronal populations (Penfield, 1958, 1959), it will be important to see how STDP rules relate to neuronal populations in the behaving animal (Sawinski et al., 2009).

Conclusion

Spike timing dependent plasticity rules have been developed mainly on the basis of in vitro experimental data and have provided a temporally specific extension to the activity-based synaptic plasticity rules first proposed by Hebb (1949). However, when one tries to apply these rules to in vivo conditions and to the behaving animal, two conundrums arise: First, in vivo, a large amount of pre- and postsynaptic activity constantly arrives at the individual synapses. This raises the possibility that in vivo, synapses are constantly adapting their synaptic efficacy as pre- and postsynaptic spikes collide, which would be energetically inefficient for the involved neurons. An alternative possibility is that a “third factor” using a neuromodulator signal may represent a selection criteria that potentially allows presynaptic activity and postsynaptic spiking to be associated, both spatially and temporally. Thus, neuromodulators might enable the neuronal networks to select certain inputs and to make them eligible for changes in efficacy. To this end a large amount of indirect in vitro experimental evidence from many brain regions as well as theoretical evidence is being amassed that this may be the case, but a direct measurement of the third factor rule in vivo has yet to be achieved (for experimental evidence, see Section “Experimental Evidence for Involvement of Neuromodulators in STDP”; for modeling approaches see: Baras and Meir, 2007; Florian, 2007; Izhikevich, 2007; Legenstein et al., 2008; Vasilaki et al., 2009; Fremaux et al., 2010; Potjans et al., 2010).

Second, if one attempts to transfer the concepts of STDP to in vivo conditions, the obvious next questions are (a) if STDP rules are actually used for behaviorally based learning, and (b) how neuromodulation might be involved in this process. Neuromodulation alone is certainly an important factor involved in behavioral learning, as demonstrated by decades of research. If neuromodulation was instrumental in shaping STDP rules during behaviorally based learning, it would require fast time scale events like pre- and postsynaptic spikes and the putatively “slowly acting” neuromodulators to interact. The existing in vitro studies are only starting to provide insights how this temporal interaction might work. And for the in vivo situation, this picture will be much more complex, as a variety of behavioral states will release different combinations of neuromodulators at different timings and at different concentrations, activating different target receptor subtypes.

Dopamine, to date, is the most investigated neuromodulator and represents an interesting case for neuromodulator-regulation of STDP, as it has both the effect of “broadening” the t-LTP window and of changing what would normally be t-LTD into t-LTP. With regards to reward-mediated learning, the implication of these experimental findings is that any spike occurring within a certain window either before or after the synaptic input will increase the synaptic efficacy, which implies that many different external events that occurred temporally around the rewarding event could be associated with the reward.

It is an open question how an animal succeeds in linking the specific neuronal activity involved in a behavior to the behavioral outcome. How neuromodulators released during different states such as attention, arousal and reward influence this linking process, is also unknown. To achieve a full understanding of the principles of how neuromodulation shapes STDP rules might represent a first step toward solving these important questions.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.