Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value.

In changing environments, animals must adaptively select actions to achieve their goals. In tasks involving goal-directed action selection, striatal neural activity has been shown to represent the value of competing actions. Striatal representations of action value could potentially bias responses toward actions of higher value. However, no study to date has demonstrated the direct effect of distinct striatal pathways in goal-directed action selection. We found that transient optogenetic stimulation of dorsal striatal dopamine D1 and D2 receptor-expressing neurons during decision-making in mice introduced opposing biases in the distribution of choices. The effect of stimulation on choice was dependent on recent reward history and mimicked an additive change in the action value. Although stimulation before and during movement initiation produced a robust bias in choice behavior, this bias was substantially diminished when stimulation was delayed after response initiation. Together, our data suggest that striatal activity is involved in goal-directed action selection.

Animals are faced with the challenge of optimally selecting actions in changing environments. Theoretically, these decisions can be implemented by estimating the value of different actions and then choosing the action of greatest value. Recently, the striatum has been implicated as an important neural structure that may mediate this process of action selection. Electrophysiological recordings in primate and rodent striatum have identified signals correlating with the expected outcomes of actions and measures of motivation for particular responses[1, 2]. Other studies have shown that the striatum encodes representations of the value of actions in free choice tasks[3, 4]. Striatal activity also parallels the learning of rewarded responses based upon previous experience[5, 6] and is essential for the acquisition and execution of goal-directed behaviors[7].

Two dorsal striatal pathways have been proposed to mediate opposing influences on the selection of actions[8-10]. Activity within the `direct pathway' has been shown to facilitate actions while activity in the `indirect pathway' has been demonstrated to inhibit behaviors[11, 12]. The direct pathway is comprised of medium spiny neurons (MSNs) that express the dopamine D1 receptor (D1R) while the indirect pathway is comprised of MSNs that express the dopamine D2 receptor (D2R)[13-15]. It has been proposed that the D1R and D2R-expressing MSNs encode a representation of the value of actions and generate a response bias towards actions of higher value[16, 17]. While this model is consistent with numerous studies, other studies seem to suggest that these populations of neurons may merely play a permissive role in learning associations[18] or only modulate the vigor of actions without affecting the animals' choice behavior[19, 20]. No study to date has definitely demonstrated the role of these two populations of neurons in the context of reward-based decision-making[13].

To directly investigate the role of striatal activity in action selection, we created a probabilistic switching task in which mice chose to enter a reward port located to their left or right side. To inform their choices in the task, mice relied on recent reward history to assess whether water will be delivered from one of two reward ports. Critically, we selected a left versus right choice design based upon previous studies showing that 1) unilateral striatal manipulations can affect lateralized body movements[11, 21], 2) neurons within the striatum encode actions for movements contralateral or ipsilateral to the recording site[22-24], and 3) brainstem motor programs mediating orienting and approach behaviors[25] are regulated by the basal ganglia[16, 25–27]. From these studies, we reasoned that unilateral manipulations to each striatal hemisphere could affect the selection of spatially lateralized responses in the context of this task. We developed a computational model that assigns a value for each action given a particular history of rewards to predict the distribution of left versus right choices. We then used optogenetic techniques to examine the impact of transient unilateral stimulation of the D1R and D2R-expressing striatal neurons during an epoch in the task when animals were choosing to approach a left or a right port.

Here, we show that transient activation of D1R-expressing striatal neurons biased choices toward the port contralateral to the side of stimulation and transient activation of D2R-expressing striatal neurons biased choices toward the ipsilateral port. However, rather than giving rise to a stereotyped and consistent motor response, optical stimulation induced a bias in the likelihood of choice responses that was dependent on the previous history of rewards for each choice. In the context of our model, we show that optical stimulation mimics a fixed additive shift in action value[28, 29]. The effect of stimulation was only effective in a narrow temporal window prior to and during the early initiation of movements. Striatal activation also sped and slowed the initiation of responses in a manner similar to changes in action value. Together, these data are consistent with the hypothesis that the striatum encodes the value of actions and demonstrates that activity within these pathways can be used to bias the selection and vigor of competing goal-directed actions.

Results

To study action selection during decision making in mice we trained adult BAC transgenic mice (n=28) expressing Cre recombinase under either dopamine D1R or D2R regulatory elements on a spatial two-alternative forced-choice probabilistic switching task. In brief, the task required animals to initiate a trial with a nose poke into a central port followed by movement to a left or a right port to obtain reward. The rewarded port delivered a water reward for 75% of correct responses, and this port was periodically switched across blocks. The length of each block was randomly distributed between 7–23 trials and the switch only took place after a rewarded trial (Fig. 1a). A `Go' cue signaled when mice could approach either choice port. Thus, the only information provided to guide the animals' choice behavior was the expectation of reward based upon the outcome of previous trials. After initial training, mice took on average 2.20±0.04 (right to left) and 2.22±0.05 (left to right) trials (± s.e.m., n=28) to switch their behavioral responses following reversals in action-outcome contingencies between blocks (Fig. 1b). When we analyzed the effects of reward history in the previous two trials on upcoming choice, we found that mice implemented a `win-stay, lose-shift' strategy in which rewards served as evidence to continue responding at a port and the lack of reward served as evidence to switch (Fig. 1c). Animals' choice probability generally tracked the reward probability at each port for various reward histories (Supplementary Fig. 1a).

Figure 1

Design of the probabilistic switching task and animal performance

(a) Sequence of events in a probabilistic switching task. Mice learned to initiate a trial at the center port and to choose a left or right peripheral port for water reinforcement. Only one peripheral port was rewarded at a time. In 25% of trials, neither port was rewarded. The rewarded port switched only following a rewarded trial. (b) Fraction of choices for left port (n=28 subjects) for trials before and after a switch of the rewarded port (at trial 0). (c) Fraction of choices for the left port from one subject for reward histories in which two consecutive choices to either the left or the right port were made during the previous two trials. Data from mixed choice histories are not shown for brevity. All error bars represent s.e.m.

Action value estimates based upon previous reward history

Next, we generated a quantitative model to describe the animals' behavior in the task. Prior electrophysiological studies have found that the activity of striatal neurons correlates with estimates of trial-to-trial values of actions[3, 4, 30]. These estimates of value assume the softmax decision rule[31], which describes the tendency of decision-makers to have more variable responses when the alternatives are more similar in value. In the case of two alternatives under the softmax rule, the contribution of previous reward history to the value of each action can be estimated by multivariate logistic regression[30]. We fit a regression model where the probability of choices at each port in the upcoming trial was determined by the animals' previous reward history (see methods).

The regression analysis demonstrated that the contribution of prior rewards declined with the passage of trials (Fig. 2a). Rewards within the previous three trials had a significant effect on choices in the upcoming trial, serving as evidence for the animal to stay at the rewarded port as indicated by the positive regression coefficient (Fig. 2a) while the lack of reward at the chosen port in the previous trial significantly promoted switching in the following trial as represented by the negative regression coefficient (Fig. 2a). The large regression coefficients for the previous two trials validated our initial attempts to analyze choice behavior by segregating trials based upon the animals' reward history in the previous two trials (Fig. 1c). Based upon the model, we generated dynamic estimates of action-values and probabilities for responding at each port based on the animals' recent reward history (Fig. 2a,b,e). These action value estimates were the sum of regression coefficients corresponding to the previous reward history for each side (Fig. 2a) determining the distribution of choices of the subject (Fig. 2b). The choice probabilities predicted by an identically generated regression model using 70% of the data recapitulated the actual distribution of choices in the remaining 30% of the data, confirming the predictive validity of our model (Supplementary Fig. 1e,f). .

Figure 2

Modeling action value and animals' choice

(a) Contributions of intrinsic bias (brown), rewarded (cyan) and unrewarded (magenta) outcomes in the previous five trials on choices in the current trial derived from logistic regression (n=28 subjects, 22726±2881 trials per subject). (b) The fraction of choices for the left port given the relative action value. Data from each subject were grouped in 10 bins and represented in a distinct color. (c) The fraction of choices for the left port given the fraction of left choices predicted by the regression model. (d) Average median withdrawal time measured from go-light signal to nose-withdrawal from center port (n=28). Withdrawal time is shorter when the relative action value for either port is higher. (e) Example data from fourteen trial blocks. In the top panel, right reward blocks are represented in green and left reward blocks in orange. The dotted line indicates the subject's probability of choosing the left port averaged across four trials while the black line indicates predicted probability of choice based upon action value estimates (bottom panel). Long ticks correspond with rewarded trials while short tics represent unrewarded trials. All error bars represent s.e.m.

Stimulation of distinct striatal neurons biases choice

To independently study the activity of D1R-expressing and D2R-expressing striatal neurons in our task, we bilaterally injected an adeno-associated virus (AAV) into the dorsal striatum that enabled Cre-dependent viral expression of channelrhodopsin (ChR2) and yellow fluorescent protein (eYFP) in transgenic mice expressing Cre under regulatory elements for either the dopamine D1R or D2R. We then chronically implanted an optic fiber in or just above the dorsal medial striatum of each hemisphere. Our Cre-dependent strategy targeted ChR2-expression to the direct or indirect pathways of the basal ganglia (Supplementary Fig. 2). In D2-Cre mice, 37.7% of putative MSNs expressed eYFP (consistent with indirect pathway targeting) and additionally 38.7% of the choline acetyltransferase (ChAT)-expressing neurons co-expressed ChR2-eYFP (Supplementary Fig. 3). Estimates suggest 80–97.7% of neurons in the striatum are MSNs and 0.3–2% are cholinergic[32, 33]. We therefore estimate in D2-Cre mice we transduced a ratio of MSNs to cholinergic neurons of roughly 50:1. By simultaneously recording and optically stimulating striatal neurons, we were able to confirm optically driven striatal neuronal firing in our system was time-locked to stimulation (Supplementary Fig. 4). Optically evoked activity could be due to direct ChR2 activation or potentially indirect activation of neurons downstream of ChR2-expressing cells.

We then sought to determine whether activation of specific neural populations in the striatum could affect the choice behavior of animals[34]. Optical stimulation was delivered at a decision point within the task when the `Go' cue signals the animal to make their choice (Fig. 3a). At this point in the task, the choice ports are equidistant in egocentric space to the left and right of the mouse, and this period marks the time when animals must select and initiate an action to acquire a potential reward (Fig. 1a). Stimulation was delivered at 5 Hz, 10 Hz, or 20 Hz for 500 ms in 6% of total trials interspersed at random. The stimulation parameters used here reflect physiologically relevant activity found in awake, freely moving striatal recordings in mice[23, 35]. The presence of stimulation and non-stimulation trials in the same session allowed us to make highly controlled within subject comparisons to determine the effect of stimulation. Stimulation sessions were interspersed with training sessions without stimulation, and the hemisphere that was stimulated alternated with every stimulation session. There were no differences in animals' choice behavior across sessions with identical stimulation parameters, therefore, data from these sessions were pooled for analysis. To rule out any possible effect of the optical stimulation or viral expression, we also injected animals with a control virus expressing Cre-dependent eYFP. These control animals were subject to identical training regimens and stimulation parameters as experimental animals.

Figure 3

Optical stimulation induces opposing biases in an animal's choice

(a) Timing of optical stimulation within the task. In 6% of trials, optical stimulation was delivered to the dorsal striatum during a 500 ms period starting at the same time as the `Go' light cues. Stimulation occurred at 5, 10 or 20 Hz delivering respectively 3, 5 or 10 pulses of 5 ms light stimulation. (b,c) Examples showing the effect of 10 Hz stimulation in the left dorsomedial striatum (DMS) of a D1-Cre animal (b) and a D2-Cre animal (c) expressing ChR2-eYFP. Individual bars represent fraction of left choices for various reward histories in trials when the animal previously made two consecutive responses at the same port. Red bars indicate stimulation trials and blue bars represent trials without stimulation. (d,e) Fraction of left choices with and without stimulation for all possible combinations of choices and outcomes in the previous two trials with more than five total occurrences. (d) Data from the same D1-Cre animal shown in (b). The frequency of trials with a given reward history are indicated by the relative size of the circle. Filled circles represent a significant change in fraction of left choice with stimulation (p<0.05, Fisher's exact test). The red curve relates the probabilities of choice with and without stimulation for a fixed odds ratio (odds ratio=e−1.33±0.20). (e) Data from the same D2-Cre animal shown in (c) (odds ratio=e1.45± 0.18). All error bars represent s.e.m.

Given that the animals' responses were dependent upon their reward history, we analyzed choice in stimulation and non-stimulation trials based on the history of reward and choice over the previous two trials. Activation of D1R-expressing neurons with optical stimulation induced a bias in choice for the port contralateral to the hemisphere in which the light was delivered. The stimulation-induced bias was greater after unrewarded trials when the animals' responses tended to be more variable (Fig. 3b). A small or insignificant bias was seen following rewarded trials (Fig. 3b).

Striatal stimulation in D2-Cre mice induced a bias to the port ipsilateral to the site of stimulation (Fig. 3c), opposite to the direction observed in D1-Cre mice. Again stimulation induced bias was greater after unrewarded trials when the animals' responses were more variable (Fig. 3c).

To further investigate the relationship between stimulation and reward history, we plotted the probability of a left choice for a given reward history in trials with and without stimulation (Fig. 3d,e). The plot revealed `bowing' of data points off the unity line in opposite directions for D1- or D2-Cre animals. This bowing can be quantified as the odds of choice without stimulation scaled by a fixed factor called the “odds ratio”.

Optical stimulation mimics a change in action value

In mice performing the switching task without stimulation, recent reward history exerted a strong effect on an animal's upcoming choice (Fig. 1). We quantified these effects by generating estimates of the value of actions (Fig. 2b), which could be used to guide the animals' selection of future actions[3, 4, 30]. Similar to rewards, striatal stimulation also had the ability to bias animals' choices. Given that neurons in striatum may encode the value of actions[3, 4], we hypothesized that striatal stimulation may mimic the effects that a change in valuation of actions (based on reward history) would have on the animals' upcoming choice.

Using our previous estimates of the relative action value for various reward and choice histories (Fig.2b), we plotted the probability of choices made with and without stimulation for individual subjects in the same sessions (Figs. 4 and 5, Supplementary Figs. 5 and 6). This analysis revealed that striatal stimulation shifted the sigmoid choice probability curve along the relative action value axis. In this way, stimulation can be interpreted as mimicking a change in the relative valuation for selecting the left versus right port by a fixed factor. In individual animals, optical stimulation of D1R-expressing striatal neurons appeared to increase the value of the contralateral port (Fig. 4a) while stimulation of the D2R-expressing neurons resembled a decrease in the value for the contralateral port (Fig. 5a). In both individual and group data, the magnitude of the shift in relative action value consistently increased with the frequency of stimulation and the direction of the shift induced by stimulation showed a consistent relationship with the stimulated hemisphere (Figs. 4b, 5b and 6a; individual data can be seen in Supplementary Figs. 5 and 6). Importantly, control animals did not demonstrate a bias in their distribution of choices between trials with or without optical stimulation. In either D1- or D2-Cre control animals expressing only eYFP without ChR2, no shift in action value was observed (Fig. 4c and 5c).

Figure 4

Dorsal striatal D1R-expressing neuron activation mimics an increase in relative action value for contralateral choice

Fraction of choices for the left port on trials with different relative action value estimates in D1-Cre animals in the presence (red) or absence (blue) of optical stimulation. (a) Representative data from one subject transduced with AAV-EF1α-DIO-ChR2-eYFP with optical stimulation in the right hemisphere. Logistic regression was used to fit the data from trials with (red line) and without stimulation (blue line). A leftward shift in the logistic curve represents a bias for the left reward port. (b) Summary data for probability of choice for the left port and relative action value pooled from all D1-Cre subjects expressing ChR2-eYFP and stimulated on the right hemisphere (top) or left hemisphere (bottom). A logistic regression fit for the no stimulation (thick blue line) and stimulation (thick red line) trial data was produced for data pooled from all subjects. Curves representing the estimated probability of choice for given relative action values in individual animals are plotted in light blue (no stimulation) or light red (with stimulation). Reported n refers to number of stimulation sites (1 per hemisphere). (c) Summary data from all D1-Cre subjects expressing eYFP only and stimulated on the right hemisphere (top) or left hemisphere (bottom). P values reported for t-tests: H0:ßstim=0 (distance between thick red and blue lines). All error bars represent s.e.m.

Figure 5

Dorsal striatal D2R-expressing neuron activation mimics a decrease in relative action value for contralateral choice

Fraction of choices for the left port on trials with different relative action value estimates in D2-Cre animals in the presence (red) or absence (blue) of optical stimulation. (a) Representative data from one subject transduced with AAV-EF1α-DIO-ChR2-eYFP with optical stimulation in the right hemisphere. Logistic regression was used to fit the data from trials with (red line) and without stimulation (blue line). A rightward shift in the logistic curve represents a bias for the right reward port. (b) Summary data for probability of choice for the left port and relative action value pooled from all D2-Cre subjects expressing ChR2-eYFP and stimulated on the right hemisphere (top) or left hemisphere (bottom). A logistic regression fit for the no stimulation (thick blue line) and stimulation (thick red line) trial data was produced for data pooled from all subjects. Curves representing the estimated probability of choice for given relative action values in individual animals are plotted in light blue (no stimulation) or light red (with stimulation). Reported n refers to number of stimulation sites (1 per hemisphere). (c) Summary data from all D2-Cre subjects expressing eYFP only and stimulated on the right hemisphere (top) or left hemisphere (bottom). P values reported for t-tests: H0:ßstim=0 (distance between thick red and blue lines). All error bars represent s.e.m.

Figure 6

Comparison of D1R- and D2R-expressing neuron activation across stimulation parameters

(a) Estimated change in the relative action value for choosing the port ipsilateral versus contralateral to the site of stimulation averaged across individuals within a group. Positive changes in relative action values correspond with an ipsilateral bias while negative changes correspond to a contralateral bias. Estimates of relative action change were derived from logistic regression analysis (see methods) for 5, 10, and 20 Hz stimulation sessions. Reported n refers to number of stimulation sites. The D1-Cre data set consisted of 6 stimulated animals expressing ChR2-eYFP and 4 control animals expressing only eYFP. The D2-Cre data set consisted of 8 stimulated animals expressing ChR2-eYFP and 5 control animals expressing only eYFP. (b) The median time taken to withdraw from the center port averaged across individual D1-Cre subjects (top) and D2-Cre subjects (bottom) expressing ChR2-eYFP in trials without stimulation (blue) or with stimulation (red) and across different relative action values for choosing the port ipsilateral versus contralateral to the site of stimulation. Positive relative action values correspond to trials in which the value of the port ipsilateral to the site of stimulation is greater than the contralateral port. Median times to withdraw are plotted for 5 Hz (left), 10 Hz (middle), and 20 Hz (right) stimulation sessions. All error bars represent s.e.m. *: p <0.05; **: p <0.01 for Wilcoxon signed-rank test. ***: p<0.001, Wilcoxon rank-sum test.

Impact of stimulation on movement initiation time

In addition to its role in action selection, activity in the striatum correlates with reaction time[1, 36]. In our task, the time a mouse takes to withdraw from the center port after trial initiation can be used to measure the latency to initiate movement. Without optical stimulation, the withdrawal time for a particular response was significantly shorter when the relative action value for that response was high (Fig. 2d). Stimulation of D1R-expresing neurons decreased center port withdrawal time when the value of the contralateral port was greater (Fig. 6b). However, when the ipsilateral port was valued more highly, stimulation of D1R-expressing neurons slowed withdrawal time (Fig. 6b). In contrast to D1-Cre mice, striatal optogenetic stimulation in the D2-Cre mice increased center port withdrawal time when the value of the contralateral port was greater (Fig. 6b). However, when the ipsilateral port was valued more highly, stimulation of D2-expressing neurons sped withdrawal times (Fig. 6b). Changes in withdrawal time with optical stimulation were still apparent when we controlled for the eventual choice response of the animals (Supplementary Fig. 7a,b). Importantly, we find that after rewarded trials when optical stimulation is not likely to affect port choice, stimulation can still be observed to affect withdrawal time.

Effective time window for optical stimulation

Previous reports have found that phasic striatal activity prior to response initiation correlated with trial-by-trial estimates of action value[3, 4]. In the experiments described above, light pulses were delivered during a 500 ms epoch coinciding with onset of a `Go' cue after which animals initiated their motor responses (Fig. 3a).

We next performed two further experiments to investigate the impact of striatal activity prior to and after response initiation. Response initiation in our task can be measured as the latency to withdraw from the center port. The median withdrawal times for animals in this study were approximately 50 ms, and the majority of response withdrawal times can be found within a window spanning approximately 30 to 140 ms (50±25% confidence interval; Supplementary Fig. 1c).

To determine whether stimulation prior to movement initiation was sufficient to bias the animals' choice behavior, a new cohort of mice were trained in a protocol in which two optical pulses separated by 50 ms (20 Hz) were delivered prior to an auditory `Go' cue (Fig. 7a). Using this second protocol, we confirmed that it is possible to bias the animals' choice behavior with striatal activation of both D1R-expressing and D2R-expressing striatal neurons prior to movement initiation (Fig. 7b–d and Supplementary Fig. 8a,b).

Figure 7

Significant bias was induced by stimulation limited to a 50 ms period before a `Go' cue

(a) The timing of events in sessions where a `Go' cue sound was delayed 70–110 ms after initiation of trial with center port entry. In these trials, two 5 ms optical pulses separated by 50 ms were delivered just prior and at the onset of the `Go cue' and before withdrawal from the center port. (b, c) Summary data for probability of choice for the ipsilateral port on trials with different relative action value estimates pooled from (b) D1-Cre and (c) D2-Cre subjects expressing ChR2-eYFP with confirmed eYFP expression sites. Positive relative action values correspond to trials in which the value of the port ipsilateral to the site of stimulation is greater than the contralateral port. A logistic regression fit for the no stimulation (thick blue line) and stimulation (thick red line) trial data was produced for data pooled from all subjects. Curves representing the estimated probability of ipsilateral choice for given relative action values in individual animals are plotted in light blue (no stimulation) or light red (with stimulation). P values reported for t-tests: H0:ßstim=0 (distance between thick red and blue lines). (d) Change in estimated relative action value for choosing the port ipsilateral versus contralateral to the site of stimulation averaged across individuals stimulated with two optical pulses prior to movement initiation. All error bars represent s.e.m.

In a third protocol, we examined the effect of delaying stimulation until after movement initiation. In a subset of mice trained in the original protocol, 10 Hz optical stimulation was delivered at either 0 ms or 150 ms following the presentation of the `Go' cue light (Fig. 8a). The bias induced by optical stimulation delayed by 150 ms was significantly weaker than the bias induced without delay (Fig. 8b–d). These data suggest that the impact of striatal activity on choice behavior decayed with time after the onset of the `Go' cue.

Figure 8

Bias induced by stimulation diminished when stimulation was delayed 150 ms after a `Go' cue

(a) The timing of events in sessions where stimulation occurred at latencies of either 0 ms or 150 ms relative to `Go light' cue onset. (b,c) Summary data for probability of choice for the ipsilateral port and relative action value pooled from (b) D1-Cre and (c) D2-Cre subjects expressing ChR2-eYFP with confirmed eYFP expression sites stimulated with 0 ms latency (left) or 150 ms latency (right) relative to `Go light' onset. A logistic regression fit for the no stimulation (thick blue line) and stimulation (thick red line) trial data was produced for data pooled from D1-Cre or D2-Cre subjects. Curves representing the estimated probability of choice for given relative action values in individual animals are plotted in light blue (no stimulation) or light red (with stimulation). P values reported for t-tests: H0:ßstim=0 (distance between thick red and blue lines). (d) Estimated change in relative action value for choosing the port ipsilateral versus contralateral to the side of stimulation averaged across individuals stimulated at 10 Hz with 0 ms and 150 ms latency within the same session. Estimates of relative action change were derived from logistic regression analysis (see methods). Reported n refers to number of stimulation sites. **: p<0.005, Wilcoxon signed-rank test. All error bars represent s.e.m.

Transient and prolonged stimulation outside of the task

Following the switching task, animals also underwent two other experiments to grossly measure the impact of transient and prolonged stimulation on the animals' locomotor behavior outside the context of the task. In a 10 cm diameter cylindrical environment, transient bursts of 20 Hz stimulation (identical to the maximum burst used in our switching task) did not produce a significant change in head or body orientation (Supplementary Fig. 9c,d). However, prolonged stimulation of the D1-expressing striatal neurons for 60 seconds at 5 Hz, 10 Hz, and 20 Hz induced a graded increase in contralateral rotations and prolonged stimulation of D2R-expressing neurons induced a decrease in contralateral rotations in an open field (Supplementary Fig. 9e,f). Together, these data are consistent with previous reports[11] and the known organization of distinct striatal pathways in the basal ganglia[8, 10, 13].

Discussion

Our data suggest that activity in distinct populations of striatal neurons exert opposing biases on the selection of goal-directed responses. Activation of D1R-expressing striatal neurons increased the occurrence of choices for the port contralateral to the side of stimulation (Figs. 3, 4 and 6) while stimulation of D2R-expressing striatal neurons increased the occurrence of ipsilateral choices (Figs. 3, 5 and 6). The effect of stimulating each population of neurons was not deterministic, but dependent on the animals recent reward history (Fig. 3). Upon closer inspection, the magnitude of this bias mimicked an additive change in the relative value of actions estimated using a simple model based upon the animals history of rewards and choices (Figs. 2b and 4–6). Stimulation also altered the latency to movement initiation as measured by the withdrawal time in manner that was dependent on the relative action value (Fig. 6b).

Qualitatively, the effects of stimulating D1R and D2R-expressing neurons match existing accounts of the opposing functions of the direct and indirect pathway within the basal ganglia. This lateralized effect may be due to the presence of dense ipsilateral descending connectivity from basal ganglia nuclei and the role that downstream efferent structures play in controlling contralateral movement[25-27]. The ability of D2R-expressing neurons to promote choices to the ipsilateral port is consistent with a larger literature suggesting that action selection involves inhibition of competing alternatives through the indirect pathway in a manner that allows or even facilitates focal promotion of desired actions by the direct pathway[37]. The proposed occurrence of targeted inhibition from basal ganglia pallidal outputs has been suggested to coordinate agonist and antagonist musculature involved in limb and visuomotor movements in primates[38,39]. Competition between the opposing actions of orienting to the left and right may also be regulated within the downstream targets of the basal ganglia. The direction of the bias observed in our study is largely consistent with previous reports in which D1R and D2R-expressing neurons are selectively activated using optogenetic techniques[11] as well as the effect of pharmacological manipulations to the dorsal striatum in rodents[21].

Optogenetic stimulation did not induce a uniform effect across all trials, but was dependent on the animals' previous reward history. Stimulation induced a larger bias on choice when animals had greater variability in their responses following unrewarded trials (Figs. 3–5). However, striatal stimulation had a weak or insignificant effect on choice behavior after recently rewarded trials when animals were likely to return to a port where water had just been delivered. Thus, the effect of stimulation could be overruled if the alternative response had a high incentive value after being recently rewarded.

The effect of optical stimulation was also time dependent. Stimulation was effective when limited to two 5 ms pulses within an epoch of the task prior to the animals' movement initiation (Fig. 7), and it became significantly weaker if delayed by 150 ms after trial initiation (Fig. 8). Together, these data suggest that striatal activation may need to take place in a “decision window” to alter the action selection of the animal. This is consistent with a number of recording studies in the striatum of rodents[22, 24] and in primates[3, 4, 36]. Finally, outside of the task, transient stimulation at the maximal experimental level did not induce head or body orientation (Supplementary Fig. 9). The interaction of stimulation with reward history, the presence of an effective decision window, and the lack of similar motor output outside of the task, demonstrate that striatal stimulation did not dictate a motor action deterministically impacting choice. We conclude striatal activity alone is not sufficient to drive the motor responses of animals in this task, but instead may also require the temporally coincident activity of other neural structures to orchestrate a complex process of action selection.

Given that the magnitude of the stimulation bias was dependent on whether previous actions had been rewarded, we hypothesized that the striatal activation may act similarly to the influence of rewards over choice. Within the context of this task, animals are required to adaptively and flexibly switch their actions across blocks due to changing reward contingencies. At a theoretical level, this process of goal-directed action selection can be modeled as a dynamic comparison between the value of actions and bias for selection of the option of highest value[40]. More specifically, we conjectured that the additional activity induced by optical stimulation mimicked the effects resulting from a change in the value of competing actions.

To demonstrate this directly, we estimated the value of actions based upon various reward and choice histories assuming the softmax decision rule[4, 31, 41]. We then re-examined quantitative features of the bias introduced by striatal activation and found that stimulation mimicked a fixed additive shift in the value of the contralateral choice (Figs 4–6 and Supplementary Figs. 5, 6 and 8) without altering sensitivity to reward (Supplementary Fig. 10). In this way, activation of D1R-expressing neurons mimics an increase in the value of the contralateral choice while activation of D2R-expressing neurons mimics a decrease of value of the contralateral choice. The nonlinear features of the softmax rule were sufficient to explain the gross tendency for stimulation to have a larger effect in a range where responses are most variable (Figs. 3–5).

While many descriptions of striatal function focus on its role in action selection, others have suggested that the basal ganglia merely regulates the “vigor of responses” without altering which response is selected[19, 20]. Here, we show that in addition to biasing the animals' action selection, striatal activity can also alter the “vigor of responses” by speeding up or slowing down the initiation of movements. We found that stimulation of D1R-expressing neurons reduced movement latencies in trials when the value of the contralateral port was greater and congruent with the direction of bias caused by stimulation (Fig. 6b). This is consistent with the canonical view that the direct pathway promotes movements[8, 10]. However, latencies following stimulation of D1R-expressing neurons were dramatically slower in trials when the ipsilateral port was valued more highly. The slowed response is perhaps due to the incongruence between bias caused by stimulation and intrinsic valuation in the action selection systems. Similarly, stimulation of D2R-expresing neurons slowed movements when the port contralateral to the site of stimulation was of greater value and it sped movements when the port ipsilateral to the stimulation site was of greater value. This data is consistent with the idea that action selection can be facilitated by the suppression of alternate reponses[38]. Our data suggest the balance of activity within striatal populations reflecting the relative value of approaching each port impacts the speed of movement initiation. These data are consistent with evidence that striatal activity correlates with reaction times[1, 36]

Our data is consistent with electrophysiological studies suggesting neural activity in the striatum more often represents the value of actions than pure motor variables[3, 4]. Striatal activity has been associated with response bias for rewarded actions[1] and successful switching following action contingency reversals[36, 42]. In these studies, neurons primarily encode a bias for contralateral responses when decisions are reported as saccades or locomotor approach[1, 22, 36]. Lesions of the dorsomedial striatum can also impair the learning of the contingencies between actions and their outcomes in various reversal tasks[7, 43, 44]. Taken together, these studies are consistent with our interpretation that striatal stimulation biases both the selection and vigor of actions based upon the value of choices.

We have attempted to best reproduce physiological conditions using optogenetics. The stimulation patterns used in our experiments parallel data from awake in vivo striatal recordings in mice[23, 35]. Other labs have also identified striatal activity in a similar decision point within a reward-based spatial task in rodents[24]. The bias induced by stimulation scaled over a range of frequency parameters, supporting the robustness of our results. However, questions still remain regarding whether optogenetic stimulation mimics physiological patterns in vivo given the large number of neurons that are synchronously recruited. In the D2-Cre mice, we also infect a small proportion of cholinergic neurons, which may contribute to our behavioral effects. An additional feature and caveat of our study is that our optical stimulation was delivered unilaterally in the context of a task where competing responses are lateralized to the left and right. This is both feature and caveat because we predict our method will likely not cause behavioral bias in alternative task designs where responses take other forms, such as up vs. down. Lastly, our results do not exclude the possibility that D1R-expressing and D2R-expressing neurons may serve other roles outside of action selection. Numerous labs have found evidence that striatal activity also correlates with a process that evaluates the outcomes of actions in rodent[24, 45] and in primates[4], and our data do not exclude these aspects of striatal function.

Striatal neurons receive massively convergent input from cortical and thalamic sources. This integration of diverse information may be used to generate a representation of action value that can be utilized to mediate goal-oriented action selection. In this framework, the updating of action values may correspond with dopamine-dependent plasticity of inputs into the striatum in concert with shifts in goal and reward-related activity from the cortex[41]. Striatal neurons may alter action selection by regulating tonic inhibition from the globus pallidus and the substantia nigra pars reticulata onto the brainstem and motor thalamus[46, 47]. This provides a mechanism by which rewards modulate the responses of premotor structures for particular actions[46, 48]. In this way, the cortico-basal ganglia system may instantiate the computations necessary for goal-oriented, highly flexible behavior[49, 50].

Materials and Methods

Animals

C57BL/6J BAC transgenic mice expressing Cre recombinase under the regulatory elements for the D1 and D2 receptor (D1-Cre and D2-Cre ER43) were obtained from Mutant Mouse Regional Resource Centers (MMRRC) and bred in our colony. All animals used in this study were adults (25–30g) and group housed under a reverse 12 h light/dark cycle (light onset at 10:00 A.M.) until surgery. Mice were given food and water ad libitum prior to water deprivation in preparation for training. All procedures were approved by the Ernest Gallo Clinic and Research Center Animal Care and Use Committee.

Construct and virus preparation

Plasmids encoding the DNA sequences for pAAV-EF1α -DIO-ChR2(H124R)-eYFP or pAAV-EF1α-DIO-eYFP were obtained from the laboratory of Karl Deisseroth. Amplification and purification of plasmids was performed using a standard plasmid maxiprep kit (Qiagen) and confirmed by sequencing. EF1α-DIO-CHR2(H124R)-eYFP and EF1α-DIO-eYFP cassettes were packaged in AAV vectors and serotyped with AAV5 coat proteins by the viral core at University of North Carolina. The final concentration was 1–2×1012 viral particles/ml.

Implantable chronic optical fibers and optic cables construction

Optical stimulators were constructed according to the published protocols[51]. Briefly, optical fibers were constructed by attaching a 200 μm, 0.37 NA optical fiber (Thor Labs) with epoxy resin into a metal ferrule that had previously been cut and scored. Fiber-ferrule units were then cut and polished. Only implants with efficiency greater than 70% and comparable efficiencies (± 10%) were used in the study. Optical-patch cables were constructed from 62.5 μm core diameter optic fiber (Thor labs) that were connected to a ferrule on one end and an FCPC connector on the other end. Cables were covered in furcation tubing to protect the fiber and to prevent light from escaping through the optic-patch cord. The ferrule at the end of the optic patch cord was fitted with a zirconium sleeve to interface with the chronic implant. The FCPC connector was coupled to a 473 nm DPSS laser (200 mW). The laser driver current was adjusted to yield 20mW output from the patch cable.

Stereotaxic AAV injection and optical implant surgery

Animals were anesthetized with either 150mg/kg ketamine and 50mg/kg xylazine or 2% isofluorane (vol/vol) gas anesthesia. Animals were placed in a stereotaxic frame and 26-gauge microinjection needles were inserted through a burr hole bilaterally into the dorsomedial striatum (coordinates from bregma: anterior 0.75, medial-lateral ±1.5, ventral −3.0 mm) of D1/D2-Cre mice to deliver 1.0 μL of either AAV-EF1-DIO-ChR2(H124R)-eYFP or AAV-EF1-DIO-eYFP ~1012 IU/mL). Injections were performed using a 1 μL Hamilton syringe through a hydraulic pump (Harvard Instruments) and took place over 10 min followed by 10 min of recovery. The length of the optic fiber protruding from the implant was cut to be 2 mm. The tip of the fiber optic from the implant was then inserted through the same burr hole as was used previously for the virus injection and was lowered 2 mm ventral to the dura. The implant was cemented to the skull using dental cement. Mice were monitored until recovery from surgery and then returned to their home cage where they were housed individually.

Anesthetized in vivo optrode recording

Recordings under conditions of anesthesia were made with a custom silicon probe (model A1×16–5mm50–413, NeuroNexus Technologies) that was attached with a 200 μm diameter core optic fiber connected to a FCPC connector with epoxy resin. One month following injection of AAV-EF1-DIO-ChR2(H124R)-eYFP into the dorsomedial striatum, animals were placed under ketamine and xylazine anesthesia (150 mg/kg ketamine and 50 mg/kg xylazine i.p.). A craniotomy was performed above the injection site and the optrode was lowered. Data was acquired using commercial systems (Plexon). Optical stimulation was delivered during recording as continuous trains of stimulation to assess the fidelity of optically induced firing as well as using the same stimulation protocols that we used during our behavioral paradigm. After each recording, the probe was moved to a new recording tract within the same animal. Following recordings, animals were euthanized for assessment of track location and viral expression. Single units were identified with principal component analysis (Offline Sorter, Plexon). The data were then imported into Matlab (MathWorks) for subsequent analysis.

Probabilistic switching (two-alternative spatial choice) task

Mice were trained on a two-alternative spatial choice task in which the location of a water reward was periodically switched at random intervals. The initiation port was located in the middle of one wall, and two choice ports were located 63.5 mm left and right of the initiation port (center-to-center; Fig. 1a). An infrared photodiode/phototransistor pair was placed on either side of the port to report the times of port entry and exit (Island Motion, Tappan, NY). The water valves (Neptune Research) were calibrated to deliver a volume of water (2 μl) for rewarded choices.

Mice initiated each trial by entering the center port, triggering `Go' lights instructing animals that water was potentially available. Mice then chose a left or right peripheral port for water reinforcement (Fig. 1a). Only one peripheral port was rewarded at a time, on 25% of trials, neither port was rewarded. The length of trial blocks was dependent on the number of rewards obtained in each block, and this number of rewards was randomly set between 7–23 rewards. After the set amount of rewards was obtained, the rewarded side was switched to the opposite port. This structure prevented the subjects from predicting the timing of the block switch.

Logistic regression analysis of behavior choices

The contribution of past rewards or lack of rewards on the subject's current choice was analyzed on a trial-by-trial basis using the following logistic regression model[30]: where PL(i) is the probability of selecting the left port in the i-th trial. The variables YL(i) or YR(i) represent whether a reward is delivered (1 or 0) at the left or right port in the i-th trial, respectively. And NL(i) or NR(i) represent the lack of reward (1 or otherwise 0) at the chosen right or left port in the i-th trial, respectively. The variable n indicates the number of past trials that were included in the model (n=5). The regression coefficients ßjReward and ßjNo-Reward represent the contribution of past rewards and lack of rewards, respectively, and ß0 indicates the intrinsic bias of the animal.

We modeled the contribution of optical stimulation on the subject's current choice as a dummy variable characterized by the ßstim coefficient: The variables Xstimulation(i) represents whether stimulation is delivered (1) or not (0) in the i-th trial. These estimated coefficients are the shifts in action value characterized in figure 6.

Optical stimulation in the behavior task

Optical stimulation was delivered at the start of the `Go' signal. Stimulation was delivered at 5 Hz, 10 Hz and 20 Hz for 500 ms with the frequency pseudo-randomly chosen prior to each session. Stimulation trials occurred in 6% of total trials. Stimulation sessions were performed every other day, interspersed by training sessions. The hemisphere that was stimulated was alternated across stimulation sessions. The infrequent occurrence of stimulation trials was to prevent any plastic or compensating adaptations from occurring during the course of a session, and the relatively long interval of days between stimulation sessions was to prevent any systemic biases in responses from arising across stimulation sessions.

To determine whether stimulation had to occur within a specific time-window relative to movement initiation we performed additional experiments. First, a subset of mice that underwent stimulation at 5 Hz, 10 Hz, and 20 Hz underwent an additional set of stimulation sessions in which two types of stimulation trials were present, each for 3% of trials. In one set of stimulation trials, 10 Hz stimulation was delivered at the same time that the `Go' light appeared, 15 ms after initiation of a trial by a center poke (as in earlier experiments). In a second set of stimulation trials, 10 Hz stimulation was delayed by 150 ms from the appearance of the `Go' light. This 150 ms delayed stimulation time corresponds to a point when movement has been initiated in >90% of trials.

To look at the effects of stimulation exclusively before the onset of the `Go' cue, we also trained new cohort of mice (n=2 D1-Cre and n=4 D2-Cre) on a variant of the stimulation experiment in which two optical pulses (interpulse interval, IPI=50 ms) were delivered prior to and coincident with a `Go' sound which was delayed by 70–100 ms after initiation of a trial by a center poke (Fig. 7a). This was to confine stimulation to an epoch of the task prior to the initiation of the animal's movement as determined by the time of withdrawal from the center poke. Stimulation trials occurred in 6% of total trials. Stimulation sessions were performed every other day interspersed by training sessions.

Optical stimulation outside of the switching task context

To measure the impact of transient stimulation on the animals' locomotor behavior outside the context of the task, animals were placed in a 10 cm diameter cylindrical environment (Supplementary Fig. 9). Optical stimulation was delivered at 20 Hz for 500 ms and repeated every 60 seconds. Each subject received 50 stimulations per stimulation site. The body orientation at stimulation onset, and 0.5 and 1 sec before and after stimulation onset (5 time points) was measured from video recording frames by a blind observer using custom software.

To measure the impact of prolonged stimulation on the animals' locomotor behavior outside the context of the task, animals were placed in an open field arena. Baseline locomotion was measured over a period of 60 seconds. Optical stimulation was then delivered at 5, 10 or 20 Hz for 60 sec, and followed by 60 seconds of recovery period (Supplementary Fig. 9). The rotation of the animals were analyzed by Anymaze software (Stoelting Co) and summed for each 60 second period (baseline, stimulation and recovery).

Statistical analysis

Fisher's exact test was used to determine whether the probability of choices with and without stimulation were significantly different. Logistic regression was used to fit data for trials with different reward histories with and without stimulation. T-test was used to determine whether the change in relative action value caused by striatal activation (ßstim) was significantly different from zero for a given stimulation condition. Wilcoxon signed-rank and rank-sum tests were used to determine whether the changes in relative action value between different stimulation conditions or between groups of subject expressing ChR2-eYFP and eYFP were significantly different. Standard errors for probabilities of choice were calculated based upon binomial statistics. α was set at 0.05.

Histology and reconstruction of optical stimulation sites

Viral expression of ChR2-eYFP and eYFP was confirmed by histology after stimulation experiments (Supplementary Fig. 2a,b). Animals were perfused with saline and 4% paraformaldehyde (wt/vol), and brain tissue was fixed for subsequent coronal sectioning. These sections were then stained to identify cell bodies (Neurotrace) alongside AAV driven expression of YFP. In the cases reported, fiber implant tracks could be identified and were found to be located in or directly above the medial half of the dorsal striatum (Supplementary Fig. 2c).

To determine whether ChR2-eYFP was expressed in medium spiny neurons as well as cholinergic neurons of the striatum, sections (coronal, 50 μm) were permeabilized in 50% alcohol for 10 min, rinsed in PBS, blocked in 10% normal donkey serum (vol/vol) (NDS) for 30 min and incubated for 48 hrs in a mixture of primary antibodies: rabbit polyclonal anti-GFP (for YFP; 1:10,000, Abcam Inc., Cambridge, MA); goat polyclonal anti-ChAT (1:500; Millipore, Temecula, CA); and mouse monoclonal anti-Kv2.1 (1:200; UC Davis and NIMH NeuroMab, Davis, CA) at 4°C[52]. Sections were then rinsed in PBS and incubated in 2% NDS for 10 min, then incubated for 4–6 hours in mixture of secondary antibodies (all made in donkey): Alexa Fluor® 488 anti-rabbit (1:300; Life Technologies; Carlsbad, CA); CF™647 anti-goat and CF™555 anti-mouse (1:300; Biotium, Inc. Hayward, CA). Sections were rinsed in PBS, mounted and coverslipped with VECTASHIELD® mounting media (VECTOR LABORATORIES, INC. Burlingame, CA). Multi-channels images of ChAT-positive cells were acquired using Zeiss LSM 510 META laser confocal microscope (Zeiss, Thornwood, NY), using 63×/1.4 NA PlanApo objective, 488 543 and 633 nm excitation lines and factory recommended detector settings.