Corticoinsular circuits encode subjective value expectation and violation for effortful goal-directed behavior.

We are presented with choices each day about how to invest our effort to achieve our goals. Critically, these decisions must frequently be made under conditions of incomplete information, where either the effort required or possible reward to be gained is uncertain. Such choices therefore require the development of potential value estimates to guide effortful goal-directed behavior. To date, however, the neural mechanisms for this expectation process are unknown. Here, we used computational fMRI during an effort-based decision-making task where trial-wise information about effort costs and reward magnitudes was presented separately over time, thereby allowing us to model distinct effort/reward computations as choice-relevant information unfolded. We found that ventromedial prefrontal cortex (vmPFC) encoded expected subjective value. Further, activity in dorsal anterior cingulate (dACC) and anterior insula (aI) reflected both effort discounting as well as a subjective value prediction error signal derived from trial history. While prior studies have identified these regions as being involved in effort-based decision making, these data demonstrate their specific role in the formation and maintenance of subjective value estimates as relevant information becomes available.

Weighing the benefits of potential rewards against the effort required to achieve them underlies successful decision-making and foraging behavior (1–4). Such choices are often made without complete access to information regarding either the expected reward or required effort; one might accept an invitation to a party without knowing the travel required or choose to take a course without being certain how good the professor or material will be. Incomplete access to choice-relevant information presents a major challenge to the process of cost/benefit valuation. Indeed, while a number of prior studies have begun to outline the circuitry underlying the evaluation of cost/benefit options under conditions of complete information (4–6), the neural mechanisms of valuation under incomplete information remain unclear.

When effort and reward information are presented simultaneously, they have generally been found to engage a network composed of dorsal anterior cingulate (dACC) and anterior insula (aI). Both dACC and aI have been implicated in effort-based decision making and have been shown to encode effort costs as well as the subjective value of decision options. Lesions to the rodent homolog of dACC [particularly cingulate (cg) 1/2] have routinely been found to induce a shift in preference away from larger rewards requiring greater effort in favor of lower effort options (7–11), with similar effects observed from dACC lesions in primates (7, 12, 13). Electrophysiological recordings of single-cell activity have also found dACC to be one of the only prefrontal areas to be sensitive to effort costs (14–18). Similarly, in tasks requiring physical effort, aI has been shown to coactivate with dACC as effort costs increase, possibly suggesting a primary role for aI in encoding effort-related costs (19–22). Finally, both animal and neuroimaging studies have demonstrated that subjective values are represented in distinct brain areas, including the dACC and aI cortices for physical effort (12, 20).

Interestingly, while substantial evidence suggests that the ventromedial prefrontal cortex (vmPFC) is a central node for the computation of subjective value during decision making (3, 23), prior human and animal research has suggested that when it comes to decisions about effort expenditure, subjective value is not computed in vmPFC, but rather in dACC and aI (24–27). This pattern of results has frequently been interpreted as suggesting that the computation of subjective value depends on the category of response cost (e.g., effort vs. delay or risk) and that effort-related decisions are unique from other forms of value-based decision making in terms of their functional anatomy.

Importantly, however, the precise nature of the computations performed by this network of regions has not been adequately addressed. Moreover, the extent to which the cost/benefit decision making presented under conditions of complete information generalizes to conditions where all choice-relevant information may not be fully available is unclear. This question is particularly relevant, given the specific brain areas previously implicated in effortful goal-directed behavior. Both the dACC and aI are among the most commonly activated brain areas across cognitive and decision-making tasks, leading to numerous theoretical accounts regarding the presence of one or more “domain general” functions. In the case of dACC, these proposals have included error detection (28), prediction error-driven reinforcement learning (29–31), conflict monitoring (32), and value-based decision making (33–35). In the context of the latter, the dACC signal appears to be inversely related to the subjective value of a given option. This has been proposed to represent an increase in the “foraging value” (i.e., the value of continuing to search for other rewards) (36), provide a “boosting signal” to overcome increasing response costs associated with lower subjective values (37), or reflect the difficulty of determining the best choice between two nearly equivalent options (38–41). Similarly, the precise role of the aI in effort-related decisions remains unclear. Like dACC, the aI is associated with a variety of functions, including interoceptive awareness (42); encoding negative prediction errors (43), particularly interoceptive prediction errors (44–46); financial risk (47); and processing pain (48). Consequently, it remains unclear what specific computations, such as effort cost encoding or updating, these regions perform in the context of effort-based decisions.

As noted above, one challenge to evaluating the role of these regions’ computations during effortful goal-directed behavior has been the use of task paradigms that provide information about costs and reward simultaneously. To better isolate the role for these regions in computations of effort cost, subjective value, choice difficulty, and prediction error signaling, we used a sequential effort-based decision-making task where trial-wise information about effort costs and reward was presented separately through time. This temporal manipulation served to decouple reward and effort, allowing for the isolation of neural signals related to each, while also providing greater ecological validity. Further, this task provided a unique opportunity to measure neural responses to the receipt of favorable or unfavorable information across different time points, allowing for the modeling of expectations and expectation violations. Using this task, we found that vmPFC encoded expected subjective value before complete information was available. We also revealed a role for dACC in subjective value discounting and prediction error signaling across all modeled computations, with the exception of effort cost encoding. Interestingly, we observed the strongest role for aI as part of a network for subjective value prediction error (SVPE) signaling. These data help elucidate multiple distinct computations performed by vmPFC and dACC during effort-based decision making, as well as evidence for the recruitment of aI to aid in prediction and prediction error encoding.

Results

Behavioral Results.

In this task, participants performed choices between options with varying rewards and physical effort (rapid button pressing). Participants decided whether to choose a “No Effort Option” for $1.00 or an “Effort Option” requiring some level of physical effort in exchange for monetary rewards of varying magnitude. The Effort Option independently varied in the required button press rate (effort) and reward magnitude. The reward magnitude was shown as a dollar amount (range: $1–$5.73; based on four bins: $1.25–$2.39, $2.40–$3.49, $3.50–$4.60, and >$4.60), and the required effort level was indicated by the height of a vertical bar (20%, 50%, 80%, or 100% of the participant’s maximum button-pressing rate). To examine neural correlates of effort and reward information separately, information about the effort and reward available for the Effort Option for each trial was presented sequentially. “Effort First” trials began with an initial presentation of effort required for the Effort Option, followed by the available reward, while “Reward First” trials had the opposite presentation order. Each trial was therefore composed of an initial cue (“Cue 1”), followed by a second cue (“Cue 2”), which was then followed by a prompt to decide between the Effort Option and the No Effort Option (“Decision Prompt”), at which point subjects made a button press indicating their selection (“Choice Phase”) (Fig. 1).

Fig. 1.

(A) Schematic of experimental task design. This image shows the time line of a trial in which effort and reward information is presented sequentially. This is an example of an Effort First trial. Each trial began with the presentation of a fixation cross, followed by the Cue 1 phase in which one piece of information was presented (either effort level or reward magnitude). After 2–6 s, the second piece of information was presented (Cue 2). After an additional 2–6 s, participants saw a Decision Prompt that prompted them to make a choice between the Effort Option presented and the No Effort Option that always paid $1.00. They were required to make their selection within 3 s. Following their selection, their choice was presented to them during the Choice Phase. (B) Proportion of effortful choices based upon effort level and reward magnitude. Participants chose more effortful options as reward increased and as effort decreased. Error bars are all SEM. (C) Individual and group average (dark blue) subjective value curves based on the results of our computational model. The group average is shown as the dark blue line with shading around it that represents the SE. The remaining colored lines each reflect a single participant, demonstrating individual differences in discounting.

We first tested whether both the size of the reward and the required effort of each choice option had an impact on participants’ choice behavior. A 4 (effort level) × 4 (reward magnitude, binned) repeated-measures ANOVA revealed that participants’ choices were strongly guided by both the required effort [F(1.45,39.10) = 64.27, P = 1.74 × 10−11, partial η2 = 0.70; Fig. 1] as well as the reward magnitude of the Effort Option [F(2.02,54.47) = 106.03, P = 1.42 × 10−19, partial η2 = 0.80; Fig. 1]. There was also an effort × reward interaction [F(4.40,118.76) = 8.88, P = 0.000001, partial η2 = 0.25]. As expected, larger rewards and smaller effort costs attracted more effortful choices. Overall, participants chose the higher effort option on 66% ± 18% of trials.

Computational model.

To better estimate how effort and reward influenced individuals’ choices, we used a two-parameter effort discounting model that had been previously shown to fit effort-based choices (49) (details are provided in ). Consistent with prior results, this model showed a superior fit (determined by both Akaike’s Information Criteria and Bayesian Information Criteria) compared with linear, parabolic, and hyperbolic discounting models (). Individual and group average subjective value discounting curves are shown in Fig. 1.

Neuroimaging Results.

We first sought to determine which regions appeared to be tracking the integration of reward and effort information at Cue 2 (when the second piece of information was presented, regardless of whether effort or reward information had come first). Using a contrast of Cue2 > Cue1, we identified a network of regions that included the insula, dACC, supplementary motor area, and striatum, regions commonly implicated in effort-based choice as well as a variety of other functions (2, 19–22) (Fig. 2).

Fig. 2.

(A) Increased BOLD signal in dACC, putamen, insula, and supplementary motor area at Cue 2. (B) Increased BOLD signal in dACC in response to decreasing subjective value. The effect size plot demonstrates the negative relationship between BOLD activity and subjective value magnitude in dACC. (C) Increased BOLD signal in vmPFC in response to subjective value magnitude. The effect size plot demonstrates the positive relationship between BOLD activity and subjective value magnitude in vmPFC.

Effort cost not encoded by any brain region.

To determine which, if any, of these regions might encode a specific “effort cost” signal, we next examined a parametric regressor contrast of presented effort requirements at Cue 1 (i.e., without any reward information). If dACC or AI is involved in primarily effort cost encoding, one would expect to see increased activation in these regions in response to increasing effort costs at Cue 1. Contrary to this hypothesis, we did not identify any significant clusters in any brain area, even at a liberal threshold of P < 0.05 (uncorrected).

Subjective value encoding in dACC and vmPFC.

While the analysis above suggested that dACC and aI were not involved in the objective encoding of effort costs, it does not rule out a possible contribution to subjective value (based on reward to be gained). To address this question, a parametric contrast reflecting model-derived subjective value estimates for the chosen option [general linear model 2 (GLM2)] was examined. Consistent with multiple prior studies of subjective value (33, 35), we found that the subjective value of the chosen option was positively associated with activity in vmPFC [x = 4, y = 32, z = −8, t = 5.02, cluster-corrected to control for family-wise error rate (pFWE) < 0.001; Fig. 2] and negatively associated with activity in the dACC (x = −8, y = 20, z = 42, t = 4.69, cluster corrected pFWE = 0.001; Fig. 2). Peak activations were located in the most ventral portion of Brodmann area (BA) 32, with cluster activation extending into BA12. In dACC, peak activation was located in dorsal BA32, with cluster activation extending into BA24. We did not identify any activation clusters in the ventral striatum (VS), even at lenient statistical thresholds (P < 0.05, uncorrected).

“Reversal of fortune” trials.

To better differentiate subjective value encoding from possible expectations that might have formed in response to Cue 1 information alone, trials from GLM1 were divided into bins based on whether the likely subjective value based on only Cue 1 information remained the same or changed based on the information provided at Cue 2 (e.g., the value of a trial beginning with a very high reward at Cue 1 might become significantly less attractive if high effort was required at Cue 2—a reversal of fortune). We examined trials that began with either high reward (reward bin 4) at Cue 1 followed by low (20–50%) or high (80–100%) effort at Cue 2 and trials that began with high effort followed by either reward values in upper half (“high reward trials”) or lower half of reward values (“low reward trials”). Time courses were extracted for both trial types within a functionally defined aI region of interest (ROI; defined by orthogonal contrast: Cue 2 > Cue 1). Greater insular activity was observed during presentation of high effort following presentation of high reward (Fig. 3). However, this activity was not observed during presentation of high effort at Cue 1 (Fig. 3), indicating that this region indeed does not encode a pure effort cost signal. Instead, it appears that this region may be tracking the formation of an expectation or expectation violation, given its involvement in trial types where the information at Cue 2 conflicts with Cue 1 information.

Fig. 3.

(A) Increased percent signal change (PSC) in aI to high-effort information at Cue 2 when presented following high reward at Cue 1 compared with low effort. (B) No observable PSC response to high effort when presented at Cue 1 regardless of reward level presented at Cue 2.

SVPE encoded by aI, dACC, and dorsomedial caudate.

Given that the reversal of fortune trials suggested that aI may track a type of prediction error in the task, we next sought to test this hypothesis more directly. Due to the involvement of both dACC and insula in error-driven reinforcement learning, as well as in both signed and unsigned prediction signals (29–31), we modeled trial-based predictions and computed expectation differentials using a sliding window analysis (GLM3). For example, when subjects saw a large reward value at Cue 1, they would likely expect a high subjective value for the trial as a whole, given the subjective value of past trials with a large reward. However, they could be “surprised” by a high-effort requirement presented at Cue 2, resulting in a negative SVPE. We calculated unsigned prediction errors based on the absolute value of the difference between the observed subjective value and the predicted subjective value based on Cue 1 information. Using a parametric contrast of trial-wise SVPE, we observed a positive association between magnitude of error and dACC (x = 8, y = 24, z = 32, t = 4.52, cluster-corrected pFWE < 0.001), dorsomedial caudate (x = 12, y = 2, z = 12, t = 4.18, cluster-corrected pFWE = 0.006), and aI (x = −36, y = 16, z = −10, t = 6.26, cluster-corrected pFWE < 0.001) activity, suggesting a role for these regions in the encoding of unsigned prediction error signals during effort-based choice (Fig. 4). For dACC, peak activation was located in dorsal BA32, with cluster activation extending into BA24. Importantly, these results remained even when controlling for regressors related to choice difficulty, subjective value, and choice outcome (Effort Option or No Effort Option), suggesting that the involvement of these regions could not be better explained by these other processes. Further, our SVPE regressor was not highly correlated with any of our other variables of interest for any participants (). A complete list of fMRI whole-brain blood oxygen-level dependent (BOLD) amplitude results is provided in .

Fig. 4.

(A) Increased BOLD activity in bilateral aI in response to unsigned SVPE generation. The effect size plot demonstrates this positive relationship between BOLD signal and SVPE. (B) BOLD activity in insula is significantly greater in response to SVPE than to subjective value. Further, within prediction error, dorsal insula activity is significantly stronger than ventral insula activity. (C) Increased BOLD activity in dACC in response to unsigned SVPE. The effect size plot demonstrates this positive relationship between BOLD signal and prediction error encoding. (D) BOLD activity in anterior dACC is significantly stronger than in posterior dACC for subjective value. In posterior dACC, greater BOLD activity was observed in response to SVPE than to subjective value alone. (E) Increased BOLD activity in caudate in response to unsigned SVPE. The effect size plot demonstrates this positive relationship between BOLD signal and prediction error encoding. (F) BOLD activity in caudate is significantly greater in response to SVPE than to subjective value. Further, posterior caudate is more active than anterior caudate for prediction error encoding. *P < 0.05; **P < 0.005; ***P < 0.001.

To better understand the functional specificity of SVPE and subjective value signals across regions, we defined ROIs using previously defined parcellations of dACC, insula, and caudate (50–52) to compare activity in distinct subregions of these structures. Within dACC, we identified an anterior/posterior spatial gradient in the encoding of unsigned prediction error signals, where more anterior subregions of dACC encoded this signal more strongly at a trend level (t = 1.81, P = 0.081; Fig. 4). We also observed that dorsal insula more strongly encoded prediction error than ventral or posterior insula, suggesting spatial specificity for this function (t = 2.70, P = 0.012; Fig. 4). Similarly, within the caudate, we observed an anterior/posterior spatial gradient, where posterior subregions of caudate encoded the SVPE signal more strongly (t = 2.21, P = 0.037; Fig. 4). Interestingly, this posterior location has been found to encode more executive functions as opposed to an action or stimulus value (52), which may explain why it is more active for evaluation of expectation differences.

Finally, using a binned trial analysis (GLMS2) (), we sought to compare this SVPE signal as a function of choice. Results of this analysis demonstrated a linear relationship between SVPE and dACC, caudate, and aI. This analysis also helped us understand if this linear relationship held across chosen and unchosen options. We observed the greatest BOLD activity in dACC for trials with the highest SVPE, which also exhibited significant overlap with the trials where participants chose the noneffortful option. In this way, we see that dACC tracks linearly with unsigned prediction error independently of choice. However, these results were also consistent with past findings suggesting that dACC may be implicated in signaling a shift away from a “default” preference (53).

vmPFC encodes expected subjective value.

We also sought to identify whether any regions encoded an expected subjective value at Cue 1 (before Cue 2 presentation). Using the same sliding window analysis, we generated an expected subjective value regressor that was used in a parametric contrast at Cue 1. Trials that began with either reward or effort information were modeled separately. While we did not observe any regions that tracked expected subjective value at Cue 1 when effort information was presented first, we did observe that expected subjective value at Cue 1 was positively associated with vmPFC activity when reward information was presented first (x = 2, y = 48, z = −8, t = 6.27, cluster-corrected pFWE < 0.001; Fig. 5), highlighting this region as engaged in forming reward-based predictions. For this vmPFC cluster, peak activation was located in dorsal BA32, with cluster activation extending into BA12 and BA10. Consistent with the idea that this activity reflected expectations rather than simply encoding the objective information presented at Cue 1 (i.e., reward magnitude, effort level), we did not identify any regions that significantly responded to the reward magnitude or effort cost of the presented option alone, even at lenient statistical thresholds (P < 0.05, uncorrected).

Fig. 5.

Increased BOLD activity at Cue 1 in response to expected subjective value when reward information is presented first in vmPFC. The effect size plot illustrates a positive relationship between BOLD signal and predicted reward magnitude.

Discussion

The goal of the current study was to investigate areas involved in the encoding of effort, reward, and their integration over time as choice-relevant information became available. Prior studies of effort-based decision making have found that dACC and aI may play an important role in the evaluation of effort-related decisions but that the vmPFC does not. In contrast, we observed a clear role for vmPFC in encoding an expected reward based on the exertion of effort. Additionally, we found evidence to suggest that engagement of dACC and aI during effort-based decision making may be driven by expectation formation and strategy updating rather than effort cost encoding per se.

Early imaging studies found that dACC activity, particularly in BA24, increased with increasing effort requirements, which was interpreted as an evaluation of effort costs (20, 24). This interpretation was further buttressed by animal lesion studies showing that damage to the rodent cg1/2 led to a marked shift in effort-related preferences (11, 54). [While the precise homolog of dACC across rodents and humans is debated, a number of authors have proposed cg1/2 as a likely candidate based on cytoarchitecture and connectivity studies (55).] The present study did not observe evidence that dACC encoded effort costs. Even when looking at presented effort costs in isolation (i.e., Cue 1) we did not find evidence that the dACC (or any region) encoded effort cost alone. Rather, we observed dACC involvement in two processes; the first was the generation of an unsigned prediction error as choice-relevant information became available. Our model for SVPE, the absolute value of the difference between the subjective value of the chosen option and the predicted subjective value, is formally similar to a standard unsigned prediction error. While both signed and unsigned prediction error signals have previously been identified in dACC in a reinforcement-learning context (29–31), the current result extends this effect to cost/benefit decision making. Intriguingly, we found that this effect was strongest when individuals made a no-effort choice, indicating that this SVPE may reach a threshold that signals a shift in strategy, as has been shown in monkeys (56). We also found evidence that lower subjective value was associated with elevated dACC activity, which is consistent with prior reports (36, 57). A potential caveat to the inverse association between dACC and subjective value is the possibility that effortful options with lower subjective values represented a more difficult choice (a more thorough description of the choice difficulty analysis is provided in ). Importantly, however, our main prediction error finding remained when controlling for either subjective value or choice difficulty.

In addition to dACC, we observed that aI and dorsomedial caudate exhibited robust responses to the SVPE. This observation is consistent with the proposed role of the insula as a hub of interoceptive processing (42), where it may be engaged in the regulation of internal states via both homeostatic and allostatic mechanisms. Given that our task required representations of physical effort expenditure, a prominent role for the insula in this valuation process would be expected. Importantly, however, we found that activity in dorsal aI was more strongly associated with prediction error than subjective value (Fig. 4), suggesting a primary role for prediction (i.e., “interoceptive inference”), rather than just effort evaluation (45, 46). A similar role for aI has also been observed in a study where individuals had to learn the value of stimuli in terms of their rewards and associated effort costs (58). Unlike this prior study, however, our task was not an explicit learning task, as the value of the current trial was not contingent on outcomes of prior trials. Nevertheless, we observed evidence for robust prediction error signals in dACC and aI, which suggests that putative value signals observed during effort-based decision-making tasks may partially reflect comparisons between the current trial and trial history.

Our study also identified a clear role for vmPFC in the context of effort-related decisions. This area spanned BA32 and BA10, which is homologous to areas 25, subgenual 32, and 14 in monkeys and infralimbic cortex in the rodent (55, 59). A large number of studies have demonstrated that vmPFC signal scales with expected reward value across a range of reward types (33, 60, 61). In the context of effort discounting, however, there has been some evidence suggesting a dissociation between vmPFC and dACC. Animal research has shown that ACC lesions in rats impaired effort discounting, whereas orbitofrontal cortical lesions only influenced delay discounting (62). These findings had been interpreted to suggest that vmPFC was not involved in effort-based decisions. In contrast, we observed that this region was strongly engaged by the expected reward to be gained in exchange for possible effort. This is consistent with results from a prior fMRI study of effort-based decision making, which found greater vmPFC activity in individuals who weighted reward information more heavily than effort information when selecting among various options (26). Consequently, this analysis suggested a role for vmPFC in the estimation of the likely subjective value of the Effortful Option based soley on the reward amount presented at Cue 1, an effect consistent with this well-established role in the integration of expected value information (23, 63).

Finally, we also observed engagement of the striatum in cost/benefit integration and SVPE signaling. The putamen was strongly engaged during integration of both effort and reward information for all trials at Cue 2, and we further observed activation of dorsomedial caudate as well as the substantia nigra and ventral tegmenal area (SN/VTA) in SVPE signaling. The dorsal striatum has been shown to play a role in negative prediction error encoding, specifically suggested to be involved in choosing between loss-predicting cues so as to avoid the worst outcome (43). In this way, the caudate might use prediction errors to prepare for action to help select the most rewarding outcome Further, involvement of the SN/VTA in prediction error signaling is consistent with research implicating the dopaminergic midbrain in the representation of prediction errors (64). There is growing research on the topographical organization of the midbrain, with evidence suggesting dissociable dopaminergic populations that project to distinct areas of the cortex and striatum (64–66). These regions’ coactivation with dACC and caudate might lend support for parallel mesocortical and mesolimbic pathways that underlie prediction error encoding and signaling.

Unexpectedly, we did not observe strong effects in the VS in response to our variables of interest, including effort or reward magnitude. This is surprising, given a number of animal studies showing that dopamine (DA) manipulations in VS, as well as lesions of VS, may induce profound shifts in effort-based decisions (4, 67–69). While ventral striatal DA is clearly necessary for effort-related decisions, the precise role for DA signaling during effort-based decisions remains unclear. DA transmission operates on multiple time scales, and recent microdialysis and signal transduction studies in animals have found that gradual changes in striatal DA release are correlated with response output to effort-related challenges (70, 71). It may be the case that such slower task-related changes in ventral striatal DA are important for effort-based decision making in humans but are not easily detected by event-related fMRI methods. That said, other studies using fast-scan cyclic voltammetry methods for measuring millisecond changes in DA release have found evidence for phasic encoding of effort discounting such that DA release increased for a reward associated with low effort compared with a reward of equal magnitude associated with greater effort (72). In humans, this type of effort discounting (varying effort levels for equal rewards) has also been associated with modulation of ventral striatal BOLD signals (24, 73). In contrast, effort-based free-choice paradigms, such as that of the current study, have often failed to identify clear VS activity (20, 26). While this absence is not fully understood, BOLD signals in VS during effort-based choice are likely affected not only by striatal DA but also by incoming glutamatergic cortical signals (74), and there is a possibility that these signals aggregate in ways that do not yield a clear parametric response detectable by fMRI. Additionally, our paradigm emphasized processing of effort and reward cues, but we did not have participants complete the selected effort until they were outside of the scanner; VS activity might be more closely linked to action selection and/or effort performance, which could also contribute to the absence of strong effects in this region. Finally, effects of striatal DA on ventral striatal activity may have been further diluted by the long trial durations resulting from our sequential trial design.

Limitations.

There are several limitations to the current study that warrant additional comment. First, our sliding window analysis, which underlies our key finding of these regions’ involvement in prediction error generation, was not tailored to individual learning rates; that is, our analysis averaged the previous five trials, but the relative weight of more recent trials may have fluctuated both across and within subjects. That said, a different trial window length would be unlikely to fundamentally alter the SVPE regressor values.

A second limitation is that our participants did not complete effort while they were in the scanner. Instead, they completed the effort they chose immediately following the scan. We did present the opportunity for participants to change their responses postscan to investigate whether fatigue associated with performing the effort in real time might influence willingness to make effortful choices. We observed near-identical choice patterns postscan as we observed during the scan (). While this may have addressed the question as to whether fatigue of effort completion influences choice, we cannot be sure that the act of completing effort in real time does not change the way participants evaluate and make decisions. This focus on effort and reward-related cues, as opposed to actual effort performance, may have limited our ability to detect effects in certain regions (e.g., VS) and may have partly accounted for our failure to identify any brain areas that appeared to selectively encode effort costs. Alternatively, the lack of evidence for any single region encoding effort costs may point to the possibility of a more distributed representation of effort in the brain (75), which is too spatially disbursed to be adequately captured by our task design. Further, given that this study only examined effort measured by rapid finger pressing, we cannot be certain how these findings might generalize to other forms of physical or cognitive effort. Finally, we note that our regressors were not optimally orthogonalized for all relevant questions, particularly related to the separation of subjective value and choice difficulty (39–41, 76). While we observed a strong unsigned SVPE, our design was not optimized to evaluate a signed prediction error signal because this regressor would be too highly correlated with subjective value. Importantly, however, our primary results remained significant when a choice difficulty regressor was included in the model ().

Conclusion.

Taken together, our results have identified unique prediction error-based functions within the context of effort-based decision making. Going forward, these data should help reveal the precise functions of vmPFC, dACC, and aI during effort-based decision making, and may help clarify the mechanisms underlying maladaptive decision-making behaviors that are commonly observed in clinical populations such as patients with major depressive disorder (27, 77, 78) and schizophrenia (79). While we predict that these functions will generalize to other cost domains outside of effort-based decisions, future studies will be needed to determine the generalizability of these computations to other forms of cost/benefit decision making (e.g., probability, delay).

Materials and Methods

Participants.

Thirty-one healthy volunteers (14 male, Mage = 20.8, SDage = 3.4; ) completed a sequential effort-based decision-making task while undergoing fMRI. All were right-handed, had normal or corrected-to-normal vision, had no history of psychiatric or neurological diseases, and had no structural brain abnormalities. Of these, three participants were excluded: one for excessive head movement, one for falling asleep, and one for behavioral evidence of inadequate task performance. This yielded datasets from 28 participants (13 male, Mage = 20.2, SDage = 2.1) for our final analysis. No statistical tests were used to predetermine sample sizes, but our sample size is within the standard range in the field (2, 24, 36, 39, 41, 80). All study procedures were reviewed and approved by the Emory University Institutional Review Board, and written informed consent was obtained from all participants.

Procedure.

The experimental task was designed to independently measure the neural responses to two dimensions of a cost/benefit decision: the effort required and the magnitude of reward. In this task, participants decided whether to perform a no-effort task for $1.00 or a higher effort task for a larger reward of varying magnitude. The higher effort option independently varied in required button press rate (effort) and reward magnitude. The reward magnitude was shown as a dollar amount (range: $1–$5.73; based on four bins: $1.25–$2.39, $2.40–$3.49, $3.50–$4.60, and >$4.60), and the required effort level was indicated as the height of a vertical bar (20%, 50%, 80%, or 100% of the participant’s maximum button-pressing rate). Before entering the scanner, participants completed three practice trials where they were asked to press a key with their left pinky finger as quickly as possible for 20 s. A participant’s maximum effort was calculated based on the average press rate across the three trials. After establishing each participant’s maximum button press rate, participants practiced completing 20%, 50%, 80%, and 100% of their maximum effort. As part of this practice, participants completed four trials of each effort level to become familiar with how effortful each value was for them. The practice trials lasted about 5 min. Participants were informed that they would not complete the physical effort component while in the scanner but would have to complete it based on the choices they made immediately following the scan.

Each trial was composed of Cue 1, Cue 2, Decision Prompt, and Choice phases. At Cue 1, participants were presented with only one piece of information from the Effort Option (either the associated effort level or reward magnitude). This first piece of information remained on the screen for a brief jittered delay of between 2 and 6 s (mean = 2.98 s); participants then saw Cue 2, which revealed the other piece of information. After another brief jittered delay of between 2 and 6 s (mean = 3.23 s), the participants were prompted to make their selection: either accept the Effort Option that has been presented or reject that option in favor of the No Effort Option that pays $1. Then, the participant’s selection was shown in the Choice phase. The interstimulus jitter was drawn from a Poisson distribution similar to that used in sequential foraging tasks (36). Because the noneffortful option was fixed, it was not presented during the task. Order of information (effort first or reward first), as well as side of presentation for effort and reward information (right or left), was counterbalanced across trials (Fig. 1). Trials were presented in the same fixed, randomized order for all participants.

While in the scanner, participants completed a total of two runs of this task. Each run lasted ∼9 min and consisted of 44 trials (11 trials per effort level and reward bin values). Stimulus presentation and response acquisition were performed using MATLAB R2013b (MathWorks) with the Psychophysics Toolbox (81). Participants responded with MRI-compatible response keypads. Participants did not complete effortful button pressing during the scan to reduce motion, as well as over the length of the scan to reduce overall task-length fatigue.

Following the scan, participants were presented with the Effort Options they selected while in the scanner. They were then asked to complete the effort required for the choices they had selected. Importantly, for each chosen trial, they were given the opportunity to change their responses. This option was given to investigate whether the fatigue of performing the effort in real time might influence a participant’s willingness to make effortful choices. We observed choice patterns postscan very consistent with those we observed during the scan, with participants choosing the same options on 97 ± 4% of trials.

Image Acquisition.

Imaging data were acquired on a Siemens 3T Tim Trio using a 32-channel, phased-array head coil. Trial presentations were synchronized to initial volume acquisition. Functional (T2*-weighted) images were acquired using a multiband sequence with the following sequence parameters: 3-mm3 isotropic voxels, repetition time (TR) = 1.0 s, echo time (TE) = 30 ms, flip angle (FA) = 65°, 52 interleaved axial slices, with slice orientation tilted 18° relative to the anterior commissure/posterior commissure plane to improve coverage of vmPFC. At the start of the imaging session, a high-resolution structural volume was also collected with the following sequence parameters: 2-mm × 1-mm × 1-mm voxels, TR = 1.9 s, TE = 2.27 ms, FA = 9°.

Behavioral Analysis.

Analyses were conducted using MATLAB 2015B (MathWorks) and SPSS v22 (IBM). To examine choice data across varying levels of effort and reward magnitude, repeated-measures ANOVAs were used. For cases that violated the sphericity assumption, a Greenhouse–Geisser correction was used.

Subjective value models.

To estimate participants’ subjective values for the offers presented on each trial, we used a two-parameter power function, which has been previously described (49). This effort-discounting model has been shown to provide better fits than the hyperbolic model previously suggested for effort discounting (20) both here and in other studies (49). The two-parameter power function estimates subjective values on each trial using Eq. , where SV is the subjective value, E is the amount of required effort [0.2, 0.5, 0.8, 1], R is the reward magnitude, and k and p are free parameters that are fit for each participant:

The subjective value of the No Effort Option, which does not require any effort to be exerted and was always worth $1, assumes a value of 1 on each trial. Importantly, the p parameter allows the two-parameter power function to take a concave or convex shape depending on the rate at which the participant devalues reward with additional effort. Hyperbolic discounting functions that have been traditionally used for delay discounting and previously suggested for effort discounting follow a convex function, where the addition of effort has a larger devaluation effect on smaller effort costs and very small devaluation effects at higher levels of effort. Alternatively, recent work has suggested that it is both intuitive and biologically plausible for effort discounting to instead take a concave shape, where additional effort has small effects on subjective value at lower levels of required effort but increases steeply as effort reaches more demanding levels (49). To verify that the two-parameter power function provided a better fit for our data, we compared it with discounting models that use hyperbolic, quadratic, and linear discounting functions previously employed to describe effort discounting (6, 20, 82). All models were each fit to a subject’s data individually using the MATLAB function “fminsearch,” and parameters were selected for each participant that optimized the likelihood of the behavioral data (a more detailed discussion of model comparison and selection is provided in ).

The Softmax function (Eq. ) was used to transform the subjective values of the two options offered on each trial into choice probabilities for selecting each option a on trial t. The Softmax function includes an inverse temperature parameter, β, which is fit as an additional free parameter for each participant for each of the discounting models. The inverse temperature parameter determines the degree to which the choice probabilities are affected by the estimated subjective value of each option, with lower values indicating random responding and higher values indicating a tendency to choose the option with the highest subjective value. The fits of the discounting models were also compared with a simple model that assumes a fixed probability of choosing each option:

SVPE.

Estimates of expected subjective value at Cue 1 (SVpredicted) were calculated using a sliding window analysis of previously experienced subjective values of the same trial type. The value of SVpredicted on each trial was derived from the Cue 1 stimulus value and recent subjective values of trials with the same stimulus value (i.e., either reward bin or effort level; more information is provided in ). Subjective values for previous trials were calculated using the two-parameter power function (Eq. ) and each participant’s best-fitting parameters. The SVPE regressor was calculated by subtracting SVpredicted from SVchosen, where SVchosen is calculated under the two-parameter power function using the pieces of information provided at both Cue 1 and Cue 2.

fMRI Analysis.

All neuroimaging data were preprocessed and analyzed in SPM12 (Wellcome Department of Imaging Neuroscience, Institute of Neurology, London). Preprocessing in SPM12 included realignment estimation and implementation, coregistration to the individual’s high-resolution structural scan, normalization to Montreal Neurological Institute space, and spatial smoothing using a Gaussian kernel (6-mm FWHM). A standard hemodynamic response function was used for all GLMs, and it was modeled based on the duration of each cue for each trial. Across all GLMs, we used the SPM default orthogonalization. When controlling for other regressors, the regressor of interest was always entered second (83).

To identify areas that encoded reward or effort signals, we implemented the first GLM (GLM1), which included eight conditions: Cue 1, Cue 2, Decision Prompt, and choice divided by order of presentation (effort first or reward first). The first three phases were associated with two parametric modulators: the reward magnitude and effort of the chosen option.

To further investigate and identify areas that encoded subjective value as well as the integration of effort and reward information, we implemented a second GLM. The second GLM (GLM2) was identical to the first, except that parametric modulators were replaced by expected subjective value at Cue 1 (calculated with a sliding window analysis) and subjective value estimates of the chosen option at Cue 2.

Last, a third GLM (GLM3) aimed to identify areas that encoded prediction as well as an unsigned prediction error. It was identical to the first except that the parametric modulators were replaced by predicted subjective value as determined by our sliding window analysis at Cue 1 as well as SVPE at Cue 2 (though SVC was not included at Cue 2 for this model).

For whole-brain analyses, we used an FWE cluster-corrected threshold of P < 0.05 (using a cluster-defining threshold of P < 0.005 and a cluster threshold of 20 voxels). Beta values were extracted from ROIs as well as from various defined regions of the medial prefrontal cortex (51) and insula (50).