Rating expectations can slow aversive reversal learning.

The process of learning allows organisms to develop predictions about outcomes in the environment, and learning is sensitive to both simple associations and higher order knowledge. However, it is unknown whether consciously attending to expectations shapes the learning process itself. Here, we directly tested whether rating expectations shapes arousal during classical conditioning. Participants performed an aversive learning paradigm wherein one image (CS+) was paired with shock on 50% of trials, while a second image (CS-) was never paired with shock. Halfway through the task, contingencies reversed. One group of participants rated the probability of upcoming shock on each trial, while the other group made no online ratings. We measured skin conductance response (SCR) evoked in response to the CS and used traditional analyses as well as quantitative models of reinforcement learning to test whether rating expectations influenced arousal and aversive reversal learning. Participants who provided online expectancy ratings displayed slower learning based on a hybrid model of adaptive learning and reduced reversal of SCR relative to those who did not rate expectations. Mediation analysis revealed that the effect of associative learning on SCR could be fully explained through its effects on subjective expectancy within the group who provided ratings. This suggests that the act of rating expectations reduces the speed of learning, likely through changes in attention, and that expectations directly influence arousal. Our findings indicate that higher order expectancy judgments can alter associative learning.

INTRODUCTION

The study of threat‐induced defensive reactions has proliferated in the last century, focusing in large part on Pavlovian classical conditioning, the critical process by which an organism learns the structure of the world and develops predictions or expectations about salient outcomes in its environment (Rescorla, 1988). Many of the defensive reactions that animals display can also be measured in humans, providing the opportunity for fundamental insights through translational work. Of course, humans can also verbalize their expectations and feelings, and thus many human threat conditioning studies ask participants to provide expectancy ratings, or estimates of the probability of shock, as an additional measure of learning. People report increased shock expectancy in response to associations acquired through classical conditioning (Boddez et al., 2013; Dunsmoor et al., 2014), verbal instructions (Mertens, Braem, et al., 2018; Mertens & De Houwer, 2016), and vicarious (social) learning (Selbing & Olsson, 2019). Expectancy ratings are also clinically relevant, as they are elevated in patients with pathological anxiety (Britton et al., 2013; Chan & Lovibond, 1996) and post‐traumatic stress disorder (Blechert et al., 2007), and meta‐analyses of fear conditioning in anxiety indicate that of subjective ratings differ between patients and controls during both acquisition and extinction, driven in part by differences in response to the safety cue (CS−) (Duits et al., 2015). For these reasons, expectancy ratings have been argued to be a valid measure of the aversive learning process (Boddez et al., 2013), similar to other defensive reactions. But does the act of providing expectancy ratings influence adaptive learning and the expression of defensive behaviors including physiological arousal?

Although expectancies are influenced by many of the same factors as other defensive reactions and behaviors, several studies indicate that they are dissociable, suggesting they may rely on distinct mechanisms. Studies have shown dissociations between expectancies and skin conductance responses (SCR; Ohman & Soares, 1998; Schell et al., 1991; Schultz et al., 2013; Schultz & Helmstetter, 2010), eyeblinks (Perruchet, 2015; Weidemann et al., 2009, 2012), fear‐potentiated startle (Kindt et al., 2009), and reaction time (Perruchet et al., 2006). These dissociations have been implicated as support for a dual process model whereby independent mechanisms support: (a) processes that are sensitive to associative learning and can exist outside of conscious awareness (e.g., defensive reactions and physiological responses) and (b) higher order processes that support propositional learning and require awareness (e.g., expectancy, subjective feelings, and instructed knowledge; LeDoux, 2013; Mineka & Öhman, 2002). Consistent with this, studies of the “Perruchet effect” (Perruchet, 2015) indicate that in partial reinforcement studies, unconditioned stimulus (US) expectancy increases after a series of trials without reinforcement (consistent with a gambler’s fallacy), whereas behavioral or physiological measures decrease when US presentations are less recent. These findings have been viewed as support for the idea that expectancy ratings depend on higher order beliefs whereas automatic responses including autonomic responses and defensive reactions are driven by associative learning (but cf. Weidemann et al., 2009, Weidemann, McAndrew, et al., 2016; Weidemann, Satkunarajah, et al., 2016).

Importantly, a substantial body of work calls the dual process model into question and indicates that higher order knowledge can shape learning‐related responses (for reviews, see Lovibond & Shanks, 2002; Mertens & Engelhard, 2020; Mitchell et al., 2009). Critiques of the dual process framework point to methodological problems in both study design (Lovibond & Shanks, 2002; Singh et al., 2013; Weidemann, McAndrew, et al., 2016) and analysis (Shanks, 2017) as well as publication bias (Mertens & Engelhard, 2020; Vadillo et al., 2016). Mechanistic studies of instructed fear indicate that higher order knowledge can modulate autonomic responses (Atlas et al., 2016; Atlas & Phelps, 2018; Costa et al., 2015; Grings et al., 1973; Mertens, Boddez, et al., 2018) and responses in learning‐related systems (Atlas et al., 2016; Doll et al., 2009; Li, Delgado, et al., 2011). Interactions between higher order knowledge and dynamic learning suggest that propositional knowledge can shape learning through a single process. However, it is unknown whether meta‐cognitive predictions in the form of expectancy ratings have similar effects as instructed knowledge on learning and defensive responses. To understand whether expectancies influence learning‐related responses, we must not only compare expectancy ratings with physiological responses, but also measure physiological responses in the absence of ratings.

Our goal was to determine whether the act of providing online expectancy ratings during threat conditioning alters dynamic learning and physiological responses, relative to conditioning in the absence of ratings. We focused on SCR, which might be particularly sensitive to higher order knowledge. For example, prior work in patients with amygdala lesions suggests that expectancy ratings may indeed modulate learning as measured by SCR, consistent with an interactive model. Patients with amygdala lesions did not express conditioned responses during standard threat conditioning, as measured by differential SCR, but displayed differential SCR when they were asked to provide expectancy ratings (Coppens et al., 2009). This indicates that the act of providing expectancy ratings engaged circuits outside of the amygdala, which in turn influenced autonomic responses. Similarly, in a study of reconsolidation, a process that depends on the amygdala (Monfils et al., 2009), participants who provided expectancy ratings exhibited enhanced fear‐potentiated startle during fear acquisition, but less evidence of reconsolidation, relative to participants who did not provide ratings (Warren et al., 2014). Ryan and colleagues (2021) also observed enhanced acquisition and extinction learning based on differential SCR when participants provided expectancy ratings relative evaluative ratings or learning without any ratings. Together, these studies suggest that the act of rating expectations can alter autonomic responses and engage different neural circuits to drive physiological responses, perhaps by increasing the contribution of higher order processes such as attention or explicit beliefs. However, it is unknown whether expectancy ratings altered dynamic learning because all studies to date measured the effects of providing expectancy ratings in conditions without any stochasticity (i.e., 100% reinforcement during conditioning or 0% reinforcement during extinction). To determine whether expectancy ratings alter learning per se, we must measure the impact of expectancy ratings on dynamic associative learning. This can only be accomplished in environments that contain some volatility, such as partial reinforcement schedules and/or contingency reversals. In both partial reinforcement and reversal learning, individuals must continually update learning as a function of experience, and this requires flexible, adaptive learning, whereby individuals increase learning rates when environments become less stable (Behrens et al., 2007; Li, Schiller, et al., 2011; Mackintosh, 1975; Schiller et al., 2008). Previous work indicates that reversal learning can be modulated through instruction (Atlas, 2019; Atlas et al., 2016; Costa et al., 2015; Grings et al., 1973; Mertens & De Houwer, 2016). How do expectancy ratings shape the dynamic adaptive learning process?

We conducted a human behavioral experiment to examine whether the act of rating expectancy alters adaptive reversal learning. One group of healthy human subjects was assigned to an Expectancy Rating Group (“Rating Group”), who provided online expectancy ratings during aversive learning, and another set of participants was assigned to a Viewing Group, who learned from experience in the absence of online ratings. Both groups underwent the same aversive reversal learning task using partial reinforcement (50% reinforcement of the CS+; see Figure 1). We combined quantitative models of adaptive learning with traditional statistical analyses and multilevel mediation analyses to ask whether online expectancy ratings influence dynamic aversive learning.

FIGURE 1

Task design. (a) Participants assigned to a Rating Group made expectancy ratings during aversive reversal learning, while Viewing Group participants viewed images and received shocks in the same task without making ratings. (b) There was a 50% reinforcement rate for the CS+ (i.e., 14 unreinforced presentations and 14 reinforced presentations). Halfway through the task, contingencies reversed and the initial CS− became the new CS+ and was reinforced with a 50% reinforcement rate for the duration of the task. (c) On each trial, the CS was presented for 4 s followed by a 12 s inter‐stimulus interval. CS+ presentation coterminated with a 200 ms shock. Two stimuli were used (purple/yellow fractal or red/green fractal) and initial CS assignment was counterbalanced across participants

We focused on a hybrid model of adaptive learning that incorporates associability, or the extent to which attention gates learning in response to changes in the environment (Mackintosh, 1975; Pearce & Hall, 1980). Consistent with previous work (Homan et al., 2019; Li, Schiller, et al., 2011; Tzovara et al., 2018; Zhang et al., 2016), we focused on SCRs as a measure of physiological arousal that is sensitive to adaptive learning, orienting, uncertainty, and attention (Ojala & Bach, 2020). We hypothesized that the timecourse of associative learning would differ between participants who provide online expectancy ratings versus those who undergo aversive learning without rating shock probability. We did not have directional hypotheses, as ratings might increase attention to contingencies (Mackintosh, 1975), consistent with enhanced differential responses during acquisition in previous work (Warren et al., 2014), or they might reduce overall arousal if they serve to enhance elaborative processing and reduce the threat value of the conditioned stimulus (e.g., through distraction).

METHOD

Participants

Participants were recruited from New York University and the surrounding community. All participants were required to be right‐handed and fluent in English. Participants were ineligible if they had participated in an experiment using shocks within the prior six months, if they were taking medication for anxiety or depression, if they might be pregnant, or if they had a history of heart or blood pressure problems. All participants provided informed consent in accordance with the Declaration of Helsinki and as approved by the New York University Institutional Review Board (IRB # 13‐9582). Eighty‐nine participants provided informed consent. Seven participants did not show measurable SCR in response to the breath‐hold task and were dismissed prior to the threat conditioning experiment. Technical difficulties prevented us from acquiring SCR or behavioral data from two additional participants. Eighty participants completed the task (47 Female, Mean Age = 22.28, SD = 3.09, missing age from 6 participants).

Stimuli and materials

Participants underwent a threat conditioning paradigm with a 50% reinforcement rate and one reversal of CS‐US contingencies (Figure 1b). Two fractal images served as conditioned stimuli (Figure 1c). Image assignment (initial CS+, initial CS−) was counterbalanced across subjects. A Grass Medical Instruments SD9 Stimulator delivered shocks to participants’ right forearms (200 ms duration) through a bar electrode (BIOPAC Systems, Inc., Goleta, CA) filled with standard NaCl electrolyte gel (Signagel from Parker Laboratories, Fairfield, NJ). Shocks coterminated with the CS presentation and consisted of a 200 ms duration train of pulses at 40 Hz.

SCR were measured through shielded Ag‐AgCl electrodes filled with 0.5% NaCl isotonic electrolyte gel (EL507; BIOPAC Systems, Inc., Goleta, CA) attached to the left palm. Data were recorded at a sample rate of 200 Hz using an MP‐150 BIOPAC system with the AcqKnowledge software (BIOPAC Systems, Inc., Goleta, CA). Acknowledge software was used for analysis. Participants also completed the Spielberger State‐Trait Anxiety Inventory STAI (form X); (Gaudry et al., 1975) and the Intolerance of Uncertainty Scale (IUS; Buhr & Dugas, 2002). The present study focuses on between‐groups differences in SCR without respect to anxiety or intolerance of uncertainty.

Procedure

Skin conductance eligibility and shock calibration

Participants provided informed consent “to take part in a research study to learn more about emotion and cognition.” Following consent, the participant completed questionnaires and was invited to a behavioral testing room where they were affixed with SCR and shock electrodes and completed the task on a computer. The experimenter remained in the room for the duration of the task. As all participants were right handed, participants used a mouse to record ratings with their right hand and skin conductance was recorded from the index and middle fingers of the left hand.

Prior to the main experiment, participants performed a breath holding task to ensure that they showed measurable SCR. Participants were asked to take a deep breath and hold it for three seconds. Participants whose skin conductance increased in response to the breath hold were eligible to continue. Seven participants were dismissed at this point.

Following the breath holding task, an electric shock stimulator was attached to the participant’s right wrist. We calibrated the shock intensity using an ascending staircase procedure, in which intensity was increased incrementally from 20 V in 5‐V increments until it reached a level that participants considered “highly annoying but not painful”. Once this level was achieved, the shock remained at this intensity throughout the conditioning task (M = 36.78 V, SD = 9.69).

Experimental design

Participants were randomly assigned to the Rating Group or the Viewing Group. Participants in the Rating Group were asked to rate the expected likelihood of shock during each CS presentation, using a continuous visual analogue scale ranging from “0%: sure of no shock” to “100%: sure of shock” (Figure 1a). This type of online US‐expectancy rating is thought to be one of the most valid measures of contingency awareness (Boddez et al., 2013). Participants were asked to record expectancy ratings using the mouse within the 4‐second CS presentation. If a participant failed to record a response during the 4‐s period, the final location of the mouse was used as the rating. On average, subjects missed fewer than 1 rating (M = 0.23, SD = 0.53). Participants in the Viewing Group viewed the CS images while making no overt responses. Participants received general contingency instructions but were not informed about the specific relationships between the CSs and outcomes. All participants were told to “try to figure out the relationship between the stimuli you see and the shocks you feel.”

All participants underwent the same aversive reversal learning task with a single reversal and 50% reinforcement rate (see Figure 1). During the first 42 trials, the original CS+ coterminated with a shock (US) on 50% of CS+ trials (i.e., 14 pairings), while the CS− was never paired with the US. Thus, the pre‐reversal phase included 14 CS− trials, 14 unreinforced CS+ trials, and 14 trials in which the CS+ coterminated with a shock US. Halfway through the task the contingencies reversed, such that the former CS+ became the CS− and vice versa for the last 42 trials. Reinforcement rates were the same, that is, the new CS+ (original CS−) had a 50% reinforcement rate (i.e., 14 unpaired trials, 14 trials paired with a shock) and there were 14 new CS− (original CS+) presentations. Participants were not instructed upon reversal. We used two trial orders, which were each pseudo‐randomized within the constraints that there were never three of the same CS image sequentially or two shocks in a row. All participants saw the same total number of each CS type (42 original CS+ trials, 42 original CS− trials) and received 28 shocks over the course of the experiment. Each CS was displayed for 4 s, followed by a 12‐s inter‐stimulus interval (ISI) during which a fixation cross was displayed. CS images were counterbalanced across participants.

Following the experimental task, participants answered a series of post‐task questions assessing declarative knowledge of the CS‐US contingencies, subjective emotional reactions to each of the CS images, and a free response item regarding any patterns or relationships observed during the study. Free responses were not included in the current analyses. Participants were then debriefed and dismissed.

Skin conductance data processing

SCR data was processed in AcqKnowledge (BIOPAC Systems, Inc., Goleta, CA) and filtered with a 25‐Hz low‐pass FIR filter and smoothed with a Gaussian kernel of 10 samples. SCRs were measured as the base‐to‐peak amplitude difference for each trial during the 0.5–4.5 s window after stimulus onset. SCR amplitudes that were less than 0.02 microSiemens were considered non‐responses and scored as 0. Amplitude estimates were square root transformed (Schlosberg & Stanley, 1953) and normalized relative to each participant’s mean square‐root‐transformed US response (Ben‐Shakhar, 1985; Fowles et al., 1981).

Statistical analyses

We used ANOVAs to analyze post‐task ratings as a function of Group (Rating vs. Viewing Only) and Stimulus type (Original CS+ vs. Original CS−) and used linear mixed models to analyze SCR outcomes on unreinforced trials (i.e., trials that were not paired with a US) as a function of Group, Stimulus, and Phase (Pre‐ vs. Post‐reversal). ANOVAs were implemented using Matlab’s “anovan.m” program and we modeled subject as random and nested in Group.

To account for the fact that conditioning and reversal learning are dynamic processes that occur over time as a function of reinforcement and experience, we used linear mixed effects models that model outcomes trial‐by‐trial and can account for effects that vary over time, as well as computational models of learning (see next section). Linear mixed models are advantageous relative to ANOVAs, which use summary statistics, average responses across trials, and cannot capture the dynamic nature of nature of learning. All linear mixed models were analyzed in R (R Core Team, 2014) using the nlme package (Bates et al., 2015). We modeled fixed effects of Group, Stimulus Type, Phase, and Trial, and all possible interactions. Slopes and intercepts were treated as random, and we modeled autoregression (AR(1)). Post‐hoc pairwise comparisons were evaluated using the R package “emmeans” (Lenth, 2020).

Computational modeling

While our linear mixed models provide tests of whether responses emerge gradually as a function of time (i.e., linear effect of Trial), computational models of reinforcement learning provide further insight on the dynamic learning process by testing how responses update not only as a function of trial, but also in response to specific outcomes. More specifically, they test whether expected value (EV) updates in response to a given reinforcement (r), which depends on the learning rate (), or the speed of updating, and the prediction error (δ) which is the deviation between EV and reinforcement (r) on a given trial.

The learning rate governs the extent to which a prediction error causes EV to update on the next trial. Thus reinforcement learning models extend insights from mixed models by not only testing whether the magnitude of the differential response increases over time (e.g., a Cue × Trial interaction), but also how learning takes place on a trial‐by‐trial basis, both in response to unexpected reinforcements (e.g., shocks) and unexpected omissions of reinforcement (e.g., reversals).

Because reversals engage contextual shifts that have been previously shown to engage adaptive learning (Behrens et al., 2007; Li, Schiller, et al., 2011), we evaluated a “hybrid model” of adaptive learning (Li, Schiller, et al., 2011) to test whether act of rating expectations modulates value‐based learning. In contrast to a standard Rescorla‐Wagner model (Rescorla & Wagner, 1972) which assumes that learning is stable over time, hybrid models are based on the Pearce‐Hall model and allow learning rates vary dynamically as a function of associability (Mackintosh, 1975; Pearce & Hall, 1980), which is inversely related to the stability of the environment. In other words, learning rates are higher in volatile environments and lower in stable environments, and depend on the recent history of prediction errors. The model is referred to as a “hybrid” model because it integrates the concept of prediction error from the Rescorla‐Wagner model with associability from the Pearce‐Hall model (Li, Schiller, et al., 2011). Prior work using hybrid models in aversive learning (Atlas et al., 2019; Li, Schiller, et al., 2011; Zhang et al., 2016) indicates that SCR reflects the joint combination of associability (i.e., the current dynamic learning rate ()) and expected value (EV). These parameters update dynamically based on two free parameters, κ and η, which control the rate at which prediction errors influence outcomes:

We fit models using Matlab’s “fmincon.m” function and minimized the sum squared errors between EV or EV and associability and each participant’s SCR on unreinforced trials. We also included a linear effect of time in all models, and fit models separately to participants in each group. Consistent with previous work (Atlas & Phelps, 2018; Miller et al., 1998; Wu, 1986), we used a “jack‐knife” approach to model fitting and iteratively left out one subject on each iteration and fit to the remaining subjects to estimate parameters across the group. This provides a distribution of scores for statistical comparison, while reducing the noise associated with individual model fits. We compared three models: (1) a hybrid model with four free parameters (κ, η, initial EV, initial ); (2) a model that assumed an initial EV of 0.5 and an initial of 1.0 (i.e., only κ and η were modeled as free parameters); (3) a standard Rescorla‐Wagner model that included a constant learning rate which was assumed to be stable over time, and the learning rate () and initial EV were modeled as free parameters.

We fit models to skin conductance on unreinforced trials throughout the entire task (i.e., pre‐ and post‐reversal) and computed Aikake’s Information Criterion for each model (Akaike, 1974), which penalizes models for extra parameters. We used Bayesian model selection implemented with SPM_bms (Stephan et al., 2009) to compare models. The hybrid model with four free parameters was determined to be the best fit across participants (see Results) and we therefore use this model for inference.

We compared group differences in each parameter using two‐sample t‐tests in Matlab based on fitted parameters from jack‐knife approaches. We also evaluated models fit to individuals and across the entire group for completeness.

Multilevel mediation analysis

To understand the relationship between expectancy ratings and observed differential responses in autonomic arousal within the Rating Group, we tested whether expectancy ratings formally mediated conditioned responses (i.e., differential effects of CS on SCR). Multi‐level mediation analyses were implemented in Matlab using the Mediation‐Moderation Toolbox (Atlas et al., 2010; Wager et al., 2009). We modeled CS type (current CS+ vs. current CS−) as the input variable (X), SCR on unreinforced trials as the outcome variable (Y), and tested for mediation by expectancy ratings. On trials on which subjects did not provide an expectancy rating (M = 1.55, SD = 2.36), we used the mouse position at the end of the 4 s CS period as a measure of expectancy. We used bootstrapping to test the significance of mediation to account for non‐normality of the indirect path (Shrout & Bolger, 2002).

Our mediation analysis differs from the multilevel models and quantitative learning models in that it tests responses only within the Rating Group. Furthermore, the mediation model measures the contribution of subjective expectancy ratings themselves both as a function of CS type and in relationship to evoked SCR, whereas the multilevel models and computational models examine SCR as a function of whether or not ratings were collected, rather than considering the trial‐by‐trial ratings themselves.

RESULTS

Post‐task ratings

There were no differences between Groups in the number of perceived reversals during the task (M RG = 1.79, SD RG = 0.95; M PG = 1.66, SD PG = 1.05; p > .5; see Figure 2). Retrospective probability ratings indicated that participants associated a higher likelihood of shock with the original CS+ during the beginning of the study and a higher likelihood of shock with the original CS− at the end of the study (Stimulus × Phase, F(1,291) = 136.07, p < .001; η 2 = 0.508). Participants also reported higher likelihood of shock at the start of the task versus the end (F(1,291) = 7.41, p = .008; η 2 = 0.007). Probability estimates did not differ by group (all p’s > .3). Retrospective affect ratings also did not differ as a function of CS Type or Group (all p’s > .2).

FIGURE 2

Retrospective ratings. Upon task completion, participants retrospectively rated (a) number of perceived reversals, (b) probability of shock associated with each stimulus at the beginning and at the end of the study; and (c) affect in response to each stimulus. Groups did not differ in any retrospective ratings. Error bars denote SEM

Participants who make expectancy ratings show sustained SCRs based on original contingencies

We used linear mixed models to examine the effects of Group, Phase (Pre‐ vs. Post‐reversal), Stimulus (Original CS+ vs. Original CS−), and Trial on SCR to unreinforced trials to measure whether rating expectations alters aversive learning (Figure 3). A fully specified linear mixed model failed to converge using the lme4 package in R with various optimization factors, so we used the lme package from nlme which allowed us to control for autoregression and evaluate the full model. We included fixed factors for Stimulus, Phase, Trial, and Stimulus*Phase interactions, and included random intercepts per subject and random factors for all effects except Trial. Results revealed significant main effects of Stimulus (p = .014), Phase (p = .003), and Trial (p = .001), as well as significant interactions between Stimulus and Phase (p < .001), Phase and Trial (p < .001), Group × Stimulus × Phase (p = .021), Group × Stimulus × Trial (p = .039), and Stimulus × Phase × Trial (p < .001). We were most interested in the Group × Stimulus × Phase interaction. Posthoc analyses separated by Group indicated that the Viewing Group showed a significant Stimulus × Phase interaction (B = 0.067, p < .001) whereas the Stimulus × Phase interaction was not significant in the Rating Group (p > .1). For complete results and results separated by group, please see Tables S1–S3. We also conducted pair‐wise comparisons post‐hoc between all factors using the R package emmeans (Lenth, 2020). Pairwise comparisons indicated that the key interaction was driven by the fact that only the Viewing Group showed significant differences between the CS+ and CS− prior to reversal (p < .001) and that responses to the original CS− increased after the reversal within the Viewing Group (p < .001). There were no significant differences as a function of Stimulus or Phase within the Rating Group in pairwise post‐hoc tests based on adjusted p‐values. For complete results of pair‐wise post‐hoc tests, see Table S4.

FIGURE 3

Skin conductance as a function of Group and Phase. Left: This figure illustrates skin conductance responses as a function of Stimulus prior to reversal (top) and following contingency reversals (bottom). Only the Viewing Group showed significant differences prior to reversal and a complete reversal of the differential response when contingencies changed. Middle: SCR and expectancy ratings show a similar timecourse on unreinforced trials within Rating Group participants, where responses do not reverse until several trials after the reversals. Top right: SCR in the Viewing Group reverses immediately upon contingency reversal. Raincloud plots are visualized using the R package raincloudplots (Allen et al., 2021)

Rating expectancy slows associative learning

Next, we fit dynamic learning models to SCR on unreinforced trials throughout the entire task (i.e., both pre‐ and post‐reversal) to test whether rating expectations alters dynamic value‐based learning. We fit models separately to each individual and used jack‐knife estimation to iteratively leave out one subject and fit estimates to remaining participants in each group, which provides a distribution of estimates for between‐group comparison that is less sensitive to noise than individual subject‐level fits. We compared a standard Rescorla‐Wagner model (Rescorla & Wagner, 1972) which assumes a stable learning rate across time with two variations of a hybrid model (Atlas et al., 2019; Li, Delgado, et al., 2011; Mackintosh, 1975; Pearce & Hall, 1980) that assumes that learning rates vary as a function of associability (see Method for model details). Model comparison using SPM_bms (Stephan et al., 2009) revealed that a hybrid model that included all parameters as free provided the best fit for our data, based on an exceedance probability of 0.76, versus the hybrid model that assumed starting parameters for learning rate and expected value (exceedance probability = .24) and the Rescorla‐Wagner model (exceedance probability = 0). We therefore make inferences based on the hybrid model that included four free parameters.

Jack‐knife estimation revealed that Viewing Group participants had significantly higher values than Rating Group participants for all parameters (Figure 4), including initial learning rate (M PG = 1.0; SE PG = 0.0; M RG = 0.66; SD RG = 0.86; t(1,78) = −19.11, p < .001), initial expected value (M PG = 0.29; SE PG = 0.05; M RG = 0. 01; SE RG = 0.004; t(1,78) = −5.49, p < .001), kappa (M PG = 0.42; SE PG = 0.02; M RG = 0.28; SE RG = 0.004; t(1,78) = −6.79, p < .001), and eta (M PG = 0.42; SE PG = 0.04; M RG = 0.23; SE RG = 0.006; t(1,78) = −2.14, p = .036). Fitting separately to each individual revealed parameters that differed in the same direction although group differences were not significant in these fits, consistent with the fact that individual estimates are noisier (see Figure S1 and Table S5).

FIGURE 4

Hybrid model learning parameters differ by group. Fitting a Rescorla‐Wagner model of reinforcement learning to SCR on unreinforced trials revealed higher learning rates in Viewing Group participants than participants in the Rating Group, whether fit to individuals or jack‐knife estimation to iteratively fit across each group using cross validation. Learning rates depicted here are from jack‐knife estimation. See Figure S1 for complete results of jack‐knife estimation and fits to individuals

Expectancies mediate differential responses in SCR

Finally, we tested whether subjective expectancies formally mediated the effects of conditioned cues on SCR within the Rating Group. Here, we analyzed differential effects, that is, effects of current contingencies including the reversal (Cue × Phase interactions), rather than original contingencies, and included all Rating Group participants. As shown in Figure 5, there was a significant differential effect (current CS+ > current CS−) on SCR on unreinforced trials (Path c = 0.04(0.01), p < .001), consistent with analyses reported above. There was also a significant differential effect on expectancy (Path a = 0.14(0.02), p < .001), such that shock expectancy was higher in response to the current CS+ than the current CS−, and an effect of expectancy on SCR, controlling for the differential effect (Path b = 0.18(0.05), p < .001), such that skin conductance was higher when subjects expected a shock. Finally, we found that expectancy ratings fully mediated the relationship between CS and SCR (Path a × b = 0.02 (0.0), p < .001), such that the differential effect on SCR was absent when controlling for expectancy rating (Path c′ = 0.01(0.01), p > .16; see Figure 3), indicating that subjective expectancy fully explains differential responses in SCR when individuals make expectancy ratings. We also examined the covariance between Paths a and b, since the mediation effect in multilevel mediation (i.e., c–c′, the difference between the direct and indirect effects) can be driven by both the product of the path coefficients and the covariance of the paths (i.e., c–c′ = a × b + cov(a,b); (Kenny et al., 2003). In this case, we observed significant negative covariance (see Figure 3), suggesting that participants who showed stronger CS effects on expectancy showed weaker additional effects of expectancy on arousal. Finally, to evaluate directionality, we tested a reversed mediation, that is, whether physiological arousal mediates effects on expectancy, as proposed by models such as the somatic marker hypothesis (Poppa & Bechara, 2017). When we tested whether CS effects on expectancy were mediated by SCR, we found that effect of cues on expectations was the same whether or not we control for cue effects on SCR (c = 0.14, STE = .02, p < .001; c′ = 0.14, STE = .02, p < .001) and there was no evidence of mediation (a × b = 0, p > .08). This provides further support for directionality of our model, that is, that cues affect expectations which in turn affect SCR.

FIGURE 5

Expectancy fully mediates differential response within participants who make ratings. Multilevel mediation revealed that trial‐by‐trial expectancy ratings fully mediated effects of the current contingencies on SCR across participants. We used bootstrap estimation to determine the significance of the mediation effect (Shrout & Bolger, 2002). Slope plots depict individual estimates in blue lines, with the 95% confidence interval depicted in the gray shaded area that surrounds the overall group effect. Analyses were conducted in the Multilevel Mediation Moderation Toolbox (Atlas et al., 2010; Wager et al., 2009). Upper left: There was a significant effect of current CS contingencies on subjective expectancy (i.e., Path a in the mediation framework). Upper right: There was a significant effect of expectancy on SCR, controlling for current CS contingencies (i.e., Path b in the mediation framework). Lower panel: There was a significant direct effect of Current CS contingencies on SCR, which was non‐significant when controlling for expectancy rating. There was a significant negative association between Path A and Path B coefficients, which suggests mediation was driven primarily by within‐subjects effects

Comparing learners and non‐learners

Our main analyses focus on quantitative models across all participants to assess how the act of providing expectancy ratings alters dynamic adaptive learning in response to reinforcement. However, classic approaches often average across responses regardless of time to discern whether individuals can be classified as learners (i.e., those who show elevated arousal in response to a CS+ relative to a CS−) or non‐learners (those who show no differences). While this approach has known limitations (Lonsdorf et al., 2017), it can still provide convergent information regarding the overall effects of expectancy ratings on learning. We used a differential response cut‐off of 0.05 µS during late acquisition to identify learners (see Figure S2). There were more learners in the Rating Group (26/40, or 65%) than the Viewing Group (18/40, or 45%), although differences were marginal based on a chi‐square test (χ 2 = 3.23, p = .072). There were no differences between Learners and Non‐learners or interactions between Group and Learning in the number of perceived reversals or affect ratings for either CS, but we did observe significant interactions with Learning Status when evaluating post‐task probability ratings (see Supplementary Results and Figure S3).

DISCUSSION

Studies of threat conditioning have provided vast insights into the mechanisms that underlie learning and memory by measuring various defensive responses, many of which are largely conserved across human and animal models. Here, we asked whether the uniquely human act of rating expectations alters learning and threat expression in the form of autonomic arousal. We found that making expectancy ratings during aversive reversal learning slowed dynamic learning, as measured by SCR to conditioned cues. Participants who made online expectancy ratings were slower to reverse conditioned responses when contingencies changed, relative to participants who underwent the task without making ratings. Within the Rating Group, expectancy ratings fully mediated the differential response in SCR, suggesting that cues affect expectations which in turn affect SCR. In this section we discuss these findings, their relationship with previous work, and questions that should be addressed in future work.

Expectancy ratings have traditionally been included in many threat conditioning studies as a measure of learning (Boddez et al., 2013). However, rating expectations engages cognitive processes that may not occur in the absence of ratings. In particular, providing subjective ratings requires attention, decision‐making, and probability inference. Our results indicate that these concurrent processes act to reduce the rate of simple associative learning, in particular making individuals less sensitive to changes in contingencies, that is, reversals. The group differences we observed are quite surprising, given the relatively high 50% reinforcement rate used. In fact, during piloting, we found that subjects who made expectancy ratings did not show SCR reversals at all when we used a 30% reinforcement rate, whereas this reinforcement rate was sufficient to induce repeated SCR reversals when expectancy ratings were not incorporated (Atlas et al., 2016). One possibility is that explicitly rating probabilities causes people to make higher order predictions that in turn guide attention. Numerous studies of associability indicate that attention can gate learning (Atlas et al., 2019; Li, Schiller, et al., 2011; Mackintosh, 1975; Pearce & Hall, 1980; Roesch et al., 2012), and that attention and learning rates decrease as an environment becomes more stable (Behrens et al., 2007; Browning et al., 2015). Individuals who explicitly believe the environment has stabilized may pay less attention to individual outcomes, and therefore may be slower to react to the contingency reversal, consistent with over‐learning. Relatedly, providing online expectancy ratings might have served as a distraction in Rating Group participants, reducing their attention to changes in the environment, or might have shifted individuals from automatic to elaborative processing. It is also possible that specific features of the task (e.g., stimulus duration, stimulus discriminability) moderate the extent to which providing online expectancy ratings alters attention. For example, our CS duration was rather short, although it was sufficient for subjects to provide expectancy ratings. Longer CS presentations might allow subjects to attend to subjective expectancy without time pressure or memory load, perhaps affording greater awareness to the contingencies. Future studies should directly measure the role of attention to differentiate between these alternatives and to understand how expectancy and attention interact to shape behavior and autonomic responses.

Interactions between expectancy and attention can also explain how our findings relate to previous work examining the relationships between expectancy ratings and physiological responses during threat conditioning (Perruchet, 2015; Ryan et al., 2021; Warren et al., 2014). In two studies using 100% reinforcement, participants who made expectancy ratings demonstrated enhanced fear acquisition and enhanced extinction retention based either on startle potentiation (Warren et al., 2014) or SCR (Ryan et al., 2021). In contrast, we combined partial reinforcement and reversal learning with a hybrid model of adaptive learning and found that the process of rating expectations reduced sensitivity to prediction errors and changes in context. We believe these findings can be easily reconciled based on differences in environmental volatility. If expectancy ratings reduced attention to individual outcomes and enhanced confidence in judgments, this would be beneficial in stable environments (e.g., tasks with 100% or 0% reinforcement during acquisition and extinction, respectively) but be deleterious in stochastic environments such as the partial reinforcement reversal task we used here (or a lower reinforcement rate with more reversals, as mentioned above). Future work should systematically manipulate reinforcement rate to formally measure the impact of expectancy ratings as a function of volatility.

While the between‐groups aspect of our study highlights how the inclusion of expectancy ratings can shape autonomic responses relative to learning without online ratings, our within‐subjects mediation approach also provides insight on the dynamic contribution of subjective expectancy ratings themselves. We found that subjective expectations fully mediated the differential SCR response within the Rating Group. These findings are relevant in light of previous work that compared the dynamics of expectancy ratings and autonomic responses within subjects and observed meaningful dissociations. In particular, studies of the so‐called Perruchet effect (Perruchet, 2015) and gamblers’ fallacy (Clark et al., 2002) indicate that the recent history of association has divergent effects on eye‐blink conditioning versus subjective expectancy. Unconditioned stimulus (US) recency is positively associated with the magnitude of the conditioned response, yet negatively related to expectancy; in other words, the greater the time since US presentation, the more people expect reinforcement, but the weaker the magnitude of the conditioned eye‐blink. This suggests expectancy ratings are sensitive to higher order beliefs such as the gamblers’ fallacy, but that conditioned eyeblink is not, which might support a dual process model. In exploratory analyses, we analyzed Rating Group responses as a function of US recency, and found that expectancy ratings did show a pattern that might be consistent with gamblers’ fallacy (i.e., higher expectancy following more unreinforced CS+ presentations; see Figure S4). However, consistent with our mediation analyses and a single process model, we did not see dissociations between SCR and expectancy ratings; both showed similar effects of US history, and in fact trial‐by‐trial expectancy ratings fully mediated the effect of conditioned cues on SCR. This suggests that SCR reflects subjective expectations (at least when ratings are made) and builds on other studies that have drawn into question the generalizability of the Perruchet effect (Weidemann, Satkunarajah, et al., 2016). Importantly, our study differed from previous work on the Perruchet effect (Perruchet, 2015; Perruchet et al., 2006; Weidemann et al., 2009) in many ways: We included CS− trials, we did not instruct participants about stimulus contingencies, and we measured SCR instead of eyeblink, although the Perruchet effect has been replicated with SCR in previous work (McAndrew et al., 2012). Future studies should use a fully balanced design with and without expectancy ratings to simultaneously evaluate the Perruchet effect within participants who provide expectancy ratings and to test whether the conditioned response differs as a function of whether ratings are collected. In addition, future studies should formally compare the impact of expectancy ratings on different measures of conditioning, as SCR, eyeblink, and startle have been shown to be differentially sensitive to expectancy and awareness in previous work (Clark et al., 2002; Hamm & Vaitl, 1996; Manns et al., 2002; Weike, 2005), although there is much debate in this area (Lovibond & Shanks, 2002; Mertens & Engelhard, 2020; Schultz & Helmstetter, 2010) and all measures are sensitive to instructed reversals in the absence of reinforcement, indicating that they can be shaped by higher order knowledge (Costa et al., 2015).

Our findings are consistent with a model whereby higher order processes like instructed knowledge, meta‐cognition, and executive function shape associative learning, that is, a single process model of threat learning (Grings, 1973; Mitchell et al., 2009). Which neural systems are likely to mediate the effects of expectancy rating on learning and autonomic responses? We and others have shown that instructed knowledge influences responses in the dorsolateral prefrontal cortex, which in turn shapes learning‐related responses in the striatum and ventromedial prefrontal cortex (Atlas et al., 2016; Li, Delgado, et al., 2011), although reversal learning in the amygdala depended on experiential learning rather than instruction (Atlas, 2019; Atlas et al., 2016, 2019). It is possible that explicit expectations, which engage meta‐cognition, act as an internally generated instruction. This seems plausible, based on prior work on amygdala lesion patients. Patients with amygdala lesions do not show SCRs during passive threat conditioning, but do show differential SCR when they make expectancy ratings (Coppens et al., 2009), indicating that higher order systems bypass the amygdala to interact with subcortical arousal circuits. Future work should directly compare neural mechanisms of threat learning with and without expectancy ratings to test whether explicitly rating expectations alters the brain responses that mediate aversive learning, or whether aversive learning systems respond similarly irrespective of whether subjects make ratings. In addition, in the present study both groups received general contingency instructions (i.e., that there would be a relationship between the stimuli and shocks) although they were not informed about the relationship. General contingency instructions differ from pure uninstructed learning, that is, when participants are not informed about relationships between CS stimuli and shock outcomes (Mertens et al., 2021). Future work should measure whether expectancy ratings interact with instructed knowledge and directly evaluate whether both types of higher order processes have similar mechanisms and downstream effects on adaptive learning.

While traditional approaches used behavioral measures as an index of emotion, affective scientists increasingly recognize the important distinction between defensive behaviors and subjective feelings (LeDoux, 2012; LeDoux, 2013). Our findings expand this conversation by demonstrating that explicitly reporting subjective predictions can actually alter physiological arousal. Importantly, it is not known whether our findings of reductions in adaptive learning are specific to expectancy ratings, or a consequence of providing any subjective rating during learning. For instance, does rating subjective fear alter learning in a different way from rating subjective expectancy? If the impact of expectancy ratings on learning is mediated by general cognitive processes such as decision making, meta‐cognition, and divided attention, then the type of rating might not matter, and all concurrent decisions might lead to reductions in adaptive learning. Alternatively, different types of concurrent ratings may have different effects on the trajectory of learning. For example, the act of rating expectancy might cause individuals to use higher order knowledge to focus on probability and reduce the impact of anxiety or fear, whereas rating subjective fear might heighten a sense of threat and anxiety and increase physiological arousal. Consistent with potential dissociations, recent findings indicate that the effect of CS‐US pairing on startle potentiation and amygdala activation is not mediated by subjective expectancy, but rather by subjective fear (Mertens, Braem, et al., 2018). Future work should use between‐subjects designs to directly compare how different subjective ratings impact associative learning to resolve these possibilities. Studies should also include an attentional control condition to control for the extent to which making any type of judgment influences the learning process. As mentioned above, future work should systematically manipulate and measure impacts of reinforcement rate, contingency instructions, and CS duration to test how these factors impact learning, attention, and the impact of expectancy ratings on multiple autonomic measures and defensive reactions. If all studies that incorporate subjective ratings also include a Viewing group, we will discover the conditions under which ratings impair or enhance learning as a field.

In conclusion, our study demonstrates that rating expectations alters learning in dynamic environments. We found that participants who provided expectancy ratings during aversive learning were slower to react when contingencies reversed, relative to participants who did not provide expectancy ratings. Furthermore, our mediation analyses indicated that subjective expectations directly shape autonomic responses within the group that provided ratings. Although the goal of learning is to generate predictions and expectations about outcomes in the environment, our work shows that making these expectations explicit can shape the dynamic process of learning itself, as measured by dynamic changes in anticipatory arousal. Studies of threat conditioning, threat related processing, and learning should quantify the extent to which concurrent measures like expectancy ratings may directly alter the behavior of interest.

AUTHOR CONTRIBUTIONS

Lauren Y. Atlas: Conceptualization; Data curation; Formal analysis; Funding acquisition; Investigation; Methodology; Project administration; Resources; Software; Supervision; Visualization; Writing—original draft; Writing—review & editing. Christina F. Sandman: Conceptualization; Data curation; Formal analysis; Investigation; Methodology; Project administration; Writing—original draft; Writing—review & editing. Elizabeth A. Phelps: Conceptualization; Funding acquisition; Project administration; Resources; Supervision; Writing—original draft; Writing—review & editing.

FIGURE S1 Learning parameters by estimation procedure

FIGURE S2 Responses during late acquisition used to define learners and non‐learners

FIGURE S3 Post task ratings as a function of learning status

FIGURE S4 Perruchet effect analysis within Rating Group participants

TABLE S1 Multilevel model measuring the effects of expectancy rating on aversive reversal learning

TABLE S2 Multilevel model measuring aversive reversal learning within Rating Group participants

TABLE S3 Multilevel model measuring aversive reversal learning within Viewing Group participants

TABLE S4 Posthoc pairwise comparisons from model measuring the effects of expectancy rating on aversive reversal learning

TABLE S5 Results of hybrid model fit to individuals

Click here for additional data file.