Reward and punishment sensitivity in children with ADHD: validating the Sensitivity to Punishment and Sensitivity to Reward Questionnaire for children (SPSRQ-C).

This study validates the Sensitivity to Punishment and Sensitivity to Reward Questionnaire for children (SPSRQ-C), using a Dutch sample of 1234 children between 6-13 years old. Factor analysis determined that a 4-factor and a 5-factor solution were best fitting, explaining 41% and 50% of the variance respectively. The 4-factor model was highly similar to the original SPSRQ factors found in adults (Punishment Sensitivity, Reward Responsivity, Impulsivity/Fun-Seeking, and Drive). The 5-factor model was similar to the 4-factor model, with the exception of a subdivision of the Punishment Sensitivity factor into a factor with 'social-fear' items and a factor with 'anxiety' items. To determine external validity, scores of three groups of children with attention deficit hyperactivity disorder (ADHD) were compared on the EFA models: ADHD-only (n = 34), ADHD and autism spectrum disorder (ADHD+ASD; n = 22), ADHD and oppositional defiant disorder (ADHD+ODD; n = 22). All ADHD groups scored higher than typical controls on Reward Responsivity and on the 'anxiety' factor (n = 75). The ADHD-only and ADHD+ODD group scored higher than other groups on Impulsivity/Fun-Seeking and Drive, while the ADHD+ASD group scored higher on Punishment Sensitivity. The findings emphasize the value of the SPSRQ-C to quickly and reliably assess a child's sensitivity to reinforcement, with the aim to provide individually-tailored behavioral interventions that utilize reward and reprimands.

Introduction

Based on operant condition principles many educational programs aim at promoting adequate behavior by using explicit rewards and ignoring inappropriate behavior. Anecdotal reports of parents, however, suggest that children with developmental problems such as attention deficit hyperactivity disorder (ADHD) respond differently to such strategies than their normal peers. Specifically, they seem to be relatively insensitive to negative feedback or reprimands while being oversensitive to rewards (see Luman et al. 2005, for an overview). It has been suggested that not only ADHD, but also other psychiatric conditions that frequently co-occur with ADHD, such as oppositional defiant disorder (ODD) and autism spectrum disorder (ASD) show alterations in sensitivity to reward and punishment (see Koot et al. 2008; Rommelse et al. 2011, for an overview). Consequently, in order to determine the future success of any educational or therapeutic intervention that systematically applies contingencies, the field would greatly benefit from tools that allow a quick and reliable assessment of a child’s individual sensitivity to reward and punishment.

Carver and White (1994) and Torrubia et al. (2001) independently developed a self-report measure consisting of a punishment and reward sensitivity scale, based on predictions from an influential theory postulated by Jeffrey Gray (Gray 1976, 1982). More recently, Gray has published a revised version of his theory (Gray and McNaughton 2000; but see also McNaughton and Corr 2004; Smillie et al. 2006), according to which there are three interactive, neurobiologically valid, systems that influence behavior. According to the theory conditioned and unconditioned appetitive signals of reward or non-punishment activate the behavioral activation system (BAS). Although the BAS is not functioning in isolation, the BAS initiates approach behavior or active avoidance and results in positive emotional experiences. The second system, the Flight, Fight and Freezing system (FFFS) is activated by conditioned and unconditioned aversive stimuli, novel stimuli, or non-rewards. The FFFS results in either behavioral activation of ‘Fight’ or ‘Flight’ responses or in ‘Freezing’. This system has been associated with feelings of rage and fear. According to the revised theory, the third system, the Behavioral Inhibition System (BIS) inhibits initiated BAS or FFFS behavior and is associated with feelings of anxiety. The BIS is activated by conflict between the BAS and FFFS. In the presence of reward the BIS inhibits the FFFS, favoring approach behavior, while in the presence of aversive stimuli the BIS inhibits the BAS, favoring escape behavior. Simultaneously, the BIS directs attention to the source of conflict when both rewarding and aversive stimuli are present in the same environment. In other words, the BIS is associated with risk-aversion and caution (Smillie et al. 2006).

The questionnaires developed by Carver and White (1994) and Torrubia et al. (2001) assess both the BAS (reward sensitivity) and FFFS (punishment sensitivity), the core elements of Gray’s model.1 The BAS scale of Carver and White is further subdivided in three factors: Impulsivity/Fun-Seeking, Reward Responsivity, and Drive. Colder and O’Connor (2004) adapted items from both questionnaires to develop a tool to be used for children aged 9 to 12. An exploratory factor solution of the items in the child version of the Sensitivity to Punishment and Sensitivity to Reward Questionnaire (SPSRQ-C) resulted in four factors similar to the factors identified by Carver and White (1994) in adults. However, the exploratory factor analysis was only performed on a small sample of 150 children with a small age-range (9–12 years old), without subsequent cross-validation. Furthermore, these questionnaires have not been externally validated in terms of a comparison between groups of children that show altered reinforcement sensitivity, such as children with ADHD.

Children with ADHD show difficulties in paying attention, they do not seem to listen when spoken to, and they get up from their seats or run around when this is not appropriate (APA 2000). Part of this behavior may be caused by a general lack of motivation to finish tasks, especially when tasks are boring or in the face of other attractive activities (Luman et al. 2005). During the past decades, several lines of experimental research have attempted to qualify the nature of reward and punishment sensitivity in ADHD (for an extensive overview, see Luman et al. 2005). Such studies show, in line with the theoretical models, that children with ADHD, compared to healthy controls, prefer small, immediate rewards over larger, delayed rewards (Rapport et al. 1986; Sonuga-Barke et al. 1992). In addition, compared to normal peers they appear to be more influenced by the last reward received than by the reinforcement history (Tripp and Alsop 1999). Children with ADHD also display larger improvements than controls in performance on a variety of cognitive tasks where responses are coupled with reward or response cost (Carlson and Tamm 2000; Konrad et al. 2000; McInerney and Kerns 2003).

Current theoretical models incorporated the ideas of Gray and suggest that ADHD is related to altered meso-limbic dopamine responsivity in reward-related circuits (Sagvolden et al. 2005; Tripp and Wickens 2008). These models propose a steeper decay of the value of reinforcement over time. A lower (intrinsic) value of future rewards is suggested to result in a relatively strong preference for immediate rewards and higher levels of impulsive behavior in ADHD. In addition, these models propose a smaller impact of the omission of rewards on behavior. This implies that behavioral responses are maintained by a child, even when these responses are not rewarded. This would result in chaotic and hyperactive behavior. According to the delay aversion hypothesis (Sonuga-Barke 2002, 2003), delay aversion, and thus the need for immediate rewards, is a developmental consequence of the failure of an impulsive child to engage effectively in delay-rich environments, resulting in attempts to avoid or escape delay.

It has been suggested that not only ADHD, but also other psychiatric conditions that frequently co-occur with ADHD, such as oppositional defiant disorder (ODD) and autism spectrum disorder (ASD) show alterations in sensitivity to reward and punishment. These disorders are highly comorbid with ADHD (up to 75% co-occurrence with ADHD; Spencer 2006; Sturm et al. 2004) and might result from a disturbed interaction between the FFFS (see Footnote 1) and BAS (Beauchaine et al. 2001). Children with ODD are thought to focus on reward, while ignoring signals of punishment, as explained by a predominant BAS (Newman and Wallace 1993). Similarly, Raine (1996) proposed that lack of fear and low autonomic arousal in antisocial behavior decreases their attention to threat related stimuli, such as punishment. Experimental studies, using a task in which the rate of winning gradually decreases, while the rate of losing increases, demonstrate that children with ODD keep responding to reward and ignore the increasing rate of punishment (Matthys et al. 2004; van Goozen et al. 2004). Additionally, in a decision making task, children with ODD, compared to typically developing peers, made more risky choices that were associated with large rewards, but also with large punishments (Luman et al. 2010).

Like children with ADHD, children with ASD seem to profit from reinforcement in clinical behavioral modification programs that were aimed to reduce their dysfunctional behavior (Matson et al. 1996). However, this may be true only for tangible reinforcement, since performance rates of children with ASD on a sustained attention task increased with tangible but not with non-tangible reinforcement such as praise (Garretson et al. 1990). Experimental literature on reward and punishment sensitivity in this group is conflicting. In one study, children with ASD compared to typical peers showed less efficient reinforcement learning in a decision-making paradigm (Johnson et al. 2006). Other reports, however, suggest that children with ASD do not differ from controls in reinforcement sensitivity (Antrop et al. 2006). Since ASD shares similarities with ADHD in brain pathology implicated in reinforcement processing, such as a deviant activation pattern of the amygdala and the orbito-frontal cortex (Amaral et al. 2003; Bachevalier and Loveland 2006) it is of interest to include this subgroup in our study.

The aim of the current study was to validate the SPSRQ-C using (a) a combination of both exploratory and confirmatory factor analyses in a group of 1234 children that attend regularly primary school (ages 6 and 13), investigating internal validity, and (b) compare different groups of children with ADHD who are predicted to show altered sensitivity to reward and punishment on the SPSRQ-C, investigating external validity. More specifically, scores on the SPSRQ-C were compared between children with ADHD-only, children with ADHD+ODD and children with ADHD+ASD. Our ultimate goal was to contribute to the development of an instrument to assess abnormalities in sensitivity to punishment and reward. Accurate assessment of a deviant behavioral response to reward and punishment may help to predict the effectiveness of their response to educational and therapeutic interventions that systematically apply such contingencies.

Method

Subjects

Study 1: Internal Validation

For the factor analysis, the Dutch version of the SPSRQ-C was sent out to a typical community sample of 5000 primary caretakers of children in the ages 6–13 years attending regular primary schools. In addition, caretakers were asked to fill out a series of other questionnaires. Parents of 1237 (mean age = 8.9, 48% boys) children filled out the SPSRQ-C.

Study 2: External Validation

Scores on the SPSRQ-C of 34 children with a clinical diagnosis of ADHD (mean age = 8.9 years; 31 boys, age 6–12 years), 22 children with a clinical diagnosis of ADHD+ODD (mean age = 7.5 years; 15 boys, age 6–12 years), and 22 children with a clinical diagnosis of ADHD + ASD (mean age = 10.6 years; all boys, age 7–12 years) according to the DSM-IV-TR (American Psychiatric Association 2000) were compared with 75 typically developing (TD) controls (mean age = 9.2 years, 55 boys, age 6–12 years). Table 1 displays the demographic characteristics of the groups. Children with ADHD and children with ADHD+ODD were recruited through the parent association for children with developmental disorders and through the university affiliated outpatient clinic for children with disruptive behavior problems. Children in the ASD group were recruited through our outpatient clinic for children with autism-spectrum disorder. Typical developing children were recruited through local regular primary schools.

Table 1

Demographic characteristics

	TD (n = 75)	ADHD-only (n = 34)	ADHD+ODD (n = 22)	ADHD+ASD (n = 22)	DBD group comparisonsa df (3,147)
Percentage Boys	73	90	68	100
Mean age (SD)	9.2 (1.3)	8.9 (1.5)	7.5 (1.5)	10.6 (1.3)
DBD parent
Inattention	2.4 (2.4)b	17.3 (6.1)	17.0 (4.5)	15.6 (5.8)	All clinical groups > TD***
Hyperactivity/Impulsivity	2.3 (2.1)	17.0 (4.9)	17.3 (4.5)	13.3 (7.3)	ADHD, ODD > ASD > TD***
ODD	1.1 (1.7)	9.4 (4.6)	12.5 (5.0)	7.6 (5.4)	ODD > ADHD > ASD > TD***
CD	0.5 (1.2)	2.4 (2.5)	3.6 (2.7)	1.6 (2.4)	ODD > ADHD, ASD > TD***
DBD teacher
Inattention	2.8 (4.4)a	12.7 (6.6)	15.6 (5.0)	10.9 (6.0)	All clinical groups > TD***
Hyperactivity/Impulsivity	1.9 (4.0)	12.9 (7.6)	13.3 (4.8)	8.8 (7.6)	ADHD,ODD > ASD > TD***
ODD	0.6 (1.6)	6.2 (5.1)	7.6 (5.4)	5.0 (5.2)	ODD > ADHD, ASD > TD***
CD	0.7 (2.1)	2.0 (2.4)	2.1 (2.1)	1.6 (2.9)	ADHD, ODD > TD***

ADHD Attention deficit/ hyperactivity disorder; ASD Autism spectrum disorder; CD Conduct disorder; DBD Disruptive Behavior Disorder Rating Scale; ODD Oppositional defiant disorder; TD Typically developing controls

aGroups were compared using ANOVA

bParent DBD of one child in the TD group and teacher DBD of one other child in the TD group was missing

***p < 0.001

All children with ADHD-only and ADHD+ODD met the following criteria: (1) a clinical diagnosis of ADHD or ADHD+ODD confirmed by the Diagnostic Interview Schedule for Children for DSM-IV, parent version (PDISC-IV; Shaffer et al. 2000), (2) a score that exceeded the 90th percentile on one of the ADHD scales (and the ODD scale for children with ADHD+ODD) of the parent version of the Disruptive Behavior Disorder rating scale (DBD), and (3) a score that exceeded the 80th percentile on one of the ADHD scales (and the ODD scale for children with ADHD+ODD) of the teacher version of the DBD (Oosterlaan et al. 2000; Pelham et al. 1992). The PDISC-IV is a widely used structured diagnostic interview based on a stringent diagnostic algorithm. For the ADHD-only group, eighteen children met criteria for ADHD combined type, nine children for ADHD inattentive type, and seven children for ADHD hyperactive/impulsive type. For the ADHD+ODD group thirteen children met criteria for ADHD combined type, five children for ADHD inattentive type, and four children for ADHD hyperactive/impulsive type. None of the children met criteria for conduct disorder (CD). The DBD consists of 42 items on a 4-point Likert scale (0 = not at all to 3 = very much). The scores on the scales may range from 0–27 (nine items) for the Inattention as well as the Hyperactivity/Impulsivity scales, 0–24 for the ODD scale and 0–48 for the CD scale. All children with ADHD-only scored >90th percentile on one of the ADHD scales of the parent version of the DBD and all (except one from which the teacher DBD score was missing) scored >80th percentile on one of the ADHD scales of the teacher version of the DBD. All children with ADHD+ODD had a score >90th percentile on the ODD scale of both the parent and teacher DBD.

In the ADHD+ASD group, the ASD diagnosis was based on extensive diagnostic assessment by a multidisciplinary autism expert team. The clinical diagnosis of ASD was confirmed by a score above the 80th percentile on the Children’s Social Behavior Questionnaire (CSBQ; Luteijn et al. 2002) as filled out by parents (15 boys even had a score above the 95th percentile). The sample consisted of two boys with autism, five with Asperger syndrome, and fifteen boys with a pervasive developmental disorder-not otherwise specified (PDD-NOS). In the ADHD+ASD group fourteen children fulfilled the ADHD criteria according to the PDISC-IV (five combined type, two hyperactive/impulsive type, and seven inattentive type); the other eight children all had scores above the 80th percentile at one of the ADHD scales of the parent DBD.

All children of the control group scored below the 90th percentile on all scales of the parent and teacher DBD. For all children (clinical groups and control children), IQ scores exceeded 70, there were no psychiatric disorders (other than ADHD, ODD, ASD in the clinical groups), there were no neurological disorders, no learning disabilities such as dyslexia, no sensory disorders, and no motor impairments.

SPSRQ-C

The children’s version of the questionnaire measuring sensitivity to punishment and reward (Colder and O’Connor 2004) contained 33 items that are rated by parents, and is divided in a Punishment Sensitivity scale (15 items), and three Reward Sensitivity scales: Reward Responsivity (7 items), Impulsivity/Fun-Seeking (7 items), and Drive (4 items) (see Appendix 1 for the items). Each item is scored on a 5-point Likert scale (1=strongly disagree, 5=strongly agree). Reliability of the questionnaire is 0.87 (coefficient alpha) for Punishment Sensitivity, 0.69 for Reward Responsivity, 0.76 for Impulsivity/Fun-Seeking, and 0.73 for Drive (Colder and O’Connor 2004). With permission of the authors, the questionnaire was translated into Dutch by an official translator, and translated back into English by a native English speaker. Discrepancies between the original English version and the Dutch version led to minor changes in the Dutch version.

Data Analysis

First, cases with more than 6 missing values were excluded from the sample. For the typical community sample (n = 1237), three cases were excluded, resulting in a total sample of 1234. Only 17 of the 1234 children had one or two missings: we imputed these remaining missing values by the average score of a specific item across all parent ratings. The pattern of missing items seemed random, as none of these values were missed by more than one participant.

For the purpose of cross validation two datasets were created. The total sample (n = 1234) was randomly split into two separate groups (A n = 617 and B n = 617). Exploratory factor analysis (EFA) was conducted on dataset A to identify factors that would optimally explain the covariation among the measures. We used the maximum likelihood procedure with promax rotation. In order to determine which EFA models were retained and used in the CFA, the EFA-derived models were evaluated against the theoretical model of Gray (Gray and McNaughton 2000). Confirmatory factor analyses (CFA) was conducted on the other dataset (Dataset B) to investigate which of the selected EFA derived models gave the best description of the current data.

The models were fitted to the covariance matrices using the maximum-likelihood procedure in LISREL 8.52 (Du Toit and Du Toit 2001). Since no clear consensus exists regarding the best goodness-of-fit indices for the evaluation of the CFA, multiple fit indices were used (Bollen 2002). In the current study, we present four frequently reported indices (see for details Bollen 2002; Schermelleh-Engel et al. 2004): χ2 goodness of fit test, the Root Mean Square Error of Approximation (RMSEA), the Comparative Fit Index (CFI), and the Akaike Information Criterion (AIC). A non-significant χ2 indicates an adequately fitting model, although this test is sensitive to sample size (which is large in the current study). An RMSEA value of 0.05 or smaller indicates a close fit, values between 0.05 and 0.08 represent a reasonable fit, values between 0.08 and 0.10 a mediocre fit, whereas values >0.10 are not acceptable. CFI values greater than 0.90 are considered as indicative of a good fit and higher values indicate a better fit. The lower the AIC value, the better a model fits. Finally, internal consistencies (coefficient alpha) of the model factors were explored, a higher consistency indicating a lower error variance. For group comparisons, a value >0.6 is acceptable.

For the clinical samples no cases were excluded due to missing values. The three clinical groups (34 ADHD, 22 ADHD+ODD and 22 ADHD+ASD) were compared to the typically developing children without behavioral problems (75 TD) using an ANOVA on scores of the selected EFA models derived from study 1. When there was a significant effect of group, group comparisons were performed using Tukey’s HSD procedure.

Results

Study 1: Internal Validation

Exploratory Factor Analysis on Dataset A

The dimensionality of the 33 items of the SPSRQ-C was explored using maximum likelihood factor analysis. The number of factors was determined using the scree-test as well as the substantive meaning of the factors. Four interpretable factor models (with 2, 3, 4, and 5 factors, respectively) were derived and each model is described below. The item loading on each of the factors is reported in the Appendix (factor loading >0.10 are included).

A model with two factors (called model EFA-2) explained 22.4% of the item variance. Factor 1 may be labeled Punishment Sensitivity and consisted of all 15 items from the original Punishment Sensitivity scale described by Colder and O’Connor (2004), although one item loaded somewhat higher on Factor 2 of the EFA-2 model (see Appendix 1). Factor 2 may be labeled Reward Sensitivity (or BAS) and consisted of all 18 items of the original Reward Sensitivity scale as described by Colder and O’Connor (2004).

A model with three factors (model EFA-3) accounted for 36.1% of the item variance. Factor 1 labeled Punishment Sensitivity consisted of items from the original Punishment Sensitivity scale. Factor 2 may be labeled Reward Responsivity & Impulsivity/Fun-Seeking and consisted mainly of items of the original scales Reward Responsivity and Impulsivity/Fun-Seeking (Colder and O’Connor 2004; 13 from the original 14 items). Factor 3 labeled Drive was composed of items related to the original Drive scale (Colder and O’Connor 2004).

A model with four factors (model EFA-4) explained 40.8% of the item variance. The first factor labeled Punishment Sensitivity was composed of items of the original Punishment Sensitivity scale, although three original Punishment Sensitivity items loaded even higher on another factor. Factor 2 labeled Reward Responsivity consisted of all items of the original Reward Responsivity scale combined with four (out of seven) items related to the Impulsivity/Fun-Seeking scale and two Punishment Sensitivity items. Factor 3 labeled Impulsivity/Fun-seeking consisted of the remaining items of the original scale Impulsivity/Fun-Seeking as well as one item of the original Punishment Sensitivity scale and one item of the original Drive scale (although this item loaded even higher on factor 4). Finally, factor 4 labeled Drive was composed of all items related to the original Drive scale, although one item loaded just as high on factor 2.

A model with five factors (model EFA-5) explained 49.5% of the variance. Factor 1 may be labeled FFFS and contained of 9 (out of 15) items from the original Punishment Sensitivity scale, but only the so-called ‘social fear items’. These items were mostly related to fear or discomfort in social situations and active avoidance. Factor 2 labeled Reward Responsivity consisted of five (out of seven) items of the Reward Responsivity scale and one item of the original Impulsivity/Fun-Seeking scale. Factor 3 labeled Impulsivity/Fun-Seeking consisted of three items related to Impulsivity/Fun-Seeking and one item related to the original Punishment Sensitivity scale, similarly to factor 3 of EFA-4 model. Factor 4 labeled Drive was made up of all items related to the original Drive scale (although one item also loaded on Factor 3) and one item related to Reward Responsivity. Factor 5 may be labeled BIS and consisted of the remaining items from the original Punishment Sensitivity scale, the so-called ‘general anxiety items’ as well as two items of the original Impulsivity/Fun-Seeking scale and one item of the original Reward Responsivity scale.

Model Selection

Three models were selected. The first model was selected based on the theory that the FFFS and BAS may act as independent subsystems (Corr 2002). This was the EFA-2 model with two factors (Punishment Sensitivity and Reward Sensitivity) that was highly similar to the 2-factor solution derived by Colder and O’Connor (2004). Secondly, the EFA-4 model was selected that was highly similar to the 4-factor solution derived by Colder and O’Connor (including a Punishment Sensitivity, Reward Responsivity, Impulsivity/Fun-Seeking and Drive scale). A third model with five factors was selected, the EFA-5 model. The EFA-5 model was similar to the EFA-4 model, with the exception of a subdivision of the Punishment Sensitivity factor into a so-called ‘FFFS’ and a ‘BAS’ factor. According to Gray’s theory (Gray and McNaughton 2000), the FFFS is particularly associated with rage and fear and behavioral initiation (e.g., children show active avoidance of an aversive stimulus / start a fight when confronted with an aversive stimulus), while the BIS (monitoring system over FFFS and BAS) is associated with anxiety and conflict reduction. Factor 1 of the EFA-5 model consists of items that are related to ‘fear or discomfort and active avoidance in social situations’, which shows theoretical overlap with the FFFS. Factor 5 of the EFA-5 model consists of items that are related to ‘general anxiety and difficulty with behavioural modulation of anxiety’, and shows theoretical overlap with the BIS.

Since EFA-2 and EFA-4 models are highly similar to the original 2-factor model and original 4-factor model of Colder and O’Conner (2004), the original factor models were left out of the CFA.

Confirmatory Factor Analysis on Dataset B

To investigate whether the three selected EFA models provided a good fit to the data, a CFA was conducted with these models. If an item had an absolute loading of 0.30 or higher on a particular factor, then this item was assigned to that factor. This implied that some items were assigned to more than one factor. When an item had no absolute factor loading higher than 0.30, this item was assigned to the factor for which it had the highest absolute loading (see Appendix).

According to RMSEA none of the EFA models gave a ‘good’ fit to the dataset. However, the fit indices RMSEA and CFI and AIC were slightly better for the EFA-4 model and the EFA-5 model than the EFA-2 model (see Table 2). Internal consistency of all factors of the EFA-4 and EFA-5 models are greater than 0.65 for all factors. Coefficient alpha’s of the EFA-4 model were 0.86, 0.83, 0.72 and 0.71 for Punishment Sensitivity, Reward Responsivity, Impulsivity/Fun-Seeking, and Drive respectively. Coefficient alpha’s of the EFA-5 model were 0.79, 0.78, 0.70, 0.65, 0.76 for FFFS, Reward Responsivity, Impulsivity/Fun-Seeking, Drive, and BIS respectively. The best fitting model, taking parsimony into account, would be the EFA-4 model, which is highly similar to the original 4-factor model of Colder and O’Connor (2004). However, the best fitting model, based on theoretical grounds, would be the EFA-5 model, since it showed most theoretical overlap with the model of Gray (Gray and McNaughton 2000).

Table 2

Fit indices for the exploratory factor analyses models of the SRSPQ-C (Dataset B)

Model	df	χ 2	p-value	RMSEA	CFI	AIC
EFA-2	492	2346.4	<0.001	0.09	0.91	2875.4
EFA-3	489	2118.7	<0.001	0.08	0.92	2501.5
EFA-4	485	2110.7	<0.001	0.08	0.92	2459.2
EFA-5	481	2116.1	<0.001	0.08	0.92	2522.2

AIC Akaike Information Criterion; CFI Comparative Fit Index; EFA exploratory factor analysis; RMSEA Root Mean Square Error of Approximation

For EFA-5, as expected, the FFFS and BIS scale showed a high correlation (r = 0.54), indicating that the defensive system and the anxiety system are related (see Gray and McNaughton 2000). In addition, there were positive correlations between the BIS factor and the Impulsivity/Fun Seeking scale (r = 0.45), and positive correlations between the BIS factor and the Reward Responsivity scale (r = 0.45). These positive relations are in line with the theory that the BIS system mediates between the defensive approach and reward-related approach (Gray and McNaughton 2000). The positive correlation between the FFFS and Reward Responsiveness (r=12) suggests that sensitivity to reward (or signals of non-punishment) is related to sensitivity to aversive stimuli (or non-rewards). This indicates that, in general, children who are more sensitive to rewards are also more sensitive to aversive stimuli, arguing against the suggestion that the FFFS and BAS act as independent subsystems (Smillie et al. 2006).

Study 2: External Validation

Group Comparison on the Selected EFA Models

Table 3 shows the outcome of the group comparisons between the clinical groups and TD group. Since the EFA-2 and EFA-4 models were highly similar to the original 2-factor model and original 4-factor model of Colder and O’Conner (2004), the group comparisons for the original factor models were not reported here. The group comparisons on all EFA models, including the EFA-3 model, are displayed in Table 3.

Table 3

Group comparisons on the factors derived by the exploratory factor analyses of the SPSRQ-C

Model group	TD (n = 75)	ADHD-only (n = 34)	ADHD+ODD (n = 22)	ADHD+ASD (n = 22)	Group comparisona df (3,147)	Tukey HSD
EFA2-1 (PunSens)	2.4 (0.6)	2.6 (0.6)	2.7 (0.5)	2.7 (0.4)	2.9*	ASD > TDb
EFA2-2 (RewSens or BAS)	2.7 (0.5)	3.5 (0.5)	3.5 (0.6)	3.1 (0.7)	25.2***	ADHD, ODD > ASD > TD
EFA3-1 (PunSens)	2.4 (0.6)	2.6 (0.6)	2.7 (0.5)	2.7 (0.4)	2.9*	ASD > TDb
EFA3-2 (RewResp, Imp/FunSeek)	2.5 (0.5)	3.4 (0.5)	3.4 (0.6)	3.1 (0.7)	32.0***	ADHD, ODD, ASD > TD
EFA3-3 (Drive)	3.0 (0.7)	3.5 (0.7)	3.6 (0.7)	2.7 (0.9)	9.6***	ADHD, ODD > ASD, TD
EFA4-1 (PunSens)	2.3 (0.6)	2.6 (0.6)	2.7 (0.5)	2.7 (0.4)	3.6**	ASD > TD
EFA4-2 (RewResp)	2.8 (0.5)	3.7 (0.5)	3.7 (0.6)	3.5 (0.7)	27.0***	ADHD, ODD, ASD > TD
EFA4-3 (Imp/FunSeek)	1.8 (0.6)	2.7 (0.7)	2.8 (0.8)	2.2 (0.8)	17.6***	ADHD, ODD > ASD, TD
EFA4-4 (Drive)	2.9 (0.7)	3.5 (0.8)	3.5 (0.7)	2.7 (1.0)	9.8***	ADHD, ODD > ASD, TD
EFA5-1 (FFFS)	2.4 (0.7)	2.5 (0.6)	2.5 (0.6)	2.7 (0.5)	1.3 ns	–
EFA5-2 (RewResp)	2.6 (0.7)	3.5 (0.8)	3.4 (0.7)	3.4 (0.8)	16.8***	ADHD, ODD, ASD > TD
EFA5-3 (Imp/FunSeek)	1.8 (0.6)	2.7 (0.7)	2.8 (0.8)	2.2 (0.8)	17.6***	ADHD, ODD > ASD, TD
EFA5-4 (Drive)	3.0 (0.7)	3.5 (0.7)	3.6 (0.6)	2.7 (0.8)	10.2***	ADHD, ODD > ASD, TD
EFA5-5 (BIS)	2.6 (0.5)	3.2 (0.6)	3.4 (0.5)	3.3 (0.6)	17.9***	ADHD, ODD, ASD > TD

ADHD Attention deficit/ hyperactivity disorder; ASD Autism spectrum disorder; CD Conduct disorder; ODD Oppositional defiant disorder; TD Typically developing controls

aGroups were compared using ANOVA. When overall group effects were significant, post-hoc group comparisons were performed using Tukey HSD

b ASD and TD comparison, p < 0.10

* p < 0.05, **p < 0.01, ***p < 0.001

When comparing groups on the EFA-2 model, children with ADHD+ASD scored higher than TD children on the Punishment Sensitivity factor, although this comparison was not significant (p = 0.06). Other group comparisons for this factor were not significant. On the Reward Sensitivity (or BAS) factor all clinical groups scored higher than the TD group (p-values <0.05), with the ADHD and ADHD+ODD group scoring even higher than the ADHD+ASD group (p-values <0.05).

When comparing groups on the EFA-4 model, children with ADHD+ASD scored higher than TD children on Punishment Sensitivity (p = 0.04). The other group comparisons for this factor were not significant. On Reward Responsivity all clinical groups scored higher than the TD group (p < 0.001). On Impulsivity/Fun-Seeking as well as on Drive both the ADHD and ADHD+ODD group scored higher than the ADHD+ASD and TD group (p < 0.05 and p < 0.001, respectively).

When comparing children on the EFA-5 model, the significant group difference between the ADHD+ASD and TD group on Punishment Sensitivity disappeared when groups were compared on the FFFS factor that included only the ‘social fear’ items (p = 0.21). On Reward Responsivity all clinical groups scored higher than the TD group (p < 0.001). On Impulsivity/Fun-Seeking as well as on Drive both the ADHD and ADHD+ODD group scored higher than the ADHD+ASD and TD group (p < 0.05 and p < 0.001). Finally, on the BIS factor that included items related to ‘general anxiety’ all clinical groups scored higher than the TD group (p < 0.001).

Discussion

The goal of the current study was to validate the Dutch version of the SPSRQ-C in children between the age of 6 and 13 years. Specifically, our two aims were (a) to investigate internal validity using a combination of EFA and CFA in a large sample of children with a wide age range, and (b) to investigate external validity by comparing children with ADHD-only, children with ADHD+ODD and children with ADHD+ASD who are all predicted to show altered sensitivity to reward and punishment on the SPSRQ-C, to a group of typically developing children.

The exploratory factor analysis of the Dutch SPSRQ-C resulted in a factor solution that contained 2–5 factors and explained between 22 and 50% of variance. From these models, three models were retained based on their theoretical validity. The confirmatory analysis determined that the 4-factor model and the 5-factor model were best fitting, explaining 41% and 50% of the variance respectively. The 4-factor solution of the Dutch version of the SPSRQ-C and the original version of Colder and O’Connor (2004) converge to a large extent. The first factor of our EFA-4 model consisted of items that are part of the original Punishment Sensitivity scale, the second factor consisted of all items that are part of the original Reward Responsivity scale (although this factor also included items of the original Impulsivity/Fun-Seeking scale), the third factor consisted of the remaining items of the original Impulsivity/Fun-Seeking scale, and the fourth factor was made up of all items of the original Drive scale. These highly consistent findings with Colder and O’Connor (2004) are promising, confirming the construct validity of the SPSRQ-C.

The small differences between the current EFA-4 model and the original 4-factor solution were caused by items from the or iginal Impulsivity/Fun-Seeking scale that were now divided over two factors in the EFA-4 model. These differences could be explained in several ways. First, an extension of the age range studied (now between 6–13 years old) may have resulted in differences between the current and original EFA derived models, since changes may occur in sensitivity to reward and punishment when children grow older (Crone et al. 2005; van Leijenhorst et al. 2010). Second, small differences in interpretation of the items by parents could have resulted in different factor loadings. For example, three items of the original Punishment Sensitivity scale that loaded high on the Punishment Sensitivity factor of the EFA-4 model, loaded even higher on other factors in the EFA-4 model. One of these items (e.g., ‘Criticism or scolding hurts your child very much’) was interpreted in Dutch as a child’s response to criticism, rather than the emotional impact of criticism on the child. This may explain the high loading of this item on Reward Responsivity. Finally, there may be small differences in the statistical procedure used to run the EFA, resulting in slightly different factors outcomes. For example, item 22 of the original Impulsivity/Fun-seeking scale loaded almost as high on the original Reward Responsivity scale (see Colder and O’Connor 2004). In our EFA-4 model, this item loaded highest on the Reward Responsivity factor.

The EFA-5 model was almost identical to the EFA-4 model, except for the Punishment Sensitivity factor that was divided in two separate factors. The factor that was theoretically related to the FFFS of Gray consisted mainly of items related to fear or discomfort and active avoidance in social situations (e.g., ‘When your child is in a group, they try to stand out as the smartest or the funniest’). The factor that was theoretically related to the BIS of Gray consisted of items related to general anxiety and difficulty with the modulation of anxiety (e.g., Your child is often afraid of new or unexpected situations). In a recent paper, Colder and colleagues (Colder et al. 2011) performed a factor analysis on the original 48-item pool of the SPSRQ-C (Colder and O’Connor 2004). Similar to the observations in the current study, Colder et al. (2011) the factor Sensitivity to Punishment split into two factors with items related to ‘fear/shyness’ and items related to ‘anxiety’. These findings further confirm the validity of our EFA-5 model.

From the perspective of parsimoniousness one would select the EFA-4 model as the optimal solution. However, from the perspective of construct validity, the EFA-5 model would be preferred, since this model was theoretically most closely related to Gray’s reinforcement theory and allowed separation of the FFFS and the BIS. Moreover, this model offers a more encompassing picture of reward and punishment sensitivity in children.

Group Comparisons

When the clinical groups were compared on Punishment Sensitivity, children with ADHD+ASD scored higher than TD children, regardless of the model solution that was chosen to assign items to this factor. In the EFA-5 model where the Punishment Sensitivity factor was subdivided into a FFFS and BIS factor, the difference between the ADHD+ASD and TD group disappeared for the FFFS factor. All clinical groups scored higher than TD children on the BIS factor that included the items related to ‘general anxiety’. These results suggest that children with ADHD+ASD are more sensitive than typical children for punishment signals, and our 5-factor solution suggest that this is particularly related to a higher score on the ‘general anxiety’ items. An increased sensitivity to punishment in children with autism-related symptoms, is in line with the observation that ASD is highly comorbid with symptoms of anxiety (Leyfer et al. 2006; South et al. 2011). Additionally, reports that children with ASD show more rigid behavior indicate a smaller sensitivity to signals of punishment that call for a change in behavior (Geurts et al. 2009; Yerys et al. 2009).

The Punishment Sensitivity findings argue against the idea that children with ADHD or ADHD+ODD are less sensitive to punishment than typical children, despite some experimental evidence supporting this idea (Luman et al. 2010; van Meel et al. 2005). Similarly, these findings argue against theoretical models that suggests a dampened punishment sensitivity as a core feature of ADHD or ADHD+ODD (Newman and Wallace 1993; Raine 1996). Children with ADHD, ADHD+ASD, and ADHD+ODD obtained similar ratings on the FFFS factor than typical children, indicating that their response to conditioned and unconditioned aversive stimuli is intact. On the other hand, these children do show higher ratings on the BIS factor that includes items related to ‘general anxiety’. These findings are in line with earlier studies (Boylan et al. 2007; Jarrett and Ollendick 2008). According to the model of Gray (Gray and McNaughton 2000; McNaughton and Corr 2004; Smillie et al. 2006), the BIS is particularly related to conflict resolution and behavioral modulation of anxiety, rather than active avoidance in response to punishment signals (associated with FFFS). These findings suggests that children with ADHD show problems with conflict resolution when signals of both punishment and reward are available, which is often the case in daily life.

In all three selected EFA models the clinical groups scored higher than TD children on Reward Responsivity, as well as the composite BAS factor of EFA-2 model. The high score on Reward Responsivity confirms altered reward sensitivity in the clinical groups and is in line with experimental studies (Garretson et al. 1990; Luman et al. 2005, 2010; Matthys et al. 2004). Further, children with ADHD-only and ADHD+ODD scored higher on Impulsivity/Fun-Seeking than TD children. A higher score on Impulsivity/Fun-Seeking for children with ADHD-only and ADHD+ODD is not surprising, since these children share impulsivity as a core symptom. The difference between the ADHD+ASD and TD group on Impulsivity/Fun-Seeking was not significant, suggesting that fun seeking, as part of reward sensitivity, is less profound in the ADHD+ASD group. In line with these findings, the ADHD+ASD group scored lower than the ADHD-only and ADHD+ODD groups on the composite BAS factor of the EFA-2 model.

On the Drive scale the ADHD+ASD group scored lower than the other clinical groups. An urge for high social status and reward-driven behavior is common in ADHD and ADHD+ODD, and may be caused by a compensatory reaction to a lack in self-confidence (Hoza et al. 2002, 2004). A Drive score in the ADHD+ASD group similar to TD children is in line with the common observation that social valuation may not important for children with ASD (Garretson et al. 1990).

In summary, the findings suggest that children with ADHD and related problems show a divergent pattern on the SPSRQ-C compared to TD children. Some clinically-referred children showed an increased sensitivity to both punishment and reward, while other children showed an increased sensitivity to reward, while being normally sensitive to punishment.

Limitations

One of the limitations is the content of the BIS factor of the EFA-5. The BIS factor was related to ‘general anxiety items’ that included a response to an unknown threat. The BIS factor was confounded, however, with three items from the original BAS scale (item 4: your child enjoys being the center of attention, item 20: your child has difficult ending a fun activity, item 24: your child has difficulty staying focused on school work in the presence of an attractive alternative). This indicates that our BIS factor is not a ‘pure’ measure of anxiety. An option is to delete these items from the questionnaire so that the BIS consists of a total of 6 items.

Secondly, since we have included only a small group of clinical children with ADHD, children with ADHD+ODD, and children with ADHD+ASD, we were not able to investigate the unique contribution of ADHD, ODD and ASD symptoms on the observed sensitivity to reward and punishment. Future studies should therefore assess these symptoms in a large typical and/or clinical sample to further explore it’s relation with reward and punishment sensitivity.

Conclusion

The Dutch SPSRQ-C seems a valid instrument to assess reward and punishment sensitivity in children. The comparisons between children with ADHD with and without comorbid ODD or ASD suggest that children with ADHD display an heightened sensitivity to reward compared to typical controls, while a heightened sensitivity to punishment was displayed particularly by children with ADHD+ASD. These findings emphasize the potential use of the SPSRQ-C in assessing a child’s sensitivity to reward and punishment with the aim to provide individually-tailored behavioral interventions. For example, in the popular positive parenting program (Triple P; Thomas and Zimmer-Gembeck 2007), adequate behavior is reinforced by using explicit rewards while undesired or inadequate behavior is ignored (but not punished). Such an approach would be highly effective for children with ADHD who show a heightened sensitivity to reward, as observed in the current study. This is in line with a meta-analysis of 77 studies (Kaminski et al. 2008) evaluating the effectiveness of parent training programs in reducing children’s externalizing behaviors showing largest effect sizes for those programs that actively taught parents how to demonstrate enthusiasm and positive attention for appropriate behavior. Oversensitivity to punishment such as observed here for children with ADHD+ASD suggests that parents, teachers and clinicians should be cautious using reprimands.

Future external validation studies could correlate the SPSRQ-C with experimental measures of punishment and reward sensitivity. One specific option is to further explore the EFA-5 model, in which the Punishment Sensitivity scale is divided in a FFFS and BIS factor, by correlating these factors with experimental measures of punishment sensitivity (e.g., such as done by Colder et al. 2011).

Table 4

Item weights of the solutions of the explorative factor analyses

	Factor solutions	EFA-2		EFA-3			EFA-4				EFA-5
	Items	1	2	1	2	3	1	2	3	4	1	2	3	4	5
2	Your child prefers not to ask for something when they are not sure they will obtain ita	0.26		0.23	0.12		0.20		0.17		0.26	0.17	0.18		–0.15
5	Your child is a shy persona	0.59	–0.25	0.65	–0.24		0.68		–0.17		0.69		–0.18
7	Whenever possible, your child avoids demonstrating their skills for fear of being embarrasseda	0.61		0.54		–0.11	0.52			–0.11	0.45	0.12		–0.14
9	When in a group, your child has difficulty thinking of something to saya	0.69		0.69			0.68				0.60				0.15
12	Your child is often afraid of new or unexpected situationsa	0.64		0.59	0.17		0.61	0.26			0.32				0.62
15	Whenever they can, your child avoids going to unfamiliar placesa	0.61		0.60	0.11		0.60	0.12			0.39				0.43
17	Your child often worries about things they said or dida	0.35	0.26	0.34	0.25	0.11	0.35	0.27		0.10	0.22			0.13	0.30
18	It is difficult for your child to talk with someone they do not knowa	0.68	–0.18	0.82	–0.27	0.22	0.84	−0.16	−0.11	0.23	0.82		−0.11	0.23
19	Your child generally tries to avoid speaking in groupsa	0.72	−0.21	0.81	−0.24	0.15	0.80	−0.24		0.15	0.82			0.13
21	Your child could do more things if it were not for their feara	0.67	0.10	0.56	0.27	−0.12	0.54	0.23		−0.12	0.29			−0.12	0.47
23	Your child is afraid of many things compared to other children their agea	0.68		0.54	0.26	−0.19	0.49		0.23	−0.20	0.20	−0.14	0.25	−0.21	0.49
26	Your child often refrains from doing something they like in order not to be rejected or disapproved of by othersa	0.47	0.25	0.38	0.34		0.27		0.58		0.20		0.60
29	Your child often refrains from doing something because of fear of being embarrasseda	0.68		0.56	0.27	−0.15	0.51	0.11	0.22	−0.15	0.42	0.16	0.24	−0.18	0.13
31	If your child thinks that something unpleasant is going to happen, they get pretty worked upa*	0.40	0.19	0.32	0.28		0.36	0.43	−0.17				−0.19		0.64
33	Criticism or scolding hurts your child very mucha*	0.29	0.32	0.23	0.36		0.25	0.42			0.13	0.24			0.31
11	The possibility of obtaining social status moves your child to action, even if this involves not playing fairb		0.45		0.40	0.13			0.64	0.13			0.65	0.13	−0.22
14	Your child often has trouble resisting the temptation of doing forbidden thingsb	0.13	0.52		0.51				0.59		−0.11		0.60	0.11
20	Your child has a lot of difficulty ending a fun activityb	0.13	0.43	0.12	0.37	0.15	0.14	0.40		0.15				0.21	0.41
22	Your child sometimes does things for quick rewardb	0.12	0.65		0.68			0.54	0.21			0.56	0.22
24	Your child has difficulty staying focused on their school work in the presence of an attractive alternativeb	0.22	0.38	0.12	0.45			0.35	0.15			0.11	0.15		0.35
25	Your child engages in risky behavior to obtain a rewardb	0.18	0.45	0.12	0.46				0.75				0.76
32	Your child craves excitement and new sensationsb*	−.39	0.38	−0.34	0.20	0.22	−0.33	0.17		0.22	−0.18	0.23		0.25	−0.22
1	The good prospect of obtaining a reward motivates your child strongly to do some thingsc		0.49		0.52			0.61			0.11	0.78	−0.12		−0.11
3	Your child often does things to be praisedc		0.53		0.62			0.60				0.65
4	Your child enjoys being the center of attentionc	−0.33	0.66	−0.36	0.54	0.17	−0.35	0.48		0.17	−0.47	0.11		0.25	0.34
6	When your child is in a group, they try to stand out as the smartest or the funniestc	−0.21	0.61	−0.15	0.41	0.33	−0.12	0.42		0.33	−0.17	0.15		0.40	0.23
8	When your child gets something they want, they feel excited and energizedc*		0.35		0.39			0.60	−0.25			0.45	−0.25		0.23
10	Your child does a lot of things for approvalc	0.15	0.60		0.61			0.52	0.13			0.46	0.14
13	Does your child generally prefer activities that involve immediate rewardc	0.19	0.60		0.69			0.58	0.16			0.61	0.17
16	Your child likes to compete and do everything they can to wind		0.56		0.22	0.58		0.18		0.58	0.11			0.65
27	Your child likes competitive activitiesd	−0.20	0.39			0.77	0.13	−0.10		0.77	0.23	−0.12		0.82	−0.11
28	Your child would like to be a socially powerful persond	−0.16	0.39		0.22	0.27		0.29		0.28		0.18		0.32
30	Your child likes displaying their physical abilities even though it may involve dangerd	−0.11	0.45		0.20	0.41			0.35	0.41		−0.11	0.35	0.46

Only factor loadings higher than |0.10| are displayed. Factor loadings higher than |0.30| are printed in bold, and factor loadings higher than |0.50| are underlined

aItem from the Sensitivity to Punishment scale (Colder and O’Connor 2004)

bItem from the Impulsivity/Fun-Seeking scale (Colder and O’Connor 2004)

cItem from the Reward Responsivity scale (Colder and O’Connor 2004)

dItem from the Drive Scale (Colder and O’Connor 2004)

* items from the BIS/BAS scales (Carver and White 1994) that were added to 29 items of the modified version of the SPSRQ (Torrubia et al. 2001)