Do online assessed self-report and behavioral measures of impulsivity-related constructs predict onset of substance use in adolescents?

INTRODUCTION: To prospectively predict the onset of use of alcohol, cigarettes and marijuana among Dutch adolescents, using behavioral and self-report measures of impulsivity-related facets. Specifically, we investigated whether behavioral measures of impulsivity predicted the onset of substance use above and beyond self-report measures of impulsivity and sensation seeking in an online sample.
METHODS: Self-report and behavioral data from 284 adolescents (195 girls, mean age = 14.8 years, SD = 1.26) were collected at four time points over a period of two years, using an online survey system. Impulsivity-related facets were assessed at time point 1 with the Delay Discounting Task, the Balloon Analogue Risk Task and the Passive Avoidance Learning Task. We conducted logistic regression analysis to examine whether behavioral and self-report measures uniquely predicted onset of alcohol use, heavy episodic drinking, smoking and marijuana use.
RESULTS: Onset of cigarette smoking was associated with behavioral assessment of impulsive decision making, but not after controlling for self-reported impulsivity and sensation seeking. Behavioral measures were sometimes associated with, but appeared not to prospectively predict, the onset of substance use in this online sample after controlling for self-report measures.
CONCLUSIONS: Based on the present results, the added value of online behavioral assessment of impulsivity-related factors in the prediction of onset of substance use was not confirmed. We suggest that factors specific to each behavioral task underlie their lack of prediction and suggest that future research addresses limitations of current behavioral tasks to increase their validity in online testing.

Introduction

Impulsivity and sensation seeking have been related rather consistently to adolescent substance use (Christiansen et al., 2012, Fernie et al., 2013, Malmberg et al., 2010, Nees et al., 2011). Although there is some evidence that adolescent substance use may result in increased impulsivity during adulthood (review: De Wit, 2009, Nasrallah et al., 2009), most empirical support has been obtained for the reverse relationship; impulsivity predicting later problems with substance use (Fernie et al., 2013, Verdejo-García et al., 2008). Impulsivity is a multifaceted concept, and research has shown incremental validity of behaviorally measured impulsivity in combination with sensation seeking in the prediction of alcohol use in laboratory-based studies (Castellanos, 2009, Nees et al., 2011). Behavioral measurement is potentially insightful in cases where participants may have limited insight into their behavior, but tends to require more elaborate assessment. Because of the time-consuming process of behavioral assessment, the present study tested whether behavioral assessment of impulsivity-related constructs prospectively predicted substance use (weekly alcohol use, heavy episodic drinking, smoking of cigarettes and marijuana) as well as above and beyond self-report measures given online assessment. Although online behavioral measurement was shown to be promising in early studies (Houben & Wiers, 2008), very few studies to date have attempted it, meaning that their potential as a tool for mass-assessment among adolescents at risk for substance use is underexplored.

A recent review (Cyders & Coskunpinar, 2011) indicated that behavioral measures of impulsivity as a whole do not correlate with self-report measures of impulsivity and sensation seeking. This suggests that the two types of measures may capture a unique variance in the construct of impulsivity, which may prove relevant to impulsivity's effect on substance use behavior. Behavioral measures of impulsivity reflect a number of processes, including: delay discounting (Castellanos, 2009, De Wit, 2009, Fernie et al., 2010), risk-taking (Fernie et al., 2010, Fernie et al., 2013, Lejuez et al., 2002, MacPherson et al., 2010, Pleskac et al., 2008) and reward–response bias (Castellanos‐Ryan et al., 2011, Castellanos, 2009). Delay discounting refers to the propensity to make decisions based on the prospect of immediate versus delayed reward (Fernie et al., 2010, Verdejo-García et al., 2008). It has been found to predict alcohol use in adolescents (Fernie et al., 2013, Field et al., 2007), cigarette smoking status in adults (Reynolds, Karraker, Horn, & Richards, 2003) and adolescents (Kollins, 2003), as well as marijuana use in adolescents (Kollins, 2003). However, other studies found no relation between delay discounting and alcohol use (e.g. Fernie et al., 2010). Risk-taking refers to the propensity to take risks in potentially rewarding situations, proximal to real-life situations. Risk-taking, often measured with the Balloon Analogue Risk Task (BART), has previously been shown to predict alcohol use (Fernie et al., 2010, Fernie et al., 2013) and use of various other substances (Lejuez et al., 2003b, Lejuez et al., 2003a). Some studies show significant correlations between the BART and self-reported impulsivity (Lejuez et al., 2007) while others do not (Skeel, Pilarski, Pytlak, & Neudecker, 2008), and at least one study has shown that the BART can predict alcohol use over and above self-report measures in a cross-sectional risk sample (Lejuez et al., 2007). Reward–response bias refers to the propensity to display reward-driven responses and is often used as a contrast to facets of impulsivity that represent non-reward-specific disinhibited behavior. Previous research has shown that the prospective prediction of heavy episodic drinking by self-reported sensation seeking is mediated by reward–response bias in an at-risk sample (Castellanos‐Ryan et al., 2011). While all of these constructs are assessed with computerized behavioral tasks, to the best of our knowledge, their assessment using internet has not yet been examined.

Self-report measures of risk personality commonly distinguish between impulsivity and sensation seeking, which are found to overlap but at the same time represent different factors of impulsivity-related personality traits. While these traits are associated, there are differences in their development during adolescence (Steinberg, 2008). Specifically, impulsivity slowly decreases throughout childhood while sensation seeking temporarily peaks during adolescence. Self-reported impulsivity is defined here as the propensity for rash, undeliberated action, specifically the inability to inhibit behavior. Self-reported sensation seeking is defined as the desire for intense and novel experiences (Zuckerman, Eysenck, & Eysenck, 1978), including the use of deliberate strategies and preparation (e.g., deep sea diving, mountaineering). Using the Substance Use Risk Profile Scale (SURPS, Woicik, Stewart, Pihl, & Conrod, 2009) to measure these traits, both impulsivity and sensation seeking have previously been related to adolescent alcohol use (Castellanos‐Ryan et al., 2013, Conrod et al., 2008, Krank et al., 2011, Woicik et al., 2009). When measured with the SURPS, high scores on impulsivity have also been shown to prospectively predict rates of heavy episodic drinking, smoking and marijuana use, while only marijuana use was additionally predicted by high scores on sensation seeking (Castellanos‐Ryan et al., 2013). Self-report measures represent a fast and easy method of measuring impulsivity that can be applied in a variety of settings. The SURPS has been used successfully to identify at-risk populations based on high scores on specific risk personality factors (Conrod, Castellanos-Ryan, & Strang, 2010). However, previous research suggests there are aspects of impulsivity untapped by self-report personality-based measures (Reynolds et al., 2003, White et al., 1994). This makes it relevant to investigate whether prediction of adolescent drug use is meaningfully improved by the inclusion of different impulsivity measures aside from the SURPS personality scores, given that the latter are fast and cheap to include, while behavioral measures are time consuming and therefore more costly.

The internet offers a potential for mass assessment, which would allow for early detection of risk for early onset of substance use. Novel techniques allow for the administration of both self-report as well as behavioral measurement via the internet. Previous literature holds no indications on whether online risk assessment using behavioral measurement is effective. To resolve this gap in knowledge, the current study investigated the incremental predictive validity of internet-based measurement of behavioral aspects of impulsivity over and above self-reported aspects in a Dutch young adolescent sample participating in a four-wave, two-year longitudinal study. The internet represents a platform for mass measurement where automation can allow researchers to include potentially large groups of participants with a relative reduction in effort. However, this comes at the cost of supervision of participation and prevents researchers from direct encouragement or assistance in cases of lack of motivation. To the best of our knowledge, this is the first study to use longitudinal internet-based assessment to examine the prediction of substance use onset. We predicted that both behavioral measures and self-report measures would prospectively predict substance use. Furthermore, we predicted that behavioral measures add to the prediction after controlling for (easily obtained) self-report measures. The longitudinal nature of the study and the young age of participants allow us to predict the observed onset of substance use during the study period. As alcohol is by far the most common psychoactive substance used by young adolescents in the Netherlands (Van Laar et al., 2011), we predicted both onset of weekly use and onset of heavy episodic drinking. We also predicted onset of substance use for the two substances most frequently used after alcohol: marijuana and cigarette smoking. Use of each of these substances has previously been linked to aspects of impulsivity in previous literature, based on offline self-report (Krank et al., 2011, Woicik et al., 2009) and behavioral assessments (Kollins, 2003, Lejuez et al., 2003a, Lejuez et al., 2003b).

Method

Participants

The current sample (N = 284, mean age 14.8, SD = 1.26, range: 12 to 18 years, 68.7% female) is defined as those participants who successfully concluded participation in the longitudinal, online survey on at least two time points. Participants for the survey were recruited from schools of secondary education participating in the Health Behaviors in School-aged Children survey (Van Dorsselaer et al., 2009). Details regarding the recruitment strategy for the online survey are described in detail in Janssen et al. (2014). Within the current sample, 209 participants completed participation at time point 2, 182 participants at time point 3, and 195 participants at time point 4.

Procedure

Data for the study were collected at four time points in 2010 and 2011 with six month intervals. At time point 1, directly after registration, the website clarified that participation was volitional and that students could cease their participation at any point. Prior to the start of the study, parents of youth interested in participation received a letter including a passive parental consent form. This form indicated that parents could object to participation by their child, which 37 parents did. The study protocol was approved by the Ethical Review Board of the Department of Psychology, University of Amsterdam. All assessments were conducted online and the participants were free to perform the assessments at a location of their choice. Each successfully completed assessment was rewarded with a 5 EUR gift voucher. Additionally, lottery prizes such as cinema tickets were awarded at each time point.

Study design

To examine whether self-report and behavioral aspects of impulsivity prospectively predicted onset of substance use, we conducted logistic regression analyses using the onset of various substances as the outcome variable. Aspects of impulsivity were measured in counterbalanced order at baseline, whereas substance use was measured at all four time points. Performance on the behavioral measures was rewarded with lottery tickets. Earning these tickets increased the likelihood of winning additional rewards for study participation. Further details about the reward structure are available in the Online supplemental materials. For all substance-related outcome measures, Onset is defined as having indicated any use (dichotomous) of that specific substance (or any heavy alcohol use episodes in the case of heavy episodic drinking), at time points 2, 3, or 4, while not having any use at time point 1. This way, aspects of impulsivity can be assumed to completely precede the onset of substance use.

Measures

Onset of weekly alcohol use

We measured average weekly alcohol use with a self-report questionnaire (Wiers, Hoogeveen, Sergeant, & Gunning, 1997). This questionnaire contained one item for each day of the week, requiring participants to indicate their average alcohol consumption on that day.

Onset of heavy episodic drinking

The participants were asked to indicate the number of heavy drinking episodes experienced in the past month (males: five or more Dutch standard (10 g of alcohol) drinks on one occasion; females: four or more Dutch standard drinks on one occasion; Wiers et al., 1997).

Onset of cigarette use and onset of marijuana use

At each time point, the participants were asked to indicate lifetime use of tobacco and marijuana. Answer categories were “Never”, “1–2 times”, “3–5 times”, “6–9 times”, “10–19 times”, “20–39 times”, and “40 or more times”.

Risk personality traits

Risk personality was measured at baseline using the SURPS (Woicik et al., 2009), which consists of 23 items assessing risk-associated personality traits, specifically impulsivity, sensation seeking, hopelessness, and anxiety sensitivity. The participants were asked to indicate for all 23 items whether they strongly disagreed, disagreed, agreed or strongly agreed with the trait-typical statements (e.g. “I often don't think things through before I speak.” for impulsivity, and “I enjoy new and exciting experiences even if they are unconventional.” for sensation seeking), and the score for each trait was calculated as the mean of scores on the associated items. The current study used the impulsivity (5 items) and sensation seeking (5 items) scales only. The impulsivity scale of the SURPS is believed to represent the inability to inhibit a prepotent response, and a tendency to act rashly, in contrast to sensation seeking, which represents the desire for intense and novel experiences. The version of the SURPS used in the present study has been translated from the original language (cf. Malmberg et al., 2010). Internal consistencies of impulsivity and sensation seeking scales used in this study were α = .61 and α = .70 respectively, which is considered acceptable for short scales and matches earlier studies (Woicik et al., 2009).

Delay Discounting Task (Richards, Zhang, Mitchell, & Wit, 1999)

In this task, the participant was required to choose between accepting an immediate virtual reward and choosing a delayed, but greater reward. The delay ranged from one to seven time points: a day, a week, a month, six months, a year, five years and 25 years. In the computerized version of this task, this choice was repeated six times for each delay period, and the immediate reward was adjusted by half the difference between the immediate and delayed reward (cf. Richards et al., 1999). The final score (indifference point) for each delay period was equal to the last chosen immediate reward within the delay period. Scores on the Delay Discounting Task were calculated as the value of the ‘Area under the Curve’ (Myerson, Green, & Warusawitharana, 2001) that emerges when scores for each delay period are plotted as a function of time delay (x) and indifference point (y). Reliability of the task, calculated as internal consistency (Cronbach's α) of the indifference points at each of the seven time delays, was .88.

BART (Lejuez et al., 2002)

In this task, the participants are shown a picture of a balloon, and instructed to inflate this balloon. Inflating the balloon increased the picture's size on screen and the associated reward. However, overinflating the balloon would result in the balloon bursting, causing the participant to lose the complete reward of that trial. Each of the 20 balloons had a different predetermined bursting point, on a scale of 1 to 128 (pumps, or units of “air”). A selection of trials was employed with an average burst point of 64 pumps in each of the two 10 trial blocks. Our version of the BART used the automatic response procedure (Pleskac et al., 2008), whereby the participants could immediately select the intended number of pumps for that specific trial and receive immediate feedback. Scores were calculated as the mean of the number of pumps into all balloons regardless of the burst event (cf. Pleskac et al., 2008), but unlike the original BART (Lejuez et al., 2002). Odd–even reliability across 20 balloons was .84.

PALT (Newman & Kosson, 1986)

Reward–response bias was assessed using the Passive Avoidance Learning Test (PALT), which presented the participant with 3 different reinforcement scenarios in which trial and error learning occurs (Newman & Kosson, 1986). In each block of this task, the participant was presented with eight two-digit numbers, four of which were “passive” and the others “active”. Responding to an “active” number was considered correct, as was withholding response to a “passive” number. The participant had to learn through trial and error which numbers were “active” and which were “passive”. Commission errors were defined as responses to “passive” numbers, omission errors as failures to respond to “active” numbers. Three blocks of 64 trials were presented, each offering different reward and punishment contingencies for different responses. In the neutral block, no reward followed either correct or incorrect responses. In the present study, two scores were calculated from this task: a neutral passive avoidance score and a “hot”, reward related passive avoidance score. The term “hot” denotes a task's presumed capacity to encourage participants to make reward-motivated decisions that encourage prompting by immediate craving for the desired reward, or immediate fear of the expected punishment. The neutral score was calculated as the number of commission errors on the neutral block. The “hot” score was calculated as the amount of commission errors on the Reward–Punishment (RP) block minus the amount of commission errors on the Punishment–Punishment (PP) block.

Analysis strategy

Missing data

To determine the relations to missing data at time points 2, 3, and 4, we correlated instances of the missing data with all study-relevant data from the baseline. This included behavioral measures, self-report measures, substance use at time point 1, age and gender. The presence of missing assessments was not associated with any impulsivity indicator, but was significantly related to weekly alcohol use and heavy episodic drinking at time point 1 at significance level p < .05. We employed Multiple Imputation with auxiliary variables, using Bayesian estimation, in Mplus (Asparouhov & Muthén, 2010) to account for the missing data. In the imputation process, the data from baseline substance use, age, gender, and baseline scores on impulsivity-related measures were used as auxiliary information. Substance use scores at time points 2, 3, and 4 were imputed while also serving as predictors for the imputation process. Ten imputed datasets were created and exported to SPSS.

Logistic regression

We first examined the associations between the impulsivity-related predictors and substance outcomes. Secondly, to predict onset of substance use, we conducted four binary logistic regressions using Onset of weekly use, Onset of heavy episodic drinking, Onset of cigarette smoking and Onset of marijuana use as dependent variables. Prior to conducting logistic regressions, we removed cases where participants had already begun using that specific substance at time point 1.1 Independent variables were included in the regressions in three steps. In the first step, Age and Gender were included as covariates. In the second step, the scores on self-report measures were included. In the third and final step, the scores on behavioral measures were included. Background variables were removed from analysis if they were determined not to be significant predictors in any of the three steps. All tests of association and logistic regressions were performed in SPSS 20. We interpret the pooled regression results from all ten imputed datasets, as is customary when working with multiple imputed datasets (see also: Van Ginkel & Kroonenberg, 2014). In SPSS 20, the pooled results are calculated using the mean across the regression results for each imputation.

Results

Descriptive statistics

Table 1 shows the descriptive statistics of substance use. This table demonstrates that during the study period, substance use prevalence increased for all substances in the current population.

Table 1

Descriptive statistics of drug use variables.

	Onset of weekly alcohol use	Onset of heavy episodic drinking	Smoking prevalence	Marijuana prevalence
	% weekly drinker (N users)	% heavy episodic drinking (N users)	% lifetime prevalence (N users)	% lifetime prevalence (N users)
T1 N = 284	23.2% (66)	13.7% (39)	16.9% (48)	5.3% (15)
T2 N = 209	29.5% (62)	16.7% (35)	20.9% (45)	5.6% (12)
T3 N = 182	40.1% (73)	25.3% (46)	28.6% (36)	7.9% (14)
T4 N = 195	58.5% (114)	27.2% (53)	24.6% (49)	8.5% (17)

Percentile values indicate the percentage of participants to indicate use of that specific substance above zero.

In Table 2, the associations for each substance with self-report measures, behavioral measures and background measures, all measured at time point 1, are displayed. Among the behavioral measures, the reward-condition of the PALT was associated with onset of heavy episodic drinking (p = 0.003) and marijuana use (p = 0.009), and the scores on the Delay Discounting Task were associated with baseline smoking prevalence (p = 0.009), while being only associated at T4 (p = 0.09) at the trend significance level. In Table 3, the correlations between the behavioral measures and self-report measures are displayed. There were no significant correlations between the behavioral measures and self-report measures or with other behavioral measures.

Table 2

Mean differences for each type of substance outcome with measures of impulsivity and background variables.

		T1 weekly alcohol use prevalence	T1 heavy episodic drinking prevalence	T1 smoking prevalence	T1 marijuana prevalence	T4 weekly alcohol use prevalence	T4 heavy episodic drinking prevalence	T4 smoking prevalence	T4 marijuana prevalence
Behavioral	BART T1	0.072	0.079	0.259	0.037	0.163	0.162	0.333#	0.195
	PALT-neutral T1	− 0.106	− 0.059	0.065	− 0.366#	0.151	0.113	− 0.281#	− 0.432⁎
	PALT-reward T1	0.076	0.179	− 0.050	− 0.093	0.142	0.353⁎⁎	− 0.071	0.435⁎⁎
	Delay discounting T1	− 0.097	− 0.270	− 0.426⁎⁎	− 0.013	0.032	0.062	− 0.268#	0.022
Self-report	Sensation seeking T1	0.228	0.379⁎	0.515⁎⁎⁎	0.192	0.437⁎⁎	0.499⁎⁎	0.659⁎⁎⁎	0.760⁎⁎
	Impulsivity T1	0.159	0.500⁎⁎	0.626⁎⁎⁎	0.515#	0.229	0.440⁎⁎	0.326#	0.603⁎
Background	Gendera	0.043	0.007	0.127	1.734	0.003	5.603⁎	0.122	1.750
	Age	1.120⁎⁎⁎	0.946⁎⁎⁎	0.521⁎⁎⁎	1.265⁎⁎⁎	0.837⁎⁎⁎	0.569⁎⁎⁎	0.291#	0.850⁎⁎⁎

Note: Results are differences in standardized mean for substance users compared to non-users. T1 = TIME point 1. T4 = time point 4.

Pearson Chi-square for independence reported.

p < .1.

p < 0.05.

p < 0.01.

p < 0.001.

Table 3

Bivariate correlations between self-report and behavioral aspects of impulsivity and age at time point 1.

		1	2	3	4	5	6	7
Behavioral	BART (1)	/	.016	.048	− .023	.009	.068	.024
	PALT-neutral (2)		/	− .026	− .051	.010	.032	− .089
	PALT-reward (3)			/	.075	− .073	− .011	− .018
	Delay discounting (4)				/	− .047	− .089	.096
Self-report	Sensation seeking (5)					/	.134	.122
	Impulsivity (6)	.					/	− .045
Background	Age (7)							/

Multiple logistic regression results

Table 4 contains the results for the logistic regressions with onsets of weekly alcohol use, heavy episodic drinking, cigarette smoking, and marijuana use as the dependent variables. We examined standardized residuals and Cook's Distance values to investigate influential outliers. It was found that the direction, strength and significance of regression coefficients remained unchanged when outliers with absolute standardized residuals of above three, and Cook's Distance values above one, were removed from the analysis. We therefore present the results with all cases included.

Table 4

Odds ratio results from logistic regression for onset of substance use.

	Predictors	Onset of weekly alcohol use OR (95% CI)	Onset of heavy episodic drinking OR (95% CI)	Onset of cigarette smoking OR (95% CI)	Onset of marijuana use OR (95% CI)
Step 1	Age	3.14⁎⁎⁎ (1.99–4.94)	1.98⁎⁎⁎ (1.40–2.79)	1.83⁎⁎ (1.22–2.73)	1.74⁎ (1.04–2.92)
Step 2	Age	3.09⁎⁎⁎ (1.91–5.00)	2.08⁎⁎⁎ (1.43–3.04)	1.83⁎⁎ (1.20–2.79)	1.69 (.98–2.92)
	Sensation seeking	1.68⁎ (1.06–2.66)	1.33 (.92–1.91)	1.63⁎ (1.12–2.38)	2.43⁎⁎ (1.32–4.46)
	Impulsivity	1.35 (.92–1.98)	1.62⁎⁎ (1.14–2.31)	1.48# (.97–2.26)	1.20 (.75–1.93)
Step 3	Age	3.29⁎⁎⁎ (1.97–5.49)	2.13 ⁎⁎⁎ (1.45–3.13)	1.80⁎ (1.15–2.81)	1.70# (.95–3.04)
	Sensation seeking	1.70⁎ (1.07–2.71)	1.32 (.91–1.91)	1.64⁎ (1.11–2.45)	2.55⁎⁎ (1.39–4.67)
	Impulsivity	1.35 (.90–2.02)	1.63⁎⁎ (1.14–2.33)	1.50# (.97–2.33)	1.15 (.71–1.88)
	BART	0.99 (.70–1.42)	1.06 (.74–1.53)	0.96 (0.65–1.43)	0.90 (.58–1.40)
	PALT neutral	1.09 (.78–1.52)	0.96 (.70–1.31)	0.77 (.43–1.39)	0.76 (.40–1.43)
	PALT reward	1.23 (.83–1.83)	1.07 (.75–1.52)	1.11 (.70–1.76)	1.75# (.95–3.23)
	Delay discounting	0.84 (.59–1.21)	0.84 (.59–1.22)	0.89 (.58–1.37)	0.78 (.43–1.41)

Note. Results indicate odds of indicating onset in the study period, relative to non-use. Results shown are pooled across 10 imputations. Higher values indicate higher odds of use.

p < .1.

p < .05.

p < .01.

p < .001.

Logistic regressions show that older age significantly predicted onset of all substances, while gender did not. The results are therefore presented with age included and gender excluded from final analyses. Self-reported sensation seeking at T1 significantly predicted onset of weekly alcohol use, onset of cigarette smoking and onset of marijuana use. Self-reported impulsivity at time point 1 significantly predicted onset of heavy episodic drinking, and predicted onset of cigarette smoking at trend-level significance (p = .07). There was a trend significant prediction of the onset of marijuana use by the RP-block of the PALT (time 1; p = .07). No other behavioral task uniquely predicted the onset of substance use over time.

Discussion

This study investigated the incremental predictive validity of behavioral assessments of impulsivity-related facets over self-reported impulsivity and sensation seeking, in the prediction of the onset of substance use. Contrary to our expectations, the results showed that although some of the behavioral measures were correlated with prospective substance use, they did not significantly add to the prediction of substance use onset after controlling for self-report measures of impulsivity and sensation seeking.

Although previous lab and field studies (Fernie et al., 2013, Field et al., 2007, Reynolds et al., 2003) have related delay discounting with all substances included in the present study, we were not able to replicate the association between scores on the Delay Discounting Task to substance use, or show that the Delay Discounting Task predicted substance use after self-reported impulsivity was controlled for. In the current study, only the onset of cigarette smoking was associated with delay discounting at the borderline significance level, but it did not predict the onset after controlling for self-report measures of impulsivity. Within the current online sample, when asked, the participants indicated in the overwhelming majority that the Delay Discounting Task was perceived as “boring”. When conducting online measurement, this may cause issues of validity as we were also unable to either instruct or physically reward participants, as is customary for this task. At the same time, internal consistency indicated that the task measurement was reliable, suggesting that participants were able to answer consistently to the task's hypothetical reward scenarios. However, it is still a possibility that their answers do not meaningfully relate to their propensity to delay reward in real life. We were unable to find earlier studies attempting online measurement of delay discounting effects, making this explanation difficult to substantiate based on the current data.

With regard to risk taking, the results specific to the BART contrast those of the earlier findings. Unlike in earlier classroom-based studies among adolescents (Lejuez et al., 2003a, Lejuez et al., 2003b), but similar to the results from the undergraduate samples (Skeel et al., 2007, Skeel et al., 2008) the scores on the BART were not associated with substance use or with self-report sensation seeking in the current sample. The earlier finding that the results from the BART predicted alcohol use over and above self-report measures (Lejuez et al., 2007) was not replicated here for the onset of substance use. The current study used the automatic response mode version of the BART, based on the version used by Pleskac et al. (2008). However, the finding from Pleskac and colleagues that this modified BART predicted substance use among undergraduates was not replicated for the onset within the current sample.

The online nature of the current study and the outcome measures used offer multiple possible explanations for non-replication of the earlier results. First, one might suspect that the unsupervised environment created by the online nature of the study resulted in greater amounts of random or casual responses, caused by a lack of motivation to complete the survey. However, we found no evidence of random or casual scoring in the task data. Additionally, when 40 participants were probed, most of them indicated that they considered the BART to be the most entertaining and motivating task in the study's battery. Furthermore, internal consistency for the task was high, suggesting that response was reliable and consistent. Secondly, to compensate for the limited time window available for behavioral measurement, the amount of trials in the task had been reduced from 30 to 20. However, given that the internal consistency for the task was high, it seems unlikely that this reduction changed the nature of the task's predictive validity. Finally, it can be argued that the automatic response version of the BART reduces the sensation of risk experienced by the participants forced to make repeated decisions to keep inflating a balloon, to growing risk. The lack of a sensation of risk could explain why trials were not experienced as acute risk scenarios prompting ‘hot’ responses, but rather as risk-hypothetical prompting ‘cold’ reasoning. This difference may be imagined as the task not featuring the urgency and immediacy with which adolescents make decisions that affect risk behavior, which are often made in the heat of the moment when reasoning and consideration of long term consequences are not possible.

Concerning reward response bias, the results revealed that the difference between commitment errors during rewarded and punished blocks in the PALT correlated significantly with the onset of heavy episodic drinking and marijuana use. The significant association with heavy episodic drinking matches earlier findings with the adolescents in an at-risk sample (Castellanos‐Ryan et al., 2011). Although the reward–response bias is a concept explored in more detail in relation to conduct disorder and criminal behavior (Arnett & Newman, 2000), we could find no other studies that tested the association between reward–response bias measured with the PALT and other substances than alcohol. In the current study, with the exception of a finding at the trend-level significance in the regression on the onset of marijuana use, the PALT did not predict the onset of substance use when other measures were controlled for. When asked, the participants indicated that the PALT was considered the most “frustrating” task to complete, as the task demanded that the participants learn on trial-and-error basis whether trials were active or passive. For this reason, we consider our findings exploratory.

The present study had a number of limitations and strengths worthy of discussion. Our sample was predominantly female, although gender was not a significant predictor in the regression analyses. Additionally, we chose not to examine lifetime use of alcohol, but rather to predict the onset of weekly alcohol use and heavy episodic drinking. We consider these outcomes to be of scientific interest because it indicates consistent substance use behavior. However, this means that unlike the prediction of the onset of cigarette smoking and marijuana use, we cannot be certain that the participants included in the regressions on alcohol results were necessarily completely alcohol-naïve at baseline. The reason for this is that alcohol use at this age may exist at levels below weekly or heavy episodic drinking. Assessment of self-report impulsivity traits was limited to the impulsivity- and sensation-seeking-scales of the SURPS. This means that the present study does not offer insights into potential relations between the behavioral assessments and other self-report impulsivity traits such as lack of perseverance and negative urgency (Whiteside and Lynam, 2001, Whiteside and Lynam, 2003). Regarding the behavioral assessments, due to time constraints, we decided to use the automatic response procedure version of the BART task. Furthermore, due to the lack of direct feedback opportunities, we decided to make changes to the reinforcement schemes employed in the behavioral tasks. Although previous research on college students had shown this version of the BART to accurately represent risk taking, non-significant prediction by this task may potentially be explained by this modification. Additionally, due to the PALT task structure, it is impossible to determine if this task suffered from lack of reliability. Finally, in interpreting the results from the logistic regression on the onset of marijuana use, it must be considered that the prevalence of marijuana use in the current sample was low, meaning that the sample was potentially underpowered to detect certain significant relations to this outcome measure.

Despite these limitations, there were also unique strengths to our design. We predicted the observed onset within the study period by excluding the participants who had already used substances at the baseline from the logistic regressions. This increased the statistical power available for detecting relations to onset, as the potential onset period extends beyond a single time point. Second, we controlled the logistical regression results for age and gender where necessary. This format allowed the present study to examine the general and unique influences of measures of impulsivity on a wider range of substances often used by adolescents. Third, we followed the design suggestions from Castellanos‐Ryan et al. (2011) by including a neutral condition for the PALT and controlling for the results on this condition in regression analyses.

Overall, the current study suggests that the use of the behavioral measures of impulsivity above self-report measures in online settings may not be warranted for the prediction of the onset of substance use among young adolescents. This conclusion is specific to the relatively “hot” reward-based tasks used in the current survey; other more “cold” behavioral measures of executive function, such as tasks measuring executive functions, may be uniquely predictive above and beyond personality in a way that the current tasks are not. Should we conclude from this that the behavioral assessment of impulsivity is useless? Maybe not. It is possible that these processes are especially relevant in predicting outcomes of rare incidence (such as indicators of conduct disorder, cf. Moffitt, 1993). When types of substance use are relatively rare and less socially endorsed, such as marijuana use at young age here, impulsivity tasks may be predictive. In our case, this could explain the statistical trend level PALT prediction of marijuana use onset.

New measures predicting risk status for engagement in substance use are of great interest when they may be applied as easily and in such great numbers as an online environment offers. However, we conclude that the online environment, where consistency, persistence and focused task completion are difficult to enforce and motivate, does not appear to lend itself to the inclusion of the measures used here. Therefore, while there may be aspects of impulsivity predictive of the onset of substance use when the self-report measures are controlled for, the tasks used in the current study do not appear to capture these aspects in an online setting. Based on this, the online use of these tasks, in their current form, to predict the early onset of substance use, is not recommended. However, further research is warranted to confirm findings regarding substance abuse outcomes of rare incidence.

Role of funding sources

Funding for this study was provided by the Netherlands Organization for Scientific Research [grant number 453-08-001], awarded to the senior author.

Contributors

All authors have materially participated in the manuscript preparation. Tim Janssen, Helle Larsen, Wouter Boendermaker, and Reinout Wiers have contributed to the study design, study execution, statistical analyses and manuscript preparation. Margot Peeters and Wilma Vollebergh have contributed to the study design and manuscript preparation.

Conflict of interest

All authors declare that they have no conflicts of interest.