Running Online Behavioral Experiments Using R: Implementation of a Response-Time Decision Making Task as an R-Shiny App.

Online experiments allow for fast, massive, cost-efficient data collection. However, uncontrolled conditions in online experiments can be problematic, particularly when inferences hinge on response-times (RTs) in the millisecond range. To address this challenge, we developed a mobile-friendly open-source application using R-Shiny, a popular R package. In particular, we aimed to replicate the numerical distance effect, a well-established cognitive phenomenon. In the task, 169 participants (109 with a mobile device, 60 on a desktop computer) completed 116 trials displaying two-digit target numbers and decided whether they were larger or smaller than a fixed standard number. Sessions lasted ~7-minutes. Using generalized linear mixed models estimated with Bayesian inference methods, we observed a numerical distance effect: RTs decreased with the logarithm of the absolute difference between the target and the standard. Our results support the use of R-Shiny for RT-data collection. Furthermore, our method allowed us to measure systematic shifts in recorded RTs related to different OSs, web browsers, and devices, with mobile devices inducing longer shifts than desktop devices. Our work shows that precise RT measures can be reliably obtained online across mobile and desktop devices. It further paves the ground for the design of simple experimental tasks using R, a widely popular programming framework among cognitive scientists. Copyright:

Introduction

Online experiments are gaining ground among behavioral and cognitive science researchers as they allow collecting massive data from diverse participants remotely, quickly, and at a low cost. Moreover, they became the principal data collection tool during the COVID-19 pandemic to keep projects running. A particularly valuable source of information in experimental psychology is participants’ response times (RTs). While RTs are crucial to infer many psychological processes (e.g. Stroop interference effect), RT measures are extremely susceptible to noise and interference. Oftentimes, millisecond-range precision is required to detect effects on RT (Plant, 2016). Thus, assessing the reliability of RT data obtained in web-based experiments is paramount.

To illustrate what an online setting implies, picture yourself riding a bus on your way back home when a message pops up on your phone inviting you to participate in a decision-making task. The bus rocks constantly, it is noisy and you are somewhat tired, but decide to partake anyway. After all, you are only asked to make a handful of simple and quick decisions in 5 minutes. Though this may be an extreme example, it shows how uncontrolled conditions in online studies contrast sharply with standard laboratory experiments. In the latter, participants are welcomed by a researcher, receive detailed instructions, and complete hundreds of trials during several minutes in front of a calibrated computer in a quiet room.

Online behavioral experiments are not novel, as attempts to use the web as a tool for psychological research date back to the late 1990s when the internet became more accessible (Krantz & Dalal, 2000; Reips, 2002). The reliability of data collected online has always been a focus of attention (Anwyl-Irvine et al., 2020; Brown et al., 2014; Crump et al., 2013; Dandurand et al., 2008; Dufau et al., 2011; Garaizar & Reips, 2019; Germine et al., 2012; Gosling et al., 2004; Grootswagers, 2020; Huber et al., 2019; Kochari, 2019; Krantz & Dalal, 2000; Pronk et al., 2020; Semmelmann & Weigelt, 2017; Simcox & Fiez, 2014). As technology develops and new tools become available, the need to verify the compatibility between online and lab experiments increases.

Some labs have developed their own native mobile apps —downloadable from an app store (e.g., Google Play)— to run gamified versions of cognitive behavioral experiments (Brown et al., 2014; Dufau et al., 2011; Zimmerman et al., 2016). Others (e.g., Crump et al., 2013) have leveraged services such as Amazon Mechanical Turk (AMT) to reproduce classical experiments with AMT users, including four tasks that required detecting within-subject RT differences as short as 50 ms.

Recent studies have shown that online experiments are useful in numerical cognition research. An online study built in HTML/JavaScript (jsPsych (de Leeuw, 2015)) replicated classical RT effects in a task in which participants had to compare a single-digit number with a predetermined standard (Kochari, 2019). Huber et al. (2019) implemented a two-digit number comparison task on Qualtrics (Qualtrics, Provo, UT), and replicated previous findings, such as the decade distance effect. To avoid variability in RTs introduced by using different devices, data collected from mobile devices was discarded (a similar approach can be found in (Cipora et al., 2019) and (Gökaydin et al., 2018). However, as mobile devices are more widely used than computers, especially in emerging economies, this methodological approach may severely constrain the total number of potential participants in online experiments (Silver et al., 2019).

Despite deriving important lessons from past experiences using online tools for behavioral experiments, there are new challenges to take upon. First, in line with calls for open-science practices (Munafò et al., 2017), we ran an experiment with open-source tools and freely shared the code online. Second, we advocate for an integrated workflow encompassing experiment design and development as well as data analysis and modelling within a single developer’s framework (R in RStudio). While switching frameworks (and programming languages) according to the job at hand is certainly possible, it can be burdensome and error-prone (e.g., code bugs/errors due to syntax switching, different OS/software requirements). Furthermore, relevant tools from other environments (e.g., Python) can be used in R via an R version/wrapper (i.e., package). Third, to leverage the power of online experiments, we need a seamless and robust mobile user interface (i.e., an app that is intuitive, responsive, and reliable across OSs and web browsers), and adequate statistical models to account for device-related effects. Most published online studies fail to jointly meet all these requisites.

Native mobile apps are good to gamify experiments, but are usually task-specific and hard to adapt to new tasks. Moreover, programming these apps, as well as most web-based experiments, may require skilled coding in HTML, CSS, and JavaScript —languages not normally used by cognitive scientists for modeling and data analysis. While coding-free alternatives are available (e.g., jsPsych Builder [http://builder.jspsych.org], Gorilla [https://gorilla.sc/], PsychoPy [https://www.psychopy.org/], Lab.js [https://lab.js.org/], LabVanced [https://www.labvanced.com/]), some may not be free, open, and/or as flexible as the more “code-involved” options.

Shiny (Chang et al., 2018) is a package that runs on R, a programming language popular among cognitive researchers. It is well integrated in RStudio (a widely-used IDE to write and run R code). This package is relatively easy to learn and use for those familiar with scientific programming, and allows for experiment coding, data collection, data analysis and modelling, and creation of Markdown reports. Yet, despite its great potential to design and develop behavioral experiments as interactive web apps (Kaufman, 2020; Steiner et al., 2020), it remains underexploited in experimental psychology. Lastly, deploying a Shiny app online within RStudio is fairly simple: a newly developed app can be deployed online in minutes on the RStudio app-hosting service shinyapps.io. Quick app testing and sporadic data collection can be effectively done under a free plan. To circumvent limitations of the service’s free plan (CPU and memory limitations, participant cap), it is possible to install a Shiny-server on a cloud service or University server.

Here, our main goal was to run a behavioral task as a Shiny app capable of measuring precise RTs at a millisecond range for participants using mobile devices, tablets and desktop computers. To this end, we implemented a two-digit number comparison task (), in which participants had to decide whether a number (target) was smaller or larger than a reference number (standard). We relied on an already-developed R package (ShinyPsych (Steiner et al., 2020)) but customized some code to implement task-specific features. Typically, RTs are shorter and error rates are lower when the absolute difference between the numbers being compared is larger (i.e., comparing 61 vs. 65 is more difficult than comparing 34 vs. 65). This long-studied phenomenon, coined the numerical distance effect, has been used to shed light on how the mind represents numbers (Dehaene et al., 1990; Dehaene, 2011; Hinrichs et al., 1981; Krajcsi & Kojouharova, 2017; Moyer & Landauer, 1967; Nuerk et al., 2001; Verguts & De Moor, 2005).

Number-comparison task. Participants had to decide whether a target number (49, in this case) was smaller or greater than a standard (65). We presented 58 target numbers: 29 were lower than the standard (from 36 to 64), and 29 were greater than the standard (from 66 to 94). Each of these targets was displayed twice, rendering a total of 116 trials. Trials were interleaved by an intertrial interval (ITI) that was randomly drawn from a uniform distribution (ITI ~ U(700 ms, 1000 ms)). Mobile device users responded on the screen by tapping a right or left box below the target, while desktop users responded by using the keyboard arrows. A left box tap/arrow keypress indicated a “smaller than 65” response and a right box tap/arrow keypress indicated a “greater than 65” response. On each trial, we recorded participants’ responses (smaller or greater) and response time (RT). Participants were instructed to be as accurate and fast as possible. To further enforce these instructions we applied a time deadline (each target was displayed for a maximum of 2 seconds) and a scoring system (+2 points for correct answers, –1 point for errors and timeouts). Finally, we included a progress bar (in light-blue at the top) to minimize dropouts due to uncertainty about how long the task would take.

This task was chosen due to three main reasons. First, it seems to reveal a universal feature of human numerical cognition (Dehaene, 2011), offering a robust benchmark to test our online tool. Indeed, several independent studies have found a linear mapping between RTs and logD —the logarithm of the absolute difference between the target (T) and the standard (S) numbers— with RTs decreasing about 100 ms per unit of logD (β ≅ –100ms), even when the standard numbers were different (e.g. 55, 65, 75) (Dehaene, 1989; Dehaene et al., 1990; Hinrichs et al., 1981).

Second, since the numerical distance effect occurs in the millisecond range, its detection needs high-precision RT measurement —a usual requirement in experimental psychology studies (Plant, 2016; Ratcliff et al., 2010). Last, the task is arguably engaging and short, rendering it more suitable for online experimental settings.

Based on the above considerations we hypothesized that:

H1: RT decreases linearly with the logarithm of D, consistent with previous results (Dehaene, 1989, 2011; Dehaene et al., 1990; Hinrichs et al., 1981; Krajcsi & Kojouharova, 2017; Moyer & Landauer, 1967, 1973; Nuerk et al., 2001; Verguts & De Moor, 2005).

To maximize recruitment, online experiments are often designed for multiple devices, meaning RT data will have several sources (e.g., iPad, Android phone). Importantly, measured RTs not only represent the time needed to make a decision, but they also include shifts (i.e. delays in recorded RTs) due to either internal processes (perceptual encoding of the stimulus and motor response execution) or external sources (). Specifically, each type of device might introduce a different delay related to hardware, OS and/or browser processing speed (Anwyl-Irvine et al., 2020; Bridges et al., 2020). Indeed, mobile devices have larger shifts than desktop computers (Holden et al., 2019; Pronk et al., 2020; Reimers & Stewart, 2008). To control for device-specific variability, we collected participants’ device information (OS and web browser), incorporated this information into a combined variable in a statistical model, and determined its impact on RT. We therefore hypothesized that:

H2: Participants’ device, OS, and web browser introduce systematic delays in recorded RTs. Specifically, we expected that mobile devices would introduce larger shifts on RTs than desktop devices.

Temporal decomposition of the recorded response time (RT). On each trial, the recorded RT is the sum of a decision time and a non-decision time or shift (θ). The shift is assumed to represent both internal processes (perceptual encoding of the stimulus and motor response execution) and external delays. The latter are exclusively technological and include delays due to hardware, OS and browser processing speed, such as the latency between button, keypad or touchscreen press to the actual recording of the response. The decision time is the time the participant takes to evaluate the evidence, and it is assumed to follow a log-Normal distribution with parameters μ and σ.

Method

Participants

Participants accessed the app online via a shortened URL link distributed across Facebook, Twitter, and WhatsApp. This link redirected them to the Shiny server hosting the app (Donweb [donweb.com] cloud server with 2GB of memory). Our aim was collecting data from at least 84 participants, twice the sample in Dehaene et al., 1990 (n = 42), as our task had half the number of trials. Within the first week of having the app online, we surpassed this minimum sample size and stopped data collection. A total of 169 participants (65 males, age range: 18–72 y.o., mean age: 31.94 ± 11.69 y.o.) completed the online task either with a mobile device (n = 109) or on a desktop computer (n = 60). All participants gave informed consent, and the local ethics committee approved the study (“Centro de Educación Médica e Investigaciones Clínicas” protocol ID 435). No monetary compensation was awarded to participants.

Task

In this two-digit number comparison task, adapted from Dehaene et al., 1990, participants had to decide whether a target number was smaller or greater than 65. We presented 58 target numbers: 29 were lower than the standard (from 36 to 64), and 29 were greater than the standard (from 66 to 94) (). Each target was displayed twice, yielding a total of 116 trials. These were pseudo-randomized so that the same target was never displayed in two consecutive trials and the correct response (greater or lower than the standard) was never the same for more than two consecutive trials. We constructed 19 different lists of numbers with these features, and one was randomly selected at the beginning of each task-session. An intertrial interval (ITI) randomly drawn from a uniform distribution, ITI ~ U(700 ms, 1000 ms) was imposed.

In each trial, a target number was presented in bold Arial font (font size 113 for mobile devices, 165 for desktop) for a maximum of 2 seconds. Desktop users responded by using the keyboard arrows, while mobile device users responded on the screen by tapping a right or left box below the target. The opacity of these boxes changed on-response as visual feedback that a response had been recorded. A left box tap/arrow keypress indicated a “<65” response, and a right box tap/arrow keypress indicated a “>65” response. We recorded RTs and response type (smaller or greater) for each trial.

Programming in Shiny

Task implementation was largely based on ShinyPsych (Steiner et al., 2020), a recently-developed R package for designing behavioral experiments on Shiny (Chang et al., 2018). This package was selected because each element in the app flow (instructions, training, experimental phase, demographic survey, and debriefing) is built as a separate “page” that is rendered when users interact with the app. Furthermore, ShinyPsych offers simple functions to specify elements within a page and includes ready-to-use templates (which we leveraged for the demographic survey). However, since ShinyPsych did not support specific aspects of our task (e.g., flexible implementation across devices and systems), we further customized the app via R/HTML/CSS/JavaScript. We included the following features: collection of device type (mobile or desktop computer), web browser and OS; variable ITIs in which a fixation cross was displayed; scoring system; task progress bar; device-dependent visual parameters (e.g., text size); device-dependent responses (key-pressing for desktop and box-tapping for mobile users); and RT collection (using differences in browser timestamps via the Date.now() JavaScript function) with a temporal deadline (upon which a custom timed-out message popped-out and remained on screen for a fixed amount of time). Importantly, the latter offers reliable time measurements, which is critical to infer the numerical-distance effect from RT data. Our app (as any Shiny app) uses JavaScript under the hood to record events and user-interactions within an HTML page. Specifically, RT is defined as the interval between two specific events: stimulus (number) onset and user response. RTs were obtained through JavaScript functions that counted the time elapsed from number onset to user-response (box-tap or key-press, for mobile and desktop devices, respectively).

Procedure

Participants clicked on a link distributed across social media platforms to access the online app. Upon connection, instructions were displayed, and informed consent was prompted. As in Dehaene et al., 1990, we instructed participants to be as accurate and fast as possible. To further enforce these instructions, we applied a time deadline (2 seconds) and a scoring system (+2 points for correct answers, –1 point for errors and timeouts). This scoring system also served as a way to “gamify” the app and promote engagement in the task (Lieberoth, 2015; Lumsden et al., 2016; Wiley et al., 2020). Users could track their progress through a progress bar, which also aided in reducing dropouts for not knowing when the task would end.

The experimental phase started after four training trials. After the task, participants reported their age, sex, and education level on an in-app survey built with the ShinyPsych package (the survey can be found at the project’s OSF repository: https://osf.io/9w28t/). Finally, performance feedback (total points) was provided and a short debriefing text was shown.

Sessions lasted 7 minutes on average, with the shortest lasting 4 minutes.

Data Analysis & Statistical Models

To determine whether our results support the numerical distance effect, rather than relying on ANOVA and OLS regressions as in Dehaene et al., 1990, we based our analyses on generalized linear mixed models (GLMM) that were estimated with Bayesian methods. We applied GLMMs because they extend the general linear models to the exponential family of probability distributions, which include the shifted log-Normal and Bernoulli probability density functions that we used to model RT and error rate, respectively. Moreover, random-effects (multi-level) modeling allowed us to represent within-subject data-dependency through a grouping variable which, at the same time, increased model parsimony and allowed for generalization in inference and interpretation. We created our GLMMs within the Stan computational framework (Carpenter et al., 2017) accessed with brms (Bürkner, 2017), which uses a No-U-Turn Sampler (NUTS, an extension of the Hamiltonian Monte Carlo sampling method) to estimate posterior probabilities. To improve convergence and prevent overfitting, we used weakly informative —yet regularizing— priors (Gelman et al., 2008).

Our modelling strategy followed a sequential rationale stemming from Bayesian workflow guidelines (Gelman et al., 2020): hypothesis-driven model specification and descriptive data-analysis, prior predictive checking (to assess our priors’ suitability), estimation, diagnosis (using the launch_shinystan function from the shinystan package (Gabry, 2018) to inspect —among other diagnostics— sampling convergence and autocorrelation), validation (via posterior predictive checks), and inference (i.e., hypothesis testing).

Next, we report the model specification for the RT and error rate models.

Response times

As shown in previous studies on numerical distance, the main predictor of mean RT is the log of the absolute difference between the target (T) and the standard (S) numbers, D = abs(T-S), i.e. RT ~ logD (Dehaene et al., 1990; Hinrichs et al., 1981; Nuerk et al., 2001). However, as RTs are positive, right-skewed, and necessarily greater than the time required to encode stimuli, RTs were modeled as a shifted-logNormal random variable (Wagenmakers & Brown, 2007):

This probability density function is defined by three parameters: μ (mean), σ (SD) and θ (non-decision time or shift). The latter accounts for both psychological (stimulus perceptual-encoding and motor response execution), and technological components (CPU, OS, and web browser) (). To render a linear relationship between mean RT and logD, we model μ as the logarithm of a linear predictor (ν), which includes, among others —as we will detail below—, the coefficient for logD.

This way, mean RT maps linearly onto logD, i.e., E[RT] = ν, and the coefficients from the linear predictor of ν can be interpreted analogously to those of previous studies.

Beyond our focus on the numerical distance effect, we also examined whether the mean, standard deviation and/or shift of RT were affected by device type, OS and web browser. To this end, the model included a predictor variable that collectively represents these features of the participant’s interface that we called system. Altogether, the predictor variables for ν, σ, and θ were:

logD: A numerical variable indicating the log of the absolute numerical distance between the target presented on each trial and the standard (values of logD range from log(1) = 0 and log(29) ≅ 3.37).

smaller: A categorical variable indicating whether the target in a given trial was smaller (1) or greater (0) than the standard — this is relevant because previous studies show that numerical comparisons are slower for smaller than for greater numbers (Dehaene et al., 1990).

system: A categorical variable indicating the participant device’s OS and web browser. Our data contained 10 categories: four from mobile devices (Android.Chrome, iPhone.Mozilla, iPhone.Safari and iPad.Mozilla) and six from desktop computers (Windows.Chrome, Windows.Firefox, Linux.Chrome, Linux.Firefox, Macintosh.Chrome and Macintosh.Safari). These categories are indexed with sub-index j (j = 1,…,9), and “Android.Chrome” was used as the reference category.

trial.number: A numerical variable, ranging from 1 to 116, that indicates trial number (centered and scaled) within each participant’s session, to control for chronological effects (e.g., fatigue, learning, hurry-to-finish).

age: A numerical variable (centered and scaled), ranging from 18 to 72, to control for participants’ age effects.

Next, we report the specification for the linear predictors of ν, σ, and θ. To aid model convergence, we used a log-link function for the linear predictors of σ and θ.

Note: id: participant ID; i: trial index; j: system index.

We represented the hierarchical structure of the data (i.e., the within-subjects design) by modelling random effects by participant ID (id) for the intercept of the linear predictor for ν and θ, and over the coefficients for the effects of logD, smaller, and trial.number on ν. Moreover, we modeled the correlation between the random effects and these coefficients, and between the intercepts of ν and θ. The corresponding covariance and correlation matrices are as follows.

Note: MVNormal: Multivariate Normal; S: covariance matrix; R: correlation matrix.

Next, we report prior specifications for all the estimated model parameters. These are weakly-informative priors that generate plausible data (assessed with prior predictive checks).

Error rate

We modeled errors as a Bernoulli random variable (i.e., a binary categorical variable where 1 = Error and 0 = Correct) with a logit link to the linear predictor of the probability π of making an error (Equation (4) below), which included as predictor variables trial.number and logD. In other words, for each trial i, we estimated the probability πi of making an error on that trial, conditional on these predictor variables.

Similarly to the RT model, we represented the hierarchical structure of the data by modelling random effects by participant ID (id) for the intercept and the trial.number coefficient of the linear predictor for π, and modeled the correlation between these random effects. Next, we show how we represented these relations.

Next, we report prior specifications for all the estimated model parameters.

To diagnose model estimation, we (i) evaluated trace autocorrelation and number of effective samples, (ii) visually inspected chains with trace plots for each model parameter, and (iii) assessed whether values were smaller than 1.01 (denoting that chains mixed well) using launch_shinystan (from the shinystan package (Gabry, 2018)). For both models, we drew 20,000 samples (4 chains of 6,000 traces each with the first 1,000 for warm up) with an adaptive delta of 0.99 to minimize divergent transitions after warm up. As a result, there were zero divergent transitions during sampling of both models.

We validated the models with posterior predictive checks (PPCs) by simulating 200 new datasets from the estimated parameters’ posterior distributions, and overlaying the simulated distributions over the observed data distribution. PPCs allowed us to understand whether (and to what degree) the estimated model captured distributional signatures of the observed data. For both models, the posterior predictive distributions were qualitatively indistinguishable from the observed distributions (). To quantify this qualitative assessment, we computed —for the error model— the proportion of errors and -for the RT model- the 0.1, 0.3, 0.5 (median), 0.7, and 0.9 quantiles and SD of the conditional (by smaller, and by system) RT distributions of each set. We then obtained the median and the quantile-based credible intervals (containing 95% of the probability density, CI95) of the distribution of these summary-values and compared them to the observed-data summaries (assessing whether the observed value was contained in the CI95). This approach assigned credibility to the model while offering a measure of uncertainty from the distribution of simulated summary statistics. For the error rate model, the CI95 of the conditional summary statistics included all the summary statistics of the observed data. Similarly, for the RT model, this was generally true, except that -mainly- for the 0.9 quantile for some categories of system, the simulations were off by a maximum of 10 ms from the observed summary statistic.

Posterior predictive checks for the RT and error rate models. (A) We simulated 200 datasets with the same structure as the observed dataset, using the RT model posterior distribution. We plotted the distribution of RT for each of these 200 simulations (light-red lines) and the observed RT distribution (black line). The high degree of overlap between the simulated and observed data distributions is indicative of the model’s ability to capture the distributional features of the observed RTs. (B) We performed a similar analysis for the error rate model but, since this variable is not continuous, we computed the mean error rate for each of the 200 simulated datasets (light-red dots) and for the observed data (black dot). Mean error rates from the simulated datasets are scattered around the observed error rate mean, which is indicative of the model’s ability to capture the main distributional feature of the observed error rate.

Additionally, we computed a measure of in-sample goodness-of-fit for each model. For the RT model, we computed a Bayesian approximation to R2 (Gelman et al., 2019) and found that R2 = 0.438 (CI95 = [0.429, 0.447]). For the error rate model, we computed the area under the Receiver Operating Characteristic curve (AUC-ROC), which reflects the model’s classification ability (as a rule-of-thumb, AUC-ROC = 0.8 denotes a good model performance (Hosmer Jr et al., 2013)), and found that AUC-ROC = 0.813.

Finally, we tested our hypotheses directly over the posterior probability distribution by computing the CI95 of the coefficient-of-interest distribution and assessing whether this interval contained (or for the logD predictor, was smaller than) 0. If the CI95 did not include 0 (or if 95% of the density was smaller than 0, for logD), the results support the hypothesis that the corresponding coefficient is different from (smaller than) that null value. To aid interpretation of the error rate model, we first exponentiated the coefficient-of-interest posterior values to obtain odds ratios, and then computed the summary values of these posterior values (median and CI95).

Results

We found evidence consistent with a numerical-distance effect on RT. As expected, the greater the absolute difference between target and standard, the faster the responses (). Regression coefficients for our model (Equations 1–3) show that, on average, RTs decrease 70 ms for every unit increase in logarithmic numerical distance, e.g. roughly the distance from 66 (log(D = 1) = 0) to 68 (log(D = 3) ≅ 1) (β = –70.3ms, CI95 = [–75.4, –65.6]ms). Strikingly, our estimated value of β closely resembles that estimated from the data shown in Dehaene et al., 19901 (β = –78.3ms), even when we used a different statistical model for RT and different technology to implement the task. Also consistent with previous studies (Dehaene, 1989; Dehaene et al., 1990; Hinrichs et al., 1981), we found that RTs for numbers smaller than 65 were slower than RTs for numbers greater than 65 (β = 55.8ms, CI95 = [39.7,72.3]ms). Again, our estimated value for β closely matches that from Dehaene et al., 1990 (β = 53.8ms).

Numerical distance effect on RT and error rate. (A) Expected and observed values for RT as a function of the numerical distance between the target and the standard (distance is 0 at the dashed vertical line). Green dots are the observed overall mean of RT for each distance. The black line and shaded area represent the median and CI95 of the RT expected values (for an average participant), respectively, obtained from 2,000 model posterior samples. (B) Expected and observed values of error rate (i.e., probability of making an error) as a function of the numerical distance. Green dots are the overall proportion of errors for each distance. The black line and shaded area represent the median and CI95 of the expected values (for an average participant), respectively, obtained from 2,000 model posterior samples. Note that, as not all systems were equally represented in our data (e.g., there was only one session from an iPad device), we obtained and represented the weighted (by the frequency of the participants’ systems) marginal means for each distance in both panels. All means were computed for mean participants’ age and at mean trial number (i.e., at the middle of the task).

We detected chronological effects on RT by which for every 35 completed trials, correct responses were made on average 20 ms faster (β = –20.6ms, CI95 = [–26.1, –15.3]ms). Further, we found that older participants were predominantly slower than younger ones (β = 10.1ms, CI95 = [–2.18, 22.4]ms). This value corresponds to an average increment of ~9 ms for every 10 years of age. Indeed, the impact of age over RTs is a well-documented effect in binary RT decision-making tasks (Ratcliff et al., 2004).

Our modeling strategy also allowed us to understand whether and how the participants’ system (i.e. device, OS, and web browser) affected their RTs. We found that the system indeed affected shift (θ) () but we did not find evidence for an effect of the system on ν nor on σ of RTs (). This means that variability associated with different user device types affected the non-decision portion of the measured RTs (modeled as a component of θ) but did not affect the mean or standard deviation of the decision portion of the RTs (modeled as ν and σ of the decision portion of the RTs, respectively). The main pattern we observed is that RTs from mobile devices (mainly Chrome browser running on Android OS) had larger shifts (θ) than RTs from desktop computers (mainly Chrome browser running on Windows OS) (Δθ = 149ms, CI95 = [128, 168]ms), which is consistent with previous reports (Holden et al., 2019; Pronk et al., 2020; Reimers & Stewart, 2008).

Shifts by participants’ system. We computed the conditional shift (θ) means by system (i.e., for each of the 10 categories of systems, see Methods) using 2,000 posterior samples from the estimated coefficients of the linear predictor of θ (Equation (3)). The colored points’ position and grey horizontal intervals represent the median and CI95 of the posterior samples for each conditional mean, respectively. We used dark pink and blue points to represent mobile and desktop devices, respectively, to show that mobile devices induce larger shifts than desktop devices. The annotation above each point reports the number of participants (N) that completed the task using that specific system. The dark pink and blue vertical dashed lines in the background are the shift means for mobile and desktop devices (weighted by the frequency of the participants’ systems, for each device type), respectively. All means were computed for mean participants’ age.

Error rates were also revealing of a numerical distance effect, by which the greater the numerical distance, the smaller the rate of errors. In line with previous studies (Dehaene et al., 1990; Hinrichs et al., 1981), we observed that participants made ~10% errors for the hardest numerical comparisons (e.g., 64 vs 65, 66 vs 65), and that the error rates dropped as the numerical distance increased (). Specifically, we found that for each unit increase in logarithmic numerical distance, the odds of making an error decreased by 50% on average (φ = –0.71, CI95 = [–0.80, –0.62]).

Although the task was rather short (116 trials), we found a relevant chronological effect (φ = –0.15, CI95 = [–0.27, –0.03]). This implies that for —roughly— every 35 completed trials, the odds of making an error decreased by 15%. This effect possibly represents a learning process by which fewer errors are made as the task unfolds.

To summarize, data collected in an online experiment is adequate to detect the numerical-distance effect on RT and error rate. Notably, the magnitude of this effect closely matches previous results. Moreover, we found that the system that participants used to complete the task (i.e., the OS and web browser of their devices) reliably affected shifts, but we did not find evidence for changes in the mean nor the SD of RTs. We quantified the change in shift for each different system and found that mobile devices generally induced larger shifts than desktop systems.

Discussion

Many paradigms in cognitive science rely on measuring RTs at the millisecond-range. Collecting behavioral data online is attractive but methodologically challenging. Since participants will use different OSs, web browsers, and devices, in potentially distracting contexts, online experiments may resemble the setting of an uncontrolled environment.

In this paper, we show that high quality data (including precise RTs) in binary decisions can be obtained with an R-Shiny application running on mobile devices and desktop systems. As a proof-of-principle, we replicated the well-established numerical distance effect, typified by decreased RTs and error rates when the absolute difference between compared numbers increases (Dehaene et al., 1990; Hinrichs et al., 1981; Krajcsi & Kojouharova, 2017; Moyer & Landauer, 1967; Verguts & De Moor, 2005).

Behavioral studies reported 35–40 years ago found that RTs map linearly onto the log of the numerical distance (logD) with a regression coefficient of β = –78.3ms when the comparison task uses 65 as the standard (Dehaene et al., 1990) with small variations when the standard is set to 55 or 75 (Dehaene, 1989; Dehaene et al., 1990; Hinrichs et al., 1981). Considering the huge technological and statistical changes occurring in the last four decades, it was far from obvious that such findings could be replicated in an online experiment involving fairly uncontrolled conditions.

The precise cognitive mechanisms underlying this phenomenon remain a matter of debate (Dehaene, 1989; Dehaene et al., 1990; Hinrichs et al., 1981; Krajcsi & Kojouharova, 2017; Moyer & Landauer, 1973; Nuerk et al., 2001; Verguts & De Moor, 2005). While our work was not aimed to discern between these accounts, it contributes to the literature by showing that reliable RTs can be measured using R-Shiny, amplifying data collection options for the field.

Our results also allowed us to identify differences in shift (θ) due to participants’ hardware and software such as web browser and OS. We collected data from 10 different combinations of OS, web browser, and type of device (mobile or desktop) that we collectively called system. However, these categories were not equally represented in our data. In fact, most participants completed the task using Chrome browser (running either on Android [mobile] or Windows [desktop] OS), limiting the interpretation of our results to comparisons between such systems. Nonetheless, we found that mobile devices have shifts that are ~150ms larger than those of desktop computers. These differences might reflect OS and browser processing times, the response mode (box-tap for mobile devices, keypress for desktops), and/or hardware limitations. Moreover, by modelling system-specific contributions to shifts we were able to control for device-dependent RT variability. While such variability may not constitute a confound in within-subjects designs, it is critical to address it in between-subjects designs.

In the last decade, many tools have been developed to perform experiments online (e.g., jsPsych, Cognition (www.cognition.run), JATOS (Lange et al., 2015), Gorilla, PsychoPy, Lab.js, LabVanced). While this diversity is positive in its own right, each tool faces a trade-off between flexibility (i.e., customizability) and ease of learning/use. Shiny stands out as an advantageous option for researchers familiarized with R, as it is highly and easily customizable (although specific features might require incorporating HTML/CSS/JavaScript code). Moreover, Shiny-app development can be readily integrated into the workflow of experimental psychologists that use R for data analysis without the need to switch back and forth between frameworks and/or programming languages.

Altogether, this study shows that a well-established RT-based cognitive effect can be robustly captured online, highlighting Shiny as a useful tool to develop replicable online experiments. We showed that online experiments are not only useful for survey-data collection but also for RT-based decision-making tasks, harnessing great power for experimental psychologists. This methodological contribution could pave the way for more massive and cost-efficient data collection protocols across cognitive science at large.

Data Accessibility statement

The datasets generated and/or analyzed in this study (containing data from 19604 trials) are available in the OSF repository: https://osf.io/6gmzs.

Code Accessibility

All code for the app and analyses are available in the OSF repository: https://osf.io/6gmzs. A demo version of the app is available in ShinyApps: https://2exp3.shinyapps.io/numbers/.

Ethics and consent

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Ethics Committee of Centro de Educación Médica e Investigaciones Clínica (protocol ID 435). The authors complied with APA ethical standards in the treatment of their sample.

Informed consent was obtained from all individual participants included in the study.

Table 1

Estimated Effects of the Participants’ System Over ν and σ.

Distributional parameter	Coefficient	Median	Lower	Upper
ν	β system, Windows.Chrome	–0.009	–0.037	0.019
ν	β system, Windows.Firefox	–0.030	–0.086	0.025
ν	β system, Linux.Chrome	–0.041	–0.097	0.015
ν	β system, Linux.Firefox	0.006	–0.130	0.141
ν	β system, Macintosh.Chrome	0.010	–0.126	0.148
ν	β system, Macintosh.Safari	0.025	–0.125	0.178
ν	β system, iPhone.Safari	–0.001	–0.043	0.042
ν	β system, iPhone.Mozilla	–0.012	–0.057	0.033
ν	β system, iPad.Mozilla	–0.061	–0.210	0.095
σ	γ system, Windows.Chrome	–0.010	–0.059	0.039
σ	γ system, Windows.Firefox	0.037	–0.052	0.123
σ	γ system, Linux.Chrome	–0.054	–0.140	0.031
σ	γ system, Linux.Firefox	–0.090	–0.229	0.047
σ	γ system, Macintosh.Chrome	–0.049	–0.202	0.100
σ	γ system, Macintosh.Safari	0.092	–0.051	0.234
σ	γ system, iPhone.Safari	0.044	–0.027	0.111
σ	γ system, iPhone.Mozilla	–0.007	–0.088	0.069
σ	γ system, iPad.Mozilla	0.119	–0.059	0.283

Note: Lower and Upper refer to the lower and upper bounds of the CI95 of each coefficient posterior distribution. Importantly, all these intervals contain 0, suggesting that the participants’ system did not have a relevant effect on either μ or σ of RTs.