Ian C Ballard1, Samuel M McClure2. 1. Neurosciences Graduate Training Program, Stanford University, Stanford, CA 94305, USA; Helen Wills Neuroscience Institute, University of California, Berkeley, CA, 94720, USA; Department of Psychology, Arizona State University, Tempe, AZ 85287, USA. Electronic address: iancballard@gmail.com. 2. Department of Psychology, Arizona State University, Tempe, AZ 85287, USA.
Abstract
BACKGROUND: Reinforcement learning models provide excellent descriptions of learning in multiple species across a variety of tasks. Many researchers are interested in relating parameters of reinforcement learning models to neural measures, psychological variables or experimental manipulations. We demonstrate that parameter identification is difficult because a range of parameter values provide approximately equal quality fits to data. This identification problem has a large impact on power: we show that a researcher who wants to detect a medium sized correlation (r = .3) with 80% power between a variable and learning rate must collect 60% more subjects than specified by a typical power analysis in order to account for the noise introduced by model fitting. NEW METHOD: We derive a Bayesian optimal model fitting technique that takes advantage of information contained in choices and reaction times to constrain parameter estimates. RESULTS: We show using simulation and empirical data that this method substantially improves the ability to recover learning rates. COMPARISON WITH EXISTING METHODS: We compare this method against the use of Bayesian priors. We show in simulations that the combined use of Bayesian priors and reaction times confers the highest parameter identifiability. However, in real data where the priors may have been misspecified, the use of Bayesian priors interferes with the ability of reaction time data to improve parameter identifiability. CONCLUSIONS: We present a simple technique that takes advantage of readily available data to substantially improve the quality of inferences that can be drawn from parameters of reinforcement learning models.
BACKGROUND: Reinforcement learning models provide excellent descriptions of learning in multiple species across a variety of tasks. Many researchers are interested in relating parameters of reinforcement learning models to neural measures, psychological variables or experimental manipulations. We demonstrate that parameter identification is difficult because a range of parameter values provide approximately equal quality fits to data. This identification problem has a large impact on power: we show that a researcher who wants to detect a medium sized correlation (r = .3) with 80% power between a variable and learning rate must collect 60% more subjects than specified by a typical power analysis in order to account for the noise introduced by model fitting. NEW METHOD: We derive a Bayesian optimal model fitting technique that takes advantage of information contained in choices and reaction times to constrain parameter estimates. RESULTS: We show using simulation and empirical data that this method substantially improves the ability to recover learning rates. COMPARISON WITH EXISTING METHODS: We compare this method against the use of Bayesian priors. We show in simulations that the combined use of Bayesian priors and reaction times confers the highest parameter identifiability. However, in real data where the priors may have been misspecified, the use of Bayesian priors interferes with the ability of reaction time data to improve parameter identifiability. CONCLUSIONS: We present a simple technique that takes advantage of readily available data to substantially improve the quality of inferences that can be drawn from parameters of reinforcement learning models.
Authors: Michael J Frank; Ahmed A Moustafa; Heather M Haughey; Tim Curran; Kent E Hutchison Journal: Proc Natl Acad Sci U S A Date: 2007-10-03 Impact factor: 11.205
Authors: Jaron T Colas; Neil M Dundon; Raphael T Gerraty; Natalie M Saragosa-Harris; Karol P Szymula; Koranis Tanwisuth; J Michael Tyszka; Camilla van Geen; Harang Ju; Arthur W Toga; Joshua I Gold; Dani S Bassett; Catherine A Hartley; Daphna Shohamy; Scott T Grafton; John P O'Doherty Journal: Hum Brain Mapp Date: 2022-07-21 Impact factor: 5.399
Authors: Bilal A Bari; Megan J Moerke; Hank P Jedema; Devin P Effinger; Jeremiah Y Cohen; Charles W Bradberry Journal: Behav Neurosci Date: 2021-09-27 Impact factor: 1.912
Authors: Nitzan Shahar; Tobias U Hauser; Michael Moutoussis; Rani Moran; Mehdi Keramati; Raymond J Dolan Journal: PLoS Comput Biol Date: 2019-02-13 Impact factor: 4.779