Literature DB >> 32127477

Science is not a signal detection problem.

Brent M Wilson1, Christine R Harris2, John T Wixted1.   

Abstract

The perceived replication crisis and the reforms designed to address it are grounded in the notion that science is a binary signal detection problem. However, contrary to null hypothesis significance testing (NHST) logic, the magnitude of the underlying effect size for a given experiment is best conceptualized as a random draw from a continuous distribution, not as a random draw from a dichotomous distribution (null vs. alternative). Moreover, because continuously distributed effects selected using a P < 0.05 filter must be inflated, the fact that they are smaller when replicated (reflecting regression to the mean) is no reason to sound the alarm. Considered from this perspective, recent replication efforts suggest that most published P < 0.05 scientific findings are "true" (i.e., in the correct direction), with observed effect sizes that are inflated to varying degrees. We propose that original science is a screening process, one that adopts NHST logic as a useful fiction for selecting true effects that are potentially large enough to be of interest to other scientists. Unlike original science, replication science seeks to precisely measure the underlying effect size associated with an experimental protocol via large-N direct replication, without regard for statistical significance. Registered reports are well suited to (often resource-intensive) direct replications, which should focus on influential findings and be published regardless of outcome. Conceptual replications play an important but separate role in validating theories. However, because they are part of NHST-based original science, conceptual replications cannot serve as the field's self-correction mechanism. Only direct replications can do that.

Keywords:  null hypothesis significance testing; replication crisis; signal detection theory

Year:  2020        PMID: 32127477      PMCID: PMC7084063          DOI: 10.1073/pnas.1914237117

Source DB:  PubMed          Journal:  Proc Natl Acad Sci U S A        ISSN: 0027-8424            Impact factor:   11.205


  37 in total

1.  False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant.

Authors:  Joseph P Simmons; Leif D Nelson; Uri Simonsohn
Journal:  Psychol Sci       Date:  2011-10-17

2.  The forgotten history of signal detection theory.

Authors:  John T Wixted
Journal:  J Exp Psychol Learn Mem Cogn       Date:  2019-06-27       Impact factor: 3.051

Review 3.  Power failure: why small sample size undermines the reliability of neuroscience.

Authors:  Katherine S Button; John P A Ioannidis; Claire Mokrysz; Brian A Nosek; Jonathan Flint; Emma S J Robinson; Marcus R Munafò
Journal:  Nat Rev Neurosci       Date:  2013-04-10       Impact factor: 34.870

4.  Response to Comment on "Estimating the reproducibility of psychological science".

Authors:  Christopher J Anderson; Štěpán Bahník; Michael Barnett-Cowan; Frank A Bosco; Jesse Chandler; Christopher R Chartier; Felix Cheung; Cody D Christopherson; Andreas Cordes; Edward J Cremata; Nicolas Della Penna; Vivien Estel; Anna Fedor; Stanka A Fitneva; Michael C Frank; James A Grange; Joshua K Hartshorne; Fred Hasselman; Felix Henninger; Marije van der Hulst; Kai J Jonas; Calvin K Lai; Carmel A Levitan; Jeremy K Miller; Katherine S Moore; Johannes M Meixner; Marcus R Munafò; Koen I Neijenhuijs; Gustav Nilsonne; Brian A Nosek; Franziska Plessow; Jason M Prenoveau; Ashley A Ricker; Kathleen Schmidt; Jeffrey R Spies; Stefan Stieger; Nina Strohminger; Gavin B Sullivan; Robbie C M van Aert; Marcel A L M van Assen; Wolf Vanpaemel; Michelangelo Vianello; Martin Voracek; Kellylynn Zuni
Journal:  Science       Date:  2016-03-04       Impact factor: 47.728

5.  Comparing meta-analyses and preregistered multiple-laboratory replication projects.

Authors:  Amanda Kvarven; Eirik Strømland; Magnus Johannesson
Journal:  Nat Hum Behav       Date:  2019-12-23

6.  Most reported genetic associations with general intelligence are probably false positives.

Authors:  Christopher F Chabris; Benjamin M Hebert; Daniel J Benjamin; Jonathan Beauchamp; David Cesarini; Matthijs van der Loos; Magnus Johannesson; Patrik K E Magnusson; Paul Lichtenstein; Craig S Atwood; Jeremy Freese; Taissa S Hauser; Robert M Hauser; Nicholas Christakis; David Laibson
Journal:  Psychol Sci       Date:  2012-09-24

7.  Rein in the four horsemen of irreproducibility.

Authors:  Dorothy Bishop
Journal:  Nature       Date:  2019-04       Impact factor: 49.962

8.  The preregistration revolution.

Authors:  Brian A Nosek; Charles R Ebersole; Alexander C DeHaven; David T Mellor
Journal:  Proc Natl Acad Sci U S A       Date:  2018-03-13       Impact factor: 12.779

9.  The impact of sleep on eyewitness identifications.

Authors:  D P Morgan; J Tamminen; T M Seale-Carlisle; L Mickes
Journal:  R Soc Open Sci       Date:  2019-12-04       Impact factor: 2.963

10.  Low replicability can support robust and efficient science.

Authors:  Stephan Lewandowsky; Klaus Oberauer
Journal:  Nat Commun       Date:  2020-01-17       Impact factor: 14.919

View more
  10 in total

1.  Consideration of Sample Size in Neuroscience Studies.

Authors: 
Journal:  J Neurosci       Date:  2020-05-20       Impact factor: 6.167

2.  Signal detection continues to be part of science.

Authors:  Jolynn Pek; Duane T Wegener; Gary H McClelland
Journal:  Proc Natl Acad Sci U S A       Date:  2020-06-16       Impact factor: 11.205

3.  Reply to Pek et al.: Science is not the signal detection problem it is ordinarily thought to be.

Authors:  Brent M Wilson; Christine R Harris; John T Wixted
Journal:  Proc Natl Acad Sci U S A       Date:  2020-06-16       Impact factor: 11.205

4.  Examining replicability in addictions research: How to assess and ways forward.

Authors:  Matthew R Pearson; Frank J Schwebel; Dylan K Richards; Katie Witkiewitz
Journal:  Psychol Addict Behav       Date:  2021-08-26

5.  Effects of age on goal-dependent modulation of episodic memory retrieval.

Authors:  Sabina Srokova; Paul F Hill; Rachael L Elward; Michael D Rugg
Journal:  Neurobiol Aging       Date:  2021-02-09       Impact factor: 5.133

6.  Sleep deprivation and memory: Meta-analytic reviews of studies on sleep deprivation before and after learning.

Authors:  Chloe R Newbury; Rebecca Crowley; Kathleen Rastle; Jakke Tamminen
Journal:  Psychol Bull       Date:  2021-11       Impact factor: 17.737

7.  Science as collaborative knowledge generation.

Authors:  Naomi Ellemers
Journal:  Br J Soc Psychol       Date:  2020-12-07

8.  Acetonitrile Adducts of Tranexamic Acid as Sensitive Ions for Quantification at Residue Levels in Human Plasma by UHPLC-MS/MS.

Authors:  Eduarda M P Silva; Luisa Barreiros; Sara R Fernandes; Paula Sá; João P Prates Ramalho; Marcela A Segundo
Journal:  Pharmaceuticals (Basel)       Date:  2021-11-23

9.  The tainted altruism effect: a successful pre-registered replication.

Authors:  Valerie Alcala; Kendra Johnson; Caroline Steele; Juanshu Wu; Donglai Zhang; Harold Pashler
Journal:  R Soc Open Sci       Date:  2022-01-26       Impact factor: 2.963

10.  Using Students' Concept-building Tendencies to Better Characterize Average-Performing Student Learning and Problem-Solving Approaches in General Chemistry.

Authors:  Regina F Frey; Mark A McDaniel; Diane M Bunce; Michael J Cahill; Martin D Perry
Journal:  CBE Life Sci Educ       Date:  2020-09       Impact factor: 3.325

  10 in total

北京卡尤迪生物科技股份有限公司 © 2022-2023.