| Literature DB >> 32483374 |
Rotem Botvinik-Nezer1,2,3, Felix Holzmeister4, Colin F Camerer5, Anna Dreber6,7, Juergen Huber4, Magnus Johannesson6, Michael Kirchler4, Roni Iwanir1,2, Jeanette A Mumford8, R Alison Adcock9,10, Paolo Avesani11,12, Blazej M Baczkowski13, Aahana Bajracharya14, Leah Bakst15,16, Sheryl Ball17,18, Marco Barilari19, Nadège Bault20, Derek Beaton21, Julia Beitner22,23, Roland G Benoit24, Ruud M W J Berkers24, Jamil P Bhanji25, Bharat B Biswal26,27, Sebastian Bobadilla-Suarez28, Tiago Bortolini29, Katherine L Bottenhorn30, Alexander Bowring31, Senne Braem32,33, Hayley R Brooks34, Emily G Brudner25, Cristian B Calderon32, Julia A Camilleri35,36, Jaime J Castrellon9,37, Luca Cecchetti38, Edna C Cieslik35,36, Zachary J Cole39, Olivier Collignon12,19, Robert W Cox40, William A Cunningham41, Stefan Czoschke42, Kamalaker Dadi43, Charles P Davis44,45,46, Alberto De Luca47, Mauricio R Delgado25, Lysia Demetriou48,49, Jeffrey B Dennison50, Xin Di26,27, Erin W Dickie51,52, Ekaterina Dobryakova53, Claire L Donnat54, Juergen Dukart35,36, Niall W Duncan55,56, Joke Durnez57, Amr Eed58, Simon B Eickhoff35,36, Andrew Erhart34, Laura Fontanesi59, G Matthew Fricke60, Shiguang Fu61,62, Adriana Galván63, Remi Gau19, Sarah Genon35,36, Tristan Glatard64, Enrico Glerean65, Jelle J Goeman66, Sergej A E Golowin55, Carlos González-García32, Krzysztof J Gorgolewski67, Cheryl L Grady21, Mikella A Green9,37, João F Guassi Moreira63, Olivia Guest28,68, Shabnam Hakimi9, J Paul Hamilton69, Roeland Hancock45,46, Giacomo Handjaras38, Bronson B Harry70, Colin Hawco71, Peer Herholz72, Gabrielle Herman71, Stephan Heunis73,74, Felix Hoffstaedter35,36, Jeremy Hogeveen75,76, Susan Holmes54, Chuan-Peng Hu77, Scott A Huettel37, Matthew E Hughes78, Vittorio Iacovella12, Alexandru D Iordan79, Peder M Isager80, Ayse I Isik81, Andrew Jahn82, Matthew R Johnson39,83, Tom Johnstone78, Michael J E Joseph71, Anthony C Juliano84, Joseph W Kable85,86, Michalis Kassinopoulos87, Cemal Koba38, Xiang-Zhen Kong88, Timothy R Koscik89, Nuri Erkut Kucukboyaci53,90, Brice A Kuhl91, Sebastian Kupek92, Angela R Laird93, Claus Lamm94,95, Robert Langner35,36, Nina Lauharatanahirun96,97, Hongmi Lee98, Sangil Lee85, Alexander Leemans47, Andrea Leo38, Elise Lesage32, Flora Li99,100, Monica Y C Li44,45,46,101, Phui Cheng Lim39,83, Evan N Lintz39, Schuyler W Liphardt102, Annabel B Losecaat Vermeer94, Bradley C Love28,103, Michael L Mack41, Norberto Malpica104, Theo Marins29, Camille Maumet105, Kelsey McDonald37, Joseph T McGuire15,16, Helena Melero104,106,107, Adriana S Méndez Leal63, Benjamin Meyer77,108, Kristin N Meyer109, Glad Mihai110,111, Georgios D Mitsis112, Jorge Moll29,67, Dylan M Nielson113, Gustav Nilsonne114,115, Michael P Notter116, Emanuele Olivetti11,12, Adrian I Onicas38, Paolo Papale38,117, Kaustubh R Patil35,36, Jonathan E Peelle14, Alexandre Pérez72, Doris Pischedda118,119,120, Jean-Baptiste Poline72,121, Yanina Prystauka44,45,46, Shruti Ray26, Patricia A Reuter-Lorenz79, Richard C Reynolds122, Emiliano Ricciardi38, Jenny R Rieck21, Anais M Rodriguez-Thompson109, Anthony Romyn41, Taylor Salo30, Gregory R Samanez-Larkin9,37, Emilio Sanz-Morales104, Margaret L Schlichting41, Douglas H Schultz39,83, Qiang Shen61,62, Margaret A Sheridan109, Jennifer A Silvers63, Kenny Skagerlund123,124, Alec Smith17,18, David V Smith50, Peter Sokol-Hessner34, Simon R Steinkamp125, Sarah M Tashjian63, Bertrand Thirion43, John N Thorp126, Gustav Tinghög127,128, Loreen Tisdall67,129, Steven H Tompson96, Claudio Toro-Serey15,16, Juan Jesus Torre Tresols43, Leonardo Tozzi130, Vuong Truong55,56, Luca Turella12, Anna E van 't Veer131, Tom Verguts32, Jean M Vettel132,133,134, Sagana Vijayarajah41, Khoi Vo9,37, Matthew B Wall135,136,137, Wouter D Weeda131, Susanne Weis35,36, David J White138, David Wisniewski32, Alba Xifra-Porxas87, Emily A Yearling44,45,46, Sangsuk Yoon139, Rui Yuan130, Kenneth S L Yuen77,108, Lei Zhang94, Xu Zhang45,46,140, Joshua E Zosky39,83, Thomas E Nichols141, Russell A Poldrack142, Tom Schonberg143,144.
Abstract
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.Entities:
Mesh：
Year: 2020 PMID： 32483374 PMCID： PMC7771346 DOI： 10.1038/s41586-020-2314-9
Source DB: PubMed Journal: Nature ISSN： 0028-0836 Impact factor: 69.504