Stephanie Noble1, Dustin Scheinost2, R Todd Constable3. 1. Interdepartmental Neuroscience Program, Yale University, USA. Electronic address: stephanie.noble@yale.edu. 2. Interdepartmental Neuroscience Program, Yale University, USA; Department of Radiology and Biomedical Imaging, Yale School of Medicine, USA; Department of Statistics and Data Science, Yale University, USA; Child Study Center, Yale School of Medicine, USA. 3. Interdepartmental Neuroscience Program, Yale University, USA; Department of Radiology and Biomedical Imaging, Yale School of Medicine, USA; Department of Neurosurgery, Yale School of Medicine, USA.
Abstract
BACKGROUND: Once considered mere noise, fMRI-based functional connectivity has become a major neuroscience tool in part due to early studies demonstrating its reliability. These fundamental studies revealed only the tip of the iceberg; over the past decade, many test-retest reliability studies have continued to add nuance to our understanding of this complex topic. A summary of these diverse and at times contradictory perspectives is needed. OBJECTIVES: We aimed to summarize the existing knowledge regarding test-retest reliability of functional connectivity at the most basic unit of analysis: the individual edge level. This entailed (1) a meta-analytic estimate of reliability and (2) a review of factors influencing reliability. METHODS: A search of Scopus was conducted to identify studies that estimated edge-level test-retest reliability. To facilitate comparisons across studies, eligibility was restricted to studies measuring reliability via the intraclass correlation coefficient (ICC). The meta-analysis included a random effects pooled estimate of mean edge-level ICC, with studies nested within datasets. The review included a narrative summary of factors influencing edge-level ICC. RESULTS: From an initial pool of 212 studies, 44 studies were identified for the qualitative review and 25 studies for quantitative meta-analysis. On average, individual edges exhibited a "poor" ICC of 0.29 (95% CI = 0.23 to 0.36). The most reliable measurements tended to involve: (1) stronger, within-network, cortical edges, (2) eyes open, awake, and active recordings, (3) more within-subject data, (4) shorter test-retest intervals, (5) no artifact correction (likely due in part to reliable artifact), and (6) full correlation-based connectivity with shrinkage. CONCLUSION: This study represents the first meta-analysis and systematic review investigating test-retest reliability of edge-level functional connectivity. Key findings suggest there is room for improvement, but care should be taken to avoid promoting reliability at the expense of validity. By pooling existing knowledge regarding this key facet of accuracy, this study supports broader efforts to improve inferences in the field.
BACKGROUND: Once considered mere noise, fMRI-based functional connectivity has become a major neuroscience tool in part due to early studies demonstrating its reliability. These fundamental studies revealed only the tip of the iceberg; over the past decade, many test-retest reliability studies have continued to add nuance to our understanding of this complex topic. A summary of these diverse and at times contradictory perspectives is needed. OBJECTIVES: We aimed to summarize the existing knowledge regarding test-retest reliability of functional connectivity at the most basic unit of analysis: the individual edge level. This entailed (1) a meta-analytic estimate of reliability and (2) a review of factors influencing reliability. METHODS: A search of Scopus was conducted to identify studies that estimated edge-level test-retest reliability. To facilitate comparisons across studies, eligibility was restricted to studies measuring reliability via the intraclass correlation coefficient (ICC). The meta-analysis included a random effects pooled estimate of mean edge-level ICC, with studies nested within datasets. The review included a narrative summary of factors influencing edge-level ICC. RESULTS: From an initial pool of 212 studies, 44 studies were identified for the qualitative review and 25 studies for quantitative meta-analysis. On average, individual edges exhibited a "poor" ICC of 0.29 (95% CI = 0.23 to 0.36). The most reliable measurements tended to involve: (1) stronger, within-network, cortical edges, (2) eyes open, awake, and active recordings, (3) more within-subject data, (4) shorter test-retest intervals, (5) no artifact correction (likely due in part to reliable artifact), and (6) full correlation-based connectivity with shrinkage. CONCLUSION: This study represents the first meta-analysis and systematic review investigating test-retest reliability of edge-level functional connectivity. Key findings suggest there is room for improvement, but care should be taken to avoid promoting reliability at the expense of validity. By pooling existing knowledge regarding this key facet of accuracy, this study supports broader efforts to improve inferences in the field.
Authors: Bharat B Biswal; Maarten Mennes; Xi-Nian Zuo; Suril Gohel; Clare Kelly; Steve M Smith; Christian F Beckmann; Jonathan S Adelstein; Randy L Buckner; Stan Colcombe; Anne-Marie Dogonowski; Monique Ernst; Damien Fair; Michelle Hampson; Matthew J Hoptman; James S Hyde; Vesa J Kiviniemi; Rolf Kötter; Shi-Jiang Li; Ching-Po Lin; Mark J Lowe; Clare Mackay; David J Madden; Kristoffer H Madsen; Daniel S Margulies; Helen S Mayberg; Katie McMahon; Christopher S Monk; Stewart H Mostofsky; Bonnie J Nagel; James J Pekar; Scott J Peltier; Steven E Petersen; Valentin Riedl; Serge A R B Rombouts; Bart Rypma; Bradley L Schlaggar; Sein Schmidt; Rachael D Seidler; Greg J Siegle; Christian Sorg; Gao-Jun Teng; Juha Veijola; Arno Villringer; Martin Walter; Lihong Wang; Xu-Chu Weng; Susan Whitfield-Gabrieli; Peter Williamson; Christian Windischberger; Yu-Feng Zang; Hong-Ying Zhang; F Xavier Castellanos; Michael P Milham Journal: Proc Natl Acad Sci U S A Date: 2010-02-22 Impact factor: 11.205
Authors: Stephanie Noble; Dustin Scheinost; Emily S Finn; Xilin Shen; Xenophon Papademetris; Sarah C McEwen; Carrie E Bearden; Jean Addington; Bradley Goodyear; Kristin S Cadenhead; Heline Mirzakhanian; Barbara A Cornblatt; Doreen M Olvet; Daniel H Mathalon; Thomas H McGlashan; Diana O Perkins; Aysenil Belger; Larry J Seidman; Heidi Thermenos; Ming T Tsuang; Theo G M van Erp; Elaine F Walker; Stephan Hamann; Scott W Woods; Tyrone D Cannon; R Todd Constable Journal: Neuroimage Date: 2016-10-13 Impact factor: 6.556
Authors: Andreia V Faria; Suresh E Joel; Yajing Zhang; Kenichi Oishi; Peter C M van Zjil; Michael I Miller; James J Pekar; Susumu Mori Journal: Neuroimage Date: 2012-04-03 Impact factor: 6.556
Authors: Jie Song; Alok S Desphande; Timothy B Meier; Dana L Tudorascu; Svyatoslav Vergun; Veena A Nair; Bharat B Biswal; Mary E Meyerand; Rasmus M Birn; Pierre Bellec; Vivek Prabhakaran Journal: PLoS One Date: 2012-12-05 Impact factor: 3.240
Authors: Janine Bijsterbosch; Samuel J Harrison; Saad Jbabdi; Mark Woolrich; Christian Beckmann; Stephen Smith; Eugene P Duff Journal: Nat Neurosci Date: 2020-10-26 Impact factor: 24.884
Authors: Ashley N Nielsen; Deanna M Barch; Steven E Petersen; Bradley L Schlaggar; Deanna J Greene Journal: Biol Psychiatry Cogn Neurosci Neuroimaging Date: 2019-11-27
Authors: Desmond J Oathes; Jared P Zimmerman; Romain Duprat; Seda S Japp; Morgan Scully; Benjamin M Rosenberg; Matthew W Flounders; Hannah Long; Joseph A Deluisi; Mark Elliott; Gavriella Shandler; Russell T Shinohara; Kristin A Linn Journal: Exp Brain Res Date: 2021-02-09 Impact factor: 1.972