S P Roels1, H Bossier2, T Loeys2, B Moerkerke2. 1. Department of Data Analysis, Ghent University, H. Dunantlaan 1, B-9000 Ghent, Belgium. Electronic address: sanne.roels@ugent.be. 2. Department of Data Analysis, Ghent University, H. Dunantlaan 1, B-9000 Ghent, Belgium.
Abstract
BACKGROUND: Carp (2012) demonstrated the large variability that is present in the method sections of fMRI studies. This methodological variability between studies limits reproducible research. NEW METHOD: Evaluation protocols for methods used in fMRI should include data-analytical stability measures quantifying the variability in results following choices in the methods. Data-analytical stability can be seen as a proxy for reproducibility. To illustrate how one can perform such evaluations, we study two competing approaches for topological feature based inference (random field theory and permutation based testing) and two competing methods for smoothing (Gaussian smoothing and adaptive smoothing). We compare these approaches from the perspective of data-analytical stability in real data, and additionally consider validity and reliability in simulations. RESULTS: There is clear evidence that choices in the methods impact the validity, reliability and stability of the results. For the particular comparison studied, we find that permutation based methods render the most valid results. For stability and reliability, the performance of different smoothing and inference types depends on the setting. However, while being more reliable, adaptive smoothing can evoke less stable results when using larger kernel width, especially with cluster size based permutation inference. COMPARISON WITH EXISTING METHODS: While existing evaluation methods focus on validity and reliability, we show that data-analytical stability enables to further distinguish between performance of different methods. CONCLUSION: Data-analytical stability is an important additional criterion that can easily be incorporated in evaluation protocols.
BACKGROUND:Carp (2012) demonstrated the large variability that is present in the method sections of fMRI studies. This methodological variability between studies limits reproducible research. NEW METHOD: Evaluation protocols for methods used in fMRI should include data-analytical stability measures quantifying the variability in results following choices in the methods. Data-analytical stability can be seen as a proxy for reproducibility. To illustrate how one can perform such evaluations, we study two competing approaches for topological feature based inference (random field theory and permutation based testing) and two competing methods for smoothing (Gaussian smoothing and adaptive smoothing). We compare these approaches from the perspective of data-analytical stability in real data, and additionally consider validity and reliability in simulations. RESULTS: There is clear evidence that choices in the methods impact the validity, reliability and stability of the results. For the particular comparison studied, we find that permutation based methods render the most valid results. For stability and reliability, the performance of different smoothing and inference types depends on the setting. However, while being more reliable, adaptive smoothing can evoke less stable results when using larger kernel width, especially with cluster size based permutation inference. COMPARISON WITH EXISTING METHODS: While existing evaluation methods focus on validity and reliability, we show that data-analytical stability enables to further distinguish between performance of different methods. CONCLUSION: Data-analytical stability is an important additional criterion that can easily be incorporated in evaluation protocols.