Michele Svanera1, Mattia Savardi2, Sergio Benini2, Alberto Signoroni2, Gal Raz3, Talma Hendler4, Lars Muckli5, Rainer Goebel6, Giancarlo Valente6. 1. Department of Information Engineering, University of Brescia, Italy; Institute of Neuroscience and Psychology, University of Glasgow, UK. Electronic address: Michele.Svanera@glasgow.ac.uk. 2. Department of Information Engineering, University of Brescia, Italy. 3. Sagol Brain Institute, Wohl Institute for Advanced Imaging, Tel-Aviv Sourasky Medical Center, Tel-Aviv, Israel; Sagol School of Neuroscience, Tel-Aviv University, Tel-Aviv, Israel; The Steve Tisch School of Film and Television, Tel-Aviv University, Tel-Aviv, Israel. 4. Sagol Brain Institute, Wohl Institute for Advanced Imaging, Tel-Aviv Sourasky Medical Center, Tel-Aviv, Israel; The School of Psychological Sciences, Tel-Aviv University, Tel-Aviv, Israel; Sagol School of Neuroscience, Tel-Aviv University, Tel-Aviv, Israel; Sackler Faculty of Medicine, Tel-Aviv University, Tel-Aviv, Israel. 5. Institute of Neuroscience and Psychology, University of Glasgow, UK. 6. Department of Cognitive Neuroscience, Maastricht University, The Netherlands.
Abstract
BACKGROUND: Deep neural networks have revolutionised machine learning, with unparalleled performance in object classification. However, in brain imaging (e.g., fMRI), the direct application of Convolutional Neural Networks (CNN) to decoding subject states or perception from imaging data seems impractical given the scarcity of available data. NEW METHOD: In this work we propose a robust method to transfer information from deep learning (DL) features to brain fMRI data with the goal of decoding. By adopting Reduced Rank Regression with Ridge Regularisation we establish a multivariate link between imaging data and the fully connected layer (fc7) of a CNN. We exploit the reconstructed fc7 features by performing an object image classification task on two datasets: one of the largest fMRI databases, taken from different scanners from more than two hundred subjects watching different movie clips, and another with fMRI data taken while watching static images. RESULTS: The fc7 features could be significantly reconstructed from the imaging data, and led to significant decoding performance. COMPARISON WITH EXISTING METHODS: The decoding based on reconstructed fc7 outperformed the decoding based on imaging data alone. CONCLUSION: In this work we show how to improve fMRI-based decoding benefiting from the mapping between functional data and CNN features. The potential advantage of the proposed method is twofold: the extraction of stimuli representations by means of an automatic procedure (unsupervised) and the embedding of high-dimensional neuroimaging data onto a space designed for visual object discrimination, leading to a more manageable space from dimensionality point of view.
BACKGROUND: Deep neural networks have revolutionised machine learning, with unparalleled performance in object classification. However, in brain imaging (e.g., fMRI), the direct application of Convolutional Neural Networks (CNN) to decoding subject states or perception from imaging data seems impractical given the scarcity of available data. NEW METHOD: In this work we propose a robust method to transfer information from deep learning (DL) features to brain fMRI data with the goal of decoding. By adopting Reduced Rank Regression with Ridge Regularisation we establish a multivariate link between imaging data and the fully connected layer (fc7) of a CNN. We exploit the reconstructed fc7 features by performing an object image classification task on two datasets: one of the largest fMRI databases, taken from different scanners from more than two hundred subjects watching different movie clips, and another with fMRI data taken while watching static images. RESULTS: The fc7 features could be significantly reconstructed from the imaging data, and led to significant decoding performance. COMPARISON WITH EXISTING METHODS: The decoding based on reconstructed fc7 outperformed the decoding based on imaging data alone. CONCLUSION: In this work we show how to improve fMRI-based decoding benefiting from the mapping between functional data and CNN features. The potential advantage of the proposed method is twofold: the extraction of stimuli representations by means of an automatic procedure (unsupervised) and the embedding of high-dimensional neuroimaging data onto a space designed for visual object discrimination, leading to a more manageable space from dimensionality point of view.
Authors: Muhammad Bilal Qureshi; Laraib Azad; Muhammad Shuaib Qureshi; Sheraz Aslam; Ayman Aljarbouh; Muhammad Fayaz Journal: Comput Math Methods Med Date: 2022-03-01 Impact factor: 2.238