Irene Sturm1, Sebastian Lapuschkin2, Wojciech Samek3, Klaus-Robert Müller4. 1. Machine Learning Group, Berlin Institute of Technology, Marchstr. 23, 10587 Berlin, Germany. Electronic address: irene.sturm@tu-berlin.de. 2. Machine Learning Group, Fraunhofer Heinrich Hertz Institute, Einsteinufer 37, 10587 Berlin, Germany. Electronic address: sebastian.lapuschkin@hhi.fraunhofer.de. 3. Machine Learning Group, Fraunhofer Heinrich Hertz Institute, Einsteinufer 37, 10587 Berlin, Germany. Electronic address: wojciech.samek@hhi.fraunhofer.de. 4. Machine Learning Group, Berlin Institute of Technology, Marchstr. 23, 10587 Berlin, Germany; Department of Brain and Cognitive Engineering, Korea University, Anam-dong, Seongbuk-gu, Seoul 136-713, Republic of Korea. Electronic address: klaus-robert.mueller@tu-berlin.de.
Abstract
BACKGROUND: In cognitive neuroscience the potential of deep neural networks (DNNs) for solving complex classification tasks is yet to be fully exploited. The most limiting factor is that DNNs as notorious 'black boxes' do not provide insight into neurophysiological phenomena underlying a decision. Layer-wise relevance propagation (LRP) has been introduced as a novel method to explain individual network decisions. NEW METHOD: We propose the application of DNNs with LRP for the first time for EEG data analysis. Through LRP the single-trial DNN decisions are transformed into heatmaps indicating each data point's relevance for the outcome of the decision. RESULTS: DNN achieves classification accuracies comparable to those of CSP-LDA. In subjects with low performance subject-to-subject transfer of trained DNNs can improve the results. The single-trial LRP heatmaps reveal neurophysiologically plausible patterns, resembling CSP-derived scalp maps. Critically, while CSP patterns represent class-wise aggregated information, LRP heatmaps pinpoint neural patterns to single time points in single trials. COMPARISON WITH EXISTING METHOD(S): We compare the classification performance of DNNs to that of linear CSP-LDA on two data sets related to motor-imaginary BCI. CONCLUSION: We have demonstrated that DNN is a powerful non-linear tool for EEG analysis. With LRP a new quality of high-resolution assessment of neural activity can be reached. LRP is a potential remedy for the lack of interpretability of DNNs that has limited their utility in neuroscientific applications. The extreme specificity of the LRP-derived heatmaps opens up new avenues for investigating neural activity underlying complex perception or decision-related processes.
BACKGROUND: In cognitive neuroscience the potential of deep neural networks (DNNs) for solving complex classification tasks is yet to be fully exploited. The most limiting factor is that DNNs as notorious 'black boxes' do not provide insight into neurophysiological phenomena underlying a decision. Layer-wise relevance propagation (LRP) has been introduced as a novel method to explain individual network decisions. NEW METHOD: We propose the application of DNNs with LRP for the first time for EEG data analysis. Through LRP the single-trial DNN decisions are transformed into heatmaps indicating each data point's relevance for the outcome of the decision. RESULTS: DNN achieves classification accuracies comparable to those of CSP-LDA. In subjects with low performance subject-to-subject transfer of trained DNNs can improve the results. The single-trial LRP heatmaps reveal neurophysiologically plausible patterns, resembling CSP-derived scalp maps. Critically, while CSP patterns represent class-wise aggregated information, LRP heatmaps pinpoint neural patterns to single time points in single trials. COMPARISON WITH EXISTING METHOD(S): We compare the classification performance of DNNs to that of linear CSP-LDA on two data sets related to motor-imaginary BCI. CONCLUSION: We have demonstrated that DNN is a powerful non-linear tool for EEG analysis. With LRP a new quality of high-resolution assessment of neural activity can be reached. LRP is a potential remedy for the lack of interpretability of DNNs that has limited their utility in neuroscientific applications. The extreme specificity of the LRP-derived heatmaps opens up new avenues for investigating neural activity underlying complex perception or decision-related processes.
Authors: Miguel Angrick; Christian Herff; Emily Mugler; Matthew C Tate; Marc W Slutzky; Dean J Krusienski; Tanja Schultz Journal: J Neural Eng Date: 2019-03-04 Impact factor: 5.379
Authors: Chenyang Shen; Yesenia Gonzalez; Peter Klages; Nan Qin; Hyunuk Jung; Liyuan Chen; Dan Nguyen; Steve B Jiang; Xun Jia Journal: Phys Med Biol Date: 2019-05-29 Impact factor: 3.609
Authors: Bettina Mieth; Alexandre Rozier; Juan Antonio Rodriguez; Marina M C Höhne; Nico Görnitz; Klaus-Robert Müller Journal: NAR Genom Bioinform Date: 2021-07-20