Vinay Jayaram1, Alexandre Barachant. 1. Empirical Inference, Max Planck Institute for Intelligent Systems, Tübingen, Germany. IMPRS for Cognitive and Systems Neuroscience, University of Tübingen, Tübingen, Germany.
Abstract
OBJECTIVE: Brain-computer interface (BCI) algorithm development has long been hampered by two major issues: small sample sets and a lack of reproducibility. We offer a solution to both of these problems via a software suite that streamlines both the issues of finding and preprocessing data in a reliable manner, as well as that of using a consistent interface for machine learning methods. APPROACH: By building on recent advances in software for signal analysis implemented in the MNE toolkit, and the unified framework for machine learning offered by the scikit-learn project, we offer a system that can improve BCI algorithm development. This system is fully open-source under the BSD licence and available at https://github.com/NeuroTechX/moabb. MAIN RESULTS: We analyze a set of state-of-the-art decoding algorithms across 12 open access datasets, including over 250 subjects. Our results show that even for the best methods, there are datasets which do not show significant improvements, and further that many previously validated methods do not generalize well outside the datasets they were tested on. SIGNIFICANCE: Our analysis confirms that BCI algorithms validated on single datasets are not representative, highlighting the need for more robust validation in the machine learning for BCIs community.
OBJECTIVE: Brain-computer interface (BCI) algorithm development has long been hampered by two major issues: small sample sets and a lack of reproducibility. We offer a solution to both of these problems via a software suite that streamlines both the issues of finding and preprocessing data in a reliable manner, as well as that of using a consistent interface for machine learning methods. APPROACH: By building on recent advances in software for signal analysis implemented in the MNE toolkit, and the unified framework for machine learning offered by the scikit-learn project, we offer a system that can improve BCI algorithm development. This system is fully open-source under the BSD licence and available at https://github.com/NeuroTechX/moabb. MAIN RESULTS: We analyze a set of state-of-the-art decoding algorithms across 12 open access datasets, including over 250 subjects. Our results show that even for the best methods, there are datasets which do not show significant improvements, and further that many previously validated methods do not generalize well outside the datasets they were tested on. SIGNIFICANCE: Our analysis confirms that BCI algorithms validated on single datasets are not representative, highlighting the need for more robust validation in the machine learning for BCIs community.
Authors: Jane E Huggins; Christoph Guger; Erik Aarnoutse; Brendan Allison; Charles W Anderson; Steven Bedrick; Walter Besio; Ricardo Chavarriaga; Jennifer L Collinger; An H Do; Christian Herff; Matthias Hohmann; Michelle Kinsella; Kyuhwa Lee; Fabien Lotte; Gernot Müller-Putz; Anton Nijholt; Elmar Pels; Betts Peters; Felix Putze; Rüdiger Rupp; Gerwin Schalk; Stephanie Scott; Michael Tangermann; Paul Tubig; Thorsten Zander Journal: Brain Comput Interfaces (Abingdon) Date: 2019-12-10