OBJECTIVE: Today, the brain-computer interface (BCI) community lacks a standard method to evaluate an online BCI's performance. Even the most commonly used metric, the information transfer rate (ITR), is often reported differently, even incorrectly, in many papers, which is not conducive to BCI research. This paper aims to point out many of the existing problems and give some suggestions and methods to overcome these problems. APPROACH: First, the preconditions inherent in ITR calculation based on Wolpaw's definition are summarized and several incorrect ITR calculations, which go against the preconditions, are indicated. Then, the issues affecting ITR estimation during the test of online BCI systems are discussed in detail. Finally, a task-oriented online BCI test platform was proposed, which may help BCI evaluations in real-world applications. MAIN RESULTS: The guidelines for ITR calculation in online BCIs testing are proposed. The platform executed in the Beijing BCI Competition 2010 shows that it can be used as a common way to compare the online performances (including the ITR) of existing BCI paradigms. SIGNIFICANCE: The proposed guidelines and task-oriented test platform may reduce the uncertainty and artifacts of online BCI performance evaluation; they provide a relatively objective way to compare different BCI's performances in real-world BCI applications, which is a forward step toward developing standards for BCI performance evaluation.
OBJECTIVE: Today, the brain-computer interface (BCI) community lacks a standard method to evaluate an online BCI's performance. Even the most commonly used metric, the information transfer rate (ITR), is often reported differently, even incorrectly, in many papers, which is not conducive to BCI research. This paper aims to point out many of the existing problems and give some suggestions and methods to overcome these problems. APPROACH: First, the preconditions inherent in ITR calculation based on Wolpaw's definition are summarized and several incorrect ITR calculations, which go against the preconditions, are indicated. Then, the issues affecting ITR estimation during the test of online BCI systems are discussed in detail. Finally, a task-oriented online BCI test platform was proposed, which may help BCI evaluations in real-world applications. MAIN RESULTS: The guidelines for ITR calculation in online BCIs testing are proposed. The platform executed in the Beijing BCI Competition 2010 shows that it can be used as a common way to compare the online performances (including the ITR) of existing BCI paradigms. SIGNIFICANCE: The proposed guidelines and task-oriented test platform may reduce the uncertainty and artifacts of online BCI performance evaluation; they provide a relatively objective way to compare different BCI's performances in real-world BCI applications, which is a forward step toward developing standards for BCI performance evaluation.
Authors: David E Thompson; Lucia R Quitadamo; Luca Mainardi; Khalil Ur Rehman Laghari; Shangkai Gao; Pieter-Jan Kindermans; John D Simeral; Reza Fazel-Rezai; Matteo Matteucci; Tiago H Falk; Luigi Bianchi; Cynthia A Chestek; Jane E Huggins Journal: J Neural Eng Date: 2014-05-19 Impact factor: 5.379
Authors: David M Brandman; Tommy Hosman; Jad Saab; Michael C Burkhart; Benjamin E Shanahan; John G Ciancibello; Anish A Sarma; Daniel J Milstein; Carlos E Vargas-Irwin; Brian Franco; Jessica Kelemen; Christine Blabe; Brian A Murphy; Daniel R Young; Francis R Willett; Chethan Pandarinath; Sergey D Stavisky; Robert F Kirsch; Benjamin L Walter; A Bolu Ajiboye; Sydney S Cash; Emad N Eskandar; Jonathan P Miller; Jennifer A Sweet; Krishna V Shenoy; Jaimie M Henderson; Beata Jarosiewicz; Matthew T Harrison; John D Simeral; Leigh R Hochberg Journal: J Neural Eng Date: 2018-04 Impact factor: 5.379
Authors: Bin He; Bryan Baxter; Bradley J Edelman; Christopher C Cline; Wendy Ye Journal: Proc IEEE Inst Electr Electron Eng Date: 2015-05-20 Impact factor: 10.961
Authors: Jane E Huggins; Christoph Guger; Brendan Allison; Charles W Anderson; Aaron Batista; Anne-Marie A-M Brouwer; Clemens Brunner; Ricardo Chavarriaga; Melanie Fried-Oken; Aysegul Gunduz; Disha Gupta; Andrea Kübler; Robert Leeb; Fabien Lotte; Lee E Miller; Gernot Müller-Putz; Tomasz Rutkowski; Michael Tangermann; David Edward Thompson Journal: Brain Comput Interfaces (Abingdon) Date: 2014-01