Alex H Barnett1, Jeremy F Magland2, Leslie F Greengard3. 1. Simons Center for Data Analysis, and Department of Mathematics, Dartmouth College, United States. Electronic address: ahb@math.dartmouth.edu. 2. Simons Center for Data Analysis, and Department of Radiology, University of Pennsylvania, United States. 3. Simons Center for Data Analysis, and Courant Institute, New York University, United States.
Abstract
BACKGROUND: The throughput of electrophysiological recording is growing rapidly, allowing thousands of simultaneous channels, and there is a growing variety of spike sorting algorithms designed to extract neural firing events from such data. This creates an urgent need for standardized, automatic evaluation of the quality of neural units output by such algorithms. NEW METHOD: We introduce a suite of validation metrics that assess the credibility of a given automatic spike sorting algorithm applied to a given dataset. By rerunning the spike sorter two or more times, the metrics measure stability under various perturbations consistent with variations in the data itself, making no assumptions about the internal workings of the algorithm, and minimal assumptions about the noise. RESULTS: We illustrate the new metrics on standard sorting algorithms applied to both in vivo and ex vivo recordings, including a time series with overlapping spikes. We compare the metrics to existing quality measures, and to ground-truth accuracy in simulated time series. We provide a software implementation. COMPARISON WITH EXISTING METHODS: Metrics have until now relied on ground-truth, simulated data, internal algorithm variables (e.g. cluster separation), or refractory violations. By contrast, by standardizing the interface, our metrics assess the reliability of any automatic algorithm without reference to internal variables (e.g. feature space) or physiological criteria. CONCLUSIONS: Stability is a prerequisite for reproducibility of results. Such metrics could reduce the significant human labor currently spent on validation, and should form an essential part of large-scale automated spike sorting and systematic benchmarking of algorithms.
BACKGROUND: The throughput of electrophysiological recording is growing rapidly, allowing thousands of simultaneous channels, and there is a growing variety of spike sorting algorithms designed to extract neural firing events from such data. This creates an urgent need for standardized, automatic evaluation of the quality of neural units output by such algorithms. NEW METHOD: We introduce a suite of validation metrics that assess the credibility of a given automatic spike sorting algorithm applied to a given dataset. By rerunning the spike sorter two or more times, the metrics measure stability under various perturbations consistent with variations in the data itself, making no assumptions about the internal workings of the algorithm, and minimal assumptions about the noise. RESULTS: We illustrate the new metrics on standard sorting algorithms applied to both in vivo and ex vivo recordings, including a time series with overlapping spikes. We compare the metrics to existing quality measures, and to ground-truth accuracy in simulated time series. We provide a software implementation. COMPARISON WITH EXISTING METHODS: Metrics have until now relied on ground-truth, simulated data, internal algorithm variables (e.g. cluster separation), or refractory violations. By contrast, by standardizing the interface, our metrics assess the reliability of any automatic algorithm without reference to internal variables (e.g. feature space) or physiological criteria. CONCLUSIONS: Stability is a prerequisite for reproducibility of results. Such metrics could reduce the significant humanlabor currently spent on validation, and should form an essential part of large-scale automated spike sorting and systematic benchmarking of algorithms.
Authors: Eric M Trautmann; Sergey D Stavisky; Subhaneil Lahiri; Katherine C Ames; Matthew T Kaufman; Daniel J O'Shea; Saurabh Vyas; Xulu Sun; Stephen I Ryu; Surya Ganguli; Krishna V Shenoy Journal: Neuron Date: 2019-06-03 Impact factor: 17.173
Authors: Réka Barbara Bod; János Rokai; Domokos Meszéna; Richárd Fiáth; István Ulbert; Gergely Márton Journal: Front Neuroinform Date: 2022-06-13 Impact factor: 3.739
Authors: David Kleinfeld; Lan Luan; Partha P Mitra; Jacob T Robinson; Rahul Sarpeshkar; Kenneth Shepard; Chong Xie; Timothy D Harris Journal: Neuron Date: 2019-09-05 Impact factor: 17.173
Authors: Julien Fournier; Christian M Mueller; Mark Shein-Idelson; Mike Hemberger; Gilles Laurent Journal: PLoS One Date: 2016-08-18 Impact factor: 3.240
Authors: Gonzalo E Mena; Lauren E Grosberg; Sasidhar Madugula; Paweł Hottowy; Alan Litke; John Cunningham; E J Chichilnisky; Liam Paninski Journal: PLoS Comput Biol Date: 2017-11-13 Impact factor: 4.475
Authors: Jason E Chung; Jeremy F Magland; Alex H Barnett; Vanessa M Tolosa; Angela C Tooker; Kye Y Lee; Kedar G Shah; Sarah H Felix; Loren M Frank; Leslie F Greengard Journal: Neuron Date: 2017-09-13 Impact factor: 17.173