Ralf Watzlawick1, Ana Antonic1, Emily S Sena1, Marcel A Kopp1, Julian Rind1, Ulrich Dirnagl1, Malcolm Macleod1, David W Howells1, Jan M Schwab2. 1. From Charité-Universitätsmedizin Berlin (R.W., M.A.K., J.R., U.D., J.M.S.), corporate member of the Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health; Department of Neurology and Experimental Neurology (R.W., M.A.K., J.R., J.M.S.), Charité Campus Mitte, Clinical and Experimental Spinal Cord Injury Research Laboratory (Neuroparaplegiology), Charité-Universitätsmedizin Berlin; Department of Neurosurgery (R.W.), Freiburg University Medical Center, Germany; Department of Neuroscience (A.A.), Central Clinical School, Monash University, Melbourne; Stroke Division (E.S.S., M.M., D.W.H.), Melbourne, Victoria, Australia; Departments of Neurology and Clinical Neurosciences (E.S.S., M.M.), University of Edinburgh, UK; Center for Stroke Research Berlin (U.D.) and Excellence Cluster Neurocure (U.D.), Charité-Universitätsmedizin, Berlin, Germany; German Center for Neurodegenerative Diseases (U.D.), Bonn; Berlin Institute of Health (M.A.K., U.D.), Germany; University of Tasmania (D.W.H.), School of Medicine, Faculty of Health, Medical Sciences Precinct, Hobart, Australia; Department of Neurology (J.M.S.), Spinal Cord Injury Medicine (Paraplegiology), and Belford Center for Spinal Cord Injury (J.M.S.), Departments of Neuroscience and Physical Medicine and Rehabilitation, The Neurological Institute, The Ohio State University, Wexner Medical Center, Columbus. 2. From Charité-Universitätsmedizin Berlin (R.W., M.A.K., J.R., U.D., J.M.S.), corporate member of the Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health; Department of Neurology and Experimental Neurology (R.W., M.A.K., J.R., J.M.S.), Charité Campus Mitte, Clinical and Experimental Spinal Cord Injury Research Laboratory (Neuroparaplegiology), Charité-Universitätsmedizin Berlin; Department of Neurosurgery (R.W.), Freiburg University Medical Center, Germany; Department of Neuroscience (A.A.), Central Clinical School, Monash University, Melbourne; Stroke Division (E.S.S., M.M., D.W.H.), Melbourne, Victoria, Australia; Departments of Neurology and Clinical Neurosciences (E.S.S., M.M.), University of Edinburgh, UK; Center for Stroke Research Berlin (U.D.) and Excellence Cluster Neurocure (U.D.), Charité-Universitätsmedizin, Berlin, Germany; German Center for Neurodegenerative Diseases (U.D.), Bonn; Berlin Institute of Health (M.A.K., U.D.), Germany; University of Tasmania (D.W.H.), School of Medicine, Faculty of Health, Medical Sciences Precinct, Hobart, Australia; Department of Neurology (J.M.S.), Spinal Cord Injury Medicine (Paraplegiology), and Belford Center for Spinal Cord Injury (J.M.S.), Departments of Neuroscience and Physical Medicine and Rehabilitation, The Neurological Institute, The Ohio State University, Wexner Medical Center, Columbus. jan.schwab@osumc.edu.
Abstract
OBJECTIVE: To determine whether and to what degree bias and underestimated variability undermine the predictive value of preclinical research for clinical translation. METHODS: We investigated experimental spinal cord injury (SCI) studies for outcome heterogeneity and the impact of bias. Data from 549 preclinical SCI studies including 9,535 animals were analyzed with meta-regression to assess the effect of various study characteristics and the quality of neurologic recovery. RESULTS: Overall, the included interventions reported a neurobehavioral outcome improvement of 26.3% (95% confidence interval 24.3-28.4). Response to treatment was dependent on experimental modeling paradigms (neurobehavioral score, site of injury, and animal species). Applying multiple outcome measures was consistently associated with smaller effect sizes compared with studies applying only 1 outcome measure. More than half of the studies (51.2%) did not report blinded assessment, constituting a likely source of evaluation bias, with an overstated effect size of 7.2%. Assessment of publication bias, which extrapolates to identify likely missing data, suggested that between 2% and 41% of experiments remain unpublished. Inclusion of these theoretical missing studies suggested an overestimation of efficacy, reducing the effect sizes by between 0.9% and 14.3%. CONCLUSIONS: We provide empirical evidence of prevalent bias in the design and reporting of experimental SCI studies, resulting in overestimation of the effectiveness. Bias compromises the internal validity and jeopardizes the successful translation of SCI therapies from the bench to bedside.
OBJECTIVE: To determine whether and to what degree bias and underestimated variability undermine the predictive value of preclinical research for clinical translation. METHODS: We investigated experimental spinal cord injury (SCI) studies for outcome heterogeneity and the impact of bias. Data from 549 preclinical SCI studies including 9,535 animals were analyzed with meta-regression to assess the effect of various study characteristics and the quality of neurologic recovery. RESULTS: Overall, the included interventions reported a neurobehavioral outcome improvement of 26.3% (95% confidence interval 24.3-28.4). Response to treatment was dependent on experimental modeling paradigms (neurobehavioral score, site of injury, and animal species). Applying multiple outcome measures was consistently associated with smaller effect sizes compared with studies applying only 1 outcome measure. More than half of the studies (51.2%) did not report blinded assessment, constituting a likely source of evaluation bias, with an overstated effect size of 7.2%. Assessment of publication bias, which extrapolates to identify likely missing data, suggested that between 2% and 41% of experiments remain unpublished. Inclusion of these theoretical missing studies suggested an overestimation of efficacy, reducing the effect sizes by between 0.9% and 14.3%. CONCLUSIONS: We provide empirical evidence of prevalent bias in the design and reporting of experimental SCI studies, resulting in overestimation of the effectiveness. Bias compromises the internal validity and jeopardizes the successful translation of SCI therapies from the bench to bedside.
Authors: Nitin B Jain; Gregory D Ayers; Emily N Peterson; Mitchel B Harris; Leslie Morse; Kevin C O'Connor; Eric Garshick Journal: JAMA Date: 2015-06-09 Impact factor: 56.272
Authors: Marca L Sipski; Amie B Jackson; Orlando Gómez-Marín; Irene Estores; Adam Stein Journal: Arch Phys Med Rehabil Date: 2004-11 Impact factor: 3.966
Authors: H M Vesterinen; E S Sena; K J Egan; T C Hirst; L Churolov; G L Currie; A Antonic; D W Howells; M R Macleod Journal: J Neurosci Methods Date: 2013-10-04 Impact factor: 2.390
Authors: Nicholas D James; Katalin Bartus; John Grist; David L H Bennett; Stephen B McMahon; Elizabeth J Bradbury Journal: J Neurosci Date: 2011-12-14 Impact factor: 6.167
Authors: Vance P Lemmon; Adam R Ferguson; Phillip G Popovich; Xiao-Ming Xu; Diane M Snow; Michihiro Igarashi; Christine E Beattie; John L Bixby Journal: J Neurotrauma Date: 2014-07-11 Impact factor: 5.269
Authors: Michael G Fehlings; Alexander Vaccaro; Jefferson R Wilson; Anoushka Singh; David W Cadotte; James S Harrop; Bizhan Aarabi; Christopher Shaffrey; Marcel Dvorak; Charles Fisher; Paul Arnold; Eric M Massicotte; Stephen Lewis; Raja Rampersaud Journal: PLoS One Date: 2012-02-23 Impact factor: 3.240
Authors: Ralf Watzlawick; Emily S Sena; Ulrich Dirnagl; Benedikt Brommer; Marcel A Kopp; Malcolm R Macleod; David W Howells; Jan M Schwab Journal: JAMA Neurol Date: 2014-01 Impact factor: 18.302
Authors: Karim Fouad; John L Bixby; Alison Callahan; Jeffrey S Grethe; Lyn B Jakeman; Vance P Lemmon; David S K Magnuson; Maryann E Martone; Jessica L Nielson; Jan M Schwab; Carol Taylor-Burds; Wolfram Tetzlaff; Abel Torres-Espin; Adam R Ferguson Journal: J Neurotrauma Date: 2019-12-06 Impact factor: 5.269
Authors: Harriet Hunter; Dana de Gracia Hahn; Amedine Duret; Yu Ri Im; Qinrong Cheah; Jiawen Dong; Madison Fairey; Clarissa Hjalmarsson; Alice Li; Hong Kai Lim; Lorcan McKeown; Claudia-Gabriela Mitrofan; Raunak Rao; Mrudula Utukuri; Ian A Rowe; Jake P Mann Journal: Elife Date: 2020-10-16 Impact factor: 8.140
Authors: Caroline Miller; Jane Cross; Joel O'Sullivan; Dominic M Power; Derek Kyte; Christina Jerosch-Herold Journal: BMJ Open Date: 2021-07-30 Impact factor: 2.692