Ali Amad1,2,3, Renaud Jardri4,5, Chloé Rousseau6, Yann Larochelle6, John P A Ioannidis7,8, Florian Naudet6. 1. Psychiatry Department and CURE Platform, Fontan Hospital, CHU Lille, Lille, France, ali.amad@outlook.com. 2. Psychiatry and Beliefs Team, SCALab, CNRS UMR-9193, University of Lille, Lille, France, ali.amad@outlook.com. 3. Department of Neuroimaging, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom, ali.amad@outlook.com. 4. Psychiatry Department and CURE Platform, Fontan Hospital, CHU Lille, Lille, France. 5. Psychiatry and Beliefs Team, SCALab, CNRS UMR-9193, University of Lille, Lille, France. 6. Inserm, CIC 1414 - Centre d'Investigation Clinique de Rennes, CHU Rennes, University of Rennes, Rennes, France. 7. Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, USA. 8. Departments of Medicine, Health Research and Policy, Biomedical Data Science, and Statistics, Stanford University, Stanford, California, USA.
Abstract
INTRODUCTION: Repetitive transcranial magnetic stimulation (rTMS) has been widely tested and promoted for use in multiple neuropsychiatric conditions, but as for many other medical devices, some gaps may exist in the literature and the evidence base for the clinical efficacy of rTMS remains under debate. OBJECTIVE: We aimed to test for an excess number of statistically significant results in the literature on the therapeutic efficacy of rTMS across a wide range of meta-analyses and to characterize the power of studies included in these meta-analyses. METHODS: Based on power calculations, we computed the expected number of "positive" datasets for a medium effect size (standardized mean difference, SMD = 0.30) and compared it with the number of observed "positive" datasets. Sensitivity analyses considered small (SMD = 0.20), modest (SMD = 0.50), and large (SMD = 0.80) effect sizes. RESULTS: A total of 14 meta-analyses with 228 datasets (110 for neurological disorders and 118 for psychiatric disorders) were assessed. For SMD = 0.3, the number of observed "positive" studies (n = 94) was larger than expected (n = 35). We found evidence for an excess of significant findings overall (p < 0.0001) and in 8/14 meta-analyses. Evidence for an excess of significant findings was also observed for SMD = 0.5 for neurological disorders. Of the 228 datasets, 0 (0%), 0 (0%), 3 (1%), and 53 (23%) had a power >0.80, respectively, for SMDs of 0.30, 0.20, 0.50, and 0.80. CONCLUSION: Most studies in the rTMS literature are underpowered. This results in fragmentation and waste of research efforts. The somewhat high frequency of "positive" results seems spurious and may reflect bias. Caution is warranted in accepting rTMS as an established treatment for neuropsychiatric conditions.
INTRODUCTION: Repetitive transcranial magnetic stimulation (rTMS) has been widely tested and promoted for use in multiple neuropsychiatric conditions, but as for many other medical devices, some gaps may exist in the literature and the evidence base for the clinical efficacy of rTMS remains under debate. OBJECTIVE: We aimed to test for an excess number of statistically significant results in the literature on the therapeutic efficacy of rTMS across a wide range of meta-analyses and to characterize the power of studies included in these meta-analyses. METHODS: Based on power calculations, we computed the expected number of "positive" datasets for a medium effect size (standardized mean difference, SMD = 0.30) and compared it with the number of observed "positive" datasets. Sensitivity analyses considered small (SMD = 0.20), modest (SMD = 0.50), and large (SMD = 0.80) effect sizes. RESULTS: A total of 14 meta-analyses with 228 datasets (110 for neurological disorders and 118 for psychiatric disorders) were assessed. For SMD = 0.3, the number of observed "positive" studies (n = 94) was larger than expected (n = 35). We found evidence for an excess of significant findings overall (p < 0.0001) and in 8/14 meta-analyses. Evidence for an excess of significant findings was also observed for SMD = 0.5 for neurological disorders. Of the 228 datasets, 0 (0%), 0 (0%), 3 (1%), and 53 (23%) had a power >0.80, respectively, for SMDs of 0.30, 0.20, 0.50, and 0.80. CONCLUSION: Most studies in the rTMS literature are underpowered. This results in fragmentation and waste of research efforts. The somewhat high frequency of "positive" results seems spurious and may reflect bias. Caution is warranted in accepting rTMS as an established treatment for neuropsychiatric conditions.
Authors: Andrada D Neacsiu; Lysianne Beynel; John P Powers; Steven T Szabo; Lawrence G Appelbaum; Sarah H Lisanby; Kevin S LaBar Journal: Psychother Psychosom Date: 2021-09-22 Impact factor: 17.659