Alfredo A Pulini1, Wesley T Kerr2, Sandra K Loo3, Agatha Lenartowicz4. 1. Paris Descartes University, Paris, France. 2. Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, Los Angeles; Department of Biomathematics, University of California, Los Angeles, Los Angeles; Department of Internal Medicine, Eisenhower Medical Center, Rancho Mirage, California. 3. Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, Los Angeles. 4. Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, Los Angeles. Electronic address: alenarto@g.ucla.edu.
Abstract
BACKGROUND: Motivated by an inconsistency between reports of high diagnosis-classification accuracies and known heterogeneity in attention-deficit/hyperactivity disorder (ADHD), this study assessed classification accuracy in studies of ADHD as a function of methodological factors that can bias results. We hypothesized that high classification results in ADHD diagnosis are inflated by methodological factors. METHODS: We reviewed 69 studies (of 95 studies identified) that used neuroimaging features to predict ADHD diagnosis. Based on reported methods, we assessed the prevalence of circular analysis, which inflates classification accuracy, and evaluated the relationship between sample size and accuracy to test if small-sample models tend to report higher classification accuracy, also an indicator of bias. RESULTS: Circular analysis was detected in 15.9% of ADHD classification studies, lack of independent test set was noted in 13%, and insufficient methodological detail to establish its presence was noted in another 11.6%. Accuracy of classification ranged from 60% to 80% in the 59.4% of reviewed studies that met criteria for independence of feature selection, model construction, and test datasets. Moreover, there was a negative relationship between accuracy and sample size, implying additional bias contributing to reported accuracies at lower sample sizes. CONCLUSIONS: High classification accuracies in neuroimaging studies of ADHD appear to be inflated by circular analysis and small sample size. Accuracies on independent datasets were consistent with known heterogeneity of the disorder. Steps to resolve these issues, and a shift toward accounting for sample heterogeneity and prediction of future outcomes, will be crucial in future classification studies in ADHD.
BACKGROUND: Motivated by an inconsistency between reports of high diagnosis-classification accuracies and known heterogeneity in attention-deficit/hyperactivity disorder (ADHD), this study assessed classification accuracy in studies of ADHD as a function of methodological factors that can bias results. We hypothesized that high classification results in ADHD diagnosis are inflated by methodological factors. METHODS: We reviewed 69 studies (of 95 studies identified) that used neuroimaging features to predict ADHD diagnosis. Based on reported methods, we assessed the prevalence of circular analysis, which inflates classification accuracy, and evaluated the relationship between sample size and accuracy to test if small-sample models tend to report higher classification accuracy, also an indicator of bias. RESULTS: Circular analysis was detected in 15.9% of ADHD classification studies, lack of independent test set was noted in 13%, and insufficient methodological detail to establish its presence was noted in another 11.6%. Accuracy of classification ranged from 60% to 80% in the 59.4% of reviewed studies that met criteria for independence of feature selection, model construction, and test datasets. Moreover, there was a negative relationship between accuracy and sample size, implying additional bias contributing to reported accuracies at lower sample sizes. CONCLUSIONS: High classification accuracies in neuroimaging studies of ADHD appear to be inflated by circular analysis and small sample size. Accuracies on independent datasets were consistent with known heterogeneity of the disorder. Steps to resolve these issues, and a shift toward accounting for sample heterogeneity and prediction of future outcomes, will be crucial in future classification studies in ADHD.
Authors: Laura O'Halloran; Zhipeng Cao; Kathy Ruddy; Lee Jollans; Matthew D Albaugh; Andrea Aleni; Alexandra S Potter; Nigel Vahey; Tobias Banaschewski; Sarah Hohmann; Arun L W Bokde; Uli Bromberg; Christian Büchel; Erin Burke Quinlan; Sylvane Desrivières; Herta Flor; Vincent Frouin; Penny Gowland; Andreas Heinz; Bernd Ittermann; Frauke Nees; Dimitri Papadopoulos Orfanos; Tomáš Paus; Michael N Smolka; Henrik Walter; Gunter Schumann; Hugh Garavan; Clare Kelly; Robert Whelan Journal: Neuroimage Date: 2017-12-21 Impact factor: 6.556
Authors: Ani Eloyan; John Muschelli; Mary Beth Nebel; Han Liu; Fang Han; Tuo Zhao; Anita D Barber; Suresh Joel; James J Pekar; Stewart H Mostofsky; Brian Caffo Journal: Front Syst Neurosci Date: 2012-08-30
Authors: Wesley T Kerr; Emily A Janio; Andrea M Chau; Chelsea T Braesch; Justine M Le; Jessica M Hori; Akash B Patel; Norma L Gallardo; Corinne H Allas; Amir H Karimi; Ishita Dubey; Siddhika S Sreenivasan; Janar Bauirjan; Eric S Hwang; Emily C Davis; Shannon R D'Ambrosio; Mona Al Banna; Rajarshi Mazumder; Ting Wu; Zachary A DeCant; Michael G Gibbs; Edward Chang; Xingruo Zhang; Andrew Y Cho; Nicholas J Beimer; Jerome Engel; Mark S Cohen; John M Stern Journal: Epilepsy Behav Date: 2020-11-13 Impact factor: 2.937