Amanda L Rodrigue1, Dana Mastrovito2, Oscar Esteban3, Joke Durnez3, Marinka M G Koenis4, Ronald Janssen5, Aaron Alexander-Bloch6, Emma M Knowles7, Samuel R Mathias7, Josephine Mollon7, Godfrey D Pearlson4, Sophia Frangou8, John Blangero9, Russell A Poldrack3, David C Glahn10. 1. Department of Psychiatry, Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts. Electronic address: amanda.rodrigue@childrens.harvard.edu. 2. Department of Psychology, Stanford University, Stanford, California. Electronic address: dmastrov@stanford.edu. 3. Department of Psychology, Stanford University, Stanford, California. 4. Department of Psychiatry, Yale University School of Medicine, New Haven, Connecticut; Olin Neuropsychiatry Research Center, Institute of Living, Hartford, Connecticut. 5. Olin Neuropsychiatry Research Center, Institute of Living, Hartford, Connecticut. 6. Department of Psychiatry, Yale University School of Medicine, New Haven, Connecticut. 7. Department of Psychiatry, Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts. 8. Department of Psychiatry, Icahn School of Medicine, Mount Sinai, New York, New York; Centre for Brain Health, University of British Columbia, Vancouver, British Columbia, Canada. 9. Department of Human Genetics and South Texas Diabetes and Obesity Institute, School of Medicine, University of Texas of the Rio Grande Valley, Brownsville, Texas. 10. Department of Psychiatry, Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts; Olin Neuropsychiatry Research Center, Institute of Living, Hartford, Connecticut.
Abstract
BACKGROUND: Progress in precision psychiatry is predicated on identifying reliable individual-level diagnostic biomarkers. For psychosis, measures of structural and functional connectivity could be promising biomarkers given consistent reports of dysconnectivity across psychotic disorders using magnetic resonance imaging. METHODS: We leveraged data from four independent cohorts of patients with psychosis and control subjects with observations from approximately 800 individuals. We used group-level analyses and two supervised machine learning algorithms (support vector machines and ridge regression) to test within-, between-, and across-sample classification performance of white matter and resting-state connectivity metrics. RESULTS: Although we replicated group-level differences in brain connectivity, individual-level classification was suboptimal. Classification performance within samples was variable across folds (highest area under the curve [AUC] range = 0.30) and across datasets (average support vector machine AUC range = 0.50; average ridge regression AUC range = 0.18). Classification performance between samples was similarly variable or resulted in AUC values of approximately 0.65, indicating a lack of model generalizability. Furthermore, collapsing across samples (resting-state functional magnetic resonance imaging, N = 888; diffusion tensor imaging, N = 860) did not improve model performance (maximal AUC = 0.67). Ridge regression models generally outperformed support vector machine models, although classification performance was still suboptimal in terms of clinical relevance. Adjusting for demographic covariates did not greatly affect results. CONCLUSIONS: Connectivity measures were not suitable as diagnostic biomarkers for psychosis as assessed in this study. Our results do not negate that other approaches may be more successful, although it is clear that a systematic approach to individual-level classification with large independent validation samples is necessary to properly vet neuroimaging features as diagnostic biomarkers.
BACKGROUND: Progress in precision psychiatry is predicated on identifying reliable individual-level diagnostic biomarkers. For psychosis, measures of structural and functional connectivity could be promising biomarkers given consistent reports of dysconnectivity across psychotic disorders using magnetic resonance imaging. METHODS: We leveraged data from four independent cohorts of patients with psychosis and control subjects with observations from approximately 800 individuals. We used group-level analyses and two supervised machine learning algorithms (support vector machines and ridge regression) to test within-, between-, and across-sample classification performance of white matter and resting-state connectivity metrics. RESULTS: Although we replicated group-level differences in brain connectivity, individual-level classification was suboptimal. Classification performance within samples was variable across folds (highest area under the curve [AUC] range = 0.30) and across datasets (average support vector machine AUC range = 0.50; average ridge regression AUC range = 0.18). Classification performance between samples was similarly variable or resulted in AUC values of approximately 0.65, indicating a lack of model generalizability. Furthermore, collapsing across samples (resting-state functional magnetic resonance imaging, N = 888; diffusion tensor imaging, N = 860) did not improve model performance (maximal AUC = 0.67). Ridge regression models generally outperformed support vector machine models, although classification performance was still suboptimal in terms of clinical relevance. Adjusting for demographic covariates did not greatly affect results. CONCLUSIONS: Connectivity measures were not suitable as diagnostic biomarkers for psychosis as assessed in this study. Our results do not negate that other approaches may be more successful, although it is clear that a systematic approach to individual-level classification with large independent validation samples is necessary to properly vet neuroimaging features as diagnostic biomarkers.
Authors: Stephen M Smith; Mark Jenkinson; Mark W Woolrich; Christian F Beckmann; Timothy E J Behrens; Heidi Johansen-Berg; Peter R Bannister; Marilena De Luca; Ivana Drobnjak; David E Flitney; Rami K Niazy; James Saunders; John Vickers; Yongyue Zhang; Nicola De Stefano; J Michael Brady; Paul M Matthews Journal: Neuroimage Date: 2004 Impact factor: 6.556
Authors: Babak A Ardekani; Ali Tabesh; Serge Sevy; Delbert G Robinson; Robert M Bilder; Philip R Szeszko Journal: Hum Brain Mapp Date: 2011-01 Impact factor: 5.038
Authors: Kristina C Skåtun; Tobias Kaufmann; Nhat Trung Doan; Dag Alnæs; Aldo Córdova-Palomera; Erik G Jönsson; Helena Fatouros-Bergman; Lena Flyckt; Ingrid Melle; Ole A Andreassen; Ingrid Agartz; Lars T Westlye Journal: Schizophr Bull Date: 2017-07-01 Impact factor: 9.306
Authors: Tobias Kaufmann; Kristina C Skåtun; Dag Alnæs; Nhat Trung Doan; Eugene P Duff; Siren Tønnesen; Evangelos Roussos; Torill Ueland; Sofie R Aminoff; Trine V Lagerberg; Ingrid Agartz; Ingrid S Melle; Stephen M Smith; Ole A Andreassen; Lars T Westlye Journal: Schizophr Bull Date: 2015-05-04 Impact factor: 9.306
Authors: Oscar Esteban; Daniel Birman; Marie Schaer; Oluwasanmi O Koyejo; Russell A Poldrack; Krzysztof J Gorgolewski Journal: PLoS One Date: 2017-09-25 Impact factor: 3.240
Authors: Anderson M Winkler; Gerard R Ridgway; Matthew A Webster; Stephen M Smith; Thomas E Nichols Journal: Neuroimage Date: 2014-02-11 Impact factor: 6.556