Robert J O'Shea1, Sophia Tsoka2, Gary Jr Cook1,3, Vicky Goh1,4. 1. Department of Cancer Imaging, School of Biomedical Engineering and Imaging Sciences, King's College London, London, UK. 2. Department of Informatics, School of Natural and Mathematical Sciences, King's College London, London, UK. 3. King's College London & Guy's and St Thomas' PET Centre, St Thomas' Hospital, London, UK. 4. Department of Radiology, Guy's and St Thomas' NHS Foundation Trust, London, UK.
Abstract
BACKGROUND: Evaluation of gene interaction models in cancer genomics is challenging, as the true distribution is uncertain. Previous analyses have benchmarked models using synthetic data or databases of experimentally verified interactions - approaches which are susceptible to misrepresentation and incompleteness, respectively. The objectives of this analysis are to (1) provide a real-world data-driven approach for comparing performance of genomic model inference algorithms, (2) compare the performance of LASSO, elastic net, best-subset selection, L 0 L 1 penalisation and L 0 L 2 penalisation in real genomic data and (3) compare algorithmic preselection according to performance in our benchmark datasets to algorithmic selection by internal cross-validation. METHODS: Five large ( n 4000 ) genomic datasets were extracted from Gene Expression Omnibus. 'Gold-standard' regression models were trained on subspaces of these datasets ( n 4000 , p = 500 ). Penalised regression models were trained on small samples from these subspaces ( n ∈ { 25 , 75 , 150 } , p = 500 ) and validated against the gold-standard models. Variable selection performance and out-of-sample prediction were assessed. Penalty 'preselection' according to test performance in the other 4 datasets was compared to selection internal cross-validation error minimisation. RESULTS: L 1 L 2 -penalisation achieved the highest cosine similarity between estimated coefficients and those of gold-standard models. L 0 L 2 -penalised models explained the greatest proportion of variance in test responses, though performance was unreliable in low signal:noise conditions. L 0 L 2 also attained the highest overall median variable selection F1 score. Penalty preselection significantly outperformed selection by internal cross-validation in each of 3 examined metrics. CONCLUSIONS: This analysis explores a novel approach for comparisons of model selection approaches in real genomic data from 5 cancers. Our benchmarking datasets have been made publicly available for use in future research. Our findings support the use of L 0 L 2 penalisation for structural selection and L 1 L 2 penalisation for coefficient recovery in genomic data. Evaluation of learning algorithms according to observed test performance in external genomic datasets yields valuable insights into actual test performance, providing a data-driven complement to internal cross-validation in genomic regression tasks.
BACKGROUND: Evaluation of gene interaction models in cancer genomics is challenging, as the true distribution is uncertain. Previous analyses have benchmarked models using synthetic data or databases of experimentally verified interactions - approaches which are susceptible to misrepresentation and incompleteness, respectively. The objectives of this analysis are to (1) provide a real-world data-driven approach for comparing performance of genomic model inference algorithms, (2) compare the performance of LASSO, elastic net, best-subset selection, L 0 L 1 penalisation and L 0 L 2 penalisation in real genomic data and (3) compare algorithmic preselection according to performance in our benchmark datasets to algorithmic selection by internal cross-validation. METHODS: Five large ( n 4000 ) genomic datasets were extracted from Gene Expression Omnibus. 'Gold-standard' regression models were trained on subspaces of these datasets ( n 4000 , p = 500 ). Penalised regression models were trained on small samples from these subspaces ( n ∈ { 25 , 75 , 150 } , p = 500 ) and validated against the gold-standard models. Variable selection performance and out-of-sample prediction were assessed. Penalty 'preselection' according to test performance in the other 4 datasets was compared to selection internal cross-validation error minimisation. RESULTS: L 1 L 2 -penalisation achieved the highest cosine similarity between estimated coefficients and those of gold-standard models. L 0 L 2 -penalised models explained the greatest proportion of variance in test responses, though performance was unreliable in low signal:noise conditions. L 0 L 2 also attained the highest overall median variable selection F1 score. Penalty preselection significantly outperformed selection by internal cross-validation in each of 3 examined metrics. CONCLUSIONS: This analysis explores a novel approach for comparisons of model selection approaches in real genomic data from 5 cancers. Our benchmarking datasets have been made publicly available for use in future research. Our findings support the use of L 0 L 2 penalisation for structural selection and L 1 L 2 penalisation for coefficient recovery in genomic data. Evaluation of learning algorithms according to observed test performance in external genomic datasets yields valuable insights into actual test performance, providing a data-driven complement to internal cross-validation in genomic regression tasks.
Authors: Benjamin Izar; Itay Tirosh; Elizabeth H Stover; Isaac Wakiro; Michael S Cuoco; Idan Alter; Christopher Rodman; Rachel Leeson; Mei-Ju Su; Parin Shah; Marcin Iwanicki; Sarah R Walker; Abhay Kanodia; Johannes C Melms; Shaolin Mei; Jia-Ren Lin; Caroline B M Porter; Michal Slyper; Julia Waldman; Livnat Jerby-Arnon; Orr Ashenberg; Titus J Brinker; Caitlin Mills; Meri Rogava; Sébastien Vigneau; Peter K Sorger; Levi A Garraway; Panagiotis A Konstantinopoulos; Joyce F Liu; Ursula Matulonis; Bruce E Johnson; Orit Rozenblatt-Rosen; Asaf Rotem; Aviv Regev Journal: Nat Med Date: 2020-06-22 Impact factor: 53.440