Lucas Schneider1, Sara Laiouar-Pedari1, Sara Kuntz1, Eva Krieghoff-Henning1, Achim Hekler1, Jakob N Kather2, Timo Gaiser3, Stefan Fröhling4, Titus J Brinker5. 1. Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany. 2. Department of Medicine III, RWTH Aachen University Hospital, Aachen, Germany; Medical Oncology, National Center for Tumour Diseases, University Hospital Heidelberg, Heidelberg, Germany. 3. Institute of Pathology, University Medical Centre Mannheim, University of Heidelberg, Mannheim, Germany. 4. Translational Medical Oncology, National Center for Tumour Diseases, German Cancer Research Center, Heidelberg, Germany. 5. Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany. Electronic address: titus.brinker@dkfz.de.
Abstract
BACKGROUND: Over the past decade, the development of molecular high-throughput methods (omics) increased rapidly and provided new insights for cancer research. In parallel, deep learning approaches revealed the enormous potential for medical image analysis, especially in digital pathology. Combining image and omics data with deep learning tools may enable the discovery of new cancer biomarkers and a more precise prediction of patient prognosis. This systematic review addresses different multimodal fusion methods of convolutional neural network-based image analyses with omics data, focussing on the impact of data combination on the classification performance. METHODS: PubMed was screened for peer-reviewed articles published in English between January 2015 and June 2021 by two independent researchers. Search terms related to deep learning, digital pathology, omics, and multimodal fusion were combined. RESULTS: We identified a total of 11 studies meeting the inclusion criteria, namely studies that used convolutional neural networks for haematoxylin and eosin image analysis of patients with cancer in combination with integrated omics data. Publications were categorised according to their endpoints: 7 studies focused on survival analysis and 4 studies on prediction of cancer subtypes, malignancy or microsatellite instability with spatial analysis. CONCLUSIONS: Image-based classifiers already show high performances in prognostic and predictive cancer diagnostics. The integration of omics data led to improved performance in all studies described here. However, these are very early studies that still require external validation to demonstrate their generalisability and robustness. Further and more comprehensive studies with larger sample sizes are needed to evaluate performance and determine clinical benefits.
BACKGROUND: Over the past decade, the development of molecular high-throughput methods (omics) increased rapidly and provided new insights for cancer research. In parallel, deep learning approaches revealed the enormous potential for medical image analysis, especially in digital pathology. Combining image and omics data with deep learning tools may enable the discovery of new cancer biomarkers and a more precise prediction of patient prognosis. This systematic review addresses different multimodal fusion methods of convolutional neural network-based image analyses with omics data, focussing on the impact of data combination on the classification performance. METHODS: PubMed was screened for peer-reviewed articles published in English between January 2015 and June 2021 by two independent researchers. Search terms related to deep learning, digital pathology, omics, and multimodal fusion were combined. RESULTS: We identified a total of 11 studies meeting the inclusion criteria, namely studies that used convolutional neural networks for haematoxylin and eosin image analysis of patients with cancer in combination with integrated omics data. Publications were categorised according to their endpoints: 7 studies focused on survival analysis and 4 studies on prediction of cancer subtypes, malignancy or microsatellite instability with spatial analysis. CONCLUSIONS: Image-based classifiers already show high performances in prognostic and predictive cancer diagnostics. The integration of omics data led to improved performance in all studies described here. However, these are very early studies that still require external validation to demonstrate their generalisability and robustness. Further and more comprehensive studies with larger sample sizes are needed to evaluate performance and determine clinical benefits.
Authors: Narmin Ghaffari Laleh; Daniel Truhn; Gregory Patrick Veldhuizen; Tianyu Han; Marko van Treeck; Roman D Buelow; Rupert Langer; Bastian Dislich; Peter Boor; Volkmar Schulz; Jakob Nikolas Kather Journal: Nat Commun Date: 2022-09-29 Impact factor: 17.694