Xingyu Li1, Min Cen1, Jinfeng Xu2, Hong Zhang1, Xu Steven Xu3. 1. Department of Statistics and Finance, School of Management, University of Science and Technology of China, Hefei, Anhui 230026, China. 2. Department of Statistics and Actuarial Science, The University of Hong Kong, Hong Kong. 3. Clinical Pharmacology and Quantitative Science, Genmab Inc., Princeton, New Jersey, USA.
Abstract
Background: Due to lack of annotated pathological images, transfer learning has been the predominant approach in the field of digital pathology. Pre-trained neural networks based on ImageNet database are often used to extract "off-the-shelf" features, achieving great success in predicting tissue types, molecular features, and clinical outcomes, etc. We hypothesize that fine-tuning the pre-trained models using histopathological images could further improve feature extraction, and downstream prediction performance. Methods: We used 100 000 annotated H&E image patches for colorectal cancer (CRC) to fine-tune a pre-trained Xception model via a 2-step approach. The features extracted from fine-tuned Xception (FTX-2048) model and Image-pretrained (IMGNET-2048) model were compared through: (1) tissue classification for H&E images from CRC, same image type that was used for fine-tuning; (2) prediction of immune-related gene expression, and (3) gene mutations for lung adenocarcinoma (LUAD). Five-fold cross validation was used for model performance evaluation. Each experiment was repeated 50 times. Findings: The extracted features from the fine-tuned FTX-2048 exhibited significantly higher accuracy (98.4%) for predicting tissue types of CRC compared to the "off-the-shelf" features directly from Xception based on ImageNet database (96.4%) (P value = 2.2 × 10-6). Particularly, FTX-2048 markedly improved the accuracy for stroma from 87% to 94%. Similarly, features from FTX-2048 boosted the prediction of transcriptomic expression of immune-related genes in LUAD. For the genes that had significant relationships with image features (P < 0.05, n = 171), the features from the fine-tuned model improved the prediction for the majority of the genes (139; 81%). In addition, features from FTX-2048 improved prediction of mutation for 5 out of 9 most frequently mutated genes (STK11, TP53, LRP1B, NF1, and FAT1) in LUAD. Conclusions: We proved the concept that fine-tuning the pretrained ImageNet neural networks with histopathology images can produce higher quality features and better prediction performance for not only the same-cancer tissue classification where similar images from the same cancer are used for fine-tuning, but also cross-cancer prediction for gene expression and mutation at patient level.
Background: Due to lack of annotated pathological images, transfer learning has been the predominant approach in the field of digital pathology. Pre-trained neural networks based on ImageNet database are often used to extract "off-the-shelf" features, achieving great success in predicting tissue types, molecular features, and clinical outcomes, etc. We hypothesize that fine-tuning the pre-trained models using histopathological images could further improve feature extraction, and downstream prediction performance. Methods: We used 100 000 annotated H&E image patches for colorectal cancer (CRC) to fine-tune a pre-trained Xception model via a 2-step approach. The features extracted from fine-tuned Xception (FTX-2048) model and Image-pretrained (IMGNET-2048) model were compared through: (1) tissue classification for H&E images from CRC, same image type that was used for fine-tuning; (2) prediction of immune-related gene expression, and (3) gene mutations for lung adenocarcinoma (LUAD). Five-fold cross validation was used for model performance evaluation. Each experiment was repeated 50 times. Findings: The extracted features from the fine-tuned FTX-2048 exhibited significantly higher accuracy (98.4%) for predicting tissue types of CRC compared to the "off-the-shelf" features directly from Xception based on ImageNet database (96.4%) (P value = 2.2 × 10-6). Particularly, FTX-2048 markedly improved the accuracy for stroma from 87% to 94%. Similarly, features from FTX-2048 boosted the prediction of transcriptomic expression of immune-related genes in LUAD. For the genes that had significant relationships with image features (P < 0.05, n = 171), the features from the fine-tuned model improved the prediction for the majority of the genes (139; 81%). In addition, features from FTX-2048 improved prediction of mutation for 5 out of 9 most frequently mutated genes (STK11, TP53, LRP1B, NF1, and FAT1) in LUAD. Conclusions: We proved the concept that fine-tuning the pretrained ImageNet neural networks with histopathology images can produce higher quality features and better prediction performance for not only the same-cancer tissue classification where similar images from the same cancer are used for fine-tuning, but also cross-cancer prediction for gene expression and mutation at patient level.
Authors: Ole-Johan Skrede; Sepp De Raedt; Andreas Kleppe; Tarjei S Hveem; Knut Liestøl; John Maddison; Hanne A Askautrud; Manohar Pradhan; John Arne Nesheim; Fritz Albregtsen; Inger Nina Farstad; Enric Domingo; David N Church; Arild Nesbakken; Neil A Shepherd; Ian Tomlinson; Rachel Kerr; Marco Novelli; David J Kerr; Håvard E Danielsen Journal: Lancet Date: 2020-02-01 Impact factor: 79.321
Authors: P Chang; J Grinband; B D Weinberg; M Bardis; M Khy; G Cadena; M-Y Su; S Cha; C G Filippi; D Bota; P Baldi; L M Poisson; R Jain; D Chow Journal: AJNR Am J Neuroradiol Date: 2018-05-10 Impact factor: 3.825
Authors: Jakob Nikolas Kather; Johannes Krisam; Pornpimol Charoentong; Tom Luedde; Esther Herpel; Cleo-Aron Weis; Timo Gaiser; Alexander Marx; Nektarios A Valous; Dyke Ferber; Lina Jansen; Constantino Carlos Reyes-Aldasoro; Inka Zörnig; Dirk Jäger; Hermann Brenner; Jenny Chang-Claude; Michael Hoffmeister; Niels Halama Journal: PLoS Med Date: 2019-01-24 Impact factor: 11.069
Authors: Jens Rittscher; Viktor H Koelzer; Korsuk Sirinukunwattana; Enric Domingo; Susan D Richman; Keara L Redmond; Andrew Blake; Clare Verrill; Simon J Leedham; Aikaterini Chatzipli; Claire Hardy; Celina M Whalley; Chieh-Hsi Wu; Andrew D Beggs; Ultan McDermott; Philip D Dunne; Angela Meade; Steven M Walker; Graeme I Murray; Leslie Samuel; Matthew Seymour; Ian Tomlinson; Phil Quirke; Timothy Maughan Journal: Gut Date: 2020-07-20 Impact factor: 23.059