Yunyan Zhang1, Daphne Hong2, Daniel McClement3, Olayinka Oladosu4, Glen Pridham3, Garth Slaney2. 1. Radiology, University of Calgary, Alberta, T2N 4N1, Canada; Department of Clinical Neurosciences, University of Calgary, Alberta, T2N 4N1, Canada; Hotchkiss Brain Institute, University of Calgary, Alberta, T2N 4N1, Canada. Electronic address: yunyzhan@ucalgary.ca. 2. Departments of Schulich School of Engineering, University of Calgary, Alberta, T2N 4N1, Canada. 3. Hotchkiss Brain Institute, University of Calgary, Alberta, T2N 4N1, Canada. 4. Hotchkiss Brain Institute, University of Calgary, Alberta, T2N 4N1, Canada; Department of Neuroscience, University of Calgary, Alberta, T2N 4N1, Canada.
Abstract
BACKGROUND: Deep learning using convolutional neural networks (CNNs) has shown great promise in advancing neuroscience research. However, the ability to interpret the CNNs lags far behind, confounding their clinical translation. NEW METHOD: We interrogated 3 heatmap-generating techniques that have increasing generalizability for CNN interpretation: class activation mapping (CAM), gradient (Grad)-CAM, and Grad-CAM++. To investigate the impact of CNNs on heatmap generation, we also examined 6 different models trained to classify brain magnetic resonance imaging into 3 types: relapsing-remitting multiple sclerosis (RRMS), secondary progressive MS (SPMS), and control. Further, we designed novel methods to visualize and quantify the heatmaps to improve interpretability. RESULTS: Grad-CAM showed the best heatmap localizing ability, and CNNs with a global average pooling layer and pretrained weights had the best classification performance. Based on the best-performing CNN model, called VGG19, the 95th percentile values of Grad-CAM in SPMS were significantly higher than RRMS, indicating greater heterogeneity. Further, voxel-wise analysis of the thresholded Grad-CAM confirmed the difference identified visually between RRMS and SPMS in discriminative brain regions: occipital versus frontal and occipital, or temporal/parietal. COMPARISON WITH EXISTING METHODS: No study has examined the CAM methods together using clinical images. There is also lack of study on the impact of CNN architecture on heatmap outcomes, and of technologies to quantify heatmap patterns in clinical settings. CONCLUSIONS: Grad-CAM outperforms CAM and Grad-CAM++. Integrating Grad-CAM, novel heatmap quantification approaches, and robust CNN models may be an effective strategy in identifying the most crucial brain areas underlying disease development in MS.
BACKGROUND: Deep learning using convolutional neural networks (CNNs) has shown great promise in advancing neuroscience research. However, the ability to interpret the CNNs lags far behind, confounding their clinical translation. NEW METHOD: We interrogated 3 heatmap-generating techniques that have increasing generalizability for CNN interpretation: class activation mapping (CAM), gradient (Grad)-CAM, and Grad-CAM++. To investigate the impact of CNNs on heatmap generation, we also examined 6 different models trained to classify brain magnetic resonance imaging into 3 types: relapsing-remitting multiple sclerosis (RRMS), secondary progressive MS (SPMS), and control. Further, we designed novel methods to visualize and quantify the heatmaps to improve interpretability. RESULTS: Grad-CAM showed the best heatmap localizing ability, and CNNs with a global average pooling layer and pretrained weights had the best classification performance. Based on the best-performing CNN model, called VGG19, the 95th percentile values of Grad-CAM in SPMS were significantly higher than RRMS, indicating greater heterogeneity. Further, voxel-wise analysis of the thresholded Grad-CAM confirmed the difference identified visually between RRMS and SPMS in discriminative brain regions: occipital versus frontal and occipital, or temporal/parietal. COMPARISON WITH EXISTING METHODS: No study has examined the CAM methods together using clinical images. There is also lack of study on the impact of CNN architecture on heatmap outcomes, and of technologies to quantify heatmap patterns in clinical settings. CONCLUSIONS: Grad-CAM outperforms CAM and Grad-CAM++. Integrating Grad-CAM, novel heatmap quantification approaches, and robust CNN models may be an effective strategy in identifying the most crucial brain areas underlying disease development in MS.
Authors: Jasjit S Suri; Sushant Agarwal; Gian Luca Chabert; Alessandro Carriero; Alessio Paschè; Pietro S C Danna; Luca Saba; Armin Mehmedović; Gavino Faa; Inder M Singh; Monika Turk; Paramjit S Chadha; Amer M Johri; Narendra N Khanna; Sophie Mavrogeni; John R Laird; Gyan Pareek; Martin Miner; David W Sobel; Antonella Balestrieri; Petros P Sfikakis; George Tsoulfas; Athanasios D Protogerou; Durga Prasanna Misra; Vikas Agarwal; George D Kitas; Jagjit S Teji; Mustafa Al-Maini; Surinder K Dhanjil; Andrew Nicolaides; Aditya Sharma; Vijay Rathore; Mostafa Fatemi; Azra Alizad; Pudukode R Krishnan; Ferenc Nagy; Zoltan Ruzsa; Mostafa M Fouda; Subbaram Naidu; Klaudija Viskovic; Mannudeep K Kalra Journal: Diagnostics (Basel) Date: 2022-06-16