Wendi Qu1, Indranil Balki1, Mauro Mendez1, John Valen1, Jacob Levman2,3, Pascal N Tyrrell4,5,6. 1. Department of Medical Imaging, University of Toronto, Toronto, ON, M5T 1W7, Canada. 2. Department of Mathematics, Statistics and Computer Science, St Francis Xavier University, Antigonish, NS, Canada. 3. Boston Children's Hospital, Harvard Medical School, Boston, MA, USA. 4. Department of Medical Imaging, University of Toronto, Toronto, ON, M5T 1W7, Canada. pascal.tyrrell@utoronto.ca. 5. Department of Statistical Sciences, University of Toronto, Toronto, ON, Canada. pascal.tyrrell@utoronto.ca. 6. Institute of Medical Science, University of Toronto, Toronto, ON, Canada. pascal.tyrrell@utoronto.ca.
Abstract
PURPOSE: Machine learning (ML) algorithms are well known to exhibit variations in prediction accuracy when provided with imbalanced training sets typically seen in medical imaging (MI) due to the imbalanced ratio of pathological and normal cases. This paper presents a thorough investigation of the effects of class imbalance and methods for mitigating class imbalance in ML algorithms applied to MI. METHODS: We first selected five classes from the Image Retrieval in Medical Applications (IRMA) dataset, performed multiclass classification using the random forest model (RFM), and then performed binary classification using convolutional neural network (CNN) on a chest X-ray dataset. An imbalanced class was created in the training set by varying the number of images in that class. Methods tested to mitigate class imbalance included oversampling, undersampling, and changing class weights of the RFM. Model performance was assessed by overall classification accuracy, overall F1 score, and specificity, recall, and precision of the imbalanced class. RESULTS: A close-to-balanced training set resulted in the best model performance, and a large imbalance with overrepresentation was more detrimental to model performance than underrepresentation. Oversampling and undersampling methods were both effective in mitigating class imbalance, and efficacy of oversampling techniques was class specific. CONCLUSION: This study systematically demonstrates the effect of class imbalance on two public X-ray datasets on RFM and CNN, making these findings widely applicable as a reference. Furthermore, the methods employed here can guide researchers in assessing and addressing the effects of class imbalance, while considering the data-specific characteristics to optimize imbalance mitigating methods.
PURPOSE: Machine learning (ML) algorithms are well known to exhibit variations in prediction accuracy when provided with imbalanced training sets typically seen in medical imaging (MI) due to the imbalanced ratio of pathological and normal cases. This paper presents a thorough investigation of the effects of class imbalance and methods for mitigating class imbalance in ML algorithms applied to MI. METHODS: We first selected five classes from the Image Retrieval in Medical Applications (IRMA) dataset, performed multiclass classification using the random forest model (RFM), and then performed binary classification using convolutional neural network (CNN) on a chest X-ray dataset. An imbalanced class was created in the training set by varying the number of images in that class. Methods tested to mitigate class imbalance included oversampling, undersampling, and changing class weights of the RFM. Model performance was assessed by overall classification accuracy, overall F1 score, and specificity, recall, and precision of the imbalanced class. RESULTS: A close-to-balanced training set resulted in the best model performance, and a large imbalance with overrepresentation was more detrimental to model performance than underrepresentation. Oversampling and undersampling methods were both effective in mitigating class imbalance, and efficacy of oversampling techniques was class specific. CONCLUSION: This study systematically demonstrates the effect of class imbalance on two public X-ray datasets on RFM and CNN, making these findings widely applicable as a reference. Furthermore, the methods employed here can guide researchers in assessing and addressing the effects of class imbalance, while considering the data-specific characteristics to optimize imbalance mitigating methods.
Keywords:
Class imbalance; Machine learning; Medical imaging; Radiology; X-ray
Authors: Justin R Camara; Roger T Tomihama; Andrew Pop; Matthew P Shedd; Brandon S Dobrowski; Cole J Knox; Ahmed M Abou-Zamzam; Sharon C Kiang Journal: J Vasc Surg Cases Innov Tech Date: 2022-05-02
Authors: Laith R Sultan; Theodore W Cary; Maryam Al-Hasani; Mrigendra B Karmacharya; Santosh S Venkatesh; Charles-Antoine Assenmacher; Enrico Radaelli; Chandra M Sehgal Journal: AI (Basel) Date: 2022-09-01