Indranil Balki1, Afsaneh Amirabadi2, Jacob Levman3, Anne L Martel4, Ziga Emersic5, Blaz Meden5, Angel Garcia-Pedrero6, Saul C Ramirez7, Dehan Kong8, Alan R Moody1, Pascal N Tyrrell9. 1. Department of Medical Imaging, University of Toronto, Toronto, Ontario, Canada. 2. Department of Medical Imaging, University of Toronto, Toronto, Ontario, Canada; Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Ontario, Canada. 3. Department of Mathematics, Statistics and Computer Science, St Francis Xavier University, Antigonish, Nova Scotia, Canada; Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts, USA. 4. Department of Medical Biophysics, University of Toronto, Toronto, Ontario, Canada. 5. Faculty of Computer and Information Science, University of Ljubljana, Ljubljana, Slovenia. 6. Department of Botany, Universidad de Valladolid, Castile and Leon, Spain. 7. Computing School, Instituto Tecnológico de Costa Rica, Cartago, Costa Rica. 8. Department of Statistical Sciences, University of Toronto, Toronto, Ontario, Canada. 9. Department of Medical Imaging, University of Toronto, Toronto, Ontario, Canada; Department of Statistical Sciences, University of Toronto, Toronto, Ontario, Canada. Electronic address: pascal.tyrrell@utoronto.ca.
Abstract
PURPOSE: The required training sample size for a particular machine learning (ML) model applied to medical imaging data is often unknown. The purpose of this study was to provide a descriptive review of current sample-size determination methodologies in ML applied to medical imaging and to propose recommendations for future work in the field. METHODS: We conducted a systematic literature search of articles using Medline and Embase with keywords including "machine learning," "image," and "sample size." The search included articles published between 1946 and 2018. Data regarding the ML task, sample size, and train-test pipeline were collected. RESULTS: A total of 167 articles were identified, of which 22 were included for qualitative analysis. There were only 4 studies that discussed sample-size determination methodologies, and 18 that tested the effect of sample size on model performance as part of an exploratory analysis. The observed methods could be categorized as pre hoc model-based approaches, which relied on features of the algorithm, or post hoc curve-fitting approaches requiring empirical testing to model and extrapolate algorithm performance as a function of sample size. Between studies, we observed great variability in performance testing procedures used for curve-fitting, model assessment methods, and reporting of confidence in sample sizes. CONCLUSIONS: Our study highlights the scarcity of research in training set size determination methodologies applied to ML in medical imaging, emphasizes the need to standardize current reporting practices, and guides future work in development and streamlining of pre hoc and post hoc sample size approaches.
PURPOSE: The required training sample size for a particular machine learning (ML) model applied to medical imaging data is often unknown. The purpose of this study was to provide a descriptive review of current sample-size determination methodologies in ML applied to medical imaging and to propose recommendations for future work in the field. METHODS: We conducted a systematic literature search of articles using Medline and Embase with keywords including "machine learning," "image," and "sample size." The search included articles published between 1946 and 2018. Data regarding the ML task, sample size, and train-test pipeline were collected. RESULTS: A total of 167 articles were identified, of which 22 were included for qualitative analysis. There were only 4 studies that discussed sample-size determination methodologies, and 18 that tested the effect of sample size on model performance as part of an exploratory analysis. The observed methods could be categorized as pre hoc model-based approaches, which relied on features of the algorithm, or post hoc curve-fitting approaches requiring empirical testing to model and extrapolate algorithm performance as a function of sample size. Between studies, we observed great variability in performance testing procedures used for curve-fitting, model assessment methods, and reporting of confidence in sample sizes. CONCLUSIONS: Our study highlights the scarcity of research in training set size determination methodologies applied to ML in medical imaging, emphasizes the need to standardize current reporting practices, and guides future work in development and streamlining of pre hoc and post hoc sample size approaches.
Authors: Reinhard Chun Wang Chau; Ming Chong; Khaing Myat Thu; Nate Sing Po Chu; Mohamad Koohi-Moghadam; Richard Tai-Chiu Hsung; Colman McGrath; Walter Yu Hang Lam Journal: PLoS One Date: 2022-06-02 Impact factor: 3.752
Authors: Andreas Vollmer; Babak Saravi; Michael Vollmer; Gernot Michael Lang; Anton Straub; Roman C Brands; Alexander Kübler; Sebastian Gubik; Stefan Hartmann Journal: Diagnostics (Basel) Date: 2022-06-06
Authors: Rachel Y L Kuo; Conrad Harrison; Terry-Ann Curran; Benjamin Jones; Alexander Freethy; David Cussons; Max Stewart; Gary S Collins; Dominic Furniss Journal: Radiology Date: 2022-03-29 Impact factor: 29.146
Authors: Saul Calderon-Ramirez; Shengxiang Yang; Armaghan Moemeni; David Elizondo; Simon Colreavy-Donnelly; Luis Fernando Chavarría-Estrada; Miguel A Molina-Cabello Journal: Appl Soft Comput Date: 2021-07-13 Impact factor: 6.725