Livia Faes1, Siegfried K Wagner2, Dun Jack Fu3, Xiaoxuan Liu4, Edward Korot5, Joseph R Ledsam6, Trevor Back6, Reena Chopra7, Nikolas Pontikos8, Christoph Kern9, Gabriella Moraes3, Martin K Schmid10, Dawn Sim2, Konstantinos Balaskas2, Lucas M Bachmann11, Alastair K Denniston12, Pearse A Keane13. 1. Department of Ophthalmology, Cantonal Hospital Lucerne, Lucerne, Switzerland; Medical Retina Department, Moorfields Eye Hospital National Health Service Foundation Trust, London, UK. 2. National Institute of Health Research Biomedical Research Center, Moorfields Eye Hospital National Health Service Foundation Trust, and University College London Institute of Ophthalmology, London, UK; Medical Retina Department, Moorfields Eye Hospital National Health Service Foundation Trust, London, UK. 3. Medical Retina Department, Moorfields Eye Hospital National Health Service Foundation Trust, London, UK. 4. National Institute of Health Research Biomedical Research Center, Moorfields Eye Hospital National Health Service Foundation Trust, and University College London Institute of Ophthalmology, London, UK; Department of Ophthalmology, University Hospitals Birmingham National Health Service Foundation Trust, Birmingham, UK; Academic Unit of Ophthalmology, Institute of Inflammation & Ageing, University of Birmingham, Birmingham, UK. 5. Medical Retina Department, Moorfields Eye Hospital National Health Service Foundation Trust, London, UK; Beaumont Eye Institute, Royal Oak, Michigan. 6. DeepMind, London, UK. 7. National Institute of Health Research Biomedical Research Center, Moorfields Eye Hospital National Health Service Foundation Trust, and University College London Institute of Ophthalmology, London, UK; Medical Retina Department, Moorfields Eye Hospital National Health Service Foundation Trust, London, UK; DeepMind, London, UK. 8. National Institute of Health Research Biomedical Research Center, Moorfields Eye Hospital National Health Service Foundation Trust, and University College London Institute of Ophthalmology, London, UK. 9. Medical Retina Department, Moorfields Eye Hospital National Health Service Foundation Trust, London, UK; Department of Ophthalmology, University Hospital of Ludwig Maximilian University, Munich, Germany. 10. Department of Ophthalmology, Cantonal Hospital Lucerne, Lucerne, Switzerland. 11. Medigntion, Zurich, Switzerland. 12. National Institute of Health Research Biomedical Research Center, Moorfields Eye Hospital National Health Service Foundation Trust, and University College London Institute of Ophthalmology, London, UK; Department of Ophthalmology, University Hospitals Birmingham National Health Service Foundation Trust, Birmingham, UK; Academic Unit of Ophthalmology, Institute of Inflammation & Ageing, University of Birmingham, Birmingham, UK; Centre for Patient Reported Outcome Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK. 13. National Institute of Health Research Biomedical Research Center, Moorfields Eye Hospital National Health Service Foundation Trust, and University College London Institute of Ophthalmology, London, UK; Medical Retina Department, Moorfields Eye Hospital National Health Service Foundation Trust, London, UK. Electronic address: pearse.keane1@nhs.net.
Abstract
BACKGROUND: Deep learning has the potential to transform health care; however, substantial expertise is required to train such models. We sought to evaluate the utility of automated deep learning software to develop medical image diagnostic classifiers by health-care professionals with no coding-and no deep learning-expertise. METHODS: We used five publicly available open-source datasets: retinal fundus images (MESSIDOR); optical coherence tomography (OCT) images (Guangzhou Medical University and Shiley Eye Institute, version 3); images of skin lesions (Human Against Machine [HAM] 10000), and both paediatric and adult chest x-ray (CXR) images (Guangzhou Medical University and Shiley Eye Institute, version 3 and the National Institute of Health [NIH] dataset, respectively) to separately feed into a neural architecture search framework, hosted through Google Cloud AutoML, that automatically developed a deep learning architecture to classify common diseases. Sensitivity (recall), specificity, and positive predictive value (precision) were used to evaluate the diagnostic properties of the models. The discriminative performance was assessed using the area under the precision recall curve (AUPRC). In the case of the deep learning model developed on a subset of the HAM10000 dataset, we did external validation using the Edinburgh Dermofit Library dataset. FINDINGS: Diagnostic properties and discriminative performance from internal validations were high in the binary classification tasks (sensitivity 73·3-97·0%; specificity 67-100%; AUPRC 0·87-1·00). In the multiple classification tasks, the diagnostic properties ranged from 38% to 100% for sensitivity and from 67% to 100% for specificity. The discriminative performance in terms of AUPRC ranged from 0·57 to 1·00 in the five automated deep learning models. In an external validation using the Edinburgh Dermofit Library dataset, the automated deep learning model showed an AUPRC of 0·47, with a sensitivity of 49% and a positive predictive value of 52%. INTERPRETATION: All models, except the automated deep learning model trained on the multilabel classification task of the NIH CXR14 dataset, showed comparable discriminative performance and diagnostic properties to state-of-the-art performing deep learning algorithms. The performance in the external validation study was low. The quality of the open-access datasets (including insufficient information about patient flow and demographics) and the absence of measurement for precision, such as confidence intervals, constituted the major limitations of this study. The availability of automated deep learning platforms provide an opportunity for the medical community to enhance their understanding in model development and evaluation. Although the derivation of classification models without requiring a deep understanding of the mathematical, statistical, and programming principles is attractive, comparable performance to expertly designed models is limited to more elementary classification tasks. Furthermore, care should be placed in adhering to ethical principles when using these automated models to avoid discrimination and causing harm. Future studies should compare several application programming interfaces on thoroughly curated datasets. FUNDING: National Institute for Health Research and Moorfields Eye Charity.
BACKGROUND: Deep learning has the potential to transform health care; however, substantial expertise is required to train such models. We sought to evaluate the utility of automated deep learning software to develop medical image diagnostic classifiers by health-care professionals with no coding-and no deep learning-expertise. METHODS: We used five publicly available open-source datasets: retinal fundus images (MESSIDOR); optical coherence tomography (OCT) images (Guangzhou Medical University and Shiley Eye Institute, version 3); images of skin lesions (Human Against Machine [HAM] 10000), and both paediatric and adult chest x-ray (CXR) images (Guangzhou Medical University and Shiley Eye Institute, version 3 and the National Institute of Health [NIH] dataset, respectively) to separately feed into a neural architecture search framework, hosted through Google Cloud AutoML, that automatically developed a deep learning architecture to classify common diseases. Sensitivity (recall), specificity, and positive predictive value (precision) were used to evaluate the diagnostic properties of the models. The discriminative performance was assessed using the area under the precision recall curve (AUPRC). In the case of the deep learning model developed on a subset of the HAM10000 dataset, we did external validation using the Edinburgh Dermofit Library dataset. FINDINGS: Diagnostic properties and discriminative performance from internal validations were high in the binary classification tasks (sensitivity 73·3-97·0%; specificity 67-100%; AUPRC 0·87-1·00). In the multiple classification tasks, the diagnostic properties ranged from 38% to 100% for sensitivity and from 67% to 100% for specificity. The discriminative performance in terms of AUPRC ranged from 0·57 to 1·00 in the five automated deep learning models. In an external validation using the Edinburgh Dermofit Library dataset, the automated deep learning model showed an AUPRC of 0·47, with a sensitivity of 49% and a positive predictive value of 52%. INTERPRETATION: All models, except the automated deep learning model trained on the multilabel classification task of the NIH CXR14 dataset, showed comparable discriminative performance and diagnostic properties to state-of-the-art performing deep learning algorithms. The performance in the external validation study was low. The quality of the open-access datasets (including insufficient information about patient flow and demographics) and the absence of measurement for precision, such as confidence intervals, constituted the major limitations of this study. The availability of automated deep learning platforms provide an opportunity for the medical community to enhance their understanding in model development and evaluation. Although the derivation of classification models without requiring a deep understanding of the mathematical, statistical, and programming principles is attractive, comparable performance to expertly designed models is limited to more elementary classification tasks. Furthermore, care should be placed in adhering to ethical principles when using these automated models to avoid discrimination and causing harm. Future studies should compare several application programming interfaces on thoroughly curated datasets. FUNDING: National Institute for Health Research and Moorfields Eye Charity.
Authors: Lena Maier-Hein; Matthias Eisenmann; Duygu Sarikaya; Keno März; Toby Collins; Anand Malpani; Johannes Fallert; Hubertus Feussner; Stamatia Giannarou; Pietro Mascagni; Hirenkumar Nakawala; Adrian Park; Carla Pugh; Danail Stoyanov; Swaroop S Vedula; Kevin Cleary; Gabor Fichtinger; Germain Forestier; Bernard Gibaud; Teodor Grantcharov; Makoto Hashizume; Doreen Heckmann-Nötzel; Hannes G Kenngott; Ron Kikinis; Lars Mündermann; Nassir Navab; Sinan Onogur; Tobias Roß; Raphael Sznitman; Russell H Taylor; Minu D Tizabi; Martin Wagner; Gregory D Hager; Thomas Neumuth; Nicolas Padoy; Justin Collins; Ines Gockel; Jan Goedeke; Daniel A Hashimoto; Luc Joyeux; Kyle Lam; Daniel R Leff; Amin Madani; Hani J Marcus; Ozanan Meireles; Alexander Seitel; Dogu Teber; Frank Ückert; Beat P Müller-Stich; Pierre Jannin; Stefanie Speidel Journal: Med Image Anal Date: 2021-11-18 Impact factor: 13.828
Authors: Nita G Valikodath; Emily Cole; Daniel S W Ting; J Peter Campbell; Louis R Pasquale; Michael F Chiang; R V Paul Chan Journal: Transl Vis Sci Technol Date: 2021-06-01 Impact factor: 3.283