Ken Chang1, Andrew L Beers1, Laura Brink2, Jay B Patel1, Praveer Singh1, Nishanth T Arun1, Katharina V Hoebel1, Nathan Gaw1, Meesam Shah2, Etta D Pisano3, Mike Tilkin4, Laura P Coombs5, Keith J Dreyer6, Bibb Allen7, Sheela Agarwal8, Jayashree Kalpathy-Cramer9. 1. Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Boston, Massachusetts. 2. American College of Radiology, Reston, Virginia. 3. Chief Research Officer (ACR), Reston, Virginia; Professor in Residence, Beth Israel Lahey/Harvard Medical School, Boston, Massachusetts. 4. Chief Information Officer and EVP for Technology (ACR), Reston, Virginia. 5. Vice President (ACR), Reston, Virginia. 6. Chief Data Science Officer, Chief Imaging Information Officer, Massachussetts General Hospital and Brigham Women's Hospital (MGH & BWH), Chief Executive, MGH & BWH Center for Clinical Data Science; Vice Chairman of Radiology - Informatics, MGH & BWH, Boston, Massachusetts; Associate Professor of Radiology,Harvard Medical School, Boston, Massachusetts; Chief Science Officer, ACR Data Science Institute, Reston, Virginia. 7. Chief Medical Office, ACR Data Science Institute, Reston, Virginia; Secretary General, International Society of Radiology, Reston, Virginia; Partner, Grandview Medical Center, Birmingham, Alabama. 8. Lennox Hill Radiology, New York, New York. 9. Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Boston, Massachusetts; Scientific Director (CCDS), Director (QTIM lab and the Center for Machine Learning), Associate Professor of Radiology, MGH/Harvard Medical School, Boston, Massachusetts. Electronic address: kalpathy@nmr.mgh.harvard.edu.
Abstract
OBJECTIVE: We developed deep learning algorithms to automatically assess BI-RADS breast density. METHODS: Using a large multi-institution patient cohort of 108,230 digital screening mammograms from the Digital Mammographic Imaging Screening Trial, we investigated the effect of data, model, and training parameters on overall model performance and provided crowdsourcing evaluation from the attendees of the ACR 2019 Annual Meeting. RESULTS: Our best-performing algorithm achieved good agreement with radiologists who were qualified interpreters of mammograms, with a four-class κ of 0.667. When training was performed with randomly sampled images from the data set versus sampling equal number of images from each density category, the model predictions were biased away from the low-prevalence categories such as extremely dense breasts. The net result was an increase in sensitivity and a decrease in specificity for predicting dense breasts for equal class compared with random sampling. We also found that the performance of the model degrades when we evaluate on digital mammography data formats that differ from the one that we trained on, emphasizing the importance of multi-institutional training sets. Lastly, we showed that crowdsourced annotations, including those from attendees who routinely read mammograms, had higher agreement with our algorithm than with the original interpreting radiologists. CONCLUSION: We demonstrated the possible parameters that can influence the performance of the model and how crowdsourcing can be used for evaluation. This study was performed in tandem with the development of the ACR AI-LAB, a platform for democratizing artificial intelligence.
OBJECTIVE: We developed deep learning algorithms to automatically assess BI-RADS breast density. METHODS: Using a large multi-institution patient cohort of 108,230 digital screening mammograms from the Digital Mammographic Imaging Screening Trial, we investigated the effect of data, model, and training parameters on overall model performance and provided crowdsourcing evaluation from the attendees of the ACR 2019 Annual Meeting. RESULTS: Our best-performing algorithm achieved good agreement with radiologists who were qualified interpreters of mammograms, with a four-class κ of 0.667. When training was performed with randomly sampled images from the data set versus sampling equal number of images from each density category, the model predictions were biased away from the low-prevalence categories such as extremely dense breasts. The net result was an increase in sensitivity and a decrease in specificity for predicting dense breasts for equal class compared with random sampling. We also found that the performance of the model degrades when we evaluate on digital mammography data formats that differ from the one that we trained on, emphasizing the importance of multi-institutional training sets. Lastly, we showed that crowdsourced annotations, including those from attendees who routinely read mammograms, had higher agreement with our algorithm than with the original interpreting radiologists. CONCLUSION: We demonstrated the possible parameters that can influence the performance of the model and how crowdsourcing can be used for evaluation. This study was performed in tandem with the development of the ACR AI-LAB, a platform for democratizing artificial intelligence.
Authors: Nalini M Singh; Jordan B Harrod; Sandya Subramanian; Mitchell Robinson; Ken Chang; Suheyla Cetin-Karayumak; Adrian Vasile Dalca; Simon Eickhoff; Michael Fox; Loraine Franke; Polina Golland; Daniel Haehn; Juan Eugenio Iglesias; Lauren J O'Donnell; Yangming Ou; Yogesh Rathi; Shan H Siddiqi; Haoqi Sun; M Brandon Westover; Susan Whitfield-Gabrieli; Randy L Gollub Journal: Neuroinformatics Date: 2022-03-28
Authors: Kanan T Desai; Brian Befano; Zhiyun Xue; Helen Kelly; Nicole G Campos; Didem Egemen; Julia C Gage; Ana-Cecilia Rodriguez; Vikrant Sahasrabuddhe; David Levitz; Paul Pearlman; Jose Jeronimo; Sameer Antani; Mark Schiffman; Silvia de Sanjosé Journal: Int J Cancer Date: 2021-12-06 Impact factor: 7.316
Authors: Aimilia Gastounioti; Shyam Desai; Vinayak S Ahluwalia; Emily F Conant; Despina Kontos Journal: Breast Cancer Res Date: 2022-02-20 Impact factor: 8.408
Authors: Adam Hanif; İlkay Yıldız; Peng Tian; Beyza Kalkanlı; Deniz Erdoğmuş; Stratis Ioannidis; Jennifer Dy; Jayashree Kalpathy-Cramer; Susan Ostmo; Karyn Jonas; R V Paul Chan; Michael F Chiang; J Peter Campbell Journal: Ophthalmol Sci Date: 2022-02-02
Authors: Jimmy S Chen; Aaron S Coyner; Susan Ostmo; Kemal Sonmez; Sanyam Bajimaya; Eli Pradhan; Nita Valikodath; Emily D Cole; Tala Al-Khaled; R V Paul Chan; Praveer Singh; Jayashree Kalpathy-Cramer; Michael F Chiang; J Peter Campbell Journal: Ophthalmol Retina Date: 2021-02-06