Robert J H Miller1,2, Ananya Singh1, Yuka Otaki1, Balaji K Tamarappoo1, Paul Kavanagh1, Tejas Parekh1, Lien-Hsin Hu1,3, Heidi Gransar1, Tali Sharir4, Andrew J Einstein5, Mathews B Fish6, Terrence D Ruddy7, Philipp A Kaufmann8, Albert J Sinusas9, Edward J Miller9, Timothy M Bateman10, Sharmila Dorbala11, Marcelo F Di Carli11, Joanna X Liang1, Damini Dey1, Daniel S Berman1, Piotr J Slomka12. 1. Departments of Medicine, Division of Artificial Intelligence, Imaging and Biomedical Sciences, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Ste. Metro 203, Los Angeles, CA, 90048, USA. 2. Department of Cardiac Sciences, University of Calgary, Calgary, AB, Canada. 3. Department of Nuclear Medicine, Taipei Veterans General Hospital, Taipei, Taiwan. 4. Department of Nuclear Cardiology, Assuta Medical Centers, Tel Aviv, Israel and Ben Gurion University of the Negev, Beer Sheba, Israel. 5. Department of Medicine, and Department of Radiology, Division of Cardiology, Columbia University Irving Medical Center and New York-Presbyterian Hospital, New York, NY, USA. 6. Oregon Heart and Vascular Institute, Sacred Heart Medical Center, Springfield, OR, USA. 7. Division of Cardiology, University of Ottawa Heart Institute, Ottawa, ON, Canada. 8. Department of Nuclear Medicine, Cardiac Imaging, University Hospital Zurich, Zurich, Switzerland. 9. Section of Cardiovascular Medicine, Department of Internal Medicine, Yale University School of Medicine, New Haven, CT, USA. 10. Cardiovascular Imaging Technologies LLC, Kansas City, MO, USA. 11. Department of Radiology, Division of Nuclear Medicine and Molecular Imaging, Brigham and Women's Hospital, Boston, MA, USA. 12. Departments of Medicine, Division of Artificial Intelligence, Imaging and Biomedical Sciences, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Ste. Metro 203, Los Angeles, CA, 90048, USA. Piotr.Slomka@cshs.org.
Abstract
PURPOSE: Artificial intelligence (AI) has high diagnostic accuracy for coronary artery disease (CAD) from myocardial perfusion imaging (MPI). However, when trained using high-risk populations (such as patients with correlating invasive testing), the disease probability can be overestimated due to selection bias. We evaluated different strategies for training AI models to improve the calibration (accurate estimate of disease probability), using external testing. METHODS: Deep learning was trained using 828 patients from 3 sites, with MPI and invasive angiography within 6 months. Perfusion was assessed using upright (U-TPD) and supine total perfusion deficit (S-TPD). AI training without data augmentation (model 1) was compared to training with augmentation (increased sampling) of patients without obstructive CAD (model 2), and patients without CAD and TPD < 2% (model 3). All models were tested in an external population of patients with invasive angiography within 6 months (n = 332) or low likelihood of CAD (n = 179). RESULTS: Model 3 achieved the best calibration (Brier score 0.104 vs 0.121, p < 0.01). Improvement in calibration was particularly evident in women (Brier score 0.084 vs 0.124, p < 0.01). In external testing (n = 511), the area under the receiver operating characteristic curve (AUC) was higher for model 3 (0.930), compared to U-TPD (AUC 0.897) and S-TPD (AUC 0.900, p < 0.01 for both). CONCLUSION: Training AI models with augmentation of low-risk patients can improve calibration of AI models developed to identify patients with CAD, allowing more accurate assignment of disease probability. This is particularly important in lower-risk populations and in women, where overestimation of disease probability could significantly influence down-stream patient management.
PURPOSE: Artificial intelligence (AI) has high diagnostic accuracy for coronary artery disease (CAD) from myocardial perfusion imaging (MPI). However, when trained using high-risk populations (such as patients with correlating invasive testing), the disease probability can be overestimated due to selection bias. We evaluated different strategies for training AI models to improve the calibration (accurate estimate of disease probability), using external testing. METHODS: Deep learning was trained using 828 patients from 3 sites, with MPI and invasive angiography within 6 months. Perfusion was assessed using upright (U-TPD) and supine total perfusion deficit (S-TPD). AI training without data augmentation (model 1) was compared to training with augmentation (increased sampling) of patients without obstructive CAD (model 2), and patients without CAD and TPD < 2% (model 3). All models were tested in an external population of patients with invasive angiography within 6 months (n = 332) or low likelihood of CAD (n = 179). RESULTS: Model 3 achieved the best calibration (Brier score 0.104 vs 0.121, p < 0.01). Improvement in calibration was particularly evident in women (Brier score 0.084 vs 0.124, p < 0.01). In external testing (n = 511), the area under the receiver operating characteristic curve (AUC) was higher for model 3 (0.930), compared to U-TPD (AUC 0.897) and S-TPD (AUC 0.900, p < 0.01 for both). CONCLUSION: Training AI models with augmentation of low-risk patients can improve calibration of AI models developed to identify patients with CAD, allowing more accurate assignment of disease probability. This is particularly important in lower-risk populations and in women, where overestimation of disease probability could significantly influence down-stream patient management.
Authors: Piotr J Slomka; Hidetaka Nishina; Daniel S Berman; Cigdem Akincioglu; Aiden Abidov; John D Friedman; Sean W Hayes; Guido Germano Journal: J Nucl Cardiol Date: 2005 Jan-Feb Impact factor: 5.952
Authors: Sanjiv S Gambhir; Daniel S Berman; Jack Ziffer; Michael Nagler; Martin Sandler; Jim Patton; Brian Hutton; Tali Sharir; Shlomo Ben Haim; Simona Ben Haim Journal: J Nucl Med Date: 2009-04 Impact factor: 10.057
Authors: Sharmila Dorbala; Karthik Ananthasubramaniam; Ian S Armstrong; Panithaya Chareonthaitawee; E Gordon DePuey; Andrew J Einstein; Robert J Gropler; Thomas A Holly; John J Mahmarian; Mi-Ae Park; Donna M Polk; Raymond Russell; Piotr J Slomka; Randall C Thompson; R Glenn Wells Journal: J Nucl Cardiol Date: 2018-10 Impact factor: 5.952
Authors: Julian Betancur; Lien-Hsin Hu; Frederic Commandeur; Tali Sharir; Andrew J Einstein; Mathews B Fish; Terrence D Ruddy; Philipp A Kaufmann; Albert J Sinusas; Edward J Miller; Timothy M Bateman; Sharmila Dorbala; Marcelo Di Carli; Guido Germano; Yuka Otaki; Joanna X Liang; Balaji K Tamarappoo; Damini Dey; Daniel S Berman; Piotr J Slomka Journal: J Nucl Med Date: 2018-09-27 Impact factor: 10.057
Authors: Lien-Hsin Hu; Tali Sharir; Robert J H Miller; Andrew J Einstein; Mathews B Fish; Terrence D Ruddy; Sharmila Dorbala; Marcelo Di Carli; Philipp A Kaufmann; Albert J Sinusas; Edward J Miller; Timothy M Bateman; Julian Betancur; Guido Germano; Joanna X Liang; Frederic Commandeur; Peyman N Azadani; Heidi Gransar; Yuka Otaki; Balaji K Tamarappoo; Damini Dey; Daniel S Berman; Piotr J Slomka Journal: J Nucl Cardiol Date: 2019-05-13 Impact factor: 5.952
Authors: Julian Betancur; Frederic Commandeur; Mahsaw Motlagh; Tali Sharir; Andrew J Einstein; Sabahat Bokhari; Mathews B Fish; Terrence D Ruddy; Philipp Kaufmann; Albert J Sinusas; Edward J Miller; Timothy M Bateman; Sharmila Dorbala; Marcelo Di Carli; Guido Germano; Yuka Otaki; Balaji K Tamarappoo; Damini Dey; Daniel S Berman; Piotr J Slomka Journal: JACC Cardiovasc Imaging Date: 2018-03-14
Authors: Ben Van Calster; David J McLernon; Maarten van Smeden; Laure Wynants; Ewout W Steyerberg Journal: BMC Med Date: 2019-12-16 Impact factor: 8.775
Authors: Yuka Otaki; Ananya Singh; Paul Kavanagh; Robert J H Miller; Tejas Parekh; Balaji K Tamarappoo; Tali Sharir; Andrew J Einstein; Mathews B Fish; Terrence D Ruddy; Philipp A Kaufmann; Albert J Sinusas; Edward J Miller; Timothy M Bateman; Sharmila Dorbala; Marcelo Di Carli; Sebastien Cadet; Joanna X Liang; Damini Dey; Daniel S Berman; Piotr J Slomka Journal: JACC Cardiovasc Imaging Date: 2021-07-14