Johanna Uthoff1,2, Nicholas Koehn1, Jared Larson1, Samantha K N Dilger1,2, Emily Hammond1,2, Ann Schwartz3, Brian Mullan1, Rolando Sanchez4, Richard M Hoffman4, Jessica C Sieren5,6. 1. Department of Radiology, University of Iowa, 200 Hawkins Drive cc704 GH, Iowa City, IA, 52242, USA. 2. Department of Biomedical Engineering, University of Iowa, 5601 Seamans Center, Iowa City, IA, 52242, USA. 3. Karmanos Cancer Institute, Wayne State University, 4100 John R St, Detroit, MI, 48201, USA. 4. Department of Internal Medicine, University of Iowa, 200 Hawkins Drive, Iowa City, IA, 52242, USA. 5. Department of Radiology, University of Iowa, 200 Hawkins Drive cc704 GH, Iowa City, IA, 52242, USA. jessica-sieren@uiowa.edu. 6. Department of Biomedical Engineering, University of Iowa, 5601 Seamans Center, Iowa City, IA, 52242, USA. jessica-sieren@uiowa.edu.
Abstract
OBJECTIVES: Post-imaging mathematical prediction models (MPMs) provide guidance for the management of solid pulmonary nodules by providing a lung cancer risk score from demographic and radiologists-indicated imaging characteristics. We hypothesized calibrating the MPM risk score threshold to a local study cohort would result in improved performance over the original recommended MPM thresholds. We compared the pre- and post-calibration performance of four MPM models and determined if improvement in MPM prediction occurs as nodules are imaged longitudinally. MATERIALS AND METHODS: A common cohort of 317 individuals with computed tomography-detected, solid nodules (80 malignant, 237 benign) were used to evaluate the MPM performance. We created a web-based application for this study that allows others to easily calibrate thresholds and analyze the performance of MPMs on their local cohort. Thirty patients with repeated imaging were tested for improved performance longitudinally. RESULTS: Using calibrated thresholds, Mayo Clinic and Brock University (BU) MPMs performed the best (AUC = 0.63, 0.61) compared to the Veteran's Affairs (0.51) and Peking University (0.55). Only BU had consensus with the original MPM threshold; the other calibrated thresholds improved MPM accuracy. No significant improvements in accuracy were found longitudinally between time points. CONCLUSIONS: Calibration to a common cohort can select the best-performing MPM for your institution. Without calibration, BU has the most stable performance in solid nodules ≥ 8 mm but has only moderate potential to refine subjects into appropriate workup. Application of MPM is recommended only at initial evaluation as no increase in accuracy was achieved over time. KEY POINTS: • Post-imaging lung cancer risk mathematical predication models (MPMs) perform poorly on local populations without calibration. • An application is provided to facilitate calibration to new study cohorts: the Mayo Clinic model, the U.S. Department of Veteran's Affairs model, the Brock University model, and the Peking University model. • No significant improvement in risk prediction occurred in nodules with repeated imaging sessions, indicating the potential value of risk prediction application is limited to the initial evaluation.
OBJECTIVES: Post-imaging mathematical prediction models (MPMs) provide guidance for the management of solid pulmonary nodules by providing a lung cancer risk score from demographic and radiologists-indicated imaging characteristics. We hypothesized calibrating the MPM risk score threshold to a local study cohort would result in improved performance over the original recommended MPM thresholds. We compared the pre- and post-calibration performance of four MPM models and determined if improvement in MPM prediction occurs as nodules are imaged longitudinally. MATERIALS AND METHODS: A common cohort of 317 individuals with computed tomography-detected, solid nodules (80 malignant, 237 benign) were used to evaluate the MPM performance. We created a web-based application for this study that allows others to easily calibrate thresholds and analyze the performance of MPMs on their local cohort. Thirty patients with repeated imaging were tested for improved performance longitudinally. RESULTS: Using calibrated thresholds, Mayo Clinic and Brock University (BU) MPMs performed the best (AUC = 0.63, 0.61) compared to the Veteran's Affairs (0.51) and Peking University (0.55). Only BU had consensus with the original MPM threshold; the other calibrated thresholds improved MPM accuracy. No significant improvements in accuracy were found longitudinally between time points. CONCLUSIONS: Calibration to a common cohort can select the best-performing MPM for your institution. Without calibration, BU has the most stable performance in solid nodules ≥ 8 mm but has only moderate potential to refine subjects into appropriate workup. Application of MPM is recommended only at initial evaluation as no increase in accuracy was achieved over time. KEY POINTS: • Post-imaging lung cancer risk mathematical predication models (MPMs) perform poorly on local populations without calibration. • An application is provided to facilitate calibration to new study cohorts: the Mayo Clinic model, the U.S. Department of Veteran's Affairs model, the Brock University model, and the Peking University model. • No significant improvement in risk prediction occurred in nodules with repeated imaging sessions, indicating the potential value of risk prediction application is limited to the initial evaluation.
Entities:
Keywords:
Area under the curve; Logistic models; Lung neoplasms; Risk assessment; Tomography, x-ray computed
Authors: S J Swensen; M D Silverstein; E S Edell; V F Trastek; G L Aughenbaugh; D M Ilstrup; C D Schleck Journal: Mayo Clin Proc Date: 1999-04 Impact factor: 7.616
Authors: Annette McWilliams; Martin C Tammemagi; John R Mayo; Heidi Roberts; Geoffrey Liu; Kam Soghrati; Kazuhiro Yasufuku; Simon Martel; Francis Laberge; Michel Gingras; Sukhinder Atkar-Khattra; Christine D Berg; Ken Evans; Richard Finley; John Yee; John English; Paola Nasute; John Goffin; Serge Puksa; Lori Stewart; Scott Tsai; Michael R Johnston; Daria Manos; Garth Nicholas; Glenwood D Goss; Jean M Seely; Kayvan Amjadi; Alain Tremblay; Paul Burrowes; Paul MacEachern; Rick Bhatia; Ming-Sound Tsao; Stephen Lam Journal: N Engl J Med Date: 2013-09-05 Impact factor: 91.245
Authors: Ali Al-Ameri; Puneet Malhotra; Helene Thygesen; Paul K Plant; Sri Vaidyanathan; Shishir Karthik; Andrew Scarsbrook; Matthew E J Callister Journal: Lung Cancer Date: 2015-03-28 Impact factor: 5.705
Authors: Michael F McNitt-Gray; Grace Hyun Kim; Binsheng Zhao; Lawrence H Schwartz; David Clunie; Kristin Cohen; Nicholas Petrick; Charles Fenimore; Z Q John Lu; Andrew J Buckler Journal: Transl Oncol Date: 2015-02 Impact factor: 4.243
Authors: Sarah J van Riel; Francesco Ciompi; Colin Jacobs; Mathilde M Winkler Wille; Ernst Th Scholten; Matiullah Naqibullah; Stephen Lam; Mathias Prokop; Cornelia Schaefer-Prokop; Bram van Ginneken Journal: Eur Radiol Date: 2017-03-14 Impact factor: 5.315
Authors: Fabien Maldonado; Cyril Varghese; Srinivasan Rajagopalan; Fenghai Duan; Aneri B Balar; Dhairya A Lakhani; Sanja L Antic; Pierre P Massion; Tucker F Johnson; Ronald A Karwoski; Richard A Robb; Brian J Bartholmai; Tobias Peikert Journal: Eur Respir J Date: 2021-04-01 Impact factor: 16.671