Diana L Miglioretti1, Laura Ichikawa2, Robert A Smith3, Diana S M Buist2, Patricia A Carney4, Berta Geller5, Barbara Monsees6, Tracy Onega7, Robert Rosenberg8, Edward A Sickles9, Bonnie C Yankaskas10, Karla Kerlikowske11. 1. Division of Biostatistics, Department of Public Health Sciences, University of California Davis School of Medicine, One Shields Ave., Med Sci 1C, Room 145, Davis, CA 95616; Kaiser Permanente Washington Health Research Institute, Seattle, Washington. Electronic address: dmiglioretti@ucdavis.edu. 2. Kaiser Permanente Washington Health Research Institute, Seattle, Washington. 3. Cancer Control Department, American Cancer Society, Atlanta, Georgia. 4. Departments of Family Medicine and Public Health and Preventive Medicine, School of Medicine: Mail Code FM, Oregon Health & Science University, Portland, Oregon. 5. Office of Health Promotion Research, Department of Family Medicine, University of Vermont, Burlington, Vermont. 6. Mallinckrodt Institute of Radiology, Washington University, St. Louis, Missouri. 7. Department of Community and Family Medicine, Dartmouth Medical School, HB 7927-Community & Family Medicine, Lebanon, New Hampshire. 8. University of New Mexico-HSC and Radiology Associates of Albuquerque, Albuquerque, New Mexico. 9. Department of Radiology, University of California, San Francisco Medical Center, San Francisco, California. 10. Department of Radiology, University of North Carolina, Chapel Hill, North Carolina. 11. Departments of Medicine and Epidemiology and Biostatistics, University of California, San Francisco, California; General Internal Medicine Section, Department of Veterans Affairs, University of California, San Francisco, California.
Abstract
RATIONALE AND OBJECTIVES: Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set. MATERIALS AND METHODS: This institutional review board-approved study randomized 83 radiologists from six Breast Cancer Surveillance Consortium registries to assess one of four test sets of 109 screening mammograms each; 48 radiologists completed a fifth test set of 110 mammograms 2 years later. Test sets differed in number of cancer cases and difficulty of lesion detection. Test set sensitivity and specificity were estimated using woman-level and breast-level recall with cancer status and expert opinion as gold standards. Clinical performance was estimated using women-level recall with cancer status as the gold standard. Spearman rank correlations between test set and clinical performance with 95% confidence intervals (CI) were estimated. RESULTS: For test sets with fewer cancers (N = 15) that were more difficult to detect, correlations were weak to moderate for sensitivity (woman level = 0.46, 95% CI = 0.16, 0.69; breast level = 0.35, 95% CI = 0.03, 0.61) and weak for specificity (0.24, 95% CI = 0.01, 0.45) relative to expert recall. Correlations for test sets with more cancers (N = 30) were close to 0 and not statistically significant. CONCLUSIONS: Correlations between screening performance on a test set and performance in clinical practice are not strong. Test set performance more accurately reflects performance in clinical practice if cancer prevalence is low and lesions are challenging to detect.
RATIONALE AND OBJECTIVES: Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set. MATERIALS AND METHODS: This institutional review board-approved study randomized 83 radiologists from six Breast Cancer Surveillance Consortium registries to assess one of four test sets of 109 screening mammograms each; 48 radiologists completed a fifth test set of 110 mammograms 2 years later. Test sets differed in number of cancer cases and difficulty of lesion detection. Test set sensitivity and specificity were estimated using woman-level and breast-level recall with cancer status and expert opinion as gold standards. Clinical performance was estimated using women-level recall with cancer status as the gold standard. Spearman rank correlations between test set and clinical performance with 95% confidence intervals (CI) were estimated. RESULTS: For test sets with fewer cancers (N = 15) that were more difficult to detect, correlations were weak to moderate for sensitivity (woman level = 0.46, 95% CI = 0.16, 0.69; breast level = 0.35, 95% CI = 0.03, 0.61) and weak for specificity (0.24, 95% CI = 0.01, 0.45) relative to expert recall. Correlations for test sets with more cancers (N = 30) were close to 0 and not statistically significant. CONCLUSIONS: Correlations between screening performance on a test set and performance in clinical practice are not strong. Test set performance more accurately reflects performance in clinical practice if cancer prevalence is low and lesions are challenging to detect.
Authors: Berta M Geller; Andy Bogart; Patricia A Carney; Edward A Sickles; Robert Smith; Barbara Monsees; Lawrence W Bassett; Diana M Buist; Karla Kerlikowske; Tracy Onega; Bonnie C Yankaskas; Sebastien Haneuse; Deirdre Hill; Matthew G Wallis; Diana Miglioretti Journal: AJR Am J Roentgenol Date: 2014-06 Impact factor: 3.959
Authors: Diana S M Buist; Melissa L Anderson; Robert A Smith; Patricia A Carney; Diana L Miglioretti; Barbara S Monsees; Edward A Sickles; Stephen H Taplin; Berta M Geller; Bonnie C Yankaskas; Tracy L Onega Journal: Radiology Date: 2014-06-24 Impact factor: 11.105
Authors: BaoLin Pauline Soh; Warwick Bruce Lee; Claudia Mello-Thoms; Kriscia Tapia; John Ryan; Wai Tak Hung; Graham Thompson; Rob Heard; Patrick Brennan Journal: J Med Imaging Radiat Oncol Date: 2015-04-01 Impact factor: 1.735
Authors: Patricia A Carney; T Andrew Bogart; Berta M Geller; Sebastian Haneuse; Karla Kerlikowske; Diana S M Buist; Robert Smith; Robert Rosenberg; Bonnie C Yankaskas; Tracy Onega; Diana L Miglioretti Journal: AJR Am J Roentgenol Date: 2012-04 Impact factor: 3.959
Authors: Joann G Elmore; Sara L Jackson; Linn Abraham; Diana L Miglioretti; Patricia A Carney; Berta M Geller; Bonnie C Yankaskas; Karla Kerlikowske; Tracy Onega; Robert D Rosenberg; Edward A Sickles; Diana S M Buist Journal: Radiology Date: 2009-10-28 Impact factor: 11.105
Authors: Nathaniel Hendrix; Brett Hauber; Christoph I Lee; Aasthaa Bansal; David L Veenstra Journal: J Am Med Inform Assoc Date: 2021-06-12 Impact factor: 4.497