Karla Horsch1, Maryellen L Giger, Charles E Metz. 1. University of Chicago, Department of Radiology, 5841 South Maryland Avenue, Chicago, IL 60637, USA. kjhorsch@uchicago.edu
Abstract
RATIONALE AND OBJECTIVES: To investigate the effect of different reporting methods and performance measures on the assessment of the benefit of computer-aided diagnosis (CAD) in characterizing malignant and benign breast lesions on mammography and sonography. MATERIALS AND METHODS: In a previous study, 10 observers provided three types of reporting data (probability of malignancy [PM] estimates, Breast Imaging Reporting and Data System [BI-RADS] ratings, and biopsy decisions), both without and with CAD. The current study compares alternative performance measures computed from the three types of reporting data. The area under the receiver operating characteristic curve (AUC) was computed from both the PM estimates and the BI-RADS ratings, whereas sensitivity and specificity were computed from all three data types. Sensitivity and specificity values calculated from either the PM estimates or the BI-RADS ratings were determined by setting both constant and user-dependent thresholds. Student's t-tests were used to evaluate the statistical significance of the differences in the performance measures without and with CAD. RESULTS: The average AUC values of the 10 observers calculated from either PM estimates or BI-RADS ratings demonstrated statistically significant improvements in performance with CAD, increasing from 0.87 to 0.92 or 0.93, respectively. However, the statistical significance of improvements in sensitivity or specificity depended on the type of reporting data used. CONCLUSIONS: Use of different types of reporting data in the computation of sensitivity and specificity may result in different conclusions concerning the benefit of CAD. Meaningful determination of sensitivity and specificity from PM estimates require the use of user-dependent thresholds.
RATIONALE AND OBJECTIVES: To investigate the effect of different reporting methods and performance measures on the assessment of the benefit of computer-aided diagnosis (CAD) in characterizing malignant and benign breast lesions on mammography and sonography. MATERIALS AND METHODS: In a previous study, 10 observers provided three types of reporting data (probability of malignancy [PM] estimates, Breast Imaging Reporting and Data System [BI-RADS] ratings, and biopsy decisions), both without and with CAD. The current study compares alternative performance measures computed from the three types of reporting data. The area under the receiver operating characteristic curve (AUC) was computed from both the PM estimates and the BI-RADS ratings, whereas sensitivity and specificity were computed from all three data types. Sensitivity and specificity values calculated from either the PM estimates or the BI-RADS ratings were determined by setting both constant and user-dependent thresholds. Student's t-tests were used to evaluate the statistical significance of the differences in the performance measures without and with CAD. RESULTS: The average AUC values of the 10 observers calculated from either PM estimates or BI-RADS ratings demonstrated statistically significant improvements in performance with CAD, increasing from 0.87 to 0.92 or 0.93, respectively. However, the statistical significance of improvements in sensitivity or specificity depended on the type of reporting data used. CONCLUSIONS: Use of different types of reporting data in the computation of sensitivity and specificity may result in different conclusions concerning the benefit of CAD. Meaningful determination of sensitivity and specificity from PM estimates require the use of user-dependent thresholds.
Authors: Brandon D Gallas; Heang-Ping Chan; Carl J D'Orsi; Lori E Dodd; Maryellen L Giger; David Gur; Elizabeth A Krupinski; Charles E Metz; Kyle J Myers; Nancy A Obuchowski; Berkman Sahiner; Alicia Y Toledano; Margarita L Zuley Journal: Acad Radiol Date: 2012-02-03 Impact factor: 3.173