Andrey A Povyakalo1, Eugenio Alberdi1, Lorenzo Strigini1, Peter Ayton2. 1. Centre for Software Reliability, City University London, Northampton Square, London, UK (AAP, EA, LS) 2. Psychology Department, City University London, Northampton Square, London, UK (PA)
Abstract
BACKGROUND: Computer aids can affect decisions in complex ways, potentially even making them worse; common assessment methods may miss these effects. We developed a method for estimating the quality of decisions, as well as how computer aids affect it, and applied it to computer-aided detection (CAD) of cancer, reanalyzing data from a published study where 50 professionals ("readers") interpreted 180 mammograms, both with and without computer support. METHOD: We used stepwise regression to estimate how CAD affected the probability of a reader making a correct screening decision on a patient with cancer (sensitivity), thereby taking into account the effects of the difficulty of the cancer (proportion of readers who missed it) and the reader's discriminating ability (Youden's determinant). Using regression estimates, we obtained thresholds for classifying a posteriori the cases (by difficulty) and the readers (by discriminating ability). RESULTS: Use of CAD was associated with a 0.016 increase in sensitivity (95% confidence interval [CI], 0.003-0.028) for the 44 least discriminating radiologists for 45 relatively easy, mostly CAD-detected cancers. However, for the 6 most discriminating radiologists, with CAD, sensitivity decreased by 0.145 (95% CI, 0.034-0.257) for the 15 relatively difficult cancers. CONCLUSIONS: Our exploratory analysis method reveals unexpected effects. It indicates that, despite the original study detecting no significant average effect, CAD helped the less discriminating readers but hindered the more discriminating readers. Such differential effects, although subtle, may be clinically significant and important for improving both computer algorithms and protocols for their use. They should be assessed when evaluating CAD and similar warning systems.
BACKGROUND: Computer aids can affect decisions in complex ways, potentially even making them worse; common assessment methods may miss these effects. We developed a method for estimating the quality of decisions, as well as how computer aids affect it, and applied it to computer-aided detection (CAD) of cancer, reanalyzing data from a published study where 50 professionals ("readers") interpreted 180 mammograms, both with and without computer support. METHOD: We used stepwise regression to estimate how CAD affected the probability of a reader making a correct screening decision on a patient with cancer (sensitivity), thereby taking into account the effects of the difficulty of the cancer (proportion of readers who missed it) and the reader's discriminating ability (Youden's determinant). Using regression estimates, we obtained thresholds for classifying a posteriori the cases (by difficulty) and the readers (by discriminating ability). RESULTS: Use of CAD was associated with a 0.016 increase in sensitivity (95% confidence interval [CI], 0.003-0.028) for the 44 least discriminating radiologists for 45 relatively easy, mostly CAD-detected cancers. However, for the 6 most discriminating radiologists, with CAD, sensitivity decreased by 0.145 (95% CI, 0.034-0.257) for the 15 relatively difficult cancers. CONCLUSIONS: Our exploratory analysis method reveals unexpected effects. It indicates that, despite the original study detecting no significant average effect, CAD helped the less discriminating readers but hindered the more discriminating readers. Such differential effects, although subtle, may be clinically significant and important for improving both computer algorithms and protocols for their use. They should be assessed when evaluating CAD and similar warning systems.
Authors: Joshua J Fenton; Guibo Xing; Joann G Elmore; Heejung Bang; Steven L Chen; Karen K Lindfors; Laura-Mae Baldwin Journal: Ann Intern Med Date: 2013-04-16 Impact factor: 25.391
Authors: David Lyell; Farah Magrabi; Magdalena Z Raban; L G Pont; Melissa T Baysari; Richard O Day; Enrico Coiera Journal: BMC Med Inform Decis Mak Date: 2017-03-16 Impact factor: 2.796
Authors: Yasser Ibraheem Abdullah; Joel S Schuman; Ridwan Shabsigh; Arthur Caplan; Lama A Al-Aswad Journal: Asia Pac J Ophthalmol (Phila) Date: 2021 May-Jun 01
Authors: Scott Monteith; Tasha Glenn; John Geddes; Peter C Whybrow; Eric Achtyes; Michael Bauer Journal: Curr Psychiatry Rep Date: 2022-10-10 Impact factor: 8.081