PURPOSE: The authors examine potential bias when using a reference reader panel as "gold standard" for estimating operating characteristics of CAD algorithms for detecting lesions. As an alternative, the authors propose latent class analysis (LCA), which does not require an external gold standard to evaluate diagnostic accuracy. METHODS: A binomial model for multiple reader detections using different diagnostic protocols was constructed, assuming conditional independence of readings given true lesion status. Operating characteristics of all protocols were estimated by maximum likelihood LCA. Reader panel and LCA based estimates were compared using data simulated from the binomial model for a range of operating characteristics. LCA was applied to 36 thin section thoracic computed tomography data sets from the Lung Image Database Consortium (LIDC): Free search markings of four radiologists were compared to markings from four different CAD assisted radiologists. For real data, bootstrap-based resampling methods, which accommodate dependence in reader detections, are proposed to test of hypotheses of differences between detection protocols. RESULTS: In simulation studies, reader panel based sensitivity estimates had an average relative bias (ARB) of -23% to -27%, significantly higher (p-value < 0.0001) than LCA (ARB--2% to -6%). Specificity was well estimated by both reader panel (ARB -0.6% to -0.5%) and LCA (ARB 1.4%-0.5%). Among 1145 lesion candidates LIDC considered, LCA estimated sensitivity of reference readers (55%) was significantly lower (p-value 0.006) than CAD assisted readers' (68%). Average false positives per patient for reference readers (0.95) was not significantly lower (p-value 0.28) than CAD assisted readers' (1.27). CONCLUSIONS: Whereas a gold standard based on a consensus of readers may substantially bias sensitivity estimates, LCA may be a significantly more accurate and consistent means for evaluating diagnostic accuracy.
PURPOSE: The authors examine potential bias when using a reference reader panel as "gold standard" for estimating operating characteristics of CAD algorithms for detecting lesions. As an alternative, the authors propose latent class analysis (LCA), which does not require an external gold standard to evaluate diagnostic accuracy. METHODS: A binomial model for multiple reader detections using different diagnostic protocols was constructed, assuming conditional independence of readings given true lesion status. Operating characteristics of all protocols were estimated by maximum likelihood LCA. Reader panel and LCA based estimates were compared using data simulated from the binomial model for a range of operating characteristics. LCA was applied to 36 thin section thoracic computed tomography data sets from the Lung Image Database Consortium (LIDC): Free search markings of four radiologists were compared to markings from four different CAD assisted radiologists. For real data, bootstrap-based resampling methods, which accommodate dependence in reader detections, are proposed to test of hypotheses of differences between detection protocols. RESULTS: In simulation studies, reader panel based sensitivity estimates had an average relative bias (ARB) of -23% to -27%, significantly higher (p-value < 0.0001) than LCA (ARB--2% to -6%). Specificity was well estimated by both reader panel (ARB -0.6% to -0.5%) and LCA (ARB 1.4%-0.5%). Among 1145 lesion candidates LIDC considered, LCA estimated sensitivity of reference readers (55%) was significantly lower (p-value 0.006) than CAD assisted readers' (68%). Average false positives per patient for reference readers (0.95) was not significantly lower (p-value 0.28) than CAD assisted readers' (1.27). CONCLUSIONS: Whereas a gold standard based on a consensus of readers may substantially bias sensitivity estimates, LCA may be a significantly more accurate and consistent means for evaluating diagnostic accuracy.
Authors: Geoffrey D Rubin; John K Lyo; David S Paik; Anthony J Sherbondy; Lawrence C Chow; Ann N Leung; Robert Mindelzun; Pamela K Schraedley-Desmond; Steven E Zinck; David P Naidich; Sandy Napel Journal: Radiology Date: 2004-11-10 Impact factor: 11.105
Authors: Marco Das; Georg Mühlenbruch; Andreas H Mahnken; Thomas G Flohr; Lutz Gündel; Sven Stanzel; Thomas Kraus; Rolf W Günther; Joachim E Wildberger Journal: Radiology Date: 2006-11 Impact factor: 11.105
Authors: Matthew S Brown; Jonathan G Goldin; Sarah Rogers; Hyun J Kim; Robert D Suh; Michael F McNitt-Gray; Sumit K Shah; Dao Truong; Kathleen Brown; James W Sayre; David W Gjertson; Poonam Batra; Denise R Aberle Journal: Acad Radiol Date: 2005-06 Impact factor: 3.173
Authors: Joshua J Fenton; Stephen H Taplin; Patricia A Carney; Linn Abraham; Edward A Sickles; Carl D'Orsi; Eric A Berns; Gary Cutter; R Edward Hendrick; William E Barlow; Joann G Elmore Journal: N Engl J Med Date: 2007-04-05 Impact factor: 91.245
Authors: David S Paik; Christopher F Beaulieu; Geoffrey D Rubin; Burak Acar; R Brooke Jeffrey; Judy Yee; Joyoni Dey; Sandy Napel Journal: IEEE Trans Med Imaging Date: 2004-06 Impact factor: 10.048
Authors: Nicholas Petrick; Berkman Sahiner; Samuel G Armato; Alberto Bert; Loredana Correale; Silvia Delsanto; Matthew T Freedman; David Fryd; David Gur; Lubomir Hadjiiski; Zhimin Huo; Yulei Jiang; Lia Morra; Sophie Paquerault; Vikas Raykar; Frank Samuelson; Ronald M Summers; Georgia Tourassi; Hiroyuki Yoshida; Bin Zheng; Chuan Zhou; Heang-Ping Chan Journal: Med Phys Date: 2013-08 Impact factor: 4.071