RATIONALE AND OBJECTIVES: Studies that evaluate the lung nodule detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the "truth"). The purpose of this study was to analyze (1) variability in the "truth" defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of "truth" in the context of lung nodule detection in computed tomographic (CT) scans. MATERIALS AND METHODS: Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules >or=3 mm in maximum diameter. Panel "truth" sets of nodules were then derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detection performance of the other radiologists was evaluated based on these panel "truth" sets. RESULTS: The number of "true" nodules in the different panel "truth" sets ranged from 15 to 89 (mean 49.8 +/- 25.6). The mean radiologist nodule detection sensitivities across radiologists and panel "truth" sets for different panel "truth" conditions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case. CONCLUSIONS: Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of "truth" on which lung nodule detection studies are based must be carefully considered, because even experienced thoracic radiologists may not perform well when measured against the "truth" established by other experienced thoracic radiologists.
RATIONALE AND OBJECTIVES: Studies that evaluate the lung nodule detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the "truth"). The purpose of this study was to analyze (1) variability in the "truth" defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of "truth" in the context of lung nodule detection in computed tomographic (CT) scans. MATERIALS AND METHODS: Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules >or=3 mm in maximum diameter. Panel "truth" sets of nodules were then derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detection performance of the other radiologists was evaluated based on these panel "truth" sets. RESULTS: The number of "true" nodules in the different panel "truth" sets ranged from 15 to 89 (mean 49.8 +/- 25.6). The mean radiologist nodule detection sensitivities across radiologists and panel "truth" sets for different panel "truth" conditions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case. CONCLUSIONS: Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of "truth" on which lung nodule detection studies are based must be carefully considered, because even experienced thoracic radiologists may not perform well when measured against the "truth" established by other experienced thoracic radiologists.
Authors: Samuel G Armato; Geoffrey McLennan; Michael F McNitt-Gray; Charles R Meyer; David Yankelevitz; Denise R Aberle; Claudia I Henschke; Eric A Hoffman; Ella A Kazerooni; Heber MacMahon; Anthony P Reeves; Barbara Y Croft; Laurence P Clarke Journal: Radiology Date: 2004-09 Impact factor: 11.105
Authors: Joseph K Leader; Thomas E Warfel; Carl R Fuhrman; Sara K Golla; Joel L Weissfeld; Ricardo S Avila; Wesly D Turner; Bin Zheng Journal: AJR Am J Roentgenol Date: 2005-10 Impact factor: 3.959
Authors: Charles R Meyer; Timothy D Johnson; Geoffrey McLennan; Denise R Aberle; Ella A Kazerooni; Heber Macmahon; Brian F Mullan; David F Yankelevitz; Edwin J R van Beek; Samuel G Armato; Michael F McNitt-Gray; Anthony P Reeves; David Gur; Claudia I Henschke; Eric A Hoffman; Peyton H Bland; Gary Laderach; Richie Pais; David Qing; Chris Piker; Junfeng Guo; Adam Starkey; Daniel Max; Barbara Y Croft; Laurence P Clarke Journal: Acad Radiol Date: 2006-10 Impact factor: 3.173
Authors: Anthony P Reeves; Alberto M Biancardi; Tatiyana V Apanasovich; Charles R Meyer; Heber MacMahon; Edwin J R van Beek; Ella A Kazerooni; David Yankelevitz; Michael F McNitt-Gray; Geoffrey McLennan; Samuel G Armato; Claudia I Henschke; Denise R Aberle; Barbara Y Croft; Laurence P Clarke Journal: Acad Radiol Date: 2007-12 Impact factor: 3.173
Authors: L H Schwartz; M S Ginsberg; D DeCorato; L N Rothenberg; S Einstein; P Kijewski; D M Panicek Journal: J Clin Oncol Date: 2000-05 Impact factor: 44.544
Authors: Lori E Dodd; Robert F Wagner; Samuel G Armato; Michael F McNitt-Gray; Sergey Beiden; Heang-Ping Chan; David Gur; Geoffrey McLennan; Charles E Metz; Nicholas Petrick; Berkman Sahiner; Jim Sayre Journal: Acad Radiol Date: 2004-04 Impact factor: 3.173
Authors: A R Larici; M Amato; P Ordóñez; F Maggi; L Menchini; A Caulo; L Calandriello; G Vallati; S Giunta; M Crecco; L Bonomo Journal: Radiol Med Date: 2012-02-10 Impact factor: 3.469
Authors: Luca Bogoni; Jane P Ko; Jeffrey Alpert; Vikram Anand; John Fantauzzi; Charles H Florin; Chi Wan Koo; Derek Mason; William Rom; Maria Shiau; Marcos Salganicoff; David P Naidich Journal: J Digit Imaging Date: 2012-12 Impact factor: 4.056
Authors: Brandon D Gallas; Heang-Ping Chan; Carl J D'Orsi; Lori E Dodd; Maryellen L Giger; David Gur; Elizabeth A Krupinski; Charles E Metz; Kyle J Myers; Nancy A Obuchowski; Berkman Sahiner; Alicia Y Toledano; Margarita L Zuley Journal: Acad Radiol Date: 2012-02-03 Impact factor: 3.173
Authors: Hani H Abujudeh; Giles W Boland; Rathachai Kaewlai; Pavel Rabiner; Elkarn F Halpern; G Scott Gazelle; James H Thrall Journal: Eur Radiol Date: 2010-03-25 Impact factor: 5.315
Authors: Nicholas Petrick; Berkman Sahiner; Samuel G Armato; Alberto Bert; Loredana Correale; Silvia Delsanto; Matthew T Freedman; David Fryd; David Gur; Lubomir Hadjiiski; Zhimin Huo; Yulei Jiang; Lia Morra; Sophie Paquerault; Vikas Raykar; Frank Samuelson; Ronald M Summers; Georgia Tourassi; Hiroyuki Yoshida; Bin Zheng; Chuan Zhou; Heang-Ping Chan Journal: Med Phys Date: 2013-08 Impact factor: 4.071
Authors: Michael F McNitt-Gray; Luc M Bidaut; Samuel G Armato; Charles R Meyer; Marios A Gavrielides; Charles Fenimore; Geoffrey McLennan; Nicholas Petrick; Binsheng Zhao; Anthony P Reeves; Reinhard Beichel; Hyun-Jung Grace Kim; Lisa Kinnard Journal: Transl Oncol Date: 2009-12 Impact factor: 4.243