Alexandre Ba1, Craig K Abbey2, Jongduk Baek3, Minah Han3, Ramona W Bouwman4, Christiana Balta4, Jovan Brankov5, Francesc Massanes5, Howard C Gifford6, Irene Hernandez-Giron7, Wouter J H Veldkamp7, Dimitar Petrov8, Nicholas Marshall8,9, Frank W Samuelson10, Rongping Zeng10, Justin B Solomon11, Ehsan Samei11, Pontus Timberg12, Hannie Förnvik12, Ingrid Reiser13, Lifeng Yu14, Hao Gong14, François O Bochud1. 1. Institute of Radiation Physics, Lausanne University Hospital, Lausanne, Switzerland. 2. Department of Psychological and Brain Sciences, University of California Santa Barbara, Santa Barbara, CA, 93106, USA. 3. School of Integrated Technology, Yonsei University, 406-840, Incheon, Korea. 4. Dutch Expert Centre for Screening, Radboud University Nijmegen Medical Centre (LRCB), P.O. Box 6873, 6503 GJ, Nijmegen, The Netherlands. 5. Department of Electrical and Computer Engineering, Illinois Institute of Technology, 3301 South Dearborn Street, Chicago, IL, 60616, USA. 6. Department of Biomedical Engineering, University of Houston, Houston, TX, 77204, USA. 7. Radiology Department, Leiden University Medical Center (LUMC), Albinusdreef 2, 2333ZA, Leiden, The Netherlands. 8. Department of Medical Physics and Quality Assessment, KU Leuven, Leuven, Belgium. 9. Department of Radiology, UZ Leuven, Leuven, Belgium. 10. Division of Imaging, Diagnostics and Software Reliability, US Food and Drug Administration, 10903 New Hampshire Ave Building 62, Room 3102, Silver Spring, MD, 20903-1058, USA. 11. Carl E. Ravin Advanced Imaging Laboratories, Departments of Radiology, Electrical and Computer Engineering, Biomedical Engineering, and Physics, Clinical Imaging Physics Group, Medical Physics Graduate Program, Duke University, Durham, NC, 27705, USA. 12. Department of Medical Radiation Physics, Translational Medicine Malmö, Lund University, Malmö, Sweden. 13. Department of Radiology, University of Chicago, 5841 S Maryland Ave, MC 2026, Chicago, IL, 60637, USA. 14. Department of Radiology, Mayo Clinic, Rochester, MN, USA.
Abstract
PURPOSE: The task-based assessment of image quality using model observers is increasingly used for the assessment of different imaging modalities. However, the performance computation of model observers needs standardization as well as a well-established trust in its implementation methodology and uncertainty estimation. The purpose of this work was to determine the degree of equivalence of the channelized Hotelling observer performance and uncertainty estimation using an intercomparison exercise. MATERIALS AND METHODS: Image samples to estimate model observer performance for detection tasks were generated from two-dimensional CT image slices of a uniform water phantom. A common set of images was sent to participating laboratories to perform and document the following tasks: (a) estimate the detectability index of a well-defined CHO and its uncertainty in three conditions involving different sized targets all at the same dose, and (b) apply this CHO to an image set where ground truth was unknown to participants (lower image dose). In addition, and on an optional basis, we asked the participating laboratories to (c) estimate the performance of real human observers from a psychophysical experiment of their choice. Each of the 13 participating laboratories was confidentially assigned a participant number and image sets could be downloaded through a secure server. Results were distributed with each participant recognizable by its number and then each laboratory was able to modify their results with justification as model observer calculation are not yet a routine and potentially error prone. RESULTS: Detectability index increased with signal size for all participants and was very consistent for 6 mm sized target while showing higher variability for 8 and 10 mm sized target. There was one order of magnitude between the lowest and the largest uncertainty estimation. CONCLUSIONS: This intercomparison helped define the state of the art of model observer performance computation and with thirteen participants, reflects openness and trust within the medical imaging community. The performance of a CHO with explicitly defined channels and a relatively large number of test images was consistently estimated by all participants. In contrast, the paper demonstrates that there is no agreement on estimating the variance of detectability in the training and testing setting.
PURPOSE: The task-based assessment of image quality using model observers is increasingly used for the assessment of different imaging modalities. However, the performance computation of model observers needs standardization as well as a well-established trust in its implementation methodology and uncertainty estimation. The purpose of this work was to determine the degree of equivalence of the channelized Hotelling observer performance and uncertainty estimation using an intercomparison exercise. MATERIALS AND METHODS: Image samples to estimate model observer performance for detection tasks were generated from two-dimensional CT image slices of a uniform water phantom. A common set of images was sent to participating laboratories to perform and document the following tasks: (a) estimate the detectability index of a well-defined CHO and its uncertainty in three conditions involving different sized targets all at the same dose, and (b) apply this CHO to an image set where ground truth was unknown to participants (lower image dose). In addition, and on an optional basis, we asked the participating laboratories to (c) estimate the performance of real human observers from a psychophysical experiment of their choice. Each of the 13 participating laboratories was confidentially assigned a participant number and image sets could be downloaded through a secure server. Results were distributed with each participant recognizable by its number and then each laboratory was able to modify their results with justification as model observer calculation are not yet a routine and potentially error prone. RESULTS: Detectability index increased with signal size for all participants and was very consistent for 6 mm sized target while showing higher variability for 8 and 10 mm sized target. There was one order of magnitude between the lowest and the largest uncertainty estimation. CONCLUSIONS: This intercomparison helped define the state of the art of model observer performance computation and with thirteen participants, reflects openness and trust within the medical imaging community. The performance of a CHO with explicitly defined channels and a relatively large number of test images was consistently estimated by all participants. In contrast, the paper demonstrates that there is no agreement on estimating the variance of detectability in the training and testing setting.
Authors: Anaïs Viry; Christoph Aberle; Thiago Lima; Reto Treier; Sebastian T Schindera; Francis R Verdun; Damien Racine Journal: Eur Radiol Date: 2021-07-29 Impact factor: 5.315