BACKGROUND: Imaging systems used in aviation security include segmentation algorithms in an automatic threat recognition pipeline. The segmentation algorithms evolve in response to emerging threats and changing performance requirements. Analysis of segmentation algorithms' behavior, including the nature of errors and feature recovery, facilitates their development. However, evaluation methods from the literature provide limited characterization of the segmentation algorithms. OBJECTIVE: To develop segmentation evaluation methods that measure systematic errors such as oversegmentation and undersegmentation, outliers, and overall errors. The methods must measure feature recovery and allow us to prioritize segments. METHODS: We developed two complementary evaluation methods using statistical techniques and information theory. We also created a semi-automatic method to define ground truth from 3D images. We applied our methods to evaluate five segmentation algorithms developed for CT luggage screening. We validated our methods with synthetic problems and an observer evaluation. RESULTS: Both methods selected the same best segmentation algorithm. Human evaluation confirmed the findings. The measurement of systematic errors and prioritization helped in understanding the behavior of each segmentation algorithm. CONCLUSIONS: Our evaluation methods allow us to measure and explain the accuracy of segmentation algorithms.
BACKGROUND: Imaging systems used in aviation security include segmentation algorithms in an automatic threat recognition pipeline. The segmentation algorithms evolve in response to emerging threats and changing performance requirements. Analysis of segmentation algorithms' behavior, including the nature of errors and feature recovery, facilitates their development. However, evaluation methods from the literature provide limited characterization of the segmentation algorithms. OBJECTIVE: To develop segmentation evaluation methods that measure systematic errors such as oversegmentation and undersegmentation, outliers, and overall errors. The methods must measure feature recovery and allow us to prioritize segments. METHODS: We developed two complementary evaluation methods using statistical techniques and information theory. We also created a semi-automatic method to define ground truth from 3D images. We applied our methods to evaluate five segmentation algorithms developed for CT luggage screening. We validated our methods with synthetic problems and an observer evaluation. RESULTS: Both methods selected the same best segmentation algorithm. Human evaluation confirmed the findings. The measurement of systematic errors and prioritization helped in understanding the behavior of each segmentation algorithm. CONCLUSIONS: Our evaluation methods allow us to measure and explain the accuracy of segmentation algorithms.
Authors: Tobias Heimann; Bram van Ginneken; Martin A Styner; Yulia Arzhaeva; Volker Aurich; Christian Bauer; Andreas Beck; Christoph Becker; Reinhard Beichel; György Bekes; Fernando Bello; Gerd Binnig; Horst Bischof; Alexander Bornik; Peter M M Cashman; Ying Chi; Andrés Cordova; Benoit M Dawant; Márta Fidrich; Jacob D Furst; Daisuke Furukawa; Lars Grenacher; Joachim Hornegger; Dagmar Kainmüller; Richard I Kitney; Hidefumi Kobatake; Hans Lamecker; Thomas Lange; Jeongjin Lee; Brian Lennon; Rui Li; Senhu Li; Hans-Peter Meinzer; Gábor Nemeth; Daniela S Raicu; Anne-Mareike Rau; Eva M van Rikxoort; Mikaël Rousson; László Rusko; Kinda A Saddi; Günter Schmidt; Dieter Seghers; Akinobu Shimizu; Pieter Slagmolen; Erich Sorantin; Grzegorz Soza; Ruchaneewan Susomboon; Jonathan M Waite; Andreas Wimmer; Ivo Wolf Journal: IEEE Trans Med Imaging Date: 2009-02-10 Impact factor: 10.048
Authors: Miguel A Lago; Aditya Jonnalagadda; Craig K Abbey; Bruno B Barufaldi; Predrag R Bakic; Andrew D A Maidment; Winifred K Leung; Susan P Weinstein; Brian S Englander; Miguel P Eckstein Journal: Curr Biol Date: 2021-01-19 Impact factor: 10.834