Jinzhong Yang1, Harini Veeraraghavan2, Samuel G Armato3, Keyvan Farahani4, Justin S Kirby5, Jayashree Kalpathy-Kramer6,7, Wouter van Elmpt8, Andre Dekker8, Xiao Han9, Xue Feng10, Paul Aljabar11, Bruno Oliveira12,13, Brent van der Heyden8, Leonid Zamdborg14, Dao Lam15, Mark Gooding11, Gregory C Sharp7. 1. Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. 2. Memorial Sloan Kettering Cancer Center, New York, NY, USA. 3. Department of Radiology, The University of Chicago, Chicago, IL, USA. 4. Cancer Imaging Program, National Cancer Institute, Bethesda, MD, USA. 5. Cancer Imaging Program, Frederick National Laboratory for Cancer Research sponsored by the National Cancer Institute, Frederick, MD, USA. 6. Harvard Medical School, Boston, MA, USA. 7. Massachusetts General Hospital, Boston, MA, USA. 8. Department of Radiation Oncology (MAASTRO), GROW - School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, The Netherlands. 9. Elekta Inc., Maryland Heights, MO, USA. 10. Department of Biomedical Engineering, University of Virginia, Charlottesville, VA, USA. 11. Mirada Medical Ltd, Oxford, UK. 12. Life and Health Sciences Research Institute (ICVS), School of Medicine, University of Minho, Braga, Portugal. 13. ICVS/3Bs - PT Government Associaste Laboratory, Braga/Guimares, Portugal. 14. Department of Radiation Oncology, Beaumont Health, Royal Oak, MI, USA. 15. Department of Radiation Oncology, Washington University School of Medicine in St. Louis, St. Louis, MO, USA.
Abstract
PURPOSE: This report presents the methods and results of the Thoracic Auto-Segmentation Challenge organized at the 2017 Annual Meeting of American Association of Physicists in Medicine. The purpose of the challenge was to provide a benchmark dataset and platform for evaluating performance of autosegmentation methods of organs at risk (OARs) in thoracic CT images. METHODS: Sixty thoracic CT scans provided by three different institutions were separated into 36 training, 12 offline testing, and 12 online testing scans. Eleven participants completed the offline challenge, and seven completed the online challenge. The OARs were left and right lungs, heart, esophagus, and spinal cord. Clinical contours used for treatment planning were quality checked and edited to adhere to the RTOG 1106 contouring guidelines. Algorithms were evaluated using the Dice coefficient, Hausdorff distance, and mean surface distance. A consolidated score was computed by normalizing the metrics against interrater variability and averaging over all patients and structures. RESULTS: The interrater study revealed highest variability in Dice for the esophagus and spinal cord, and in surface distances for lungs and heart. Five out of seven algorithms that participated in the online challenge employed deep-learning methods. Although the top three participants using deep learning produced the best segmentation for all structures, there was no significant difference in the performance among them. The fourth place participant used a multi-atlas-based approach. The highest Dice scores were produced for lungs, with averages ranging from 0.95 to 0.98, while the lowest Dice scores were produced for esophagus, with a range of 0.55-0.72. CONCLUSION: The results of the challenge showed that the lungs and heart can be segmented fairly accurately by various algorithms, while deep-learning methods performed better on the esophagus. Our dataset together with the manual contours for all training cases continues to be available publicly as an ongoing benchmarking resource.
PURPOSE: This report presents the methods and results of the Thoracic Auto-Segmentation Challenge organized at the 2017 Annual Meeting of American Association of Physicists in Medicine. The purpose of the challenge was to provide a benchmark dataset and platform for evaluating performance of autosegmentation methods of organs at risk (OARs) in thoracic CT images. METHODS: Sixty thoracic CT scans provided by three different institutions were separated into 36 training, 12 offline testing, and 12 online testing scans. Eleven participants completed the offline challenge, and seven completed the online challenge. The OARs were left and right lungs, heart, esophagus, and spinal cord. Clinical contours used for treatment planning were quality checked and edited to adhere to the RTOG 1106 contouring guidelines. Algorithms were evaluated using the Dice coefficient, Hausdorff distance, and mean surface distance. A consolidated score was computed by normalizing the metrics against interrater variability and averaging over all patients and structures. RESULTS: The interrater study revealed highest variability in Dice for the esophagus and spinal cord, and in surface distances for lungs and heart. Five out of seven algorithms that participated in the online challenge employed deep-learning methods. Although the top three participants using deep learning produced the best segmentation for all structures, there was no significant difference in the performance among them. The fourth place participant used a multi-atlas-based approach. The highest Dice scores were produced for lungs, with averages ranging from 0.95 to 0.98, while the lowest Dice scores were produced for esophagus, with a range of 0.55-0.72. CONCLUSION: The results of the challenge showed that the lungs and heart can be segmented fairly accurately by various algorithms, while deep-learning methods performed better on the esophagus. Our dataset together with the manual contours for all training cases continues to be available publicly as an ongoing benchmarking resource.
Authors: Carlos E Cardenas; Abdallah S R Mohamed; Jinzhong Yang; Mark Gooding; Harini Veeraraghavan; Jayashree Kalpathy-Cramer; Sweet Ping Ng; Yao Ding; Jihong Wang; Stephen Y Lai; Clifton D Fuller; Greg Sharp Journal: Med Phys Date: 2020-06 Impact factor: 4.071