Delaney Stuhr1, Ying Zhou1, Hai Pham1, Jian-Ping Xiong1, Shi Liu1, James G Mechalakos1, Sean L Berry2. 1. Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York. 2. Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York. Electronic address: BerryS@MSKCC.org.
Abstract
PURPOSE: This study aimed to perform a longitudinal analysis of the performance of our automated plan checking software by retrospectively evaluating the number of errors identified in plans delivered to patients in 3, month-long, data collection periods between 2017 and 2020. METHODS AND MATERIALS: Eleven automated checks were retrospectively run on 1169 external beam radiation therapy treatment plans identified as meeting the following criteria: planning target volume-based multifield photon plans receiving a status of treatment approved in March 2017, March 2018, or March 2020. The number of passes (true positives) and flags were recorded. Flags were subcategorized into false negatives, false negatives due to naming conventions, and true negatives. In addition, 2 × 2 contingency tables using a 2-tailed Fisher's exact test were used to determine whether there were nonrandom associations between the output of the automated plan checking software and whether the check was manual or automated at the original time of treatment approval. RESULTS: A statistically significant decrease in flags between the pre- and postautomation data sets was observed for 4 contour-based checks, namely adjacent structures overlap, empty structures and missing slices, overlap between body and couch, and laterality, as well as a check that determined whether the plan's global maximum dose was within the planning target volume. A review of the origins of false negatives was fed back into the design of the checks to improve the reliability of the system and help avoid warning fatigue. CONCLUSIONS: Periodic and longitudinal review of the performance of automated software was essential for monitoring and understanding its impact on error rates, as well as for optimization of the tool to adapt to regular changes of clinical practice. The automated plan checking software has demonstrated continuous contributions to the safe and effective delivery of external beam radiation therapy to our patient population, an impact that extends beyond its initial implementation and deployment.
PURPOSE: This study aimed to perform a longitudinal analysis of the performance of our automated plan checking software by retrospectively evaluating the number of errors identified in plans delivered to patients in 3, month-long, data collection periods between 2017 and 2020. METHODS AND MATERIALS: Eleven automated checks were retrospectively run on 1169 external beam radiation therapy treatment plans identified as meeting the following criteria: planning target volume-based multifield photon plans receiving a status of treatment approved in March 2017, March 2018, or March 2020. The number of passes (true positives) and flags were recorded. Flags were subcategorized into false negatives, false negatives due to naming conventions, and true negatives. In addition, 2 × 2 contingency tables using a 2-tailed Fisher's exact test were used to determine whether there were nonrandom associations between the output of the automated plan checking software and whether the check was manual or automated at the original time of treatment approval. RESULTS: A statistically significant decrease in flags between the pre- and postautomation data sets was observed for 4 contour-based checks, namely adjacent structures overlap, empty structures and missing slices, overlap between body and couch, and laterality, as well as a check that determined whether the plan's global maximum dose was within the planning target volume. A review of the origins of false negatives was fed back into the design of the checks to improve the reliability of the system and help avoid warning fatigue. CONCLUSIONS: Periodic and longitudinal review of the performance of automated software was essential for monitoring and understanding its impact on error rates, as well as for optimization of the tool to adapt to regular changes of clinical practice. The automated plan checking software has demonstrated continuous contributions to the safe and effective delivery of external beam radiation therapy to our patient population, an impact that extends beyond its initial implementation and deployment.
Authors: Lakshmi Santanam; Coen Hurkmans; Sasa Mutic; Corine van Vliet-Vroegindeweij; Scott Brame; William Straube; James Galvin; Prabhakar Tripuraneni; Jeff Michalski; Walter Bosch Journal: Int J Radiat Oncol Biol Phys Date: 2012-01-13 Impact factor: 7.038
Authors: M B Altman; J A Kavanaugh; H O Wooten; O L Green; T A DeWees; H Gay; W L Thorstad; H Li; S Mutic Journal: Phys Med Biol Date: 2015-06-17 Impact factor: 3.609
Authors: Koren S Smith; Kendra M Harris; Louis Potters; Rajiv Sharma; Sasa Mutic; Hiram A Gay; Jean Wright; Michael Samuels; Xiaobu Ye; Eric Ford; Stephanie Terezakis Journal: J Oncol Pract Date: 2014-08-05 Impact factor: 3.840
Authors: X Allen Li; An Tai; Douglas W Arthur; Thomas A Buchholz; Shannon Macdonald; Lawrence B Marks; Jean M Moran; Lori J Pierce; Rachel Rabinovitch; Alphonse Taghian; Frank Vicini; Wendy Woodward; Julia R White Journal: Int J Radiat Oncol Biol Phys Date: 2009-03-01 Impact factor: 7.038
Authors: Eric Ford; Leigh Conroy; Lei Dong; Luis Fong de Los Santos; Anne Greener; Grace Gwe-Ya Kim; Jennifer Johnson; Perry Johnson; James G Mechalakos; Brian Napolitano; Stephanie Parker; Deborah Schofield; Koren Smith; Ellen Yorke; Michelle Wells Journal: Med Phys Date: 2020-04-15 Impact factor: 4.071
Authors: Elizabeth L Covington; Xiaoping Chen; Kelly C Younge; Choonik Lee; Martha M Matuszak; Marc L Kessler; Wayne Keranen; Eduardo Acosta; Ashley M Dougherty; Stephanie E Filpansick; Jean M Moran Journal: J Appl Clin Med Phys Date: 2016-11-08 Impact factor: 2.102
Authors: Petra Reijnders-Thijssen; Diana Geerts; Wouter van Elmpt; Todd Pawlicki; Andrew Wallis; Mary Coffey Journal: Tech Innov Patient Support Radiat Oncol Date: 2020-06-12
Authors: Sean L Berry; Ying Zhou; Hai Pham; Sharif Elguindi; James G Mechalakos; Margie Hunt Journal: J Appl Clin Med Phys Date: 2020-03-20 Impact factor: 2.102