Tiffany Shao1, Lisa Wang1, Arnoud J Templeton1, Raymond Jang1, Francisco W Vera-Badillo1, Mairead G McNamara1, Myles Margolis1, Tae Kyoung Kim1, Mehrdad Sinaei1, Hassan Shoushtari1, Ian F Tannock2. 1. Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS). 2. Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS). ian.tannock@uhn.ca.
Abstract
BACKGROUND: "Waterfall plots" are used to describe changes in tumor size observed in clinical studies. Here we assess criteria for generation of waterfall plots and the impact of measurement error in generating them. METHODS: We reviewed published waterfall plots to investigate variability in criteria used to define them. We then compared waterfall plots generated by different observers for 24 patients enrolled in a completed phase I study of solid tumors with available computed tomography (CT) scans. Tumor measurements were made independently from CT scans according to Response Evaluation Criteria in Solid Tumors 1.1 by four board-certified radiologists and four medical oncologists. Interobserver variability was quantified and compared with reference measurements reported for the phase 1 study. All statistical tests were two-sided. RESULTS: There was substantial variability in criteria used to generate published waterfall plots. In the internal study, the results were statistically significantly different between all eight readers (P = .01, variance = 197.1, SD = 14.0) and between the oncologists (P = .01, variance = 319.0, SD = 17.9), but not between the radiologists (P = .68, variance = 70.8, SD = 8.4). Different observers classified one to five patients as having a partial response and 12-19 patients as having stable disease. Similar variability in categorization of response was observed when these error rates were applied to published waterfall plots. CONCLUSION: Waterfall plots are subject to substantial variability in criteria used to define them and are influenced by measurement errors; they should be generated by trained radiologists. Caution should be exercised when interpreting results of waterfall plots in the context of clinical trials.
BACKGROUND: "Waterfall plots" are used to describe changes in tumor size observed in clinical studies. Here we assess criteria for generation of waterfall plots and the impact of measurement error in generating them. METHODS: We reviewed published waterfall plots to investigate variability in criteria used to define them. We then compared waterfall plots generated by different observers for 24 patients enrolled in a completed phase I study of solid tumors with available computed tomography (CT) scans. Tumor measurements were made independently from CT scans according to Response Evaluation Criteria in Solid Tumors 1.1 by four board-certified radiologists and four medical oncologists. Interobserver variability was quantified and compared with reference measurements reported for the phase 1 study. All statistical tests were two-sided. RESULTS: There was substantial variability in criteria used to generate published waterfall plots. In the internal study, the results were statistically significantly different between all eight readers (P = .01, variance = 197.1, SD = 14.0) and between the oncologists (P = .01, variance = 319.0, SD = 17.9), but not between the radiologists (P = .68, variance = 70.8, SD = 8.4). Different observers classified one to five patients as having a partial response and 12-19 patients as having stable disease. Similar variability in categorization of response was observed when these error rates were applied to published waterfall plots. CONCLUSION: Waterfall plots are subject to substantial variability in criteria used to define them and are influenced by measurement errors; they should be generated by trained radiologists. Caution should be exercised when interpreting results of waterfall plots in the context of clinical trials.
Authors: Puey Ling Chia; Craig Gedye; Paul C Boutros; Paul Wheatley-Price; Thomas John Journal: J Natl Cancer Inst Date: 2016-05-31 Impact factor: 13.506
Authors: Nikhil Goyal; Andrea B Apolo; Eliana D Berman; Mohammad Hadi Bagheri; Jason E Levine; John W Glod; Rosandra N Kaplan; Laura B Machado; Les R Folio Journal: J Digit Imaging Date: 2017-06 Impact factor: 4.056