Jinzhong Yang1, Wendy A Woodward2, Valerie K Reed2, Eric A Strom2, George H Perkins2, Welela Tereffe2, Thomas A Buchholz2, Lifei Zhang3, Peter Balter3, Laurence E Court3, X Allen Li4, Lei Dong5. 1. Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas. Electronic address: jyang4@mdanderson.org. 2. Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, Texas. 3. Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas. 4. Department of Radiation Oncology, Medical College of Wisconsin, Milwaukee, Wisconsin. 5. Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas; Scripps Proton Therapy Center, San Diego, California.
Abstract
PURPOSE: To develop a new approach for interobserver variability analysis. METHODS AND MATERIALS: Eight radiation oncologists specializing in breast cancer radiation therapy delineated a patient's left breast "from scratch" and from a template that was generated using deformable image registration. Three of the radiation oncologists had previously received training in Radiation Therapy Oncology Group consensus contouring for breast cancer atlas. The simultaneous truth and performance level estimation algorithm was applied to the 8 contours delineated "from scratch" to produce a group consensus contour. Individual Jaccard scores were fitted to a beta distribution model. We also applied this analysis to 2 or more patients, which were contoured by 9 breast radiation oncologists from 8 institutions. RESULTS: The beta distribution model had a mean of 86.2%, standard deviation (SD) of ±5.9%, a skewness of -0.7, and excess kurtosis of 0.55, exemplifying broad interobserver variability. The 3 RTOG-trained physicians had higher agreement scores than average, indicating that their contours were close to the group consensus contour. One physician had high sensitivity but lower specificity than the others, which implies that this physician tended to contour a structure larger than those of the others. Two other physicians had low sensitivity but specificity similar to the others, which implies that they tended to contour a structure smaller than the others. With this information, they could adjust their contouring practice to be more consistent with others if desired. When contouring from the template, the beta distribution model had a mean of 92.3%, SD ± 3.4%, skewness of -0.79, and excess kurtosis of 0.83, which indicated a much better consistency among individual contours. Similar results were obtained for the analysis of 2 additional patients. CONCLUSIONS: The proposed statistical approach was able to measure interobserver variability quantitatively and to identify individuals who tended to contour differently from the others. The information could be useful as feedback to improve contouring consistency.
PURPOSE: To develop a new approach for interobserver variability analysis. METHODS AND MATERIALS: Eight radiation oncologists specializing in breast cancer radiation therapy delineated a patient's left breast "from scratch" and from a template that was generated using deformable image registration. Three of the radiation oncologists had previously received training in Radiation Therapy Oncology Group consensus contouring for breast cancer atlas. The simultaneous truth and performance level estimation algorithm was applied to the 8 contours delineated "from scratch" to produce a group consensus contour. Individual Jaccard scores were fitted to a beta distribution model. We also applied this analysis to 2 or more patients, which were contoured by 9 breast radiation oncologists from 8 institutions. RESULTS: The beta distribution model had a mean of 86.2%, standard deviation (SD) of ±5.9%, a skewness of -0.7, and excess kurtosis of 0.55, exemplifying broad interobserver variability. The 3 RTOG-trained physicians had higher agreement scores than average, indicating that their contours were close to the group consensus contour. One physician had high sensitivity but lower specificity than the others, which implies that this physician tended to contour a structure larger than those of the others. Two other physicians had low sensitivity but specificity similar to the others, which implies that they tended to contour a structure smaller than the others. With this information, they could adjust their contouring practice to be more consistent with others if desired. When contouring from the template, the beta distribution model had a mean of 92.3%, SD ± 3.4%, skewness of -0.79, and excess kurtosis of 0.83, which indicated a much better consistency among individual contours. Similar results were obtained for the analysis of 2 additional patients. CONCLUSIONS: The proposed statistical approach was able to measure interobserver variability quantitatively and to identify individuals who tended to contour differently from the others. The information could be useful as feedback to improve contouring consistency.
Authors: Arya Amini; Jinzhong Yang; Ryan Williamson; Michelle L McBurney; Jeremy Erasmus; Pamela K Allen; Mandar Karhade; Ritsuko Komaki; Zhongxing Liao; Daniel Gomez; James Cox; Lei Dong; James Welsh Journal: Int J Radiat Oncol Biol Phys Date: 2012-03-01 Impact factor: 7.038
Authors: He Wang; Lei Dong; Ming Fwu Lii; Andrew L Lee; Renaud de Crevoisier; Radhe Mohan; James D Cox; Deborah A Kuban; Rex Cheung Journal: Int J Radiat Oncol Biol Phys Date: 2005-03-01 Impact factor: 7.038
Authors: Daniel M Landis; Weixiu Luo; Jun Song; Jennifer R Bellon; Rinaa S Punglia; Julia S Wong; Joseph H Killoran; Rebecca Gelman; Jay R Harris Journal: Int J Radiat Oncol Biol Phys Date: 2007-02-01 Impact factor: 7.038
Authors: Valerie K Reed; Wendy A Woodward; Lifei Zhang; Eric A Strom; George H Perkins; Welela Tereffe; Julia L Oh; T Kuan Yu; Isabelle Bedrosian; Gary J Whitman; Thomas A Buchholz; Lei Dong Journal: Int J Radiat Oncol Biol Phys Date: 2008-09-17 Impact factor: 7.038
Authors: Liza J Stapleford; Joshua D Lawson; Charles Perkins; Scott Edelman; Lawrence Davis; Mark W McDonald; Anthony Waller; Eduard Schreibmann; Tim Fox Journal: Int J Radiat Oncol Biol Phys Date: 2010-03-16 Impact factor: 7.038
Authors: C W Hurkmans; J H Borger; B R Pieters; N S Russell; E P Jansen; B J Mijnheer Journal: Int J Radiat Oncol Biol Phys Date: 2001-08-01 Impact factor: 7.038
Authors: Jinzhong Yang; Benjamin Haas; Raymond Fang; Beth M Beadle; Adam S Garden; Zhongxing Liao; Lifei Zhang; Peter Balter; Laurence Court Journal: Phys Med Biol Date: 2017-11-14 Impact factor: 3.609
Authors: Rongrong Zhou; Zhongxing Liao; Tinsu Pan; Sarah A Milgrom; Chelsea C Pinnix; Anhui Shi; Linglong Tang; Ju Yang; Ying Liu; Daniel Gomez; Quynh-Nhu Nguyen; Bouthaina S Dabaja; Laurence Court; Jinzhong Yang Journal: Radiother Oncol Date: 2016-12-08 Impact factor: 6.280
Authors: Wookjin Choi; Ming Xue; Barton F Lane; Min Kyu Kang; Kruti Patel; William F Regine; Paul Klahr; Jiahui Wang; Shifeng Chen; Warren D'Souza; Wei Lu Journal: Med Phys Date: 2016-10 Impact factor: 4.071
Authors: Ingrid Kristensen; Kristina Nilsson; Måns Agrup; Karin Belfrage; Anna Embring; Hedda Haugen; Anna-Maja Svärd; Tommy Knöös; Per Nilsson Journal: Tech Innov Patient Support Radiat Oncol Date: 2017-11-04