A prospective in silico analysis of interdisciplinary and interobserver spatial variability in post-operative target delineation of high-risk oral cavity cancers: Does physician specialty matter?

BACKGROUND: The aim of this study was to determine the interdisciplinary agreement in identifying the post-operative tumor bed.
METHODS: Three radiation oncologists (ROs), four surgeons, and three radiologists segmented post-operative tumor and nodal beds for three patients with oral cavity cancer. Specialty cohort composite contours were created by STAPLE algorithm implementation results for interspecialty comparison. Dice similarity coefficient and Hausdorff distance were utilized to compare spatial differentials between specialties.
RESULTS: There were significant differences between disciplines in target delineation. There was unacceptable variation in Dice similarity coefficient for each observer and discipline when compared to the STAPLE contours. Within surgery and radiology disciplines, there was good consistency in volumes. ROs and radiologists have similar Dice similarity coefficient scores compared to surgeons.
CONCLUSION: There were significant interdisciplinary differences in perceptions of tissue-at-risk. Better communication and explicit description of at-risk areas between disciplines is required to ensure high-risk areas are adequately targeted.

Introduction

Intensity modulated radiotherapy (IMRT) is a conformal radiation technique that enables the generation of steep dose gradients within complex geometries [1]. The widespread adoption of this modality has resulted in improved dose sparing of organs at risk, ultimately resulting in improved delivery of tumoricidal dose and dose-toxicity profiles. A shift from traditional two-dimensional (2D) treatment to use highly conformal IMRT treatment has greatly reduced concurrent and late-onset toxicity sequelae. However, this problem continues to be a challenge as even minor variability in treatment setup and/or execution may result in significant under dosage of at-risk areas and/or over dosage of surrounding normal tissues [2].

Modern evaluation and treatment of head and neck cancer (HNC) patients typically involves, and is dependent upon, the collective, coordinated expertise of multidisciplinary care teams with heavy input from radiologic, surgical, and radiation oncologic specialties. The multidisciplinary input in patient care has become widely accepted as “best practice,” having demonstrated measureable improvements in clinical quality indicators [3], [4]. The teams are continuously challenged with effectively communicating at all stages of the process (diagnosis, staging, treatment, support, rehabilitation, and follow-up) to maximize benefit to the patient. While communication itself can be challenging, the process may be further complicated by differing use (or understanding) of specialty-specific vernacular and/or therapeutic decision making algorithms.

Furthermore, the radiation oncologist, particularly in a non-academic setting, is dependent upon the descriptive language of the surgeons and/or radiologists in the post-operative setting if adequate pre-operative imaging is not available for comparison. Given the potential for significant adverse patient outcomes based on a variable interdisciplinary understanding of fundamental radiation oncology treatment paradigms we consequently sought to investigate the variation in delineation of target volumes in post-operative HNC patients recommended to receive adjuvant radiation therapy by all parties involved in a typical case. We evaluated whether any discrepancy in nomenclature, particularly ‘post-operative tumor bed’ and ‘post-operative nodal bed’, between disciplines was present, necessitating the need to establish a standardized set of definitions. Fundamentally, we sought to determine whether, when specialist head and neck surgeons, radiologists and radiation oncologists discuss the “post-operative tumor bed” they were actually talking about the same spatial region; furthermore, did the ‘surgical’, ‘radiological’ and ‘radiation oncological’ post-operative tumor bed mean the same within a specialty to differing physicians?

This study is a prospective in silico human performance evaluation to identify and quantify the intradisciplinary and interdisciplinary observer variability in post-operative target volume delineation using a standardized case set and accepted spatial metrics as a surrogate for shared understanding of where radiation should be directed in high-risk cases.

Materials and Methods

Imaging and contouring of cases

Three standardized cases of patients with resected oral cavity cancers and recommended to receive post-operative radiation therapy (PORT) were selected for this study. Patients were randomly chosen from an Institutional Review Board (IRB) approved dataset and real patient data were extracted from patient records. The patients had a planning CT scan acquired (CTAqSim, Philips Medical Systems) without IV contrast utilizing immobilization devices, including tongue-depressing oral stent, and head and neck thermoplastic mask [5], [6]. CT imaging was obtained from the vertex to the carina with 2 mm thickness slice thickness reconstruction [7]. Clinical information and planning CT images were anonymized and used for target delineation. After receiving a standardized set of explicit instructions [Appendix A], radiation oncologists (RO) specializing in HNC, head and neck surgeons (HNS), and head and neck radiologists (NR) contoured the post-operative tumor and nodal beds, respectively, using DICOM images in a commercial treatment planning/segmentation software (Pinnacle v9.0, Philips Medical Systems); expertise levels by specialty and years of experience are listed in Table 1. For users who were unfamiliar with the software interface (e.g. HNS and NR physicians, who do not use segmentation software daily), a skilled segmentation software user [BD/ASRM] was present throughout the initial contouring process to answer software interface questions only. Physicians were allowed access to all pertinent anonymized patient records including pre-operative imaging and any operative, pathologic or clinical note relevant to their task, excluding the actual delivered radiation treatment plan or planning notes. One radiation oncologist and one radiologist did not complete a majority (>50%) of contouring. Therefore their volumes were excluded from the analysis.

Table 1

Observers and their corresponding specialty and years of experience.

Specialty	Observer	Years of Experience
Radiation Oncology	RadOnc_A	4
	RadOnc_B	22
	RadOnc_C	2
Head and Neck Surgery	Surgeon_A	3
	Surgeon_B	10
	Surgeon_C	9
	Surgeon_D	5
Radiology	Radiologist_A	12
	Radiologist_B	5
	Radiologist_C	8

Comparison of volumes

Contour information was subsequently exported and analyzed using the EvaluateSegmentation program and metrics as described by Taha and Hanbury [8]. Contours were compared for agreement. The Warfield’s simultaneous truth and performance level estimation (STAPLE) algorithm was used to generate a consensus contour representing the ‘ground truth’ volume. Warfield’s STAPLE is an algorithm which incorporates multiple unordered and assumed independent segmentations to create an estimate of the hidden true segmentation, enabling characterization of the performance level of each observer [9]. The STAPLE volume was compared to each observer’s volumes, allowing direct comparison between observers’ volumes and the ‘ground truth’ volume [9].

The following metrics were included in the analysis:where A represents the observer dataset and B represents the STAPLE dataset [11] (Fig. 5).

Fig. 5

Illustration of the overlap metric where A represents the observer and G represents the gold standard (STAPLE).

Dice similarity coefficient (overlap based) – measures the similarity between two sets of segmentations and is calculated using the formula [10]

where true positive (TP) represents positive voxels in STAPLE and observer and false negative (FP) represents the positive voxels in observer segmentation but not in STAPLE.

Sensitivity and specificity (Information theoretic based) – Sensitivity, also known as the True Positive Rate (TPR), measures the voxels that are labeled positive by both the observer and STAPLE and is calculated by the formula: [8]

Similarly, specificity, also known as True Negative Rate (TNR), measures the voxels that are labeled negative by both observer and STAPLE: [8], where TN represents true negative, and FP represents false negative.

Hausdorff distance (Spatial distance based) measures the maximum distance between contours and is measured from one point in one set of segmentation to the closest point in another set of segmentation [8].

Statistical analysis

Individual and intradisciplinary results were concatenated and analyzed as a cohort using the JMP v.12.1.0 software package (SAS Institute; Cary, NC). The analyzed metrics for post-operative primary and nodal tumor bed delineation by all users were pooled according to user specialty. The non-parametric Steel-Dwass Method was used to compare all pairs of measurements whilst controlling the overall error rate [12]. A variability gauge modeled with main effect was generated and Bayesian analysis was performed to measure the variation in Dice coefficient similarity between observers and disciplines.

Results

Three patients with resected high-risk oral cavity tumors were chosen for this study. All three patients had T2-T3 N1-2 oral cavity squamous cell carcinomas resected with clear margins and were recommended to receive post-operative radiation therapy to improve locoregional control. A total of three ROs, four head and neck surgeons, and three radiologists completed all contours.

Interobserver variability

The volumes (in cc) contoured by each observer for each case are illustrated in Fig. 1. Dice similarity coefficients were used as a measure of conformity between observers. As demonstrated in Fig. 2, there was a high degree of variability existed between observers with an Acceptable Percent Measurement Variation of 99.7% (16.5% Repeatability and 83.5% Reproducibility). According to the proposed guidelines for acceptable Repeatability and Reproducibility percent by Barrentine [13], a measurement of <21% is acceptable, 21–30% is marginally acceptable and >30% is considered unacceptable.

Fig. 1

Variability chart illustrating the each observer’s volume (in cc) of contours by case and by discipline. The green line represents the mean volume for each observer.

Fig. 2

Variability gauge chart (top chart) depicting observer DICE similarity coefficient by case and by discipline, compared to the STAPLE (All). The bottom chart displays the standard deviation for the variability chart above. The green line in both charts represents the mean value for each observer. Abbreviations: NR – Radiology; RO – Radiation Oncology; HNS – Head and Neck Surgery.

Interdisciplinary variability

Table 2 summarizes the results for all metrics between disciplines. In summary, there were significant differences between disciplines for most metrics. Radiologists and radiation oncologists segmented relatively similar post-operative volumes compared to surgeons (Fig. 4). On review of individual contours, surgeons tend to delineate anatomical compartments whilst radiologists tend to contour the surgical bed. Radiation oncologists’ volumes were of an approximate average of surgeons and radiologists, but tend to be more similar to radiology colleagues, as demonstrated by the Hausdorff distance (Table 2), average volume contoured and Dice coefficient similarity (Fig. 3). The variability between interdisciplinary contours is depicted in Fig. 2 as well.

Table 2

Summary of metrics measurements and comparisons for each discipline. *depicts significant p-value (<0.05). Abbreviations: HNS – head and neck surgery; NR – radiology, RO – radiation oncology.

Metric	Discipline	Median	Range	Nonparametric Comparison Pairs	P-value
Volume (in cc)	RO	116.3	22.1–179.1	RO: HNS	0.0316*
	HNS	223.9	107.4–333.3	RO: NR	0.8569
	NR	206.6	36.9–173.6	HNS: NR	0.0025*
Sensitivity	RO	0.99	0.83–1	RO: HNS	0.6513
	HNS	0.98	0.43–1	RO: NR	0.0333*
	NR	1.00	0.66–1	HNS: NR	0.0094*
Specificity	RO	0.99	0.99–1	RO: HNS	0.0099*
	HNS	0.99	0.99–1	RO: NR	0.2520
	NR	0.99	0.99–1	HNS: NR	<0.0001*
Dice Similarity Coefficient (Dice)	RO	0.56	0.03–0.91	RO: HNS	0.0196*
	HNS	0.71	0.25–0.98	RO: NR	0.0623
	NR	0.33	0.14–0.60	HNS: NR	<0.0001*
Hausdorff Distance (in mm)	RO	49.38	19.21–257.18	RO: HNS	0.0029*
	HNS	22.95	3.16–251.13	RO: NR	0.9083
	NR	43.32	19.24–243.13	HNS: NR	0.0060*

Fig. 4

An example of contours delineated by radiation oncologist (red), radiologist (blue), and surgeon (green), and STAPLE contour (yellow).

Fig. 3

Box plots illustrating the distribution of volumes contoured and Dice similarity coefficient (minimum, maximum, median and interquartile range) between disciplines. Abbreviations: All – STAPLE; HNS – head and neck surgery; NR – radiology, RO – radiation oncology.

Intradisciplinary variability

Fig. 2 shows the intraspecialty interobserver variability for analyzed metric. As a specialty, the surgeons and radiologists were relatively consistent in delineating post-operative volumes with almost no significant interobserver difference existed for surgeons amongst the evaluated metrics. In the radiation oncology group, radiation oncologists A and C were consistent in delineating post-operative volumes whilst radiation oncologists B’s contours was less consistent (Fig. 2).

Discussion

The cardinal principle in radiation oncology planning is ‘do not miss the tumor’. This rule is usually achievable when there is intact tumor with normal (usually undistorted) anatomy that can be easily visualized on planning scan. This proves to be rather challenging in post-operative cases where the gross tumor has been resected and the patient’s anatomy has been altered by removal of tissue (resection), addition of new tissue (reconstruction) or surgical hardware and/or inflammation (healing). In the post-operative setting, radiation oncologists rely heavily on pre-operative images and operative reports to delineate the tumor and/or nodal beds accurately with assistance from his/ her diagnostic radiology and surgical colleagues to delineate the areas at risk or concern. It is recognized that in a pre-treatment setting, multidisciplinary team input has significant impact on patient’s treatment plan and potentially improving patients’ outcomes [4]. However, in the post-operative setting, it appears that these same multidisciplinary teams may have varied understanding or consensus of what consists of a ‘post-operative tumor bed’ and ‘post-operative nodal bed’. Our study investigated the interspecialty and intraspecialty consistency in delineation of target volumes in the post-operative setting for head and neck cancer patients.

Our results showed overall there were significant differences in segmentation between specialties. This translates to a difference in the understanding of the region treated in a post-operative setting. In a study with intact gross tumors, Horan et al. [14] reported good concordance of gross tumor delineation between radiologists and radiation oncologists, especially in head and neck tumor cases. When assessing interdisciplinary contouring variation, there was good consistency within each discipline, particularly among the surgeons. Within radiation oncology discipline, there was one person (RadOncB) with significantly different contours. This may be secondary to the experience of the clinician as Cardenas et al. [15] demonstrated that the experience of the treating radiation oncologist significantly affected the agreement between pre- and post-quality assurance segmentations. Previous studies evaluating interobserver variations in target delineation in head and neck cancer were predominantly in the definitive setting, comparing gross tumor volume delineation, between radiation oncologists with or without radiologists [14], [16], [17]. To our knowledge, this is the first prospective study comparing post-operative head and neck delineation. Studies in other disease sites have shown that when target volumes delineated by radiation oncologists were compared between institutions, a large discrepancy in target volumes was observed [18], [19]. However, when target volumes were compared within an institution, there was good concordance in gross tumor delineation between radiation oncologists and radiologists [17], consistent with our findings. Despite no significant interobserver difference in gross tumor volume, these studies showed that the volumes on PET/CT images were the most consistent between observers compared to volumes on CT [17], [20], [21] or MRI [21]. The role of PET/CT imaging is limited in post-operative cases, as the gross tumor has been resected.

In this study, we have demonstrated that there is an interdisciplinary difference in understanding of ‘at risk’ tissue in a post-operative setting. This can have significant implication for patient management. For example, a surgeon may perform a smaller resection with the perception that the treating radiation oncologist will irradiate the whole anatomical compartment. In this case, if the surgeon did not communicate his or her concerns of areas at risk accurately, the surgical ‘high-risk’ area may be not be included in the high radiation dose region, thus placing the patient at higher risk of local recurrence. On the contrary, if the treating surgeon was not aware that the radiation oncologist would treat all areas perturbed by the surgeon, the surgeon may make an incision quite a distance away from the tumor (e.g. to achieve better cosmetic outcome) without realizing that the patient will subsequently be treated with a larger radiation field to encompass the surgically-perturbed area, thereby increasing treatment toxicities. Similarly, a mutual understanding of the high-risk post-operative area is imperative between radiation oncologists and radiologists. We have shown that radiologists delineated smaller volumes than radiation oncologists. This can be an issue especially in the post-treatment setting, whereby a lesion detected on radiological scan may be reported as favoring a recurrence rather than post-treatment inflammation-related changes as the radiologist may have thought that the lesion was located outside the radiotherapy fields. This may result in additional and likely unnecessary further investigations, some of which may be invasive, to the patient.

Our study demonstrated significant interobserver and interdisciplinary differences in tumor bed delineation, emphasizing that peer review quality assurance remained of utmost importance to ensure accurate target delineation [22]. A recent single institution study by Cardenas et al. [15] indicated that peer review quality assurance led to major plan changes in 14% of patients. In a central quality assurance review of 687 treatment plans for a large international phase 3 head and neck study, Peters et al. [23] reported that 25% of patients had noncompliant plans, and 47% of noncompliant plans may have resulted in a significant adverse tumor control outcome. Furthermore, patients who received treatment with major deficiencies in treatment plan had significantly poorer clinical outcome in terms of freedom from locoregional failure and overall survival compared to those who had a protocol compliant treatment plan. While quality assurance requires additional clinical and treatment planning time, a simulated study on a head and neck cancer study by the EORTC 22071–24071 [24] showed that a quality assurance program not only improved patient outcomes but was also cost-effective. Therefore, it is advocated that every radiation oncology center should implement a peer review program, adding to the quality of patient care. In addition, we advocate extending this paradigm, such that a radiation plan review might ideally (at some interval) also provides feedback and information to the treating surgeons and radiologists. We suggest that radiation oncology treatment plans be made readily available and easily accessible in every patient’s medical record, allowing other non-radiation oncology physicians to review and better understand the regions treated. This will also allow rapid assessment and determination if any future normal tissue complications or recurrences may or may not be within the radiotherapy fields.

This study has its own limitations. To recapitulate real-life clinical workflow, the observers were allowed to access anonymized patients’ records, imaging reports, and diagnostic images. As some observers were the direct treating physicians for some of these patients, a potential bias could be imposed due to prior knowledge or recall of the intricacies of the case. Furthermore, this is a single institution study in a high volume cancer center, possibly reducing the magnitude of segmentation differences between disciplines and radiation oncologists. A study across different institutions is required to understand the general interdisciplinary agreement and/or understanding of ‘post-operative bed’. Although each discipline may have their own anatomical and radiological definition of ‘post-operative bed’, it is of utmost importance that each clinician communicates clearly with their colleagues when it comes to treating post-operative patients as it may alter patients’ radiotherapy fields and consequently, dose to target and normal tissues, and potential side effects to patients.

Nonetheless, to our knowledge, our data represents the first prospective rigorously executed multi-disciplinary (surgery, radiology and radiation oncology) specialist assessment of quantitative differences in defining the post-operative tumor bed. At a technical level, the importance of quantification is innovative; however, more important is the observation that different specialties are often talking about different and distinctive regions when discussing “the post-operative bed”. We recommend that future cooperative group target delineation efforts include non-radiation oncologists when incorporating post-operative radiotherapy. The rise of minimally invasive transoral robotic surgery (TORS) trials, such as the ECOG-3311 trial (ClinicalTrials.gov Identifier: NCT1898494) necessitates this move. It would be poor form to over-irradiate unnecessarily in trials where surgery is designed as a radiation dose de-escalation strategy, based only on failed nomenclature for the target volumes. This effort represents a first step towards such approaches, and brings the imperative nature of multidisciplinary care not only to clinical practice, but into the realm of spatial, not just conceptual agreement.

Conclusion

Overall, our study demonstrated significant variations in interdisciplinary target delineation, highlighting the importance of good communication and explicit description of ‘areas of concern’ amongst multidisciplinary care teams to ensure proper target delineation and adequate radiation coverage of at-risk post-operative areas. To our knowledge, this is the first prospective study comparing target delineation variations between all three main disciplines in post-operative oral cavity cases. As there were intra-discipline and inter-observer variations observed, peer-to-peer review of target delineation is recommended to ensure adequate target coverage while avoiding ‘over-contouring’ that may result in increased radiation-related acute and/or late toxicity to the patient. Although there are contouring guidelines available, post-operative cases target delineation can prove to be more difficult due to anatomical distortion. In these cases, peer-to-peer review is of utmost importance, in addition to communication with the treating surgical and radiological colleagues.

Prior presentation

Preliminary analyses and portions of this data were presented as a poster at the 2013 American Society of Radiation Oncology (ASTRO) Annual Meeting, September 22–25, 2013, Atlanta, GA, USA.

Co-author specific contributions

All listed co-authors performed the following:

Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work;

Drafting the work or revising it critically for important intellectual content

Final approval of the version to be published

Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Specific additional individual cooperative effort contributions to study/manuscript design/execution/interpretation, in addition to all criteria above are listed as follows:

SPN, BAD, MJA, ASRM, CDF - Manuscript writing, statistical analysis, oversight of all image segmentation, supervised statistical analysis.

JKC, CC – Segmentation analysis.

GBG, JP, MZ, JMD, CML, RRC, MEK, JH, BH, NGT, SL, DR, BH, JH – Image segmentation, contributory analytic support, clinical review and review of manuscript.

CDF- Corresponding author; conceived, coordinated, and directed all study activities, responsible for data collection, project integrity, manuscript content and editorial oversight and correspondence; direct oversight of trainee personnel. Statistical analysis and guarantor of statistical quality.

Conflict of interest

None

Multiple funders/agencies contributed to contributing personnel salary or project support during the study execution and manuscript preparation interval. This work was supported by Andrew Sabin Family Fellowship. Drs. Lai, Mohamed and Fuller receive funding support from the National Institutes of Health (NIH)/National Institute for Dental and Craniofacial Research (NIDCR) (1R01DE025248-01/R56DE025248-01). Drs. Canahuate, Marai, Vock, Mohamed and Fuller were previously funded via the National Science Foundation (NSF), Division of Mathematical Sciences, Joint NIH/NSF Initiative on Quantitative Approaches to Biomedical Big Data (QuBBD) Grant (NSF DMS-1557679) and are currently supported by the NIH National Cancer Institute (NCI)/Big Data to Knowledge (BD2K) Program (1R01CA214825-01). Dr. Marai received further support from the National Science Foundation (NSF IIS-1541277). Dr. Fuller received/(s) grant and/or salary support from the NIH/NCI Head and Neck Specialized Programs of Research Excellence (SPORE) Developmental Research Program Career Development Award (P50CA097007-10); the NCI Paul Calabresi Clinical Oncology Program Award (K12 CA088084-06); a General Electric Healthcare/MD Anderson Center for Advanced Biomedical Imaging In-Kind Award; an Elekta AB/MD Anderson Department of Radiation Oncology Seed Grant; the Center for Radiation Oncology Research (CROR) at MD Anderson Cancer Center Seed Grant; and the MD Anderson Institutional Research Grant (IRG) Program. Dr. Fuller has received speaker travel funding from Elekta AB. Supported in part by the NIH/NCI Cancer Center Support (Core) Grant CA016672 to The University of Texas MD Anderson Cancer Center (P30 CA016672). Dr. Ng received grant funding from the Royal Australian and New Zealand College of Radiologists (RANZCR) and the Radiological Society of North America (RSNA).