Prospective Clinical Evaluation of Integrating a Radiation Anatomist for Contouring in Routine Radiation Treatment Planning.

Purpose: A radiation anatomist was trained and integrated into clinical practice at a multi-site academic center. The primary objective of this quality improvement study was to determine whether a radiation anatomist improves the quality of organ-at-risk (OAR) contours, and secondarily to determine the impact on efficiency in the treatment planning process. Methods and Materials: From March to August 2020, all patients undergoing computed tomography-based radiation planning at 2 clinics at Memorial Sloan Kettering Cancer Center were assigned using an "every other" process to either (1) OAR contouring by a radiation anatomist (intervention) or (2) contouring by the treating physician (standard of care). Blinded dosimetrists reported OAR contour quality using a 3-point scoring system based on a common clinical trial protocol deviation scale (1, acceptable; 2, minor deviation; and 3, major deviation). Physicians reported time spent contouring for all cases. Analyses included the Fisher exact test and multivariable ordinal logistic regression.
Results: There were 249 cases with data available for the primary endpoint (66% response rate). The mean OAR quality rating was 1.1 ± 0.4 for the intervention group and 1.4 ± 0.7 for the standard of care group (P < .001), with subset analysis showing a significant difference for gastrointestinal cases (n = 49; P <.001). Time from simulation to contour approval was reduced from 3 days (interquartile range [IQR], 1-6 days) in the control group to 2 days (IQR, 1-5 days) in the intervention group (P = .007). Both physicians and dosimetrists self-reported decreased time spent contouring in the intervention group compared with the control group, with a decreases of 8 minutes (17%; P < .001) and 5 minutes (50%; P = .002), respectively. Qualitative comments most often indicated edits required to bowel contours (n = 14). Conclusions: These findings support improvements in both OAR contour quality and workflow efficiency with implementation of a radiation anatomist in routine practice. Findings could also inform development of autosegmentation by identifying disease sites and specific OARs contributing to low clinical efficiency. Future research is needed to determine the potential effect of reduced physician time spent contouring OARs on burnout.

Introduction

Precise delineation of organs at risk (OARs) is essential to optimal quality and safety of radiation treatment plans. Multiple studies have shown that interobserver variation in the delineation of OARs exists in disease sites including the central nervous system, head and neck, thorax, and pelvis.1, 2, 3, 4 In gastrointestinal cancers, variation in OAR contouring has been associated with worse dosimetry and increased clinical toxicity., Poor-quality OAR contours have also been noted to be limitations in clinical trial efforts in radiation oncology. To date, efforts to standardize normal-tissue contouring have focused on dissemination of consensus guidelines (including atlases) by several organizations and institutions, which have become increasingly available as radiation planning has become more complex; however, variation persists.

Normal structure delineation is also time consuming, providing a potential rate-limiting step in the treatment planning process. A study conducted by the German Society of Radiation Oncology has shown that manual segmentation is the most time-consuming task for physicians, who spend, on average, 74 minutes contouring for each patient. One strategy to decrease time spent contouring is autosegmentation. However, most validated algorithms still require additional modification by clinical experts, and therefore, they have to-date contributed only modest time savings.12, 13, 14

Therefore, there is an apparent need to identify additional strategies to standardize OAR contour quality while improving efficiency in the treatment planning process. Our institution trained a full-time radiation anatomist to contour OARs for our regional network. We hypothesized that a dedicated radiation anatomist would improve consistency of OAR contours while reducing physician workload and not prolonging time to contour approval.

Methods and Materials

Study design

This was an observational quality improvement study in which data were prospectively collected as part of routine care and approved by the institutional review board at Memorial Sloan Kettering Cancer Center for retrospective analysis. From March to August 2020, all patients who received radiation treatment requiring computed tomography (CT)–based planning (ie, 3-dimensional conformal radiation therapy, intensity modulated radiation therapy, and stereotactic body radiation therapy [SBRT]) at 2 regional clinics within our institutional network were included. Treatments for all disease sites were included except for the prostate, owing to an ongoing autosegmentation evaluation. Using an “every other” binning process (ie, alternating cases between the 2 study groups), we assigned patients to have OARs contoured by either the treating physician (standard of care [SOC]) or a radiation anatomist (intervention). The treating radiation oncologists included in this study were a median of 7 years out from medical school (range, 6-14 years). Contour quality, as defined below, was selected as the primary endpoint to highlight the potential patient-centered clinical effects of implementing an anatomist in routine practice. Physician efficiency and time to contour approval were included as secondary endpoints.

Radiation anatomist onboarding

The radiation anatomist previously received a master's degree in anatomy and physiology but had no prior knowledge of cross-sectional imaging. The anatomist's onboarding involved (1) developing a set of anonymized cases previously contoured by physicians for self-directed practice simulation with immediate contour comparison, (2) referencing consensus guideline atlases, (3) receiving case-specific feedback from physicians via an ARIA 17.0 (Varian Medical Systems, Inc, Palo Alto, California) task with 1 multiple choice question using the protocol deviation scale and 1 open-ended section for comments, and (4) reviewing difficult cases and questions with a radiologist. Training was structured such that 1 disease site at a time was added to the case list in routine practice and was considered complete at 8 months when the anatomist was contouring all disease sites with consistently high ratings from the physician scoring task. Once onboarded, the radiation anatomist contoured on average 50 cases per week and allotted 10% of time to education, quality assurance (QA), and research, including autosegmentation development.

OAR contour workflow and assessment

The overall workflow is outlined in the schema in Figure 1. For every eligible simulation scan sent to the contouring platform, MIM 7.1.6. (MIM Software Inc, Cleveland, Ohio), the radiation anatomist loaded and saved a session before any contouring to blind the evaluating dosimetrists to the person who performed the initial OAR contours. A physician survey was assigned (via ARIA task) for every eligible patient during the study, which included self-reported time spent contouring and, for the intervention group only, a question regarding OAR quality rating using the 3-point protocol deviation scale.

Figure 1

Study schema and randomization process for standard-of-care and intervention study arms.

Given evidence for its association with clinical outcomes, a contour quality scoring system commonly used in the clinical trial setting to assign protocol deviations was selected, whereby "1" signified that OAR contours were acceptable with no edits, "2" signified that OAR contours were acceptable with minor edits not likely to affect the treatment plan, and "3" signified that OAR contours were unacceptable and major edits were made that would likely affect the treatment plan.

Subsequently, physicians were allowed to make edits to the anatomist-generated OARs for quality assurance. In both study groups, once OARs had been approved by the physicians, they were rated by blinded dosimetrists using the same 3-point protocol deviation scoring system via a second short survey (ARIA task). If contours were rated as 2 or 3, the survey prompted the dosimetrist to provide additional comments, which were qualitatively reviewed for themes. Dosimetrists also reported their total time spent making contour edits in preparation for planning. One training session was held with all dosimetrists to review the scoring criteria, and clarifications were answered ad hoc by the study team.

Time to contour approval

Time to contour approval (which initiates the treatment planning process) is a known rate-limiting step in the time to treatment sequence. To measure this secondary endpoint, time stamps from MIM Software were collected retrospectively for the point at which the case was ready to contour (after simulation) and upon MD contour approval.

Statistical analysis and sample size

The primary outcome of interest was OAR contour quality, which we analyzed as a categorical variable. Interrater reliability between the physician's and dosimetrist's ratings of radiation anatomist–contoured OARs was calculated using the joint agreement probability method. Means and standard deviations for OAR quality score are reported, and the Fisher exact test was used to compare radiation anatomist and physician OAR contour ratings. Given that the OAR quality rating was an ordinal categorical variable with 3 levels, an ordinal logistic regression model was constructed to estimate the effect of various predictors on the outcome of interest—OAR quality rating—assuming proportional odds. We verified that the assumption of proportional odds (ie, the odds ratio is similar across all levels of OAR quality scoring) was reasonable. The primary independent variable of interest was who performed initial OAR contouring, the radiation anatomist (intervention) or an MD (standard of care). Other independent variables included treatment technique and disease site. The Wilcoxon rank sum test was used to assess differences in time spent contouring and time to contour approval between the standard-of-care and intervention groups. A 2-sided P value < .05 was set as the threshold of significance, with the exception of Bonferroni correction applied for subset analyses with an adjusted 2-sided P value of .005. All statistical analysis was performed using RStudio (R foundation for Statistical Computing, Vienna, Austria). Qualitative comments from physicians and dosimetrists were organized thematically.

To determine study sample size, we reviewed available retrospective data (n = 88), which suggested an average score (mean ± SD) of 1.50 ± 0.37 for physician OAR contours and 1.15 ± 0.35 for anatomist contours. Sample size calculation (for comparing 2 means from 2 independent samples with α = 0.05 and β = 0.8) suggested only 36 patients would be required. To further assess patient subsets by anatomic disease sites of greatest interest (including head and neck, gastrointestinal, lung, metastases, and spine), we scaled up the sample size by 5 times for a total of 180. To account for potential dropout owing to ineligibility, we aimed to include 200 patient scans.

Results

Study cohort characteristics

A total of 379 eligible patients undergoing a radiation treatment planning scan were included in our study cohort as detailed in Figure 1. Of those, 183 CT simulation scans (48%) were sent to the treating physician to contour OARs per standard of care, and 196 CT simulation scans (52%) were allocated to the intervention group for the radiation anatomist to contour OARs. In the SOC group, OAR contours were performed by 9 physicians across 2 regional clinics, with each physician treating a median of 18 patients (interquartile range [IQR], 14-23 patients) during the 6-month study period. A single radiation anatomist contoured the OARs for all patients in the intervention group. The list of disease sites most represented in the patient cohort included gastrointestinal (n = 63 [17%]), metastatic disease (n = 67 [18%]), head and neck (n = 53 [14%]), lung (n = 49 [13%]), breast (n = 46 [12%]), brain (n = 40 [11%]), and spine (n = 28 [7%]). Table 1 shows the patient characteristics in each group, with no significant differences in clinical location, disease sites treated, and radiation treatment techniques.

Table 1

Study cohort characteristics

Variable	Anatomist, No. (%) (n = 196)	MD, No. (%) (n = 183)	P value*
Treatment location			>.99
Clinical site 1	75 (38)	70 (38)
Clinical site 2	121 (62)	113 (62)
Disease site			.31
Brain	23 (12)	17 (9.3)
Breast	23 (12)	23 (13)
GI	40 (20)	23 (13)
H&N	21 (11)	32 (17)
Lung	23 (12)	26 (14)
Metastases	33 (17)	34 (19)
Other†	19 (9.7)	14 (7.7)
Spine	14 (7.1)	14 (7.7)
RT technique			.53
3D CRT	30 (15)	36 (20)
IMRT/VMAT	107 (55)	96 (52)
SRS/SBRT	59 (30)	51 (28)

Abbreviations:3D CRT = 3-dimensional conformal radiation therapy; GI = gastrointestinal; H&N = head and neck; IMRT = intensity modulated radiation therapy; Mets =;RT = radiation therapy; SBRT = stereotactic body radiation therapy; SRS = stereotactic radiosurgery; VMAT = volumetric modulated arc therapy.

χ2 test of independence.

Genitourinary, gynecological, sarcoma, lymphoma, and skin.

Contour quality scoring

Comparison between the physician and dosimetrist rating of OARs contoured by the radiation anatomist showed high interrater agreement probability (88%), suggesting physicians did not frequently adjust OARs generated by the radiation anatomist and that dosimetrist rating of OARs was a reasonable proxy to assess contour quality. A dosimetrist report of OAR contour quality was available for 249 patient scans, with response rates of 63% and 68% in the SOC and intervention groups, respectively. Of OAR contour sets in the intervention group, 87% received a quality score of 1 (acceptable), compared with 65% of OAR contour sets in the SOC group (Fig 1). For all disease sites combined, the mean OAR quality score was 1.1 ± 0.4 for the intervention group (anatomist) and 1.4 ± 0.7 for the SOC group (P < .001), indicating, on average, higher-quality OAR contouring by an anatomist than a physician Fig. 2. Subset analysis by disease site showed that OARs contoured by the radiation anatomist were rated higher in quality than OARs contoured by physicians for gastrointestinal cases (P < .001) (Table 2). On multivariable analysis, an ordinal logistic regression model controlling for radiation treatment technique and disease site estimated that OARs contoured by an MD were 3.9 times more likely to be rated either a 2 or 3, indicating lower OAR quality, compared with contours by the radiation anatomist (P < .001) (Table 3). Review of dosimetrist open-ended comments with scores of 2 and 3 suggested bowel was the most common structure requiring edits (n = 14). Variation in OAR contour quality was noted among physicians (see Fig. E1).

Figure 2

Organ-at-risk quality score frequency in the standard of care and radiation anatomist intervention groups.

Table 2

Subset analysis of OAR quality ratings between SOC and anatomist groups, by disease site

Disease site and study arm	OAR rating, No. (%)
	1	2	3	Mean	P value1
All OARs (n = 249)
MD	75 (65)	30 (26)	10 (9)	1.44	< .001*
Anatomist	117 (87)	15 (11)	2 (2)	1.14
Head and neck (n = 40)
MD	12 (50)	9 (38)	3 (13)	1.63	.17
Anatomist	13 (81)	3 (19)	0 (0)	1.19
Lung (n = 35)
MD	17 (100)	0	0	1.00	.10
Anatomist	14 (78)	4 (22)	0	1.22
GI (n = 49)
MD	10 (56)	7 (39)	1 (6)	1.50	< .001*
Anatomist	30 (97)	1 (3)	0	1.03
Mets (n = 44)
MD	16 (76)	3 (14)	2 (10)	1.33	.39
Anatomist	21 (91)	1 (4)	1 (4)	1.13
Spine (n = 23)
MD	5 (45)	4 (36)	2 (18)	1.73	.03
Anatomist	11 (92)	0	1 (8)	1.17

Abbreviations: OAR = organ at risk; SOC = standard of care.

Indicates statistical significance (P < .005) by Fisher exact test.

Table 3

Factors associated with OAR contour quality on multivariable ordinal logistic regression

Variable	Odds ratio	95% CI	P value
Contoured byradiation anatomistMD	[Reference]3.91	-1.87-6.73	<.001
Treatment technique* 3D RT Other	[Reference]0.41	-0.02-8.07	.09
Disease site† GI, H&N, and spine Other	[Reference]1.45	-0.77-2.71	.24

Abbreviations: 3D RT = 3-dimensional radiation therapy; CI = confidence interval; GI = gastrointestinal; H&N = head and neck; OAR = organ at risk.

Treatment technique is a dichotomized variable of 3D conformal or volumetric modulated arc therapy / intensity modulated radiation therapy and stereotactic radiosurgery/ stereotactic body radiation therapy.

Disease site is a dichotomized variable; “other” includes brain, breast, genitourinary, gynecological, lung, metastatic, sarcoma, lymphoma, and skin.

Time spent contouring

On average, physicians reported a median of 33 minutes (IQR, 25-45 minutes) spent contouring (including both targets and OARs) in the SOC group (n = 114) versus a median of 25 minutes (IQR, 18-30 minutes) spent contouring in the intervention group (n = 87), reflecting an 8-minute (17% relative) reduction (P < .001) (Fig 3). Similarly, dosimetrists in the intervention group (n = 39) reported a median of 5 minutes (IQR, 0-12 minutes) spent contouring versus 10 minutes (IQR, 5-15 minutes) in the SOC group (n = 48), for a 5-minute (50%) reduction (P = .002). For specific disease sites, the largest reduction in physician time spent contouring was observed for spine cases (primarily stereotactic body radiation), with physicians reporting a median time of 45 minutes in the SOC group and 25 minutes in the intervention group (n = 18; P = .02), for a reduction of 20 minutes (44%).

Figure 3

Time spent contouring by physician in the standard-of-care (without anatomist) and intervention (with anatomist) groups.

The average time to contour approval was 3 days (IQR, 1-6 days) for SOC versus 2 days (IQR, 1-5 days) for the intervention group, which is a significant reduction in time (P = .007).

Discussion

To our knowledge, this is the first study to describe and evaluate the implementation of a radiation anatomist for OAR contouring in routine radiation oncology practice, which confirmed improvements in both quality and efficiency in the treatment planning process. The role of the radiation anatomist, at minimum, appears to be to reduce OAR contour variation and reduce time to contour approval. During routine care, an anatomist provides physicians with accurate OAR contours, which decreases their overall time spent contouring, and as a result, reduces total time to contour approval. Physician time savings and shorter time to contour approval could have additional benefits of reducing physician burnout and improving patient experience, but this warrants further investigation. We provide considerations for the integration of an anatomist with autosegmentation efforts, because advancements are rapidly occurring in this space.

To comprehensively assess the quality of OAR contours in this study, we used a clinical trial protocol system for recording deviations. Previous data have shown that radiation therapy protocol deviations are an independent predictor of worse clinical outcomes, including an increased risk of treatment failure and overall mortality. Specifically, quality of bowel contours has been demonstrated to directly correlate with gastrointestinal toxic effects; a retrospective quality assurance analysis of the Radiation Oncology Group 0411 phase 2 study for locally advanced pancreatic cancer reported an increased incidence of grade 3 toxic effects for patients with radiation therapy protocol deviations compared with guideline-concordant plans. Gastrointestinal and spine cases showed the greatest benefit to having a radiation anatomist, most likely owing to the cumbersome and complex nature of bowel contouring. Although this study was underpowered to assess differences in less common disease sites such as gynecologic and genitourinary (owing to planned omission of the prostate), it seems reasonable to extrapolate these findings and focus radiation anatomist effort on these disease sites in the absence of useful autosegmentation tools.

Efforts to reduce physician time spent contouring are of high priority to improve efficiency in treatment planning, particularly as the burden of 3-dimensional and highly conformal treatment planning increases. Although most studies to date have evaluated autosegmentation, herein we evaluated manual segmentation by an anatomist to augment this approach as automated algorithms mature. We found that time savings with a radiation anatomist contouring OARs resulted in an average of 8 minutes per case (and up to 20 minutes for spine [mostly stereotactic body radiation] cases), for a total of 17% relative time savings, on average. This is comparable with 9 minutes noted in a prior randomized controlled study of OAR autosegmentation in head and neck cancer. At the time of the current study, our institution was testing autosegmentation for prostate-only radiation; thus, these patients were excluded. Our prostate study and a similar rectal cancer study ultimately showed an approximate 30% reduction in physician time savings. In our study, dosimetrists often reported bowel as an OAR that required editing (n = 14). This is corroborated by a systematic review evaluating 3 different commercial software solutions for autosegmentation that highlighted the rectum and bowel as volumes needing the most manual corrections by physicians. Although autosegmenting OARs has been shown to save time, gaps still exist for some disease sites and OARs. The current study helps identify areas of high priority for improving efficiency in the clinical workflow. Such technological work can be done alongside a radiation anatomist, as is now the practice at our institution.

Although it is evident that autosegmentation algorithms decrease contouring time, prior studies have not demonstrated significant improvements in overall efficiency in the planning process, as in the current study. A recent study quantifying the resources required for radiation pretreatment tasks highlighted increased workload in the past decade with the patient population increased by 45%, whereas the time required to complete these tasks increased by 150%. Meanwhile, the staffing levels only increased by 29% in the same period. Therefore, human resource use may be needed to meet the substantial demands associated with technological changes, and initiatives such as a radiation anatomist should therefore be considered. The cost-effectiveness of this program can be estimated from physician time savings (8 minutes per case) and radiation anatomist case volume (50 cases per week) for approximately a 0.2 physician full-time equivalent. Alternatively, time spent contouring OARs for just 3 complex cases (20 minutes per spine case) would equal the typical amount of time for 1 new patient consult. Importantly, reducing physician time spent contouring should facilitate more focus on meaningful clinical and academic activities, and evidence suggests that physician burnout is correlated with not operating at the top of one's license.

Data presented in this study are likely relevant to radiation oncology practices broadly, given the inclusion of all disease sites and the persistence of physician contouring of OARs in routine practice. A recent Twitter poll (n = 232) found that OARs are most often contoured by physicians (48%), followed by dosimetrists (42%), autosegmentation (6%), and least commonly, by anatomists (3%). In justifying the hiring of a new position, our department has considered additional opportunities for anatomists to contribute through adaptive planning, quality assurance practices (chart rounds and contour review), and professional education (including trainees and new physician and dosimetry staff). Additionally, radiation anatomist–generated OARs will likely improve quality of training data sets for autosegmentation algorithms development. In the absence of a radiation anatomist, formal training and guidance in OAR contouring for dosimetrists could be an alternate strategy to achieve consistency and efficiency in routine practice.

Limitations

This study has several limitations. First, an “every other” patient scan assignment is not technically a randomization, although it provides a pragmatic approach to identifying a comparison (control) group that accounts for variation over time and equally distributes the cases between the study groups. Second, the subjective quality scoring was performed by dosimetrists, which was selected for pragmatic purposes to allow for blinded review. Related, the presence of a 2-step editing process itself in the intervention group may inherently improve the quality of OAR contours. Nonetheless, interrater agreement was high between OAR ratings by physicians and dosimetrists when both were available in the intervention arm, reducing the likelihood that significant edits were made by the physician. Future work should evaluate dosimetric outcomes of contouring errors. Third, although self-reported time spent contouring may be subject to recall bias, it is a reliable pragmatic approach compared with a controlled timed setting., Fourth, these data represent the experience of a small group of physicians within a single academic institution with a single anatomist, potentially limiting generalizability. However, the study was conducted at 2 community-based regional practice clinic sites to ensure that distribution of the diseases treated was comparable to a typical radiation oncology practice. Finally, the study size potentially limited our power to detect disease-specific differences in quality and time savings in subset analyses. We decided a priori to analyze disease sites with 10 or more patients per study arm to reduce underpowered analyses and limit multiple testing.

Conclusions

This study quantified improvements in both quality and workflow efficiency with the implementation of a radiation anatomist for contouring OARs. This study took place in community-based practice sites in an academic network, making this information potentially applicable to routine radiation oncology practice. The greatest benefits of anatomist contouring were in bowel contouring, coinciding with evidence from clinical trial QA suggesting that the quality of bowel contours may affect clinical outcomes. In addition to immediate clinical workflow advantages, consistent radiation anatomist–generated OAR contours may facilitate development of useful and accurate autosegmentation algorithms and standardization for patients treated on trial. Removing the burden of OAR contouring could reduce physician burnout by allowing physicians to practice at their level of training, but this warrants further investigation. Additional opportunities for radiation anatomists may include educating radiation professionals in contouring complex OARs and peer review QA, specifically contour-specific chart rounds.