FIELDRT: an open-source platform for the assessment of target volume delineation in radiation therapy.

OBJECTIVES: Target volume delineation (TVD) has been identified as a weakness in the accuracy of radiotherapy, both within and outside of clinical trials due to the intra/interobserver variations affecting the TVD quality. Sources of variations such as poor compliance or protocol violation may have adverse effect on treatment outcomes. In this paper, we present and describe the FIELDRT software developed for the ARENA project to improve the quality of TVD through qualitative and quantitative feedbacks and individual and personalized summary of trainee"s performance.
METHODS: For each site-specific clinical case included in the FIELDRT software, reference volumes, minimum and maximum "acceptable" volumes and organ at risk were derived by outlines of consultants and senior trainees. The software components currently developed include: (a) user-friendly importing interface (b) analysis toolbox to compute quantitative and qualitative (c) visualiser and (d) structured report generator for personalised feedback. The FIELDRT software was validated by comparing the performance of 63 trainees and by measuring performance over time. In addition, a trainee evaluation day was held in 2019 to collect feedback on FIELDRT.
RESULTS: Results show the trainees' improvement when reoutlining a case after reviewing the feedback generated from the FIELDRT software. Comments and feedback received after evaluation day were positive and confirmed that FIELDRT can be a useful application for training purposes.
CONCLUSION: We presented a new open-source software to support education in TVD and ongoing continuous professional development for clinical oncology trainees and consultants. ARENA in combination with FIELDRT implements site-specific modules with reference target and organs at risk volumes and automatically evaluates individual performance using several quantitative and qualitative feedbacks. Pilot results suggests this software could be used as an education tool to reduce variation in TVD so to guarantee high quality in radiotherapy. ADVANCES IN KNOWLEDGE: FIELDRT is a new easy and free to use software aiming at supporting education in TVD and ongoing continuous professional development. The software provides quantitative/qualitative feedback and an exportable report with an individual and personalised summary of trainee's performance.

Introduction

Radiotherapy (RT) uses ionising radiation to kill cancer cells with the aim of cure or effective palliation and is used in the treatment of over 50% of all patients with cancer.[1] Current practice in RT requires target volume delineation (TVD) usually on a planning CT by a clinician or non-medical outliner (NMO). Macroscopic tumour (defined by the relevant diagnostic investigations) is manually delineated as the “gross tumour volume” (GTV). This is expanded to create the clinical target volume (CTV) to encompass areas of possible microscopic spread surrounding the GTV, and for some tumour sites may include nodal regions at risk. A volumetric expansion is applied to the CTV to create the planning target volume (PTV) to allow for potential patient set-up discrepancies and organ motion. Organs at risk (OARs) relevant to the particular tumour site, which, if receiving a significant RT dose may lead to significant toxicity for the patient and risk of long-term complications respectively,[2] are also delineated to allow dose to be calculated.

The accuracy of TVD may potentially adversely impact patient outcome through both an effect on normal tissue toxicity and tumour control. “Overdelineation” of target volumes may lead to unnecessary normal issue toxicity whereas “underdelineation” may lead to geographical miss of the target regions.[3,4] Failure to adhere to the protocol has been shown to affect the outcome in several trials and although the focus has historically been on the treatment planning aspects of the protocol, recent interest has focused more on the TVD.[5-8] TVD is an essential step within the RT planning pathway. However, it has been identified as potentially the “weakest link” because it may be affected by intra/interobserver variations.[9]

This variation has been attributed to quality of imaging to assist in TVD, the extent and quality of training in TVD and personal bias.[10,11] To minimise intra/interobserver TVD variation, a number of interventions have been employed.[12,13] One example includes access to a trial protocol and an outlining atlas, which has shown to improve consistency in RT outlining in prostate cancer[14] and rectal cancer,[15] respectively. However, one study in lung TVD demonstrated considerable TVD variation despite a protocol guidance due to participating clinicians reverting to pre-protocol practice. Educational sessions or workshops are another source of TVD training. They have been shown to reduce TVD variation in a range of settings, such as prostate[16] and lung cancer.[17] Examples includes the fellowship in anatomic delineation and contouring educational project (FALCON)[18] developed by the European society for radiotherapy and oncology (ESTRO) which aimed to promote e-learning activities and teaching in addition to providing a platform for contouring workshops[19] organised annually at the ESTRO congress and in the UK, the Royal College of radiologists has also organised a series of outlining workshops using the AQUILAB software (AQUILAB, Loos, France). However, these sessions carry limitations including small number of participants due to logistical reasons and costs[20] and the omission of/inability to finish contouring exercises due to time constraints.[21]

TVD is a craft and developing these core skills at a training level will inevitably influence future practice and habits. TVD teaching is therefore a key component in clinical/ radiation oncology training. However, in the UK, inadequate RT training is cited as the main training concern for clinical oncology trainees, with only 78 and 61% considering themselves competent in palliative and radical RT planning, respectively.[22] This is attributed to a loss of protected TVD time to other clinical duties and lack of feedback on from consultant supervisors.

Given the potential variation of RT training for trainees across tumour sites and training centres, our group established the ARENA (Assurances in Radiotherapy through EducatioN and Assessment) project in 2017 with the aim of standardising high-quality TVD training for UK clinical oncology trainees, based on experience acquired from RT trials quality assurance.[23] To determine the preferred format for these training packages, in 2018 we surveyed 131 UK clinical oncology trainees across all training grades regarding TVD training quality and preferential format for TVD modules.[24] This survey highlighted the fact that self-directed learning remains one of the most common methods of TVD training, and that most trainees would value a tool supplementing consultant-led TVD teaching in the form of site-specific TVD modules. In response to this survey, the ARENA project team has developed educational packages that comprise of site-specific introductory TVD modules detailing outlining instructions with accompanying interactive cases that can be hosted within appropriate software. Following delineation and submission of the case, semi-automated feedback is provided on outlining performance. In this paper, we present and describe the Feedback on Individual Education in anatomicaL Delineation in RadioTherapy (FIELDRT) software developed for the project.

Methods and materials

Site-specific clinical case development

Given the expertise of the ARENA project team, prostate and oesophagus were identified as pilot tumour sites followed by lung, head and neck, rectum and breast. Step-by-step guidance for TVD in a power-point format, based on relevant UK RT trial protocols, were developed. For example, the oesophageal cancer TVD guidance was adapted from the UK SCOPE 2 RT trial.[25]. A number of UK prostate RT trial protocols were adapted to create TVD guidance for the prostate cancer case. A clinical vignette was written for each case including anonymised patient and diagnostic information, accompanied by PET-CT and MRI images. The process was then repeated for the other tumour sites, with site-specific leads being identified.

Reference volumes

For each case, up to six UK clinical oncologists comprising of both site-specific consultants and senior trainees (post Fellowship of the Royal College of Radiologists (FRCR), developing a subspeciality interest in the respective tumour site) delineated the relevant case target volumes on the clinical cases identified (Table 1), following the delineation guidance in the TVD module. Before importing the pre-defined reference cases in the FIELDRT software, all outlines were processed using CERR[26,27] (Computational Environment for Radiological Research), an open-source platform custom built for the purpose of analysing and sharing RT data. Using the same approach adopted by the Radiation Therapy Oncology Group (RTOG) when developing contouring atlases for prostate[28] and rectal cancer,[29] a single contour was created from the constituent outlines using the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm available in CERR with a 95% confidence.[30] As with the RTOG groups, as a final step, this contour was reviewed again by the site-specific lead consultant for the respective tumour site and agreed to be the reference volume, hereafter referred to as gold -standard (GS).

Table 1.

Tumour sites with the related volumes and OARs outlined by a team of experts and processed to populate the package of reference clinical sites available in FIELDRT.

Tumour site	Volumes	OARs
Oesophagus	GTVCTVACTVBCTVCPTV	AortaAzygous veinLeft lungLeft main bronchusLiverPericardium/great vesselsRight lungStomachVertebra
Prostate	CTVpCTVpsv	BladderBowelLeft femoral headPenile bulbRectumRight femoral head

CTV, clinical target volume; OAR, organ at risk; PTV, planning target volume.

Minimum and maximum “acceptable” volumes

Given the variation in TVD even among experts, along with the difficulties in defining the “ground truth”, we also developed a minimum and maximum “acceptable” volume for the relevant target outlines made up of the constituent outlines as shown in S1 Fig. These were edited by the site-specific leads, omitting “incorrect” and “nonsense” volumes to create a final minimum and maximum “acceptable” volume, with the intention that a trainee contour is within this range would be considered to be clinically acceptable.

Organs at risk

On the basis of feedback from the UK trainee survey that qualitative feedback was most useful for training,[31] we wanted to include descriptive feedback on outlining performance, within the limitations of an “offline” feedback platform. Due to the critical nature of the incorrect inclusion/extension to surrounding OARs, we developed a “flag” tool within our software, based on pre-outlined OARs for each case (Table 1). “Dummy” OARs were created by reducing the OAR contours by 3 mm circumferentially.[32] If a trainee contour extended beyond this 3 mm limit this was considered unacceptable and would be included in the feedback report.

Architecture and workflow of FIELDRT software

In order to provide a platform for the interactive cases to be analysed, we developed software components including the importing interface, analysis computation toolbox, visualiser and report generator. Figure 1 shows the workflow and the internal architecture of the FIELDRT software.

Figure 1.

Workflow and internal architecture of the FIELDRT software. User’s volumes in DICOM format are exported from the outlining software choosen by the user and imported and processed in the FIELDRT software. Quantitative/qualitative feedback is provided and converted to an exportable document with a personalised summary of the performance.

Importing interface

The FIELDRT software uses the same routine and file format as CERR to import and store RT treatment planning and contour data to be compared with the corresponding reference volumes for each clinical case. The software also includes a module that keeps track of previous attempts. This allows the user to directly view their analysed performances using either the visualizer or personalised performance report without the need to re-compute quantitative and qualitative analyses. Figure 2 shows the FIELDRT importing interface to upload and analyse user’s volumes in DICOM format.

Figure 2.

FIELDRT importing interface to select the clinical site to use as a reference for the analysis and to import the user’s volumes in DICOM format. The FIELDRT logo at the top left with blue and red contours representing the colour-coded feedback provided by the software.

Analysis toolbox

To assess the user’s performance a range of quantitative and qualitative evaluations are computed to assess the comparison between their contours and the expert contours. The analysis includes operations such as intersection, union and difference between the user volumes and the corresponding reference volume as shown in S2 Fig. Any overlap between the user volumes and the OARs is also detected and reported. In addition, a comprehensive quantitative analysis of the user volumes is performed in terms of 2D (slice by slice) and 3D (whole volume) comparison against the reference volume using the Jaccard index. The Jaccard similarity index (also known as Jaccard similarity coefficient) compares two sets or volumes to determine the extent of their overlap with a range from 0% (two distinct and separate volumes) to 100% (volumes completely overlapping) and is calculated by:

where and are the intersection and union of the user’s volume with the corresponding reference volume (GS). A more detailed description of Jaccard and other measures of conformity be found in the work Gwynne et al[33].

Visualiser

The interface of the FIELDRT visualiser shows the CT planning scan and target volumes in different anatomical planes and related quantitative and qualitative feedback. Qualitative evaluations of the FIELDRT visualiser include a visual display of:

user’s and reference volumes with maximum/minimum acceptable contours (Figure 3a);

Figure 3.

Qualitative and quantitative feedback of the user’s performance in the FIELDRT software: user’s and GS volumes with maximum/minimum acceptable outlining areas (a), user’s volumes and OARs with a 'red flag' for inappropriately contoured volumes including encroachment into OAR (b) and user’s and GS volumes with (c) or without (d) over/undercontoured areas highlighted. OAR, organ at risk.

user’s volumes and OARs with a 'red flag' for inappropriately contoured volumes including unacceptable extension into OAR (Figure 3b);

over- and undercontoured areas defined as regions present and absent in the GS reference volume, respectively (Figure 3c);

user’s and reference volumes (Figure 3d);

For each user’s contours, the volume and 2D/3D Jaccard conformity indexes are also provided and displayed as quantitative feedback (Figure 3c and d).

Individual report

The FIELDRT report generator processes both quantitative and qualitative feedback to create an exportable document with an individual summary of the user’s performance. The report includes flagged slices in which performance is below a threshold (2D Jaccard <0.50) or includes inappropriately contoured volumes that excessively overlap with OARs (S3 Fig). In addition, a table containing quantitative feedback (Figure 4a) and a colour-coded graph of the local performance of the user (Figure 4b) are included in the report. Whole volume and slice by slice JCIs are colour-coded with in red, yellow and green corresponding to values of <0.50, 0.50–0.75 and >0.75, respectively. These predetermined levels were set based on previous work undertaken by the group.[34]

Figure 4.

Personalised summary of quantitative feedback of the trainee’s performance with whole volume (a) and slice by slice (b) indexes highlighted in red, yellow and green if Jaccard value is <0.50, 0.50–0.75 and >0.75, respectively. Slices not included in the reference volume are highlighted in dark grey.

Additional measures of performance

In addition to the analyses discussed above, we also compared the performance of a candidate against both their own and their peers’ performance over time. To develop such features, we analysed the mean 3D Jaccard values computed by the FIELDRT software when processing 63 oesophagus and prostate cancer contours submitted by national and international trainees who attended a FRCR part 2B preparation course in Cardiff in September 2019.

Results

Validation of FIELDRT

The FIELDRT software successfully analysed and generated personalised quantitative and qualitative feedback of the oesophageal and prostate volumes outlined by the 63 candidates. The processing time required to import each set of user volumes, analyse them and produce the exportable document with a summary of the user’s performance was approximately 4 min on a conventional machine (Intel Core e5-1620, 32 GB RAM, 3.50 GHz). The table containing quantitative feedback of the slice by slice and whole volume performance of the trainees are shown in Figure 5. The mean values for volume and 3D Jaccard obtained by each trainee for the oesophageal and prostate cases are showed in S1 and S2 Tables.

Figure 5.

Summary of the quantitative feedback of all 63 trainees’ performance for oesophagus (a) prostate (b) with slice by slice and whole volume and global indexes highlighted in red, yellow and green if Jaccard value is <0.50, 0.50–0.75 and >0.75. Slices not included in the reference volume are highlighted in dark grey. CTV, clinical target volume; GTV, gross tumour volume; PTV, planning target volume.

To perform a longitudinal validation of the FIELDRT software, three randomly selected candidates were requested to reoutline the same oesophagus case after reviewing the quantitative and qualitative feedback generated from the FIELDRT software at the end of each outlining session. A further three randomly selected candidates were asked to do the same for the prostate case. For the oesophagus case, the overall mean percent change of 3D Jaccard of all attempts from baseline (first attempt) for the three candidates was +10.24%, +17.61% and +22.17%, respectively. For the prostate case, a mean percent change of 3D Jaccard of +33.84%, +33.95% and +4.86% was observed for the three candidates, respectively. The percent changes for each candidate’s attempt are showed in S3 and S4 Tables. The improvement in the performance based on whole volume assessment is showed in Figure 6. For the two oesophagus and prostate cases, variations of 2D Jaccard index for slices in the reference range and the offset in the selection of the lower and upper slices outlined by the trainees with respect to the GS are provided in S4, S5 and S6 Figs.

Figure 6.

Improvement of six trainees when reoutlining the same case after reviewing the quantitative and qualitative feedback generated from the FIELDRT software. For each attempt, the mean 3D Jaccard value computed by the FIELDRT software for each volume is reported.

Evaluation of FIELDRT

As part of the ongoing development of the material for the ARENA project and the FIELDRT software, we held a trainee evaluation day in 2019 with seven local trainees, who were asked to outline the oesophagus and prostate cases in advance of the day. The cases were then analysed in real time and the trainees asked to provide a score from 1 to 10 and comments for different components present in the FIELDRT software. The overall score, expressed as mean ± standard deviation (SD), was 8.71 ± 1.70, 8.77 ± 0.25 and 8.57 ± 0.74 for the importing interface, the visualiser and the individual report, respectively. A table with the specific scores for each tool present in the FIELDRT software is provided in Supplementary Material 1 (S5 Table). The comments provided by the trainees were very positive. All seven trainees found the interface very easy to use and navigate, commenting on software that was “self-explanatory and simple”. Five out of seven found it useful to have their contour and the volume on the same screen and liked the minimum/maximum “acceptable” volume tool, enhancing understanding of the fact that there can be a range of acceptable volumes. Highlighting the areas where there was unacceptable extension into OARs was appreciated, and trainees stated they would bear this in mind when attempting future outlining. All trainees (seven out of seven) appreciated the quantitative assessment of their performance to assess the conformity of their outlining and valued the PDF report, commenting that it provided good background information on conformity indices and could serve as an essential part of their RT training logs. All the seven trainees appreciated being informed of areas of concern where conformity was low and found it useful being able to see screenshots of their contours.

Click here for additional data file.

Discussion

The ARENA project, using FIELDRT software, aims to facilitate a standardised approach for TVD training. The key features of the FIELDRT software include: (a) training packages which include a site-specific RT TVD document and corresponding clinical cases for users to practice TVD; (b) a user-friendly platform and interface to import and upload attempts at TVD which are automatically analysed and assessed; (c) image analysis package to compute quantitative (i.e. conformity metrics such as volumes, 2D/3D Jaccard) and qualitative evaluation of the user’s performance (user outline vs reference volume, maximum and minimum acceptable volumes, over- and undercontoured regions and a “red flag” for volumes inappropriately including OARs); (d) a user-friendly viewer to display CT planning images and contoured structures in axial, coronal, and sagittal planes; (e) a structured report generator that produces a personalised summary of the quantitative and qualitative feedback of the user’s performance.

The base FIELDRT package (i.e. importing interface, analysis computation toolbox, visualiser, report generator) with a prostate case (including the expert contours and dummy OARs) is freely available at the following address https://githubcom/concettapiazzese/FIELDRT-GitHub. In addition to the desktop version that requires a MATLAB license, the FIELDRT software is also available as a standalone executable that includes all the required runtime libraries.

The main differences between the FIELDRT software and other contouring software (i.e. eContour,[35] EduCase,[36] WorkflowBox,[37] ProKnow[38] and other contouring tools[39] are the following: (a) FIELDRT software and the cases are opensource and freely accessible (the software is available for both Macintosh and Windows operating systems), (b) new educational material and cases can be created by any user and uploaded into the software, (c) the quantitative and qualitative feedback created by software are saved locally and can reviewed within the visualizer, (d) an individual report is generated as an exportable document with some unique features for qualitative and quantitative feedback (i.e. colour-coded graph of the local performance of the user, flagged slices in which performance is below a threshold or inappropriately contoured volumes that excessively overlap with OARs) and (e) FIELDRT software doesn’t include any contouring tools as we decided at the start of the project to not replicate the high quality treatment planning systems already used in clinical practice and with which trainees and consultants are more familiar.

From the work achieved so far, we have demonstrated that this software is a platform that is able to successfully upload, analyse and provide useful qualitative and quantitative analysis on TVD performance for training purposes. Experience and feedback received so far show that the FIELDRT software can be a useful application to prepare training material, evaluate trainees’ delineations and provide useful qualitative and quantitated of information on TVD performance for training purposes.

Conclusions

We developed a software to support education in TVD and ongoing continuous professional development for clinical oncology trainees and consultants. The ARENA project in combination with the FIELDRT software implements site-specific modules with reference target and OARs volumes and automatically evaluates individual performance using several quantitative and qualitative feedbacks. Pilot results and quantitative improvement of trainee performance suggest this software could be used as an education tool to reduce variation in TVD so to improve the quality of RT. The availability of the FIELDRT software as an opensource software will enable the wider RT community to create their own educational material and cases to be hosted locally or shared with others to create a large library of educational material in TVD.

ARENA project, using the FIELDRT software, aims at supporting TVD education and ongoing continuous professional development.

The software is easy and free to use, and it quickly and automatically evaluates the trainee’s performance.

Quantitative/qualitative feedback is provided through the user-friendly viewer.

An exportable report with an individual and personalised summary of trainee’s performance is generated.