Conversion of a colorectal cancer guideline into clinical decision trees with assessment of validity.

OBJECTIVE: The interpretation and clinical application of guidelines can be challenging and time-consuming, which may result in noncompliance to guidelines. The aim of this study was to convert the Dutch guideline for colorectal cancer (CRC) into decision trees and subsequently implement decision trees in an online decision support environment to facilitate guideline application.
METHODS: The recommendations of the Dutch CRC guidelines (published in 2014) were translated into decision trees consisting of decision nodes, branches and leaves that represent data items, data item values and recommendations, respectively. Decision trees were discussed with experts in the field and published as interactive open access decision support software (available at www.oncoguide.nl). Decision tree validation and a concordance analysis were performed using consecutive reports (January 2016-January 2017) from CRC multidisciplinary tumour boards (MTBs) at Amsterdam University Medical Centers, location AMC.
RESULTS: In total, we developed 34 decision trees driven by 101 decision nodes based on the guideline recommendations. Decision trees represented recommendations for diagnostics (n = 1), staging (n = 10), primary treatment (colon: n = 1, rectum: n = 5, colorectal: n = 9), pathology (n = 4) and follow-up (n = 3) and included one overview decision tree for optimal navigation. We identified several guideline information gaps and areas of inconclusive evidence. A total of 158 patients' MTB reports were eligible for decision tree validation and resulted in treatment recommendations in 80% of cases. The concordance rate between decision tree treatment recommendations and MTB advices was 81%. Decision trees reported in 22 out of 24 non-concordant cases (92%) that no guideline recommendation was available.
CONCLUSIONS: We successfully converted the Dutch CRC guideline into decision trees and identified several information gaps and areas of inconclusive evidence, the latter being the main cause of the observed disagreement between decision tree recommendations and MTB advices. Decision trees may contribute to future strategies to optimize quality of care for CRC patients.

Introduction

Clinical practice guidelines are developed to facilitate evidence-based medicine, optimize quality of care and reduce unjustified variation in clinical practice [1, 2]. The number, length and complexity of available guidelines in oncology have rapidly increased over the past two decades [3], but providers’ adherence to recommendations in guidelines varies widely and is suboptimal [4]. Several factors have been identified that adversely impact adherence [5, 6].

Obviously, disease and patient-related factors influence the application of guideline recommendations, but guideline-related barriers such as guideline usability or lacking or contradictory information in guidelines may contribute to non-adherence [7-9]. Guidelines are historically text-based that make their reading and interpretation challenging and time-consuming and complicate application in clinical practice. Improving the usability of narrative guidelines may dissolve guideline-related barriers to guideline adherence [10]. Conversion of guidelines into decision trees that are suitable for implementation in an interactive decision support system may be a viable strategy to overcome this barrier and facilitate guideline interpretation [11-15].

Decision trees are algorithms structured by decision nodes, branches and leaves that represent data items, data values (representing different outcomes) and (treatment) recommendations, respectively. A recent study demonstrated the feasibility of transforming a complex multidisciplinary oncology guideline into systematically designed decision trees and their subsequent implementation in an interactive decision support application [11].

However, little is known about the entire process of development, validation, implementation and evaluation of decision trees in clinical practice. We believe that this process would benefit from strictly following these subsequent phases: (i) conversion of guideline into data-driven decision trees [11]; (ii) validation of decision trees using real patient cases; (iii) concordance analysis between treatment recommendations of decision trees versus multidisciplinary tumour board (MTB) advices; (iv) implementation of decision trees in electronic health records by using standardized forms; and (v) evaluation of decision tree application in clinical practice and their impact on guideline adherence.

For colorectal cancer (CRC), one of the most prevalent cancers, suboptimal adherence to guideline recommendations and substantial practice variation have been demonstrated [16-19]. Although several reasons may explain non-adherence, unawareness of treatment recommendations and the complexity of the guideline itself may have contributed. We therefore aim to convert the Dutch multidisciplinary CRC guideline [20] into decision trees and report on the first three phases of this process (conversion, validation and concordance). We hypothesize that translating the CRC guidelines into decision trees improves the awareness of variables needed for decision-making and hereby the applicability of guidelines in clinical practice, which will ultimately contribute to the optimization of quality of care for CRC patients.

Methods

Treatment of CRC

CRC is classified into non-metastatic (stages I–III) and metastatic disease (stage IV or metachronous metastases). In summary, treatment of patients with non-metastatic disease consists of neoadjuvant treatment for a minority of (mainly rectal cancer) patients and surgery of the primary tumour. A subgroup of colon cancer patients is eligible for adjuvant chemotherapy, based on tumour stage (high risk stage II and stage III) and microsatellite status. For patients with metastatic CRC, different treatment modalities such as systemic therapy (chemotherapy, targeted therapy and immunotherapy for a subgroup of patients) and local treatment of metastases are integrated depending on the resectability of metastases.

Conversion of CRC guidelines into data-driven decision trees

The Dutch CRC guideline [20], consisting of 238 pages, was converted into decision trees by the following approach. First, we extracted guideline recommendations from the 12 text-based chapters of the Dutch CRC guideline. Subsequently, we performed a backward evaluation for each guideline recommendation to create an overview of decision nodes (i.e. data items, e.g. pT stage) and branches (i.e. data values, e.g. pT1) that were required to result in a guideline recommendation.

A decision node consisted of a clinical (e.g. comorbidity), topographic (e.g. colon or rectum), pathologic (e.g. tumour differentiation grade) or molecular (e.g. mismatch repair status) feature. A branch originating from a decision node could lead to a subsequent decision node, a recommendation or a recommendation in combination with a link to another decision tree (e.g. recommendation: perform a total mesorectal excision, and a link to the following decision tree: pathologic staging after resection of the primary tumour). We decided to develop separate decision trees for colon and rectal cancers and for metastatic versus non-metastatic setting. In addition, CRC care was subdivided into diagnosis, staging, primary treatment, adjuvant treatment and follow-up.

Decision tree drafts were developed in Microsoft Visio (2016) in cooperation with a multidisciplinary panel consisting of delegates from the following departments: medical oncology, surgery, pathology, radiology, radiation oncology and specialists in clinical informatics. Ultimately, the developed decision trees were translated into open access decision support software (available at www.oncoguide.nl). In addition, text-based background information from the original guideline (e.g. considerations and scientific evidence behind treatment recommendations) was added to the decision nodes and leaves with recommendations.

Validation of decision trees with MTB reports

The study population consisted of CRC patients, who were discussed in MTBs at the Amsterdam University Medical Centers (location AMC, The Netherlands) between January 2016 and January 2017. Consecutive patient reports were collected in this timeframe and included reports on different phases of patients’ disease course: before primary treatment after diagnosis, adjuvant treatment, disease recurrence/progression after surgical and/or systemic treatment, or treatment options for metastatic disease. MTB reports of patients who were discussed multiple times (e.g. before primary treatment and adjuvant treatment after surgery) were included as separate cases.

Patient data, such as patient and disease characteristics, and conclusions and treatment recommendations, were retrospectively extracted from MTB reports, and were manually entered into a Case Report Form using Castor EDC version 1.4 (Castor Research Inc). Subsequently, individual Case Report Forms were used to run though the decision trees by using Oncoguide software (available at www.oncoguide.nl). We distinguished between MTB reports that successfully or unsuccessfully led to a decision tree recommendation. The reasons for an unsuccessful run through decision trees were registered.

Concordance analysis

We performed a concordance analysis with the subset of MTB reports that successfully resulted in a decision tree recommendation in the validation analysis. We compared treatment advices from original MTB reports with decision tree recommendations, which were considered concordant if treatment recommendations from both sources were consistent. Otherwise, treatment recommendations were labelled as non-concordant.

Statistical analysis

The number of developed decision trees were counted and classified accordingly: diagnostics, staging, primary treatment, pathology and follow-up. Descriptive statistics of patient data extracted from MTB reports were presented as frequencies with percentages. Percentages of MTB reports that successfully or unsuccessfully resulted in a decision tree treatment recommendation were calculated. Ultimately, concordance between successfully generated recommendations from decision trees and original MTB advices was presented by a percentage. Analyses were conducted using SPSS (version 25).

Results

Conversion of guideline into data-driven decision trees

A total of 34 separate decision trees were developed, driven by 101 decision nodes (i.e. data items). Decision trees focused on recommendations for diagnostics (n = 1), staging (n = 10), primary treatment (colon: n = 1, rectum: n = 5, both: n = 9), pathology (n = 4), follow-up (n = 3) and included one overview decision tree for optimal online navigation.

Several information gaps were identified during decision tree development, for example the treatment of metastatic CRC with ≥2 extrahepatic metastatic sites (Figure 1); indication for local treatment related to a specific number of lung metastases; follow-up schedule after non-surgical local treatment of metastases (e.g. radiofrequency ablation); time between neoadjuvant treatment and restaging; and background information on contraindications for specific systemic therapies (e.g. oxaliplatin). Information gaps were described in the decision trees to preserve agreement between decision trees and the original guideline (i.e. ‘no guideline recommendation is available’) (Figure 1). The same principle was applied to guideline recommendations in areas of inconclusive evidence (e.g. consider additional MRI liver, PET-CT and/or Spiral CT to rule out extrahepatic metastases) (Figure 2).

Figure 1

Details of a larger decision tree with information gap (treatment of metastatic colorectal cancer with ≥2 extrahepatic metastatic sites).

Figure 2

Representation of ‘Addition diagnostic work-up’ and ‘Primary treatment’ decision tree. Decision tree ‘Additional diagnostic work-up’ includes inconclusive information (consider additional MRI liver, PET-CT and/or Spiral CT to rule out extrahepatic metastases).

Decision tree validation and concordance analysis

A total of 178 MTB reports derived from 129 patients were initially included for analyses. MTB reports from 17 non-CRC patients were excluded. Three additional patients were excluded because no MTB recommendation was registered in the electronic health record, resulting in 158 MTB reports derived from 109 patients. The 158 MTB reports included 68 patients who were discussed once, 35 patients were discussed twice (70 MTB reports), 5 patients three times (15 MTB reports) and 1 patient was discussed five times (5 MTB reports). The baseline characteristics are presented in Table 1.

Table 1

Baseline characteristics of study population

		n (%)
Age (years)a	<50	13 (12)
	50–65	50 (46)
	>65	46 (42)
Gendera	Female	40 (37)
	Male	69 (63)
WHO performance scorea	0	78 (72)
	1	14 (13)
	2	2 (2)
	3	1 (1)
	Missing	14 (13)
Reason for MTB referralb	Primary diagnosis and initial treatment plan	77 (49)
	Re-staging after induction therapy with systemic regiments	2 (1)
	Postoperative restaging	49 (31)
	Diagnosis of metachronous metastases	13 (8)
	Response evaluation after systemic therapy for metastases	17 (11)
Tumour localizationa,c	Colon	71 (65)
	Rectum	39 (35)
TNM stagingb	Stage I	29 (18)
	Stage II	31 (20)
	Stage III	28 (18)
	Stage IV	40 (25)
	Locoregional recurrence	5 (3)
	Metachronous metastases	25 (16)

Based on 109 patients.

Based on 158 MTB reports derived from 109 patients.

Variable includes 110 cases; one patient had a tumour in both colon and rectum.

MTB = multidisciplinary tumour board.

Decision tree validation

Figure 3 shows the distribution of included MTB reports across the main branches of the decision tree. The ‘emergency treatment’ (e.g. presentation with obstruction or perforation), ‘post-neoadjuvant colon cancer’, and ‘post-neoadjuvant post-surgery’ branches were not covered by the included MTB reports.

Figure 3

Overview decision tree. Representation of included MTB reports and covering of different decision trees.

Overall, 127 out of 158 MTB reports (80%) successfully ran through the decision trees and resulted in a treatment recommendation compared to 31 MTB reports that unsuccessfully ran through the decision trees (Figure 4). Reasons for not reaching a treatment recommendations were decision tree-related (n = 24), due to missing data from electronic health records (n = 2) or other reasons (n = 5). Decision tree-related reasons included: missing values at decision nodes (n = 6), no subsequent branch for a specific value in a decision node (n = 5), no subsequent branch after a previous recommendation (n = 6), another inadequate loop to another decision tree (n = 5) or other (n = 2). All MTB reports from patients with stage II or III CRC except for one stage II patient, or locoregional recurrence, passed decision trees successfully, whereas MTB reports of patients with stage I or IV disease, or MTB reports of patients with metachronous metastases failed in 34%, 40% and 16%, respectively (Figure 4).

Figure 4

Overview of validation and concordance analyses.

A total of 127 out of 158 MTB reports resulted in decision tree recommendations in the validation analysis. The majority of treatment recommendations from these reports (103 out of 127) were in accordance with MTB advices, resulting in a concordance rate of 81%. Decision trees reported in 22 out of 24 non-concordant cases (92%) that no guideline recommendation was available. This was predominantly found in MTB reports concerning stage IV disease (33%), locoregional recurrence (100%), or cases with metachronous metastases (38%) (Figure 4). The remaining two non-concordant recommendations included cases where radiotherapy (5x5Gy) followed by systemic therapy for metastatic rectal cancer was recommended by the MTB, while only systemic therapy was recommended by the decision trees.

Discussion

Statement of principal findings

We successfully converted the Dutch CRC guideline into decision trees that were published in open access interactive online decision support software (available at www.oncoguide.nl). Decision tree validation revealed that several challenges persist, as not all patient cases that were run through the decision trees led to a treatment recommendation (success rate: 80%), although this was particularly due to information gaps in the guideline. The concordance rate of recommendations between successfully generated decision tree recommendations and MTB advices concerned 81%. Overall, the concept of decision trees offers many opportunities for future strategies to optimize quality of care for CRC patients.

Strengths and limitations

This is the first study that focused on translation of guideline recommendations into decision trees and included a validity and concordance analysis. This concept is applicable to other (oncological) guidelines. We experienced several challenges during decision tree development. A potential risk of converting text-based guideline recommendations and considerations into decision trees is losing nuance. We addressed this issue by describing information gaps, inconclusive treatment recommendations and guideline considerations below the guideline recommendations in the decision trees, although this may hamper quick interpretation.

A practical difficulty of current decision trees on the Oncoguide platform is the manual input of cases, which is both sensitive to errors and time-consuming. Future integration of decision trees with patient data derived from electronic health records seems essential. The development of a decision support system that is interoperable with electronic health records requires standardized terminology and synoptic reporting [14, 15].

MTB reports from the Amsterdam University Medical Centers, location AMC, were included for decision tree validation. This hospital provides tertiary care, which means that a substantial number of patients are referred from other (regional or extra-regional) hospitals. This may have contributed to an imbalance in MTB reports with complex cases and potential selection bias as a result. The use of MTB reports from different hospitals would have resulted in a better reflection of the CRC population. Therefore, validation and concordance rates are possibly underestimated in the current patient population due to lower success rates as a result from the complexity of cases. Moreover, we evaluated concordance by comparing decision tree recommendations with MTB advices, but MTB advices could also be non-adherent to CRC guidelines. Although non-concordant cases should not automatically be interpreted as decision tree errors, the concordance analysis with MTB advices resulted in a critical appraisal of the content of decision tree recommendations. MTB advices were used as a reference value, as discussing cancer patients in MTBs is generally considered best practice and recommended in (inter)national guidelines. This strategy for the evaluation of decision support systems has also been used in previous studies [21, 22].

The success rate of decision tree treatment recommendations was 80%. Dropout of MTB reports (20% in total) was substantial in cases with stage I and stage IV disease (34% and 40%). Treatment recommendations could be generated in 98% of MTB reports with stages II and III CRC, which implicates that the current decision trees are particularly applicable for more straightforward cases. However, decision trees could assist in the identification of cases that require a more consensus-based approach if guideline recommendations are lacking or scientific evidence is inconclusive. MTB discussions will remain leading in such cases.

Last, dependency of data should be considered in our analyses, because multiple MTB reports from a single patient were included as separate cases, resulting in non-independent observations. However, the variables needed for the different decision trees (e.g. for the ‘post-surgery’ (mainly pathology items) and ‘primary diagnosis and treatment’ decision trees) do not, or hardly, overlap. We therefore suggest that if bias has occurred, it is limited.

Interpretation within the context of the wider literature

The concept of decision tree algorithms and computer interpretable guidelines has been widely described, and several initiatives have explored the concept of guidelines based on decision trees [12, 13, 23]. An important challenge relies in the requirement of decision trees that are suitable for both clinical practice and implementation in a decision support system [11, 24]. A previous study described the translation of the Dutch guideline for non-metastatic breast cancer into decision trees, but this study did not include the (usually more complex) metastatic part of the guideline and no validation or concordance analysis [11].

Implications for policy, practice and research

The concept and future use of decision trees offer new strategies to improve quality of care. First, decision trees that are integrated during multidisciplinary tumour boards contribute to awareness of determinants required for decision-making, which may improve guideline adherence [24]. Evaluation of non-adherent cases that have been run through decision trees provide insight into reasons for guideline deviation. Second, registration and follow-up of cases facilitate learning from clinical cases that may improve tailored treatment of future CRC patients. Third, the Oncoguide platform may facilitate data collection for cancer registries if a link between the decision trees and registries is generated. Fourth, decision trees provide insight into information gaps in the guideline, which may offer opportunities for clinical trials, and the Oncoguide platform may improve patient participation in clinical trials if eligibility criteria of clinical trials are linked to the decision trees. Last, the inherent modular character of decision trees fits the current modular revision process that has been implemented for an increasing number of guidelines in Dutch oncology care and may even facilitate in the identification and delineation of relevant modules. Moreover, decision trees could assist in prioritizing the update of certain guideline modules based on revealed information gaps. All implications described above may ultimately contribute to improve quality of care for CRC patients.

The usability of decision trees in clinical practice is subject to future investigations. We are currently planning pilot studies for the clinical applicability of the decision trees. Decision trees may assist in (i) the consideration whether a patient should be discussed in a local MTB or should be referred to a more centralized MTB; (ii) which medical disciplines should be present for discussion and (iii) whether difficult discussion is expected related to the complexity of the case. In other words, we would like to study whether decision trees are able to differentiate between complex treatment decisions and more straightforward cases, hereby offering opportunities for optimizing patient planning in MTBs.

Conclusions

Conversion of the CRC guideline into decision trees was feasible and resulted in the development and publication of open access software of decision tree–based guidelines. Future studies must confirm the usability of the tool and explore the quality improvement opportunities of decision trees that should ultimately lead to the optimization of quality of care for CRC patients.

Click here for additional data file.