Conventional regression analysis and machine learning in prediction of anastomotic leakage and pulmonary complications after esophagogastric cancer surgery.

BACKGROUND AND OBJECTIVES: With the current advanced data-driven approach to health care, machine learning is gaining more interest. The current study investigates the added value of machine learning to linear regression in predicting anastomotic leakage and pulmonary complications after upper gastrointestinal cancer surgery.
METHODS: All patients in the Dutch Upper Gastrointestinal Cancer Audit undergoing curatively intended esophageal or gastric cancer surgeries from 2011 to 2017 were included. Anastomotic leakage was defined as any clinically or radiologically proven anastomotic leakage. Pulmonary complications entailed: pneumonia, pleural effusion, respiratory failure, pneumothorax, and/or acute respiratory distress syndrome. Different machine learning models were tested. Nomograms were constructed using Least Absolute Shrinkage and Selection Operator.
RESULTS: Between 2011 and 2017, 4228 patients underwent surgical resection for esophageal cancer, of which 18% developed anastomotic leakage and 30% a pulmonary complication. Of the 2199 patients with surgical resection for gastric cancer, 7% developed anastomotic leakage and 15% a pulmonary complication. In all cases, linear regression had the highest predictive value with the area under the curves varying between 61.9 and 68.0, but the difference with machine learning models did not reach statistical significance.
CONCLUSION: Machine learning models can predict postoperative complications in upper gastrointestinal cancer surgery, but they do not outperform the current gold standard, linear regression.

acute respiratory distress syndrome

American Society of Anesthesiologists

area under the curve

confidence interval

Dutch Upper Gastrointestinal Cancer Audit

enhanced recovery after surgery

k‐Nearest Neighbors

Least Absolute Shrinkage and Selection Operator

receiver‐operating characteristics

Support Vector Machine

INTRODUCTION

The incidence of esophageal cancer in the western world has increased over the past decades and is currently the seventh most common malignancy worldwide and accounts for 5% of the cancer‐related mortality in 2018. Although the incidence of gastric cancer decreased over the last years, it is still the fifth most common malignancy worldwide and was responsible for 8% of the cancer‐related mortality in 2018. Curative treatment of these upper gastrointestinal cancers consists in most cases of (neo)adjuvant therapy and surgical resection. These resections are complex procedures. Present‐day, the 5‐year survival rates of resectable esophageal and gastric carcinomas lie around 28%–42%. Although in centers of excellence, postoperative mortality is around 2%, the overall complication rate of around 60%–65% after esophagectomy is high compared to most procedures for gastrointestinal malignancies. Of all severe postoperative complications, anastomotic leakage and pulmonary complications are the most common. , , , The incidence of major complications (Clavien–Dindo ≥ IIIa) is 20%–31%, with a failure‐to‐rescue rate of 13%–25%. , Postoperative complications are associated with higher tumor recurrence and lower overall (cancer‐related) survival. Reduction of postoperative complications will enhance recovery, lead to fewer readmissions, and may increase long‐term quality of life.

With the present increase of data‐driven approaches in healthcare, preoperative risk factors can be appraised by analyzing large data sets. Machine learning holds the potential to unravel subtle associations that are not—or cannot be—identified using conventional regression analyses. In the current literature, no consensus exists on the added value of machine learning in predicting postoperative outcomes. ,

The aim of this study is twofold. First, to investigate the added value of machine learning methods in predicting postoperative outcomes after esophageal and gastric carcinoma surgeries and compare it to conventional regression analyses. Second, to use the best‐performing method to develop a predictive model for anastomotic leakage and cardiopulmonary complications after esophagectomy and gastrectomy.

METHODS

Data source and study population

Data were retrieved from the Dutch Upper GI Cancer Audit (DUCA). A prospective nationwide audit, initiated in 2011, containing all patients undergoing surgery with the intention of resection for esophageal or gastric cancers in the Netherlands. Participation in the DUCA has been incorporated as a mandatory quality standard, leading to data completeness of 99.8% and accuracy of 94%–100%. Validation of completeness and accuracy of this data registration has been performed by external data verification. All Dutch hospitals register detailed patient, tumor, and treatment characteristics, pathology, 30‐day morbidity, and 30‐day/in‐hospital mortality.

Patient selection

Patients who underwent elective surgery with curative intent for primary esophageal or gastric cancers were selected. Only patients with histologically proven adenocarcinoma or squamous cell carcinoma, a known surgery date between 2011 and 2017 and a recorded surgical technique were included. Patients with surgery with palliative or prophylactic intent and patients with nonepithelial tumors were excluded. Additionally, patients with missing essential values (age, sex, length, weight, surgical approach, American Society of Anesthesiologists (ASA) score, preoperative therapy and tumour, node and metastasis stage) were excluded.

Definitions of complications

The studied postoperative outcomes were anastomotic leakage and pulmonary complications in patients with esophageal carcinoma, and anastomotic leakage in patients with gastric carcinoma. Anastomotic leakage was defined as any clinically or radiologically proven anastomotic leakage. Pulmonary complications entailed: pneumonia, pleural effusion, respiratory failure, pneumothorax and/or acute respiratory distress syndrome (ARDS).

Statistical analysis

For each outcome, the data set was randomly divided into training (75%) and testing (25%) data. All models used variables documented in the DUCA, which covers patient characteristics, comorbidity, treatment characteristics, and outcome ; a total of 28 prognostic variables were included. The following machine learning models, which are frequently described in the literature, were used: k‐Nearest Neighbors (KNN), Support Vector Machine (SVM), Neural Networks Random Forest, AdaBoost, and SuperLearner. , , These models were compared with linear regression, for which a generalized linear model (GLM) was used. Background information on the models used can be found in Table 1. Afterward, nomograms were constructed using a regression model fit. The predictive strength of the models was measured by the area under the receiver‐operating characteristics (ROC) curve (AUC). Odds ratios with 95% confidence intervals and p values were reported for each variable to assess the impact on the risk of all patient characteristics. All analyses were done using R version 3.6.1 in RStudio. The Caret packages were used for pipelining and data splitting. ROC curves and AUC scores were calculated using the pROC package, plots were made using the ggplot2 package. The rms package was used to make the nomograms.

Table 1

Explanation of the models used.

Logistic regression

Describes the relationship between a discrete binary outcome and one or several predictor variables. The outcome is expressed as the log odds of one class over the other. This can be transformed into odds or probabilities.

Lasso regression

The difference between the Logistic regression model and the Lasso model is that the Lasso model can exclude coefficients that have little weight in the solution. This may increase interpretability.

k‐Nearest Neighbors (KNN)

Predicts new instances of a class by looking at k other instances in the neighborhood. The predictor variables are transformed by centering and scaling to improve numerical stability. For each outcome a separate KNN model is fit.

Neural Networks (NNs)

The inspiration for NN comes from the architecture of the human brain. The idea is that artificial neurons send the next neuron a signal based on the input they are receiving. A network of artificial neurons is called a NN. A NN consists of layers. The first is an input layer (the predictor variables), followed by one or more hidden layers (the artificial neurons) and finally resulting in an output layer (the prediction). For each outcome in the data, a NN is fit.

Support Vector Machine (SVM)

A classification (and regression) algorithm that can classify nonlinearly separable data by constructing a hyperplane (or a set of hyperplanes) in high dimensional space. An SVM tries to find a hyperplane that best separates two groups. This is the hyperplane whose distance to the nearest element of each class is the largest. For data that is not linearly separable the kernel trick is used. This is a method of adding dimensions to the data while at the same time keeping the calculations feasible. For each outcome, a polynomial (kernel) SVM and a radial (kernel) SVM is fit.

Random Forest

A Random Forest is an ensemble of decision trees. The model is trained with a technique called bootstrap aggregation (bagging). Bagging reduces variance and avoids overfitting in ensemble methods. With this technique, many bootstrap samples are taken and a decision tree is trained on each sample. The outcome of all trees together is averaged, which leads to the final outcome. For each outcome, a Random Forest is trained.

Adaboost

Boosting is similar to a Random Forest. The main differences are that the trees are now built sequentially and the results are averaged along the way. Boosting is an ensemble method that combines weak classifiers to output a single strong predicted response. The technique is considered to be an improvement over Random Forests on some occasions. For each outcome, an Adaboost.m1 model is trained.

Super Learner

The Super Learner finds an optimally weighted combination of candidate learners. The candidate learners can be any prediction algorithm. The Super Learner itself is a prediction algorithm as well. The performance of the candidate learners is assessed by cross‐validation. For each outcome, a Super Learner model is trained. The candidate learners consist of all models mentioned above. With the exception of Adaboost.m1, which is replaced by XGBoost (an alternative boosting algorithm).

Abbreviation: Lasso, Least Absolute Shrinkage and Selection Operator.

RESULTS

Study population

Between 2011 and 2017, 8173 patients were included in the DUCA. Of these, 6427 were included in the final data set (Figure 1). Of the excluded patients, 403 were a result of missing essential values; the outcomes of these patients were not significantly different from those included. In total, 4228 patients underwent a surgical resection for esophageal carcinoma, of which 2540 had a postoperative complication (60%). Of the 2199 patients with a resection for gastric carcinoma, 883 patients had a postoperative complication (40%). Patient characteristics are described in Table 2, and Figure 2 presents an overview of the type of postoperative complications.

Figure 1

Patient selection.

Table 2

Clinical characteristics.

	Esophageal cancer resection (n = 4228)	Gastric cancer resection (n = 2199)
Age, median (IQR) [range]	66 (59–71) [19–89]	70 (62–77) [22–92]
Gender
Male	3272 (77%)	1366 (62%)
Female	956 (23%)	833 (38%)
BMI
<20	276 (7%)	170 (8%)
20–24	1614 (38%)	949 (43%)
25–29	1663 (39%)	789 (36%)
≥30	675 (16%)	291 (13%)
Comorbidity
None	998 (24%)	433 (20%)
Yes	3229 (76%)	1764 (80%)
Cardiac	978 (30%)	676 (31%)
Diabetes	639 (20%)	375 (17%)
Pulmonary	769 (18%)	352 (16%)
Thrombotic	173 (5%)	154 (7%)
Unknown	1 (<1%)	2 (<1%)
Preoperative weight loss
None	1138 (27%)	639 (29%)
1–5 kg	1184 (28%)	543 (25%)
6–10 kg	891 (21%)	473 (22%)
11–15 kg	279 (7%)	146 (7%)
16–20 kg	106 (3%)	55 (3%)
21–35 kg	56 (1%)	24 (1%)
Unknown	574 (14%)	319 (15%)
Previous surgery a
No	2943 (70%)	1313 (40%)
Yes	1276 (30%)	882 (40%)
Unknown	9	4
Histology
Adenocarcinoma	3383 (80%)	2195 (>99%)
Squamous cell carcinoma	845 (20%)	4 (<1%)
Type of surgery
Transhiatal	1395 (33%)	‐
Transthoracic	2833 (67%)	‐
McKeown	1353 (48%)
Subtotal gastrectomy	‐	1275 (58%)
Total gastrectomy	‐	924 (42%)
cTNM‐7 stage
Stage 0	6 (<1%)	16 (1%)
Stage I	566 (13%)	465 (21%)
Stage II	1116 (26%)	842 (38%)
Stage III	2155 (51%)	185 (8%)
Stage IV	40 (1%)	39 (2%)
Stage X	345 (8%)	652 (30%)
Neoadjuvant treatment
None	314 (7%)	848 (39%)
Chemotherapy	286 (7%)	1316 (60%)
Chemoradiotherapy	3628 (86%)	35 (2%)
ASA score
I	712 (17%)	305 (14%)
II	2592 (61%)	1237 (56%)
III	908 (22%)	639 (29%)
IV	16 (<1%)	18 (1%)
Steroid use
No	4093 (97%)	2118 (96%)
Yes	107 (3%)	46 (2%)
Unknown	28 (1%)	35 (2%)

Thoracic and/or abdominal surgeries.

Figure 2

Type of complications after surgery after (A) esophagectomy and (B) gastrectomy.

Esophageal carcinoma

Anastomotic leakage occurred in 31% (799 of 4228) patients following esophagectomy and pulmonary complications in 54% (1380 of 4228) (Figure 2). From all prediction models, the GLM had the highest AUC, both for anastomotic leakage (61.9; 95% confidence interval [CI]: 57.9–65.9) and for pulmonary complications (64.4; 95% CI: 60.9–67.9) (Figure 3A,B). Closely followed by the machine learning models: Neural Networks (AUC: 61.7; 95% CI: 57.7–65.6), Least Absolute Shrinkage and Selection Operator (Lasso) (AUC: 61.7; 95% CI: 57.7–65.7), and SuperLearner (AUC: 61.7; 95% CI: 57.7–65.8) for anastomotic leakage. And the machine learning model Lasso (AUC: 64.3; 95% CI: 60.9–67.8) for pulmonary complications. For preoperative prediction, nomograms, based on a GLM, have been constructed for anastomotic leakage (Figure 5A) and pulmonary complications (Figure 5B). For anastomotic leakage: steroid use, advanced tumor stage, distant metastasis, surgical approach, and preoperative weight loss factors with the most prognostic value. For pulmonary complications, these are weight loss, ASA III/IV, advanced tumor stage, type of resection, and location of anastomosis.

Figure 3

(A) Anastomotic leakage after esophageal cancer resection. (B) Pulmonary complications after esophageal cancer resection. AUC, area under the curve; CI, confidence interval; KNN, k‐Nearest Neighbors; Lasso, Least Absolute Shrinkage and Selection Operator; SVM, Support Vector Machine.

Gastric carcinoma

After gastrectomy, anastomotic leakage was reported in 18% (156 of 2199) patients. GLM had the highest AUC (68.0; 95% CI: 60.2–75.8) (Figure 4), followed by the machine learning model Neural Networks (AUC: 67.9; 95% CI: 60.4–75.5). A nomogram for the preoperative prediction of anastomotic leakage after gastric resection is displayed in Figure 5C. Tumor histology and lymph node involvement are factors with the most prognostic value.

Figure 4

Anastomotic leakage after gastric cancer resection. AUC, area under the curve; CI, confidence interval; KNN, k‐Nearest Neighbors; Lasso, Least Absolute Shrinkage and Selection Operator; SVM, Support Vector Machine.

(A) Nomogram for the prediction of anastomotic leakage after esophageal cancer resection. AUC: 61.9%. (B) Nomogram for the prediction of pulmonary complications after esophageal cancer resection. AUC: 64.4%. (C) Nomogram for the prediction of anastomotic leakage after gastric cancer resection. AUC: 68.1%. AUC, area under the curve.

DISCUSSION

This study presents the development of various (machine learning) models for the prediction of anastomotic leakage and pulmonary complications in a population‐based cohort of 4228 patients who underwent esophagectomy and 2199 patients who underwent gastrectomy in the Netherlands. Linear regression had the highest accuracy of all models for the prediction of anastomotic leakage after esophagectomy and gastrectomy, as well as for pulmonary complications after esophagectomy. Of the machine learning models, the Neural Networks had the highest accuracy for predicting anastomotic leakage after esophageal cancer surgery and after gastric cancer surgery. Lasso had the highest accuracy of the machine learning models for the prediction of pulmonary complications following esophageal cancer surgery. Furthermore, the highest accuracy of all studied models was 68.0%, suggesting that none of the models had a superior predictive ability for postoperative complications in this patient cohort.

It is thought that for the development of machine models, a large population is necessary to adequately train the models. For example, the study of Nudel et al. included over 436,000 patients and 40 different variables. In their study, artificial neural networks and gradient boosting machines outperformed the traditional linear regression in predicting anastomotic leakage after weight loss surgery. However, a study that included merely 321 patients successfully designed a support vector classification model to predict postoperative complications in patients undergoing gastrectomy, using 23 clinical features. Their model had an accuracy of 78% in external validation. Like in the current study, age and tumor stage were the most predictive of the development of complications. With a broad array of machine learning models available, it is difficult to decide which model to use for each particular outcome. The systematic review of Elfanagely et al. reviewed 45 papers published between 2015 and 2020. They concluded that machine learning in surgical research is still in its infancy, but these early‐published papers show potential. However, they found great heterogeneity exists between the different studies; various models are being used, and different variables and outcomes are being investigated. They have also shown large variation in sample sizes, ranging from 71 to 130,945, implying that sample size is not the only factor for successful machine learning models. However, it might be possible that a certain publication and confirmation bias is present in the current literature, which could be deceiving.

In line with current literature, our study has demonstrated that ASA score ≥III is associated with anastomotic leakage and pulmonary complications. , Both patients with advanced age and high ASA classification are thought to have lower healing capacity causing higher susceptibility to postoperative complications. Patients with a more advanced tumor stage may require more extensive resections and technically more demanding surgery to reach an R0 resection. This may lead to more intraoperative organ damage and subsequent postoperative complications. Furthermore, lymph node involvement and, therefore, extensive lymph node dissections and possibly additional splenic resection are high‐risk procedures. As shown in this study, chronic use of steroids preoperatively is associated with postoperative complications, which is thought to be due to a reduced healing capacity. ,

In daily practice, it is difficult to estimate the surgical risk and make treatment decisions based on individual predictive factors. Therefore incorporating multiple factors into prediction models can be used to combine the information in a simple and more useful manner. However, the use of these models is often limited since they are often created in a selected patient population or specialized centers, making generalization hard; hence this study used a nationwide population‐based cohort. , Furthermore, clinical judgment and expertise are still needed for the correct interpretation and usage of clinical prediction models. With the current more data‐driven approach to health care and the availability of nationwide clinical audits, big data become available, eliminating this limitation of generalization of models. In addition, with big data, the interest in machine learning for prediction models has increased. In our study, linear regression was superior to the machine learning models. Another study, which used a more extensive amount of variables, did show favor for machine learning models. However, one could question the use in daily clinical practice when using more extensive models, which subsequently leads to more administrative burden to include all variables unless automated. Furthermore, in some machine learning models (e.g., neural networks) individual prognostic factors are not known, in contrast to linear regression. As demonstrated in this study, linear regression can be used to construct nomograms, which are easy to use in daily practice and might expose improvable prognostic factors (e.g., weight loss, steroid use) for postoperative complications. Subsequently, nomograms can easily be formed into web‐based models of mobile phone applications, which might increase usability in daily practice.

As part of the currently ongoing implementation and standardization of perioperative care into enhanced recovery after surgery (ERAS) protocols, preoperative optimization of patients has gained interest. Research toward improving perioperative care for upper gastrointestinal surgery also focuses on identifying preoperative high‐risk patients and developing prehabilitation programs for these patients. Upper gastrointestinal cancer patients are at high risk for malnutrition due to the anatomical localization of the tumor. Therefore, nutritional interventions are important in preoperative prehabilitation. Prehabilitation programs for patients with esophageal carcinoma have been shown to improve objective measures of physical fitness, but are less clear on postoperative outcomes. However, good physical fitness and nutritional status are widely recognized as a protective factor against postoperative complications. It has shown to lead to a sooner return to bowel function, oral feeding, and restored metabolic equilibrium, and is therefore currently being standardized and implemented into ERAS protocols. , This may indicate that prehabilitation programs might have to be more specific toward certain risk factors.

Although our study provides insight into a different aspect of the clinical applicability of machine learning models, it has some limitations. The use of Dutch national audit data, DUCA, might lead to less generalizability to other countries. However, in the Netherlands, participation in the DUCA has been incorporated as a mandatory quality standard, leading to an exceptionally complete and reliable database. Voluntary participation of some other audits and registries could give a distorted view if their participation did not concern all patients. External validation of the models did not occur in this study. However, the accuracy of the models was tested using a random internal sample of 25% of patients. Another limitation of the study is that the experience and expertise of the individual centers and/or surgeons could not be included. Hospital volume is thought to be a predictor of mortality after high‐risk surgery. Patients treated in high‐volume hospitals benefit from more experience and more advanced expertise. However, according to the DUCA research regulations, no data are provided that can be used to derive individual hospitals. If these restrictions are lifted in the future, this variable could be implemented in the model to improve accuracy. Additionally, accuracy may be improved by adding more variables, which are currently not in the DUCA registry, such as preoperative laboratory results (e.g., C‐reactive protein, albumin) and other predictors such as smoking and alcohol usage , were not included in our models, whereas these variables may serve as strong preoperative predictors. The use of intraoperative variables such as intraoperative hypotension or blood transfusion may improve the predictive accuracy of the model. However, these factors cannot be used during patient selection for prehabilitation programs or for surgery. , , Additionally, the anastomotic leakage rate of 18% following esophagectomy in the Netherlands is relatively high compared to other countries, an explanation is the learning curve for new techniques (e.g., minimally invasive) in recent years. Around 2010 minimally invasive surgery was introduced and many surgeons changed from a McKeown to an Ivor Lewis technique. Furthermore, both clinically and radiologically proven anastomotic leakages were included. Finally, all patients included in this study were selected to be fit for surgery by expert opinion preoperatively, leading to allocation bias. However, the occurrence of this type of bias is unavoidable in this type of study.

CONCLUSION

This study demonstrates that the studied machine‐learning models are able to predict postoperative complications in upper gastrointestinal cancer surgery, but they are not superior to the current gold standard, logistic regression. However, the accuracy of all studied models was relatively low. Furthermore, the use of prediction models does serve a purpose for preoperative risk estimation and treatment decisions, but clinical expertise is still needed. Additionally, identifying predictive individual factors within prediction models (e.g., malnutrition) may improve perioperative care and might lead to the improved preoperative physical fitness of patients, which can improve ERAS protocols and therewith surgical outcomes.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.

SYNOPSIS

Curative treatment of upper gastrointestinal malignancies includes surgical resections with a high risk of postoperative morbidity. The present study shows that predicting postoperative complications after upper gastrointestinal cancer surgery using machine learning is feasible. However, machine learning models do not outperform the current gold standard, the generalized linear model.