External validation of risk prediction platforms for pancreatic fistula after pancreatoduodenectomy using nomograms and artificial intelligence.

Purpose: Postoperative pancreatic fistula (POPF) is a life-threatening complication following pancreatoduodenectomy (PD). We previously developed nomogram- and artificial intelligence (AI)-based risk prediction platforms for POPF after PD. This study aims to externally validate these platforms.
Methods: Between January 2007 and December 2016, a total of 1,576 patients who underwent PD in Seoul National University Hospital, Ilsan Paik Hospital, and Boramae Medical Center were retrospectively reviewed. The individual risk scores for POPF were calculated using each platform by Samsung Medical Center. The predictive ability was evaluated using a receiver operating characteristic curve and the area under the curve (AUC). The optimal predictive value was obtained via backward elimination in accordance with the results from the AI development process.
Results: The AUC of the nomogram after external validation was 0.679 (P < 0.001). The values of AUC after backward elimination in the AI model varied from 0.585 to 0.672. A total of 13 risk factors represented the maximal AUC of 0.672 (P < 0.001).
Conclusion: We performed external validation of previously developed platforms for predicting POPF. Further research is needed to investigate other potential risk factors and thereby improve the predictability of the platform.

INTRODUCTION

Postoperative pancreatic fistula (POPF) is one of the most fatal complications after pancreatoduodenectomy (PD) and is associated with postoperative hemorrhage, intraabdominal infections, and increased mortality [12]. The incidence of POPF remains high despite advances in surgical techniques and perioperative management [2]. Particularly for clinically relevant POPF (CR-POPF), defined by the grading system of the International Study Group of Pancreatic Fistula (ISGPF) [3], the reported incidence was 14.5% in a previous meta-analysis [4].

Many studies have analyzed the risk factors for POPF. Based on traditional risk factors, there were attempts to develop platforms for predicting POPF [56]. However, due to the limited predictive values of previous platforms [78], we developed new risk prediction platforms using nomograms [9] and artificial intelligence (AI) technology [10]. The nomogram consisting of 6 preoperatively available data provides a better insight into the risk factors and their contributions. The AI model was based on an in-depth analysis of risk factors using machine learning algorithms. The model finally included 16 preoperative and intraoperative variables. Both models are readily available in the form of a calculator at http://popf.smchbp.org and http://popfrisk.smchbp.org.

In this study, we performed external validation of the aforementioned platforms with multicenter datasets to verify the reproducibility and generalizability of the models and to determine their clinical utilities.

METHODS

This study was approved by the Institutional Review Boards of Samsung Medical Center (Seoul, Korea; No. 2020-09-181), Seoul National University Hospital (Seoul, Korea; No. SNUH 2010-147-116), Ilsan Paik Hospital (Goyang, Korea; No. 2021-06-009), and Boramae Medical Center (Seoul, Korea; No. 30-2021-72). The study was performed in accordance with the Declaration of Helsinki and written informed consent was waived due to its retrospective nature.

Patient database

The cohort for external validation included 1,576 patients who underwent PD between January 2007 and December 2016 at 3 different centers: Seoul National University Hospital, Ilsan Paik Hospital, and Boramae Medical Center. The patients’ demographic data, preoperative laboratory results, imaging findings and surgical outcomes were retrospectively reviewed.

Perioperative data and risk calculation

The individual risks of CR-POPF were calculated using previously developed nomogram- and AI-based web calculators, which are available at the websites (Fig. 1). The nomogram was based on the following 6 preoperative variables: sex, body mass index (BMI), the American Society of Anesthesiology physical status (ASA PS) classification, serum albumin, tumor location, and the diameter of the main pancreatic duct (MPD) measured via CT or magnetic resonance cholangiopancreatography. The AI calculator was developed using the following preoperative and intraoperative variables: age, sex, BMI, underlying heart disease, ASA PS classification, preoperative platelet count, serum albumin, serum lipase, preoperative endoscopic biliary drainage, neoadjuvant radiotherapy, amount of intraoperative fluid infusion, pancreatic texture, the diameter of MPD, portal vein resection, coexisting pancreatitis detected preoperatively or intraoperatively, and tumor location.

Fig. 1

(A) The web-based nomogram calculator (http://popf.smchbp.org). (B) The web-based artificial intelligence (AI) calculator (http://popfrisk.smchbp.org).

There were no missing values in the categorical variables of the validation cohort. Median imputation was used for the missing continuous variables to replace the missing data with medians.

Postoperative outcomes

POPF was diagnosed and graded according to the 2016 ISGPF definition and grading. POPF grades B and C (CR-POPF) were included as the outcomes in the analysis, and biochemical leak was not counted.

Statistical analysis and external validation

Receiver operating characteristic (ROC) curve analyses were performed using IBM SPSS Statistics ver. 26 (IBM Corp., Armonk, NY, USA). The area under the curve (AUC) for the logistic regression model was reported with 95% confidence intervals (CIs). The AUC values with P-values of less than 0.05 were regarded as statistically significant.

Backward elimination was performed to obtain the optimal AUC value of the AI model by selecting features that had no significant prognostic value. The AUC for each model was calculated based on the stepwise selection in accordance with the development process.

RESULTS

The clinical demographics and surgical outcomes of 1,576 patients in the validation cohort are presented in Table 1. The patients’ mean age was 63.6 years, and 697 patients (44.2%) had underlying heart disease including hypertension. Preoperative endoscopic biliary drainage was performed in 825 patients (52.3%). Intraoperatively, 1,056 patients (67.0%) had soft pancreas and the mean diameter of MPD was 3.5 mm. CR-POPF was developed in 270 patients (17.1%).

Table 1

Demographic and clinical characteristics, and surgical outcomes of the validation cohort

Values are presented as number only, mean ± standard deviation, or number (%).

ASA PS, American Society of Anesthesiologist physical status; CCRT, concurrent chemoradiotherapy; POPF, postoperative pancreatic fistula; BCL, biochemical leakage.

a)Data were not available in 1, 688, and 2 patients in order.

In the external validation of the nomogram with 1,576 patients, the ROC curve was drawn with the AUC of 0.679 (95% CI, 0.645–0.713; P < 0.001) (Fig. 2A). Fig. 3 shows the values of the AUC after backward elimination. All 16 independent variables were entered into the analysis first and each variable was eliminated one by one. The maximal AUC was 0.672 (95% CI, 0.637–0.706; P < 0.001) (Fig. 2B), including the following 13 variables: the diameter of MPD, BMI, serum albumin, amount of intraoperative fluid infusion, age, preoperative platelet count, tumor location, portal vein resection, coexisting pancreatitis, serum lipase, neoadjuvant radiotherapy, ASA PS classification, and sex.

Fig. 2

(A) The receiver operating characteristic (ROC) of the nomogram. Area under the curve (AUC) = 0.679, P < 0.001. (B) The ROC of the artificial intelligence predictor. AUC = 0.672, P < 0.001.

Fig. 3

The area under the curve (AUC) values with backward elimination.

DISCUSSION

In the absence of a standardized management protocol for POPF until now, early prediction and recognition are crucial to identify patients at high risk of POPF requiring careful observation [2]. A series of traditional risk scoring systems, such as the original fistula risk score (o-FRS) [5] and the alternative fistula risk score (a-FRS) [6], have been used. However, several studies performed external validation and the predictability varied with the study population [781112]. Therefore, our institution suggested new predictive models using nomograms and AI [910] and this study validated the predictability of the new platforms with an external cohort.

The nomogram including 6 simple variables had an AUC value of 0.709 in the development process [9], and 0.679 in the external validation. A few other nomograms were recently proposed. Huang et al. [13] suggested a new nomogram with the following 3 variables: BMI, the diameter of MPD, and drain fluid amylase level (DFA) on postoperative day (POD) 1. The AUC value was 0.744 in the external validation. Another nomogram by Suzuki et al. [14] also included drain fluid lipase level on POD 1 and decreased rate of DFA, which was defined as a change in levels from POD 1 to 3. The accuracy of the nomogram was 0.810, as stated in the study. Compared to these recently developed nomograms, our nomogram showed limited predictive value in the external validation. The AUC value might have decreased in the process of external validation with highly heterogeneous data collected from different centers. But most of all, the predictive power of the variables in the nomogram could also be limited. Previous studies suggested that DFA is a strong predictive factor of POPF [1516], and DFA on POD 1 is currently widely accepted as an indicator for early drain removal after PD [17]. Considering that our nomogram is composed of only preoperative and intraoperative factors, updating the platform with DFA may improve the accuracy and predictability of the model.

Machine learning is a branch of AI technology designed to enable rapid analytical model building. It has been used in various medical fields including surgery. As far as we know, we invented the first AI-based prediction model for CR-POPF. The most remarkable advantage of machine learning is that it can identify complex structures in high-dimensional data [18] and detect latent variables, which are not directly measured using conventional analytical methods [19]. The new AI model yielded the maximal AUC value of 0.74 with 16 variables, and in the external validation with backward elimination, the AUC was 0.672 with 13 variables. This value is acceptable but leaves considerable room for improvement. First, there were quite a few missing values in both development and validation cohorts. Despite attempts to handle missing data in deep learning [20], it remains the main obstacle to model development. Also, the AI model includes a few variables, which cannot be objectively measured. For example, coexisting pancreatitis or pancreatic texture detected by surgeons intraoperatively can be highly subjective. In order to resolve this issue, several studies are underway to objectively measure those variables using preoperative images and machine learning [2122]. These efforts are expected to establish a foothold for improving predictability of the platforms in the future.

The current study has several limitations. Since this study is based on retrospectively reviewed multicenter datasets, the results might have been affected by selection and information bias. Some potential factors that could influence the development of POPF, such as anastomosis technique and postoperative drain management, varied considerably among the surgeons and the institutions. Also, as previously stated, the platforms were developed and validated with datasets including missing values, which reduces the sample representativeness and complicates the analysis. Further studies with prospectively collected high-quality data are needed to upgrade the platforms with improved predictability.

In conclusion, this study was performed to externally validate the previously developed prediction platforms for POPF. The results suggest the need for improvement and future studies to build better prediction models with higher accuracy.