Validation of the ACS NSQIP Surgical Risk Calculator for Patients with Early Gastric Cancer Treated with Laparoscopic Gastrectomy.

PURPOSE: The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) risk calculator is useful in predicting postoperative adverse events. However, its accuracy in specific disorders is unclear. We validated the ACS NSQIP risk calculator in patients with gastric cancer undergoing curative laparoscopic surgery.
MATERIALS AND METHODS: We included 207 consecutive early gastric cancer patients who underwent laparoscopic gastrectomy between January 2018 and January 2019. The preoperative characteristics and risks of the patients were reviewed and entered into the ACS NSQIP calculator. The estimated risks of postoperative outcomes were compared with the observed outcomes using C-statistics and Brier scores.
RESULTS: Most of the patients underwent distal gastrectomy with Roux-en-Y reconstruction (74.4%). We did not observe any cases of mortality, venous thromboembolism, urinary tract infection, renal failure, or cardiac complications. The other outcomes assessed were complications such as pneumonia, surgical site infections, any complications requiring re-operation or hospital readmission, the rates of discharge to nursing homes/rehabilitation centers, and the length of stay. All C-statistics were <0 and the highest was for pneumonia (0.65; 95% confidence interval: 0.58-0.71). Brier scores ranged from 0.01 for pneumonia to 0.155 for other complications. Overall, the risk calculator was inconsistent in predicting the outcomes.
CONCLUSIONS: The ACS NSQIP surgical risk calculator showed low predictive ability for postoperative adverse events after laparoscopic gastrectomy for patients with early gastric cancer. Further research to adjust the risk calculator for these patients may improve its predictive ability.

INTRODUCTION

The incidence of gastric cancer varies among different countries and regions. Korea and Japan show the highest incidence of gastric cancer, whereas Europe and the United States have shown a steady decrease in newly diagnosed gastric cancers [1]. Despite the decline in incidence, gastric cancer remains the second leading cause of cancer deaths and the fourth most common malignancy worldwide [2345]. In South Korea, gastric cancer remains the most common cancer and the fourth most common cause of cancer-related deaths [6].

Surgical resection remains the curative modality for treating gastric adenocarcinoma. Among patients with gastric cancer, those with early gastric cancer (EGC) have an excellent prognosis when treated with curative surgical resection and appropriate lymphadenectomy. The long-term survival rate is almost 99% when the tumor is limited to the mucosal layer and 96% when the cancer is confined to the submucosa [78]. The number of EGC cases is increasing in Korea owing to the endoscopic screening program supported by the government. Consequently, minimally invasive resection is being increasingly used to treat the patients with EGC [910].

Surgical treatment for gastric cancer has now been refined in various ways, leading to reduced surgical morbidity and subsequently facilitating better recovery. Following the initial report on laparoscopic gastrectomy for EGC [11], other retrospective studies and randomized clinical trials have also shown the short-term benefits of laparoscopic surgery over conventional open gastrectomy and comparable long-term outcomes between the 2 methods [12131415]. As a result, laparoscopic approach has currently become a popular treatment strategy in gastric cancer, particularly for EGC.

The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has established an online surgical risk calculator. This risk calculator is based on a database review and an analysis of preoperative patient data and postoperative complications, involving more than a million surgical patients from 393 hospitals across the United States [1617]. ACS NSQIP variables have been quite useful for predicting mortality in several procedures and for improving surgical quality in the participating hospitals [18].

The accuracy of the ACS NSQIP surgical risk calculator was found to be varying for patients with gastric cancer, as shown in the results from the United States Gastric Cancer Collaborative [19]. Given the differences between the West and the East in terms of patient characteristics, surgical approach, and outcomes [20], our aim in this study was to examine the ability of the ACS NSQIP risk calculator to estimate short-term surgical outcomes among a Korean patients with gastric cancer, treated with a minimally invasive approach.

MATERIALS AND METHODS

A prospectively collected database of 207 consecutive patients who underwent laparoscopic gastrectomy for EGC at Asan Medical Center Hospital, Seoul, Korea between January 2018 and January 2019, was reviewed. All the procedures performed in the study complied with the ethical standards of the institutional and national research committee. The study was conducted in accordance with the Declaration of Helsinki. Formal consent was not required as this was a retrospective study. The Institutional Review Board of Asan Medical Center approved this study (2019-1466).

Subsequently, the data were retrospectively analyzed using ACS NSQIP variables including sex, age group, weight and height, the American Society of Anesthesiologists (ASA) class, presence of comorbidities like diabetes, chronic obstructive pulmonary disease, hypertension, congestive heart failure, dyspnea, ascites, dialysis, renal failure and sepsis, ventilator dependence, and disseminated cancer. The actual surgical and non-surgical outcomes were documented.

The preoperative patient factors were entered into the NSQIP online risk calculator at the ACS NSQIP surgical risk calculator’s website (http://www.riskcalculator.facs.org) [16]. The estimated results were then compared with actual patient outcomes. Preoperative characteristics and the current procedural terminology (CPT) code (43621 - Gastrectomy, total; with Roux-en-Y reconstruction, 43631 - Gastrectomy, partial, distal; with gastroduodenostomy, 43632 - Gastrectomy, partial, distal; with gastrojejunostomy, 43633 - Gastrectomy, partial, distal; with Roux-en-Y reconstruction) were entered for each patient separately. The results of the calculator included all the complications comprising serious complications (cardiac arrest, myocardial infarction, pneumonia, progressive renal insufficiency, acute renal failure, pulmonary embolism, deep vein thrombosis, deep incisional surgical site infection, organ/space surgical site infection, systemic sepsis, unplanned intubation, and wound disruption), pneumonia, cardiac complication, surgical site infection, urinary tract infection, venous thromboembolism, renal failure, return to the operating room, readmission, discharge to nursing/rehab, length of hospital stay, sepsis, and death. The available option of the surgeon adjustment score (SAS) was set at the default value of 1.

Statistical analysis

Patient demographics and clinical characteristics were summarized using descriptive statistics, including frequency and percentage for categorical variables and median and interquartile range for continuous variables. The predicted rates of complications in these patients were then compared with the actual patient outcomes.

The performance of the ACS NSQIP calculator in predicting the risk was evaluated using 2 metrics: C-statistic and Brier score. C-statistic is a test that represents the area under the curve (AUC) of a receiver operating characteristic curve. It is a measure of discrimination and plots the sensitivity (true positive rate) versus 1-specificity (false positive rate). If the variable under the study, which in this case is the ACS NSQIP calculated risk, accurately predicts the patients who will have complications versus those who will not, the AUC will be 1. If the variable fails to distinguish between those who will have a complication and those who will not, the AUC will be 0.5. In general, an AUC value of >0.7 is indicates relatively good predictability and an AUC value >0.8 is indicates good predictability [21].

The Brier score is a simultaneous measure of calibration and discrimination. It is reported as a score between 0 and 1. A score of 0 indicates no difference between the predicted and actual outcome, thus indicating the best possible test result. A score of 1 indicates that the test did not predict the outcome. The Brier score is compared with a Brier score cut-off, which is partially based on the incidence in the sample, and a score above the cut-off is considered not useful [1622]. The length of hospital stay was excluded from the Brier score analysis because it is not a binary outcome. All statistical analyses were performed using STATA® Data Analysis Statistical Software (College Station, TX, USA) [23]. A P-value of <0.05 was considered statistically significant.

RESULTS

A total of 207 consecutive patients who underwent laparoscopic gastrectomy for EGC were included in the study. The preoperative risk variables are shown in Table 1. The most commonly performed procedure was total laparoscopic distal gastrectomy with Roux-en-Y reconstruction (n=154, 74.4%), followed by laparoscopic total gastrectomy with overlap anastomosis (n=28, 13.5%). Majority of the included patients were below 65 years old (n=183, 66.7%), men (n=132, 63.8%), independent in terms of the functional status (n=206, 99.5%), classified as ASA class II with mild systemic disease (n=143, 69%), and underwent elective surgery (n=207, 100.0%).

Table 1

Correlation of preoperative risk factors with actual outcomes (complications vs. no complications)

Characteristics		No. (%)	Complications	No complications	P-value*
Age (yr)					0.954
	<65	138 (66.7)	26 (18.8)	112 (81.2)
	65–74	44 (21.3)	9 (20.5)	35 (79.5)
	75–84	24 (11.6)	5 (20.8)	19 (79.2)
	>84	1 (0.5)	0	1 (100.0)
Type of surgery					0.314
	Billroth I	2 (1.0)	1 (50.0)	1 (50.0)
	Billroth II	22 (10.6)	5 (22.7)	17 (77.3)
	Roux-en-Y GJ	155 (74.9)	26 (16.8)	129 (83.2)
	Roux-en-Y EJ	28 (13.5)	8 (28.6)	20 (71.4)
HTN					0.742
	Yes	68 (32.9)	14 (20.6)	54 (79.4)
	No	139 (67.1)	26 (18.7)	113 (81.3)
DM					0.491
	Yes	34 (16.4)	8 (23.5)	26 (76.5)
	No	173 (83.6)	32 (18.5)	141 (81.5)
BMI					0.587
	Normal (18.5–24.9)	125 (60.4)	22 (17.6)	103 (82.4)
	Overweight (25–29.9)	72 (34.8)	15 (20.8)	57 (79.2)
	Obese (30 and above)	10 (4.8)	3 (30.0)	7 (70.0)
Smoking in 1 year					0.657
	Yes	36 (17.4)	34 (94.4)	137 (380.6)
	No	171 (82.6)	6 (3.5)	30 (17.5)
Cardiac event					0.323
	Yes	4 (1.9)	0	4 (100.0)
	No	203 (98.1)	40 (19.7)	163 (80.3)
Functional dependence					0.041
	Independent	206 (99.5)	39 (18.9)	167 (81.1)
	Partially dependent	1 (0.5)	1 (100.0)	0
ASA class					0.373
	1	60 (29.0)	8 (13.3)	52 (86.7)
	2	143 (69.1)	31 (21.7)	112 (78.3)
	3	4 (1.9)	1 (25.0)	3 (75.0)
	4, 5	0	0	0
Steroid use					0.041
	Yes	1 (0.5)	1 (100.0)	0
	No	206 (99.5)	39 (18.9)	167 (81.1)
Ascites					0.481
	Yes	2 (1.0)	0	2 (100.0)
	No	205 (99.0)	40 (19.5)	165 (80.5)

Values are presented as number of patients (%).

GJ = gastrojejunostomy; EJ = esophagojejunostomy; HTN = hypertension; DM = diabetes mellitus; BMI = body mass index; ASA = American Society of Anesthesia.

*P<0.05 was considered significant.

The analysis of preoperative factors in correlation with the actual events showed a positive association for the patients' functional status (P=0.041) and steroid use (P=0.041). All the other factors showed insignificant associations. There was no mortality in the study group and there were no actual events of cardiac complications, urinary tract infections, venous thromboembolism, and renal failure in the postoperative results.

All comparisons demonstrated that the ACS NSQIP calculator with default SAS did not accurately predict postoperative complications. Table 2 displays the receiver operating characteristic curves of the outcomes with the observed events. The C-statistics were poor (<0.7) for all the outcomes (0.57–0.65) (Fig. 1). The C-statistic results were close to a fair value for pneumonia (AUC=0.65) but showed low results for any complications (AUC=0.57).

Table 2

AUC and Brier scores for patients with and without adverse events

Outcomes	With adverse events		Without adverse events		OR (95% CI)	P-value	AUC	Brier score	R2
	No. of cases	Median risk	No. of cases	Median risk
Serious complications	12	13.9%	195	13.5%	1.099 (0.95–1.27)	0.201	0.6	0.054	0.025
Any complications	40	16.8%	167	16.3%	1.07 (0.98–1.16)	0.113	0.57	0.154	0.015
Pneumonia	2	2.9%	205	2.1%	1.26 (0.6–2.6)	0.771	0.65	0.009	0.103
Surgical site infection	13	9.1%	194	8.2%	1.017 (0.91–1.13)	0.749	0.6	0.058	0.089
Readmission	4	9.05%	203	8.7%	1.22 (0.7–2.1)	0.472	0.57	0.018	0.032
Return to operating room	7	5.5%	200	5.3%	1.14 (0.63–2.06)	0.641	0.61	0.032	0.069

AUC = area under the curve; OR = odds ratio; CI = confidence interval.

Fig. 1

ROC curve analysis showing the calculation of AUC, for postoperative outcomes. (A) Surgical site infection, (B) return to operating room, (C) readmission, (D) any complications, (E) serious complications, and (F) pneumonia.

ROC = receiver operating characteristics; AUC = area under the curve.

The Brier scores were contradictory to the AUC results. They showed a considerably impressive values by approaching 0.0 for all of the outcomes (Brier score range, 0.01–0.06), except for any complication (Brier score, 0.15). The best score result was demonstrated for pneumonia (0.01) and all results were below the cut-off, which indicates its validity.

In the regression analysis based on the R2 statistic, all values were insignificant with the R2 results (0.015–0.103). Furthermore, the odds ratio results proved to be insignificant. The mean length of hospital stay did not show any significance (R2=0.016; P=0.074; 95% confidence interval, 0.07–1.5), even though there was a higher rate than that predicted by the surgical risk calculator among those who had a complication, and a lower rate than that predicted in the non-event group. Taking the modal analyses results into consideration, the ACS NSQIP appeared to be variable and somewhat inconsistent in predicting outcomes for patients with EGC treated with laparoscopy.

DISCUSSION

Although no model accurately reflects reality, some statistical models can test the predictive ability of risk calculators. In the development of the ACS NSQIP surgical risk calculator, many metrics were suggested for evaluation [16]. The C-statistic and Brier score were the most commonly used methods for this purpose. AUC is used primarily for discrimination but it has a limitation of focusing on categorical comparisons and not evaluating the accuracy of prediction directly [24]. The Brier score indicates discrimination and calibration simultaneously, but it is limited by the prevalence of cases [25]. Demonstrating positive results with both tests is an indication of a valid risk calculator. In the current study, we showed variable values of AUC and Brier scores. The results may have been affected by the homogeneity of the case-mix and a low number of complications, overall. In a western study by Beal et al. [14], the accuracy of the ACS NSQIP online risk calculator for estimating risk after gastrectomy was assessed using the United States Gastric Cancer Collaborative database. Nine hundred and sixty-five patients were included in the study from the period between 2000 to 2012. Most patients underwent total gastrectomy with Roux-en-Y reconstruction (404 patients, 41.9%) and only 1 patient was classified as emergent. The complication rates were stratified in the cohort by stages and stage IA and IB were included in a group (407 patients, 25.6%). In this group, the rate of serious complications was 27.1% (67 patients) with mortality of 2.8% (7 patients) and any complications were seen in 37.7% (93 patients). The study demonstrated an increase in complication rates with higher stages. In their cohort, the C-statistic was highest for venous thromboembolism (0.69) and lowest for renal failure (0.54). All C-statistics were less than 0.7. Brier scores ranged from 0.010 for venous thromboembolism to 0.238 for any complication. They concluded that the general estimates of risk for the cohort were variable in terms of accuracy. In the present study, we had a low rate of surgical complications. However, there were similar variable results of AUC and Brier scores regarding the ability to predict outcomes.

In general, gastrectomy is a surgical procedure that carries unique risks because of its high technical demand. Two Western multicenter trials found a morbidity rate of 26%–46% and a 30-day mortality rate of 3%–11% after gastrectomy [20]. In comparison, the Eastern trials showed a morbidity rate of 7.3%–28.1% and a 30-day mortality of 0.0%–0.8% [20]. In laparoscopic gastrectomy, Kitano et al. [26] reported morbidity and mortality rates of 14.8% and 0.0%, respectively, and Kim et al. [27] reported similar results of 13.0% and 0.6%, respectively. In the current study, we demonstrated a comparable mortality rate of 0.0% and a high rate of any complication of 19.3%, as defined by the ACS NSQIP outcome. When the Clavien-Dindo classification [28] grade I was excluded, the rate of morbidity was 12%, which was comparable with the currently available data on its incidence. Incorporating such a universal definition for complications to define serious and any complication outcomes in the ACS NSQIP calculator makes the outcomes standardized and amenable for accurate comparison in other studies.

Accurate evaluation of surgical procedure results must involve assessment in terms of postoperative morbidity and mortality. These results are influenced by different variables, including the patient's preoperative characteristics, the type of surgical procedure, and the surgeon's experience [29]. Predicting postoperative risks is a critical component of operative planning, informed consent, and sharing the medical decision with patients and their families [19]. Multiple models of risk prediction, including the physiological and operative severity score for the enumeration of mortality and morbidity (POSSUM), its variations P-POSSUM and O-POSSUM [29], and the Acute Physiology and Chronic Health Evaluation (APACHE) II scoring system [30], have been published. The ideal model should predict postoperative morbidity and mortality as accurately as possible, be quick and easy to apply, and must be adaptable to different surgical procedures. Moreover, it should be applicable to all health institutes. Most of the current predictive models involve operative factors, which limit their use in preoperative assessment. However, one of the most valuable points of the ACS NSQIP risk calculator is its complete dependability on preoperative factors.

The ACS NSQIP surgical risk calculator gains its value from being a decision-supporting tool based on reliable multi-institutional clinical data, which can be used to estimate the risks of most operations. It has been designed to assist surgeons and patients in making decisions, using estimated patient-specific postoperative risks. Yet, by analyzing patients with EGC, our study demonstrates that the current calculator is not an effective tool to predict the risks of the surgical outcome. We believe that this is because our study focused on a specific group of patients who underwent a specific treatment, i.e. laparoscopic gastrectomy. The prediction results could be improved by using a new tool that includes preoperative variables with clinical staging and the variables involving intra-operative procedures, such as the extent of lymphadenectomy or the type of surgical approach.

Although the SAS function allows surgeons to predict the complication risk of surgeries based on their clinical evaluation and general impression of the patient, it lacks a clear determinant and does not support lowering the risk in a suitable clinical scenario. In our study, we demonstrated a lower rate of most of the surgical complications after laparoscopic gastrectomy than that predicted by the risk calculators (Fig. 2). Therefore, the calculator outcomes should be adjusted to a lower percentage in such interventions. Augmenting the calculator with other valuable predictors such as albumin level, nutritional factor, and the type of procedure may improve the accuracy for patients with gastric cancer [3132].

Fig. 2

Actual percentage of outcomes vs. predicted mean of surgical outcomes.

This study had several limitations. First, the analysis included only the laparoscopic gastrectomy procedure performed by a single surgeon at a high volume center, which may affect the generalizability of the results. Future studies using the ACS NSQIP risk calculator should include more surgeons at both low and high-volume institutions, in order to overcome this limitation. Second, the number of complications to be assessed was limited. A large sample size might yield a higher detection rate of complications, which would overcome the Brier score limitations and the homogeneity of the patients. Third, this study was a retrospective analysis. Finally, in our study, most of the patients (74.9%) underwent laparoscopic distal gastrectomy with Roux-en-Y gastrojejunostomy. Consequently, a relatively small portion of the patients underwent laparoscopic distal gastrectomy with Billroth I, laparoscopic distal gastrectomy with Billroth II, or laparoscopic total gastrectomy with Roux-en-Y esophagojejunostomy procedures. Due to such disproportionate distribution, we were unable to classify and analyze the extent of gastrectomy or the type of reconstruction by the CPT codes.

Despite these limitations, this study had 2 main strengths. First, this is the first study to evaluate the effectiveness of the surgical risk calculator for predicting outcomes for laparoscopic gastrectomy among a specific group of patients with EGC. Second, it assessed the outcomes of consecutive patients in an area where gastric cancer is the most prevalent.

In conclusion, the ACS NSQIP surgical risk calculator is a valuable tool in predicting short-term surgical outcomes. However, the current ACS NSQIP variables showed a low predictive ability for postoperative adverse events after laparoscopic gastrectomy for EGC. This limits the clinical applicability of the tool for patients with EGC after laparoscopic gastrectomy.

Adding disease-specific and operation-specific variables may lead to further evolution of the ACS NSQIP risk calculator for patients with gastric cancer, allowing valuable prediction of adverse outcomes for laparoscopic gastric surgery.