Development and Validation of a Clinical Risk Score for Intensive Care Resource Utilization After Colon Cancer Surgery: a Practical Guide to the Selection of Patients During COVID-19.

BACKGROUND: The purpose of this study was to develop and validate a prediction model and clinical risk score for Intensive Care Resource Utilization after colon cancer surgery.
METHODS: Adult (≥ 18 years old) patients from the 2012 to 2018 ACS-NSQIP colectomy-targeted database who underwent elective colon cancer surgery were identified. A prediction model for 30-day postoperative Intensive Care Resource Utilization was developed and transformed into a clinical risk score based on the regression coefficients. Model performance was assessed using the area under the receiver operating characteristic curve (AUC) and Hosmer-Lemeshow goodness-of-fit test. The model was validated in a separate test set of similar patients.
RESULTS: In total, 54,893 patients underwent an elective colon cancer resection, of which 1224 (2.2%) required postoperative Intensive Care Resource Utilization. The final prediction model retained six variables: age (≥ 70; OR 1.90, 95% CI 1.68-2.14), sex (male; OR 1.73, 95% CI 1.54-1.95), American Society of Anesthesiologists score (III/IV; OR 2.52, 95% CI 2.15-2.95), cardiorespiratory disease (yes; OR 2.22, 95% CI 1.94-2.53), functional status (dependent; OR 2.81, 95% CI 2.22-3.56), and operative approach (open surgery; OR 1.70, 95% CI 1.51-1.93). The model demonstrated good discrimination (AUC = 0.73). A clinical risk score was developed, and the risk of requiring postoperative Intensive Care Resource Utilization ranged from 0.03 (0 points) to 19.0% (8 points). The model performed well on test set validation (AUC = 0.73).
CONCLUSION: A prediction model and clinical risk score for postoperative Intensive Care Resource Utilization after colon cancer surgery was developed and validated.

Introduction

Colon cancer is the second most common cause of cancer-related mortality in the USA [1], and it is anticipated that there will 147,950 new cases diagnosed in the year 2020 [2]. While surgical resection is the mainstay of curative therapy, the COVID-19 pandemic has greatly impacted the delivery of care to colon cancer patients. As the number of new cases of SARS-CoV-2 infection continues to grow across North America, elective surgeries have come to a halt, with cancer cases presenting a unique and difficult challenge [3]. Hospital administrators are faced with the task of balancing the risk of SARS-CoV-2 infection and healthcare resource utilization with that of disease progression associated with treatment delays [4]. In the management of colon cancer, this task becomes especially difficult, as no accepted treatment alternative exists for most cases. Colon cancer surgery is thus considered a medically necessary and time-sensitive procedure, even during the COVID-19 pandemic [5], and societal and provincial guidelines have prioritized these cases among the most urgent of procedures.

Among all hospital resources, intensive care unit (ICU) beds, equipment, and personnel are perhaps the most precious in the current healthcare environment. Based on data from Italy and China, it is estimated that approximately 15% of patients presenting with SARS-CoV-2 will require ICU admission [6, 7], and ventilator shortages have been of paramount concern. Surgical patients are typically a large consumer of Intensive Care Resource Utilization (ICRU), both in the emergency and elective setting [8]. Therefore, the potential for postoperative ICU care is an important consideration when selecting patients for cancer surgery during the COVID-19 pandemic.

The purpose of this study was to develop and validate a prediction model and clinical risk score for ICRU after colon cancer surgery, with the goal of facilitating healthcare decisions surrounding medically necessary colon cancer surgery during the COVID-19 pandemic.

Materials and Methods

This prediction model study is reported in accordance with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist [9].

Patients

Training Set for Model Development

After institutional review board approval, we identified all adult (≥ 18 years old) patients from the 2012 to 2018 ACS-NSQIP colectomy-targeted patient user file (PUF) who underwent an elective colectomy for colon cancer, according to Current Procedural Terminology codes and International Classification of Disease codes. Patient data from the colectomy-targeted PUF was then linked to the general PUF for demographic and general operative and postoperative data, and only patients with data in both databases were further included in the training set. Patients with an American Society of Anesthesiologists (ASA) score ≥ 5 and disseminated cancer or who were preoperatively ventilated or in septic shock were excluded. Patients with missing data on preoperative T-stage, N-stage, operative approach, or ASA score were also excluded.

Test Set for Model Validation

The test set consisted of a similar cohort of patients from the 2012 to 2018 ACS-NSQIP general PUF who underwent an elective colectomy for colon cancer, but who did not have a match in the colectomy-targeted PUF. The same inclusion and exclusion criteria were applied, and the outcome and variables were defined in an identical manner.

Dependent Variable

The primary outcome was 30-day postoperative Intensive Care Resource Utilization (ICRU), defined as a composite outcome including any one of the following postoperative events: (1) unplanned reintubation; (2) ventilator requirements > 48 h; (3) cardiac arrest; (4) septic shock; or (5) death. Unplanned reintubation was defined as the need for endotracheal tube insertion (or similar) and mechanical ventilation for reasons other than re-operation (e.g., refractory hypotension or inability to protect the airway). Septic shock was defined as signs and symptoms of sepsis with documented organ and/or circulatory dysfunction.

Candidate Variables

For the development of a clinically applicable prediction model and risk score, only preoperative factors were considered. The following patient, disease, and treatment factors were considered: age (dichotomized as < 70 and ≥ 70, given the risk for severe SARS-CoV-2 illness in individuals ≥ 70) (6), sex, ASA score (I/II vs. III/IV), obesity (defined as body mass index ≥ 30 kg/m2), diabetes, smoking, cardiorespiratory disease (defined as either chronic obstructive pulmonary disease, congestive heart failure, or dyspnea), hypertension, bleeding disorder, steroid use, preoperative weight loss ≥ 10%, functional status (partially or totally dependent for activities of daily living vs. independent), procedure (right colectomy vs. left colectomy vs. total abdominal colectomy), preoperative chemotherapy within 90 days, tumor depth (early: T1/2 vs. locally advanced: T3/4), nodal status (negative: N0 vs. positive: N1/2), distant metastases, operative approach (minimally invasive surgery (MIS) vs. open surgery), and bowel preparation (any: mechanical and/or oral antibiotic vs. none). Cases of conversion to open surgery were grouped with MIS, as they reflected an operation that the surgeon felt was amenable to an MIS approach. Missing data on bowel preparation was imputed using the Multiple Imputation by Chained Equations (MICE) package in R [10]. No other data was missing, allowing for a complete case analysis.

Model Development and Validation

Patients who required postoperative ICRU were compared with those who did not require ICRU within the training set. Patient, disease, and treatment factors were compared between the two groups on univariate analysis. A multiple logistic regression model was then developed, using a 0.05 forward and backward stepwise selection method for candidate variables. Selected variables in the regression model were then ranked according to their effect size (Wald chi-square statistic) and relative contribution to the overall model, taking into account both the strength of association (odds ratio) and the incidence of the variable. In order to develop a practical clinical risk score with as few variables as possible while retaining the model’s predictive capabilities, variables with an individual contribution of < 5% were dropped; all other variables were retained in the final prediction model. The area under the receiver operating characteristic curve (AUC) (equivalent to the c-statistic) was used to assess for model discrimination and the Hosmer-Lemeshow goodness-of-fit test for model calibration. The final model was then validated in the test set using the same measures of performance as described for model development. A clinical risk score was then developed based on the relative differences in regression coefficients. The risk of postoperative ICRU was displayed across the range of possible points. All data analyses were performed with R v3.5.1 and SAS 9.4.

Sensitivity Analysis

To test the robustness of the clinical risk score, a sensitivity analysis was performed, whereby the following three postoperative events were added to the composite outcome of 30-day postoperative ICRU: pulmonary embolism, pneumonia, and myocardial infarction. While these events do not always require postoperative ICU care, their inclusion may help account for the variability in ICU admission criteria that exists among hospitals. A new model was developed applying the same steps as outlined above, and model discrimination was similarly assessed.

Proof-of-Concept of Clinical Risk Score During COVID-19

To demonstrate the application of the clinical risk score for the selection of elective colon cancer cases during the COVID-19 pandemic, we retrospectively reviewed all planned colon cancer resection at three McGill University–affiliated hospitals from March 13, 2020, to April 20, 2020. Two hospitals (Jewish General Hospital and Montreal General Hospital) were designated level-1/condition-0 hospitals according to the American College of Surgeons Surgery COVID-19 Activation and Response Plan, while one hospital (St. Mary’s Hospital) was in the alert phase [11]. We compared the median clinical risk score among patients who underwent surgery as planned versus those whose who had their surgery plans altered (delayed or referred to a less acute-phase COVID hospital for surgery) due to higher risks of ICRU and mortality from COVID-19, if acquired.

Results

Of 54,893 patients who underwent an elective colon cancer resection in the training set, 655 (1.2%) had an unplanned reintubation, 455 (0.82%) required > 48 h of postoperative ventilation, 221 (0.40%) suffered a cardiac arrest, 524 (0.95%) had septic shock, and 525 (0.96%) died within 30 days of their index surgery. In total, 1224 (2.2%) patients met the definition for postoperative ICRU. In comparison to patients who did not require ICRU, patients who required ICRU were more likely to be ≥ 70 years old (63.3% vs. 40.0%, p < 0.001); had a higher ASA score (III/IV 83.7% vs. 57.9%, p < 0.001); were more likely to have comorbidities such as diabetes (28.7% vs. 18.9%, p < 0.001), cardiorespiratory disease (28.2% vs. 10.5%, p < 0.001), and hypertension (70.5% vs. 52.4%, p < 0.001); and were more likely to be functionally dependent (7.1% vs. 1.5%, p < 0.001). Patients who required ICRU also had more advanced tumors (locally advanced T-stage, 63.7% vs. 58.5%, p < 0.001) and were more likely to undergo open surgery (32.9% vs. 19.6%, p < 0.001) (Table 1).

Table 1

Comparison of patients who required postoperative Intensive Care Resource Utilization to those who did not

Variable	ICRU (−) n = 53,669	ICRU (+) n = 1224	p value
Age ≥ 70, n (%)	21,488 (40.0)	775 (63.3)	< 0.001
Male sex, n (%)	27,728 (51.6)	777 (63.4)	< 0.001
ASA score III/IV, n (%)	31,078 (57.9)	1025 (83.7)	< 0.001
Obesity (BMI ≥ 30), n (%)	18,985 (35.3)	448 (36.6)	0.39
Diabetes, n (%)	10,171 (18.9)	352 (28.7)	< 0.001
Smoking, n (%)	6760 (12.5)	214 (17.4)	< 0.001
Cardiorespiratory disease, n (%)	5687 (10.5)	346 (28.2)	< 0.001
Hypertension, n (%)	28,135 (52.4)	863 (70.5)	< 0.001
Bleeding disorder, n (%)	1232 (2.2)	59 (4.8)	< 0.001
Preoperative weight loss, n (%)	1874 (3.4)	63 (5.1)	0.002
Functionally dependent, n (%)	847 (1.5)	87 (7.1)	p < 0.001
Procedure, n (%)	-	-	< 0.001
Right colectomy	13,858 (25.8)	358 (29.2)	-
Left colectomy	38,616 (71.9)	826 (67.4)	-
Total abdominal colectomy	1195 (2.2)	40 (3.2)	-
Preoperative chemotherapy, n (%)	5803 (10.8)	88 (7.1)	< 0.001
T-stage, n (%)	-	-	< 0.001
Early (T1/2)	22,288 (41.5)	445 (36.3)	-
Advanced (T3/4)	31,381 (58.5)	779 (63.7)	-
N-stage, n (%)	-	-	0.84
Negative (N0)	35,552 (66.3)	807 (66.0)	-
Positive (N1/2)	18,117 (33.7)	417 (34.0)	-
Distant metastasis, n (%)	-	-	0.23
Yes	971 (1.8)	25 (2.0)	-
No	45,337 (84.4)	1012 (82.6)	-
Unknown	7361 (13.7)	187 (15.2)	-
Operative approach, n (%)	-	-	< 0.001
Open surgery	10,559 (19.6)	403 (32.9)	-
MIS	43,110 (80.4)	821 (67.1)	-
Bowel preparation, n (%)	43,248 (80.5)	883 (72.1)	< 0.001

ICRU Intensive Care Resource Utilization; ASA American Society of Anesthesiologists; BMI body mass index; MIS minimally invasive surgery

Using stepwise selection, 11 candidate variables were assessed on multiple logistic regression. All variables were significantly associated with the outcome (Table 2). The original model demonstrated good discrimination (AUC = 0.75) and calibration (goodness-of-fit, p = 0.12). Six variables were retained in the final model based on the Wald chi-squared statistic demonstrating an individual contribution to the model of > 5%: age, sex, ASA score, cardiorespiratory disease, functional status, and operative approach (Table 2). Collectively, these 6 variables contributed 91% to the original prediction model, while the 5 excluded variables together contributed 9%. A restricted model was then fit with the 6 retained variables: age (≥ 70; OR 1.90, 95% CI 1.68–2.14), sex (male; OR 1.73, 95% CI 1.54–1.95), ASA score (III/IV; OR 2.52, 95% CI 2.15–2.95), cardiorespiratory disease (yes; OR 2.22, 95% CI 1.94–2.53), functional status (dependent; OR 2.81, 95% CI 2.22–3.56), and operative approach (open surgery; OR 1.70, 95% CI 1.51–1.93) (Table 3). The restricted model maintained good discrimination (AUC = 0.73) and calibration (goodness-of-fit, p = 0.072) (Fig. 1a). Using the regression coefficients, a clinical risk score for postoperative ICRU was developed (Table 3). Based on the possible sum of points, the risk of ICRU ranged from 0.3 (0 points) to 19.0% (8 points) (Table 4).

Table 2

Original prediction model for Intensive Care Resource Utilization, arranged by individual contribution (Wald chi-square) to the model

Variable	OR	95% CI	Wald chi-square	Contribution (%)
Cardiorespiratory disease	2.05	1.79–2.34	382.0	36
ASA score III/IV	2.25	1.91–2.64	235.4	22
Age ≥ 70	1.89	1.67–2.15	109.9	10
Functionally dependent	2.75	2.17–3.49	81.6	8
Male sex	1.70	1.51–1.92	81.0	8
Open surgery	1.67	1.48–1.89	73.4	7
Bowel preparation	0.69	0.61–0.79	31.0	3
Smoking	1.56	1.34–1.83	27.0	3
Hypertension	1.31	1.14–1.50	20.5	2
Steroid use	1.46	1.12–1.90	7.9	1
Diabetes	1.18	1.03–1.35	6.1	1

ASA American Society of Anesthesiologists

Table 3

Final prediction model for Intensive Care Resource Utilization with point assignment

Variable	Coefficient	OR	95% CI	Points
Age ≥ 70	0.63	1.90	1.68–2.14	≥ 70 = 1, < 70 = 0
Male sex	0.55	1.73	1.54–1.95	Male = 1, Female = 0
ASA score III/IV	0.92	2.52	2.15–2.95	III/IV = 1.5, I/II = 0
Cardiorespiratory disease	0.80	2.22	1.94–2.53	Yes = 1.5, No = 0
Functionally dependent	1.03	2.81	2.22–3.56	Yes = 2, No = 0
Open surgery	0.53	1.70	1.51–1.93	Open = 1, MIS = 0

ASA American Society of Anesthesiologists; MIS minimally invasive surgery

Fig. 1

Receiver operating characteristic curve for prediction model for postoperative Intensive Care Resource Utilization. a Training set. b Test set

Table 4

Clinical risk score for 30-day postoperative Intensive Care Resource Utilization

Risk score points	Risk of ICRU (%)	Risk of ICRU (sensitivity analysis)a (%)
0	0.3	0.7
1	0.8	1.6
1.5	0.9	1.8
2	1.5	3.0
2.5	1.9	3.7
3	2.8	6.1
3.5	3.7	6.2
4	5.0	8.4
4.5	5.7	9.7
5	6.8	10.8
5.5	7.6	13.9
6	11.9	17.2
6.5	14.3	18.6
7	15.5	22.7
8	19.0	28.6

Age, ≥ 70 = 1; < 70 = 0

Sex, male = 1; female = 0

Operative approach: open surgery = 1; MIS = 0

ASA score: III/IV = 1.5; I/II = 0

Cardiorespiratory disease: yes = 1.5; no = 0

Functional status: dependent = 2; independent = 0

aIncludes pulmonary embolism, pneumonia, and myocardial infarction in composite outcome

ICRU Intensive Care Resource Utilization; MIS minimally invasive surgery; ASA American Society of Anesthesiologists

Given that the final clinical risk score did not include any colectomy-targeted variables, patients from the ACS-NSQIP general PUF who did not have a match within the colectomy-targeted PUF (initially excluded for generation of the training set) could be used for validation. The test set included 33,838 patients who underwent an elective colectomy for colon cancer. With respect to the 6 variables included in the clinical risk score, the training set and test set were reasonably similar (Table 5). A total of 828 (2.5%) patients required 30-day postoperative ICRU in the test set. In applying the training set final prediction model equation to the test set data, the model demonstrated good discrimination (AUC = 0.73) (Fig. 1b).

Table 5

Comparison of variables included in the final prediction model between training set and test set

Variable	Training set, n = 54,893	Test set, n = 33,838
Age ≥ 70, n (%)	22,263 (40.6)	14,262 (42.2)
Male sex, n (%)	28,505 (51.9)	17,573 (51.9)
ASA score III/IV, n (%)	32,103 (58.5)	19,602 (57.9)
Cardiorespiratory disease, n (%)	6033 (11.0)	4073 (12.0)
Functionally dependent, n (%)	934 (1.7)	681 (2.0)
Operative approach, n (%)	-	-
Open surgery	10,962 (20.0)	12,119 (35.8)
MIS	43,931 (80.0)	21,719 (64.2)

MIS minimally invasive surgery

On sensitivity analysis, the incidence of 30-day postoperative ICRU increased to 4.0% with the inclusion of pulmonary embolism, pneumonia, and myocardial infarction. Model development retained the same 6 variables, and the regression coefficients were similarly distributed to those of the original prediction model to allow for the same point assignment. The sensitivity analysis model maintained good predictive capabilities (AUC = 0.71), and the range of ICRU risk is presented in Table 4.

In total, 27 patients at three McGill University–affiliated hospitals were scheduled for an elective colon cancer resection during the 5-week period affected by the COVID-19 pandemic: 17 patients had their planned operation, while 10 patients had their surgery plans altered due to COVID-19 (7 patients had their operation delayed and 3 patients had their care transferred to a less acute-phase COVID hospital). The median clinical risk score was higher among patients who had their surgery plans altered compared with that among those who had their planned operation (2.5 (1.0–6.0) vs. 1.0 (0.0–2.0), p = 0.004). All three patients with a clinical risk score ≥ 6 (risk of ICRU = 11.9%) had their operations delayed due to COVID-19. None of the operated patients required postoperative ICRU.

Discussion

As the COVID-19 pandemic continues to unfold across North America, the delivery of quality and timely surgical care will remain a challenge. Patients with colon cancer present a particular burden, given the relatively high incidence of the disease, the potential for patients to become acutely symptomatic, and the risk of progression with delays to surgery [4]. Furthermore, no well-established alternative treatment strategy exists for colon cancer, rendering surgery a medically necessary and time-sensitive procedure during the COVID-19 pandemic [5].

In the present study, we have developed and validated a prediction model and clinical risk score for 30-day postoperative ICRU after colon cancer surgery. While other outcomes, such as postoperative complications and length of stay, are important concerns as well, the need for ICRU is one of the most current pressing considerations when contemplating medically necessary cancer surgery. In the American College of Surgeons’ recently published guidelines for the triage of cancer surgery patients, they stated that “for elective cases with a high likelihood of postoperative ICU or respirator utilization, it will be more imperative that the risk of delay to the individual patient is balanced against the imminent availability of these resources for patients with COVID-19” [3]. Using multicenter data from ACS-NSQIP, we reported an overall postoperative ICRU of 2.2% after colon cancer surgery. However, based on a variety of easily obtainable clinical preoperative factors, the risk for ICRU ranged from 0.3 to 19.0%.

Depending on a hospital’s available resources and current phase in the COVID-19 response, this clinical risk score can be used as a tool for surgeons and hospital administrators when deciding on an acceptable postoperative ICRU threshold to proceed with surgery during this dynamic crisis. In reviewing our own institutional data, we observed a higher median clinical risk score among those patients whose surgery plans were altered due to COVID-19. In our hospital system, decisions regarding urgent elective cancer cases are made by a multidisciplinary leadership team on a daily basis [12]. Factors taken into consideration at these daily meetings are available critical resources, urgency of surgery, lack of other suitable treatment alternatives, likelihood to require ICU admission, and risk of acquiring COVID-19 that would result in a poor outcome. Thus, while this small cohort represents only a 5-week snapshot in time and other factors are likely contributory, it highlights the current reality of ICRU importance when making operative decisions during the pandemic and demonstrates the applicability of the clinical risk score.

The 6 variables included in the clinical risk score were mainly patient-level factors, reflecting their overall health status and comorbidities. Many of the included factors are also associated with worse outcomes with SARS-CoV-2 infection, such as older age and comorbidities [7]. Thus, in addition to balancing the need for postoperative ICRU, these factors are also relevant when considering the potential exposure to SARS-CoV-2 associated with hospitalization [13]. Interestingly, none of the cancer-specific variables was included in the final risk score, including the stage of the cancer. This suggests that surgeons may reassuringly prioritize more advanced tumors, so long as the patient’s preoperative functional and medical status is acceptable according to the clinical risk score. Open surgery was also associated with ICRU, and may reflect case complexity. While the majority of colon cancer cases across the USA are performed via an MIS approach [14], open surgery may still be employed in more difficult cases, such as re-operative abdomens. It is less likely that open surgery reflected an intraoperative complication, as cases of unplanned conversion from MIS to open surgery were analyzed according to their initial approach.

There are several strengths to the current study and clinical risk score. The major advantage is the ability to use this clinical risk score at the point of care. The prediction model includes 6 variables, all of which can be easily obtained and/or estimated based on the patient’s medical history and planned operation. The transformation of the prediction model into a simple clinical risk score also allows for easy use when making clinical decisions regarding operative candidacy during COVID-19. The clinical risk score may equally be useful during the recovery phase of the pandemic when operative services resume and a back-log of colon cancer cases needs to be prioritized. The prediction model demonstrated good discrimination and predictive capabilities (AUC = 0.73). The strong performance may be attributed to the robust and data-driven approach to the method of candidate variable selection into the final restricted model. The model was also validated in a separate cohort of similar patients, which is important to assess for any statistical deficiencies during model development and to evaluate the generalizability of the prediction model [15]. Finally, data collection for ACS-NSQIP is prospectively performed by well-trained and audited reviewers who all use detailed prescribed definitions for each variable [16].

There are also several important limitations that must be considered when interpreting the results of this study. First, ICU admission is not a variable in the ACS-NSQIP database, and ICRU was defined as a composite outcome including several pre-defined variables. Therefore, even though the outcomes selected—unplanned reintubation, ventilator requirements > 48 h, cardiac arrest, septic shock, and death—almost always include ICU care at some point in their course, there may be an element of outcome misclassification. To account for this, a sensitivity analysis was performed to include less severe postoperative events that may still require ICU care. The model retained good predictive capabilities on sensitivity analysis. Second, the prediction model was validated in a separate cohort of patients who did not have a match within the colectomy-targeted PUF. There may be some unmeasured differences between the training and test sets, as hospitals that choose to contribute data to the colectomy-targeted PUF may differ from those who do not. Lastly, the clinical risk score is not applicable to rectal cancer patients. In addition to rectal cancer patients having a different risk set, their clinicians can choose from a gamut of therapies including neoadjuvant radiotherapy, chemotherapy, or a combination of these modalities; all of which may downstage the tumor with an acceptable delay to surgery [17]. Patients with colon cancer are typically more in need of immediate surgery. Future work should aim to prospectively validate the clinical risk score in a larger cohort of patients during the COVID-19 era, to better evaluate its application in clinical decision-making during these resource-limited times.

Conclusion

We have developed and validated a prediction model for 30-day postoperative Intensive Care Resource Utilization after colon cancer surgery. The model demonstrated good predictive capabilities and was transformed into a simple clinical risk score to allow for easy use at the point of care. The clinical risk score may aid surgeons and hospital administrators in identifying appropriate colon cancer patients for medically necessary surgery during the COVID-19 pandemic, in an attempt to minimize the use of limited ICU resources.