Predictive performance of the American College of Surgeons universal risk calculator in neurosurgical patients.

OBJECTIVE The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) universal Surgical Risk Calculator is an online decision-support tool that uses patient characteristics to estimate the risk of adverse postoperative events. Further validation of this risk calculator in the neurosurgical population is needed; therefore, the object of this study was to assess the predictive performance of the ACS NSQIP Surgical Risk Calculator in neurosurgical patients treated at a tertiary care center. METHODS A single-center retrospective review of 1006 neurosurgical patients treated in the period from September 2011 through December 2014 was performed. Individual patient characteristics were entered into the NSQIP calculator. Predicted complications were compared with actual occurrences identified through chart review and administrative quality coding data. Statistical models were used to assess the predictive performance of risk scores. Traditionally, an ideal risk prediction model demonstrates good calibration and strong discrimination when comparing predicted and observed events. RESULTS The ACS NSQIP risk calculator demonstrated good calibration between predicted and observed risks of death (p = 0.102), surgical site infection (SSI; p = 0.099), and venous thromboembolism (VTE; p = 0.164) Alternatively, the risk calculator demonstrated a statistically significant lack of calibration between predicted and observed risk of pneumonia (p = 0.044), urinary tract infection (UTI; p < 0.001), return to the operating room (p < 0.001), and discharge to a rehabilitation or nursing facility (p < 0.001). The discriminative performance of the risk calculator was assessed using the c-statistic. Death (c-statistic 0.93), UTI (0.846), and pneumonia (0.862) demonstrated strong discriminative performance. Discharge to a rehabilitation facility or nursing home (c-statistic 0.794) and VTE (0.767) showed adequate discrimination. Return to the operating room (c-statistic 0.452) and SSI (0.556) demonstrated poor discriminative performance. The risk prediction model was both well calibrated and discriminative only for 30-day mortality. CONCLUSIONS This study illustrates the importance of validating universal risk calculators in specialty-specific surgical populations. The ACS NSQIP Surgical Risk Calculator could be used as a decision-support tool for neurosurgical informed consent with respect to predicted mortality but was poorly predictive of other potential adverse events and clinical outcomes.