Yaoming Liu1, Mark E Cohen2, Bruce L Hall3, Clifford Y Ko4, Karl Y Bilimoria5. 1. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL. 2. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL. Electronic address: markcohen@facs.org. 3. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, Washington University in St. Louis; Center for Health Policy and the Olin Business School at Washington University in St Louis; John Cochran Veterans Affairs Medical Center; and BJC Healthcare, St Louis, MO. 4. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, University of California Los Angeles David Geffen School of Medicine and the VA Greater Los Angeles Healthcare System, Los Angeles, CA. 5. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Surgical Outcomes and Quality Improvement Center (SOQIC), Department of Surgery and Center for Healthcare Studies, Feinberg School of Medicine and Northwestern Medicine, Northwestern University, Chicago, IL.
Abstract
BACKGROUND: The American College of Surgeon (ACS) NSQIP Surgical Risk Calculator has been widely adopted as a decision aid and informed consent tool by surgeons and patients. Previous evaluations showed excellent discrimination and combined discrimination and calibration, but model calibration alone, and potential benefits of recalibration, were not explored. Because lack of calibration can lead to systematic errors in assessing surgical risk, our objective was to assess calibration and determine whether spline-based adjustments could improve it. STUDY DESIGN: We evaluated Surgical Risk Calculator model calibration, as well as discrimination, for each of 11 outcomes modeled from nearly 3 million patients (2010 to 2014). Using independent random subsets of data, we evaluated model performance for the Development (60% of records), Validation (20%), and Test (20%) datasets, where prediction equations from the Development dataset were recalibrated using restricted cubic splines estimated from the Validation dataset. We also evaluated performance on data subsets composed of higher-risk operations. RESULTS: The nonrecalibrated Surgical Risk Calculator performed well, but there was a slight tendency for predicted risk to be overestimated for lowest- and highest-risk patients and underestimated for moderate-risk patients. After recalibration, this distortion was eliminated, and p values for miscalibration were most often nonsignificant. Calibration was also excellent for subsets of higher-risk operations, though observed calibration was reduced due to instability associated with smaller sample sizes. CONCLUSIONS: Performance of NSQIP Surgical Risk Calculator models was shown to be excellent and improved with recalibration. Surgeons and patients can rely on the calculator to provide accurate estimates of surgical risk.
BACKGROUND: The American College of Surgeon (ACS) NSQIP Surgical Risk Calculator has been widely adopted as a decision aid and informed consent tool by surgeons and patients. Previous evaluations showed excellent discrimination and combined discrimination and calibration, but model calibration alone, and potential benefits of recalibration, were not explored. Because lack of calibration can lead to systematic errors in assessing surgical risk, our objective was to assess calibration and determine whether spline-based adjustments could improve it. STUDY DESIGN: We evaluated Surgical Risk Calculator model calibration, as well as discrimination, for each of 11 outcomes modeled from nearly 3 million patients (2010 to 2014). Using independent random subsets of data, we evaluated model performance for the Development (60% of records), Validation (20%), and Test (20%) datasets, where prediction equations from the Development dataset were recalibrated using restricted cubic splines estimated from the Validation dataset. We also evaluated performance on data subsets composed of higher-risk operations. RESULTS: The nonrecalibrated Surgical Risk Calculator performed well, but there was a slight tendency for predicted risk to be overestimated for lowest- and highest-risk patients and underestimated for moderate-risk patients. After recalibration, this distortion was eliminated, and p values for miscalibration were most often nonsignificant. Calibration was also excellent for subsets of higher-risk operations, though observed calibration was reduced due to instability associated with smaller sample sizes. CONCLUSIONS: Performance of NSQIP Surgical Risk Calculator models was shown to be excellent and improved with recalibration. Surgeons and patients can rely on the calculator to provide accurate estimates of surgical risk.
Authors: T Sammour; L Cohen; A I Karunatillake; M Lewis; M J Lawrence; A Hunter; J W Moore; M L Thomas Journal: Tech Coloproctol Date: 2017-10-28 Impact factor: 3.781
Authors: Scott K Sherman; Jennifer E Hrabe; Emily Huang; John W Cromwell; John C Byrn Journal: J Gastrointest Surg Date: 2018-04-23 Impact factor: 3.452
Authors: Ira L Leeds; Joseph K Canner; Jonathan E Efron; Nita Ahuja; Elliott R Haut; Elizabeth C Wick; Fabian M Johnston Journal: J Surg Res Date: 2017-09-18 Impact factor: 2.192
Authors: Ruchika Goel; Cassandra D Josephson; Eshan U Patel; Molly R Petersen; Sarah Makhani; Steven M Frank; Paul M Ness; Evan M Bloch; Eric A Gehrie; Parvez M Lokhandwala; Marianne M Nellis; Oliver Karam; Beth H Shaz; Ravi M Patel; Aaron A R Tobian Journal: Pediatrics Date: 2020-03-20 Impact factor: 7.124