Mark E Cohen1, Yaoming Liu2, Clifford Y Ko3, Bruce L Hall4. 1. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL. Electronic address: markcohen@facs.org. 2. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL. 3. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, University of California Los Angeles, David Geffen School of Medicine, Los Angeles, CA; VA Greater Los Angeles Healthcare System, Los Angeles, CA. 4. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, Washington University in St Louis, St Louis, MO; Center for Health Policy and the Olin Business School, Washington University in St Louis, St Louis, MO; John Cochran Veterans Affairs Medical Center, St Louis, MO; BJC Healthcare, St Louis, MO.
Abstract
BACKGROUND: The American College of Surgeons NSQIP offers a Surgical Risk Calculator (SRC) that provides detailed, patient-level, risk assessments for many adverse outcomes to surgeons, patients, and the general public. The SRC calculator was designed to help guide discussion and decisions by providing generally applicable (not hospital-specific) information about surgical risk using easily understood and broadly available preoperative variables. Although large, internal evaluations have shown that the SRC has good accuracy (model discrimination and calibration), external validations have been inconsistent and tend to favor a conclusion of inadequate performance. STUDY DESIGN: External studies, attempting to validate the SRC, were examined with respect to 3 design features: sample size (small samples reduce reliability), case-mix homogeneity (homogeneity reduces discrimination); and number of institutions providing data (few institutions reduces generalizability). The impact of each feature was then examined in several sets of simulation studies. RESULTS: Each of the 3 design features has the potential to act as an artifactual cause for apparent SRC predictive failure. In addition, demonstrations that SRC estimates are inferior to those from models that use additional (sometimes operation-specific) predictor variables were seen as not relevant with respect to the SRC's intended scope. CONCLUSIONS: The SRC predictive failures, reported by studies with the described design limitations, should not be misunderstood as disqualifying the SRC as an accurate and appropriate tool for its intended purpose of providing a general purpose risk calculator, applicable across many surgical domains, using easily understood and generally available predictive information.
BACKGROUND: The American College of Surgeons NSQIP offers a Surgical Risk Calculator (SRC) that provides detailed, patient-level, risk assessments for many adverse outcomes to surgeons, patients, and the general public. The SRC calculator was designed to help guide discussion and decisions by providing generally applicable (not hospital-specific) information about surgical risk using easily understood and broadly available preoperative variables. Although large, internal evaluations have shown that the SRC has good accuracy (model discrimination and calibration), external validations have been inconsistent and tend to favor a conclusion of inadequate performance. STUDY DESIGN: External studies, attempting to validate the SRC, were examined with respect to 3 design features: sample size (small samples reduce reliability), case-mix homogeneity (homogeneity reduces discrimination); and number of institutions providing data (few institutions reduces generalizability). The impact of each feature was then examined in several sets of simulation studies. RESULTS: Each of the 3 design features has the potential to act as an artifactual cause for apparent SRC predictive failure. In addition, demonstrations that SRC estimates are inferior to those from models that use additional (sometimes operation-specific) predictor variables were seen as not relevant with respect to the SRC's intended scope. CONCLUSIONS: The SRC predictive failures, reported by studies with the described design limitations, should not be misunderstood as disqualifying the SRC as an accurate and appropriate tool for its intended purpose of providing a general purpose risk calculator, applicable across many surgical domains, using easily understood and generally available predictive information.
Authors: Kota Sahara; Anghela Z Paredes; Katiuscha Merath; Diamantis I Tsilimigras; Fabio Bagante; Francesca Ratti; Hugo P Marques; Olivier Soubrane; Eliza W Beal; Vincent Lam; George A Poultsides; Irinel Popescu; Sorin Alexandrescu; Guillaume Martel; Workneh Aklile; Alfredo Guglielmi; Tom Hugh; Luca Aldrighetti; Itaru Endo; Timothy M Pawlik Journal: J Gastrointest Surg Date: 2019-04-01 Impact factor: 3.452
Authors: Jason B Liu; Julie A Sosa; Raymon H Grogan; Yaoming Liu; Mark E Cohen; Clifford Y Ko; Bruce L Hall Journal: JAMA Surg Date: 2018-01-17 Impact factor: 14.766
Authors: J Madison Hyer; Diamantis I Tsilimigras; Anghela Z Paredes; Kota Sahara; Susan White; Timothy M Pawlik Journal: World J Surg Date: 2020-01 Impact factor: 3.352
Authors: Jeffrey B Walker; Augustyna Gogoj; Brian D Saunders; Daniel J Canter; Kathleen Lehman; Jay D Raman Journal: Int Urol Nephrol Date: 2019-06-10 Impact factor: 2.370
Authors: Scott K Sherman; Jennifer E Hrabe; Emily Huang; John W Cromwell; John C Byrn Journal: J Gastrointest Surg Date: 2018-04-23 Impact factor: 3.452
Authors: Tyler J Loftus; Patrick J Tighe; Amanda C Filiberto; Philip A Efron; Scott C Brakenridge; Alicia M Mohr; Parisa Rashidi; Gilbert R Upchurch; Azra Bihorac Journal: JAMA Surg Date: 2020-02-01 Impact factor: 14.766
Authors: Neel P Chudgar; Shi Yan; Meier Hsu; Kay See Tan; Katherine D Gray; Daniela Molena; Tamar Nobel; Prasad S Adusumilli; Manjit Bains; Robert J Downey; James Huang; Bernard J Park; Gaetano Rocco; Valerie W Rusch; Smita Sihag; David R Jones; James M Isbell Journal: Ann Thorac Surg Date: 2020-10-16 Impact factor: 4.330