John R Bergquist1, Cornelius A Thiels1, David A Etzioni2, Elizabeth B Habermann3, Robert R Cima4. 1. Department of Surgery, Mayo Clinic, Rochester, MN; Robert D and Patricia E Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN. 2. Robert D and Patricia E Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN; Division of Colon and Rectal Surgery, Mayo Clinic, Scottsdale, AZ. 3. Robert D and Patricia E Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN; Division of Health Care Policy and Research, Mayo Clinic, Rochester, MN. 4. Robert D and Patricia E Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN; Division of Colon and Rectal Surgery, Mayo Clinic, Rochester, MN. Electronic address: cima.robert@mayo.edu.
Abstract
BACKGROUND: Colorectal surgical site infections (C-SSIs) are a major source of postoperative morbidity. Institutional C-SSI rates are modeled and scrutinized, and there is increasing movement in the direction of public reporting. External validation of C-SSI risk prediction models is lacking. Factors governing C-SSI occurrence are complicated and multifactorial. We hypothesized that existing C-SSI prediction models have limited ability to accurately predict C-SSI in independent data. STUDY DESIGN: Colorectal resections identified from our institutional ACS-NSQIP dataset (2006 to 2014) were reviewed. The primary outcome was any C-SSI according to the ACS-NSQIP definition. Emergency cases were excluded. Published C-SSI risk scores: the National Nosocomial Infection Surveillance (NNIS), Contamination, Obesity, Laparotomy, and American Society of Anesthesiologists (ASA) class (COLA), Preventie Ziekenhuisinfecties door Surveillance (PREZIES), and NSQIP-based models were compared with receiver operating characteristic (ROC) analysis to evaluate discriminatory quality. RESULTS: There were 2,376 cases included, with an overall C-SSI rate of 9% (213 cases). None of the models produced reliable and high quality C-SSI predictions. For any C-SSI, the NNIS c-index was 0.57 vs 0.61 for COLA, 0.58 for PREZIES, and 0.62 for NSQIP: all well below the minimum "reasonably" predictive c-index of 0.7. Predictions for superficial, deep, and organ space SSI were similarly poor. CONCLUSIONS: Published C-SSI risk prediction models do not accurately predict C-SSI in our independent institutional dataset. Application of externally developed prediction models to any individual practice must be validated or modified to account for institution and case-mix specific factors. This questions the validity of using externally or nationally developed models for "expected" outcomes and interhospital comparisons.
BACKGROUND: Colorectal surgical site infections (C-SSIs) are a major source of postoperative morbidity. Institutional C-SSI rates are modeled and scrutinized, and there is increasing movement in the direction of public reporting. External validation of C-SSI risk prediction models is lacking. Factors governing C-SSI occurrence are complicated and multifactorial. We hypothesized that existing C-SSI prediction models have limited ability to accurately predict C-SSI in independent data. STUDY DESIGN: Colorectal resections identified from our institutional ACS-NSQIP dataset (2006 to 2014) were reviewed. The primary outcome was any C-SSI according to the ACS-NSQIP definition. Emergency cases were excluded. Published C-SSI risk scores: the National Nosocomial Infection Surveillance (NNIS), Contamination, Obesity, Laparotomy, and American Society of Anesthesiologists (ASA) class (COLA), Preventie Ziekenhuisinfecties door Surveillance (PREZIES), and NSQIP-based models were compared with receiver operating characteristic (ROC) analysis to evaluate discriminatory quality. RESULTS: There were 2,376 cases included, with an overall C-SSI rate of 9% (213 cases). None of the models produced reliable and high quality C-SSI predictions. For any C-SSI, the NNIS c-index was 0.57 vs 0.61 for COLA, 0.58 for PREZIES, and 0.62 for NSQIP: all well below the minimum "reasonably" predictive c-index of 0.7. Predictions for superficial, deep, and organ space SSI were similarly poor. CONCLUSIONS: Published C-SSI risk prediction models do not accurately predict C-SSI in our independent institutional dataset. Application of externally developed prediction models to any individual practice must be validated or modified to account for institution and case-mix specific factors. This questions the validity of using externally or nationally developed models for "expected" outcomes and interhospital comparisons.
Authors: Robert R Cima; John R Bergquist; Kristine T Hanson; Cornelius A Thiels; Elizabeth B Habermann Journal: J Gastrointest Surg Date: 2017-05-03 Impact factor: 3.452
Authors: Matthew G Mullen; Robert B Hawkins; Lily E Johnston; Puja M Shah; Florence E Turrentine; Traci L Hedrick; Charles M Friel Journal: Dis Colon Rectum Date: 2018-05 Impact factor: 4.585
Authors: M R Weiser; M Gonen; S Usiak; T Pottinger; P Samedy; D Patel; S Seo; J J Smith; J G Guillem; L Temple; G M Nash; P B Paty; A Baldwin-Medsker; C E Cheavers; J Eagan; J Garcia-Aguilar Journal: Br J Surg Date: 2018-07-04 Impact factor: 6.939
Authors: Riccardo Lemini; Aaron C Spaulding; James M Naessens; Zhuo Li; Amit Merchea; Julia E Crook; David W Larson; Dorin T Colibaseanu Journal: Int J Colorectal Dis Date: 2018-07-21 Impact factor: 2.571
Authors: Anuradha R Bhama; Farwa Batool; Stacey D Collins; Jane Ferraro; Robert K Cleary Journal: J Gastrointest Surg Date: 2017-10-02 Impact factor: 3.452
Authors: Nicholas P McKenna; Katherine A Bews; Robert R Cima; Cynthia S Crowson; Elizabeth B Habermann Journal: J Gastrointest Surg Date: 2019-06-26 Impact factor: 3.452
Authors: Sarah S Jackson; Surbhi Leekha; Laurence S Magder; Lisa Pineles; Deverick J Anderson; William E Trick; Keith F Woeltje; Keith S Kaye; Timothy J Lowe; Anthony D Harris Journal: Clin Infect Dis Date: 2017-09-01 Impact factor: 9.079