Christopher M Quinn1, Karl Y Bilimoria2, Jeanette W Chung1, Clifford Y Ko3, Mark E Cohen4, Jonah J Stulberg5. 1. Surgical Outcomes and Quality Improvement Center, Department of Surgery, Northwestern University, Chicago, IL. 2. Surgical Outcomes and Quality Improvement Center, Department of Surgery, Northwestern University, Chicago, IL; Division of Optimal Research and Patient Care, American College of Surgeons, Chicago, IL. 3. Division of Optimal Research and Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, University of California, Los Angeles, Los Angeles, CA. 4. Division of Optimal Research and Patient Care, American College of Surgeons, Chicago, IL. 5. Surgical Outcomes and Quality Improvement Center, Department of Surgery, Northwestern University, Chicago, IL. Electronic address: jonah.stulberg@nm.org.
Abstract
BACKGROUND: Surgeon performance profiling is of great interest to surgeons, hospitals, health plans, and the public, yet efforts to date have been contested, with stakeholders at odds over the selection, reliability, and validity of metrics used. We sought to create surgeon-level comparative assessments within the Illinois Surgical Quality Improvement Collaborative. STUDY DESIGN: American College of Surgeons NSQIP data were obtained for 51 Illinois hospitals covering a 30-month period from 2014 to 2016. Surgeon-level, risk-adjusted outcomes rates were estimated from 3-level crossed random effects logistic regression models and classified as low, as expected, or high for each of 7 postoperative outcomes. Model intra-class correlations and provider-specific reliability statistics were calculated. RESULTS: A total of 123,141 cases were analyzed for 2,724 surgeons. Median provider case volume was 17 (interquartile range 4 to 54). Overall crude complication rates ranged from 0.62% to 7.14% across the 7 outcomes investigated. Surgeon-level variance estimates were low (intra-class correlation coefficients between 0.007 and 0.074). No performance outliers were detected for 3 of the outcomes measures, while a small number of outliers were identified for any morbidity (11 surgeons), surgical site infection (10 surgeons), death or serious morbidity (8 surgeons), and reoperation (1 surgeon). Among all physicians, median reliability was below 0.1 for each outcome. CONCLUSIONS: Few individual surgeon performance outliers could be detected in NSQIP clinical registry data for a statewide hospital collaborative over a 30-month period using postoperative patient outcomes. Low surgeon-specific case volumes and minimal variance between surgeons may limit the utility of American College of Surgeons NSQIP outcomes measures for individual profiling. Alternative metrics, such as process measures, patient experience, composite measures, or technical skill assessments should be explored for surgeon-level measurement.
BACKGROUND: Surgeon performance profiling is of great interest to surgeons, hospitals, health plans, and the public, yet efforts to date have been contested, with stakeholders at odds over the selection, reliability, and validity of metrics used. We sought to create surgeon-level comparative assessments within the Illinois Surgical Quality Improvement Collaborative. STUDY DESIGN: American College of Surgeons NSQIP data were obtained for 51 Illinois hospitals covering a 30-month period from 2014 to 2016. Surgeon-level, risk-adjusted outcomes rates were estimated from 3-level crossed random effects logistic regression models and classified as low, as expected, or high for each of 7 postoperative outcomes. Model intra-class correlations and provider-specific reliability statistics were calculated. RESULTS: A total of 123,141 cases were analyzed for 2,724 surgeons. Median provider case volume was 17 (interquartile range 4 to 54). Overall crude complication rates ranged from 0.62% to 7.14% across the 7 outcomes investigated. Surgeon-level variance estimates were low (intra-class correlation coefficients between 0.007 and 0.074). No performance outliers were detected for 3 of the outcomes measures, while a small number of outliers were identified for any morbidity (11 surgeons), surgical site infection (10 surgeons), death or serious morbidity (8 surgeons), and reoperation (1 surgeon). Among all physicians, median reliability was below 0.1 for each outcome. CONCLUSIONS: Few individual surgeon performance outliers could be detected in NSQIP clinical registry data for a statewide hospital collaborative over a 30-month period using postoperative patient outcomes. Low surgeon-specific case volumes and minimal variance between surgeons may limit the utility of American College of Surgeons NSQIP outcomes measures for individual profiling. Alternative metrics, such as process measures, patient experience, composite measures, or technical skill assessments should be explored for surgeon-level measurement.
Authors: William O Cooper; David A Spain; Oscar Guillamondegui; Rachel R Kelz; Henry J Domenico; Joseph Hopkins; Patricia Sullivan; Ilene N Moore; James W Pichert; Thomas F Catron; Lynn E Webb; Roger R Dmochowski; Gerald B Hickson Journal: JAMA Surg Date: 2019-09-01 Impact factor: 14.766