Kristopher M Huffman1, Mark E Cohen, Clifford Y Ko, Bruce Lee Hall. 1. *Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL †Department of Surgery, University of California Los Angeles David Geffen School of Medicine, and VA Greater Los Angeles Healthcare System, Los Angeles, CA; and ‡Department of Surgery, Washington University in St. Louis; Center for Health Policy and the Olin Business School at Washington University in St Louis; John Cochran Veterans Affairs Medical Center; and BJC Healthcare, St. Louis, MO.
Abstract
OBJECTIVE: To assess statistical reliability of hospital profiling models in ACS NSQIP (American College of Surgeons' National Surgical Quality Improvement Program). BACKGROUND: The ACS NSQIP January 2013 Semiannual Report provided risk-adjusted hospital quality assessments for 137 models. METHODS: Median reliability and percentage of hospitals achieving acceptable reliability were computed for each model. Average median reliability was computed across models with common outcomes. RESULTS: Median reliability varied across the 137 models, from a high of 0.91 for "All Cases Morbidity" to a low of 0.005 for "Procedure-Targeted Total Hip Arthroplasty Surgical Site Infection." Generally, reliability was greatest for models with larger sample sizes and higher outcome event rates. Among "Essentials" models, 72% attained a median reliability of 0.40 or more, and 24% of 0.70 or more. Among "Procedure-Targeted" models, 29% attained a median reliability of 0.40 or more, and 3% of 0.70 or more. Percentage of hospitals achieving an acceptable reliability of 0.40 ranged from 98% for "All Cases Morbidity" to 0% for "Procedure-Targeted Pancreatectomy Mortality." For Essentials models, average median reliability for each outcome, except mortality, was more than 0.40. However, for Procedure-Targeted models the average median was less than 0.40. CONCLUSIONS: For a large proportion of ACS NSQIP Essentials models, statistical reliability is adequate for assessing surgical quality and differentiating hospital performance. The Procedure-Targeted program is evolving in terms of statistical reliability, with promising results to date. These results also argue for broader discussions of statistical reliability in performance assessments for the profession.
OBJECTIVE: To assess statistical reliability of hospital profiling models in ACS NSQIP (American College of Surgeons' National Surgical Quality Improvement Program). BACKGROUND: The ACS NSQIP January 2013 Semiannual Report provided risk-adjusted hospital quality assessments for 137 models. METHODS: Median reliability and percentage of hospitals achieving acceptable reliability were computed for each model. Average median reliability was computed across models with common outcomes. RESULTS: Median reliability varied across the 137 models, from a high of 0.91 for "All Cases Morbidity" to a low of 0.005 for "Procedure-Targeted Total Hip Arthroplasty Surgical Site Infection." Generally, reliability was greatest for models with larger sample sizes and higher outcome event rates. Among "Essentials" models, 72% attained a median reliability of 0.40 or more, and 24% of 0.70 or more. Among "Procedure-Targeted" models, 29% attained a median reliability of 0.40 or more, and 3% of 0.70 or more. Percentage of hospitals achieving an acceptable reliability of 0.40 ranged from 98% for "All Cases Morbidity" to 0% for "Procedure-Targeted Pancreatectomy Mortality." For Essentials models, average median reliability for each outcome, except mortality, was more than 0.40. However, for Procedure-Targeted models the average median was less than 0.40. CONCLUSIONS: For a large proportion of ACS NSQIP Essentials models, statistical reliability is adequate for assessing surgical quality and differentiating hospital performance. The Procedure-Targeted program is evolving in terms of statistical reliability, with promising results to date. These results also argue for broader discussions of statistical reliability in performance assessments for the profession.
Authors: Sean M O'Brien; Jeffrey P Jacobs; David M Shahian; Marshall L Jacobs; J William Gaynor; Jennifer C Romano; Michael G Gaies; Kevin D Hill; John E Mayer; Sara K Pasquali Journal: Ann Thorac Surg Date: 2018-09-15 Impact factor: 4.330
Authors: Jacob K Greenberg; Margaret A Olsen; John Poe; Christopher F Dibble; Ken Yamaguchi; Michael P Kelly; Bruce L Hall; Wilson Z Ray Journal: Spine (Phila Pa 1976) Date: 2021-09-01 Impact factor: 3.241
Authors: Jacob K Greenberg; Margaret A Olsen; Christopher F Dibble; Justin K Zhang; Brenton H Pennicooke; Ken Yamaguchi; Michael P Kelly; Bruce L Hall; Wilson Z Ray Journal: Spine J Date: 2021-06-20 Impact factor: 4.166