Bruce Lee Hall1, Kristopher M Huffman2, Barton H Hamilton3, Jennifer L Paruch4, Lynn Zhou2, Karen E Richards2, Mark E Cohen2, Clifford Y Ko5. 1. Department of Surgery, Washington University in St Louis; the Center for Health Policy at Washington University in St Louis; St. Louis Veterans Affairs Medical Center; and BJC Healthcare, St Louis, MO; Olin Business School at Washington University in St Louis, St Louis, MO; Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL. Electronic address: hallb@wustl.edu. 2. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL. 3. Olin Business School at Washington University in St Louis, St Louis, MO. 4. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, University of Chicago Pritzker School of Medicine, Chicago, IL. 5. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, University of California Los Angeles David Geffen School of Medicine and the VA Greater Los Angeles Healthcare System, Los Angeles, CA.
Abstract
BACKGROUND: There is increasing interest in profiling the quality of individual medical providers. Valid assessment of individuals should highlight improvement opportunities, but must be considered in the context of limitations. STUDY DESIGN: High quality clinical data from the American College of Surgeons NSQIP, gathered in accordance with strict policies and specifications, was used to construct individual surgeon-level assessments. There were 39,976 cases evaluated, performed by 197 surgeons across 9 hospitals. Both 2-level (cases by surgeon) and 3-level (cases by surgeon by hospital) risk-adjusted, hierarchical regression analyses were performed. Outcomes were 30-day postoperative morbidity, surgical site infection, and mortality. Surgeon performance was compared in both absolute and relative terms. "Signal-to-noise" reliability was calculated for surgeons and models. Projected case requirements for reliability levels were generated. RESULTS: Surgeon performances could be distinguished to different degrees: morbidity distinguished best, mortality least. Outliers could be identified for morbidity and infection, but not mortality. Reliability was also highest for morbidity and lowest for mortality. Even models with high overall reliability did not assess all providers reliably. Incorporating institutional effects had predictable effects: penalizing providers at "good" institutions, benefiting providers at "poor" institutions. CONCLUSIONS: Individual surgeon profiles can, at times, be distinguished with moderate or good reliability, but to different degrees in different models. Absolute and relative comparisons are feasible. Incorporating institutional level effects in individual provider modeling presents an interesting policy dilemma, appearing to benefit providers at "poor-performing" institutions, but penalizing those at "high-performing" ones. No portrayal of individual medical provider quality should be accepted without consideration of modeling rationale and, critically, reliability.
BACKGROUND: There is increasing interest in profiling the quality of individual medical providers. Valid assessment of individuals should highlight improvement opportunities, but must be considered in the context of limitations. STUDY DESIGN: High quality clinical data from the American College of Surgeons NSQIP, gathered in accordance with strict policies and specifications, was used to construct individual surgeon-level assessments. There were 39,976 cases evaluated, performed by 197 surgeons across 9 hospitals. Both 2-level (cases by surgeon) and 3-level (cases by surgeon by hospital) risk-adjusted, hierarchical regression analyses were performed. Outcomes were 30-day postoperative morbidity, surgical site infection, and mortality. Surgeon performance was compared in both absolute and relative terms. "Signal-to-noise" reliability was calculated for surgeons and models. Projected case requirements for reliability levels were generated. RESULTS: Surgeon performances could be distinguished to different degrees: morbidity distinguished best, mortality least. Outliers could be identified for morbidity and infection, but not mortality. Reliability was also highest for morbidity and lowest for mortality. Even models with high overall reliability did not assess all providers reliably. Incorporating institutional effects had predictable effects: penalizing providers at "good" institutions, benefiting providers at "poor" institutions. CONCLUSIONS: Individual surgeon profiles can, at times, be distinguished with moderate or good reliability, but to different degrees in different models. Absolute and relative comparisons are feasible. Incorporating institutional level effects in individual provider modeling presents an interesting policy dilemma, appearing to benefit providers at "poor-performing" institutions, but penalizing those at "high-performing" ones. No portrayal of individual medical provider quality should be accepted without consideration of modeling rationale and, critically, reliability.
Authors: David G Brauer; Kerri A Ohman; David P Jaques; Cheryl A Woolsey; Ningying Wu; Jingxia Liu; M B Majella Doyle; Ryan C Fields; William C Chapman; Steven M Strasberg; William G Hawkins Journal: J Am Coll Surg Date: 2017-10-19 Impact factor: 6.113
Authors: Jacob K Greenberg; Margaret A Olsen; John Poe; Christopher F Dibble; Ken Yamaguchi; Michael P Kelly; Bruce L Hall; Wilson Z Ray Journal: Spine (Phila Pa 1976) Date: 2021-09-01 Impact factor: 3.241
Authors: Ewen M Harrison; Thomas M Drake; Stephen O'Neill; Catherine A Shaw; O James Garden; Stephen J Wigmore Journal: BMJ Open Date: 2016-10-31 Impact factor: 2.692
Authors: Jeffrey Landercasper; Oluwadamilola M Fayanju; Lisa Bailey; Tiffany S Berry; Andrew J Borgert; Robert Buras; Steven L Chen; Amy C Degnim; Joshua Froman; Jennifer Gass; Caprice Greenberg; Starr Koslow Mautner; Helen Krontiras; Luis D Ramirez; Michelle Sowden; Barbara Wexelman; Lee Wilke; Roshni Rao Journal: Ann Surg Oncol Date: 2017-11-22 Impact factor: 5.344
Authors: Paul G Barnett; Josephine C Jacobs; Jeffrey G Jarvik; Roger Chou; Derek Boothroyd; Jeanie Lo; Andrea Nevedal Journal: JAMA Netw Open Date: 2020-07-01