BACKGROUND: Surgeons and hospitals are being increasingly assessed by third parties regarding surgical quality and outcomes, and much of this information is reported publicly. Our objective was to compare various methods used to classify hospitals as outliers in established surgical quality assessment programs by applying each approach to a single data set. METHODS: Using American College of Surgeons National Surgical Quality Improvement Program data (7/2008-6/2009), hospital risk-adjusted 30-day morbidity and mortality were assessed for general surgery at 231 hospitals (cases = 217,630) and for colorectal surgery at 109 hospitals (cases = 17,251). The number of outliers (poor performers) identified using different methods and criteria were compared. RESULTS: The overall morbidity was 10.3% for general surgery and 25.3% for colorectal surgery. The mortality was 1.6% for general surgery and 4.0% for colorectal surgery. Programs used different methods (logistic regression, hierarchical modeling, partitioning) and criteria (P < 0.01, P < 0.05, P < 0.10) to identify outliers. Depending on outlier identification methods and criteria employed, when each approach was applied to this single dataset, the number of outliers ranged from 7 to 57 hospitals for general surgery morbidity, 1 to 57 hospitals for general surgery mortality, 4 to 27 hospitals for colorectal morbidity, and 0 to 27 hospitals for colorectal mortality. CONCLUSIONS: There was considerable variation in the number of outliers identified using different detection approaches. Quality programs seem to be utilizing outlier identification methods contrary to what might be expected, thus they should justify their methodology based on the intent of the program (i.e., quality improvement vs. reimbursement). Surgeons and hospitals should be aware of variability in methods used to assess their performance as these outlier designations will likely have referral and reimbursement consequences.
BACKGROUND: Surgeons and hospitals are being increasingly assessed by third parties regarding surgical quality and outcomes, and much of this information is reported publicly. Our objective was to compare various methods used to classify hospitals as outliers in established surgical quality assessment programs by applying each approach to a single data set. METHODS: Using American College of Surgeons National Surgical Quality Improvement Program data (7/2008-6/2009), hospital risk-adjusted 30-day morbidity and mortality were assessed for general surgery at 231 hospitals (cases = 217,630) and for colorectal surgery at 109 hospitals (cases = 17,251). The number of outliers (poor performers) identified using different methods and criteria were compared. RESULTS: The overall morbidity was 10.3% for general surgery and 25.3% for colorectal surgery. The mortality was 1.6% for general surgery and 4.0% for colorectal surgery. Programs used different methods (logistic regression, hierarchical modeling, partitioning) and criteria (P < 0.01, P < 0.05, P < 0.10) to identify outliers. Depending on outlier identification methods and criteria employed, when each approach was applied to this single dataset, the number of outliers ranged from 7 to 57 hospitals for general surgery morbidity, 1 to 57 hospitals for general surgery mortality, 4 to 27 hospitals for colorectal morbidity, and 0 to 27 hospitals for colorectal mortality. CONCLUSIONS: There was considerable variation in the number of outliers identified using different detection approaches. Quality programs seem to be utilizing outlier identification methods contrary to what might be expected, thus they should justify their methodology based on the intent of the program (i.e., quality improvement vs. reimbursement). Surgeons and hospitals should be aware of variability in methods used to assess their performance as these outlier designations will likely have referral and reimbursement consequences.
Authors: Karl Y Bilimoria; Mark E Cohen; Angela M Ingraham; David J Bentrem; Karen Richards; Bruce L Hall; Clifford Y Ko Journal: Ann Surg Date: 2010-07 Impact factor: 12.969
Authors: S F Khuri; J Daley; W Henderson; K Hur; J Demakis; J B Aust; V Chong; P J Fabri; J O Gibbs; F Grover; K Hammermeister; G Irvin; G McDonald; E Passaro; L Phillips; F Scamman; J Spencer; J F Stremple Journal: Ann Surg Date: 1998-10 Impact factor: 12.969
Authors: Chester L Davis; John R Pierce; William Henderson; C David Spencer; Christine Tyler; Robert Langberg; Jennan Swafford; Gladys S Felan; Martha A Kearns; Brigitte Booker Journal: J Am Coll Surg Date: 2007-03-02 Impact factor: 6.113
Authors: Mark E Cohen; Justin B Dimick; Karl Y Bilimoria; Clifford Y Ko; Karen Richards; Bruce Lee Hall Journal: J Am Coll Surg Date: 2009-10-17 Impact factor: 6.113
Authors: Shukri F Khuri; William G Henderson; Jennifer Daley; Olga Jonasson; R Scott Jones; Darrell A Campbell; Aaron S Fink; Robert M Mentzer; Janet E Steeger Journal: J Am Coll Surg Date: 2007-06 Impact factor: 6.113
Authors: Deborah S Keller; Juan R Flores-Gonzalez; Sergio Ibarra; Nisreen Madhoun; Reena Tahilramani; Ali Mahmood; Eric M Haas Journal: Surg Endosc Date: 2015-09-28 Impact factor: 4.584
Authors: Matthew G Mullen; Alex D Michaels; J Hunter Mehaffey; Christopher A Guidry; Florence E Turrentine; Traci L Hedrick; Charles M Friel Journal: JAMA Surg Date: 2017-08-01 Impact factor: 14.766
Authors: I E Ceyisakar; N van Leeuwen; Diederik W J Dippel; Ewout W Steyerberg; H F Lingsma Journal: BMC Med Res Methodol Date: 2021-01-06 Impact factor: 4.615
Authors: Ryan P Merkow; Ying Shan; Aakash R Gupta; Anthony D Yang; Pradeep Sama; Mark Schumacher; David Cooke; Cynthia Barnard; Karl Y Bilimoria Journal: Jt Comm J Qual Patient Saf Date: 2020-07-03