Evaluating trauma center quality: does the choice of the severity-adjustment model make a difference?

CONTEXT: The Major Trauma Outcome Study (MTOS) database was created by the American College of Surgeons over 20 years ago to establish national norms for trauma care. The primary trauma outcome prediction models used for evaluating the quality of trauma care, TRISS and ASCOT (A Severity Characterization of Trauma), were developed using the MTOS database.
OBJECTIVE: First, to determine whether TRISS and ASCOT agree on hospital quality. Second, to determine whether TRISS and ASCOT accurately reflect contemporary outcomes in trauma care. DESIGN, SETTING AND PATIENTS: A retrospective cohort study based on 91,112 patients admitted to 69 hospitals between 2000 and 2001 in the National Trauma Databank. Using TRISS and ASCOT, the ratio of the observed to expected mortality rate (O/E ratio) was calculated for each hospital. Hospitals whose O/E ratio was statistically different from 1 were identified as quality outliers. Kappa analysis was used to assess the degree to which TRISS and ASCOT agreed on the identity of hospital quality outliers.
RESULTS: TRISS and ASCOT disagreed on the outlier status of 35 of the 69 hospitals. Kappa analysis revealed only fair agreement (kappa = 0.23; p = 0.0015) between TRISS and ASCOT in identifying quality outliers. Thirty-eight hospitals were identified by the TRISS method as high-performance hospitals.
CONCLUSION: First, TRISS and ASCOT exhibit substantial disagreement on the identity of quality outliers within the NTDB. Second, an unrealistically high number of hospitals were identified as high-performance outliers using either TRISS or ASCOT. These findings have important implications for the use of TRISS and ASCOT for benchmarking performance and quality improvement.