Predicting risk-adjusted mortality for CABG surgery: logistic versus hierarchical logistic models.

BACKGROUND: In recent years, several studies in the medical and health service research literature have advocated the use of hierarchical statistical models (multilevel models or random-effects models) to analyze data that are nested (eg, patients nested within hospitals). However, these models are computer-intensive and complicated to perform. There is virtually nothing in the literature that compares the results of standard logistic regression to those of hierarchical logistic models in predicting future provider performance.
OBJECTIVE: We sought to compare the ability of standard logistic regression relative to hierarchical modeling in predicting risk-adjusted hospital mortality rates for coronary artery bypass graft (CABG) surgery in New York State. DESIGN, SETTING AND PATIENTS: New York State CABG Registry data from 1994 to 1999 were used to relate statistical predictions from a given year to hospital performance 2 years hence. MAIN OUTCOME MEASURES: Predicted and observed hospital mortality rates 2 years hence were compared using root mean square errors, the mean absolute difference, and the number of hospitals whose predicted mortality rate data was within a 95% confidence interval around the observed mortality rate.
RESULTS: In these data, standard logistic regression performed similarly to hierarchical models, both with and without a second level covariate. Differences in the criteria used for comparison were minimal, and when the differences could be statistically tested no significant differences were identified.
CONCLUSIONS: It is instructive to compare the predictive abilities of alternative statistical models in the process of assessing their relative performance on a specific database and application.