OBJECTIVES: Using the public reports of the Pennsylvania Health Care Cost Containment Council on coronary artery bypass graft surgery for 1990 to 1992 as a case study, the authors assess the sensitivity of results to the choice of data and statistical methodology. METHODS: Using the Council's public-release data, surgical mortality and utilization were reanalyzed by standard linear models, empirical Bayes methods, Monte Carlo simulations, and hierarchical statistical models. RESULTS: Statistical power calculations demonstrate that the annual volume of bypass surgery for many hospitals and for most surgeons is too small for meaningful mortality comparisons. The number of hospitals and physicians designated as mortality "outliers" in the Council's reports results in part from a failure to adjust critical P values for multiple comparisons. Hierarchical statistical models implemented by mixed effects logistic regression, by contrast, can detect true differences in performance without producing false outliers. Mortality analyses are sensitive to the choice of comorbidities used for severity adjustment of a mortality model. Small-area analyses indicate large differences in the rates of bypass surgery across Pennsylvania, with lower population-based rates of surgery associated with higher population-based inpatient mortality. CONCLUSIONS: Analyses of mortality by operative procedure, rather than by patient diagnosis, should consider the potential for selection bias caused by the decision to elect surgery. The clinical and statistical issues of operative mortality are sufficiently complex to merit review by independent experts before public release of hospital and physician performance measures.
OBJECTIVES: Using the public reports of the Pennsylvania Health Care Cost Containment Council on coronary artery bypass graft surgery for 1990 to 1992 as a case study, the authors assess the sensitivity of results to the choice of data and statistical methodology. METHODS: Using the Council's public-release data, surgical mortality and utilization were reanalyzed by standard linear models, empirical Bayes methods, Monte Carlo simulations, and hierarchical statistical models. RESULTS: Statistical power calculations demonstrate that the annual volume of bypass surgery for many hospitals and for most surgeons is too small for meaningful mortality comparisons. The number of hospitals and physicians designated as mortality "outliers" in the Council's reports results in part from a failure to adjust critical P values for multiple comparisons. Hierarchical statistical models implemented by mixed effects logistic regression, by contrast, can detect true differences in performance without producing false outliers. Mortality analyses are sensitive to the choice of comorbidities used for severity adjustment of a mortality model. Small-area analyses indicate large differences in the rates of bypass surgery across Pennsylvania, with lower population-based rates of surgery associated with higher population-based inpatient mortality. CONCLUSIONS: Analyses of mortality by operative procedure, rather than by patient diagnosis, should consider the potential for selection bias caused by the decision to elect surgery. The clinical and statistical issues of operative mortality are sufficiently complex to merit review by independent experts before public release of hospital and physician performance measures.