The Limited Utility of Ranking Hospitals Based on Their Colon Surgery Infection Rates.

BACKGROUND: The Centers for Medicare and Medicaid Services (CMS) use colon surgical site infection (SSI) rates to rank hospitals and apply financial penalties. The CMS' risk-adjustment model omits potentially impactful variables that might disadvantage hospitals with complex surgical populations.
METHODS: We analyzed adult patients who underwent colon surgery within facilities associated with HCA Healthcare from 2014 to 2016. SSIs were identified from National Health Safety Network (NHSN) reporting. We trained and validated 3 SSI prediction models, using (1) current CMS model variables, including hospital-specific random effects (HCA-adapted CMS model); (2) demographics and claims-based comorbidities (expanded-claims model); and (3) demographics, claims-based comorbidities, and NHSN variables (claims-plus-electronic health record [EHR] model). Discrimination, calibration, and resulting rankings were compared among all models and the current CMS model with published coefficient values.
RESULTS: We identified 39 468 colon surgeries in 149 hospitals, resulting in 1216 (3.1%) SSIs. Compared to the HCA-adapted CMS model, the expanded-claims model had similar performance (c-statistic, 0.65 vs 0.67, respectively), while the claims-plus-EHR model was more accurate (c-statistic, 0.70; 95% confidence interval, .67-.73; P = .004). The sampling variation, due to the low surgical volume and small number of infections, contributed 74% of the total variation in observed SSI rates between hospitals. When CMS model rankings were compared to those from the expanded-claims and claims-plus-EHR models, 18 (15%) and 26 (22%) hospitals changed quartiles, respectively, and 10 (8.3%) and 12 (10%) hospitals changed into or out of the lowest-performing quartile, respectively.
CONCLUSIONS: An expanded set of variables improved colon SSI risk predictions and quartile assignments, but low procedure volumes and SSI events remain a barrier to effectively comparing hospitals.

Colon surgeries are common procedures in US hospitals and are associated with high surgical site infection (SSI) rates, compared to other procedure types, with estimated attributable costs exceeding $3 billion annually [1, 2]. Because of both the high volume of these procedures and the relatively high rate of infectious complications, the Centers for Medicare and Medicaid Services (CMS) requires acute-care hospitals to report colorectal SSI outcomes in order to receive full CMS reimbursement as part of the Inpatient Prospective Payment System [3, 4]. Hospital rankings based on colorectal SSI outcomes—specifically, those ranking in the lowest-performing quartile—are used along with other quality metrics to determine hospital reimbursements through both the CMS Hospital Acquired Conditions and Value-Based Purchasing pay-for-performance systems. In fiscal year 2015, the total amount of reimbursement at risk in the Hospital Acquired Conditions and Value-Based Purchasing programs was approximately $1.4 billion [5].

In order to account for differences in patient complexity and baseline SSI risk, hospital performance is risk-adjusted, and hospitals with less than 1 expected SSI are excluded from rankings. For colon surgery, CMS uses a risk adjustment model that includes age, gender, American Society of Anesthesiologists (ASA) score, diabetes, body mass index (BMI), primary closure, and oncology hospital (Table 1). This scoring omits several risk factors that have been associated with increased SSI risk; for example, tobacco use [6-10] and comorbidities such as renal disease [11], vascular disease [12], cirrhosis [13], and malignancy [13]. Electronically available comorbidities derived from International Classification of Disease (ICD) codes have been shown to improve risk adjustment and lead to significant changes in hospital rankings [14]. Additionally, concomitantly performed colon or other noncolon intraabdominal surgeries might affect the SSI risk [15], and are available to CMS via procedural billing codes.

Table 1.

Eligible Covariates for Each Risk Adjustment Model

CMS	HCA-Adapted-CMS	Expanded Claims	Claims Plus EHR
Age	Age	Age	Age
ASA score	ASA score	ASA score	ASA score
Diabetes status	Diabetes status	Diabetes status	Diabetes status
Body mass index	Body mass index	Body mass index	Body mass index
Use of primary closure	Use of primary closure	Use of primary closure	Use of primary closure
Oncology hospital	Oncology hospital	Oncology hospital	Oncology hospital
Sex	Sex	Sex	Sex
…	…	Charlson/Elixhauser comorbiditiesa	Charlson/Elixhauser comorbiditiesa
…	…	Concomitant colon proceduresa	Concomitant colon proceduresa
…	…	Concomitant noncolon intraabdominal proceduresa	Concomitant noncolon intraabdominal proceduresa
…	…	…	Use of anesthesia b
…	…	…	Procedure durationb
…	…	…	Laparoscopic techniqueb
…	…	…	Medical school affiliationb
…	…	…	Hospital size ≥/<500 bedsb
…	…	…	Wound classb
…	…	…	Emergency surgeryb
…	…	…	Smoking status (current, former, or never)
…	…	…	Race

Abbreviations: ASA, American Society of Anesthesiologists; CMS, Centers for Medicare and Medicaid Services; EHR, electronic health record; HCA, HCA Healthcare.

aData obtained from diagnostic or procedural claims codes.

Data obtained from the National Healthcare Safety Network.

Less than optimal risk adjustment affects the ability to use SSI outcomes to identify opportunities for improvement or to meaningfully compare SSI performance between health-care facilities. By omitting recognized risk factors for SSI, CMS’ adjustment model might be unfairly weighted against hospitals that perform colorectal surgery on more complex patients with multiple comorbidities. We hypothesized that better risk adjustment models could be made using additional data relevant to the SSI risk, including both claims-based and other patient-level clinical data that are available in electronic health records (EHRs). We set out to determine whether an expanded set of preoperative data could help build more effective risk adjustment models that yield significantly different hospital rankings than those derived from the current CMS model, especially with respect to the lowest-performing quartile.

METHODS

We performed a retrospective analysis of adult patients who underwent colon surgery within facilities affiliated with HCA Healthcare (HCA) in 21 US states from January 2014 through December 2016. The median HCA hospital size is 260 beds (range 32–1013). Only the first eligible episode of colon surgery for each individual was included.

We used data that HCA-affiliated hospitals had submitted to the Centers for Disease Control and Prevention National Healthcare Safety Network to identify patients who underwent colon surgery. We also obtained patient- and hospital-level data from both the NHSN submissions and from HCA’s central data repository. The full set of variables is shown in Table 1. Deep incisional or organ/space colon SSIs were determined by each hospital’s infection prevention staff using NHSN criteria [16]. Each HCA facility validates NHSN data twice per year (3 cases), and CMS validation is performed at 30 facilities per year. Superficial incisional SSIs were excluded from the analyses, as these are not included in the CMS SSI outcome measures [17]. All SSIs present at the time of surgery were also excluded. Using ICD-9 and ICD-10 procedure codes (Supplementary Appendix 1), we identified instances of secondary colon surgery or other noncolon intraabdominal surgery that occurred on the same date as the index colon procedure for use as covariates, but did not count them as separate procedures. The Harvard Pilgrim Healthcare Institutional Review Board approved this study.

We computed Charlson and Elixhauser comorbidities using diagnosis codes, according to the methods of Quan et al [18]. Baseline demographics and clinical variables were compared between patients with and without SSI using mixed effects models. The sample was randomly split into a training set (2/3) and a validation set (1/3) within each hospital. We used mixed-effects logistic models to assess the SSI risk attributable to each patient-level variable, with a random intercept for facility to account for the clustering effect at the hospital level. We used a backwards selection approach with a threshold of P < .05 for elimination. Using the training set, we built an HCA-adapted CMS model using the variables in the existing CMS model; an expanded-claims model using only variables selected from a set including basic demographics (age, gender, race, ASA score) and claims-based data (comorbidity scores and concomitant colon or other noncolon intraabdominal surgeries); and a claims-plus-EHR model using all available NHSN patient- and hospital-level data, in addition to demographics and claims-based data. For the claims-plus-EHR model, the laparoscopic variable was included by default [19, 20]. For comparison of the resulting hospital rankings, we also used the CMS risk-adjustment model with published coefficient values for each variable [17]. This differed from the HCA-adapted CMS model, which had coefficients built from our training set and used random effects.

In the model-building stage, we included random effects to account for clustering within hospitals in order to compare model performance. Without including the random effects, in general we would not expect the predicted risks to agree with the observed risk, unless there was no heterogeneity across hospitals (ie, no clustering within a hospital). Patient-level predicted probabilities of SSI were generated from each model in validation data sets. We calculated c-statistics with 95% confidence intervals (CIs) and calibration plots for each model [21]. Hospitals’ predicted and observed SSI rates were graphed by the hospital procedure volume to assess differences in prediction accuracy for large versus small hospitals.

For ranking purposes, we calculated the expected individual-level SSI risk from each model using the validation data set. The random effects were not included, in order to remove the influence of hospital characteristics so that the resulting value represented the expected risk for an individual in the same reference hospital after adjusting only for individual characteristics. Based on the expected individual SSI risk, we estimated the expected number of SSI events and created standardized infection ratios for each hospital by dividing the observed number of SSI events by the expected number estimated from each model. We ranked hospitals according to the standardized infection ratios calculated from the different models and assessed the differences in ordinal ranks and by quartile in comparison to the CMS model [14, 22]. Hospitals were excluded if they had <1 expected SSI. Bubble plots were made to show changes in quartile ranks between models. Finally, we decomposed the total variability in observed SSI rates across hospitals into the between-hospital variability and the within-hospital sampling variability, and estimated the coefficient of variation in SSI rates across hospitals using the methods of Hayes and Moulton [23]. Analyses were performed using SAS version 9.3.

RESULTS

We identified 39 468 instances of eligible colon surgery at 149 hospitals. In total, 1216 (3.1%) subjects developed a deep incisional or organ/space SSI. Subjects with SSIs differed significantly from those without for many attributes; those with an SSI were slightly younger, had higher ASA scores, were less likely to have primary closure, were less likely to undergo laparoscopic procedures, had higher BMIs, had longer procedure durations, were less likely to be in hospitals with medical school affiliations, were more likely to have trauma, were more likely to have an emergency procedure, were more likely to have malnutrition, were more likely to have contaminated or dirty wounds, were more likely to have a concomitant noncolon intraabdominal procedure, and had higher Elixhauser and Charlson comorbidity scores. No significant differences were observed for race, gender, smoking status, concomitant colon surgery, hospital bed size, or several other comorbidities (Table 2).

Table 2.

Baseline Demographics by Surgical Site Infection Status

	Total, N = 39 468	Subjects without SSI, n = 38 252	Subjects with SSI, n = 1216	P Value
Mean age (SD)	62.7 (15.31)	62.7 (15.31)	60.9 (15.33)	<.0001
Race	…	…	…	.0789
Asian	741 (1.9%)	723 (1.9%)	18 (1.5%)
Black	4264 (10.8%)	4115 (10.8%)	149 (12.3%)
Hispanic	694 (1.8%)	672 (1.8%)	22 (1.8%)
Other	3025 (7.7%)	2933 (7.7%)	92 (7.6%)
White	30 746 (77.9%)	29 809 (77.9%)	935 (76.9%)
Female gender	21 345 (54.1%)	20 707 (54.1%)	638 (52.5%)	.3606
ASA score	…	…	…	.0008
1	836 (2.1%)	816 (2.1%)	20 (1.6%)
2	12 643 (32%)	12 302 (32.2%)	341 (28%)
3	20 118 (51%)	19 479 (50.9%)	639 (52.5%)
4	5487 (13.9%)	5278 (13.8%)	209 (17.2%)
5	384 (1%)	377 (1%)	7 (0.6%)
Use of anesthesia	38 953 (98.7%)	37 740 (98.7%)	1213 (99.8%)	.1145
Primary closure	39 083 (99%)	37 898 (99.1%)	1185 (97.5%)	<.0001
Laparosopic technique	14 233 (36.1%)	13 883 (36.3%)	350 (28.8%)	<.0001
Mean BMI (SD)	28.6 (6.69)	28.5 (6.67)	30.2 (7.22)	<.0001
Mean procedure length, minutes (SD)	137.3 (72.19)	136.6 (71.63)	160.3 (84.83)	<.0001
Medical school affiliation	16 499 (41.8%)	16 041 (41.9%)	458 (37.7%)	.0532
Trauma	171 (0.4%)	141 (0.4%)	30 (2.5%)	<.0001
Emergency surgery	5299 (13.4%)	5096 (13.3%)	203 (16.7%)	.0012
Hospital bed size ≥500	7415 (18.8%)	7178 (18.8%)	237 (19.5%)	.5836
Malnutrition	3579 (9.1%)	3364 (8.8%)	215 (17.7%)	<.0001
Wound class	…	…	…	<.0001
Clean-contaminated	31 469 (79.7%)	30 624 (80.1%)	845 (69.5%)
Contaminated	4707 (11.9%)	4523 (11.8%)	184 (15.1%)
Dirty	3293 (8.3%)	3106 (8.1%)	187 (15.4%)
Tobacco use category	…	…	…	.1433
Current smoker	6488 (16.4%)	6274 (16.4%)	214 (17.6%)
Former smoker	11 813 (29.9%)	11 441 (29.9%)	372 (30.6%)
Never smoker	18 248 (46.2%)	17 710 (46.3%)	538 (44.2%)
Missing	2921 (7.4%)	2829 (7.4%)	92 (7.6%)
≥1 concomitant colon procedure	20 639 (52.3%)	19 998 (52.3%)	641 (52.7%)	.9034
≥1 concomitant intraabdominal procedure	15 728 (39.9%)	15 135 (39.6%)	593 (48.8%)	<.0001
Elixhauser components	…	…	…
Congestive heart failure	3043 (7.7%)	2921 (7.6%)	122 (10%)	.0026
Cardiac arrhythmia	7590 (19.2%)	7256 (19%)	334 (27.5%)	<.0001
Cardiac valve disease	1085 (2.7%)	1049 (2.7%)	36 (3%)	.5811
Pulmonary vascular disease	832 (2.1%)	787 (2.1%)	45 (3.7%)	.0002
Peripheral vascular disease	2820 (7.1%)	2695 (7%)	125 (10.3%)	<.0001
Hypertension	18 052 (45.7%)	17 506 (45.8%)	546 (44.9%)	.7705
Complicated hypertension	3981 (10.1%)	3824 (10%)	157 (12.9%)	.0006
Paralysis	709 (1.8%)	698 (1.8%)	11 (0.9%)	.0108
Neurological disease	2186 (5.5%)	2070 (5.4%)	116 (9.5%)	<.0001
Chronic pulmonary disease	6707 (17%)	6457 (16.9%)	250 (20.6%)	.0003
Diabetes	6196 (15.7%)	5975 (15.6%)	221 (18.2%)	.0129
Complicated diabetes	1571 (4%)	1507 (3.9%)	64 (5.3%)	.0194
Hypothyroidism	4912 (12.4%)	4752 (12.4%)	160 (13.2%)	.3785
Renal disease	4070 (10.3%)	3917 (10.2%)	153 (12.6%)	.0062
Liver disease	1464 (3.7%)	1400 (3.7%)	64 (5.3%)	.0038
Peptic ulcer disease	476 (1.2%)	448 (1.2%)	28 (2.3%)	.0005
AIDS	59 (0.1%)	58 (0.2%)	1 (0.1%)	.5807
Lymphoma	304 (0.8%)	294 (0.8%)	10 (0.8%)	.7802
Metastatic cancer	3829 (9.7%)	3675 (9.6%)	154 (12.7%)	.0019
Solid tumor	13 318 (33.7%)	12 891 (33.7%)	427 (35.1%)	.6026
Rheumatoid arthritis	1033 (2.6%)	994 (2.6%)	39 (3.2%)	.1674
Coagulopathy	2262 (5.7%)	2131 (5.6%)	131 (10.8%)	<.0001
Obesity	5176 (13.1%)	4932 (12.9%)	244 (20.1%)	<.0001
Weight loss	5060 (12.8%)	4757 (12.4%)	303 (24.9%)	<.0001
Fluid and electrolyte disorder	7385 (18.7%)	7058 (18.5%)	327 (26.9%)	<.0001
Blood loss anemia	1105 (2.8%)	1059 (2.8%)	46 (3.8%)	.0369
Iron deficiency anemia	2028 (5.1%)	1958 (5.1%)	70 (5.8%)	.3658
Alcohol abuse	1533 (3.9%)	1463 (3.8%)	70 (5.8%)	.0004
Illicit drug abuse	679 (1.7%)	647 (1.7%)	32 (2.6%)	.0102
Psychosis	248 (0.6%)	236 (0.6%)	12 (1%)	.1532
Depression	4405 (11.2%)	4225 (11%)	180 (14.8%)	<.0001
Median Elixhauser score (IQR)	3.0 (1.0–4.0)	2.0 (1.0–4.0)	3.0 (2.0–5.0)	<.0001
Charlson components
Myocardial infarction	1087 (2.8%)	1057 (2.8%)	30 (2.5%)	.6107
Coronary heart disease	2905 (7.4%)	2791 (7.3%)	114 (9.4%)	.0076
Cerebrovascular disease	842 (2.1%)	811 (2.1%)	31 (2.5%)	.3650
Dementia	629 (1.6%)	613 (1.6%)	16 (1.3%)	.4165
Chronic pulmonary disease	6532 (16.5%)	6294 (16.5%)	238 (19.6%)	.0018
Rheumatologic disease	928 (2.4%)	896 (2.3%)	32 (2.6%)	.4707
Liver disease	613 (1.6%)	585 (1.5%)	28 (2.3%)	.0248
Diabetes	7082 (17.9%)	6863 (17.9%)	219 (18%)	.8639
Complicated diabetes	876 (2.2%)	846 (2.2%)	30 (2.5%)	.5363
Paraplegia	556 (1.4%)	547 (1.4%)	9 (0.7%)	.0299
Renal disease	4059 (10.3%)	3906 (10.2%)	153 (12.6%)	.0056
Malignancy	4095 (10.4%)	3931 (10.3%)	164 (13.5%)	.0014
Severe liver disease	178 (0.5%)	169 (0.4%)	9 (0.7%)	.1561
Metastatic cancer	3829 (9.7%)	3675 (9.6%)	154 (12.7%)	.0017
HIV/AIDS	59 (0.1%)	58 (0.2%)	1 (0.1%)	.5807
Peripheral vascular disease	1111 (2.8%)	1061 (2.8%)	50 (4.1%)	.0047
Median Charlson score (IQR)	1.0 (0–3.0)	1.0 (0–3.0)	2.0 (0–3.0)	<.0001

The P values reflect the comparisons of subjects with and without SSI.

Abbreviations: ASA, American Society of Anesthesiologists; BMI, body mass index; HIV, human immunodeficiency virus; IQR, interquartile range; SD, standard deviation; SSI, surgical site infection.

The analysis of the training data set yielded an expanded-claims multivariable model that included age, ASA score, presence of concomitant colon procedure(s), total Elixhauser score, and the Elixhauser components of cardiac arrhythmia, paralysis, neurological disorder, coagulopathy, obesity, weight loss, and fluid/electrolyte disorder (Table 3). Of these, only age and ASA score are present in the CMS model. The claims-plus-EHR model included laparoscopy (by default), age, ASA score, total Elixhauser score, and the Elixhauser components of cardiac arrhythmia, paralysis, neurological disorder, and weight loss, as well as wound class, procedure length, BMI, and use of primary closure. The c-statistic for the CMS model was 0.59 (95% CI .56–.61) in the validation set. A significant improvement in the prediction accuracy was observed using the HCA-adapted CMS model, with a c-statistic of 0.65 (95% CI, .62–.69; P = .001 for comparison). The expanded-claims model yielded similar accuracy as the HCA-adapted CMS model, with a c-statistic of 0.67 (95% CI, .64–.71; P = .15). The claims-plus-EHR model further improved the prediction accuracy, as compared to the HCA-adapted CMS model, with a c-statistic of 0.70 (95% CI, .67–.73; P = .004).

Table 3.

Associations Between Variables and Surgical Site Infection in Each Prediction Model

		Odds Ratio (95% CI)	C-statistic (95% CI) from Validation Data Set
CMS model, fixed coefficients		…	.59 (.56–.61)
HCA-adapted CMS model			.65 (.62–.69)
ASA scorea	1	.86 (.51–1.45)
	2	.80 (.67–.94)
	3/4/5, reference	…
Diabetes		1.08 (.89–1.31)
Male gender		1.01 (.87–1.16)
Age / 10		.93 (.88–.97)
BMI ≥0		1.45 (1.25–1.68)
Non-primary closure		.37 (.23–.59)
Expanded-claims model			.67 (.64–.71)
ASA score	1	1.09 (.65–1.85)
	2, reference	…
	3	1.01 (.84–1.21)
	4	.77 (.60–1.00)
	5	.24 (.09–.67)
Age		.99 (.98–.99)
Cardiac arrhythmia		1.59 (1.34–1.90)
Paralysis		.22 (.09–.53)
Neurological disease		1.42 (1.09–1.84)
Coagulopathy		1.74 (1.37–2.21)
Obesity		1.81 (1.51–2.17)
Weight loss		2.27 (1.89–2.72)
Fluid and electrolyte disorder		1.34 (1.13–1.59)
Depression		1.28 (1.05–1.60)
Multiple colon procedures on the index date		1.30 (1.13–1.51)
Claims-plus-EHR Model			.70 (.67–.73)
ASA score	1	1.20 (.71–2.03)
	2, reference	…
	3	.90 (.75–1.08)
	4	.67 (.51–.88)
	5	.22 (.08– .60)
Wound class	CC, reference	…
	CO	1.48 (1.20–1.83)
	D	1.77 (1.41–2.22)
Age		.99 (.99–1.00)
Cardiac arrhythmia		1.28 (1.06–1.54)
Coronary heart disease		.67 (.50–.90)
Diabetes, Charlson		.69 (.56–.85)
Complicated diabetes, Elixhauser		.55 (.32–.95)
Hypertension		.80 (.67–.96)
Paralysis		.23 (.09–.57)
Weight loss		1.78 (1.46–2.18)
Hypothyroidism		.76 (.60–.96)
Renal disease, Elixhauser		.72 (.55–.96)
Total Elixhauser score		1.24 (1.18–1.30)
Laparoscopy		.85 (.72–1.01)
Trauma		3.76 (1.99–7.11)
Primary closure		.59 (.35–.98)
BMI, continuous		1.03 (1.02–1.04)
Procedure length, minutes		1.00 (1.00–1.01)

Abbreviations: ASA, American Society of Anesthesiologists; BMI, body mass index; CC, clean-contaminated; CI, confidence interval; CMS, Centers for Medicare and Medicaid Services; CO, contaminated; D, dirty; EHR, electronic health record; HCA, HCA Healthcare.

aThe ASA score is treated as a continuous variable with a value of 1, 2, or 3/4/5 in the CMS model.

For comparisons of ranking, 29 hospitals were excluded for having 0 observed SSIs and <1 expected, based on the claims-plus-EHR model, leaving 120 hospitals. When rankings from the CMS model were compared to those from the expanded-claims and the claims-plus-EHR models, 89 (74%) and 93 (78%) of 120 hospitals had a change in rank, respectively; the median absolute changes were 3 and 3.5 ranks, respectively, and the number of hospitals that changed quartiles were 18 (15%) and 26 (22%), respectively. There were 10 (8.3%) and 12 (10%) hospitals, respectively, that were reclassified into or out of the lowest-performing quartile (Figure 1). Comparing the expanded-claims and the claims-plus-EHR models, 10 (8.3%) hospitals were reclassified into or out of the lowest-performing quartile.

Figure 1.

A–D, Comparison of hospital rankings, in quartiles, between the CMS model, HCA-adapted CMS model, expanded claims model, and claims-plus-EHR model. The number in each circle represents the number of hospitals that fall into the quartiles indicated on each axis, with “1” being the best-performing quartile and “4” being the worst-performing quartile. Each axis represents the resulting hospital rankings of 1 of the 4 modeling approaches used in this study. Abbreviations: CMS, Centers for Medicare and Medicaid Services; EHR, electronic health record; HCA, HCA Healthcare.

Graphing of the hospitals’ predicted and observed SSI rates demonstrated modest agreement using the HCA-adapted CMS, expanded-claims, and claims-plus-EHR models (Figure 2; Supplementary Appendix 2). In our sample, the median total surgical volume for the entire 3-year period was 207 and the median number of SSIs was only 5. The total procedure volume was 101 or less for 25% of the hospitals. The sampling variation, due to a low surgical volume and a small number of infections, contributed to 74% of the total variation in observed SSI rates across hospitals. The heterogeneity in SSI rates across hospitals was substantial: the coefficient of variation was estimated to be 47%, which corresponds to an intraclass correlation of 26%.

Figure 2.

A–F, Predicted versus observed SSI rates, stratified by median procedure volume (validation data set). Abbreviations: CMS, Centers for Medicare and Medicaid Services; EHR, electronic health record; HCA, HCA Healthcare; SSI, surgical site infection.

DISCUSSION

Optimizing the risk adjustment is essential for meaningful interhospital comparisons and appropriately assigning financial penalties. The current CMS risk adjustment model for colon surgery omits several important variables that proved to be significant risk factors in our models. The addition of claims data and clinical data that hospitals already submit to NHSN improved the discrimination of the model using variables currently employed by CMS from a c-statistic of 0.65 (95% CI, .62–.70) to 0.70 (95% CI, .67–.73). This confirms the importance of previously identified SSI risk factors included in our models. This improved model assigned 22% of hospitals to a different quartile than did the CMS model, including 10% of hospitals that moved into or out of the lowest-performing quartile.

The agreements between the predicted SSI rates based on these models and the observed SSI rates were moderate for all models, as shown in Figure 2. Low procedure volumes and the small number of predicted events in individual hospitals are likely to be major limiting factors. Even though our sample included 3 years of data for each hospital, the median number of procedures included in our study was 207 (69 per year) and the median number of colon SSIs was only 5 (1.67 per year). The low surgical volume and small number of infections led to a large amount of uncertainty in estimating hospital-specific SSI rates. The heterogeneity in SSI rates across hospitals was substantial. While adjustment for patient, surgeon, and hospital characteristics (eg, surgeon’s personal procedure volume or surgical process measures) might be useful, the variation resulting from low procedure volumes and event rates will remain a serious barrier to effectively characterizing hospitals’ “true” SSI rates.

The improved performance of the HCA-adapted CMS model versus the CMS model suggests that coefficients obtained from a different patient population may have low portability. This also highlights the disadvantage of using a “1-size-fits-all” national model, as different populations may be better risk-adjusted using different variables. Additionally, the enhanced prediction demonstrated by the expanded-claims model versus the CMS model cannot be attributed to better variable selection, but instead is due to training this model in our specific population.

Our findings come with certain caveats. Our models were trained using data from a large network of community hospitals; therefore, our findings may not be applicable to populations at traditional academic centers or specialty hospitals. However, these results are likely to apply to a large proportion of US hospitals. As our primary goal was to build models for prediction, the calculated coefficients used for adjustment in our models do not have a causal interpretation. SSI diagnoses were based on NHSN reporting by hospital-based infection prevention staff, and SSI detection methods may vary by hospital. We did not exclude patients who died within the 30-day postoperative period; this is consistent with the limitations of current surveillance practices at CMS, and a possible explanation for the lower SSI rates seen with increasing ASA scores in the expanded-claims and claims-plus-EHR models.

This study’s strengths include a large sample of individual patients and SSI outcomes, as well as a geographically diverse network of hospitals that represent a wide range of facility sizes and care environments.

CONCLUSIONS

An expanded set of claims-based comorbidities and patient-level clinical data improved SSI risk prediction, compared to current CMS methods, in a population of adult colon surgery patients. This improved predictive ability changed hospital rankings for the majority of hospitals, and resulted in a change in quartile for 22%, including 10% that moved into or out of the lowest-performing quartile. However, all adjusted ranking systems were of limited value, because of low hospital-specific procedure volumes and the small number of predicted infections. Because SSI following colon surgery is a relatively rare outcome, coupled with low procedure volumes, we believe that current ranking systems and readily foreseeable improvements will be largely incapable of distinguishing hospitals based on their SSI rates. This limitation may justify returning to assessing compliance with evidence-based process measures; for example, appropriate selection and timing of the administration of perioperative antibiotic prophylaxis.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Click here for additional data file.