Impact of Financial Incentives on Early and Late Adopters among US Hospitals: observational study.

OBJECTIVE: To examine how hospitals that volunteered to be under financial incentives for more than a decade as part of the Premier Hospital Quality Incentive Demonstration (early adopters) compared with similar hospitals where these incentives were implemented later under the Hospital Value-Based Purchasing program (late adopters).
DESIGN: Observational study.
SETTING: 1189 hospitals in the USA (214 early adopters and 975 matched late adopters), using Hospital Compare data from 2003 through 2013. PARTICIPANTS: 1 371 364 patients aged 65 years and older, using 100% Medicare claims. MAIN OUTCOME MEASURES: Clinical process scores and 30 day mortality.
RESULTS: Early adopters started from a slightly higher baseline of clinical process scores (92) than late adopters (90). Both groups reached a ceiling (98) a decade later. Starting from a similar baseline, just below 13%, early and late adopters did not have significantly (P=0.25) different mortality trends for conditions targeted by the program (0.05% point difference quarterly) or for conditions not targeted by the program (-0.02% point difference quarterly).
CONCLUSIONS: No evidence that hospitals that have been operating under pay for performance programs for more than a decade had better process scores or lower mortality than other hospitals was found. These findings suggest that even among hospitals that volunteered to participate in pay for performance programs, having additional time is not likely to turn pay for performance programs into a success in the future. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Introduction

OECD (Organisation for Economic Co-operation and Development) countries use financial incentives to improve the quality of healthcare.1 These incentives are increasingly being used in low and middle income countries.2 The USA and the UK are probably the furthest ahead, but other countries are monitoring the experiences of these countries to determine what they might do. Providing incentives to improve the performance of hospitals has become common in the USA over the last decade.3 4 5 The Premier Hospital Quality Incentive Demonstration (HQID), a voluntary program run by the Centers for Medicare and Medicaid Services,6 7 ran from 2003 to 2009. It became the model for the national Hospital Value-Based Purchasing (HVBP) program, adopted after the passage of the Affordable Care Act,8 9 which has run since 2011 (details on both programs are shown in web appendices 1 to 3). The hospitals that were invited by Centers for Medicare and Medicaid Services and then participated in the HQID (early adopters) have been under financial incentives to improve quality for more than a decade and are likely to have a comparative advantage, having had more time to refine their delivery of care and strategically focus on improving patient outcomes. Previous evaluations of the HQID and the HVBP program showed limited impact on process measures but no improvements in patient outcomes or in cost reduction.10 11 12 13 14 15 16 17 18 19 20 However, advocates argue that it takes time for hospitals to make meaningful improvements and that we need patience to better understand how delivery of care under pay for performance programs changes care.

Improving outcomes is difficult; it can require changes to workflows, restructuring the way providers are paid, and alignment of information technology systems.21 22 The national HVBP program, which attaches incentives directly to outcomes of Medicare beneficiaries, has been in effect for a few years, and early evidence suggests that it has had little impact on patient outcomes.18 One might expect that most hospitals have not yet been able to equip themselves adequately to make meaningful improvements to patient care. However, the early adopters of pay for performance programs, the Premier HQID hospitals, have been under financial incentives since 2003 (except for 2009 to 2010 when the HQID ended and before the HVBP program was implemented).23 24 These early adopters have likely had enough time to make the difficult structural changes which are necessary to improve outcomes. Given that early adopters volunteered to be in the HQID, they likely represent the best case scenario of how much improvement we might expect over a longer period. Whether the early adopters have outpaced late adopters or whether the HQID had spillover effects on the late adopters is unclear – and empirical evidence would be helpful for healthcare policymakers as the new administration makes changes to, or replaces, the Affordable Care Act.

As we continue to shift to value based payments in healthcare,25 it is crucial to understand the long term effects of financial incentives on quality of care and effectiveness of HVBP programs. Therefore, we used national Medicare claims data from 2003 to 2013 to answer three questions. First, how do early adopters that had participated in the HQID perform under the HVBP program compared with similar hospitals that did not participate in the HQID (late adopters) in terms of clinical process scores? Second, how do early adopters compare with late adopters on 30 day mortality outcomes for the three target conditions: acute myocardial infarction, congestive heart failure, and pneumonia? And third, are there any spillover effects for 30 day mortality on non-target conditions?

Methods

Data and matching

We identified all 2702 American acute care hospitals participating in Hospital Value-Based Purchasing (HVBP) programs in 2013 using publicly available Hospital Compare data. Of these hospitals, 233 had also voluntarily participated in the Hospital Quality Incentive Demonstration (HQID) from 2003 through 2009, and we defined these hospitals as early adopters (for details see web appendix 4). We obtained baseline hospital characteristics from the 2003 American Hospital Association Annual Survey.26

Previous work has shown that different hospital types are important to consider when examining patient outcomes under pay for performance programs.27 To limit the potential bias arising from differences in observable hospital characteristics between early and late adopters, including volunteering to participate in the HQID, we applied coarsened exact matching.28 This matching method, like other matching methods, prunes observations from the data. The remaining data have a better balance between the treated and control groups for the baseline of the outcome measure and for the following observed hospital characteristics: size, region, ownership, teaching status, location, presence of an intensive care unit, and safety net status (defined as the top 25% of hospitals with the largest disproportionate share index). The advantage of coarsened exact matching over other matching methods is that the resulting data are exactly balanced and do not need to be controlled further because both groups have exactly the same observed hospital characteristics (for details see web appendix 4).28

Data on clinical process scores (range from 0 to 100) for each hospital were obtained from Hospital Compare.29 30 Clinical process scores were available for the three HVBP program target conditions (acute myocardial infarction, congestive heart failure, and pneumonia) from 2004 through 2014 (latest available year) (see web appendix 1 and 4). The clinical process score used in this study is the measure that the Centers for Medicare and Medicaid Services uses to determine the clinical process score of hospitals. These clinical process scores assess whether “what is known to be ‘good’ medical care has been applied.”31 32 These process measures indicate whether or not a healthcare provider gives the recommended care to patients with a particular condition.33

Using the matched dataset, we identified the subset of Medicare beneficiaries admitted to the hospital from the 100% Medicare inpatient fee-for-service claims for 2003 through 2013 (latest available year at time of analysis). We included 1 371 364 patients with any of the three HVBP program target conditions or a selected non-target condition (stroke, gastroenteritis and esophagitis, gastrointestinal bleed, urinary tract infection, metabolic disorder, arrhythmia, and renal failure) (see web appendix 4).

Statistical analysis

Using the dataset of matched hospitals and their associated patients, we performed a segmented linear regression analysis to estimate differences in trends over time for outcome measures for both early and late adopters. Using these segmented linear models, we assessed the trends in outcome measures over three periods: the HQID period (fourth quarter of 2003 to fourth quarter of 2009), the pre-HVBP period (first quarter of 2010 to second quarter of 2011), and the HVBP period (third quarter 2011 to fourth quarter of 2013), and the associated years for the clinical process scores which are based on annual as opposed to quarterly data. We expected differences in trends to increase over time as the duration of the exposure to incentives increases.

Clinical process scores

Using the annual hospital level data, we estimated the relation between early adopter status and clinical process scores. The linear segmented regression model (see web appendix 4 for a formal description of all models used) included a binary variable to determine whether a hospital was an early adopter and further corrected for the underlying annual time trend in the HQID, pre-HVBP, and HVBP periods as well as the interactions between the time and early adopter variables. We included a stratum fixed effect that represented the strata of comparable hospitals based on the coarsened exact matching and a hospital random effect following the Centers for Medicare and Medicaid Services methods to account for correlation between patients within each hospital and for correlation over time. We weighted our analyses for hospital volume. We confirmed robustness of our findings using a hospital fixed effect, controlling for unobserved hospital characteristics that were time invariant (not shown).

Day mortality

Using quarterly patient level data, we estimated a similar model for mortality within 30 days after admission, standardized for age, gender, and comorbidities. We examined 30 day mortality data for each quartile for the combination of target conditions, for the three target conditions separately, and for the non-target conditions over the fourth quarter of 2003 to the fourth quarter of 2013. In addition, the segmented regression models included a correction for seasonality; the target condition for which a patient was admitted; the underlying time trend in the HQID, pre-HVBP, and HVBP periods by quarter; and patient characteristics (age, sex, race, and comorbidities) (see web appendix 4).

Differences in trends

Based on the estimates from the different linear segmented regressions, we compared differences between early and late adopters for each of the three periods and subsequently compared these differences for the HVBP period with those in the pre-HVBP period to determine whether the outcomes improved differently over time for early and late adopters.

All analyses were performed using software packages SAS version 9.4 or Stata version 14.0.

Patient involvement

No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for implementation of the study. No patients were asked to advise on interpretation or writing up of results. There are no plans to disseminate the results of the research to study participants or the relevant patient community.

Results

Late adopters were frequently dropped during the matching process for being dissimilar to the early adopters with regards to the following characteristics: small size; located in the north east and west; for profit or public; non-teaching; small rural and large rural; and safety net hospital status. This resulted in a matched dataset of 1189 hospitals that are mostly medium or large, private not for profit, and based in urban areas. Table 1 shows the baseline hospital characteristics before and after matching. The associated sample of individuals consists of 263 088 patients admitted to a hospital classed as an early adopter, and 1 108 276 admitted to a late adopter. Table 2 shows the patient characteristics before and after matching. As we would expect from a matching on hospital characteristics, the patient characteristics across early and late adopters still differed after matching, mostly in terms of race, with early adopters caring for a larger share of white patients (87.4% v 84.9%, P<0.001). All observed patient characteristics were controlled for in the regression models.

Table 1

Hospital characteristics at baseline. Values are percentages unless stated otherwise

Characteristics	Before matching				After matching				Not matched*
	Early adopters	Late adopters	P value		Early adopters	Late adopters	P value
No	233	2469			214	975			1492
Size:
Small	12.9	25.1	<0.001		13.1	13.1	1.00		34.1
Medium	61.8	61.0			64.0	64.0			51.7
Large	25.3	13.9			22.9	22.9			14.2
Region:
North east	13.7	18.0	0.002		14.0	14.0	1.00		14.8
Midwest	22.7	24.8			23.4	23.4			21.9
South	50.2	37.8			50.0	50.0			39.8
West	13.3	19.4			12.6	12.6			23.7
Ownership:
For profit	0.0	18.0	<0.001		0.0	0.0	1.00		29.4
Private not for profit	91.0	66.2			91.6	91.6			49.6
Public	9.0	15.8			8.4	8.4			21.0
Teaching status:
Non	59.2	70.3	0.001		59.3	59.3	1.00		9.2
Major	14.6	9.3			14.5	14.5			19.3
Minor	26.2	20.4			26.2	26.2			71.5
Location:
Urban	75.5	63.8	<0.001		77.6	77.6	1.00		55.3
Suburban	6.4	4.0			4.7	4.7			5.7
Large rural	14.6	22.4			14.0	14.0			25.0
Small rural	3.4	9.8			3.7	3.7			14.0
Has intensive care unit	89.7	83.2	0.010		91.6	91.6	1.00		71.1
Safety net hospital	21.9	26.4	0.130		19.6	19.6	1.00		31.3
Medicare patients	44.4	43.4	0.209		44.5	43.6	0.370		42.2
Medicare to non-Medicare ratio	0.8	0.8	0.209		0.8	0.8	0.370		0.7
Mean age (years) of Medicare patients	78.7	78.8	0.130		78.7	78.9	0.051		78.7

Twenty one of the non-matched hospitals had missing data and are not included

Table 2

Patient characteristics at baseline. Values are percentages unless stated otherwise

Characteristics	Before matching				After matching
	Early adopters	Late adopters	P value		Early adopters	Late adopters	P value
No	275 354	1 428 540			263 088	1 108 276
Mean age (years)	78.6	78.7	0.04		78.8	78.7	0.32
Male	41.3	41.4	0.89		40.5	41.0	0.08
Race:
White	86.8	84.7	<0.001		87.4	84.9	<0.001
Black	9.9	11.6			9.5	11.4
East Asian	1.0	1.0			1.0	1.0
Native American	0.6	0.7			0.6	0.7
Hispanic	1.2	1.4			1.0	1.3
Other	0.6	0.7			0.6	0.7
Comorbidities:
Chronic pulmonary disease	26.3	26.9	0.11		27.0	27.3	0.50
Congestive heart failure	14.7	15.3	0.01		14.7	15.3	0.01
Depression	5.6	5.5	0.55		5.8	5.4	0.01
Diabetes	25.4	25.3	0.77		25.6	25.5	0.67
Hypertension	46.6	45.4	0.01		46.2	45.3	0.01
Ischemia heart disease	23.7	23.4	0.54		23.0	22.5	0.23
Liver disease	1.0	1.0	0.33		1.0	1.0	0.04
Renal failure	7.1	7.5	0.06		6.9	7.3	0.02

Clinical process scores

Clinical process scores were incentivized during both the Hospital Quality Incentive Demonstration (HQID) and the current Hospital Value-Based Purchasing (HVBP) periods. Figure 1 shows that early adopters started from a slightly higher baseline clinical process score in 2004. Table 3 shows that early adopters had an average score of 91.5 versus 89.9 for late adopters in the HQID period for the combined target conditions. Improvements among the early adopters were smaller during the HQID period (difference −0.21, 95% confidence interval −0.31 to −0.11), although early adopters continued to perform at a slightly higher level than the late adopters (−0.55, −1.01 to −0.10) during the pre-HVBP period. Over the HVBP period, early and late adopters no longer differed in their clinical process scores. In the HVBP period, the increase in the clinical process score reached a ceiling where early and late adopters approach the same level: 98.5 versus 98.2 with a difference of −0.27 (95% confidence interval −0.77 to 0.22). Comparing the difference in the differences in the trend for the HVBP period with the pre-HVBP period, we found that there is no significant difference (P=0.19) across early and late adopters (last two columns of table 3). Estimates for the individual target conditions confirmed these patterns (web appendix 5).

Fig 1

Clinical process scores for target conditions

Table 3

Trends in clinical process scores over time. Values are clinical process scores (range 0 to 100) unless stated otherwise

Period	Average clinical process score				Yearly change in clinical process scores
	Early adopters	Late adopters	Difference (95% CI)		Early adopters	Late adopters	Difference in trends (95% CI)	Difference in differences in trends HVBP−pre-HVBP (95% CI)	P value
Target conditions*
HQID	91.5	89.9	−1.59 (−1.98 to −1.20)		2.44	2.65	−0.21 (−0.31 to −0.11)	0.24 (−0.12 to 0.59)	0.19
pre-HVBP	97.3	96.7	−0.55 (−1.01 to −0.10)		0.38	0.70	−0.32 (−0.52 to −0.12)
HVBP	98.5	98.2	−0.27 (−0.77 to 0.22)		0.53	0.61	−0.08 (−0.28 to 0.12)

HQID=Hospital Quality Incentive Demonstration; HVBP=Hospital Value-Based Purchasing.
*Acute myocardial infarction, congestive heart failure, and pneumonia

Day mortality

Figure 2 shows that mortality fell for both early and late adopters during the study period. Both groups started from a similar baseline (14.9% and 14.8% for the early and late adopters in the fourth quarter of 2003) and ended at the same rate of 9.9% for both groups in the fourth quarter of 2013. Table 4 shows that the average mortality is slightly higher among late adopters for each period. This pattern is not confirmed by the underlying mortality from individual target conditions (web appendix 6). For the non-targeted conditions, we found no significant differences (P=0.48) in average mortality by period. In formal testing with the linear segmented regression models, we found that the reduction in mortality was comparable for early and late adopters during the HQID period (difference 0.00% points, 95% confidence interval −0.01 to 0.01). The reduction in mortality (−0.04% points for early adopters and −0.02% points for late adopters) slowed down after the abolition of the financial incentives. There continued to be no noticeable differences in mortality reductions between early and late adopters over the pre-HVBP (difference −0.02% points, 95% confidence interval −0.06 to 0.02) and HVBP (0.02, −0.02 to 0.07) period. Comparing the difference in the differences in the trends for the HVBP versus pre-HVBP period, we found no noticeable effect (0.05, −0.03 to 0.13). This suggests that the HQID did not have a meaningful effect on mortality through 2013, even though the hospitals had a decade of experience and volunteered to participate in the demonstration. These results, as expected, did not lead to spillover effects on mortality for the non-targeted conditions. The reduction in mortality from non-targeted conditions during the HQID period was comparable across early and late adopters (0.00, 0.00 to 0.01). After the abolition of the financial incentives, there continued to be no noticeable differences in mortality reductions between early and late adopters (0.00, −0.03 to 0.03) nor were these differences present over the HVBP period (−0.02, −0.05 to 0.01). Comparing the difference in the differences in the trends for the HVBP with the pre-HVBP period, we found no noticeable effect (−0.02, −0.07 to 0.03). Finally, when examining the individual target conditions, we found qualitatively similar effects (web appendix 7).

Fig 2

Standardized 30 day mortality for target conditions. HQID=Hospital Quality Incentive Demonstration; HVBP=Hospital Value-Based Purchasing

Table 4

Trends in mortality over the three periods. Values are percentages unless stated otherwise

Period	Average mortality				Quarterly change in mortality
	Early adopters	Late adopters	PP Difference(95% CI)		Early adopters	Late adopters	PP Difference in trends (95% CI)	PP Difference in difference in trends pre-HVBP v HVBP (95% CI)	P value
Target conditions*
HQID	12.2	12.5	0.27(0.03 to 0.50)		−0.19	−0.19	0.00(−0.01 to 0.01)	0.05(−0.03 to 0.13)	0.25
pre-HVBP	9.8	10.0	0.29(0.01 to 0.56)		−0.04	−0.02	−0.02(−0.06 to 0.02)
HVBP	9.4	9.7	0.28(0.01 to 0.56)		−0.05	−0.07	0.02(−0.02 to 0.07)
Non-target conditions
HQID	9.1	9.2	0.09(−0.08 to 0.27)		−0.11	−0.11	0.00(0.00 to 0.01)	−0.02(−0.07 to 0.03)	0.48
pre-HVBP	7.4	7.6	0.14(−0.06 to 0.34)		−0.06	−0.06	0.00(−0.03 to 0.03)
HVBP	7.2	7.3	0.15(−0.04 to 0.35)		−0.04	−0.02	−0.02(−0.05 to 0.01)

PP=Percentage point; HQID=Hospital Quality Incentive Demonstration; HVBP=Hospital Value-Based Purchasing.
*Acute myocardial infarction, congestive heart failure, and pneumonia

Discussion

While hospitals should be rewarded for better outcomes, not just for the number of inputs, the impact of pay for performance programs has been limited and disappointing. However, there has been a longstanding view that it may take time to make meaningful improvements to care under financial incentive programs. We examined how a group of hospitals, having effectively been under a pay for performance program for more than a decade, fared under the current Hospital Value-Based Purchasing (HVBP) program compared with control hospitals with far less experience in a pay for performance program. We found that despite substantial time, the early adopter Premier Hospital Quality Incentive Demonstration (HQID) hospitals did not have better outcomes and had only marginally better adherence to clinical process measures compared with a group of controls over time. Clinical process scores reached a ceiling for both groups, suggesting that the comparative advantage of the early adopters reduced over time, allowing the late adopters to catch up.

We found, contrary to our hypothesis, that the improvements in clinical process scores were smaller for the early adopters during the HQID period, even though they did perform at a slightly higher level. We further found no noticeable difference in the trends across early and late adopters for the HVBP versus pre-HVBP period. It might have been the case that the definition and communication of specific, measurable processes to provide good quality care as part of the highly visible HQID, resulted in non-participating hospitals also improving their processes, because of a more intrinsic motivation to provide good quality care. The inevitable interaction between early and late adopters might have further led to spillover effects by healthcare personnel teaching each other about standards and approaches to improve quality.

Our findings have important implications for the way policymakers should approach pay for performance programs. These findings provide evidence that having additional time is not likely to turn these programs into a success, at least as far as patient outcomes are concerned. Even for clinical processes, while HQID hospitals began with better performance at baseline, by the end of the study period, the gap between early and late adopters was gone – presumably because of a ceiling effect. This suggests that there is a need to change the measures for clinical process scores.

The limited effects after more than a decade of financial incentives might be explained by different factors. First, the incentives are very small (see web appendix 1 for size and timing of performance bonuses), and given that these incentives only cover a small set of conditions and that hospitals receive revenue from a multitude of payers, this modest incentive is diluted, limiting its impact.34 However, hospital margins tend to be small, usually only a few percentage points, so one might assume that these Medicare payments could still motivate many hospitals.7 Second, the program was extremely complex (in terms of its structure and how hospitals were incentivized), making it more difficult for hospitals to engage meaningfully in the program. Previous work has shown that when the incentivized measure is clear and simple (eg, hospital readmissions in the Hospital Readmission Reduction Program),35 it is possible to observe the impact of financial incentives on performance even within a year after implementation. Finally, given that studies suggest that people tend to discount future gains when the delay in receiving the benefit increases, it is possible that having to wait until the end of the year to receive bonuses or penalties might have reduced the impact.36

Weaknesses of this study

There are limitations to this study. First, observable hospital characteristics used for coarsened exact matching and observable patient characteristics used as covariates in the linear segmented regression models might not sufficiently capture unobserved differences between the early adopters who voluntarily joined the demonstration and the matched late adopters. This might lead to an overestimation of the effect of having more time to adapt to financial incentives because part of this effect might be driven by unobserved characteristics that make those who voluntarily joined different from those who did not (ie, primed to be successful). However, we found virtually no effects so an upwards bias seems unlikely, and we also have a relatively good comparison group so there is no obvious reason to expect large unobserved differences between both groups of acute care hospitals and their patients. Second, generalization to the general population is limited because this study only looks at patients aged 65 and older. However, Medicare covers more than 55 million people and accounts for more than 20% of the total American health spending.37 Third, given the other interventions being implemented simultaneously by Centers for Medicare and Medicaid Services, especially the public reporting under the Hospital Compare initiative for process measures (since 2005) and of 30 day mortality (since 2008), it is difficult to determine the pure effect of pay for performance financial incentives. This might have led to an underestimation of the effects studied here. However, because these interventions were implemented across all hospitals studied, both early and late adopters, we would expect an effect on the outcome levels but not on the differences in the trends across both groups. In other words, the average clinical process scores across early and late adopters might have changed as a result of public reporting, but we expect these changes to be similar across both groups, therefore not affecting the differences in the trends across both groups. Related to this, because the indicators for the HQID were published, late adopters might have already been aware of quality measures to be used for the targeted conditions, in turn already improving their clinical process scores. Next, when comparing trends in clinical processes of HQID and non-HQID hospitals, given that clinical process measures are top coded, it is possible that it may represent an artifact of top coding as most hospitals may be approaching their asymptote.14 Finally, we could not study the cost effectiveness of either of the two pay for performance programs. So far there have been mixed findings on the effectiveness of HVBP programs to simultaneously improve quality and control costs.38

Conclusion

We found that hospitals that have been under financial incentives for more than a decade have not been able to reduce patient mortality more than the late adopters, which had only been under financial incentives for less than three years. The HVBP program as currently structured, is not living up to the promise advocates originally envisioned. Given its cost, policymakers in the USA should consider one of two things: revise the current program or potentially end it. Given major efforts to shift more and more payments toward a value based framework, it is more likely that the program will be revised rather than completely stopped. There are a few ways that this may be done. First, the program should consider increasing the incentives. Currently, the small amount of money at stake is arguably not enough to change the way hospitals are doing business. Second, policy makers should focus on a few measures that matter most to patients (mortality, patient experience, and functional status). The current structure includes numerous measures that are difficult to track and measure by hospital leaders, and therefore, hard to improve. Given the growing worldwide interest in pay for performance programs and the unclear American health policy agenda,39 these findings should be considered by policymakers when assuming that programs like these simply need more time to have a meaningful effect.

Previous studies have found that pay for performance programs have had limited impact on process measures and no impact on patient outcomes

No study has examined whether early adopters of pay for performance programs (ie, Premier Hospital Quality Incentive Demonstration) out performed late adopters of pay for performance programs (ie, Hospital Value-Based Purchasing)

Clinical process scores or 30 day mortality for Medicare beneficiaries were not found to be better at hospitals that have been operating under pay for performance programs for more than a decade

Pay for performance programs as currently implemented are unlikely to be successful in the future, even if their timeframes are extended