Center Variability in Medicare Claims-Based Publicly Reported Transcatheter Aortic Valve Replacement Outcome Measures.

Background Public reporting of transcatheter aortic valve replacement (TAVR) claims-based outcome measures is used to identify high- and low-performing centers. Whether claims-based TAVR outcomes can reliably be used for center-level comparisons is unknown. In this study, we sought to evaluate center variability in claims-based TAVR outcomes used in public reporting. Methods and Results The study sample included 119 554 Medicare beneficiaries undergoing TAVR between January 2014 and October 2018 based on procedure codes in 100% Medicare inpatient claims. Multivariable hierarchical logistic regression was used to estimate center-specific adjusted rates and reliability (R) of 30-day mortality, discharge not to home/self-care, 30-day stroke, and 30-day readmission. Reliability was defined as the ratio of between-hospital variation to the sum of the between- and within-hospital variation. The median (interquartile range [IQR]) center-level adjusted outcome rates were 3.1% (2.9%-3.4%) for 30-day mortality, 41.4% (31.3%-53.4%) for discharge not to home, 2.5% (2.3%-2.7%) for 30-day stroke, and 14.9% (14.4%-15.5%) for 30-day readmission. Median reliability was highest for the discharge not to home measure (R=0.95; IQR, 0.94-0.97), followed by the 30-day stroke (R=0.92; IQR, 0.87-0.94), 30-day mortality (R=0.86; IQR, 0.81-0.91), and 30-day readmission measures (R=0.42; IQR, 0.35-0.51). Across outcomes, there was an inverse relationship between center volume and measure reliability. Conclusions Claims-based TAVR outcome measures for mortality, discharge not to home, and stroke were reliable measures for center-level comparisons, but readmission measures were unreliable. Stakeholders should consider these findings when evaluating claims-based measures to compare center-level TAVR performance.

Medicare Provider Analysis and Review

transcatheter aortic valve replacement

Clinical Perspective

What Is New?

The rapid growth in transcatheter aortic valve replacement (TAVR) has led to the development of claims‐based outcome measures for public reporting, which rely on risk adjustment models to detect differences in outcomes across centers.

Our analysis of national Medicare claims data suggests that claims‐based TAVR outcome measures for mortality, discharge not to home, and stroke can reliably detect differences among hospitals, but readmission measures were unreliable.

Center‐level case volume was directly related to the reliability of TAVR outcome measures, with low‐volume centers frequently having unreliable outcome measures.

What Are the Clinical Implications?

Measures of TAVR mortality, discharge not to home, and stroke can reliably detect center‐level differences over and above statistical noise, which could be used to encourage improvements in TAVR quality.

Readmission rates for TAVR may not be an appropriate measure for public reporting, because it may reflect issues beyond the quality of care provided by a center.

Ensuring TAVR quality in low‐volume centers is critical to improving access, but reliably measuring quality in these centers remains challenging.

Transcatheter aortic valve replacement (TAVR) is a minimally invasive therapy for patients suffering from symptomatic severe aortic stenosis, the most common heart valve condition in adults. Since its approval in 2012, clinical trials have shown TAVR to be similarly safe and effective compared with surgery across all surgical risk categories, which has driven rapid growth in its use. , , , To facilitate continued expansion of TAVR across the country, the Centers for Medicare and Medicaid Services adopted less‐restrictive reimbursement requirements for hospitals providing or seeking to provide TAVR. At each step, providers, payers, and policymakers have sought to ensure that quality of care is maintained as TAVR expands to new patients and hospitals, often through tracking and benchmarking hospital performance using risk‐adjusted outcomes. , Recently, US News & World Report began publicly rating hospital performance according to TAVR outcomes in 2020.

Whether outcome measures derived from administrative claims data can reliably detect differences in outcomes among hospitals remains unclear. Prior studies have already shown variability in hospital‐level mortality and readmission rates for TAVR. , It is assumed that differences in outcomes among hospitals can be attributed to systematic differences in quality after adjusting for patient risk. , On the other hand, variation in outcomes among hospitals could be the product of random statistical variation, particularly in hospitals with low case volumes. , The statistical reliability of TAVR outcome measures used to compare hospital performance is critical to future benchmarking, quality improvement efforts, and potentially reimbursement.

Therefore, this study sought to evaluate variability in claims‐based TAVR outcome measures used in public reporting. We estimated center‐level rates of 30‐day mortality, discharge not to home/self‐care, 30‐day stroke, and 30‐day readmission after TAVR, and the proportion of center‐level outcome variation that can be attributed to systematic differences among hospitals, referred to as reliability (R). Additionally, this study sought to establish the case‐volume threshold needed to achieve adequate reliability for hospital‐level comparisons for each outcome measure, and the proportion of current hospitals that meet the threshold.

Methods

This study was deemed exempt from human subject protections by the University of Michigan Institutional Review Board (HUM00163969). M.P.T. and H.H. both have full access to all of the data in the study and take responsibility for their integrity and the data analysis. Medicare data will not be made publicly available because of data use agreement restrictions, but methods and materials can be made available to any researcher for purposes of reproducing the results. All analyses were performed using SAS 9.4 (SAS Institute, Cary, NC), and statistical tests were deemed significant at α<0.05 (2‐sided).

Data Sources and Study Sample

Medicare administrative claims data from 2014 to 2018 were used for this study, including the 100% Medicare Provider Analysis and Review (MedPAR) and Medicare Beneficiary Summary File data sets (Medicare Data Use Agreement No. 21521). The MedPAR data include information for all inpatient discharges, including beneficiary demographics, admission and discharge information, and diagnosis and procedure codes. The Medicare Beneficiary Summary File includes information on beneficiary Medicare enrollment and dates of death. Additionally, hospital characteristics were obtained from the American Hospital Association Annual Survey and Medicare Impact Files.

The study sample included any Medicare beneficiary with a procedure code for TAVR during an inpatient admission and fully entitled to Medicare Part A and Part B for the 6 months before admission and at 90 days after discharge. International Classification of Diseases Ninth Revision (ICD‐9; 35.05, 35.06) and Tenth Revision (ICD‐10; 02RF3*) procedure codes were used to identify TAVR during inpatient admissions.

Outcomes

The primary outcomes for this study were 30‐day mortality, discharge not to home, 30‐day stroke, and 30‐day readmission, which are currently used to profile hospital TAVR performance in US News & World Report. Date of procedure from the MedPAR data and date of death from the Medicare Beneficiary Summary File data were used to define death within 30 days of procedure. Discharge location was used to define discharge not to home, which included discharge to a skilled nursing facility, intermediate care facility, swing bed, inpatient rehabilitation, long‐term care facility, or other health care facility. Stroke within 30 days was defined as an inpatient admission with diagnosis of stroke within 30‐days of the procedure (ICD‐9: 433*, 434*, 436*, 437.9, 997.02; ICD‐10: I63*, I97.81, I97.82). Beneficiaries who died in‐hospital were excluded from the readmission definition. Admission and discharge dates from MedPAR were used to identify readmissions within 30‐days of discharge. Readmissions that were potentially planned readmissions as defined by Medicare were also excluded from this definition.

Covariates

Our study included covariates at the beneficiary and hospital level. Beneficiary‐level factors were based on published criteria and drawn from the MedPAR and the Medicare Beneficiary Summary File, and included age, sex (men versus women), transfer status, year of admission, Medicare eligibility status (age, disability, or end‐stage renal disease), Medicaid dual eligibility, and 29 Elixhauser comorbidities. , , Race or ethnicity and elective procedure status were not included in risk‐adjusted models because they are not used in adjustment for publicly reported outcomes.

Hospital Risk‐Adjusted Outcomes

Hospital risk‐adjusted outcomes were estimated for TAVR using a 2‐stage approach applied in publicly reported outcomes. The first stage used a standard logistic regression model to estimate a beneficiary‐level probability of the outcome adjusting for beneficiary factors to generate the expected probability. The second stage used a hierarchical logistic regression model to estimate the probability of the outcome adjusting for the expected probability generated from the first model and a hospital‐level random effect to get the predicted probability. Predicted and expected outcome probabilities were summed at the hospital level and used to estimate the ratio of predicted to expected for each hospital . Finally, we multiplied the hospital‐specific predicted to expected ratio with the overall sample outcome rate to calculate the hospital risk‐adjusted outcome rate. Model fit information for publicly reported TAVR outcomes can be found in Table S1.

Statistical Analysis

Reliability of hospital risk‐adjusted outcome measures were estimated using established methods. , , Briefly, measure reliability is defined as the ratio of between‐hospital variation in outcome rates (the signal) to the sum of the between‐hospital (the signal) and within‐hospital variation in outcome rates (the noise), or signal/(signal+noise). In other words, an outcome measure’s R value (range, 0–1) is the proportion of hospital‐level variation that can be attributed to systematic differences among hospitals, which are assumed to be a proxy for hospital quality. The between‐hospital variation estimate was drawn from the hierarchical logistic regression models as the variance in hospital random intercepts and is uniform across all hospitals. The within‐hospital variation was estimated as the standard error of a proportion, or √[P(1−P)/n], where P represents the probability of the outcome within a given hospital (ie, the hospital observed mortality or readmission rate), and n is the hospital case volume. Because both the observed outcome rates and case volume vary by hospital, each hospital has a unique within‐hospital variation estimate and therefore a unique estimate of reliability.

Dot plots were created to show the relationship between a hospital’s case volume and reliability estimates for individual outcome measures. A fitted Loess curve was added to illustrate the relationship between reliability and case volume at the hospital level. Median reliability (interquartile range [IQR]) was estimated for unadjusted and adjusted TAVR outcomes. Using prior studies, we used common benchmarks of reliability for group‐level comparisons (R=0.7 and 0.9) and estimated the procedural volume threshold required to exceed those benchmarks. , , Finally, we estimated the number of reliable risk‐adjusted outcome measures per hospital (range, 0–4) for both benchmark levels. As a sensitivity analysis, we repeated our analysis using a 3‐year period (2016–2018), which reflects time horizons used in other TAVR quality measures in the Society of Thoracic Surgeons/American College of Cardiology Transcatheter Valve Therapy Registry.

Results

Between 2014 and 2018, a total of 119 554 Medicare beneficiaries underwent TAVR at 576 hospitals, with a mean of 211 beneficiaries per hospital (SD, 226). The characteristics of TAVR beneficiaries are shown in Table 1. Within this sample, 3762 (3.2%) died within 30 days, 50 672 (43.2%) were discharged not to home/self‐care, 2980 (2.5%) had a stroke within 30 days, and 17 498 (14.9%) were readmitted within 30 days of discharge.

Table 1

Characteristics of Patients With Transcatheter Aortic Valve Replacement Outcome Status

Characteristic	No. (%) or mean (SD)
Age, y, mean (SD)	82 (8)
Men, n (%)	63 341 (53.0)
Year of admission, n (%)
2014	12 410 (10.4)
2015	17 586 (14.7)
2016	25 435 (21.3)
2017	31 934 (26.7)
2018	32 189 (26.9)
Medicare eligibility, n (%)
Aged without ESRD	113 260 (94.7)
Aged with ESRD	3832 (3.2)
Disabled without ESRD	1475 (1.2)
Disabled with ESRD	751 (0.6)
ESRD only	236 (0.2)
Dual eligibility, n (%)	13 990 (11.7)
Transferred patient, n (%)	6461 (5.4)
Clinical factors, n (%)
Acquired immune deficiency syndrome	66 (0.1)
Alcohol abuse	1609 (1.4)
Chronic blood loss anemia	2895 (2.4)
Chronic pulmonary disease	39 535 (33.1)
Coagulopathy	20 795 (17.4)
Congestive heart failure	15 021 (12.6)
Deficiency anemias	38 168 (31.9)
Depression	13 323 (11.1)
Diabetes with chronic complications	24 515 (20.5)
Diabetes without chronic complications	29 353 (24.6)
Drug abuse	464 (0.4)
Fluid and electrolyte disorders	35 977 (30.1)
Hypertension	77 462 (64.8)
Hypothyroidism	28 569 (23.9)
Liver disease	3775 (3.2)
Lymphoma	1692 (1.4)
Metastatic cancer	999 (0.8)
Obesity	25 258 (21.1)
Other neurological disorders	11 137 (9.3)
Paralysis	3610 (3.0)
Peptic ulcer disease/bleeding	1241 (1.0)
Peripheral vascular disease	35 153 (29.4)
Psychoses	1561 (1.3)
Pulmonary circulation disease	1369 (1.2)
Renal failure	45 735 (38.3)
Rheumatoid arthritis	6969 (5.8)
Solid tumor without metastasis	3722 (3.1)
Valvular disease	21 588 (18.1)
Weight loss	6707 (5.6)

ESRD indicates end‐stage renal disease.

The distribution of crude hospital‐level outcomes can be found in Figure 1. After risk‐adjustment, the median (IQR) outcome rates were as follows: 3.1% (2.9%–3.4%) for 30‐day mortality, 41.4% (31.3%–53.4%) for discharge not to home/self‐care, 2.5% (2.3%–2.7%) for 30‐day stroke, and 14.9% (14.4%–15.5%) for 30‐day readmission.

Figure 1

Distribution of center‐level TAVR outcomes.

TAVR indicates transcatheter aortic valve replacement.

The relationship between hospital‐level volume and reliability for 30‐day risk‐adjusted outcomes are shown in Figure 2. In general, as volume increased, the reliability of TAVR outcome measures increased. Median reliability was greatest for the 30‐day stroke measure for the discharge not to home/self‐care measure (R=0.95; IQR, 0.94–0.97), which exceeds both reliability benchmarks for group comparisons (Table 2). In other words, 95% of the variation in 30‐day stroke among hospitals can be attributed to hospital‐level differences, with the remaining 5% representing statistical noise. Median reliability for 30‐day stroke (R=0.92; IQR, 0.87–0.94) and 30‐day mortality measures (R=0.86; IQR, 0.81–0.91) were also above the benchmarks for group‐level comparisons. The reliability of the 30‐day readmission measure was R=0.42 (IQR, 0.35–0.51), which was below the benchmark for group‐level comparisons.

Figure 2

Relationship between center volume and reliability of TAVR outcome (n=567).

TAVR indicates transcatheter aortic valve replacement.

Table 2

Median Reliability of Transcatheter Aortic Valve Replacement 30‐Day Outcome Measures

Measure	Frequency (%)	Risk‐adjusted rate, median (IQR)	Between‐hospital variance	Reliability, median (IQR)	Volume threshold for R=0.70, median (IQR)	Hospitals with R≥0.70, n (%)	Volume threshold for R=0.90, median (IQR)	Hospitals with R≥0.90, n (%)
30‐day mortality	3762 (3.2)	3.1% (2.9%–3.4%)	0.08	0.86 (0.81–0.91)	24 (17–34)	527 (93.0%)	368 (255–508)	165 (29.1%)
Discharge not to home*	50 672 (43.2)	41.4% (31.3%–53.4%)	0.81	0.95 (0.94–0.97)	1.8 (1.5–2.0)	567 (100%)	27.5 (22.2–30.2)	529 (93.3%)
30‐day stroke	2980 (2.5)	2.5% (2.3%–2.7%)	0.10	0.92 (0.87–0.94)	11 (6–16)	552 (97.4%)	161 (89–238)	324 (57.1%)
30‐day readmissions*	17 498 (14.9)	14.9% (14.4%–15.5%)	0.02	0.42 (0.35–0.51)	1438 (1220–1650)	4 (0.7%)	21 400 (18 148–24 544)	2 (0.4%)

IQR indicates interquartile range; and R, reliability.

Excludes patients who died in‐hospital: n=2263.

The volume thresholds for highly reliable measures also differed by outcome. Hospitals would need to have at least 28 cases to have a highly reliable measure (R=0.90 benchmark) for discharge not to home/self‐care, which was met by 93.3% of hospitals. The volume requirements for highly reliable 30‐day stroke and mortality measures were 161 and 368 cases over the 5‐year measurement period. These volume thresholds were met by 324 (57.1%) and 165 (29.1%) hospitals in this sample for 30‐day stroke and mortality, respectively. For hospitals to have acceptable reliability in 30‐day readmission (R=0.7), they would need to have at least 1438 cases over the 5‐year measurement period. This threshold was only met by 4 hospitals (0.7%) in the sample.

Using a 3‐year period, estimates of between‐hospital variation and measure reliability were qualitatively similar for all measures (Table S2). However, the shorter time horizon led to fewer hospitals having reliable outcome measures for all measures because of smaller volume estimates.

When using the R=0.7 benchmark for reliable outcome measures, only 3 hospitals (0.5%) met the benchmark for all 4 outcome measures (Table 3). A total of 509 (89.8%) hospitals achieved reliability for 3 of the 4 measures, and 42 (7.4) hospitals achieved reliability for 2 of the 3 measures. Using the benchmark for high reliability (R=0.9), no hospitals achieved the benchmark for all 4 measures, and 109 (19.2%) achieved this benchmark for 3 of 4 measures. A plurality of hospitals met the high‐reliability benchmark for only 2 of the 4 measures.

Table 3

Number of Reliable Outcome Measures Per Hospital (n=567)

Reliability benchmark	No. (%) of reliable outcome measures
	0	1	2	3	4
Reliability ≥0.70	0 (0.0)	13 (2.3)	42 (7.4)	509 (89.8)	3 (0.5)
Reliability ≥0.90	14 (2.5)	211 (37.2)	233 (41.1)	109 (19.2)	0 (0.0)

Discussion

The usefulness of hospital‐level quality measures is dependent upon its ability to identify systematic differences among hospitals over and above random statistical noise. Through this analysis, 5‐year TAVR outcome measures for 30‐day mortality, discharge not to home/self‐care, and 30‐day stroke demonstrated high reliability to detect systematic difference over random statistical noise. However, 30‐day readmission rates for TAVR did not demonstrate sufficient reliability for group‐level comparisons, with only 42% of the variability in readmission rates attributed to the hospital. We also found a strong relationship between center volume and measure reliability, with lower volume centers exhibiting less‐reliable outcome measures.

There are limitations to our study that should be considered. First, Medicare claims data lack important clinical data on severity and extent of disease, which may explain some of the observed hospital‐level variation in TAVR outcomes. However, the purpose of this study was to evaluate publicly reported outcomes that are derived from administrative claims data. Second, our study focused on a single example of publicly reported outcome measures for TAVR. Different risk‐adjustment models may result in different estimates of between‐hospital variation, and should be evaluated when they become available. Third, TAVR volumes have continually risen during and after the time period used in our analysis (2014–2018), which were reflected in our sample. Future work should seek to understand how the inclusion of more recent data may impact the findings of our study as TAVR volume continues to grow. , Fourth, our study did not analyze the same data used in public reporting. Although attempts were made to mimic existing approaches, differences in data management may produce results different from current publicly reported measures. Finally, our study sample includes only Medicare fee‐for‐service beneficiaries, which was done to mimic current publicly reported TAVR outcomes that use Medicare claims data. These findings may not be generalizable to non‐Medicare fee‐for‐service populations.

The findings from this study have important implications for ensuring TAVR quality through public reporting. Performance profiling is reliant on the ability to distinguish systematic differences among providers. Our results suggest that claims‐based TAVR outcomes for mortality, discharge location, and stroke, achieves this purpose. However, a critical question remains: Are claims‐based TAVR outcomes valid measures of performance? There is evidence to suggest that hospital performance derived from claims data may differ from performance derived from clinical registries. Nevertheless, it is likely that public reporting will have significant implications for TAVR quality. Evaluations of public reporting for other cardiovascular procedures have demonstrated significant improvements in quality, , but also unintended effects on access to care and disparities. , , More research is needed to better understand the concordance between publicly reported TAVR outcomes derived from claims data and clinical registries, and the subsequent impact of public reporting on TAVR care.

One unanticipated finding of our study pertains to the high reliability of the 30‐day stroke measure for TAVR. An analysis of the Society of Thoracic Surgeons/American College of Cardiology Transcatheter Valve Therapy Registry demonstrated that a risk model for in‐hospital stroke using clinical data had difficulty predicting the outcome using rich clinical registry data (C statistic, 0.622). The present study did not seek to develop a prediction model for 30‐day stroke following TAVR, but rather estimated the extent to which patient‐level variation in 30‐day stroke outcomes can be attributed to the hospital in which the patient was treated. Similar to our study, an analysis of postoperative stroke rates following cardiac surgery identified a strong hospital‐level effect after adjusting for patient risk. The analytic approach used in this study and in publicly reported outcome measures more broadly assumes that variation in risk‐adjusted outcomes attributed to the hospital level is the direct result of differences in quality. However, other hospital‐specific practices unrelated to quality may also explain systematic differences in outcomes, such as diagnostic and coding patterns. A comparison of post‐TAVR stroke outcomes in clinical trial data versus administrative claims data showed stroke rates in trial data to be higher than those captured in claims data, with the sensitivity of claims‐based algorithms to detect stroke between 19% and 67%. This could be explained by active stroke surveillance in clinical trials or limitations in administrative claims coding of stroke. More work is needed to better understand the mechanisms underlying between‐hospital differences in 30‐day stroke rates for TAVR.

Using readmission rates to profile hospital TAVR quality may be more problematic. Readmission following surgery is widely accepted as a measure of care quality. The Society of Thoracic Surgeons benchmarks hospitals on readmissions after surgical aortic valve replacement, although they use risk models and comprehensive clinical data from the Society of Thoracic Surgeons Adult Cardiac Surgery Database, rather than Medicare administrative claims. Perhaps more importantly, readmission measures are foundational in Medicare pay‐for‐performance and public reporting programs, although they have not yet been implemented in aortic valve replacement. Prior work has shown wide variability in TAVR readmissions across hospitals, suggesting it may be a useful quality measure. , However, our findings show that less than half of the between‐hospital variation in readmissions was attributed to the hospital itself, because the variance estimate attributed to the hospital is markedly lower compared with other measures. Our findings add to the literature addressing the reliability of readmission rates for cardiac procedures, which have had mixed findings for coronary artery bypass grafting. , These results may not be surprising, because studies have consistently demonstrated that hospital readmissions are often a reflection of socioeconomic factors, rather than the clinical care provided in the initial admission. , Readmissions may still be an important quality indicator for TAVR, and efforts to reduce readmissions may be beneficial to patients, but their use in performance profiling may be limited.

Profiling TAVR outcomes in low‐volume hospitals using claims‐based outcomes may continue to be a challenge. Current hospital performance profiling methods are often sensitive to volume, with lower‐volume hospitals exhibiting greater statistical noise. , Our findings similarly highlight a strong relationship between TAVR volume and outcome measure reliability, with lower‐volume hospitals exhibiting less‐reliable outcomes. The claims‐based outcomes evaluated in this study attempt to address the low‐volume problem by using a rolling 5‐year window for measurement, which will be less sensitive to year‐to‐year changes in performance. If the measurement window were shortened to include fewer years, the resulting decline in volume would likely have detrimental effects to the reliability of TAVR outcome measures. Alternative methods to minimizing statistical noise in low‐volume hospitals may be preferred, referred to as reliability adjustment or shrinkage methods. , However, these methods may mask legitimate outcome deviations in low‐volume hospitals, where consistent evidence has demonstrated volume‐outcome relationships with outcomes improving as volume increases. , , , , Concerns about ensuring quality in low‐volume TAVR centers have intensified with Medicare’s recent decision to lower volume requirements for hospitals seeking to maintain or begin TAVR programs. Professional societies have also echoed these concerns, and proposed additional paths to ensure quality in low‐volume hospitals, such as requiring root cause analyses of adverse events. Policymakers should consider our findings when developing and implementing performance metrics to track and benchmark quality in aortic valve replacement.

Conclusions

Claims‐based TAVR outcome measures for mortality, discharge not to home, and stroke were reliable measures for hospital comparisons, but readmission measures were not reliable. Stakeholders should consider our findings when deciding to use claims‐based measures to evaluate and compare hospital‐level TAVR performance.

Sources of Funding

This project was supported by the Institute for Healthcare Policy and Innovation at the University of Michigan. Access to data was supported by Dr Dimick through a federal grant from the National Institute on Aging (R01AG039434).

Disclosures

Dr Thompson receives partial salary support from Blue Cross Blue Shield of Michigan and receives funding from the Agency for Healthcare Research and Quality (K01HS027830‐01A1) outside of the submitted work. Dr Brescia is supported by the National Research Service Award postdoctoral fellowship (5T32HL076123) outside of the submitted work. Dr Sukul receives partial salary support from Blue Cross Blue Shield of Michigan outside of the submitted work and is a member of the Government Relations Committee for the Society for Cardiovascular Angiography & Interventions, which had no role in the work herein. Dr Likosky is a consultant to the American Society of Extracorporeal Technology and receives funding from the Agency for Healthcare Research and Quality and National Institutes of Health, all outside of the submitted work. The remaining authors have no disclosures to report.

Tables S1–S2

Click here for additional data file.