Disagreement in Readmission Rates After Total Hip and Knee Arthroplasty Across Data Sets.

BACKGROUND: In 2014, the Affordable Care Act Hospital Readmissions Reduction Program began penalizing hospitals for excessive readmission rates 30 days after total hip arthroplasty (THA) and total knee arthroplasty (TKA). Various data sets with nonstandardized validation processes report readmission data, which may provide conflicting outcome values for the same patient populations.
METHODS: We queried 4 separate data sets: the American Joint Replacement Registry, Centers for Medicare and Medicaid Services billing data, the Vizient data set, and an advanced analytics integration (Cognos) report from our electronic medical record. We identified 2763 patients who underwent primary TKA and THA at a single academic medical center from June 2016 to June 2019. We then matched 613 surgery encounters in all 4 databases. Our primary outcome metric was 30-day readmissions. Fleiss' Kappa was used to measure agreement among the different data sets.
RESULTS: Of the 613 THA and TKA patients, there were 45 (7.3%) readmissions noted. Data collected from the Centers for Medicare and Medicaid Services flagged 41 (6.7%) readmissions, Vizient flagged 11 (1.8%) readmissions, and the American Joint Replacement Registry and Cognos report both flagged 6 (0.98%) readmissions each. None of the readmissions were identified by all 4 data sets. There was significant disagreement among data sets using Fleiss' Kappa (kappa = -0.1318, P = .03).
CONCLUSION: There is disagreement in readmission rates in databases receiving the same patient data after THA and TKA. Care must be taken to establish standard validation processes and reporting methods and scrutiny applied when interpreting readmission rates from various data sets.

Introduction

As hospital reimbursement strategies shift toward value-based models, policy-makers have emphasized using an outcomes-based approach to rate provider performance and quality of care [[1], [2], [3]]. In orthopedics, readmission rate due to complications of surgery is considered an indicator of quality of care [4]. In 2014, the Affordable Care Act Hospital Readmissions Reduction Program began penalizing hospitals for excessive hospital readmissions within 30 days of total hip arthroplasty (THA) and total knee arthroplasty (TKA). These provisions effectively linked hospital payment to quality measures in an attempt to be more representative of the patient experience [1,5].

Even before the advent of this new outcomes-based payment structure, hospital-related quality data were compiled in various data sets, including administrative claims databases and clinical registries. Postoperative complications as outcome measures are often considered to have clinical relevance and validity as indicators of quality over process of care measures [1,5]. Administrative claims data are submitted by hospitals for billing purposes and allow access to a large volume of patient data, including longitudinal encounters over a vast geographical range [5,6]. In contrast, clinical registry data are generally collected for clinical quality surveillance. Owing to these factors, each data set has potential advantages and disadvantages. Clinical registry data have been suggested to have greater validity for quality metrics over administrative claims data sets but with the potential of increased burden placed on the hospital for data collection [[7], [8], [9]].

While these data sets are intended to represent similar information for the same groups of patients, previous comparisons among clinical registries and administrative claims databases have shown stark differences in the data for patients with similar clinical profiles. This has been shown in primary TKA where there are significant variations in complication rates when reviewing similar patient profiles using the same data abstraction method [[10], [11], [12]]. These variations could potentially lead to inaccurate representation of the true outcomes for an institution depending on the selected database.

Prior studies compared similar patient profiles between databases but used deidentified information to do so [10,13]. To our knowledge, no prior study has linked specific patients across databases to compare reported outcomes after TKA and THA [10,13]. We therefore compared 30-day readmissions after TKA and THA among databases supported by our institution.

Material and methods

A retrospective observational study was performed of primary THA and TKA cases performed at a single academic medical center from June 2016 to June 2019. Our institution submits readmission data of TKA and THA patients to 4 databases: the American Joint Replacement Registry (AJRR), the Centers for Medicare and Medicaid Service (CMS), the Vizient database, and our electronic medical record Epic (Cognos) reporting data set. The AJRR is a national clinical data registry operated by the American Academy of Orthopedic Surgeons [14]. CMS collects Medicare administrative claims data based on reimbursement and payment information for beneficiaries [15]. Vizient Incorporated (Irving, TX) is a health-care performance improvement company that has aligned with the University Health System Consortium to create a member-owned health care company that collects member data to form the Vizient Clinical Data Base [16]. This allows our health-care quality team the ability to evaluate our own outcomes data and to compare and contrast this with other participating institutions across the nation. Our institutions' electronic medical record Epic (Verona, WI) provides access to an advanced analytics integration report (IBM Cognos Analytics, Ottawa, Ontario, Canada), which is able to query specific discrete outcomes data across regional health-care systems on the same platform [17]. Each database had different collection, validation processes, and exclusion criteria [[18], [19], [20], [21]] (Table 1). Basic demographic data common to all 4 databases, including age, gender, and race, were collected. Primary payer status was collected but was not available in the AJRR data set.

Table 1

Inclusion and exclusion criteria for each data set.

AJRR	CMS	Vizient	Cognos
Included:• All-cause readmissions to the index event facility within 90 d of index procedure date Excluded:• N/A	Included:• Readmissions at any facility • Elective admissions • Patients enrolled in Medicare FFS Part A for the 12 mo before the date of admission Excluded:• Anyone under 65 y of age • Planned readmissions • Transfers	Included:• All-cause readmissions to the index event facility within 30 d • Transfers Excluded:• Hospice • Chemotherapy • Radiation therapy • Rehabilitation • Dialysis • Delivery/birth • Mental diseases • Alcohol & drug use	Included:• All-cause readmissions to index event facility or partner facility within 30 d Excluded:• Transfers

We identified a total of 2763 primary TKA and THA surgeries, with 2481 in the AJRR data set, 694 in the CMS data set, 2217 in the Vizient data set, and 2287 in the Cognos report. We then used patient identifiers to match 613 surgery encounters between the 4 data sets (Fig. 1). Out of 694 in the CMS data set, only 613 patients were included in all 4 data sets. Patient demographic characteristics were compared among the 4 databases and our matched cohort (Table 2). Our primary outcome was 30-day readmission after primary TKA and THA. The Vizient data set and Cognos reports only offer 30-day readmission data, while the CMS and AJRR data sets allow for either 30- or 90-day readmission data. An Euler diagram was created to identify readmissions that were identified in each data set and those that were present in multiple data sets (Fig. 2) [22]. Fleiss’ Kappa analysis was used to determine the level of agreement between the data sets. Data were managed and analyzed using R software, version 3.5.1 (www.r-project.org).

Figure 1

Euler diagram, matched encounters between data sets.

Table 2

Demographics across datasets and matched cohort.

▪	Matched cohort	AJRR	CMS	Vizient	Cognos	P value
No. of patients	613	2481	694	2217	2287
Sex, N (%) male	231 (37.7)	1022 (41.2)	260 (37.5)	910 (41)	937 (41.0)	.33
Age, mean (SD)	71.9 (9.3)	63.9 (13.3)	72.1 (9.5)	62.9 (13.9)	65.1 (13.8)	<.001
Race, N (%) white	580 (94.6)	2277 (91.8)	629 (90.6)	2025 (91.3)	2086 (91.2)	.78
Primary payer		Not available				<.001
Medicare, N (%)	613 (100)		694 (100)	1128 (50.9)	1157 (50.6)
Commercial, N (%)	0 (0)		0 (0)	762 (34.4)	910 (39.8)
Other	0 (0)		0 (0)	327 (14.7)	220 (9.6)

P values represent comparison of AJRR, CMS, Vizient, and Cognos databases.

Figure 2

Euler diagram, overlap of readmissions between data sets.

Results

The matched cohort included 613 primary THA and TKA procedures. The mean patient age (and standard deviation) was 71.9 + 9.3 years, 231 (37.7%) patients were male, and 580 (94.6%) patients considered themselves Caucasian. As one of the databases was comprised entirely of CMS data, 100% of patients in the final matched cohort had Medicare as the primary payer (Table 2).

Of the 613 THA and TKA patients matched across 3 databases, 45 (7.3%) total readmissions were identified. Data collected from the CMS flagged 41 (6.7%) readmissions, Vizient flagged 11 (1.8%) readmissions, and the AJRR and Cognos report both flagged 6 (0.98%) readmissions each. No single patient with a readmission was identified commonly across all 4 entities (Fig. 2).

Fleiss’ Kappa analysis showed significant disagreement noted among data sets, with a kappa value of - 0.131 (P = .03) (Table 2). This indicated that the data sets had less agreement than would be expected by chance.

Discussion

Value-based reimbursement rewards or penalizes hospitals based on patient outcomes, leading to concern regarding the validity of clinical data sets used by insurers and government payers. This study demonstrates significant discrepancies in 30-day readmission rates after THA and TKA for patient-matched data in 4 clinical and administrative claims data sets supported by our institution.

The disagreement in readmission rates among patient-matched data is likely due to a number of factors, including variations in institutional data collection, validation processes, and reporting methods. Our study showed that the CMS database captured the highest number of readmissions. Furthermore, all matched patients were covered by CMS as a primary payer because of the inclusion of CMS billing data. The average age in the CMS cohort was 72.1 years, which is almost 10 years older than the average patient cohort age of AJRR and Vizient. This age gap correlates with an elevated rate in comorbidities, increasing both the risk for surgical complications and number of readmissions [23,24]. It may be expected that because of the population captured under the CMS data, there would be a higher tendency to have readmissions in those patients. Based on these expectations, it is possible that CMS has established better mechanisms to capture these data in their collection and validation processes.

The history of a database is another influential factor in the development of collection and validation strategies, as is the scope of its institutional network. While CMS has been functioning since 1965, the AJRR was implemented in 2011 as a national registry to provide performance reports for improving care. Heckmann et al. performed an early analysis comparing AJRR performance to other national registries and estimated that the AJRR reported only 28% of total joint arthroplasty procedures in 2016 [25]. Furthermore, a majority of the participating hospitals were academic facilities [25]. The inclusion of nonacademic institutions in the future can potentially alter the range and type of patient data reported.

The AJRR data verification system underwent a rapid transition from ICD-9 to ICD-10-PCS format in 2015, requiring hospital coders to translate surgical procedures into a stricter and more granular clinical coding system [26]. This steep learning curve for hospital administrative coders may have led to a higher level of operator bias. In comparison, data from the CMS claims database are manually submitted by administrative coders for reimbursement purposes. While the Medicare claims database is considered to be one of the largest data sources available, several prior studies have alluded to implications for the laxity in the claims filing process [9,27]. Bozic et al. suggested that a lack of stricter guidelines leads to overrepresentation of complications and diagnoses that increase payment [27].

Finally, Vizient markets itself as a health-care performance improvement company, collecting patient outcome data for quality and cost analytics. As this service procures additional costs, it is limited to hospitals and medical facilities that can afford it. Future analysis of participating hospitals and their patient demographics may help determine how Vizient processes and reports data to cater to its customer base, and how this may affect the data it provides.

Our study was limited by its analysis of patient data from a single institution. Although this may limit generalizability in terms of the specific data verification and reporting processes used at each institution, the utility of this study is found in the direct patient-to-patient comparisons across databases. To our knowledge, existing literature compares outcome measures between larger patient data sets based on similar patient profiles. However, a lack of direct patient matching prevents the identification of specific lapses in information between data sets. Our direct comparison of patient-matched data is a more accurate representation of the differences in data reported by each source. While our single institution-specific reporting may be limited in their generalizability, it is our belief that the discrepancies we experienced between databases may be similar to those of other institutions reporting information to multiple sources. The data sets evaluated in this study have specific collection and validation processes, which are not unique to our institution, and this suggests that other institutions may have similar results.

There is an increased risk for selection bias as a retrospective cohort study. However, the study cases are an accurate representation of the data typically reported by our institution to outside databases for quality measures. In addition, we reported only one patient outcome value, 30-day readmission. It is unknown whether the reporting of other variables such as early and late postoperative infection or reoperations may have higher agreement between data sets than the readmissions data we gathered.

Our aim was not to identify which data set provides the most accurate information. Rather, the aim was to report that these data sets disagree significantly. This should be taken into account when using data sets to determine patient outcomes for hospital and physician performance analysis. Further studies are warranted to determine which, if any, databases are more successful at reporting specific types of patient outcome information. In addition, only 613 of the 694 cases from the CMS data set were available in all data sets. This suggests that not only are the readmissions not tracked consistently among the data sets but also not all eligible cases make it into the data sets for which they are eligible.

Conclusions

Our findings demonstrate significant disagreement in 30-day readmission rates after THA and TKA in patient-matched data across 4 prominent data sets. Variation across data sets was likely the result of different collection and validation processes established by each of these entities. Therefore, diligence must be used to evaluate sources of data for their strengths and limitations before use, especially in the current context of value-based reimbursement. In the future, expansion of this patient-matched cohort approach at a multi-institutional level could provide more definitive evidence for the cause of disagreement in clinical registry and administrative claims data.

Conflicts of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this article.