Comparison of LACE and HOSPITAL Readmission Risk Scores for CMS Target and Nontarget Conditions.

This study evaluated the utility and performance of the LACE index and HOSPITAL score with consideration of the type of diagnoses and assessed the accuracy of these models for predicting readmission risks in patient cohorts from 2 large academic medical centers. Admissions to 2 hospitals from 2011 to 2015, derived from the Vizient Clinical Data Base and regional health information exchange, were included in this study (291 886 encounters). Models were assessed using Bayesian information criterion and area under the receiver operating characteristic curve. They were compared in CMS diagnosis-based cohorts and in 2 non-CMS cancer diagnosis-based cohorts. Overall, both models for readmission risk performed well, with LACE performing slightly better (area under the receiver operating characteristic curve 0.73 versus 0.69; P ≤ 0.001). HOSPITAL consistently outperformed LACE among 4 CMS target diagnoses, lung cancer, and colon cancer. Both LACE and HOSPITAL predict readmission risks well in the overall population, but performance varies by salient, diagnosis-based risk factors.

Thirty-day hospital readmission rates are in the spotlight as a major health care issue. Excess readmissions are associated with increased costs and low quality of care.[1] In response to these substantial cost burdens and quality-of-care issues, in 2012 US policymakers, as part of the Affordable Care Act, initiated the Hospital Readmissions Reduction Program (HRRP) to penalize hospitals with high readmission rates. This program continues to affect hospitals apart from other changes to the Affordable Care Act. Although numerous interventions have been designed, implemented, and evaluated for their effectiveness in reducing hospital readmissions,[2-5] as of FY2021, about 82% of hospitals subject to HRRP were still penalized.[6] Facing limited resources and capacities, hospitals must prioritize the identification of patients who are most likely to be readmitted and implement interventions targeted to these high-risk groups.[1,7-9]

Therefore, having validated and accurate predictive models with which to estimate the risk of 30-day readmission is critical for the success of any readmission reduction strategy.[10] Such models enable health care providers to identify high-risk patients before discharge, thereby increasing the likelihood that patients at the highest risk of readmission will receive appropriate and timely care.[10-13] However, identifying and predicting risk factors for 30-day readmission is more challenging than, for instance, predicting mortality, because readmission is determined by a complex dynamic of personal, medical, and socioeconomic factors.[7,12] Owing to this complex context, models must draw from various data sources; some validated prediction models rely on administrative data, whereas others incorporate data from electronic health records and laboratory data to identify patients at high risk of readmission.[12,14]

The LACE index and HOSPITAL scores are feasible and validated readmission prediction models with high utility and practicality, since they both determine readmission risk by using administrative data that hospitals routinely collect.[15] The LACE index was initially created from a randomly selected Canadian population as a profiling tool to predict the risk of early death or unplanned readmission within 30 days after discharge from an acute care hospital.[16] The LACE index—an acronym for Length of stay, Acuity of the admission, Comorbidity of the patient (the total Charlson Comorbidity Index [CCI]),[17] and Emergency department attendances in the last 6 months—assigns patients scores from 0 to 19.[16] The HOSPITAL score was developed from the US population to predict the risk of 30-day avoidable readmission. It is calculated by using 7 routinely collected and easily available data elements: Hemoglobin at discharge, discharge from an Oncology service, Sodium level at discharge, Procedure during the index admission, Index Type of admission, number of Admissions during the last 12 months, and Length of stay. Using the HOSPITAL score, the readmission risk is categorized into 3 levels: low (0–4 points), intermediate (5–6 points), and high risk (7 points or above).[18] In contrast to LACE, which incorporates the risk of death or unplanned readmission, the HOSPITAL model is designed to calculate only a patient’s risk of avoidable 30-day readmission before hospital discharge.[1]

Several recent studies have compared independent validations of the LACE index and HOSPITAL score. One study validated the utility of the 2 prediction models in a cohort of 432 adult medical patients at a university-affiliated US hospital and found that the HOSPITAL score had better discrimination and calibration to predict the 30-day readmission risk than did the LACE index.[15] Two other studies, 1 conducted in Switzerland (346 patients) and 1 in Denmark (19 277 patients), also found that the HOSPITAL score was significantly more effective at identifying patients at high risk of avoidable readmission than LACE.[7,19] Other studies used machine learning techniques to refine LACE with additional factors, including patient demographics, acuity of admission, comorbidities (from the CCI[17]), and laboratory variables, and found that the refined models outperformed the original LACE index in predicting 30-day readmission risk.[20,21] Another study demonstrated that there was no statistically significant difference between the HOSPITAL and LACE prediction models in readmission risk identification.[22]

Despite the important findings in the existing literature, there is still a paucity of research comparing the utility and performance of the LACE and HOSPITAL models in large academic medical centers. Moreover, to the best of the authors’ knowledge, no studies have compared these models in cohorts of patients with different diagnoses. Identifying and understanding how the type of diagnosis and hospital setting (eg, acute care hospital, cancer center) affect the risk of readmission are important for increasing the accuracy of predictive models. Therefore, the objectives of this study were to evaluate the utility and performance of the LACE index and the HOSPITAL score with consideration of the type of diagnoses and to assess the accuracy of these models in patient cohorts from 2 large academic medical centers—Houston Methodist (HM), focused on acute care, and MD Anderson Cancer Center (MDACC), as a major cancer center.

Methods

Data

The primary data source used for this study was Vizient’s Clinical Data Base (CDB)/Resource Manager.[23] Both participating institutions are contributing subscribers to the CDB/Resource Manager product (hereafter, CDB) from Vizient, Inc. (Irving, TX). Use of the data for this study was approved by Vizient.

Aggregate inpatient and emergency department (ED) visit data covering the period from January 1, 2011 to December 31, 2015 were extracted for both institutions. An additional 30 days of data were pulled from January 2016 to capture any 30-day readmissions of patients discharged in December of 2015. This study was reviewed and approved by both participating facilities’ Institutional Review Boards (IRBs), with the HM Hospital IRB as the primary oversight IRB for the project.

Creation of the Inter-institutional Master Patient Index

In addition to the CDB data, the study team extracted data from institutional sources to enable the creation of an inter-institutional master patient index (MPI) to elucidate how many patients appeared in both data sets so that patient identifiers could be recoded in the aggregate set. This enriched data set also included additional laboratory values that are required for the calculation of the LACE index and HOSPITAL scores but were not routinely submitted to the CDB until 2014.

Research teams at each institution separately extracted the full set of fully identified patient demographics for all patients admitted or discharged for an inpatient encounter or seen in the ED from January 1, 2011 to December 31, 2015, whether or not they were admitted after evaluation in the ED (see Supplementary Appendix 1, Supplemental Digital Content 1, available at http://links.lww.com/AJMQ/A56, which displays patient data elements used to create the MPI). These sets of data were submitted to a mutual health information exchange trading partner, Greater Houston HealthConnect (GHH), the regional health information exchange for both participating institutions. GHH was engaged to determine overlapping patients who appeared in the data sets submitted by both of the participating institutions. Each data set was subsumed into the community MPI employed by GHH by using both probabilistic and deterministic algorithms. The list of recoded patients that appeared in each set was returned to each respective institution for further aggregation. In summary, each institution received a simple list of new globally unique identifiers along with the institutionally supplied identifiers to enable patient identity mapping in the data sets housed at each institution.

Recoding and Deidentification of Patient and Encounter Data

The list of new identifiers supplied by GHH was recoded again (recoding pass 2) at each institution separately. The full patient lists at each facility were then recoded (recoding pass 3) with new identifiers to further protect patient confidentiality. After the third round of identifier recoding, all unnecessary PHI was removed from each institution’s data sets by separate teams housed at each institution to produce the core sets of recoded data for aggregation. The date of encounter was the lone remaining PHI element in the data sets before aggregation. The recoded data sets were then aggregated for analysis at the Houston Methodist Hospital data center. At no time were identifiable patient data from one institution accessed or viewed by study team members from the other institution.

Calculation of LACE Indices and HOSPITAL Scores

The aggregated data set was further processed to derive CCI scores,[17] LACE indices,[16] and HOSPITAL[18] scores using the parameters and algorithms as specified in the source manuscripts.

Calculation of Inter-institutional Centers for Medicare and Medicaid Services Readmission Rates

Published guidelines for inclusion and exclusion criteria from the Centers for Medicare and Medicaid Services (CMS) were used to derive 30-day readmission rates within the aggregate data set for CMS-defined categories of acute myocardial infarction, heart failure, chronic obstructive pulmonary disease, and pneumonia.[24,25] Three new categories were crafted to capture the readmission rates for patients with diagnoses of any colon cancer, any lung cancer, or cerebrovascular accident with infarction (CVA) (see Supplementary Appendix 2, Supplemental Digital Content 1, available at http://links.lww.com/AJMQ/A56, which demonstrates codes used for the CMS and custom cancer disease cohorts). Encounters for which the primary admitting diagnosis was associated with colon cancer or lung cancer were marked as members of the colon cancer cohort or lung cancer cohort, respectively. Encounters for which the primary diagnosis was cerebrovascular accident with an infarction were classified into the CVA cohort. There were no exclusions for the 2 cancer cohorts or the CVA cohort.

Statistical Analysis

Logistic regression analysis was used to examine the contribution of prognostic variables to the patient’s risk of readmission within 30 days of discharge. The performance of the LACE and HOSPITAL scores was evaluated based on their Bayesian information criterion and area under the receiver operating characteristic curve (AUC). Pairwise comparison of AUCs between the 2 risk scores was conducted. All analyses were performed on Stata version 15.1 (StataCorp LP, College Station, TX). A P value of <0.05 was considered statistically significant.

Results

The aggregated data set contained 291 886 encounters, of which 41 423 (14.2%) were classified as unplanned 30-day readmissions. MDACC accounted for 124 313 (42.6%) of the aggregate encounters, and HM accounted for the balance of 167 573 (57.4%) of the aggregate set. The overall, all-cause, unplanned 30-day readmission rates were 15.6% for MDACC and 13.2% for Houston Methodist (P ≤ 0.001; Figure 1 and see Supplementary Appendix 3, Supplemental Digital Content 1, available at http://links.lww.com/AJMQ/A56, which indicates the results of the univariate logistic regression). Slightly more than half of the patients in the aggregated data set were over 60 years of age (51.4%), and 9.9% of the total population was over 80 years of age. The aggregate population had a median age of 60 years (interquartile range, 48.0–70.0 y) at admission. In the aggregated data set, 50.1% of patients were female, and 61.1% were white (Supplementary Appendix 3, Supplemental Digital Content 1, available at http://links.lww.com/AJMQ/A56). Patients who had a 30-day all-cause unplanned readmission were slightly younger than the aggregate, with a median age of 59 years (interquartile range, 47.0–69.0 y) (P ≤ 0.001) at readmission. The population of patients in this study hailed from all over the world, with concentrations from the southern, southeastern, and midwestern continental United States (Figure 2).

Figure 1.

Flowchart of the study sample.

Figure 2.

Geographic distribution of all patient home addresses with encounters at the participating medical centers from Q1-2011 to Q3-2015, by ZIP code.

Both the LACE and HOSPITAL predictive models for readmission risk performed well in the aggregated data set from both participating academic medical centers. In this aggregate data set, without consideration of diagnosis or procedure classifications, the LACE algorithm performed slightly better, with an AUC of 0.73 (95% confidence interval [CI], 0.73-0.73), while the HOSPITAL algorithm had a slightly lower AUC of 0.69 (95% CI, 0.68-0.69; P ≤ 0.001; Table 1 and see Supplementary Appendix 4, Supplemental Digital Content 1, available at http://links.lww.com/AJMQ/A56, which demonstrates receiver operating characteristic curves comparing LACE and HOSPITAL risk scores). The authors next examined the performance of the LACE and HOSPITAL algorithms using just a patient cohort with the CMS-defined medical diagnoses of acute myocardial infarction, heart failure, chronic obstructive pulmonary disease, and pneumonia; any encounter classified into any of these CMS cohorts was included in this analysis. The algorithms performed the same, with LACE having an AUC of 0.69 (95% CI, 0.67-0.71) and HOSPITAL having an AUC of 0.69 (95% CI, 0.66-0.71; P = 0.380; Table 1 and Figure 3).

Table 1.

Prediction Performance of LACE and HOSPITAL Scores Based on 291 886 Encounters at 2 Facilities

	OR (95% CI)	P  a	BIC	AUC (95% CI)	LACE vs HOSPITAL, P [b]
30-day unplanned readmission					≤0.001
LACE	1.27 (1.26-1.27)	≤0.001	215526	0.73 (0.73-0.73)
HOSPITAL	1.37 (1.36-1.38)	≤0.001	214839	0.69 (0.68-0.69)
30-day unplanned readmission, CMS criteria					0.380
LACE	1.19 (1.16-1.23)	≤0.001	4859	0.69 (0.67-0.71)
HOSPITAL	1.33 (1.26-1.39)	≤0.001	4776	0.69 (0.66-0.71)

Bold font indicates statistical significance P < 0.05.

P value of the logistic regression model.

Comparison of the AUCs of LACE vs HOSPITAL.

Abbreviations: AUC, area under the receiver operating characteristic curve; BIC, Bayesian information criterion; OR, odds ratio.

Figure 3.

Receiver operating characteristic curves for LACE and HOSPITAL risk scores.

When the authors dove deeper and examined the performance of the LACE and HOSPITAL algorithms in the context of each of the 4 CMS diagnosis-defined cohorts and the 3 novel colon cancer, lung cancer and cerebrovascular accident diagnosis cohorts, they found that the HOSPITAL algorithm consistently outperformed the LACE algorithm in all but 1 of the diagnosis categories: colon cancer, for which the 2 algorithms performed equally well (LACE: AUC, 0.66 [95% CI, 0.60-0.72]; HOSPITAL: AUC 0.65 [95% CI, 0.60-0.69]; P = 0.762) (Table 2).

Table 2.

Predictive Performance of LACE and HOSPITAL Scores (Based on Encounters) in Individual CMS or Custom Cohorts

Cohort	OR (95% CI)	P  a	BIC	AUC (95% CI)	LACE vs HOSPITAL, P [b]
Acute myocardial infarction (CMS) Cohort (n = 2829)					≤0.001
LACE	1.00 (0.83-1.20)	0.983	167	0.50 (0.34-0.66)
HOSPITAL	1.69 (1.23-2.31)	0.001	156	0.72 (0.58-0.87)
Any colon cancer (custom) cohort (n = 1746)					0.762
LACE	1.18 (0.82-1.71)	0.378	44	0.66 (0.60-0.72)
HOSPITAL	1.24 (0.66-2.33)	0.507	45	0.65 (0.60-0.69)
Any lung cancer (custom) cohort (n = 2973)					0.026
LACE	1.02 (0.8-1.31)	0.856	86	0.51 (0.41-0.61)
HOSPITAL	1.27 (0.81-1.97)	0.297	85	0.62 (0.46-0.78)
Heart failure (CMS) cohort (n = 5512)					≤0.001
LACE	1.10 (1.03-1.17)	0.003	1303	0.57 (0.53-0.61)
HOSPITAL	1.55 (1.42-1.69)	≤0.001	1198	0.71 (0.68-0.75)
Chronic obstructive pulmonary disease (CMS) cohort (n = 2309)					≤0.001
LACE	0.92 (0.8-1.05)	0.214	309	0.58 (0.48-0.68)
HOSPITAL	1.33 (1.13-1.57)	0.001	290	0.66 (0.56-0.75)
Cerebrovascular accident (custom) cohort (n = 3063)					≤0.001
LACE	1.10 (0.92-1.32)	0.286	209	0.57 (0.43-0.70)
HOSPITAL	2.21 (1.64-2.96)	≤0.001	176	0.83 (0.72-0.93)
Pneumonia (CMS) cohort (n = 9683)					≤0.001
LACE	1.09 (1.00-1.18)	0.042	670	0.57 (0.51-0.63)
HOSPITAL	1.49 (1.33-1.68)	≤0.001	618	0.73 (0.68-0.78)
NONE cohort (n = 255 710)	All are without 30-day readmission based on CMS criteria
LACE
HOSPITAL

Bold font indicates statistical significance P < 0.05.

P value of the logistic regression model.

Comparison of the AUCs of LACE vs HOSPITAL.

Abbreviations: AUC, area under the receiver operating characteristic curve; BIC, Bayesian information criterion; OR, odds ratio.

Discussion

Both the LACE and HOSPITAL predictive models for readmission risk performed well in the aggregated data set from both participating academic medical centers. However, for the patient cohorts with 4 CMS-defined diagnoses and lung cancer, the HOSPITAL algorithm consistently outperformed the LACE algorithm for identifying patients at the greatest risk for readmission. This information is important because hospital systems struggle with HRRP penalties associated with patients with specific primary discharge diagnoses (or procedures). Predicting readmissions from routinely collected and available data likely does not capture the full spectrum of salient risk factors that should be considered when attempting to identify patients at highest risk for readmission. Other patient characteristics are relevant (eg, social determinants of health), but these data are not routinely collected nor available for use in this context. Neither the LACE nor the HOSPITAL algorithms considers these important attributes. Doing so may further improve their performance as the US health care system struggles to apply its limited—and dwindling—resources towards reducing unplanned readmissions.

For this study, the authors elected to use Vizient CDB data that both participating institutions collect and submit to Vizient as part of their ongoing hospital quality improvement operations. The data set is normalized by data dictionary definitions and undergoes quality assurance as a routine part of the data collection and submission process. Having the same data from different institutions, harmonized and cleaned in a consistent, reproducible manner, was as a significant advantage of this cross-institutional study.

This study, like most studies of this nature, which benefit from a full, comprehensive, longitudinal health care use history of patients, is limited in that such data were not available for this analysis. All encounter data encompassing inpatient and ED use across both participating institutions were combined, but some encounters are likely to be missing if patients sought care outside of the participating institutions during the study period. These missing encounters may have affected the conclusions drawn from this analysis. The study is also limited to encounters in 2 academic medical centers located in the same city which may have influenced the findings of this study. The authors were fortunate that the patients included in this study came from a very large geographic cross-section covering the majority of the continental United States, thereby strengthening the generalizability of the results. However, the characteristics of these 2 academic medical centers that are nationally recognized should be considered when assessing its generalizability.

The findings of this study have implications for hospital operation and future research. The results of this study suggest hospital leaders consider readmission risk-stratification as a factor when prioritizing high-resource and high-intensity interventions on patients more accurately by their diagnosis-based risk factors. By including both CMS target and nontarget conditions in the comparison of 2 predictive models, the findings of this study contribute to advancing the HRRP implementation and reconsidering the risk factors by more specified diagnoses. For future research, this study will serve as a model to continue refining predictive models to account for disease-specific and setting-specific differences. Future studies need to incorporate dynamic risk factors of readmissions, such as social determinants of health, which are not currently incorporated into the HOSPITAL or LACE instruments, to further fine-tune the performance of these algorithms. Lastly, further research incorporating more qualitative information derived from focus groups or perhaps Delphi sessions with clinical leaders and policymakers to further enhance the quantitative data. Thematic analysis and multi-method triangulation may uncover new salient elements in our endeavors to appropriately allocate resources in the most effective ways to achieve safer, higher quality, and more cost-effective health care.

Conclusion

The LACE and HOSPITAL readmission risk-prediction algorithms performed well and equivalently in a heterogeneous and diverse population of patients cared for at 2 academic medical centers in Houston, Texas. These readmission risk tools may prove effective in identifying patients at the highest risk for readmission while patients are still in-house, so that early-intervention resources can be directed towards these patients to reduce their risk for unplanned readmission. Much further study is needed to elucidate which readmission risk-reduction programs are effective at reducing readmissions across the diverse population of patients being cared for in the United States.

Acknowledgments

The authors gratefully acknowledge support for this study from Texas Medical Center Health Policy Institute, Houston Methodist, and MD Anderson Cancer Center. The authors wish to acknowledge Jacob M. Kolman, MA, of the Center for Outcomes Research at Houston Methodist Hospital and Amy Ninetto, PhD, ELS, of Scientific Publications at MD Anderson Cancer Center for their contributions to editing the manuscript for language and conformance to journal requirements.

Conflicts of Interest

The authors have no conflicts of interest to disclose.

Funding

Funding for this study was provided by the Texas Medical Center Health Policy Institute. The sponsor had no input into the study design, analysis, or interpretation of the results. The sponsor was not involved in the preparation of this manuscript at any time.

Disclaimer

The content is solely the responsibility of the authors and does not necessarily represent the official views of their respective institutions, the Texas Medical Center Health Policy Institute, Greater Houston HealthConnect, or Vizient, Inc.