Nonelective Rehospitalizations and Postdischarge Mortality: Predictive Models Suitable for Use in Real Time.

BACKGROUND: Hospital discharge planning has been hampered by the lack of predictive models.
OBJECTIVE: To develop predictive models for nonelective rehospitalization and postdischarge mortality suitable for use in commercially available electronic medical records (EMRs).
DESIGN: Retrospective cohort study using split validation.
SETTING: Integrated health care delivery system serving 3.9 million members. PARTICIPANTS: A total of 360,036 surviving adults who experienced 609,393 overnight hospitalizations at 21 hospitals between June 1, 2010 and December 31, 2013. MAIN OUTCOME MEASURE: A composite outcome (nonelective rehospitalization and/or death within 7 or 30 days of discharge).
RESULTS: Nonelective rehospitalization rates at 7 and 30 days were 5.8% and 12.4%; mortality rates were 1.3% and 3.7%; and composite outcome rates were 6.3% and 14.9%, respectively. Using data from a comprehensive EMR, we developed 4 models that can generate risk estimates for risk of the combined outcome within 7 or 30 days, either at the time of admission or at 8 AM on the day of discharge. The best was the 30-day discharge day model, which had a c-statistic of 0.756 (95% confidence interval, 0.754-0.756) and a Nagelkerke pseudo-R of 0.174 (0.171-0.178) in the validation dataset. The most important predictors-a composite acute physiology score and end of life care directives-accounted for 54% of the predictive ability of the 30-day model. Incorporation of diagnoses (not reliably available for real-time use) did not improve model performance.
CONCLUSIONS: It is possible to develop robust predictive models, suitable for use in real time with commercially available EMRs, for nonelective rehospitalization and postdischarge mortality.

Adults discharged from the hospital have substantially elevated risk for morbidity and mortality,1–3 which has led Krumholz to postulate the existence of “posthospital syndrome.”4 The Affordable Care Act established the Hospital Readmissions Reduction Program in which the Centers for Medicare and Medicaid Services report risk-adjusted rehospitalization rates and reduce payment for certain conditions with “excess” 30-day rehospitalization rates.5,6 Discharge planners must also address the fact that hospitalized patients’ risk for specific events (such as rehospitalization for an acute illness) may vary over time. This is likely because nonelective rehospitalizations and posthospital mortality involve complex interactions between medical, social, and health services factors.

Lack of accurate predictive models continues to limit descriptive analyses and attempts at implementing interventions to support discharge planning.7,8 Most existing models rely on administrative data and cannot be calculated before patient discharge. Incorporating clinical data improves the ability to predict for other outcomes (eg, inpatient mortality9,10), but few models that could support discharge planning have incorporated clinical and/or patient-reported data.11–17

We have described use of granular electronic medical record (EMR) data for analyzing in-hospital deterioration,18,19 risk adjustment for intensive care unit (ICU) patients,20 and risk adjustment for all hospitalized patients.21–23 The setting for our work is Kaiser Permanente Northern California (KPNC), an integrated health care delivery system that routinely uses severity of illness and longitudinal comorbidity scores for internal quality assurance. As is the case with some university hospitals,24 KPNC is starting to embed predictive models into the EMR. In November of 2013, KPNC began a 2-hospital early warning system pilot project that provides clinicians in the emergency department (ED) and general medical-surgical wards with a severity of illness score and longitudinal comorbidity score (previously described in this journal23), as well as an in-hospital deterioration risk estimate, in real time.25,26

Our objective was to develop a set of predictive models that take advantage of the existing electronic infrastructure at the 2 KPNC pilot sites. We aimed to develop models with these attributes: (1) it should be possible to instantiate them in a commercially available EMR so that risk estimates can be available in real time; (2) focusing on the immediate postdischarge period (7 and 30 d), they target 2 overlapping postdischarge events: nonelective rehospitalizations and postdischarge mortality; (3) their statistical characteristics should be well described, and they should have good calibration; and (4) timing of the models should address the differing operational requirements of an integrated health care delivery system.

METHODS

This project was approved by the KPNC Institutional Review Board.

Under a mutual exclusivity arrangement, salaried physicians of The Permanente Medical Group Inc., care for 3.9 million Kaiser Foundation Health Plan Inc. members at facilities owned by Kaiser Foundation Hospitals Inc. All KPNC facilities use the same information systems with a common medical record number.27 The Epic EMR (http://www.epicsystems.com), known internally as KP HealthConnect (KPHC), was fully deployed in 2010.

Our setting consisted of 21 hospitals described previously.18,19,21,23 We constructed our cohort by initially identifying all hospital stays for patients meeting these criteria: (1) patient survived an overnight inpatient hospitalization that began from June 1, 2010 through December 31, 2013 (excluding hospitalizations for observation or 1-day surgery not resulting in an overnight stay); (2) initial hospitalization occurred at a KPNC hospital (if transfer from an outside hospital occurred, the first hospital stay was covered by KPNC); (3) age 18 years and above at the time of admission; (4) hospitalization was not for childbirth (postdelivery pregnancy complications were included); and (5) KPHC was functioning at the hospital for ≥3 months. As previously reported, we then linked hospital stays for transferred patients21 and identified initial admission location.18,19,23 Rehospitalizations were assigned to the discharging hospital. Using this methodology, we could classify all hospitalizations as being an index hospitalization, a rehospitalization, or both.

Because the purpose of our model was to obtain point estimates of the probability of a postdischarge event (as opposed to explaining the effects of individual predictors), we included all hospitalizations experienced by the patients in our cohort, ignoring within-patient clustering (random) effects. In addition, we used admissions up to 7 days or 8–30 days before the index admission as predictors in the model. We ignored the correlation structure generated by multiple admissions for a single patient because point estimates are known to be insensitive to misspecification of the random effects distribution.28 However, to explore the effect of multiple admissions on the final predictions, we conducted sensitivity analyses where we built a model in which each patient only contributed 1 index hospitalization in the development dataset and validated the model in the same validation dataset described above.

We developed 4 predictive models (ED 7, ED 30, discharge day 7, and discharge day 30) based on: (1) timing of risk estimate provision (in the ED or at 08:00 on the discharge day), and (2) outcome time frame (7 or 30 d after discharge). The principal dependent variable was occurrence of a composite outcome (death and/or nonelective rehospitalization) within 7 or 30 days after discharge. Although a number of methods exist to define unplanned or nonelective admission,29,30 currently, no standard definition exists for defining a nonelective readmission. Consequently, we defined a nonelective rehospitalization as one having at least one of these characteristics: it was due to an ambulatory care sensitive condition (thus potentially preventable) as defined by the Agency for Healthcare Research and Quality31; admission occurred through the ED; and/or, at readmission, the patient had a Laboratory Acute Physiology Score, version 2 (LAPS223)≥60, which is associated with a mortality risk of 7.2% and approximates the average mortality risk among inpatients who triggered a real-time alert in the study of Kollef et al.24 We included this criterion to ensure that we did not miss urgent transfers from the clinic to the hospital that bypassed the ED. Linked state mortality data were unavailable for the entire study period, so we ascertained 7- and 30-day mortality based on the combination of KPNC patient demographic databases and publicly available files of deceased patients.

Half of the patients (regardless of their number of hospitalizations) were randomized into the development dataset and the rest into the validation dataset. After randomization, all hospitalizations for a given patient were included in analyses. Models were developed and tested on the development dataset and the final performance evaluation was conducted on the validation dataset. We evaluated the following independent variables, selected because they can be obtained in real time, as potential predictors: age; sex; admission venue (ED or not); admission LAPS223; illness severity at 08:00 on the day of discharge (LAPS2dc); longitudinal comorbidities (COmorbidity Point Score, version 2, or COPS223); admission and discharge care directives (classified as “full code” or “restricted resuscitation preference,” which included “partial code,” “do not resuscitate,” and “comfort care only” orders23); total index hospital length of stay (LOS); time and day of discharge; and whether a patient experienced any overnight inpatient hospitalization in the first 7 and separately in the 8–30 days preceding the index hospitalization. For descriptive purposes, we also quantified other patient characteristics, including whether or not the patient was admitted to intensive care and, for surgical patients, whether they had a second stay in the operating room (a proxy for a surgical complication). We combined Health Care Utilization Project (http://www.ahrq.gov/data/hcup) single-level diagnosis clinical classification software categories to group all possible International Classification of Diseases principal diagnosis codes into 30 groups, which we refer to as Primary Conditions.23

As elaborated in Section 3 of the Appendix (Supplemental Digital Content 1, http://links.lww.com/MLR/B40) we first considered various models for the discharge day 30 outcome, including ANCOVA, saturated ANOVA with smoothing by logistic regression,32 random forests,33 conditional inference recursive partition,34 neural networks,35–37 recursive-partition-then-logistic regression, and a type of nearest-neighbor analysis.38 All methods were compared on their performance in a random set of patients held out from the derivation data; the winning model’s performance was checked again on the validation data. The best model was selected based on a high c-statistic with a penalty for the number of covariates and the model complexity. The discharge day 30 model ultimately selected was a single logistic regression of longitudinal comorbidities (COmorbidity Point Score, version 2, or COPS223), admission illness severity (LAPS2), total index hospital LOS (truncated at 30 d), discharge care directives (classified as “full code” or not), and whether the patient experienced any hospitalizations in the 7 and 30 days preceding this index hospitalization. The discharge day 7 model mirrors the discharge day 30 model.

The ED 7 and ED 30 models use a more restricted set of covariates that would be available in the ED (age, sex, COPS2, and LAPS2). These were first processed using a recursive partitioning algorithm that yielded 6 terminal nodes. Recursive partitioning algorithms create a decision tree that classifies observations in the population by recursively splitting the population into subpopulations (tree branches, or nodes) based on a set of predictors. The process is called recursive partitioning because each tree branch splits the population recursively until a stopping criterion is met. Predicted values are calculated as a summary statistic or by a predictive model based on the subpopulation remaining in the last partitions (tree leaves, or terminal nodes). Using a separate logistic regression per terminal node and only those observations that reside in each respective node, the same restricted set of covariates were then used to fit on the composite outcome.

For the final 4 models selected we calculated the relative contribution of predictors as described by Render et al.39 All analyses were performed in SAS version 9, SAS JMP version 7, and R version 3. As a sensitivity analysis to address the possible impact of multiple patient admissions, we remodeled using a development dataset where each patient contributed only 1 randomly selected index hospitalization and validated the model in the same validation dataset described above (ie, representing the actual operational environment where the model will be used).

During the course of the ongoing pilot, we have found that accurate patient diagnoses cannot be retrieved reliably for all patients in real time, so we did not include these in our models; however, we did assess their impact on model discrimination and explanatory power as recommended by Cook,40 Pencina et al,41 and Pepe et al.42 We also assessed the performance of our models for specific primary conditions and within the following patient subsets: males, females, age below 65/age 65 years and above, surgical/nonsurgical patients, and whether or not a patient required intensive care. For comparison purposes, we also assigned every hospitalization episode 2 scores currently in operational use in some Kaiser Permanente regions: a Charlson score43 and the LACE (Length of stay, Acuity, Charlson, Emergency department use) score of van Walraven et al.13

RESULTS

We identified eligible hospital stays within or outside KPNC involving 360,036 patients during the study period; these were concatenated into 609,393 linked index hospitalizations (episodes) for our analyses. Table 1 shows hospitalization characteristics Section 4 of the (Appendix, Supplemental Digital Content 1, http://links.lww.com/MLR/B40 provides a patient level description of the cohort). The table shows that the total rehospitalization rate was 5.8% at 7 days and 14.5% at 30 days postdischarge, with the corresponding nonelective rehospitalization rate being 5.1% and 12.4%, respectively. Mortality was 1.3% at 7 days and 3.7% at 30 days, whereas the composite outcome rate was 6.3% and 14.9%, respectively. The distribution of inclusion criteria (eg, presence of an ambulatory care sensitive condition) for the rehospitalizations is provided in Section 2 of the Appendix (Supplemental Digital Content 1, http://links.lww.com/MLR/B40).

TABLE 1

Cohort Characteristics*

Table 2 shows cohort characteristics stratified by the composite outcome. The following predictors were strongly associated with the study outcome: male sex; age; admission or discharge illness severity; comorbidity burden; and LOS; restricted care directives; ED admission; admission to the ICU; unplanned transfer to the ICU; and a proxy for the occurrence of a surgical complication.

TABLE 2

Selected Characteristics of Index Hospitalizations With and Without the 30-Day Composite Outcome*†

Table 3 summarizes the performance of the ED 30 and discharge day 30 models and compares them to the LACE. Similar results were obtained for the 7-day models (Section 6 Appendix, Supplemental Digital Content 1, http://links.lww.com/MLR/B40). Adding diagnosis did not result in a significant improvement in model performance (Section 10 Appendix, Supplemental Digital Content 1, http://links.lww.com/MLR/B40). For the discharge day 30 model, the c-statistic across subgroups ranged from 0.669 to 0.822. Section 7 of the Appendix (Supplemental Digital Content 1, http://links.lww.com/MLR/B40) also shows graphically that the discharge day 30 model had the best calibration (eg, the LACE as well as our ED models did not assign any patients a risk above 80%). For the discharge day 30 model, the relative contribution of predictors was as follows: COPS2, 37%; admission LAPS2, 25%; LOS, 9%; discharge care directive, 13%; and prior hospitalization category, 16%. Performance of the models was not substantially affected by multiple admissions; for example, when we restricted the 30-day discharge model to a dataset in which each patient only contributed 1 index hospitalization, the c-statistic was 0.756 but the Nagelkerke R2 dropped to ∼0.14, suggesting that using all patient admissions is beneficial.

TABLE 3

30-Day Models’ Performance Characteristics in Validation Dataset

Table 4 summarizes the ED and discharge day 30 models’ characteristics. From a clinician’s perspective, the most important metric is the workup to detection ratio (number of patients one would need to evaluate to identify a patient who would have experienced the composite outcome). Table 4 and Section 9 of the Appendix (Supplemental Digital Content 1, http://links.lww.com/MLR/B40) show that the discharge day 30 model was the most efficient. Figure 1 shows a Kaplan-Meier plot for the combined outcome after division of the study cohort into risk terciles based on the discharge day 30 model in the validation dataset. A strong correlation between rehospitalization and mortality rates is present once the composite outcome risk is ≥10%. The mortality rate was 0.32% among patients with a predicted risk for the composite outcome of <10%, where most outcomes were rehospitalizations where the patient survived. In contrast, among patients with a predicted composite risk of ≥50%, the death rate was 24% (81% of which were at home or in hospice), the rehospitalization rate with survival was 27%, and the rehospitalization rate with subsequent 30-day mortality was 7% (Section 13 Appendix, Supplemental Digital Content 1, http://links.lww.com/MLR/B40).

TABLE 4

Operational Characteristics of 30-Day Models (Validation Dataset)

FIGURE 1

Kaplan-meier survival curves for composite outcome across high, medium, and low risk groups using the validation dataset. Vertical axis shows the percentage of patients without the composite outcome at a given point after discharge from the index hospitalization. Horizontal axis shows the number of days after discharge. Cohort was divided into terciles based on predicted risk for the combined outcome described in the text. The model used the 30-day electronic medical record-based discharge model (discharge day 30). Dotted line (•••) shows the lowest risk tercile (predicted risk up to 6.2%; 11.1% of all combined outcomes, 1.5% of all deaths); dashed line (- - -), hospitalizations with predicted risk of 6.2%–9.6% (24.3% of all combined outcomes, 7.1% of all deaths); and the solid line (—), hospitalizations with predicted risk of 9.6% or more (64.6% of all combined outcomes, 91.4% of all deaths). Boundaries around each line indicate 95% equal-precision confidence bands. Similar curves for the other 3 models are provided in section 13 of the Appendix (Supplemental Digital Content 1, http://links.lww.com/MLR/B40).

DISCUSSION

We have developed a set of predictive models explicitly designed for use with workflows involving comprehensive EMRs. In an integrated system where patient care is structured around the EMR, careful consideration of different caregivers’ workflows and needs is essential. For example, hospital physicians specifically requested a 7-day model because KPNC quality assurance processes place responsibility for prevention of early rehospitalizations on hospitalists and hospital-based discharge planners; further, some KPNC hospitals are experimenting with hospitalists making house calls during that first week. In contrast, outpatient clinicians and case managers, some operating within specific chronic condition management programs, are responsible for later rehospitalizations (7–30 d after discharge). Thus, the basic workflow envisioned for using our models is one where preliminary 7 and 30-day risk estimates are generated in the ED. These preliminary risk estimates would be available to hospital caregivers and can also be transmitted electronically to outpatient case managers, giving them provisional notification of patients likely to require services. Subsequently, an updated probability estimate can be generated on the day of discharge.

Our results highlight the role data from current EMRs can and cannot play in predicting post-discharge trajectories. On the one hand, they permit enhancing hospital-based predictive models by incorporating physiologic and care directive data (54% of our discharge day 30 model’s predictive ability was due to these predictors). They also can improve the screening efficiency across different levels of risk. For example, the administrative data-based LACE score currently in use in some Kaiser Permanente regions does not assign any observations a risk above 80%, whereas the number needed to evaluate—which determines the cost of screening—is generally lower for our models. In contrast, their overall explanatory power remains low, highlighting the need to incorporate other predictors (eg, socioeconomic status from geocoding,44 variables obtained using natural language processing or direct patient interviews, such as social support or functional status). Importantly, we found that diagnosis did not improve model performance, and our 30-day model does not include age and sex. With hindsight, this is not that surprising, given that the predictive value of many diagnoses is probably mediated through physiological derangement, which is captured by the LAPS2. In addition, our findings show that the distinction between tools used for retrospective analyses (eg, LAPS2, COPS2) and tools used for real-time risk prediction is becoming blurred. This mixed picture of both promise and limitations highlights the fact that, as a profession, we are in a very early stage of development of EMR-based predictive models.

Consideration of replication of our models in other settings also presents a mix of promise and limitations. All of the predictors we used are generic and can be found in any comprehensive EMR, and our algorithms are available in the public domain. The first version of the LAPS (that only included laboratory data) has been externally validated,22 and our team is working with research partners at several universities to validate the second version (LAPS2). Although not all entities may have longitudinal data, it would still be possible for them to calculate a COPS2 using present on admission codes without significant degradation of statistical performance,23 as the primary advantages of the COPS2 are related to its use in real time. However, the fact remains that many health care entities (including many of those that have comprehensive EMRs) cannot replicate our methods. Most likely, this is due to staffing limitations (not having trained personnel capable of extracting data from the EMR “back end”), but other reasons may also exist, including the absence of incentives.45

A number of additional limitations need to be stressed. Our work is based on a care delivery system whose level of integration is extremely high. Further, KPNC has an insured population that enjoys a broad array of disease-specific preventive programs as well as a variety of generic preventive care systems (eg, call centers with immediate access to physicians, electronic portals for communicating with primary care providers). Thus, our models have been developed in a population where the baseline adverse outcomes rates are already low. For our models to be used elsewhere, recalibration is indicated, as baseline rehospitalization and mortality rates may be different. Another important limitation is that our models do not provide guidance on what may trigger a given rehospitalization, nor do they actually provide indication of what should be done to prevent adverse postdischarge events. This limitation can be addressed by future research that incorporates data from free text notes, other laboratory tests, and admission and discharge diagnoses. Our models also have the limitation (due to an explicit design consideration) that they do not provide separate risk estimates for different types of outcomes (eg, rehospitalization where the patient survived the rehospitalization, rehospitalization where the patient died during the rehospitalization, death at home, or death in other settings). However, both of these limitations can be addressed by conducting additional research and by adjusting operational workflows. With respect to the research component, new predictive models can be developed that adjust baseline risk estimates generated by the EMR by incorporating patient-reported predictors such as social support and/or functional status. These predictors need not necessarily be available in the EMR. One can use a mixed approach currently in use in KPNC for newborns suspected of sepsis. In this approach, a first model generates a baseline risk based on maternal risk factors available in the EMR46 and then clinicians can manually incorporate new information (the newborn’s clinical examination). Likelihood ratios developed from a second model47 can then be used by an online calculator to generate a new, posterior probability.48 A similar strategy could be used for hospitalized adults—baseline risk from an EMR model is modified by additional, nonclinical patient information.

Our primary reasons for using a composite outcome were operational. They included factors such as the need to minimize alert fatigue due to proliferation of predictive models as well as KPNC organizational policy. At KPNC, discharge and advance care planning are being linked as part of a comprehensive approach (Respecting Choices)49 to ensure that patient preferences around end of life care are respected. Given the strong correlation between rehospitalization and death described above and the fact that end of life care issues are common among rehospitalized patients,50 this approach makes sense for KPNC. Other health systems with different policies and operational constraints may prefer to develop separate models.

In summary, we have developed a set of predictive models for nonelective rehospitalization and postdischarge mortality that use electronic algorithms already in use in an ongoing pilot. These models are efficient and can serve as the platform for future research (eg, as risk adjustment tools for comparative effectiveness studies). In addition to being used to support discharge planning, these models will be used for risk adjustment in internal benchmarking systems, to support randomized interventions, and to enhance existing quality assurance and case management efforts. Our team is hopeful that rehospitalization prevention pilots using these models can begin within the next 6–9 months.