Stratification of risk for hospital admissions for injury related to fall: cohort study.

OBJECTIVE: To determine whether the ability to stratify an individual patient's hazard for falling could facilitate development of focused interventions aimed at reducing these adverse outcomes.
DESIGN: Clinical and sociodemographic data from electronic health records were utilized to derive multiple logistic regression models of hospital readmissions for injuries related to falls. Drugs used at admission were summarized based on reported adverse effect frequencies in published drug labeling.
SETTING: Two large academic medical centers in New England, United States. PARTICIPANTS: The model was developed with 25,924 individuals age ≥ 40 with an initial hospital discharge. The resulting model was then tested in an independent set of 13,032 inpatients drawn from the same hospital and 36,588 individuals discharged from a second large hospital during the same period. MAIN OUTCOME MEASURE: Hospital readmissions for injury related to falls.
RESULTS: Among 25,924 discharged individuals, 680 (2.6%) were evaluated in the emergency department or admitted to hospital for a fall within 30 days of discharge, 1635 (6.3%) within 180 days of discharge, 2360 (9.1%) within one year, and 3465 (13.4%) within two years. Older age, female sex, white or African-American race, public insurance, greater number of drugs taken on discharge, and score for burden of adverse effects were each independently associated with hazard for fall. For drug burden, presence of a drug with a frequency of adverse effects related to fall of 10% was associated with 3.5% increase in odds of falling over the next two years (odds ratio 1.04, 95% confidence interval 1.02 to 1.05). In an independent testing set, the area under the receiver operating characteristics curve was 0.65 for a fall within two years based on cross sectional data and 0.72 with the addition of prior utilization data including age adjusted Charlson comorbidity index. Portability was promising, with area under the curve of 0.71 for the longitudinal model in a second hospital system.
CONCLUSIONS: It is potentially useful to stratify risk of falls based on clinical features available as artifacts of routine clinical care. A web based tool can be used to calculate and visualize risk associated with drug treatment to facilitate further investigation and application. © Castro et al 2014.

Introduction

An important contributor to preventable healthcare costs is adverse effects of drug treatment,1 particularly among older patients exposed to multiple drugs with overlapping adverse effect profiles. Among these, the consequences of falls can be substantial.2 3 They represent the leading cause of death caused by injury among elderly patients,4 a major contributor to placement in long term care facilities,5 and in 2010 accounted for $30 000m in direct costs in the United States.6 Notably, both the risk of falls and use of specific drugs among older individuals represent targets for stage 2 meaningful use standards from the Centers for Medicare and Medicaid Services, and a range of interventions to reduce risk such as strength and balance training have been studied.7 8 9 A means of identifying individuals at particularly high risk for adverse outcomes solely based on readily identifiable clinical risk factors could allow for targeted interventions aimed at reducing risk, particularly those that might otherwise be too costly to apply in unselected populations of patients.5

To date, most such prediction rules require assessment of patients by a skilled clinician3 10 11 12 or use of a specialized device13 and focus on relatively small or specific subgroups.3 Many prior studies have investigated overall burden of anticholinergic adverse effects,14 15 16 for example, but these models do not account for other cognitive effects, such as excessive sedation, which might not be associated with anticholinergic mechanisms.

We developed and validated a novel risk stratification tool based on sociodemographic and clinical features readily available in the electronic health records as artifacts of routine care. We utilized data from a large New England hospital system that acts as a tertiary referral center as well as a large community hospital and then validated the resulting models in an independent group of patients from the same system, as well as in a second large New England hospital system.

Methods

Cohort derivation and overall design

Our primary analysis examined the association between sociodemographic and clinical features and hospital admission or emergency department visits for injury related to a fall, defined as in prior reports as ICD-9 (international classification of diseases, ninth revision) codes 800-847 (injuries including fracture, dislocation, strain and sprain), 850-854 (intracranial injury), 920-924 (contusion), or E-codes (external causes of injury) 880, 881, 884, 885, and 888 (accidental falls, excluding falls out of building, into hole, or as a result of pushing by another).

The cohort at risk was defined as all individuals aged ≥40 admitted to a large New England hospital system (“hospital 1”) in 2007 or 2008 for any indication other than fall and for whom drug reconciliation was available from discharge. For individuals with multiple admissions in this period, we used only the initial admission in the primary analysis; sensitivity analyses incorporating all admissions and clustering by individual did not yield meaningfully different results and are not presented here. This dataset was divided randomly into a training set comprising two thirds of individuals (“hospital 1-model building set”) and a testing set of the remaining third (“hospital 1-testing set”). To examine portability, a validation dataset utilized an identically derived cohort drawn from admissions to a second large New England hospital system (“hospital 2”) in 2007-08. While rare, it was possible for individuals to be admitted to both hospitals; individuals were assigned to the site of their index admission during the study period. (In sensitivity analysis that excluded these individuals, results were unchanged.)

We developed two sets of models using multiple logistic regression. The first utilized only data available for a single hospital admission cross sectionally—that is, it did not assume the availability of any prior data regarding that individual, such as previous outpatient visits or admissions. The second extended that model by incorporating summary measures of prior visits or admissions to estimate the improvement in prediction afforded by such longitudinal data.

Derivation of clinical variables

We utilized i2b2 server software (i2b2 v1.5, Boston, MA)17 (see box) to access and manipulate data from the electronic health records of two large hospitals based in Boston, each with their own network of primary and specialty care settings. The i2b2 system18 19 is a scalable computational framework, deployed at over 100 academic health centers, for managing human health data. The electronic health records from each hospital system include sociodemographic data, billing codes, laboratory results, problem lists, drugs, vital signs, procedure reports, and narrative notes.

What is i2b2?

Informatics for Integrating Biology and the Bedside, or i2b2, is an NIH-funded National Center for Biomedical Computing that has developed tools for managing human health data. The software (www.i2b2.org/software/index.html) allows data to be extracted and integrated from electronic health records and similar datasets. A typical electronic health record can comprise numerous individual databases (for example, containing billing information, laboratory results, narrative notes, and so forth); the i2b2 workbench allows data to be aggregated and queried across these diverse data sources, and integrated into a standard database format. For the present study, i2b2 software was used to extract data from the electronic health records from the two hospitals, which could then be exported into flat files readable by standard statistical packages.

Both hospital systems utilize a formal drug reconciliation protocol to ensure that drugs are correctly documented in the electronic health record at the time of hospital discharge. Confirmation that drugs prescribed were subsequently dispensed to patients is not available for research purposes because of restrictions imposed by the pharmacy data provider.

Characterization of discharge drugs

While adverse effects of drugs are recognized as a contributor to falls, inclusion of all individual drugs in prediction models risks over-fitting and potentially reduces generalizability of a model, particularly as standard treatments change over time. We therefore incorporated three drugs related measures. The first was a simple drug count—that is, how many drug prescriptions were issued at discharge. The second was the anticholinergic risk scale score,20 which has previously been reported in small cohorts to predict risk. A limitation of this method, however, is that it captures only one mechanism by which drugs might increase falls—for example, sedation or gait instability associated with anticonvulsants would not be included. Therefore, we developed and applied another empirical method based on frequencies of adverse effects drawn from the SIDER side effect resource21 databases, which include Food and Drug Administration and international drug labeling information, as well as postmarketing surveillance data (fig 1). The fall burden score represents an estimate of how likely a patient is to experience at least one side effect that could contribute to the risk of falling, based on the reported frequency of each adverse effect in drug labeling.

Fig 1 Process of generation of model to assess risk of falls in patients discharged from hospital

First, the senior author (RP) manually curated the categories of adverse effects to identify those corresponding to the risk for fall; this list is provided in supplemental table 1 in the appendix. The fall burden score is the sum of frequencies of relevant adverse effects related to falls aggregated over all drugs. So, for example, in a list of eight drugs, if one has dizziness (frequency of 10%) and gait instability (10%) as labeled adverse effects, and another has dizziness (15%), the burden score would be 0.1 + 0.1 + 0.15 = 0.35. (A further refinement of the score considering putative drug-drug pharmacokinetic interactions via cytochrome p450 is described in supplemental methods in the appendix; modeling these interactions explicitly led to modest gains in prediction; see supplemental text in appendix).

Analysis

We used multiple logistic regression to examine admissions or visits to emergency departments for fall related injury, with data censored after 24 months or time of death, whichever came first. (We used logistic regression in lieu of survival analysis because the time to event might be less relevant here than the presence or absence of an event—that is, falls at 30 days or 700 days are both problematic from a clinical perspective. Supplemental materials in the appendix examine the impact of censoring at shorter time horizons than 24 months). Our primary analysis included only the first admission or visit to an emergency department for a non-fall diagnosis in 2007-08 for each individual who was not initially admitted for a fall (that is, incident falls). First, the regression model included only sociodemographic or clinical data available cross sectionally for a given admission—that is, age (coded with linear and quadratic terms), sex, payment type, and self identified race/ethnicity—as well as admission for a primary psychiatric diagnostic code (ICD-9 codes 290-319). We used two terms to describe drugs prescribed at discharge: total count and drug associated burden of risk of an adverse event related to a fall. Next, we fitted an extended model to the data to incorporate features of medical history and overall severity of illness readily calculated from the entire coded electronic health record data (that is, incorporating prior visit data). These included age adjusted Charlson comorbidity score as well as log transformed count of prior hospital admissions and outpatient visits. (We used Charlson score and aggregated visit counts, rather than specific diagnoses or procedures, to maximize generalizability, in light of variability in validity of specific diagnoses.22 23) The goal for the extended model was to maximize discrimination in terms of prediction of subsequent falls, as well as to understand the additional informativeness of considering longitudinal versus cross sectional data frames.

Given the relatively limited number of predictor variables we selected and the frequent exhortation by statistical consultants to not substitute automated approaches to variable selection, we did not use stepwise variable addition or elimination. There were no missing variables, so no imputation was required.

Analyses utilized Stata 13.1 (Statacorp, College Station, TX) with the somersd,24 hl,25 and nriidi26 packages.

Model validation

We examined the resulting logistic regression models in an independent testing cohort drawn from the same hospital system to characterize model discrimination and calibration. Then, to understand portability and generalizability, we examined the same models in a second hospital system. This system has distinct clinical facilities and protocols and generally serves a distinct population, although it is located in the same city as the first hospital and health records are accessible at both hospitals through a similar interface.

As there is no standard clinical risk stratification model for falls to compare with these models, we used the area under receiver operating characteristics curve (AUC), a summary measure of discrimination, as a primary performance metric. We have also reported other metrics of improvement in fit, including the likelihood ratio test (for nested regression models) and net reclassification improvement.27 We determined calibration—the extent to which predicted risk matches observed risk—using Hosmer-Lemeshow goodness of fit.28

Development of risk score calculation and visualization tool

As the only modifiable risk factor included here relates to drug treatment, and to facilitate further study or application of drug risk scores, we developed a web based tool to calculate and visualize risk scores. Of note, while the i2b2 platform was utilized for model development, it is not required to implement the calculator itself. The web tool allows manual entry of a drug list with autocompletion and applies the algorithms described above to calculate and display the burden associated with each drug and with the regimen as a whole. Comparison with reference clinical populations is also provided, based on the data presented here. The tool is accessible at http://clearer.mghcedd.org.

Results

We randomly divided 38 956 individuals discharged from hospital 1 into a training (n=25 924) and testing (n=13 032) cohort. Table 1 summarizes the sociodemographic and clinical features of individuals from hospital 1 and a second hospital cohort (hospital 2) subsequently used for validation. Among the training cohort, 680 (2.6%) were evaluated in the emergency department or admitted for a fall within 30 days of discharge, 1635 (6.3%) within 180 days of discharge, 2360 (9.1%) within one year, and 3465 (13.4%) within two years. In a logistic regression model including age, sex, race, insurance type, admission type (via emergency department or not), psychiatric versus non-psychiatric admission, and number of drugs (table 2), the area under the curve was 0.64 (95% confidence interval 0.63 to 0.65). Addition of the drug adverse effect score yielded modest but significant improvement in the area under the curve (0.64, 0.63 to 0.65; P<0.001 for comparison with simpler model; net reclassification improvement 10.9%; P<0.001). The drug adverse effect burden, adjusted for all other terms in the model, yielded odds ratio of 1.41 (95% confidence interval 1.26 to 1.58). Addition of the anticholinergic risk score did not significantly improve the area under the curve (0.643, 0.63 to 0.65; P=0.2). Alternative measures of model fit yielded similar results (supplemental table 2 in appendix).

Table 1

 Sociodemographic and clinical characteristics of two hospital cohorts* used to develop and test model for prediction of risk of falls. Figures are numbers (percentage) of patients unless stated otherwise

	Hospital 1 (n=38 956)	Hospital 2 (n=36 585)
Men	20 243 (52)	16 739 (46)
White	33 114 (85)	28 558 (78)
African-American	1584 (4)	3164 (9)
Private insurance	15 387 (40)	17 134 (47)
Other insurance/uninsured	2150 (6)	2120 (6)
Index admission:
Via emergency room	14 717 (38)	11 074 (30)
Primary psychiatric	2411 (6)	1915 (5)
Mean (SD):
Age at index admission (years)	64.3 (13.3)	62.6 (13.1)
Age adjusted Charlson index	6.4 (4.6)	5.6 (4.1)
Total No of drugs	8.3 (4.8)	8.6 (4.9)
Anticholinergic risk scale score	0.5 (1.2)	0.5 (1.2)
Adverse effect burden score	0.6 (0.5)	0.6 (0.5)
Prior admissions	1.6 (3.3)	1.6 (3.5)
Prior outpatient visits	46.3 (76.5)	32.6 (60.7)

*P<0.001 for all univariate comparisons except “other insurance” (P=0.1) and “anticholinergic risk scale score” (P=0.2).

Table 2

 Logistic regression for prediction of fall risk within two years of index admission to hospital

Variable	Odds ratio (95% CI)
	Cross sectional model	Longitudinal model
Fall burden score (frequency × 10)	1.03 (1.02 to 1.05)	1.03 (1.02 to 1.04)
Anticholinergic risk scale score	1.02 (0.99 to 1.06)	1.02 (0.99 to 1.05)
White	1.37 (1.20 to 1.56)	1.24 (1.09 to 1.42)
African-American	1.46 (1.19 to 1.80)	1.27 (1.02 to 1.57)
Male	0.82 (0.76 to 0.88)	0.87 (0.80 to 0.94)
Private insurance	0.69 (0.63 to 0.76)	0.81 (0.73 to 0.89)
No insurance	0.80 (0.67 to 0.94)	1.26 (1.06 to 1.51)
Age at admission (years)	1.01 (1.01 to 1.01)	1.01 (1.01 to 1.01)
Total No of drugs	1.03 (1.02 to 1.04)	1.01 (1.00 to 1.02)
Primary psychiatric diagnosis at admission	1.22 (1.05 to 1.40)	1.30 (1.12 to 1.51)
Admission via emergency department	1.83 (1.70 to 1.97)	1.50 (1.39 to 1.63)
Age adjusted Charlson index	—	0.99 (0.98 to 1.01)
Prior admissions (log10(count+1))	—	1.27 (1.19 to 1.36)
Prior outpatient visits (log10(count+1))	—	1.32 (1.28 to 1.36)

Next, we developed an extended model incorporating longitudinal measures of severity of illness and utilization (table 2). This allowed us to examine the extent to which drug burden represented solely a proxy for severity, as well as potentially improving discrimination. In these models (table 2), the area under the curve was significantly improved: 0.71, 95% confidence interval 0.70 to 0.72; P<0.001 for comparison with cross sectional model. After inclusion of prior admission and visit counts, the association between Charlson comorbidity index and outcome was no longer significant. (As insurance and race/ethnicity might be less relevant in non-US health systems, we developed additional models omitting these variables; see supplemental materials in appendix).

To reduce optimism associated with training and assessing model performance in the same dataset, we examined the performance of the two primary models (cross sectional and longitudinal) in the independent hospital 1 testing set, which had not been included in model development. Here the area under the curve was 0.65 (95% confidence interval 0.63 to 0.66) for the cross sectional model and 0.72 (0.71 to 0.73) for the longitudinal model. We also considered a range of sensitivity thresholds, as might be applied in defining high risk cohorts. Specificity was 18% at 95% sensitivity; 31% at 90% sensitivity; and 49% at 80% sensitivity.

Figure 2 illustrates the calibration for the longitudinal model in the testing set; Hosmer-Lemeshow goodness of fit χ2 (5 df) was 1.76 (P=0.9), indicating good fit. Figure 3 illustrates survival curves based on fifths of the risk distribution. The observed incidence of falls within two years ranged linearly from 4.9% in the lowest fifth to 32.8% in the highest fifth. The top fifth included 42.9% of subsequent injurious falls, the top two fifths included 67.3%.

Fig 2 Calibration of models for prediction of falls within two years in patients discharged from hospital

Fig 3 Kaplan-Meier survival curves for time to readmission for fall, by fifths, in testing cohort

Finally, to understand portability, we considered the two models in an independent testing set from hospital 2. Table 1 shows the general differences between these two hospital systems. Results in terms of discrimination were quite similar to those obtained in the previous two analyses: the area under the curve was 0.651 (95% confidence interval 0.641 to 0.661) for the cross sectional model and 0.708 (0.699 to 0.717) for the longitudinal model. Calibration was somewhat poorer (χ2 for goodness of fit was 226.75 (df=5), P<0.001). While risk increased incrementally by fifth, absolute risk was systematically overestimated, as expected given the difference in baseline risk of fall between the cohorts. Recalibration of the model in the hospital 2 dataset yielded acceptable calibration (Hosmer-Lemeshow χ2 9.69 (df=5); P=0.08).

For illustrative purposes, figure 4 shows a typical output screen from a web based calculator of adverse effect burden. The fall risk score is presented at the top, followed by the individual drug risk scores, facilitating interventions based on minimizing risk associated with drug treatment.

Fig 4 Illustration of risk visualization tool for risk of falls

Discussion

Cross sectional or longitudinal electronic hospital record data can be utilized to derive clinically relevant risk stratification for admissions to hospital for fall related injury. They illustrate a simple but novel method for incorporating aggregate measures of adverse drug effects based on an automated method for integrating reported frequencies of adverse effects from drug labeling. The inclusion of a second hospital system provides further support for the portability of this algorithm. Discrimination for the models is promising, with the area under the receiver operating characteristics curve exceeding 0.70 in both hospitals; calibration curves (figs 2 and 3 ) illustrate the differential outcomes between fifths of risk.

Targeted prevention

A range of possible interventions could be considered in the highest risk group, in which the hazard for fall is more than six times higher than in the lowest risk group (33% versus 5%), and which captures about 43% of discharged individuals who subsequently fall. Strength and balance training,7 8 9 intensive education, home visits, or ambulatory monitors,29 alone or in combination,30 could be used to minimize fall risk, interventions that might be more feasibly and cost effectively applied only in individuals at high risk. In addition, while the adverse effect burden only modestly increased prediction, it represents the only other potentially modifiable risk factor among the variables identified in this cohort. Individuals at high risk of falls at hospital admission could have their drug regimens re-examined in terms of drugs with additive adverse effects and simplified if possible before hospital discharge, if prospective studies support this effect. By allowing targeted intervention, the model we describe could increase cost effectiveness compared with interventions applied to all comers, although particularly in the case of drug simplification, such interventions require further investigation considering other potential benefits beyond reduction in risk of falls. Moreover, while we cannot exclude the possibility that adverse effect burden is simply a proxy for medical and psychiatric morbidity, we note that independent effects of this variable persist after adjustment for multiple other measures of comorbidity. Even if residual confounding remains, we suggest that these scores are still useful in predicting outcomes, providing a readily calculated metric to capture risk not otherwise included in age, Charlson index, and other measures.

A key characteristic of the present analysis is its reliance on data routinely available in electronic health records, rather than requiring additional assessments or more complex parsing. While a recent report suggests that fall detection in electronic health records is improved by use of natural language processing, clinical application of this strategy in real time might not be feasible in all settings.31 Undoubtedly prediction could be improved further with additional clinical detail, or incorporation of specific risk measures, including pre-discharge checklists; the intent of this report is to describe a simple, automated, and scalable approach requiring minimal additional clinical effort that can serve as a baseline for future efforts. One extension might be inclusion of specific individual diagnoses, although the differential utilization of these codes across health systems might detract from generalizability.

Falls in patients are a major preventable contributor to healthcare costs, and one with an incidence that seems to be increasing based on emergency department data.32 Concern for this risk, as well as the broader consequences of adverse drug effects among older patients, led the Centers for Medicare and Medicaid Services to prioritize them among meaningful use strategies. To date, however, there has been no effort to automate the process of risk stratification based on drug profiles and more generally using data from electronic health records. While we propose one simple means of aggregating risk across medicines based on labeling information, we recognize that numerous other approaches to generating such scores could be considered, given that drug labels often reflect primarily short term use in a specific diagnosis.

Strengths and limitations

A notable limitation of this analysis is that neither of the two hospitals examined here is an entirely closed healthcare system, so data are not available for individuals who experience an event of interest and are admitted to hospital elsewhere. The presence of missed events will likely diminish accuracy of classification (for instance, because individuals who actually experienced an event but were admitted elsewhere are mislabeled), although we cannot exclude the possibility that some apparent predictors of lower risk actually predict failure to follow-up within these health systems. Our models also do not deal with adherence, dosing, or other clinical parameters that could modify the pharmacokinetic properties of a treatment. Whether incorporation of these data can improve prediction accuracy merits further investigation. We note, however, that a prior report found that calculated anticholinergic risk was more useful in prediction than direct measurement of serum anticholinergic activity,33 suggesting that attempting to capture blood concentrations might not be necessary. Moreover, we found that incorporating prediction of CYP450 interactions (supplemental text in appendix) did not yield improvement in discrimination for the longitudinal model. This result suggests these drug-drug interactions might not be useful on average to predict risk of falls, but their usefulness in identifying contraindicated drugs or other adverse events merits further investigation. Finally, we focused on individuals who were not initially admitted for a fall; for patients with an index admission for a fall, the risk for a subsequent fall is likely to be substantially greater and thus specific interventions are more likely to be appropriate in most cases.

Extension of findings

A key next step in understanding these results and considering implementation of these models will be further investigation in additional hospital systems, including systems outside the US. The two hospitals examined here are academic medical centers that also act as large community hospitals—that is, while they treat regional and national referrals, most patients represent local referrals, and the two hospitals have different catchment areas. As such, given the broad and diverse patient population (table 1) reflecting the region as a whole, we would anticipate generalizability to be good. A further strength of the models is that they are well calibrated, such that the estimated risk (rather than just discrimination at a single cut point34) can be incorporated in planning interventions. On the other hand, it is highly likely that further refinement will be required in other health systems to reflect differing base rates of fall and system specific risk profiles. In particular, to facilitate these efforts in countries where insurance type and race might be less relevant to outcomes, we include a model omitting these factors (supplemental materials in appendix).

A previous study suggested the utility of the anticholinergic risk score in predicting mortality in 71 patients with hip fracture.16 Our work extends this effort by introducing a new burden score, integrating it with other sociodemographic and clinical measures, using them to predict specific outcomes, and validating a prediction tool in more than 75 000 patients admitted across two hospital systems. Despite the enthusiasm for application of large datasets to improve healthcare outcomes, there are few examples of this approach being reduced to practice. This investigation suggests one simple strategy by which large health datasets can be integrated with other data mining strategies21 33 to facilitate risk stratification in a clinical setting. These models could provide a useful starting point for future efforts to more precisely estimate risk for falls and to develop and target appropriate interventions.

Risk of fall represents a major preventable contributor to healthcare costs and a target of meaningful use standards from the US Centers for Medicare and Medicaid Services

Older individuals at high risk for falling after hospital discharge can be identified by using routine data available from electronic health records.