Predicting Adverse Outcomes in Heart Failure Patients Using Different Frailty Status Measures.

Frailty is an important outcome predictor in older patients. We randomly sampled 12,000 veterans with heart failure diagnosed in 2010. The topic modeling method was applied to identify frailty-related topics from the clinical notes in the electronic medical records. The frailty topics were classified into five deficit areas including physical functioning (PF), role-physical (RP), general health (GH), social functioning (SF), and mental health (MH). We experimented with different covariates and four different frailty measures: individual frailty topics, number of distinct frailty topics, a dichotomous deficit category, and the number of distinct deficits, respectively. A total of 8,531 (71.1%) patients had at least one frailty topic. The prevalence of GH, PF, MH, SF, and RP deficits were 89.0%, 61.3%, 56.9%, 40.6%, and 9.5%, respectively. PF deficits (yes/no) and the number of distinct deficits were the most consistent, significant predictors of adverse outcomes of rehosptalization or death.

Introduction

Frailty commonly occurs in older adults and is an important determinant of health outcomes [3; 14; 16; 26]. Frailty is distinct from comorbidity and disease, and is a multifaceted combination of fatigue, weakness, malnutrition, and mobility [5, 15]. Frailty not only affects patients and caregivers, but is also a leading indicator of worsening health outcomes, including death. It is a barometer of how well patients may respond to treatment. However, frailty measurements are rarely collected in a quantitative, reliable fashion in routine patient care.

Frailty is specifically an important measure to assess cardiac peri-operative risk, morbidity, and mortality. Older patients with heart failure (HF) comprise a growing proportion of the cardiac surgery population. The Society for Thoracic (STS) risk model is often used to estimate risk, but includes no variables directly related to patient frailty [1]. Recently, smaller studies have demonstrated the added predictive value of frailty measures, for example the 5-minute gait speed [2] and a comprehensive frailty assessment [20]. Still, frailty is a key dimension that continues to be absent from quantitative risk prediction, in part because it is challenging to capture this metric on a large scale.

Fortunately, clinicians commonly document various aspects of frailty in clinical notes, especially when treatment plans change and/or when a patient’s quality of life is impacted, suggesting that perceptions of frailty is a component of clinician’s mental model of the patient’s status. The Department of Veteran Affairs (VA) national electronic health record (EHR) database is a particularly rich data source with an extremely large, older patient population with a comprehensive collection of different types of clinical notes.

In this study, we expanded on prior work examining the association between the number of frailty topics and adverse outcomes. We used existing frailty assessment instruments for ontological guidance, and created four different measures of frailty status to evaluate their role in predictive modeling.

Prior Work

In our prior work [19], we used a case control study design to estimate the association of the number of frailty with poor outcomes among HF patients. The outcome of interest was ≥2 hospitalizations following index HF hospitalization or death. A total of 709,389 notes were included from 12,000 patients. For training, we randomly selected 50,000 notes from 4,000 patients with the outcome, and 50,000 notes from 8,000 patients without the outcome. We ran the latent Dirichlet allocation (LDA) program from a java software package called MAchine Learning and LanguagE Toolkit (MALLET), on the 100,000 notes, with the initial number of topics set to be 700.

To identify stable topics that were consistently present in different LDA runs, we first independently applied each of the 3 learned LDA models to the 709,389 notes using the topic inference tool included in MALLET. This step yielded 3 topic proportions per note for each note. Next, we defined a stable topic to be present in a note if at least 2 of the 3 topics in the topic triple corresponding to the stable topic had a proportion of 0.02 or higher in that note. This step produced 556 stable topics. An informatician and a physician independently reviewed all the stable topics and identified 53 topics that were related to frailty. The inter-rater agreement between reviewers measured by Kappa was found to be 0.818. We used the labels assigned by the informatics expert as human interpretations.

We discovered that increased number of frailty topics was statistically associated with increased risk of poor outcomes. Each additional frailty topic was associated with a 7% higher risk of adverse outcomes. Compared to patients with <4 frailty topics, those with >=4 frailty topics had two times greater risk of developing an adverse outcome within 1 year following the initial HF diagnosis.

While the results were promising, further studies are needed because the area under ROC curve (AUC) was suboptimal in the prior work, at just above 0.6 regardless of the way we parameterized the frailty variable (coding it as a continuous or binary variable). This is partly because only a small number of covariates were used. However, when we included a larger number of covariates, the statistical significance of the number of frailty topics was diminished. Thus, more robust frailty measurements are needed to predict adverse outcomes.

Methods

Data Source

In this study, we used the Veterans Administration Informatics and Computing Infrastructure (VINCI) as the data source. VINCI contains comprehensive patient health and medical information from the US nationwide veterans’ EMR, which include both structured (i.e., race, gender, diagnosis code) and unstructured data (data in text documents, i.e., clinical notes).

Study Population

Patients in our study population were the same as those identified in the prior work [19]. They were 12,000 randomly sampled veterans with one International Classification of Disease 9th Clinical Modification (ICD-9-CM) HF diagnosis of 428.0–428.9 in 2010. They were composed of 4,000 veterans who experienced death or >=2 HF-caused hospitalizations during the year after the first HF diagnosis and 8,000 veterans who did not experience death and had at most 1 HF hospitalization during the year after first HF diagnosis.

Outcomes and Predictors

The main outcome of this study was defined as >=2 all-cause hospitalizations or death within 1 year after the first diagnosis of HF.

The predictors and covariates were identified from both structured and unstructured data. For the structured data, we used patient birthdate, gender, and ICD-9-CM diagnoses at each visit. Age was calculated as baseline age at the first diagnosis of HF. Patients’ comorbidities represented by ICD codes during the year before the first HF diagnosis were captured. The Charlson Comorbidity Index (CCI) was calculated based on these ICD codes via the methods described by Quan et al. [18].

For the unstructured data, we extracted topics from the Text Information Utility (TIU) notes. All the TIU notes dated within one year prior to the first HF diagnosis were extracted. Frailty indicators, which were not available in the structured data, were extracted from the TIU notes using the topic modeling technique.

Frailty Measurement Development

As described in the introduction section, we identified 53 frailty topics. Using these topics, we created four types of frailty measurements using the ontology knowledge from SF-36 and Frailty Index [23]. SF-36 and Frailty Index are commonly used among a number of frailty assessment instruments. We grouped the frailty topics into five deficit domains (called “deficits”), including physical functioning (PF), role-physical (RP), general health (GH), social functioning (SF), and mental health (MH). Topics relating to deficits in physical activities were grouped as PF; topics relating to deficits in role activities were grouped as RP; topics relating to general health perception or vitality (energy and fatigue) were grouped as GH; topics relating to deficits in social activity were grouped as SF; and topics relating to mental health were grouped as MH. The purpose of grouping is to see if some deficit domains are better than others to predict adverse outcome and if grouped deficits are more predictive than individual frailty topics. These comparisons are critical for developping frailty ontology in our future study. Since we are investigating how to measure frailty, each frailty topic and frailty deficit domain were first treated as individual variables. We then calculated an aggregate of the total number of distinct frailty topics and the total number of distinct deficits as additional variables.

Logistic Regression Models

We used logistic regression models to analyze the association of frailty and the outcome at the individual level. We assessed each of the four frailty measures (individual frailty topics, number of distinct frailty topics, deficit category, and number of distinct deficits, respectively) as predictors for the outcome. We also experimented with 4 different sets of covariates: set #1 including age, gender, and CCI; set #2 including age, gender, and individual comorbid conditions identified by ICD-9 diagnoses; set #3 including age, gender, individual comorbid conditions, and individual topics not related to frailty; and set #4 including all structured data (age, gender, CCI, ICD-9 diagnosis, Current Procedure Terminology [CPT] procedure codes, medications, and medical note type) and non-frailty topics. For each set of covariates, we built four logistic regression models seperately with including one of four frailty measures in each model. Given the large number of comorbid conditions and topics, we used automatic stepwise selection methods to set entry p-value at 0.2 and p-value at 0.05 to choose predictors other than the four frailty measures. A multi-linearity test was also conducted for each model to make sure there was no variable with variance inflation factor (VIF) higher than 10.

Results

Demographics

Among the studied cohort, 97.9% were males. The mean age was 69.7 (SD 12.1) years old, with 48.9% veterans aged 60–69, 26.1% aged 70–79, 21.9% aged 80–89, and 3.1% aged 90 or older. Just under a third of veterans (31.4%) had CCI of 0 (based on ICDs dated within one year of the first HF diagnosis) indicating no comorbid conditions, 34.3% veterans had CCI of 1–2, and 34.3% veterans had CCI of 3 or above.

Frailty Measurements and Prediction Models

Among the total 53 frailty topics, 22 were grouped as PF deficits, 19 as GH deficits, 7 as MH deficits, 4 as SF deficits, and one as a RF deficit. The grouping of frailty topics is described in Table 1. A total of 8,531 (71.1%) patients had at least one frailty topic in their medical notes, among which 89.0%, 61.3%, 56.9%, 40.6%, and 9.5% had frailty topics in GH, PF, MH, SF, and RP, respectively.

Table 1

Deficit and Key Words for Frailty Topics Variables

Deficit	Topics	Top Five Keywords
PF	133;132;102	fall risk score morse gait
	698;229;28	bed call reach light position
	24;144;324	walker cane gait wheelchair ambulation
	488;37;4	mobility functional goals balance transfers
	520;470;564	risk fall falls patient high
	547;20;641	resident fall bed lock risk
	419;394;578	ot adl dressing functional shower
	503;383;345	point patient assistance bathing independence
	439;104;250	assistance independent level mobility bowel
	437;218;399	feces shift output bm fall
	484;169;492	shoes wear issued fit size
	144;213;206	ability position eats discomfort occasionally
	543;231;663	staff minutes tolerated support set-up
	98;577;120	patient place fall assistance bed
	576;455;230	assist care cc independent bladder
	653;177;191	assessment risk fall patient sounds
	514;589;654	assist independent shift needed care
	366;588;404	provided support assist activity performance
	571;68;644	provided activity support adl occur
	529;422;226	risk patient fall assessment skin
	325;553;336	patient fall pain assess monitor
	472;70;490	fall risk patient assessment bed
RP	74;620;477	activities leisure activity group social
GH	194;401;354	bed resting monitor noted distress
	303;135;385	home care services health va
	540;605;131	skin dry intact bed warm
	206;499;637	appointment scheduled clinic show letter
	335;303;687	family patient member significant members
	247;171;303	call message left phone called
	249;359;139	care patient needed diet rehabilitation
	413;170;359	procedure sedation patient performed consent
	591;375;661	diet eat foods meals milk
	47;536;339	long term short rehab patient
	205;88;18	care patient days moving behaviors
	466;560;35	visit hbpc home caregiver medication
	157;120;289	good poor fair appetite time
	643;411;9	comments additional date staff assistance
	344;639;167	patient current condition complain detailed
	387;148;660	low diet high fat salt
	553;380;575	problems feeling trouble days difficult
	607;400;510	staff provided patient shift mod
	169;292;285	activity staff occur resident assistance
SF	693;666;317	daughter son home called spoke
	530;507;467	writer veteran stated contact contacted
	130;509;628	wife time son house years
	181;108;203	patient caregiver action spouse criteria
MH	225;99;107	alert oriented x3 distress acute
	90;361;691	oriented alert place time status
	36;78;476	memory cognitive average speech evaluation
	54;469;320	suicide risk thoughts plan suicidal
	106;221;92	suicidal ideation homicidal hallucinations mood
	281;604;604	restraint restraints patient family behavior
	158;567;319	wife home states dementia husband

As shown in Table 2a–d, we created four sets of predictive models with each set using the same covariates and one of the four frailty measures as predictors, respectively. For each set of models, when the same covariates were used, the accuracy was very similar across the models, with AUC of 0.66, 0.80, 0.81, and 0.86 for set #1, #2, #3 and #4 of models, respectively.

Table 2a

Prediction Performance of Four Different Measures of Frailty Topics With Set #1 Coveriates*

	Variable	OR	95% CI	AUC
Frailty Topics	133;132;102	1.37	1.21–1.56	0.667
	520;470;564	1.17	1.04–1.31
	547;20;641	1.38	1.11–1.73
	503;383;345	1.46	1.24–1.71
	194;401;354	1.27	1.08–1.49
	303;135;385	1.23	1.07–1.41
	206;499;637	1.19	1.08–1.31
	387;148;660	0.81	0.73–0.91
	693;666;317	1.41	1.24–1.61
	530;507;467	1.20	1.05–1.37
	90;361;691	1.20	1.05–1.38
	106;221;92	1.23	1.07–1.42
#Distinct Frailty Topics		1.08	1.07–1.09	0.659
Deficits	PF	1.61	1.47–1.76	0.660
	RP	1.18	1.00–1.39
	GH	0.95	0.86–1.05
	SF	1.50	1.36–1.65
	MH	1.38	1.26–1.52
#Distinct Deficits		1.32	1.29–1.36	0.659

Set #1 covariates: age, gender, and CCI

Table 2d

Prediction Performance of Four Different Measures of Frailty Topics With Set #4 Covariates*

	Variable	OR	95% CI	AUC
Frailty Topic	503;383;345	1.45	1.18–1.78	0.870
	466;560;35	1.47	1.02–2.11
#Distinct Frailty Topics		1.00	0.98–1.01	0.868
Deficits	PF	1.21	1.07–1.38	0.869
	RP	0.87	0.69–1.09
	GH	1.00	0.88–1.14
	SF	1.09	0.95–1.24
	MH	1.02	0.90–1.16
#Distinct Deficits		1.06	1.01–1.11	0.868

Set #4 covariates: all structured data (age, gender, CCI, ICD-9 diagnosis, Current Procedure Terminology [CPT] procedure codes, medications, and medical note type) and non-frailty topics

Frailty Topics

Using individual frailty topics (rather than grouped into larger domains) sometimes resulted in a small (0.002 in AUC) improvement in prediction accuracy when compared with the other three frailty measures. As we included more covariates from set #1 to #4, the AUC improved from 0.66 to 0.87, regardless of what frailty measure we used. Including more covariates also caused fewer individual frailty topics to be significant (decreased from 12 in table 2a to 2 in table 2d). Most frailty topics did not consistently predict the outcome.

The topic variable “503;383;345” was the only one that was always significantly associated with increased risk of outcome. This variable was a topic related to physical functioning with top 5 key words of “point patient assistance bathing independence.” This is because self-care activity is an important risk factor for hospitalization or death, which is not captured in ICDs.

Deficits

The number of distinct deficits was consistently associated with the outcome and the association was always significant except for the model including age, gender, and ICD covariates. Compared to the number of distinct frailty topics, the number of distinct deficits was more predictive and significant. The odds ratio estimate of 1.02 to 1.32 for the number of distinct deficits were consistently higher than the odds ratio estimate of 0.99 to 1.08 for the number of distinct frailty topics in each corresponding set of models.

Among individual deficit variables, the PF was the only deficit that was consistently associated with increased risk of outcome. The GH was not predictive in any model; other deficits were significantly associated with the outcome occasionally depending on what covariates were included in the model..

When all structured data and other non-frailty topics were controlled, one additional deficit would increase risk of adverse outcomes by 6%. Although 6% seems small, the effect magnitude may be larger since the number of distinct deficits can go from 0 to 5. For example, compared to patients with 0 deficit, the risk of adverse outcomes would be increased by 12% and 34% among patients with 2 and 5 deficits, respectively.

Discussion

In this study, 71.1% veterans had at least one frailty topic in their medical notes. Since our cohort consisted of patients with HF, the prevalence of frailty was expected to be relatively higher than the general population. It also indicates the centrality of frailty in the mental models of cardiologists. Our findings are congruent with other work in frailty. A review study found that the prevalence of frailty ranged from 15–74%, depending on the study population and method of assessment [22].

This study shows the importance of using ontological knowledge to aggregate frailty findings. Individual frailty topics and the number of distinct topics are not the best predictors, when compared to the burden from aggregated deficits. Frailty is an aggregation of deficits as others have noted. When using frailty topics as individual variables, most topics became insignificant, because of the lower prevalence and because frailty cannot be determined by a single finding. When aggregated, the impact of frailty became more pronounced. For example, patients with >=3 deficits had 1.16 times risk to have adverse outcomes compared with those with <3 deficits.

Frailty is not the only predictor of outcome. Depending on the covariates included in the models, the AUC ranged from 0.659–0.667 to 0.868–0.870. More covariates resulted in higher AUC, and once individual ICDs instead of CCI were included in the model, then AUC improved substantially. According to the generally accepted rule for AUC, models adjusting for set #1 covariates were inferior to good, since the AUC was less than 0.7 but higher than 0.5; models adjusting for set #2, #3, or #4 covariates were strong, since the AUC were around 0.8 or above [10].

Many studies have reported that frailty is independently associated with a higher risk of death and other adverse outcomes among people with cardiovascular disease or the elderly in both short term and long term [6; 8; 11; 13; 21]. These studies in general reported frailty prediction models with lower accuracy than ours, regardless of frailty measure methods. These studies also used many different instruments to measure frailty including frailty index, modified frailty index, frailty-related phenotype, frailty criteria, frailty survey, and frailty related signs and symptom scales, to predict adverse outcomes [4;7;9;24;25]. The AUC of these prediction models were in the range of 0.55–0.77, which was lower than those of our models [4; 7; 9; 24; 25].

Among all the deficits, physical functioning was the only persistent predictor of the outcomes. Frailty, however, goes beyond physical functioning. The number of distinct deficits was robustly significant even when we included many other predictors besides age, gender, and ICDs. These findings suggest that the physical deficits are possibly related to a generalized functional status latent variable and perhaps an indicator of the patient’s overall burden. Our finding fills a research gap in understanding what deficit of frailty topics and if increased number of deficits would be significantly associated with the higher risk of the adverse outcomes, which remained unknown in the previous studies [12; 17].

Our study has some limitations. First, we used the most commonly used HF ICD-9-CM codes to identify the study population, which might miss patients that should be included otherwise. Second, this study explored a simple ontology of frailty by classifying the frailty topics according to patients’ physical, mental, social function and general health status. Third, we set an equal weight for different deficits, which may not be optimal, since these deficits may have variable effect magnitudes on adverse outcomes. Fourth, in this study, we focused on the overall outcome, not the outcome following specific treatments, which might be more useful.

Our final goal is to create a more detailed ontology to improve the identification of deficits and guide the aggregation of deficits into meaningful frailty levels. In the future studies, we will conduct a survey among HF patients to identify deficits from their perspective. We will evaluate patients’ frailty levels following major cardiac procedures and examine the association between frailty level and time-to-events.

Conclusions

In summary, aggregate frailty deficit measurements created based on frailty topics and ontology knowledge significantly and strongly predicted adverse outcomes among heart failure patients.

Table 2b

Prediction Performance of Four Different Measures of Frailty Topics With Set #2 Covariates*

	Variable	OR	95% CI	AUC
Frailty Topics	419;394;578	0.78	0.63–0.97	0.801
	503;383;345	1.35	1.12–1.62
	472;70;490	0.63	0.41–0.97
	303;135;385	1.21	1.03–1.41
	466;560;35	1.39	1.02–1.90
	387;148;660	0.81	0.72–0.92
	693;666;317	1.19	1.03–1.39
#Distinct Frailty Topics		0.99	0.98–1.00	0.796
Deficits	PF	1.15	1.04–1.28	0.798
	RP	0.75	0.61–0.91
	GH	0.91	0.82–1.01
	SF	1.04	0.93–1.16
	MH	1.07	0.97–1.19
#Distinct Deficits		1.02	0.99–1.05	0.798

Set #2 covariates: age, gender, and individual comorbid conditions

Table 2c

Prediction Performance of Four Different Measures of Frailty Topics With Set #3 Covariates*

	Variable	OR	95% CI	AUC
Frailty Topic	503;383;345	1.41	1.17–1.71	0.816
	466;560;35	1.42	1.02–1.96
#Distinct Frailty Topics		1.01	0.99–1.02	0.814
Deficits	PF	1.25	1.12–1.40	0.815
	RP	0.86	0.70–1.07
	GH	1.03	0.91–1.15
	SF	1.06	0.94–1.20
	MH	1.13	1.01–1.27
#Distinct Deficits		1.10	1.05–1.15	0.814

Set #3 covariates: age, gender, individual comorbid conditions, and non-frailty topics