Development and validation of multivariable mortality risk-prediction models in older people undergoing an interRAI home-care assessment (RiskOP).

BACKGROUND: Currently, one-year survival of older people with complex co-morbidities is unpredictable. Identifying older adults with a reduced life expectancy will lead to more targeted care and better healthcare resource allocation.
METHODS: Development and validation of one-year and three-month mortality risks in people aged ≥65 years who had completed an International Resident Assessment Instrument-Home Care (interRAI-HC) assessment between July 2012 and March 2018. Data was split into development (90%) and validation data sets (10%). A multivariable logistic regression model using data from 108 interRAI questions across multiple domains was developed and validated using discrimination metrics and calibration curves. Variables each explaining at least 1% of the model were then used to develop and validate a parsimonious model. Subgroups by sex, age, ethnicity, and comorbidities were evaluated.
FINDINGS: There were 104,436 persons (60.2% female; mean age 82.1 years) in the study cohort of whom 20,972 (20.1%) died within one year. The full multivariable model had area under the curves (AUCs) of 0.778 to 0.795 in the 5 validation datasets and was well calibrated. After variable reduction a parsimonious model consisted of 16 variables and was well calibrated and the AUC remained high: 0.773 (0.769 to 0.777). The three-month parsimonious model comprised 22 variables and was well calibrated with an AUC of 0.843 (95%CI: 0.839 to 0.848).
INTERPRETATION: These community-based risk prediction models accurately predict mortality in older people with complex co-morbidities. They may contribute to both forecasting for policy making and clinical decision making regarding an individual's needs. FUNDING: The New Zealand Health Research Council.

Evidence before this study

One-year survival of older people with complex comorbidities is difficult to predict. Previous studies have investigated the links between 1) individual factors and mortality and 2) variables from comprehensive ranges of health data used to create scores that predict mortality. The latter models have been optimised using improved statistical techniques to predict mortality of nursing home residents. The increased availability of large, linked health datasets has led to further attempts to predict mortality.

We searched PubMed for research articles published between January 2000 and July 2019 using the terms “mortality”, “risk-prediction models”, and “older adults”. No language restrictions were applied. We identified one new scale that predicts mortality in institutionalized older people. The Changes in Health, End-Stage Disease, Signs, and Symptoms (CHESS) scale was developed using 12 items from the minimum interRAI hospitalised patient dataset, which contains information on complex care needs that predict mortality. This recently has been optimised using improved statistical techniques for nursing home residents with two models using a more recent version of the Minimum Data Set. The increased availability of large, linked health datasets also has led to further attempts to predict mortality. For example, a recent study identified a range of risk factors associated with mortality based on UK Biobank data, which utilised medical investigations to predict mortality in older people.

Added value of this study

We report the results of the first study to validate a broad and comprehensive model to predict one-year mortality of older people with complex comorbidities using modern statistical analyses of interRAI data collected during routine home assessments. While most previous studies have predicted mortality from multiple sources of data, our study used routinely-collected data without examining patients.

Implications of all the available evidence

Our models could be used for shared decision-making of treatment options. The clinician, patient, and their family could make informed decisions about the patient's treatment. This also could help inform decision-making regarding who requires an end-of-life care plan, advanced directives or who should proceed with a more invasive elective operation or procedure. Policy makers within a jurisdiction could choose risk thresholds to inform health system needs, such as the need for palliative care and hospice services.

Alt-text: Unlabelled box

Introduction

Currently predicting up to one-year survival of older people with complex comorbidities is unreliable [1,2]. With the growing older population, identifying older adults with a reduced life expectancy is a core issue in clinical decision-making and health policy development. Accurate mortality prediction would enable clinicians, patients and families to make informed decisions about patient care, including whether to proceed with operations and procedures or to adopt a supportive end-of-life approach [3,4].

To address the challenges of the increasing ageing population, New Zealand has mandated a nationwide standardised comprehensive clinical assessment for all older people with complex needs. The assessment is known as the international Residential Assessment Instrument (interRAI) and has been developed and validated by clinicians and academics. InterRAI assessments are now performed in over 36 countries and the assessments are usually performed by trained health professionals (usually nurses or social workers). The assessment records 236 items across multiple domains of capability, medical diagnoses, physical function, and social and psychosocial wellbeing. However, apart from assessing weight, height and walking speed no physical examinations or other measurements are included in the assessment. Thus, a large linked comprehensive dataset has been generated representing a significant portion of older people in New Zealand using routinely collected health data. This data can readily be linked with national mortality data.

Multiple previous studies have looked at the link between individual factors and mortality [5,6]. More recently studies have used variables from a comprehensive range of health data to create scores that predict mortality [7,8]. The Changes in Health, End-Stage Disease, Signs, and Symptoms (CHESS) scale was developed using 12 items from the minimum interRAI hospitalised patient dataset, which contains information on complex care needs that predict mortality [9]. This model has been recently optimised using improved statistical techniques for nursing home residents with two models using a more recent version of the Minimum Data Set [10,11].

The increased availability of large linked health datasets has led to further attempts to predict mortality. For example, a recent study identified a range of risk factors associated with mortality based on UK Biobank data, which utilised medical investigations including bone density, blood pressure, and spirometry to predict mortality [5]. Additionally, another UK study, QMortality focused on predicting mortality using a large cohort of clinical data [12].

In this study, we develop and validate a broad and comprehensive model to predict one-year mortality of older people with complex comorbidities using modern statistical techniques with interRAI data measured during routine home assessments.

Methods

Design

We developed and validated a risk prediction model using data collected from older persons living at home using the interRAI home care instrument (interRAI-HC). The interRAI-HC is administered to individuals thought to need home support or entry to a long-term care facility. These assessments consist of over 236 questions across 20 domains. Data quality and completeness has been previously demonstrated [13].

Participants

Participants included all people aged ≥65 years who completed an interRAI between 1 July 2012 and 11 March 2018, and who had consented for their data to be used for research. Mortality data was available until 12 March 2019. Repeat assessments and assessments undertaken while not living at home at the time of the assessment were excluded (Fig. 1). All participants were followed for a minimum of 12 months or until death. The interRAI-HC assessment data was provided by New Zealand's Technical Advisory Services. The National Mortality Collection Register administered by the New Zealand Ministry of Health provided the dates of deaths. Linkage between the two data sets was made by encrypted national health index numbers as every person who has ever interacted with the public health system in New Zealand has a unique number.

Fig. 1

Flowchart of data acquisition and splitting.

Outcomes

The primary outcome of interest was one-year mortality and the secondary outcome was three-month mortality.

Statistical analysis

Data are presented as n (%) for categorical variables, mean and standard deviation for normally distributed quantitative variables and median (lower quartile and upper quartile) for non-normally distributed variables.

A large set of variables was chosen a priori. These included variables that are part of the CHESS and updated CHESS scores previously shown to be associated with mortality, [3,4] variables that are known or thought to be associated with mortality by geriatricians, and variables across the domains of the interRAI-HC assessment. Ethnicity was not included to increase generalizability beyond New Zealand.

Variable reduction before model fitting was by determining how well each variable was predicted by the remaining variables using a flexible parametric additive model (the redun function in the Hmisc R-package) [14]. Variables that were predicted by other variables with an R2 > 0.9 were dropped. The remaining variables were used to predict one-year mortality using a logistic regression model. Apart from age, all variables were categorical variables.

The data were randomly split into development (90%) and validation (10%) cohorts. The beta coefficients from the logistic regression analysis of each development data set were then applied in the corresponding validation data to obtain a risk prediction for each participant. Performance was then assessed in the validation data set by a measure of discrimination, the Area Under the receiver-operating characteristic Curve (AUC; which is a measure of the ability to discriminate between those at higher and lower risk) and calibration by plotting the predicted versus observed risk. Additionally, we present the Brier score which is a measure of the agreement between observed and predicted risk (between 0 and 1, the smaller the better) and Nagelkerke R2 which is a measure of the proportion of the variation in risk the model explains. This process was repeated 5 times. This enables assessment of robustness of the model and choice of variables for a parsimonious model.

Following development and validation of the full model, to aid implementation we developed a parsimonious model with fewer variables. The variables chosen each contribute ≥1% to the performance of the full model based on their χ2 and numbers of degrees of freedom (d.f.) in the full model [10]. The percentage contribution of a variable is 100 x (variable χ2 – variable d.f) / (sum of all variables χ2 – sum of all variables d.f).

Following assessment of discrimination and calibration the final parsimonious model was developed on the entire dataset and the equation to enable prediction of individual probability of death within 1 year is presented.

We further assessed calibration performance in the complete dataset in subgroups of interest according to sex, age, ethnicity, and co-morbidities of cancer, congestive heart failure (CHF), chronic obstructive pulmonary disorder (COPD), coronary heart disease (CHD), and body mass index.

We used the same procedure to develop as a secondary outcome a model for prediction of 3-month mortality. Results are presented in the supplement.

The Transparent Reporting of Multivariable Prediction Model (TRIPOD) guidelines were used to guide this report [15]. All calculations were performed in R version 3.5.2 [16]. Ethics approval was obtained from the Ministry of Health and Disability Ethics Committee (14/STH/140). Funding was from the Health Research Council.

Role of funding sources

This study was funded by the Health Research Council of New Zealand, which had no role in the study.

Results

There were 104,436 unique persons who underwent a first interRAI-HC assessment during the study period of whom 20,972 (20.1%) died within one year (Table S1; Fig. 1). There were more females (60.2%) than males (39.8%). The mean age (SD) was 82.1 (7.5) years. Forty-six percent had been admitted to hospital at least once in the preceding 90 days.

Pain variables measuring pain frequency, intensity, and consistency were all highly predicted by the other variables, so were not included in the model. Similarly, ability to transfer, self-reported anxiousness, and self-reported sadness were predicted; therefore, excluded.

Full one-year model

The five development datasets each comprised 93,992 people of whom 18,897 to 18,924 (20.1%) died within one-year. These multivariable models had good discrimination with AUCs of 0.792 to 0.795, R2 of 0.277 to 0.280 and Brier scores of 0.126 to 0.127 (Table S2). In the five validation data sets (n=10,044 of whom 2048 to 2101 [19.6% to 21.1%] died), the model retained its strong discriminatory performance with AUCs of 0.778 to 0.795 and was well calibrated (Fig. 2). The odds ratios for all variables for each data set are in supplementary Table S2.

Fig. 2

Calibration of the Full and parsimonious models using the first of the five validation data sets. Line segments are 95% confidence intervalw. The data was split into 20 intervals of equal range on the Predicted percentage scale. Perfect calibration is indicated by the dotted line.

Parsimonious one-year model

Sixteen variables explained >1% of the proportion of the variation in risk each of the model in all five datasets (Fig. 3, Table 1). One of the five models had an additional variable explaining 1% of the model (Unstable conditions). Only the 16 variables common across all five datasets were included in the parsimonious model. Being male, underweight or increasing age, a diagnosis of cancer, dyspnoea, or chronic heart failure, having weight loss, fatigue, oxygen therapy, recent hospital stay, or a decline in ADL status all demonstrated an association with increased mortality. On the other hand, being overweight or obese, spending time out of the house, exercising, and diagnosed with Parkinson's or having had a stroke were all associated with decreased mortality, Table 1. These latter two variables contributed the least to the model of the 16 variables included in the parsimonious models. The presence of cancer was the strongest predictor in each of the five models with an odds ratio of 2.48 to 2.54 (Table S4).

Fig. 3

Variable contribution to the full model using the first of the five data sets. The 16 top contributors contributed to 84% of the model. BMI: Body Mass Index; CHF: Chronic Heart Failure; ADL: Activities of Daily Living. Suffixes are the interRAI-HC question numbers.

Table 1

Odds Ratios for the variables in the parsimonious one year and three months models

Variables		One-year model	Three-month model
(Intercept)		0.014 (0.011 to 0.017)	0.006 (0.004 to 0.008)
Cancer I1t	Not Diagnosed	Reference 1	Reference 1
	Diagnosed	2.509 (2.405 to 2.617)	2.745 (2.589 to 2.911)
Dyspnea	No	Reference 1	Reference 1
	Yes moderate	1.217 (1.165 to 1.271)	1.27 (1.184 to 1.363)
	Yes normal	1.498 (1.429 to 1.571)	1.534 (1.427 to 1.648)
	Yes rest	2.022 (1.896 to 2.156)	1.895 (1.732 to 2.072)
CHF I1m	Not Diagnosed	Reference 1	Reference 1
	Diagnosed	1.643 (1.576 to 1.714)	1.493 (1.403 to 1.588)
Gender	Female	Reference 1	Reference 1
	Male	1.621 (1.567 to 1.678)	1.449 (1.369 to 1.534)
Scale BMI	Normal	Reference 1	Reference 1
	Underweight	1.401 (1.306 to 1.503)	1.332 (1.204 to 1.473)
	Overweight	0.761 (0.723 to 0.802)	0.853 (0.785 to 0.927)
	Obese	0.619 (0.581 to 0.659)	0.636 (0.572 to 0.707)
	Unknown	1.016 (0.974 to 1.06)	1.015 (0.95 to 1.084)
Age	per year	1.028 (1.025 to 1.03)	1.016 (1.012 to 1.019)
Weight loss K2a	No	Reference 1	Reference 1
	Yes	1.47 (1.409 to 1.534)	1.58 (1.488 to 1.677)
Fatigue	None	Reference 1	Reference 1
Fatigue J5Minimal	Minimal	1.087 (1.036 to 1.142)	1.161 (1.063 to 1.268)
Fatigue J5Moderate	Moderate	1.338 (1.272 to 1.408)	1.445 (1.326 to 1.576)
Fatigue J5Severe plus	Severe+	1.941 (1.831 to 2.057)	2.085 (1.903 to 2.285)
Oxygen therapy N2e	No	Reference 1	Reference 1
	Yes	1.747 (1.587 to 1.923)	1.573 (1.405 to 1.762)
Days went out G4	None	Reference 1	Reference 1
	Usually	0.785 (0.733 to 0.84)	0.889 (0.807 to 0.979)
	From 1to2d	0.657 (0.627 to 0.69)	0.681 (0.633 to 0.733)
	Three d	0.511 (0.485 to 0.538)	0.474 (0.434 to 0.518)
Other skin ulcer L3	No	Reference 1
	Yes	1.661 (1.542 to 1.79)
Hours exercise G4	None	Reference 1	Reference 1
	Less 1h	0.812 (0.779 to 0.848)	0.841 (0.792 to 0.894)
	From 1to2h	0.668 (0.637 to 0.7)	0.614 (0.569 to 0.662)
	More than 2h	0.627 (0.587 to 0.67)	0.557 (0.493 to 0.629)
Time since last hospital stay A14	None	Reference 1	Reference 1
	From 31 to 90d	1.224 (1.163 to 1.287)	1.234 (1.132 to 1.345)
	From 8 to 30d	1.347 (1.274 to 1.425)	1.553 (1.423 to 1.695)
	From 0 to 7d	1.621 (1.544 to 1.702)	1.842 (1.709 to 1.985)
Change in ADL status G6	Improved/None	Reference 1	Reference 1
	Declined	1.348 (1.297 to 1.402)	1.686 (1.576 to 1.804)
	Uncertain	1.17 (1.065 to 1.286)	1.266 (1.08 to 1.483)
Stroke I1j	Not Diagnosed	Reference 1	Reference 1
	Diagnosed	0.903 (0.863 to 0.944)	0.725 (0.675 to 0.778)
Parkinsons I1h	Not Diagnosed	Reference 1	Reference 1
	Diagnosed	0.74 (0.672 to 0.816)	0.663 (0.564 to 0.781)
Timed walk G3b	Fast		Reference 1
	Average		1.023 (0.939 to 1.114)
	Slow		1.132 (1.035 to 1.237)
	Unable/unwilling		1.866 (1.719 to 2.026)
Marital_Status	Not married		Reference 1
	Married/CivilUnion/Defacto		0.897 (0.793 to 1.015)
	Widowed		0.98 (0.865 to 1.11)
	Separated/Divorced		0.935 (0.804 to 1.087)
	Other		1.265 (0.981 to 1.631)
Lonely F2	No		Reference 1
	Yes		0.718 (0.669 to 0.769)
VomitingJ3n	No		Reference 1
	Yes		1.721 (1.481 to 2)
Peripheral edema J3u	No		Reference 1
	Yes		1.262 (1.19 to 1.338)
Fluid intake K2c	Normal		Reference 1
	<1000cc/day		1.528 (1.407 to 1.659)
Bladder continence H1	Continent		Reference 1
	Continent with ostomy		1.303 (1.183 to 1.435)
	Infrequently incontinent		0.997 (0.907 to 1.095)
	Occasionally incontinent		0.988 (0.902 to 1.082)
	Frequently incontinent		0.981 (0.911 to 1.057)
	Incontinent		1.491 (1.337 to 1.663)

The parsimonious model developed in the five data sets has AUCs from 0.772 to 0.774, only fractionally smaller than for the full model. The validation AUCs ranged from 0.763 to 0.782. The calibration was good in each of the five validation data sets (see supplement).

Final one-year predictive model

The final predictive model was the model constructed on the entire data set with the 16 variables of the parsimonious models. The AUC was 0.773 (0.769 to 0.777), R2 0.241, and Brier score 0.132.

Sub-group analysis

The full parsimonious model performed similarly for males and females (AUCs of 0.76 [0.763 to 0.774] and 0.766 [0.761 to 0.771], respectively; Fig. 4). Discriminatory performance was highest in the youngest age group (65 to 74 years: AUC 0.797 [0.788 to 0.806]) and progressively poorer in older age groups (Fig. 4). However, discrimination was still good in the oldest age group (95 or older: AUC 0.721[0.702 to 0.739]). A substantial minority (46.4%) of participants had at least one of the comorbidities: cancer, CHF, COPD, CHD. The model performed similarly across all four comorbidities with point estimates from 0.737 (CHF) to 0.760 (CHD). Amongst the 53.6% who had none of those comorbidities, the AUC was 0.738 [0.731 to 0.745]. The model performed similarly across the ethnic groups with point estimates from 0.762 (Māori) to 0.782 (Other). Those with a normal BMI had a marginally greater point estimate AUC (0.766) than those underweight (0.744) and those overweight (0.758) or obese (0.751).

Fig. 4

The Area Under the receiver operating characteristic Curve (95% confidence interval) for the parsimonious model in sub-groups. BMI: Body Mass Index; CHF: Chronic Heart Failure; COPD: Chronic Obstructive Pulmonary Disorder; CHD: Coronary Heart Disease.

In all sub-groups the model was very well calibrated (Supplement figures).

Parsimonious three-month model

Twenty-two variables (Table 1) explained >1% each of the model in at least three of the five datasets (18 variables were in all five data sets). These included all the variables in the one-year model with the exception of other skin ulcer, and with the addition of loneliness, marital status, timed walk, fluid intake, vomiting, and peripheral oedema. Slow walking speed, poor fluid intake (less than 1000cc/day), vomiting in the last 3 days or having peripheral edema increase the risk. Compared to never being married, being married or in a defacto relationship slightly reduced risk, as did being lonely (Supplementary Table S9). Unlike for the one-year model age was one of the weaker contributors. Cancer remained the strongest contributor followed by dyspnea, weight loss, CHF, recent hospital stay, sex and fatigue. The final parsimonious model was well calibrated and had an AUC of 0.843 (0.839 to 0.848) (Supplementary figure S10).

Discussion

In this cohort of older people with complex needs who had a mandated standardised interRAI-HC assessment, predictive models with 16 and 22 variables respectively provided accurate and precise risk predictions for one-year and three-month mortality. The calibration was excellent and the AUCs, a measure of the ability of the model to discriminate between those who do and do not die, of 0.78 and 0.84 were good and were high compared with other similar models. While the full models were slightly better than the parsimonious models, an advantage of the parsimonious models are there are fewer variables making it easier to measure and calculate.

There is no validated mortality score for use with the interRAI-HC instrument. However, in 2003, a 6-level mortality score (CHESS) was developed in Canadian nursing home patients for use with the interRAI instruments. Nevertheless, this score has been applied in the home care context and found to be associated with mortality [17]. All items in the interRAI CHESS score, but two (end-stage disease, decrease in food or fluid) were included in the full models, while change in ADL status and weight loss were the only two variables that made it into the parsimonious models. The CHESS score includes a measure (end-stage disease) which incorporates a judgement about whether the patient has six months or less to live which can be difficult to use in such judgements. Additionally, staff judgement about whether the patient is end stage is not a replicable variable.

An updated CHESS developed in US Medicare beneficiaries admitted to a nursing home summed each ADL item so that anyone with an ADL score of 23 or higher was considered to have severe physical impairment, whereas our full model included each ADL individually (not dressing upper body and walking) [11]. The Minimum Data Set 3.0 was developed specifically for nursing home residents and predicted mortality within 30 days, 60 days, and 1-year of assessment. The 1-year assessment had a C-statistic of 0.655 (95% CI: 0.654–0.657). Similar to the CHESS 3.0 and the MRS3 (Mortality Risk Score 3.0) were developed for nursing home residents using the MDS 3.0 assessment [10]. The MRS3 was used to predict 30- and 60-day mortality with a C-statistic of 0.744 (95% CI: 0.741–0.747) for 30-days and 0.709 (95% CI: 0.706–0.711) for 60-days. The CHESS 3.0 does not include age or sex as variables and the MRS3 does not include sex [10]. All ordinal variables were dichotomized before analysis. For both scores, and in contrast to our model, any variables that may mitigate the risk of death were not included.

Other large studies have integrated a broader range of health data into mortality prediction models. The UK Biobank study used a cohort of about 500,000 people aged 40–70 years who attended centres where a standardised assessment was performed that included a questionnaire and clinical, biochemical tests [5]. Some of the questions in the interRAI-HC assessment were derived from clinical information such as disease diagnoses whereas, the UK Biobank questionnaire was for individuals to be able to answer the questions themselves. Questions included in both the UK Biobank and our model were age, tobacco smoking, illnesses, walking pace, anxiety and depression, and cancer. Of these, age and cancer were in the parsimonious model. The UK Biobank study had a C-statistic of 0.80 (95% CI: 0.77–0.83) for 5-year mortality. This is similar to our result but required all participants to undergo blood tests and participate in detailed physical and biological measurements.

The Qmortality study is perhaps the most similar to ours, although it did not use the interRAI instrument [12]. This UK Qmortality study developed a one-year mortality prediction model from 1.47 million patients aged 65 years and older who registered with a primary care clinical practice. Mortality contained 33 items, and age, hospital stay, dementia, Parkinson's disease, CHD, CHF, COPD, cancer, diabetes mellitus, smoking tobacco, alcohol, and BMI were used in both the Qmortality and our full models. Activities of daily living and questions relating to exercise were not included. For one-year mortality, the Qmortality score had a C-statistic of 0.854 (95% CI: 0.850-0.859) for females and 0.844 (0.839-0.849) for males. While this compares favourably with the AUC in our one-year parsimonious model 0.773 (0.769 to 0.777), that study utilised a broader range of health information including a deprivation score and blood tests.

Our models could be used for shared decision-making for treatment options. The clinician, patient, and their family could make informed decisions about the patient's treatment. This could also help inform decision making regarding who requires an end-of-life care plan, advanced directives or who should proceed with a more invasive elective operation or procedure. Policy makers within a jurisdiction may choose risk thresholds to inform health system needs, such as the need for palliative care and hospice services. This has international implications as over three million frail adults have interRAI-HC annually.

In previous work we have shown 93.1% of individuals undergoing an interRAI assessment provide consent for their data to be used in research [13]. We cannot exclude the possibility of bias having been introduced by the non-random exclusion of those who did not provide consent, but given the small proportion of potential participants any bias is likely to be small. Those excluded because of less than one-year follow-up had similar demographics and responses to interRAI questions to those included.

These mortality prediction models were developed in a New Zealand cohort. While internal validation was strong, the mortality model needs validating in an international context. The advantage of the interRAI suite of instruments are that they have been developed for use in multiple jurisdictions and are supported by comprehensive guidelines for assessors.

The two models developed are amongst the most comprehensive models developed for community-dwelling older adults. These well-validated mortality prediction models have strong predictive ability giving estimations of risk that are well calibrated. Additionally, assessments are usually preformed entirely by non-medical health professionals and do not require blood tests or physical examination. Following appropriate real-world assessment these models may contribute to both forecasting for policy making clinical decision-making for an individual's needs.

Funding statement

The project was funded by Health Research Council of New Zealand grant 18/006

Contributors

HJ designed the study, obtained the data and ethics; JP led the statistical analysis with assistance from HA and RAN; all authors contributed to writing the manuscript

Data sharing statement

The dataset contains anonymised data from New Zealand. Currently New Zealand Ethics Regulations do not allow transfer of the data out of New Zealand. Visitors to New Zealand can view the data with permission from a team member.

Declaration of Competing Interest

All authors declare no conflicts of interest.