Developing and validating a scoring system for measuring frailty in patients with hip fracture: a novel model for predicting short-term postoperative mortality.

Objectives: Frailty is common among patients with hip fracture and may, in part, contribute to the increased risk of mortality and morbidity after hip fracture surgery. This study aimed to develop a novel frailty score for patients with traumatic hip fracture that could be used to predict postoperative mortality as well as facilitate further research into the role of frailty in patients with hip fracture.
Methods: The Orthopedic Hip Frailty Score (OFS) was developed using a national dataset, retrieved from the Swedish National Quality Registry for Hip Fractures, that contained all adult patients who underwent surgery for a traumatic hip fracture in Sweden between January 1, 2008 and December 31, 2017. Candidate variables were selected from the Nottingham Hip Fracture Score, Sernbo Score, Charlson Comorbidity Index, 5-factor modified Frailty Index, as well as the Revised Cardiac Risk Index and ranked based on their permutation importance, with the top 5 variables being selected for the score. The OFS was then validated on a local dataset that only included patients from Orebro County, Sweden.
Results: The national dataset consisted of 126,065 patients. 2365 patients were present in the local dataset. The most important variables for predicting 30-day mortality were congestive heart failure, institutionalization, non-independent functional status, an age ≥85, and a history of malignancy. In the local dataset, the OFS achieved an area under the receiver-operating characteristic curve (95% CI) of 0.77 (0.74 to 0.80) and 0.76 (0.74 to 0.78) when predicting 30-day and 90-day postoperative mortality, respectively. Conclusions: The OFS is a significant predictor of short-term postoperative mortality in patients with hip fracture that outperforms, or performs on par with, all other investigated indices. Level of evidence: Level III, Prognostic and Epidemiological. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Frailty is common among patients with hip fracture and contributes to the high risk of postoperative mortality observed among this population. While many frailty indices have been tested in the context of patients with hip fracture, most tend to be cumbersome and difficult to implement in the emergency setting.

This study develops and validates the Orthopedic Hip Frailty Score, a frailty score specifically targeted toward patients with hip fracture that outperforms, or performs on par with, all other investigated indices.

Further research is required to determine how the Orthopedic Hip Frailty Score can be most effectively used to aid in resource allocation and clinical decision-making.

Background

Hip fractures are associated with a high incidence of postoperative mortality with rates as high as 10%, 16% and 27% being reported after 30, 90 and 365 days, respectively.1 2 These rates have remained high despite a multitude of strategies being employed to reduce unfavorable outcomes, such as orthopedic innovations, multidisciplinary management, and fast-track programs.1–3 The healthcare burden associated with hip fractures is also predicted to increase in the coming years as a result of a rise in the incidence of hip fractures driven by an aging global population.4 5 These patients tend to both be older and suffer from a higher comorbidity burden compared with the general population; simultaneously, patients with hip fracture exhibit a high degree of heterogeneity.6–8 Risk stratification tools may subsequently be a vital asset for managing this patient population. While clinical decisions should be made by the treating physician based on the individual case, predictive tools offer the advantage of facilitating the distribution of healthcare resources and expertise with greater efficiency while also permitting the early identification of patients with an excess risk of deterioration.

Numerous studies have been performed investigating the utility of different indices that could potentially fill this role; however, with few exceptions, these indices have been measurements of patients’ comorbidity burden or fitness for surgery.6 7 9–11 Frailty, a reduced ability to tolerate external stressors due to a reduced physiologic reserve,12 13 is common among patients with hip fracture and may also help explain the increased risk of mortality and morbidity after hip fracture surgery.12–15 Nevertheless, the utility of frailty for predicting postoperative mortality in hip fractures has been less thoroughly studied, and the scales that have been used tend to be cumbersome and difficult to implement in the emergency setting.16 17 This study aimed to develop a novel frailty score for patients with traumatic hip fracture that could be used to predict postoperative mortality as well as facilitate further research into the role of frailty in patients with hip fracture.

Methods

The study was approved by the Swedish National Review Authority (ref: 2021-05403-02) and adhered to both the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines as well as the Declaration of Helsinki.18 Two datasets were used to create the Orthopedic Hip Frailty Score (OFS); the larger, national dataset was used to develop the score whereas the local dataset was used to validate the score. The national dataset included all consecutive adult patients (18 or older) who underwent surgery for a traumatic hip fracture in Sweden between January 1, 2008 and December 31, 2017. The local dataset contained all consecutive adult patients who underwent surgery for a traumatic hip fracture in Orebro County, Sweden between January 1, 2013 and December 31, 2017. Orebro county consists of three hospitals: Orebro University Hospital and two university-affiliated regional hospitals. To keep the sample population homogenous, reoperations and pathological hip fractures were excluded along with conservatively managed hip fractures. Cases missing data required to calculate any of the indices were also excluded to allow for a complete case analysis. Cases in the local dataset were excluded from the national dataset.

Both datasets were originally retrieved from the Swedish National Quality Registry for Hip Fractures (RIKSHÖFT).19 Variables retrieved from this database included age, sex, American Society of Anesthesiologists (ASA) classification, and measurements of frailty such as non-independent functional status, living arrangements, walking ability, and use of walking aids.20 The national dataset was cross-referenced with the Swedish National Board of Health and Welfare’s Cause of Death and Patient Registers, which are contributed to by all healthcare providers nationally, to obtain variables pertaining to mortality and comorbidities. For the local dataset, this information was obtained from the patients’ electronic medical records.

Calculating alternative indices

Nottingham Hip Fracture Score

The Nottingham Hip Fracture Score (NHFS) was predominantly calculated according to the original study by Maxwell et al.21 Patients received 3 points if they were between 66 and 85 years of age or 4 points if they were 86 or older. Patients also received 1 point if they were male, had an admission hemoglobin <100 g/L, were institutionalized, had a malignant tumor (excluding non-invasive neoplasms of the skin), or had ≥2 comorbidities. However, since a mini-mental test score was unavailable, patients instead received 1 point if they had dementia, in accordance with previous studies investigating the NHFS’s association with postoperative mortality.22–25

Sernbo Score

The Sernbo Score is determined using four variables: age, use of walking aids, institutionalization, and mental status.26 Patients receive 5 points for each of the following: age <80, using no walking aid or a single cane, living in their own home, or an alert mental status. Patients received 2 points for an age ≥80, requiring an advanced walking aid (two canes, a walking frame/rollator, or being bedridden), institutionalization, or confusion. Due to its reversed scoring system, the Sernbo Score differs compared with other risk indices in that a lower score, rather than a higher score, is considered worse.

Charlson Comorbidity Index

The age-adjusted Charlson Comorbidity Index (CCI) is based on both the age of the patient as well as the presence of specific comorbidities.27 Patients received 1 point if their age was between 50 and 59, 2 points if between 60 and 69, 3 points if between 70 and 79, and 4 points if ≥80. Patients also received 1 point for the presence of a previous myocardial infarction, congestive heart failure (CHF), peripheral vascular disease, previous cerebrovascular events, dementia, chronic obstructive pulmonary disease (COPD), connective tissue disease, peptic ulcer disease, mild liver disease, or uncomplicated diabetes mellitus. Patients with moderate-to-severe liver disease received 3 points while those with end-organ damage secondary to diabetes mellitus received 2 points. Patients also received 2 points for hemiplegia, moderate-to-severe chronic kidney disease (CKD), leukemia, lymphoma, or localized cancer. The presence of metastatic cancer or AIDS gave 6 points individually.

5-Factor modified Frailty Index

The 5-factor modified Frailty Index (5-mFI) was calculated as the sum of all the variables included in the index that were present in a patient.28 These variables included respiratory pathology (COPD or ongoing pneumonia), diabetes mellitus, hypertension requiring medication, CHF, and non-independent functional status.

Revised Cardiac Risk Index

The Revised Cardiac Risk Index (RCRI) score was determined based on the following variables: history of ischemic heart disease, CHF, history of cerebrovascular disease, preoperative insulin therapy, preoperative creatinine above 2 mg/dL, and high-risk surgery. Each variable resulted in 1 point if present. Hip fracture surgery is considered intermediate risk surgery according to the American College of Cardiology and the American Heart Association guidelines, so no patients received a point for high-risk surgery.29 Patients who had end-organ damage resulting from diabetes mellitus, but did not receive preoperative insulin therapy, also received 1 point to reflect the severity of their diabetes.6

Developing the OFS

Variables that could be considered as potential markers of frailty were selected from the NHFS, Sernbo Score, CCI, 5-mFI, and RCRI.20 All variables were simplified and dichotomized, so that they could be recorded as true or false. This resulted in the following candidate variables being selected for the OFS: age ≥65, age ≥80, age ≥85, dementia, CHF, diabetes mellitus, hypertension, COPD, previous myocardial infarction, history of malignancy (any previous localized or metastatic cancer, excluding non-invasive neoplasms of the skin), confusion, institutionalization, non-independent functional status (requiring assistance with activities of daily life), and requiring an advanced walking aid (two canes, a walking frame/rollator, or being bedridden).

Using the national dataset, a logistic regression (LR) model was constructed with these candidate variables as predictors and 30-day postoperative mortality as the outcome.30 The relative importance of the candidate variables in the model was determined using the permutation importance (PI).31 The PI was evaluated based on how much a predetermined value (1−area under the receiver-operating characteristic curve (AUC)) was affected by the omission of a specific variable. Instead of merely eliminating the variable from the dataset, this method replaces it with noise from other cases. This is achieved by rearranging the values for the variable, which masks the information of the variable during evaluation. To account for the inherent uncertainty associated with the use of permutations, this process was repeated 10 times. The relative importance of each variable in the model was then presented as the average increase in 1−AUC compared with the AUC in a model including all variables without masking.

The top five most important variables, based on their relative importance, were then selected for inclusion in the OFS. Each variable was worth 1 point, which meant the OFS could range between 0 and 5.

Validating the OFS

Using the variables previously determined to be part of the OFS, the OFS was calculated for each patient in the local dataset. To evaluate the predictive ability of the OFS, a LR model was fitted to the local dataset with 30-day mortality as the response variable and the OFS as the predictor.30 The AUC, along with the sensitivity and specificity that maximized Youden’s index (sensitivity+specificity–1), was calculated for this model along with their respective 95% CIs. The CIs for the AUCs were determined using the variance of the AUC as defined by DeLong et al,32 using the algorithm by Sun and Xu.33 The remaining CIs were estimated using 2000 stratified bootstrap replicates. This process was repeated with 90-day mortality as the outcome as well as the NHFS, Sernbo Score, CCI, 5-mFI, ASA Classification, and RCRI as predictors. The AUCs were compared using the process described by DeLong et al for paired curves while employing the algorithm by Sun and Xu.32 33

Quantifying the risk of mortality associated with frailty

The threshold which maximized Youden’s index for the OFS was also selected as a cut-off for defining frailty. Based on this threshold, patients were classified as either frail or not frail. To determine the association between frailty and postoperative mortality in the local dataset, Poisson regression models with robust standard errors were used. 30-day and 90-day postoperative mortality was included as the outcome while the predictors were frailty (dichotomized as true or false), age, sex, type of fracture, type of surgery, previous myocardial infarctions, previous cerebrovascular events, peripheral vascular disease, dementia, diabetes mellitus, CKD, COPD, connective tissue disease, and liver disease. The results of these analyses are presented as incident rate ratios (IRRs) with 95% CIs.

Statistical analysis

Statistical significance was defined as a two-sided p value <0.05. Analyses were performed using the statistical programming language R (R Foundation for Statistical Computing, Vienna, Austria) using the tidyverse, DALEX, haven, lubridate, parallel, pROC, robustbase, and rlist packages.34

Results

After applying the previously mentioned inclusion and exclusion criteria, 126,065 patients remained in the national dataset and 2365 patients remained in the local dataset. A comparison of the datasets can be found in online supplemental table 1. Based on the permutation importance calculated using the national dataset, the top five most important variables were CHF, institutionalization, non-independent functional status, an age ≥85, and a history of malignancy (figures 1 and 2). Using the national dataset, the OFS calculated using these variables exhibited an AUC (95% CI) of 0.72 (0.72 to 0.73) when predicting 30-day postoperative mortality (table 1).

Table 1

Predictive ability of the OFS based on the national and local dataset for 30-day mortality

Dataset	AUC (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	P value for difference in AUC
National	0.72 (0.72 to 0.73)	0.80 (0.79 to 0.81)	0.54 (0.53 to 0.55)	Reference
Local	0.77 (0.74 to 0.80)	0.84 (0.78 to 0.89)	0.56 (0.49 to 0.61)	0.006

The OFS was developed using the national dataset and validated using the local dataset.

AUC, area under the receiver-operating characteristic curve; OFS, Orthopedic Hip Frailty Score.

Relative importance of predictors for 30-day postoperative mortality with a logistic regression model. COPD, chronic obstructive pulmonary disease.

Figure summarizing the Orthopedic Hip Frailty Score.

In the local dataset, the OFS achieved an AUC (95% CI) of 0.77 (0.74 to 0.80) when predicting 30-day postoperative mortality. It thereby had the same overall predictive ability as the NHFS [AUC (95% CI): 0.76 (0.73 to 0.79), p=0.620]. Furthermore, the OFS outperformed all the remaining indices, including the Sernbo Score [AUC (95% CI): 0.73 (0.69 to 0.76), p=0.011], CCI [AUC (95% CI): 0.69 (0.66 to 0.73), p<0.001], 5-mFI [AUC (95% CI): 0.69 (0.66 to 0.73), p<0.001], ASA Classification [AUC (95% CI): 0.67 (0.64 to 0.71), p<0.001], and RCRI [AUC (95% CI): 0.59 (0.55 to 0.63), p<0.001] (table 2). The same pattern was observed when predicting 90-day postoperative mortality, with the OFS demonstrating the highest predictive ability [AUC (95% CI): 0.76 (0.74 to 0.78)] (table 3). The mortality distribution for all indices can be found in online supplemental table 2.

Table 2

Predictive ability of the OFS compared with other indices for 30-day mortality

Index	AUC (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	P value for difference in AUC
OFS	0.77 (0.74 to 0.80)	0.84 (0.78 to 0.89)	0.56 (0.49 to 0.61)	Reference
NHFS	0.76 (0.73 to 0.79)	0.67 (0.60 to 0.74)	0.71 (0.65 to 0.76)	0.620
Sernbo Score	0.73 (0.69 to 0.76)	0.64 (0.57 to 0.71)	0.71 (0.63 to 0.75)	0.011
CCI	0.69 (0.66 to 0.73)	0.55 (0.48 to 0.63)	0.72 (0.65 to 0.77)	<0.001
5-mFI	0.69 (0.66 to 0.73)	0.71 (0.64 to 0.78)	0.57 (0.49 to 0.62)	<0.001
ASA Classification	0.67 (0.64 to 0.71)	0.75 (0.68 to 0.81)	0.50 (0.41 to 0.56)	<0.001
RCRI	0.59 (0.55 to 0.63)	0.62 (0.55 to 0.69)	0.51 (0.43 to 0.58)	<0.001

ASA, American Society of Anesthesiologists; AUC, area under the receiver-operating characteristic curve; CCI, Charlson Comorbidity Index; 5-mFI, 5-factor modified Frailty Index; NHFS, Nottingham Hip Fracture Score; OFS, Orthopedic Hip Frailty Score; RCRI, Revised Cardiac Risk Index.

Table 3

Predictive ability of the OFS compared with other indices for 90-day mortality

Index	AUC (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	P value for difference in AUC
OFS	0.76 (0.74 to 0.78)	0.81 (0.76 to 0.85)	0.59 (0.53 to 0.62)	Reference
NHFS	0.75 (0.73 to 0.78)	0.62 (0.56 to 0.68)	0.75 (0.69 to 0.78)	0.337
Sernbo Score	0.73 (0.70 to 0.75)	0.62 (0.56 to 0.68)	0.73 (0.67 to 0.76)	0.002
CCI	0.70 (0.67 to 0.73)	0.72 (0.67 to 0.77)	0.55 (0.50 to 0.60)	<0.001
5-mFI	0.68 (0.65 to 0.71)	0.69 (0.63 to 0.74)	0.59 (0.52 to 0.63)	<0.001
ASA Classification	0.67 (0.64 to 0.70)	0.75 (0.70 to 0.79)	0.52 (0.45 to 0.56)	<0.001
RCRI	0.59 (0.56 to 0.62)	0.62 (0.57 to 0.68)	0.52 (0.46 to 0.57)	<0.001

ASA, American Society of Anesthesiologists; AUC, area under the receiver-operating characteristic curve; CCI, Charlson Comorbidity Index; 5-mFI, 5-factor modified Frailty Index; NHFS, Nottingham Hip Fracture Score; OFS, Orthopedic Hip Frailty Score; RCRI, Revised Cardiac Risk Index.

In the adjusted Poisson regression analysis, frailty (defined as an OFS ≥2) was associated with a 244% increased risk of 30-day postoperative mortality [adjusted IRR (95% CI): 3.44 (2.12 to 5.57), p<0.001]. Frailty was also associated with a 168% increase in the risk of 90-day postoperative mortality [adjusted IRR (95% CI): 2.68 (1.94 to 3.69), p<0.001], after adjusting for age, sex, type of fracture, type of surgery and comorbidities. A more detailed description of the local dataset can be found in online supplemental table 3.

Based on the logistic regression models from the local dataset, the following equations could be used to predict the probability of a patient dying within 30 and 90 days after hip fracture surgery:

Discussion

Our study is the first to develop and validate a frailty score tailored to the assessment of mortality in patients with hip fracture. In this study, the OFS outperformed all other indices in predicting both 30-day and 90-day mortality, except the NHFS which performed at a similar level. The cut-off that maximized Youden’s index was OFS ≥2, indicating that this can be considered as a threshold for defining patients as frail. Patients with an OFS ≥2 were 3.4 times more likely to die within 30 days postoperatively and 2.7 times more likely to die within 90 days postoperatively, compared with patients with a lower OFS. The OFS could accordingly be a useful and simple tool for incorporating frailty into research and clinical decision-making among patients with hip fracture.

To develop a frailty score, it is first essential to agree on a definition of frailty. Unfortunately, there remains a clinical equipoise regarding this subject. However, frailty is often characterized as a condition in which patients have a reduced physiological reserve to withstand stressors due to the degeneration of multiple organ systems, which results in an increased risk of morbidity, disability, and mortality.14 35–38 It should also be considered disparate from the concept of comorbidity.39 At the same time, these two concepts cannot be definitively extricated from each other; while frailty can promote the development of diseases, comorbidities may also precipitate the progression of frailty.39

The question then remains, if the OFS actually succeeds in capturing the concept of frailty. Most of the variables included in the score, institutionalization, non-independent functional status, and a history of malignancy, have been used previously to measure frailty and are widely accepted and validated as markers of frailty.20 28 40 41 Including age might be considered to be a more contentious decision. It is important to note that frailty is an independent process from aging; however, frailty also becomes more prevalent at higher ages and has been used previously as a component when assessing frailty.42 43 Furthermore, with the OFS, an age ≥85 on its own is insufficient for a patient to be classified as frail. The inclusion of CHF as a variable also bears discussing. While it is clearly a comorbidity, it should also be seen as a marker of frailty. Heart failure can be defined as an inability for the heart to maintain an adequate cardiac output to meet the body’s demands, or the inability to maintain an adequate output without compensatory mechanisms.44 45 This fits together neatly with the concept of frailty, as according to this definition, CHF would indicate that a patient has a reduced physiological reserve to respond to external stressors. This is further corroborated by the inclusion of CHF in the 5-mFI.28 40

Of the indices that are compared with the OFS, only the NHFS and Sernbo Score can be considered to achieve an acceptable predictive ability with AUCs ≥0.7 for both 30-day and 90-day mortality. However, it is worth noting that the underlying reason for this discriminative ability differs significantly. While the NHFS and Sernbo Score demonstrate superior specificity, the OFS instead excels in sensitivity; the OFS is the only score with a diagnostic statistic ≥0.8 at the threshold that maximizes Youden’s Index. A uniting factor for all frailty indices is that none truly measure frailty itself, instead they all use different surrogates for frailty to identify which patients can be classified as frail. Consequently, if the goal of the OFS is to identify frail patients with an elevated risk of postoperative mortality, then a high sensitivity can be argued to be more important than a high specificity in identifying those patients who have the most to gain from additional interventions.

The OFS could accordingly be useful for identifying these high-risk patients early on, which could aid in a more effective allocation of expertise and resources, such as multidisciplinary interventions.15 It could also be a useful tool for communicating with patients and their relatives. Frailty as a concept can be challenging for patients to understand, which is understandable given that even clinicians at times struggle with the subject.46 However, this process could be simplified if a number with a concrete reasoning behind it could be provided. The OFS may also be useful for further research into frailty among patients with hip fracture. Most retrospective databases lack the granular data required to capture frailty, but the variables used in the OFS tend to be readily available, which might allow more of these databases to be used in investigating the role of frailty in hip fracture management and outcomes.

The OFS’s utility in research is further aided by its simplicity. Many frailty indices are significantly more complex or require measurements which may be challenging or unfeasible to assess in the emergency setting.47–49 The OFS, on the other hand, only requires five binary variables, which can easily be retrieved from a patient’s electronic medical records or determined with little delay after arrival to the emergency room. Despite this simplicity, the OFS still demonstrates the same level of discriminatory ability for mortality as the NHFS. This is despite the NHFS making use of preoperative blood tests along with requiring more variables that each contribute different amounts of points to the final score. For any screening tool used for risk stratification in clinical practice, simplicity is essential. This is of particular importance in orthopedic surgery, where there is a preference for straightforward tools over more complex or purely physical measures.46

There are several limitations in the current study that bear mentioning. While the local dataset is based on patients from all hospitals within Orebro County, the generalizability may be limited due to this geographic restriction. Future validating studies will consequently be required. This study also focused on predicting short-term postoperative mortality in patients with hip fracture. However, there is nothing in the OFS that limits it to either this particular outcome or patient population. Future studies should accordingly also determine the OFS’s discriminative ability for alternative adverse outcomes as well as consider other patient populations. The weaknesses inherent in using retrospective datasets is also apparent when comparing the predictive ability of the OFS in the local dataset with the national dataset. While the Swedish National Quality Registry for Hip Fractures constitutes a prospectively collected, nationwide sample population that is contributed to by almost all Swedish orthopedic departments and boasts a high case coverage between 80% and 90%,50 certain variables were less readily registered than others, particularly in regard to institutionalization and non-independent functional status. In the local dataset, missing data as well as any errors that occurred during registration could be corrected using the patients’ electronic medical records, which could explain why the OFS demonstrated a higher performance in the validation dataset.

Conclusion

The OFS is a significant predictor of short-term postoperative mortality in patients with hip fracture that outperforms, or performs on par with, all other investigated indices. The score adequately measures frailty and demonstrates potential utility within both clinical practice and research.