Interrater Reliability of the Clinical Frailty Scale by Geriatrician and Intensivist in Patients Admitted to the Intensive Care Unit.

BACKGROUND: The Clinical Frailty Scale (CFS) is a commonly used frailty measure in intensive care unit (ICU) settings. We are interested in the test characteristics, especially interrater reliability, of the CFS in ICU by comparing the scores of intensivists to geriatricians.
METHODS: We conducted a prospective cohort study on a convenience sample of newly admitted patients to an ICU in Edmonton, Canada. An intensivist and a resident in Geriatric Medicine (GM) independently assigned a CFS score on 158 adults within 72 hours of admission. A specialist in Geriatric Medicine assigned a CFS score independently of 20 of the 158 patients to assess agreement between the two raters trained in geriatrics. Predictive validity was captured using mortality and length of stay.
RESULTS: Agreement on CFS score was fair for intensivists vs. GM resident (kappa 0.32) and for intensivists vs. GM specialist (0.29), but substantial for GM resident vs. staff (0.79). Despite this, the CFS remained prognostically relevant, regardless of rater background. Frailty (CFS ≥ 5) as assessed by either intensivist or GM resident was a strong predictor of in-hospital mortality (odds ratio [OR] 3.6; 95% CI, 1.6-8.4, p = .003 and OR 3.0; 95% CI 1.3-6.9; p = .01, respectively). Frailty was also positively correlated with age, illness severity measured by APACHE II score, and length of hospital stay.
CONCLUSIONS: The interrater reliability of the CFS in ICU settings is fair for intensivists vs. geriatricians.

INTRODUCTION

With an aging population, there is a need to understand how best to support patients with complex medical and psychosocial needs who develop critical illness.( That complexity is well captured in the construct of frailty—a dynamic, multi-dimensional syndrome characterized by diminished physiologic reserve and heightened vulnerability to poor outcomes. Frailty has a high prevalence in patients admitted to the Intensive Care Unit (ICU), and is associated with prolonged hospital stay, death, and disability.(

The choice of the most appropriate frailty assessment tool depends on the purpose, setting, intended population, and available time and skill of the operator.( Despite widespread use in acute care, the CFS was originally designed and validated to measure frailty in an ambulatory setting concomitant with completion of a comprehensive geriatric assessment (CGA).( Its use has since been extrapolated, and the CFS has become the most common method of measuring frailty in ICU settings worldwide.( It has been promoted for use in surgical patients without prior CGA because it can be completed in less than one minute.(

Inherent to any judgment-based frailty measure, applying the CFS presupposes that the assessor can recognize frailty subjectively, based on the information that each rater judges to be most relevant. However, in ICU settings there are potential challenges to judging whether a patient may be frail, such as the need to rely on proxies for robust information at the time of critical illness, inability to interview the patient due to altered level of consciousness, and the potential to overestimate baseline frailty by relying on features that temporally worsened due to the acute illness.

A recent systematic review on the feasibility of measuring frailty in critically ill patients found that the majority of existing research has focused on predictive validity.( Six studies, in addition to the original validation done in community dwelling outpatients, examined concurrent validity.( Of these, three had a primary objective of evaluating the reliability of frailty assessment in the ICU; however, none of them explicitly tested agreement with geriatricians based on CGA.( Instead, CFS ratings were obtained from chart review rather than direct patient or family contact,( or included only raters without formal training in geriatrics.( Given this existing gap in the literature, we sought to evaluate the test characteristics of the CFS, comparing geriatricians and intensivists in a critically ill population. We postulated poor agreement in the CFS score between geriatricians and intensivists.

METHODS

This study was approved by the Research Ethics Board at the University of Alberta, Edmonton (File # Pro00056591). The ethics board did not require informed consent as the CFS score was integrated into standard care practices for all patients admitted to intensive care, including non-study participants.

Study Design

As part of a larger prospective cohort study,( we evaluated the interrater reliability of the CFS among three raters—Geriatric Medicine specialist, Geriatric Medicine resident, and intensivist—in newly admitted patients to the ICU. We used Cohen’s kappa to represent interrater reliability for comparisons amongst the three rater categories.

The Clinical Frailty Scale (CFS) was selected as the preferred tool, as it is relatively intuitive to clinicians and can be completed briefly based on information about a patient’s medical history and functional status. In the case of ICU settings, only the CFS has been broadly tested. Frailty was operationalized using the nine-point CFS, with scores of 1–3 being not frail, 4 as vulnerable, and 5–8 as frail.( Frailty assessment using the CFS is part of the routine admission ICU documentation and was performed on all patients.

The admitting intensivist and the Geriatric Medicine resident assigned CFS scores independently. In order to determine whether the CGA and frailty assessment done by the Geriatric Medicine resident served as an accurate surrogate for an assessment by a geriatrician, a random sample of 20 of the 158 patients was evaluated by a staff consultant in Geriatric Medicine within the 72-hour window. The two raters from geriatrics were mutually blinded to the other assessment so that reliability between Geriatric Medicine specialist and resident could be determined.

Setting

Assessments were conducted in a 32-bed mixed medical surgical ICU at the University of Alberta Hospital in Edmonton, Alberta.

Population

We assembled a convenience sample of new admissions to the ICU between October 2016 and July 2017. All adult admissions (age 18 or older) in whom the CFS had been completed by the admitting intensivist within 72 hours were eligible. Approximately once per week, at variable times of day, the Geriatric Medicine resident reviewed the current ICU census, and included all who met study criteria as potential participants.

Intervention

The specialist and resident in Geriatric Medicine independently performed comprehensive geriatric assessment supported by the CGA form that was used in the original CFS validation work (see https://www.dal.ca/sites/gmr/our-tools/comprehensive-geriatric-assessment.html), then assigned a CFS for each patient using patient and family interview and chart review. Information encompassed the key CGA elements, including an assessment of prior impairment in activities of daily living, cognitive or mood symptoms, geriatric syndromes, weight loss, exercise tolerance, and prior level of care, all leading to a synthesized problem list. The Geriatric Medicine resident and specialist established the pre-illness baseline level of function through interview of the family or proxy based on functional ability two weeks prior to their admission.

The intensivists received prior education and training on assigning CFS scores aligned with the Scale’s provincial implementation in the electronic health record, and generally use informed clinical judgment based on the information available around the time of ICU admission.

Each individual was assessed according to the CFS categories of robust (CFS 1–3), vulnerable (CFS 4), and frail (CFS 5–8), and the cohort was described by CFS category in terms of age, sex, reason for admission, common comorbid illnesses, Acute Physiology and Chronic Health Evaluation (APACHE) II score,( Sequential Organ Failure Assessment (SOFA) score,( and the Charlson Comorbidity Index.( The APACHE II score was completed within 24 hours of ICU admission, assigning a score between 0 and 71, with higher scores corresponding to more severe disease and higher risk of death. The SOFA score tracks organ failure status during an ICU stay based on the sum of scores across six systems, including respiratory, cardiovascular, hepatic, coagulation, renal, and neurologic, with a maximum score of 24 points representing the most organ failure and highest risk of morbidity and mortality. The Charlson Comorbidity Index, which includes 20 conditions each assigned a score of 1,2,3 or 6 based on estimated risk of death with a maximum score of 43, was calculated using ICD-10 codes.( The Charlson Comorbidity Index values were reported as interquartile ranges (lower quartile (25% quartile) = 0, median quartiles = 1, upper quartile (75% quartile) = 2). Mortality and length of stay (LOS) were recorded for both the ICU and the index hospital stay.

We measured predictive validity of the CFS in a critically ill population using hospital mortality and LOS. As measures of construct validity, we compared the CFS with age and Charlson Comorbidity Index. Additionally, to determine if severity of baseline frailty correlated with severity of acute illness, we compared the CFS scores with SOFA and APACHE II scores, respectively.

Data Sources and Analysis

We used the provincial clinical information system (eCritical Alberta), coupled with a data warehouse and clinical analytics system (eCritical TRACER), to obtain data. eCritical TRACER has been widely used to support health services research.( The Charlson Comorbidity Index scores were ascertained by linkage of eCritical data with Alberta Health Services (AHS) Discharge Abstract Data, housed in the AHS Data Repository for Reporting.(

Normally or near normally distributed data were reported as means with standard deviations (SD), and non-normally distributed continuous data were reported as medians with interquartile ranges (IQR). The Wilcoxon test was then utilized to determine the correlation between the CFS scores and characteristics of patient illness severity (Table 1).

TABLE 1

Wilcoxon test (p value) used to demonstrate association between CFS score assignment and age, Charlson Comorbidity Index, APACHE II, and SOFA score

Characteristics	GR CFS Score Assignment as Not Frail (≤4) vs. Frail (≥ 5)	ICU CFS Score Assignment as Not Frail (≤4) vs. Frail (≥5)
Age	<.0001	.0389
Charlson Comorbidity Index score	.0034	.0130
APACHE II score	.0003	.1611
SOFA score	.0068	.2777

P values <.05 considered statistically significant.

We used the Kappa coefficient to measure interrater reliability between Geriatric Medicine resident (GR), intensivist (ICU), and Geriatric Medicine staff physician (GS). Agreement and disagreement between two CFS raters was illustrated using enhanced Bland-Altman plots. Interobserver agreement is conventionally described as 0–0.19 “slight”, 0.20–0.39 “fair”, 0.40–0.59 “moderate”, 0.60–0.79 “substantial”, and 0.8–1.0 “almost perfect”.(

We performed all analyses using the Statistical Analysis System (SAS) Enterprise Guide 7.1. Two-sided p values of < .05 were considered as statistically significant.

RESULTS

Of 170 admissions that were screened, the CFS had not been completed by admitting intensivists for 12 (7.1%), leaving a sample of 158 subjects. Of these, the median (IQR) age was 60 years (46–69), 56% were male, most were admitted for medical reasons (72%), the median (IQR) Charlson Comorbidity Index score was 1 (0–2), the mean (SD) APACHE II score was 22.4 (7.9), and mean (SD) SOFA score was 8.6 (4.1) (Table 2). In total, 12% died in ICU and 18% in hospital.

TABLE 2

Patient’s characteristics by ICU CFS

Characteristic	Total (n=158)	ICU CFS 1–3 (n=73; 46%)	ICU CFS 4 (n=24; 15%)	ICU CFS 5–8 (n=61; 39%)
Age, median (IQR)	60.5 (46.0–69.0)	55.0 (41.0–65.0)	64.0 (54.5–68.5)	61.0 (54.0–71.0)
Sex, n (%)
Female	69 (43.7)	31 (42.5)	12 (50.0)	26 (42.6)
Male	89 (56.3)	42 (57.5)	12 (50.0)	35 (57.4)
Surgery, n (%)
Elective	12 (7.6)	10 (13.7)	1 (4.2)	1 (1.6)
Emergency	21 (13.3)	8 (11.0)	2 (8.3)	11 (18.0)
Non-operative	125 (79.1)	55 (75.3)	21 (87.5)	49 (80.3)
Admission Category, n (%)
Medical	113 (71.5)	49 (67.1)	19 (79.2)	45 (73.8)
Neurological	2 (1.3)	0 (0.0)	0 (0.0)	2 (3.3)
Surgical	36 (22.8)	19 (26.0)	3 (12.5)	14 (23)
Trauma	7 (4.5)	5 (6.8)	2 (8.3)	0 (0.0)
Comorbidity Disease, n (%)
Cardiac	33 (21.0)	8 (10.9)	11 (45.8)	14 (23.0)
Neurological	14 (8.8)	5 (6.9)	1 (4.2)	8 (13.1)
Chronic Pulmonary Disease	20 (12.7)	8 (11.0)	2 (8.3)	10 (16.4)
Connective Tissue Disease	4 (2.5)	2 (2.7)	0 (0.0)	2 (3.3)
Peptic Ulcer Disease	5 (3.2)	3 (4.1)	0 (0.0)	2 (3.3)
Liver Disease	36 (22.8)	16 (22)	7 (29.1)	14 (21.8)
Diabetes	48 (30.4)	21 (28.8)	9 (47.5)	18 (29.5)
Renal Disease	18 (11.4)	1 (1.4)	3 (12.5)	14 (23)
Cancer	37 (23.4)	14 (19.2)	4 (16.7)	19 (31.2)
Admission APACHE II, mean (SD)	22.4 (7.9)	20.1 (7.8)	26.8 (8.2)	23.5 (7.2)
SOFA, mean (SD)	8.6 (4.1)	7.9 (4.2)	9.5 (4.1)	8.9 (3.9)
ICU Mortality	19 (12.0)	5 (6.9)	4 (16.7)	10 (16.4)
Hospital Mortality	28 (18.0)	6 (8.3)	4 (16.7)	18 (30.0)
ICU LOS, median (IQR)	4.2 (2.6–9.0)	4.0 (2.7–9.7)	3.0 (2.3–7.7)	4.4 (3.1–9.0)
Hospital LOS, median (IQR)	18.1 (8.5–35.9)	16.8 (7.2–26.1)	18.4 (9.6–36.8)	23.8 (8.6–46.9)
Charlson Index, median (IQR)	1.0 (0.0–2.0)	1.0 (0.0–1.0)	2.0 (0.5–2.0)	2 (1.0–2.0)

IQR = intraquartile range; SD = standard deviation; ICU = intensive care unit.

In total 39% (n=61) and 49% (n=82) of patients were screened as frail by intensivists and Geriatric Medicine, respectively. Compared to patients who were not deemed to be frail, those assessed by the intensivists to have CFS score ≥ 5 had higher APACHE II scores (24 [20-27] vs. 19 [13-26], p < .001), longer hospital stay (24 [9-47] vs. 17 [7-26] days, p < .001), and greater in-hospital mortality (30% vs. 10%; p = .003).

There was a significant positive correlation between the CFS with age and the Charlson Comorbidity Index (p < .0001 and p = .0034, respectively). Correlations were similarly significant for intensivists and the Geriatric Medicine resident. The correlation of the CFS with the APACHE II score was significant when rated by Geriatric Medicine (p = .0003) but not when rated by intensivists (p = .1611) (Table 1).

As shown in Table 3, agreement on whether a patient was categorized as frail was fair between the intensivist and Geriatric Medicine resident (kappa 0.32; n=158, p = < .0001) (Figure 1), and between intensivist and Geriatric Medicine specialist (kappa 0.29; n=18, p = .1632) (Figure 2). Agreement was substantial between the Geriatric Medicine resident and specialist (kappa 0.79; n=20, p = .0004) (Figure 3).

TABLE 3

Kappa coefficient indicating interrater reliability between Geriatric Medicine resident (GR), intensivist (ICU), and Geriatric Medicine staff consultant (GS)

	Frail (CFS ≥ 5) vs. Not Frail (CFS 1–4)
	Kappa Coefficient	%95 CI	P valuea
GR vs. ICUn=158	0.32	(0.17, 0.46)	<.0001
GR vs. GSn=20	0.79	(0.52, 1.00)	.0004
GS vs. ICUn=18	0.29	(−0.11, 0.69)	.1632

P values obtained via two-sided test.

FIGURE 1

Enhanced Altman plot comparison of CFS by geriatric resident (GR_CFS) and by intensivist (ICU_CFS) with confidence intervalsa

aAccurate prediction with 10% homogeneous error.

FIGURE 2

Enhanced Altman plot comparison of Clinical Frailty Scale by Geriatric Medicine specialist (GS_CFS) and by intensivist (ICU_CFS) with confidence intervalsa

aAccurate prediction with 10% homogeneous error.

FIGURE 3

Enhanced Altman plot comparison of Clinical Frailty Scale by Geriatric Medicine resident (GR_CFS) and by Geriatric Medicine specialist (GS_CFS) with confidence intervalsa

aAccurate prediction with 10% homogeneous error.

Patients with a CFS ≥ 5 had higher in-hospital mortality when assessed by both intensivists (frail 30% vs. not frail 10%; odds ratio [OR] 3.6; 95% CI, 1.6–8.4; p = .003) and Geriatric Medicine (frail 26% vs. not frail 9%; OR 3.0; 95% CI, 1.3–6.9; p = .01).

DISCUSSION

In this prospective evaluation of the Clinical Frailty Scale in ICU settings, we found that agreement between physicians trained in Critical Care and Geriatric Medicine was fair, whereas agreement between the Geriatric Medicine specialist and resident was substantial. Interestingly, despite variable agreement, the CFS showed consistently good predictive validity regardless of the rater. Patients screened as frail had a higher in-hospital mortality, whether assessed by intensivists or Geriatric Medicine.

We found good construct validity when the CFS was compared to the Charlson Comorbidity Index and age. However, when comparing the CFS with measures of severity of acute illness, construct validity was inconsistent. The correlation of the CFS to the SOFA score was poor, as might be expected since the SOFA reflects burden of acute organ dysfunction rather than baseline functional status and frailty. Alternately, the CFS showed good construct validity with the APACHE II score when rated by Geriatric Medicine but not intensivists. Unlike the SOFA, the APACHE II includes additional elements relevant to frailty, such as age and four chronic comorbidities.

A recent multi-centre study examining interrater reliability (IRR) of the CFS in ICU patients found a higher level of agreement amongst raters with different backgrounds (kappa 0.74). The raters were intensivists, nurses, and physiotherapists. The lowest level of agreement was found between physician/nurse pairings, while the highest level of agreement was found between nurses and physiotherapists.( In contrast, in our study, the comparison was between intensivist assessment and the comprehensive geriatric assessment.

One important limitation is generalizability. Our findings were based on a single centre with only one resident and one specialist in Geriatric Medicine. While there was substantial agreement between these two, it is possible that agreement between other specialists in Geriatric Medicine would not be as strong. A second limitation is that we compared CFS ratings by physicians only. In an interprofessional setting such as ICU, nurses and additional allied health professionals also gather rich assessment information that could be used for frailty assessment, both independently and in collaboration with physicians. A third limitation is the time lag between the assessments. While the Geriatric Medicine resident and specialist completed comprehensive geriatric assessments and CFS assignments within a 72-hour window, the intensivists would often complete their CFS assessments within the first 12–48 hours from the time of ICU admission. It is possible that the Geriatric Medicine raters were using more contextual information available to them not only because of their different approach to information gathering (i.e., CGA), but also because additional time had transpired. Easier or more timely access by intensivists to family for collateral history, or information from the multidisciplinary team assessment may have modified their determination of frailty severity and resulted in better agreement.

The optimal timing for completion of a frailty assessment in a critically ill patient is unknown, prompting the need to explore the benefits of frailty assessment at different times during ICU course. Indeed, completing a more comprehensive assessment at the time of ICU discharge may allow for more robust data. However, deferring a frailty assessment to the end could forfeit the opportunity to establish a baseline frailty status and prognosis which would otherwise help inform early discussions of the care plan at the time of ICU admission.

Evaluation of frailty in a critically ill population holds significant promise to improve case-finding, anticipate outcomes, and tailor care strategies that align with the best interests and preferences of individual patients and their families. However, several challenges remain in measurement of frailty in ICU settings. First, there is a high reliance on proxies for frailty assessment, as critically ill patients are generally unable to communicate due to altered level of consciousness or sedation. Proxy ratings may be subject to recall bias, and do not necessarily correspond to the patient’s own report of their functional ability or quality of life prior to acute illness.( Secondly, clinicians may inadvertently ascribe features of acute illness to baseline frailty status, resulting in an overestimate of the burden of baseline frailty.( Different responses to these challenges may explain differences in CFS scoring by Geriatric Medicine and intensivists. For example, the assessment framework and cognitive biases of geriatricians may favour the use of collateral history and prior functional performance in forming an impression of frailty. Likewise, dynamic physiological parameters and past functional recovery in acute care may influence intensivists.

In our study, interrater reliability between intensivists and Geriatric Medicine of approximately 0.3 supports the idea that the two groups are indeed working from different constructs of frailty. Our data do support the role of the CFS in estimating prognosis, whether by intensivist or by Geriatric Medicine. While the use of frailty status has the potential to help frame the prognostic expectations of patients and families in relation to the ICU, our findings suggest the need to exercise caution in making inferences about qualitative aspects of frailty using a case-finding measure such as the CFS.

Despite these challenges, there are potential benefits to consider in routine case-finding. The proposed role of judgment-based tools such as the CFS in the acute care setting is to identify those who may be most vulnerable or at risk of adverse outcomes; however, a positive screen does not constitute a robust assessment of frailty.( We propose that case-finding could trigger tailored care pathways that include a comprehensive geriatric assessment to identify the distinct domains contributing to frailty. Such an assessment could inform goals of care discussions and reinforce realistic survivorship expectations of patients and families, or could be used by health systems to optimize resource and funding allocation decisions. For example, for those at risk of immobility or incident disability, robust data support early mobilization for reducing length of stay and improving functional outcomes in acutely ill patients.( This may also help to manage expectations for recovery and guide decision-making about treatment options, including choices about the duration and extent of ICU support.

Failing to recognize frailty has the potential to result in both unrealized enhancements in care and unmitigated harms. The current approach of managing disease entities in acute care using a single-illness lens is poorly designed for older adults living with frailty, multimorbidity, and functional limitations. The ultimate goal of frailty case-finding is to enhance care practices using CGA, then bridge towards a carefully considered individualized care plan. This, in turn, has the potential to support a shift towards a more sustainable health-care system.( Further, the recognition of frailty should inform decisions on procedures that introduce exaggerated harms. For example, coronary artery bypass surgery in frail patients causes death in 15% of patients, with another 50% becoming severely disabled postoperatively.( To accomplish these goals, instrument reliability is needed in addition to simple predictive validity.

Our study adds to our knowledge of the application of frailty measurement in ICU settings by demonstrating that the reliability of the CFS appears to be inconsistent, perhaps influenced by the operator, the timing of administration, and the information used when it is scored. Still, this stands in contrast to strong construct and predictive validity, regardless of these variables. These findings could be further enriched by determining the test characteristics of the CFS in relation to other ICU professionals and at other times in the care trajectory, such as at the time of ICU discharge.

CONCLUSION

The interrater reliability of the CFS between admitting intensivists and Geriatric Medicine showed minimal-to-moderate agreement, suggesting that geriatricians and intensivists utilize different cues and parameters when using a judgment-based frailty measure, and may have a different conception of how frailty manifests in acutely ill patients. The construct and predictive validity for in-hospital mortality of the CFS were similar for both groups of raters. Further research is needed to evaluate the reliability of frailty assessment tools in ICU settings, elucidate the optimal methods to complete a CFS in this population, and determine the essential elements and amount of time required for a frailty assessment to be reliably completed.