Recalibration and validation of the Charlson Comorbidity Index in acute kidney injury patients underwent continuous renal replacement therapy.

BACKGROUND: Comorbid conditions impact the survival of patients with severe acute kidney injury (AKI) who require continuous renal replacement therapy (CRRT). The weights assigned to comorbidities in predicting survival vary based on type of index, disease, and advances in management of comorbidities. We developed a modified Charlson Comorbidity Index (CCI) for use in patients with AKI requiring CRRT (mCCI-CRRT) and improved the accuracy of risk stratification for mortality.
METHODS: A total of 828 patients who received CRRT between 2008 and 2013, from three university hospital cohorts was included to develop the comorbidity score. The weights of the comorbidities were recalibrated using a Cox proportional hazards model adjusted for demographic and clinical information. The modified index was validated in a university hospital cohort (n = 919) using the data of patients treated from 2009 to 2015.
RESULTS: Weights for dementia, peptic ulcer disease, any tumor, and metastatic solid tumor were used to recalibrate the mCCI-CRRT. Use of these calibrated weights achieved a 35.4% (95% confidence interval [CI], 22.1%-48.1%) higher performance than unadjusted CCI in reclassification based on continuous net reclassification improvement in logistic regression adjusted for age and sex. After additionally adjusting for hemoglobin and albumin, consistent results were found in risk reclassification, which improved by 35.9% (95% CI, 23.3%-48.5%).
CONCLUSION: The mCCI-CRRT stratifies risk of mortality in AKI patients who require CRRT more accurately than does the original CCI, suggesting that it could serve as a preferred index for use in clinical practice.

Introduction

Despite advances in critical care medicine over the past decades, treatment outcomes for patients admitted to the intensive care unit (ICU) with acute kidney injury (AKI) remain poor [1,2]. Since continuous renal replacement therapy (CRRT), developed by Kramer in 1977, has become an indispensable treatment option for critical care, the mortality rate of severe AKI patients who require CRRT has remained very high over the past 40 years, likely due to the increased severity of underlying disease. Since the number and severity of underlying comorbid diseases are among the main causes of such an increase [1-3], predicting prognosis using an appropriate scoring system is important in treating critically ill AKI patients as well as for providing and allocating medical resources.

The Charlson Comorbidity Index (CCI) is the most widely used tool to evaluate the effects of comorbid diseases on patient prognosis, and its use has been validated in various acute/chronic conditions, including kidney diseases [4]. However, since the CCI was created originally as an index for general ward-admitted patients with various comorbid diseases, it is questionable whether each disease covered by the index exerts the same impact in severe AKI patients requiring CRRT. Since the index was developed 30 years ago, there is an inherent disadvantage that the effects of the diseases addressed might have changed with treatment technology. Recently, various tools such as the Davies index and the Index of Coexistent Disease score [5-7] have been created to evaluate the burden of coexistent chronic diseases. Nevertheless, the CCI remains the most widely used and clinician-friendly scoring system [4]. Although a series of studies has recalibrated and validated CCI for patients undergoing peritoneal dialysis, hemodialysis, and renal transplantation, research regarding CRRT patients is lacking [8-10]. Therefore, we recalibrated the weight of comorbidities on mortality in severe AKI patients requiring CRRT by modifying the original CCI (mCCI) and validated the performance of the new mCCI index compared with that of the original CCI.

Methods

Data collection in the development cohort

We collected data from 858 AKI patients aged ≥18 years who received CRRT from Seoul National University Hospital (n = 439), Seoul National University Boramae Medical Center (n = 218), or Dongguk University Ilsan Hospital (n = 201) between 2008 and 2013, as the development cohort. We excluded patients who had incomplete data regarding hemoglobin and albumin levels (n = 30). Therefore, the following demographic and clinical information were obtained for 828 AKI patients: sex, age, hemoglobin and albumin at baseline, 28-day mortality, and the 15 comorbidities constituting the CCI (peripheral vascular disease, dementia, myocardial infarction, congestive heart failure, cerebrovascular disease, hemiplegia, diabetes, diabetes with end organ damage, chronic pulmonary disease, connective tissue disease, peptic ulcer disease, any tumor, metastatic solid tumor, mild liver disease, and moderate to severe liver disease), defined based on the International Classification of Disease, the 10th Revision (ICD-10). Although there is a criterion for classifying moderate/severe renal disease in the original CCI appendix, scoring for underlying kidney disease was excluded. This is because our study population was composed of CRRT patients, who are those with severe AKI. These data were confirmed by trained clinicians at each medical center.

This study is in compliance with the Declaration of Helsinki and received full approval from the Institutional Review Boards of Seoul National University Hospital (No. H-1404-028-568), Seoul National University Boramae Medical Center (No. 26-2014-15), Yonsei University Severance Hospital (No. 4-2021-0082), and Dongguk University Ilsan Hospital (No. DUIH 2018-12-010-012). Informed consent was waived for this study because retrospective data were used.

Data collection in the validation cohort

A total of 1,144 patients who received CRRT at Yonsei University Severance Hospital participated in the validation cohort between 2009 and 2015. We excluded 225 patients with missing information regarding albumin (n = 216) and comorbidities (n = 9). Finally, information on age, sex, albumin, hemoglobin, and CCI score of 919 patients was collected.

Statistical analyses

Baseline characteristics in the development and validation cohorts were described using mean and standard deviation for continuous variables and frequency and percentile for categorical variables. The primary outcome of this study was 28-day all-cause mortality, which was observed in each hospitalization. To estimate the calibrated weights of 15 comorbidities in AKI patients, Cox proportional hazard models were used after being stratified by age (<50, 50–59, and ≥60 years) and treatment center in the development cohort and adjusted for sex, albumin, hemoglobin, and the 15 comorbidities. We tested the proportional hazard assumption using the Schoenfeld test. After estimating the hazard ratios (HRs) of the comorbidities in the survival model, the modified CCI of CRRT patients (mCCI-CRRT) was calculated using each significant HR among the CCI diseases divided by the lowest significant HR. The comorbidity score of the mCCI-CRRT for each patient was the sum of the rounded weights. We obtained Kaplan-Meier curves stratified into three categories (<2, 2–4, ≥5) for the original CCI and the mCCI-CRRT scores in the development and validation cohorts to compare the performances. The index discriminations between CCI and mCCI-CRRT were assessed by C-statistics and continuous net reclassification improvement (cNRI). The C-statistic was the area under the receiver operator curve between sensitivity and 1-specificity. To overcome the disadvantages of C-statistics, such as difficulty in clinical interpretation and unclear decision threshold, cNRI, which is the sum of the proportion in event prediction of the event group (cNRIevent) and the non-event prediction of the non-event group (cNRInon-event) was used [11]. Improvement in the mortality prediction of mCCI-CRRT compared to the age-adjusted and original CCI was estimated using a logistic regression model adjusted for sex and age in model 1 and additionally adjusted for hemoglobin and albumin in model 2. All analyses were conducted using R software, version 4.0.2 (R Foundation for Statistical Computing, Vienna, Austria) and p < 0.05 was considered statistically significant.

Results

Baseline characteristics of both the development and validation cohorts are presented in Table 1. In the development cohort, most participants were in the ≥60-year age group at initiation of CRRT therapy, and 61.0% of patients were male. Similar distributions were observed regarding age and sex in the validation cohort. The average follow-up duration and 28-day mortality rate during hospitalization were 14.45 ± 11.96 days and 61.5% (n = 509) in the development cohort, respectively, and 18.92 ± 11.25 days and 58.8% (n = 540) in the validation cohort. In the development cohort, most subjects (82.6%) had one or more comorbidities. Among the 15 comorbidities, tumor was the most prevalent (24.4%), followed by diabetes without end organ damage (19.6%), moderate to severe liver disease (18.1%), and congestive heart failure (18.0). Additionally, 34.5% of the patients in the validation cohort had diabetes without end organ damage, followed by any tumor (31.9%), diabetes with end organ damage (24.4%), and congestive heart disease (16.5%).

Table 1.

Descriptive characteristics in the development and validation cohorts

Variable	Development cohort	Validation cohort
No. of subjects	828	919
Age (yr)
<50	111 (13.4)	162 (17.6)
50–59	132 (15.9)	172 (18.7)
≥60	585 (70.7)	585 (63.7)
Sex
Male	505 (61.0)	567 (61.7)
Female	323 (39.0)	352 (38.3)
28-Day mortality	509 (61.5)	540 (58.8)
Follow-up duration (day)	14.45 ± 11.96	18.92 ± 11.25
Hemoglobin (g/dL)	9.86 ± 2.25	9.66 ± 2.26
Albumin (g/dL)	2.68 ± 0.55	2.71 ± 0.52
Comorbidity
No comorbidity	144 (17.4)	140 (15.2)
Peripheral vascular disease	23 (2.8)	36 (3.9)
Dementia	32 (3.9)	33 (3.6)
Myocardial infarction	98 (11.8)	86 (9.4)
Congestive heart failure	149 (18.0)	152 (16.5)
Cerebrovascular disease	74 (8.9)	91 (9.9)
Hemiplegia	44 (5.3)	20 (2.2)
Diabetes, without end organ damage	162 (19.6)	317 (34.5)
Diabetes, with end organ damage	134 (16.2)	224 (24.4)
Chronic pulmonary disease	31 (3.7)	66 (7.2)
Connective tissue disease	26 (3.1)	22 (2.4)
Peptic ulcer disease	49 (5.9)	105 (11.4)
Any tumor	202 (24.4)	293 (31.9)
Metastatic solid tumor	25 (3.0)	89 (9.7)
Mild liver disease	69 (8.3)	119 (12.9)
Moderate to severe liver disease	150 (18.1)	76 (8.3)
Charlson Comorbidity Index score	3.4 ± 2.35	3.16 ± 2.28
APACHE II score	28.16 ± 7.69	27.47 ± 7.65
SOFA score	12.83 ± 3.41	13.33 ± 3.59

Data are expressed as number only, number (%), or mean ± standard deviation.

APACHE II, Acute Physiology And Chronic Health Evaluation II; SOFA, sequential organ failure assessment.

Table 2 lists the β coefficients, adjusted HR, and weights for each comorbidity in the Cox proportional hazard model. We confirmed a relationship between time and residuals in proportional hazard assumption using the Schoenfeld test (p = 0.76). The original CCI calculated the adjusted relative risk of each item for 1-year mortality. A relative risk of <1.2 was excluded from the weight calculation. Relative risk from 1.2 to <1.5 was set as a weight of 1, from 1.5 to <2.5 as a weight of 2, from 2.5 to <3.5 as a weight of 3, and relative risk ≥3.5 was set as a weight of 6 [4]. Unlike the original CCI, weighting of the mCCI-CRRT was derived by dividing the significant HR of each item by the significant smallest adjusted HR (peptic ulcer disease). Metastatic solid tumor scored 5 points and was the strongest predictor of mortality among the comorbidities, followed by a score of 3 for any tumor (including leukemia and lymphoma) and a score of 1 for dementia and peptic ulcer disease.

Table 2.

Weights of mCCI-CRRT based on Cox proportional hazard model compared with original CCI

Variable	Coef	SE	HR	p-value	CCI	mCCI-CRRT
Female sex	0.07	0.09	1.08	0.43
Albumin	–0.44	0.09	0.65	0.001
Hemoglobin	0.03	0.02	1.03	0.22
Peripheral vascular disease	–0.14	0.29	0.87	0.62	1	0
Dementia	–0.58	0.27	0.56	0.03	1	1
Myocardial infarction	–0.06	0.17	0.94	0.72	1	0
Congestive heart failure	–0.16	0.15	0.85	0.29	1	0
Cerebrovascular disease	–0.17	0.16	0.84	0.28	1	0
Hemiplegia	–0.46	0.24	0.63	0.06	2	0
Diabetes, without end organ damage	–0.12	0.12	0.89	0.31	1	0
Diabetes, with end organ damage	–0.19	0.13	0.83	0.17	2	0
Chronic pulmonary disease	0.26	0.2	1.30	0.20	1	0
Connective tissue disease	0.14	0.25	1.15	0.57	1	0
Peptic ulcer disease	–0.68	0.22	0.51	0.001	1	1
Any tumor	0.40	0.11	1.50	0.001	2	3
Metastatic solid tumor	0.87	0.24	2.38	0.001	6	5
Mild liver disease	0.00	0.17	1.00	0.99	1	0
Moderate to severe liver disease	0.14	0.12	1.15	0.24	3	0

CCI, Charlson Comorbidity Index; Coef, coefficient; HR, hazard ratio; mCCI-CRRT, modified CCI-continuous renal replacement therapy; SE, standard error.

The total scores of the original and recalibrated weights were allocated to each patient in the development cohort. The median CCI and mCCI-CRRT in the development cohort were 2 and 0, respectively (Fig. 1). We categorized the CCI and mCCI-CRRT into three score categories using the same cutoff values (<2, 2–4, ≥5) to compare the probability of survival using CCI and mCCI-CRRT. Fig. 2 illustrates the Kaplan-Meier curves for the CCI (Fig. 2A, C) and mCCI-CRRT (Fig. B, D) differentiated by the three risk groups in both cohorts. When the survival probability was classified by CCI in the development (Fig. 2A) and validation (Fig. 2C) cohorts, survival curves for the ≥5 and 2 to 4 groups intersected and overlapped, while group differences based on the log-rank test were significant in the validation cohort. However, survival curves of mCCI-CRRT (Fig. 2B) showed differences in survival probability according to the classification of risk scores in the development cohort (p < 0.01). In addition, survival probability analysis showed a decrease in mCCI-CRRT (Fig. 2D) as risk increased, with significant between-group differences in the validation cohort.

Figure 1.

Distribution of the CCI (A) and mCCI-CRRT (B) in the development cohort.

CCI, Charlson Comorbidity Index; mCCI-CRRT, modified CCI-continuous renal replacement therapy.

Figure 2.

Kaplan-Meier curve of CCI and mCCI-CRRT score groups in development (A, B) and validation (C, D) cohorts.

CCI, Charlson Comorbidity Index; mCCI-CRRT, modified CCI-continuous renal replacement therapy.

We estimated the mortality prediction of mCCI-CRRT compared to those of CCI and age-adjusted CCI in the validation cohort using C-statistics (Table 3). Significant difference was observed in the comparison between CCI (0.54; 95% confidence interval [CI], 0.50–0.58) and mCCI-CRRT (0.59; 95% CI, 0.55–0.62) (p = 0.01). However, the C-statistics difference in the age-adjusted CCI (0.58; 95% CI, 0.54–0.61) compared with that of the mCCI-CRRT was not significant (p = 0.60). Our calibrated weights in the mCCI-CRRT achieved a 35.4% (95% CI, 22.1%–48.1%) higher performance than did CCI in reclassification, based on cNRI in logistic regression adjusted for age and sex. After additionally adjusting for hemoglobin and albumin, consistent results were found in risk reclassification improvement by 35.9% (95% CI, 23.3%–48.5%). In comparison with age-adjusted CCI, mCCI-CRRT significantly improved the net risk reclassification of mortality by 22.9% (95% CI, 9.9%–35.8%) in model 1 and by 15.2% (95% CI, 2.1%–28.2%) in model 2.

Table 3.

Model performance of the mCCI-CRRT in the validation cohort

Variable	C-statistics	cNRI[a]	cNRI[b]
CCI vs. mCCI-CRRT
CCI	0.54 (0.50–0.58)	35.4 (22.1–48.1)	35.9 (23.3–48.5)
mCCI-CRRT	0.59 (0.55–0.62)
p-value	0.01	0.001	0.001
Age-adjusted CCI vs. mCCI-CRRT
Age-adjusted CCI	0.58 (0.54–0.61)	22.9 (9.9–35.8)	15.2 (2.1–28.2)
mCCI-CRRT	0.59 (0.55–0.62)
p-value	0.6	0.001	0.02

CCI, Charlson Comorbidity Index; cNRI, continuous net reclassification improvement; mCCI-CRRT, modified CCI-continuous renal replacement therapy.

Adjusted for age and sex;

adjusted for age, sex, hemoglobin, and albumin.

Discussion

In this study, we modified the original CCI using the AKI-CRRT cohort database of three hospitals in Korea to provide better risk stratification in AKI patients requiring CRRT. Furthermore, we validated the accuracy of the modified index by comparing its performance with that of the original index using data of independent hospitals for the development cohort. We found that mCCI-CRRT exhibited better risk stratification performance for mortality in incident AKI-CRRT patients compared with that provided by the original CCI.

In patients with severe AKI requiring CRRT, the effect of comorbidities on outcome is considerable and warrants development of a comorbidity index [12,13]. However, for the original CCI, the scoring system was created with a development cohort of patients admitted to a general ward with various disease entities, while validation was performed on breast cancer patients [4]. Therefore, it might not be appropriate to apply the original CCI in AKI patients requiring CRRT. To our knowledge, there have been few attempts to recalibrate the CCI of patients requiring CRRT. Therefore, this study is meaningful in that it used a large sample of incident AKI-CRRT patients and validated the index using an independent observational cohort.

The disease weights in the original CCI showed many changes when using the mCCI-CRRT. When considering only HR without statistical significance, the more prevalent causes of septic AKI (metastatic solid tumor, any tumor, connective tissue disease, chronic pulmonary disease, and moderate to severe liver disease) showed higher HR values. In contrast, HRs tended to be <1.0 in common causes of ischemic AKI, hypovolemic AKI, and postoperative AKI (peptic ulcer disease, dementia, hemiplegia, cerebrovascular disease, congestive heart failure, and myocardial infarction) [14].

Metastatic solid tumors show the highest weight, even in mCCI-CRRT [15-18]. Although target therapy and immunotherapy, which did not exist in the 1980s when the original CCI was developed, are used widely and new anticancer drugs are being developed, the mortality and risk of sepsis and septic AKI in cancer patients remain high. This highest weight also might be due to cancer treatment-related immune dysfunction and secondary infections [19,20].

Although the difference was not statistically significant, the weight of connective disease in mCCI-CRRT increased from 1 to 2 compared with the original CCI. Recently, biologics such as tumor necrosis factor-α or interleukin-6 blocking monoclonal antibodies, which did not exist 25 years ago, have been used for various autoimmune diseases and connective tissue disorders [21]. The disturbance of the immune system following the use of these biologics is believed to cause sepsis more frequently and is more difficult to treat in patients with connective tissue disease [22,23]. On the other hand, the weights of diabetes and hemiplegia were lower in the mCCI-CRRT group than in the original CCI group. Since CCI was first developed in 1977, the guidelines for diabetes control have become more sophisticated, and the target serum glucose level has been lowered, which might decrease the risk of mortality due to diabetes. In this study, peptic ulcer disease showed a statistically significant hazard ratio. Previous studies have shown that the incidence of stress-induced ulcers was high in critically ill patients receiving ICU care [24-26]. In one observational retrospective study, 90-day mortality was significantly higher in a bleeding group among patients admitted to the ICU, but there was no difference in mortality at <28 days compared to the non-bleeding group [24]. In another study, critically important gastrointestinal (GI) bleeding was not significantly related to 90-day mortality in patients who received ICU care for more than 7 days, and 90-day mortality was increased by comorbidities such as liver disease, renal replacement therapy, and coagulopathy [27]. Finally, based on a recent randomized trial comparing proton pump inhibitor and placebo groups in critically ill patients admitted to the ICU, 90-day mortality and the number of clinically important events did not show significant difference between the two groups [28]. These studies show that GI bleeding had relatively little influence on mortality in critically ill patients compared to other comorbidities, which is consistent with the results of this study.

In both the original CCI and mCCI-CRRT, there was no statistically significant difference in mortality between men and women, as shown in Table 2. Previous studies have found similar lack of association. Zettersten et al. [29] reported no difference in mortality between men and women in a retrospective cohort analysis of approximately 9,000 patients admitted to a Swedish university hospital ICU between 2006 and 2016. Sakr et al. [30] reported that the overall ICU mortality rate did not differ significantly between males and females in 3,902 patients admitted to one of 24 participating medical and/or surgical ICUs between April 2006 and September 2006.

Our study has some limitations. First, risk stratification for underlying chronic diseases was performed without considering the differences in acute antecedent factors such as myocardial infarction, bleeding, or sepsis that cause AKI. In this situation, the acute medical condition requiring CRRT is severe, and the contribution of underlying chronic diseases can be underestimated. Second, this dataset only included serum hemoglobin and albumin levels among many laboratory measures. Thus, we could not adjust for other variables such as baseline serum creatinine, quantitative proteinuria, or prothrombin time for recalibrating the weights of comorbidities. In addition, since there were no patients with acquired immunodeficiency syndrome in either cohort, this condition was not taken into consideration. Third, this is a scoring system created from a cohort composed of a single racial group, and it might not be generalizable or applicable to other racial groups and populations. Additional validation using disease-specific cohorts and other national data is necessary. Fourth, since the underlying comorbidity information is based on the ICD-10 code registered in the diagnosis window of electronic health records, specific relative risk of mortality could be underestimated. Finally, since information on CRRT dose, which is an important factor for CRRT patients, was not collected, there might be a difference in CRRT dose between the development and validation cohorts. However, since the sample size of each cohort was large and the same CRRT protocol was used, a large difference in CRRT dose between the two cohorts was less likely. In addition, our study focused on pre-existing conditions, i.e., underlying comorbidities prior to CRRT initiation. Therefore, we did not consider post-CRRT factors.

A major strength of our study is that the new index was created based on a relatively large cohort from a multi-center ICU, and it was validated with an independent, similarly sized cohort, which is quite large in terms of CRRT-related studies. Lastly, the mCCI-CRRT demonstrated superior performance in a head-to-head comparison with the original CCI.

In conclusion, we suggest use of the new mCCI-CRRT scoring system that offers superior risk stratification for mortality in incident AKI-CRRT patients compared with the original CCI. In addition, this could be a preferred scoring system in clinical practice and statistical analysis in epidemiological research. As treatment options for various diseases develop, the effects of individual chronic diseases on mortality will change. Therefore, this edition of the mCCI-CRRT is not final, and it will require periodic updates to remain current and fit the trends over time.