Nomogram to predict the risk of septic acute kidney injury in the first 24 h of admission: an analysis of intensive care unit data.

Background: Acute kidney injury (AKI) is a significant cause of morbidity and mortality, especially in sepsis patients. Early prediction of AKI can help physicians determine the appropriate intervention, and thus, improve the outcome. This study aimed to develop a nomogram to predict the risk of AKI in sepsis patients (S-AKI) in the initial 24 h following admission.
Methods: Sepsis patients with AKI who met the Sepsis 3.0 criteria and Kidney Disease: Improving Global Outcomes criteria in the Massachusetts Institute of Technology critical care database, Medical Information Mart for Intensive Care (MIMIC-III), were identified for analysis. Data were analyzed using multiple logistic regression, and the performance of the proposed nomogram was evaluated based on Harrell's concordance index (C-index) and the area under the receiver operating characteristic curve.
Results: We included 2917 patients in the analysis; 1167 of 2042 patients (57.14%) and 469 of 875 patients (53.6%) had AKI in the training and validation cohorts, respectively. The predictive factors identified by multivariate logistic regression were blood urea nitrogen level, infusion volume, lactate level, weight, blood chloride level, body temperature, and age. With the incorporation of these factors, our model had well-fitted calibration curves and achieved good C-indexes of 0.80 [95% confidence interval (CI): 0.78-0.82] and 0.79 (95% CI: 0.76-0.82) in predicting S-AKI in the training and validation cohorts, respectively.
Conclusion: The proposed nomogram effectively predicted AKI risk in sepsis patients admitted to the intensive care unit in the first 24 h.

Background

Acute kidney injury (AKI) is a common and economically significant disease. It affects approximately 5–7% of hospitalized patients [1,2], particularly in the intensive care unit (ICU) where it is reported in 20–50% of patients [3-5]. Sepsis is a major devastating disease in the ICU. The outcomes for ICU admissions owing to the two syndromes are significant, with the mortality rates for septic acute kidney injury (S-AKI) ranging from 22.0 to 70.2% [6-8].

Currently, AKI is diagnosed based on elevation of serum creatinine levels or a decline in urine output according to the Kidney Disease: Improving Global Outcomes (KDIGO) guidelines [9]. In February 2016, JAMA reported the newest Sepsis-3 criteria for clinical care, which highlighted the strong association between infection and organ failure [10]. Early diagnosis and intervention delay progression of AKI to severe stages and development of chronic kidney disease (CKD) after hospital discharge. For these reasons, the prediction of risk is worthy of investigation [11].

Many prediction efforts have been made over the past decade, most of which focused on blood- or urine-based injury or novel biomarkers [12-14]. Unfortunately, these markers are not only rarely available but often lack sensitivity or specificity. With advances in machine learning techniques, other investigators have built models that focus on other specific patient subsets [15-17].

Because of the current inability to predict S-AKI in ICU settings, the development of a predictive model that considers relevant factors associated with S-AKI based on electronic health records appears to be essential. Of the available models, a nomogram can provide an individualized, evidence-based, highly accurate risk estimation. Nomograms are easy to use and can facilitate management-related decision making. To our knowledge, there are no previous nomograms for predicting AKI risk estimation in patients with sepsis.

Methods

Source of data

An openly available US-based critical care database named the Medical Information Mart for Intensive Care (MIMIC)-III v 1.4 was analyzed [18]. MIMIC-III is a large, integrated, de-identified, comprehensive clinical dataset that includes data from all patients admitted to the ICUs of Beth Israel Deaconess Medical Center in Boston, MA from June 1, 2001 to October 31, 2012. Three hundred and eighty laboratory measurements and hourly records made by ICU nurses were recorded. The database also includes data on basic information, laboratory results, imaging examinations, diagnoses, and others. When patients were discharged, International Classification of Diseases, 9th revision (ICD-9) were entered for precise diseases by the hospital staff. Because this study was an analysis of a third-party anonymized publicly available database with preexisting institutional review board approval, IRB approval from our institution was waived. The study was conducted according to the recommendations of the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [19].

Participants

The inclusion criteria were as follows: (1) Sepsis-3 criteria [10] (suspected infection, sequential organ failure assessment (SOFA) scores calculated using data from the first 24 h of the ICU stay); (2) KDIGO criteria [9] (serum creatinine ≥ 26.4 μmol/L or increase by ≥ 1.5-fold to two-fold from baseline, > two-fold to three-fold from baseline, and > three-fold from baseline; or serum creatinine ≥ 354 μmol/L with an acute increase of at least 44 μmol/L. Individuals who received continuous renal replacement therapy (CRRT) were considered to have met the criteria of AKI III regardless of their serum creatinine value. Baseline serum creatinine level was evaluated according to KDIGO standards for all patients); and (3) ICU admission from 2008 to 2012, because antibiotic prescriptions were only recorded from 2003 onwards [20]. We excluded patients with CKD, who were less than 18 years of age, or those who had incomplete clinical data. Variables with >20% missing values were excluded from the model building process. Eligible patients were selected randomly from 70% of the sample for the training set, and the data were independently validated using the remaining 30% of patients. The training group was used to train the model weight parameters, and the validation group was used to verify the accuracy of the model in predicting AKI in sepsis patients. Finally, we excluded several items with extreme values and considered a range of 0.5–99.5%.

Clinicopathologic variables

The clinicopathologic variables in the study are displayed in Table 1. The basic data included covariates, comorbidities, vital signs, interventions, and laboratory results. Age, sex, weight, height, and ethnicity were collected from the original dataset. Respiratory rate, temperature, and heart rate at ICU admission were collected from the CHARTENENTS in MIMIC_III. Laboratory parameters, including white blood cell (WBC) count, hemoglobin level, platelet count, and blood urea nitrogen (BUN), potassium, bicarbonate, chloride, and lactate levels were recorded in the LABENENTS in MIMIC-III. All data were recorded in the first 24 h following admission. SOFA scores were calculated using data from the initial 24 h of the ICU stay. All comorbidities were identified on the basis of the recorded International Classification of Diseases, 9th revision (ICD-9) codes. According the Bayesian Information Criterion (BIC) principle, six factors were selected to build the model. Bayesian Information Criterion (BIC) principle formula is: is the maximized value of likelihood function of the model M, i.e. = p(x |M), where are the parameter values that maximize the likelihood function, and x is the observed data; n means the number of data points in x, the number of observations, or equivalently, the sample size; k is the number of parameters estimated by the model. For example, in multiple linear regression, the estimated parameters are the intercept, the q slope parameters, and the constant variance of the errors, thus, k is q add 2.

Table 1.

Baseline demographic, intervention and laboratory characteristics of patients.

	Cohort
Variable	Training (n = 2042)	Validation (n = 875)	p Value
Demographic variables
Age (years)	63.01 (18.06)	63.92 (18.03)	0.22
Gender male (n)	1130	510	0.14
weight (mean (SD))	81.71 (22.1)	81.18 (22.16)	0.35
Elixhauser score	3.14 (6.96)	3.16 (6.9)	0.94
Sofa	5.44 (3.26)	5.41 (3.09)	0.80
Ethnicity
White (n)	1483	620	0.33
Black (n)	135	69	0.22
Hispanic (n)	69	33	0.60
Other race (n)	355	153	0.95
Comorbidities
Congestive heart failure (n)	410	157	0.182
Atrial fibrillation (n)	459	221	0.104
Liver disease (n)	201	82	0.693
Chronic obstructive pulmonary disease (n)	266	117	0.8
Coronary artery disease (n)	438	172	0.275
Stroke (n)	195	70	0.182
Malignancy tumor (n)	449	189	0.816
Cardiac arrhythmias (n)	403	201	0.048
Valvular disease (n)	111	43	0.564
Pulmonary circulation (n)	109	44	0.731
Peripheral vascular (n)	150	53	0.21
Paralysis (n)	78	35	0.817
Diabetes with uncomplicated (n)	397	189	0.183
Diabetes complicated (n)	83	25	0.114
Hypothyroidism (n)	212	97	0.572
Coagulopathy (n)	371	157	0.885
Obesity (n)	143	68	0.463
Anemias (n)	438	209	0.147
Psychoses (n)	141	44	0.057
Depression (n)	277	124	0.663
Interventions
Sedative use (1st 24 h (n))	1174	490	0.456
Vasopressor use (n)	8.14 (18.07)	8.93 (20.53)	0.741
Mechanical ventilation use time (h)	40.75 (82.7)	33.51 (68.96)	0.114
Infusion volume(1st 24h (ml))	4574.52 (3285.75)	4597.09 (3257.39)	0.671
Vital signs
Heart rate	108.21 (19.33)	107.51 (19.27)	0.37
Breathing rate	27.95 (6.2)	27.99 (5.99)	0.87
temperature (°C)	37.61 (0.89)	37.65 (0.86)	0.19
Laboratory test
Hemoglobin	12.21 (2.07)	12.08 (2.13)	0.13
Potassium	4.65 (0.9)	4.62 (0.91)	0.39
Bicarbonate	24.65 (4.25)	24.65 (4.23)	1.00
BUN	26.84 (19.44)	26.28 (18.43)	0.46
WBC	15.16 (7.6)	14.63 (7.6)	0.062
Platelet	247.56 (137.24)	245.45 (124.9)	0.983
Sodium	140.42 (5.01)	140.07 (5.01)	0.09
Chloride	108.09 (6.13)	107.8 (6.18)	0.25
Lactate	2.96 (2.23)	2.91 (2.13)	0.51
INR	1.58 (0.89)	1.57 (0.83)	0.28
Hematocrit	36.58 (6.01)	36.17 (6.18)	0.10

ALT: alanine aminotransferase; AST: Aspartate aminotransferase; BUN: blood urea nitrogen; CVP: central venous pressure; CK: creatine kinase; INR: international normalized ratio; Sofa: Sequential organ failure assessment; WBC: White blood cell count.

Statistical analysis

Continuous clinical data are expressed as means and were compared using the independent Student t-test or rank-sum test, as appropriate. Categorical variables were compared using the Chi-square or Fisher exact test. The significance of potential risk factors in the training cohort was assessed using univariate logistic regression analysis. A stepwise logistic regression model was used to select significant variables associated with AKI. A nomogram was developed based on the results of the multivariate logistic regression analysis using the rms package of R, version 3.6 (http://www.r-project.org/). The predictive performance of the nomogram was evaluated using Harrell’s concordance index (C-index) and calibration with 1000 bootstrap samples was performed to decrease the overfit bias. Accuracy of the optimal cutoff value was assessed using sensitivity, specificity, positive and negative predictive values, and likelihood ratios.

Statistical significance in all analyses was set at the 0.05 level. All analyses were performed using R, version 3.6. The outcomes of our study are completely reproducible, and the data are available online from the MIMIC-III database.

Results

After applying our selection criteria, 2917 of 23,620 ICU admissions met the Sepsis 3.0 criteria, with 2042 and 875 patients with and without AKI, respectively. The differences in the characteristics of the patients between the training and validation sets are shown in Table 1. The baseline clinicopathologic data were similar between the training set and validation set. According to the KDIGO criteria, AKI was detected in 1167 (1167/2042) and 496 (496/875) patients in the training and validation sets, respectively.

Development and validation of an S-AKI predicting nomogram

All variables shown in Table 1 were used in the univariate logistic analysis (Table 2). Multivariate logistic regression was performed using variables with p < 0.05 in the univariate logistic analysis and those with not more than 20% missing values. We selected another six factors based on the Bayesian Information Criterion (BIC) principle in the ‘bestglm’ package in R; however, considering its clinical significance, age was also taken into account [21,22]. On multivariate analysis, with results reported as odds ratio (95% CI), BUN level (mg/dL) (1.08 [1.07–1.09]), infusion volume (ml) (1.00 [1.00–1.00]), lactate level (mmol/L) (1.18 [1.12–1.26]), weight (kg) (1.01 [1.01–1.02]), chloride level (mEq/L) (1.04 [1.02–1.06]), temperature (°C) (0.83 [0.73-0.94]), and age(year) (1.01 [1.00–1.01]) were independently associated with the occurrence of S-AKI (Table 3). Because they were calculated based on basic units instead of 10-year and L, the ORs for age and infusion volume were relatively small.

Table 2.

Univariate logistic regression analysis of S-AKI based on first 24 h data in the training set.

Characteristics	OR	95% CI	p Value
Demographic variables
Age	1.01	1.01–1.02	<0.001
Gender male	0.88	0.74–1.05	0.17
Weight	1.01	1.01–1.01	<0.001
Elixhauser score	1.03	1.01–1.04	<0.001
Sofa	1.27	1.23–1.32	<0.001
Ethnicity
White	1.13	0.93–1.38	0.21
Black	1.17	0.82–1.67	0.39
Hispanic	0.50	0.30–0.82	0.01
other race	0.92	0.72–1.16	0.46
Comorbidities
Congestive heart failure	1.34	1.07–1.68	0.01
Atrial fibrillation	1.50	1.21–1.86	<0.001
Liver disease	1.58	1.16–2.17	<0.01
Chronic obstructive pulmonary disease	1.05	0.81-1.37	0.70
Coronary artery disease	0.86	0.70–1.07	0.17
Stroke	0.61	0.45-0.82	<0.01
Malignancy tumor	0.92	0.74–1.13	0.42
Cardiac arrhythmias	1.61	1.28-2.02	<0.001
Valvular disease	0.92	0.62–1.37	0.67
Pulmonary circulation	1.51	1.01–2.27	0.05
Peripheral vascular	0.95	0.66-1.35	0.76
Paralysis	0.59	0.37–0.93	0.02
Diabetes with uncomplicated	1.38	1.11–1.73	<0.01
Diabetes complicated	1.76	1.11–2.89	0.02
Hypothyroidism	1.31	0.99–1.74	0.06
Coagulopathy	1.55	1.22-1.96	<0.001
Obesity	1.85	1.31–2.64	<0.01
Anemias	1.13	0.92–1.41	0.25
Psychoses	1.06	0.75–1.51	0.74
Depression	1.20	0.93–1.55	0.17
Interventions
Sedative use	0.95	0.80–1.13	0.57
Vasopressor use	1.47	1.29–1.69	<0.001
Mechanical ventilation use time	1.00	1.00–1.00	<0.01
Infusion volume	1.00	1.00–1.00	<0.001
Vital signs
Heart rate	1.00	1.00–1.01	0.13
Breathing rate	1.03	1.01–1.04	<0.001
temperature (°C)	0.79	0.71–0.88	<0.001
Laboratory test
Hemoglobin	0.99	0.95–1.03	0.561
Potassium	1.58	1.42–1.78	<0.001
Bicarbonate	0.93	0.91–0.95	<0.001
BUN	1.09	1.08–1.10	<0.001
WBC	1.03	1.02–1.05	<0.001
Platelet	1.00	1.00–1.00	0.31
Sodium	1.02	1.00–1.04	0.03
Chloride	1.03	1.02–1.05	<0.001
Lactate	1.25	1.19–1.31	<0.001
INR	1.46	1.28–1.68	<0.001
Hematocrit	1.00	0.99–1.02	0.57

CI: confidence interval; OR: odds rate.

Table 3.

Multivariate logistic regression analysis of S-AKI based on first 24 h data in the training set.

Variable	OR	95% CI	p Value
Infusion volume (ml)	1.00	1.00–1.00	<0.001
BUN (mg/dL)	1.08	1.07–1.09	<0.001
Lactate (mmol/L)	1.18	1.12–1.26	<0.001
Weight (kg)	1.01	1.01–1.02	<0.001
Temperature (°C)	0.83	0.73–0.94	<0.01
Chloride (mEq)	1.04	1.02–1.06	<0.001
Age (year)	1.01	1.00–1.01	<0.01

CI: confidence interval; OR: odds rate.

These factors were then used to develop an S-AKI risk estimation nomogram (Figure 1) as continuous variables, meaning that the parameters were not categorized as normal, low, or high. The resulting model was internally validated using the bootstrap validation method. The nomogram demonstrated good accuracy for estimating the risk of S-AKI in the first 24 h, with an unadjusted C-index of 0.80 (95% CI: 0.78–0.82) (Figure 2(A)); the C-index for the estimation of S-AKI risk in the validation set was 0.79 (95% CI: 0.76–0.82) (Figure 2(B)).

Figure 1.

Flow chart of patient selection.

Figure 2.

(A) Calibration of the training set. Evaluation of the predictive performance for estimating the risk of S-AKI of the nomogram in the training cohort (n = 2042); (B) Calibration of the validation set. Evaluation of the predictive performance for estimating the risk of S-AKI of the nomogram in the validation set (n = 875). C-index: concordance index.

Risk of S-AKI based on the nomogram scores

Our model performed well with sensitivity, specificity, positive predictive value, and negative predictive value for differentiating between the presence and absence of S-AKI of 73.86, 68.46, 75.75, and 65.54% in the training set, and 71.98, 69.39, 75.48, and 65.42% in the validation cohort, respectively. The AUC, used for evaluation of the accuracy of the nomogram in the prediction of AKI in septic patients, was 0.80 (95% CI, 0.78–0.82) in the training set and 0.79 (95% CI, 0.76–0.82) in validation set (Figure 3).

Figure 3.

Receiver operating characteristic curve of the nomogram. Receiver operating characteristic curve for predicting AKI within 24 h of admission to the intensive care unit in sepsis patients. AUC = area under the receiver operating characteristic curve. The AUC of the nomogram for the prediction of AKI in septic patients was 0.80 [95% confidence interval (CI) 0.78–0.82] in the training set and 0.79 (95% CI, 0.76–0.82) in the validation set.

Discussion

The first hours in the ICU are considered the critical ‘golden hours’ because appropriate decision-making during this period about treatment and examination, such as fluid and hemodynamic management and targeted contrast-enhanced ultrasound, and drugs causing nephrotoxicity may alter the risk of AKI [23,24].

In this study, we found that AKI according to the KDIGO criteria was diagnosed in sepsis patients on the first day of admission 8–10 times more often than that in hospitalized patients. Further, the incidence of S-AKI in the ICU was significantly high. Moreover, we developed and validated a risk prediction nomogram, which was shown to have good performance for predicting AKI, using routine laboratory parameters and vital signs. We found that BUN level, infusion volume, lactate level, body weight, blood chloride level, body temperature, and age were significantly associated with AKI risk. Overall, our results demonstrated that the combination of these seven indicators may be used for more sensitive and specific diagnostic tests and requires further study.

AKI prediction models reported in previous studies focused on subsets of patients such as those with contrast-induced AKI [17], those who underwent cardiac surgery [16,25] and those who received living donor liver transplants [26]. However, to our knowledge, our nomogram is the first to be developed for AKI prediction in sepsis patients. Another study [27] reported a recurrent neural network (RNN) model that demonstrated extraordinary practical value; however, the authors did not publicly release the code and the model is not available. Furthermore, the application of such a model would require specific computer software with programming ability, and it cannot be used on portable devices using basic clinical data, thereby limiting its application. Among currently available prediction tools, a nomogram can provide highly accurate information that can be generated simply and expressed graphically.

Our nomogram performed well as indicted by the C-index values. The bootstrapped calibration curves demonstrate the difference between a prediction and observation. Moreover, because the data for this model were derived from septic patients with a baseline probability of developing AKI, the values of these indicators although within the normal range may contribute to morbidity due to AKI.

For example, consider a 56-year-old man with sepsis and severe pneumonia who had received a transfusion. The patient’s volume of infusion was 4000 mL, BUN level 40 mg/dL, lactate level 2 mmol/L, weight 60 kg, chloride level 115 mEq/L, and last measured temperature 37.9 °C. According to Figure 4, the score corresponding to each individual parameter in this case should be based on the first row. For example for age, the score will be approximately 3 based on an age of 56 years; the final score is calculated as the sum of points for all parameters (3(age) + 3(infusion) + 31(BUN) + 3(lactate) + 8(weight) + 5(temperature) + 14(chloride) = 67). This score corresponds to a risk of developing AKI in 24 h of approximately 73%. If data for one or two indicators are missing for a patient, the sensitivity of the model may decline, with inaccurate scores leading to underestimation of the risk.

Figure 4.

Nomogram to estimate the risk of AKI in sepsis patients. To use the nomogram, we first draw a line from each parameter value to the score axis for the score. The points for all the parameters are then added. Finally, a line from the total score axis is drawn to determine the risk of AKI on the lower line of the nomogram.

Previous studies have shown BUN level [28], infusion volume [29], and hyperlactatemia [30] to be associated with an increased likelihood of S-AKI incidence. The OR for infusion volume was 1.00009 because the unit of this parameter was ml, and in clinical settings, patients usually receive several hundred ml of infusion at a time. Accordingly, if a patient’s infusion volume is 1000 mL, the OR becomes 1.09 (1.000091000) for patients receiving no infusion. Elevated BUN levels and hyperlactatemia, in the clinical setting, are common indicators for physicians to carefully evaluate the condition of the patient and select suitable treatments or medications. The most sensitive indicator for the occurrence of AKI is the BUN level; a decline in the BUN level is associated with a reduced need for CRRT or diuretic use in patients. In this study, the ORs of the seven factors in the univariate logistic regression and multivariate logistic regression analyses showed a similar trend. Although body temperature can be controlled to maintain it as close to normal as possible, it is a complex parameter that is affected by several factors and may rise or fall secondary to factors such as cold shock.

We intend to develop more advanced machine learning algorithms, including random forest, support vector machine, and even RNNs for sepsis patients in the future. We plan to integrate these models into our local electronic health records system, from which the data and models can be extracted readily and automatically. Further, we intend to update this database and the models on a monthly basis by retraining and validating them using latest patient data. Although some data in a few algorithms may be difficult to interpret, the overall aim of this endeavor would be to enhance the specificity and sensitivity of the predictive ability.

Our study has several limitations. First, these data were derived from a single institution between 2008 and 2012; therefore, the model needs to be validated using data from other sources. Second, the risk of S-AKI in the first 24 h of admission to the ICU was predicted and no time series analysis was undertaken. Furthermore, with the accumulation of further laboratory and clinical data, a model-driven development approach should be employed. Finally, we only considered traditional parameters and did not consider new biomarkers of sepsis. With the inclusion of variables, such as neutrophil gelatinase-associated lipocalin (NGAL) [31], matrix metallopeptidase 9 (MMP-9) [32], and cystatin C, [33] the accuracy of the model may improve.

Conclusion

Our model effectively predicts S-AKI risk upon ICU admission. The S-AKI nomogram demonstrated high sensitivity and specificity and can be used widely if routine clinical data are available. The application of this nomogram in clinical care settings can help physicians in decision-making regarding treatment and management, which may in turn help reduce mortality caused by the development of AKI.