Risk Prediction for Acute Kidney Injury in Patients Hospitalized With COVID-19.

Rationale & Objective: Acute kidney injury (AKI) is common in patients hospitalized with COVID-19, but validated, predictive models for AKI are lacking. We aimed to develop the best predictive model for AKI in hospitalized patients with coronavirus disease 2019 and assess its performance over time with the emergence of vaccines and the Delta variant. Study Design: Longitudinal cohort study. Setting & Participants: Hospitalized patients with a positive severe acute respiratory syndrome coronavirus 2 polymerase chain reaction result between March 1, 2020, and August 20, 2021 at 19 hospitals in Texas. Exposures: Comorbid conditions, baseline laboratory data, inflammatory biomarkers. Outcomes: AKI defined by KDIGO (Kidney Disease: Improving Global Outcomes) creatinine criteria. Analytical Approach: Three nested models for AKI were built in a development cohort and validated in 2 out-of-time cohorts. Model discrimination and calibration measures were compared among cohorts to assess performance over time.
Results: Of 10,034 patients, 5,676, 2,917, and 1,441 were in the development, validation 1, and validation 2 cohorts, respectively, of whom 776 (13.7%), 368 (12.6%), and 179 (12.4%) developed AKI, respectively (P = 0.26). Patients in the validation cohort 2 had fewer comorbid conditions and were younger than those in the development cohort or validation cohort 1 (mean age, 54 ± 16.8 years vs 61.4 ± 17.5 and 61.7 ± 17.3 years, respectively, P < 0.001). The validation cohort 2 had higher median high-sensitivity C-reactive protein level (81.7 mg/L) versus the development cohort (74.5 mg/L; P < 0.01) and higher median ferritin level (696 ng/mL) versus both the development cohort (444 ng/mL) and validation cohort 1 (496 ng/mL; P < 0.001). The final model, which added high-sensitivity C-reactive protein, ferritin, and D-dimer levels, had an area under the curve of 0.781 (95% CI, 0.763-0.799). Compared with the development cohort, discrimination by area under the curve (validation 1: 0.785 [0.760-0.810], P = 0.79, and validation 2: 0.754 [0.716-0.795], P = 0.53) and calibration by estimated calibration index (validation 1: 0.116 [0.041-0.281], P = 0.11, and validation 2: 0.081 [0.045-0.295], P = 0.11) showed stable performance over time. Limitations: Potential billing and coding bias. Conclusions: We developed and externally validated a model to accurately predict AKI in patients with coronavirus disease 2019. The performance of the model withstood changes in practice patterns and virus variants.

As the COVID-19 pandemic evolved, acute kidney injury (AKI) has remained a common complication in patients hospitalized with COVID-19. We developed models to predict the occurrence of AKI in patients hospitalized with COVID-19. We built models using patient characteristics, comorbid conditions, and laboratory values. The model with the most accurate prediction included patient demographic characteristics, comorbid conditions, and inflammatory blood tests acquired on hospital admission. We also validated that the model remained effective as the pandemic progressed and the Delta variant emerged. The model was compared between the initial cohort and 2 cohorts of patients hospitalized later in the pandemic as the Delta variant emerged. We found that the model was stable and could predict AKI.

The coronavirus disease 2019 (COVID-19) may have a multiorgan involvement, and acute kidney injury (AKI) is a known sequela. Potential mechanisms include acute tubular injury because of hypotension, intravascular volume depletion leading to prerenal injury, and acute interstitial nephritis from antibiotics and antiviral treatments. However, COVID-19–specific mechanisms are also likely to be involved in kidney injury, as evidenced by case reports and postmortem autopsies., Observational studies that controlled for typical AKI risk factors have shown that COVID-19 is associated with AKI, suggesting that part of the pathogenesis may be specific to the virus itself. COVID-19–related mechanisms may include kidney ischemia from hypercoagulability, inflammation-related kidney injury in the setting of cytokine storm, and cardiorenal syndrome.

Older age; male sex; and a history of diabetes mellitus, chronic kidney disease (CKD), and hypertension are associated with AKI in patients with COVID-19.5, 6, 7, 8, 9, 10 However, predictive models with temporal external validation to prognosticate the development of AKI in the setting of hospitalization for COVID-19 are lacking. This is especially pertinent given the evolution of the pandemic over time because of changes in clinical management, vaccination, and emergence of new variants.

With the rollout of COVID-19 vaccines, cases began to decline in late 2020 and early 2021. However, the emergence of the Delta variant in March 2021 led to a subsequent rebound in cases. Delta soon became the dominant variant, with 70% and 98% of US COVID-19 cases attributable to the variant by July 1, 2021 and by August 1, 2021, respectively. The United States saw a rapid increase in hospitalizations because of a marked increase in transmissibility and number of breakthrough infections with the Delta variant compared to the original strains., Increasing evidence suggests that infection with the Delta variant leads to more severe illness in unvaccinated individuals13, 14, 15 and an increased risk of hospitalization compared with the previously prominent variants., It is unclear, however, what effect the Delta variant has on AKI pathogenesis, incidence, and outcomes.

We, therefore, aimed to (1) derive a prognostic model to predict the risk of AKI using granular demographic, clinical, and laboratory variables, including inflammatory biomarkers, during an index hospitalization with COVID-19; (2) externally evaluate the model’s predictive performance based on discrimination and calibration with out-of-time cohorts; and (3) assess data shifts in the pandemic because of varying treatments and emergence of the Delta variant.

Methods

Study Design and Participants

We conducted a longitudinal study using the University of Texas Southwestern (UTSW) COVID-19 Registry Collaborative database that contains granular demographic, clinical, and laboratory data on all COVID-19–related hospital admissions at 19 hospitals in North Texas, including UTSW Clements University Hospital and 18 hospitals in the Texas Health Resources Health System. Ethics approval and written informed consent were not required because the research was declared exempt by the institutional review board at UTSW as nonhuman research using existing data from the COVID-19 registry. For the development cohort, we included all individuals aged 18 years or older with a positive SARS-CoV-2 polymerase chain reaction test admitted between March 1, 2020, and January 1, 2021. The first hospital admission after the first positive polymerase chain reaction or up to 10 days prior was used. Only the first hospital admission was used if an individual had multiple admissions within the timeframe. We excluded the following patients: (1) those admitted for observation, (2) those without a COVID-19-associated billing diagnosis, (3) those with end-stage kidney disease or kidney transplantation, (4) those without comorbid condition information or a serum creatinine measurement during hospitalization, and (5) those in whom AKI was already present on admission (Fig S1). Data were collected for the entire hospitalization or a minimum of 30 days. Two out-of-time validation cohorts were used. The validation cohort 1 included patients with a positive SARS-CoV-2 polymerase chain reaction result admitted between January 2, 2021, and June 30, 2021, whereas the validation cohort 2 included those admitted between July 1, 2021, and August 20, 2021. The same exclusion criteria were applied to the validation cohorts. Data for July and August 2021 were grouped together because the COVID-19 Delta variant was the predominant strain at both UTSW and in Texas at-large based on variant testing.

Clinical and Laboratory Variables

Demographic variables, past medical history, and medications before admission were extracted from the electronic health records (Epic Systems). We collected treatments during hospitalization, including medications and mechanical ventilation, intensive care unit use, and kidney replacement therapy. We obtained laboratory variables at admission and during hospitalization. Comorbid conditions were considered present if the condition was listed as active on the patient’s problem list, medical history, or hospital problem and/or used 2 or more times (to improve specificity) as an encounter or billing diagnosis. Computable condition definitions were expressed as intentional (rule-based) value sets defined using standard Systemized Nomenclature of Medicine-Clinical Terms (SNOMED-CT) nomenclature as previously described (125 condition definitions are included as an online Supplement to this reference). Computable definitions of laboratory variables and inflammatory biomarkers were expressed using an internally vetted value set field within the Epic electronic health record. This field (“Common Name”) is used to determine which laboratory results can be displayed together for clinical viewing within the electronic health record to meet pathology and laboratory accreditation requirements.

Weekly sampling was performed for COVID-19 variant testing from August 2020 to August 2021 in patients from the UTSW Medical Center. All COVID-19–positive specimens with a cycle threshold value <35 by real-time polymerase chain reaction (approximately half of the positive samples) were sent for Sanger sequencing to determine the variant of coronavirus (Fig 1).

Figure 1

COVID-19 variant proportions over time by week at the University of Texas Southwestern Medical Center.

AKI Definitions

A detailed prespecified algorithm was used to define AKI according to standard KDIGO (Kidney Disease: Improving Global Outcomes) criteria: stage 1—increase in serum creatinine level by 0.3 mg/dL within 48 hours or 1.5-1.9 times increase from the baseline; stage 2—2.0- 2.9 times increase; and stage 3—≥3.0 times increase or initiation of renal replacement therapy (Fig S2). Urine output criteria were not used, given the variability in documentation. Patients were stratified according to the highest AKI stage attained during hospitalization. AKI was categorized as present on admission if the creatinine level at admission was higher than that at baseline and met one of the AKI stage criteria. We defined “baseline creatinine” by adapting and modifying a published algorithm, based on the availability or absence of prior serum creatinine values (Fig S3).

Outcome Measures

The prespecified primary outcome was in-hospital development of AKI that was not present on admission.

Statistical Analysis

Categorical data were presented as percentages and continuous data as means ± standard deviation (SD) or median (25th, 75th percentile). To compare categorical variables between patients with and without AKI, Fisher exact or χ2 tests were used as indicated. For continuous variables, Mann-Whitney U test was applied. For comparisons among the development, validation 1, and validation 2 data sets, continuous variables were compared using the nonparametric analysis of variance (Kruskal-Wallis test) followed by Dunn’s test. Categorical variables were compared using the Pearson’s χ2 test. Pairwise comparisons between data sets were performed if the overall test was significant. P values were adjusted with the Holm’s method. For comparison across AKI stages, Cochran-Armitage trend test was used for categorical variables and Jonckheere-Terpstra test was used for continuous variables.

To build a predictive model for AKI, we assessed a group of patients without AKI on admission and excluded the treatments and laboratory tests performed after the development of AKI while hospitalized. Prespecified covariates selected based on clinical relevance and biological plausibility were age, sex, race, ethnicity, smoking, hypertension, CKD, diabetes mellitus, coronary artery disease (CAD), congestive heart failure (CHF), use of an angiotensin-converting enzyme inhibitor (ACEI) or angiotensin receptor blocker (ARB), initial levels of white blood cell (WBC) count, high-sensitivity C-reactive protein (hs-CRP) level, hemoglobin level, ferritin level, and D-dimer level. Lactate dehydrogenase level was not included because of 26% missing values. Nested multivariable logistic models were built on top of a base model 1 that included demographic characteristics and comorbid conditions (age, sex, race, ethnicity, smoking, hypertension, diabetes mellitus, CKD, CAD, and CHF), as well as ACEI/ARB use. Model 2 added admission laboratory values commonly obtained in the setting of COVID-19—WBC count, hs-CRP level, and hemoglobin level—to model 1. Model 3 added additional inflammatory biomarkers, ferritin and D-dimer, to model 2. Nested models were compared with likelihood ratio tests after adding each biomarker group. A separate missing category for categorical variables was created, and complete case analysis was used for modeling. The estimated effect was reported by adjusted odds ratios with their 95% confidence intervals. We assessed the models for multicollinearity using variance inflation factors, where a variance inflation factor >10 suggested multicollinearity. No multicollinearity was found by variance inflation factor analysis. A mathematic model was derived to compute the log odds of AKI that could be transformed to predict the probability of AKI. The variable importance was assessed by the absolute value of the z statistic for each variable in the models.

The performance of the prognostic models was internally validated by 10-fold stratified cross-validation and externally validated with out-of-time validation over 2 intervals. Model discrimination was measured using areas under the receiver operating characteristic curve. Optimal predicted probability cut-offs were determined by Youden’s index from receiver operating characteristic analyses. Sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood ratio, and negative likelihood ratio were reported as an average (± SD) across 10-fold cross-validation. Model calibration was evaluated with calibration curves and estimated calibration index, which is the mean squared difference between predicted AKI probabilities and estimated observed probabilities based on the flexible calibration curves fitted by locally weighted scatterplot smoothing method.23, 24, 25 Bootstrap percentile-based confidence intervals were obtained from 2,000 bootstrap samples. Data shift was measured with population stability index (PSI), which summarizes the difference between 2 statistical distributions. A PSI < 0.10 indicated no significant changes, PSI between 0.10 and 0.25 indicated a small change that needs to be assessed, and PSI > 0.25 suggested a significant change.

Results

Baseline Characteristics

The development cohort, validation cohort 1, and validation cohort 2 contained 5,676 individuals, 2,917 individuals, and 1,441 individuals, respectively. Demographic characteristics, comorbid conditions, laboratory values, and clinical outcomes of the 3 cohorts are shown in Table 1. Comparisons by presence and stage of AKI in the development cohort are reported in Table S1. In the development cohort, 2,812 (49.5%) were men, 844 (15.8%) were African American, and 1,657 (30.2%) were Hispanic. The development cohort contained more Hispanic patients than the validation cohorts, driven by a peak in COVID-19 cases among the Hispanic population in the early summer of 2020. Patients in the validation cohort 2 were younger than patients in the development cohort or validation cohort 1; the mean ages were 54 ± 16.8 years versus 61.4 ± 17.5 years and 61.7 ± 17.3 years, respectively (adjusted P < 0.001). The patients in the validation cohort 2 had lower rates of hypertension, diabetes mellitus, CAD, CHF, and CKD than those in the development cohort and validation cohort 1. The patients in the validation cohort 2 also had higher hs-CRP levels (adjusted P < 0.01) than those in the development cohort and higher levels of ferritin than those in the development cohort and validation cohort 1 (adjusted P < 0.001).

Table 1

Baseline Characteristics and Outcomes of the Development and Validation Cohorts With Population Stability Indices of Variables in the Validation Cohorts

	DevelopmentMarch-December 2020N = 5,676	Validation 1January-June, 2021N = 2,917	Validation 2July-August 2021N = 1,441	OverallP Value
Baseline characteristic, n (%)
AKI incidence	776 (13.7%)	368 (12.6%)	179 (12.4%)	0.26
Age, y, mean ± SD	61.4 ± 17.5a	61.7 ± 17.3f	54.0 ± 16.8	<0.001
Male sex	2,812 (49.5%)	1448 (49.6%)	725 (50.3%)	0.87
Hispanic ethnicity	1,657 (30.2%)a,b	704 (24.7%)g	310 (21.9%)	<0.001
African American race	844 (15.8%)	418 (14.9%)	240 (17.0%)	0.20
Smoker (smoked at any time)	1,913 (34.1%)	1037 (36.0%)h	446 (31.3%)	0.008
Hypertension	3,380 (59.5%)a	1747 (59.9%)f	602 (41.8%)	<0.001
Diabetes mellitus	1,967 (34.7%)a	996 (34.1%)f	344 (23.9%)	<0.001
CKD	1,238 (21.8%)a,c	570 (19.5%)f	212 (14.7%)	<0.001
CAD	689 (12.1%)a	368 (12.6%)f	106 (7.4%)	<0.001
CHF	545 (9.6%)a	292 (10.0%)f	77 (5.3%)	<0.001
ACEI/ARB (on presentation)	2,195 (38.7%)a	1192 (40.9%)f	372 (25.8%)	<0.001
Laboratory variable, median (IQR)
WBC count, initial, K/μL (median (IQR))	6.8 (5.1-9.3)a,b	7.2 (5.2-9.8)f	6.5 (4.9-8.8)	<0.001
hs-CRP level, initial, mg/L (median (IQR))	74.5 (32.0-135.4)d	78.9 (34.2-141.1)	81.7 (43.2-137.0)	0.004
Hemoglobin level, initial, g/dL (median (IQR))	13.4 (12.0-14.6)a	13.4 (12.1-14.6)f	13.9 (12.5-15.0)	<0.001
Ferritin level, initial, ng/mL (median (IQR))	444.0 (205.1-961.1)a,b	495.8 (237.0-1068.0)f	696.0 (298.2-1511.0)	<0.001
D-dimer level, initial, mg/L (median (IQR))	0.9 (0.6-1.7)e	1.0 (0.6-1.8)h	0.9 (0.6-1.6)	<0.001
Outcome, n (%)
ICU	895 (15.8%)b	369 (12.6%)	193 (13.4%)	<0.001
Mechanical ventilation	427 (7.5%)	184 (6.3%)	105 (7.3%)	0.11
In-hospital mortality within 30 d	387 (6.8%)	185 (6.3%)	107 (7.4%)	0.40
CRRT for AKI	33 (0.6%)	10 (0.3%)	10 (0.7%)	0.23
HD for AKI	126 (2.2%)	60 (2.1%)	20 (1.4%)	0.14
CRRT and HD for AKI	7 (0.1%)	5 (0.2%)	1 (0.1%)	0.67
CRRT/HD for AKI	152 (2.7%)	65 (2.2%)	29 (2.0%)	0.23
PSI
AKI		0.001	0.001
Age		0.003	0.191
Male sex		0	0
Hispanic ethnicity		0.019	0.045
African American race		0.008	0.037
Smoker (smoked at any time)		0.002	0.003
Hypertension		0	0.128
Diabetes mellitus		0	0.056
CKD		0.003	0.034
CAD		0	0.027
CHF		0	0.027
ACEI/ARB (on presentation)		0.002	0.076
WBC count, initial, K/μL		0.012	0.017
hs-CRP level, initial, mg/L		0.009	0.037
Hemoglobin level, initial, g/dL		0.004	0.061
Ferritin level, initial, ng/mL		0.018	0.135
D-dimer level, initial, mg/L		0.008	0.016

Note: Percentages for categorical variables were obtained using the total number of patients with available variable as the denominator. For overall P value, continuous variables were compared using nonparametric analysis of variance (Kruskal-Wallis test), followed by Dunn’s test. Categorical variables were compared using Pearson’s χ2 test. Pairwise comparisons among data sets were performed if an overall test was significant. P values were adjusted with the Holm’s method. PSI summarizes the difference between the development and validation cohorts. A PSI < 0.10 indicates no significant change, PSI between 0.10 and 0.25 indicates a small change, and PSI > 0.25 indicates a significant change.

Abbreviations: ACEI/ARB, angiotensin-converting enzyme inhibitors/angiotensin II receptor blocker; AKI, acute kidney injury; CAD, coronary artery disease; CHF, congestive heart failure; CKD, chronic kidney disease; CRRT, continuous renal replacement therapy; HD, hemodialysis; hs-CRP, high-sensitivity C-reactive protein; ICU, intensive care unit; IQR, interquartile range; PSI, population stability index; SD, standard deviation; WBC, white blood cell.

P<0.001: Pairwise comparison between the development cohort and validation cohort 2.

P<0.001: Pairwise comparison between the development cohort and validation cohort 1.

P<0.05: Pairwise comparison between the development cohort and validation cohort 1.

P<0.01: Pairwise comparison between the development cohort and validation cohort 2.

P<0.01Pairwise comparison between the development cohort and validation cohort 1.

P<0.001: Pairwise comparison between the validation cohort 1 and validation cohort 2.

P<0.05: Pairwise comparison between the validation cohort 1 and validation cohort 2.

P<0.01: Pairwise comparison between the validation cohort 1 and validation cohort 2.

Outcomes

The outcomes of the 3 cohorts are illustrated in Table 1. The outcomes of the development cohort grouped by presence and stage of AKI are shown in Table S1. There were 776 (13.7%) patients in the development cohort, 368 (12.6%) in validation cohort 1, and 179 (12.4%) in validation cohort 2 who developed AKI. There was no difference in AKI rates among the cohorts. The rates of mechanical ventilation, in-hospital mortality, and kidney replacement therapy were also similar among the 3 cohorts. The rate of intensive care unit admission was significantly higher in the development cohort than in the validation cohort 1 (adjusted P < 0.001). Age, hypertension, and ferritin level were found to show a small change over time in the validation cohort 2 (0.10 < PSI < 0.25) (Table 1).

COVID-19 Variant Testing

Figure 1 shows the results of weekly testing for COVID-19 variants at UTSW from August 2020 to August 2021. The Delta variant was first seen in April 2021 but did not become the dominant variant until the week of June 27, 2021, when it comprised 77% of the total COVID-19 cases. The Delta variant made up 97% of all samples by the week of July 11, 2021 and 100% by August 8, 2021.

Predictive Models for AKI During Hospitalization

Receiver operating characteristic curves for nested models for the development cohort are illustrated in Fig 2A. Model 1, the base model (age, sex, race, ethnicity, smoking status, underlying hypertension, diabetes mellitus, CKD, CAD, CHF, and ACEI/ARB use at baseline), had an area under the curve (AUC) (95% confidence interval) of 0.753 (0.734-0.771). Model 2 (which added admission WBC, hs-CRP, and hemoglobin to model 1), had a higher AUC of 0.764 (95% confidence interval, 0.746-0.783) compared with model 1 (likelihood ratio test adjusted P < 0.001). Model 3, which contained all variables in model 2 plus admission ferritin and D-dimer levels, had the highest AUC of 0.781 (0.763-0.799) (adjusted P < 0.001 for comparison with model 2). Each subsequent model showed a statistically significant improvement with the likelihood ratio test. The adjusted odds ratios for all variables included in the models are shown in Table 2. The metrics used to compare the performance of the AKI predictive models from internal validation are shown in Table S2. Variable importance for each model showed that the most important variable for all 3 models was the history of CKD; the second most important variable was hs-CRP level for model 2 and ferritin level for model 3 (Fig S4). Model 3 had a sensitivity of 0.69 (SD, 0.10), specificity of 0.76 (SD, 0.08), negative predictive value of 0.94 (SD, 0.01), positive predictive value of 0.32 (SD, 0.04), positive likelihood ratio 3.02 (SD, 0.62), and negative likelihood ratio of 0.40 (SD, 0.09) when using Youden’s index as the probability cut off for AKI.

Figure 2

Receiver operating characteristic curves for nested acute kidney injury models for (A) development cohort, (B) validation cohort 1, and (C) validation cohort 2. Model 1 contains age, sex, race, ethnicity, smoking status, hypertension, diabetes mellitus, chronic kidney disease, coronary artery disease, congestive heart failure, and angiotensin-converting enzyme inhibitor/angiotensin receptor blocker use. Model 2 contains all variables in model 1 plus initial white blood cell count, high-sensitivity C-reactive protein level, and hemoglobin level. Model 3 contains all variables in model 2 plus initial ferritin and D-dimer levels. Abbreviation: AUC, area under the curve.

Table 2

Predictive Models for AKI in the Development Cohort

N = 5,676	Model 1		Model 2		Model 3
	aOR (95%CI)	P Value	aOR (95%CI)	P Value	aOR (95%CI)	P Value
Age, per year	1.011 (1.005-1.017)	<0.001	1.010 (1.004-1.016)	0.001	1.009 (1.003-1.015)	0.006
Male sex	1.369 (1.160-1.616)	<0.001	1.382 (1.159-1.649)	<0.001	1.214 (1.012-1.457)	0.04
Hispanic	1.467 (1.192-1.805)	<0.001	1.368 (1.108-1.688)	0.004	1.373 (1.101-1.700)	0.004
African American	1.281 (1.005-1.632)	0.05	1.234 (0.965-1.577)	0.09	1.114 (0.867-1.433)	0.40
Smoker	0.922 (0.772-1.101)	0.37	0.932 (0.779-1.115)	0.44	0.976 (0.814-1.170)	0.79
Hypertension	1.450 (1.157-1.817)	0.001	1.454 (1.158-1.828)	0.001	1.448 (1.149-1.825)	0.002
Diabetes	1.448 (1.220-1.719)	<0.001	1.416 (1.191-1.685)	<0.001	1.518 (1.272-1.811)	<0.001
CKD	4.766 (3.994-5.687)	<0.001	4.588 (3.829-5.499)	<0.001	4.288 (3.567-5.154)	<0.001
CAD	1.065 (0.844-1.342)	0.60	1.070 (0.846-1.355)	0.57	1.110 (0.874-1.409)	0.39
CHF	1.244 (0.979-1.579)	0.07	1.256 (0.984-1.602)	0.07	1.296 (1.013-1.659)	0.04
ACEI/ARB (on admission)	0.976 (0.816-1.168)	0.79	1.013 (0.844-1.215)	0.89	1.070 (0.889-1.289)	0.47
WBC count, per K/µL increase			1.010 (1.001-1.018)	0.03	1.009 (1.001-1.018)	0.04
hs-CRP level, per mg/L increase			1.004 (1.003-1.004)	<0.001	1.003 (1.002-1.004)	<0.001
Hemoglobin level, per g/dL increase			0.928 (0.889-0.968)	<0.001	0.938 (0.899-0.980)	0.004
Ferritin level, per ng/mL increase					1.0002 (1.0001-1.0003)	<0.001
D-dimer level, per mg/L increase					1.055 (1.034-1.077)	<0.001

Note: Model 1 contains age, sex, race, ethnicity, smoking status, hypertension, diabetes mellitus, CKD, CAD, CHF, and ACEI/ARB use. Model 2 contains all variables in model 1 plus initial WBC count, hs-CRP level, and hemoglobin level. Model 3 contains all variables in model 2 plus initial ferritin and D-dimer levels. P values obtained using Wald’s test. Those with AKI present on admission and those with missing information on comorbid conditions or continuous variables were excluded.

Abbreviations: ACEI/ARB, angiotensin-converting enzyme inhibitor/angiotensin receptor blocker; aOR, adjusted odds ratio; CAD, coronary artery disease; CHF, chronic heart failure; CI, confidence interval; CKD, chronic kidney disease; hs-CRP, high-sensitivity C-reactive protein; WBC, white blood cell.

The following is the resultant formula for the AKI prediction model (model 3) that computes the log odds (logit) of AKI:

Log odds of AKI = −3.4524 + 0.0087 (Age) + 0.1942 (Male∗) + 0.3171 (Hispanic∗) + 0.4016 (Hispanic missing#) + 0.1082 (African American∗) + 0.1111 (African American missing#) – 0.0248 (Smoker∗) + 0.5228 (Smoker missing#) + 0.3701 (Hypertension∗) + 0.4171 (Diabetes∗) + 1.4558 (CKD∗) + 0.1043 (CAD∗) + 0.2595 (CHF∗) + 0.068 (ACEI_ARB∗) + 0.009 (WBC) − 0.0636 (Hemoglobin) + 0.0025 (CRP) + 0.0002 (Ferritin) + 0.0536 (D-dimer)

where ∗ indicates for male insert 1, for female insert 0; for Hispanic ethnicity insert 1, for non-Hispanic ethnicity insert 0; for African American race insert 1, for non-African American race insert 0; for smoker insert 1, for nonsmoker insert 0; for patients treated with ACEI or ARB or with hypertension, diabetes, CKD, CAD, or CHF insert 1 in the corresponding term, otherwise insert 0 and # indicates for missing Hispanic ethnicity, missing African American race, missing smoker, insert 1.

The log odds of AKI can be transformed into the predicted probability of developing AKI with the following formula:

Predicted AKI probability = 1/[1 + e(−1∗Logit AKI)]

Model Evaluation Over Time

Model discrimination was measured using AUC. Receiver operating characteristic curves for the nested AKI prediction models in validation cohorts 1 and 2 are illustrated in Fig 2B and C. The adjusted P values comparing the AUC for each model of the development cohort to both validation cohorts were all not significant, indicating stable discrimination over time (Table 3).

Table 3

Discrimination and Calibration Measures in the Development Cohort, Validation Cohort 1, and Validation Cohort 2

		Development CohortMarch-December 2020	Validation Cohort 1January-June 2021	Validation Cohort 2July-August 2021	Development vs Validation 1Adjusted P	Development vs Validation 2Adjusted P	Validation 1 vs Validation 2Adjusted P
Model 1
Discrimination	AUC	0.753	0.757	0.740
	(95% CI)a	(0.734-0.771)	(0.730-0.784)	(0.699-0.781)	1.00	1.00	1.00
Calibration	ECI	0.005	0.059	0.115
	(95% CI) a	(0.004-0.043)	(0.019-0.179)	(0.070-0.340)	0.11	<0.001	0.24
Model 2
Discrimination	AUC	0.764	0.775	0.732
	(95% CI)a	(0.746- 0.783)	(0.749-0.800)	(0.690-0.774)	0.52	0.33	0.29
Calibration	ECI	0.004	0.038	0.144
	(95% CI)a	(0.003-0.046)	(0.016-0.133)	(0.083-0.356)	0.13	<0.001	0.12
Model 3
Discrimination	AUC	0.781	0.785	0.754
	(95% CI)a	(0.763-0.799)	(0.760-0.810)	(0.716-0.795)	0.79	0.53	0.53
Calibration	ECI	0.020	0.116	0.081
	(95% CI)a	(0.006-0.073)	(0.041-0.281)	(0.045-0.295)	0.11	0.11	0.93

Note: P value was adjusted with Holm’s method. Model 1 contains age, sex, race, ethnicity, smoking status, hypertension, diabetes mellitus, chronic kidney disease, coronary artery disease, congestive heart failure, and angiotensin-converting enzyme inhibitor/angiotensin receptor blocker use. Model 2 contains all variables in model 1 plus initial white blood cell count, high-sensitivity C-reactive protein level, and hemoglobin level. Model 3 contains all variables in model 2 plus initial ferritin and D-dimer levels.

Abbreviations: AUC, area under the curve; CI, confidence interval; ECI, estimated calibration index.

Bootstrap percentile-based confidence interval was obtained from 2,000 bootstrap samples for all CIs.

The calibration of all models for all cohorts is shown by calibration plots (Fig S5). The calibration plots for model 3 showed agreement between the predicted probabilities of AKI and the empirical probability, which aligned close to the diagonal reference line, suggesting good calibration. When comparing the development cohort to validation cohort 2, there was a statistically significant decrease in the calibration in both models 1 and 2. Model 3 showed no statistically significant changes in the estimated calibration index when the groups were compared, suggesting stable calibration over time (Table 3).

Discussion

To our knowledge, we report the first internally and externally out-of-time validated AKI prediction model in patients hospitalized with COVID-19. We generated 3 models for the development of AKI in a contemporary cohort of nearly 6,000 patients across 19 hospitals. Our models, which included clinically available demographic characteristics, comorbid conditions, and inflammatory biomarkers routinely obtained during hospitalization with COVID-19, were validated in 2 cohorts (a total of 4,358 patients hospitalized later in the pandemic) to account for emerging virus variants and increasing vaccination rates. Our final model maintained stable discrimination and calibration despite these developments in the pandemic, supporting the clinical utility of this model.

As the pandemic evolved, so did the reports of AKI incidence. Initial reports from China showed rates of AKI ranging from 0%-15%.28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39 Later studies from the United States reported higher but inconsistent incidence of AKI, ranging from 14%-69%.7, 8, 9, 10,40, 41, 42, 43, 44 This variability may be because of differences in definitions and demographic characteristics between cohorts. For example, not all studies included patients requiring intensive care. Although most studies defined AKI according to KDIGO guidelines, methods for determining baseline creatinine levels differed. Changes in the predominant circulating variants over time may affect predilection for AKI. Practice patterns also evolved, with changes in available evidence to support the use of corticosteroids and remdesivir, which may have affected AKI incidence. The emergence and adoption of vaccines likely also contributed to shifts in the populations most at risk of severe illness and AKI.

Predictive models for AKI in the setting of hospitalization with COVID-19 have been reported in smaller samples but have not been temporally validated. Fisher et al. developed a predictive model for stages 2-3 AKI that included respiratory rate, WBC count, neutrophil/lymphocyte ratio, and lactate dehydrogenase level. This model did not exclude AKI present on admission; thus, it is possible that some of these variables may reflect collinearity with pre-existing AKI as opposed to the prediction of AKI development during hospitalization. To address this limitation in the existing literature, we excluded patients with AKI present on admission to determine risk factors before AKI onset. Our findings extend prior knowledge by developing a comprehensive predictive model and resultant formula for AKI to help prognosticate AKI in patients hospitalized with COVID-19.

Well-calibrated clinical models that maintain their ability to accurately predict clinical risk over time aid providers in decision making to improve patient outcomes, prioritize how resources are allocated, and decrease health care costs. The development of treatment strategies including vaccination and the emergence of the more contagious Delta variant led to the evolution of the pandemic, highlighting the importance of models that are resilient over time. Although our findings showed data shift in independent variables in our 3 models, discrimination remained stable. The calibration of our models deteriorated over time mainly in the validation cohort 2, which corresponded to the increase in the Delta variant. However, our final, most extensive model, which included ferritin and D-dimer levels, showed stable calibration. Patients hospitalized with COVID-19 when the Delta variant was predominant were younger, with fewer comorbid conditions, and higher markers of inflammation than those patients admitted earlier in the pandemic. This is further illustrated by the higher PSI values reflecting unstable change in variables for age, history of hypertension, and ferritin level for the validation cohort 2. To date, limited information is available on differences in demographic characteristics, comorbid conditions, and laboratory abnormalities in patients with the Delta variant versus previous variants. The inclusion of inflammatory markers, specifically ferritin, not only improved the performance of the model but also maintained stability in discrimination and calibration despite a shifting patient population.

This study has several strengths. We studied a more contemporary COVID-19 cohort than previously reported. Our study included 10,000 patients across multiple hospitals in a large metropolitan area, enrolling a diverse sample to adequately power the analyses. The variables used are commonly available among patients hospitalized with COVID-19. We also note several limitations. The comorbid conditions identified by billing or coding data may not be completely accurate. Missing values may not have been missing at random, but there were few such patients excluded from the study, so this likely had little impact on the generalizability of our findings. Changes in practice patterns and the effect of vaccination on the population at risk of severe infection may have affected the results. We addressed this limitation by validating our model using 2 cohorts later in the pandemic. Finally, although this tool could be very useful to identify the most vulnerable patients who could develop AKI, it may not be as useful with the emergence of new variants of the virus with significantly different behavior. Model monitoring would be needed over time to continue to evaluate performance as the pandemic further evolves.

In conclusion, using demographic characteristics, comorbid conditions, admission laboratory values, and markers of inflammation, we developed and validated predictive models for the development of AKI during hospitalization with COVID-19 and demonstrated that our final model was robust to evolving changes in the pandemic from 2020 to 2021 and during the Delta variant surge.