Acute Kidney Injury in the Outpatient Setting: Developing and Validating a Risk Prediction Model.

RATIONALE &
OBJECTIVE: Risk factors for acute kidney injury (AKI) in the hospital have been well studied. Yet, risk factors for identifying high-risk patients for AKI occurring and managed in the outpatient setting are unknown and may differ. STUDY
DESIGN: Predictive model development and external validation using observational electronic health record data. SETTING & PARTICIPANTS: Patients aged 18-90 years with recurrent primary care encounters, known baseline serum creatinine, and creatinine measured during an 18-month outcome period without established advanced kidney disease. NEW PREDICTORS & ESTABLISHED PREDICTORS: Established predictors for inpatient AKI were considered. Potential new predictors were hospitalization history, smoking, serum potassium levels, and prior outpatient AKI. OUTCOMES: A ≥50% increase in the creatinine level above a moving baseline of the recent measurement(s) without a hospital admission within 7 days defined outpatient AKI. ANALYTICAL APPROACH: Logistic regression with bootstrap sampling for backward stepwise covariate elimination was used. The model was then transformed into 2 binary tests: one identifying high-risk patients for research and another identifying patients for additional clinical monitoring or intervention.
RESULTS: Outpatient AKI was observed in 4,611 (3.0%) and 115,744 (2.4%) patients in the development and validation cohorts, respectively. The model, with 18 variables and 3 interaction terms, produced C statistics of 0.717 (95% CI, 0.710-0.725) and 0.722 (95% CI, 0.720-0.723) in the development and validation cohorts, respectively. The research test, identifying the 5.2% most at-risk patients in the validation cohort, had a sensitivity of 0.210 (95% CI, 0.208-0.213) and specificity of 0.952 (95% CI, 0.951-0.952). The clinical test, identifying the 20% most at-risk patients, had a sensitivity of 0.494 (95% CI, 0.491-0.497) and specificity of 0.806 (95% CI, 0.806-0.807). LIMITATIONS: Only surviving patients with measured creatinine levels during a baseline period and outcome period were included.
CONCLUSIONS: The outpatient AKI risk prediction model performed well in both the development and validation cohorts in both continuous and binary forms.

We used electronic health record data from a multisite health care system to develop a predictive model for acute kidney injury (AKI) occurring and managed in the outpatient setting, or outpatient AKI, over an 18-month period among a primary care–receiving population. The continuous form of the model was transformed into 2 separate binary forms for 2 distinct purposes: identifying high-risk patients for closer clinical monitoring or for research in outpatient AKI. The continuous model was subsequently externally validated using national electronic health record data from the Veterans Health Administration and showed good performance. The performance of the binary tests in the validation cohort was also described. This model, in its various forms, may be used to identify high-risk patients for outpatient AKI.

Acute kidney injury (AKI) occurs frequently among hospitalized patients, including as many as half of critically ill patients. AKI also occurs in the ambulatory setting, with only a subset of patients subsequently admitted to a hospital.3, 4, 5, 6, 7, 8 Regardless of the setting, patients with AKI are at an increased risk of adverse outcomes, including the progression of chronic kidney disease (CKD), end-stage kidney disease, cardiovascular disease, and death.8, 9, 10, 11, 12, 13, 14 Among hospitalized patients, the risk factors for AKI have become increasingly well recognized, and prediction models for patients in the hospital have been published.15, 16, 17, 18, 19, 20, 21 Yet, there has been no such examination of the predictability of AKI in the ambulatory setting or support tool developed to assess AKI risk in outpatients. A substantial proportion of AKI cases may occur in the outpatient setting and be managed by outpatient providers,, and although the short-term mortality risk is lower, the long-term mortality risk appears to be similar for hospitalized patients with AKI and those not hospitalized at the time of AKI after statistical adjustment. Therefore, identifying patients at high risk for outpatient AKI is important. This may lead to improved pre- or post-AKI care in the ambulatory setting and research in preventive interventions in this under-recognized group.

We sought to develop and externally validate a risk prediction model for AKI occurring and managed in the outpatient setting using covariates widely available in electronic health records (EHRs). Such a predictive model could be used to both define and subsequently identify “high-risk” patients for clinical interventions (eg, avoidance of nonsteroidal anti-inflammatory drugs) or recruitment into clinical trials.

Methods

Study Population

A model-development cohort was derived from the M Health Fairview health system (of the greater Minneapolis, Minnesota area), beginning with its current EHR adoption November 29, 2000, through June 1, 2018. A baseline period was defined by the receipt of at least 2 primary care encounters over at least 18 months (to allow both an adequate number of encounters and time to document comorbid conditions and obtain laboratory and vital sign measurements). All baseline periods ended by December 1, 2016, to allow the completion of a subsequent 18-month outcome period for all participants.

An external validation cohort was derived from national Veterans Affairs (VA) data via VA Informatics and Computing Infrastructure. Owing to a longer history of VA EHR use, the validation cohort was limited to patients with a second qualifying primary care encounter after December 31, 2009. Data were accessed through January 1, 2019 (with all baseline periods ending by July 1, 2017).

Inclusion criteria included at least one outpatient-measured serum creatinine value before the end of the baseline period and also at least one from the outcome period. Participants were aged 18-90 years and without baseline end-stage kidney disease, kidney transplantation, or CKD stage 5 (Figs 1 and 2).

Figure 1

Patient study eligibility flow diagram for the development cohort, derived from the M Health Fairview health care system electronic health record. Abbreviations: AKI, acute kidney injury; eGFR, estimated glomerular filtration rate.

Figure 2

Patient study eligibility flow diagram for the validation cohort, derived from the Veterans Affairs health care system electronic health record. Abbreviations: AKI, acute kidney injury; eGFR, estimated glomerular filtration rate.

Baseline Characteristics

Baseline laboratory- and vital signs–based covariates were defined by the final measurement during the baseline period. The baseline estimated glomerular filtration rate (eGFR) and proteinuria have been identified as independent risk factors for inpatient AKI., For this study, eGFR was treated as a continuous variable and independently calculated by the Modification of Diet in Renal Disease Study equation using the “175” coefficient. The Modification of Diet in Renal Disease Study equation was selected as it is used in the VA EHR,, whereas the eGFR equation used by M Health Fairview was not consistent across the study timeframe.

To avoid widespread missing or old data, proteinuria was defined as the most recent spot albuminuria, total proteinuria, or semiquantitative urinalysis-albumin concentration. In the event of concurrent measurements or uninterpretable results, spot albuminuria was the first choice (because of precalculated ratios) and urinalysis was last. “Severe” proteinuria was defined as spot albuminuria or total proteinuria of ≥300 mg/g or urinalysis-albumin concentration of “≥300” or equivalent. “Moderate” proteinuria was 30-300 mg/g or urinalysis-albumin concentration of “30-300” or equivalent. Proteinuria below these thresholds was considered “normal.”

Owing to assumed nonrandomness in missing data, binary “missingness” covariates were introduced to coincide with laboratory measurements and vital signs., This allowed modeling AKI risk on both laboratory measurements/vital signs and whether these covariates had been measured. When missing, the original covariate was entered as zero. Occasional nonphysiologic values were treated as missing (Table S1). Laboratory measurements and vital signs were included according to the risk factors identified in inpatient AKI models: serum sodium, albumin, hemoglobin A1c, calcium, hemoglobin, and systolic and diastolic blood pressures (BPs). Serum potassium level was also included because of widespread availability on metabolic panels.

AKI was divided into 2 mutually exclusive categories because of potential differences between inpatient and outpatient AKIs (Fig S1). Cases of AKI that developed within 7 days preceding hospitalization, the so-called community-acquired AKI,4, 5, 6, 7, were categorized with cases of hospital-acquired AKI as “inpatient AKI.” This was done because of the similar mortality risk observed by others after either community-acquired AKI and hospital-acquired AKI and a dissimilar mortality pattern after AKI without hospitalization. To avoid missing cases of inpatient AKI that developed before admission, inpatient AKI was defined as an increase in inpatient-measured creatinine level of ≥50% above the most recent outpatient-measured creatinine level from 7-365 days before admission or, if none existed, the most recent creatinine level from any setting more than 365 days prior. Baseline inpatient AKI was defined before the end of the baseline period.

Outpatient AKI was defined by a ≥50% increase in outpatient-measured creatinine level compared with a moving baseline. The moving baseline creatinine level was defined as the average of the 3 most recent outpatient-measured creatinine values 25-365 days prior. If no baseline outpatient creatinine was available during that window, the most recent creatinine measurement >365 days prior was used. Creatinine values between 0 and 25 days prior were not included in the calculation of the moving baseline to avoid missing cases of slowly peaking AKI (which otherwise would result in the inclusion of increasing creatinine measurements in the moving baseline). This was similar to prior definitions for AKI originating in the outpatient setting., A creatinine increase of ≥0.3 mg/dL over 48 hours or ≥50% over ≤7 days, was not used as outpatient laboratory measurements are rarely obtained with that frequency. Outpatient AKI was defined as a baseline risk factor when occurring before the end of the baseline period.

Other comorbid diseases were defined by ≥2 International Classification of Diseases-9 and 10 and/or Current Procedural Terminology billing codes. These included diabetes mellitus, hypertension, cardiovascular disease (defined as the composite of coronary artery disease, peripheral vascular disease, congestive heart failure, or stroke), hypertension, cancer, and liver disease. The race was defined as “Black” if ever documented as Black in the EHR. Smoking status was defined as “ever smoker” or “never smoker.” The hospitalization history was defined by any hospitalization during which serum creatinine level was measured to exclude nonmedical/surgical hospitalizations.

Study Outcome

The study outcome was outpatient AKI occurring during a fixed 18-month outcome period starting at the end of the baseline period. The same definition for outpatient AKI was again used. An 18-month outcome period was chosen to allow the inclusion of patients seeking care as infrequently as yearly (plus an additional 6 months for allowable tardiness), as lacking at least one outcome period creatinine measurement was a study exclusion criterion.

Statistical Analyses

Logistic regression was used to assess the risk of outpatient AKI during the outcome period. Restricted cubic splines were initially applied to all continuous variables to assess nonlinearity. Nonlinear and interaction terms specified a priori were included only in the subsequent variable selection if both were statistically significantly associated with outpatient AKI and if meaningful nonlinear fitting or effect modification of AKI risk was observed on visual inspection.

The final model was developed using backward stepwise elimination of covariates with N = 1,000 bootstrap sampling based on the Akaike information criterion. If either a continuous or paired missingness covariate was selected to remain in the model, the pair was forced into the model. The risk equation was subsequently applied to the validation cohort and assessed by C statistic for discrimination and Hosmer-Lemeshow goodness-of-fit test and visually assessed by decile of predicted AKI risk.

To demonstrate the clinical significance of predicting outpatient AKI, time-to-mortality after the outpatient AKI period was determined in the subsets of patients in the highest and lowest deciles of predicted AKI risk. Death was defined in the VA EHR. Censoring was defined by the final encounter, vital sign, or laboratory measurement. Crude analyses for mortality predicted by outpatient AKI were performed using the log-rank test. Unadjusted and adjusted hazard ratios were calculated using Cox proportional hazards modeling. Covariates for adjustment were age, sex, race, smoking history, eGFR category (eGFR of ≥60, 45-59, 30-44, or 15-29 mL/min per 1.73 m2), baseline inpatient and outpatient AKI history, cardiovascular disease, diabetes mellitus, hypertension, liver disease, cancer, and hospitalization history.

The original model was subsequently transformed into 2 binary tests based on potential uses: (1) identification of high-risk research patients or (2) identification of patients warranting additional clinical monitoring or interventions. Each positive-negative cut-point in the continuous AKI risk prediction was selected by weighting the utility or cost of true and false positive and negative cases based on perceived risks and benefits. The maximum value of the resulting “total utility” was calculated in the development cohort and the optimal cut-point applied to the validation cohort. The research test was defined by setting the utility of true positives at 100 times the cost of false positives to promote an adequate number of AKI outcomes for well-powered research. This was balanced by weighting the utility of true negatives at 10 times the cost of false negatives owing to the risk and financial cost of study inclusion incurred from those not truly at high risk. The clinical test was similarly defined by setting the utility of true positives at 20 times the cost of false positives to promote diagnosing cases of AKI, but without the strict concern for statistical power, and the utility of true negatives at 1/20th the cost of false negatives due to an assumed lower risk and cost incurred from unnecessary clinical monitoring than a research study.

The Institutional Review Boards of the University of Minnesota and Minneapolis VA approved this study (applications 1502M63126 and 4739-B, respectively). Informed consent was waived due to deidentified data. Cohorts were defined in SQL (Microsoft). Statistical analyses were performed in R (R Foundation for Statistical Computing).

Results

Cohort Characteristics

There were 152,371 patients in the development cohort, of whom 4,611 experienced outpatient AKI during the 18-month outcome period. In the validation cohort, 115,744 of 4,864,576 patients experienced outpatient AKI. The median ages were 55 and 63 years in the development and validation cohorts, respectively. The median baseline serum creatinine values were similar: 0.9 and 1.0 mg/dL for the development and validation cohorts, respectively. Baseline characteristics and their linear associations with outpatient AKI for both cohorts are shown in Tables 1 and 2. The distribution of “most recent” proteinuria measurements used to define baseline proteinuria is shown in Table S2.

Table 1

Baseline Characteristics of Model-Development Cohort, Overall and by Outpatient Acute Kidney Injury

Baseline Characteristic	All Patients (N = 152,371)	With Outpatient AKIa (N = 4,611)	Without Outpatient AKIa (N = 147,760)	P valueb
Age, y, median (IQR)	55.1 (43.1-66.7)	60.7 (48.3-71.8)	55.0 (43.0-66.6)	<0.001
Male, N (%)	69,799 (46%)	1,789 (39%)	68,010 (46%)	<0.001
Black, N (%)	9,185 (6.0%)	365 (7.9%)	8,820 (6.0%)	<0.001
Ever smoker, N (%)	66,388 (44%)	2,254 (49%)	64,134 (43%)	<0.001
Final creatinine level of the baseline period, mg/dL, median (IQR)	0.9 (0.8-1)	0.9 (0.7-1.1)	0.9 (0.8-1)	0.03
eGFRc stage, N (%)				<0.001
eGFR, ≥60	127,976 (84%)	3,269 (71%)	124,707 (84%)
Stage 3A, eGFR, 45-59	17,732 (12%)	779 (17%)	16,953 (12%)
Stage 3B, eGFR, 30-44	5,518 (3.6%)	401 (8.7%)	5,117 (3.5%)
Stage 4, eGFR, 15-29	1,145 (0.8%)	162 (3.5%)	983 (0.7%)
Most recent proteinuria elevated, N (%)d	12,245 (8.0%)	715 (16%)	11,530 (7.8%)	<0.001
Most recent proteinuria not elevated, N (%)d	78,138 (51%)	2,402 (52%)	75,736 (51%)	0.26
Missing, N (%)	61,988 (41%)	1,494 (32%)	60,494 (41%)	<0.001
Baseline proteinuria is “moderate,” N (%)d	10,514 (6.9%)	550 (12%)	9,964 (6.7%)	<0.001
Baseline proteinuria is “severe,” N (%)d	1,731 (1.1%)	165 (3.6%)	1,566 (1.1%)	<0.001
Baseline history of outpatient AKIe, N (%)	4,336 (2.8%)	492 (11%)	3,844 (2.6%)	<0.001
Baseline history of inpatient AKIf, N (%)	766 (0.5%)	98 (2.1%)	668 (0.5%)	<0.001
Number creatinine values at baseline, median (IQR)	2 (1-4)	3 (1-6)	2 (1-4)	<0.001
Diabetes mellitus, N (%)	25,696 (17%)	1,353 (29%)	24,343 (17%)	<0.001
Hypertension, N (%)	65,787 (43%)	2,487 (54%)	63,300 (43%)	<0.001
Cardiovascular disease, N (%)g	19,511 (13%)	1,113 (24%)	18,398 (13%)	<0.001
Coronary artery disease, N (%)g	12,833 (8.4%)	700 (15.2%)	12,133 (8.2%)	<0.001
Congestive heart failure, N (%)g	3,951 (2.6%)	382 (8.3%)	3,569 (2.4%)	<0.001
Stroke, N (%)g	4,221 (2.8%)	224 (4.9%)	3,997 (2.7%)	<0.001
Peripheral vascular disease, N (%)g	3,789 (2.5%)	272 (5.9%)	3,517 (2.4%)	<0.001
Liver disease, N (%)	4,758 (3.1%)	269 (5.8%)	4,489 (3.0%)	<0.001
Cancer, N (%)	13,443 (8.8%)	559 (12%)	12,884 (8.7%)	<0.001
Systolic blood pressure, mm Hg, median (IQR)	124 (114-135)	126 (114-138)	124 (114-135)	<0.001
Missing, N (%)	6,166 (4.0%)	279 (6.1%)	5,887 (4.0%)	<0.001
Diastolic blood pressure, mm Hg, median (IQR)	76 (69-82)	74 (67-82)	76 (69-82)	<0.001
Missing, N (%)	6,244 (4.1%)	281 (6.1%)	5,963 (4.0%)	<0.001
Sodium level, mmol/L, median (IQR)	140 (138-142)	140 (138-142)	140 (138-142)	<0.001
Missing, N (%)	5,901 (3.9%)	142 (3.1%)	5,759 (3.9%)	0.005
Potassium level, mmol/L, median (IQR)	4.1 (3.9-4.4)	4.1 (3.9-4.5)	4.1 (3.9-4.4)	0.29
Missing, N (%)	3,205 (2.1%)	95 (2.1%)	3,110 (2.1%)	0.84
Calcium level, mg/dL, median (IQR)	9.2 (9.0-9.5)	9.2 (8.9-9.5)	9.2 (9.0-9.5)	<0.001
Missing, N (%)	7,050 (4.6%)	191 (4.1%)	6,859 (4.6%)	0.11
Hemoglobin, g/dL, median (IQR)	14.0 (12.9-15.0)	13.3 (11.9-14.4)	14.0 (13.0-15.0)	<0.001
Missing, N (%)	27,023 (18%)	650 (14%)	26,373 (18%)	<0.001
Albumin level, g/dL, median (IQR)	4.1 (3.9-4.4)	4 (3.7-4.3)	4.1 (3.9-4.4)	<0.001
Missing, N (%)	43,216 (28%)	1,153 (25%)	42,063 (29%)	<0.001
Hemoglobin A1c in percentage, median (IQR)	6.1 (5.6-7.0)	6.5 (5.8-7.6)	6.1 (5.6-7.0)	<0.001
Missing, N (%)	108,953 (72%)	2,770 (60%)	106,183 (72%)	<0.001
Baseline history of hospitalization, N (%)	16,222 (11%)	1,004 (22%)	15,218 (10%)	<0.001

Note: Conversion factors for units: creatinine in mg/dL to μmol/L, ×88.4; calcium in mg/dL to mmol/L, ×0.2495.

Abbreviations: AKI, acute kidney injury; eGFR, estimated glomerular filtration rate; IQR, interquartile range.

Outcome of AKI occurring and managed in the outpatient setting over an 18-month period.

Two sample t test or χ test, as applicable, applied to those with outpatient AKI versus those without.

eGFR, by the Modification of Diet in Renal Disease Study equation, in mL/min per 1.73 m2.

Proteinuria defined as the most recent of spot albuminuria, spot total proteinuria, or semiquantitative albuminuria by urinalysis, derived from Table 7 of the Kidney Disease Improving Global Outcomes 2012 Clinical Practice Guideline for the Evaluation and Management of Chronic Kidney Disease. “Severe” = spot albuminuria or total proteinuria ≥300 mg/g or albuminuria by urinalysis of “300” or equivalent. “Moderate” = spot albuminuria or total proteinuria ≥30 mg/g or albuminuria by urinalysis of “30” or equivalent. “Normal” or not elevated if otherwise measured.

Baseline history of AKI occurring and managed in the outpatient setting.

Baseline history of AKI managed in the inpatient setting regardless of setting of onset.

Cardiovascular disease defined as the composite of coronary artery disease, congestive heart failure, stroke, or peripheral vascular disease as nonmutually exclusive categories.

Table 2

Baseline Characteristics of Model Validation Cohort, Overall and by Outpatient Acute Kidney Injury

Baseline Characteristic	All Patients (N = 4,864,576)	With Outpatient AKIa (N = 115,744)	Without Outpatient AKIa (N = 4,748,832)	P valueb
Age, y, median (IQR)	63.3 (52.4-70.2)	63.9 (56.8-70.6)	63.2 (52.3-70.2)	<0.001
Male, N (%)	4,492,178 (92%)	106,449 (92%)	4,385,729 (92%)	<0.001
Black, N (%)	815,167 (17%)	23,457 (20%)	791,710 (17%)	<0.001
Ever smoker, N (%)	3,466,441 (71%)	90,231 (78%)	3,376,210 (71%)	<0.001
Final creatinine of the baseline period, mg/dL, median (IQR)	1.0 (0.9-1.1)	1.0 (0.8-1.2)	1.0 (0.9-1.1)	<0.001
eGFRc stage, N (%)				<0.001
eGFR, ≥60	4,080,191 (84%)	87,880 (76%)	3,992,311 (84%)
Stage 3A, eGFR, 45-59	572,480 (12%)	15,938 (14%)	556,542 (12%)
Stage 3B, eGFR, 30-44	177,056 (3.6%)	8,129 (7.0%)	168,927 (3.6%)
Stage 4, eGFR, 15-29	34,849 (0.7%)	3,797 (3.3%)	31,052 (0.7%)
Most recent proteinuria elevatedd, N (%)	623,912 (13%)	29,135 (25%)	594,777 (13%)	<0.001
Most recent proteinuria not elevatedd, N (%)	3,797,304 (78%)	77,568 (67%)	3,719,736 (78%)	<0.001
Missing, N (%)	443,360 (9.1%)	9,041 (7.8%)	434,319 (9.1%)	<0.001
Baseline proteinuria is “moderate”d, N (%)	569,567 (12%)	23,851 (21%)	545,716 (12%)	<0.001
Baseline proteinuria is “severe”d, N (%)	54,345 (1.1%)	5,284 (4.6%)	49,061 (1.0%)	<0.001
Baseline history of outpatient AKIe, N (%)	322,564 (6.6%)	20,484 (18%)	302,080 (6.4%)	<0.001
Baseline history of inpatient AKIf, N (%)	119,253 (2.5%)	9,184 (7.9%)	110,069 (2.3%)	<0.001
Number creatinine values at baseline, median (IQR)	8 (4-17)	12 (5-24)	8 (4-17)	<0.001
Diabetes mellitus, N (%)	1,311,503 (27%)	51,945 (45%)	1,259,558 (27%)	<0.001
Hypertension, N (%)	3,024,529 (62%)	90,739 (78%)	2,933,790 (62%)	<0.001
Cardiovascular diseaseg, N (%)	1,259,851 (26%)	45,748 (40%)	1,214,103 (26%)	<0.001
Coronary artery diseaseg, N (%)	1,006,004 (21%)	34,841 (30%)	971,163 (21%)	<0.001
Congestive heart failureg, N (%)	221,907 (4.6%)	14,361 (12%)	207,546 (4.4%)	<0.001
Strokeg, N (%)	148,727 (3.1%)	5,739 (5.0%)	142,988 (3.0%)	<0.001
Peripheral vascular diseaseg, N (%)	271,887 (5.6%)	12,792 (11%)	259,095 (5.5%)	<0.001
Liver disease, N (%)	142,127 (2.9%)	7,114 (6.1%)	135,013 (2.8%)	<0.001
Cancer, N (%)	816,634 (17%)	26,079 (23%)	790,555 (17%)	<0.001
Systolic blood pressure, mm Hg, median (IQR)	129 (120-138)	131 (120-142)	129 (120-138)	<0.001
Missing, N (%)	18,296 (0.4%)	517 (0.4%)	17,779 (0.4%)	<0.001
Diastolic blood pressure, mm Hg, median (IQR)	77 (70-84)	76 (68-84)	77 (70-84)	<0.001
Missing, N (%)	21,091 (0.4%)	593 (0.5%)	20,498 (0.4%)	<0.001
Sodium level, mmol/L, median (IQR)	139 (138-141)	139 (137-141)	139 (138-141)	<0.001
Missing, N (%)	11,946 (0.2%)	325 (0.3%)	11,621 (0.2%)	0.01
Potassium level, mmol/L, median (IQR)	4.2 (4.0-4.5)	4.2 (3.9-4.5)	4.2 (4.0-4.5)	<0.001
Missing, N (%)	33,930 (0.7%)	816 (0.7%)	33,114 (0.7%)	0.76
Calcium level, mg/dL, median (IQR)	9.3 (9.0-9.6)	9.3 (9.0-9.6)	9.3 (9.0-9.6)	<0.001
Missing, N (%)	259,971 (5.3%)	4,733 (4.1%)	255,238 (5.4%)	<0.001
Hemoglobin, g/dL, median (IQR)	14.5 (13.5-15.4)	13.8 (12.5-14.9)	14.5 (13.5-15.4)	<0.001
Missing, N (%)	77,238 (1.6%)	1,614 (1.4%)	75,624 (1.6%)	<0.001
Albumin level, g/dL, median (IQR)	4.1 (3.8-4.3)	4.0 (3.6-4.2)	4.1 (3.8-4.3)	<0.001
Missing, N (%)	330,596 (6.8%)	5,750 (5.0%)	324,846 (6.8%)	<0.001
Hemoglobin A1c in percentage, median (IQR)	5.9 (5.5-6.5)	6.1 (5.6-7.2)	5.9 (5.5-6.5)	<0.001
Missing, N (%)	1,212,012 (25%)	20,102 (17%)	1,191,910 (25%)	<0.001
Baseline history of hospitalization, N (%)	1,069,398 (22%)	44,850 (39%)	1,024,548 (22%)	<0.001

Note: Conversion factors for units: creatinine in mg/dL to μmol/L, ×88.4; calcium in mg/dL to mmol/L, ×0.2495.

Abbreviations: AKI, acute kidney injury; eGFR, estimated glomerular filtration rate; IQR, interquartile range.

Outcome of AKI occurring and managed in the outpatient setting over an 18-month period.

Two sample t test or χ test, as applicable, applied to those with outpatient AKI versus those without.

eGFR, by the Modification of Diet in Renal Disease Study equation, in mL/min per 1.73 m2.

Proteinuria defined as the most recent of spot albuminuria, spot total proteinuria, or semiquantitative albuminuria by urinalysis, derived from Table 7 of the Kidney Disease Improving Global Outcomes 2012 Clinical Practice Guideline for the Evaluation and Management of Chronic Kidney Disease. “Severe” = spot albuminuria or total proteinuria ≥300 mg/g or albuminuria by urinalysis of “300” or equivalent. “Moderate” = spot albuminuria or total proteinuria ≥30 mg/g or albuminuria by urinalysis of “30” or equivalent. “Normal” or not elevated if otherwise measured.

Baseline history of AKI occurring and managed in the outpatient setting.

Baseline history of AKI managed in the inpatient setting regardless of setting of onset.

Cardiovascular disease defined as the composite of coronary artery disease, congestive heart failure, stroke, or peripheral vascular disease as nonmutually exclusive categories.

Model Development

Linear versus restricted cubic spline fitting for continuous covariates were compared (Table S3). The majority of interaction terms were not included in the variable selection (Table S4). The final model included age, sex, smoking history, outpatient AKI history, eGFR, proteinuria, cardiovascular disease, diabetes mellitus, hypertension, liver disease, serum sodium, potassium, hemoglobin A1c, albumin, hemoglobin, systolic and diastolic BP, and hospitalization history. Included interaction terms were age and hospitalization history, sex and systolic BP, and sex and diastolic BP. The model is shown in Table S5. The resulting model performed well in the development cohort with a C statistic of 0.717 (95% CI, 0.710-0.725).

Model Validation

The model performed well in the validation cohort, with C statistic of 0.722 (95% CI, 0.720-0.723). The Hosmer-Lemeshow goodness-of-fit test suggested appreciable differences between the predicted and observed risk of outpatient AKI in the validation cohort (P < 0.001). The performance of the model in the validation cohort is shown in Fig 3 by decile of predicted AKI risk and did not suggest overfitting of the model.

Figure 3

Observed risk versus mean predicted risk of outpatient acute kidney injury in the 18 months after a primary care baseline period by decile of predicted risk in the validation cohort (N = 4,864,576). Observed and mean predicted risks by deciles are 0.5% versus 0.9% for the 1st decile, 0.8% versus 1.3% for the 2nd decile, 1.0% versus 1.6% for the 3rd decile, 1.2% versus 1.9% for the 4th decile, 1.5% versus 2.2% for the 5th decile, 1.9% versus 2.6% for the 6th decile, 2.3% versus 3.1% for the 7th decile, 2.9% versus 4.0% for the 8th decile, 4.0% versus 5.4% for the 9th decile, and 7.7% versus 12.1% for the 10th decile, respectively. N = 486,458 for odd-numbered deciles and 486,457 for even-numbered deciles. Abbreviation: AKI, acute kidney injury.

Clinical Significance of the Model

In the lowest decile of predicted AKI risk in the validation cohort, there were 5,149 deaths among 450,160 patients with documented follow-up after the AKI period (73 deaths were among 2,321 patients with outpatient AKI). Median follow-up after the AKI period was 3.7 years. Kaplan-Meier curves reflected greater mortality among those experiencing outpatient AKI in the uniformly low-AKI risk group (P < 0.001) (Fig 4). The unadjusted and adjusted hazard ratios for death after outpatient AKI were 2.81 (95% CI, 2.23-3.54) and 2.54 (95% CI, 2.02-3.20), respectively.

Figure 4

Kaplan-Meier survival curves for (A) the lowest decile of predicted outpatient acute kidney injury (AKI) risk (N = 450,160) and (B) the highest decile of predicted outpatient AKI risk (N = 481,058) in the validation cohort, beginning at the end of the 18-month AKI period, among those not lost to follow-up before the end of the AKI period, and censored on final encounter, vital sign, or laboratory measurement.

In the highest decile of AKI risk, there were 191,892 deaths among 481,058 patients with 4.5 years median follow-up (17,452 deaths were among 36,973 patients with outpatient AKI). Kaplan-Meier curves also displayed greater mortality among those with outpatient AKI than those without (P < 0.001) (Fig 4). The unadjusted and adjusted hazard ratios for death after outpatient AKI were 1.36 (95% CI, 1.33-1.38) and 1.59 (95% CI, 1.56-1.61), respectively.

Model Performance as 2 Binary Tests

The performance of the original model when transformed into research-oriented and clinically oriented tests is shown in Table 3. A “positive” result for the research test, derived from the development cohort, was ≥9.5% risk, resulting in a positive predictive value of 0.096 (95% CI, 0.094-0.097) and a negative predictive value of 0.980 (95% CI, 0.980-0.980) in the validation cohort. A “positive” result for the clinical test was ≥4.5% risk, resulting in a positive predictive value of 0.058 (95% CI, 0.058-0.059) and a negative predictive value of 0.985 (95% CI, 0.985-0.985).

Table 3

Performance in the Validation Cohort (Unless Otherwise Specified) of 2 Binary Tests Produced From the Risk Prediction Model for Outpatient Acute Kidney Injury in the 18 Months After a Primary Care Baseline Period

Statistic	Research Test	Clinical Test
Predicted risk cut-pointa	≥9.5%	≥4.5%
Sensitivity	0.210 (95% CI, 0.208-0.213)	0.494 (95% CI, 0.491-0.497)
Specificity	0.952 (95% CI, 0.951-0.952)	0.806 (95% CI, 0.806-0.807)
Positive predictive value	0.096 (95% CI, 0.094-0.097)	0.058 (95% CI, 0.058-0.059)
Negative predictive value	0.980 (95% CI, 0.980-0.980)	0.985 (95% CI, 0.985-0.985)
Positive likelihood ratio	4.34 (95% CI, 4.29-4.39)	2.55 (95% CI, 2.53-2.57)
Negative likelihood ratio	0.83 (95% CI, 0.83-0.83)	0.63 (95% CI, 0.62-0.63)
Percent of development cohort “negative” by the test (percentile of cut-point)	96.7%	85.5%
Percent of validation cohort “negative” by the test (percentile of cut-point)	94.8%	80.0%
P value for predicting outpatient AKIb	<0.001	<0.001

Abbreviations: AKI, acute kidney injury; CI, confidence interval.

Threshold defining a “positive” test, as selected based on model performance in the development cohort.

By χ2 test, for the association between outpatient AKI and test positivity.

Discussion

We developed and externally validated a (continuous) risk prediction model for AKI occurring and managed in the outpatient setting. The continuous form of our model was also transformed into 2 binary tests with different applications: one for identifying patients for closer clinical monitoring or intervention and another for identifying research participants for randomized trials. In the continuous form, the model performed equally well in both the development and external validation cohorts (C statistic, ∼0.72), supporting generalizability. This performance was similar to that observed in inpatient AKI risk models, which have generally produced C statistics ranging from 0.7-0.8.18, 19, 20,

The model also performed well in its binary forms. The clinical test, as designed, offers primary care providers a tool with which they can determine which patients may most benefit from closer clinical monitoring for outpatient AKI, which may lead to improved preventive care, prognostication, and post-AKI care. The sensitivity and specificity of the clinical test indicate that approximately one-half of cases of outpatient AKI would be identified by this test, and only 1 in 5 of those who would not experience outpatient AKI over 18 months are falsely identified as high-risk patients. Eighty percent of patients are deemed low-risk by this test.

The research test, with a higher positivity threshold identifying the 5% most at-risk patients, could alternatively serve to identify patients for clinical trials of outpatient AKI prevention. Although only 1 in 5 cases of outpatient AKI is captured by the research test, approximately 10% of those who test “positive” would have the outcome (compared with 2%-3% in the general population). Should clinical providers or researchers weigh the risks or benefits of being labeled “high-risk” differently than we did for either test, our model (when applied to a population) could be reconfigured with alternate weights.

Our study results stand in contrast to prior risk prediction models focused on AKI among hospitalized patients. Several risk factors known to associate with inpatient AKI overlapped with the risk factors identified in this study of outpatient AKI, although other novel risk factors were identified. In the study by Malhotra et al, like our study, CKD, heart disease, liver disease, hypertension, and anemia were identified as predictive of AKI. Others have identified albuminuria, diabetes mellitus, hypoalbuminemia, and hyponatremia as postsurgical AKI risk factors. Other known predictors of AKI that are typically limited to the inpatient setting, including acidemia, mechanical ventilation or hypoxia, or sepsis,15, 16, 17, 18, were not included in our model for variable selection.

Novel predictors of AKI in the outpatient setting identified included hospitalization history, smoking history, and serum potassium levels. Surprisingly, hypotension has not been frequently identified as a risk factor for predicting inpatient AKI. We identified a greater impact on AKI risk by diastolic BP over the physiologic range observed than by systolic BP, although both were selected for model inclusion. We also recognized interactions between age and prior hospitalizations and between sex and systolic and diastolic BP on their association with outpatient AKI risk. Also identified was the predicted risk of outpatient AKI associated with lacking a baseline laboratory or vital sign measurement, which, to our knowledge, has not been previously reported for AKI in any setting. The study by Tomasev et al did not provide risk estimates for individual covariates. Although lacking these prior measurements may be a general marker of wellness or a marker of infrequent or incomplete care by which outpatient AKI may be missed, not all missingness covariates predicted a reduced risk of observing outpatient AKI. Rather, lacking a prior proteinuria, hemoglobin A1c, or serum albumin measurement increased the predicted risk. Not identified in our model as risk factors for outpatient AKI were race, cancer, or serum calcium, despite being identified in studies of inpatient AKI., Although prior outpatient AKI was identified as a risk factor for subsequent outpatient AKI, inpatient AKI was unexpectedly not identified as an outpatient AKI risk factor.

Our model did display differences between observed and predicted risk in the validation cohort. This was not unexpected owing to our study’s statistical power. Although validation cohorts are typically recommended to contain at least 100 cases and 100 noncases,, ours far exceeded those figures. Despite this, a reasonable visual agreement was seen between observed and predicted risk for outpatient AKI. Our model did typically overestimate the AKI risk in the validation cohort, which may be due to important differences between the 2 cohorts not captured by the covariates included in our study. Nonetheless, the model performed well in its continuous and binary forms in the validation cohort.

Although not the focus of this study, we observed that outpatient AKI was more common than inpatient AKI, consistent with our prior work and highlighting the potential impact of predicting outpatient AKI. In this study, we showed that the mortality risk was not fully captured by the presence of AKI risk factors (despite likely overlap in risk factors) and was heightened by the occurrence of outpatient AKI. On an absolute scale, this effect was greatest in the highest AKI risk group. On a relative scale, the occurrence of outpatient AKI impacts the mortality risk greatest among patients who, other than AKI, are the most well.

Additional strengths of this study include large cohort sizes and a lack of excluding patients for or imputation of missing data. They also include the use of objective variables in the EHR that could be used by both clinicians and researchers to identify high-risk patients.

Study limitations include its retrospective design. Additional cases of outpatient AKI not reflected in the EHR may have been missed. Cases of inpatient AKI that persisted into the outpatient setting may also have been misclassified in some cases. Cases of progressive CKD may have been incorrectly categorized as AKI if creatinine had been measured only infrequently. We were unable to retrospectively determine which creatinine measurements used isotope-dilution mass spectrometry–traceable methods and which did not. Our study design also restricted the cohorts to those patients surviving the 18-month AKI period. Early mortality after outpatient AKI is reassuringly low. Although some cases of outpatient AKI soon followed by death may have been missed, this design avoided bias from classifying those who died before AKI into the “no AKI” group. Finally, medications were not considered in this study, in part, because of the limited accuracy of outpatient medication lists outside managed care organizations.

Our outpatient AKI risk prediction model performed well in both our development and validation cohorts. The risk predicted by our model is calculable from readily available vital signs, laboratory tests, and diagnosed comorbid conditions in addition to incorporating the predictive quality of lacking these measurements. Such a tool can be adapted into a binary test to categorize patients as high-risk or not based on weighing risks and benefits and could be used to guide clinical monitoring or identify patients for research. Future studies will be needed to prospectively validate our model and to determine whether interventions in patients with an elevated risk of outpatient AKI can reduce the incidence of AKI or lead to a reduction in the mortality associated with outpatient AKI.