External validation of the Madrid Acute Kidney Injury Prediction Score.

BACKGROUND: The Madrid Acute Kidney Injury Prediction Score (MAKIPS) is a recently described tool capable of performing automatic calculations of the risk of hospital-acquired acute kidney injury (HA-AKI) using data from from electronic clinical records that could be easily implemented in clinical practice. However, to date, it has not been externally validated. The aim of our study was to perform an external validation of the MAKIPS in a hospital with different characteristics and variable case mix.
METHODS: This external validation cohort study of the MAKIPS was conducted in patients admitted to a single tertiary hospital between April 2018 and September 2019. Performance was assessed by discrimination using the area under the receiver operating characteristics curve and calibration plots.
RESULTS: A total of 5.3% of the external validation cohort had HA-AKI. When compared with the MAKIPS cohort, the validation cohort showed a higher percentage of men as well as a higher prevalence of diabetes, hypertension, cardiovascular disease, cerebrovascular disease, anaemia, congestive heart failure, chronic pulmonary disease, connective tissue diseases and renal disease, whereas the prevalence of peptic ulcer disease, liver disease, malignancy, metastatic solid tumours and acquired immune deficiency syndrome was significantly lower. In the validation cohort, the MAKIPS showed an area under the curve of 0.798 (95% confidence interval 0.788-0.809). Calibration plots showed that there was a tendency for the MAKIPS to overestimate the risk of HA-AKI at probability rates ˂0.19 and to underestimate at probability rates between 0.22 and 0.67.
CONCLUSIONS: The MAKIPS can be a useful tool, using data that are easily obtainable from electronic records, to predict the risk of HA-AKI in hospitals with different case mix characteristics.

INTRODUCTION

The incidence of hospital-acquired acute kidney injury (HA-AKI) ranges between 5% and 15%, or 30–45 cases per 1000 hospital admissions per year, but it shows an increasing trend as hospitalized patients are older and subjected to many diagnostic and treatment interventions, as well as exposure to the effects of nephrotoxic drugs [1-3]. HA-AKI is associated with high morbidity and increased mortality rates [4-6]. Since a large majority of HA-AKI episodes are due to potentially avoidable causes, knowing precisely the individual risk of each patient as soon as possible after hospital admission is crucial to the implementation of preventive measures aimed at reducing the incidence of HA-AKI [7-9]. Different models based on demographic data and chronic comorbidities have been developed for this purpose [10-14]. One of the most recently published predictive models is the Madrid Acute Kidney Injury Prediction Score (MAKIPS) [15]. This model can automatically use data from electronic clinical records and can be implemented easily in clinical practice. However, to date, it has not been externally validated. Independent external validation is essential to determine whether the model can be considered as a clinical predictive model by ruling out potential overfitting or deficiencies in statistical modelling in the developing cohort and to evaluate the applicability of the model in different case mix populations [16, 17].

The objective of our study was to perform an external validation of the MAKIPS as a model to predict HA-AKI in a hospital centre with different with different case-mix characteristics.

MATERIALS AND METHODS

This retrospective observational external validation cohort study of the MAKIPS was performed in adult (≥18 years) patients hospitalized in Hospital Arnau de Vilanova in Lleida, Spain, from April 2018 to September 2019. Hospital Arnau de Vilanova is a high-complexity tertiary centre that serves a population of 430 217 inhabitants in Lleida that provides medical, surgical and endovascular catheter-guided interventions, with the exception of cardiac surgery and lung, liver, kidney or bone marrow transplantation services.

Patient comorbidities, diagnoses and procedural interventions were obtained from electronic records of medical data and classified according to the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM), applying the same codes used to develop the MAKIPS. Biochemical data from inpatient settings were obtained from electronic laboratory databases. Patients were included if they were ≥18 years of age, were admitted for at least 24 h in hospital and had at least two serum creatinine measurements during their hospital stay. Patients who had AKI within the first 48 h of hospital admission were excluded, as they were considered to have community-acquired AKI (CA-AKI). Patients on chronic dialysis treatment were also excluded.

Baseline kidney function

Our patient care system integrates the laboratory databases of both hospital and primary care registers, thus allowing historical data to be obtained for all patients who are hospitalized, provided that these data had been previously recorded in those registers. Baseline kidney function was obtained from electronic records of laboratory data from the primary healthcare register and defined as the most recent glomerular filtration rate (GFR), as estimated by the Chronic Kidney Disease Epidemiology Collaboration equation, within the last 12 months prior to hospital admission. For patients with no serum creatinine measurement available within 12 months prior to hospitalization, the baseline kidney function was the lowest serum creatinine measurement taken during hospitalization.

Definition of AKI

AKI was defined and classified according to severity stages based on the Kidney Disease: Improving Global Outcomes criteria [18]. HA-AKI was defined as an increase in serum creatinine ≥0.3 mg/dL or >50% above the baseline occurring within the first 48 h to any time during hospital admission.

AKI detection

Software integrated into the hospital electronic laboratory database was used to perform repeat comparisons of all serum creatinine measurements taken for each patient during their hospital stay and generated an identification code, with ‘1’ assigned when AKI criteria were met and ‘0’ assigned when not. It also assigned the level of AKI severity according to the maximum differences in serum creatinine levels detected. The number of the admission episode, which is unique for each patient, was used as a filter so that patients with more than one AKI episode during their hospital stay were recorded on the database only once, with the entry corresponding to the more severe AKI episode.

The research team members responsible for data analysis had access to anonymized data only and were blinded to any other data. The study was conducted in accordance with the Declaration of Helsinki and Spanish law and was approved by the ethics committees of the two participating centres, which considered that informed consent was not necessary.

Statistics

The incidence calculations were based on the total number of admissions. For patients who developed more than one AKI episode during their hospital stay, only the most severe episode was included in the study. Patients were considered to be at risk on each hospital admission and therefore patients who, during the study period, were admitted two or more times, were included in the calculations for each admission, except when readmission occurred within 30 days after hospital discharge. Results are given as the mean ± standard deviation (SD) or as the median and 25th–75th percentiles (P25–P75). Differences in risk factors between groups were calculated using the unpaired Student’s t-test for quantitative variables or the chi-squared test for categorical variables. A P-value ˂0.05 was considered statistically significant. The individual risk of developing HA-AKI was estimated by the MAKIPS [15], assigning a value of zero to cardiac surgery. Discrimination of the MAKIPS was evaluated using the C statistic and the area under the receiver operating characteristics curve (AUROC). Calibration diagrams were used to calculate the goodness-of-fit of the MAKIPS in the external validation cohort. Statistical analyses were performed using the Statistical Package for the Social Sciences for Windows version 20.0 (IBM, Armonk, NY, USA) and R software version 3.6.3 (R Foundation for Statistical Computing, Vienna, Austria).

RESULTS

Clinical and biochemical characteristics of patients

During the study period there were 26 362 hospital discharges. Figure 1 shows the flow chart of patient selection. The final study group comprised 21 787 patients. Of this cohort, 1155 patients (5.3%) developed AKI, with an incidence of 53 AKI episodes per 1000 hospital admissions. Distributions by AKI stages were as follows: stage 1, n = 785 (68%); stage 2, n = 219 (19%); and stage 3, n = 151 (13%).

FIGURE 1:

Flow chart showing patient selection.

Table 1 summarizes the demographic, clinical and admission characteristics of the study group and those of the MAKIPS cohort of patients. When compared with the MAKIPS cohort, patients from our study group showed a higher percentage of men as well as significantly higher prevalence of diabetes, hypertension, cardiovascular disease, cerebrovascular disease, anaemia, congestive heart failure, chronic pulmonary disease, connective tissue diseases and renal disease, whereas the prevalence of peptic ulcer disease, liver disease, malignancy, metastatic solid tumours and acquired immune deficiency syndrome (AIDS) was significantly lower. The percentages of both surgical patients and urgent admissions were significantly higher in our cohort of patients.

Table 1.

Comorbidity and admission characteristics of the external validation and MAKIPS cohorts

Variables	External validation cohort	MAKIPS cohort	P-value
Patients, n	21 787	47 466
Men, % (n)	46 (9932)	43.5 (20 647)	<0.0001
Mean age (years), mean (SD)	60.1 (19.7)	62.1 (20.1)	nd
Diabetes, % (n)	13.2 (2876)	12.2 (5786)	0.0002
Hypertension, % (n)	32 (6972)	30.3 (14 392)	<0.0001
Cardiovascular disease, % (n)	8.1 (1765)	7.6 (3596)	0.0167
Cerebrovascular disease, % (n)	6.9 (1486)	6 (2842)	<0.0001
Anaemia, % (n)	12 (2614)	11 (5205)	0.0035
Myocardial infarction, % (n)	3 (654)	2.8 (1363)	0.0888
Congestive heart failure, % (n)	7.5 (1634)	6.7 (3222)	0.0007
Peripheral vascular disease, % (n)	4 (851)	3.9 (1867)	0.8675
Dementia, % (n)	0.8 (172)	0.6 (319)	0.0967
Chronic pulmonary disease, % (n)	14.4 (3102)	13.4 (6385)	0.0052
Connective tissue disease, % (n)	3.6 (790)	1.7 (809)	<0.0001
Peptic ulcer disease, % (n)	0.38 (83)	0.5 (265)	<0.0001
Liver disease, % (n)	4.2 (915)	5.3 (2535)	<0.0001
Hemiplegia, % (n)	1.1 (240)	1.0 (506)	0.6700
Renal disease, % (n)	8 (1743)	6.0 (2849)	<0.0001
Malignancy, % (n)	14.3 (3115)	15.0 (7142)	0.0103
Metastatic solid tumour, % (n)	4 (871)	6.5 (3107)	<0.0001
AIDS/HIV, % (n)	0.4 (86)	0.6 (294)	0.0003
Urgent admission, % (n)	66.3 (14 445)	54.6 (25 916)	<0.0001
Surgical admission, % (n)	49 (10 675)	45.6 (21 633)	<0.0001
Admission department			<0.0001
Intensive care unit, % (n)	4.5 (980)	0.78 (372)	–
Nephrology, % (n)	1.5 (372)	0.42 (200)	–
Cardiology, % (n)	10.7 (2340)	6.3 (2986)	–
Cardiac surgery, % (n)	0	0.48 (228)	–
Vascular surgery, % (n)	3.6 (792)	1.8 (854)	–
Urology, % (n)	8.8 (1918)	6 (2835)	–
General surgery, % (n)	22.8 (4982)	11.8 (5596)	–
Other, % (n)	47.9 (10 449)	72.4 (34 395)	–

HIV, human immunodeficiency virus.

Table 2 summarizes the demographic characteristics and comorbidities of the external validation cohort of patients classified according to the presence of HA-AKI. Patients with HA-AKI were older and predominantly male compared with non-AKI patients. Comorbidities, including diabetes, cardiovascular disease, anaemia, hemiplegia, congestive heart failure, liver disease, malignancy and renal disease, were more frequent in AKI patients. Patients with AKI also showed significantly higher rates of urgent and surgical admissions. AKI patients had higher levels of uric acid, urea, glucose and potassium, as well as higher leucocyte counts, compared with non-AKI patients.

Table 2.

Demographic and clinical characteristics of the external validation cohort, classified according to the presence or absence of HA-AKI

Variables	Total	AKI	Non-AKI	P-value
Patients, n (%)	21 787	1155 (5.3)	20 632 (94.7)
Male, n (%)	10 022 (46.0)	647 (56.0)	9375 (45.4)	<0.0001
Age (years), mean (SD)	55.8 (21.3)	75.4 (24.1)	54.7 (20.3)	<0.0001
Diabetes, n (%)	2876 (13.2)	316 (27.4)	2560 (12.4)	<0.0001
Cardiovascular disease, n (%)	1.765 (8.1)	231 (20.0)	1534 (7.4)	<0.0001
Anaemia, n (%)	2614 (12.0)	312 (27.0)	2302 (11.1)	<0.0001
Hemiplegia, n (%)	240 (1.1)	29 (2.5)	211 (1.0)	<0.0001
Congestive heart failure, n (%)	1634 (7.5)	323 (28.0)	1311 (6.3)	<0.0001
Liver disease, n (%)	915 (4.2)	127 (11.0)	788 (3.8)	<0.0001
Malignancy, n (%)	3115 (14.3)	283 (24.5)	2832 (13.7)	<0.0001
Renal disease, n (%)	1743 (8.0)	337 (29.2)	1406 (6.8)	<0.0001
Urgent admission, n (%)	14.445 (66.3)	901 (78.0)	13 544 (65.6)	<0.0001
Surgical admission, n (%)	10 675 (49.0)	665 (57.5)	10 010 (48.5)	<0.0001
Estimated GFR (mL/min/1.73 m2), median (P25–P75)	94.1 (75–114.6)	76.2 (51–98.3)	95.4 (77–1189)	<0.0001
Uric acid (mg/dL), median (P25–P75)	4.9 (3.7–6.8)	6.1 (4.7–7.6)	4 (3.5–4.6)	<0.0001
Urea (mg/dL), median (P25–P75)	39 (31.0–45.0)	50 (41.0–72.0)	39 (28.0-55.0)	<0.0001
Calcium (mg/dL), median (P25–P75)	9.1 (8.4–9.6)	8.8 (8.1–9.4)	9.3 (8.2–9.5)	<0.0001
Glucose (mg/dL), median (P25–P75)	94 (83.0–124.0)	114 (98.0–155.0)	93 (82.0–116.0)	<0.0001
Sodium (mEq/L), median (P25–P75)	138 (136.0–141.0)	137 (135.0–142.0)	139 (134.0–143.0)	<0.0001
Potassium (mEq/L), median (P25–P75)	4.2 (3.7–4.6)	4.3 (3.9–4.7)	4.1 (3.8–4.4)	<0.0001
Leucocytes (n/µL), median (P25–P75)	8.23 (5.1–11.9)	10.7 (6.6–12.3)	8.6 (5.8–10.9)	<0.0001

Predictive value and goodness-of-fit of the MAKIPS algorithm in the external validation cohort

The MAKIPS showed an AUROC of 0.798 [95% confidence interval (CI) 0.788–0.809] (Figure 2).

FIGURE 2:

AUROC of the MAKIPS to predict HA-AKI in the external validation cohort.

Calibration plots for the association between predicted probabilities and observed event rates showed that with a 95% CI there was a tendency for the MAKIPS to overestimate the observed risk of HA-AKI at probability rates ˂0.21 and to underestimate at probability rates between 0.22 and 0.67 (Figure 3).

FIGURE 3:

Calibration plot of the MAKIPS in the external validation cohort (n = 21 787). Calibration plots for the association between predicted probabilities and observed event rates showed that with a 95% CI, there was a tendency for the MAKIPS to overestimate the observed risk of HA-AKI at probability rates <0.21 and underestimate at probability rates between 0.22 and 0.67.

DISCUSSION

In this study we have carried out the first external validation of the MAKIPS score in a hospital that, in relation to the center where the original model was performed, lacks cardiac surgery and presents differential characteristics, both in the clinical profile and in the distribution of patients when compared with the hospital studied when developing the original model.

The overall incidence of HA-AKI reported in different studies varies, depending on the definition criteria of CA-AKI and the percentage of patients who come from intensive care units (ICUs), with an incidence of ~50% in the latter [13, 19–22]. The percentage of patients with CA-AKI in our study was very similar to that described in the MAKIPS cohort [15]. On the other hand, although the proportion of admissions to ICUs was significantly higher in our cohort, these patients represented only a small percentage of the total in both centres. Therefore the incidence of HA-AKI, in both cases, was very similar to previous reports in non-critically ill patients [23]. When comparing our cohort of patients with the MAKIPS cohort, we observed statistically significant differences in the prevalence of most of the chronic comorbidities analysed, in spite of the fact that in both cohorts the same ICD-9 codes were applied when classifying clinical conditions. These differences may be due to dissimilarities in the case mix between the hospitals, but may also be due to biases associated with potential discrepancies in assigning administrative codes to clinical conditions [24, 25]. There were also between-group differences in other variables involved in the calculation of the risk of HA-AKI, such as the total percentage of urgent or surgical admissions and the type of surgical intervention performed in each centre. Although not the only one, the most notable difference was related to exposure to cardiac surgery, since this intervention was not performed in the external validation centre.

External validation of a predictive model involves quantifying the model’s discrimination and calibration performance using an external source of data that were not used to develop the model [26]. Discrimination is the ability of a model to differentiate between patients with different outcomes and is usually measured by the AUROC and C statistic. Calibration analyses the agreement between predicted and observed risks, and can be visualized by plotting observed against predicted risks across categories of predicted risk, using a calibration plot with a smooth, non-linear curve [17, 27]. When a predictive model is externally validated, the discrimination power is expected to be lower in the external validation cohort due to overfitting from derivation modelling [28]. Data obtained in our study indicate that, despite the aforementioned differences between both cohorts of patients, the discrimination of the MAKIPS in the external validation cohort was comparable to that reported in the original cohort and was not affected by differences in the prevalence of the variables involved in risk calculation. Moreover, the absence of a significant decrease in discrimination in the external validation cohort indicates that correct adjustment was made to the original score to avoid overfitting.

Calibration of the model in the external validation cohort showed a similar trend to that observed in the derivation cohort. There was a tendency for the MAKIPS to overestimate slightly the risk of HA-AKI at category risks ˂0.19 and to underestimate the risk at category risks between 0.22 and 0.67. In both studies, this overestimating and underestimating tendency could be explained by the fact that the risk of developing HA-AKI depends not only on demographic data, chronic comorbidities and surgical procedures, but also on risk factors related to the inflammatory environment, haemodynamic status and exposure to contrast media or nephrotoxic drugs during the hospital stay, among others [29-31]. This last set of variables involves acute precipitants and may arise throughout the hospitalization period and can lead to relevant changes in the risk profile of patients that cannot be identified with predictive models such as the MAKIPS, which do not include these variables as predictors. The inclusion of dynamic changes of potential acute precipitants into predictive models is technically complex and is a challenge for future research. It could lead to a significant improvement in the discrimination of predictive models and could also generate dynamic predictive models capable of detecting changes in the risk profile of patients throughout their hospital stay.

Notwithstanding all these limitations and with more external validation data still awaited from more hospitals, including wider case mix scenarios, the data from our external validation cohort indicate that the MAKIPS can be a useful tool using data that are easily obtainable from electronic records to predict HA-AKI in hospitals with different case-mix populations.

CONFLICT OF INTEREST STATEMENT

None declared.