A Validation Study of Administrative Health Care Data to Detect Acute Kidney Injury in the Pediatric Intensive Care Unit.

BACKGROUND: Large studies evaluating pediatric acute kidney injury (AKI) epidemiology and outcomes are lacking, partially due to underuse of large administrative health care data.
OBJECTIVE: To assess the diagnostic accuracy of administrative health care data-defined AKI in children admitted to the pediatric intensive care unit (PICU).
DESIGN: Retrospective cohort study utilizing chart and administrative data.
SETTING: Children admitted to the PICU at 2 centers in Montreal, QC. PATIENTS: Patients between 0 and 18 years old with a provincial health insurance number, without end-stage renal disease and admitted to the PICU between January 1, 2003, and March 31, 2005, were included. MEASUREMENTS: The AKI was defined from chart data using the Kidney Disease: Improving Global Outcomes (KDIGO) definition (Chart-AKI). The AKI defined using administrative health data (Admin-AKI) was based on International Classification of Disease, Ninth Revision (ICD-9) AKI codes.
METHODS: Data available from retrospective chart review, including baseline and PICU patient characteristics, and serum creatinine (SCr) and urine output (UO) values during PICU admission, were merged with provincial administrative health care data containing diagnostic and procedure codes used for ascertaining Admin-AKI. Sensitivity, specificity, positive, and negative predictive value of Admin-AKI compared with Chart-AKI (reference standard) were calculated. Univariable associations between Admin-AKI and hospital mortality were evaluated.
RESULTS: A total of 2051 patients (55% male, mean age at admission 6.1 ± 5.8 years, 355 cardiac surgery, 1696 noncardiac surgery) were included. The AKI defined by SCr or UO criteria occurred in 52% of cardiac surgery patients and 24% of noncardiac surgery patients. Overall, Admin-AKI detected Chart-AKI with low sensitivity, but high specificity in cardiac and noncardiac surgery patients. Sensitivity increased by 1.5 to 2 fold with each increase in AKI severity stage. Admin-AKI was associated with hospital mortality (13% in Admin-AKI vs 2% in non-AKI, P < .001). LIMITATIONS: These data were generated in a PICU population; future research should study non-PICU populations.
CONCLUSIONS: Use of administrative health care data to define AKI in children leads to AKI incidence underestimation. However, for detecting more severe AKI, sensitivity is higher, while maintaining high specificity.

What was known before

Acute kidney injury (AKI) is associated with poor short- and long-term outcomes in children. There have been no studies in children evaluating diagnostic accuracy of AKI diagnostic codes from administrative databases.

What this adds

Understanding the accuracy of administrative data for identifying AKI in children, and studying their relationship with outcomes, will strongly enhance future research aimed at determining pediatric AKI health outcomes using administrative data.

Introduction

Acute kidney injury (AKI) in children admitted to the pediatric intensive care unit (PICU) is associated with increased PICU and hospital mortality, prolonged length of stay, and prolonged mechanical ventilation.[1-4] Robust data in adults and emerging data in children suggest that AKI is also a risk factor for long-term renal disease and associated cardiovascular morbidity.[1,2] Most published pediatric AKI research to date has consisted of retrospective chart reviews, which are expensive, time-consuming, and challenging to perform as multicenter studies.

Administrative health care databases are a rich source of population-based information, which may be used to evaluate outcomes of large populations in a cost-effective manner and have fewer logistic barriers compared to traditional prospective cohort designs.[5,6] However, the extent to which research using administrative databases, which mainly use diagnostic and procedure codes, accurately capture AKI diagnosis and reflect associations of AKI with outcomes in children is unclear.

In adults, several studies have evaluated the validity of administrative database AKI diagnostic and procedure codes, by comparing administrative data-defined AKI with the reference standard definition from chart review.[6] In general, administrative databases in adults have shown poor sensitivity, but high specificity to identify AKI.[6] No studies have formally evaluated the diagnostic accuracy of administrative database-defined AKI in critically ill children, despite the fact that recent studies have begun using such methods to study pediatric AKI outcomes.[7] Understanding the accuracy of administrative data for identifying AKI in children, and studying their relationship with outcomes, will strongly enhance future research aimed at determining pediatric AKI health outcomes.

We performed a diagnostic accuracy study of administrative health care data-defined AKI in children admitted to the PICU, using chart review as the reference standard. A secondary objective was to determine the extent to which AKI defined using administrative health care data is associated with hospital mortality, as a surrogate evaluation of prognostic value and validity.

Methods

Study Design and Population

This diagnostic accuracy study was a secondary analysis using data previously collected from a retrospective cohort study of children admitted to 2 tertiary care PICUs (Montreal Children’s Hospital [MCH] and Centre Hospitalier Universitaire de Sainte-Justine [CHUSJ]) between January 1, 2003, and March 31, 2005.[8,9] Inclusion criteria for the original study were PICU admission during the study period and age 0 to 18 years on the day of PICU admission. Only the first admission during the study period per subject was used. Exclusion criteria included a diagnosis of end-stage renal disease (ESRD) (dialysis or transplant) at PICU entry, on the kidney transplant list prior to PICU admission or ESRD diagnosis made during the index hospitalization, or missing a provincial health insurance number. Subjects were further excluded from this analysis if they had neither serum creatinine (SCr) nor urine output (UO) PICU data available from chart review. Approvals from institutional research ethics boards and the Commission d’accès à l’information du Québec (CAI, provincial ethics body) were obtained and requirement for patient consent was waived.

Data Source and Collection for Retrospective Chart Review

Medical charts and electronic medical records of eligible patients were reviewed retrospectively. A standardized case report form was developed by the investigators and iteratively refined by consensus, followed by evaluation for test-retest and interrater reliability. Data collected included baseline patient demographics, PICU diagnostic and illness severity data, PICU medication and complications, and PICU and hospital outcomes. Pertinent to the current study, pre-PICU (baseline) SCr, highest daily PICU SCr, and daily UO values (in 8-hour time periods) were recorded for the purpose of AKI status determination. Baseline SCr was defined as the lowest SCr value in the 3 months before PICU admission. If baseline SCr was unavailable, it was estimated by back-calculation using a previously validated method in children.[10]

Data Sources and Collection for Administrative Health Care Data

Chart data described above from PICU admission was merged with data from the Régie de l’assurance maladie du Québec (RAMQ) and Med-Echo (provincial health care administrative databases), and with the Quebec Vital Statistics Registry database (containing mortality data). The database pertinent to this study is an acute care hospitalization database, including admission and discharge dates and primary diagnoses (International Classification of Disease, Ninth Revision [ICD-9]), physician information, procedures, and up to 15 secondary diagnoses (ICD-9 codes). Individual subject data for chart review were de-identified and merged with the administrative data by an external biostatistician (MD) for analysis, to protect patient confidentiality.

AKI Definition by Reference Standard Chart Review and Administrative Data

Medical chart review served as the reference standard method to define AKI. Chart-defined AKI (referred hereafter as Chart-AKI) was defined in 3 ways based on the Kidney Disease: Improving Global Outcomes (KDIGO) definition (Supplementary Table 1)[11,12]: using SCr alone (Chart-AKISCr), using UO criteria alone (Chart-AKIUO), and using the combined SCr and/or UO criteria (Chart-AKISCrUO). For the latter definition, if a patient fulfilled criteria for either SCr and/or UO, they were classified as having Chart-AKISCrUO. The maximum AKI stage defined by SCr or UO was used to classify Chart-AKI severity (eg, if classified as stage 2 by SCr but stage 3 by UO, the patient was assigned stage 3 Chart-AKISCrUO). We also specifically evaluated dialysis-requiring AKI by chart review. The Chart-AKI definition was based on the KDIGO definition because it is the latest AKI definition developed, is the currently internationally accepted definition, and includes pediatric-specific criteria.[11,13]

To define AKI using administrative health care data, we developed an algorithm including ICD-9 diagnostic and procedure codes. During our study period (2003-2005), Quebec was using ICD-9 codes only; transition to use of International Classification of Diseases, Tenth Revision (ICD-10) codes was in 2006. We began by reviewing literature for similar studies in adults, followed by “pediatric-specific” examination of these administrative data code algorithms, retaining only relevant pediatric diagnostic and procedure codes, to refine the algorithm (face validity). The algorithm evaluated for this study (Supplementary Table 2) only included ICD-9 codes. Pertinent ICD-10 codes identified from our review are shown in Supplementary Table 2 for reference and future research. We primarily evaluated 2 algorithms: one was composed of diagnostic codes related to AKI only (hereafter referred to as Admin-AKI, shown in Supplementary Table 2), and another was composed of AKI diagnostic codes and codes related to the dialysis procedures to define dialysis-requiring AKI (or Admin-AKI+D, shown in Supplementary Table 2).

Clinical Outcome

The primary clinical outcome was mortality during the index hospitalization defined using the Quebec Vital Statistics Registry database.

Statistical Analysis

Characteristics between groups were compared using student t tests, Mann-Whitney U tests, or chi-square tests, depending on data distribution. Sample size requirements were calculated for the original retrospective cohort study. For the current study, the level of precision achievable when estimating sensitivity and specificity was calculated,[14] using data from a recent similar study performed in adults (17% sensitivity and 98% specificity of Admin-AKI to detect Chart-AKI)[15] and known pertinent data in this cohort for the noncardiac surgery PICU patients (approximately 24% prevalence of Chart-AKISCrUO in 1696 patients with available data). This power calculation (assuming α = 0.05) revealed that sensitivity and specificity could be calculated with a ±2% and ±1% level of precision, respectively. Analyses were performed by a biostatistician using SAS statistical software, release 9.2 (SAS Institute Inc, North Carolina). Reporting was performed as per guidelines on reporting of diagnostic accuracy studies and observational cohort studies.[16]

We evaluated the diagnostic characteristics (sensitivity, specificity, negative, and positive predictive values) for Admin-AKI definitions to detect Chart-AKI by each of the Chart-AKI definitions (SCr, UO, or both) and by Chart-AKI severity (any Chart-AKI, Chart-AKI stage 2 or worse, stage 3 Chart-AKI). These analyses were carried out in 2 separate cohorts: (1) patients admitted to the PICU post-cardiac surgery during the hospitalization and (2) noncardiac surgery patients. The decision to analyze these 2 populations was made a priori, as their AKI disease patterns and causes differ significantly, and because health care providers may also differ significantly (eg, cardiologists and cardiac surgeons specifically caring for post-cardiac surgery patients and thus responsible for discharge summaries). Moreover, differing findings between these populations would imply different needs for knowledge translation strategies with regard to AKI diagnosis and education.[17] We performed 2 additional analyses in the noncardiac surgery population only (too few events in the cardiac surgery population). First, we evaluated the diagnostic characteristics of Admin-AKI+D in children with AKI requiring dialysis by chart review. Second, we evaluated the association between AKI and mortality using univariable logistic regression for both Chart-AKI and Admin-AKI definitions (expressed as unadjusted odds ratio [OR] with 95% confidence interval [CI]). We have previously published the multivariable association of Chart-AKI with mortality in this cohort.[9,10] We therefore evaluated the association of Admin-AKI with hospital mortality, adjusting for age and sex.

Results

Study Population and Characteristics

Of 2,499 eligible patients, a further 24 patients were excluded due to missing health number or repeat admissions during the study period (N = 2475, Figure 1). Of these patients, 424 patients had neither an SCr nor a UO measured during PICU admission and were excluded, leaving 2051 patients available for analysis (355 cardiac surgery and 1696 noncardiac surgery patients with either SCr or UO data available to evaluate Chart-AKISCrUO; see Figure 1 for distribution of patients available for different Chart-AKI definition analyses).

Figure 1.

Study flow diagram.

Note. Figure 1 depicts subject selection flow, leading to the final analysis population. In the lower portion of the diagram, numbers of subjects for whom measures were available for the different methods of defining AKI are provided. PICU = pediatric intensive care unit; SCr = serum creatinine; UO = urine output.

Tables 1 and 2 display characteristics of patients with vs without AKISCrUO in the cardiac surgery (Table 1) and noncardiac surgery (Table 2) cohorts. As shown, in both cohorts, variables relating to illness severity (calculated death risk, use of mechanical ventilation, vasopressors and nephrotoxic medication, longer mechanical ventilation, and prolonged length of stay) were higher in the AKISCrUO groups (Tables 1 and 2). In the cardiac surgery group only, patients with AKISCrUO were also significantly younger (Table 1).

Table 1.

Baseline and PICU Patient Characteristics of the Cardiac Surgery Study Cohort.

Characteristic	AKISCrUO n = 185	No AKISCrUO n = 170
Age at PICU admission, mean (SD) years	3.0 (4.8)[a]	3.3 (4.1)
Male, n (%)	108 (58.4)	94 (55.3)
CHUSJ study site (vs MCH), n (%)	107 (57.8)	83 (48.8)
PRISM calculated death rate, mean (SD)	19.2 (18.0)[a]	11.3 (11.6)
Invasive mechanical ventilation, n (%)	183 (98.9)[a]	151 (88.8)
Days of invasive mechanical ventilation, mean (SD)	5.6 (5.3)[a]	2.8 (3.3)
Vasopressors, n (%)	154 (83.2)[a]	104 (61.2)
Nephrotoxic antibiotics,[b] n (%)	60 (32.4)[a]	18 (10.6)
Steroids, n (%)	34 (18.4)[a]	16 (9.4)
Nephrology consultation, n (%)	18 (9.7)[a]	5 (2.9)
Renal replacement therapy, n (%)	1 (0.5)	0 (0.0)
Length of PICU stay, mean (SD) days	10.5 (20.4)[a]	4.7 (7.5)
Length of hospital stay, mean (SD) days	26.7 (39.1)[a]	17.4 (35.1)

Note. PICU = pediatric intensive care unit; AKISCrUO = acute kidney injury defined using serum creatinine and urine output criteria; CHUSJ = Centre Hospitalier Universitaire de Sainte-Justine; MCH = Montreal Children’s Hospital; PRISM calculated death rate = the logarithmic equation calculation of death rate using the Pediatric Risk of Mortality Score.[18]

Denotes that there was a statistically significant difference between AKI and non-AKI group for the variable.

Nephrotoxic antibiotics for which data were collected included aminoglycosides, vancomycin, nephrotoxic anti-virals including acyclovir, and amphotericin B.

Table 2.

Baseline and PICU Patient Characteristics of the Noncardiac Surgery Study Cohort.

Characteristic	AKISCrUO n = 406	No AKISCrUO n = 1290
Age at PICU admission, mean (SD) years	7.3 (5.9)[a]	6.5 (5.9)
Male, n (%)	222 (54.7)	712 (55.2)
CHUSJ study site (vs MCH), n (%)	295 (72.7)[a]	817 (63.3)
PRISM calculated death rate, mean (SD)	12.2 (20.7)[a]	4.7 (8.8)
Primary PICU admission reason, n (%)
Cardiac (nonsurgical)	31 (7.6)	72 (5.6)
Trauma	45 (11.1)	155 (12.0)
Renal	15 (3.7)[a]	4 (0.3)
Infection (excludes bronchiolitis)	98 (24.1)[a]	252 (19.5)
Neurological/neurosurgical	46 (11.3)[a]	222 (17.2)
Gastroenterological	33 (8.1)[a]	46 (3.6)
Oncological (nontransplant)	16 (3.9)	44 (3.4)
Respiratory	44 (10.8)	126 (9.8)
Diabetes	18 (4.4)[a]	17 (1.3)
Other	60 (14.8)[a]	352 (27.3)
Post-operative (noncardiac), n (%)	111 (27.3)[a]	485 (37.6)
Invasive mechanical ventilation, n (%)	232 (57.1)[a]	526 (40.8)
Days of invasive mechanical ventilation, mean (SD)	4.1 (6.1)[a]	1.8 (3.5)
Vasopressors, n (%)	119 (29.3)[a]	90 (7.0)
Nephrotoxic antibiotics,[b] n (%)	161 (39.7)[a]	292 (22.6)
Steroids, n (%)	152 (37.4)[a]	346 (26.8)
Nephrology consultation, n (%)	94 (23.2)[a]	51 (4.0)
Renal replacement therapy, n (%)	17 (4.2)[a]	0 (0.0)
Length of PICU stay, mean (SD) days	9.3 (35.3)[a]	3.2 (7.1)
Length of hospital stay, mean (SD) days	31.8 (58.6)[a]	19.4 (57.9)

Note. PICU = pediatric intensive care unit; AKISCrUO = acute kidney injury defined using serum creatinine and urine output criteria; CHUSJ = Centre Hospitalier Universitaire de Sainte-Justine; MCH = Montreal Children’s Hospital; PRISM calculated death rate = the logarithmic equation calculation of death rate using the Pediatric Risk of Mortality Score.[18]

Denotes that there was a statistically significant difference between AKI and non-AKI group for the variable.

Nephrotoxic antibiotics for which data were collected included aminoglycosides, vancomycin, nephrotoxic anti-virals including acyclovir, and amphotericin B.

Diagnostic Characteristics of Admin-AKI to Detect Chart-AKI in Cardiac Surgery Patients

In the cardiac surgery population, 170 of 355 (47.9%) patients developed Chart-AKISCr, 51 of 204 (25.0%) had Chart-AKIUO, and 185 of 355 (52.1%) developed Chart-AKISCrUO in the PICU. Only one cardiac surgery patient was treated with dialysis. Of the ICD-9 Admin-AKI diagnostic codes shown in Supplementary Table 2, all cardiac surgery patients with Admin-AKI attained a diagnosis by either one of the AKI codes, AKI unspecified, or a combination of both codes. None of the Admin-AKI patients had codes for abnormal renal function, Pregnancy-AKI, or glomerulonephritis (GN).

Table 3 shows that in cardiac surgery patients, Admin-AKI detected the presence of Chart-AKI with low sensitivity, but high specificity. Overall, the sensitivity of Admin-AKI to detect AKIUO was higher than for detecting AKISCr, with minimal impact on specificity (shown throughout Table 3). The diagnostic characteristics of Admin-AKI to detect AKISCrUO were extremely similar to those for detecting AKISCr (right side of Table 3).

Table 3.

Diagnostic Characteristics of Administrative Data-Defined AKI (Admin-AKI) in Post-Cardiac Surgery Patients.

Detecting any Chart-AKITotal n = 355 for AKISCr and for AKISCrUO; n = 204 for AKIUO
Diagnostic characteristic[a]	SCr criteria170 AKI events	UO criteria51 AKI events	SCr or UO criteria185 AKI events
Sensitivity	5.9% (4.6-7.6)	13.7% (10.1-18.5)	5.9% (4.7-7.6)
Specificity	99.5% (98.6-99.8)	98.5% (97.5-99.2)	100% (99.2-100)
PPV	90.9% (77.3-97.4)	70.0% (54.2-82.4)	100% (88.2-100)
NPV	53.5% (51.5-55.4)	82.0% (80.0-83.8)	49.4% (47.5-51.4)
Detecting Chart-AKI stage 2 or worse
	SCr criteria64 AKI events	UO criteria11 AKI events	SCr or UO criteria64 AKI events
Sensitivity	14.1% (10.8-18.2)	18.2% (8.7-32.6)	14.1% (10.8-18.2)
Specificity	99.3% (98.7-99.7)	96.7% (95.6-97.6)	99.3% (98.7-99.7)
PPV	81.8% (67.4-91.2)	20.0% (9.6-35.5)	81.8% (67.4-91.2)
NPV	84.0% (82.5-85.4)	96.3% (95.2-97.2)	84.0% (82.5-85.4)
Detecting Chart-AKI stage 3
	SCr criteria32 AKI events	UO criteria7 AKI events	SCr or UO criteria35 AKI events
Sensitivity	15.6% (10.6-22.3)	14.3% (4.0-34.1)	14.3% (9.7-20.5)
Specificity	98.1% (97.4-98.7)	96.4% (95.2-97.2)	98.1% (97.3-98.7)
PPV	45.5% (31.7-59.8)	100% (2.8-24.7)	45.5% (31.7-59.8)
NPV	92.2% (90.9-93.2)	97.6% (96.5-98.3)	91.3% (90.0-92.4)

Note. Chart-AKI = AKI defined by medical chart review (reference standard); AKISCr = acute kidney injury defined using serum creatinine criteria; AKIUO = acute kidney injury defined using urine output criteria; AKISCrUO = acute kidney injury defined using serum creatinine and urine output criteria; SCr = serum creatinine; UO = urine output; PPV = positive predictive value; NPV = negative predictive value.

Estimates of sensitivity, specificity, PPV, and NPV are presented with 95% confidence intervals.

With increasing severity of Chart-AKI definition (eg, stage 2 or worse AKI), the sensitivity of Admin-AKI to detect Chart-AKI generally increased (to a range of 14%-18%), with slight reduction in specificity (Table 3).

Diagnostic Characteristics of Admin-AKI to Detect Chart-AKI in Noncardiac Surgery Patients

In the noncardiac surgery population, 345 of 1648 (20.9%) patients developed Chart-AKISCr in the PICU, 109 of 760 (14.3%) had Chart-AKIUO, and 406 of 1696 (24.1%) had Chart-AKISCrUO. There were 17 noncardiac surgery patients with dialysis-requiring AKI. Of ICD-9 diagnostic codes shown in Supplementary Table 2, all Admin-AKI patients attained Admin-AKI status with codes for AKI, AKI unspecified, or a combination of both codes. One patient had a diagnosis of GN, but also had a diagnostic code for AKI. No patients fulfilled Admin-AKI criteria with the abnormal renal function or Pregnancy-AKI codes.

Table 4 shows that in noncardiac surgery patients, the overall pattern of low sensitivity and high specificity of Admin-AKI to detect Chart-AKI was similar to what was seen in cardiac surgery patients. However, overall, sensitivity for detecting Chart-AKI was higher (as high as 70% for detecting stage 3 AKIUO, Table 4). Similarly, as severity of Chart-AKI increased, the sensitivity and negative predictive value for Admin-AKI to detect Chart-AKI increased while the specificity and positive predictive value decreased (Table 4).

Table 4.

Diagnostic Characteristics of Administrative Data-Defined AKI (Admin-AKI) in Noncardiac Surgery Patients.

Detecting any Chart-AKITotal n = 1648 for AKISCr; n = 760 for AKIUO; n = 1696 AKISCrUO
Diagnostic characteristic[a]	SCr criteria345 AKI events	UO criteria109 AKI events	SCr or UO criteria406 AKI events
Sensitivity	15.7% (14.3-17.2)	20.2% (17.3-23.4)	13.8% (12.6-15.1)
Specificity	98.9% (98.7-99.1)	95.2% (94.5-95.8)	99.1% (98.8-99.3)
PPV	79.4% (75.1-83.1)	41.5% (36.2-47.1)	82.4% (78.2-85.8)
NPV	81.6% (80.9-82.3)	87.7% (86.8-88.6)	78.5% (77.8-79.2)
Detecting Chart-AKI stage 2 or worse
	SCr criteria163 AKI events	UO criteria37 AKI events	SCr or UO criteria178 AKI events
Sensitivity	25.8% (23.3-28.5)	46.0% (39.2-52.8)	23.6% (21.3-26.1)
Specificity	98.2% (98.0-98.5)	95.0% (94.4-95.6)	98.3% (98.0-98.5)
PPV	61.8% (57.0-66.3)	32.1% (27.1-37.6)	61.8% (57.0-66.3)
NPV	92.3% (91.8-92.8)	97.2% (96.6-97.6)	91.7% (91.1-92.1)
Detecting Chart-AKI stage 3
	SCr criteria81 AKI events	UO criteria23 AKI events	SCr or UO criteria83 AKI events
Sensitivity	43.2% (39.0-47.6)	69.6% (60.4-77.4)	42.2% (38.0-46.5)
Specificity	97.9% (97.6-98.1)	95.0% (94.3-95.5)	98.0% (97.7-98.2)
PPV	51.5% (46.7-56.2)	30.2% (25.3-35.6)	51.5% (46.7-56.2)
NPV	97.1% (96.8-97.4)	99.0% (98.6-99.3)	97.1% (96.7-97.3)

Note. Chart-AKI = AKI defined by medical chart review (reference standard); AKISCr = acute kidney injury defined using serum creatinine criteria; AKIUO = acute kidney injury defined using urine output criteria; AKISCrUO = acute kidney injury defined using serum creatinine and urine output criteria; SCr = serum creatinine; UO = urine output; PPV = positive predictive value; NPV = negative predictive value.

Estimates of sensitivity, specificity, PPV, and NPV are presented with 95% confidence intervals.

Of the 17 patients who received dialysis for AKI (determined by chart review), 14 patients were identified as having Admin-AKI+D using administrative data. The sensitivity, specificity, positive, and negative predictive values for Admin-AKI+D to detect dialysis-requiring AKI were 82.4% (95% CI = 71.8%-89.7%), 99.6% (95% CI = 99.5%-99.7%), 70.0% (95% CI = 60.0%-78.4%), and 99.8% (95% CI = 99.7%-99.9%), respectively.

Association of Admin-AKI With Hospital Mortality

Only 5 patients in the cardiac surgery population died during admission; therefore, this analysis was focused on the noncardiac surgery population only. In the noncardiac surgery population, 74 of 1696 (4.4%) patients died during index admission. Table 5 shows that regardless of how Chart-AKI was defined (using SCr, UO, or both criteria), there was a strong association of Chart-AKI with hospital mortality (Table 5). There was also a strong association between Admin-AKI and Admin-AKI+D with hospital mortality (Table 5, unadjusted ORs approximately 5 and 10, respectively, P < .0001). The association between Admin-AKI and hospital mortality remained statistically significant when adjusting for age and sex (adjusted OR = 6.0, 95% CI = 3.0-11.9).

Table 5.

Hospital Mortality in Noncardiac Surgery Patients With Versus Without AKI Defined by Chart Review (Chart-AKI) and by Administrative Health Data (Admin-AKI), With AKI Versus Non-AKI Odds Ratio.

Definition method	Mortality n (%) with AKI	Mortality n (%) without AKI[a]	Odds ratio (95% CI)[b]
Chart-AKISCr
Any AKI	46/345 (13.3%)	27/1303 (2.1%)	7.3 (4.5-11.9)
Stage 2 or worse AKI	34/163 (20.9%)	39/1485 (2.6%)	9.8 (6.0-16.0)
Stage 3 AKI	23/81 (28.4%)	50/1567 (3.2%)	12.0 (6.9-21.0)
Chart-AKIUO
Any AKI	19/109 (17.4%)	18/651 (2.8%)	7.4 (3.8-14.7)
Stage 2 or worse AKI	9/37 (24.3%)	28/723 (3.8%)	8.0 (3.4-18.5)
Stage 3 AKI	8/23 (34.8%)	29/737 (3.9%)	13.0 (5.1-33.2)
Chart-AKISCrUO
Any AKI	51/406 (12.6%)	23/1290 (1.8%)	7.9 (4.8-13.1)
Stage 2 or worse AKI	36/178 (20.2%)	38/1518 (2.5%)	9.9 (6.1-16.1)
Stage 3 AKI	24/83 (28.9%)	50/1613 (3.1%)	12.7 (7.3-22.1)
Admin-AKI	12/68 (17.7%)	62/1628 (3.8%)	5.4 (2.8-10.6)
Admin-AKI requiring dialysis	6/20 (30.0%)	68/1676 (4.1%)	10.1 (3.8-27.2)

Note. AKISCr = acute kidney injury defined using serum creatinine criteria; AKIUO = acute kidney injury defined using urine output criteria; AKISCrUO = acute kidney injury defined using serum creatinine and urine output criteria; Admin-AKI+D = acute kidney injury with renal replacement therapy treatment, defined by administrative health data (diagnostic codes plus dialysis procedure codes); CI = confidence interval.

Without AKI refers to no AKI as per the definition method used for analysis (eg, for stage 2 AKI or worse, without AKI refers to patients with no AKI or stage 1 AKI, or no stage 2 AKI).

Odds ratio calculated using logistic regression. All odds ratios are statistically significant.

Discussion

Our results demonstrate that administrative health data detect the presence of chart-defined AKI with low sensitivity in both cardiac and noncardiac surgery patients. With increasing severity of Chart-AKI staging, the sensitivity and negative predictive value of Admin-AKI to detect Chart-AKI increase, with a minor reduction in specificity and larger reduction in positive predictive value. Despite its low sensitivity, Admin-AKI was highly specific and associated with hospital mortality, a well-studied outcome associated with pediatric AKI,[1-4] which may support the use of administrative health care databases to study more severe AKI in children.

Diagnostic accuracy studies of AKI diagnostic and procedure codes in adult populations have shown very similar results, namely, a poor sensitivity (11%-17%), but high specificity (>98%) to identify AKI, with increasing sensitivity to detect more severe AKI.[15] However, we noted some differences from previously published adult studies. Contrary to a recent, large adult study, in both cardiac surgery and noncardiac surgery patients, we found that the sensitivity of Admin-AKI to detect Chart-AKI was higher when using only the UO criteria to define Chart-AKI (vs using SCr criteria).[15] A recent multinational study of pediatric AKI in the PICU showed that oliguria was more strongly associated with hospital mortality than SCr-defined AKI.[19,20] Thus, in children, when oliguria is present, often leading to fluid overload and need for intervention (eg, diuretics), it is possible that the presence of AKI is more highly noted by physicians and therefore recorded in discharge summaries.

We found that with each increase in Chart-AKI severity stage, the sensitivity of Admin-AKI increased 1.5 to 2 fold; this rise in sensitivity was associated with a minimal decrease in specificity. Future studies aimed at defining highly accurate prevalence data or examining AKI outcomes for which there is little knowledge or data published should consider the fact that using Admin-AKI data will likely only reflect more severe forms of AKI and that non-Admin-AKI control groups will contain substantial proportions of patients with mild AKI. Future research should aim to identify other variables (eg, non-AKI-related diagnostic/procedure codes) associated with AKI (and non-AKI) to further help improve the sensitivity of Admin-AKI detection.

We found substantial differences between the cardiac and noncardiac surgery populations. In pediatrics, cardiac surgery patients may often be in different PICUs or even have different ICU specialty physicians caring for them. For example, surgeons tend to be highly implicated in the care of post-operative cardiac surgery patients and not noncardiac surgery patients. Cardiac surgery patients are often found to have SCr rise or oliguria very shortly after surgery (sometimes immediately post-operatively); this is well known to occur and most often transient.[21,22] This commonplace immediate post-operative SCr rise or oliguria in these patients may explain why the sensitivity of Admin-AKI to detect Chart-AKI was substantially lower in the cardiac surgery patients; when AKI does occur immediately post-operatively in this patient population, it may simply seem less noteworthy to health care providers. Despite these findings, we cannot exclude the possibility that our reduced sample size of cardiac surgery patients contributed to lack of power and reduced estimates of sensitivity due to chance alone. Moreover, it was not possible in this study to determine the extent to which health care providers considered immediate post-operative SCr rise or oliguria more or less indicative of true AKI, compared with later appearance of AKI in the PICU.

To our knowledge, this is the first published detailed Admin-AKI algorithm, aimed at use in children admitted to the PICU. A recent study published on hospitalized children with nephrotoxin-associated AKI also evaluated an AKI diagnostic algorithm utilizing ICD-9 discharge diagnoses.[23] This study similarly demonstrated that sensitivity for detection of AKI by SCr criteria was low (approximately 20%) and specificity was high (approximately 95%) but did not evaluate different AKI severity levels and was focused on nephrotoxin-mediated AKI. Although we only evaluated validity of ICD-9 codes due to our pre-ICD-10 study period, our study provides a more extended algorithm for consideration by future researchers, including ICD-9 and ICD-10 codes.[17] Our algorithm also provides corresponding procedural codes for dialysis, thereby enhancing the sensitivity for detecting dialysis-requiring (most severe) AKI. Our study was limited to patients admitted to the PICU, the highest risk patient population for poor AKI outcomes.

This study had limitations. First, although this is one of the largest published retrospective PICU AKI cohorts, the sample size for the cardiac surgery cohort was low. This study included data from 2 hospitals within the same geographic region, limiting generalizability. Center-specific practice patterns (eg, fluid management; acute dialysis initiation practice; urinary catheter insertion and removal practice) and center-specific beliefs on what constitutes a diagnosis of AKI may impact ascertainment of both Chart-AKI and Admin-AKI. It is thus important that similar research in other geographic regions be performed to determine the extent to which our results are generalizable. The cohort included patients admitted between 2003 and 2005, because the original goal of the study was to evaluate long-term outcomes of AKI in children admitted to the PICU using administrative health care data (therefore, we chose a somewhat older cohort, to have a minimum of 5 years follow-up after PICU admission). It is conceivable that awareness of AKI, and thus inclusion of AKI as a discharge or admission diagnosis on discharge summaries, has improved in the last decade and that sensitivity of Admin-AKI may be higher today. A previous adult study from the United States showed that billing codes for AKI have increased in recent years.[15] However, there have not been any specific knowledge translation or awareness enhancement interventions performed to target increased identification of AKI at our centers. Our analysis included only patients who had SCr or UO data collected during PICU admission. This likely led to a study population biased toward patients with higher illness severity. However, it is those patients who are of higher interest in terms of evaluating AKI outcomes and identifying the presence of AKI using administrative health data.

In conclusion, we have shown that using administrative health data including hospitalization admission and discharge diagnostic codes and dialysis procedural codes to identify the presence of AKI in the PICU is highly specific, but not sensitive. It is therefore important to exercise caution when using administrative data to assign AKI diagnosis and/or evaluate its association with patient outcomes and to consider the impact of this AKI ascertainment method’s strength and limitations on specific study objectives. Studies targeting more severe forms of AKI can improve sensitivity of Chart-AKI detection as high as approximately 70% and studies including noncardiac surgery patients will also have higher likelihood of better sensitivity for Chart-AKI detection. Our study highlights a probable lack of awareness of AKI as an important health care problem in PICU patients, on the part of PICU health care providers, as evidenced by the common occurrence of lack of AKI diagnostic coding in patients with mild and moderate AKI. Future research should include intervention studies aimed at increasing awareness of AKI and encouraging the identification of AKI in critically ill patients. We also found that Admin-AKI is associated with hospital mortality, albeit with somewhat lower magnitude to what is known for Chart-AKI. Although this finding supports that Admin-AKI may have prognostic value, more research is needed with larger sample sizes to determine whether this association holds in large multivariable models (considering comorbidities, acute illness severity, and treatment variables). Future research should also determine the extent to which Admin-AKI in children is associated with post-hospital discharge outcomes including health care utilization, chronic kidney disease, and mortality, to evaluate validity and prognostic value of Admin-AKI for studying AKI outcomes.

Click here for additional data file.

Supplemental material, Supplementary_material_DArienzo_et_al_Revision_2_Clean for A Validation Study of Administrative Health Care Data to Detect Acute Kidney Injury in the Pediatric Intensive Care Unit by David D’Arienzo, Erin Hessey, Rami Ali, Sylvie Perreault, Susan Samuel, Louise Roy, Jacques Lacroix, Philippe Jouvet, Genevieve Morissette, Marc Dorais, Jean-Philippe Lafrance, Veronique Phan, Michael Pizzi, Rahul Chanchlani and Michael Zappitelli in Canadian Journal of Kidney Health and Disease