Creatinine Assay Attainment of Analytical Performance Goals Following Implementation of IDMS Standardization: Further Improvements Required.

BACKGROUND: The international initiative to standardize creatinine (Cr) assays by tracing reference materials to Isotope Dilution Mass Spectrometry (IDMS) assigned values was implemented to reduce interlaboratory variability and improve assay accuracy.
OBJECTIVE: The aims of this study were to examine whether IDMS standardization has improved Cr assay accuracy (bias), interlaboratory variability (precision), total error (TE), and attainment of recommended analytical performance goals.
METHODS: External Quality Assessment (EQA) data (n = 66 challenge vials) from Ontario, Canada, were analyzed. The bias, precision, TE, and the number of EQA challenge vials meeting performance goals were determined by assay manufacturer before (n = 32) and after (n = 34) IDMS implementation.
RESULTS: The challenge vials with the worst bias and precision were spiked with known common interfering substances (glucose and bilirubin). IDMS standardization improved assay bias (10.4%-1.6%, P < .001), but precision remained unchanged (5.0%-4.7%, P = .5) with performance goals not consistently being met. Precision and TE goals based on biologic variation were attained by only 29% to 69% and 32% to 62% of challenge vials.
CONCLUSIONS: While IDMS standardization has improved Cr assay accuracy and thus reduced TE, significant interlaboratory variability remains. Contemporary Cr assays do not currently meet the standards required to allow for accurate and consistent estimated glomerular filtration rate assessment and chronic kidney disease diagnosis across laboratories. Further improvements in Cr assay performance are needed.

What was known before

Analytical performance limitations of serum creatinine (Cr) assays have been long recognized. Standardization of Cr assays using Isotope Dilution Mass Spectrometry (IDMS) assigned reference materials is an international initiative to improve assay accuracy and reduce interlaboratory variability. Data on the impact of this initiative on Cr assay performance and attainment of analytical performance goals are lacking.

What this adds

Our study demonstrates that interlaboratory variation of Cr assays still exists despite IDMS standardization and that recommended analytical performance goals are not consistently being met. The results demonstrate the need for further analytical improvements in Cr measurement to allow for consistent and more accurate chronic kidney diagnosis and serial monitoring of kidney function across laboratories.

Background

Serum or plasma creatinine (Cr) is the most common biomarker used to assess kidney function and is included in most glomerular filtration rate (GFR) estimation equations.[1,2] In many countries, laboratories report an estimated GFR (eGFR) when Cr is ordered. Among other things, the eGFR is used to diagnose, classify, and monitor the progression of chronic kidney disease (CKD).

Inaccuracy (bias) and interlaboratory variability (precision) in Cr have been well documented and discussed.[3-5] The analytical variability is multifactorial and attributed to differences in (1) methods (Jaffe vs enzymatic) with different biases, method precision, and susceptibility to interferences (medications, other endogenous molecules such as bilirubin, glucose, ketones, proteins); (2) assay manufacturers with heterogeneous reference materials and procedures; (3) platforms (instruments, models) with different performance characteristics; and (4) other inherent measurement uncertainties.[3,6,7]

Nonstandardized calibration of Cr assays was identified as one of most important modifiable factors contributing to measurement inaccuracy and interlaboratory variability in the 2006 National Kidney Disease Education Program (NKDEP) Laboratory Working Group report.[3] To address this, Isotope Dilution Mass Spectrometry (IDMS) was chosen as a high-order reference method, and the practice of tracing assay reference materials against IDMS-assigned values has since been progressively implemented internationally by most manufacturers.[6,8]

Another potentially modifiable factor contributing to interlaboratory variability and assay inaccuracy is assay specificity (ability to measure unequivocally the analyte of interest in the presence of other components that may be present).[3,7] Assay nonspecificity is a well-documented problem with Cr assays. There are numerous known endogenous and exogenous substances that interfere with Cr assays.[3,7] At different times, individual patients may have different types and amounts of interfering substances, and their impact will differ at different concentrations and with different assay types.[3,7] Enzymatic methods have been shown to have an improved specificity as compared with Jaffe methods and require smaller sample volumes but are considerably more expensive.[6,7,9] It is important to note that the nonspecificity and other contributors to assay precision (pipetting, reagents, temperature fluctuations, etc) are not expected to be impacted by IDMS standardization.[3]

Overall assay performance is assessed by its total error (TE), which combines both accuracy and precision. Total error allowable (TEa) performance goals (minimum, desirable, and optimum) are often based on the analyte’s day-to-day biologic variation.[10] For Cr, the “desirable” proposed TEa is <7.0%, with bias <3.4% and analytical precision <2.2%.[10] In 2006, the NKDEP Laboratory Working Group advocated for more “realistic” goals of <5% for bias and <8% for precision for Cr values >88.4 µmol/L.[3] This translates to a TEa of 18% for Cr and a “clinically acceptable” corresponding TEa of <10% for eGFRs <60 mL/min/1.73 m2.[3] They do not specify performance goals for lower Cr values, commenting that manufacturers should improve assay precision for Cr <88.4 µmol/L.[3]

Using laboratory data from Canada before IDMS standardization, Komenda et al[5] showed a very high average TE of 24% and significant interlaboratory variability (TE ranging from 4% to 54%). Data from a variety of other countries suggest that IDMS standardization has not always resulted in consistent improvement in Cr measurement.[7-9,11,12] No study has examined whether performance goals for Cr are currently being attained on a large scale.

The aims of this study were to determine whether implementation of IDMS standardization has reduced interlaboratory variability and resulted in improved attainment of recommended TEa goals by analyzing the Quality Management Program–Laboratory Services (QMP-LS), External Quality Assessment (EQA) Cr data before and after IDMS implementation in Ontario, Canada.[13] QMP-LS is a mandated provincially funded proficiency testing program which is responsible for assessing the quality of all licensed Ontario laboratories and since 2014 is called the Institute for Quality Management in Healthcare.[14]

Methods

Data

Data were recorded from the QMP-LS EQA program survey reports from 2004 to 2012.[13] QMP-LS provides 3 challenge vials of fresh frozen sera to approximately 200 laboratories in Ontario in January, May, and October of each year. For each challenge vial, it compiles the resulting data by manufacturers and/or methods and compares the measured values against reference values, all methods’ means (AMM), or peer group means (PGM).[13] Data for individual laboratories are not available in the generic survey reports. QMP-LS started using IDMS-assigned Cr values for each challenge vial in May 2008. Prior to this change, reference values were obtained by high-performance liquid chromatography (HPLC). Between May 2008 and January 2010, there was a transition period during which Ontario laboratories converted to IDMS traceable methods.

Analysis

Data from all challenge vials between 2004 and 2012 were collected, collated, and analyzed. For each challenge vial, we compared the AMM and PGM with the assigned reference value examining both bias and precision. The bias was assessed as the percentage difference [bias = (AMM − reference value) × 100 / reference value]. The precision was assessed as the coefficient of variation (CVa = standard deviation / AMM) × 100). The same calculations and comparisons of Cr data were done separately for each of the 3 main manufacturers (PGM) in use during this time period—Ortho Clinical Diagnostics (OCD), Beckman Coulter, and Roche BMC—and TE was calculated (TE = |(% bias)| + 1.65 × CVa) for each manufacturer.[10] Mean bias, CVa, and TE were compared pre-IDMS (prior to May 2008) and post-IDMS (after May 2008) using 2-tailed t tests assuming unequal variances.

To examine with greater granularity, the impact of IDMS traceability, we calculated overall method bias, precision, and TE in IDMS versus non-IDMS traceable laboratories during the transitional period between May 2008 and January 2010 (n = 16 challenge samples). This provides us with data that have less contamination than the single time point of May 2008.

QMP-LS data do not allow for the determination of the number of individual laboratories achieving performance goals. However, the number and percentage of challenge vials where the specific manufacturer bias, CVa, and TE met biologic variation (TEa < 7.0%, bias < 3.4%, and precision < 2.2%), and NKDEP performance goals (TEa < 18%, bias < 5%, and precision < 8% for Cr > 88.4 µmol/L) before and after May 2008 were calculated.[3,10] A 2-tailed Fisher’s exact test was used to compare the proportions.

Results

There were 72 QMP-LS challenge vials between January 2004 and January 2012. Six challenge vials were excluded from the analysis as reference values were not provided on survey reports. The final study results are therefore based on data from a total of 66 challenge vials. The mean number of participating laboratories was 186 ± 11. The 3 manufacturers that were most represented per challenge vial were OCD (n = 105 ± 15), Beckman Coulter (n = 28 ± 3), and Roche BMC (n = 18 ± 3). There were 31 ± 15 laboratories using a total of 4 other manufacturers per challenge vial.

Bias

All method and by manufacturer biases are shown in Figure 1a and Table 1. The AMM bias fell from 10.4% ± 4.2% prior to May 2008 to 1.6% ± 3.3% after May 2008 (P < .001). Among the manufacturers, OCD improved the most with its mean bias decreasing from 13.6% to 3.8%, while the mean Beckman Coulter bias changed from +5.8% to −4.3%. There were 2 challenge vials with particularly elevated AMM biases: one challenge vial (#25, Cr 86 µmol/L) was spiked with bilirubin (108 µmol/L) and the other (#53, Cr 79 µmol/L) had an elevated glucose (16.6 mmol/L) (Figure 1a). Both had low Cr values, where the impact of interferences is more pronounced.[3,7]

Figure 1.

Bias (a) and precision (b) of creatinine AMM for test vials from January 2004 to January 2012.

Note. Solid lines (mean) and dashed lines (standard deviation) before (●) and after (○) May 2008. Arrows indicate challenge vials with very poor biases and precisions and known interfering substances. AMM = all methods’ means.

Table 1.

QMP-LS Bias, Precision (CVa), and TE Before and After May 2008.

Instrument	Mean % bias (SD)			Mean CVa (%)			Mean TE (%)
	Pre-May 2008	Post-May 2008	P value	Pre-May 2008	Post-May 2008	P value	Pre-May 2008	Post-May 2008	P value
All methods[a]	10.4 (4.2)	1.6 (3.3)	<.001	5.0	4.7	.582	18.6	10.5	<.001
Ortho Clinical[b] Diagnostics (enzymatic)	13.6 (6.8)	3.8 (4.2)	<.001	3.0	2.1	.040	18.6	7.5	<.001
Beckman Coulter[b] (modified Jaffe)	5.8 (3.8)	−4.3 (7.0)	<.001	4.0	3.8	.801	12.3	12.7	.825
Roche BMC[b] (enzymatic and modified Jaffe)	4.3 (4.3)	−0.1 (4.2)	<.001	3.8	3.0	.109	10.0	8.3	.160

Note. QMP-LS = Quality Management Program–Laboratory Services; CVa = coefficient of variation; TE = total error.

All methods: n = 32 challenge vials pre-May 2008 and n = 34 challenge vials post-May 2008.

Specific manufacturers: % Bias (n = 23 challenge vials pre-May 2008, n = 34 challenge vials post-May 2008), CVa, and TE (n = 20 challenge vials pre-May 2008, n = 34 challenge vials post-May 2008). The number of challenge vials pre-May 2008 for specific manufacturer is lower than for all methods as QMP-LS reports did not present results by manufacturer for 12 of the challenge vials.

Precision

Summary data for precision are shown in Figure 1b and Table 1. There was no improvement in overall precision with the mean CVa remaining unchanged (5.0% vs 4.7%) pre- and post-May 2008 (P = .58). Only 1 of the 3 main manufacturers (OCD) showed an improvement in assay precision. Examination of Figure 1b reveals 3 challenge vials (#25, #27, #31), where the mean precision was very poor (CVa > 10%). All 3 of these challenge vials were spiked with bilirubin (108, 245, and 148 µmol/L) and had low Cr values (86, 119, and 96 µmol/L).

Total Error

TE improved significantly only for OCD from 18.6% to 7.5% (P < .001) pre- and post-May 2008 (Table 1).

Transition Time Bias/Precision/TE

There were 16 challenge vials during the transition period with available data according to whether laboratories had IDMS traceable assays or not (Table 2). No significant change in precision was observed when assessed by the manufacturer. Bias and TE improved with IDMS traceability for OCD only.

Table 2.

QMP-LS Bias, Precision (CVa), and TE During IDMS Transition (May 2008-January 2010); 184 to 188 Laboratories; 10 to 16 Challenge Vials.

Instrument	Mean % bias (SD)			Mean CVa (%)			Mean TE (%)
	No IDMS	With IDMS	P value	No IDMS	With IDMS	P value	No IDMS	With IDMS	P value
Ortho Clinical Diagnostics (enzymatic)	17.9 (11.8)	3.9 (4.8)	.002	2.3	2.1	.295	22.4	7.5	<.001
Beckman Coulter (modified Jaffe)	3.8 (5.0)	−4.3 (8.8)	.010	4.2	4.2	.982	11.4	15.0	.336
Roche BMC (enzymatic and modified Jaffe)	NA[a]	1.2 (3.3)		NA[a]	3.1		NA[a]	7.7

Note. Data are presented for laboratories with and without IDMS standardization. Ortho Clinical Diagnostics: No IDMS, n = 12 challenge vials; With IDMS, n = 14 challenge vials. Beckman Coulter: No IDMS, n = 10 challenge vials; With IDMS, n = 14 challenge vials. Roche BMC: With IDMS, n = 16 challenge vials. aRoche BMC had already transitioned to IDMS traceable assays by May 2008. QMP-LS = Quality Management Program–Laboratory Services; CVa = coefficient of variation; TE = total error; IDMS = Isotope Dilution Mass Spectrometry; NA = not available.

Analytical Performance Goals

Before May 2008, the bias, precision, and TE were within desirable performance goals based on biologic variation in only a small minority of challenge vials (Table 3). OCD’s enzymatic assay using dry slide technology did not meet bias goals in any challenge vial before May 2008 but this increased to 65% after May 2008. Roche BMC assays achieved bias targets more frequently than the other manufacturers before May 2008 likely because they had already instituted IDMS traceability by then. Beckman Coulter bias did not improve significantly. Beckman Coulter had the poorest results with an average TEa target achieved in 15% (pre-May 2008) and 32% (post-May 2008) of challenge vials. Overall, TE improved significantly for OCD only reflecting improvements in both bias and CVa. Roche BMC and Beckman Coulter did not show significant improvement in TEs.

Table 3.

Percentage of QMP-LS Challenge Vials Meeting Biologic Variation–Related Analytical Performance Goals: Pre- and Post-May 2008.

Instrument	% challenge vials with mean % bias < 3.4%			% challenge vials with mean CVa < 2.2%			% challenge vials with mean TE < 7.0%
	Pre-May 2008	Post-May 2008	P value	Pre-May 2008	Post-May 2008	P value	Pre-May 2008	Post-May 2008	P value
Ortho Clinical[a] Diagnostics (enzymatic)	0	65	<.001	45	65	.254	5	59	<.001
Beckman[a] Coulter (modified Jaffe)	22	35	.379	25	29	1.000	15	32	.208
Roche BMC[a] (enzymatic and modified Jaffe)	57	65	.587	15	29	.329	25	44	.244

Note. QMP-LS = Quality Management Program–Laboratory Services; CVa = coefficient of variation; TE = total error.

Specific manufacturers: % Bias (n = 23 challenge vials pre-May 2008, n = 34 challenge vials post-May 2008), CVa, and TE (n = 20 challenge pre-May 2008, n = 34 challenge vials post-May 2008).

More challenge vials attained the less stringent precision and TE goals of the NKDEP Laboratory Working Group (Table 4). Only 47% to 71% of challenge vials reached bias goals, and only OCD demonstrated significant improvement in bias and TE after May 2008.

Table 4.

Percentage of QMP-LS Challenge Vials Meeting NKDEP’s Analytical Performance Goals: Pre- and Post-May 2008.

Instrument	% challenge vials with mean %bias < 5%			% challenge vials with mean CVa < 8%			% challenge vials with mean TE < 18%
	Pre-May 2008	Post-May 2008	P value	Pre-May 2008	Post-May 2008	P value	Pre-May 2008	Post-May 2008	P value
Ortho Clinical[a] Diagnostics (enzymatic)	13	71	<.001	95	100	.370	55	100	<.001
Beckman[a] Coulter (modified Jaffe)	39	47	.597	95	97	1.0	90	82	.695
Roche BMC[a] (enzymatic and modified Jaffe)	61	71	.569	95	100	.370	90	100	.133

Note. QMP-LS = Quality Management Program–Laboratory Service; NKDEP = National Kidney Disease Education Program; CVa = coefficient of variation; TE = total error.

Specific manufacturers: % Bias (n = 23 challenge vials pre-May 2008, n = 34 challenge vials post-May 2008), CVa, and TE (n = 20 challenge pre-May 2008, n = 34 challenge vials post-May 2008).

Discussion

This study demonstrates that, despite improvement in Cr assay bias with implementation of IDMS standardization, bias performance goals, as proposed by the NKDEP Laboratory Working Group, are not being reached consistently by all manufacturers (47%-71% of challenge vials). As expected, individual manufacturer assay precision is mostly unchanged with IDMS standardization as the provision of traceable reference materials will have no impact on assay nonspecificity due to interfering factors or on the other contributors to assay precision such as instrument performance, reagents characteristics, pipetting, temperature fluctuations, and so on. Because the NKDEP precision goal is generous at <8%, most EQA challenges meet these goals as well as the corresponding TEa goal. Precision and TEa goals based on biologic variation are attained by only 29% to 69% and 32% to 62% of challenge vials. Overall, OCD’s enzymatic assay EQA performance is the strongest of the manufacturers with the lowest bias, best precision, and lowest TE. The challenge vials with the worst bias and precision were spiked with known interfering substances (glucose and bilirubin), highlighting the ongoing problems with assay nonspecificity.

There are very few studies examining EQA data since introduction of IDMS standardization. Carobene et al[15] examined Italian EQA data from 2006 and 2011, involving 450 laboratories. Surprisingly, there was no major improvement in overall assay bias in 2011 as compared with 2006 despite introduction of IDMS standardization. In 2011, 7% of laboratories were using enzymatic methods and in that smaller cohort, performance goals were achieved far more frequently.[15] In 2011, EQA proficiency testing results for 3 challenge vials from the College of American Pathologists were analyzed by instrument.[8] The mean percent bias ranged between −6.9% and 8.1% which was an improvement over 2003 results when mean percent biases ranged between −7% and 34%. As in the current study, there were differences in performance between manufacturers with some performing much better than others. Analytical bias and CVa were compared directly between one challenge sample from 2003 and one challenge sample from 2011 with similar Cr concentrations around 75 µmol/L.[8] In 2003, bias ranged between 1.0 and 17.7 µmol/L, and CVa between 0.1% and 11.3%; in 2011, bias improved ranging from −2.6 to 5.3 µmol/L, but CVa was unchanged at 3.1% to 11.2%.

Many EQA programs have performance goals for Cr that are more stringent than the NKDEP goals but less stringent than biologic variation goals. Ontario’s QMP-LS has a TEa of ±9 µmol/L for Cr <100 µmol/L and <9% for Cr > 100 µmol/L.[16] The Royal College of Pathologists of Australia (RCPA) have a “minimal performance” TEa of ±8 µmol/L for Cr <100 µmol/L and <8% for Cr >100 µmol/L.[17] In the United Kingdom, the RCPA have established the Minimum Analytical Performance Standards (MAPS) group, and they propose a “desirable” TEa goal at 75 µmol/L Cr of 8.2%.[18,19] The outlier is the US Federal Agency’s Center for Medicare and Medicaid Services (CMS), which is responsible for implementation of the Clinical Laboratory Improvement Amendments (CLIA) regulations that governs all laboratory testing in the United States.[20] The CLIA TEa for Cr is set at ±27 µmol/L or 15% (whichever is greater).[20]

An acceptable TEa for Cr yields a TEa for corresponding eGFR with various consequences. Consider the January 2015 QMP-LS 1501 challenge vial 3 which has an assigned Cr value of 85 µmol/L.[13] According to the QMP-LS TEa performance goals, Cr values between 76 and 94 µmol/L would be acceptable for this sample. For a 65-year-old non–African American female, the assigned value of 85 µmol/L translates to an eGFR-EPI of 62 mL/min/1.73 m2 with corresponding acceptable eGFR ranging between 55 and 71 mL/min/1.73 m2. The range of observed Cr values was slightly wider at 74 to 100 µmol/L in the 212 participating laboratories,[13] yielding corresponding eGFR-EPI (Chronic Kidney Disease Epidemiology Collaboration) results between 51 and 74 mL/min/1.73 m2 and substantial differences in whether the sample was consistent with a diagnosis of CKD or not. For the clinician, a fixed diagnostic cutoff requires an appreciation of the TE of a result due to both analytical and biologic variation. Only results that are significantly different from a diagnostic cutoff have 95% probability of being truly diagnostic. For the eGFR cutoff of 60 mL/min/1.73 m2, a value of 49 mL/min/1.73 m2 (TE ± 11 mL/min/1.73 m2) has been proposed.[21] In addition to the biologic and analytic variation of Cr, the clinician must consider nonanalytical factors such as the various non-GFR determinants of serum Cr concentrations (age, gender, diet, muscle mass, hepatic synthetic function) when interpreting the eGFR.[22]

Strengths of our study include the high-quality comprehensive QMP-LS data from before, during, and after implementation of IDMS standardization in Ontario using fresh frozen plasma samples. Weaknesses of our study include the lack of individual laboratory results as these are not available with performance, therefore ascertained using aggregate data by manufacturer. In addition, QMP-LS does not distinguish between Jaffe and enzymatic methods so we could not separate out the enzymatic from the modified Jaffe Roche BMC results. Only 3 manufacturers were included in the analysis to ensure that results generally represented more than 10 laboratories. There are a number of additional manufacturers that were not as common during the study time period, and some of their performances appeared to be worse than the 3 included here.[13] Manufacturers did not all transition to IDMS traceability in May 2008 with some transitioning sooner or later than May 2008 leading to some contamination. However, the analysis done of the transition period May 2008 and Jan 2010 (n = 16 challenge vials) comparing IDMS traceable laboratories with those that were not provided similar results to those in the primary analyses (Table 2).

Conclusions

Ongoing efforts to further reduce Cr bias and precision and interlaboratory variability are needed. Clinicians and their collaborating clinical biochemists should be advocating within their own setting for the phasing out of Jaffe assays and adoption of enzymatic assays as the preferred method for Cr measurement as has been repeatedly recommended.[1,6,9] Given that the cost for enzymatic methods has remained significantly higher for unclear and questionable reasons than that for Jaffe methods, many institutions and laboratories have not yet made the switch. EQA schemes could help validate this effort by providing summary statistics and graphical analysis based on the performance of these 2 method groups independently.

Finally, recommended TEa goals should be reviewed to ensure that the minimal acceptable performance is aligned with test characteristics that permit optimal patient care. It might be asked whether, with a fixed CKD diagnosis cutoff of 60 mL/min/1.73 m2, a TEa of 9 µmol/L for Cr <100 µmol/L (QMP-LS TEa goal) is reasonable, given that this translates to an “acceptable” eGFR-EPI of 53 to 71 mL/min/1.73 m2 (65-year-old non–African American female with an assigned Cr value of 85 µmol/L). More stringent TEa goals should stimulate invigorated manufacturer motivation for improvements in method performance. This is especially relevant in the era of increasing enthusiasm for early CKD detection by organizations wishing to decrease the poor outcomes and economic impacts of progressive CKD.[23-25]