The comparison of analytical performances of Mindray CL-1000i and Beckman Coulter Access II Troponin I methods in the light of recent guidelines and the quality requirements.

BACKGROUND: The results of cardiac troponin I (cTnI) methods used in the diagnosis of acute myocardial infarction (AMI) are highly variable. In this study, it was aimed to compare the analytical performance of the Mindray CL-series TnI method with the Beckman Coulter-Access II AccuTnI+3 method.
METHODS: Analytical performance and method comparison experiments for cTnI measurement with Mindray CL-1000i and Beckman Coulter-Access II instruments were with the Clinical and Laboratory Standards Institute (CLSI) documents EP15-A3 and EP9-A3. Precision studies were performed with commercially available third-party quality control (QC) materials. Method comparison experiments were performed by using patient samples. Furthermore, the limit of quantification (LoQ), total analytical error (TAE), and sigma metrics of both methods was determined.
RESULTS: The repeatability CV% for the three levels of Mindray CL-series TnI were 1.86, 1.75, and 1.08, while within the laboratory, CV% values were 3.36, 5.27, and 5.82. The quantification limits for Mindray CL-series and Beckman AccuTnI+3 were found to be 0.0085 and 0.0366 ng/mL with a CV level of less than 10%, respectively. The Mindray CL-series TnI results in the method comparison study were higher compared to the Beckman Coulter Access II AccuTnI+3 method.
CONCLUSIONS: With low repeatability, low bias, and low LoQ, The Mindray CL-series cTnI method shows it may be used safely in its category. The significant difference between the two methods in the method comparison study reveals cTnI methods in the market should be standardized to ensure global traceability. 2020 Annals of Translational Medicine. All rights reserved.

Introduction

Diagnosis of acute myocardial infarction (AMI) is from cardiac markers (preferably cardiac troponin) above the 99th percentile upper reference limit (URL) of the healthy population and with one of the following: (I) classic ischemic symptoms; (II) new ST-segment changes or left bundle branch block; (III) pathologic Q waves on electrocardiogram; (IV) evidence of myocardial injury in imaging; (V) demonstration of coronary thrombus in angiography or autopsy (1).

The cardiac marker journey began between 1950–60s with AST and CK, CK-MB mass in the 1980s, first-generation troponins (Tn) in the 1990s, new generation Tns in the 2000s, and the high-sensitive Tn (hs-Tn) continues while different molecules including new proteins and miRNAs are added daily, although few of them are used in routine laboratory practice (2-4).

Cardiac Tns (cTns) have become an important biomarker for the noninvasive detection of acute myocardial injury in many cardiovascular diseases, and elevated levels of cTns are considered the gold standard biomarker of AMI (1,2). Both cardiac troponin I (cTnI) and cTnT perform equivalent performance in terms of the diagnostic capability of AMI, according to the proper cut-off values (5). While the cTnT method exists in only one manufacturer and has no standardization problem, cTnI methods are marketed by many manufacturers, which needs global standardization. Although having CVs lower than 10% on their 99th percentile URL, it is reported there are essential variations up to 20 times in the results of many commercially available TnI assays (6). Also, it is reported that most available TnI methods have poor precision on the 99th percentile URL of the reference population.

This is the first study about Mindray troponin I, about the comparison of Mindray and Beckman troponins and the first study evaluating the analytical performances of Mindray and Beckman troponins according to CLSI EP15-A3 or CLSI EP9-A3 guidelines. This study aimed to compare the analytical performance of the Mindray CL-seriesTnI method with the Beckman Coulter Access II AccuTnI+3 method, currently used in the routine and to verify the precision and bias claims of the manufacturers. We present the following article in accordance with the MDAR reporting checklist (available at http://dx.doi.org/10.21037/atm-20-6104).

Methods

Assays

TnI measurements were performed in two chemiluminescent systems as Beckman Coulter-Access II (Beckman Coulter Diagnostics Division Headquarters, Brea, CA, USA) and Mindray CL-1000i (Mindray Bio-Medical Electronics Co., Ltd., Shenzhen, China). Both are bench-top analyzers for small to mid-sized laboratories. Beckman Coulter-Access II analyzer has been using for cardiac marker and hormone tests in our laboratory for many years, while Mindray CL-1000i was used for the first time. A Mindray CL-1000i analyzer uses a CL-series TnI assay, a chemiluminescent immunoassay for the quantitative determination of TnI. The Beckman Coulter AccuTnI+3 kit was used in the Beckman Coulter-Access II analyzer, which is also a chemiluminescent immunoassay for the quantitative determination of TnI. The sensitivity of Mindray CL-seriesTnI assay was 0.006ng/mL with a measurement range of 0.006–50 ng/mL. Beckman Coulter-Access II AccuTnI+3 assay had a sensitivity of 0.01 ng/mL and a measurement range of 0.01–100 ng/mL. The 99th percentile URL for troponin I was 0.04 ng/mL for both the Mindray and Beckman methods, as shown in their package inserts.

Verification of precision and estimation of bias

After a one-month familiarization period for Mindray CL-1000i analyzer, a precision and bias study was performed for both analyzers simultaneously. Analyzers were calibrated at the start of the study, and the quality control (QC) results of both analyzers were within QC limits. AccuTnI+3 kits of Beckman Coulter (lot number: 724243) and Mindray CL-series TnI kit (lot number: 171101) were used for the evaluation experiments.

A third-party QC material (MAS, CardioImmune, XL, lot CXL2001, Microgenics, Thermo Scientific, Massachusetts, USA) was used for evaluating precision and bias by using the Clinical and Laboratory Standards Institute (CLSI) EP15-A3 protocol (7). Instead of patient samples or pools, we used QC materials since they have target values which are needed for calculation of bias. Three levels of QC materials were measured for five consecutive days as five replicates. The target values of QC materials for Mindray and Access II TnI, which were provided by the QC manufacturer, are presented in .

Table 1

The verification data of precision and bias study

PI claims or user estimates	Mindray CL 1000i				Beckman Coulter Access II
	Level 1	Level 2	Level 3		Level 1	Level 2	Level 3
PI claims
Mean (ng/mL)	0.410	*	34.873		0.070	0.910	13.970
Repeatability (CV%)	5.15	*	4.15		5	2	2
WL imprecision (CV%)	9.06	*	7.11		7	5	5
User estimates
N	25	25	25		25	25	25
Repeatability (CV%)	1.86	1.75	1.08		2.58	3.13	2.00
UVL for repeatability imprecision (CV%)	6.90	6.90	5.56		6.70	2.68	2.68
WL imprecision (CV%)	3.36	5.27	5.82		12.01	8.60	9.13
UVL for WL imprecision (CV%)	14.13	14.13	11.09		10.29	8.30	8.30
Target value of QC material (ng/mL)	0.78	2.52	27.1		0.22	0.88	12.1
Overall mean, n=25 (ng/mL)	0.775	2.486	27.023		0.234	0.935	11.437
Verification interval for bias (ng/mL)	0.74–0.82	2.30–2.74	24.36–29.84		0.17–0.27	0.75–1.02	10.29–13.91
Bias %	−0.64	0.04	−0.28		6.36	6.25	−5.48
Total analytical error %	6.18	8.74	9.88		26.19	20.44	20.55
Sigma metric for TEa =27.91%	8.12	5.29	4.75		1.8	2.52	2.46
Sigma metric for TEa =10%	0.56	1.89	1.67		0.31	0.44	0.5

*, the manufacturer has no claim for level 2. PI, package insert; WL, within laboratory; UVL, upper verification limit; TEa, total allowable error.

Method comparison and bias estimation using patient samples

The method comparison study was performed by analyzing 115 serum samples in Beckman Coulter Access II, AccuTnI+3 and Mindray CL-series TnI assay systems according to CLSI EP9-A3 protocols (8). Serum samples were collected randomly from leftover serum samples after routine analysis for whom a TnI test was requested. None of the samples interfered with hemolysis, lipemia, or icterus. The study was conducted according to the principles of the Declaration of Helsinki (as revised in 2013). Since the serum samples were collected after routine measurements were finished, the conducted research did not need ethical approval or patient consent. The serum samples were stored at minus 20 °C until the analysis. In this method comparison study, the Beckman Coulter Access II AccuTnI+3 assay was accepted as a comparative method, and the Mindray CL-series TnI assay was accepted as a candidate method. The concentrations of TnI in all samples were not covering the whole measuring interval of both comparative (0.01–100 ng/mL) and candidate (0.006–50 ng/mL) measurement procedures. Duplicate measurements were made in both analyzer systems, and the mean of duplicate measurements was compared. Data were collected by printing the results daily, and the possible transcription errors in typing the data were prevented by careful inspection of each entry.

The limit of quantification (LoQ) study

A pool by serums with elevated levels of cTnI was prepared. Then the pool was diluted with serums that have no or extremely low levels of cTnI. Thus, serum samples with decreasing concentrations of TnI were obtained. Ten measurements for each concentration were performed on the same day. The CV% of ten measurements for each concentration was calculated, and 10%CV was accepted for LoQ.

Statistical analysis

A two-sided Grubbs test found no outliers. One-way analysis of variance (ANOVA) was used to calculate the repeatability (intra-day) and between-run variation. Within-laboratory (WL) variability (total imprecision) was then calculated from these two parameters. The estimated CVs were compared with the manufacturer’s declared values. If the estimated CVs were higher than the values given by the manufacturer, they were checked whether they exceeded the upper verification limit (UVL). For this purpose, the degrees of freedom were determined to calculate the UVL of the CVs given by the manufacturer. The data obtained from these values were multiplied by the manufacturer’s CVs, and UVLs were found.

The mean value of the 25 results (overall mean) for each control level in the precision study was used to determine bias. The standard error of the overall mean is calculated from repeatability and WL CVs. Then, the combined standard error resulting from both measurement variability and defining of the target value of the control material was calculated according to CLSI EP15-A3. 95% confidence intervals of target values of the QC materials (verification interval) were found, considering the degrees of freedom. Bias% was calculated by using the overall means obtained from the precision study and the target value of QC materials with the formula, [(overall mean − QC target value)/QC target value) × 100]. Finally, it was evaluated whether the overall means obtained from the precision study were within the calculated verification interval. The statistical analyses for precision and bias estimation were calculated using MedCalc Statistical Software version 19.1 (Oostende, Belgium).

In addition to bias, total analytical error (TAE) and sigma metrics were also calculated with the following formulas, (|Bias%| + 1.65 WL CV%), (TEa− |bias%|/WL CV%), respectively.

Scatter diagram, Bland Altman difference plot, and Passing Bablok regression analysis were performed by MedCalc Statistical Software version 19.1. The Bland-Altman difference plot was used to assess the distribution of the difference between the two methods throughout the measurement range. Scatter plots are used to evaluate the correlation between the two methods. The mathematical relationship between the two methods was determined by using the Passing-Bablok regression analysis. The mean systematic difference (bias) between the two methods was defined at a medical decision level of 0.04 ng/mL (URL).

Results

presents the detailed data of the precision and bias verification studies performed according to the protocol of CLSI EP15-A3 guideline for Mindray CL-series TnI and Beckman Coulter Access II AccuTnI+3 methods. Mindray CL-series TnI imprecision values were below the limits of the manufacturer’s claim. It was found Beckman Coulter Access II AccuTnI+3 imprecisions were higher in repeatability CV for L2 and in WL CVs for all three levels compared to the values in the manufacturer’s claim.

The bias verification study data according to CLSI EP15-A3 protocol are given in . It was found that the calculated values in all three levels for Mindray CL-series TnI were within the verification interval. The biases in all three levels for the Beckman Coulter Access II AccuTnI+3 assay was within the verification interval.

In the patient-based method comparison study according to CLSI EP9-A3, it was determined there was an average of 17.5% difference between the two methods between 0.04–10 ng/mL measuring range (). As observed in the Bland-Altman difference plot, the Mindray CL-series TnI method gave higher results compared to Beckman Coulter Access II AccuTnI+3. It was found this difference exceeded 100% in low concentrations and 30% in high concentrations (). In the Passing-Bablok regression analysis, the relationship between the two methods was found as Mindray CL-1000i TnI = Beckman Coulter Access II AccuTnI+3 × 1.208+0.00110 (). When the medical decision level (0.04 ng/mL) was used in the regression equation instead of Beckman Coulter Access II AccuTnI+3, which was the comparative method, the systematic difference (bias) between two methods was found 170% at 99th percent URL. In other words, the systematic difference in medical decision level was found to be higher than the TEa suggested by the RCPA and the biological variation studies 10% and 27.91% (9), respectively.

Figure 1

There was an average of 17.5% difference between the two methods between 0.04–10 ng/mL measuring range. As observed in the Bland-Altman difference plot, the Mindray CL-series TnI method gave higher results compared to Beckman Coulter Access II AccuTnI+3. It was found this difference exceeded 100% in low concentrations and 30% in high concentrations.

Figure 2

In the Passing-Bablok regression analysis, the relationship between the two methods was found as Mindray CL-1000i TnI = Beckman Coulter Access II AccuTnI+3×1.208+0.00110.

It was found the Mindray CL-series TnI method had a LoQ of approximately 0.0085 ng/mL, where the Beckman Coulter Access II AccuTnI+3 method had a LoQ of approximately 0.0366 ng/mL under 10%CV ().

Figure 3

Mindray CL-series TnI method had a LoQ of approximately 0.0085 ng/mL under 10%CV.

Figure 4

Beckman Coulter Access II AccuTnI+3 method had a LoQ of approximately 0.0366 ng/mL under 10%CV.

Discussion

Rapid diagnosis of AMI is essential for prompt treatment decisions and reduction of mortality. Also, the rapid exclusion of AMI will reduce the length of stay in the emergency department. cTns are preferred markers for the diagnosis of myocardial injury due to their high sensitivity as well as almost absolute myocardial tissue specificity that reflects the microscopic damage to myocardial necrosis (10). Increased value in cTn is defined as a measurement above the 99th percentile URL. The optimal CV on the 99th percentile URL for cTn assays was defined as <10% (11). The sensitivity and specificity of many Tn assays available on the market vary, although the more sensitive cTn methods are being produced.

There are essential differences in the analytical performance of the commercially available cTn methods in cTnI levels close to the 99th percentile URL. In this study, the analytical performance of Mindray CL series cTnI methods (with a sensitivity of 0.006 ng/mL and a measurement range of 0.006–50 ng/mL) was compared with Beckman Coulter Access II AccuTnI+3 (with a sensitivity of 0.01 ng/mL and a measurement range of 0.01–100 ng/mL). Following the CLSI EP15-A3 protocol, it was evaluated if the repeatability and WL imprecision of both methods met the manufacturer’s claims and the quality requirements or not. It was determined the Mindray CL-series TnI method had lower repeatability and WL CVs at all three control levels than the manufacturer’s declared values. However, the repeatability imprecision for L2 and the WL imprecision for all three levels of the Beckman Coulter Access II AccuTnI+3 method were above the manufacturer’s declaration and UVL. Comparing the repeatability and WL CVs of both methods, it was determined the Mindray CL 1000i system had a better Beckman Coulter Access II AccuTnI+3 method. cTns can be divided into three groups according to their analytical performance at the concentrations of the 99th percentile of the healthy population; acceptable (CV% <10), clinically usable (CV% =10–20), and not accepted CV%>20) (12,13). With WL imprecision values as; 12.01%, 8.60% and 9.13% for level 1, level 2 and level 3, respectively (), Beckman Coulter Access II AccuTnI+3 method may be accepted as clinically usable and acceptable (12,13), although it has a problem in our verification study.

The CLSI EP15-A3 protocol recommends the data obtained from the precision study be used for bias validation without the need for an added bias study. Comparisons of averages from the 25 results obtained from the control level are within the verification interval, following the protocol above. To find 95% confidence intervals of the verification interval, the variability resulting from the measurement precision (repeatability and WL CV) is suggested to combine the variability resulting from the determination of the target values of the control material. Since we did not know the uncertainty of the detection at the target values of the control material, we took this value as zero, as suggested in the CLSI EP15-A3 guideline. Thus, the combined standard error is lower, resulting in a narrower target range; in other words, the possibility of a false rejection may occur due to the narrowing of the verification interval. Despite this, both methods were determined to be within the verification interval in the third-party controls. Although the controls used in the bias and precision verification of both methods were the same, the target values were given separately by the control supplier for both methods. Although the biases for both methods were low, the overall means were found to be within the verification interval. The differences between the two methods in overall means in Level 1, Level 2, and Level 3 controls were found as 331%, 265%, and 236%, respectively. The fact that the control providers set the target values according to the analytical system, it is difficult to determine the absolute accuracy. Although the URL of both methods claimed by the manufacturers in their kit inserts is very close, the differences varying from 236% to 331% between overall means of two methods can result in leading to erroneous clinical interpretation at the results close to URL or in minimal myocardial damage. False-positive results will cause loss of labor and an increase in cost, while false negative results will result in severe clinical outcomes, including missing patients with minimal myocardial ischemia.

There are some disadvantages of bias calculation by using control materials, including matrix effect. Therefore, we compare both methods according to the CLSI EP9-A3 protocol to determine the effect of the matrix. Compared to the Beckman Coulter Access II AccuTnI+3 method, the Mindray Cl series TnI method was found to give higher results in medical decision levels accepted by both manufacturers; 170% on the 99th percentile (0.04 ng/mL) URL, although both manufacturers have approximately the same cut-off values. These findings show there is a significant difference between the two methods in which the detection of abnormal results in values close to the upper limit of the reference range, contrary to high TnI values. Although manufacturers recommend using their reference values or cut-off limits when making clinical interpretations, this may not always give a correct clinical interpretation because it may be a false positive or false negative, and may also compromise the traceability of results when various analytical systems measure a patient’s result. The systematic difference between these two methods, which is higher than TEa (14,15), reveals how it is necessary to evaluate and verify the reference intervals and cut-off points of such methods before the routine use in clinical laboratories.

Clinical laboratories should evaluate whether the method’s quantification limit meets the targeted analytical and medical requirements when deciding on the routine use of the TnI method. One of the performance criteria to be considered when selecting a method is the lower measurement limits. For this purpose, the concepts of detection limit (LoD), functional sensitivity (FS), and LoQ are used. The analytical system manufacturers prefer to write one of these in their kit inserts. There is a confusion in the use and interpretation of these concepts. LoQ has been described as the lowest concentration at which the CV is <10%. The CVs of the Tn methods are expected to be <10% in the URL. It was determined in our study that the quantification limit of Mindray CL-series TnI was lower than Beckman Coulter-Access II AccuTnI+3, 0.0085and 0.0366 ng/mL, respectively (). The ratio between the 99th percentile URL (0.04 ng/mL) and LOQ value is an essential analytical characteristic and is suggested to be higher than 1 (16). The ratios for Mindray and Beckman Access II in our study were found 4.7 and 1.1, respectively. Both results are over 1, but Mindray’s analytical performance is4.3-fold higher, presumed to be the sign of a better performance. Although the lower limit of measuring range was declared as the analytical sensitivity value in the Beckman Coulter Access II AccuTnI+3 package insert, CV at that level was found to be 14%. The difference may be a global problem for Beckman Coulter-Access II AccuTnI+3 users because, in an external quality assessment program used in our laboratory, some levels around 0.02 ng/mL, the peer group’s CV levels were over 25%. This fact shows the necessity of highly sensitive cTnI kits.

As shown in , we calculate the other analytical performance characteristics of both methods by using WL CVs and bias obtained from the precision and bias verification study. When we compared the performance data with the biological variation study, it was found that bias and TAE of the Beckman The Coulter-Access II AccuTnI+3 method was acceptable, but its WL CVs were not. However, the TEa of 27.91% we used in our study is derived from the biological variation studies (9,14) and is high. Because some organizations, for example, the Royal College of Pathologists of Australasia (RCPA) accept TEa as 10% (15); here, it was determined the TAE of the Beckman Coulter-Access II AccuTnI+3 method could not meet this requirement at all three control levels.

There are also variations in TEa, including changes in the analytical performance of cTnI methods. For example, TEa from the biological variation study is approximately three times higher than the RCPA (15); this leads to different outcomes when evaluating the TAE of the cTnI methods. Currently, different TEas also affect the level of sigma metrics. As shown in in our study, when the value derived from the biologic variation study was used, sigma metrics were higher compared to the values obtained by using TEa of RCPA. Hence, the same analytical performance can be determined differently when different TEa values are taken. These findings are misleading, suggesting that laboratories should be careful when evaluating their performance with sigma metrics. While the manufacturers have progressively lowered the sensitivity and imprecision of cTnI methods, it is needed TEas should also be rearranged.

❖ The Mindray CL-series TnI has better repeatability and WL CVs than the manufacturer’s claim, and meets the quality requirements;

❖ The Mindray CL-series TnI method has a lower quantification limit than that of Beckman Coulter-Access II AccuTnI+3 method;

❖ In patient sample-based comparison, there were significant differences between the two methods, the results of Mindray CL-series TnI were higher, especially at low levels;

❖ The Mindray CL-series TnI method has an excellent analytical performance in its category, but the analytical performance of existing TnI methods may vary and may yield different results to the same patient;

❖ The proper reference range or cut-off points should be used for each method, but this cannot solve all problems, so in order to prevent confusion standardization of TnI methods is required worldwide;

❖ It was thought that TEas for TnI assays should be updated in parallel with the continuous improvement of TnI methods’ analytical performance.

The article’s supplementary files as