External Quality Assessment of Glycated Hemoglobin in Iran: Comparison of Five Different Commercial Methods with Two Different Total Allowable Errors.

BACKGROUND: Glycated hemoglobin (HbA1c) measuring has a critical role in the monitoring and diagnosis of diabetes. So, the analytical performance of its measuring method must be acceptable. Clinical laboratories should continuously monitor the performance of their commercial methods, both by using proper internal quality control (IQC) and by participating in external quality assessment schemes (EQAS).
METHODS: In January and August 2016, two different freshly prepared commutable patient QC samples were sent to over 1000 laboratories, but 682 and 925 different laboratories which were used five common commercial methods for measuring HbA1c, included in this study during 23th and 24th runs of the external quality assessment program (EQAP), respectively. Target values for total group and also for peer groups were calculated. The performance of each method and laboratory were determined according to two different allowable total errors (TEa), including ±6% and ±20%, which are suggested by the National Glycohemoglobin Standardization Program (NGSP) and Reference Health Laboratory of Iran, respectively.
RESULTS: Considering TEa of ±20% in evaluating HbA1c commercial methods and laboratory performances, pass rates ranged from 97% to 98% during EQAP-23 and EQAP-24, respectively. But when this evaluation was performed according to TEa of ±6%, pass rates decreased significantly to 60% and 62%, respectively.
CONCLUSIONS: Using improper analytical goals has led to misinterpretation of EQA results. In order to maintain the clinical usefulness of HbA1c results, we need to reduce TEa of ±20% to ±6% and improve HbA1c measuring method performance. Although, with TEa of ±6% our pass rates are not so bad.

Introduction

According to the critical role of the level of glycated hemoglobin (HbA1c) in the monitoring and diagnosis of diabetes (1, 2) the acceptable precision and accuracy of different commercial methods in clinical laboratories for measuring HbA1c is essential (3). There are many different methods such as ion-exchange chromatography, capillary electrophoresis, boronate affinity chromatography, immunoassay and enzymatic methods, which separate glycated hemoglobins from nonglycated ones, based on existing differences in structure, charge, affinity or immunity (2).

The manufacturers should produce commercial methods that their results could be traceable to the higher-order references and be able to achieve acceptable analytical goals of measurement (3). Analytical goals must be defined in such a way that the test could save its clinical usefulness. On the other hand, it is the responsibility of clinical laboratories to monitor the performance of their methods, both by using a proper internal quality control (IQC) and participating in external quality assessment schemes (EQAS) (3). In addition, before routinely using any commercial methods for analyzing patient specimens, each laboratory should verify method producer claims or validate method performance according to the laboratory analytical goals (4). In each above experiment, including IQC, EQAS, and method validation, judgement of method performance acceptability depends on the established allowable total error (TEa). TEa is a maximum tolerable error that, if it occurs, will not impair the result’s clinical usefulness (5, 6, 7).

In order to significantly reduce differences between results obtained by various commercial methods, standardization of the methods of HbA1c measurement has been necessary. This can be achieved by the International Federation of Clinical Chemistry (IFCC) and National Glycohemoglobin Standardization Program (NGSP) (5). The American Diabetes Association (ADA) recommends that laboratories use only NGSP-certified HbA1c methods and participate in the College of American Pathologists (CAP) fresh sample proficiency testing survey (6, 7). Using NGSP certified methods is not the only step in achieving a precise and accurate analytical HbA1c measuring method. Continuous monitoring of the performance of the method is needed. In this regard, defining analytical goals with which method performance must be evaluated, has profound effects on detection of analytical errors.

The exact number of errors made can be quantified by employing sigma metrics in the laboratory. According to selected TEa along with method bias and imprecision, we can calculate sigma of the method which is a valuable criterion for assessing the quality of the analytical phase. A high sigma value indicates good performance. A sigma value of 2 is defined as the minimum allowable value for routine performance (8).

Unfortunately, there are few reports about the performance of clinical laboratories measuring HbA1c in EQAS and proficiency testing (PT) of different countries. In the United States, CAP publishes these data regularly and some reports can be found from some other countries, including Italy and China. In this study, for the first time in Iran, we evaluated the effects of two different analytical goals, defined as TEa, on interpretation of EQAS HbA1c results. TEa encompasses the imprecision and bias of a single test measurement and is used to evaluate laboratory tests performance in EQAS (9). We also evaluated the analytical performance of the methods by calculating the sigma value of each method according to different TEa.

Materials and Methods

Sample Preparation

Pooled whole bloods were collected from diabetic patients who suffered from polycythemia and should have regular phlebotomy to reduce their red blood cell count. The bloods were collected in EDTA containing bags and then four or five isogroup bloods were mixed in order to have enough blood (about 1500 mL) for filling each HbA1c vial with 1.0 mL pooled whole blood. In this preparation, only EDTA, with no preservatives was added as a coagulant which is recommended by method producer. So in comparison to routine blood collection for HbA1c measuring, the matrix of the blood was not changed and we can consider them commutable.

Explanation of the Procedure

During twenty third and twenty fourth runs of the external quality assessment program (EQAP-23 and EQAP-24), in January 2016 and August 2016, two different freshly prepared commutable patient whole blood samples in EDTA-containing vials were sent to 966 and 1393 participant laboratories, respectively, at 4 °C to 8 °C. In order to protect the blood samples against possible degradation and leakage during mailing, the plastic tubes were filled with the whole blood and sealed with stoppers and inserted into a triple container. The first container is a cold box with adequate ice bag for cooling the filled tubes. The second one is the Styrofoam container which acts as a thermal insulation and a water barrier and the last one is a carton on which the names and the addresses of the providers and participants together with the information about the mailing cautions have been recorded.

Using commutable samples, the results of different commercial methods can be compared (10). Before sending to participating laboratories, homogeneity of whole blood vials and after sending, the stability of these vials was assessed and confirmed. These assessments and confirmation were done according to WHO requirements (11). For homogeneity, about one percent of total vials, including ten and fourteen vials were randomly selected during EQAP-23 and EQAP-24, respectively, and their HbA1c determined by Tosoh G8 Auto HPLC in duplicate. Then the coefficient of variation (CV) of the results was calculated and compared with one fourth of an allowable total error of 6%, or 1.5%. For stability, one whole blood vial in each run was refrigerated by the end of the program and then the HbA1c was measured in duplicate and the average of the results compared with the homogeneity mean value ± modified allowable total error (mTEa%).

Although there are more than ten commercial methods for monitoring of HbA1c in Iran. This study was focused on common methods for which the number of using laboratories was at least twenty. So their statistical analysis could be valid, and their performance had also been evaluated (12). These included Biosystem, Nycocard, Pars Azmon, Pishtaz Teb, and Roche kits which their assay principles were cation-exchange chromatography, boronate-affinity chromatography, immunoturbidimetry, enzymatic, and immunoturbidimetry, respectively.

In EQAP-23, 682 participating laboratories used the desired commercial methods. These laboratories were grouped in five peer groups, including Biosystem, NycoCard, Pars Azmon, Pishtaz Teb, and Roche with 172, 233, 120, 137, and 20 participating laboratories, respectively. In EQAP-24, 925 participating laboratories used the desired commercial methods. These laboratories were grouped in five peer groups, including Biosystem, NycoCard, Pars Azmon, Pishtaz Teb, and Roche with 209, 337, 149, 205, and 25 participating laboratories, respectively. Each participant laboratory had to examine the sent control material as a routine patient sample according to the instructions of commercial method manufacturers and had to calibrate and control its measuring method by the calibrators and control materials, as an internal quality control, provided by method manufacturers. After measuring HbA1c, the results were sent to EQAP and statistical analyses were done. According to the used commercial method, the results were grouped into five peer groups. Then, mean, SD, and CV of each peer group and also total results were calculated. In EQA, the mean of each peer group is used as target value to evaluate each laboratory performance. In this regard, it is necessary to delete outliers which are out of Mean ± 2SD or 3SD (13). In EQAP, Mean ± 2.5SD is used. After deleting outliers, calculation of mean and SD was repeated until there were no outliers. The last calculated mean, termed as weighted mean, was used as the target value. Statistical analysis was done by SPSS 20 and Med Calc 13.3.1 software.

Target values for total group and also for peer groups were calculated. In Iran, laboratory performance is evaluated according to standard deviation interval (SDI). SDI is commonly calculated by the following formula (6, 7, 8, 9, 10, 11, 12, 13, 14):

but adjusted SD instead of peer group SD is used. Adjusted SD is calculated by the following formula:

in which, CCV (Chosen Coefficient of Variation) is defined by the Reference Health Laboratory of Iran and equals 10% for HbA1c methods. SDI ≤ 2 is considered as acceptable result (12). This CCV and acceptability criteria represent TEa of 20%; i.e. SDI = 2 shows that the result is 2 SD far from mean target value and SD equals 10% of mean target value, so TEa = 2 SD = 2 × 10% = 20%.

For calculating the relative bias of the mean value of each commercial method, in comparison to the mean value of all groups, we subtracted peer group mean value from the all groups mean value. According to NGSP, bias up to 0.3% of absolute HbA1c was considered acceptable.

Total analytical error (TAE) has two parts, including bias and imprecision which are representative of systematic and random errors, respectively. For calculating bias% of each commercial method, first the TAE% of each method is calculated by the following formula:

TAE% is commonly calculated as bias % + 2 CV% (15, 16). The components of analytical errors of a single measurement result include random and systematic errors. In comparison with a single measurement, when two or more replicate measurements are made, the mean of the obtained values is more likely to be closer to the true or target value, since repeated measurements decrease the effect of the random error of the mean. Ultimately, when an infinite number of measurements are made, the effect of random error on the mean will be eliminated (16). The effect of a random error component of TAE is inversely related to the square root of the number of measurements (16, 17). If a test is repeated, its random error will decrease by in which n is the number of repeats (17). Thus, according to the number of measurement repeats, the effects of random error (imprecision) and systematic error (bias) on TAE can be calculated. In calculating bias%, we used the following formula to eliminate the effect of random error on TAE according to the number (n) of participant laboratories in each peer group:

We can define modified total analytical error (mTEa) for when the test is repeated n times. In this way, we can determine TEa for situations when the test is repeated, for example 2, 4, 10 or 20 times.

this formula was derived from when test is repeated n times.

Also, we calculated pass rates of laboratories having acceptable results during EQAP-23 and EQAP-24 with TEa of ±20% suggested by the Reference Health Laboratory of Iran and TEa of ±6% suggested by NGSP and CAP (5, 18). In this comparison, we used total mean as target value.

Commercial methods quality, as the sigma value of each method performance, was calculated according to the above two different TEa along with Bias and CV of each method (9):

Results

In EQAP-23, and EQAP-24, the CVs of homogeneity studies were 0.8% and 0.9% respectively, which were less than the allowable CV% of 1.5%. So, the hemogeneity of the vials containing whole blood samples was confirmed. In the case of stability, the results of stored vials at 4 °C by the end of the program were 8.20% in EQAP-23 and 7.95 in EQAP-24, which were in the 6% acceptable limit of related homogeneity target values, 7.88 in EQAP-23 and 7.70 in EQAP-24, respectively. The results of Mean (target value), SD, CV, TAE, bias, method quality (sigma level with TEa of 6% and 20%) of each peer group and also target value, SD, and CV of total group for participating laboratories using the desired commercial methods in EQAP-23 and EQAP-24, have been shown in . According to CAP allowable bias% of absolute 0.3%, all mean methods results were with in the acceptable range in both EQAP-23 and EQAP-24.

As shown in , when TEa of ±20% was used to evaluate HbA1c method performance, about 97% and 98% of participant laboratories had acceptable performance during EQAP-23 and EQAP-24, respectively. But when this evaluation was performed according to TEa of ±6% pass rates decreased significantly to 62% and 60%, respectively.

Data related to HbA1c methods and laboratory performance from EQAP-23 and EQAP-24 are demonstrated as multiple variable graphs in and , respectively. In these graphs, positions of different method means in respect to total mean, distribution of data around related means, error bars representing ±2SD interval, and allowable intervals for TEa of 6% and 20% are shown.

Performance during EQAP-23. Error bars represent ±2SD interval from the means (filled squares). The continuous horizontal line represents the total mean. Dotted and dashed horizontal lines show the allowable interval for total allowable errors of 6% and 20, respectively.

Performance during EQAP-24. Error bars represent ±2SD interval from the means (filled squares). The continuous horizontal line represents the total mean. Dotted and dashed horizontal lines show the allowable interval for total allowable errors of 6% and 20, respectively.

Discussion

Reliability of methods for measurements of blood HbA1c has been studied extensively throughout the world (19, 20, 21, 22, 23, 24). This reliability for methods which are commonly used in Iran has also been studied previously (12). This study and the present study have three important differences in relation to the other studies. First, in the other studies one method, usually an HPLC method, was chosen as a reference or comparative method and other methods, as test methods, were com pared with that one. In this study, we had no access to a reference or comparative method which has no analytical error and whose results could be reliable for comparing the results of other methods. To resolve this problem, commutable fresh patient blood specimens are used instead of noncommutable commercial quality materials and means of all groups are used as target values. It seems this approach is a better approach to obtain results with the least error. Second, in the most of other studies, methods were grouped according to principles of HbA1c separation, including ion-exchange chromatography, affinity chromatography, immunoturbidimetry, enzymatic, etc. But, in this study, methods were grouped according to commercial methods. In spite of using similar principles, products of different manufacturers may have different performance. For example, in this study, both Roche and Pars Azmon kits used immunoturbidimetric methods, but their performance (bias%, CV%, and sigma level) was completely different. Third, in this study, we used a formulation to account the effect of the number of repeating measurements in reducing random error and calculating analytical bias properly. This may be the first time that such a formula has been used in the calculation of bias, TAE, and mTEa.

Allowable total error (TEa) is a parameter used to define acceptable analytical performance which defines the amount of error that can be tolerated without invalidating the medical usefulness of the analytical result. If TAE is less than TEa, then the performance of the test is considered acceptable. If this error is larger than the TEa, corrections should be made to reduce the analytical error or the method should be replaced. This process ensures that laboratory test results give accurate information to physicians to manage their patients effectively (25).

According to the Diabetes Control and Complications Trial (DCCT), HbA1c results <7.0% show good glycemic control and results >8.0% show poor glycemic control. In order to properly classify a patient with an HbA1c value of 7.5%, total analytical error of measuring should be maximally ±0.5% (as the absolute value of HbA1c), which equals relative total error of ±6.7%. If TAE is greater, a patient with an HbA1c value of 7.5% will be incorrectly classified in good or poor glycemic control groups, which is obviously not acceptable (3).

Results of our study show that using CCV% = 10% and SDI > 2 for evaluating HbA1c method performance (equals to TEa of 20%) is not proper according to clinical needs and leads to misclassifying about 37% of laboratories as acceptable. So, if we want the HbA1c results to be useful for managing diabetic patients, it is necessary to use TEa = 6% which is now suggested by NGSP and used in CAP surveys for evaluating laboratory performance in measuring HbA1c (26).

As shown in , when we used TEa by 6%, quality (sigma metrics) of all the methods were unacceptable (sigma level less than 2.0). But, when we used TEa by 20%, this quality was acceptable, except for Biosystem. Means of each method bias, imprecision, and quality were determined according to TEa of 6% and 20%. According to these means, Biosystem had the least bias and the greatest imprecision. In contrast, Roche had the least imprecision and the greatest bias. Mean sigma quality showed that according to TEa of 6%, all methods had poor performance (sigma level less than 1.0), but with TEa of 20%, all methods had acceptable performance (sigma level more than 2.0), except for Biosystem.

In 2015, Huysal K. and Budak Y. (8) reported the results of their study on MQ-2000 PT HbA1c analyzer which uses HPLC and separates HbA1c by cation exchange. They used TEa of 10%, which is recommended by the Clinical Laboratory Improvement Amendments (CLIA) regulations (8, 9), and calculated the sigma level for assessment of quality of this analytical performance (9). They obtained a sigma level of less than 1.5 and showed that the analytical quality of the MQ-2000 PT HbA1c analyzer was not appropriate when evaluated on the sigma scale (9). In comparison to using TEa about 6%, when we use TEa by 10%, method performance improves and method quality sigma value increases. But the meaning of this quality improvement is not the improvement in the clinical usefulness of HbA1c results. Thus, in order to in crease clinical usefulness of HbA1c results, we should decrease TAE, but not in crease TEa.

In 2007, the CAP used wide acceptance limits of ±15% for evaluating the performance of laboratories measuring HbA1c. In 2008, the CAP narrowed this limit to ±12%, and then in 2009 to ±10%, in 2010 to ±8%, in 2011 to ±7%, and finally in 2013 to ±6% (18). If we use the sigma value to evaluate method performance in the CAP survey, surprisingly, the method performance would be too low. In this survey, acceptable limit (TEa), bias and CV are ±6%, 0.3%, and 4%, respectively. Here, limit bias is given according to the absolute amount of HbA1c. With a target value of 7%, bias% will be 4.3%. So, a method with maximum acceptable bias% and CV% could have a very low sigma of 0.4! In 2016, in the GH5-B survey, for example, Roche Cobas Integra 400 had a bias% and CV% of about 2.3% and 3.6%, respectively. In other words, this method was 1.0 sigma.

In 2015, Mosca et al. sent two fresh blood samples to 206 Italian laboratories asking for the determination of their HbA1c concentration (27). Here, 193 laboratories using analytical systems from five manufacturers (Bio-Rad Laboratories, A. Menarini Diagnostics, Roche Diagnostics, Sebia and Tosoh) obtained a global variety of 5.3% (in terms of CV) and of 3.8% at an HbA1c value of 5.6% (sample 1) and 7.8% (sample 2), respectively. With a TEa of 6.0%, pass rates were 70% and 77% of samples 1 and 2, respectively. All methods had a mean bias of ≤ 2.8% with respect to the target values, with the exception of Tosoh (bias of +6.1 and +5.8%, for samples 1 and 2, respectively) (27). Our results are comparable to theirs. Our bias is somewhat better and our inter-laboratory CVs% and pass rates are worse than theirs.

Conclusion

According to our study and comparison with the results of other related studies in different countries, we can conclude that (1) the calculated sigma and HbA1c performance methods in Iran are comparable to the other countries and still there is a need to improve these method performances all over the world (2). Our laboratory pass rate is low and inter-laboratory CV% is high and these must be improved by using closed and NGSP-certified methods. And we need to gradually tighten the TEa from the improper ±20% to proper ±6% and the Iranian clinical laboratories that have not achieved acceptable results must improve their performance methods gradually, or replace their methods with an acceptable HbA1c measuring method (3).

Table I

Weighted mean (target value), SD, CV, TAE, Bias%, and method quality according to two different TEa in EQAP-23 and EQAP-24.

Commercial Methods	No. labs	Mean (%)	SD	CV (%)	TAE (%)	Bias (%)	Method quality (sigma)
							TEa = 6%	TEa = 20%
EQAP-23
Biosystem	172	7.93	0.83	10.5	- 0.50	- 0.44	0.61	1.95
NycoCard	233	8.05	0.52	6.5	+ 1.00	+ 0.89	0.79	2.96
Pasr Azmon	120	7.99	0.73	9.1	+ 0.25	+ 0.21	0.63	2.17
Pishtaz Teb	137	7.78	0.59	7.6	- 2.38	- 2.04	1.06	2.91
Roche	20	8.12	0.32	3.9	+ 1.88	+ 1.30	1.19	4.75
Total	682	7.97	0.62	7.8	–	–	–	–
EQAP-24
Biosystem	209	7.66	0.90	11.8	+ 0.13	+ 0.11	0.50	1.69
NycoCard	337	7.63	0.53	7.0	- 0.26	- 0.24	0.90	2.91
Pasr Azmon	149	7.70	0.61	7.9	+ 0.65	+ 0.56	0.69	2.45
Pishtaz Teb	205	7.60	0.57	7.5	- 0.65	- 0.57	0.88	2.74
Roche	25	7.88	0.39	5.0	+ 3.01	+ 2.15	0.78	3.61
Total	925	7.65	0.62	8.1	–	–	–	–

Abbreviations: TAE, total analytical error; TEa, allowable total error; HbA1c, glycated hemoglobin; EQAP-23, twenty-third run of external quality assessment program; EQAP-24, twenty-fourth run of external quality assessment program.

Table II

Acceptable range and pass rates of HbA1c in EQAP-23 and EQAP-24 with different allowable total error (TEa).

Commercial Methods	No. Labs	Reported Range (%)	TEa = 6%			TEa = 20%
			Acceptable range (%)	Pass Rates		Acceptable range (%)	Pass Rates
				N	%		N	%
EQAP-23
Biosystem	172	5.90–10.00		86	50		159	92
NycoCard	233	6.80–9.20		163	70		233	100
Pasr Azmon	120	6.20–9.80		62	52		115	96
Pishtaz Teb	137	6.40–9.10	7.49–8.45	92	67	6.38–9.56	137	100
Roche	20	7.69–8.69		17	85		20	100
Total	682	5.90–10.0		420	62		664	97
EQAP-24
Biosystem	209	5.56–9.80		96	46		189	90
NycoCard	337	6.40–8.90		224	66		337	100
Pasr Azmon	149	6.20–9.10	7.19–8.11	86	58	6.12–9.18	149	100
Pishtaz Teb	205	6.20–9.00		132	64		205	100
Roche	25	7.01–8.45		17	68		25	100
Total	925	5.56–9.80		555	60		905

Abbreviations: HbA1c, glycated hemoglobin; EQAP-23, twenty-third run of external quality assessment program; EQAP-24, twenty-fourth run of external quality assessment program.