Investigation and Analysis of Hemoglobin A1c Measurement Systems' Performance for 135 Laboratories in China.

BACKGROUND: Hemoglobin A1c (HbA1c) measurement is of great value for the diagnosis and monitoring of diabetes. Many manufacturers have developed various experiments to determine the HbA1c concentration. However, the longitudinal use of these tests requires strict quality management. This study aimed to analyze the quality of HbA1c measurement systems in China using six sigma techniques to help improve their performances.
METHODS: A total of 135 laboratories were involved in this investigation in 2015. Bias values and coefficients of variation were collected from an HbA1c trueness verification external quality assessment program and an internal quality control program organized by the National Center of Clinical Laboratories in China. The sigma (σ) values and the quality goal index (QGI) were used to evaluate the performances of different groups, which were divided according to principles and instruments.
RESULTS: The majority of participants (88, 65.2%) were scored as "improvement needed (σ < 3)", suggesting that the laboratories needed to improve their measurement performance. Only 8.2% (11/135) of the laboratories were scored as "world class (σ ≥ 6)". Among all the 88 laboratories whose σ values were below 3, 52 (59.1%) and 23 (26.1%) laboratories needed to improve measurement precision (QGI <8.0) and trueness (QGI >1.2), respectively; the remaining laboratories (13, 14.8%) needed to improve both measurement precision and trueness. In addition, 16.1% (5/31) and 15.0% (3/20) of the laboratories in "TOSOH" and "ARKRAY" groups, respectively, were scored as "world class", whereas none of the laboratories in "BIO-RAD" group were scored as "world class".
CONCLUSIONS: This study indicated that, although participating laboratories were laboratories with better performance in China, the performances were still unsatisfactory. Actions should be taken to improve HbA1c measurement performance before we can include HbA1c assays in diabetes diagnosis in China.

Introduction

Glycated hemoglobin concentrations (most commonly hemoglobin A1c [HbA1c]) reflect time-averaged blood glucose during the previous 2–3 months and are used as the gold standard for long-term follow-up of glycemic control.[1] HbA1c has recently become an attractive target in the diagnosis of diabetes. The American Diabetes Association (ADA) has formally included HbA1c ≥6.5% as a diagnostic criterion for diabetes in the “Standards of Medical Care in Diabetes”[2] and “Diagnosis and Classification of Diabetes Mellitus”[3] since 2010. In addition, the World Health Organization (WHO) has published guidelines for the use of HbA1c in the diagnosis of diabetes mellitus and concluded that HbA1c can be used as a diagnostic test for diabetes, provided that stringent quality assurance tests are in place and assays are standardized to criteria aligned to the international reference values.[4] The HbA1c of 6.5% has been recommended as a cutoff point for diagnosing diabetes.

However, the “Guidelines for Clinical Application of Blood Glucose Monitoring in China” stated that “Although 6.5% was recommended as cut point for diagnosing diabetes by ADA and WHO, it is not recommended in China for now as the HbA1c measurements are short of widespread use, lack of standardization and the measurement performance cannot meet clinical requirement, etc”.[5] Hence, the accuracy of HbA1c assays is essential in China. Internal quality control (IQC) and traditional external quality assessment (EQA) programs can help evaluate the measurement accuracy of participant laboratories rather than trueness.

Therefore, to evaluate the measurement of trueness of HbA1c assays for laboratories and facilitate further improvements in diagnostic approaches, an HbA1c trueness verification EQA program was organized by the National Center for Clinical Laboratories (NCCL) in China. Here, we reported the results of an analysis of the trueness of HbA1c assays in laboratories participating in this program.

Methods

Ethical approval

All laboratories were voluntary to participate the investigation, and this study was approved by the Ethics Committee of Beijing Hospital.

Study design

A total of 135 laboratories in China were included in this investigation in 2015. Sigma (σ) value and quality goal index (QGI) were used to evaluate the performance of HbA1c assays. To calculate σ value and QGI, we determined bias, allowable total error (TEa), and coefficients of variation (CV), as described below.

Bias

An HbA1c trueness verification EQA program was organized by the NCCL in China. The remaining HbA1c assay samples in different clinical laboratories were collected and then stored at −80°C for at least 1 week to fracture the erythrocytes. All the collected samples were thawed at 4°C overnight after collecting adequate blood samples. Samples were analyzed at four concentrations as needed, and blood clots were eliminated. The blood mixtures were then divided into 200-μl plastic cryopreserved tubes (100–200 μl/tube) and stored at −80°C. Each participating laboratory received 12 samples with four concentrations of HbA1c (three samples for each concentration). Four samples (lots 201511, 201512, 201513, and 201514) were measured on three different working days, with each sample evaluated five times. Therefore, a total of sixty results were obtained for each sample. Laboratories were required to report all results with principles, instruments, reagents, and calibrators used for HbA1c assays online. The target values were assigned using the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) reference measurement procedure. The percentage difference between the laboratory-tested value and the target value was defined as the bias. The bias for each laboratory was represented by the average bias from the four concentrations of the samples.

Coefficients of variation

An HbA1c IQC investigation was initiated by the NCCL in 2015. Participant laboratories were asked to provide IQC data, including cumulative CVs of results in-control online. To calculate the cumulative CV, laboratories collected all the results of controls from the first day on which the same lot of the control was used to the last day until June 2015. Outliers were removed for analysis. For laboratories in which two levels of quality controls were used, the total CVt of each sample was calculated as CVt= ([CVlevel 12 + CVlevel 22]/2)1/2, whereas for laboratories in which only one level of QC was used, CVt = CVlevel 1.

Tolerance limits of measurement procedures

The TEa (8.0%) of the HbA1c EQA program set by the NCCL was used as the TEa; 1/3 the TEa (2.7%) was used as the allowable imprecision (CVa). The allowable bias (4.5%) of the HbA1c trueness verification EQA program was used as the allowable bias (Biasa) in this study.

Sigma value

The σ values for point-of-care glucose meters in the participating laboratories were calculated based on the following equation: σ = (TEa− |bias|)/CV.[678] Here, TEa was the allowable total error, as described above. Bias, as specified, was the average bias obtained in the HbA1c trueness verification EQA program. CVs were obtained from the IQC investigation. The performance of the participating laboratories was scored based on the calculated σ values: σ ≥ 6 was scored as “world class”; 5 ≤ σ < 6 was scored as “excellent”; 4 ≤ σ < 5 was scored as “good”; 3 ≤ σ < 4 was scored as “marginal”; and σ < 3 was scored as “improvement needed”.

Quality goal index

If the measurement procedure was categorized as “improvement needed (σ < 3)”, the QGI was calculated based on the following equation: QGI = |bias|/(1.5 × CV). QGI values of <0.8 indicated that the precision of the measurement procedure needed improvement; QGI values of >1.2 indicated that the trueness needed to be improved; and values of 0.8≤ QGI ≤1.2 indicated that both the precision and trueness needed to be improved.[9]

Statistical analysis

In this study, laboratories were divided into different groups according to principles and instruments. Excel 2010 (Microsoft Corporation, Redmond, WA, USA) was used to calculate biases, CVs, σ values, and QGIs for each laboratory. The percentages of laboratories in each group meeting bias criteria, imprecision criteria, and both bias and imprecision criteria were calculated. The constituent ratios of σ values for each group were calculated as were the QGIs and constituent ratio QGIs for laboratories with σ values of <3.

Results

General and grouping situation of analytic systems

Principles, instruments, reagents, and calibrators used by the participating laboratories are shown in Table 1. All laboratories were divided into three principle groups according to different principles, as follows: “high-performance liquid chromatography” (HPLC; automated cation exchange HPLC: 115 laboratories, and automated affinity chromatography HPLC: 6 laboratories); “enzymatic method” (3 laboratories); and “immunoturbidimetry” (11 laboratories). There were 121 laboratories in the group “HPLC”, and the instruments used in the different laboratories varied greatly. Sixty laboratories employed Bio-Rad instruments (D-10: 26 laboratories; Variant II: 17 laboratories; Variant II Turbo: 17 laboratories), 20 laboratories employed instruments from ARKRAY Inc., Japan (HA-8160: 10 laboratories; HA-8180: 10 laboratories), 31 laboratories employed instruments from TOSOH, Japan (G7: 4 laboratories; G8: 27 laboratories), six laboratories employed instruments from PRIMUS (Primus HPLC, USA), and four laboratories employed instruments from Hui-zhong (MQ-2000PT, China). Since different measurement systems may have different performances, we divided the participating laboratories into seven groups, as follows: “BIO-RAD”, “TOSOH”, “ARKRAY”, “PRIMUS”, “Hui-zhong”, “enzymatic method”, and “immunoturbidimetry”. A system was considered homogeneous if the same manufacturer supplied instruments, reagents, and calibrators; all the other systems were considered heterogeneous.[10] With the exception of one, three, and six laboratories that used heterogeneous analytic systems for HbA1c assays in the “PRIMUS”, “enzymatic method”, and “immunoturbidimetry” groups, respectively, all laboratories employed homogeneous analytic systems in this study.

Table 1

Principles, instruments, reagents, and calibrators used by all laboratories in this study

Principles and instruments	Reagents	Calibrators	Number of laboratories
Automated cation exchange high-performance liquid chromatography
ARKRAY HA-8160	ARKARY	ARKARY	10
ARKRAY HA-8180	ARKARY	ARKARY	10
BIO-RAD D-10	Bio-Rad	Bio-Rad	26
BIO-RAD Variant II	Bio-Rad	Bio-Rad	17
BIO-RAD Variant II Turbo	Bio-Rad	Bio-Rad	17
TOSOH G7	Tosoh	Tosoh	4
TOSOH G8	Tosoh	Tosoh	27
Hui-zhong MQ-2000 PT	Hui-zhong	Hui-zhong	4
Automated affinity chromatography HPLC
PRIMUS HPLC	Primus	Bio-Rad	1
PRIMUS HPLC	Primus	PRIMUS	5
Enzymatic method
Beckman AU	First Chemistry	First Chemistry	2
Mindray BS	Mindray	Mindray	1
Immunoturbidimetry
Abbott architect	Simes Sikma	Simes Sikma	1
Beckman Synchron	Medical System	Medical System	1
Roche cobas	Roche	Roche	4
Roche others	Roche	Roche	1
Dade Behring Dimension	Siemens	Siemens	1
Hitachi	Medical System	Medical System	1
Hitachi	Randox	Randox	1
Hitachi	Zhe-jiang Dongou	Zhe-jiang Dongou	1

HPLC: High-performance liquid chromatography.

Evaluation of bias and coefficients of variation

Among the 135 laboratories who reported their HbA1c results, 77.0% (104/135) met Biasa criteria, 62.2% (84/135) met CVa criteria, and 51.1% (69/104) met both Biasa and CVa criteria. The percentages of laboratories meeting Biasa and/or CVa criteria varied among the groups. More laboratories in the “BIO-RAD” (53.3%, 32/60) and “TOSOH” (61.3%, 19/31) groups and fewer laboratories in the “immunoturbidimetry” group (27.3%, 3/11) met both Biasa and CVa criteria. These data are presented in Table 2.

Table 2

Laboratories meeting bias criteria, imprecision criteria, and both criteria in this study

Principle and instrument groups	Number of laboratories	Laboratories meeting different criteria, n (%)
		Biasa	CVa	Both biasa and CVa
HPLC	121	97 (80.2)	76 (62.8)	64 (52.9)
BIO-RAD	60	53 (88.3)	36 (60.0)	32 (53.3)
TOSOH	31	26 (83.9)	22 (71.0)	19 (61.3)
ARKRAY	20	11 (55.0)	13 (65.0)	8 (40.0)
PRIMUS	6	3 (50.0)	3 (50.0)	3 (50.0)
Hui-zhong	4	4 (100.0)	2 (50.0)	2 (50.0)
Enzymatic method	3	2 (66.7)	2 (66.7)	2 (66.7)
Immunoturbidimetry	11	5 (45.5)	6 (54.6)	3 (27.3)
All	135	104 (77.0)	84 (62.2)	69 (51.1)

HPLC: High-performance liquid chromatography; Biasa: Allowable bias; CVa: Allowable coefficient of variation.

Sigma metric

Laboratories were categorized based on their σ values, which were calculated as described above. Our results indicated that the majority of participating laboratories (65.2%) were scored as “improvement needed”, with less-than-optimal measurement performance. Only 8.2% (11/135) of laboratories were scored as “world class (σ ≥ 6)”. In addition, 18 (13.3%), 11 (8.2%), and seven (5.2%) laboratories were scored as “excellent (3 ≤ σ < 4)”, “good (4 ≤ σ < 5)”, and “marginal (5 ≤ σ < 6)”, respectively. Laboratories in the “TOSOH” group showed relatively high σ levels, whereas the σ values of laboratories in the “PRIMUS” group were relatively low. Constituent ratios of σ values for different groups are shown in Table 3.

Table 3

Constituent ratios of sigma values for different groups in this study

Principle and instrument groups	Number of laboratories	Number of laboratories (%)
		σ<3	3≤σ<4	4≤σ<5	5≤σ<6	σ≥6
HPLC	121	80 (66.1)	16 (13.2)	10 (8.3)	7 (5.8)	8 (6.6)
BIO-RAD	60	43 (71.7)	8 (13.3)	5 (8.3)	4 (6.7)	(0.0)
TOSOH	31	15 (48.4)	4 (12.9)	5 (16.1)	2 (6.5)	5 (16.1)
ARKRAY	20	14 (70.0)	2 (10.0)	(0.0)	1 (5.0)	3 (15.0)
PRIMUS	6	5 (83.3)	1 (16.7)	(0.0)	(0.0)	(0.0)
Hui-zhong	4	3 (75.0)	1 (25.0)	(0.0)	(0.0)	(0.0)
Enzymatic method	3	1 (33.3)	1 (33.3)	(0.0)	(0.0)	1 (33.3)
Immunoturbidimetry	11	7 (69.6)	1 (9.1)	1 (9.1)	(0.0)	2 (18.2)
All	135	88 (65.2)	18 (13.3)	11 (8.2)	7 (5.2)	11 (8.2)

HPLC: High-performance liquid chromatography.

Among the 135 participating laboratories, there were 88 laboratories whose HbA1c measurement performances needed to be improved. For these laboratories, QGIs were further calculated to provide additional advice on improvements. As shown in Table 4, 59.1% (52/88) of laboratories were scored as “improvement needed”, with regard to needing to improve the precision of the measurement procedure (QGI <0.8). In addition, 26.1% (23/88) of laboratories needed to improve the trueness of the measurement procedure (QGI >1.2). Finally, 14.8% (13/88) of laboratories needed to improve both the precision and trueness of the measurement procedure.

Table 4

Constituent ratios of QGI for laboratories with σ<3 in this study

Principle and instrument groups	Number of laboratories	Number of laboratories (%)
		QGI <0.8	0.8≤ QGI ≤1.2	QGI >1.2
HPLC	80	47 (58.8)	13 (16.3)	20 (25.0)
BIO-RAD	43	33 (76.8)	3 (7.0)	7 (16.3)
TOSOH	15	9 (60.0)	1 (6.7)	5 (33.3)
ARKRAY	14	3 (21.4)	5 (35.7)	6 (42.9)
PRIMUS	5	1 (20.0)	2 (40.0)	2 (40.0)
Hui-zhong	3	1 (33.3)	2 (66.7)	(0.0)
Enzymatic method	1	1 (100.0)	(0.0)	(0.0)
Immunoturbidimetry	7	4 (57.1)	(0.0)	3 (42.9)
All	88	52 (59.1)	13 (14.8)	23 (26.1)

HPLC: High-performance liquid chromatography; QGI: Quality goal index.

Discussion

Studies have shown that 11.6% of Chinese adults (113.9 million individuals) have diabetes.[11] In addition, a series of Diabetes Control and Complication Trials of insulin-dependent diabetes mellitus showed that the risk of chronic complications associated with diabetes is reduced by 35–45% as the HbA1c level decreases by 1%.[1213141516] Thus, HbA1c assays are essential in the diagnosis and treatment of diabetes. Moreover, compared with blood glucose testing, HbA1c assays have major advantages, including no requirement for fasting or collection of blood samples at specific times. Accordingly, HbA1c assays are important diagnostic indicators in diabetes.

However, HbA1c is not often used as a diagnostic indicator of diabetes because the measurement performance of HbA1c assays is not sufficient. Thus, in this study, we investigated the current state of HbA1c standardization in clinical laboratories in China. Results from different laboratories were compared and assessed to obtain their measurement performances. EQA programs using commutable materials with values assigned by reference methods are essential. In the present study, the target values were assigned by the IFCC reference measurement procedure. Trueness and precision were used to evaluate measurement performance in clinical laboratories. Compared with other similar reports, the number of participating laboratories in this study was not small (135 laboratories). Among all the participating laboratories, 55.1% of laboratories were within both the 4.5% limit for trueness and 2.7% limit for precision. Moreover, 62.2% of CVs were within the CVa. In a study in Norway,[17] 45% of laboratories met the limit for HbA1c trueness and the limit for imprecision, and almost all CVs were <2%. In our investigation, the overall pass rates of bias were 59.1% with a 4.5% limit for trueness, which was better than that in a German study,[18] in which the pass rates were about 57% for ±5% with a percentage limit of 5% for bias.

Although more than half of the participating laboratories could satisfy the trueness and imprecision requirements separately, only 34.8% of laboratories achieved minimal sigma values (σ ≥ 3) and 8.2% of laboratories were scored as “world class (σ ≥ 6)”. Laboratories should continue improving their analytical quality to obtain a better sigma level, even when they have satisfied trueness and imprecision performance requirements. To provide additional advice on problems in measurement procedures, QGIs were calculated for laboratories scored as “improvement needed”. More than half of the participating laboratories should focus on improving the precision of HbA1c assays, and 26.1% of laboratories should pay more attention to trueness of measurement. However, 14.8% of laboratories still needed to improve both precision and trueness.

Various instruments are available for detecting HbA1c, as shown in Table 1, similar to a report of HbA1c measurement in Norway.[17] Different measurement systems have different constituent ratios for σ metric values. Most of the participating laboratories used HPLC for HbA1c testing; Bio-Rad, TOSOH, and ARKRAY instruments accounted for the majority of results. Notably, 16.1% and 15.0% of laboratories in the “TOSOH” and “ARKRAY” groups were scored as “world class”, respectively, whereas no laboratories in the “BIO-RAD” were scored as “world class”. Compared with the ARKRAY instrument, the TOSOH instrument appeared to perform better, showing fewer laboratories (48.4%) with σ values of <3. However, in another study, the two groups showed similar performance.[19] Consistent with these results, more laboratories scored as “improvement needed” in the “BIO-RAD” group compared with those in the “TOSOH” and “ARKRAY” groups. Only one laboratory in the “enzymatic method” group was scored as “improvement needed”, showing poor precision performance. In addition, seven laboratories in the “immunoturbidimetry” group had σ values <3, among which three laboratories needed to improve trueness performance, whereas the remaining four laboratories needed to improve precision performance.

Laboratories can evaluate measurements and obtain information regarding necessary improvements using the σ metric and QGI. To calculate σ values and QGIs, it is necessary to determine biases, TEas, and CVs.[2021] In this study, CVs were from routine operation data in laboratories (i.e., cumulative CVs of IQC results in-control) and may partly reflect the actual situation. Bias is estimate of the systematic measurement error, while trueness is the closeness of agreement between the average of an infinite number of replicate-measured quantity values and the reference quantity value. The measurement of trueness is usually expressed in terms of bias.[22] Trueness verification EQA programs, which are not affected by matrix effects of control materials, can overcome the deficiencies of traditional EQA programs. Ideally, these programs can simultaneously evaluate measurement trueness for hundreds or even thousands of laboratories, thus contributing to a comprehensive understanding of the overall status of HbA1c measurement in China.

To achieve worldwide standardization, the IFCC developed a reference measurement procedure for higher metrological order, which is embedded in a global network of reference laboratories in Europe, Asia, and the United States of America.[23] However, from our data, the performance of laboratories was not as good as expected. The reliability of HbA1c measurement depends on bias (related to proper calibration) and precision (related to the reproducibility of the method). In terms of bias, EQA program providers, manufacturers, and laboratories all have responsibilities. For example, EQA/PT providers are responsible for conducting and reporting on investigations to ensure that the results of participating laboratories meet the evaluation criteria. When the participating laboratories obtain unacceptable results, proper advice and instructions should be provided to facilitate further improvements. The manufacturers should ensure the traceability of results obtained in clinical laboratories, as required by the European directive 98/79 IEC on in vitro diagnostic medical devices. Laboratories themselves can verify trueness through either purchasing certified reference material from reference material producers or participating in trueness verification EQA programs (reference materials are provided by EQA providers uniformly, and laboratories are required to test reference materials in accordance with the established procedures). Precision, as determined by the intra-laboratory CV, reflects the reproducibility and stability of the assay, the precision of the instrument, and the lot-to-lot consistency of the reagents and calibrators. For laboratories with large intra-laboratory CVs, it is imperative to improve the frequency of calibration to guarantee the stability of the assay.

From our investigation, it can be observed that the percentages of laboratories that could meet the bias limits were higher (77.0%) than those (62.2%) that could meet the CVa limits. Among the 88 laboratories that needed improvements, more than half (59.1%) needed to improve precision. Therefore, improving the intra-laboratory precision appeared to be more important than improving trueness in the current state in China. Accordingly, more effort should be made to improve trueness and precision.

In the present study, σ indexes were used to assess the performance of laboratories, providing a new perspective on assay performance. However, this study was limited by the small number of laboratories in some groups, which may have contributed to deviations in the data. However, since trueness verification EQA programs have high transport and storage condition demands, the participating laboratories in this study were all EQA customers of the NCCL, with more tertiary hospitals and fewer second-class or other hospitals. Thus, these hospitals may have shown better performance and laboratory practices than some other hospitals in China and may not be representative of all hospitals in China. We hope that more economical methods can be developed in the future, allowing us to include more laboratories in trueness verification EQA programs.

In conclusion, although the participating laboratories were laboratories with better performance in China, the performances of these laboratories were still unsatisfactory, with more than half of the laboratories scored as “improvement needed (σ < 3)”. Actions should be taken to improve HbA1c measurement performance before we can include HbA1c assays in diabetes diagnosis in China. The σ metric is a useful tool that can be used to evaluate measurement performance and facilitate the identification of directions for improving assays (trueness, precision, or both). Laboratories are advised to take an active part in EQA programs, set suitable performance goals, and strive to obtain these goals.

Financial support and sponsorship

This study was supported by grants from the National High Technology Research and Development Program of China (863 Program; No. 2011AA02A102 and No.2011AA02A116).

Conflicts of interest

There are no conflicts of interest.