Standardization Status of Total Cholesterol Concentration Measurement: Analysis of Korean External Quality Assessment Data.

BACKGROUND: Total cholesterol concentration measurement is important in the diagnosis of dyslipidemia and evaluation of cardiovascular disease risk factors. Measurement reliability for obtaining an accurate total cholesterol concentration requires procedure standardization. We evaluated the standardization status for total cholesterol concentration measurement through Korean external quality assessment (EQA) data analysis.
METHODS: This study involved 1,670 laboratories that participated in the EQA of total cholesterol concentration measurements in 2019 for 32 products from different manufacturers. The target concentrations of three quality control (QC) materials (samples A, B, and C) were measured using the reference method and compared with EQA data. The performance criteria for total cholesterol concentration measurement were based on the National Cholesterol Education Program guidelines, with ±3% inaccuracy.
RESULTS: The target values and inaccuracies of the QC material based on the reference method measurements were 254.65±7.64, 108.30±3.25, and 256.29±7.69 mg/dL (6.59±0.20, 2.80±0.08, and 6.63±0.20 mmol/L) for samples A, B, and C, respectively. The performance criteria were not met in 42.7% laboratories for sample A, 68.4% of laboratories for sample B, and 38.0% laboratories for sample C.
CONCLUSIONS: Despite significant efforts to accurately measure total cholesterol concentrations, further actions are needed for measurement standardization. Manufacturers reporting values that differ from target values should check calibrator traceability; additional efforts to accurately measure total cholesterol concentrations are required for laboratories that use products from these manufacturers.

INTRODUCTION

Total cholesterol concentration is used to diagnose dyslipidemia and determine the risk factors for cardiovascular diseases. The recent National Cholesterol Education Program (NCEP) guidelines [1] set specific medical decision points for serum total cholesterol (e.g., desirable concentrations: <200 mg/dL [5.2 mmol/L]; high and undesirable concentrations: >240 mg/dL [6.2 mmol/L]). The medical decision points for total cholesterol concentration were based on national population studies conducted by the United States Centers for Disease Control and Prevention (CDC) and were standardized to the decision points determined using the Abell-Kendall reference method [2]. To use these decision points, clinical total cholesterol concentration measurement procedures must be standardized and harmonized within clinically meaningful limits. Using results that are neither standardized nor harmonized may lead to significantly erroneous clinical decisions. Manufacturers have made various efforts to prepare products for measuring total cholesterol concentration accurately for routine clinical use. However, accurate total cholesterol concentration measurement remains challenging with respect to traceability to the reference material, altered matrix characteristics of calibrators and controls, and other factors [2].

The degree of standardization and harmonization of total cholesterol concentration measurement procedures can be evaluated by the external quality assessment (EQA). EQA is a widely accepted tool for monitoring and improving method performance in clinical laboratories and plays a central role in achieving trueness verification standardization and harmonization [3]. The Korean Association of External Quality Assurance Service (KEQAS) has been the nation’s top authorized EQA institute for the standardization and quality management of laboratory tests in Korea since its foundation in 1965 [4]. The KEQAS was certified as an EQA provider according to ISO/IEC 17043 in 2015 [5].

Only a few studies have evaluated the accuracy of total cholesterol concentration measurement worldwide using EQA, and no such studies have been conducted in Korea [6, 7]. We report the total cholesterol concentration measurement standardization status of products from various manufacturers using KEQAS data analysis for the first time in Korea; our findings can indirectly indicate reagent performance.

MATERIALS AND METHODS

This retrospective study involved 1,670 laboratories that participated in the KEQAS assessment of 32 products from different manufacturers for measuring total cholesterol concentration in 2019. Total cholesterol concentrations in three quality control (QC) materials (samples A, B, and C) were measured in the first round of KEQAS in February 2019. The samples included a human serum powder (Bio-Rad Laboratories, Hercules, CA, USA) and were shipped to the participating laboratories in a box with ice packs under refrigeration. The total cholesterol concentration in each sample was measured using routine methods without modification of routine practice. The laboratories reported their results, methods, and instruments used.

The target values of the three QC materials (A, B, and C) were measured at the National Medical Reference Laboratory (NMRL) of the Korea Disease Control and Prevention Agency (KDCA), a member laboratory of the Cholesterol Reference Method Laboratory Network that participates in an EQA scheme for reference laboratories of the Joint Committee on Traceability in Laboratory Medicine (JCTML). Total cholesterol concentration was measured by isotope-dilution gas chromatography-mass spectrometry (ID/GC/MS), which was certified by the JCTML, and then converted to the predicted Abell-Kendall method value using a previously reported equation [8-11]. The target values of the three QC materials were determined as the means of three replicates.

Statistical analysis

Results from participating laboratories were evaluated using the standard deviation index (SDI) among peer groups in the KEQAS in April 2019. SDI was calculated as follows: (laboratory result – peer group mean)/peer group SD. An SDI >3 was considered unacceptable by the KEQAS. The peer group mean and SD were not presented when the number of laboratories per manufacturer was <10; the corresponding manufacturer data were excluded from the comparative analysis with a target value, although these data were used to calculate the total mean value. Therefore, data from 1,598 laboratories and 24 manufacturers were used in the comparative analysis. Manufacturer standardization degree was evaluated based on the performance criteria suggested by the NCEP [1, 12]; the performance criteria have a bias of ≤3% of the target value. The NCEP-based performance criteria and the KEQAS acceptable range based on the peer group mean were compared. We examined whether the KEQAS acceptable range for each laboratory group using the same manufacturer’s product met the performance criteria determined using the standard NMRL measurement procedure. Statistical analyses were performed using Microsoft Excel (Microsoft Corporation, Redmond, WA, USA).

RESULTS

The mean and SD of the total cholesterol concentration measurements reported by different manufacturers are summarized in Table 1. The KEQAS acceptable range (≤3 SD), target value measured at the NMRL of the KDCA, and the performance criteria are shown in Fig. 1. The NCEP-based performance criteria of samples A, B, and C by the NMRL were 254.65±7.64, 108.30±3.25, and 256.29±7.69 mg/dL (6.59±0.20, 2.80±0.08, and 6.63±0.20 mmol/L), respectively. The total mean values of samples A, B, and C in the KEQAS data from 1,670 laboratories were 254.5, 103.7, and 255.3 mg/dL (6.58, 2.68, and 6.60 mmol/L), respectively. Values in 42.7%, 68.4%, and 38.0% of the laboratories using samples A, B, and C, respectively, were outside the performance criteria range. In contrast to those for samples A and C, the total mean value of sample B was outside the performance criteria range. The acceptable ranges for all manufacturers were wider than the range for the target value performance criteria. Additionally, in the case of a few manufacturers, the peer group means were outside the target value performance criteria range. Three manufacturers using sample A, nine using sample B, and four using sample C had a peer group mean outside the performance criteria range. At lower concentrations (sample B), participating laboratories showed poor accuracy compared that at high concentrations (sample A and C). In the case of peer groups with high SD, even if the peer group mean falls within the performance criteria, many laboratories were found to be out of the performance criteria because the acceptable range was wide as 3 SD.

Table 1

Statistics of laboratories participating in total cholesterol concentration measurement

Manufacturer	Laboratories (N)	Sample A[*]	Sample B[*]	Sample C[*]
		Mean ± SD (mmol/L)
Total[†]	1,670	6.58 ± 0.21	2.68 ± 0.13	6.60 ± 0.20
Sekisui Medical, Tokyo, Japan	285	6.61 ± 0.16	2.74 ± 0.07	6.62 ± 0.13
Roche Diagnostics, Mannheim, Germany	273	6.44 ± 0.12	2.58 ± 0.06	6.46 ± 0.14
Beckman Coulter Inc., Brea, CA, USA	224	6.66 ± 0.16	2.57 ± 0.09	6.68 ± 0.17
Wako Diagnostics, Osaka, Japan	98	6.60 ± 0.18	2.76 ± 0.06	6.58 ± 0.16
Shenzhen Mindray Bio-Medical Electronics, Shenzhen, China	91	6.65 ± 0.12	2.72 ± 0.06	6.65 ± 0.15
Stanbio Laboratory, Boerne, TX, USA	82	6.36 ± 0.18	2.62 ± 0.14	6.45 ± 0.24
JW Medical, Seoul, Korea	76	6.65 ± 0.26	2.81 ± 0.11	6.66 ± 0.20
Biosystems, Barcelona, Spain	64	6.79 ± 0.23	2.69 ± 0.09	6.90 ± 0.18
Kyowa Medex, Tokyo, Japan	58	6.57 ± 0.09	2.75 ± 0.05	6.57 ± 0.10
Siemens Healthineers, Erlangen, Germany	49	6.39 ± 0.14	2.73 ± 0.06	6.38 ± 0.13
GDLKorea, Anyang, Korea	47	6.62 ± 0.19	2.74 ± 0.11	6.57 ± 0.13
ELITechGroup, Puteaux, France	33	6.54 ± 0.30	2.72 ± 0.14	6.67 ± 0.35
DiaSys Diagnostic Systems, Holzheim, Germany	31	6.78 ± 0.28	2.68 ± 0.15	6.80 ± 0.25
Hanlab, Chungju, Korea	27	6.61 ± 0.10	2.75 ± 0.05	6.61 ± 0.09
IVDLab, Uiwang, Korea	27	6.66 ± 0.22	2.79 ± 0.13	6.75 ± 0.26
LC Diagnostics, Quezon, The Philippines	22	6.78 ± 0.16	2.78 ± 0.11	6.82 ± 0.24
Asan Pharm, Seoul, Korea	21	6.67 ± 0.21	2.82 ± 0.09	6.75 ± 0.27
Spinreact, Girona, Spain	15	6.76 ± 0.21	2.75 ± 0.06	6.86 ± 0.16
Shinyang Diagnostics, Seoul, Korea	14	6.45 ± 0.22	2.55 ± 0.03	6.48 ± 0.21
Erba diagnostics, Miami Lakes, FL, USA	14	6.49 ± 0.21	2.70 ± 0.08	6.51 ± 0.16
Shino-Test Corporation, Tokyo, Japan	13	6.33 ± 0.13	2.65 ± 0.06	6.34 ± 0.12
Embiel, Gunpo, Korea	12	6.49 ± 0.26	2.61 ± 0.12	6.58 ± 0.24
GeneMatrix, Yongin, Korea	12	6.74 ± 0.29	2.88 ± 0.10	6.78 ± 0.19
Shinsung Pharm, Suwon, Korea	10	6.71 ± 0.36	2.94 ± 0.19	6.79 ± 0.21

*The target value and performance criteria of sample A, B, and C: 6.59±0.20, 2.80±0.08, and 6.63±0.20 mmol/L. †Manufacturers with <10 laboratories are not listed in this table but were counted in the total number.

Abbreviation: SD, standard deviation.

Fig. 1

Distribution of total cholesterol concentrations according to manufacturers, and target values and performance criteria of quality control materials (samples A, B, and C). The peer group mean is the average value of all laboratories in the same manufacturer group. Numbers from 1 to 24 were given to each manufacturer group in ascending order of peer group mean. (A) Data obtained using sample A. (B) Data obtained using sample B. (C) Data obtained using sample C.

Abbreviation: SD, standard deviation.

DISCUSSION

Although all manufacturers have tried to maintain traceability to standard material, the actual inaccuracy of products from various manufacturers was revealed by comparing the true values of QC materials used in the KEQAS. This approach of using the peer group mean as the reference has the potential to create errors when evaluating EQA results. Although the KEQAS data, which are based on peer group mean and 3 SD, suggest that <0.27% of laboratories were outside the allowed range, this study, based on true values and performance criteria, showed that a significant number of laboratories did not meet the performance criteria (42.7% using sample A, 68.4% using sample B, and 38.0% using sample C). The acceptable ranges for all manufacturers in the KEQAS were outside the range of performance criteria. A high SD group has a high probability of deviating from the performance criteria, indicating that a high SD manufacturer is more vulnerable to QC than other manufacturers.

The KEQAS results show that, despite efforts to standardize total cholesterol concentration measurement worldwide, differences in test results still exist across manufacturers and can lead to serious errors in the practical application of various clinical guidelines, including those of the NCEP. There are a variety of factors underlying errors in test results performed in clinical laboratories, and analytical errors depend largely on the manufacturers. The reproducibility of the equipment and the traceability of the calibrator affect the test results. In approaches using peer group mean, analytical performance depends on the precision of the peer group using the same manufacturer’s product, since the peer group mean is regarded as the target value. Therefore, as long as manufacturers only deal with reproducibility issues, the EQA results of user laboratories are not of concern. Statistically, only a small number of laboratories outside the 3 SD (<0.27%) will obtain unacceptable results; thus, biases originating from a specific manufacturer cannot be detected by interpreting the EQA results using the peer group mean. For globally standardized tests such as total cholesterol, accuracy-based EQA can detect bias and imprecision simultaneously because of the target value of the QC materials.

There are many options for analytical performance criteria based on clinical outcomes, including the clinician’s expert opinion, biological variability, and accrediting agency. We selected the performance criteria based on published professional recommendations [1, 12].

Although QC through the KEQAS has improved in many clinical laboratories in Korea, additional efforts are required to keep pace with the recent standardization of lipid testing worldwide. Monitoring EQA result trends and providing feedback to individual manufacturers are also necessary. One of the factors underlying the differences between manufacturers may be calibrator traceability. Some calibrators have been identified as untraceable; even those marked as traceable they show different values when measured using a standard procedure.

Overall industry performance can be elucidated using data from these accreditation and regulatory agencies (such as the KEQAS). However, the manufacturers in this study were not ranked, because the test results were not determined solely by the manufacturer’s products. Pre-analytical elements, such as technicians, transportation, vial-filling imprecision for the QC materials, varying storage stability, shipping, and in-use stability of QC materials owing to handling at user laboratories could also affect the test results. This study has a limitation that a commutability test to exclude matrix effects was not performed [13].

In conclusion, we demonstrated the biases between each peer group means and target values in total cholesterol concentration measurements. EQA data represent a useful source for comparable analysis of in vitro diagnostics. Although significant efforts have been made to accurately measure total cholesterol concentrations to date, further actions are needed by manufacturers to standardize total cholesterol concentration measurement and determine calibrator traceability accurately.