The Evaluation of Analytical Performance of Immunoassay Tests by using Six-sigma Method.

BACKGROUND: The Six-Sigma Methodology is a quality measurement method in order to evaluate the performance of the laboratory. In the present study, it is aimed to evaluate the analytical performance of our laboratory by using the internal quality control data of immunoassay tests and by calculating process sigma values.
METHODS: Biological variation database (BVD) are used for Total Allowable Error (TEa). Sigma values were determined from coefficient of variation (CV) and bias resulting from Internal Quality Control (IQC) results for 3 subsequent months. If the sigma values are ≥6, between 3 and 6, and <3, they are classified as »world-class«, »good« or »un - acceptable«, respectively.
RESULTS: A sigma value >6 was found for TPSA and TSH for the both levels of IQC for 3 months. When the sigma values were analyzed by calculating the mean of 3 months, folate, LH, PRL, TPSA, TSH and vitamin B12 were found >6. The mean sigma values of CA125, CA15-3, CA19-9, CEA, cortisol, ferritin, FSH, FT3, PTH and testosteron were >3 for 3-months. However, AFP, CA125 and FT4 produced sigma values <3 for varied months.
CONCLUSION: When the analytical performance was evaluated according to Six-Sigma levels, it was generally found as good. It is possible to determine the test with high error probability by evaluating the fine sigma levels and the tests that must be quarded by a stringent quality control regime. In clinical chemistry laboratories, an appropriate quality control scheduling should be done for each test by using Six-Sigma Methodology.

Introduction

In clinical laboratories, medical laboratory processes can be devided in three basic stages; pre-analytical, analytical and post-analytical. Errors that will appear in each phase may negatively affect the test results and the magnitude of total error should be calculated for each phase (1, 2). When the laboratory errors are evaluated according to these stages, it is found that errors mostly occur at the pre-analytical phase, secondly at post-analytical phase and finally the lowest rate is found at the analytical phase (3).

There are two types of measurement errors: random and systematic errrors. Inaccuracy and imprecision, which are the typical features of analytical performance, are basic parameters for systematic and random errors (4). These parameters are expressed as bias and coefficient of variation (CV), and can be used to calculate the total error (TE) (5). The total error of a test is calculated by: TE = Bias + 1.65CV.

TEa is a useful parameter for determinating required laboratory test quality which combines the effects of systematic and random errors (6). The TEa values of several clinical chemistry analytes have been reported and they are accessible (7, 8).

The Six-Sigma Methodology may be used to evaluate the quality of the analytical phase by combining bias, imprecision and TEa (9). In 1986, Motorola Inc. started to use Sigma metrics as a statistical-based method in order to reduce the variation in electronic manufacturing processes in the USA. It contains 5 phases including define, measure, analyze, improve and control (DMAIC) (5). These phases are universal, and they can be performed in industry, business and health sectors (2). The sigma value of a test enables to determine targets for improving the quality of the test in laboratory, or to accept the current quality of the test if the quality is adequate (10).

Sigma Metric is calculated by using the formula of sigma (s)=(TEa–bias)/CV (5, 11). High sigma values means low analytical errors and acceptable test results (6). Low sigma metric value is accepted as an error or a defect. The defect value is measured in defects per million (DPM) (12). The Six-Sigma is focused to control a process in 6 standard deviations (SD) and it is equal to 3.4 DPM. The success with Six Sigma Quality is accepted as the perfection standard. A performance at the 3-sigma level is considered as the minimum quality for manufacturing process (2, 5).

In the present study, we evaluated the analytical phase by determining the analytical performances and calculated sigma values by using internal quality control (IQC) data of 18 tests.

Materials and Methods

The present study was conducted in the clinical chemistry laboratory of the Ahi Evran University Research and Education Hospital. Internal quality control (IQC) data of 18 analytes were analyzed retrospectively over a period of 3 months from June 2015 to August 2015 using Cobas e601 analyzer (Roche Diagnostics, Germany). 18 serum immunoassay tests were included: α1-fetoprotein (AFP), cancer antigen 19-9 (CA19-9), carcinoembryonic antigen (CEA), cortisol, ferritin, folate, follicle-stimulating hormone (FSH), free thyroxine (FT4), free triiodothyronine (FT3), luteinizing hormone (LH), parathyroid hormone (PTH), prolactin (PRL), testosteron. Thyroid-stimulating hormone (TSH), total prostate-specific antigen (TPSA) and vitamin B12. All reagents were obtained from Roche and used according to the manufacturer’s package inserts.

Both normal (IQC1) and pathological (IQC2) levels of QC materials were assayed before analysing of patient samples every day. Cobas PreciControl Tumor Marker, PreciControl Varia and PreciControl Universal QC materials were belong to Roche Diagnostics company and QC values were based on the reference method. The instruments was calibrated regularly. IQC data were obtained from Laboratory Data Management System (SARUS LIS, Labıratory Information System). Faulty values arising from false control samples were excreted.

Following the determination of mean and SD values, CV, bias and sigma values were calculated according to the following formulations. The target mean, labaratory mean and SD values of each test are presented in .

Coefficient of variation calculation

Imprecision, expressed as coefficient of variation (% CV) was determined from the calculated mean and Standard deviation evaluated from internal quality control (IQC) data.

CV is the ratio of the SD which is obtained from a data set to the mean (x̄) and it is expressed as a percentage of variance to the mean; CV(%)=(SD/Mean of IQC data) × 100.

Determination of Bias

Bias was calculated as the percentage difference of the average of observed results for each analyte from the target values provided in the roche control package inserts. Percent bias values of each test were calculated separately between June, July and August 2015.

%Bias= [(our laboratory mean of IQC data–target mean of IQC data)/target mean of IQC data]×100

Allowable total error (TEa)

The sigma metrics were calculated using TEa goals from one source in order to understand the effect of TEa on estimates of Sigma metrics: the Desirable Biological Variation Database (8). These source is regularly updated and can be freely accessed through http://www.westgard.com.The TEa values of each test are presented in .

Sigma metric calculation

Sigma (σ) value was used in order to determine the analytical performance characteristics of sigma value tests by using CV (obtained from IQC data), Bias% and TEa values. Sigma value calculated using the standard equation:

Sigma metric (σ)= (%TEa-%Bias) / %CV

Sigma values were used to determine the analytical performance characteristics of the test. A sigma level <3 is an indication of a poor performance procedure, whilst a good performance is indicated by a sigma level >3. Above six sigma level is a world class performance (13).

Results

shows the target mean, laboratory mean and the calculated standard deviation values of the two levelsnamely normal (IQC1) and pathological (IQC2) quality controls run in our laboratory for the different parameters.

The %CV values of pathological and normal level of IQC were found as < 5 for the tests including CEA, ferritin, FT3, FT4, FSH, LH, PRL, testosterone and TSH for 3 subsequent months. The CV values of AFP, CA125, CA15-3, CA19-9, cortisol, folat, PTH, TPSA and vitamin B12 varied as 5 or >5 according to the months and IQC levels, and they didn’t exceed the value of 10%. highlights TEa, bias and coefficient of variation (CV) sigma values of the two levels of quality control for the different parameters.

The sigma values of several tests were found as >6 by using desirable biological variability TEa targets, and the maximum value was determined as 26,05. However, several analytes produced sigma values <3: AFP (IQC2, June, IQC1-IQC2, August) CA125 (IQC1, July); FT4 (IQC2, June, IQC1-IQC2, July, IQC1-IQC2, August). The mean sigma values of CA125, CA15-3, CA19-9, CEA, cortisol, ferritin, FSH, FT3, PTH and testosteron were >3 for 3-months. The sigma value >6 was observed for TPSA and TSH for both levels of IQC for 3 months. When the sigma values were analyzed by the mean of 3 months; folate, LH, TPSA, PRL, TSH and Vitamin B12 for both levels of IQC were found as >6, CA 125, CA 15-3, testesteron and FSH for both levels of IQC were found as 3–6 for 3 months. Complete Sigma metrics for 18 assays are shown in and .

Discussion

In order to evalute the imprecision and accuracy of laboratory tests in clinical biochemistry laboratories, internal and external quality controls were studied at different levels. Westgard rules are followed during the evaluation of internal quality. Quality control materials are used to follow the performance of analytical methods (14). The Six-Sigma Method is one of the important quality control analyses which are used in the evaluation of quality and performance and, it is based on statistical calculations (15).

The Six-Sigma Method enables the quantitative comparison of various auto analyzers, laboratories and methods around the world (16). If the 6 standard deviation between the mean of a test and the upper and lower limits of the test are conserved, the errors can be minimized in the laboratory (17).

Systematic error indicator, bias, random error indicator, CV and TEa are required in order to calculate the sigma level that is used for determining analytical process (2).

The tolerance limits of the laboratory were expressed as TEa. If the difference between the actual analytical concentration of the patient samples and reported concentration is higher than TEa value, it is thought that the result is not confidential (17). The TEa is the degree of variation in a test which is used for an important clinical decision about an advanced survey or treatment (5). In the present study, Desirable Biological Variation Database including the TEa values of all analyzed parameters were used (12).

The CV is used to define the variation of a test and it is expressed as a percentage of variance to the mean. It gives a general idea about the performance of a method. The CV values ≤5% and ≥10% mean that analytical method and analyzer have a good performance or have an inadequate performance, respectively (14).

In a study performed at Architect i2000SR auto-analyzer by Litten J et al. (18) the control CV% of CEA, total PSA, FT3, FT4, TSH, ferritin, FSH and vitamine B12 immunassay tests varied between 1.34% and 18.87%; and the most of CV values were found below 5%. In the present study, the CV% values of pathological and normal level of IQC were found as < 5 for the tests including CEA, ferritin, FSH, FT3, FT4, TSH, LH, PRL and testosterone for 3 subsequent months. The CV values of other tests varied as <5 or >5 over the months, but didn’t exceed the value of 10%. The reasons for variability of CV and bias may be errors during the calibrator preparation, instability of the IQC during transport or storage and sample handling of laboratory technicians. For better CV and bias we should identify a protokol for transportation, preparation and aliquoting the IQC and calibrator samples to prevent differences between laboratory technicians during assay.

Gulbahar et al. (19) performed a study at Roche/Cobas e602 autoanalyzer and they compared the two level IQC sigma values of TSH, FT3 and FT4 with two immunoassay analyzer. When the sigma values were calculated, TSH and FT4 were found as »world-class« and »unacceptable«, respectively, in both analyzers, and FT3 was found as »unacceptable« and »good« for two level IQC of the first analyzer and the second analyzer, respectively.

In a study performed by Ercan et al. (20) found at Beckman Coulter UniCel® DxI800 Immunoassay System autoanalyzer, sigma values were found as 5.18/6.14, 0.35/1.14, 1.88/0.85 for first and second level IQC, respectively.

In the present study, TSH was determined as >6 sigma in both level quality control for 3 months. The sigma value for FT3 was between 3 and 6 in first level quality control for 3 months while it was >6 and between 3 and 6 in the second level control in June and July, and August, respectively. For TSH, FT4 and FT3, the mean sigma values of three months were found as 13,06/16,13, 3,97/3,69, 3,75/6,57 for first and second level, respectively. According to these values, the TSH was in »World-class« for both levels; FT3 was in »good« and »World-class« for first and second level, respectively, and FT4 was »good« for both levels. Since sigma values differ according to imprecision and/or bias analytical concentration, they may vary between the different levels of a control.

Aksoy et al. (21) found that the sigma values for AFP, cortisol, ferritin and total PSA were 2,49/ 3,20/3,53/1,21/6,46, respectively, by using the TEa values of DBV database in Beckman Coulter DXI 800 autoanalyzer.

In the present study, the sigma values for AFP, cortisol, ferritin and total PSA tests were found as 6,98/11,17, 4,15/6,33, 6,77/7,35, 13,62/13,42 for the first and second level IQC, respectively.

In a study performed by Ercan et al. (20) at Beck man Coulter UniCel® DxI800 Immunoassay System autoanalyzer, the sigma values for vitamin B12 and folate tests were found as 4.38/4.01, 8.12/9.7 for the first and second level IQC, respectively. In our study, the sigma values for vitamin B12 and folate tests were 12,52/10,67, 7,89/9,83 for the first and second level IQC, respectively.

The differences in sigma values between our study and other studies may depend on the autoanalyzer, quality control materials or pre-nanalytical and post-analytical conditions (22, 23). Sigma metric values are necessary for the determination of acceptability criteria of IQC and, the design and application of rational control design according to sigma values with the aid of Westgard Operational Specifications Chart (OPSpecs chart) in clinical biochemistry laboratory (9, 17).

The Six-Sigma scale is generally evaluated between 0 and 6, and it may exceed the 6 Sigma value in case of low variability. 3-Sigma is acceptable for a process and it is evaluated as minimum performance. If the performance is below 3 Sigma, the process is evaluated as unstable and unacceptable (24). For 6 Sigma (or more) values, n (number of controls per day) and the control limits should be two and 3.5 SD, respectively; for 5 Sigma value, n and the control limits should be 2 and 3.0 SD, respectively; for 4 Sigma value, n and the control limits should be 4 and 2.5 SD for multirule procedures, respectively; for 3 Sigma value, n should be 6 or 8 for multirule procedures. Method performance should be developed before performing a routine study below 3 Sigma value (9, 17).

For parameters like, folate, LH, PRL, TPSA, TSH and vitamin B12 sigma metrics value in the mean of 3 months is above 6. So, for these parameters, the QC protocol does not need any change and patient results can be released. For parameters like, CA 19-9, CA 15-3, CEA, ferritin, PTH, FT3, cortizol, FSH and testesteron, the sigma metrics values in the mean of 3 months are between 3 to 6. For these parameters, QC monitoring should be done, but still it is acceptable.

However, several analytes produced low Sigma values <3: CA125 AFP and FT4. For these parameters especially for FT4 a very stringent internal QC has to be followed, and the frequency of internal QC should be increased and corrective action should be taken for these parameters. In our laboratory calibrations of parameters with low six sigma levels are more frequently performed now and the number of daily IQC have been increased.

The Six-Sigma Methodology is an effective method for the evaluation of analytical stage, the quality measurement of the laboratory tests and the optimization of quality control rules according to sigma values. IQC practices should be specific to the test and they should be generated in accordance with the sigma values of each test. In the present study, according to the 6 sigma values the analytical performance of our laboratory is found as »world-class« or »good«. IQC studies should be more controlled for parameters with sigma values <3, CA125, AFP and FT4. In addition to analytical process, the analyses of pre-analytical and post-analytical processes should be performed for the evaluation of the general performance of the laboratory.

Table I

The target mean, laboratory mean and SD values of each test.

Assay name	IQC 1							IQC 2
	Target mean	June		July		August			June		July		August
		Lab mean	SD	Lab mean	SD	Lab mean	SD		Lab mean	SD	Lab mean	SD	Lab mean	SD
AFP	9.28	8.64	0.44	8.63	0.41	8.64	0.44	123	103.4	2.39	108.98	3.61	113.50	6.24
CA125	32.1	35.41	2.95	35.60	3.46	35.41	2.95	107	108.10	7.20	114.00	9.00	113.30	5.30
CA15-3	20.7	19.68	0.68	19.33	0.71	19.68	0.68	97.9	94.33	4.05	87.89	2.36	93.50	4.80
CA19-9	22.8	21.81	0.99	20.47	0.90	21.81	0.99	86	99.78	7.42	98.80	3.96	94.34	8.56
CEA	4.8	4.64	0.20	4.41	0.15	4.64	0.20	54.2	47.88	0.78	47.06	0.86	50.88	2.29
TPSA	4	3.80	0.17	3.66	0.10	3.80	0.17	40.7	40.12	0.94	39.39	0.79	40.17	2.02
Vitamin B12	476	491.70	18.58	479.00	5.40	491.70	18.58	896	943.50	47.91	895.40	21.08	938.80	16.48
Ferritin	146	143.55	6.48	141.60	4.56	143.55	6.48	928	918.65	12.71	896.00	33.52	897.50	39.59
Folat	3.89	3.83	0.24	3.86	0.20	3.83	0.24	11.7	10.98	0.32	11.84	0.73	11.02	0.63
PTH	62.8	63.70	2.86	56.69	3.71	63.70	2.86	195	173.55	3.00	171.00	9.02	190.45	6.38
FT3	3.96	4.06	0.10	4.05	0.10	4.06	0.10	18.6	18.57	0.22	18.55	0.33	18.98	0.49
FT4	1.24	1.23	0.03	1.22	0.03	1.23	0.03	4.8	4.70	0.13	4.61	0.11	4.66	0.13
TSH	1.5	1.51	0.04	1.50	0.02	1.51	0.04	8.38	8.38	0.08	8.38	0.16	8.44	0.16
LH	9.91	9.20	0.35	9.34	0.30	9.20	0.35	50.5	47.22	1.24	48.59	1.97	48.12	1.54
PRL	11.5	10.61	0.49	10.92	0.32	10.61	0.49	42	40.05	0.92	39.55	1.36	38.95	1.39
Testesteron	5.62	5.80	0.14	5.66	0.19	5.80	0.14	2.36	2.30	0.06	2.34	0.07	2.45	0.05
Cortizol	14.32	14.92	0.64	14.63	0.80	14.92	0.64	31.3	31.26	0.90	32.07	1.14	32.68	1.14
FSH	16.3	15.51	0.57	15.17	0.54	15.51	0.57	51.8	47.09	1.17	48.57	2.08	48.97	1.84

Table II

TEa, bias and CV values of the two levels of quality control for the assays.

Assay name	TEa (%)	IQC 1						IQC 2
		June		July		August		June		July		August
		%CV	% Bias	%CV	% Bias	%CV	% Bias	%CV	% Bias	%CV	% Bias	%CV	% Bias
AFP	21.9	3.73	10.56	4.75	7.00	5.07	6.90	2.31	15.9	3.31	11.40	5.50	7.72
CA125	35	6.38	12.34	9.72	10.90	8.33	10.30	6.66	1.03	7.89	6.54	4.68	5.89
CA15-3	20.8	2.90	11.30	3.67	6.62	3.46	4.93	4.29	3.65	2.69	10.22	5.13	4.49
CA19-9	46.0	6.52	11.18	4.40	10.22	4.54	4.34	7.44	16.02	4.01	14.88	9.07	9.70
CEA	24.7	2.24	8.12	3.40	8.12	4.31	3.33	1.63	11.66	1.83	13.17	4.49	6.13
TPSA	34	2.35	7.50	2.81	8.50	4.58	5.00	2.34	1.43	2.01	3.22	5.03	1.30
Vitamin B12	30.0	6.27	2.12	1.13	0.63	3.78	3.30	5.08	5.30	2.35	0.07	1.76	4.78
Ferritin	16.9	1.79	2.43	3.22	3.01	4.51	1.68	1.38	1.01	3.74	3.45	4.41	3.29
Folat	39.0	4.12	0.26	5.18	0.77	6.27	1.54	2.91	6.15	6.15	1.20	5.70	5.81
PTH	30.2	6.09	7.93	6.54	9.73	4.49	1.43	1.73	11.00	5.27	12.31	3.35	2.33
FT3	11.3	2.27	2.40	2.40	2.27	2.46	2.53	1.20	0.16	1.78	0.27	2.56	2.04
FT4	8.0	1.35	1.61	2.38	1.61	2.44	0.81	2.77	2.08	2.28	3.96	2.79	2.92
TSH	23.7	1.73	0.00	1.47	0.00	2.50	0.33	1.00	0.00	1.87	0.06	1.91	0.66
LH	27.9	3.05	7.47	3.21	5.75	3.80	7.16	2.63	6.50	4.05	3.78	3.20	4.71
PRL	29.4	2.66	3.65	2.88	5.04	4.59	7.74	2.30	4.64	3.44	5.83	3.57	7.26
Testesteron	13.6	2.43	0.89	3.27	0.71	2.40	3.20	2.61	2.54	2.82	0.85	2.12	3.81
Cortizol	22.8	3.59	1.19	5.47	2.13	4.29	4.19	2.88	0.13	3.55	2.46	3.47	4.41
FSH	21.2	3.21	8.22	3.55	6.93	3.68	4.85	2.48	9.10	4.27	6.24	3.76	5.46

Table III

The sigma metrics for 3 months and overall sigma metrics for the assays.

Assay Lname	June		July		August		overall 3 month sigma metrics
	IQC 1 sigma metrics	IQC 2 sigma metrics	IQC 1 sigma metrics	IQC 2 sigma metrics	IQC 1 sigma metrics	IQC 2 sigma metrics	IQC 1	IQC 2
AFP	3.04	2.58	3.14	3.17	2.96	2.58	3.05	2.78
CA125	3.62	5.16	2.52	3.66	3.01	6.31	3.05	5.04
CA15-3	3.27	4.00	3.86	3.94	4.59	3.18	3.91	3.71
CA19-9	5.35	4.04	8.14	7.77	9.18	4.00	7.56	5.27
CEA	7.38	8.00	4.87	6.31	4.96	4.14	5.74	6.15
TPSA	11.10	13.75	8.92	15.15	6.25	6.42	8.76	11.77
Vitamin B12	4.45	4.86	26.05	12.71	7.07	14.37	12.52	10.65
Ferritin	8.08	11.49	4.31	3.60	3.37	3.09	5.25	6.06
Folat	9.40	11.27	7.38	6.15	5.98	5.82	7.59	7.75
PTH	3.66	11.11	3.13	3.39	6.41	8.32	4.40	7.61
FT3	3.92	9.32	3.77	6.20	3.56	3.62	3.75	6.38
FT4	4.73	2.14	2.69	1.77	2.95	1.82	3.46	1.91
TSH	13.67	23.64	16.16	12.61	9.35	12.07	13.06	16.11
LH	6.70	8.16	6.90	5.95	5.46	7.25	6.35	7.12
PRL	9.67	10.78	8.44	6.85	4.72	6.20	7.61	7.94
Testesteron	5.23	4.24	3.95	4.52	4.34	4.62	4.51	4.46
Cortizol	6.02	7.87	3.78	5.72	4.34	5.30	4.71	6.30
FSH	4.04	4.88	4.02	3.50	4.45	4.19	4.17	4.19

Table IV

The distribution of groups and tests according to sigma values.

Sigma metrics	June		July		August
	IQC 1	IQC 2	IQC 1	IQC 2	IQC 1	IQC 2
Grup 1 (<3)		AFP FT4	CA125 FT4	FT4	AFP FT4	AFP FT4
Grup 2 (3–6)	AFP	CA125	AFP	AFP	CA125	CA15-3
	CA125	CA15-3	CA15-3	CA125	CA15-3	CA19-9
	CA15-3	CA19-9	CEA	CA15-3	CEA	CEA
	CA19-9	Vitamin B12	Ferritin	Ferritin	Ferritin	Ferritin
	Vitamin B12	Testesteron	PTH	PTH	Folat	Folat
	PTH	FSH	FT3	Testesteron	FT3	FT3
	FT3		Testesteron	Cortizol	PRL	Testesteron
	FT4		Cortizol	FSH	Testesteron	Cortizol
	Testesteron		FSH		Cortizol	FSH
	FSH				FSH
Grup 3 (>6)	CEA	CEA	CA19-9	CA19-9	CA19-9	CA125
	TPSA	TPSA	TPSA	CEA	TPSA	TPSA
	Ferritin	Ferritin	Vitamin B12	TPSA	Vitamin B12	Vitamin B12
	Folat	Folat	Folat	Vitamin B12	PTH	PTH
	TSH	PTH	TSH	Folat	TSH	TSH
	LH	FT3	LH	FT3	LH	LH
	PRL	TSH	PRL	TSH		PRL
	Cortizol	LH		LH
		PRL		PRL
		Cortizol