Analysis of performance of clinical biochemistry laboratory using Sigma metrics and Quality Goal Index.

BACKGROUND: Unreliable and ingenuine results issued by clinical laboratories have serious consequences for the patients. Sigma metrics is a standardized tool for Quality assessment for test performance in a laboratory.
OBJECTIVE: To evaluate the performance of routine biochemistry laboratory at MMIMSR, Mullana in terms of Sigma metrics and Quality Goal Index.
MATERIAL AND METHODS: This cross sectional study evaluated performance of 14 routine chemistry parameters using retrospective Internal Quality Control data of two levels on Siemens Dimension Rxl from Feb to Jul 2019 for CV% and EQAS reports from CMC, Vellore for Bias%. Sigma metrics was calculated using total allowable error targets as per CLIA and Biological Variability database guidelines.
RESULTS: For level-2 IQC; TG, Chol, ALP showed excellent performance with σ ​> ​6 while σ ​< ​3 was observed for AST, Total Protein, Glucose, BUN and ALT using CLIA guidelines while in IQC Level-3 poor performers were only BUN and ALT with Ca, TG and Chol showing σ ​> ​6. Further by using Biological Variability data guidelines; 10 parameters of IQC Level-2 and 5 of IQC level-3 were poor performers with σ ​< ​3.
CONCLUSION: Sigma metrics is an excellent tool for performance analysis of tests performed in a clinical laboratory. Lack of precision in terms of CV% was seen for majority of the poor performers. Total allowable error targets using Biological Variability data revealed σ ​< ​3 for 10 parameters while using CLIA guidelines σ ​< ​3 was seen for only 5 parameters of IQC level-2.

Introduction

Clinical laboratories are the backbone of any health care system as all decisions undertaken on the patients by the physicians primarily depends upon the results provided by the laboratories. Laboratory results influence 70%–75% of medical diagnosis, as a result the quality of the laboratory service directly affects the quality of health care. Laboratory results need to be as accurate as possible, and at the same time the laboratory operations need to be reliable with timely reporting in order to be useful in a clinical setting. If inaccurate results are provided by the laboratories, plethora of serious consequences may result ranging from unnecessary treatment, complications, lack of proper treatment, delay in correct diagnosis and unnecessary diagnostic testing [1].

The laboratory is a complex system, involving many steps of activity and many people. The entire set of operations that occur in testing is called the Path of Workflow. The Path of Workflow begins with the patient and ends in reporting and subsequently interpretation of the results. In any clinical laboratory, mistakes are inevitable considering the ever increasing number of samples, the limited number of individuals who handle these samples, and numerous steps involved in the testing process. Any mistake during sample analysis in any part of the cycle (TTP) Total Testing process can lead to poor laboratory results. A reliable method for detection of errors within the framework of TTP is required to ensure quality results [2].

Quality is defined as conformance to the requirements of the end users. Quality is assessed in terms of accuracy (closeness to true value), precision, senstivity and specificity. In clinical laboratories, for Quality assessment in an objective and quantitative manner, sigma metrics have been used [3]. Sigma metric is a composite measure of the total allowable error (method specific), bias (EQAS) and imprecision (CV% of IQC). Analysis of sigma metrics acts as a standardized scale for comparing the quality of test performance. Higher Sigma metric values represents few analytical errors with fewer questionable tests being accepted; to the tune that six sigma assay indicates 99.99966% of results being error free; i.e. 3.4 defects per million oppurtunities. Further, based on the sigma metrics, appropriate QC rules are applied [[4], [5], [6]].

The present study intends to evaluate the performance of Siemens Dimensions Rxl at routine biochemistry laboratory at MMIMSR, Mullana in terms of Sigma metrics and quality Goal index for routine biochemical parameters.

Materials and methods

This retrospective study was conducted at routine biochemistry lab at MMIMSR, Mullana. The secondary Quality Control data was obtained from Internal Quality Control of Randox Clinical Chemistry (Level 2 and 3) and EQAS reports from CMC, Vellore for the period from Feb 2019 to Jul 2019. The parameters included in the study were Aspartate Transaminase (AST), Alkaline Phosphatase (ALP), Albumin, Total protein, Glucose, Blood Urea Nitrogen (BUN), Uric acid, Total Bilirubin, Alanine Transaminase (ALT), Triglycerides (TG), Calcium and High Density Lipoprotein-Cholestrol (HDL-Chol) and Total Cholesterol and were analysed on Siemens Dimensions Rxl.

For all these parameters, imprecision was estimated using CV% which is a measure of variability of an assay and indicator of random errors [7]. Bias however is an indicator of accuracy and systematic errors in analysis. Bias % was calculated for each parameter by using the Monthly EQAS report from CMC, Vellore.

Total allowable error (TEa) or the tolerance limit is the total allowable variation for the performance of an analyte. Sigma metrics were calculated using Total allowable goals as per i) Clinical Laboratory Improvement Amendments (CLIA) guidelines from US and ii) the biological variation database specifications. Sigma metrics was calculated as:Sigma ​= ​TEa - Bias/ CV

The minimum acceptance limit for sigma was considered to be 3 sigma level. Sigma was calculated using CV% for both levels of Internal Quality control Level 2 and 3 using TEa targets from both CLIA guidelines and the biological variation database specifications [8,9].

Quality Goal index (QGI) indicates the reason behind a lower value of sigma i.e either because of lower precision or lower accuracy or combination of both. QGI was calculated using the following formula and interpreted as per Table 1 [10].QGI ​= ​Bias/1.5 x CV%

Table 1

Criteria for interpretation of Quality Goal Index.

QGI	Problem
<0.8	imprecision
0.8–1.2	imprecision and inaccuracy
>1.2	inaccuracy

Results

The various parameters tested a long with their measurement units, methods used and target values with expected range is represented in Table 2. For IQC - Level 2, average CV% ranged from 2.94 (HDL) to 6.56 (ALT) while for IQC - Level 3 it ranged from 10.01 (T.Bil) to 2.45 (T protein). IQC level 2 analysis revealed CV% less than 5% for all analytes except for 4 parameters: AST, BUN, Creatinine and ALT where as for IQC level 3, CV% for all analytes was less than 5% for all analytes except for Total Bilirubin (Table 3). Poor precision for these analytes was due to temperature fluctuations affecting the performance of enzymatic reagents used for assay, due to other ran-dom errors in reconstitution, pipetting of reagent and sample. Elevated bias (poor accuracy) observed in Cholesteol, Calcium is due to significant difference between expected range of the Internal Quality control samples and reported value of EQAS.

Table 2

Represents the measurement units, methods used and target values with expected range of various test parameters of Randox Clinical Chemistry (Level 2 and 3) analysed on Siemens Dimensions Rxl.

	Method used	Unit	Mean (range)			Method used	Unit	Mean (range)
			L2	L3				L2	L3
AST	IFCC (UV with P5P)	U/L	53 (43–63)	185 (146–218)	CRE	Alkaline picrate	mg/dL	1.48 (1.2–1.77)	4.21 (3.37–5.05)
ALP	IFCC (ALP - AMP)	U/L	154 (131–177)	296 (252–340)	TBi	Jendrassik- Grof	mg/dL	1.61 (1.27–1.9)	4.4 (3.47–5.3)
ALB	BCP	g/dL	4.18 (3.55–4.8)	2.27 (2.36–3.2)	ALT	IFCC (UV with P5P)	U/L	41 (33–49)	142 (113–171)
TP	Biuret	g/dL	6.25 (5–7.5)	4.73 (3.8–5.7)	CAL	OCPC method	mg/dL	8.38 (7.54–9.2)	12.3 (11.1–13.5)
GLU	Hexokinase	mg/dL	120 (102–138)	279 (238–320)	TGL	Enzyme end point (GPO)	mg/dL	94.7 (79.5–110)	255 (214–296)
BUN	Urease/UV	mg/dL	42.7 (36.3–49.1)	116 (98.6–133)	HDL	Direct measure PEG	mg/dL	45.5 (39–52)	125 (107–143)
URI	Uricase/UV	mg/dL	5.6 (4.8–6.3)	8.99 (7.8–10.2)	CHO	CHOD/esterase/POD	mg/dL	145 (126–164)	267 (233–301)

Table 3

Represents the CV% for all analytes for IQC - Level 2 and Level-3 from Feb 2019 to July 2019 along with the mean value of CV%.

	Feb 19		Mar 19		Apr 19		May 19		Jun 19		Jul 19		Avg
	L2	L3	L2	L3	L2	L3	L2	L3	L2	L3	L2	L3	L2	L3
AST	5.08	1.7	6.44	3.67	4.58	2.32	7.35	2.7	6.28	2.55	8.45	5.51	6.36	3.07
ALP	4.65	4.16	7.67	5.28	3.45	7.69	2.49	2.55	3.64	4.26	3.93	3.82	4.3	4.62
ALB	1.62	1.9	3.63	4.77	7.95	7.89	2.3	3.66	7.27	3.54	3.28	4.43	4.34	4.36
TP	1.45	1.65	7.17	1.83	6.71	4.6	1.93	2.51	1.78	1.89	1.66	2.25	3.45	2.45
GLU	3.61	3.02	3.28	3.15	3.31	2.56	2.41	2.07	6.5	3.1	3.36	4.07	3.74	2.99
BUN	6.15	5.19	5.85	7.24	4.15	2.67	5.34	2.96	5.71	7.61	8.93	4.26	6.02	4.99
URIC	4.07	3.18	1.78	2.02	3.43	2.35	3.5	4.27	8.01	6.79	7.64	11.65	4.74	5.04
CRE	3.75	2.85	6.16	2.61	6.8	3.4	4.92	4.01	7.04	3.12	7.18	4.74	5.97	3.45
TBi	2.66	5.03	3.4	9.04	3.89	6.56	3.97	5.34	6.66	8.15	4.54	4.76	4.19	6.48
ALT	4.44	2.61	6.92	4.54	7.07	5.27	6.68	3.07	6.84	7.73	7.44	7.52	6.56	5.12
CAL	2.97	2.97	3.85	2.72	3.51	2.41	3.35	3.81	7.95	2.44	3.98	2.89	4.27	2.87
TGL	1.87	2.02	6.27	4.01	5.28	4.92	2.41	5.21	6.6	2.04	2.73	2.43	4.19	3.44
HDL	2.33	4.59	2.9	3.19	3.35	1.83	1.93	5.99	2.73	4.45	4.41	6.83	2.94	4.48
CHOL	1.99	1.83	3.76	2.87	3.31	1.95	3.5	5.26	3.91	1.31	3.22	2.4	3.28	2.6

Table 4 summarizes Bias % obtained from EQAS report by CMC Vellore for all analytes from Feb 2019 to July 2019 along with the mean value of Bias%.

Table 4

Bias % obtained from External Quality Assurance Scheme from CMC Vellore for various biochemical parameters from Feb to July 2019.

	Feb	March	April	May	June	July	Average
AST	6.6	13.4	6.2	−0.21	−12.38	0.21	2.3
ALP	−15.5	5.3	0.3	−2.82	−0.82	9.66	−0.64
ALB	−5.3	−14.3	−7.3	−3.76	−14	9.72	−5.82
TP	−3.3	−6.5	7.5	0.42	6.41	0.58	0.851
GLU	−7	−1	1.9	4.76	−7.68	2.46	−1.09
BUN	−1.1	−13.4	−0.1	1.15	9.25	−0.85	−0.84
URIC	−8.9	1.3	−4.3	0.06	−16.43	8.16	−3.35
CRE	0	−7.7	−3.7	−10.61	−10.69	15.38	−2.88
TBi	0	2.9	−5.3	4.93	−0.37	−2.99	−0.14
ALT	8.2	24.1	15.2	4.86	7.27	4.79	10.73
CAL	−2.1	−3.7	5.9	−16.57	−21.66	0.11	−6.24
TGL	−6.7	−0.8	−5.5	−23.56	−1.57	−7.73	−7.64
HDL	−1	17.8	6	13.6	−3.35	13.94	7.83
CHOL	−23.8	−27.8	−6.2	1.13	−18.78	−18.27	−15.62

Table 5 indicates the sigma metrics and QGI ratio for all analytes for IQC - Level 2 and Level-3 using Average of CV%, Bias % and TEa values from CLIA guidelines and biological variation database specifications.

Table 5

Sigma metrics and QGI ratio all analytes for IQC - Level 2 and Level-3 using TEa values from CLIA guidelines and biological variation database specifications.

	CV%		BIAS%		TEa CLIA	TEa BVD	Sigma CLIA		Sigma BVD		QGI		Problem
	L2	L3	L2	L3			L2	L3	L2	L3	L2	L3	L2	L3
AST	6.36	3.07	2.3	2.3	20	16.69	2.7	5.76	2.26	4.68	0.24	0.5	imp	imp
ALP	4.3	4.62	−0.64	−0.64	30	12.04	7.12	6.63	2.94	2.74	−0.09	−0.09	imp	imp
ALB	4.34	4.36	−5.82	−5.82	10	4.07	3.64	3.62	2.27	2.26	−0.89	−0.89	Imp,inac	Imp, inac
TP	3.45	2.45	0.85	0.85	10	3.63	2.65	3.73	0.80	1.13	0.16	0.23	imp	imp
GLU	3.74	2.99	−1.09	−1.09	10	6.96	2.96	3.71	2.15	2.69	−0.19	−0.24	imp	imp
BUN	6.02	4.99	−0.84	−0.84	9	15.55	1.63	1.97	2.7	3.28	−0.09	−0.11	imp	imp
URI	4.74	5.04	−3.35	−3.35	17	11.97	4.29	4.03	3.23	3.03	−0.47	−0.44	imp	imp
CRE	5.97	3.45	−2.88	−2.88	15	8.87	3.01	5.18	1.96	3.41	−0.32	−0.55	imp	imp
TBi	4.19	6.48	−0.14	−0.14	20	26.94	4.81	3.11	6.46	4.18	−0.02	−0.01	imp	imp
ALT	6.56	5.12	10.7	10.73	20	27.48	1.41	1.81	2.55	3.27	1.09	1.39	Imp,inac	inacc
CAL	4.27	2.87	−6.24	−6.24	11	2.55	4.03	6.01	2.05	3.06	−0.97	−1.44	Imp,inac	inacc
TGL	4.19	3.44	−7.64	−7.64	25	25.99	7.78	9.49	8.02	9.77	−1.21	−1.48	inac	inac
HDL	2.94	4.48	7.83	7.83	30	11.63	7.54	4.94	1.29	0.84	1.77	1.16	inac	imp inac
CHO	3.28	2.6	−15.6	−15.6	10	9.01	7.81	9.8	7.51	9.47	−3.17	−4.01	inac	Inac

Table 5 indicates the performance of various analytes in terms of sigma metrics using Total allowable error targets from CLIA and Biological variation database guidelines.

For IQC-Level 2, average CV% ranged from 2.94 (HDL) to 6.56 (ALT) while for IQC - Level 3 it ranged from 10.01 (T. Bil) to 2.45 (T protein). IQC level 2 analysis revealed CV% less than 5% for all analytes except for 4 parameters: AST, BUN, Creatinine and ALT whereas for IQC level 3, CV% for all analytes was less than 5% for all analytes except for Total Bilirubin. Poor precision for these analytes was due to temperature fluctuations affecting the performance of enzymatic reagents used for assay, due to other random errors in reconstitution, pipetting of reagent and sample. Elevated bias (poor accuracy) observed in Cholesterol, Calcium is due to significant difference between expected range of the Internal Quality control samples and reported value of EQAS.

The Sigma metrics analysis for IQC- Level 2 showed sigma value < 3 for AST, Total protein, Glucose, BUN and ALT using CLIA guidelines indicating failure to perform minimum sigma quality performance. In contrast, while using biological variation database specifications, sigma value lesser than 3 was observed for all parameters except for Total Bilirubin, Triglycerides and Cholesterol. For 4 parameters; ALP, TG, HDL and Cholesterol, sigma metrics was more than 6, which is a marker of world class quality.

Similarly, sigma metrics for IQC level-3 also revealed sigma values ​> ​6 for ALP, TG, Cholesterol and Calcium using CLIA guidelines. TG and Cholesterol evaluation by using biological variation database specifications also showed sigma value ​> ​6. Marginal performance i.e. sigma metrics between 3-6 was noted for 8 parameters namely AST, Albumin, Total Protein, Glucose, uric acid, creatinine, Total Bilirubin and HDL. Sigma value ​< ​3 was observed for only 2 analytes BUN and ALT using CLIA guidelines indicating failure to perform minimum sigma quality performance.

For parameters with sigma metrics <3, the reason behind their poor performance was evaluated and it showed that the main problem was imprecision for AST, Total protein, Glucose, BUN (for IQC Level 2) and for BUN (for IQC Level 3). The lower performance of ALT in IQC Level 2 was however attributable to both lack of precision and accuracy while for IQC Level 3, it was due to inaccuracy.

Further the performance analysis of various test parameters in terms of sigma metrics revealed significantly different σ values using same bias and CV% but different guidelines for Total Allowable errors from CLIA and Biological Variability data as depicted in Table 6.

Table 6

Performance analysis of various test parameters in terms of sigma metrics using Total allowable error targets from CLIA and Biological variation database guidelines.

	Sigma values L2		Sigma values L3			Sigma values L2		Sigma values L3
	CLIA	BVD	CLIA	BVD		CLIA	BVD	CLIA	BVD
AST	2.7	2.26	5.76	4.68	CRE	3.01	1.96	5.18	3.41
ALP	7.12	2.94	6.63	2.74	TBi	4.81	6.46	3.11	4.18
ALB	3.64	2.27	3.62	2.26	ALT	1.41	2.55	1.81	3.27
TP	2.65	0.80	3.73	1.13	CAL	4.03	2.05	6.01	3.06
GLU	2.96	2.15	3.71	2.69	TGL	7.78	8.02	9.49	9.77
BUN	1.63	2.7	1.97	3.28	HDL	7.54	1.29	4.94	0.84
URIC	4.29	3.23	4.03	3.03	CHOL	7.81	7.51	9.8	9.47

Discussion

The assay performance of any analyte can be evaluated in terms of sigma metrics with σ value ​≥ ​6 indicating world class performance, σ value ​≥ ​5 as excellent performance, σ value ​≥ ​4 as good, σ ​≥ ​3 as marginal, σ value ​≥ ​2 as poor and σ ​< ​2 as unacceptable performance.

In the present study we evaluated the performance of 15 routine chemistry parameters being carried out on Siemens Dimensions Rxl at clinical biochemistry laboratory at MMIMSR, Mullana in terms of sigma metrics. The previous studies undertaken by scientists across the country evaluated sigma metrics using TEa goals from CLIA [[11], [12], [13], [14], [15], [16]]. In the present study we used TEa goals from two sources CLIA and biological variability database as undertaken by Hens K et al. from Belgium and Xia J et al. from China [3,17]. Further for analytes showing poor performance in terms of σ ​< ​3, the cause for poor performance was evaluated using QGI index similar to study performed by Verma M et al. from Rohtak and Kumar BV et al. [18,19].

Sigma metric analysis using TEa specifications as per CLIA guidelines revealed that for 3 parameters (ALP, TG and Chol) in both levels of IQC; σ value was >6 similar to the observation by Vijatha Thomas et al. [11,14]. For HDL, σ value was >6 in IQC level 2 but in IQC level 3 it was 4.94. Similarly for Ca; σ value was >6 in IQC level 3 but in IQC level 2 it was 4.03. Albumin, Uric acid, Creatinine and Total Bil exhibited marginal performance (σ 3–6) in both levels of IQC indicating a scope for improvement [12,13,16]. Performance analysis of AST, Total protein, Glucose revealed σ ​< ​3 for IQC level 2 and between 3-6 for IQC level 3. For ALT and BUN σ ​< ​3 was obtained for both levels of IQC indicating poor performance [14]. The discrepancy observed in the evaluation of various analytes in terms of sigma metrics can be attributed to a combinational of multiple factors ranging from different methodology used, differences in IQC material, differences in the reported bias % by the different Proficiency test providers [14].

Comparison of the performance of two levels of IQC showed that parameters of level 3 performed better than level 2 as Sigma value ​< ​3 was observed for only 2 analytes BUN and ALT of level 3 using CLIA guidelines. However our lab decided to use the poor performer (Level 2 IQC) as a marker for further comparison in subsequent IQC analysis.

Root cause analysis in terms of Quality Goal Index for poor performers (AST, Total protein, Glucose, BUN) in IQC Level 2 and BUN in IQC Level 3 revealed imprecision as the cause for poor performance [19]. In contrast the reason attributable for poor performance of ALT in IQC Level 3 was found to be inaccuracy while in IQC level 2; it was due to combination of both inaccuracy and imprecision [18].

Further after considering the results of sigma metrics and QGI for poor performers (AST, Total protein, Glucose, BUN) in IQC Level 2 and BUN in IQC Level 3, we tried to improve the performance by adopting NABL/NABH guidelines for better quality achievement. We standardized our method for reconstitution, handling and storing the QC material. Temperature fluctuation was found to be a major culprit as we used enzymatic reagents. Further as a result strict temperature monitoring was undertaken for the lab and the refrigerator where the kits were stored.

In the present study we noted that significantly different σ values were obtained using same bias and CV% but different Total Allowable error targets from CLIA and Biological Variability data similar to the observations by Hens K et al. and Xia J et al. [3,17] Using biological variability data, σ metrics ranged from 0.8 to 8.02 for Level-2 and from 0.84 to 9.77 for Level-3. Performance analysis for 14 parameters revealed σ ​< ​3 (poor performance) for 5 parameters of level 2 IQC (AST, Total protein, Glucose, BUN and ALT), 2 analytes of level 3 IQC (BUN, ALT) using TEa values as per CLIA guidelines while using Biological Variability data guidelines; poor performance (σ ​< ​3) was seen for 10 parameters of level 2 IQC (AST, ALP, Alb, Total protein, Glucose, BUN, creatinine, ALT, Ca and HDL), 5 analytes of level 3 IQC (ALP, Alb, Total Protein, Glucose and HDL). Xia et al. in their study evaluated the sigma metrics for various analytes using 4 different TEa targets and concluded that although many targets for TEa exist, the optimal ones should be individualized by the laboratory such that the requirements are neither too high nor too low [17]. Our findings are similar to the observations of Hens K et al. wherein they concluded that TEa guidelines from Biological Variability data is highly demanding for analysing the performance of various test parameters resulting in false rejection of acceptable test reports [3]. Total Allowable error targets from CLIA seem to be more appropriate and realistic as per findings in our study as well.

Conclusion

Sigma metrics is an excellent self assessment tool for performance analysis of various test parameters in the laboratory. On applying the same to our routine biochemistry laboratory at Maharishi Markandeshwar Institute of Medical Sciences and Research, Mullana we observed that TG, Chol, ALP and Calcium showed world class/excellent performance. Albumin, Uric acid, Creatinine and Total Bil exhibited marginal performance (σ 3–6) in both levels of IQC indicating a scope for improvement. However, for ALT and BUN σ ​< ​3 was obtained for both levels of IQC indicating poor performance due to lack of precision for BUN and lower accuracy for ALT.

CRediT authorship contribution statement

Parul Goel: Conceptualization, Conception and design of study, Formal analysis, Analaysis and/or interpretation of data, Writing - original draft, Drafting the manuscript, Revising the manuscript critically for important intellectual content. Gagandeep Malik: Conceptualization, Conception and design of study, Funding acquisition, Acquisition of Data, Formal analysis, Analaysis and/or interpretation of data, Writing - original draft, Drafting the manuscript, Revising the manuscript critically for important intellectual content. Suvarna Prasad: Formal analysis, Analaysis and/or interpretation of data, Revising the manuscript critically for important intellectual content. Isha Rani: Funding acquisition, Acquisition of Data, Writing - original draft, Drafting the manuscript. Sunita Manhas: Formal analysis, Analaysis and/or interpretation of data.