Assessment of the performance of blood glucose monitoring systems for monitoring dysglycaemia in neonatal patients.

OBJECTIVE: To validate a three-step protocol that assesses the clinical risk associated with using blood glucose monitoring systems (BGMS) in neonates for the management of dysglycaemia.
METHOD: The three-step validation approach included confirmation of the accuracy of the reference method using National Institute of Standards and Technology (NIST) glucose standards, assessment of analytical risk performed on whole blood collected from paediatric patients routinely tested for glucose and a clinical risk assessment performed using heel stick capillary samples collected from 147 new-born babies and neonates admitted to intensive care. BGMS glucose measurements were compared with the NIST aligned laboratory reference method.
RESULTS: The accuracy of the laboratory reference method was confirmed with the NIST standards. Specificity studies demonstrated that the accuracy of one of the BGMS was affected, particularly, in the hypoglycaemic range, by known interference factors including haematocrit, ascorbic acid, lactose, galactose, N-acetylcysteine and glutathione. The accuracy of the other BGMS was unaffected. The clinical performance of this BGMS in neonates met the system accuracy criteria of Clinical and Laboratory Standards Institute (CLSI) POCT 12-A3 standard for evaluating hospital BGMS with 95.1% of glucose measurements within±0.67 mmol/L for samples ≤5.55 mmol/L and 95.6% within±12.5% for samples>5.55 mmol/L.
CONCLUSIONS: This three-step validation protocol provides a challenging approach for determining the accuracy and reliability of BGMS for managing dysglycaemia in neonates. StatStrip BGMS achieved analytical and clinical performance criteria confirming its suitability for use in neonates. We advocate that this validation approach should be considered for performance evaluations of both BGMS and continuous glucose monitoring systems going forward.

There is increasing debate about the accuracy and reliability of current bedside blood glucose monitoring systems (BGMSs).

Published case studies and real-time as well as simulation modelling studies have highlighted that the use of inaccurate BGMS can lead to mismanagement of dysglycaemia resulting in adverse clinical outcomes.

Evaluations to assess clinical performance and risk often neglect to confirm the accuracy of the reference method, questioning the validity of data analysis.

A three-step protocol to assess the clinical accuracy risk of a diagnostic method that includes: (1) isotope dilution mass spectrometry alignment of the reference method, (2) an analytical risk assessment focusing on specificity studies and (3) a clinical risk assessment performed on the target patient populaton.

The three-step validation protocol used provides a platform for more effective validation of the performance of BGMS in neonates.

First demonstration of BGMS meeting Clinical and Laboratory Standards Institute Point-of-Care Testing (POCT) system accuracy criteria in a neonatal population.

Introduction

Accurately identifying and managing dysglycaemia is important in high-risk neonates. Neonates with prolonged or recurrent episodes of hypoglycaemia can have a poor prognosis and are at risk of developing long-term neurocognitive impairments. This is most marked in symptomatic hypoglycaemia, but long-term neurological sequelae have also been associated with asymptomatic hypoglycaemia in small-for-gestational age babies, babies of diabetic mothers or preterm babies.1 2 Hyperglycaemia is common in preterm babies particularly extremely low birthweight babies during the first days of life. Unmanaged hyperglycaemia is associated with short-term negative influence on neurological and behavioural outcomes and is a risk factor for early mortality in very preterm infants.3 Close monitoring of blood glucose levels in high-risk babies is important in avoiding the consequences of hypoglycaemia and hyperglycaemia.4 Bedside handheld blood glucose monitoring systems (BGMCs) are widely used for measuring glucose in neonates.5 However, there is increasing debate about the accuracy and reliability of current bedside BGMS.4–7 Analytical accuracy evaluations, published case studies and real-time as well as simulation modelling studies have highlighted that the use of inaccurate BGMS can lead to mismanagement of dysglycaemia resulting in adverse clinical outcomes.8–14 Falsely elevated BGMS glucose values in adult and neonatal hospitalised patients particularly those in intensive care can be associated with pathophysiological factors or medication present in the patient’s whole blood.8 9 Abnormal haematocrit values, for example, have been shown to adversely affect the accuracy of BGMS glucose measurements in neonates with low haematocrit causing falsely high glucose values.15 The consequence of this can result in a missed neonatal hypoglycaemia, and it has been reported that currently used BGMS affected by haematocrit can miss up to 50% of hypoglycaemias in neonates.16 The failure to recognise hypoglycaemia could have clinical risk consequences for preterm neonates. The increased scrutiny of BGMS accuracy has led to the introduction of new guidelines and standards for assessing glucose metre performance. Prior to 2012, the only international standard available was the International Organization for Standardization (ISO) 15197 applied only to blood glucose systems used for self-monitoring.17 In 2013, this was updated to include tighter system accuracy criteria.18 In 2012, the first standard for use in hospital patients was published from the Clinical and Laboratory Standards Institute (CLSI POCT 12-A3) with system accuracy criteria tighter than included in ISO 15197:2013.19 To date, there are limited studies assessing the clinical risk accuracy of BGMS in neonates in comparison with the CLSI performance criteria. In China, there has also been little validation of the wide range of blood glucose metres used in hospitals. Many of the glucose metres in use are those that are intended for use only for self-monitoring in individuals with diabetes but have migrated into hospital setting. A recent international multisite regulatory compliant clinical risk assessment study advocates that an isotope dilution mass spectrometry (IDMS) aligned laboratory reference method should be used as the comparator method in BGMS performance evaluations.20 The aim of this study was to establish and validate a three-step protocol to evaluate the clinical risk associated with using BGMS in neonatal and paediatric patients for management of dysglycaemia, using an IDMS traceable reference method.

Material and methods

Study design

A three-step validation approach was used to assess BGMS clinical accuracy risk as according to Chinese Consensus Glucose Performance Guidelines for hospital BGMS21 and CLSI POCT 12-A3 guideline.19 The three-step approach comprised: (1) validation of the accuracy and IDMS traceability of the laboratory reference method with international reference, (2) an analytical risk assessment including method correlation, precision and specificity studies performed on two BGMS, and (3) a clinical accuracy risk assessment was performed with the BGMS demonstrating better specificity and the lowest analytical risk.

IDMS calibration alignment of the reference method

The validation of the IDMS alignment of the central laboratory reference method was performed with National Institute of Standards and Technology (NIST) standard reference materials 917c (NIST certified pure glucose, purity 99.7%±0.3%) and 965b (glucose in frozen serum with NIST-certified concentration values), both of which were assessed by isotope dilution gas chromatography mass spectrometry. The reference laboratory method used glucose oxidase run on a Hitachi 7180 laboratory analyser (Roche Diagnostics, Indianapolis, Indiana, USA).

Analytical risk assessment

For the analytical risk assessment, venous whole blood specimens were collected in lithium heparin blood collection tubes from paediatric patients routinely tested for glucose as part of our institution’s glycaemic control programme. Each whole blood specimen was tested directly after collection using the two BGMS systems: StatStrip Glucose (Nova Biomedical, Waltham, Massachusetts, USA), AccuChek Inform II (Roche Diagnostics). The two BGMSs were randomly used throughout the analytical study. Following BGMS testing, the samples were immediately centrifuged, and the plasma was tested on the hospital central laboratory reference analyser glucose method. For the specificity studies, the influence of interfering factors such as ascorbic acid, lactose, galactose, β-hydroxybutyrate, N-acetylcysteine, glutathione and haematocrit on the accuracy of glucose measurements was assessed. Adult donated blood specimens were used for preparation of specificity samples. The donated blood was left overnight for glycolysis to occur resulting in the removal of native glucose present in the blood sample. Blood samples were prepared with five different glucose concentrations (1.1–3.3, 6.9–9.7, 12.5–15.3, 18.1–20.8 and 25.4–29.2 mmol/L), and at each glucose level, two different concentrations of each interference were tested along with a zero concentration interference control. The concentrations of the interference factors tested reflected the levels that can be seen in patients at the upper end of the clinical and therapeutic ranges and were based on published guidelines.19 22 Five different blood glucose levels were also tested in combination with five different levels of haematocrit representing the abnormal ranges seen in neonatal intensive care unit (NICU) patients. The interference blood samples were tested using the two BGMS and by the NIST aligned central laboratory reference method.

Clinical risk assessment

Heel stick capillary samples were collected from hospitalised new-born babies including neonates admitted to intensive care. To ensure sufficient glucose levels within the hyperglycaemic or hypoglycaemic range, 20 neonatal specimens were spiked with a glucose solution or left to undergo glycolysis in order to ensure sufficient glucose levels within the hyperglycaemic or hypoglycaemic range. Each whole blood specimen was tested directly after collection using the StatStrip Glucose BGMS, and the haematocrit level was determined in each whole blood specimen using microhaematocrit centrifugation (StatSpin VT Centrifuge, Statspin, Norwood, Massachusetts, USA). Following this, each sample was immediately centrifuged, and the plasma was tested on the hospital central laboratory analyser glucose method.

Statistical analysis

All data analysis and calculations were performed using Analyse-It software V.4.0 (Analyse-IT Software, Leeds, UK) and Windows Excel. Passing-Bablok regression analysis was used for the determination of correlation coefficient, intercept and slope. For assessment of method correlation the mean, bias, SD and % bias difference between the BGMS and the laboratory reference glucose oxidase method were determined. Bias plot analysis was used to assess BGMS system accuracy in comparison with the criteria defined in Chinese Consensus Glucose Performance Guidelines21 and CLSI POCT 12-A3 guideline19 for hospital blood glucose monitoring. The % bias difference between the glucose value for the control interference samples (zero), and the glucose value for each interference dosed samples was calculated for each interfering factor and for each BGMS

Results

The glucose oxidase laboratory reference method showed good agreement and alignment with the NIST SRM917c and 965b tests confirming IDMS alignment and the accuracy of the reference method (online supplementary table S1) for use in the analytical and clinical risk assessment studies.

The analytical interference study showed that ascorbic acid, lactose, galactose, β-hydroxybutyrate, N-acetylcysteine and glutathione did not interfere with the accuracy of StatStrip BGMS glucose measurements. For the AccuChek Inform II BGMS, only β-hydroxybutyrate did not interfere with the accuracy of glucose measurements. The interference factors particularly influenced the accuracy of the low level glucose measurements for the AccuChek BGMS indicating a risk of misidentifying hypoglycaemia (figure 1). Varying haematocrit (HCT) levels did not interfere with the accuracy of StatStrip BGMS glucose measurements, but for the AccuChek Inform II BGMS, the medium and higher glucose levels glucose readings were inversely proportional to the HCT levels (figure 2). Both StatStrip and AccuChek BGMS showed a good initial correlation to the NIST aligned reference method (online supplementary figure 1). The overall coefficient of correlation (r) for StatStrip compared with the NIST aligned reference method was 0.998 with a slope of 0.943 and intercept of 0.041 and an overall mean % bias difference of −4.46%. The overall coefficient of correlation (r) for AccuChek Inform II compared with the NIST aligned reference method was 0.998 with a slope of 0.907 and intercept of 0.254 and an overall mean % bias difference of −3.44%. The agreement between the two BGMS and the NIST aligned reference method was also analysed using the Bland-Altman plot (online supplementary figure 2).

Figure 1

Influence of interference factors on the accuracy of BGMS glucose measurements: (A) ascorbic acid, (B) lactose, (C) β-hydroxybutyrate, (D) N-acetylcysteine, (E) glutathione and (F) galactose. (1) Each data point represents an average of five measurements of the same specimen. (2) In the presence of ascorbic acid or L-glutathione, the accuracy of the laboratory analyser glucose measurement is affected. (3) GOD is glucose oxidase laboratory plasma method. BGMS, blood glucose monitoring systems.

Figure 2

Influence of varying levels of haematocrit on the accuracy of BGMS glucose measurements: (A) StatStrip BGMS and (B) AccuChek BGMS. (1) The tudy was conducted on five glucose levels. The graphs show the typical results for 1 of the 5 glucose levels (around 8.33 mmol/L). (2) Each data point represents an average of five measurements of the same specimen. BGMS, blood glucose monitoring systems.

Although both BGMS showed a good correlation to the laboratory reference method with the residual specimens, we only selected the StatStrip BGMS for the clinical risk assessment validation step because the accuracy of AccuChek Inform 2 was affected by substances that may be present in the whole blood matrix of hospitalised neonatal and paediatric patients.

Heel stick capillary blood specimens were collected and tested from 147 neonatal patients including 86 males and 61 females. The mean age of the neonatal population was 9.6 days and the ages ranged from 1 to 28 days old. Demographic breakdown by gender and additional information on birth weight and clinical disorder categories can be seen in online supplementary table S2. Each heel stick capillary blood specimen was analysed using StatStrip BGMS and the NIST aligned reference laboratory glucose method. The StatStrip BGMS achieved concordance to both the Chinese consensus performance criteria and the CLSI POCT12-A3 guideline system accuracy criteria (table 1, figure 3). Haematocrit present in the neonatal samples, values ranged from 27% to 70% (mean 49%), also had no effect on the accuracy of glucose measurements (figure 4).

Table 1

System accuracy concordance of StatStrip BGMS compared with NIST aligned glucose oxidase method

Glucose level	Criteria	Number	% within
Criteria based on Chinese consensus guidelines for hospital blood glucose monitoring
<5.5 mmol/L	95% within ±0.83 mmol/L	99	97.0 (96/99)
≥5.5 mmol/L	95% within ±15%	48	95.8 (46/48)
Criteria based on CLSI POCT 12-A3 system accuracy for hospital blood glucose monitoring
≤5.55 mmol/L	95% within ±0.67 mmol/L	102	95.1 (97/102)
>5.55 mmol/L	95% within ±12.5%	45	95.6 (43/45/)
4.2 mmol/L	98% within ±0.83 mmol/L	147	98.0 (144/147)
	≤4.2 mmol/L and within ±20%
	>4.2 mmol/L

BGMS, blood glucose monitoring systems; CLSI, Clinical and Laboratory Standards Institute; NIST, National Institute of Standards and Technology.

Figure 3

Bias distribution depicting difference of neonatal glycaemia values between StatStrip BGMS and the reference method plotted against the reference method, with dash lines representing CLSI POCT 12-A3 system accuracy limits. BGMS, blood glucose monitoring systems; CLSI, Clinical and Laboratory Standards Institute.

Figure 4

Bias distribution depicting difference between StatStrip BGMS and the reference method plotted against the HCT level in neonatal patient samples (n=147). HCT values ranged from 27% to 70% (mean 49%). BGMS, blood glucose monitoring systems; HCT, haematocrit.

Discussion

Accurate glucose measurements are important for identifying and managing neonatal dysglycaemia. Handheld BGMSs are widely used for this purpose despite concern about their accuracy for identifying and managing neonatal hypoglycaemia.1 4

However, it is also recognised that continuous glucose monitoring systems, although useful for monitoring glycaemic variability and with improving overall performance, are still less accurate than BGMS for use in hospitalised patients.23 Blood gas analysers are regularly used for glucose measurements in patients in NICU, but a recent report indicates that these may not show good agreement with central laboratory results and may not meet the accuracy criteria of new hospital use performance guidelines.24 The advent of CLSI POCT 12-A3 guideline as the first guideline published providing performance criteria for use in hospital patients provides a protocol for evaluating clinical risk prior to selection of a BGMS. In this study, this protocol was adopted for the first time in China in a three-step validation approach to determine the safest BGMS option for use in a neonatal patient population. A key step in the process is to ensure the accuracy and traceability of the reference method glucose measurements, which was established in our study using international NIST standards. Confirmation of NIST IDMS alignment is rarely presented in published accuracy studies undertaken in neonatal patients. In our three-step protocol, the analytical risk assessment step raised concerns about the neonatal application of AccuChek BGMS because of the influence of some of the interference substances (which can be present in the whole blood matrix of hospitalised neonates) on the accuracy of glucose measurements. This has been previously reported for AccuChek Inform II in accuracy assessments performed on adult patient populations with the BGMS not achieving the CLSI performance criteria.25 26 Erroneous blood glucose readings have been reported due to galactose interference in neonates with galactosaemia-inherited disease due to transferase deficiency leading to mismanagement and misdiagnosis of galactosaemia.11 27 However, transient non-persistent galactosemia can also occur in neonates due to liver disease, delay in liver maturation and portasystemic shunting,28 29 and if unrecognised could lead to inaccurate glucose readings. Lactose is a key component of breast milk and non-glucose sugars including lactose, and galactose can be present in neonatal formula feeds. It is not known if at the concentrations present in blood this could impact on neonatal blood glucose readings, but it has been reported that the AccuChek metre is susceptible to interference by low concentrations of galactose that could potentially influence the ability to detect hypoglycaemia.30 The accuracy of StatStrip BGMS was unaffected by these interference factors, and a third validation step was undertaken to assess clinical accuracy risk. In previously published studies, StatStrip was reported to be acceptable for use in neonates, but in these studies, SMBG and not hospital glucose performance criteria was used to assess accuracy.31 32 In our study, StatStrip BGMS met these tighter system accuracy criteria of CLSI POCT 12-A3. This outcome is similar to that recently reported in a large multisite study performed on critically ill adult patients.20 In this larger study, a trend analysis of glucose measurements in the hypoglycaemic range was performed to confirm the accuracy in this range. A limitation of our study was the small number of glucose measurements in the hypoglycaemic range limiting the scope for a trend analysis. Our findings indicate that StatStrip BGMS is accurate across the glycaemic range found in neonates, and two recent publications have reported improved clinical outcomes in paediatric and neonatal patients following a switch to using StatStrip BGMS.33 34

This three-step validation approach is a useful tool for identifying clinical risk associated with using a BGMS in a neonatal setting. The laboratory glucose oxidase method used in our hospital is IDMS traceable and as such is validated for undertaking an evaluation of the accuracy of BGMS. StatStrip shows strong measurement accuracy in neonates either in normal inpatient ward or in NICU. Consideration should be given to adopting the three-step validation approach for performance evaluations of both BGMS. At present, we do not use continuous glucose monitoring systems (CGMS) in our hospital neonatal setting but recognise that CGMS is being evaluated for neonatal use and as such we believe that our three-step protocol should also be considered for determining clinical accuracy of CGMS.