Bedside Glucose Monitoring-Is it Safe? A New, Regulatory-Compliant Risk Assessment Evaluation Protocol in Critically Ill Patient Care Settings.

OBJECTIVES: New data have emerged from ambulatory and acute care settings about adverse patient events, including death, attributable to erroneous blood glucose meter measurements and leading to questions over their use in critically ill patients. The U.S. Food and Drug Administration published new, more stringent guidelines for glucose meter manufacturers to evaluate the performance of blood glucose meters in critically ill patient settings. The primary objective of this international, multicenter, multidisciplinary clinical study was to develop and apply a rigorous clinical accuracy assessment algorithm, using four distinct statistical tools, to evaluate the clinical accuracy of a blood glucose monitoring system in critically ill patients.
DESIGN: Observational study.
SETTING: Five international medical and surgical ICUs. PATIENTS: All patients admitted to critical care settings in the centers.
INTERVENTIONS: None.
MEASUREMENTS AND MAIN RESULTS: Glucose measurements were performed on 1,698 critically ill patients with 257 different clinical conditions and complex treatment regimens. The clinical accuracy assessment algorithm comprised four statistical tools to assess the performance of the study blood glucose monitoring system compared with laboratory reference methods traceable to a definitive standard. Based on POCT12-A3, the Clinical Laboratory Standards Institute standard for hospitals about hospital glucose meter procedures and performance, and Parkes error grid clinical accuracy performance criteria, no clinically significant differences were observed due to patient condition or therapy, with 96.1% and 99.3% glucose results meeting the respective criteria. Stratified sensitivity and specificity analysis (10 mg/dL glucose intervals, 50-150 mg/dL) demonstrated high sensitivity (mean = 95.2%, SD = ± 0.02) and specificity (mean = 95. 8%, SD = ± 0.03). Monte Carlo simulation modeling of the study blood glucose monitoring system showed low probability of category 2 and category 3 insulin dosing error, category 2 = 2.3% (41/1,815) and category 3 = 1.8% (32/1,815), respectively. Patient trend analysis demonstrated 99.1% (223/225) concordance in characterizing hypoglycemic patients.
CONCLUSIONS: The multicomponent, clinical accuracy assessment algorithm demonstrated that the blood glucose monitoring system was acceptable for use in critically ill patient settings when compared to the central laboratory reference method. This clinical accuracy assessment algorithm is an effective tool for comprehensively assessing the validity of whole blood glucose measurement in critically ill patient care settings.

Inpatient glycemic management (IPGM) has become widely accepted as a standard of care (1). Proper glucose measurement is key to safe and effective IPGM (2, 3). Bedside glucose monitoring with blood glucose meters is an essential component of IPGM, but has been shown to create confounding analytical and clinical factors (4, 5). This occurred when self-monitoring blood glucose meters (SMBG) designed for diabetic patient self-use migrated into the hospital. Subsequently, numerous studies demonstrated confounding factors affecting clinical outcomes in acute care settings. This multicenter observational study is the first to present an algorithm combining four statistical tools to evaluate the analytical and clinical accuracy of a blood glucose monitoring system (BGMS) in critical care patient settings.

In the 1990s through the first decade of the new millennium, glycemic management programs were developed, implemented, and studied to determine the clinical outcome of glycemic control through IV intensive insulin therapy (IIT) in critically ill patients (6–9). The initial outcomes of these glycemic management programs were profound; they significantly reduced postsurgical infections, blood transfusion, acute kidney injury, polyneuropathy, ICU length of stay, and in-hospital mortality (6–8). Unfortunately, follow-up studies reported increased risk for hypoglycemia with an associated enhanced mortality in critically ill patients who received IV IIT (10–12). Central to these adverse events was the unreliability and lack of standardization of glucose measurement.

Historically, the quality of glucose measurement for diabetic patients was assessed using measurement validation protocols established by regulatory and standards agencies in cooperation with manufacturers, in which the comparative device was the YSI 2300 Glucostat (Yellow Springs Instruments, Yellow Springs, OH) and not a commutable central laboratory reference analyzer. As outlined recently, these measurement validation protocols continue to be revised in response to clinical performance concerns in various hospital patient populations (13). In 2003, ISO 15197 required that SMBG analytical accuracy meet the expectation that 95% of glucose measurement fall within 20% of the reference method for values greater than 75 mg/dL and less than or equal to 15 mg/dL for glucose measurements less than 75 mg/dL (14). In 2013, the ISO 15197 analytical accuracy target was tightened (15). Subsequently for hospital use, these targets were commonly achieved in laboratory analytical investigations but were found to be unacceptable in clinical use (3, 16). In October 2016, the U.S. Food and Drug Administration (FDA) published new guidelines (16) defining a general approach for verification and validation of performance with separate and distinct criteria for SMBG and BGMS. As evidenced by these new guidelines, the criteria for assessing glucose measurement are still evolving and there is open dialogue on how to properly assess clinical accuracy. Clinical accuracy is a qualitative measure to relate clinical treatment decisions based on a glucose result with clinical outcomes (17). Methods and statistical tools including sensitivity and specificity analyses, error grid analyses, and Monte Carlo simulation models have been individually used to assess clinical accuracy for glucose meters in ambulatory and acute care settings (17–19). As previously reported, individually these tools have limitations across the glucose measurement range because they represent different aspects of assessing analytical performance, for example, bias only, bias + imprecision, estimated total analytical error, and probability of an erroneous glucose result contributing to insulin dosing error (17, 20). To date, no investigation has combined these statistical tools to assess glucose measurement clinical accuracy in critically ill patients.

The purpose of this international, multicenter, multidisciplinary observational study was to evaluate the combination of specific statistical tools to assess clinical accuracy of a glucose result in critically ill patients in relation to clinical treatment decisions.

This algorithm, using four distinct statistical tools, was used to determine the accuracy of a BGMS glucose result in critically ill patients compared with a laboratory reference method traceable to definitive method, as recommended by the American Diabetes Association, FDA, and Clinical and Laboratory Standards Institute (CLSI) (16, 21, 22).

MATERIALS AND METHODS

Study Design

The study used paired prospective testing and retrospective chart review (February 2013 to February 2014) of critically ill patients aged 2 months to 99 years admitted to ICUs at five international clinical sites (Netherlands, Belgium, and United States). The sites (A, B, C, D, and E) comprised medical, surgical, and burn intensive care patients (Table ), where institution-specific IV intensive insulin procedures for maintaining glycemic control were employed as the standard of care (Table 1). Patient medical conditions were classified according to the World Health Organization’s International Statistical Classification of Diseases and Related Health Problems, 10th Revision (23). Medications were classification based on the “United States Pharmacopeia” (24). The FDA preapproved the pre-Investigational Device Exemption study protocol, which was subsequently approved by each institution’s ethics committee as part of their standard of care.

TABLE 1.

Study Investigation Sites, Patient Populations, and Target Glycemic Control Ranges

Patient Testing

Peripheral and central arterial and venous whole blood specimens were collected in lithium heparin blood collection tubes from patients routinely tested for glucose as part of each institution’s glycemic control programs. Capillary whole blood specimens were not included at this time due to the FDA’s comments during protocol development referencing their adverse events database (25), and also based on the recommendations in international consensus guidelines (3) that the standard of care requires the use of arterial or venous specimens. Each whole blood specimen was tested directly after collection using the StatStrip Glucose (Nova Biomedical, Waltham, MA) BGMS and was then immediately centrifuged and the plasma tested on the hospital central laboratory method within 15 minutes. For sites A, B, C, and D, the central laboratory method was plasma glucose hexokinase performed on the Modular P800 platform (Roche Diagnostics, Indianapolis, IN). Sites A, B, C, and D were aligned to National Institute of Standards and Technology (NIST) standard reference materials 917c and 965b. At sites A and B, the plasma hexokinase glucose method NIST alignment was also confirmed directly to an internal gas chromatography isotope dilution mass spectrometry (IDMS) glucose method. For site E, laboratory glucose measurement was performed using a glucose oxidase NIST aligned (917c and 965b) method on the UniCel Synchron DxC (Beckman Coulter, Brea, CA). Additional information about calibration and commutability are provided in Supplemental Table 1 (Supplemental Digital Content 1, http://links.lww.com/CCM/C303).

Clinical Accuracy and Risk Modeling Tools

Four clinical accuracy and risk assessment modeling tools were used to evaluate the risk of mismanagement of dysglycemia associated with BGMS measurements.

Clinical and Laboratory Standards Institute POCT12-A3 Guideline Analysis. Clinical accuracy was assessed using BGMS performance criteria as defined in the CLSI’s 2013 “POCT12-A3: Point-of-care blood glucose testing in acute and chronic care facilities; approved guideline—third edition” (22).

Parkes Error Grid Analysis. The Parkes error grid analysis incorporating recent recommendations for clinical accuracy studies (18) was performed to assess clinical risk associated with glucose measurement differences between the BGMS and central laboratory methods.

Monte Carlo Simulation Modeling. Application of Monte Carlo Simulation of Clinical Risk to the BGMS Trial Data. Simulation of the influence of bias and precision of the BGMS results on the risk of insulin dosing error in critically ill patients was reported in 2013 (19). The region of risk on the contour plot associated with this study was depicted graphically by overlaying a scatter plot of patient data (19). Individual data points were plotted at the average coefficient of variation for the BGMS (3.25%), and the bias values were determined by the difference in BGMS and reference glucose values. The region of clinical risk for operation of the BGMS in critically ill patients is shown by the cluster of data points, and the associated clinical risk is shown by the contour lines (Fig. ). The straight solid and dashed lines of total analytical error from the original publication were also included as an interpretative guide for the cluster of patient data. Quantitative results were shown in the attached tables of the fraction of all patient data (n = 1,815) bound by specific contour lines. As previously reported, the insulin dosing error rates were categorized as one dosage unit of insulin error, two dosage units of insulin error, and three dosage units of insulin error (19). Further details on the method used to overlay published Monte Carlo simulation contour plots of clinical risk in critical care adult patients to the BGMS trial data in critical care adult patients are presented as supplemental data (Supplemental Digital Content 1, http://links.lww.com/CCM/C303).

Stratified Clinical Sensitivity and Specificity Analysis. Stratified clinical sensitivity and specificity analysis was conducted in order to determine that the BGMS measurement was sufficient for intervention and therapeutic purposes at the medical decision limits of glucose values. The laboratory reference glucose and BGMS glucose results were stratified into glucose categories for insulin dosing and the frequency distributions determined. In each laboratory reference glucose category, sensitivity was assessed by determining the fraction of corresponding BGMS measurements within ± 1 reference method category. The false negative percentage is 100% × (1 – sensitivity). In each BGMS category, the specificity was assessed by the percentage of laboratory reference glucose measurements within ± 1 reference method category. The false positive percentage is 100% × (1 – specificity). Further details on the method used to calculate the sensitivity and specificity in each stratum are presented as supplemental data (Supplemental Digital Content 1, http://links.lww.com/CCM/C303).

Statistical Analysis

Data were analyzed using Analyze-it (version 3.50) for Excel 2007 (Analyse-it Software, Leeds, United Kingdom) and STATA/MP (version 11; StataCorp LP, College Station, TX) and included least squares linear regression. The BGMS mean absolute bias and percent (%) bias were calculated. The percent bias of the BGMS result compared with the hexokinase reference method result was calculated and assessed according to POCT12-A3 (22). Passing and Bablok regression analysis was used for hypoglycemic patient trend analysis.

RESULTS

Complexity of the Patient Population

The retrospective analysis of 1,815 paired glucose measurements from critically ill patients included n = 1,698 patients (paired glucose measurements from 1,692 individual patients and 123 glucose measurements from six burn patients), which included 19 different and complex medical condition categories representing 257 different and specific clinical conditions (23). On average, each patient received 14 medications from 33 different parent drug classes with 144 drug subcategories and 8,016 compounds administered in complex treatment regimens (24). The patient population investigated ranged in age from 2 months to 99 years and represented a wide spectrum of severity of illness (Table ), receiving multiple therapeutic and polypharmacy medications. The patient population had abnormal ranges of confounding physiological and biochemical substances known to affect the accuracy of the BGMS measurement (4). Detailed information about the breakdown of patient age and glucose ranges, medication classes, and range of confounding physiological and biochemical parameters are provided in Supplemental Tables 2–4 (Supplemental Digital Content 1, http://links.lww.com/CCM/C303).

BGMS Analytical Performance Analysis

Analytical performance was evaluated through comparison of paired patient specimen analyses for the BGMS versus the certified gas chromatography IDMS aligned plasma hexokinase method used in sites A and B (n = 1,245), where the coefficient of correlation for the BGMS versus comparative method was 0.995 with a slope of 1.05 and an intercept of –3.9 mg/dL. The mean percent bias difference between the BGMS and comparative method was –1.35%. These data demonstrated that the BGMS is analytically equivalent to the gas chromatography IDMS aligned reference hexokinase.

Risk Modeling

The data from all five sites demonstrated that 99.3% (1,802 results) of the BGMS measurements were within zone A of the Parkes error grid (Fig. ). The remaining 0.7% fell into zone B and breakdown analysis showed that these were not clinically significant and would not result in any untoward clinical intervention. Analysis of the collated data from all study sites showed that the performance of the BGMS met the clinical accuracy performance criteria outlined in POCT12-A3 with 95.4% (606/635) of patient sample results within ± 12 mg/dL for glucose values less than 100 mg/dL and 96.5% (1,139/1,180) of patient sample results within ± 12.5% for glucose values greater than 100 mg/dL.

Parkes error grid analysis: Patient data (n = 1,815) by blood glucose monitoring system (BGMS) related to the reference glucose method. The data are displayed over a solid line of equivalence and the risk levels: (A) < 20% deviation from reference glucose or both BGMS and reference glucose < 70 mg/dL, (B) deviation from reference glucose > 20% but leads to no treatment or benign treatment, (C) overcorrection of acceptable blood glucose monitoring levels, (D) dangerous failure to detect and treat blood glucose monitoring errors, and (E) erroneous treatment (17).

Stratified Clinical Sensitivity and Specificity Analysis. Stratified clinical sensitivity and specificity analysis showed that BGMS measurements were highly sensitive (mean = 95.2%, sd = ± 0.02) and highly specific (mean = 95.8%, sd = ± 0.03) over the glycemic range tested (10 mg/dL intervals between 50 and 150 mg/dL) (Fig. ).

Stratified clinical sensitivity and specificity analysis: Frequency distribution of glucose measurements (n = 1,815) by the reference method (gray) and blood glucose monitoring system (BGMS) (white) in each strata of glucose for a sliding insulin dosing scale (A). B, Depicts the percentage of false negative results (i.e., 1 – sensitivity) in each stratum where the BGMS results do not match the corresponding reference method glucose ± 1 BGMS category. B, Also depicts the percentage of false positive results (i.e., 1 – specificity) in each stratum where the BGMS results do not match the corresponding reference method glucose ± 1 reference method category.

Clinical glucose data were overlaid on plots to determine the anticipated probability of insulin dosing error for greater than or equal to 1, greater than or equal to 2, and greater than or equal to 3 insulin dosing error categories. Most BGMS data falls within the boundary of 15% total error and within 0.05% probability of three or more categories of insulin dosing error (Fig. ). Tabulated analysis was performed with all BGMS data, and the analysis for the simulated sliding insulin scale showed that 1.8% (32/1,815) of critically ill patients had greater than 0.5% chance of three or more insulin dosing errors during treatment and 2.3% (41/1,815) had greater than 20% chance of two or more insulin dosing errors during treatment (Fig. 3).

Figure 3.

Monte Carlo simulation modeling: Contour plot of insulin dosing error rates of three or more insulin dosing error categories created using Monte Carlo simulation. The color contours represent the probability that the blood glucose monitoring system (BGMS) glucose method will generate a result associated with three or more insulin dosing error categories (0.05–90%). Total analytic error zones of 10%, 15%, and 20% are presented as black solid, dashed, and dotted-dashed lines, respectively (19).A random sample of BGMS results (n = 200) are plotted in (A) for illustration at 3.25% coefficient of variation with bias of individual observations determined by the difference in the BGMS and reference glucose method. Summary results for ≥ 2 and ≥ 3 insulin category dosing errors with all patient results (n = 1,815) are in tabular form as a cross reference to the contour plot in (B). CV = coefficient of variance.

A Passing and Bablok bias trend analysis was performed on the clinical dataset from three of the study sites to identify any potential safety issues with the use of the BGMS in the critically ill patient population subset with hypoglycemia. An analysis was not performed on the clinical data from sites C and E due to the small number of hypoglycemic patients: one at site E and none at site C. The BGMS demonstrated 99.1% (223/225) concordance to the central laboratory reference methods in characterizing hypoglycemic patients with glucose less than 70 mg/dL (< 3.9 mmol/L).

DISCUSSION

The variability observed in glycemic control studies has been associated with nonstandardized glucose methods that are not validated in critically ill patients (2, 3). These limitations have resulted in inconsistent outcomes and diminished the utility of BGMS use in high-risk populations (26, 27). In response to the published concerns about the suitability of BGMS use in IV IIT, the study team elected to develop and apply a combination of specific statistical tools to thoroughly evaluate the clinical accuracy and the estimated total analytical error of the study BGMS in order to evaluate the device’s suitability for use in critically ill patient care settings. There were no exclusion criteria for patients participating in the study, where the patient population was not limited to diabetic patients, but included all the patients admitted in to the critical care units in each study center. As such, the study included patients with a significant array of medical conditions with abnormal pathophysiologic factors and a vast range of medications known to interfere with the accuracy of many routinely used glucose meters and other glucose measurement methods (4, 5).

There were no clinically significant interferences observed, and the BGMS demonstrated substantial equivalence to gas chromatography IDMS traceable plasma hexokinase aligned laboratory reference methods. The use of the different clinical accuracy assessments algorithm demonstrated that the BGMS is accurate and reliable for use in critically ill patients. The significant volume of data generated by this study permitted thorough investigation of device performance across the analytical measuring range and, particularly, the hyper- and hypoglycemic ranges.

In patients identified as hypoglycemic (< 70 mg/dL) using the reference method (225 patients), two (2/225, 0.9%) were identified as normoglycemic using the BGMS. Further review of these two patients did not identify any medical condition, drug combination, or clinical trend, and therefore, the result cannot be explained based upon available information.

The BGMS demonstrated substantial equivalence to the central laboratory reference methods in characterizing hypoglycemic patients with glucose less than 70 mg/dL (< 3.9 mmol/L). Previously published accuracy guidelines and studies lacked sufficient data to properly evaluate clinical accuracy (16). Use of simulation modeling was a beneficial contribution to assess the probability of an inaccurate glucose measurement resulting in an insulin dosing error. The introduction of the stratified clinical sensitivity and specificity analysis as a new tool provided further insight into the quantifiable probability of error (i.e., uncertainty) at the critical decision limits for managing insulin administration. Collectively, the four mathematical and statistical modeling tools enabled the study team to complete an effective and comprehensive clinical accuracy assessment. It is important to note that the study protocol and data analyses have not been applied to other whole blood, point-of-care devices and glucose methods, but, more importantly, the study design and this clinical accuracy algorithm can serve as a model for future studies of in vitro diagnostic methods in critically ill patient care settings, including point-of-care devices and other methods. In the United States, regulation of BGMS has undergone considerable debate following the FDA’s publication of guidelines for manufacturers that separates SMBG from prescription and professional use BGMS, which manufacturers are now required to test in critically ill patient care settings (16, 28, 29). Subsequently, the Centers for Medicare and Medicaid Services announced that BGMS not cleared for use with critically ill patients would be considered “off-label” when used in these patient populations (30, 31), requiring users to comply with the Clinical Laboratory Improvement Amendments of 1988 (32).

Although this study addresses venous and arterial whole blood, an additional study is required using the same clinical accuracy algorithm to determine if capillary whole blood specimens can safely be used in critically ill patient care settings with the study BGMS and other whole blood methods.

The question remains: are all whole blood glucose methods clinically accurate and acceptable for use in critically ill patient care settings? The algorithm used in this study is a rigorous approach combining several statistical tools to effectively validate the clinical accuracy of using bedside glucose monitoring systems in critically ill patients. This algorithm and clinical accuracy assessment approach can be applied to other routinely used whole blood glucose measurement methods, including emerging continuous glucose monitoring systems.

In summary, this clinical accuracy assessment algorithm is an effective tool for comprehensively assessing the validity of using whole blood glucose measurement in critically ill patient care settings.

This study approach and subsequent analyses may serve as an example for future evaluations of other whole blood bedside test systems in critically ill patients.

TABLE 2.

Patient Information by Site Including 19 Complex Medical Conditions