Development of the Diabetes Technology Society Blood Glucose Monitor System Surveillance Protocol.

BACKGROUND: Inaccurate blood glucsoe monitoring systems (BGMSs) can lead to adverse health effects. The Diabetes Technology Society (DTS) Surveillance Program for cleared BGMSs is intended to protect people with diabetes from inaccurate, unreliable BGMS products that are currently on the market in the United States. The Surveillance Program will provide an independent assessment of the analytical performance of cleared BGMSs.
METHODS: The DTS BGMS Surveillance Program Steering Committee included experts in glucose monitoring, surveillance testing, and regulatory science. Over one year, the committee engaged in meetings and teleconferences aiming to describe how to conduct BGMS surveillance studies in a scientifically sound manner that is in compliance with good clinical practice and all relevant regulations.
RESULTS: A clinical surveillance protocol was created that contains performance targets and analytical accuracy-testing studies with marketed BGMS products conducted by qualified clinical and laboratory sites. This protocol entitled "Protocol for the Diabetes Technology Society Blood Glucose Monitor System Surveillance Program" is attached as supplementary material.
CONCLUSION: This program is needed because currently once a BGMS product has been cleared for use by the FDA, no systematic postmarket Surveillance Program exists that can monitor analytical performance and detect potential problems. This protocol will allow identification of inaccurate and unreliable BGMSs currently available on the US market. The DTS Surveillance Program will provide BGMS manufacturers a benchmark to understand the postmarket analytical performance of their products. Furthermore, patients, health care professionals, payers, and regulatory agencies will be able to use the results of the study to make informed decisions to, respectively, select, prescribe, finance, and regulate BGMSs on the market.

The Need for the DTS BGMS Surveillance Program

The use of blood glucose monitoring systems (BGMSs) plays a vital role in the management of type 1, type 2, and gestational diabetes. People with diabetes who perform self-monitoring of blood glucose (SMBG) depend on the accuracy of their BGMS to ensure proper diabetes therapy including insulin and other medication dosing, exercise, meal planning and other necessary activities.[1]

In the USA, BGMS products are class II medical devices that require clearance from the Food and Drug Administration (FDA) to be marketed. Acceptable analytical accuracy data are critical for a meter to be cleared for the market.

Several recent standards and guidelines define accuracy requirements of BGMSs. ISO 15197:2013[2] requires that at least 95% of meter test results fall within ±15mg/dl (for BG < 100mg/dl) and within ±15% (for BG ≥ 100 mg/dl) of an accepted reference method procedure (ie, comparative method).

On January 7, 2014, the FDA released a draft guidance document for BGMSs intended for over-the-counter use that contains additional criteria.[3]

One shortcoming of the current process for clearing a BGMS for market is that, although many companies perform their own postmarket surveillance throughout the lifespan of the meter systems and test strips, no additional testing and/or reporting is required to be submitted to regulatory authorities once the system is on the market. Exceptions would include FDA audits and follow up of any medical device reporting incident (MDR) that may occur and/or warning letters from the FDA. Concerns about poorly performing BGMSs on the market have been expressed by many stakeholders including: regulatory agencies, health care providers, patients and payers. Many manufacturers of BGMSs have worked diligently to improve the performance of these systems. However, poorly performing BGMSs are currently available on the market.[4] Recent efforts by the FDA to improve accuracy standards for clearance will have little positive impact if the performance of the cleared BGMS is not maintained postmarket.

Current Performance of Cleared BGMS Products

Klonoff and Prahalad reviewed the literature from 2010-2014 of BGMSs that were tested according to ISO 15197:2003 and ISO 15197:2013, which are the standards used by FDA to clear most BGMSs currently on the market. They found that 75% met ISO:15197:2003 and 48% met ISO 15197:2013.[4] Such inaccuracy noted in the literature can be due to: manufacturing errors; scale up; changes in components between strip lots; other production issues; or improper shipping or storage. Eventual performance of a system might no longer correspond to the sponsor’s initial accuracy data submitted to the FDA.

The performance standards stated in the ISO and FDA standards ensure thorough evaluation of blood glucose values. Consequently, noncompliance with these standards could cause incorrect evaluation of blood glucose levels. Unreliably measured blood glucose levels can lead to incorrect therapeutic decisions and adverse health outcomes. This Surveillance Program is intended to protect people with diabetes from such poorly performing, inaccurate BGMSs that are currently on the market in the United States.[4] The program will provide an independent assessment of the performance of BGMSs following clearance by the FDA. The Surveillance Program will (1) establish a benchmark performance level for manufacturers to understand the postmarket accuracy of their products; (2) generate information that can assist diabetes patients, health care providers, and payers in making educated selection of BGMS products; and (3) provide data that can be used by regulatory agencies to support postmarket surveillance activities.

Such a program is needed because, once a BGMS product has been cleared for use by the FDA, there is currently no systematic postmarket Surveillance Program that can ensure ongoing product quality. One limitation of the research studies in the literature is that they are limited in time and scope. Some of them have different designs making comparisons among studies as well as assessments over time very difficult. A Surveillance Program (rather than many independent research studies) is needed to assess analytical performance on an ongoing basis, using a generally accepted protocol that allows comparison of performance over time and among BGMSs.

The Creation of the Program

The mission of Diabetes Technology Society (DTS) is to promote development and use of technology to help people with diabetes. Because BGMS products are one of the most important technologies for diabetes management, DTS has addressed the issues outlined above by establishing the DTS BGMS Surveillance Program for postmarket surveillance of BGMSs. The protocol for testing the marketed BGMS is the subject of this article.

On May 21, 2013, DTS presented a public meeting in Arlington, Virginia, to determine whether BGMSs on the market still met regulatory standards.[5] The meeting consisted of 4 sessions in which FDA, diabetes experts and leading academic clinicians and clinical chemists participated. This meeting was devoted to determining whether BGMSs on the market maintain their performance after they have been cleared by FDA. This was a concern because several articles in the literature at that time[5] presented data showing that, in fact, in some cases BGMSs did not meet the standards under which they were cleared. Meeting participants stated that if products on the market do not meet standards for which they were cleared, then people with diabetes will receive incorrect information and possibly make incorrect decisions putting them at increased risk for hypoglycemia and hyperglycemia. The FDA participated in the meeting and expressed interest in reaching a better understanding of the problem.

DTS sponsored a subsequent public meeting on this issue in Bethesda, Maryland, on September 9, 2013, Verifying the Performance of Blood Glucose Monitors Following FDA Clearance, that was attended by academia, patients, industry, nonprofit diabetes organizations, the FDA, and other government agencies. The event generated much interest and agreement that verification of the accurate performance of cleared BGMSs is needed. Plans for a Surveillance Program were introduced by DTS at that meeting.

On May 20, 2014, following much preparatory work and many negotiations, DTS announced the launch of the DTS BGMS Surveillance Program for Cleared Blood Glucose Monitoring Systems. The data obtained with this program will help manufactures with identifying problems in their quality assurance (QA) systems and will provide other stakeholders with scientifically sound data to initiate actions to address problems and concerns. The lack of sound scientific data seems to be one major challenge preventing key stakeholders from acting on the accuracy problems. The data obtained with this new Surveillance Program can support current efforts and facilitate actions and activities aimed toward improving BGMS and patient safety. The FDA has stated that they will be interested in the information received from the DTS-BGM Surveillance Programs’ Protocol. No other program in the USA with an expert-developed consensus protocol currently exits to independently assess the performance of multiple BGMSs.

In summary, this program is intended to protect people with diabetes from inaccurate BGMSs that might be currently on the market. This program will provide independent assessments of the performance of BGMSs that have received FDA clearance and provide information that can assist diabetes patients, health care providers, and payers in making informed product selection and can provide important information for regulators on the postmarket performance of regulated products.

Methods

Steering Committee

A first step in the process was the formation of a Steering Committee consisting of experts in blood glucose monitoring from academia, medical practice, clinical chemistry, and industry, as well as US government agencies and medical organizations. The Steering Committee that developed the protocol described in this manuscript included members from

Centers for Disease Control and Prevention (CDC)

Food and Drug Administration (FDA)

National Institutes of Health (NIH)

US Army

American Association of Clinical Chemists (AACC)

American Association of Diabetes Educators (AADE)

American Diabetes Association (ADA)

College of American Pathologists (CAP)

Diabetes Technology Society (DTS)

Juvenile Diabetes Research Foundation (JDRF)

The Endocrine Society (TES)

Albert Einstein College of Medicine

Isala, Zwolle, the Netherlands

Mills-Peninsula Health Services

New York-Presbyterian/Weill Cornell

University of Missouri School of Medicine

University of Virginia, Center for Diabetes Technology

Vanderbilt University School of Medicine

Joan Lee Parkes Consulting, Inc

Sampson Consulting, Inc

Abbott Diabetes Care

Industry

Once this Steering Committee was formed, the committee assembled in-person for all-day meetings on 3 different occasions in the Washington, DC, area to create a protocol for testing the marketed BGMS with an optimal study design. In addition to the in-person meetings of the entire Steering Committee, three separate Steering Committee Subcommittees consisting of experts in various fields, were formed that met by way of teleconferences to achieve consensus on specific topics regarding the best practices for BGMS testing.

The committee is confident that the BGMS testing protocol created is unbiased, transparent and fair to persons with diabetes as well as health care professionals, regulatory agencies, payers and manufacturers of BGMS products.

Decisions

Some of the specific areas where the committees worked through various approaches to reach decisions about best practices for a surveillance screening program for postmarket surveillance were:

Number of subjects, number of samples per subject

Inclusion and exclusion criteria

Testing to be conducted by subject or study staff

Required glucose range to be tested

Number of test strips to be tested per BGMS

Numbers of test strip lots to be tested per BGMS

Number of blood glucose meters to be used in a study

Methods for procuring BGMS products to best represent real-world experience

Hematocrit requirements

Environmental conditions and requirements

Blood sampling method either directly from the fingertip or from a capillary tube

Qualification of glucose reference instruments (ie, comparative glucose analyzers)

Choice of analytic accuracy target levels for passing

Incorporation of confidence intervals into the passing and failing scores

Incorporation of clinical accuracy metrics (ie, Error Grid analysis) into BGMS performance evaluation

Requirements for the DTS seal of approval

Testing sites for BGMSs and comparative glucose analyzer to be at the same or different locations to reduce potential investigator bias

Qualifications for a clinical trial site and a reference laboratory site

An accurate measurement method of SMBG performance in the hypoglycemic range in glycolyzed blood samples without introducing experimental artifacts

Criteria to select and eliminate data points from analysis

Statistical methods to analyze data

Vehicles to communicate findings to professionals and lay public

Retesting criteria if a BGMS fails

All of the above issues and options, along with many other considerations, were carefully worked through to achieve a consensus protocol acceptable to all on the Steering Committee. The committee was confident that the final protocol can be carried out within reasonable cost and time constraints, especially compared to the greater cost associated with clearance of a BGMS and they hoped such a study could be carried out soon.

Specific Controversial Topics

Four specific topics that generated the most discussion to achieve consensus included (1) choice and qualification of glucose comparative instruments, (2) a method of blood sampling, (3) whether to test BGMS and comparative glucose analyzer results at the same or different locations, and (4) modification of blood samples. These topics are discussed in this manuscript.

A separate section describes the statistical analysis including pass-fail criteria for analytical accuracy and clinical accuracy. The protocol itself (Protocol for the Diabetes Technology Society Blood Glucose Monitor System Surveillance Program) is attached as supplementary material.

Results

Study Objectives

The primary objective is to determine whether BGMSs marketed in the US meet predetermined analytical accuracy performance criteria determined by consensus of the DTS BGMS Surveillance Program Steering Committee. The program is not designed to evaluate whether devices are FDA compliant.

Other objectives of the protocol are to

Determine whether BGMSs marketed in the US meet predetermined clinical performance criteria (determined by error grid analysis);

Obtain data on BGMS device failures or device related problems that may occur during the study;

Provide study results to all interested parties through creation of the DTS BGMS Surveillance Website.

The protocol covers a series of in vitro diagnostic, Phase 4, BGMS clinical studies to determine accuracy of the systems.

All activities will be conducted according to good clinical practice including institutional review board approval of the protocol and supporting materials, and subject informed consent.

At least 110 and not more than 125 subjects will be enrolled in the study to ensure that at least 100 evaluable natural BGMS test results are obtained.

Limited demographic and medical history information will be collected from subjects.

To facilitate obtaining a wide range of blood glucose values, a subject may be tested in a fasting or postprandial state.

Trained study staff will perform both shallow and deep finger punctures on subjects using lancing devices to collect capillary blood for, respectively, testing on the BGMSs and testing on a comparative glucose instrument.

Subjects will not do any self-testing. Understanding by the end user of instructions for use and human factors analysis are not within the scope of this protocol.

A subject’s capillary blood will be directly applied to the test strip of the BGMS as per labeling of the device.

All values obtained by the BGMS assayed at the clinical site will be compared with capillary glucose values of plasma from the same subjects.

Capillary blood plasma samples will be properly prepared at the clinical site, frozen and shipped on dry ice to a Clinical Laboratory Improvement Amendments (CLIA)–certified, accredited laboratory for analysis on a glucose comparative instrument.

Glucose comparative instruments will be tested with appropriate manufacturers controls as well as with National Institute of Standards and Technology (NIST) 965b standards to ensure accuracy and performance (analytical imprecision ≤ 2.9% a bias of ≤ 2.2%, and a total error ≤ 6.9%) according to ADA and National Academy of Clinical Biochemistry (NACB) guidelines.[6]

In some cases, a subject’s capillary-fingerstick blood will also be collected into a micro-tube and glycolyzed to obtain blood samples in the hypoglycemic range not safely obtained directly from the subject. Glycolyzed blood samples will only be assayed on BGMSs that are insensitive to oxygen.

Testing of blood will include glucose concentration and hematocrit.

Trained personnel will oversee the studies, monitor the clinical and laboratory sites, conduct Part 11 compliant data management, perform statistical analysis, create reports and post results on clintrials.gov and on the DTS BGMS Surveillance Program website.

Study Design

This protocol describes a series of studies, to be conducted in a similar fashion, to test the accuracy of marketed BGMSs. Each study will test a different BGMS. The protocol allows for more than a single BGMS to be tested within any study. The first part of a study will be conducted at a clinical site where blood glucose will be measured by a BGMS. The second part of a study will be conducted at a reference clinical laboratory where blood glucose will be measured by a protocol-specified comparable method that is traceable to established reference methods and materials.

Subjects must be at least 18 years old. Most subjects will have a diagnosis of diabetes. Up to 33% of subjects with no diagnosis of diabetes may be recruited to obtain values in the low blood glucose range (ie, <81 mg/dl) where BGMS accuracy is critical.

For each study, subjects will make one visit to the clinical trial site. Study sessions will be conducted by study staff members who are familiar with all aspects of the protocol, use of the BGMSs, data collection, and good clinical practice (GCP). Subjects will complete the informed consent process before any study procedures as performed.

Limited demographic and medical information about the subjects will be collected, including: age, gender, ethnicity, race, and diabetes type (ie, type 1, type 2, prediabetes, or no diabetes). All tests will be performed by trained staff professionals.

The following information will be collected but will not result in exclusion from the study. If the subject consumed the following medications within 48 hours of the testing procedure: acetaminophen, ibuprofen, salicylate-containing drugs, or ascorbic acid-containing drugs. However, if these substances are contraindicated in the BGMS labeling then the data from that subject will be excluded from the analysis.

Staff Prestudy Orientation and Training

Study staff training will be conducted prior to the initial BGMS assay for the DTS BGMS Surveillance Program and periodically thereafter, at no duration greater than 1-year intervals, but sooner if there is any question or cause for retraining.

The Study Manager will be responsible for overseeing the entire study, including but not limited to: study progress at both the clinical and laboratory sites, training of study staff on the protocol and acquisition/use of the products, adequate monitoring of the study, and the final close out of the study. The Study Manager will review all study documents, procedures, and specific instructions for use of the BGMSs being tested by the site staff.

GCP training will be provided for the study staff through the CITI Program (the Collaborative Institutional Training Initiative of the University of Miami) or another qualified organization or body. A GCP course completion report for each member of the study staff at the clinical and laboratory sites will be provided to the Sponsor.

Subject Enrollment

At least 110 and up to 125 subjects who meet the inclusion–exclusion criteria will be enrolled into each study to ensure approximately 100 evaluable natural samples results for each BGMS being tested.

Site study staff will perform the following steps when the subject arrives at the clinical trial site: complete the informed consent process; ensure the subject meets all inclusion–exclusion criteria and day of study criteria; assign a unique number to each subject; record subject demographic, medical, and diabetes history information as well as medications taken in the last 48 hours; and offer some subjects something to eat prior to testing to obtain blood glucose values in the high range.

Assaying Glucose in Natural Blood Samples

Staff will perform a shallow fingerstick on the subject; the capillary blood will be applied directly from the finger onto a test strip of the BGMS being tested. Blood from a separate, deep fingerstick will then be collected immediately (but no more than 10 minutes later) into a microtube for centrifugation (within 5 minutes of collection) to separate plasma from cells. The plasma will be carefully transferred to another labeled tube, frozen, and prepared for shipment to the research laboratory for analysis on the comparative glucose instrument.

To facilitate obtaining a wide range of blood glucose values, sometimes a subject may be asked to come to the clinic in a fasting state and have a test performed. Other times, a nonfasting subject may be given something to eat and then be tested within 60 to 120 minutes to obtain a high BG value.

In total, there will be approximately 100 evaluable natural blood sample results from at least 110 subjects and no more than 125 subjects for each type of BGMS tested. Blood will also be collected to determine the hematocrit.

There are no criteria in this program for the number of test strip lots to be tested. This study is designed to test whatever strip lots are obtained from the sourcing process in a manner similar to how a person with diabetes would obtain test strips. This method mimics the real life situation.

Assaying Glucose in Glycolyzed Blood Samples

A BGMS will be tested across a wide range of blood glucose values. To obtain an adequate numbers of test results in the hypoglycemic range (Table 1), some blood samples will also be allowed to glycolyze to lower the blood glucose concentration. This method for generating hypoglycemic specimens protects subjects from the risks of hypoglycemia associated with manipulating their blood glucose levels downward.

Table 1.

Required Distribution of Data Points per Protocol According to Capillary Blood Glucose Level for Oxygen-Sensitive and Oxygen-Insensitive BGMSs.

	Capillary blood glucose
Type of BGMS according to oxygen sensitivity	<81 mg/dl	81-250 mg/dl	>250 mg/dl
Oxygen-insensitive systems that can handle blood specimen modification to generate hypoglycemic data points	25 ± 10%[b]	50 ± 10%[b]	25 ± 10%[b]
Oxygen-sensitive systems for which modified specimens cannot be used[a] to generate hypoglycemic data points	10 ± 10%[b]	65 ± 10%[b]	25 ± 10%[b]

As described earlier in the Assaying Glucose in Glycolyzed Blood Samples section, the blood samples for some BGMSs cannot be modified to generate data points in the <81 mg/dl range.

This value is an absolute percentage and not a relative percentage of the total.

Glycolyzed blood samples will be assayed only on BGMS products that are oxygen insensitive. This is because glycolysis changes the oxygen concentration of blood samples and the accuracy of some BGMSs is affected by the ambient oxygen concentration. BGMSs using glucose oxidase chemistry are particularly sensitive to oxygen levels.

To achieve blood sample glycolyzation, after the subject participates in the initial testing period, staff will perform an additional deep fingerstick on a subject using a Tenderlett™ or similar device. The capillary blood will be collected into a microtube(s) containing lithium heparin and be allowed to glycolyze in a water bath at 32°C ±2 degrees to lower the glucose level.

A blood drop from the tube of glycolyzing blood will be placed onto parafilm and immediately transferred to the test strip of the BGMS. If the blood glucose value is not less than 81 mg/dl, the tube will be recapped and placed back into the water bath for further glycolysis. Once the desired glucose concentration of the modified blood in the tube is achieved, an additional meter test will be performed. The goal is to obtain a range of low glucose values between 20 and 80 mg/dl. The remaining glycolyzed blood in the tube will be centrifuged (within 5 minutes of collection) and the plasma placed into a new tube, properly labeled and frozen for shipment to the clinical laboratory for assay on the comparative glucose instrument.

Safety Procedures

Study staff will clean and disinfect the BGMS after each use according to the manufacturer’s instructions, and at any time a meter becomes contaminated with blood. A new or properly cleaned and disinfected meter must be used for each subject. A new lancing device and a new lancet must be used for each subject. Safety precautions will be used throughout the study to prevent exposure to biohazard material.

Capillary Blood Glucose Target Ranges

The goal is to obtain at least 100 capillary blood samples (natural and/or modified) distributed across 3 glucose ranges (<81 mg/dl, 81-250 mg/dl, and >250 mg/dl) as shown in Table 1.

It is desired to obtain at least 10 natural samples below 81 mg/dl, although this may not always be possible.

Device Failures

Any functional problems with a BGMS will be documented by study staff. Study staff will be specific about describing the problem and the sequence of events that led to it. All information will be documented, including BGMS (meter) serial number or identification number as well as the test strip lot number. It is important to distinguish user error from BGMS failures. If 5% or more of BGMS devices exhibit a true system failure, then that system will be defined as a failed system despite its accuracy.

See the attached supplementary material for specific details of all testing procedures.

Data Recording

Study staff at the clinical site will record meter results via Electronic Data Capture (EDC) and/or paper case record forms (CRFs). Meter memory (if available) will also be reviewed to ensure that values were correctly documented. Staff at the laboratory site will record results of the comparative glucose instrument on EDC and/or paper CRFs. It is critical that proper labeling is performed to ensure that BGMS results will be properly matched to corresponding comparative glucose assay results. The results recorded for both the BGMS and the corresponding comparative glucose assay results will be sent to the data manager at the Contract Research Organization (CRO) for processing and analysis by the statistician.

Temperature and Humidity Monitoring at the Clinical Site

Study staff will measure the temperature and humidity of the testing area twice per day during the BGMS testing using thermometer/hygrometer(s). Staff will record results and ensure that environmental conditions are within the labeling of the BGMS. If physical conditions at the testing site are out of range, then the environment will have to be changed.

Qualifications of the Research Laboratory

The laboratory will be CLIA-certified, accredited laboratory and must participate in CAP or other proficiency testing glucose surveys. The laboratory must be using a comparative glucose method with records of quality control for at least the previous 6 months.

Proficiency testing (PT) should be performed as required by law in the United States (CLIA 88). The PT provider will send samples to the laboratory, which needs to analyze the samples in a manner identical to that used for patient samples. Details are in CLSI document GP27A2.[7]

Measurement of Glucose by Comparative Glucose Analyzer at Research Laboratory

It is imperative that the comparative methods to which the BGMSs are being compared are appropriately accurate, precise, and traceable to established reference methods and materials. All laboratory methods have bias. The bias of the comparative method will be determined by analyzing reference standards (also known as secondary reference materials) of known glucose concentration (established by mass spectrometry) with each batch of subject specimens. This will allow both direct comparison of BGMS results to a laboratory comparative method and estimation of BGMS bias to the mass spectrometry glucose reference method.

Before a research laboratory site can be qualified to participate in the study, the suitability of the instrument for use as the glucose comparative method at that site will be assessed by comparing the measurement bias with predefined performance criteria.

The glucose comparative method to which the BGMS will be compared should be calibrated following the manufacturer’s instructions. NIST 965 Standards (4 levels) will be used to assess the accuracy of the glucose comparative instrument with respect to the reference method. Each sample should be analyzed in duplicate in a single run and bias estimated by calculating the difference between the comparative method results and the expected results for the sample established by the reference mass spectrometry method. A NIST Certificate of Analysis will be included in the study file with all Standard Reference Material 965b Glucose in Frozen Human Serum that is used in this study. This Certificate of Analysis defines the performance of the Standard Reference Material.

Only an instrument/method that meets predefined performance criteria will be used as a glucose comparative analyzer for this study. Any comparative method that meets the predefined criteria for accuracy and precision is acceptable.

As per ADA and NACB guidelines predefined performance criteria for the comparative method are analytical imprecision ≤ 2.9%, a bias of ≤ 2.2%, and a total error ≤ 6.9%.[6] The comparative method analyzer results for that day’s batch analysis will not be used if either the manufacturer controls are outside the acceptable QC ranges or any of the 4 NIST standard reference samples do not meet predefined criteria for precision of sample replicates.

Statistical Analysis

Accuracy Testing

The DTS BGMS Surveillance Program will determine if performance of a marketed BGMS meets prespecified criteria defined below. If it does, then the BGMS will receive a DTS seal of approval. To grant or deny a seal of approval, the surveillance problem is formulated in terms of statistical hypothesis testing, which establishes to what extent the meter test results are consistent with ISO 15197: 2013. Two hypotheses are used to create 3 zones for the probability that a BGMS would perform in compliance with ISO 2013, as follows:

A: The Null Hypothesis is H0: “the meter does not meet the ISO 15197: 2013 level of performance.” In this case the meter is presumed inaccurate until this is rejected. Statistically, H0 will be rejected if the device generates a number K of readings within the ISO 15197: 2013 standard limits, such that K≥K0 where K0 is a certain threshold or “critical” value.

B: The Null Hypothesis is H1: “the meter does meet the ISO 15197: 2013 level of performance.” In this case the meter is presumed accurate until this is rejected. Statistically, H1 will be rejected if the device generates a number K of readings within the ISO 2013 standard limits that is K

Intuitively K0>K1 and, due to inherent statistical uncertainty, there is an “indeterminate zone” for K between these 2 numbers that depends on the sample size N: increasing the sample size will bring K0 and K1 closer, narrowing the indeterminate zone. Given the 2 critical values K0 and K1, the zones are defined as follows:

Clear Acceptance with at least 95% probability of adequate performance (K≥K0);

Indeterminate Performance with between 5% and 95% probability of performance (K1

Clear Rejection with less than 5% probability of adequate performance (K

Following the testing of these hypotheses, each meter will receive (1) a notation as to whether or not the device received the program’s seal of approval, which will be awarded to a BGMS in the Clear Acceptance and Indeterminate Performance zones and will be withheld from a BGMS in the Clear Rejection zone; (2) an estimate of its analytical accuracy—the percentage of data points that were compliant with the ISO 15197 2013’s analytical accuracy requirements; and (3) a probability of adequate performance defined through a Binomial model as described in the Statistical Analysis section of the protocol (Appendix A of the supplementary material). Figure 1 illustrates this process and presents the probability that a device would pass the accuracy test given that it had generated K accurate readings out of N=100 tests:

Figure 1.

Number readings K within the ISO 15197:2013 Standard limits (out of N=100).

Retesting of BGMS Products That Fail Their Initial Accuracy Test

Upon initial testing, devices in the Clear Acceptance and Indeterminate Performance zones will receive the program seal of approval; devices in the Clear Rejection Zone will not receive the program seal of approval, which is equivalent to rejecting the hypothesis H1: “the meter does meet the ISO 15197: 2013 level of performance.” Thus, if a device fails its initial test and comes for retesting, H1 is no longer applicable as a null hypothesis—the device will have to be assumed not meeting ISO 15197: 2013 because it had failed its first test. It follows that during retesting the device will have to show higher level of performance—sufficient to reject the null hypothesis H0—to be awarded the program seal of approval. For example, suppose that on initial testing a device yields 88 readings meeting ISO 15197: 2013 out of 100 total readings and is therefore rejected because 88 < K1 = 91. Upon retesting, this device will have to yield 98 or more accurate readings (out of 100) to pass the test.

Graphs and Additional Analyses

Several additional graphs and analyses will be computed and offered to further clarify the nature of device errors and to identify outliers in device readings:

Surveillance error grid (SEG) analysis

The SEG Analysis for clinical accuracy[8] will be computed using the SEG software[9] and the results will be presented graphically on the color-coded surveillance plot. Two separate SEG plots will be generated: one for natural and one for glycolyzed samples. The results will be also presented in table format as shown in Table 3 in the Statistical Analysis section of the protocol (Appendix A of the supplementary material).

Bland–Altman plots

Modified Bland–Altman plots (difference between evaluation device results and reference results plotted against reference results) will be constructed for visual inspection of device errors with respect to reference glucose levels. To reflect the clinical risk of large deviations, the Modified Bland–Altman plots will also be color-coded in the same way as the Surveillance Error Grid plot—each data point will be coded according to its Surveillance Grid coordinates. As detailed in the Statistical Analysis section of the protocol (Appendix A of the supplementary material), 2 separate modified Bland–Altman plots will be presented: one for natural and one for glycolyzed samples.

Additional accuracy zone chart

In addition to the ISO 15197: 2013 standard the number of readings within accuracy limits of ±15 mg/dl (< 100 mg/dl) and within ±15% (≥ 100 mg/dl) will be presented together with number of readings within narrower zones as suggested by the 2014 FDA Draft Guidance[3] (see Table 4 in the Statistical Analysis section of the protocol—Appendix A of the supplementary material).

Regression and correlation

The correlation and regression coefficients between reference values and device reading will be reported. This analysis will use all samples, not differentiating natural from glycolyzed samples (see Table 5 in the Statistical Analysis of the protocol—Appendix A of the supplementary material).

Future

The DTS BGMS Surveillance Program will assess the performance of cleared BGMSs, verify adherence to the protocol by clinical trial and reference laboratory test sites, and award a program seal of approval to accurate products. The published protocol will also be available to independent investigators who wish to test the performance of BGMSs outside of the DTS program.

Discussion: Achieving Consensus on Four Specific Controversial Topics

As mentioned above, 4 controversial topics in particular generated lively discussion to achieve consensus. These included: (1) choice and qualification of comparative glucose instruments; (2) a method of blood sampling; (3) whether to test BGMS and comparative glucose analyzer results at the same or different locations; and (4) modification of blood samples.

Choice of and Qualification of Comparative Glucose Instruments

There was much discussion concerning what comparative glucose instrument to employ. Many BGMS manufacturers prefer the glucose oxidase–based Yellow Springs Instrument (YSI) glucose analyzer not only to test the accuracy of their systems but also in the development of their systems. Other manufacturers use large laboratory glucose analyzers such as the hexokinase-based Hitachi 917 or Cobas 6000 c501 (Roche Diagnostics GmbH, Mannheim, Germany).

The Steering Committee decided to accept any comparative glucose analyzer that met prespecified accuracy criteria as follows: Glucose Comparative instruments will be tested with appropriate manufacturer’s controls as well as with NIST 965b standards to ensure accuracy and performance. The following must be met: analytical imprecision ≤2.9%; a bias of ≤2.2%; and a total error ≤6.9%.[6]

The committee decided that each BGMS would be assayed at a clinical site, and the plasma samples would be shipped for testing at a different location (CLIA-certified, accredited laboratory) for analysis on a glucose comparative instrument.

Method of Blood Sampling

It was decided the subject’s capillary blood will be directly applied to the test strip of the BGMS as per labeling of the device. The method of collecting blood from a separate shallow fingerstick for assay on the BGMS and from a deep fingerstick for the comparative glucose instrument and then comparing the samples has its limitations. However, the method of applying blood directly from the subject’s finger onto the BGMS test strip conforms to the product labeling, thus it was chosen as the method for this program. The alternative would have been to collect capillary blood from a deep fingerstick into a tube, use a drop of blood from the tube to assay on the BGMS and centrifuge the remainder to obtain plasma to assay on the comparative glucose analyzer.

Whether to Test BGMS and Comparative Glucose Analyzer at the Same or Different Locations

Many BGMS accuracy studies are conducted at a single clinical site. After measurement of the subject’s capillary glucose on the BGMS, a sample of whole blood or plasma from the same subject is assayed on a YSI glucose analyzer at the same location.

The DTS BGMS Surveillance Steering Committee decided to separate the two parts of the study. That is, the BGMS testing will be conducted at a clinical trial site, while the plasma will be frozen and sent to a participating laboratory site for analysis on a comparative glucose instrument. One of the advantages of conducting the study in this manner is to reduce potential investigator bias. It allows for a three-way blind so that an independent third party assembles and collates the data from the BGMS and the corresponding comparative laboratory glucose generated value.

Modification of Blood Samples

A BGMS will be tested across a wide range of blood glucose values. BGMS accuracy in the low glucose range is critically important for patient self-management. It is expected that some naturally low glucose specimens (considered to be <81 mg/dl in this protocol) will occur in every test subject population. These natural hypoglycemic specimens will be tested by all BGMSs. To safely achieve adequate numbers of tests results across the hypoglycemic range, some blood samples (from at least 20 subjects) will also be modified, that is, allowed to glycolyze to lower the blood glucose concentration.

The accurate performance of some BGMS products is particularly sensitive to oxygen levels, especially those with glucose oxidase chemistry. Because glycolysis changes the oxygen concentration of a blood sample, glycolyzed blood samples will be assayed only on BGMSs that are oxygen insensitive. Most BGMSs with glucose dehydrogenase-based chemistry are oxygen insensitive.

Therefore, fewer hypoglycemic data points will be included in the distribution (by glucose level) for oxygen-sensitive BGMSs than for oxygen-insensitive BGMSs. Accordingly, caution is needed if results from surveillance of oxygen-sensitive systems and oxygen-insensitive systems are ever compared, because the distribution of data points in the hypoglycemic range will differ between oxygen-sensitive and oxygen-insensitive BGMSs.

Conclusions

The DTS BGMS Surveillance Program for Cleared Blood Glucose Monitoring Systems is intended to protect people with diabetes from inaccurate, poorly performing BGMS products that are currently on the market in the United States.[4] This Surveillance Program will provide an independent assessment of the analytical performance of BGMSs following clearance by the FDA as well as generate information that can assist people with diabetes, health care providers, and payers in making educated selection of BGMSs. Such a program is needed, because once a BGMS product has been cleared for use by the FDA, there is currently no systematic postmarket surveillance program that can ensure ongoing product quality. Poorly performing, inaccurate BGMS can negatively impact the ability of people with diabetes to reliably measure their blood glucose levels, and can lead to adverse health effects.

An explanation often given for the poor analytical accuracy of some marketed BGMSs and their failure to meet the analytical accuracy data that they originally submitted to the FDA, is intentional selection. According to this theory, the best performing meters and test strip lots were chosen by the manufacturers for their initial studies submitted for clearance. In an effort to avoid this type of “cherry-picking,” the DTS BGMS Surveillance protocol specifies that products to be tested will be obtained from the same types of channels that patients use to obtain their diabetes supplies, rather than directly from the manufacturers to represent a more “real-world experience.”

The results of this program are intended to alert people with diabetes to inaccurate, poorly performing BGMS products that are currently on the market in the United States. The Surveillance Program will provide an independent assessment of the performance of BGMS products following clearance by the FDA. The results of the program will assist manufacturers to understand how their products are performing in the “real world.” The program will also generate information to assist diabetes patients, health care providers, payers, and regulators to understand the performance of these products that they must routinely select, prescribe, finance, and regulate.