Harald Kirchsteiger1, Lutz Heinemann2, Guido Freckmann3, Volker Lodwig4, Günther Schmelzeisen-Redeker4, Michael Schoemaker4, Luigi Del Re5. 1. Institute for Design and Control of Mechatronical Systems, Johannes Kepler University, Linz, Austria harald.kirchsteiger@jku.at. 2. Science & Co, Düsseldorf, Germany. 3. Institute for Diabetes-Technology GmbH, at Ulm University, Ulm, Germany. 4. Roche Diagnostics GmbH, Mannheim, Germany. 5. Institute for Design and Control of Mechatronical Systems, Johannes Kepler University, Linz, Austria.
Abstract
BACKGROUND: The ongoing progress of continuous glucose monitoring (CGM) systems results in an increasing interest in comparing their performance, in particular in terms of accuracy, that is, matching CGM readings with reference values measured at the same time. Most often accuracy is evaluated by the mean absolute relative difference (MARD). It is frequently overseen that MARD does not only reflect accuracy, but also the study protocol and evaluation procedure, making a cross-study comparison problematic. METHODS: We evaluate the effect of several factors on the MARD statistical properties: number of paired reference and CGM values, distribution of the paired values, accuracy of the reference measurement device itself and the time delay between data pairs. All analysis is done using clinical data from 12 patients wearing 6 sensors each. RESULTS: We have found that a few paired points can have a potentially high impact on MARD. Leaving out those points for evaluation thus reduces the MARD. Similarly, accuracy of the reference measurements greatly affects the MARD as numerical and graphical data show. Results also show that a log-normal distribution of the paired references provides a significantly different MARD than, for example, a uniform distribution. CONCLUSIONS: MARD is a reasonable parameter to characterize the performance of CGM systems when keeping its limitations in mind. To support clinicians and patients in selecting which CGM system to use in a clinical setting, care should be taken to make MARD more comparable by employing a standardized evaluation procedure.
BACKGROUND: The ongoing progress of continuous glucose monitoring (CGM) systems results in an increasing interest in comparing their performance, in particular in terms of accuracy, that is, matching CGM readings with reference values measured at the same time. Most often accuracy is evaluated by the mean absolute relative difference (MARD). It is frequently overseen that MARD does not only reflect accuracy, but also the study protocol and evaluation procedure, making a cross-study comparison problematic. METHODS: We evaluate the effect of several factors on the MARD statistical properties: number of paired reference and CGM values, distribution of the paired values, accuracy of the reference measurement device itself and the time delay between data pairs. All analysis is done using clinical data from 12 patients wearing 6 sensors each. RESULTS: We have found that a few paired points can have a potentially high impact on MARD. Leaving out those points for evaluation thus reduces the MARD. Similarly, accuracy of the reference measurements greatly affects the MARD as numerical and graphical data show. Results also show that a log-normal distribution of the paired references provides a significantly different MARD than, for example, a uniform distribution. CONCLUSIONS: MARD is a reasonable parameter to characterize the performance of CGM systems when keeping its limitations in mind. To support clinicians and patients in selecting which CGM system to use in a clinical setting, care should be taken to make MARD more comparable by employing a standardized evaluation procedure.
Authors: Richard L Weinstein; Sherwyn L Schwartz; Ronald L Brazg; Jolyon R Bugler; Thomas A Peyser; Geoffrey V McGarraugh Journal: Diabetes Care Date: 2007-03-02 Impact factor: 19.112
Authors: Timothy S Bailey; Andrew Ahmann; Ronald Brazg; Mark Christiansen; Satish Garg; Elaine Watkins; John B Welsh; Scott W Lee Journal: Diabetes Technol Ther Date: 2014-04-07 Impact factor: 6.118
Authors: Edward R Damiano; Katherine McKeon; Firas H El-Khatib; Hui Zheng; David M Nathan; Steven J Russell Journal: J Diabetes Sci Technol Date: 2014-04-21
Authors: Satish K Garg; James Smith; Christie Beatson; Benita Lopez-Baca; Mary Voelmle; Peter A Gottlieb Journal: Diabetes Technol Ther Date: 2009-02 Impact factor: 6.118
Authors: Edward R Damiano; Firas H El-Khatib; Hui Zheng; David M Nathan; Steven J Russell Journal: Diabetes Care Date: 2012-12-28 Impact factor: 19.112
Authors: J Kropff; D Bruttomesso; W Doll; A Farret; S Galasso; Y M Luijf; J K Mader; J Place; F Boscari; T R Pieber; E Renard; J H DeVries Journal: Diabetes Obes Metab Date: 2014-09-10 Impact factor: 6.577
Authors: Florian Reiterer; Philipp Polterauer; Michael Schoemaker; Guenther Schmelzeisen-Redecker; Guido Freckmann; Lutz Heinemann; Luigi Del Re Journal: J Diabetes Sci Technol Date: 2016-09-25