Michael Willis1, Pierre Johansen2, Andreas Nilsson2, Christian Asseburg2. 1. The Swedish Institute for Health Economics, Box 2127, SE-220 02, Lund, Sweden. michael.willis@ihe.se. 2. The Swedish Institute for Health Economics, Box 2127, SE-220 02, Lund, Sweden.
Abstract
BACKGROUND: The Economic and Health Outcomes Model of Type 2 Diabetes Mellitus (ECHO-T2DM) was developed to address study questions pertaining to the cost-effectiveness of treatment alternatives in the care of patients with type 2 diabetes mellitus (T2DM). Naturally, the usefulness of a model is determined by the accuracy of its predictions. A previous version of ECHO-T2DM was validated against actual trial outcomes and the model predictions were generally accurate. However, there have been recent upgrades to the model, which modify model predictions and necessitate an update of the validation exercises. OBJECTIVES: The objectives of this study were to extend the methods available for evaluating model validity, to conduct a formal model validation of ECHO-T2DM (version 2.3.0) in accordance with the principles espoused by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Society for Medical Decision Making (SMDM), and secondarily to evaluate the relative accuracy of four sets of macrovascular risk equations included in ECHO-T2DM. METHODS: We followed the ISPOR/SMDM guidelines on model validation, evaluating face validity, verification, cross-validation, and external validation. Model verification involved 297 'stress tests', in which specific model inputs were modified systematically to ascertain correct model implementation. Cross-validation consisted of a comparison between ECHO-T2DM predictions and those of the seminal National Institutes of Health model. In external validation, study characteristics were entered into ECHO-T2DM to replicate the clinical results of 12 studies (including 17 patient populations), and model predictions were compared to observed values using established statistical techniques as well as measures of average prediction error, separately for the four sets of macrovascular risk equations supported in ECHO-T2DM. Sub-group analyses were conducted for dependent vs. independent outcomes and for microvascular vs. macrovascular vs. mortality endpoints. RESULTS: All stress tests were passed. ECHO-T2DM replicated the National Institutes of Health cost-effectiveness application with numerically similar results. In external validation of ECHO-T2DM, model predictions agreed well with observed clinical outcomes. For all sets of macrovascular risk equations, the results were close to the intercept and slope coefficients corresponding to a perfect match, resulting in high R 2 and failure to reject concordance using an F test. The results were similar for sub-groups of dependent and independent validation, with some degree of under-prediction of macrovascular events. CONCLUSION: ECHO-T2DM continues to match health outcomes in clinical trials in T2DM, with prediction accuracy similar to other leading models of T2DM.
BACKGROUND: The Economic and Health Outcomes Model of Type 2 Diabetes Mellitus (ECHO-T2DM) was developed to address study questions pertaining to the cost-effectiveness of treatment alternatives in the care of patients with type 2 diabetes mellitus (T2DM). Naturally, the usefulness of a model is determined by the accuracy of its predictions. A previous version of ECHO-T2DM was validated against actual trial outcomes and the model predictions were generally accurate. However, there have been recent upgrades to the model, which modify model predictions and necessitate an update of the validation exercises. OBJECTIVES: The objectives of this study were to extend the methods available for evaluating model validity, to conduct a formal model validation of ECHO-T2DM (version 2.3.0) in accordance with the principles espoused by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Society for Medical Decision Making (SMDM), and secondarily to evaluate the relative accuracy of four sets of macrovascular risk equations included in ECHO-T2DM. METHODS: We followed the ISPOR/SMDM guidelines on model validation, evaluating face validity, verification, cross-validation, and external validation. Model verification involved 297 'stress tests', in which specific model inputs were modified systematically to ascertain correct model implementation. Cross-validation consisted of a comparison between ECHO-T2DM predictions and those of the seminal National Institutes of Health model. In external validation, study characteristics were entered into ECHO-T2DM to replicate the clinical results of 12 studies (including 17 patient populations), and model predictions were compared to observed values using established statistical techniques as well as measures of average prediction error, separately for the four sets of macrovascular risk equations supported in ECHO-T2DM. Sub-group analyses were conducted for dependent vs. independent outcomes and for microvascular vs. macrovascular vs. mortality endpoints. RESULTS: All stress tests were passed. ECHO-T2DM replicated the National Institutes of Health cost-effectiveness application with numerically similar results. In external validation of ECHO-T2DM, model predictions agreed well with observed clinical outcomes. For all sets of macrovascular risk equations, the results were close to the intercept and slope coefficients corresponding to a perfect match, resulting in high R 2 and failure to reject concordance using an F test. The results were similar for sub-groups of dependent and independent validation, with some degree of under-prediction of macrovascular events. CONCLUSION: ECHO-T2DM continues to match health outcomes in clinical trials in T2DM, with prediction accuracy similar to other leading models of T2DM.
Authors: Thomas J Hoerger; John S Wittenborn; Joel E Segel; Nilka R Burrows; Kumiko Imai; Paul Eggers; Meda E Pavkov; Regina Jordan; Susan M Hailpern; Anton C Schoolwerth; Desmond E Williams Journal: Am J Kidney Dis Date: 2010-02-08 Impact factor: 8.860
Authors: Silvio E Inzucchi; Richard M Bergenstal; John B Buse; Michaela Diamant; Ele Ferrannini; Michael Nauck; Anne L Peters; Apostolos Tsapas; Richard Wender; David R Matthews Journal: Diabetes Care Date: 2015-01 Impact factor: 19.112
Authors: William C Cushman; Gregory W Evans; Robert P Byington; David C Goff; Richard H Grimm; Jeffrey A Cutler; Denise G Simons-Morton; Jan N Basile; Marshall A Corson; Jeffrey L Probstfield; Lois Katz; Kevin A Peterson; William T Friedewald; John B Buse; J Thomas Bigger; Hertzel C Gerstein; Faramarz Ismail-Beigi Journal: N Engl J Med Date: 2010-03-14 Impact factor: 91.245
Authors: P M Clarke; A M Gray; A Briggs; A J Farmer; P Fenn; R J Stevens; D R Matthews; I M Stratton; R R Holman Journal: Diabetologia Date: 2004-10-27 Impact factor: 10.122
Authors: David M Eddy; William Hollingworth; J Jaime Caro; Joel Tsevat; Kathryn M McDonald; John B Wong Journal: Med Decis Making Date: 2012 Sep-Oct Impact factor: 2.583
Authors: Michael Willis; Christian Asseburg; April Slee; Andreas Nilsson; Cheryl Neslusan Journal: Pharmacoeconomics Date: 2021-02-13 Impact factor: 4.981
Authors: Nasuh C Büyükkaramikli; Maureen P M H Rutten-van Mölken; Johan L Severens; Maiwenn Al Journal: Pharmacoeconomics Date: 2019-11 Impact factor: 4.981
Authors: Mi Jun Keng; Jose Leal; Marion Mafham; Louise Bowman; Jane Armitage; Borislava Mihaylova Journal: Value Health Date: 2021-10-27 Impact factor: 5.101