Lei Si1, Michael S Willis2, Christian Asseburg3, Andreas Nilsson2, Michelle Tew4, Philip M Clarke5, Mark Lamotte6, Mafalda Ramos7, Hui Shao8, Lizheng Shi9, Ping Zhang10, Phil McEwan11, Wen Ye12, William H Herman13, Shihchen Kuo14, Deanna J Isaman12, Wendelin Schramm15, Fabian Sailer15, Alan Brennan16, Daniel Pollard16, Harry J Smolen17, José Leal18, Alastair Gray19, Rishi Patel19, Talitha Feenstra20, Andrew J Palmer21. 1. The George Institute for Global Health, UNSW Sydney, Kensington, Australia; Menzies Institute for Medical Research, University of Tasmania, Hobart, Tasmania, Australia. 2. The Swedish Institute for Health Economics, Lund, Sweden. 3. ESiOR Oy, Kuopio, Finland. 4. Centre for Health Policy, School of Population and Global Health, The University of Melbourne, Victoria, Australia. 5. Centre for Health Policy, School of Population and Global Health, The University of Melbourne, Victoria, Australia; Health Economics Research Centre, Nuffield Department of Population Health, University of Oxford, United Kingdom. 6. Global Health Economics and Outcomes Research, IQVIA, Zaventem, Belgium. 7. Global Health Economics and Outcomes Research, IQVIA, Lisbon, Portugal. 8. Department of Pharmaceutical Outcomes and Policy, College of Pharmacy, University of Florida, Gainesville, Florida. 9. Department of Global Health Management and Policy, School of Public Health and Tropical Medicine, Tulane University, New Orleans, Louisiana, USA. 10. Division of Diabetes Translation, Centers for Disease Control and Prevention, Atlanta, Georgia, USA. 11. Health Economics and Outcomes Research Ltd, Cardiff, United Kingdom. 12. Department of Biostatistics, University of Michigan, Ann Arbor, Michigan, USA. 13. Departments of Internal Medicine and Epidemiology, University of Michigan, Ann Arbor, Michigan, USA. 14. Department of Internal Medicine, University of Michigan, Ann Arbor, Michigan, USA. 15. Centre for Health Economics and Outcomes Research, GECKO Institute, Heilbronn University, Heilbronn, Germany. 16. School of Health and Related Research, University of Sheffield, Sheffield, United Kingdom. 17. Medical Decision Modeling Inc., Indianapolis, Indiana, USA. 18. National Institute for Public Health and the Environment (RIVM), Bilthoven, The Netherlands. 19. Health Economics Research Centre, Nuffield Department of Population Health, University of Oxford, United Kingdom. 20. National Institute for Public Health and the Environment (RIVM), Bilthoven, The Netherlands; University of Groningen, Faculty of Science and Engineering, Groningen, The Netherlands. 21. Menzies Institute for Medical Research, University of Tasmania, Hobart, Tasmania, Australia; Centre for Health Policy, School of Population and Global Health, The University of Melbourne, Victoria, Australia. Electronic address: andrew.palmer@utas.edu.au.
Abstract
OBJECTIVES: The cardiovascular outcomes challenge examined the predictive accuracy of 10 diabetes models in estimating hard outcomes in 2 recent cardiovascular outcomes trials (CVOTs) and whether recalibration can be used to improve replication. METHODS: Participating groups were asked to reproduce the results of the Empagliflozin Cardiovascular Outcome Event Trial in Type 2 Diabetes Mellitus Patients (EMPA-REG OUTCOME) and the Canagliflozin Cardiovascular Assessment Study (CANVAS) Program. Calibration was performed and additional analyses assessed model ability to replicate absolute event rates, hazard ratios (HRs), and the generalizability of calibration across CVOTs within a drug class. RESULTS: Ten groups submitted results. Models underestimated treatment effects (ie, HRs) using uncalibrated models for both trials. Calibration to the placebo arm of EMPA-REG OUTCOME greatly improved the prediction of event rates in the placebo, but less so in the active comparator arm. Calibrating to both arms of EMPA-REG OUTCOME individually enabled replication of the observed outcomes. Using EMPA-REG OUTCOME-calibrated models to predict CANVAS Program outcomes was an improvement over uncalibrated models but failed to capture treatment effects adequately. Applying canagliflozin HRs directly provided the best fit. CONCLUSIONS: The Ninth Mount Hood Diabetes Challenge demonstrated that commonly used risk equations were generally unable to capture recent CVOT treatment effects but that calibration of the risk equations can improve predictive accuracy. Although calibration serves as a practical approach to improve predictive accuracy for CVOT outcomes, it does not extrapolate generally to other settings, time horizons, and comparators. New methods and/or new risk equations for capturing these CV benefits are needed.
OBJECTIVES: The cardiovascular outcomes challenge examined the predictive accuracy of 10 diabetes models in estimating hard outcomes in 2 recent cardiovascular outcomes trials (CVOTs) and whether recalibration can be used to improve replication. METHODS: Participating groups were asked to reproduce the results of the Empagliflozin Cardiovascular Outcome Event Trial in Type 2 Diabetes MellitusPatients (EMPA-REG OUTCOME) and the Canagliflozin Cardiovascular Assessment Study (CANVAS) Program. Calibration was performed and additional analyses assessed model ability to replicate absolute event rates, hazard ratios (HRs), and the generalizability of calibration across CVOTs within a drug class. RESULTS: Ten groups submitted results. Models underestimated treatment effects (ie, HRs) using uncalibrated models for both trials. Calibration to the placebo arm of EMPA-REG OUTCOME greatly improved the prediction of event rates in the placebo, but less so in the active comparator arm. Calibrating to both arms of EMPA-REG OUTCOME individually enabled replication of the observed outcomes. Using EMPA-REG OUTCOME-calibrated models to predict CANVAS Program outcomes was an improvement over uncalibrated models but failed to capture treatment effects adequately. Applying canagliflozin HRs directly provided the best fit. CONCLUSIONS: The Ninth Mount Hood Diabetes Challenge demonstrated that commonly used risk equations were generally unable to capture recent CVOT treatment effects but that calibration of the risk equations can improve predictive accuracy. Although calibration serves as a practical approach to improve predictive accuracy for CVOT outcomes, it does not extrapolate generally to other settings, time horizons, and comparators. New methods and/or new risk equations for capturing these CV benefits are needed.
Authors: Michael Willis; Christian Asseburg; April Slee; Andreas Nilsson; Cheryl Neslusan Journal: Pharmacoeconomics Date: 2021-02-13 Impact factor: 4.981
Authors: Mi Jun Keng; Jose Leal; Marion Mafham; Louise Bowman; Jane Armitage; Borislava Mihaylova Journal: Value Health Date: 2021-10-27 Impact factor: 5.101
Authors: Michelle Tew; Michael Willis; Christian Asseburg; Hayley Bennett; Alan Brennan; Talitha Feenstra; James Gahn; Alastair Gray; Laura Heathcote; William H Herman; Deanna Isaman; Shihchen Kuo; Mark Lamotte; José Leal; Phil McEwan; Andreas Nilsson; Andrew J Palmer; Rishi Patel; Daniel Pollard; Mafalda Ramos; Fabian Sailer; Wendelin Schramm; Hui Shao; Lizheng Shi; Lei Si; Harry J Smolen; Chloe Thomas; An Tran-Duy; Chunting Yang; Wen Ye; Xueting Yu; Ping Zhang; Philip Clarke Journal: Med Decis Making Date: 2021-12-15 Impact factor: 2.749