Isaac Corro Ramos1, George A K van Voorn2, Pepijn Vemer3, Talitha L Feenstra4, Maiwenn J Al5. 1. Institute for Medical Technology Assessment, Erasmus University Rotterdam, Rotterdam, The Netherlands. Electronic address: corroramos@imta.eur.nl. 2. Biometris, Wageningen University and Research, Wageningen, The Netherlands. 3. Department of Epidemiology, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands; Pharmacoepidemiology and Pharmacoeconomics (PE2), Groningen University, Groningen, The Netherlands. 4. Department of Epidemiology, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands; National Institute for Public Health and the Environment (RIVM), Bilthoven, The Netherlands. 5. Institute of Health Policy and Management, Erasmus University Rotterdam, Rotterdam, The Netherlands.
Abstract
BACKGROUND: The validation of health economic (HE) model outcomes against empirical data is of key importance. Although statistical testing seems applicable, guidelines for the validation of HE models lack guidance on statistical validation, and actual validation efforts often present subjective judgment of graphs and point estimates. OBJECTIVES: To discuss the applicability of existing validation techniques and to present a new method for quantifying the degrees of validity statistically, which is useful for decision makers. METHODS: A new Bayesian method is proposed to determine how well HE model outcomes compare with empirical data. Validity is based on a pre-established accuracy interval in which the model outcomes should fall. The method uses the outcomes of a probabilistic sensitivity analysis and results in a posterior distribution around the probability that HE model outcomes can be regarded as valid. RESULTS: We use a published diabetes model (Modelling Integrated Care for Diabetes based on Observational data) to validate the outcome "number of patients who are on dialysis or with end-stage renal disease." Results indicate that a high probability of a valid outcome is associated with relatively wide accuracy intervals. In particular, 25% deviation from the observed outcome implied approximately 60% expected validity. CONCLUSIONS: Current practice in HE model validation can be improved by using an alternative method based on assessing whether the model outcomes fit to empirical data at a predefined level of accuracy. This method has the advantage of assessing both model bias and parameter uncertainty and resulting in a quantitative measure of the degree of validity that penalizes models predicting the mean of an outcome correctly but with overly wide credible intervals.
BACKGROUND: The validation of health economic (HE) model outcomes against empirical data is of key importance. Although statistical testing seems applicable, guidelines for the validation of HE models lack guidance on statistical validation, and actual validation efforts often present subjective judgment of graphs and point estimates. OBJECTIVES: To discuss the applicability of existing validation techniques and to present a new method for quantifying the degrees of validity statistically, which is useful for decision makers. METHODS: A new Bayesian method is proposed to determine how well HE model outcomes compare with empirical data. Validity is based on a pre-established accuracy interval in which the model outcomes should fall. The method uses the outcomes of a probabilistic sensitivity analysis and results in a posterior distribution around the probability that HE model outcomes can be regarded as valid. RESULTS: We use a published diabetes model (Modelling Integrated Care for Diabetes based on Observational data) to validate the outcome "number of patients who are on dialysis or with end-stage renal disease." Results indicate that a high probability of a valid outcome is associated with relatively wide accuracy intervals. In particular, 25% deviation from the observed outcome implied approximately 60% expected validity. CONCLUSIONS: Current practice in HE model validation can be improved by using an alternative method based on assessing whether the model outcomes fit to empirical data at a predefined level of accuracy. This method has the advantage of assessing both model bias and parameter uncertainty and resulting in a quantitative measure of the degree of validity that penalizes models predicting the mean of an outcome correctly but with overly wide credible intervals.
Authors: Brandon K Bellows; Natalia Ruiz-Negrón; Kirsten Bibbins-Domingo; Jordan B King; Mark J Pletcher; Andrew E Moran; Valy Fontil Journal: Circ Cardiovasc Qual Outcomes Date: 2019-06-05