Ka Keat Lim1,2, Rositsa Koleva-Kolarova3, Julia Fox-Rushby4,5. 1. Faculty of Life Sciences and Medicine, School of Life Course and Population Sciences, King's College London, London, UK. ka_keat.lim@kcl.ac.uk. 2. National Institute for Health Research (NIHR) Biomedical Research Centre, Guy's and St Thomas' NHS Foundation Trust and King's College London, London, UK. ka_keat.lim@kcl.ac.uk. 3. Nuffield Department of Population Health, Health Economics Research Centre, University of Oxford, Oxford, UK. 4. Faculty of Life Sciences and Medicine, School of Life Course and Population Sciences, King's College London, London, UK. 5. National Institute for Health Research (NIHR) Biomedical Research Centre, Guy's and St Thomas' NHS Foundation Trust and King's College London, London, UK.
Abstract
OBJECTIVES: The aim of this study was to examine whether and how the content of six checklists (Caro, Consensus on Health Economic Criteria [CHEC]-Extended, European Network of Health Economic Databases [EURONHEED], National Institute for Health and Care Excellence [NICE], Philips, Welte) affect the consistency in findings on methodological quality and transferability, using 10 model-based economic evaluations of genetic-guided pharmacotherapy for venous thromboembolism. METHODS: Each checklist was categorised by domain (structure, data, consistency, etc.) and type of assessment (presence vs. appropriateness) and was applied to each study by two independent reviewers who agreed on ratings via consensus, and discussion with a third reviewer when necessary. Methodological quality scores and rankings were examined using Spearman correlation tests, with subgroup analyses for domains and types of assessment. We compared overall ratings of transferability qualitatively, including how content may affect what is considered 'transferable'. RESULTS: The checklists had similar proportions of items judging presence and appropriateness, but varying proportions of items across domains. For methodological quality, ranking consistencies were the highest between CHEC-Extended-Philips, Philips-NICE and NICE-Caro, with similar consistencies for domains and type of assessment. For transferability, NICE and Caro identified the same study, which scored high on EURONHEED, as transferable to the UK, while Welte, which considered methodological quality, identified none as transferable. CONCLUSIONS: We found that the choice of checklist can affect findings on study quality and decisions about whether study results are transferable, indicating that different checklists may shortlist different sets of studies in formulating policy recommendations, leading to different policy decisions. Our systematic approach for evaluating the content of methodological quality and transferability checklists of economic evaluations can be extended to other checklists.
OBJECTIVES: The aim of this study was to examine whether and how the content of six checklists (Caro, Consensus on Health Economic Criteria [CHEC]-Extended, European Network of Health Economic Databases [EURONHEED], National Institute for Health and Care Excellence [NICE], Philips, Welte) affect the consistency in findings on methodological quality and transferability, using 10 model-based economic evaluations of genetic-guided pharmacotherapy for venous thromboembolism. METHODS: Each checklist was categorised by domain (structure, data, consistency, etc.) and type of assessment (presence vs. appropriateness) and was applied to each study by two independent reviewers who agreed on ratings via consensus, and discussion with a third reviewer when necessary. Methodological quality scores and rankings were examined using Spearman correlation tests, with subgroup analyses for domains and types of assessment. We compared overall ratings of transferability qualitatively, including how content may affect what is considered 'transferable'. RESULTS: The checklists had similar proportions of items judging presence and appropriateness, but varying proportions of items across domains. For methodological quality, ranking consistencies were the highest between CHEC-Extended-Philips, Philips-NICE and NICE-Caro, with similar consistencies for domains and type of assessment. For transferability, NICE and Caro identified the same study, which scored high on EURONHEED, as transferable to the UK, while Welte, which considered methodological quality, identified none as transferable. CONCLUSIONS: We found that the choice of checklist can affect findings on study quality and decisions about whether study results are transferable, indicating that different checklists may shortlist different sets of studies in formulating policy recommendations, leading to different policy decisions. Our systematic approach for evaluating the content of methodological quality and transferability checklists of economic evaluations can be extended to other checklists.