Jan L Brozek1, Carlos Canelo-Aybar2, Elie A Akl3, James M Bowen4, John Bucher5, Weihsueh A Chiu6, Mark Cronin7, Benjamin Djulbegovic8, Maicon Falavigna9, Gordon H Guyatt1, Ami A Gordon10, Michele Hilton Boon11, Raymond C W Hutubessy12, Manuela A Joore13, Vittal Katikireddi11, Judy LaKind14, Miranda Langendam15, Veena Manja16, Kristen Magnuson10, Alexander G Mathioudakis17, Joerg Meerpohl18, Dominik Mertz19, Roman Mezencev20, Rebecca Morgan19, Gian Paolo Morgano21, Reem Mustafa22, Martin O'Flaherty23, Grace Patlewicz24, John J Riva25, Margarita Posso26, Andrew Rooney5, Paul M Schlosser20, Lisa Schwartz19, Ian Shemilt27, Jean-Eric Tarride28, Kristina A Thayer29, Katya Tsaioun30, Luke Vale31, John Wambaugh24, Jessica Wignall10, Ashley Williams10, Feng Xie19, Yuan Zhang32, Holger J Schünemann1. 1. Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Medicine, McMaster University, Hamilton, Ontario, Canada; McMaster GRADE Centre & Michael DeGroote Cochrane Canada Centre, McMaster University, Hamilton, Ontario, Canada. 2. Department of Paediatrics, Obstetrics and Gynaecology, Preventive Medicine, and Public Health. PhD Programme in Methodology of Biomedical Research and Public Health. Universitat Autònoma de Barcelona, Bellaterra, Spain; Iberoamerican Cochrane Center, Biomedical Research Institute (IIB Sant Pau-CIBERESP), Barcelona, Spain. 3. Department of Internal Medicine, American University of Beirut, Beirut, Lebanon. 4. Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Toronto Health Economics and Technology Assessment (THETA) Collaborative, Toronto, Ontario, Canada. 5. National Toxicology Program, National Institute of Environmental Health Sciences, Durham, NC, USA. 6. Department of Veterinary Integrative Biosciences, Texas A&M University, College Station, TX, USA. 7. School of Pharmacy and Chemistry, Liverpool John Moores University, Liverpool, UK. 8. Center for Evidence-Based Medicine and Health Outcome Research, Morsani College of Medicine, University of South Florida, Tampa, Florida, USA. 9. Institute for Education and Research, Hospital Moinhos de Vento, Porto Alegre, Rio Grande do Sul, Brazil. 10. ICF International, Durham, NC, USA. 11. Institute of Health & Wellbeing, University of Glasgow, Glasgow, UK. 12. Department of Immunization, Vaccines and Biologicals, World Health Organization, Geneva, Switzerland. 13. Clinical Epidemiology and Medical Technology Assessment, Maastricht University Medical Centre+, Maastricht, the Netherlands. 14. LaKind Associates, LLC, Catonsville, MD, USA; Department of Epidemiology and Public Health, University of Maryland School of Medicine, Baltimore, MD, USA. 15. Department of Clinical Epidemiology, Biostatistics, and Bioinformatics, Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands. 16. Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Surgery, University of California Davis, Sacramento, CA, USA; Department of Medicine, Department of Veterans Affairs, Northern California Health Care System, Mather, CA, USA. 17. Division of Infection, Immunity and Respiratory Medicine, University Hospital of South Manchester, University of Manchester, Manchester, UK. 18. Institute for Evidence in Medicine, Medical Center, University of Freiburg, Freiburg-am-Breisgau, Germany; Cochrane Germany, Freiburg-am-Breisgau, Germany. 19. Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada. 20. National Center for Environmental Assessment, U.S. Environmental Protection Agency, Washington, DC, USA. 21. Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; McMaster GRADE Centre & Michael DeGroote Cochrane Canada Centre, McMaster University, Hamilton, Ontario, Canada. 22. Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Medicine, University of Kansas Medical Center, Kansas City, KS, USA. 23. Institute of Population Health Sciences, University of Liverpool, Liverpool, UK. 24. National Center for Computational Toxicology, U.S. Environmental Protection Agency, Durham, NC, USA. 25. McMaster GRADE Centre & Michael DeGroote Cochrane Canada Centre, McMaster University, Hamilton, Ontario, Canada; Department of Family Medicine, McMaster University, Hamilton, Ontario, Canada. 26. Iberoamerican Cochrane Center, Biomedical Research Institute (IIB Sant Pau-CIBERESP), Barcelona, Spain. 27. EPPI-Centre, Institute of Education, University College London, London, UK. 28. Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Programs for Assessment of Technology in Health, McMaster University, Hamilton, Ontario, Canada. 29. Department of Medicine, Department of Veterans Affairs, Northern California Health Care System, Mather, CA, USA. 30. Evidence-Based Toxicology Collaboration, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA. 31. Health Economics Group, Institute of Health and Society, Newcastle University, Newcastle upon Tyne, UK. 32. Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Health Quality Ontario, Toronto, Ontario, Canada.
Abstract
OBJECTIVES: The objective of the study is to present the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) conceptual approach to the assessment of certainty of evidence from modeling studies (i.e., certainty associated with model outputs). STUDY DESIGN AND SETTING: Expert consultations and an international multidisciplinary workshop informed development of a conceptual approach to assessing the certainty of evidence from models within the context of systematic reviews, health technology assessments, and health care decisions. The discussions also clarified selected concepts and terminology used in the GRADE approach and by the modeling community. Feedback from experts in a broad range of modeling and health care disciplines addressed the content validity of the approach. RESULTS: Workshop participants agreed that the domains determining the certainty of evidence previously identified in the GRADE approach (risk of bias, indirectness, inconsistency, imprecision, reporting bias, magnitude of an effect, dose-response relation, and the direction of residual confounding) also apply when assessing the certainty of evidence from models. The assessment depends on the nature of model inputs and the model itself and on whether one is evaluating evidence from a single model or multiple models. We propose a framework for selecting the best available evidence from models: 1) developing de novo, a model specific to the situation of interest, 2) identifying an existing model, the outputs of which provide the highest certainty evidence for the situation of interest, either "off-the-shelf" or after adaptation, and 3) using outputs from multiple models. We also present a summary of preferred terminology to facilitate communication among modeling and health care disciplines. CONCLUSION: This conceptual GRADE approach provides a framework for using evidence from models in health decision-making and the assessment of certainty of evidence from a model or models. The GRADE Working Group and the modeling community are currently developing the detailed methods and related guidance for assessing specific domains determining the certainty of evidence from models across health care-related disciplines (e.g., therapeutic decision-making, toxicology, environmental health, and health economics).
OBJECTIVES: The objective of the study is to present the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) conceptual approach to the assessment of certainty of evidence from modeling studies (i.e., certainty associated with model outputs). STUDY DESIGN AND SETTING: Expert consultations and an international multidisciplinary workshop informed development of a conceptual approach to assessing the certainty of evidence from models within the context of systematic reviews, health technology assessments, and health care decisions. The discussions also clarified selected concepts and terminology used in the GRADE approach and by the modeling community. Feedback from experts in a broad range of modeling and health care disciplines addressed the content validity of the approach. RESULTS: Workshop participants agreed that the domains determining the certainty of evidence previously identified in the GRADE approach (risk of bias, indirectness, inconsistency, imprecision, reporting bias, magnitude of an effect, dose-response relation, and the direction of residual confounding) also apply when assessing the certainty of evidence from models. The assessment depends on the nature of model inputs and the model itself and on whether one is evaluating evidence from a single model or multiple models. We propose a framework for selecting the best available evidence from models: 1) developing de novo, a model specific to the situation of interest, 2) identifying an existing model, the outputs of which provide the highest certainty evidence for the situation of interest, either "off-the-shelf" or after adaptation, and 3) using outputs from multiple models. We also present a summary of preferred terminology to facilitate communication among modeling and health care disciplines. CONCLUSION: This conceptual GRADE approach provides a framework for using evidence from models in health decision-making and the assessment of certainty of evidence from a model or models. The GRADE Working Group and the modeling community are currently developing the detailed methods and related guidance for assessing specific domains determining the certainty of evidence from models across health care-related disciplines (e.g., therapeutic decision-making, toxicology, environmental health, and health economics).
Authors: Howard Balshem; Mark Helfand; Holger J Schünemann; Andrew D Oxman; Regina Kunz; Jan Brozek; Gunn E Vist; Yngve Falck-Ytter; Joerg Meerpohl; Susan Norris; Gordon H Guyatt Journal: J Clin Epidemiol Date: 2011-01-05 Impact factor: 6.437
Authors: Gordon H Guyatt; Andrew D Oxman; Gunn Vist; Regina Kunz; Jan Brozek; Pablo Alonso-Coello; Victor Montori; Elie A Akl; Ben Djulbegovic; Yngve Falck-Ytter; Susan L Norris; John W Williams; David Atkins; Joerg Meerpohl; Holger J Schünemann Journal: J Clin Epidemiol Date: 2011-01-19 Impact factor: 6.437
Authors: Andrew H Briggs; Milton C Weinstein; Elisabeth A L Fenwick; Jonathan Karnon; Mark J Sculpher; A David Paltiel Journal: Value Health Date: 2012 Sep-Oct Impact factor: 5.725
Authors: Monica Hultcrantz; David Rind; Elie A Akl; Shaun Treweek; Reem A Mustafa; Alfonso Iorio; Brian S Alper; Joerg J Meerpohl; M Hassan Murad; Mohammed T Ansari; Srinivasa Vittal Katikireddi; Pernilla Östlund; Sofia Tranæus; Robin Christensen; Gerald Gartlehner; Jan Brozek; Ariel Izcovich; Holger Schünemann; Gordon Guyatt Journal: J Clin Epidemiol Date: 2017-05-18 Impact factor: 6.437
Authors: Yuan Zhang; Pablo Alonso-Coello; Gordon H Guyatt; Juan José Yepes-Nuñez; Elie A Akl; Glen Hazlewood; Hector Pardo-Hernandez; Itziar Etxeandia-Ikobaltzeta; Amir Qaseem; John W Williams; Peter Tugwell; Signe Flottorp; Yaping Chang; Yuqing Zhang; Reem A Mustafa; María Ximena Rojas; Holger J Schünemann Journal: J Clin Epidemiol Date: 2018-02-13 Impact factor: 6.437
Authors: Teegwendé V Porgo; Susan L Norris; Georgia Salanti; Leigh F Johnson; Julie A Simpson; Nicola Low; Matthias Egger; Christian L Althaus Journal: Res Synth Methods Date: 2019-01-08 Impact factor: 5.273
Authors: Jan M Stratil; Renke L Biallas; Jacob Burns; Laura Arnold; Karin Geffert; Angela M Kunzler; Ina Monsef; Julia Stadelmaier; Katharina Wabnitz; Tim Litwin; Clemens Kreutz; Anna Helen Boger; Saskia Lindner; Ben Verboom; Stephan Voss; Ani Movsisyan Journal: Cochrane Database Syst Rev Date: 2021-09-15
Authors: Rob B M De Vries; Michelle Angrish; Patience Browne; Jan Brozek; Andrew A Rooney; Daniele S Wikoff; Paul Whaley; Stephen W Edwards; Rebecca L Morgan; Ingrid L Druwe; Sebastian Hoffmann; Thomas Hartung; Kristina Thayer; Marc T Avey; Brandiese E J Beverly; Maicon Falavigna; Catherine Gibbons; Katy Goyak; Andrew Kraft; Fernando Nampo; Amir Qaseem; Meg Sears; Jasvinder A Singh; Catherine Willett; Erin Y Yost; Holger Schünemann; Katya Tsaioun Journal: ALTEX Date: 2021-04-08 Impact factor: 6.250
Authors: Andrew Anglemyer; Theresa Hm Moore; Lisa Parker; Timothy Chambers; Alice Grady; Kellia Chiu; Matthew Parry; Magdalena Wilczynska; Ella Flemyng; Lisa Bero Journal: Cochrane Database Syst Rev Date: 2020-08-18
Authors: Jacob Burns; Ani Movsisyan; Jan M Stratil; Renke Lars Biallas; Michaela Coenen; Karl Mf Emmert-Fees; Karin Geffert; Sabine Hoffmann; Olaf Horstick; Michael Laxy; Carmen Klinger; Suzie Kratzer; Tim Litwin; Susan Norris; Lisa M Pfadenhauer; Peter von Philipsborn; Kerstin Sell; Julia Stadelmaier; Ben Verboom; Stephan Voss; Katharina Wabnitz; Eva Rehfuess Journal: Cochrane Database Syst Rev Date: 2021-03-25