Stefan Schandelmaier1, Yaping Chang2, Niveditha Devasenapathy3, Tahira Devji2, Joey S W Kwong4, Luis E Colunga Lozano2, Yung Lee5, Arnav Agarwal6, Neera Bhatnagar2, Hannah Ewald7, Ying Zhang8, Xin Sun9, Lehana Thabane10, Michael Walsh11, Matthias Briel12, Gordon H Guyatt11. 1. Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University of Basel and University Hospital Basel, Spitalstrasse 12, 4056 Basel, Switzerland. Electronic address: s.schandelmaier@gmail.com. 2. Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada. 3. Indian Institute of Public Health-Delhi, Public Health Foundation of India, Plot 47, Sector 44, Institutional Area, Gurgaon, 122002 Haryana, India. 4. JC School of Public Health and Primary Care, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong. 5. Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Michael G. DeGroote School of Medicine, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada. 6. Department of Medicine, University of Toronto, 190 Elizabeth Street, R. Fraser Elliott Building, 3-805, Toronto, Ontario M5G 2C4, Canada. 7. Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University of Basel and University Hospital Basel, Spitalstrasse 12, 4056 Basel, Switzerland. 8. Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Center for Evidence-based Chinese Medicine, Beijing University of Chinese Medicine, 11 Bei San Huan Dong Lu, Chaoyang, Beijing 100029, China. 9. Chinese Evidence-Based Medicine Center, West China Hospital, Sichuan University, Chengdu 610041, China. 10. Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Biostatistics Unit, St Joseph's Healthcare - Hamilton, 50 Charlton Street East, Hamilton, Ontario L8N 4A6, Canada. 11. Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Medicine, McMaster University, 1200 Main Street West, Hamilton, Ontario L8S 4L8, Canada. 12. Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University of Basel and University Hospital Basel, Spitalstrasse 12, 4056 Basel, Switzerland.
Abstract
OBJECTIVE: The objective of the study was to systematically survey the methodological literature and collect suggested criteria for assessing the credibility of effect modification and associated rationales. STUDY DESIGN AND SETTING: We searched MEDLINE, Embase, and WorldCat up to March 2018 for publications providing guidance for assessing the credibility of effect modification identified in randomized trials or meta-analyses. Teams of two investigators independently identified eligible publications and extracted credibility criteria and authors' rationale, reaching consensus through discussion. We created a taxonomy of criteria that we iteratively refined during data abstraction. RESULTS: We identified 150 eligible publications that provided 36 criteria and associated rationales. Frequent criteria included significant test for interaction (n = 54), a priori hypothesis (n = 49), providing a causal explanation (n = 47), accounting for multiplicity (n = 45), testing a small number of effect modifiers (n = 38), and prespecification of analytic details (n = 39). For some criteria, we found more than one rationale; some criteria were connected through a common rationale. For some criteria, experts disagreed regarding their suitability (e.g., added value of stratified randomization; trustworthiness of biologic rationales). CONCLUSION: Methodologists have expended substantial intellectual energy providing criteria for critical appraisal of apparent effect modification. Our survey highlights popular criteria, expert agreement and disagreement, and where more work is needed, including testing criteria in practice.
OBJECTIVE: The objective of the study was to systematically survey the methodological literature and collect suggested criteria for assessing the credibility of effect modification and associated rationales. STUDY DESIGN AND SETTING: We searched MEDLINE, Embase, and WorldCat up to March 2018 for publications providing guidance for assessing the credibility of effect modification identified in randomized trials or meta-analyses. Teams of two investigators independently identified eligible publications and extracted credibility criteria and authors' rationale, reaching consensus through discussion. We created a taxonomy of criteria that we iteratively refined during data abstraction. RESULTS: We identified 150 eligible publications that provided 36 criteria and associated rationales. Frequent criteria included significant test for interaction (n = 54), a priori hypothesis (n = 49), providing a causal explanation (n = 47), accounting for multiplicity (n = 45), testing a small number of effect modifiers (n = 38), and prespecification of analytic details (n = 39). For some criteria, we found more than one rationale; some criteria were connected through a common rationale. For some criteria, experts disagreed regarding their suitability (e.g., added value of stratified randomization; trustworthiness of biologic rationales). CONCLUSION: Methodologists have expended substantial intellectual energy providing criteria for critical appraisal of apparent effect modification. Our survey highlights popular criteria, expert agreement and disagreement, and where more work is needed, including testing criteria in practice.
Keywords:
Clinical trials as topic (MeSH); Epidemiologic methods (MeSH); Health care evaluation mechanisms (MeSH); Meta-analysis as topic (MeSH); Precision medicine (MeSH); Subgroup analysis
Authors: Stefan Schandelmaier; Matthias Briel; Ravi Varadhan; Christopher H Schmid; Niveditha Devasenapathy; Rodney A Hayward; Joel Gagnier; Michael Borenstein; Geert J M G van der Heijden; Issa J Dahabreh; Xin Sun; Willi Sauerbrei; Michael Walsh; John P A Ioannidis; Lehana Thabane; Gordon H Guyatt Journal: CMAJ Date: 2020-08-10 Impact factor: 8.262
Authors: Andreas D Meid; Carmen Ruff; Lucas Wirbka; Felicitas Stoll; Hanna M Seidling; Andreas Groll; Walter E Haefeli Journal: Clin Epidemiol Date: 2020-11-02 Impact factor: 4.790
Authors: Stefan Schandelmaier; Andreas M Schmitt; Amanda K Herbrand; Dominik Glinz; Hannah Ewald; Matthias Briel; Gordon H Guyatt; Lars G Hemkens; Benjamin Kasenda Journal: BMJ Open Date: 2020-05-30 Impact factor: 2.692