Michelle Gates1, Allison Gates1, Gonçalo Duarte2, Maria Cary3, Monika Becker4, Barbara Prediger4, Ben Vandermeer1, Ricardo M Fernandes5, Dawid Pieper4, Lisa Hartling6. 1. Department of Pediatrics, Alberta Research Centre for Health Evidence, University of Alberta, Edmonton, Alberta, Canada. 2. Clinical Pharmacology Unit, Instituto de Medicina Molecular, University of Lisbon, Lisbon, Portugal. 3. Centre for Health Evaluation & Research (CEFAR), National Association of Pharmacies, Lisbon, Portugal. 4. Department für Humanmedizin, Institut für Forschung in der Operativen Medizin, Universität Witten/Herdecke, Witten, Germany. 5. Clinical Pharmacology Unit, Instituto de Medicina Molecular, University of Lisbon, Lisbon, Portugal; Department of Pediatrics, Santa Maria Hospital, Lisbon, Portugal. 6. Department of Pediatrics, Alberta Research Centre for Health Evidence, University of Alberta, Edmonton, Alberta, Canada. Electronic address: hartling@ualberta.ca.
Abstract
OBJECTIVE: The objective of the study was to evaluate the inter-rater and intercenter reliability, usability, and utility of A MeaSurement Tool to Assess systematic Reviews (AMSTAR), AMSTAR 2, and Risk Of Bias In Systematic reviews (ROBIS). STUDY DESIGN AND SETTING: This is a prospective evaluation using 30 systematic reviews of randomized trials, undertaken at three international centers. RESULTS: Reviewers completed AMSTAR, AMSTAR 2, and ROBIS in median (interquartile range) 15.7 (11.3), 19.7 (12.1), and 28.7 (17.4) minutes and reached consensus in 2.6 (3.2), 4.6 (5.3), and 10.9 (10.8) minutes, respectively. Across all centers, inter-rater reliability was substantial to almost perfect for 8/11 AMSTAR, 9/16 AMSTAR 2, and 12/24 ROBIS items. Intercenter reliability was substantial to almost perfect for 6/11 AMSTAR, 12/16 AMSTAR 2, and 7/24 ROBIS items. Intercenter reliability for confidence in the results of the review or overall risk of bias was moderate (Gwet's first-order agreement coefficient (AC1) 0.58, 95% confidence intervals [CI]: 0.30 to 0.85) to substantial (AC1 0.74, 95% CI: 0.30 to 0.85) for AMSTAR 2 and poor (AC1 -0.21, 95% CI: -0.55 to 0.13) to moderate (AC1 0.56, 95% CI: 0.30 to 0.83) for ROBIS. It is not clear whether using the appraisals of any tool as an inclusion criterion would alter an overview's findings. CONCLUSIONS: Improved guidance may be needed to facilitate the consistent interpretation and application of the newer tools (especially ROBIS).
OBJECTIVE: The objective of the study was to evaluate the inter-rater and intercenter reliability, usability, and utility of A MeaSurement Tool to Assess systematic Reviews (AMSTAR), AMSTAR 2, and Risk Of Bias In Systematic reviews (ROBIS). STUDY DESIGN AND SETTING: This is a prospective evaluation using 30 systematic reviews of randomized trials, undertaken at three international centers. RESULTS: Reviewers completed AMSTAR, AMSTAR 2, and ROBIS in median (interquartile range) 15.7 (11.3), 19.7 (12.1), and 28.7 (17.4) minutes and reached consensus in 2.6 (3.2), 4.6 (5.3), and 10.9 (10.8) minutes, respectively. Across all centers, inter-rater reliability was substantial to almost perfect for 8/11 AMSTAR, 9/16 AMSTAR 2, and 12/24 ROBIS items. Intercenter reliability was substantial to almost perfect for 6/11 AMSTAR, 12/16 AMSTAR 2, and 7/24 ROBIS items. Intercenter reliability for confidence in the results of the review or overall risk of bias was moderate (Gwet's first-order agreement coefficient (AC1) 0.58, 95% confidence intervals [CI]: 0.30 to 0.85) to substantial (AC1 0.74, 95% CI: 0.30 to 0.85) for AMSTAR 2 and poor (AC1 -0.21, 95% CI: -0.55 to 0.13) to moderate (AC1 0.56, 95% CI: 0.30 to 0.83) for ROBIS. It is not clear whether using the appraisals of any tool as an inclusion criterion would alter an overview's findings. CONCLUSIONS: Improved guidance may be needed to facilitate the consistent interpretation and application of the newer tools (especially ROBIS).
Authors: Brenda S Bauer; Amaya Azcoaga-Lorenzo; Utkarsh Agrawal; Adeniyi Francis Fagbamigbe; Colin McCowan Journal: PLoS One Date: 2022-05-19 Impact factor: 3.752
Authors: Natalie C Momen; Kai N Streicher; Denise T C da Silva; Alexis Descatha; Monique H W Frings-Dresen; Diana Gagliardi; Lode Godderis; Tom Loney; Daniele Mandrioli; Alberto Modenese; Rebecca L Morgan; Daniela Pachito; Paul T J Scheepers; Daria Sgargi; Marília Silva Paulo; Vivi Schlünssen; Grace Sembajwe; Kathrine Sørensen; Liliane R Teixeira; Thomas Tenkate; Frank Pega Journal: Environ Int Date: 2021-11-30 Impact factor: 9.621
Authors: Dawid Pieper; Irma Hellbrecht; Linlu Zhao; Clemens Baur; Georgia Pick; Sarah Schneider; Thomas Harder; Kelsey Young; Andrea C Tricco; Ella Westhaver; Matthew Tunis Journal: Syst Rev Date: 2022-08-22