Rita Banzi1, Michela Cinquini2, Marien Gonzalez-Lorenzo3, Valentina Pecoraro4, Matteo Capobussi3, Silvia Minozzi5. 1. IRCCS-Istituto di Ricerche Farmacologiche Mario Negri, Milan, Italy. Electronic address: rita.banzi@marionegri.it. 2. IRCCS-Istituto di Ricerche Farmacologiche Mario Negri, Milan, Italy. 3. Department of Biomedical Sciences for Health, University of Milan, Milan, Italy. 4. Ospedale Civile S. Agostino Estense, Azienda USL of Modena, Modena, Italy. 5. Department of Epidemiology, Lazio Regional Health Service, Rome, Italy.
Abstract
OBJECTIVES: The objective of the study was to assess the inter-rater reliability (IRR) of AMSTAR and ROBIS in judging individual domains and overall methodological quality/risk of bias of systematic reviews, the concurrent validity of the tools, and the time required to apply them. STUDY DESIGN AND SETTING: This is a cross-sectional study. Five raters independently read 31 systematic reviews and applied AMSTAR and ROBIS. Fleiss' k for multiple raters for individual domains and overall methodological quality/risk of bias was calculated. Similar domains assessed by both tools and final scores were matched to explore the concurrent validity, using the Kendall tau correlation. RESULTS: IRR ranged from fair to perfect for AMSTAR and from moderate to substantial for ROBIS. Kappa for overall quality/risk of bias was 0.73 (95% confidence interval [CI] 0.65-0.81) for AMSTAR and 0.64 (95% CI 0.54-0.74) for ROBIS. We judged most of the reviews at intermediate quality with AMSTAR (53%), while judgments were split in high (53%) and low (47%) risk of bias with ROBIS. The correlation between judgments on similar domains ranged from moderate to high, while it was fair on the overall judgment (K = 0.35, 95% CI 0.21-0.49). The mean time to complete ROBIS was about double that for AMSTAR. CONCLUSION: AMSTAR and ROBIS offer similar IRR but differ in their construct and applicability.
OBJECTIVES: The objective of the study was to assess the inter-rater reliability (IRR) of AMSTAR and ROBIS in judging individual domains and overall methodological quality/risk of bias of systematic reviews, the concurrent validity of the tools, and the time required to apply them. STUDY DESIGN AND SETTING: This is a cross-sectional study. Five raters independently read 31 systematic reviews and applied AMSTAR and ROBIS. Fleiss' k for multiple raters for individual domains and overall methodological quality/risk of bias was calculated. Similar domains assessed by both tools and final scores were matched to explore the concurrent validity, using the Kendall tau correlation. RESULTS: IRR ranged from fair to perfect for AMSTAR and from moderate to substantial for ROBIS. Kappa for overall quality/risk of bias was 0.73 (95% confidence interval [CI] 0.65-0.81) for AMSTAR and 0.64 (95% CI 0.54-0.74) for ROBIS. We judged most of the reviews at intermediate quality with AMSTAR (53%), while judgments were split in high (53%) and low (47%) risk of bias with ROBIS. The correlation between judgments on similar domains ranged from moderate to high, while it was fair on the overall judgment (K = 0.35, 95% CI 0.21-0.49). The mean time to complete ROBIS was about double that for AMSTAR. CONCLUSION: AMSTAR and ROBIS offer similar IRR but differ in their construct and applicability.
Authors: Josep M García-Alamino; Manuel López-Cano; Leonard Kroese; Frederik Helgstrand; Filip Muysoms Journal: World J Surg Date: 2019-12 Impact factor: 3.352
Authors: Maya M Jeyaraman; Nameer Al-Yousif; Reid C Robson; Leslie Copstein; Chakrapani Balijepalli; Kimberly Hofer; Mir S Fazeli; Mohammed T Ansari; Andrea C Tricco; Rasheda Rabbani; Ahmed M Abou-Setta Journal: Syst Rev Date: 2020-02-12
Authors: Maya M Jeyaraman; Rasheda Rabbani; Nameer Al-Yousif; Reid C Robson; Leslie Copstein; Jun Xia; Michelle Pollock; Samer Mansour; Mohammed T Ansari; Andrea C Tricco; Ahmed M Abou-Setta Journal: Syst Rev Date: 2020-01-13
Authors: Pascal Djiadeu; Joseph Nguemo; Chantal Mukandoli; Apondi J Odhiambo; David Lightfoot; Lawrence Mbuagbaw; LaRon E Nelson Journal: BMJ Open Date: 2019-01-30 Impact factor: 2.692