Dawid Pieper1, Livia Puljak2, Marien González-Lorenzo3, Silvia Minozzi4. 1. Institute for Research in Operative Medicine, Evidence-based Health Services Research, Faculty of Health, School of Medicine, Witten/Herdecke University, Ostmerheimer Str, Cologne 200 51109, Germany. Electronic address: Dawid.pieper@uni-wh.de. 2. Catholic University of Croatia, Ilica 242, Zagreb 10000, Croatia. 3. Department of Biomedical Sciences, Humanitas University, Milan, Italy; IBD Center, Humanitas Clinical and Research Center, Milan, Italy. 4. Department of Biomedical Sciences for Health, University of Milan, Milan, Italy.
Abstract
OBJECTIVE: To compare A Measurement Tool to Assess Systematic Reviews (AMSTAR 2) with a tool to assess risk of bias in systematic reviews (ROBIS) in terms of validity, reliability, and applicability. STUDY DESIGN AND SETTING: We analyzed 30 systematic reviews (SRs) that included randomized and nonrandomized studies, with Cochrane and non-Cochrane SRs sampled in 1:1 ratio. Four reviewers assessed independently all 30 SRs with AMSTAR 2, followed by ROBIS. We calculated Fleiss' Kappa as a measure of inter-rater reliability (IRR) across 4 raters. RESULTS: The IRR for scoring the overall confidence in the SRs with AMSTAR 2 and the overall domain in ROBIS was fair (AMSTAR 2: κ = 0.30, 95% [confidence interval] CI: 0.17 to 0.43; ROBIS: κ = 0.28, 95% CI: 0.13 to 0.42). AMSTAR 2 confidence in review ratings strongly correlated with the overall domain rating in ROBIS (Spearman rs = 0.84). Mean time for scoring AMSTAR 2 was slightly higher than for ROBIS (18 vs. 16 min), with huge differences between the reviewers. CONCLUSION: Both AMSTAR 2 and ROBIS can be applied to SRs including both randomized controlled trials (RCTs) and non-RCTs. Measurement properties of ROBIS seemed not to be much different when comparing with other studies that include only SRs of RCTs.
OBJECTIVE: To compare A Measurement Tool to Assess Systematic Reviews (AMSTAR 2) with a tool to assess risk of bias in systematic reviews (ROBIS) in terms of validity, reliability, and applicability. STUDY DESIGN AND SETTING: We analyzed 30 systematic reviews (SRs) that included randomized and nonrandomized studies, with Cochrane and non-Cochrane SRs sampled in 1:1 ratio. Four reviewers assessed independently all 30 SRs with AMSTAR 2, followed by ROBIS. We calculated Fleiss' Kappa as a measure of inter-rater reliability (IRR) across 4 raters. RESULTS: The IRR for scoring the overall confidence in the SRs with AMSTAR 2 and the overall domain in ROBIS was fair (AMSTAR 2: κ = 0.30, 95% [confidence interval] CI: 0.17 to 0.43; ROBIS: κ = 0.28, 95% CI: 0.13 to 0.42). AMSTAR 2 confidence in review ratings strongly correlated with the overall domain rating in ROBIS (Spearman rs = 0.84). Mean time for scoring AMSTAR 2 was slightly higher than for ROBIS (18 vs. 16 min), with huge differences between the reviewers. CONCLUSION: Both AMSTAR 2 and ROBIS can be applied to SRs including both randomized controlled trials (RCTs) and non-RCTs. Measurement properties of ROBIS seemed not to be much different when comparing with other studies that include only SRs of RCTs.