OBJECTIVES: To assess the reliability of the Cochrane Risk of Bias (ROB) tool between individual raters and across consensus agreements of pairs of reviewers and examine the impact of study-level factors on reliability. STUDY DESIGN AND SETTING: Two reviewers assessed risk of bias for 154 randomized controlled trials (RCTs). For 30 RCTs, two reviewers from each of four centers assessed risk of bias and reached consensus. We assessed interrater agreement using kappas and the impact of study-level factors through subgroup analyses. RESULTS: Reliability between two reviewers was fair for most domains (κ=0.24-0.37), except sequence generation (κ=0.79, substantial). Reliability results across reviewer pairs: sequence generation, moderate (κ=0.60); allocation concealment and "other sources of bias," fair (κ=0.37-0.27); and other domains, slight (κ=0.05-0.09). Reliability was influenced by the nature of the outcome, nature of the intervention, study design, trial hypothesis, and funding source. Variability resulted from different interpretation of the tool rather than different information identified in the study reports. CONCLUSION: Low agreement has implications for interpreting systematic reviews. These findings suggest the need for detailed guidance in assessing the risk of bias.
OBJECTIVES: To assess the reliability of the Cochrane Risk of Bias (ROB) tool between individual raters and across consensus agreements of pairs of reviewers and examine the impact of study-level factors on reliability. STUDY DESIGN AND SETTING: Two reviewers assessed risk of bias for 154 randomized controlled trials (RCTs). For 30 RCTs, two reviewers from each of four centers assessed risk of bias and reached consensus. We assessed interrater agreement using kappas and the impact of study-level factors through subgroup analyses. RESULTS: Reliability between two reviewers was fair for most domains (κ=0.24-0.37), except sequence generation (κ=0.79, substantial). Reliability results across reviewer pairs: sequence generation, moderate (κ=0.60); allocation concealment and "other sources of bias," fair (κ=0.37-0.27); and other domains, slight (κ=0.05-0.09). Reliability was influenced by the nature of the outcome, nature of the intervention, study design, trial hypothesis, and funding source. Variability resulted from different interpretation of the tool rather than different information identified in the study reports. CONCLUSION: Low agreement has implications for interpreting systematic reviews. These findings suggest the need for detailed guidance in assessing the risk of bias.
Authors: Lukas Schwingshackl; Sven Knüppel; Carolina Schwedhelm; Georg Hoffmann; Benjamin Missbach; Marta Stelmach-Mardas; Stefan Dietrich; Fabian Eichelmann; Evangelos Kontopantelis; Khalid Iqbal; Krasimira Aleksandrova; Stefan Lorkowski; Michael F Leitzmann; Anja Kroke; Heiner Boeing Journal: Adv Nutr Date: 2016-11-15 Impact factor: 8.701
Authors: Abdullah Pandor; Daniel Horner; Sarah Davis; Steve Goodacre; John W Stevens; Mark Clowes; Beverley J Hunt; Tim Nokes; Jonathan Keenan; Kerstin de Wit Journal: Health Technol Assess Date: 2019-12 Impact factor: 4.014
Authors: Susan Armijo-Olivo; Jorge Fuentes; Maria Ospina; Humam Saltaji; Lisa Hartling Journal: BMC Med Res Methodol Date: 2013-09-17 Impact factor: 4.615