Silvia Minozzi1, Michela Cinquini2, Silvia Gianola3, Marien Gonzalez-Lorenzo2, Rita Banzi4. 1. Department of Epidemiology, Lazio Regional Health Service, Rome, Italy; Laboratory of Clinical Research Methodology, Istituto di Ricerche Farmacologiche Mario Negri IRCCS, Milan, Italy. Electronic address: minozzi.silvia@gmail.com. 2. Laboratory of Clinical Research Methodology, Istituto di Ricerche Farmacologiche Mario Negri IRCCS, Milan, Italy. 3. Unit of Clinical Epidemiology, IRCCS Istituto Ortopedico Galeazzi, Milan, Italy. 4. Center for Health Regulatory Policies, Istituto di Ricerche Farmacologiche Mario Negri IRCCS, Milan, Italy.
Abstract
OBJECTIVE: The objective of the study is to assess the interrater reliability (IRR) and usability of the revised Cochrane risk of bias tool for randomized trials (RoB 2). STUDY DESIGN AND SETTING: This is a cross-sectional study. Four raters independently applied RoB 2 on the primary outcome of a random sample of individually randomized parallel-group trials (randomized controlled trials (RCTs)). We calculated the Fleiss' kappa for multiple raters, the time needed to complete the tool, and discussed the application of RoB 2 to identify difficulties and reasons for disagreement. RESULTS: A total of 70 outcomes from 70 RCTs were included. IRR was slight for overall judgment (IRR 0.16, 95% confidence interval (CI) 0.08-0.24); individual domain analysis gave IRR as moderate for "randomization process" (IRR 0.45, 95% CI 0.37-0.53), slight for "deviations from intended intervention" for RCTs assessing the effect of the assignment to an intervention (IRR 0.04, 95% CI -0.06 to 0.14), fair for those assessing the effect of adhering (IRR 0.21, 95% CI 0.11-0.31), and fair for the other domains, ranging from 0.22 (95% CI 0.14-0.30) for "missing outcome data" to 0.30 (95% CI 0.22-0.38) for "selection of reported results". Mean time to apply the tool was 28 minutes (standard deviation 13.4) per study outcome. The main difficulties were due to poor knowledge of the subject matter of primary studies, new terminology, different approaches for some domains compared with the previous tool, and way of formulating signaling questions. CONCLUSIONS: RoB 2 is a detailed and comprehensive tool but difficult and demanding, even for raters with substantial expertise in systematic reviews. Calibration exercises and intensive training are needed before its application, to improve reliability.
OBJECTIVE: The objective of the study is to assess the interrater reliability (IRR) and usability of the revised Cochrane risk of bias tool for randomized trials (RoB 2). STUDY DESIGN AND SETTING: This is a cross-sectional study. Four raters independently applied RoB 2 on the primary outcome of a random sample of individually randomized parallel-group trials (randomized controlled trials (RCTs)). We calculated the Fleiss' kappa for multiple raters, the time needed to complete the tool, and discussed the application of RoB 2 to identify difficulties and reasons for disagreement. RESULTS: A total of 70 outcomes from 70 RCTs were included. IRR was slight for overall judgment (IRR 0.16, 95% confidence interval (CI) 0.08-0.24); individual domain analysis gave IRR as moderate for "randomization process" (IRR 0.45, 95% CI 0.37-0.53), slight for "deviations from intended intervention" for RCTs assessing the effect of the assignment to an intervention (IRR 0.04, 95% CI -0.06 to 0.14), fair for those assessing the effect of adhering (IRR 0.21, 95% CI 0.11-0.31), and fair for the other domains, ranging from 0.22 (95% CI 0.14-0.30) for "missing outcome data" to 0.30 (95% CI 0.22-0.38) for "selection of reported results". Mean time to apply the tool was 28 minutes (standard deviation 13.4) per study outcome. The main difficulties were due to poor knowledge of the subject matter of primary studies, new terminology, different approaches for some domains compared with the previous tool, and way of formulating signaling questions. CONCLUSIONS: RoB 2 is a detailed and comprehensive tool but difficult and demanding, even for raters with substantial expertise in systematic reviews. Calibration exercises and intensive training are needed before its application, to improve reliability.
Authors: Kim Luijken; Bryan J M van de Wall; Lotty Hooft; Luke P H Leenen; R Marijn Houwert; Rolf H H Groenwold Journal: Eur J Trauma Emerg Surg Date: 2022-07-09 Impact factor: 3.693
Authors: Garrett S Bullock; Timothy C Sell; Ryan Zarega; Charles Reiter; Victoria King; Hailey Wrona; Nilani Mills; Charlotte Ganderton; Steven Duhig; Anu Räisäsen; Leila Ledbetter; Gary S Collins; Joanna Kvist; Stephanie R Filbay Journal: Sports Med Date: 2022-08-13 Impact factor: 11.928
Authors: Montserrat Lara-Velazquez; Jack M Shireman; Eric J Lehrer; Kelsey M Bowman; Henry Ruiz-Garcia; Mitchell J Paukner; Richard J Chappell; Mahua Dey Journal: Front Oncol Date: 2021-05-11 Impact factor: 6.244
Authors: Xiaoqin Wang; Grace Martin; Behnam Sadeghirad; Andrea J Darzi; Rachel J Couban; Ivan D Florez; Holly N Crandon; Elena Kum; Yaping Chang; Meisam Abdar Esfahani; Laxsanaa Sivananthan; Fatemeh Mehrabi; Neil K Sengupta; Preksha Rathod; Rami Z Morsi; D Norman Buckley; Gordon H Guyatt; Y Raja Rampersaud; Christopher J Standaert; Thomas Agoritsas; Jason W Busse Journal: BMJ Open Date: 2021-07-09 Impact factor: 2.692