Silvia Minozzi1, Kerry Dwan2, Francesca Borrelli3, Graziella Filippini4. 1. Department of Epidemiology, Lazio Regional Health Service, Rome, Italy. Electronic address: minozzi.silvia@gmail.com. 2. Department of Cochrane Editorial and Methods, London. 3. Department of Pharmacy, School of Medicine and Surgery, University of Naples Federico II, Naples, Italy. 4. Scientific Director's Office, Carlo Besta Foundation and Neurological Institute, Milan, Italy.
Abstract
OBJECTIVE: to assess the inter-rater reliability (IRR) of the revised Cochrane risk-of-bias tool for randomised trials (RoB2). METHODS: Four raters independently applied RoB2 on critical and important outcomes of individually randomized parallel-group trials (RCTs) included in the Cochrane Review "Cannabis and cannabinoids for people with multiple sclerosis." We calculated Fleiss' Kappa for multiple raters and time to complete the tool; we performed a calibration exercise on five studies, then we developed an implementation document (ID) specific for the condition, and the intervention addressed by the review with instructions on how to answer the signalling questions of RoB2 tool. We measured IRR before and after the ID adoption RESULTS: Eighty results related to seven outcomes from 16 RCTs were assessed. During calibration exercise we reached no agreement for overall judgment (IRR -0.15); IRR for individual domains ranged from no agreement to fair. Mean time to apply the tool was 168.5 minutes per study. Time to complete the calibration exercise and develop the ID was about 40 hours. After the ID adoption ID, overall agreement increased to slightly (IRR 0.11) for the first five studies and moderate (IRR 0.42) for the remaining 11. IRR for individual domains ranged from no agreement to almost perfect. Mean time to apply the tool decreased to 41 minutes. CONCLUSION: RoB2 tool is comprehensive but complex even for high experienced raters. The development of an ID specific for the review may improve reliability substantially.
OBJECTIVE: to assess the inter-rater reliability (IRR) of the revised Cochrane risk-of-bias tool for randomised trials (RoB2). METHODS: Four raters independently applied RoB2 on critical and important outcomes of individually randomized parallel-group trials (RCTs) included in the Cochrane Review "Cannabis and cannabinoids for people with multiple sclerosis." We calculated Fleiss' Kappa for multiple raters and time to complete the tool; we performed a calibration exercise on five studies, then we developed an implementation document (ID) specific for the condition, and the intervention addressed by the review with instructions on how to answer the signalling questions of RoB2 tool. We measured IRR before and after the ID adoption RESULTS: Eighty results related to seven outcomes from 16 RCTs were assessed. During calibration exercise we reached no agreement for overall judgment (IRR -0.15); IRR for individual domains ranged from no agreement to fair. Mean time to apply the tool was 168.5 minutes per study. Time to complete the calibration exercise and develop the ID was about 40 hours. After the ID adoption ID, overall agreement increased to slightly (IRR 0.11) for the first five studies and moderate (IRR 0.42) for the remaining 11. IRR for individual domains ranged from no agreement to almost perfect. Mean time to apply the tool decreased to 41 minutes. CONCLUSION: RoB2 tool is comprehensive but complex even for high experienced raters. The development of an ID specific for the review may improve reliability substantially.
Authors: Samuel E Cooper; Eva A M van Dis; Muriel A Hagenaars; Angelos-Miltiadis Krypotos; Charles B Nemeroff; Shmuel Lissek; Iris M Engelhard; Joseph E Dunsmoor Journal: Neuropsychopharmacology Date: 2022-04-30 Impact factor: 8.294