OBJECTIVE: To evaluate the reliability of risk of bias assessments based on published trial reports, for determining trial inclusion in meta-analyses. DESIGN: Reliability evaluation of risk of bias assessments. DATA SOURCES: 13 published individual participant data (IPD) meta-analyses in cancer were used to source 95 randomised controlled trials. REVIEW METHODS: Risk of bias was assessed using the Cochrane risk of bias tool (RevMan5.1) and accompanying guidance. Assessments were made for individual risk of bias domains and overall for each trial, using information from either trial reports alone or trial reports with additional information collected for IPD meta-analyses. Percentage agreements were calculated for individual domains and overall (<66%= low, ≥ 66% = fair, ≥ 90% = good). The two approaches were considered similarly reliable only when agreement was good. RESULTS: Percentage agreement between the two methods for sequence generation and incomplete outcome data was fair (69.5% (95% confidence interval 60.2% to 78.7%) and 80.0% (72.0% to 88.0%), respectively). However, percentage agreement was low for allocation concealment, selective outcome reporting, and overall risk of bias (48.4% (38.4% to 58.5%), 42.1% (32.2% to 52.0%), and 54.7% (44.7% to 64.7%), respectively). Supplementary information reduced the proportion of unclear assessments for all individual domains, consequently increasing the number of trials assessed as low risk of bias (and therefore available for inclusion in meta-analyses) from 23 (23%) based on publications alone to 66 (66%) based on publications with additional information. CONCLUSIONS: Using cancer trial publications alone to assess risk of bias could be unreliable; thus, reviewers should be cautious about using them as a basis for trial inclusion, particularly for those trials assessed as unclear risk. Supplementary information from trialists should be sought to enable appropriate assessments and potentially reduce or overcome some risks of bias. Furthermore, guidance should ensure clarity on what constitutes risk of bias, particularly for the more subjective domains.
OBJECTIVE: To evaluate the reliability of risk of bias assessments based on published trial reports, for determining trial inclusion in meta-analyses. DESIGN: Reliability evaluation of risk of bias assessments. DATA SOURCES: 13 published individual participant data (IPD) meta-analyses in cancer were used to source 95 randomised controlled trials. REVIEW METHODS: Risk of bias was assessed using the Cochrane risk of bias tool (RevMan5.1) and accompanying guidance. Assessments were made for individual risk of bias domains and overall for each trial, using information from either trial reports alone or trial reports with additional information collected for IPD meta-analyses. Percentage agreements were calculated for individual domains and overall (<66%= low, ≥ 66% = fair, ≥ 90% = good). The two approaches were considered similarly reliable only when agreement was good. RESULTS: Percentage agreement between the two methods for sequence generation and incomplete outcome data was fair (69.5% (95% confidence interval 60.2% to 78.7%) and 80.0% (72.0% to 88.0%), respectively). However, percentage agreement was low for allocation concealment, selective outcome reporting, and overall risk of bias (48.4% (38.4% to 58.5%), 42.1% (32.2% to 52.0%), and 54.7% (44.7% to 64.7%), respectively). Supplementary information reduced the proportion of unclear assessments for all individual domains, consequently increasing the number of trials assessed as low risk of bias (and therefore available for inclusion in meta-analyses) from 23 (23%) based on publications alone to 66 (66%) based on publications with additional information. CONCLUSIONS: Using cancer trial publications alone to assess risk of bias could be unreliable; thus, reviewers should be cautious about using them as a basis for trial inclusion, particularly for those trials assessed as unclear risk. Supplementary information from trialists should be sought to enable appropriate assessments and potentially reduce or overcome some risks of bias. Furthermore, guidance should ensure clarity on what constitutes risk of bias, particularly for the more subjective domains.
Authors: Claire L Vale; Sarah Burdett; David J Fisher; Neal Navani; Mahesh K B Parmar; Andrew J Copas; Jayne F Tierney Journal: Clin Lung Cancer Date: 2014-11-22 Impact factor: 4.785
Authors: Matthew J Page; Julian P T Higgins; Gemma Clayton; Jonathan A C Sterne; Asbjørn Hróbjartsson; Jelena Savović Journal: PLoS One Date: 2016-07-11 Impact factor: 3.240
Authors: Sunita Rehal; Tim P Morris; Katherine Fielding; James R Carpenter; Patrick P J Phillips Journal: BMJ Open Date: 2016-10-07 Impact factor: 2.692
Authors: Fariba Ahmadizar; N Charlotte Onland-Moret; Anthonius de Boer; Geoffrey Liu; Anke H Maitland-van der Zee Journal: PLoS One Date: 2015-09-02 Impact factor: 3.240