Patrick Vavken1, Ronald Dorotka. 1. Department of Orthopedic Surgery, Children's Hospital Boston, Harvard Medical School, 300 Longwood Avenue, Enders 1016, Boston, MA 02115, USA. Patrick.vavken@childrens.harvard.edu
Abstract
BACKGROUND: Meta-analyses of randomized, controlled trials are considered the highest level-of-evidence, thus strongest source of information. However, questions concerning the validity of meta-analyses in orthopaedic surgery emerged recently. Among the most common sources for errors is publication bias. This describes the fact that studies with small or non-significant outcomes are less likely to be published, thus less likely to be identified and included in systematic reviews and meta-analyses. In this study we asked three questions: (1) Whether publication bias is assessed in orthopaedic meta-analyses, (2) What the actual prevalence of publication bias is, and (3) what effect publication bias has on the outcomes of orthopaedic meta-analyses. METHODS: Using the estimate of 35 ± 20% for the prevalence of publication bias in meta-analyses obtained from earlier research we calculated a required samples size of 22 (plus 20% to account for attrition) and randomly selected 26 orthopaedic meta-analyses. To answer our first question we calculated the percentage of papers that report on formal assessment of publication bias. For our second question we obtained all primary studies (n = 321) from the included meta-analyses and used Egger's regression to search for evidence for publication bias. Third, we used the trim-and-fill method to assess the impact of publication bias, if present. This method estimates publication bias in a meta-analysis and adds hypothetical studies to reduce this bias; it thus produces an estimate of adjusted, unbiased outcomes that can be compared with the unadjusted, publication-biased outcomes to assess the effect of publication bias. RESULTS: We found that only 35% (95% CI 20-57) of all orthopaedic meta-analyses published between 1992 and 2008 in English and German assessed publication bias. Most studies used funnel plots, which are rather insensitive. The prevalence of publication bias, based on a sensitivity analysis, ranged between 12 and 19%. Adjustment for publication bias did not produce significantly different results, but the magnitude of the pooled estimates in the affected meta-analyses changed by 29% (95% CI 0-63) on average. CONCLUSION: We found a rather low prevalence of publication bias in orthopaedic meta-analyses, but recommend assessing for it and its effects, which might be substantial.
BACKGROUND: Meta-analyses of randomized, controlled trials are considered the highest level-of-evidence, thus strongest source of information. However, questions concerning the validity of meta-analyses in orthopaedic surgery emerged recently. Among the most common sources for errors is publication bias. This describes the fact that studies with small or non-significant outcomes are less likely to be published, thus less likely to be identified and included in systematic reviews and meta-analyses. In this study we asked three questions: (1) Whether publication bias is assessed in orthopaedic meta-analyses, (2) What the actual prevalence of publication bias is, and (3) what effect publication bias has on the outcomes of orthopaedic meta-analyses. METHODS: Using the estimate of 35 ± 20% for the prevalence of publication bias in meta-analyses obtained from earlier research we calculated a required samples size of 22 (plus 20% to account for attrition) and randomly selected 26 orthopaedic meta-analyses. To answer our first question we calculated the percentage of papers that report on formal assessment of publication bias. For our second question we obtained all primary studies (n = 321) from the included meta-analyses and used Egger's regression to search for evidence for publication bias. Third, we used the trim-and-fill method to assess the impact of publication bias, if present. This method estimates publication bias in a meta-analysis and adds hypothetical studies to reduce this bias; it thus produces an estimate of adjusted, unbiased outcomes that can be compared with the unadjusted, publication-biased outcomes to assess the effect of publication bias. RESULTS: We found that only 35% (95% CI 20-57) of all orthopaedic meta-analyses published between 1992 and 2008 in English and German assessed publication bias. Most studies used funnel plots, which are rather insensitive. The prevalence of publication bias, based on a sensitivity analysis, ranged between 12 and 19%. Adjustment for publication bias did not produce significantly different results, but the magnitude of the pooled estimates in the affected meta-analyses changed by 29% (95% CI 0-63) on average. CONCLUSION: We found a rather low prevalence of publication bias in orthopaedic meta-analyses, but recommend assessing for it and its effects, which might be substantial.