Jeffrey L Jackson1, Ethan M Balk2, Noorie Hyun3, Akira Kuriyama4. 1. General Medicine Section, Zablocki VAMC, Milwaukee, WI, USA. jjackson@mcw.edu. 2. Center for Evidence Synthesis in Health, Brown University School of Public Health, Providence, RI, USA. 3. Institute of Health and Equity, Center for Advancing Population Science, Medical College of Wisconsin, Milwaukee, WI, USA. 4. Emergency and Critical Care Center, Kurashiki Central Hospital, Okayama, Japan.
Abstract
BACKGROUND: Selective or non-reporting of study outcomes results in outcome reporting bias. OBJECTIVE: We sought to develop and assess tools for detecting and adjusting for outcome reporting bias. DESIGN: Using data from a previously published systematic review, we abstracted whether outcomes were reported as collected, whether outcomes were statistically significant, and whether statistically significant outcomes were more likely to be reported. We proposed and tested a model to adjust for unreported outcomes and compared our model to three other methods (Copas, Frosi, trim and fill). Our approach assumes that unreported outcomes had a null intervention effect with variance imputed based on the published outcomes. We further compared our approach to these models using simulation, and by varying levels of missing data and study sizes. RESULTS: There were 286 outcomes reported as collected from 47 included trials: 142 (48%) had the data provided and 144 (52%) did not. Reported outcomes were more likely to be statistically significant than those collected but for which data were unreported and for which non-significance was reported (RR, 2.4; 95% CI, 1.9 to 3.0). Our model and the Copas model provided similar decreases in the pooled effect sizes in both the meta-analytic data and simulation studies. The Frosi and trim and fill methods performed poorly. LIMITATIONS: Single intervention of a single disease with only randomized controlled trials; approach may overestimate outcome reporting bias impact. CONCLUSION: There was evidence of selective outcome reporting. Statistically significant outcomes were more likely to be published than non-significant ones. Our simple approach provided a quick estimate of the impact of unreported outcomes on the estimated effect. This approach could be used as a quick assessment of the potential impact of unreported outcomes.
BACKGROUND: Selective or non-reporting of study outcomes results in outcome reporting bias. OBJECTIVE: We sought to develop and assess tools for detecting and adjusting for outcome reporting bias. DESIGN: Using data from a previously published systematic review, we abstracted whether outcomes were reported as collected, whether outcomes were statistically significant, and whether statistically significant outcomes were more likely to be reported. We proposed and tested a model to adjust for unreported outcomes and compared our model to three other methods (Copas, Frosi, trim and fill). Our approach assumes that unreported outcomes had a null intervention effect with variance imputed based on the published outcomes. We further compared our approach to these models using simulation, and by varying levels of missing data and study sizes. RESULTS: There were 286 outcomes reported as collected from 47 included trials: 142 (48%) had the data provided and 144 (52%) did not. Reported outcomes were more likely to be statistically significant than those collected but for which data were unreported and for which non-significance was reported (RR, 2.4; 95% CI, 1.9 to 3.0). Our model and the Copas model provided similar decreases in the pooled effect sizes in both the meta-analytic data and simulation studies. The Frosi and trim and fill methods performed poorly. LIMITATIONS: Single intervention of a single disease with only randomized controlled trials; approach may overestimate outcome reporting bias impact. CONCLUSION: There was evidence of selective outcome reporting. Statistically significant outcomes were more likely to be published than non-significant ones. Our simple approach provided a quick estimate of the impact of unreported outcomes on the estimated effect. This approach could be used as a quick assessment of the potential impact of unreported outcomes.