OBJECTIVE: Optimism bias refers to unwarranted belief in the efficacy of new therapies. We assessed the impact of optimism bias on a proportion of trials that did not answer their research question successfully and explored whether poor accrual or optimism bias is responsible for inconclusive results. STUDY DESIGN: Systematic review. SETTING: Retrospective analysis of a consecutive-series phase III randomized controlled trials (RCTs) performed under the aegis of National Cancer Institute Cooperative groups. RESULTS: Three hundred fifty-nine trials (374 comparisons) enrolling 150,232 patients were analyzed. Seventy percent (262 of 374) of the trials generated conclusive results according to the statistical criteria. Investigators made definitive statements related to the treatment preference in 73% (273 of 374) of studies. Investigators' judgments and statistical inferences were concordant in 75% (279 of 374) of trials. Investigators consistently overestimated their expected treatment effects but to a significantly larger extent for inconclusive trials. The median ratio of expected and observed hazard ratio or odds ratio was 1.34 (range: 0.19-15.40) in conclusive trials compared with 1.86 (range: 1.09-12.00) in inconclusive studies (P<0.0001). Only 17% of the trials had treatment effects that matched original researchers' expectations. CONCLUSION: Formal statistical inference is sufficient to answer the research question in 75% of RCTs. The answers to the other 25% depend mostly on subjective judgments, which at times are in conflict with statistical inference. Optimism bias significantly contributes to inconclusive results.
OBJECTIVE: Optimism bias refers to unwarranted belief in the efficacy of new therapies. We assessed the impact of optimism bias on a proportion of trials that did not answer their research question successfully and explored whether poor accrual or optimism bias is responsible for inconclusive results. STUDY DESIGN: Systematic review. SETTING: Retrospective analysis of a consecutive-series phase III randomized controlled trials (RCTs) performed under the aegis of National Cancer Institute Cooperative groups. RESULTS: Three hundred fifty-nine trials (374 comparisons) enrolling 150,232 patients were analyzed. Seventy percent (262 of 374) of the trials generated conclusive results according to the statistical criteria. Investigators made definitive statements related to the treatment preference in 73% (273 of 374) of studies. Investigators' judgments and statistical inferences were concordant in 75% (279 of 374) of trials. Investigators consistently overestimated their expected treatment effects but to a significantly larger extent for inconclusive trials. The median ratio of expected and observed hazard ratio or odds ratio was 1.34 (range: 0.19-15.40) in conclusive trials compared with 1.86 (range: 1.09-12.00) in inconclusive studies (P<0.0001). Only 17% of the trials had treatment effects that matched original researchers' expectations. CONCLUSION: Formal statistical inference is sufficient to answer the research question in 75% of RCTs. The answers to the other 25% depend mostly on subjective judgments, which at times are in conflict with statistical inference. Optimism bias significantly contributes to inconclusive results.
Authors: R J Prescott; C E Counsell; W J Gillespie; A M Grant; I T Russell; S Kiauka; I R Colthart; S Ross; S M Shepherd; D Russell Journal: Health Technol Assess Date: 1999 Impact factor: 4.014
Authors: B Djulbegovic; M Lacevic; A Cantor; K K Fields; C L Bennett; J R Adams; N M Kuderer; G H Lyman Journal: Lancet Date: 2000-08-19 Impact factor: 79.321
Authors: Christopher M Booth; Aurélie Le Maître; Keyue Ding; Kristen Farn; Michael Fralick; Cameron Phillips; David W Cescon; Ralph M Meyer Journal: J Clin Oncol Date: 2009-07-20 Impact factor: 44.544
Authors: Benjamin Djulbegovic; Ambuj Kumar; Heloisa P Soares; Iztok Hozo; Gerold Bepler; Mike Clarke; Charles L Bennett Journal: Arch Intern Med Date: 2008-03-24
Authors: Daniel M Benjamin; David R Mandel; Tristan Barnes; Monika K Krzyzanowska; Natasha Leighl; Ian F Tannock; Jonathan Kimmelman Journal: Oncologist Date: 2020-08-26
Authors: Kerry Dwan; Douglas G Altman; Lynne Cresswell; Michaela Blundell; Carrol L Gamble; Paula R Williamson Journal: Cochrane Database Syst Rev Date: 2011-01-19