Joseph M Unger1, William E Barlow1, Scott D Ramsey2, Michael LeBlanc1, Charles D Blanke3, Dawn L Hershman4. 1. SWOG Statistical Center, Fred Hutchinson Cancer Research Center, Seattle, Washington. 2. Fred Hutchinson Cancer Research Center, Seattle, Washington. 3. SWOG Group Chair's Office/Knight Cancer Institute, Oregon Health and Science University, Portland. 4. Columbia University, New York, New York.
Abstract
IMPORTANCE: Positive phase 3 cancer clinical trials are widely hailed, while trials with negative results are often interpreted as scientific failures. We hypothesized that these interpretations would be reflected in the scientific literature. OBJECTIVE: To compare the scientific impact of positive vs negative phase 3 cancer clinical treatment trials. DESIGN, SETTING, AND PARTICIPANTS: We examined the phase 3 trial history of SWOG, a national cancer clinical trials consortium, over a 30-year period (1985-2014). Scientific impact was assessed according to multiple publication and citation outcomes. Citation data were obtained using Google Scholar. Citation counts were compared using generalized estimating equations for Poisson regression. Any trial that was formally evaluated for the randomized treatment comparison was included for analysis of publication and citation outcomes. Trials were categorized as positive if they achieved a statistically significant result in favor of the new experimental treatment for the protocol-specified primary end point. Trials were categorized as negative if they achieved a statistically significant result in favor of standard therapy or a null result with no statistically significant benefit for either the experimental or standard therapy. MAIN OUTCOMES AND MEASURES: Impact factors for the journals publishing the primary trial results, and the number of citations for the primary trial articles and all secondary articles associated with the trials. RESULTS: Ninety-four studies enrolling n = 46 424 patients were analyzed. Twenty-eight percent of trials were positive (26 of 94). The primary publications from positive trials were published in journals with higher mean (SD) 2-year impact factors (28 [19] vs 18 [13]; P = .007) and were cited twice as often as negative trials (mean per year, 43 vs 21; relative risk, 2.0; 95% CI, 1.1-3.9; P = .03). However, the number of citations from all primary and secondary articles did not significantly differ between positive and negative trials (mean per year, 55 vs 45; relative risk, 1.2; 95% CI, 0.7-2.3; P = .53). CONCLUSIONS AND RELEVANCE: The scientific impact of the primary articles from positive phase 3 randomized cancer clinical trials was twice as great as for negative trials. But when all of the articles associated with the trials were considered, the scientific impact between positive and negative trials was similar. Positive trials indicate clinical advances, but negative trials also have a sizeable scientific impact by generating important scientific observations and new hypotheses and by showing what new treatments should not be used.
IMPORTANCE: Positive phase 3 cancer clinical trials are widely hailed, while trials with negative results are often interpreted as scientific failures. We hypothesized that these interpretations would be reflected in the scientific literature. OBJECTIVE: To compare the scientific impact of positive vs negative phase 3 cancer clinical treatment trials. DESIGN, SETTING, AND PARTICIPANTS: We examined the phase 3 trial history of SWOG, a national cancer clinical trials consortium, over a 30-year period (1985-2014). Scientific impact was assessed according to multiple publication and citation outcomes. Citation data were obtained using Google Scholar. Citation counts were compared using generalized estimating equations for Poisson regression. Any trial that was formally evaluated for the randomized treatment comparison was included for analysis of publication and citation outcomes. Trials were categorized as positive if they achieved a statistically significant result in favor of the new experimental treatment for the protocol-specified primary end point. Trials were categorized as negative if they achieved a statistically significant result in favor of standard therapy or a null result with no statistically significant benefit for either the experimental or standard therapy. MAIN OUTCOMES AND MEASURES: Impact factors for the journals publishing the primary trial results, and the number of citations for the primary trial articles and all secondary articles associated with the trials. RESULTS: Ninety-four studies enrolling n = 46 424 patients were analyzed. Twenty-eight percent of trials were positive (26 of 94). The primary publications from positive trials were published in journals with higher mean (SD) 2-year impact factors (28 [19] vs 18 [13]; P = .007) and were cited twice as often as negative trials (mean per year, 43 vs 21; relative risk, 2.0; 95% CI, 1.1-3.9; P = .03). However, the number of citations from all primary and secondary articles did not significantly differ between positive and negative trials (mean per year, 55 vs 45; relative risk, 1.2; 95% CI, 0.7-2.3; P = .53). CONCLUSIONS AND RELEVANCE: The scientific impact of the primary articles from positive phase 3 randomized cancer clinical trials was twice as great as for negative trials. But when all of the articles associated with the trials were considered, the scientific impact between positive and negative trials was similar. Positive trials indicate clinical advances, but negative trials also have a sizeable scientific impact by generating important scientific observations and new hypotheses and by showing what new treatments should not be used.
Authors: Erin E Kent; Sandra A Mitchell; Kathleen M Castro; Darren A DeWalt; Arnold D Kaluzny; Judith A Hautala; Oren Grad; Rachel M Ballard; Worta J McCaskill-Stevens; Barnett S Kramer; Steven B Clauser Journal: J Clin Oncol Date: 2015-07-20 Impact factor: 44.544
Authors: Anna Spreafico; Aaron R Hansen; Albiruni R Abdul Razak; Philippe L Bedard; Lillian L Siu Journal: Cancer Discov Date: 2021-04 Impact factor: 38.272
Authors: Yoon-Dong Park; Wei Sun; Antonio Salas; Avan Antia; Cindy Carvajal; Amy Wang; Xin Xu; Zhaojin Meng; Ming Zhou; Gregory J Tawa; Jean Dehdashti; Wei Zheng; Christina M Henderson; Adrian M Zelazny; Peter R Williamson Journal: MBio Date: 2016-08-02 Impact factor: 7.867
Authors: Joseph M Unger; Riha Vaidya; Dawn L Hershman; Lori M Minasian; Mark E Fleury Journal: J Natl Cancer Inst Date: 2019-03-01 Impact factor: 13.506
Authors: Joseph M Unger; Charles D Blanke; Michael LeBlanc; William E Barlow; Riha Vaidya; Scott D Ramsey; Dawn L Hershman Journal: JAMA Netw Open Date: 2020-04-01
Authors: Emmanuelle Kempf; Jennifer A de Beyer; Jonathan Cook; Jane Holmes; Seid Mohammed; Tri-Long Nguyên; Iveta Simera; Marialena Trivella; Douglas G Altman; Sally Hopewell; Karel G M Moons; Raphael Porcher; Johannes B Reitsma; Willi Sauerbrei; Gary S Collins Journal: Br J Cancer Date: 2018-10-24 Impact factor: 7.640