Levi Waldron1, Benjamin Haibe-Kains1, Aedín C Culhane1, Markus Riester1, Jie Ding1, Xin Victoria Wang1, Mahnaz Ahmadifar1, Svitlana Tyekucheva1, Christoph Bernau1, Thomas Risch1, Benjamin Frederick Ganzfried1, Curtis Huttenhower1, Michael Birrer1, Giovanni Parmigiani2. 1. Affiliations of authors: City University of New York School of Public Health, Hunter College, New York, NY (LW); Department of Biostatistics and Computational Biology, Dana-Farber Cancer Institute, Boston, MA (LW, AC, MR, JD, XVW, ST, TR, BG, GP); Department of Biostatistics, Harvard School of Public Health, Boston, MA (LW, AC, CH, GP); Center for Cancer Research, Massachusetts General Hospital, Boston, MA (MB); Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada (BH); Medical Biophysics Department, University of Toronto, Toronto, Ontario, Canada (BH);Institute for Medical Information Sciences, Biometry, and Epidemiology, LMU Munich, Munich, Germany (CB). 2. Affiliations of authors: City University of New York School of Public Health, Hunter College, New York, NY (LW); Department of Biostatistics and Computational Biology, Dana-Farber Cancer Institute, Boston, MA (LW, AC, MR, JD, XVW, ST, TR, BG, GP); Department of Biostatistics, Harvard School of Public Health, Boston, MA (LW, AC, CH, GP); Center for Cancer Research, Massachusetts General Hospital, Boston, MA (MB); Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada (BH); Medical Biophysics Department, University of Toronto, Toronto, Ontario, Canada (BH);Institute for Medical Information Sciences, Biometry, and Epidemiology, LMU Munich, Munich, Germany (CB). gp@jimmy.harvard.edu.
Abstract
BACKGROUND: Ovarian cancer is the fifth most common cause of cancer deaths in women in the United States. Numerous gene signatures of patient prognosis have been proposed, but diverse data and methods make these difficult to compare or use in a clinically meaningful way. We sought to identify successful published prognostic gene signatures through systematic validation using public data. METHODS: A systematic review identified 14 prognostic models for late-stage ovarian cancer. For each, we evaluated its 1) reimplementation as described by the original study, 2) performance for prognosis of overall survival in independent data, and 3) performance compared with random gene signatures. We compared and ranked models by validation in 10 published datasets comprising 1251 primarily high-grade, late-stage serous ovarian cancer patients. All tests of statistical significance were two-sided. RESULTS: Twelve published models had 95% confidence intervals of the C-index that did not include the null value of 0.5; eight outperformed 97.5% of signatures including the same number of randomly selected genes and trained on the same data. The four top-ranked models achieved overall validation C-indices of 0.56 to 0.60 and shared anticorrelation with expression of immune response pathways. Most models demonstrated lower accuracy in new datasets than in validation sets presented in their publication. CONCLUSIONS: This analysis provides definitive support for a handful of prognostic models but also confirms that these require improvement to be of clinical value. This work addresses outstanding controversies in the ovarian cancer literature and provides a reproducible framework for meta-analytic evaluation of gene signatures.
BACKGROUND: Ovarian cancer is the fifth most common cause of cancer deaths in women in the United States. Numerous gene signatures of patient prognosis have been proposed, but diverse data and methods make these difficult to compare or use in a clinically meaningful way. We sought to identify successful published prognostic gene signatures through systematic validation using public data. METHODS: A systematic review identified 14 prognostic models for late-stage ovarian cancer. For each, we evaluated its 1) reimplementation as described by the original study, 2) performance for prognosis of overall survival in independent data, and 3) performance compared with random gene signatures. We compared and ranked models by validation in 10 published datasets comprising 1251 primarily high-grade, late-stage serous ovarian cancer patients. All tests of statistical significance were two-sided. RESULTS: Twelve published models had 95% confidence intervals of the C-index that did not include the null value of 0.5; eight outperformed 97.5% of signatures including the same number of randomly selected genes and trained on the same data. The four top-ranked models achieved overall validation C-indices of 0.56 to 0.60 and shared anticorrelation with expression of immune response pathways. Most models demonstrated lower accuracy in new datasets than in validation sets presented in their publication. CONCLUSIONS: This analysis provides definitive support for a handful of prognostic models but also confirms that these require improvement to be of clinical value. This work addresses outstanding controversies in the ovarian cancer literature and provides a reproducible framework for meta-analytic evaluation of gene signatures.
Authors: Stephen A Cannistra; Robert C Bast; Jonathan S Berek; Michael A Bookman; Christopher P Crum; Paul D DePriest; Judy E Garber; Wui-Jin Koh; Maurie Markman; William P McGuire; Peter G Rose; Eric K Rowinsky; Gordon J S Rustin; Steven J Skates; Paul A Vasey; Laura King Journal: J Clin Oncol Date: 2003-05-15 Impact factor: 44.544
Authors: John P A Ioannidis; David B Allison; Catherine A Ball; Issa Coulibaly; Xiangqin Cui; Aedín C Culhane; Mario Falchi; Cesare Furlanello; Laurence Game; Giuseppe Jurman; Jon Mangion; Tapan Mehta; Michael Nitzberg; Grier P Page; Enrico Petretto; Vera van Noort Journal: Nat Genet Date: 2008-01-28 Impact factor: 38.330
Authors: R Sabatier; P Finetti; J Bonensea; J Jacquemier; J Adelaide; E Lambaudie; P Viens; D Birnbaum; F Bertucci Journal: Br J Cancer Date: 2011-06-07 Impact factor: 7.640
Authors: Benjamin Frederick Ganzfried; Markus Riester; Benjamin Haibe-Kains; Thomas Risch; Svitlana Tyekucheva; Ina Jazic; Xin Victoria Wang; Mahnaz Ahmadifar; Michael J Birrer; Giovanni Parmigiani; Curtis Huttenhower; Levi Waldron Journal: Database (Oxford) Date: 2013-04-02 Impact factor: 3.451
Authors: Wolfgang Huber; Vincent J Carey; Robert Gentleman; Simon Anders; Marc Carlson; Benilton S Carvalho; Hector Corrada Bravo; Sean Davis; Laurent Gatto; Thomas Girke; Raphael Gottardo; Florian Hahne; Kasper D Hansen; Rafael A Irizarry; Michael Lawrence; Michael I Love; James MacDonald; Valerie Obenchain; Andrzej K Oleś; Hervé Pagès; Alejandro Reyes; Paul Shannon; Gordon K Smyth; Dan Tenenbaum; Levi Waldron; Martin Morgan Journal: Nat Methods Date: 2015-02 Impact factor: 28.547
Authors: Fanny Dao; Brooke A Schlappe; Jill Tseng; Jenny Lester; Alpa M Nick; Susan K Lutgendorf; Scott McMeekin; Robert L Coleman; Kathleen N Moore; Beth Y Karlan; Anil K Sood; Douglas A Levine Journal: Gynecol Oncol Date: 2016-03-11 Impact factor: 5.482
Authors: Denis Bertrand; Kern Rei Chng; Faranak Ghazi Sherbaf; Anja Kiesel; Burton K H Chia; Yee Yen Sia; Sharon K Huang; Dave S B Hoon; Edison T Liu; Axel Hillmer; Niranjan Nagarajan Journal: Nucleic Acids Res Date: 2015-01-08 Impact factor: 16.971
Authors: Gregory M Chen; Lavanya Kannan; Ludwig Geistlinger; Victor Kofia; Zhaleh Safikhani; Deena M A Gendoo; Giovanni Parmigiani; Michael Birrer; Benjamin Haibe-Kains; Levi Waldron Journal: Clin Cancer Res Date: 2018-07-03 Impact factor: 12.531
Authors: Markus Riester; Wei Wei; Levi Waldron; Aedin C Culhane; Lorenzo Trippa; Esther Oliva; Sung-Hoon Kim; Franziska Michor; Curtis Huttenhower; Giovanni Parmigiani; Michael J Birrer Journal: J Natl Cancer Inst Date: 2014-04-03 Impact factor: 13.506