Vahid Montazerhodjat1,2, Shomesh E Chaudhuri1,3, Daniel J Sargent4, Andrew W Lo1,3,5,6. 1. Laboratory for Financial Engineering, MIT Sloan School of Management, Cambridge, Massachusetts. 2. Department of Computer Science, Boston College, Chestnut Hill, Massachusetts. 3. Department of Electrical Engineering and Computer Science, MIT, Cambridge, Massachusetts. 4. Mayo Clinic, Rochester, Minnesota. 5. Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, Massachusetts. 6. AlphaSimplex Group LLC, Cambridge, Massachusetts.
Abstract
IMPORTANCE: Randomized clinical trials (RCTs) currently apply the same statistical threshold of alpha = 2.5% for controlling for false-positive results or type 1 error, regardless of the burden of disease or patient preferences. Is there an objective and systematic framework for designing RCTs that incorporates these considerations on a case-by-case basis? OBJECTIVE: To apply Bayesian decision analysis (BDA) to cancer therapeutics to choose an alpha and sample size that minimize the potential harm to current and future patients under both null and alternative hypotheses. DATA SOURCES: We used the National Cancer Institute (NCI) Surveillance, Epidemiology, and End Results (SEER) database and data from the 10 clinical trials of the Alliance for Clinical Trials in Oncology. STUDY SELECTION: The NCI SEER database was used because it is the most comprehensive cancer database in the United States. The Alliance trial data was used owing to the quality and breadth of data, and because of the expertise in these trials of one of us (D.J.S.). DATA EXTRACTION AND SYNTHESIS: The NCI SEER and Alliance data have already been thoroughly vetted. Computations were replicated independently by 2 coauthors and reviewed by all coauthors. MAIN OUTCOMES AND MEASURES: Our prior hypothesis was that an alpha of 2.5% would not minimize the overall expected harm to current and future patients for the most deadly cancers, and that a less conservative alpha may be necessary. Our primary study outcomes involve measuring the potential harm to patients under both null and alternative hypotheses using NCI and Alliance data, and then computing BDA-optimal type 1 error rates and sample sizes for oncology RCTs. RESULTS: We computed BDA-optimal parameters for the 23 most common cancer sites using NCI data, and for the 10 Alliance clinical trials. For RCTs involving therapies for cancers with short survival times, no existing treatments, and low prevalence, the BDA-optimal type 1 error rates were much higher than the traditional 2.5%. For cancers with longer survival times, existing treatments, and high prevalence, the corresponding BDA-optimal error rates were much lower, in some cases even lower than 2.5%. CONCLUSIONS AND RELEVANCE: Bayesian decision analysis is a systematic, objective, transparent, and repeatable process for deciding the outcomes of RCTs that explicitly incorporates burden of disease and patient preferences.
IMPORTANCE: Randomized clinical trials (RCTs) currently apply the same statistical threshold of alpha = 2.5% for controlling for false-positive results or type 1 error, regardless of the burden of disease or patient preferences. Is there an objective and systematic framework for designing RCTs that incorporates these considerations on a case-by-case basis? OBJECTIVE: To apply Bayesian decision analysis (BDA) to cancer therapeutics to choose an alpha and sample size that minimize the potential harm to current and future patients under both null and alternative hypotheses. DATA SOURCES: We used the National Cancer Institute (NCI) Surveillance, Epidemiology, and End Results (SEER) database and data from the 10 clinical trials of the Alliance for Clinical Trials in Oncology. STUDY SELECTION: The NCI SEER database was used because it is the most comprehensive cancer database in the United States. The Alliance trial data was used owing to the quality and breadth of data, and because of the expertise in these trials of one of us (D.J.S.). DATA EXTRACTION AND SYNTHESIS: The NCI SEER and Alliance data have already been thoroughly vetted. Computations were replicated independently by 2 coauthors and reviewed by all coauthors. MAIN OUTCOMES AND MEASURES: Our prior hypothesis was that an alpha of 2.5% would not minimize the overall expected harm to current and future patients for the most deadly cancers, and that a less conservative alpha may be necessary. Our primary study outcomes involve measuring the potential harm to patients under both null and alternative hypotheses using NCI and Alliance data, and then computing BDA-optimal type 1 error rates and sample sizes for oncology RCTs. RESULTS: We computed BDA-optimal parameters for the 23 most common cancer sites using NCI data, and for the 10 Alliance clinical trials. For RCTs involving therapies for cancers with short survival times, no existing treatments, and low prevalence, the BDA-optimal type 1 error rates were much higher than the traditional 2.5%. For cancers with longer survival times, existing treatments, and high prevalence, the corresponding BDA-optimal error rates were much lower, in some cases even lower than 2.5%. CONCLUSIONS AND RELEVANCE: Bayesian decision analysis is a systematic, objective, transparent, and repeatable process for deciding the outcomes of RCTs that explicitly incorporates burden of disease and patient preferences.
Authors: Steven R Alberts; Daniel J Sargent; Suresh Nair; Michelle R Mahoney; Margaret Mooney; Stephen N Thibodeau; Thomas C Smyrk; Frank A Sinicrope; Emily Chan; Sharlene Gill; Morton S Kahlenberg; Anthony F Shields; James T Quesenberry; Thomas A Webb; Gist H Farr; Barbara A Pockaj; Axel Grothey; Richard M Goldberg Journal: JAMA Date: 2012-04-04 Impact factor: 56.272
Authors: Benjamin Djulbegovic; Ambuj Kumar; Heloisa P Soares; Iztok Hozo; Gerold Bepler; Mike Clarke; Charles L Bennett Journal: Arch Intern Med Date: 2008-03-24
Authors: Christopher J L Murray; Charles Atkinson; Kavi Bhalla; Gretchen Birbeck; Roy Burstein; David Chou; Robert Dellavalle; Goodarz Danaei; Majid Ezzati; A Fahimi; D Flaxman; Sherine Gabriel; Emmanuela Gakidou; Nicholas Kassebaum; Shahab Khatibzadeh; Stephen Lim; Steven E Lipshultz; Stephanie London; Michael F MacIntyre; A H Mokdad; A Moran; Andrew E Moran; Dariush Mozaffarian; Tasha Murphy; Moshen Naghavi; C Pope; Thomas Roberts; Joshua Salomon; David C Schwebel; Saeid Shahraz; David A Sleet; Jerry Abraham; Mohammed K Ali; Charles Atkinson; David H Bartels; Kavi Bhalla; Gretchen Birbeck; Roy Burstein; Honglei Chen; Michael H Criqui; Eric L Ding; E Ray Dorsey; Beth E Ebel; Majid Ezzati; S Flaxman; A D Flaxman; Diego Gonzalez-Medina; Bridget Grant; Holly Hagan; Howard Hoffman; Nicholas Kassebaum; Shahab Khatibzadeh; Janet L Leasher; John Lin; Steven E Lipshultz; Rafael Lozano; Yuan Lu; Leslie Mallinger; Mary M McDermott; Renata Micha; Ted R Miller; A A Mokdad; A H Mokdad; Dariush Mozaffarian; Mohsen Naghavi; K M Venkat Narayan; Saad B Omer; Pamela M Pelizzari; David Phillips; Dharani Ranganathan; Frederick P Rivara; Thomas Roberts; Uchechukwu Sampson; Ella Sanman; Amir Sapkota; David C Schwebel; Saeid Sharaz; Rupak Shivakoti; Gitanjali M Singh; David Singh; Mohammad Tavakkoli; Jeffrey A Towbin; James D Wilkinson; Azadeh Zabetian; Jerry Abraham; Mohammad K Ali; Miriam Alvardo; Charles Atkinson; Larry M Baddour; Emelia J Benjamin; Kavi Bhalla; Gretchen Birbeck; Ian Bolliger; Roy Burstein; Emily Carnahan; David Chou; Sumeet S Chugh; Aaron Cohen; K Ellicott Colson; Leslie T Cooper; William Couser; Michael H Criqui; Kaustubh C Dabhadkar; Robert P Dellavalle; Daniel Dicker; E Ray Dorsey; Herbert Duber; Beth E Ebel; Rebecca E Engell; Majid Ezzati; David T Felson; Mariel M Finucane; Seth Flaxman; A D Flaxman; Thomas Fleming; Mohammad H Forouzanfar; Greg Freedman; Michael K Freeman; Emmanuela Gakidou; Richard F Gillum; Diego Gonzalez-Medina; Richard Gosselin; Hialy R Gutierrez; Holly Hagan; Rasmus Havmoeller; Howard Hoffman; Kathryn H Jacobsen; Spencer L James; Rashmi Jasrasaria; Sudha Jayarman; Nicole Johns; Nicholas Kassebaum; Shahab Khatibzadeh; Qing Lan; Janet L Leasher; Stephen Lim; Steven E Lipshultz; Stephanie London; Rafael Lozano; Yuan Lu; Leslie Mallinger; Michele Meltzer; George A Mensah; Catherine Michaud; Ted R Miller; Charles Mock; Terrie E Moffitt; A A Mokdad; A H Mokdad; A Moran; Mohsen Naghavi; K M Venkat Narayan; Robert G Nelson; Casey Olives; Saad B Omer; Katrina Ortblad; Bart Ostro; Pamela M Pelizzari; David Phillips; Murugesan Raju; Homie Razavi; Beate Ritz; Thomas Roberts; Ralph L Sacco; Joshua Salomon; Uchechukwu Sampson; David C Schwebel; Saeid Shahraz; Kenji Shibuya; Donald Silberberg; Jasvinder A Singh; Kyle Steenland; Jennifer A Taylor; George D Thurston; Monica S Vavilala; Theo Vos; Gregory R Wagner; Martin A Weinstock; Marc G Weisskopf; Sarah Wulf Journal: JAMA Date: 2013-08-14 Impact factor: 56.272
Authors: Jennifer E Flythe; Derek Forfang; Nieltje Gedney; David M White; Caroline Wilkie; Kerri L Cavanaugh; Raymond C Harris; Mark Unruh; Grace Squillaci; Melissa West; Carol Mansfield; Cindy S Soloe; Katherine Treiman; Dallas Wood; Frank P Hurst; Carolyn Y Neuland; Anindita Saha; Murray Sheldon; Michelle E Tarver Journal: Kidney360 Date: 2022-05-05