PURPOSE: Comparative efficacy research performed using population registries can be subject to significant bias. There is an absence of objective data demonstrating factors that can sufficiently reduce bias and provide accurate results. METHODS: MEDLINE was searched from January 2000 to October 2016 for observational studies comparing two treatment regimens for any diagnosis of cancer, using SEER, SEER-Medicare, or the National Cancer Database. Reporting quality and statistical methods were assessed using components of the STROBE criteria. Randomized trials comparing the same treatment regimens were identified. Primary outcome was correlation between survival hazard ratio (HR) estimates provided by the observational studies and randomized trials. Secondary outcomes included agreement between matched pairs and predictors of agreement. RESULTS: Of 3,657 studies reviewed, 350 treatment comparisons met eligibility criteria and were matched to 121 randomized trials. There was no significant correlation between the HR estimates reported by observational studies and randomized trials (concordance correlation coefficient, 0.083; 95% CI, -0.068 to 0.230). Forty percent of matched studies were in agreement regarding treatment effects (κ, 0.037; 95% CI, -0.027 to 0.1), and 62% of the observational study HRs fell within the 95% CIs of the randomized trials. Cancer type, data source, reporting quality, adjustment for age, stage, or comorbidities, use of propensity weighting, instrumental variable or sensitivity analysis, and well-matched study population did not predict agreement. CONCLUSION: We were unable to identify any modifiable factor present in population-based observational studies that improved agreement with randomized trials. There was no agreement beyond what is expected by chance, regardless of reporting quality or statistical rigor of the observational study. Future work is needed to identify reliable methods for conducting population-based comparative efficacy research.
PURPOSE: Comparative efficacy research performed using population registries can be subject to significant bias. There is an absence of objective data demonstrating factors that can sufficiently reduce bias and provide accurate results. METHODS: MEDLINE was searched from January 2000 to October 2016 for observational studies comparing two treatment regimens for any diagnosis of cancer, using SEER, SEER-Medicare, or the National Cancer Database. Reporting quality and statistical methods were assessed using components of the STROBE criteria. Randomized trials comparing the same treatment regimens were identified. Primary outcome was correlation between survival hazard ratio (HR) estimates provided by the observational studies and randomized trials. Secondary outcomes included agreement between matched pairs and predictors of agreement. RESULTS: Of 3,657 studies reviewed, 350 treatment comparisons met eligibility criteria and were matched to 121 randomized trials. There was no significant correlation between the HR estimates reported by observational studies and randomized trials (concordance correlation coefficient, 0.083; 95% CI, -0.068 to 0.230). Forty percent of matched studies were in agreement regarding treatment effects (κ, 0.037; 95% CI, -0.027 to 0.1), and 62% of the observational study HRs fell within the 95% CIs of the randomized trials. Cancer type, data source, reporting quality, adjustment for age, stage, or comorbidities, use of propensity weighting, instrumental variable or sensitivity analysis, and well-matched study population did not predict agreement. CONCLUSION: We were unable to identify any modifiable factor present in population-based observational studies that improved agreement with randomized trials. There was no agreement beyond what is expected by chance, regardless of reporting quality or statistical rigor of the observational study. Future work is needed to identify reliable methods for conducting population-based comparative efficacy research.
Authors: Tyler A Janz; Evan M Graboyes; Shaun A Nguyen; Mark A Ellis; David M Neskey; E Emily Harruff; Eric J Lentsch Journal: Otolaryngol Head Neck Surg Date: 2018-08-21 Impact factor: 3.497
Authors: Michel Bolla; Geertjan Van Tienhoven; Padraig Warde; Jean Bernard Dubois; René-Olivier Mirimanoff; Guy Storme; Jacques Bernier; Abraham Kuten; Cora Sternberg; Ignace Billiet; José Lopez Torecilla; Raphael Pfeffer; Carmel Lino Cutajar; Theodore Van der Kwast; Laurence Collette Journal: Lancet Oncol Date: 2010-10-07 Impact factor: 41.316
Authors: Jacques E Rossouw; Garnet L Anderson; Ross L Prentice; Andrea Z LaCroix; Charles Kooperberg; Marcia L Stefanick; Rebecca D Jackson; Shirley A A Beresford; Barbara V Howard; Karen C Johnson; Jane Morley Kotchen; Judith Ockene Journal: JAMA Date: 2002-07-17 Impact factor: 56.272
Authors: Reshma Jagsi; Justin E Bekelman; Aileen Chen; Ronald C Chen; Karen Hoffman; Ya-Chen Tina Shih; Benjamin D Smith; James B Yu Journal: Int J Radiat Oncol Biol Phys Date: 2014-09-01 Impact factor: 7.038
Authors: Joseph M Unger; William E Barlow; Diane P Martin; Scott D Ramsey; Michael Leblanc; Ruth Etzioni; Dawn L Hershman Journal: J Natl Cancer Inst Date: 2014-03-13 Impact factor: 13.506
Authors: Catherine M Tangen; Jeannette Schenk; Cathee Till; Phyllis J Goodman; Wendy Barrington; M Scott Lucia; Ian M Thompson Journal: Cancer Epidemiol Date: 2019-10-19 Impact factor: 2.984
Authors: Filippo Pederzoli; Marco Bandini; Alberto Briganti; Elizabeth R Plimack; Günter Niegisch; Evan Y Yu; Aristotelis Bamias; Neeraj Agarwal; Srikala S Sridhar; Cora N Sternberg; Ulka N Vaishampayan; Christine Théodore; Jonathan E Rosenberg; Lauren C Harshman; Joaquim Bellmunt; Matthew D Galsky; Andrea Gallina; Andrea Salonia; Francesco Montorsi; Andrea Necchi Journal: Eur Urol Date: 2019-07-11 Impact factor: 20.096
Authors: Christina L Roland; Elizabeth G Grubbs; Matthew H G Katz; Mediget Teshome; Kelly K Hunt; Ching-Wei Tzeng Journal: J Surg Oncol Date: 2020-03-09 Impact factor: 3.454
Authors: Robert T Dess; Holly E Hartman; Brandon A Mahal; Payal D Soni; William C Jackson; Matthew R Cooperberg; Christopher L Amling; William J Aronson; Christopher J Kane; Martha K Terris; Zachary S Zumsteg; Santino Butler; Joseph R Osborne; Todd M Morgan; Rohit Mehra; Simpa S Salami; Amar U Kishan; Chenyang Wang; Edward M Schaeffer; Mack Roach; Thomas M Pisansky; William U Shipley; Stephen J Freedland; Howard M Sandler; Susan Halabi; Felix Y Feng; James J Dignam; Paul L Nguyen; Matthew J Schipper; Daniel E Spratt Journal: JAMA Oncol Date: 2019-07-01 Impact factor: 31.777
Authors: Kristian D Stensland; Krystal DePorto; James Ryan; Samuel Kaffenberger; Lael S Reinstatler; Matthew Galsky; David Canes; Ted A Skolarus; Alireza Moinzadeh Journal: Urol Oncol Date: 2020-11-27 Impact factor: 3.498
Authors: Nicholas G Zaorsky; Xi Wang; Sara M Garrett; Eric J Lehrer; Christine Lin; David J DeGraff; Daniel E Spratt; Daniel M Trifiletti; Amar U Kishan; Timothy N Showalter; Henry S Park; Jonathan T Yang; Vernon M Chinchilli; Ming Wang Journal: Int J Cancer Date: 2021-08-27 Impact factor: 7.396