Lars G Hemkens1, Despina G Contopoulos-Ioannidis1, John P A Ioannidis1. 1. Stanford Prevention Research Center, Department of Medicine (Hemkens, Ioannidis), Stanford University School of Medicine, Stanford, Calif.; Basel Institute for Clinical Epidemiology and Biostatistics (Hemkens), University Hospital Basel, Basel, Switzerland; Department of Pediatrics, Division of Infectious Diseases (Contopoulos-Ioannidis), Stanford University School of Medicine, Stanford, Calif.; Palo Alto Medical Foundation Research Institute (Contopoulos-Ioannidis), Palo Alto, Calif.; Department of Health Research and Policy (Ioannidis), Stanford University School of Medicine; Meta-Research Innovation Center at Stanford (METRICS) (Ioannidis), Stanford, Calif.
Abstract
BACKGROUND: Studies that use routinely collected health data (RCD studies) are advocated to complement evidence from randomized controlled trials (RCTs) for comparative effectiveness research and to inform health care decisions when RCTs would be unfeasible. We aimed to evaluate the current use of routinely collected health data to complement RCT evidence. METHODS: We searched PubMed for RCD studies published to 2010 that evaluated the comparative effectiveness of medical treatments on mortality using propensity scores. We identified RCTs of the same treatment comparisons and evaluated how frequently the RCD studies analyzed treatments that had not been compared previously in randomized trials. When RCTs did exist, we noted the claimed motivations for each RCD study. We also analyzed the citation impact of the RCD studies. RESULTS: Of 337 eligible RCD studies identified, 231 (68.5%) analyzed treatments that had already been compared in RCTs. The study investigators rarely claimed that it would be unethical (6/337) or difficult (18/337) to perform RCTs on the same question. Evidence from RCTs was mentioned or cited by authors of 213 RCD studies. The most common motivations for conducting the RCD studies were alleged limited generalizability of trial results to the "real world" (37.6%), evaluation of specific outcomes (31.9%) or specific populations (23.5%), and inconclusive or inconsistent evidence from randomized trials (25.8%). Studies evaluating "real world" effects had the lowest citation impact. INTERPRETATION: Most of the RCD studies we identified explored comparative treatment effects that had already been investigated in RCTs. The objective of such studies needs to shift more toward answering pivotal questions that are not supported by trial evidence or for which RCTs would be unfeasible.
BACKGROUND: Studies that use routinely collected health data (RCD studies) are advocated to complement evidence from randomized controlled trials (RCTs) for comparative effectiveness research and to inform health care decisions when RCTs would be unfeasible. We aimed to evaluate the current use of routinely collected health data to complement RCT evidence. METHODS: We searched PubMed for RCD studies published to 2010 that evaluated the comparative effectiveness of medical treatments on mortality using propensity scores. We identified RCTs of the same treatment comparisons and evaluated how frequently the RCD studies analyzed treatments that had not been compared previously in randomized trials. When RCTs did exist, we noted the claimed motivations for each RCD study. We also analyzed the citation impact of the RCD studies. RESULTS: Of 337 eligible RCD studies identified, 231 (68.5%) analyzed treatments that had already been compared in RCTs. The study investigators rarely claimed that it would be unethical (6/337) or difficult (18/337) to perform RCTs on the same question. Evidence from RCTs was mentioned or cited by authors of 213 RCD studies. The most common motivations for conducting the RCD studies were alleged limited generalizability of trial results to the "real world" (37.6%), evaluation of specific outcomes (31.9%) or specific populations (23.5%), and inconclusive or inconsistent evidence from randomized trials (25.8%). Studies evaluating "real world" effects had the lowest citation impact. INTERPRETATION: Most of the RCD studies we identified explored comparative treatment effects that had already been investigated in RCTs. The objective of such studies needs to shift more toward answering pivotal questions that are not supported by trial evidence or for which RCTs would be unfeasible.
Authors: Nancy A Dreyer; Sebastian Schneeweiss; Barbara J McNeil; Marc L Berger; Alec M Walker; Daniel A Ollendorf; Richard E Gliklich Journal: Am J Manag Care Date: 2010-06 Impact factor: 2.229
Authors: Nancy A Dreyer; Sean R Tunis; Marc Berger; Dan Ollendorf; Pattra Mattox; Richard Gliklich Journal: Health Aff (Millwood) Date: 2010-10 Impact factor: 6.301
Authors: Sripal Bangalore; Deepak L Bhatt; Joachim Röther; Mark J Alberts; Julie Thornton; Kathy Wolski; Shinya Goto; Alan T Hirsch; Sidney C Smith; Franz T Aichner; Raffi Topakian; Christopher P Cannon; P Gabriel Steg Journal: Circulation Date: 2010-08-30 Impact factor: 29.690
Authors: Maria Rosa Costanzo; R S Johannes; Michael Pine; Vikas Gupta; Mitchell Saltzberg; Joel Hay; Clyde W Yancy; Gregg C Fonarow Journal: Am Heart J Date: 2007-08 Impact factor: 4.749
Authors: Benjamin Kasenda; Stefan Schandelmaier; Xin Sun; Erik von Elm; John You; Anette Blümle; Yuki Tomonaga; Ramon Saccilotto; Alain Amstutz; Theresa Bengough; Joerg J Meerpohl; Mihaela Stegert; Kelechi K Olu; Kari A O Tikkinen; Ignacio Neumann; Alonso Carrasco-Labra; Markus Faulhaber; Sohail M Mulla; Dominik Mertz; Elie A Akl; Dirk Bassler; Jason W Busse; Ignacio Ferreira-González; Francois Lamontagne; Alain Nordmann; Viktoria Gloy; Heike Raatz; Lorenzo Moja; Rachel Rosenthal; Shanil Ebrahim; Per O Vandvik; Bradley C Johnston; Martin A Walter; Bernard Burnand; Matthias Schwenkglenks; Lars G Hemkens; Heiner C Bucher; Gordon H Guyatt; Matthias Briel Journal: BMJ Date: 2014-07-16
Authors: Sinéad M Langan; Sigrún A J Schmidt; Kevin Wing; Vera Ehrenstein; Stuart G Nicholls; Kristian B Filion; Olaf Klungel; Irene Petersen; Henrik T Sørensen; William G Dixon; Astrid Guttmann; Katie Harron; Lars G Hemkens; David Moher; Sebastian Schneeweiss; Liam Smeeth; Miriam Sturkenboom; Erik von Elm; Shirley V Wang; Eric I Benchimol Journal: CMAJ Date: 2019-06-24 Impact factor: 8.262
Authors: Sinéad M Langan; Sigrún Aj Schmidt; Kevin Wing; Vera Ehrenstein; Stuart G Nicholls; Kristian B Filion; Olaf Klungel; Irene Petersen; Henrik T Sorensen; William G Dixon; Astrid Guttmann; Katie Harron; Lars G Hemkens; David Moher; Sebastian Schneeweiss; Liam Smeeth; Miriam Sturkenboom; Erik von Elm; Shirley V Wang; Eric I Benchimol Journal: BMJ Date: 2018-11-14
Authors: Aukje L Kreuger; Rutger A Middelburg; Erik A M Beckers; Karen M K de Vooght; Jaap Jan Zwaginga; Jean-Louis H Kerkhoffs; Johanna G van der Bom Journal: PLoS One Date: 2018-08-15 Impact factor: 3.240
Authors: Stephen J McCall; Mahrukh Imran; Lars G Hemkens; Kimberly Mc Cord; Linda Kwakkenbos; Margaret Sampson; Sena Jawad; Merrick Zwarenstein; Clare Relton; Sinéad M Langan; David Moher; Ole Fröbert; Brett D Thombs; Chris Gale; Edmund Juszczak Journal: J Clin Epidemiol Date: 2021-09-12 Impact factor: 6.437