Adrian Gheorghe1,2, Tracy Roberts3, Karla Hemming4, Melanie Calvert5. 1. Primary Care Clinical Sciences and MRC Midlands Hub for Trials Methodology Research, University of Birmingham, Birmingham, UK. adrian.e.gheorghe@gmail.com. 2. Department of Global Health and Development, London School of Hygiene and Tropical Medicine, 15-17 Tavistock Place, London, WC1H 9SH, UK. adrian.e.gheorghe@gmail.com. 3. Health Economics Unit, University of Birmingham, Birmingham, UK. 4. Public Health, Epidemiology and Statistics, University of Birmingham, Birmingham, UK. 5. Primary Care Clinical Sciences and MRC Midlands Hub for Trials Methodology Research, University of Birmingham, Birmingham, UK.
Abstract
BACKGROUND: Few randomised controlled trials (RCTs) recruit centres representatively, which may limit the external validity of trial results. OBJECTIVE: The aim of this study was to propose a proof-of-concept method of assessing the generalisability of the clinical and cost-effectiveness findings of a given RCT. METHODS: We developed a generalisability index (Gix), informed by centre-level characteristics, as a measure of centre and trial representativeness. The centre-level Gix quantifies how representative a centre is in relation to its jurisdiction, e.g. a country or health authority. The trial-level Gix quantifies how representative trial recruitment is in relation to clinical practice in the jurisdiction. Taking a real-world RCT as a case study and assuming trial-wide results to represent 'true jurisdiction values', we used simulation methods to recreate 5000 RCTs and investigate the relationship between trial representativeness, reflected by the standardised trial-Gix, and the deviation of simulated trial results from the 'true values'. RESULTS: The simulation study provides evidence that trial results (odds ratio for the primary outcome and incremental quality-adjusted life-years) were influenced by the representativeness of the sample of recruiting centres. Simulated RCTs with the closest results to the 'true values' were those whose recruitment closely mirrored the jurisdiction-wide context. Results appeared robust to six alternative specifications of the Gix. CONCLUSIONS: Our findings suggest that an unrepresentative selection of centres limits the external validity of trial results. The Gix may be a valuable tool to help facilitate rational selection of trial centres and ensure the generalisability of results at the jurisdiction level.
BACKGROUND: Few randomised controlled trials (RCTs) recruit centres representatively, which may limit the external validity of trial results. OBJECTIVE: The aim of this study was to propose a proof-of-concept method of assessing the generalisability of the clinical and cost-effectiveness findings of a given RCT. METHODS: We developed a generalisability index (Gix), informed by centre-level characteristics, as a measure of centre and trial representativeness. The centre-level Gix quantifies how representative a centre is in relation to its jurisdiction, e.g. a country or health authority. The trial-level Gix quantifies how representative trial recruitment is in relation to clinical practice in the jurisdiction. Taking a real-world RCT as a case study and assuming trial-wide results to represent 'true jurisdiction values', we used simulation methods to recreate 5000 RCTs and investigate the relationship between trial representativeness, reflected by the standardised trial-Gix, and the deviation of simulated trial results from the 'true values'. RESULTS: The simulation study provides evidence that trial results (odds ratio for the primary outcome and incremental quality-adjusted life-years) were influenced by the representativeness of the sample of recruiting centres. Simulated RCTs with the closest results to the 'true values' were those whose recruitment closely mirrored the jurisdiction-wide context. Results appeared robust to six alternative specifications of the Gix. CONCLUSIONS: Our findings suggest that an unrepresentative selection of centres limits the external validity of trial results. The Gix may be a valuable tool to help facilitate rational selection of trial centres and ensure the generalisability of results at the jurisdiction level.
Authors: M F Drummond; B S Bloom; G Carrin; A L Hillman; H C Hutchings; R P Knill-Jones; G de Pouvourville; K Torfs Journal: Int J Technol Assess Health Care Date: 1992 Impact factor: 2.188
Authors: David J Costa; Michel Amouyal; Philippe Lambert; Dermot Ryan; Holger J Schünemann; Jean Pierre Daures; Jean Bousquet; Philippe J Bousquet Journal: J Allergy Clin Immunol Date: 2011-01-26 Impact factor: 10.793
Authors: Andrew W Hoel; Ahmed Kayssi; Soma Brahmanandam; Michael Belkin; Michael S Conte; Louis L Nguyen Journal: J Vasc Surg Date: 2009-08 Impact factor: 4.268
Authors: Nicholas Shortt; Alexander Martin; Kyley Kerse; Gabrielle Shortt; Iva Vakalalabure; Luke Barker; Joseph Singer; Bianca Black; Angela Liu; Allie Eathorne; Mark Weatherall; Marius Rademaker; Mike Armour; Richard Beasley; Alex Semprini Journal: EClinicalMedicine Date: 2022-07-15
Authors: Cristina M Caperchione; Mitch J Duncan; Richard R Rosenkranz; Corneel Vandelanotte; Anetta K Van Itallie; Trevor N Savage; Cindy Hooker; Anthony J Maeder; W Kerry Mummery; Gregory S Kolt Journal: Contemp Clin Trials Commun Date: 2015-12-21
Authors: Alex Semprini; Joseph Singer; Irene Braithwaite; Nick Shortt; Darmiga Thayabaran; Melanie McConnell; Mark Weatherall; Richard Beasley Journal: BMJ Open Date: 2019-05-14 Impact factor: 2.692
Authors: Wim van Biesen; Catherine Van Der Straeten; Sigrid Sterckx; Johan Steen; Lisa Diependaele; Johan Decruyenaere Journal: BMC Med Inform Decis Mak Date: 2021-03-06 Impact factor: 2.796
Authors: Rebecca Mercieca-Bebber; Michael J Palmer; Michael Brundage; Melanie Calvert; Martin R Stockler; Madeleine T King Journal: BMJ Open Date: 2016-06-15 Impact factor: 2.692