Luke Farrow1,2, William T Gardner3, Andrew D Ablett4, Vladislav Kutuzov5, Alan Johnstone3. 1. Institute of Applied Health Sciences, University of Aberdeen, Foresterhill, Aberdeen, AB25 2ZD, Scotland, UK. luke.farrow@abdn.ac.uk. 2. Aberdeen Royal Infirmary, Aberdeen, UK. luke.farrow@abdn.ac.uk. 3. Aberdeen Royal Infirmary, Aberdeen, UK. 4. Edinburgh Royal Infirmary, Edinburgh, UK. 5. Charing Cross Hospital, London, UK.
Abstract
INTRODUCTION: The recent past has seen a significant increase in the number of trauma and orthopaedic randomised clinical trials published in "the big five" general medical journals. The quality of this research has, however, not yet been established. METHODS: We therefore set out to critically appraise the quality of available literature over a 10-year period (April 2010-April 2020) through a systematic search of these 5 high-impact general medical journals (JAMA, NEJM, BMJ, Lancet and Annals). A standardised data extraction proforma was utilised to gather information regarding: trial design, sample size calculation, results, study quality and pragmatism. Quality assessment was performed using the Cochrane Risk of Bias 2 tool and the modified Delphi list. Study pragmatism was assessed using the PRECIS-2 tool. RESULTS: A total of 25 studies were eligible for inclusion. Over half of the included trials did not meet their sample size calculation for the primary outcome, with a similar proportion of these studies at risk of type II error for their non-significant results. There was a high degree of pragmatism according to PRECIS-2. Non-significant studies had greater pragmatism that those with statistically significant results (p < 0.001). Only 56% studies provided adequate justification for the minimum clinically important difference (MCID) in the population assessed. Overall, very few studies were deemed high quality/low risk of bias. CONCLUSIONS: These findings highlight that there are some important methodological concerns present within the current evidence base of RCTs published in high-impact medical journals. Potential strategies that may improve future trial design are highlighted. LEVEL OF EVIDENCE: Level 1.
INTRODUCTION: The recent past has seen a significant increase in the number of trauma and orthopaedic randomised clinical trials published in "the big five" general medical journals. The quality of this research has, however, not yet been established. METHODS: We therefore set out to critically appraise the quality of available literature over a 10-year period (April 2010-April 2020) through a systematic search of these 5 high-impact general medical journals (JAMA, NEJM, BMJ, Lancet and Annals). A standardised data extraction proforma was utilised to gather information regarding: trial design, sample size calculation, results, study quality and pragmatism. Quality assessment was performed using the Cochrane Risk of Bias 2 tool and the modified Delphi list. Study pragmatism was assessed using the PRECIS-2 tool. RESULTS: A total of 25 studies were eligible for inclusion. Over half of the included trials did not meet their sample size calculation for the primary outcome, with a similar proportion of these studies at risk of type II error for their non-significant results. There was a high degree of pragmatism according to PRECIS-2. Non-significant studies had greater pragmatism that those with statistically significant results (p < 0.001). Only 56% studies provided adequate justification for the minimum clinically important difference (MCID) in the population assessed. Overall, very few studies were deemed high quality/low risk of bias. CONCLUSIONS: These findings highlight that there are some important methodological concerns present within the current evidence base of RCTs published in high-impact medical journals. Potential strategies that may improve future trial design are highlighted. LEVEL OF EVIDENCE: Level 1.
Authors: Søren T Skou; Ewa M Roos; Mogens B Laursen; Michael S Rathleff; Lars Arendt-Nielsen; Ole Simonsen; Sten Rasmussen Journal: N Engl J Med Date: 2015-10-22 Impact factor: 91.245
Authors: Elliott E Ridgeon; Paul J Young; Rinaldo Bellomo; Marta Mucchetti; Rosalba Lembo; Giovanni Landoni Journal: Crit Care Med Date: 2016-07 Impact factor: 7.598
Authors: Victor A van de Graaf; Julia C A Noorduyn; Nienke W Willigenburg; Ise K Butter; Arthur de Gast; Ben W Mol; Daniel B F Saris; Jos W R Twisk; Rudolf W Poolman Journal: JAMA Date: 2018-10-02 Impact factor: 56.272
Authors: Raine Sihvonen; Mika Paavola; Antti Malmivaara; Ari Itälä; Antti Joukainen; Heikki Nurmi; Juha Kalske; Teppo L N Järvinen Journal: N Engl J Med Date: 2013-12-26 Impact factor: 91.245
Authors: Matthew L Costa; Juul Achten; Ruth Knight; Julie Bruce; Susan J Dutton; Jason Madan; Melina Dritsaki; Nick Parsons; Miguel Fernandez; Richard Grant; Jagdeep Nanchahal Journal: JAMA Date: 2020-02-11 Impact factor: 56.272
Authors: Antony J R Palmer; Vandana Ayyar Gupta; Scott Fernquest; Ines Rombach; Susan J Dutton; Ramy Mansour; Simon Wood; Vikas Khanduja; Tom C B Pollard; Andrew W McCaskie; Karen L Barker; Tony J M D Andrade; Andrew J Carr; David J Beard; Sion Glyn-Jones Journal: BMJ Date: 2019-02-07
Authors: Ada Keding; Helen Handoll; Stephen Brealey; Laura Jefferson; Catherine Hewitt; Belen Corbacho; David Torgerson; Amar Rangan Journal: Trials Date: 2019-09-18 Impact factor: 2.279
Authors: Cristina E Firanescu; Jolanda de Vries; Paul Lodder; Alexander Venmans; Marinus C Schoemaker; Albert J Smeets; Esther Donga; Job R Juttmann; Caroline A H Klazen; Otto E H Elgersma; Frits H Jansen; Alexander V Tielbeek; Issam Boukrab; Karen Schonenberg; Willem Jan J van Rooij; Joshua A Hirsch; Paul N M Lohle Journal: BMJ Date: 2018-05-09
Authors: Muhammad Shariq Usman; Harriette G C Van Spall; Stephen J Greene; Ambarish Pandey; Darren K McGuire; Ziad A Ali; Robert J Mentz; Gregg C Fonarow; John A Spertus; Stefan D Anker; Javed Butler; Stefan K James; Muhammad Shahzeb Khan Journal: Nat Rev Cardiol Date: 2022-05-17 Impact factor: 49.421