Terence J Quinn1, Jennifer K Burton2, Ben Carter3, Nicola Cooper4, Kerry Dwan5, Ryan Field6, Suzanne C Freeman4, Claudia Geue6, Ping-Hsuan Hsieh6,7, Kris McGill8, Clareece R Nevill4, Dikshyanta Rana6, Alex Sutton2, Martin Taylor Rowan2, Yiqiao Xin6. 1. Institute of Cardiovascular and Medical Sciences, University of Glasgow, New Lister Building Campus, Alexandra Parade, Glasgow, G31 2ER, UK. terry.quinn@glasgow.ac.uk. 2. Institute of Cardiovascular and Medical Sciences, University of Glasgow, Glasgow, UK. 3. Institute of Psychiatry, Psychology and Neuroscience Kings College London, London, UK. 4. Department of Health Sciences, University of Leicester, Leicester, UK. 5. Cochrane Methods Support Unit, Cochrane, UK, Oxford, UK. 6. Health Economics and Health Technology Assessment, University of Glasgow, Glasgow, UK. 7. Tri-Service General Hospital, National Defence Medical Center, Taipei, Taiwan. 8. NMAHP Research Unit, Glasgow Caledonian University, Glasgow, UK.
Abstract
BACKGROUND: Following the initial identification of the 2019 coronavirus disease (covid-19), the subsequent months saw substantial increases in published biomedical research. Concerns have been raised in both scientific and lay press around the quality of some of this research. We assessed clinical research from major clinical journals, comparing methodological and reporting quality of covid-19 papers published in the first wave (here defined as December 2019 to May 2020 inclusive) of the viral pandemic with non-covid papers published at the same time. METHODS: We reviewed research publications (print and online) from The BMJ, Journal of the American Medical Association (JAMA), The Lancet, and New England Journal of Medicine, from first publication of a covid-19 research paper (February 2020) to May 2020 inclusive. Paired reviewers were randomly allocated to extract data on methodological quality (risk of bias) and reporting quality (adherence to reporting guidance) from each paper using validated assessment tools. A random 10% of papers were assessed by a third, independent rater. Overall methodological quality for each paper was rated high, low or unclear. Reporting quality was described as percentage of total items reported. RESULTS: From 168 research papers, 165 were eligible, including 54 (33%) papers with a covid-19 focus. For methodological quality, 18 (33%) covid-19 papers and 83 (73%) non-covid papers were rated as low risk of bias, OR 6.32 (95%CI 2.85 to 14.00). The difference in quality was maintained after adjusting for publication date, results, funding, study design, journal and raters (OR 6.09 (95%CI 2.09 to 17.72)). For reporting quality, adherence to reporting guidelines was poorer for covid-19 papers, mean percentage of total items reported 72% (95%CI:66 to 77) for covid-19 papers and 84% (95%CI:81 to 87) for non-covid. CONCLUSIONS: Across various measures, we have demonstrated that covid-19 research from the first wave of the pandemic was potentially of lower quality than contemporaneous non-covid research. While some differences may be an inevitable consequence of conducting research during a viral pandemic, poor reporting should not be accepted.
BACKGROUND: Following the initial identification of the 2019 coronavirus disease (covid-19), the subsequent months saw substantial increases in published biomedical research. Concerns have been raised in both scientific and lay press around the quality of some of this research. We assessed clinical research from major clinical journals, comparing methodological and reporting quality of covid-19 papers published in the first wave (here defined as December 2019 to May 2020 inclusive) of the viral pandemic with non-covid papers published at the same time. METHODS: We reviewed research publications (print and online) from The BMJ, Journal of the American Medical Association (JAMA), The Lancet, and New England Journal of Medicine, from first publication of a covid-19 research paper (February 2020) to May 2020 inclusive. Paired reviewers were randomly allocated to extract data on methodological quality (risk of bias) and reporting quality (adherence to reporting guidance) from each paper using validated assessment tools. A random 10% of papers were assessed by a third, independent rater. Overall methodological quality for each paper was rated high, low or unclear. Reporting quality was described as percentage of total items reported. RESULTS: From 168 research papers, 165 were eligible, including 54 (33%) papers with a covid-19 focus. For methodological quality, 18 (33%) covid-19 papers and 83 (73%) non-covid papers were rated as low risk of bias, OR 6.32 (95%CI 2.85 to 14.00). The difference in quality was maintained after adjusting for publication date, results, funding, study design, journal and raters (OR 6.09 (95%CI 2.09 to 17.72)). For reporting quality, adherence to reporting guidelines was poorer for covid-19 papers, mean percentage of total items reported 72% (95%CI:66 to 77) for covid-19 papers and 84% (95%CI:81 to 87) for non-covid. CONCLUSIONS: Across various measures, we have demonstrated that covid-19 research from the first wave of the pandemic was potentially of lower quality than contemporaneous non-covid research. While some differences may be an inevitable consequence of conducting research during a viral pandemic, poor reporting should not be accepted.
Authors: Lucy Turner; Larissa Shamseer; Douglas G Altman; Laura Weeks; Jodi Peters; Thilo Kober; Sofia Dias; Kenneth F Schulz; Amy C Plint; David Moher Journal: Cochrane Database Syst Rev Date: 2012-11-14
Authors: Benedikt Schmid; Peter Kranke; Rudolf Lucas; Patrick Meybohm; Bernhard Zwissler; Sandra Frank Journal: Trials Date: 2022-04-04 Impact factor: 2.279
Authors: Tom Jefferson; Carl J Heneghan; Elizabeth Spencer; Jon Brassey; Annette Plüddemann; Igho Onakpoya; David Evans; John Conly Journal: Viruses Date: 2022-07-22 Impact factor: 5.818
Authors: John P A Ioannidis; Eran Bendavid; Maia Salholz-Hillel; Kevin W Boyack; Jeroen Baas Journal: Proc Natl Acad Sci U S A Date: 2022-07-07 Impact factor: 12.779
Authors: Sebastian Ziemann; Irina Paetzolt; Linda Grüßer; Mark Coburn; Rolf Rossaint; Ana Kowark Journal: BMC Med Res Methodol Date: 2022-01-20 Impact factor: 4.615