James Thomas1, Steve McDonald2, Anna Noel-Storr3, Ian Shemilt4, Julian Elliott5, Chris Mavergames6, Iain J Marshall7. 1. EPPI-Centre, UCL Social Research Institute, University College London, London, UK. Electronic address: james.thomas@ucl.ac.uk. 2. Cochrane Australia, School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia. 3. Radcliffe Department of Medicine, University of Oxford, Oxford, UK; Cochrane, London, UK. 4. EPPI-Centre, UCL Social Research Institute, University College London, London, UK. 5. Department of Infectious Diseases, Monash University and Alfred Hospital, Melbourne, Australia. 6. Cochrane, London, UK. 7. School of Population Health & Environmental Sciences, Kings College London, London, UK.
Abstract
OBJECTIVES: This study developed, calibrated, and evaluated a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews. METHODS: A machine learning classifier for retrieving randomized controlled trials (RCTs) was developed (the "Cochrane RCT Classifier"), with the algorithm trained using a data set of title-abstract records from Embase, manually labeled by the Cochrane Crowd. The classifier was then calibrated using a further data set of similar records manually labeled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification. RESULTS: The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs), and our bootstrap validation found the classifier had recall of 0.99 (95% confidence interval 0.98-0.99) and precision of 0.08 (95% confidence interval 0.06-0.12) in this data set. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published. CONCLUSIONS: The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane Reviews, with a very low and acceptable risk of missing eligible RCTs. This classifier now forms part of the Evidence Pipeline, an integrated workflow deployed within Cochrane to help improve the efficiency of the study identification processes that support systematic review production.
OBJECTIVES: This study developed, calibrated, and evaluated a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews. METHODS: A machine learning classifier for retrieving randomized controlled trials (RCTs) was developed (the "Cochrane RCT Classifier"), with the algorithm trained using a data set of title-abstract records from Embase, manually labeled by the Cochrane Crowd. The classifier was then calibrated using a further data set of similar records manually labeled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification. RESULTS: The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs), and our bootstrap validation found the classifier had recall of 0.99 (95% confidence interval 0.98-0.99) and precision of 0.08 (95% confidence interval 0.06-0.12) in this data set. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published. CONCLUSIONS: The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane Reviews, with a very low and acceptable risk of missing eligible RCTs. This classifier now forms part of the Evidence Pipeline, an integrated workflow deployed within Cochrane to help improve the efficiency of the study identification processes that support systematic review production.
Authors: Julian Elliott; Rebecca Lawrence; Jan C Minx; Olufemi T Oladapo; Philippe Ravaud; Britta Tendal Jeppesen; James Thomas; Tari Turner; Per Olav Vandvik; Jeremy M Grimshaw Journal: Nature Date: 2021-12 Impact factor: 49.962
Authors: Jesse Dawson; Yannick Béjot; Louisa M Christensen; Gian Marco De Marchis; Martin Dichgans; Guri Hagberg; Mirjam R Heldner; Haralampos Milionis; Linxin Li; Francesca Romana Pezzella; Martin Taylor Rowan; Cristina Tiu; Alastair Webb Journal: Eur Stroke J Date: 2022-06-03
Authors: Monika Geretsegger; Laura Fusar-Poli; Cochavit Elefant; Karin A Mössler; Giovanni Vitale; Christian Gold Journal: Cochrane Database Syst Rev Date: 2022-05-09
Authors: Elizabeth Allen; Alice R Rumbold; Amy Keir; Carmel T Collins; Jennifer Gillis; Hiroki Suganuma Journal: Cochrane Database Syst Rev Date: 2021-10-21
Authors: Anna H Noel-Storr; Patrick Redmond; Guillaume Lamé; Elisa Liberati; Sarah Kelly; Lucy Miller; Gordon Dooley; Andy Paterson; Jenni Burt Journal: BMC Med Res Methodol Date: 2021-04-26 Impact factor: 4.615