Chris Cooper1, Joanna Varley-Campbell2, Andrew Booth3, Nicky Britten4, Ruth Garside5. 1. Systematic Review Initiative, NHS Blood & Transplant, University of Oxford, Oxford OX3 9BQ, UK. Electronic address: chris.cooper@ndcls.ox.ac.uk. 2. Centre of Outcomes Research and Effectiveness, University College London, London WC1 7HB, UK. 3. HEDS, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield S10 2TN, UK. 4. Institute of Health Research, University of Exeter Medical School, Exeter EX4, UK. 5. European Centre for Environment and Human Health, University of Exeter Medical Schooll, Exeter EX4, UK.
Abstract
OBJECTIVES: To identify the metrics or methods used by researchers to determine the effectiveness of literature searching where supplementary search methods are compared to bibliographic database searching. We also aimed to determine which metrics or methods are summative or formative and how researchers defined effectiveness in their studies. STUDY DESIGN AND SETTING: Systematic review. We searched MEDLINE and Embase to identify published studies evaluating literature search effectiveness in health or allied topics. RESULTS: Fifty studies met full-text inclusion criteria. Six metrics (sensitivity, specificity, precision, accuracy, number needed to read, and yield) and one method (capture recapture) were identified. CONCLUSION: Studies evaluating effectiveness need to identify clearly the threshold at which they will define effectiveness and how the evaluation they report relates to this threshold. Studies that attempt to investigate literature search effectiveness should be informed by the reporting of confidence intervals, which aids interpretation of uncertainty around the result, and the search methods used to derive effectiveness estimates should be clearly reported and validated in studies.
OBJECTIVES: To identify the metrics or methods used by researchers to determine the effectiveness of literature searching where supplementary search methods are compared to bibliographic database searching. We also aimed to determine which metrics or methods are summative or formative and how researchers defined effectiveness in their studies. STUDY DESIGN AND SETTING: Systematic review. We searched MEDLINE and Embase to identify published studies evaluating literature search effectiveness in health or allied topics. RESULTS: Fifty studies met full-text inclusion criteria. Six metrics (sensitivity, specificity, precision, accuracy, number needed to read, and yield) and one method (capture recapture) were identified. CONCLUSION: Studies evaluating effectiveness need to identify clearly the threshold at which they will define effectiveness and how the evaluation they report relates to this threshold. Studies that attempt to investigate literature search effectiveness should be informed by the reporting of confidence intervals, which aids interpretation of uncertainty around the result, and the search methods used to derive effectiveness estimates should be clearly reported and validated in studies.
Authors: Elena Stallings; Andrea Gaetano-Gil; Noelia Alvarez-Diaz; Ivan Solà; Jesús López-Alcalde; Daniel Molano; Javier Zamora Journal: BMC Med Res Methodol Date: 2022-04-10 Impact factor: 4.615
Authors: Jeffrey V Lazarus; Adam Palayew; Lauge Neimann Rasmussen; Tue Helms Andersen; Joey Nicholson; Ole Norgaard Journal: J Med Internet Res Date: 2020-11-26 Impact factor: 5.428