Margaret Sampson1, Berry de Bruijn2, Christine Urquhart3, Kaveh Shojania4. 1. Library and Media Services, Children's Hospital of Eastern Ontario, 401 Smyth Road, Ottawa, Ontario K1H 8L1, Canada. Electronic address: msampson@cheo.on.ca. 2. National Research Council-Information and Communications Technologies Portfolio (NRC-ICT), 1200 Montreal Rd, bldg M-50, Ottawa, Ontario K1A 0R6, Canada. 3. Department of Information Studies, Sunnybrook Health Sciences Centre, Aberystwyth University, Llanbadarn Campus, Aberystwyth, Wales SY23 3AL, United Kingdom. 4. Centre for Quality Improvement and Patient Safety, Room H468, 2075 Bayview Avenue Toronto, Ontario M4N 3M5, Canada.
Abstract
OBJECTIVES: To maximize the proportion of relevant studies identified for inclusion in systematic reviews (recall), complex time-consuming Boolean searches across multiple databases are common. Although MEDLINE provides excellent coverage of health science evidence, it has proved challenging to achieve high levels of recall through Boolean searches alone. STUDY DESIGN AND SETTING: Recall of one Boolean search method, the clinical query (CQ), combined with a ranking method, support vector machine (SVM), or PubMed-related articles, was tested against a gold standard of studies added to 6 updated Cochrane reviews and 10 Agency for Healthcare Research and Quality (AHRQ) evidence reviews. For the AHRQ sample, precision and temporal stability were examined for each method. RESULTS: Recall of new studies was 0.69 for the CQ, 0.66 for related articles, 0.50 for SVM, 0.91 for the combination of CQ and related articles, and 0.89 for the combination of CQ and SVM. Precision was 0.11 for CQ and related articles combined, and 0.11 for CQ and SVM combined. Related articles showed least stability over time. CONCLUSIONS: The complementary combination of a Boolean search strategy and a ranking strategy appears to provide a robust method for identifying relevant studies in MEDLINE.
OBJECTIVES: To maximize the proportion of relevant studies identified for inclusion in systematic reviews (recall), complex time-consuming Boolean searches across multiple databases are common. Although MEDLINE provides excellent coverage of health science evidence, it has proved challenging to achieve high levels of recall through Boolean searches alone. STUDY DESIGN AND SETTING: Recall of one Boolean search method, the clinical query (CQ), combined with a ranking method, support vector machine (SVM), or PubMed-related articles, was tested against a gold standard of studies added to 6 updated Cochrane reviews and 10 Agency for Healthcare Research and Quality (AHRQ) evidence reviews. For the AHRQ sample, precision and temporal stability were examined for each method. RESULTS: Recall of new studies was 0.69 for the CQ, 0.66 for related articles, 0.50 for SVM, 0.91 for the combination of CQ and related articles, and 0.89 for the combination of CQ and SVM. Precision was 0.11 for CQ and related articles combined, and 0.11 for CQ and SVM combined. Related articles showed least stability over time. CONCLUSIONS: The complementary combination of a Boolean search strategy and a ranking strategy appears to provide a robust method for identifying relevant studies in MEDLINE.