Healthcare decision-making needs to be informed by high quality and timely research evidence. The randomized controlled trial has long been considered the ‘gold standard’ in the hierarchy of evidence. Trials involving sufficient numbers of participants are essential to distinguish reliably between the effects of healthcare interventions and the effects of bias or chance. The synthesis of the results of these trials in systematic reviews can provide reliable evidence about the effects of these interventions.The Cochrane Collaboration, an international organization dedicated to improving healthcare through the preparation of systematic reviews, has focused on the systematic electronic searching of Medline and EMBASE and the systematic handsearching of over 2000 general and specialized healthcare journals for reports of randomized controlled trials. This involves reading each document in a journal to decide if a report is a randomized trial, according to Cochrane eligibility criteria.1 The efforts of the many volunteers working within the Cochrane Collaboration have added a substantial number of previously ‘buried’ reports of randomized controlled trials to the Cochrane Central Register of Controlled Trials (CENTRAL) published in The Cochrane Library. Some of these reports of trials may have been ‘buried’ as a result of inconsistencies in indexing (indexing bias), a lack of cover-to-cover indexing, or because they have been published in journals not indexed in the major healthcare databases such as Medline and EMBASE (database bias) or in journals published in languages other than English (language bias). A good example of language bias was provided in a study of 68 Spanish general medicine journals, which found that only six were indexed in Medline.2 The Bahrain Branch of the UK Cochrane Centre is contributing to efforts to minimize bias through a comprehensive handsearching of journals published in the Arab world.
Methods
All issues of the Annals of Saudi Medicine (July 1981 to April 2005) were searched by hand from cover to cover for reports of trials involving prospective assignment to one of two or more interventions. These were classified as randomized controlled trials (RCTs) or controlled clinical trials (CCTs) according to Cochrane eligibility criteria.1 The handsearcher classified reports of trials as RCTs if the author reported that the groups compared in the trial were established by random allocation. If the author(s) did not state explicitly that the trial was randomized but randomization could not be ruled out, the report was classified as a CCT. CCT was also applied to quasi- or possibly quasi-randomized studies where the method of allocation was known but not considered strictly random (e.g. alternation, date of birth, medical record number). Photocopies of the bibliographic details and of the pages describing the study design were sent to the UK Cochrane Centre for verification and processing for submission to the US Cochrane Center and publication in CENTRAL.EMBASE (via Ovid Web) and Medline were also searched to determine if reports found by the handsearch were already included in either of these databases. The Annals has been indexed in EMBASE since January 1990, but in Medline only since the beginning of 2004. We then determined the added value of the handsearch in minimizing the effects of indexing bias by assessing the sensitivity (proportion of total number of known randomized trials identified by the search)3 of (i) the EMBASE index term RANDOMIZED CONTROLLED TRIAL and (ii) the Medline index term RANDOMIZED CONTROLLED TRIAL. The “gold standard” for calcuation of sensitivity was the handsearch. Added value was defined as the total number of reports of trials published in these journals and not indexed as randomized controlled trials in EMBASE or appearing before indexing, and therefore not easily identified except through a handsearch of these journals.
Results
We found 26 RCTs and 3 CCTs by handsearching. The distribution by country of principal investigator was highest for Saudi Arabia (18), followed by Turkey (2) and Iran (2). Of the 26 reports of RCTs found by the handsearch, 21 were in EMBASE but only 8 of these had been given the index term RANDOMIZED CONTROLLED TRIAL (Figure 1). Six of the trials (all randomized controlled trials) identified in the handsearch were published before January 1990, so the sensitivity of the EMBASE search was 40% (8/20). The overall added value of the handsearch in relation to EMBASE was 21 additional articles found, so that 72% of controlled trials in the journal were discovered by handsearching. Only 3 of the 29 reports found by the handsearch were in Medline, but indexing started in January of 2004. The added value of the handsearch in relation to Medline was 26 additional articles found (90% of controlled trials identified by handsearching).
Figure 1
Added value of handsearching vs. search of EMBASE for reports of randomized controlled trials and controlled clinical trials in the Annals of Saudi Medicine.
Discussion
To minimize bias due to the selective availability of data, systematic reviewers need to identify as many relevant studies as possible to provide reliable evidence on which to base healthcare decisions. However, the identification of trials from bibliographic databases can be problematic.4 When searching EMBASE or Medline for reports of randomized controlled trials, it is not advisable to rely solely on the term RANDOMIZED CONTROLLED TRIAL. Our study confirmed that the sensitivity of the term was poor. Only 8 of the 20 (40%) reports of trials found since 1990 by the handsearch were also in EMBASE. However, the term was only introduced in 1994 and so would not have been applied to some of the reports.Because of the lack of availability of appropriate indexing terms and inconsistencies in indexing (indexing bias), The Cochrane Collaboration has carried out systematic electronic searches of Medline and EMBASE using extensive search strategies designed to be sensitive, i.e. to avoid missing reports of trials. The reports of trials, which were identified by an assessment of the titles and abstracts only, using these sensitive search strategies for Medline and EMBASE are included in CENTRAL. However, despite sensitive searching of electronic databases, it has been found that handsearching still provides additional reports of trials missed by the electronic searches.5 As a result of our handsearching of the Annals, 21 previously unidentified RCTs and CCTs are now included in CENTRAL. In addition, our findings are a useful contribution to the ongoing development of the Collaboration’s s investigation of terms that might be of potential value in retrieving reports of randomized trials.Further research is required to assess the quality of the trials identified by handsearching or by systematic electronic searching and to assess how many trials were duplicated. Additionally, comparisons need to be made in the quality of trials and the treatment effects of trials reported in Arabic with those reported in English to determine whether there might be differences which could lead to bias being introduced into reviews based exclusively on English language reports.6 In addition, it would be helpful if authors of trials report study designs clearly and comply with published guidance (CONSORT—Consolidated Standards of Reporting Trials)7 on the better reporting of randomized controlled trials, which would in turn help indexers to apply appropriate index terms and thereby improve the retrieval of reports in electronic searches.In conclusion, the handsearching programme of the newly established Bahrain Branch of the UK Cochrane Centre provides a valuable and unique contribution from the Arab region to the global effort by The Cochrane Collaboration. This will help to close the gap between the number of reports of trials that exist and the number of reports of trials accessible to authors of Cochrane reviews and others needing to make informed decisions about the effects of healthcare interventions. It will also contribute to a more comprehensive assessment of the biomedical research output of Arab countries.
Figure 1