Availability of renal literature in six bibliographic databases.

BACKGROUND: When searching for renal literature, nephrologists must choose between several different bibliographic databases. We compared the availability of renal clinical studies in six major bibliographic databases.
METHODS: We gathered 151 renal systematic reviews, which collectively contained 2195 unique citations referencing primary studies in the form of journal articles, meeting articles or meeting abstracts published between 1963 and 2008. We searched for each citation in three subscription-free bibliographic databases (PubMed, Google Scholar and Scirus) and three subscription-based databases (EMBASE, Ovid-MEDLINE and ISI Web of Knowledge). For the subscription-free databases, we determined which full-text journal articles were available free of charge via links to the article source.
RESULTS: The proportion of journal articles contained within each of the six databases ranged from 96 to 97%; results were similar for meeting articles. Availability of meeting abstracts was poor, ranging from 0 to 37% (P < 0.01) with ISI Web of Knowledge containing the largest proportion [37%, 95% confidence interval (95% CI) 32-43%]. Among the subscription-free databases, free access to full-text articles was highest in Google Scholar (38% free, 95% CI 36-41%), and was only marginally higher (39%) when all subscription-free databases were searched. After 2000, free access to full-text articles increased to 49%.
CONCLUSIONS: Over 99% of renal clinical journal articles are available in at least one major bibliographic database. Subscription-free databases provide free full-text access to almost half of the articles published after the year 2000, which may be of particular interest to clinicians in settings with limited access to subscription-based resources.

Introduction

The Internet has emerged as an important tool for modern clinical practice. It provides health professionals with access to current, best evidence and facilitates the translation of research to clinical care. According to a recent survey, nephrologists search for medical information to guide the treatment of patients at least once per month [1]. To access best evidence, physicians can no longer rely on browsing a few key journals as relevant literature is widely dispersed. In nephrology alone, relevant literature is published across over 400 journals [2].

Numerous tools are available to healthcare professionals when searching for primary literature online. These include subscription-free databases such as PubMed, Google Scholar and Scirus (which are freely accessible via the Internet), and subscription-based bibliographic databases such as EMBASE, MEDLINE and ISI Web of Knowledge. Subscription-free resources enable global access to important clinical information. This is especially important in developing nations where physicians and institutions may lack the funds needed to maintain database and journal subscriptions.

While nephrologists increasingly rely on bibliographic databases for clinical information, evaluations of database performance are limited [3, 4]. Evaluating the performance of a bibliographic database involves the consideration of several factors [5-8]. For example, an important article may not be retrieved from a database such as PubMed because (i) the article of interest is not contained in the database holdings, or (ii) the search strategy failed to retrieve the article. To thoroughly evaluate the search performance of a database, it is first necessary to consider the ‘content coverage’, which quantifies the availability of relevant literature contained in the database. To date, no such evaluation has been performed for renal content [9-14].

Here, we evaluate and compare the availability of renal literature in six major bibliographic databases (PubMed, Google Scholar, Scirus, EMBASE, Ovid MEDLINE and ISI Web of Knowledge). We searched each database for 2195 journal articles, meeting articles and conference abstracts pertinent to renal medicine. We compared the searching features and content coverage among the subscription-free and subscription-based databases, and also compared how often full-text journal articles were available for free in the subscription-free databases.

Materials and methods

Bibliographic database characteristics

We considered six databases: three subscription-free databases (PubMed, Google Scholar and Scirus) and three subscription-based databases (EMBASE, Ovid MEDLINE and ISI Web of Knowledge). We searched official home pages to determine the content and searching characteristics of each bibliographic database [15-22].

Set of renal studies

A content coverage analysis quantifies the proportion of important literature in a specific field that is contained in a specific database [6]. As users in different medical disciplines have specialized needs, discipline-specific evaluations are the most informative. To avoid bias, we identified studies independently of the bibliographic databases to be tested by extracting the citations of the primary studies included in high-quality, renal systematic reviews. A well-conducted systematic review uses a variety of methods to identify all high-quality primary studies for a particular clinical question.

We compiled renal systematic reviews from peer-reviewed journals using the EvidenceUpdates service [23]. This service directs users to high-quality systematic reviews and meta-analyses that meet strict methodological criteria and have a high potential for clinical relevance [16]. We searched the EvidenceUpdates service in November 2009 and selected the option to view all reviews for the discipline of nephrology; our search yielded 207 systematic reviews. Two nephrologists used a standardized checklist previously developed by our team to independently assess whether each systematic review was pertinent to renal care [24]. A sample of 151 renal reviews was identified and included in the study (Appendix eList 1). The reviews were published between 1995 and 2009; 146 (97%) reviews targeted therapies for renal conditions. For each included review, we extracted the citations of all the primary studies. This included abstracting the full title, including any non-English titles, author names, year of publication, page numbers and journal or meeting title. We also noted whether the citation referenced a journal article, an article from a meeting or a meeting abstract. In total, we extracted 2838 citations to primary studies from all reviews. After removing duplicates, 2195 unique citations remained, which included 1577 (72%) journal articles, 280 (13%) meeting articles and 338 (15%) meeting abstracts; all published between 1963 and 2008. Among the journal articles, 34 (2%) were published in a language other than English.

Study availability in each bibliographic database

We searched for each citation in each bibliographic database (all searches performed between March and October, 2010). Our method of searching was similar to previous coverage studies [12, 13]. Using advanced search features where necessary, one author searched for each study in each of the six major scientific databases: PubMed, Google Scholar, Scirus, EMBASE, Ovid MEDLINE and ISI Web of Knowledge. This involved various combinations of the manuscript's title (both English and non-English titles), the authors' names, journal title and the year published. All links to candidate matches to confirm a true match were followed. In Google Scholar, we always selected the option to view all versions of a candidate article, and all links were attempted. If a citation was not found in a database, a second rater performed further searches to confirm its absence.

For each found article, two raters recorded information about the citation (authors, journal name, issue, volume and date of publication) and whether the complete abstract was available. In addition, when searching for journal or meeting articles in the three subscription-free databases, the raters documented whether the link(s) provided to access the article resulted in free full-text viewing, or if payment or subscription was required for full-text viewing. When searching the subscription-free databases, we ensured that no-pay-for privilege access was used. For articles not found in any of the databases, we further searched the Cochrane Library database to determine whether the articles were included in their holdings.

Analyses

We compiled and analyzed all citations using Microsoft Excel 2007 (Microsoft Corporation, Redmond, WA). We compared the proportion of journal articles, meeting articles and meeting abstracts contained in the six databases. To assess potential biases in coverage, we further considered articles published in a language other than English and articles published before 1995 [8] (this year marked milestones in computer and Internet advancements: Yahoo! was incorporated and Microsoft introduced the Windows 95 operating system and Internet Explorer version 1.0) [17, 25]. For subscription-free databases, we also compared the proportion of full-text articles that were available free of charge. As PubMed is the most frequently used database by nephrologists [1], we further examined the articles not included in this database and analyzed whether other databases contained these missed articles in their holdings.

We used the chi-squared test to determine whether the proportions of articles available in the databases were statistically different. If a test yielded a significant result, we conducted pairwise tests between the database with the largest yield and the remaining four sources. To reduce the risk of Type I error, we used a conservative P-value of 0.01 to interpret significance for all comparisons. We used the Wilson's score method to calculate 95% confidence intervals (95% CI) for all proportions [26].

Results

General characteristics of the six databases are contrasted in Table 1.

Table 1.

Characteristics of six bibliographic resources

Characteristica	PubMedb	Google Scholarb	Scirusb	EMBASE (Ovid)c	MEDLINE (Ovid)c	ISI Web of Knowledgec
Content
Number of journals	Over 23 000 cited; 5511 indexed (through MEDLINE)	NA	NA	Over 5000	5511	23 000
Disciplines	Biomedical	Multidisciplinary	Multidisciplinary	Biomedical	Biomedical	Multidisciplinary
Period covered	1948–present	NA	NA	1974–present	1948–present	1864–present
Number of entries	20 423 752	NA	410 million science-specific web pages	Nearly 13 million	19 398 391	NA
Biomedical databases included	MEDLINE, OldMedline, ‘in-process’ and ‘out-of-scope’ citations	NA	PubMed, LexisNexis, ScienceDirect, other	EMBASE	MEDLINE, OldMedline, ‘in-process’ and other non-indexed citations	Web of Science, Biological Abstracts, BIOSIS Previews, ISI Proceedings, MEDLINE
Frequency of updates	Daily	Several times a week	NA	Daily	Daily	Daily
Developer/owner or provider and country	National Library of Medicine, USA	Google Inc., USA	Reed Elsevier, The Netherlands	Developer: Elsevier Provider: Wolters Kluwer Netherlands	Wolters Kluwer, The Netherlands	Thomson Reuters, Canada
Search Features
Number of entries which can be displayed/accessed for each search	No limit	1000	1000	No limit	No limit	No limit
Tracks the number of times articles are cited by other publications	No	Yes	No	No	No	Yes
Allows users to view citing articles	No	Yes	No	Yes	Yes	Yes
Allows use of controlled vocabulary (e.g. MeSH terminology)	Yes	No	No	Yes	Yes	Yes (limited)
Indicates whether articles are available as free full texts	Yes	Yes	No	N/A	N/A	N/A
Allows article citations to be imported into bibliography managers (e.g. Reference Manager)	Yes	Yes (can import only one citation at a time)	No	Yes	Yes	Yes
Provides searching by limits (e.g. age, publication type)	Yes	Yes (very limited)	Yes (very limited)	Yes	Yes	Yes
Allows linking to institutions for subscription access (e.g. link to university library)	Yes	Yes	Yes	Yes	Yes	Yes
Provides email alerts for prespecified searches	Yes	Yes (introduced in 2010)	No	Yes	Yes	Yes
Allows users to view related articles for an article of interest	Yes	Yes	Yes	Yes	Yes	Yes (not all records)
Allows access to search filters (e.g. clinical queries)	Yes	No	No	Yes	Yes	No
When searching, algorithm searches the full text of publications	No	Yes (not all records)	Yes (not all records)	No	No	No
Sorts results by relevance	No	Yes	Yes	Yes (very limited)	Yes (very limited)	No
Provides spell checking for misspelled search terms	Yes	Yes	Yes	No	No	No
Stores search history	Yes	No	No	Yes	Yes	Yes

NA, not applicable.

aAs of March, 2011.

bSubscription-free databases available for free access via the Internet.

cSubscription-based database; requires a subscription to access.

Content coverage

We describe the results separately for journal articles, meeting articles and meeting abstracts.

Journal Articles

The proportion of journal articles contained within the holdings of each of the six databases were similar, ranging from 96 to 97% (Figure 1); content coverage did not differ between the subscription-free databases and the subscription-based databases (P = 0.6). All databases contained similar proportions of articles published before 1995 (88–94%), while the few articles published in languages other than English (n = 34) were most prevalent in EMBASE (82%) and least prevalent in ISI Web of Knowledge (29%); P < 0.01. Five journal articles were not contained in any of the databases; we examined the methods of the originating reviews and determined that three of the citations originated from a specialized database in China [27], and two were found via hand searching [28, 29].

Fig. 1.

Proportion of all journal articles, older articles and non-English language articles included in the six bibliographic databases. Error bars represent 95% confidence interval calculated using the Wilson's score method. *Global chi-squared test: P = 0.6. **Global chi-squared test: P < 0.01; pairwise chi-squared tests compared with Google Scholar–EMBASE: P < 0.01, Scirus: P = 0.4, PubMed: P = 0.4, Ovid MEDLINE: P = 0.4, ISI Web of Knowledge: P = 1. ***Global chi-squared test: P < 0.01; pairwise chi-squared tests compared with EMBASE–Google Scholar: P < 0.01, Scirus: P = 0.5, PubMed: P = 0.5, Ovid MEDLINE: P = 0.5, ISI Web of Knowledge: P < 0.01. Detailed breakdowns of the formats (citation only, full abstract, subscription or payment full-text option, free full text) that the articles were found in the subscription-free databases are presented in the Supplementary data Figure S1.

Of the 1577 journal articles, 60 (4%) were not contained in PubMed, 43 (72%) originated from journals not indexed by the database and 14 (23%) were published in years when the source journals were not indexed by PubMed. Failure to locate the final three articles in PubMed resulted from errors in indexing. While the database indicated that the originating journal volumes were available (and other articles from the journals volumes were found), the three articles could not be located. EMBASE contained the largest proportion of articles not indexed by PubMed (73%; Table 2). When combined, EMBASE and PubMed contained 99% of all journal articles.

Table 2.

Proportion of articles not contained in PubMed that were contained in the other bibliographic databases

	Proportion (%) of articles (n = 60)
Ovid MEDLINE	0
Scirus	16
ISI Web of Knowledge	40
Google Scholar	53
EMBASE	73

Meeting articles and meeting abstracts

Table 3 lists the proportion of meeting articles and meeting abstracts included in the six databases. While over 98% of meeting ‘articles’ were contained in all the databases, the distribution of meeting ‘abstracts’ showed significant variation, ranging from 0 to 37% (P < 0.01). ISI Web of Knowledge contained the largest proportion of meeting abstracts (37%, 95% CI 32–43%). The largest source of meeting abstracts among those found by ISI Web of Knowledge originated from the American Society of Nephrology Renal Week meetings. When combined, 57% of abstracts were not included in any of the databases. Further investigation revealed that the Cochrane Library database included 162 of the 193 (84%) abstracts not found by any of the databases.

Table 3.

Proportion of meeting articles and meeting abstracts included in the six bibliographic databases

	Proportion (%) of articles (95% CIa)
	Meeting articles (n = 280)b	Meeting abstracts (n = 338)c
EMBASE	100.0 (98.7–100.0)	0.0 (0.0–1.1)
Scirus	99.6 (98.0–99.9)	4.2 (2.5–6.8)
ISI Web of Knowledge	99.3 (97.4–99.8)	37.3 (32.3–42.6)
Google Scholar	99.3 (97.4–99.8)	9.5 (6.8–13.1)
PubMed	98.9 (96.9–99.6)	0.0 (0.0–1.1)
Ovid MEDLINE	98.9 (96.9–99.6)	0.0 (0.0–1.1)

aCI, confidence interval calculated using the Wilson's score method.

bGlobal chi-squared test: P = 0.6.

cChi-squared test: P < 0.01; all pairwise chi-squared tests compared with ISI Web of Knowledge: P < 0.01.

Availability of free full-text publications

Among the subscription-free databases, free full-text access to journal articles (via links to external sources offering free full-text access) was greatest in Google Scholar (38%), followed by Scirus (36%) and PubMed (33%); P < 0.01 (Figure 2). A similar pattern was seen for meeting articles (P = 0.3; Figure 2). Altogether, free full-text access to journal articles among the subscription-free databases was 39%, with free access increasing after the year 2000 when almost half of the articles were available for free (Figure 3).

Fig. 2.

Proportion of journal articles and meeting articles available as free full texts in Google Scholar, Scirus and PubMed. Error bars represent 95% confidence interval calculated using the Wilson's score method. *Global chi-squared test: P < 0.01; pairwise chi-squared tests compared with Google Scholar–Scirus: P = 0.2, PubMed: P < 0.01. **Global chi-squared test: P = 0.3.

Fig. 3.

Proportion of journal articles available as free full texts in Google Scholar, Scirus or PubMed by era.

Discussion

In this study, we evaluated the availability of high-quality, renal practice evidence in six major bibliographic databases (PubMed, Google Scholar, Scirus, EMBASE, Ovid MEDLINE and ISI Web of Knowledge). We found that over 99% of renal journal articles were contained in at least one of these databases.

We were surprised to learn that the availability of renal journal articles did not differ between subscription-free databases and subscription-based databases. Further, nearly 40% of full-text journal articles could be accessed freely through links provided by the subscription-free databases. Free access to full-text articles increased from 20 to 49% for articles published before 1980 compared with articles published in 2000 or later. These findings indicate that journals appear slow to grant free access to older materials, which typically involves conversion of print materials to electronic PDF files. The increasing availability of free access to literature is particularly important for developing nations where physicians and institutions may lack the resources needed to maintain subscriptions to databases and journals [30]. Even in developed countries such as Canada and the United States, the burden of paying for knowledge is felt: some academic databases and journals have raised their fees for university subscriptions by up to 400% over a 1-year period [31-33].

In contrast to the excellent coverage of journal articles, we found that meeting abstracts are poorly indexed. Abstracts were nonexistent in most databases, with the exception of ISI Web of Knowledge, which indexed 37% of meeting abstracts. This finding suggests that authors of systematic reviews are not solely relying on bibliographic databases but are also searching conference websites and supplements as is recommended. To assist with comprehensive searching for reviews, The Cochrane Collaboration provides detailed guidelines on searching for gray literature (including hand searching) through the Cochrane Handbook, available online at http://www.cochrane-handbook.org/.

To our knowledge, this is the first study to quantify the content coverage of renal literature in the major bibliographic databases. While previous evaluations of other medical disciplines showed greater variation in content coverage, we found very little variation with respect to journal articles [13, 14]. For the purpose of conducting systematic reviews, numerous studies recommend searching several bibliographic databases to develop comprehensive reviews [13, 34–40]. Although our results indicate only a minor increase in content yield when searching multiple databases (the greatest increase in yield was only 3% when combining MEDLINE and EMBASE), these results only reflect differences in the renal content of the databases. Differences between this study and earlier evaluations may be due to improved database performance over time, possibly due to competition between databases.

We designed this study to overcome the limitations of previous publications. We used systematic reviews to identify an unbiased, representative set of renal clinical studies. However, due to the lengthy systematic review process, publication process and required time to complete the current study, this resulted in articles being tested that were at least 2 years old and were cited by at least one publication. Nonetheless, we reduced the likelihood that possible selection bias influenced the results by using this objective method of sampling renal literature. Because we selected primary studies included in systematic reviews, we were unable to test the coverage of articles that some physicians find relevant (studies of lower methodological quality, narrative reviews, case reports, animal studies, commentaries etc.). However, we engaged in the widely accepted principles of the hierarchy of evidence to identify the most important articles to retrieve when answering clinical questions. Having chosen articles from English language systematic reviews, we likely limited the number of non-English renal articles in our sample (only 34 journal articles were published in a language other than English). Thus, our results may not generalize well to non-English renal content. Finally, we acknowledge that the Internet is a dynamic environment and encourage future research to evaluate whether the performance of databases varies with time, as well as to monitor the improvement of the online sources.

As shown in Table 1, bibliographic databases vary with respect to search features and each offers certain advantages. All the databases allow access to institutional library links, which can be used to retrieve the full text or publications within their library's subscription packages. In general, the subscription-based databases provide a more structured interface, which enables more complex searches, such as combinations of two or more searches, use of complex search filters and a controlled vocabulary. They also allow a user to search several databases simultaneously and provide more options for efficient data management. In contrast, the subscription-free databases, such as Scirus and Google Scholar, offer a simpler search interface, but limit the number of entries, which can be displayed or accessed for each search. With the exception of PubMed, subscription-free databases also lack specialized features for clinicians. PubMed, EMBASE and MEDLINE provide indexed content that is directly relevant to clinicians, including extensive clinical controlled vocabularies (e.g. MeSH), limits, access to discipline-specific search filters and the ‘Clinical Queries’ filters, which help clinicians find articles of high methodological quality for clinical questions of therapy, diagnosis, prognosis and etiology [15, 41–43].

Content coverage analysis is only the first necessary step in evaluating the quality of bibliographic databases. A comprehensive content coverage of a database does not directly translate into better search performance; indexing database systems, syntax, searching capabilities and available limits all influence the successful retrieval of evidence. For example, the EMBASE database is well known for its extensive indexing of studies in pharmacology and drug therapy: a search performed in March 2011 for the term ‘valsartan’ identified 6620 records in EMBASE and only 1852 in PubMed. Future research should focus on the other properties of bibliographic databases that are of interest to clinicians, namely the ability to retrieve relevant literature when conducting a search, the relative effort required to retrieve this literature and the ability to find newly published evidence.

In conclusion, the major bibliographic databases appear sufficient in indexing the majority of renal articles. However, poor indexing of meeting abstracts means that a comprehensive search will require searching other sources in addition to bibliographic databases. Finally, nephrologists searching for full-text articles should first search the subscription-free databases before paying for access.

Supplementary Data

Supplementary data is available online at http://ckj.oxfordjournals.org.

Conflict of interest statement

None declared.