Integrity of Databases for Literature Searches in Nursing: Avoiding Predatory Journals.

The quality of literature used as the foundation to any research or scholarly project is critical. The purpose of this study was to analyze the extent to which predatory nursing journals were included in credible databases, MEDLINE, Cumulative Index to Nursing and Allied Health Literature (CINAHL), and Scopus, commonly used by nurse scholars when searching for information. Findings indicated that no predatory nursing journals were currently indexed in MEDLINE or CINAHL, and only one journal was in Scopus. Citations to articles published in predatory nursing journals are not likely found in a search using these curated databases but rather through Google or Google Scholar search engines.

Research, evidence-based practice, quality improvement studies, and other scholarly projects typically begin with a literature review. In research, the review of the literature describes existing knowledge about the topic, reveals gaps and further research questions to be answered, and provides a rationale for engaging in a new study. In evidence-based practice, the literature review provides evidence to answer clinical questions and make informed decisions. Quality improvement studies also begin with a search of the literature to gather available knowledge about a problem and explore interventions used in other settings. The appearance of journals that are published by predatory publishers has introduced the danger that reviews of the literature include inadequate, poorly designed, and low-quality information being used as “evidence”—raising the possibility of risky and harmful practice. Researchers and authors should be confident in the literature they cite; readers should have assurance that the literature review is based on sound, authoritative sources. When predatory journals are cited, that trust is eroded. No matter what type of study or project is being done, the quality of literature is critical for the development of nursing knowledge and for providing up-to-date information, concepts, theories, and approaches to care.1

An effective literature review requires searching various reliable and credible databases such as MEDLINE (through PubMed or Ovid) and the Cumulative Index to Nursing and Allied Health Literature (CINAHL), among others that are relevant to the topic. The ease of searching using a web browser (now commonly referred to as “googling”) has increased the risk of finding sources published in predatory and low-quality journals that have not met the standards of research and scholarship that can be trusted as credible and reliable evidence.

The purpose of this article is to present an analysis of the extent to which predatory nursing journals are included in MEDLINE, CINAHL, and Scopus databases, used by nurse researchers and other nurses when searching for information, and in the Directory of Open Access Journals. This directory indexes “high-quality, open access, peer-reviewed journals” and should not include any predatory journals.2

Statement of Significance

What is known or assumed to be true about this topic?

The quality of nursing literature used is vital for the development of research studies, application of evidence in clinical settings, and other scholarly projects. Nurse scholars need to be confident as they search the literature that they are accessing sound information sources and not articles from predatory nursing journals, which do not adhere to quality and ethical publishing standards. Citations of articles in predatory nursing journals may be found when searching Google and Google Scholar, making these citations easy to access but potentially resulting in the integration of poor quality research into the nursing literature. On the other hand, searches through credible databases—MEDLINE, CINAHL, and Scopus—are less likely to yield citations from predatory publications.

What this article adds:

This study helps validate the trustworthiness of these databases for conducting searches in nursing.

PREDATORY JOURNALS

Many studies have documented the problem of predatory journals. These journals do not adhere to quality and ethical publishing standards, often use deceptive language in emails to encourage authors to submit their manuscripts to them, are open access but may not be transparent with the article processing charge, may have quick but questionable peer review, and may publish inaccurate information on their Web sites such as impact factor and indexing.3–6 Predatory publishing is an issue in many fields including nursing. In a recent study, 127 predatory journals were identified in nursing.7

Citations acknowledge the ideas of others and give credit to the authors of the original work. When articles are cited in a subsequent publication, those citations disseminate the information beyond the original source, and the article in which it is cited might in turn be referenced again, transferring knowledge from one source to yet another. When articles in predatory journals are cited, the same process occurs. Those citations transfer knowledge from the predatory publication beyond that source. Studies have found that authors are citing articles published in predatory journals in nursing as well as other fields.7–10 Nurse scholars need to be confident as they search the literature that they are accessing sound information sources and not articles from predatory journals.

NATIONAL LIBRARY OF MEDICINE INFORMATION RESOURCES

The National Library of Medicine (NLM) supports researchers and clinicians through its multiple health information resources including PubMed, MEDLINE, and PubMed Central (PMC). PubMed serves as the search engine to access the MEDLINE database, PMC, and books, chapters, and other documents that are indexed by the NLM. PubMed is free and publicly available: by using PubMed, researchers can search more than 30 million citations to the biomedical literature.11 The majority of records in PubMed are from MEDLINE, which has citations from more than 5200 scholarly journals. For inclusion in MEDLINE, journals are assessed for their quality by the Literature Selection Technical Review Committee.12 Five areas are included in this assessment: scope of the journal (ie, in a biomedical subject); quality of the content (validity, importance of the content, originality, and contribution of the journal to the coverage of the field); editorial standards and practices; production quality (eg, layout and graphics); and audience (content addresses health care professionals).

PMC includes journal citations and full-text articles that are selected by the NLM for digital archiving. To be included in PMC, journals are evaluated for their scope and scientific, editorial, and technical quality.13 Journals considered for inclusion are evaluated by independent individuals both inside and outside PMC.14 PMC serves as the repository for articles to meet the compliance requirements of the National Institutes of Health (NIH) and other funding agencies for public access to funded research. About 12% of the articles in PMC are deposited by individual authors to be in compliance with funders and 64% by publishers, scholarly societies, and other groups.15 Beginning in June 2020, as a pilot program, preprints reporting research funded by the NIH also can be deposited in PMC.16

CINAHL AND SCOPUS

The journal assessment and indexing processes for CINAHL and Scopus are similar to those used by the NLM. However, as private corporations, EBSCO (CINAHL) and Elsevier (Scopus) are not required to make journal selection processes publicly available or explicit. CINAHL has an advisory board for journal selection. A CINAHL representative provided the following criteria for indexing of journals in CINAHL: high impact factor; usage in reputable subject indexes (eg, the NLM catalog); peer-reviewed journals covered by other databases (eg, Web of Science and Scopus); top-ranked journals by industry studies; and article quality (avoiding low-quality journals) (personal communication, October 19, 2020).

Elsevier's Scopus provides a webpage referring to the journal selection and assessment processes. Journals being considered for indexing in Scopus are evaluated by the Content Selection and Advisory Board and must meet the following criteria: peer-reviewed with a publicly available description of the peer review process; published on a regular basis; has a registered International Standard Serial Number (ISSN); includes references in Roman (Latin) script; has English language titles and abstracts; and has publicly available publication ethics and publication malpractice statements.17

LITERATURE REVIEW

Studies have shown that in health care fields, researchers, clinicians, faculty, and students regularly search MEDLINE for their research and other scholarly and clinical information.18–21 De Groote et al18 found that 81% of health science faculty used MEDLINE to locate articles for their research. MEDLINE was used by the majority of faculty in each individual health care field including nursing (75%) and medicine (87.5%) for searching the literature and finding articles. In another study of 15 different resources, medical faculty and residents reported that PubMed was used most frequently for searching the databases of the NLM, primarily MEDLINE.20 Few studies have focused on the search practices of nurses. In a review of the literature, Alving et al22 found that hospital nurses primarily searched Google for information on evidence-based nursing. They used Google more than bibliographic databases.

The quality of content that is retrieved when using PubMed as a search engine is important considering its widespread use for accessing scholarly and clinical information in nursing and other fields. Manca et al23 reported that articles published in predatory journals were being retrieved when conducting searches using PubMed and were a concern for researchers. Based on their studies of predatory journals in neurology24 and rehabilitation,25 they concluded that predatory journals “leaked into PubMed” through PMC because of less stringent criteria for inclusion of journals.23 Citations to articles from predatory journals then could be found using the PubMed search engine. However, in a letter to the editor, Topper et al26 from the NLM clarified that individual articles published in predatory journals might be deposited in PMC to meet the requirements of research funding and be searchable in PubMed. Topper and colleagues make a clear distinction between journals indexed in MEDLINE or PMC and citations of individual articles that were deposited in PMC to meet funder requirements.

Purposes

The aim of this study was to determine whether predatory nursing journals were included in databases used by nurse researchers and other nurses when searching for information. These databases included MEDLINE (searched via PubMed), CINAHL (EBSCO), and Scopus (Elsevier) and in the Directory of Open Access Journals.

METHODS

In an earlier study, 127 predatory nursing journals were identified and assessed for characteristics of predatory publications. That dataset was used for the current study. For each predatory nursing journal, information was retrieved from the NLM Catalog, Ulrichsweb, and journal and publisher Web sites. Ulrichsweb27 provides bibliographic and publisher information on academic and scholarly journals, open access journals, peer-reviewed titles, magazines, newspapers, and other publications. Journal titles of the predatory journals were often similar to nonpredatory journals and could be easily mistaken. To ensure accuracy, the information for each journal was checked for consistency between these sources using the ISSN, exact journal title, and publisher name. The purpose of an ISSN is to identify a publication and distinguish it from other publications with similar names. An ISSN is mandatory for all publications in many countries and having one assigned is considered a journal best practice.28 For each predatory journal, the following data were collected if available: complete journal title; abbreviated journal title; acronym; ISSN (electronic and/or print); DOI prefix; publisher name and Web site URL; NLM index status; number of predatory journal articles cited in MEDLINE and PMC (when searching using PubMed), in CINAHL, and in Scopus; if the journal was indexed in the Directory of Open Access Journals; status in Ulrichsweb; and Google Scholar profile URL.

Counts of articles cited were checked individually by journal title, publisher, and/or ISSN. Once ISSNs (both electronic and print where available) were assembled, a search algorithm was created, which included all retrieved journal ISSNs. MEDLINE was searched via PubMed using a combination of NLM journal title abbreviations and ISSNs. CINAHL, Scopus, and the Directory of Open Access Journals were searched using a combination of ISSN, journal title abbreviation, full title, and publisher. Results were visually inspected for accuracy and alignment with dataset fields.

Data analysis

Data were collected between January and April 2020. Data were entered into an Excel spreadsheet and organized by predatory journal name; abbreviated journal title; acronym; ISSN (electronic, print); DOI prefix; Web site URL; entry in NLM Catalog (yes/no); index status; number of articles cited in PubMed, CINAHL, and Scopus; Directory of Open Access Journals (included/not included); Ulrichsweb status (active/ceased); publisher; and Google Scholar profile URL. Frequencies and medians are reported.

RESULTS

Of the 127 predatory nursing journals in the dataset, only 102 had ISSNs to use for the search. Eighteen of the journals had records in the NLM Catalog, but only 2 of those had ever been indexed in MEDLINE, and neither are currently indexed. These 2 journals had been published earlier by a reputable publisher but then were sold to one of the large predatory publishers. The NLM Catalog record for these journals indicates that citations of articles from them appeared in MEDLINE through 2014 for one of the journals and 2018 for other, but following their transition to the new publisher are no longer included. Consistent with the MEDLINE results, these same 2 journals had been indexed in Scopus as well. Citations of articles from one of these journals were added to Scopus up to 2014, with no articles cited thereafter. Articles from the second journal continue to be added through 2020. One additional journal from the predatory journal dataset is currently in Scopus, however, only through 2014. None of the predatory nursing journals were indexed in CINAHL based on full journal title, title abbreviation, ISSN, or publisher. Two journals in the dataset were found in the Directory of Open Access Journals.

When searching PubMed, we found citations of articles from 16 predatory nursing journals. The number of citations ranged from 1 to 372 citations (from one of the journals indexed earlier in MEDLINE but sold to a predatory publisher). The second highest number of citations (n = 168) was of articles from a predatory nursing journal that had been depositing articles in PMC (and thus were retrievable when searching PubMed) but is no longer adding new material to PMC. The other citations were of articles deposited in PMC to meet requirements of NIH and other research funding. The predatory journals in which these articles were published, however, are not indexed in MEDLINE or PMC.

There were no articles from predatory nursing journals cited in CINAHL. Scopus has citations from the 2 predatory nursing journals that are no longer indexed there: 616 that were published in one of the journals and 120 from the other. Articles from a third predatory nursing journal in the study dataset, which is currently indexed in Scopus, totaled 173 (see Table).

This analysis documented that none of the predatory nursing journals in the study dataset were currently indexed in MEDLINE or CINAHL, and only one journal is still in Scopus. Most of the citations of articles from predatory journals found in a search of these databases are from earlier years before the journals were sold to one of the large predatory publishers. Other citations are to articles deposited in PMC in compliance with research funder requirements.

DISCUSSION

By using PubMed as a search engine and entry point to the databases of the NLM, researchers can search millions of records included in MEDLINE, or in process for inclusion, and articles from PMC deposited by publishers or authors for compliance with funders. Six million records, and about 5500 journals, can be searched in CINAHL Complete,29 and Scopus, the largest of the proprietary databases, provides access to 24000 journals and 60 million records.30 Results from this study show that very few articles published in predatory nursing journals find their way into a search done using PubMed and Scopus and none into CINAHL.

In a prior study, 814 citations of articles in predatory nursing journals were found in articles published in nonpredatory nursing journals.7 Based on this current study, the conclusion can be made that these citations are not coming from searches in MEDLINE/PubMed, CINAHL, or Scopus and are likely from searches done using Google or Google Scholar as the search engine. The databases examined in this study are curated by organizations with a vested interest in maintaining and improving the quality of the research literature in those databases.

Searching multiple databases using different search engines can be frustrating and time consuming. There is overlap among MEDLINE, CINAHL, and Scopus. However, these are curated databases and, as this study found, are unlikely to return many, if any, predatory citations as part of the search results. Still, it falls on the searcher to eliminate duplicates and redundant citations. Further, certain types of literature, such as theses, dissertations, and fugitive (or “gray” literature),31 are unlikely to be found in any of these databases, even though those citations may be important or relevant sources. Given this, it is easy to understand the intuitive appeal of Google Scholar, which provides “one stop shopping”: “From one place, you can search across many disciplines and sources: articles, theses, books, abstracts and court opinions, from academic publishers, professional societies, online repositories, universities and other web sites. Google Scholar helps you find relevant work across the world of scholarly research.”32 Google and Google Scholar were founded with a mission to become the most comprehensive search engines in the world. While this allows someone to scour the World Wide Web and Internet for some of the most obscure facts available, at the same time, little is done to verify or validate the results that are returned. Thus, it falls on the searcher to be diligent and evaluate the results of a Google or Google Scholar search, which will include citations of articles in predatory journals. This is easily confirmed by the fact that many predatory journal Web sites promote the Google Scholar logo as a sign of indexing or a badge of legitimacy.

Another vexing issue that was revealed in this study is that of reputable journals that have been bought by predatory publishers. This study found 2 journals in this category. Brown33 reported on 16 medical specialty journals that were purchased from 2 Canadian commercial publishers by a predatory publisher. In all these cases, it is the same predatory publisher, although some of the purchases were made under a different business imprint, adding further confusion to an already muddied situation. Jeffrey Beall, who coined the term “predatory publisher” and maintained the blog “Scholarly Open Access” for almost a decade, was quoted by Brown33: “[The company] is not only buying journals, it is buying metrics and indexing, such as the journals' impact factors and listing in Scopus and PubMed, in order to look legitimate.” One positive finding from this study was that the 2 purchased journals that were identified were quickly de-accessioned by the NLM and are no longer indexed in MEDLINE, although citations from their pre-predatory era remain intact.

Recommendations

All of this presents a confusing picture, but it is possible to make some specific recommendations to aid researchers, clinicians, faculty, and students in their literature searches. First, become familiar with the journals and publications in your field. This is a basic foundation of scholarship. As you read articles, remember where they were published, learn journal titles, and focus on sources as well as the content. As you come across predatory journals in nursing and health care, make note of them and learn their titles too. Remember that many predatory journals adopt names that are intended to be confusing and may differ from a legitimate journal by only one letter, such as “Africa” and “African.”

Second, consider carefully how to approach your search from the outset. If you choose to start with MEDLINE (searched via PubMed), CINAHL, or Scopus, then you can have some assurance that the results will not return citations from predatory journals—although you should still verify every citation that you receive. On the other hand, Google and Google Scholar can be a “quick and easy” way to get started but will require that you carefully review and evaluate the results. If you need to venture to other more specialized databases, such as PsycInfo or ERIC (Education Resources Information Center), it is important to carefully inspect the results that you receive. To reduce the risk of including a predatory journal article in research, nursing scholars should use reputable bibliographic databases, which have clear criteria for journal indexing, for their searches.

Third, when you come across a journal title that is not familiar, take time to research it, visit the journal Web site and evaluate the information at the Web site, and determine whether it is a credible source to include in your results. If something seems irregular, then it is worth your time to do more investigating—either on your own or by enlisting the help of a knowledgeable colleague or librarian. Journals change publishers all the time, and while most of these business transfers are benign and probably will not impact you as an end consumer of the literature, that is not always the case. Likewise, the major publishers in the world today are large, multinational conglomerates that regularly spin off or purchase other companies. While this probably will not impact you on a day-to-day basis, it is important to investigate any irregularities when conducting a search of the literature.

Last, because these issues are complex and multifaceted, it is always wise to consult with a librarian who can assist you in every step of the search process. Their knowledge and expertise in information literacy, data sources, and searching techniques can help to ensure that you find the information you need from sources that are reliable and credible.

SUMMARY

Researchers, clinicians, faculty, and students need to be careful not to include citations from predatory sources in their literature searches and articles. Predatory journals publish low-quality studies and citing this work erodes the scholarly literature in nursing. The findings of this study offer some reassurance to those who search the professional nursing literature: if you begin a search in a database such as MEDLINE, CINAHL, or Scopus, then the results will probably not include citations to predatory publications. Google and Google Scholar searches, however, may very well include predatory citations, and in that case, it is the searcher's responsibility to carefully evaluate the output and discard findings from nonlegitimate sources. Enlisting the help of a librarian is always beneficial and highly recommended.

Table.

Citations to Articles From Predatory Nursing Journals

Predatory Nursing Journalsa	Number of Citations
	PubMedb	Scopus	CINAHL
A	372	616	0
B	168	173	0
C	12	0	0
D	7	0	0
E	5	120	0
F	3	0	0
G	3	0	0

Abbreviation: CINAHL, Cumulative Index to Nursing and Allied Health Literature.

aPredatory nursing journals with 3 or more citations to articles.

bSearch using PubMed.