Assessing the impact of predatory journals on policy and guidance documents: a cross-sectional study protocol.

INTRODUCTION: Many predatory journals fail to follow best publication practices. Studies assessing the impact of predatory journals have focused on how these articles are cited in reputable academic journals. However, it is possible that research from predatory journals is cited beyond the academic literature in policy documents and guidelines. Given that research used to inform public policy or government guidelines has the potential for widespread impact, we will examine whether predatory journals have penetrated public policy. METHODS AND ANALYSIS: This is a descriptive study with no hypothesis testing. Policy documents that cite work from the known predatory publisher OMICS will be downloaded from the Overton database. Overton collects policy documents from over 1200 sources worldwide. Policy documents will be evaluated to determine how the predatory journal article is used. We will also extract epidemiological details of the policy documents, including: who funded their development, the discipline the work is relevant to and the name of the organisations producing the policy. The record of scholarly citations of the identified predatory articles will also be examined. Findings will be reported with descriptive statistics using counts and percentages. ETHICS AND DISSEMINATION: No ethical approval was required for this study since it does not involve human or animal research. Study findings will be discussed at workshops on journalology and predatory publishing and will be disseminated through preprint, peer-reviewed literature and conference presentations. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

We use Overton, a new database that is the most comprehensive source of international policy documents, to assess the penetration of predatory journals.

We employ rigorous study design and data extraction standards to ensure reliability of data.

Use of the single largest predatory publisher, OMICS, to determine penetration of predatory journals in policy literature may not be representative of broader predatory publishers/journals.

Introduction

Predatory journals and publishers are defined as ‘entities that prioritise self-interest at the expense of scholarship and are characterised by false or misleading information, deviation from best editorial and publication practices, a lack of transparency, and/or the use of aggressive and indiscriminate solicitation practices’.1 In contrast, legitimate journals generally publish according to a set of best practices.2 Predatory, or illegitimate journals, often do not perform the expected quality checks that legitimate journals conduct. For instance, predatory journals typically forego the peer-review process and publish low-quality work.3 Predatory journals may also accept research that has not received ethical approval.4 Additionally, these publications are increasingly finding their way into reputable databases,5–7 which is problematic.

Despite growing evidence for the negative impact of predatory journals, the extent to which these journals are being incorporated into the scholarly record through citation has only recently been examined.4 8 9 To date, this research has focused on the citation of articles published in predatory journals by authors publishing in legitimate ones.10–12 It is also possible that predatory journals are cited beyond the traditional academic literature in policy guidelines or guidance documents. In the context of healthcare, an individual practitioner reading a flawed article may have a relatively limited impact, but policy guidelines have the potential to influence clinical practice on a much larger scale. Flawed guidelines, like those historically encouraging widespread opioid prescribing, can stem from low-quality untenable and misleading research.13–16 These guidelines can cause serious systemic issues, such as for patient safety, resource use and economic burden.13–15

Pilot work from our group identified multiple policy documents or guidelines citing randomised controlled trials published in predatory journals; similarly, these findings raised the possibility that science published in predatory journals influences policy and guidelines more generally. Here we present a protocol to broadly evaluate the impact of predatory journals on policy and guidance documents.

Objective

The objectives of this study are to: (1) determine whether predatory journal articles are cited in policy and guidance documents; and (2) if so, describe how predatory journal articles are cited and used in these documents; and (3) assess whether or not correlations or patterns exist between how predatory journal articles are cited and used in these policy documents, and how these same predatory journal articles are cited and used in the traditional scientific literature.

Methods

This protocol will be registered on the Open Science Framework and all study materials and data will be shared openly. Our final report will use elements of the Strengthening the Reporting of Observational Studies in Epidemiology reporting guideline17 that are applicable to our study design, where policy documents are analogous to participants. This study does not involve patient or community partners. This is a descriptive study with no hypothesis testing.

Policy sampling strategy

We will use the Overton database to identify policy and/or guidance documents which have cited articles published in predatory journals.18 The Overton database is the largest searchable index of policy documents, guidelines, think tank publications and working papers. Launched in 2019, it collects data from 182 countries and more than 1200 sources worldwide including governments, intergovernmental organisations, and some think tanks and non-governmental organisations. As of November 2021, it contains over 5.1 million policy documents. Overton defines policy documents broadly as ‘documents written primarily for or by policymakers’, and aims to collect working papers, reports, case studies, policy briefs, testimony, clinical guidelines and government documents that are not behind a paywall or clearly in the scholarly record (eg, journal articles). Overton organises policy documents by topic and identifies relationships with other policy documents, academic research and media. This database allows researchers, think tanks, funding agencies and other users to search these documents to determine where research is being cited and how it is shaping policy.

Predatory journal sampling strategy

At present, there is no broadly accepted standard practice to identify predatory journals. The nature of predatory journals is that they are not systematically indexed, meaning they can be hard to identify and retrieve. Lists of journals deemed predatory that are available online are not transparently or systematically curated,19 or are not publicly available.20 To keep our research to a manageable and feasible scope, we will use journals published by OMICS Group, an established predatory publisher. OMICS Group was fined US$50 million in 2019 by the US Federal Trade Commission for deceptive business practices including falsely claiming peer review, listing scientists as journal editors without their knowledge, using fake impact factors, and unauthorised use of logos implying that journals were indexed in the US National Library of Medicine, PubMed Central and Medline.21 22 Currently, OMICS Group has over 700 open-access journals organised by subject area. Overton allows for identification of policy documents that have cited work by the OMICS Group. We will export all policy documents that have cited OMICS articles into Distiller SR (Evidence Partners, Ottawa, Canada), which is cloud-based and audit-ready software that will allow for screening and data extraction.

Eligibility criteria

The Overton database includes some documents which do not meet our definition of policy documents. As a result, we will screen exported documents to ensure they meet our definition of policy documents, which we have based off definitions from Overton and the US Department of Commerce. We will include statements from any agency or organisation that put forward a policy on a statutory, regulatory, or technical issue, or interpretation thereof, or documents otherwise primarily intended for policymakers.23 This definition will include: working papers, briefs, clinical guidelines, regulatory submissions, and other documents released by these agencies that report on their policies or are otherwise obviously intended for policymakers. We will exclude: original scientific research (articles, journals and conference proceedings, with the exception of clinical guidelines published as research articles), and documents unrelated to policy and their audience. We will not exclude documents based on language of publication; however, for those not in English, we will take note of which language they were written in. If two team members are not fluent in the language used in the policy, manually extracted questions for that document will be marked as unable to assess. We will include policy documents published since 2012 when the concept of predatory journals was first widely disseminated.24

Data extraction

Once we have identified our sample of policy/guidance documents citing articles from the OMICS Group journals, we will extract all available meta-data from the Overton database on these documents. This includes both information on the policy/guidance document and the OMICS journal article that was cited. For the policy/guidance document, these data include: the title of the document, the URL, the source (eg, WHO, Center for Disease Control, including categorisation of the source and country if applicable), unique policy document ID number, citations by other Overton policy documents and the date of publication of the document. For the journal article cited, this information includes but is not limited to: the title of the article, the DOI, the journal it was published in and the publication year. From the policy documents, we will manually extract and code the funding source, and discipline of the policy (eg, health, non-health). For the articles, we will extract information including: article type (eg, clinical trial, review, cohort study, etc), affiliations of corresponding authors, number of authors, funding, ethical approval, population (eg, adults), discipline of article (eg, health economics), total number of references, sample size and how the predatory article is used in the policy document. Details on how the predatory article is being cited represent a key component of our study. This will provide a basic understanding around whether articles are being cited simply as background information, or if they are being used in a manner that may directly inform policy recommendations. Data automatically extracted from Overton will be manually reviewed to screen for irregularities (ie, inconsistencies or oddities in the data). Should irregularities be identified, we will confirm the automatically extracted information manually. For example, in pilot searches of Overton, we noted that some automatically generated article DOIs are incorrect. The full-text article will be reviewed to manually check the DOI for accuracy. If more than one version of a policy document is found, the most recent version (or the English version if published in multiple languages) will be retained. In order to reduce bias and between extractor variation, we have created standardised and objective data extraction forms to collect the above-mentioned data. These are available in online supplemental appendix 1. We have also created a ‘codebook’ with objective question and answer descriptions, and will require all extractors to pilot test the extraction forms on the same 10 articles, to ensure consistency, as part of their training. Once the data extraction forms have been pilot-tested and agreed on by the study team, data extraction will be performed by one reviewer and audited by a second reviewer. If necessary, any conflicts will be resolved by discussion or by senior investigators—KDC and MML.

Once the cited predatory articles have been identified using the above-mentioned method and workflow, we will use their DOI to obtain information on whether the articles have been cited, and if so, by whom. To do so, at least two sources would be used among Google Scholar, Scite and The Lens. We will determine the choice of these sources on the basis of two criteria: (1) the possibility to freely reproduce data without infringing on possible copyright issues and (2) the possibility of implementing automated or semiautomated procedures for extracting scholarly citations. We will describe the scope and limitations of the retained sources for this extraction. Other types of data including self-citations and altmetrics might be collected depending on the tools that are retained for data extraction and potential copyrighted information. We will make sure that the scholarly citations extracted on the basis of DOIs match the records of predatory journal articles found in Overton by reviewing their associated metadata (journal title, article title, publisher, etc). As citation counts are dynamic, we will record the dates when data were extracted for individual entries. We will use the label ‘unworkable’ for DOIs that are not found in the chosen sources for scholarly citation extraction. These data will allow us to determine potential correlations between the number of citations of predatory journal articles in policy documents and the number of scholarly citations of these same articles. We anticipate that the data will not be normally distributed; therefore, an inverse transformation will be applied to the data. A Pearson correlation test will then be used to test for an association between the number of times an OMICS article is cited in Overton and the number of times the article was cited in the academic literature.

Data analysis

Both characteristics of the citing policy documents as well as the articles published in an OMICS journal, and their records of scholarly citations, will be summarised with descriptive statistics using count data and percentages. There will be no hypothesis testing performed; the study will exclusively be descriptive.

Study timeline

The Overton database search was conducted on 18 June 2021. Exporting Overton data, preparing data for extraction, and recruiting and training extractors will be completed in February 2022. Extraction will follow with completion expected by June 2022. Data analysis is expected to be complete by August 2022.

Patient and public involvement

This work does not involve any patients or members of the public.

Ethics and dissemination

Ethics

No ethical approval is required for this study since it does not involve human or animal research.

Dissemination

We aim to make all study results readily accessible to researchers, policymakers, and the community at large to raise awareness around predatory publishing and associated risks. Study findings will be discussed at workshops on journalology and predatory publishing and will be disseminated through preprint, peer-reviewed publications and conference presentations.

Discussion

Results from this study will provide critical insight into the citation of predatory journals in policy and guidance documents. Determining whether predatory journals are cited in policy/guidelines is important for several reasons. First, if predatory journals are cited in policy/guidelines, low-quality research may influence policymaker decisions that could have repercussions in broad disciplines such as healthcare delivery or economic and climate reform. Second, citation of these journals in policy documents increases the credibility of these journals, which may sow confusion for the public, researchers and clinicians. This risk of confusion may be amplified or vary according to the record of scholarly citations of these predatory journal articles.

The proposed study is not without its limitations. Although Overton is the largest database of policy documents and guidelines currently available, it is not a complete list of all policy documents. Overton tracks over 1200 various policy sources, but we may not be able to generalise our findings to policy documents it does not capture. To obtain our sample of predatory journals, we selected journals from the known predatory publisher, OMICS. OMICS is a very large predatory publisher—a search of The Lens estimates OMICS published 92 662 journal articles from 2012 to 2019—but it is not known what proportion of predatory journals OMICS represents; journals from this publisher may not be representative of predatory journals more broadly. Due to the nature of predatory publishing, it is a challenge to estimate the true scale of the problem, with the last key study in 2015 unlikely to reflect the current landscape.25 This means we will only identify a portion of the predatory journal articles cited in policy documents, and cannot completely generalise the citation patterns observed among OMICS journals to a broader sample of predatory journals/publishers. Finally, we are not evaluating the quality of cited OMICS articles; certainly high-quality work (that should be cited) may find its way into OMICS journals; however, we would note that evaluations to date have demonstrated that articles published in potentially predatory journals are of extremely low quality.4 26 An evaluation of these articles may be done in a future study. Despite these limitations, the proposed work is the first to address this question, and will thus provide initial evidence of the penetration of predatory journals in public policy.