Impact factor: Mutation, manipulation, and distortion.

Currently, there is an increased dependency on the impact factor in scientific research publications. Sometimes the overzealousness in implementing the impact factor value to a publication is detrimental to the growth of the scientific authors, especially the junior lot, for no fault of theirs. The inept and myopic application of the impact factor defeats the purpose of making the value judgment and hence has been criticized by many learned authors. The scientist Eugene Garfield, who formulated the impact factor, feels wrong that it is being not judiciously used. A corollary is the invention dynamite by Alfred Nobel which instead of helping, aiding, and reducing human effort and endeavor it was more misused for human annihilation and pained the scientist who gave it to the world. The authors reexamine the application of impact factor to scientific manuscripts for rightful application of the value judgment. Copyright:

“In 1955, it did not occur to me that the impact would one day become so controversial. Like nuclear energy, the impact factor is a mixed blessing. I expected it to be used constructively while recognizing that in the wrong hands it might be abused.” –

Background

The journal impact factor (JIF) has become an important indicator of the quality of research publication. Hence, while considering the research impact, most consider the JIF as a barometer of research. Although it was never intended to be used to evaluate individual scientists, but rather as a measure of the quality of academic journals, JIF has been increasingly misused in this way.[1] Researchers are often ranked on the basis of their publication in the journals with high IF, and in some countries, publication in a journal with an impact factor <5.0 is officially of no value.[2] Science ministries in certain countries offer cash rewards to scientists publishing in high IF journals viz Nature, Science, Cell etc.[34] Thus, it has become imperative for scientists to publish their work in journals with high IF. However, serious concerns have been raised about the use of JIF as a surrogate marker for the quality of research, individual articles, or a researcher itself.[25]

Impact factor calculation

The JIF was devised by Eugene Garfield in 1955 to help research libraries differentiate between journals when deciding which one to subscribe to.[6] The term, IF was first used in 1961, after publication in Science Citation Index (SCI) in 1963, and the first ranking of the journals on the basis of IF was published in 1972. As a part of SCI and the social sciences citation index, Thomson Reuters began publishing Journal Citation Report (JCR) annually in 1975.[6] The JIF is actually a measure of how frequently the articles published in that journal are cited. The IF of a journal for any specific year is calculated by dividing the total number of citations received by the articles published in the journal during the preceding 2 years (numerator), by the total number of articles published in the same 2 years (denominator).[7] For example, if the IF of a journal is 3.0 in 2015, it reflects that on average the articles published in 2013 and 2014 were cited thrice in the collection of all Thomson Reuters indexed journals published in 2015.

Impact factor distortions

Scientific publications contribute to the dissemination of the research findings and knowledge to improve life. However, these fundamental tenets of research publications have probably been forgotten in IF mania, and misuse of JIF is extensive. Based on the notion that a journal is representative of its articles, publishing in high-IF journal is considered to evaluate the author's scientific achievement.[1] This, in turn, has led to IF-based assessment for the appointment, allocation of research grants, and academic advancement of the researchers.[89] Researchers also tend to publish their manuscripts in journals with high IF and are more concerned about “where they publish, rather than what they publish.”[10] The contradiction is evident; on one hand we want our journals to attain international standards but when it comes to publishing our exemplary findings, we prefer international journals rather than the Indian ones.[11] This takes even longer for our journals to get recognition and improve IF. The 2014 edition of JCR contained 8474 science and technology journals from all over the world, including 98 from India. Among 98 of these Indian journals only two had IF >2.000.[12]

Since the JIF is calculated over a period of 2 years after being indexed in Thomson Reuters, a recently launched journal or a journal not indexed in Thomson Reuters cannot have an IF. Moreover, there are many peer-reviewed journals that are not indexed in Thomson Reuters and, therefore, do not have an IF. Researchers dislike publishing their findings in journals with no or low IF. Taking advantage of such existing IF craze, many agencies have started allocating fake IFs to the journals on payment basis, which may resemble the original IF.[1314] These bogus IF agencies seem to be hand in glove with “predatory journals”[15] displaying fake IFs predominantly on their websites, and we all have our mailboxes filled with their e-mails with a soliciting manuscripts. The sole aim of these dubious journals is to earn from publishing fees. Due to this demand and supply culture, budding researchers and even academic institutions fall prey to them. However, some researchers knowingly use sham publications and fake ” scientometrics” for their academic advancement based on the poor quality articles posing serious threat to the academic standard and integrity.[15]

Impact factor manipulation

Over the years, critics have argued that the JIF, per se, may not reflect anything informative about the quality of empirical research.[9] It is not an appropriate metric to measure the scientific content of individual articles or a scientist's credibility, and if applied to individual researchers, publications, or grants, it exerts an increasingly detrimental influence on the scientific enterprise. There is a great degree of mutation and manipulation in the evaluation of IF, and we enumerate some of these in the following text.

Eugene Garfield, the inventor of the IF never predicted that it would be used in the scientific community as a criterion for judging the quality of a scientist and determining the provision for research grants. Unfortunately, the IF of a journal is not statistically representative of its individual articles and Garfield himself has reported a poor correlation between the IF of a journal and the actual citation rates of its articles.[6] It has been seen that citations of many articles may not peak until after the second year of publication and is beyond the brief period of time considered for calculating IF.[16] In fact, Lariviere and Sugimoto, in their six-point critique of a JIF, explain that a 2-year period for citations could accidentally favor certain disciplines over others.[17] Moreover, JIF can be skewed by publication of more reviews (which tend to be cited more frequently) or by self-citation of the articles. Recent example is of a journal Acta Crystallographica Section A: Foundations of Crystallography (pISSN 0108—7673) which had an IF of 2.051 in 2008, which changed to 49.926 in 2009 and then increased to 54.333 in 2010 and the IF in 2014 was 2.3074, reason being a single review article receiving a large number of citations.[7]

Journals are also under continuous pressure to raise their IF, which can lead to editorial misconduct. Sometimes a journal may request authors to include references from its own previous publications in order to inflate its IF.[18] Such practice was recently brought to light, where three Brazilian journals conspired to cite each other's published papers in a mutual effort to increase their JIF.[19] Moreover, some papers are cited multiple times for negative reasons and yet these negative citations contribute to improving JIF.[18] An article published in Science showed that many studies that have been proven to be fraudulent are not even retracted and continue to be cited.[20]

While calculating the JIF, only original papers and review articles are counted in denominator while all the published materials (editorials, letters to editor, news, book reviews, etc.) including original papers and review articles are accepted in the numerator. This significantly boosts the JIF. Interestingly, even some of the reputed journals such as Nature and Science have been found to do so in order to boost their JIF.[2122] Also, the continuous pressure for publication in high-IF journals leads to “performance anxiety” among researchers and they indulge in unethical publication practices (data falsification and fabrication).[2324] Such cases are mostly reported from countries where regulatory bodies demand academic faculty to regularly publish in high-IF journals. The pressure to publish creates a bias that discourages high-risk research and reduces the likelihood of unexpected breakthrough discoveries. Vannevar Bush had commented nearly 70 years ago that “Basic research is performed without thought of practical ends…. Many of the most important discoveries have come as a result of experiments undertaken with very different purposes in mind.”[25]

Suggestions for reforms

Scientist's unhealthy obsession with JIF has been widely criticized, yet many are trapped into this value system when submitting their own work or judging the work of others. A recent study by Madhan et al. pointed out that cumulative JIFs were still being utilized as a criterion for prestigious awards such as the Tata Innovation Fellowship, Innovative Young Biotechnologist Award, National Bioscience Awards for Career Development, and so on.[26] Similarly, the Indian Council of Medical Research routinely uses average JIF as a measure of the performance of its various laboratories.

In order to stop JIF misuse, the researchers should put a halt on the relentless chase for IF and rather focus on the originality and quality of their research work. In this regard, American Society of Microbiology (ASM), on 11th July 2016 announced to remove the IF from its journals and website, as well as from marketing and advertising, a move which was appreciated by many.[27] Of note are prestigious journals such as Nature, Science, The New England Journal of Medicine, The Lancet, etc., which have existed and prospered for long, some even for centuries, before the advent of IF.

The misuse of the JIF as a metric of an individual scientist's or article's importance has been decried in a consensus statement from the San Francisco Declaration of Research Assessment (DORA).[28] The aim of DORA was to put an end to the practice of using JIF as a valuation metric of individual researchers. The declaration states that “the impact factor must not be used as a surrogate measure of the quality of individual research article, to assess an individual scientist's contributions, or in hiring, promotion or funding decisions.” The DORA has made one general and 17 specific recommendations, which are enumerated in Table 1.

Table 1

Recommendations made by San Francisco Declaration of Research Assessment (DORA)

DORA recommendations[22]
General recommendation	Do not use journal-based metrics, such as JIFs, as surrogate measures of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions
For organizations that supply metrics	Be transparentProvide access to dataDiscourage data manipulationProvide different metrics for primary literature and reviews
For publishers	Cease to promote journals by IF; provide an array of metricsFocus on article-level metricsIdentify different author contributionsOpen the bibliographic citation dataEncourage primary literature citations
For research institutions	When hiring and promoting, stating scientific content of a paper, not the JIF of the journal where it was published, is what mattersConsider value from all outputs and outcomes generated by research
For funding agencies	Stating scientific content of a paper, not the JIF of the journal where it was published, is what mattersConsider value from all outputs and outcomes generated by research
For researchers	Focus on contentCite primary literatureUse a range of metrics to show the impact of your workChange the culture!

JIF: journal impact factor

In order to boost the growth of quality research in our country, a policy statement was released by the Indian National Science Academy on Dissemination and Evaluation of Research Output in India.[29] This document elaborately discusses basic policy parameters such as promoting preprint repositories and incorporating quality peer review, minimizing interference caused by predatory journals as well as predatory conferences, policies for categorizing and evaluating research effort and rationalizing payment policies in the Indian scenario. However, such recommendations are yet to be executed, in actuality.

Over the years, several approaches have evolved to address the limitations posed by JIF in the valuation of researchers and research publications.[30] Some of them are enumerated in Table 2. However, due to the inherent lacunae, there is no one size that fits all, set of metrics that can assess the credibility of the researchers or their publications. Research organizations should be consistent about which valuation metrics to be used while maintaining the ethos and values of scholarly scientific publishing over the mere accumulation of publications in prestigious journals.

Table 2

Alternative and diverse evaluation metrics to measure the scientific impact

Alternative approaches to IF	Description
h-index[31]	It is an author-level metric that attempts to measure both the productivity and the citation impact of the publications of a researcher.It evaluates the research impact of a scientist by the total number of citations that author’s work has received and includes citation to journal articles only.
Y-factor[3233]	It is a combination of both the traditional IF and the weighted page rank measure for the average article in the journal.Used to rank and rate journals
Eigenfactor[34]	The Eigen factor score measures the importance of a journal to the scientific community by considering the origin of incoming citations.It reflects how frequently an average researcher would access content from the journal.
Altmetric widget[35]	Allows authors, readers, and publishers to track online activity around scholarly articles.It gives a virtually instantaneous view of how research publications are being shared and the amount of attention they are receiving.
Article usage statistics[36]	It gives a breakdown of how many times the abstract or the full-text version of an article has been viewed online and the number of PDF downloads.

Conclusion

Despite having widespread recognition that the IF is being misused, the misuse continues and is likely to continue because of the diverse confluence of forces within the scientific community that encourage, promote, and perpetuate it. We submit that the JIF remains a relatively crude index for evaluating the quality of a journal, its scientific content or the credibility of a researcher. Doing so will not only affect the research scientists involved but may even discourage ethical research and hamper the overall scientific progress. A comprehensive scientific evaluation of an article requires a multidimensional approach and is beyond the scope of a single metric such as IF. While evaluating the performance of a researcher, academic administrators should focus on contribution and content rather than on publication venue. However, changing the existing culture will be slow since the researchers are so deeply entrenched with JIF mania that it may take time to wean off from its influence. The removal of JIF from websites as done by ASM is a bold step in this direction and should be followed by others with right earnestly. To err is human but to correct is divine.

One must note that the traditional method of evaluation continues to be peer review, and there is no substitute for reading the article for assessing the research worthiness of authors, rather than reading the title of the paper or the title of the journal or its IF.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.