Blockbuster effect of COVID-19 on the impact factor of infectious disease journals.

The recent release of the 2021 impact factor (IF) showed a considerable increase in the score of journals in the infectious disease (ID) category and turned the classification of these journals upside down. Although IF was originally developed as a tool to help librarians index scientific journals, it has become a widely used tool for ranking the scientific impact and quality of journals and, by extension, researchers [1]. In many countries, credits and grants are allocated to scientists publishing in journals with the highest impact factors. Given the importance of the IF for researchers, it seems legitimate to question the causes of such an increase.

The journal impact factor (JIF) is calculated as the number of citations received each year by documents published in the previous two years, divided by the number of documents published in the previous two years. One of its curious features is that documents considered to have no “substantive” research content (such as editorials or viewpoints), called “non-citable” documents, are not included in the denominator but citations of these same documents are included in the numerator [1].

To investigate the determinant of this tremendous IF increase, we extracted citation reports from the Web of Science database for the 99 indexed ID Journals (‘NN category’) as of July 1st, 2022 (methods are detailed in the appendix). We selected the 47 journals that have published at least 100 “citable” research articles or reviews per year since 2014 and we computed their JIF from 2016 to 2021 (Figure S1-S2). Between 2014 and 2020, 131,128 documents were published, of which 93,184 were “citable” (71.1%), with an increase of 22% in 2020 compared to 2019 (21,590 versus 17,699).

While the mean JIF was stagnant from 2016 to 2019, it rose up dramatically over the last two years of the COVID-19 pandemic, from 4.52 in 2019 (standard-deviation (SD) ±3.83) to 9.49 in 2021 (SD±12.00). Based on their titles, a sixth (3505, 16%) of all documents published in 2020 referred to COVID-19 (i.e. included the terms “COVID” or “SARS-CoV-2”). Journals with the highest relative increase in JIF were those who published the higher proportion of documents related to the pandemic (r = 0.68, p-value < 0.001; Figure S3). When articles referring to COVID-19 were removed from the IF computation (Figure 1 ), the average JIF for 2021 remained similar to that of 2019 at 4.55 (SD±3.18). Hence, COVID-19 pandemic was the main cause of the increase in IF among ID journals.

Figure 1

Evolution of 2-year journal impact factors for infectious diseases journals from 2019 to 2021. Two (2) year impact factor (IF) for the selected 47 infectious diseases journals (see text) in 2019 and 2021. CIF is computed for all articles or letters issued from the Web of Science database (left), without the 10 most cited publications in each journal (centre) or without publications related to COVID-19 (right). Colour circles corresponds to journals.

To ensure this rise reflected a scientific output of consistent quality and interest, we examined the citations of the 41 journals with an increase in JIF. Whereas the weight of the top-10 most cited articles in the 2019 JIF was 13.9% in median (IQR 11.1%-19.0%), it increased to 25.6% in 2021 (IQR 18.1%-33.5.6%) (Figure S4). Among the 28 journals with an increase in JIF ≥ 1, the 2020 top-10 articles were responsible for 55.3% (IQR 39.7%-61.4%) of the increase in JIF (Figure 1). To roughly assess the scientific quality of these “blockbuster” publications, we analysed the abstracts of the 100 most cited articles published in 2020/21 by ID journals (Table S1). All articles were about COVID-19 and half (47%) had been published at the beginning of the pandemic, before July 2020. Only 2 were randomized-controlled trials (RCTs), 12 were prospective cohort studies, 24 were retrospective cohort studies, 10 were cross-sectional studies, 10 were modelling studies, 7 were case reports or case series, and 14 were narrative reviews or opinion papers. Apart from RCTs, the median number of patients enrolled in longitudinal or cross-sectional studies was only 76 (IQR 33-550). Albeit limited, this analysis suggests that these publications likely reflect the critical need for information in the early times of the pandemic rather than a strong long-term scientific effort. Since then, most of these articles have probably become outdated. For example, 2 out of the 10 most cited articles argued in favour of the use of hydroxychloroquine, a drug which has eventually proven to be ineffective for COVID-19.

The computation of JIF is based on all citations, either “positive” or “negative”, as an article might be cited because its results are interesting and useful to the scientific community, or on the contrary, to criticise its quality, methodology or conclusions. Controversial papers might thus have higher impact on the JIF than studies universally recognized as scientifically robust. Here again, the first papers supporting the use of hydroxychloroquine have been highly cited, notably for criticising their methodology or refute their conclusions [2]. As an illustration, we randomly selected 100 citations (out of 2687) of the most cited paper about hydroxychloroquine and found that a third (35) were criticizing the study or contradicting its conclusions (Table S2). Overall, it seems that the pandemic has exacerbated the “blockbuster effect” inherent in the metric [3], which is poorly correlated with the intrinsic quality of the research.

Although widespread for ID journals, this phenomenon has probably not spared other disciplines. Many of the top-ranked general medicine journals have more than doubled their IF. Even within other medical fields, leading journals have often seen their JIF increase, likely because of COVID-19 pandemic. For instance, the 2020/21 most-cited documents in the Web of Science categories “Neurology”, “Gastroenterology” or “Dermatology” were COVID-19 related papers published before July 2020.

The IF metric has several well-known limitations that have been exploited by journals to maximise their JIF. For example, the publication of a large proportion of documents that are not “citable” research articles (e.g. editorials), the multiplication of self-citations or the maintenance of articles in “online first” status for a long period of time, mechanically lead to an artificial inflation of the JIF [4]. And beyond the criticisms, it should be borne in mind that, within a journal, articles are very heterogeneously cited (as our analyses show) and their scientific quality may vary. Using the JIF to judge the quality of an article or a researcher is therefore meaningless. In a few years, moving away from the onset of the pandemic, we can expect JIF to decrease for many journals: not because of a decline in the quality of research, but most likely because the “blockbuster effect” will fade.

One can argue that alternative metrics have been proposed, in particular, the h-index [5]. Like the IF, this simple metric does not exclude self-citations and it cannot distinguish between positive and negative ones [6]. It is computed as the least number of publications (h) in a journal, cited at least h times. As the h-index does not rely on the average number of citations, it is less sensitive to the “blockbuster effect” than the JIF. However, the h-index cannot decrease, and it is strongly influenced by the age of the journal, penalising those that are recent. It is also depends on the absolute number of citable articles in the journal, and even if we consider a limited time window (e.g. 2-years) to compute the h-index, it remains strongly correlated with the JIF (in our data, r = 0.58, p-value < 0.001) [7]. If many other bibliometric indicators exist [5,8], none has proven to be reliable in reflecting the scientific value of a researcher or a journal. Instead of single metric, some authors have proposed to use mixed approaches based on a combination of multiples indicators, quantitative to measure scientific output and citations (e.g. JIF), and qualitative to measure other dimensions of scientific work such as collaboration, fidelity or impact on health services [9]. However, given that most of these mixed indicators have not yet been developed, establishing a globally recognised combination will demand a considerable collaborative academic work, if ever achievable. Furthermore, as mentioned above, it would be interesting to consider the “negative” citations, which is now possible thanks to advances in natural language processing [10].

To conclude, we have shown that the tremendous increase in IF was caused by COVID-19 and driven by a “blockbuster effect” poorly correlated to research quality. As the pandemic illustrates, the evaluation of journals, researchers and the allocation of research funds should not be determined by quantitative metrics alone, including the impact factor metric, at the risk of promoting sensationalism over true science. More generally, whichever metric we use to evaluate scientific production, we should be aware of its limits and apply it responsibly. It is time the scientific community be less addicted to impact factors and recognize that a simple metric cannot capture the complexity of scientific value.

Funding

None to declare.

Transparency Declaration

The authors declare no conflict of interest.

Authors Contribution

A.M. and T.D. conceptualized the commentary. A.M. extracted data and performed statistical analyses. A.M. and T.D. wrote the manuscript.