Does retraction after misconduct have an impact on citations? A pre-post study.

BACKGROUND: Retracted articles continue to be cited after retraction, and this could have consequences for the scientific community and general population alike. This study was conducted to analyse the association of retraction on citations received by retracted papers due to misconduct using two-time frames: during a postretraction period equivalent to the time the article had been in print before retraction; and during the total postretraction period.
METHODS: Quasiexperimental, pre-post evaluation study. A total of 304 retracted original articles and literature reviews indexed in MEDLINE fulfilled the inclusion criteria. Articles were required to have been published in a journal indexed in MEDLINE from January 2013 through December 2015 and been retracted between January 2014 and December 2016. The main outcome was the number of citations received before and after retraction. Results were broken down by journal quartile according to impact factor and the most cited papers during the preretraction period were specifically analysed.
RESULTS: There was an increase in postretraction citations when compared with citations received preretraction. There were some exceptions however: first, citations received by articles published in first-quartile journals decreased immediately after retraction (p<0.05), only to increase again after some time had elapsed; and second, postretraction citations decreased significantly in the case of articles that had received many citations before their retraction (p<0.05).
CONCLUSIONS: The results indicate that retraction of articles has no association on citations in the long term, since the retracted articles continue to be cited, thus circumventing their retraction. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

There is scarce evidence regarding if retracted articles continue to be cited after retraction.

Our study suggests that retraction of articles has no association with the number of citations in the long term across all quartiles. However, our results have shown an association of retraction on citations received by articles highly cited before the retraction.

To prevent this lack of effect on citations after retraction, more effective mechanisms to avoid citing retracted papers should be established by the scientific journals. Options available with reference managers should be used by authors.

Introduction

In recent years, there has been an increase in the number of retracted articles, even when the increase in the number of indexed publications is taken into account, a development that is of concern to the scientific community.1 A paper’s retraction informs the scientific community that the study in question is not considered reliable, ethical or both, should be effectively eliminated from the literature and should therefore cease to be cited, especially when its retraction is attributable to scientific misconduct.2 Even so, there is evidence to indicate that retracted articles continue to be cited even after their retraction.3 A classic example of this phenomenon is the case of the paper published in 1998 by Wakefield et al, which reported an association between the administration of measles, mumps and rubella vaccine and autism. According to a study conducted in 2019 by Suelzer et al, Wakefield’s paper had been cited a total of 1211 times until March 2019, and of these citations, 881 occurred after a partial retraction of the article in March 2004.4 5 This paper was finally retracted in 2010.6

Postretraction citations pose a problem and can have consequences, both for the scientific community and also for the general population, since they perpetuate the misconduct and/or erroneous results, and continue lending visibility and, implicitly, credibility to the research and its authors. On the one hand, citing retracted articles may lead to other researchers accepting this information as accurate and basing their studies on it; and on the other hand, if health professionals make use of and base themselves on dishonest or misinterpreted information in their own clinical practice, their patients’ health could be put at risk.7 Most postretraction citations may be attributed to two causes: first, the author’s ignorance of the retraction because, in many cases, news of the retraction does not appear in the text or, alternatively, the author does not access the article but instead places their trust in the citations of other authors;8 9 and second, under certain circumstances, the author may consider it useful to mention these results as erroneous in the context of their own study.

The frequency of retracted publications is an important indicator of both the quality of the peer-review process and the pressure on researchers to publish, widely known as ‘publish or perish’.1 10 The number of retracted papers has risen.11 12 However, a decreasing trend can be observed for the first time in the number of retracted papers since 2016 to the present days. The total number of retracted publications in 2013 was 573 out of a total of 1 148 851, representing a proportion of 5 per 10 000 publications. However, in 2016, this proportion was 3.8 per 10 000 publications (484 retracted publications out of a total of 1 279 814). The proportions were calculated by dividing the total number of retracted publications by the total number of registries in PubMed for these 2 years.

Despite the reduction in the proportion of retracted publications in recent years, it should be highlighted that proportion of misconduct among retracted papers did not change over the time.13 The situation is rendered more serious by the fact that only a small proportion of cases of scientific misconduct are known.14 15 For these reasons, there is a need to generate scientific evidence that would allow for systematic and rigorous analysis of the characteristics of the retracted articles and the causes of their retraction.1 A number of authors have conducted studies focusing on postretraction citations and the association of retraction on these, and report a reduction of 35%–69% in the number of citations received by such articles after their retraction.7 16 17 However, these studies have neither focused on the characteristics of the journals in which the retracted articles are published, nor considered a minimum period of publication for retracted articles that would afford an opportunity for citations to be generated.

Accordingly, the main aim of this study was to analyse whether or not citations received by retracted articles decrease after retraction, and the association of such retraction on citations received according to the journal’s relative position (impact factor, IF); and, to assess if the number of citations on articles that are highly cited prior to their retraction is reduced.

Methods

Study design

This was a quasiexperimental, pre–post evaluation study in which the intervention consisted of an article’s retraction for reasons of misconduct, and the unit of analysis was the article published and subsequently retracted. We selected original research papers and literature reviews published in journals included in Journal Citation Reports (citation-based research analytics tool belonging to Clarivate Analytics). By way of inclusion criteria, articles were required to have been published in MEDLINE from January 2013 through December 2015 and been retracted between January 2014 and December 2016, both inclusive. The cause of retraction must have been research misconduct, defined by the Office of Research Integrity as follows: “fabrication, falsification or plagiarism in proposing, performing or reviewing research, or in reporting research results… Research misconduct does not include honest error or differences of opinion’.18

The characteristics of retracted studies included have been described in other previous reports.13 19 All types of publications that were not original research papers or literature reviews were excluded, as were those retracted for any reason other than misconduct. To ensure that any given retracted article enjoyed the same opportunity of being cited preretraction and postretraction, the period during which it was in print before and after retraction was required to be equal, with a minimum of 12 months each (preretraction and postretraction periods). This meant that all the studies included were published a minimum of 12 months prior to their retraction and were ‘at risk of citation’ for a further 12 months thereafter, with a minimum total time to retrieve citations of 24 months. An extra follow-up time for all papers included was added until December 2019. The time between retraction and such date was called ‘total postretraction period’ and we collected all citations received during such period.

Data collection

We designed our own database to include all information from the retracted papers included in this study. Some of this information was recorded manually and other was collected from Web of Science (Clarivate Analytics). The information used for this study was the following:

Data related to the retracted article: first author’s country of affiliation, number of authors, institutional affiliation of author(s), date of publication and date of retraction.

Data about the journal in which the retraction occurred: category of area of knowledge (and in any case where a journal appeared in more than one such area, we selected the one in which it ranked best), IF, relative position (by quartile) and country of publication. Relative position by quartile was calculated for a specific journal publishing the retraction using the category where it ranked best at the Journal Citation Reports (Clarivate analytics) database. This database provides the quartile belonging to each journal for a given year.

Data on citations received: title of the citing article, name of its author(s) and name of the journal. Number of citations received by the retracted article across the total preretraction period, and number of citations received both in a postretraction period equivalent to the preretraction period and across the total postretraction period. In addition, we calculated total citations, defined as the sum of citations in the preretraction and postretraction periods. All citation data were obtained by conducting a manual search in the Web of Science database. All the information regarding the retracted paper characteristics, citations and time of citation was collected in a database designed to this end by the authors.

Statistical analysis

We performed a descriptive analysis of the characteristics of the retracted articles by reference to the variables of interest. We also showed descriptive information on retracted papers excluding those with the first author from China. To do this, we calculated absolute numbers distribution with percentages and medians with 25th and 75th percentiles. Afterwards, we analysed the association between citations received preretraction and postretraction stratified by journal quartile and also specifically for a smaller group of retracted papers comprising those most cited papers before retraction. For the quartile calculation, we only considered journals where retracted papers have been published. We first compared citations received during the preretraction period against those received during an equivalent postretraction period; and we then compared these preretraction citations against total postretraction citations received until December 2019. As these were matched data, the Wilcoxon signed-rank test was used for analysis. Using the same method, a specific analysis was performed on the 21 most cited articles prior to their retraction, in order to ascertain whether their postretraction citation pattern was affected to a relevant degree. All analyses were performed using IBM SPSS V.22 (IBM, Armonk, NY, USA).

Results

A total of 304 articles fulfilled the inclusion criteria. The retracted articles included in the study were published in 178 different biomedical journals. A breakdown of the principal characteristics of the retracted articles is shown in table 1: 65.0% of retracted articles were written by more than four authors, and 77.0% of these came from university institutions; China was the country of affiliation of 45.7% of first authors, followed by the USA with 14.1% and finally, one-third of all retracted articles analysed were published in first-quartile journals. It can be observed that excluding retracted papers from China did not change importantly the number of authors of retracted papers and other variables, though the number of papers retracted in first quartile journals increased from 32.9% to 45.5%.

Table 1

Characteristics of retracted articles (n=304)

Variables	N (%)	Papers from China excludedn (%)
Country of first author
China	139 (45.7)	–
USAmerica	43 (14.1)	43 (26.1)
Iran	28 (9.2)	28 (17.0)
Other countries	94 (31.0)	94 (56.9)
Number of authors*
1–2	37 (12.3)	27 (16.4)
3–4	69 (22.8)	39 (23.6)
>4	196 (64.9)	99 (60.0)
Institution†
University	233 (77.0)	127 (77.4)
Hospital	39 (12.8)	8 (4.9)
Research centre	28 (9.2)	26 (15.9)
Other	3 (1.0)	3 (1.8)
Scope of the journal
Biochemistry and molecular biology	43 (14.1)	11 (6.7)
Oncology	35 (11.5)	11 (6.7)
Medicine, research and experimental	23 (7.6)	1 (0.6)
Multidisciplinary sciences	22 (7.2)	15 (9.1)
Pathology	19 (6.3)	16 (9.7)
Other	162 (53.3)	121 (67.2)
Relative position of journal
Q1	100 (32.9)	75 (45.5)
Q2	78 (25.7)	44 (26.7)
Q3	91 (29.9)	28 (17.0)
Q4	35 (11.5)	18 (10.8)
Country of journal
USA	97 (31.9)	70 (42.4)
UK	91 (29.9)	56 (33.9)
The Netherlands	40 (13.2)	9 (5.5)
Other	76 (25)	30 (18.2)

*Missing information: two cases missing for all papers and two when papers from China are excluded.

†Missing information: one case missing for all papers.

The citations received by retracted articles are shown in table 2. A median of 2 preretraction citations were received by the articles included, with this number being equal to that received in the equivalent postretraction period. In the case of median total postretraction citations, however, this figure rose to 4. These results did not differ when we excluded papers with the first author from China (data not shown).

Table 2

Characteristics of retracted articles linked to citations

Variables	All papers
Median preretraction follow-up time and range (months)	18 (12–39)
Median equivalent postretraction period follow-up time and range (months)	18 (12–39)
Median total postretraction follow-up time and range (months)	50 (37–72)
Median total citations (25th and 75th percentiles)	7 (3–15)
Median citations in the preretraction period (25th and 75th percentiles)	2 (1–5)
Median citations in the equivalent postretraction period (25th and 75th percentiles)	2 (1–5)
Median total postretraction citations (25th and 75th percentiles)	4 (2–8)

Table 3 and figure 1 show the citations received by retracted articles according to the journal quartile in which they were published. In all cases (ie, in both the preretraction and postretraction periods), the median number of citations was higher for first-quartile (Q1) journals. While there was a significant reduction in citations in first-quartile journals comparing preretraction period to the equivalent postretraction period (five median citations; interquartilic range 3–13 to 4 median citations; interquartilic range 2–8), this was not the case in the remaining quartiles. When comparing preretraction citations with the total postretraction period citations, we observed that the number of citations increased significantly for second and third quartiles (2–4 median citations for the second quartile and 1–3 for the third quartile), with the fourth quartile showing a close to significant difference (2–3 median citations). There was no significant difference for the first quartile for this comparison. Of note, the median number of citations for the total postretraction period were, for the first, second, third and fourth quartiles, 7, 4, 3 and 3, respectively. This means that these papers continue to be cited regardless of the quartile of publication.

Table 3

Citations received preretraction and postretraction, by journal category

Variables	Q1	Q2	Q3	Q4
Median total citations (25th and 75th percentiles)	14 (7–24.25)	6 (2.75–11)	5 (2–8)	5 (2–9)
Median citations in the preretraction period (25th and 75th percentiles)	5 (3–12.75)	1 (0–4)	1 (0–3)	1 (0–3)
Median citations in the equivalent postretraction period (25th and 75th percentiles)	4 (2–8)	2 (1–5)	1 (0–3)	2 (1–4)
Comparison between preretraction period and equivalent postretraction period	p<0.001	p=0.245	p=0.392	p=0.488
Median total postretraction citations (25th and 75th percentiles)	7 (4–12)	4 (1–7)	3 (2–6)	3 (1–6)
Comparison between preretraction and total postretraction periods	p=0.332	p<0.001	p<0.001	p=0.054

Figure 1

Preretraction and postretraction citations by journal quartile.

Table 4 describes the characteristics of the 21 most cited articles in the sample, and analyses the citations received before and after retraction (during both the equivalent and total postretraction periods). There was a statistically significant decrease in the median number of citations between the preretraction and equivalent postretraction periods (p<0.05); thereafter, citations registered an increase, though this was not significant (p=0.06).

Table 4

Characteristics of the most cited articles prior to retraction

Variables
Median preretraction follow-up time and range (months)	27 (15–39)
Median equivalent postretraction period follow-up time and range (months)	27 (15–39)
Median total postretraction follow-up time and range (months)	51 (37–59)
Median total citations (25th and 75th percentiles)	39 (30.5–70)
Median citations in the preretraction period (25th and 75th percentiles)	24 (18–33.5)
Median citations in the equivalent postretraction period (25th and 75th percentiles)	13 (7.5–23)
Comparison between preretraction period and equivalent postretraction period	p=0.01
Median total postretraction citations (25th and 75th percentiles)	17 (11–26.5)
Comparison between preretraction and postretraction periods	p=0.06

Discussion

This study highlights the fact that articles retracted due to scientific misconduct continue to be cited, thus showing retraction to be an inadequate measure for eliminating invalid knowledge as a means of preventing its use by the scientific community. On the contrary, the results of this study suggest that retraction only leads to an initial decrease in the number of citations in first-quartile journals, and does not reduce the number of citations in articles published in second, third or fourth-quartile journals.

The potential effect of retraction on citations has been studied for over three decades, however, the published studies are not comparable with ours in terms of study design. Most of the studies analysing the association of retraction with citations do not have a pre–post design to be specifically targeted at analysing postretraction citations of individual articles retracted due to scientific misconduct, during a postretraction period equivalent to the preretraction period and also during the entire postretraction period.

Different authors have studied the association of retraction with the citations of retracted articles, with a reported reduction of 35%–69% in the number of citations received by articles following retraction.7 16 17 Instead of a reduction, however, our results show a twofold increase in postretraction citations. Even so, our results coincide with other previous studies, in which most of the articles analysed received more citations following retraction.9 20 It should be stressed here that these previous studies are not comparable with the present study in terms of the design included (ie, clinical trials),16 knowledge areas, such as radiology or dentistry9 21 or time of publication (some studies were published prior to 2006).7 17

In all, 32.9% of the retracted articles included in this study had been published in first-quartile journals. These results are in line with those of earlier studies.9 22 This may be because papers that appear in high-IF journals and are retracted due to misconduct, are more likely to be detected, possibly owing to the additional postpublication scrutiny to which such articles are subjected as a result of being used more often than those published in lower-IF journals.7 22

With respect to citations of articles published in first-quartile journals, these decrease significantly during the equivalent postretraction period but then subsequently increase once again, receiving more citations than those received by articles in the other quartiles. This finding is in line with those of previous studies which report a positive correlation between IF and postretraction citations, namely, that the higher the journal’s IF, the more citations an article will receive after its retraction.17 23 24

Different studies have found that papers which receive many citations prior to being retracted continue to be highly cited thereafter.24 25 This study shows a significant decrease between the citations received by the most highly cited articles prior to retraction and those received in the equivalent postretraction period. After this period, the number of citations begins to rise again, though this increase is not significant with respect to the number received preretraction. However, despite the fact that there is a certain association on postretraction citations in these types of articles, the total postretraction citations they receive are almost double the number of postretraction citations received by the articles included in this study.

Bearing in mind that 94% of all postretraction citations refer to the article as valid, without mentioning its retracted status,8 one possible explanation for the results obtained by us is that, if a retracted article is cited in a publication with no mention of its retracted status, this may give rise to ‘secondhand’ citations. This amounts to citing an article that has previously been cited by another author, without having accessed the original and thus being ignorant of the fact that one is citing a retracted article.9 20 According to Simkin and Roychowdhurys study, 70%–90% of all citations are copied from references used in other articles,26 reflecting poor scholarship. This could cause a chain reaction and thus increase citations of a retracted article long after its retraction.

One possible explanation for the decrease in citations in the immediate postretraction period and the subsequent increase thereafter in first-quartile journals may be linked to the phenomenon of self-citations. According to the study undertaken by Madlock-Brown and Eichmann, authors may be capable of influencing the way in which their retracted paper is viewed by means of self-citation.23 Hence, there is the possibility that, in some cases, authors, on observing a decrease in citations as a consequence of retraction, cite themselves without mentioning their article’s retracted status, thereby boosting the number of citations that their retracted article receives.23

This study has some limitations, one of which could be the low number of retracted articles. This is due to the inclusion criterion that required retracted articles to have been in print for at least 1 year without having been retracted. However, this inclusion criterion meant that each article had the opportunity to receive a sufficient number of citations to analyse a potential association with retraction, something that would not have been feasible if the article had been published for a short time only. Furthermore, there is the possibility that a number of postretraction citations may have been incorrectly classified, since some came from articles accepted for publication and already available online before the article cited had been retracted, consequently leaving the authors unable to delete the relevant citation. Yet even if this were so, we nonetheless observed that the number of citations increased in general and that there was only an association when the retracted paper was published on journals belonging to the first quartile. This means that authors who published in high-IF journals were aware of such retractions, in contrast to what happened in journals belonging to the other quartiles. Hence, while the period of time that elapsed between the dispatch and publication of a given article might have had some association with the results of this study, this can be assumed to have been the same across all quartiles. Additional limitations are having only included articles published in IF journals, and therefore we do not know the results of retraction in other types of journals and a further limitation is not having collected reasons for exclusion of excluded papers.

This study’s principal advantage lies in its pre–post design, since, to the best of our knowledge, it is the first study to compare the citations received by retracted articles before retraction against those received during an equivalent postretraction period of time. Moreover, the number of citations received, both before and after retraction, was ascertained on an article-by-article basis, thereby making it possible to ascertain the association of different characteristics of the retracted papers on the number of citations.

In conclusion, this study indicates that retraction of articles has no association with the number of citations in the long term, since the retracted articles continue to be cited, thus circumventing their retraction. While it is true that, for a period of time, postretraction citations are seen to decrease in first-quartile journals, they ultimately do start to increase again.

Therefore, retractions must be announced and monitored to prevent spreading of retracted studies. One notable example of monitoring retractions is the Retraction Watch database that was created in 2010 and has a catalogue more than 20 000 retracted items.27 Even though this database is not complete, it is available and can be used by any interested person to know different details of retracted papers.

However, more effective mechanisms should be established to prevent the citation of retracted articles in scientific papers: these could possibly include a warning in the publication guidelines of all scientific journals advising authors to check for citations of retracted articles.