Retractions in orthopaedic research: A systematic review.

OBJECTIVES: Despite the fact that research fraud and misconduct are under scrutiny in the field of orthopaedic research, little systematic work has been done to uncover and characterise the underlying reasons for academic retractions in this field. The purpose of this study was to determine the rate of retractions and identify the reasons for retracted publications in the orthopaedic literature.
METHODS: Two reviewers independently searched MEDLINE, EMBASE, and the Cochrane Library (1995 to current) using MeSH keyword headings and the 'retracted' filter. We also searched an independent website that reports and archives retracted scientific publications (www.retractionwatch.com). Two reviewers independently extracted data including reason for retraction, study type, journal impact factor, and country of origin.
RESULTS: One hundred and ten retracted studies were included for data extraction. The retracted studies were published in journals with impact factors ranging from 0.000 (discontinued journals) to 13.262. In the 20-year search window, only 25 papers were retracted in the first ten years, with the remaining 85 papers retracted in the most recent decade. The most common reasons for retraction were fraudulent data (29), plagiarism (25) and duplicate publication (20). Retracted articles have been cited up to 165 times (median 6; interquartile range 2 to 19).
CONCLUSION: The rate of retractions in the orthopaedic literature is increasing, with the majority of retractions attributed to academic misconduct and fraud. Orthopaedic retractions originate from numerous journals and countries, indicating that misconduct issues are widespread. The results of this study highlight the need to address academic integrity when training the next generation of orthopaedic investigators.Cite this article: J. Yan, A. MacDonald, L-P. Baisi, N. Evaniew, M. Bhandari, M. Ghert. Retractions in orthopaedic research: A systematic review. Bone Joint Res 2016;5:263-268. DOI: 10.1302/2046-3758.56.BJR-2016-0047.

To determine the rate of retractions and identify the reasons for retracted publications in the orthopaedic literature

The number of retracted publications in orthopaedic research has been increasing exponentially over the past 20 years.

Fraud, plagiarism, and other forms of academic misconduct are the leading causes of retraction in orthopaedic research.

Journals have inconsistent methods on how to handle existing publications that have been retracted.

Two reviewers independently reviewed three databases with a high inter-reviewer reliability (0.95, 95% confidence interval 0.93 to 0.98).

There is no established search methodology for retractions.

Some retraction notices are vague in their phrasing of the reason for retraction.

Introduction

Retractions in the scientific literature occur when the authors of a publication are found to have committed research misconduct or errors that have compromised the integrity of the study or the validity of the conclusions. While retractions still represent a minority of the published medical literature, there has recently been a rapid increase in the rate of retractions.[1-4] Invalid published medical research can mislead other researchers down unfruitful lines of investigation, waste resources and grant funding, and tarnish the public perception of medicine.[5,6] More importantly, invalidated published research may lead to incorrect treatment guidelines that may lead to patient harm.[7]

There has recently been a greater awareness of retractions in both academic and public circles, highlighted by the creation of the website “Retraction Watch” in 2010, which is dedicated to tracking and archiving retracted publications in the scientific literature.[8] The types of infringements that lead to retractions have been detailed in the guidelines set by the Committee on Publication Ethics (COPE).[9] It has previously been postulated that the majority of retracted studies are due to honest error on behalf of the authors.[10,11] However, more recent comprehensive studies have revealed that the most common reason for retractions in the scientific literature is academic misconduct.[12-18] Common examples of academic misconduct reported in the literature include proven fraud, suspected fraud, duplicate publication and plagiarism.[12,16]

Despite the fact that fraud and misconduct are under scrutiny in the field of orthopaedic research,[19] there has been little work done to uncover and characterise the underlying reasons for retractions in this field. It is imperative that the next generation of orthopaedic investigators become aware of, and adhere to, high standards of research ethics. The purpose of this study was to determine the rate of, and reasons for, retractions in orthopaedic research.

Materials and Methods

Search strategy

On September 9, 2015, we searched three online databases (EMBASE, MEDLINE and Cochrane), as well as the website Retraction Watch (www.retractionwatch.com), for retractions in orthopaedic research. Within the databases we used the following key words in our search: “orthopaedic”; “bone”; “musculoskeletal”; “cartilage”; “sports medicine”; “joints”; “spine”; “rheumatology”; “rehabilitation”; and “peri-operative”. Our search was limited to studies published between 1995 and 2015. We used the Boolean operator “or” between each orthopaedic search term. In MEDLINE, we selected the ‘retraction’ filter to limit our search strategy to retracted articles. In EMBASE, which does not have this filter, we also used the following keywords to indicate that we were narrowing our search to retracted studies: “retraction of study”; “retracted study”; “notice of retraction”; “retraction notice”; “retracted publication”; and “retraction of publication”, connected by the Boolean operator “or”. We then also used the Boolean operator “and” between the combination of orthopaedic search terms and the combination of the retraction search terms.

Study screening

Two reviewers (JY, AM) independently screened titles, abstracts, and full texts of the retrieved studies. We used the following inclusion criteria: studies involving any musculoskeletal science or musculoskeletal conditions; all levels of evidence; studies on humans and animals, including cadaver studies; and basic science studies. We used the following exclusion criteria: topics unrelated to orthopaedic research; duplicate papers; full article unavailable; and publication not available in English. Two reviewers noted any discrepancies during this search and discussed their findings to settle disagreements with respect to inclusion. When unresolved conflicts arose, a third senior reviewer (NE) was brought in to mediate discussion until a consensus was reached.

Data extraction

Two reviewers (AM, L-PB) independently extracted relevant study data from the final pool of included articles and recorded the data in an electronic database. The first 10% of studies underwent data extraction in duplicate in order to calibrate agreement between the two reviewers. The reviewers then divided the remaining studies equally to undergo data extraction. Extracted data included author names, year of publication, year of retraction, country of origin, journal name, journal impact factor, number of citations, reasons for retraction, availability of retracted papers online, retraction watermark, study design, and musculoskeletal subspecialty. The Web of Science database was used to determine journal impact factor.[20]

Statistical analysis

We reported discrete variables as counts or proportions, normally distributed continuous variables as means with standard deviations (sd), and skewed continuous variables as medians with interquartile ranges (IQRs). In order to assess inter-reviewer agreement, we calculated a weighted κ (kappa) for the article screening at the title-abstract screening stage. Agreement was categorised as follows: κ > 0.61 to indicate substantial agreement; 0.21 < κ < 0.60 to indicate moderate agreement; and κ < 0.20 to indicate slight agreement.[21] A 95% interval was calculated for the kappa statistic. All analyses were performed using Microsoft Excel (Microsoft, Santa Rosa, California).

Results

Retractions identified and dates of retraction

We identified 1192 potentially eligible studies from our search strategy, of which we excluded 1082 (Fig. 1), for a total of 110 studies for data extraction. The kappa statistic for inter-reviewer agreement for study inclusion was 0.95 (95% CI 0.93 to 0.98) which indicates substantial agreement. In total, 25 papers were retracted in the first ten years (1995 to 2004), while 85 papers were retracted between the years 2005 and 2015 (Fig. 2). The mean time from article publication to retraction was 2.8 years (sd 3.3).

Fig. 1

Search strategy process.

Fig. 2

Number of orthopaedic retraction notifications issued per year in five-year intervals between 1995 and 2015.

Journal and author characteristics

The retracted studies originated from institutions based in 22 different countries. The 110 retracted papers were found across 67 different journals. Two of these journals are no longer in print. The highest 2014 impact factor of a journal with a retracted study was 13.26. The most highly cited retracted publication has been cited 165 times,[22] however, the median number of times a retracted paper has been cited was 6 (IQR 2 to 19).

We identified several authors who were named on multiple retractions. One author is listed on 17 retractions, and another on 11 retractions. In addition, one author is listed on four retractions, and another on three retractions. There are two authors with two retracted papers. In total, 39 of the total 110 retracted studies involved these six authors.

Reasons for retraction

The most common reasons for retraction were fraudulent data (29), plagiarism (25), duplicate publication (20), and data errors (14) (Table I). A total of 28 studies were retracted for other reasons such as authorship misattribution, falsified peer reviewer details, or lack of ethics approval. Two studies were removed on the basis of committing both plagiarism and scientific fraud. One study misreported ethics approval. One study was retracted for both plagiarism and duplication. Occasionally, retraction notices were vague, with phrases such as “…the article contains several examples of incorrect presentation of scientific data…”,[23] or “…account of concerns over the validity of the data reported therein based on further consideration by the journal’s Editor…”[24] obscuring whether an article was retracted due to error or an act of misconduct.

Table I.

Reasons for article retraction[*]

Reason for retraction	n
Fraud	29
Plagiarism	25
Duplicate	20
Error	14
Peer review process manipulation	8
No ethics approval	7
No reason given	3
Data ownership/Copyright issue	3
Other	5

Two studies counted as both plagiarism and fraud, one study counted in both fraud and other. One study counted in both plagiarism and duplication

Method of retraction notification

Nine of the studies were unavailable online from the publisher’s website (indicating that they had been removed), and 34 were available online without any indication that they had been retracted. Among the remaining 67 studies, 37 were labelled as retracted with translucent watermarks, 22 with opaque watermarks, five with headers, footers, edited titles, and/or forewords, and three with both a watermark (one opaque, two translucent) and a header (Table II).

Table II.

Methods employed by journals to label original articles that were retracted

Method of identifying original retracted articles	Frequency of occurrence
Removed/inaccessible	9
Translucent watermark	39
Opaque watermark	23
No indication of retraction	34
Other	8
Total[*]	113

Three studies were labelled as retracted with both a watermark and another method

Discussion

Our study is the first to identify the reasons for retractions within the orthopaedic and musculoskeletal literature. As the number of retractions continues to grow, this study will raise awareness of the most common reasons for retraction and therefore highlight the importance of research integrity. The majority of orthopaedic retractions are due to research misconduct, with fraud and plagiarism accounting for nearly half of the retractions identified. While we did uncover several “repeat offenders”, we note that retractions originate from a variety of institutions and countries and have been published by a wide variety of academic journals.

Limitations

The main limitation of our study is that, without standard search methodologies for identifying retractions, some retractions may have been missed in our investigation. The nature of our electronic searches accounted only for papers that were actually retracted, and ignores any flawed studies that have not been retracted. Additionally, there is a time delay between when an article is formally retracted and when the retraction is updated on a searchable database. One study found this latency period to be up to six months[25] and indicates that some retractions in our search window may have been missed. The relatively small number of relevant retractions identified also limits our investigation. However, we examined three large and commonly used databases in our search strategy, as well as a popular website well known for leading the documentation of retracted articles.[8]

A final limitation of this study is that we were only able to assess the reason for retraction based on the information available from the retraction notice. Other more detailed sources unavailable to us include archived reports from the Office of Research Integrity and direct interviews with the pertinent institutions.[8,12] Nonetheless, the retraction notices used were published according to COPE guidelines.[9]

Relation to previous work

Our findings are consistent with a number of other studies outside the orthopaedic literature that have identified an increasing incidence of retracted studies in the general scientific literature.[12-14] Overall, we found that 82.7% of orthopaedic retractions were due to academic misconduct, which included plagiarism, fraudulent data, duplicate publishing, failure to obtain ethics approval, peer review process manipulation, and copyright violation. This compares to 77.4% of retractions in the overall scientific literature determined to be due to misconduct, defined by Fang et al[12] as fraud or suspected fraud, duplicate publication, and plagiarism. Van Noorden[15] found 44% of studies retracted due to misconduct, defined as fabrication or falsification, self-plagiarism, and plagiarism. Redman et al[17] stated that only 17% of studies were retracted due to research misconduct, but had identified plagiarism as a separate entity which accounted for 17% of retractions. “Inability to reproduce” was the reason for another 20% of retractions and the authors speculated that these retracted works were actually due to misconduct. Therefore, some discrepancy in the literature may be attributed to the varying definitions or categorisations of research misconduct.[26]

Several earlier studies on retractions found that the majority of retractions were due to published errors.[10,11] However, more recent studies have applied greater scrutiny and reclassified many of the studies labelled ‘retracted due to error’ as ‘misconduct’.[12-17] Van Noorden et al[15] showed that only 28% of retractions could be attributed to honest mistakes, while Fang et al[12] found that only 21.3% of retractions are due to error. The current study found that error was the cause for retraction in only 12.7% of the orthopaedic-related studies.

Implications

In total, 85 of the 110 identified orthopaedic studies (77.3%) were retracted in the most recent ten years of our search. It is possible that this increase in retractions is due to increased awareness and availability of retraction notices online.[15,18,27,28] In addition, advancement in detection software has enhanced methods of screening for plagiarism.[15] Improved information and communications technology allows for easier reporting and widespread online communication of errors and misconduct. The internet has allowed the post-publication peer review process to develop into a more open and transparent format. The development and successful growth of sites such as Retraction Watch and PubPeer[29] has increased the extent to which a published paper can be scrutinised. The format of PubPeer, which provides anonymity to commenters, can potentially provide protection for whistleblowers pointing out errors or misconduct in a paper.[29-31] As more vigilance is applied to publication standards, we can expect the number of retractions reported in the future to increase until further measures are put in place to discourage research misconduct. These measures may include a collective database for retractions, greater transparency in the reasons for retractions, tighter and carefully constructed research practice guidelines in conjunction with more stringent and enforced reporting guidelines, and greater resources and training dedicated towards replication and reproduction of studies.[4,13,15,27]

The method by which the journals handled the original retracted article varied considerably. Most of the journals included a watermark across the text that presented in some form the word “retraction”, however, these watermarks could either be translucent so that a reader could still read the original text, or opaque which obscured sections of the text. Moreover, there were many original articles that gave no indication that they were retracted nor notice that they contained errors or evidence of misconduct. This lack of consistency regarding the handling of original articles can have serious consequences as it can allow misinformation to perpetuate.[3] If these retracted articles continue to be cited by other authors or transition into clinical practice, it is possible that their erroneous conclusions could lead to detrimental medical care or misinformed public knowledge.[17]

The lack of consistency in issuing retraction notices and in the handling of retracted articles may reflect the fact that academic journals and editors, as a community, have not yet developed retraction standards. COPE provides guidelines for handling research misconduct, but the guidelines do not specify how a retraction is to be issued.[9,28,32] Greater awareness of, and adherence to, the guidelines set by COPE would help improve overall standards of quality among retractions.[9] In 2006, Sox and Rennie outlined, as designated by the Office of Research Integrity, the responsibilities of academic institutions, journal editors, and authors on taking action against fraudulent articles.[5]

The author who was involved in 17 retracted articles is trained as an anaesthesiologist and all of the associated retracted publications were published in anaesthesia journals.[33] We included these retractions as they studied anaesthetic techniques used for orthopaedic procedures and, therefore, would apply to the orthopaedic patient population.[34] The majority of the authors who had multiple retractions had their publications retracted due to misconduct, which is consistent with the findings by Fang et al,[12] whose study found that these cases are among the most notorious examples of misconduct in the field of research ethics.[12] Similarly, Steen et al[14] found that “repeat offenders” were the first named author in the majority of fraudulent papers. Retractions from “repeat offenders” skews the data for both specific journal and year, as detailed investigations often expose a large clump of such cases within a narrow time frame.[35,36]

Future directions

We noted in our findings that there was inconsistency in how journals and editors release and phrase retraction notices. Additionally, we came across instances where the phrasing in the notices was vague and these findings are consistent with those of previous studies.[6,16,18] It is difficult to assess why these vague statements are made and perhaps journals are hesitant to publish statements that could tarnish an author’s reputation.[6] However, the editors of Retraction Watch have found that authors who openly self-report honest mistakes (as opposed to attempting to obfuscate fraudulent actions), do not suffer negative consequences in their research careers.[18] It would be worthwhile as a future study to survey orthopaedic journals that have issued retraction notices to understand the processes behind the retractions and the reasons for the choice of words used in the retraction notices, in particular with respect to the effect of the retraction on the reputation of the journal itself and the authors involved.

The pace of retractions in the musculoskeletal literature is increasing, with the majority of retractions occurring due to academic misconduct and fraud. Orthopaedic retractions originate from numerous journals and countries, indicating that misconduct issues are widespread. The results of this study raise awareness of the need to address academic integrity when training the next generation of orthopaedic investigators.