Can the similarity index predict the causes of retractions in high-impact anesthesia journals? A bibliometric analysis.

BACKGROUND: The overall similarity index (OSI) and highest similarity scores (HSSs) from a single source might help to predict the potential reasons for the retraction from the anesthesia journals.
METHODS: Retracted publications, from five highest impact anesthesia journals, were retrieved from the MEDLINE and journal archives and analyzed using a plagiarism detection software (iThenticate) and manually verified for citation characteristics, OSI, HSS, and the presence, extent, and location of the duplicate text. The validity of the OSI including and excluding quotations and references and the HSS in predicting the potential reasons for retraction were tested using the receiver operating characteristic curves.
RESULTS: Of the total 138 retracted original and corresponding articles identified, 131 articles were analyzed. Most of them had the HSS more than 40% arising from a single source. Extensive degree of plagiarism (OSI score >35%) was identified through the main text of all analyzed retracted articles. The areas under the curves indicate that the OSI including and excluding quotations and bibliography and the HSS had reasonable ability to predict plagiarism and fabrication with a perfect sensitivity rate and low specificity but were weaker at distinguishing ethical misconduct or inconsistent or erroneous contents.
CONCLUSIONS: The study highlights the presence of significant plagiarism in the retracted anesthesia publications irrespective to the reasons for retraction. The high OSI and the HSS could be useful tools to identify the potential manuscripts with high risks for plagiarism and fabrication.

Introduction

The high-impact anesthesia journals have the commitments to maintain the integrity of the clinical research for offering safe and effective patients' care and building the public trust in healthcare systems.[1] The editors have the responsibilities to retract the flawed publications to correct the scientific literature. There is a significant increase in the number of retracted scientific publications which could reflect the changes in the behavior of journals' editors, authors, and institutions.[2] Retractions in the medical literature are usually due to compromised peer review,[3] ethical misconduct, plagiarism, duplicate publications, fabrication, falsification, inappropriate data management, authorship issues, journal issues, review process, conflict of interest, or unknown reasons.[45]

The World Association of Medical Editors defines plagiarism as “the use of others' published and unpublished ideas or words (or other intellectual property) without attribution or permission and presenting them as new and original rather than derived from an existing source.”[6] The commercial plagiarism-checking software is regularly used by the journals for detecting plagiarism. Currently, the Crossref Similarity Check (iThenticate) is one of the most commonly used software for examination of the similarity index (SI) of the text.[7]

There is no current consensus about the cutoff SI score to detect plagiarism. Of note, the low SI score does not necessarily mean eliminating plagiarism.[8] Higgins and others[9] found good sensitivity and specificity for the cutoff SI score of 15% (84.8% and 80.5%, respectively) to detect plagiarism among the manuscripts submitted to a single specialized medical journal (Genetics in Medicine) over a 1-year period. Taylor has developed an effective screening algorithm for the detection of plagiarism in the manuscripts submitted to the American Journal of Roentgenology including (1) the overall SI including quotations and references, (2) the highest similarity score (HSS) from a single source, and (3) to submit manuscripts with an average value of more than 12% for further review.[10] I believe that Taylor's screening algorithm could be extrapolated to the anesthesia journals not only for detecting plagiarism but also predicting the reason for retractions.

Up to the best of my knowledge, no previous publications have studied the predictability of the overall SI score for the reasons of retraction from high-impact anesthesia journals.

I hypothesized that the overall similarity index (OSI) score and HSS from a single source might help to predict the potential reasons for the retraction from the high-impact anesthesia journals.

The investigator aimed to examine the overall SI scores, the HSS from a single source, and location of the duplicate text in the retracted publications from the five high-impact anesthesia journals.

Methods

Ethics committee approval for this research was waived. Five nonspecialized anesthesia journals with the highest impact factors including Anesthesiology, British Journal of Anaesthesia, Anesthesia and Analgesia, European Journal of Anaesthesiology, and Anaesthesia were selected based on the Journal Citation Reports by Clarivate Analytics in 2017.

All retracted articles from these five journals were included in the subsequent analyses. The retracted articles, which were left blank or only available in the scanned forms, were excluded from the analysis because the iThenticate software is unable to identify duplication in images.

Search strategy

The full search strategy aimed to include any retracted article from the selected five journals for any reason including plagiarized, erroneous, inconsistent, falsified, or fabricated content or ethical misconduct. Pertinent articles were independently searched in BioMedCentral, PubMed, and the journals' archives (updated November 3, 2018) by an expert librarian familiar with the literature search. The databases were searched using the following search keywords: “retracted,” “retraction,” “retract,” “note,” “notice,” “notification,” “misconduct,” “fabrication,” “falsification,” “duplicated,” and “plagiarism.” No time or language restrictions were imposed. Subsequently, all the abstracts and full-text articles of the included retracted publications and pertinent correspondences were reviewed.

Outcomes measured

The recruited retracted articles were analyzed using the iThenticate plagiarism detection software [http://www.ithenticate.com] and then verified with a detailed manual review.

The primary outcome included the OSI including and excluding the quotes and bibliography. The OSI was defined as the percentage of duplicated text from all sources whether it was originated from a single source or multiple sources.[10]

Secondary outcomes were the type of retracted article, reason of retraction (plagiarism, duplicate publication, fabrication, ethical misconduct, or inconsistent or erroneous contents), HSS, defined as a SI greater than 10% arising from a single source, whether the publication contains copied portions of text from one or more sources, extent of plagiarism, and the location of the duplicated text in the manuscript. Similar to Taylor,[10] plagiarism was defined as “an OSI equal or higher than 25% with the HSS from a single source equal to or higher than 10%.”

The validity of the data

Searching bias was assessed by a second investigator independently assessed compliance to selection criteria and selected publications for the final analysis.

Statistical analysis

Data were tested for normality using the Kolmogorov–Smirnov test. Categorical and ordinal data were analyzed using the Chi-square test followed by a modified Bonferroni correction. One-way analysis of variance was used to compare the continuous and normally distributed values in the five journals. Corrections for family-wise multiple comparisons using the Student–Newman–Keuls test were performed.

Univariate variables included the type of retracted article, the OSI including and excluding quotations and references, the HSSs more from a single source, the location of the duplicated text in the manuscript, and reasons for retraction. To identify the independent predictors that influenced reasons for retraction, these variables were examined in a stepwise manner in a multivariate logistic regression model, with entry and retention set at a significance level of P < 0.05.

The receiver operating characteristic (ROC) curves were generated to evaluate the accuracy of the OSI including and excluding quotations and references and the HSS in predicting the potential causes for retraction. A ROC area of 1.0 is characteristic of an ideal model, whereas an area of 0.5 indicates a model of no diagnostic value. The cutoff values for the OSI and HSSs with the best predicting sensitivity and specificity were derived from the ROC curve analyses.

Data were expressed as mean (standard deviation) or number (proportion). A value of P < 0.05 was considered to be statistically significant for continuous, categorical, and ordinal data, for which a Bonferroni correction was used.

Results

Database searches identified a total of 138 retracted articles from the five high-impact anesthesia journals. Seven articles were excluded from analysis because the original contents were removed, or the article is only available in a scanned form.

All remaining 131 articles were analyzed. The characteristics of the retracted articles, including the type of retracted publications and reasons for retraction, are presented in Table 1. Most of the retracted publications were original articles and were mainly retracted because of fabrication or ethical misconduct.

Table 1

Characteristics of the retracted articles

Characteristics	Anesthesiology	British Journal of Anaesthesia	Anesthesia and Analgesia	European Journal of Anaesthesiology	Anaesthesia
Number of retracted publications	10	24	64	25	15
Analyzed publications with iThenticate
Yes n (%)	9 (90%)*	13 (54.2%)*	64 (100%)	23 (92%)*	15 (100%)
No n (%)	1 (10%)*	11 (45.8%)*	0 (0%)	2 (8%)*	0 (0%)
Reason for a failure of analysis	Original text removed	Original text removed		Scanned form
Type of retracted publications n (%)
Original articles	8 (80%)	22 (91.7%)	64 (100%)	23 (92%)	14 (93.3%)
Correspondences	2 (20%)	2 (8.3%)	0 (0%)	2 (8%)	1 (6.7%)
Reasons for retraction n (%)
Plagiarism	0 (0%)	0 (0%)	2 (3.1%)	3 (12%)	0 (0%)
Duplicate publication	1 (10%)	0 (0%)	1 (1.55%)	0 (0%)	0 (0%)
Fabrication	2 (20%)	12 (50%)	33 (51.6%)	10 (40%)	5 (33.3%)
Ethical misconduct	2 (20%)	12 (50%)	24 (37.5%)	9 (36%)	7 (46.7%)
Inconsistent or erroneous contents	5 (50%)	0 (0%)	4 (6.25%)	2 (8%)	1 (6.7%)*
Unknown		0	0 (0%)	0 (0%)	1 (4%)	2 (13.3%)

Data are presented as number or number (proportion). *P<0.045 significant compared with the anesthesia and analgesia

The OSI scores including and excluding the quotes and bibliography were shown in Figure 1. The retracted articles from the European Journal of Anaesthesiology differed significantly from the Anaesthesia journal in terms of the OSI scores including the quotes and bibliography(P = 0.025). Compared with the British Journal of Anaesthesia, the Anesthesia and Analgesia and European Journal of Anaesthesiology had significantly higher OSI scores excluding the quotes and bibliography (P < 0.044).

Figure 1

The overall similarity index (OSI) (%) including and excluding the quotes and bibliography in the five journals.Note. Data are presented as mean (SD). BJA: British Journal of Anaesthesia; EJA: European Journal of Anaesthesiology. *P= 0.025 significant compared with the Anaesthesia journal, †P < 0.044 significant compared with the British Journal of Anaesthesia

The retracted articles from the five journals did not differ significantly in terms of the HSS from one source, origin of duplicated texts, or extent or location of plagiarism[Table 2]. The vast majority of the analyzed retracted publications had the HSSs more than 40% arising from a single source. Major degrees of plagiarism (OSI score >35%) were identified through the main text of all analyzed retracted articles irrespective of the reasons for retraction [Table 2].

Table 2

Secondary outcome data

Outcomes	Anesthesiology	British Journal of Anaesthesia	Anesthesia and Analgesia	European Journal of Anaesthesiology	Anaesthesia
Highest similarity score from one source
Mean (SD)	59.0±11.14	58.8±15.10	58.7±15.13	65.8±15.61	53.3±17.21
Number of articles
0–10%	0 (0%)	0 (0%)	2 (3.1%)	0 (0%)	0 (0%)
11–20%	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
21–30%	0 (0%)	0 (0%)	0 (0%)	1 (4.4%)	0 (0%)
31–40%	0 (0%)	1 (7.7%)	3 (4.7%)	0 (0%)	3 (20%)
41–50%	3 (33.4%)	3 (23.1%)	9 (14.1%)	2 (8.7%)	3 (20%)
51–60%	2 (22.2%)	4 (30.7%)	20 (31.3%)	5 (21.7%)	7 (46.6%)
61–70%	2 (22.2%)	1 (7.7%)	16 (25%)	7 (30.4%)	0 (0%)
71–80%	2 (22.2%)	3 (23.1%)	11 (17.1%)	5 (21.7%)	1 (6.7%)
81–90%	0 (0%)	1 (7.7%)	2 (3.1%)	1 (4.4%)	0 (0%)
91–100%	0 (0%)	0 (0%)	1 (1.6%)	2 (8.7%)	1 (6.7%)
Identified plagiarism1n (%)	9 (100%)	13 (100%)	64 (100%)	23 (100%)	15 (100%)
Origin of duplicated texts
Single source	7 (77.8%)	11 (84.6%)	32 (50%)	19 (82.6%)	13 (86.7%)
Multiple sources	2 (22.2%)	2 (15.4%)	32 (50%)	4 (17.4%)	2 (13.3%)
Extent of plagiarism
Minor (OSI score<25%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
Moderate (OSI score of 25–35%)	0 (0%)	0 (0%)	1 (1.6%)	0 (0%)	0 (0%)
Major (OSI score>35%)	9 (100%)	13 (100%)	63 (98.4%)	23 (100%)	15 (100%)
Location of the duplicated text
Abstract	0 (0%)	1 (7.7%)	1 (1.6%)	0 (0%)	0 (0%)
Introduction	0 (0%)	0 (0%)	0 (0%)	0 (0%)	1 (6.7%)
Methods	0 (0%)	0 (0%)	0 (0%)	0 (0%)	1 (6.7%)
Results	0 (0%)	1 (7.7%)	0 (0%)	0 (0%)	1 (6.7%)
Discussion	0 (0%)	0 (0%)	0 (0%)	0 (0%)	2 (13.2%)
Methods and results	0 (0%)	1 (7.7%)	0 (0%)	0 (0%)	1 (6.7%)
Methods and discussion	0 (0%)	2 (15.4%)	1 (1.6%)	0 (0%)	0 (0%)
Results and discussion	0 (0%)	2 (15.4%)	0	1 (4.4%)	0 (0%)
Main text (introduction, methods, results, and discussion)	8 (88.9%)	2 (15.4%)	7 (10.9%)	0 (0%)	9 (60%)
Main text and abstract	1 (11.1%)	4 (30.7%)	55 (85.9%)	22 (95.6%)	0 (0%)

Data are presented as mean (SD) or number (proportion). OSI=The overall similarity index. 1Plagiarism was defined as “an OSI equal or higher than 25% with the highest similarity score from a single source equal to or higher than 10%”

The type of retracted article, the OSI including and excluding quotations and references, the HSS more from a single source, and the location of the duplicated text in the manuscript showed no significant correlation with the reasons for retraction [Table 3].

Table 3

Independent factors for the reasons for retractions

	OSI including the quotes and bibliography		OSI excluding the quotes and bibliography		Highest similarity score from a single source
	Correlation coefficient	P	Correlation coefficient	P	Correlation coefficient	P
Plagiarism	0.058	0.519	0.061	0.503	0.105	0.244
Duplicate publication	0.096	0.290	0.103	0.253	−0.028	0.760
Fabrication	0.101	0.264	0.087	0.339	0.127	0.159
Ethical misconduct	−0.116	0.200	−0.048	0.593	−0.128	0.156
Inconsistent or erroneous contents	−0.072	0.426	−0.098	0.279	−0.063	0.488

OSI: The overall similarity index

The ROC curves of validating the OSI including and excluding quotations and references and the HSS in predicting the potential causes for retractions are shown in Table 4 and Figure 2. The areas under the curves indicate that the OSI including and excluding quotations and bibliography and the HSS had reasonable ability to predict plagiarism and fabrication with a perfect sensitivity rate and low specificity but were weaker at distinguishing ethical misconduct or inconsistent or erroneous contents. The optimal cut points are presented in Table 4.

Table 4

Sensitivity and specificity of different outcome measures for the identification of the causes of retraction

	OSI including the quotes and bibliography				OSI excluding the quotes and bibliography				Highest similarity score
	AUC	Sensitivity (%)	Specificity (%)	Cutoff point	AUC	Sensitivity (%)	Specificity (%)	Cutoff point	AUC	Sensitivity (%)	Specificity (%)	Cutoff point
Plagiarism	0.586	80	42	75.5	0.589	80	40	73.5	0.655	80	31	51.5
Duplicate publication	0.809	100	20	68.5	0.833	100	15	59.5	0.411	0	58	61.5
Fabrication	0.575	68	41	74.5	0.581	68	39	72	0.573	65	44	54.5
Ethical misconduct	0.383*	16	61	83.5	0.404	17	59	81.5	0.405	16	59	66.5
Inconsistent or erroneous contents	0.430	25	56	80.5	0.404	25	57	78.5	0.439	50	55	60.5

OSI: The overall similarity index, AUC=area under the ROC curve. *P=0.03

Figure 2

The Receiver Operating Characteristic (ROC) curves analyses for validation of the overall similarity index including and excluding quotations and references and the highest similarity score in predicting the potential causes for retractions

Discussion

Plagiarism is a grave unethical problem that could negatively impact the integrity of scientific research. That has raised the attention of the journals' editors to use different online software solutions for the detection of plagiarized contents in the submitted manuscripts.[11] The SI, generated with the well-developed iThenticate software, could be used for plagiarism detection.[10]

The present study showed that the analyzed retrieved retracted publications from the five high-impact anesthesia journals included a major degree of plagiarism (OSI greater than 35%) and HSSs more than 40% arising from a single source, irrespective the reasons for retraction. The OSI including and excluding quotations and bibliography and the HSS could be able to predict both plagiarism and fabrication.

In the present study, the identified significant degree of plagiarism in the all analyzed retracted anesthesia publications may be partially inflated by our tight definition of plagiarism similarly to Taylor,[10] including “an OSI ≥25% with the HSS from a single source ≥10%.” However, interestingly most retracted articles had OSI scores >35% and the HSSs >40% arising from a single source. Most of the retracted publications were original research which could be explained by the fact that the original articles are one of the most powerful types of publications. Compared with the Anesthesia and Analgesia and European Journal of Anaesthesiology, the retracted publications from the United Kingdom anesthesia journals (the British Journal of Anaesthesia and Anaesthesia) had lower OSI scores.

The reasons for retraction of scientific publications vary significantly among different medical specialties. Article retractions occur across the biomedical field because of errors, plagiarism, duplicate publication, fraud, and invalid peer review.[12] The commonest reasons from retraction from the surgical journals are duplication, ethical violations, and falsified data.[13] The present study has shown that most of the retracted publications from the five anesthesia journals were because of fabrication or ethical misconduct. Similarly, Yan and others[14] found that fraudulent data, plagiarism, and duplicate publication are the commonest reasons for 110 retracted publications from the orthopedic journals. Wang and colleagues[15] have identified 97 retracted neurosurgical publications over a 5-year period because of duplicate publishing and plagiarism.

Similar to the previous study,[10] an extensive amount of similarity was detected throughout the main text and abstract of the retracted publications.

Similarly to Taylor,[10] the results of the present study demonstrated that the use of OSI including and excluding quotations and bibliography and the HSS from a single source might be useful to identify plagiarism and fabrication with a good sensitivity and a low specificity. Additionally, the cutoff points for these outcome variables have been described. Further studies are required to test whether these findings generalize to all impacted anesthesia journals.

The present study might have an impact in terms of drawing the attention of the editors to subject the submitted manuscripts with the high OSI and highest SI from one source scores for further editorial review for the possibilities of plagiarism or fabrication.

The present study had several limitations. First, the retracted anesthesia articles were retrieved only from five high-impact anesthesia journals. Thus, our results could not be generalized to the other anesthesia journals. Second, the iThenticate software was unable to examine 6% of the retracted anesthesia publications because either the original contents were removed or they were available in a scanned form. Of note, Baždarić and others demonstrated the ability of a combined use of the plagiarism detection software solutions including the TBLAST, CrossCheck, and WCopyfind and manual verification to detect plagiarized manuscripts. Similarly, I used the iThenticate software in conjunction with manual verification to examine the identified retracted publications.[16] Third, the present study was not powered to test the hypothesis. Finally, the OSI score was not validated earlier to identify plagiarism.

In conclusion, the study highlights the presence of significant plagiarism in the retracted anesthesia publications irrespective to the reasons for retraction. The high OSI and the HSS could be the useful tools to identify the potential manuscripts with high risks for plagiarism and fabrication.

Financial support and sponsorship

Nil.

Conflicts of interest

I received free airway device samples from Ambu in April 2014 and from Airtraq in March 2015 for use in another three published studies. He has no direct financial or other interests in Ambu, Airtraq, or any other industry (in the context of this and other studies).

Authors' contributions

Mohamed R. EL-Tahan designed the study, employed searching, recruiting, analysis, and interpretation the data, and wrote and revised the manuscript.