Assessing the effect of highly cited papers on the impact factor of journals in the field of public health.

BACKGROUND: The aim of this study was to appraisal the effect of highly cited papers in the field of public health and find out whether the unusual citations affect the ranking order of the journals in this field or not. A total number of 142 journals titles were listed in Journal Citation Report (ISI Thomson) in the field of "Public, Environmental & Occupational Health". All but one of them had published papers at least for a year from 2009 to 2010. Journal title, number of citations and publication year of 45685 papers were collected from ISI web of knowledge database at December 25, 2011. About half of the papers (23226) had no citations and 89.4% (40835) had less than 6 citations. We concluded that the ranking of journals in the field of public health is not affected by the individual papers with unusual number of citations.

Introduction

The first scientific journal published in 1665 called le Journal des Savantsand(1). Since the beginning of 19th century, it has become important to evaluate the individual papers using the number of citations (2). However measuring the papers through their citations was a controversial issue from then (3). In 1955, when Impact Factor (IF) was introduced as a method to evaluate the scientific journals, it was supported by the National Institute of Health (NIH); Since then many objections have been reported about the usage and definition of this index, such as “Journal impact factors correlate poorly with actual citations of individual articles” and “ Citations to “non-citable” items are erroneously included in the database” (4). Another objection regarding IF is that it is highly affected by papers with unusual number of citations. The rationale behind this objection is that IF uses Mean, the statistical central index, which is applicable in semi-normal distributions; but citation distribution is not semi-normal (4–5).

In this paper, we intend to analyze the effect of highly cited papers in the field of public health and find out whether the unusual citations affect the ranking order of the journals in this field or not.

Methods

A total number of 142 journals titles were listed in Journal Citation Report (ISI Thomson) in the field of “Public, Environmental & Occupational Health”. All but one of them had published papers at least for a year from 2009 to 2010. Journal title, number of citations and publication year of 45685 papers were collected from ISI web of knowledge database at December 25, 2011. It is noteworthy to say that one journal (WHO Technical Report Series) had different title for different issues but we considered them as WHO Technical Report Series.

As the IF index has not a normal distribution, Spearman’s non-parametric correlation coefficient was used to compute the magnitude of relationship of journals ranking with and without unusual citations. To distinguish usual and unusual citations, we used different percentiles as cut-off points.

Results

Table 1 shows the percentiles for the number of citations of papers in the field of public health.

Table 1:

Percentiles for number of citations

Percentile	50	75	80	90	95	99
Number of Citations	0	2	3	6	9	19

About half of the papers (23226) had no citations and 89.4% (40835) had less than 6 citations. Spearman’s correlation coefficient test was calculated as 0.9344 (P <0.001) when papers with greater citations or equal to 6 (10.6% of papers) considered as unusual papers. Although 6 considered as a cut-off point in this table, Fig. 1 shows the Spearman’s correlation coefficient for other cut-off points.

Fig. 1:

Spearman’s Correlation Coefficient for different cut-off point

Discussion

Our findings showed that the ranking of journals in the field of public health is not affected by the individual papers with unusual number of citations. As 90% of the papers had less than 6 citations, we consider 6 as a cut-off point; however, Spearman’s correlation coefficient did not indicate significant difference between the rankings of journals with and without unusual papers. Even for higher cut-off points, the Spearman’s Rho increased. Weale et al. (6) assessed the effect of non-cited papers in immunology and surgery journals. Based on their results, the correlation of rankings before and after removal of non-cited papers was high. Although it is statistically inappropriate to use mean as a central index for non-normal distributions, but this results along with ours, demonstrate that the effect of outlier observations is not high enough to completely avoid Impact Factor as a quality measure for comparison of journals which was pointed by many studies (4–5).