Evaluation of the scientific outputs of researchers with similar h index: a critical approach.

BACKGROUND AND
PURPOSE: h-index has been always reviewed as one of the most useful criteria for evaluating the scientific outputs of researchers by the sciencometric experts. In this study, the h-index of 40 Iranian researchers accompanied with its relationship to assessment criteria of scientific outputs such as the number of articles, scientific age, number of citations and self-citation were reviewed.
MATERIALS AND METHODS: The first part of this study was related to the literature review. But the information of 40 Iranian researchers' Citation Reports was observational extracted from WOS database and the Pearson correlation coefficient test was used to answer the research hypotheses.
RESULTS: Citation analysis showed that 40 selected researchers published 877 articles in web of science up to 9 January 2013. These articles have been cited 3858 time. The average of their h-index was estimated 38.5 ±12.12 Correlation coefficient test showed that there was a significant and direct relationship between the h-index and the number of papers, the number of citations and self-citation (Sig>0.05) but there was no significant relationship between scientific age and h-index (Sig> 0.05).
CONCLUSIONS: Analysis of the data showed that the quantitative and qualitative indicators of the researchers with the same h-index had considerable differences. Therefore, only the h-index should not be a criterion for scientific ranking of the researchers and other complementary indexes such as M parameter and G index along with h-index must be used to be able to more accurately determine the degree of scientific influence of the researchers with the same h.

1. INTRODUCTION

The importance of citations in articles and printed materials has offered in many sources (1-9). The h-index is a useful summary measure of output and quality of health services researchers(10). In 2005 Hirsch created a new approach in the assessment of scientific outputs of researchers by presentation the h-index(11, 12). H-index is one of the important and reliable indexes of sciencometry. This index assesses the quantity and quality of scientific outputs of researchers and all the other indexes which were presented in this field completed the h-index or were based on this index.

The literature review shows that despite many advantages of h-index it has many flaws. Although this index as an important tool has been accepted to measure the scientific outputs of the researchers by the scientific community due to its advantages over other sciencometric tools, there are still many debates about the certain validity of this index (13). The complementary tools were created to overcome the shortcomings of h-index but none of them could replace with h index or was common like it.

The articles whose citations are less than the h-index value are excluded from the calculation. The number of citations in this index is also influenced by the self-citation and colleagues’ citation. This means that the citation rate and along with it the h-index can be increased by recommendation to the friends, colleagues and self-citation (14). The various features of scientific fields have caused the h-index become more considerable in some majors. So in all fields of human knowledge cannot be used a single index because the distribution of citations in various scientific fields is different. On the other hand, many scientists have had an undeniable impact on the world science in spite of their low productions of science and low rate of citation to their articles which have led to a low h-index. For example, although the h-index for Einstein was between 4 and 5, his great influence on science could not be denied (15)

In addition to the above factor, many components affect the amount of citation. These components are such as the country of residence, the language of researchers, the number of working researchers in a country or a science center and such factors. Considering each of these in this index is a complex and difficult task.

It also cannot be easy to test the purpose of its citation and false. And about the articles of several authors, the role of each of them is not equal and its determination is difficult. They solve the problem by proposing the national h-index. It means that the national h-index is initially obtained according to the effective factors of each country then the h-index of the researchers is compared with the national h-index and the better results and more fair assessment can be achieved(16).

As mentioned above the extensive studies have been done in eliminating the shortcomings of h-index in recent years and usually each of them has led to a new index. However, none of the new indexes not only is universal as h-index but also can remove all flaws of h-index. So, this research showed the main disadvantages of the h-index clearly, the importance of each of the evaluation criteria of scientific outputs of researchers (number of articles, number of citations and self- citation, and the scientific age of researcher) in determining the h-index. ­And the present study determined the value difference of each of the above mentioned criteria among the researchers with the same h-index, too.

2. MATERIALS AND METHODS

This sciencometric study was done by using citation analysis. The samples of this study were 40 researchers of Babol City and their h indicators were between 2 to 12 in the web of science (WOS) database. The information of the citations of these researchers was extracted from the citation analysis part of its database on 9 January 2013. In addition to the information of the citation report other required information such as scientific age of researcher and the number of citations of each article were observational collected and transferred to the checklist. Tables and graphs were used to describe the data and the Pearson correlation coefficient was used to answer the research hypotheses.

3. RESULTS

Citation analysis showed that 40 selected researchers published 877 articles in web of science (WOS) up to 9 January 2013. These articles have been cited 3858 times. Of total citations, 645 cases were self-citations. The average of their h index was estimated 38.5 ±12.12.

The above table also shows that every researcher has averagely published 1.05± 21.92 articles in WOS citation database, each researcher has 96.45 ± 10.07 citations for his/her published papers averagely. The mean of net citations and self-citation of each researcher was 96.45 ± 10.07 and 2.31 ±12.16, respectively.

Moreover, the analysis of the data showed that each article had averagely 4/39 citation. Of these citations 3.66, and 0.73 were net citations and self-citation, respectively and totally, 20% of the citations were self-citation.

According to the above table, the correlation coefficient shows that there is a direct and significant relationship between the h-index and the number of researchers’ papers, the number of citations and self–citation (Sig <0.05) but there is no significant relationship between the scientific age and h-index (Sig> 0.05).

Also from total investigated researchers 9 (22.5%), 8 (20%), 7 (17.5%), 5 (12.5%), 2(5%), 5 (12.5%) and 3 persons had 5, 3, 6, 7, 4, 8 and 3 h index, respectively. The information of 9 researchers who had the five h-index was shown in figure 1 for the exact evaluation of the researchers who had the similar five h-index in terms of quantitatively (total number of articles, scientific age) and qualitatively (the total number of citations and self-citations).

Figure 1

Comparison of qualitative and quantitative indexes of scientific outputs of researchers with similar 5 h index

As shown in the above figure although the h index of all 9 researchers was 5, there were considerable differences in all assessment indexes of scientific outputs (scientific age, number of papers, number of citations and self-citations). For example among 9 researchers the differences between the lowest and highest numbers of citations and articles were 109 and 24, respectively. It indicated that the h index of a researcher who received the citations 153 times was equal with that of a researcher who received 44 citations.

4. DISCUSSION

The results of this study also showed that there were many writers who had the same h-index but the number of their citations was completely different in Hirsch’s core and non-core. And applying the same h-index was unfair and perhaps irrational for this group of writers.

For example, although the 9 investigated scholars in this study had the same 5 h-index, the total number of their citations and the Hirsch core were different from each other? In other words, among 9 scholars the differences between the lowest and highest numbers of citations and articles were 109 and 24, respectively. It indicated that the h-index of a researcher who received the citations 153 times was equal with that of a researcher who received 44 citations (Figure 1).

Therefore, such findings indicated that the influence rate of the academic researchers could not be precisely measured only relying on the value of the h-index. The researchers of this study believed that the minimal changes in the data of each indicator (scientific age, the number of citations, articles, and self-citation) should also caused changes in the value of h-index.

Another disadvantage of h-index is that not all citations of a researcher are involved in the calculation and the h-index value may not increase with the increase of citations and the changes in Hirsch core.

In addition, this weakness with varying severity has been seen in 8 other indicators. The articles with more citations (especially in basic core) do not have more weight and effects. There was a significant and positive correlation between the h-index and the total numbers of citations according to the findings of the present study (Sig <0.05). Therefore, it is better for researchers to consider all citations received by a given researcher in evaluating the scientific outputs. Among all the indicators, the citation is considered as the most important parameter. So, if an article receives low citation, it should not be a good reason to ignore it.

As a result, the writers and scholars who have recently entered the academic area are excluded due to the low number of articles and their citations. However, in this way some researchers tried to solve the problem by giving weight to the articles with more citations (17). But there is still the problem of no calculation of the articles which have fewer citations than the value of h-index.

Although the citations are important parameters in calculating the assessment indicators of scientific outputs, the self-citation is always a challenge for scientists in sciencometric area (18). The findings also showed that 20 percent of citations were related to the self-citation and almost one in five citations was self-citation (Table 1). Moreover, there was significant and positive correlation coefficient between h-index and the amount of self-citation (Table 2) and this was corroborated that the self-citation had an important role in calculating the h-index. This issue expresses the necessity of self-citation study. However, the inappropriate citations and consequently the invalid citations can increase the false h-index.

Table 1

Citation Report of 40 Selected Researchers of Babol city in WOS

Table 2

The correlation coefficient between the h-index and the indicators of scientific outputs

In the present study, the significant correlation was not found between scientific age and the value of h-index (Table 2). There is no similar study to compare this result so the generalization of this finding requires further research. Another flaw of this index can be that the h index will not decreased over time of researchers’ working life even if they do not do any research during few years.

Jin and et al. presented an index in 2007 to resolve this problem. Their index is obtained from the square root of total citations of Hirsch-core divided by the number of years since the first publication (19). Thus, this study proposes to pay attention the scientific age of articles in addition to the age of scientific researchers because previous studies showed that as the age of articles increased the amount of citations to them increased, too (20). One of the other problems of h-index is its numerical low so the amount of this index cannot exceed the number of published articles. And this factor causes the problem to compare different h-indicators to some extent.

5. CONCLUSIONS

H-index cannot accurately determine the scientific influence of researchers (especially with similar h). Therefore, it is recommended to use other complementary indicators along with h-index such as m parameter and g index in order to delineate the scientific influence of researchers with similar h. Although the supplemental indicators also have their own advantages and disadvantages, the problems of other suggested indicators are likely.