h-Indices in a university department of anaesthesia: an evaluation of their feasibility, reliability, and validity as an assessment of academic performance.

BACKGROUND: The h-index is a tool that is increasingly used to measure individual research productivity. It is unknown whether its use as an evaluation of individual research impact is reliable and valid within the context of anaesthesia.
METHODS: We calculated the h-indices of 268 faculty members of a university department of anaesthesia using Scopus™ and Web of Science(®). Agreement between the databases was investigated with a Bland-Altman plot. The construct validity was examined by comparing the h-indices for faculty grouped by academic rank.
RESULTS: The mean bias between the Scopus™ and Web of Science(®) h-indices was 0.09 but 1.96 sd limits of agreement were -5.7 to 5.9. The Web of Science(®)-derived h-indices showed a statistically significant difference between the different academic ranks (P<0.001): median h-indices were 0 for lecturers, 2 for assistant professors, 9 for associate professors, and 16 for full professors. The Scopus™-derived h-indices also showed a statistically significant difference between the different academic ranks (P<0.001): median h-indices were 0 for lecturers, 1 for assistant professors, 9 for associate professors, and 17 for full professors. Post hoc testing found statistically significant differences in all comparisons between academic ranks (all P<0.01). Ignoring self-citations did not affect construct validity of the h-index. We found no evidence that the h-index is superior to counting the total number of publications.
CONCLUSIONS: Agreement between the two databases was problematic. There was evidence of construct validity; however, the overlap between academic ranks limits the discriminative power of a low h-index.