Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings.

We applied a set of standard bibliometric indicators to monitor the scientific state-of-arte of 500 universities worldwide and constructed a ranking on the basis of these indicators (Leiden Ranking 2010). We find a dramatic and hitherto largely underestimated language effect in the bibliometric, citation-based measurements of research performance when comparing the ranking based on all Web of Science (WoS) covered publications and on only English WoS covered publications, particularly for Germany and France.

The WoS data system covers a number of journals in non-English languages, particularly in German and in French. Publications in these non-English language journals are counted as part of the output of countries, but they generally have a very low impact because only a few scientists outside Germany, Austria and Switzerland are able to read German; we have a similar situation for French. Thus, these non-English publications will ‘dilute’ considerably the average impact of countries such as Germany, Austria and France. This is particularly the case for the more application-oriented fields such as clinical medicine and engineering, and also for the social sciences and humanities. Because clinical medicine represents a considerable part of the entire scientific output of a country (particularly in western countries), this language effect directly influences the overall impact position of universities (van Leeuwen et al. 2001).

This effect has serious consequences for all citation-per-paper based rankings, but also for citations per staff rankings. The most striking result is a too low position of most German and France universities in these rankings, particularly universities with a medical school. As citations play an important role in university rankings such as the Times Higher Education, QS, Shanghai (ARWU) and the Leiden Ranking, all these rankings suffer from this language effect.

Thus, for ranking and comparison with countries such as the US, UK and all countries where almost all WoS-covered publications are written in English, calculation of bibliometric indicators based on only-English publications is a more fair procedure.

We show results in Fig. 1 for in total 69 German and French universities. We see that the ‘English-only’ citation impact is systematically higher than the ‘all publications’ impact, and the ranking position based on ‘English-only’ citation impact is systematically lower (thus, ‘better’) than based on ‘all publications’ impact, with a few exceptions due to differences in inter-group changes of ranking positions. In Fig. 2 the language effect is shown in a slightly different way for the 25 German universities with the highest citation impact. A comparison of the rankings for the 500 largest universities in the world based on all WoS publications as well as on English-only publications is given in our ranking website (Leiden Ranking 2010).

Fig. 1

Language effect in citation impact of all German and French universities. The upper figure shows the difference in impact (numerical values of the field-normalized citation score) in the case of all WoS (imp-all, horizontal axis) and only-English WoS publications (imp-eng, vertical axis). The lower figure shows the difference in citation impact ranking position in the case of all WoS (r-all, horizontal axis) and only-English WoS publications (r-eng, vertical axis)

Fig. 2

Language effect in citation impact of the 25 highest impact German universities. The upper figure shows the difference in impact (numerical values of the field-normalized citation score) in the case of all WoS and only-English WoS publications. The lower figure shows the difference in citation impact ranking positions in the case of all WoS and only-English WoS publications

In the above context, we focus on the language effect of, particularly European universities in the German and French language areas, but a similar effect will also be present for Italian and Spanish universities. The issue of publication language is related to the problem of ‘local journals’ (but covered in the Web of Science). These journals play only a marginal role in the scientific communication process at an international scale. The publications in these journals mainly transfer knowledge to local or national end-users. For instance, medical knowledge transferred to non-academic hospitals to improve clinical practice. In the technical sciences we find transfer of new knowledge to the engineering practice. As most fields contain local journals, these journals should ideally be excluded from the sets forming the basis for bibliometric studies in international comparison. This is not a simple task and it requires further bibliometric studies.

One can expect that similar language effects will occur when using Scopus data and that this effect will seriously aggravate by the extension of the coverage of the WoS and Scopus with national language journals.