HIV research productivity and structural factors associated with HIV research output in European Union countries: a bibliometric analysis.

OBJECTIVES: To assess HIV/AIDS research productivity in the 27 countries of the European Union (EU), and the structural level factors associated with levels of HIV/AIDS research productivity.
METHODS: A bibliometric analysis was conducted with systematic search methods used to locate HIV/AIDS research publications (period of 1 January 2002 to 31 December 2011; search databases: MEDLINE (Ovid, PubMed), EMBASE, ISI-Thomson Web of Science; no language restrictions). The publication rate (number of HIV/AIDS research publications per million population in 10 years) and the rate of articles published in HIV/AIDS journals and selected journals with moderate to very high (IF ≥3) 5-year impact factors were used as markers for HIV research productivity. A negative binomial regression model was fitted to assess the impact of structural level factors (sociodemographic, health, HIV prevalence and research/development indicators) associated with the variation in HIV research productivity.
RESULTS: The total numbers of HIV/AIDS research publications in 2002-2011 by country ranged from 7 to 9128 (median 319). The median publication rate (per million population in 10 years) was 45 (range 5-150) for all publications. Across all countries, 16% of the HIV/AIDS research was published in HIV/AIDS journals and 7% in selected journals with IF ≥3. Indicators describing economic (gross domestic product), demographic (size of the population) and epidemiological (HIV prevalence) conditions as well as overall scientific activity (total research output) in a country were positively associated with HIV research productivity.
CONCLUSIONS: HIV research productivity varies noticeably across EU countries, and this variation is associated with recognisable structural factors. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

To the best of our knowledge, this is the only study to date that used bibliometric methods to evaluate the output of peer-reviewed publications on HIV/AIDS from Europe for the past decade.

We present data on three different measures of research productivity. The measures used were highly correlated, indicating measurement of the same underlying phenomenon.

Defining research by publication in peer-reviewed journals and presentations (abstracts) at the leading international conferences of the field underestimates the total HIV research productivity.

The databases searched consist largely of English-language journals, thereby possibly contributing to selection bias due to language barriers.

Introduction

HIV infection is of major public health and research importance in Europe.1 In 2011, 28 038 HIV diagnoses were reported by countries in the European Union (EU) and European Economic Area.2 Across the region, there is a significant variation in the rates of new HIV diagnoses: the four countries with the highest rates of HIV diagnoses per 100 000 population in 2011 were Estonia (27.3), Latvia (13.4), Belgium (10.7) and the UK (10). The lowest rates were reported by the Czech Republic (1.5) and Slovakia (0.9).2

An analysis by Ramos et al3 indicated that countries with the highest numbers of individuals living with HIV are not leading the scientific output on HIV/AIDS. An analysis based on articles published in three journals focusing on HIV/AIDS from 1986 to 2003 revealed that the USA and Western Europe together, and the five most developed world regions (of nine), accounted for the great majority of the world’s research production on HIV/AIDS (83% and 92%, respectively).4

Results from bibliometric analyses can be a critically important source of objective information about the quantity and quality of scientific work.5 The primary assumption supporting the use of bibliometrics is that exchange and recognition of research results is desired and is one of the key driving forces in the advancement of science.6 Bibliometric studies of HIV research have described the distribution and variation in scientific output over time7–13 in addition to impact and linkage.5

We present a study aimed at describing HIV/AIDS research productivity in EU countries, and at assessing the structural level factors associated with HIV/AIDS research output in these countries.

Methods

Evaluation of scientific output

A bibliometric analysis was used to assess scientific output and to compare the relative scientific contributions of the 27 member states of the EU.14 Systematic search methods were used to locate HIV/AIDS research publications.

Data sources and sample

Criteria for considering studies for this review

We included published research results (conference abstract and journal articles; hereafter referred to as publications), articles in the press as well as records that were ‘In Process’ (not yet fully indexed), irrespective of the study design (all publications, primary and secondary, commentaries, editorials, etc, were retained for the analysis). The search was limited to the period of 1 January 2002 to 31 December 2011 and to the 27 countries of the EU at that time.

Search methods for identification of studies

We searched the following bibliographic databases: MEDLINE (Ovid, PubMed; includes citations from approximately 5600 scholarly journals published), EMBASE (includes citations from approximately 8000 scholarly journals published), ISI-Thomson Web of Science (enables access to the published literature from conferences, symposia, seminars; selected conferences for the period 2002–2011: International AIDS conferences, Conferences on Retroviruses and Opportunistic Infections; The International Society for Sexually Transmitted Diseases Research conferences). To be comprehensive, we used a combination of controlled vocabulary (MeSH, Emtree, DeCS, including exploded terms) and free-text terms (considering spelling variants, synonyms, acronyms and truncation), and the filter used to search HIV-related publications was based on the search terms defined by Volmink et al15 (see online supplementary annex 1). This search filter was modified and adjusted to the search engine properties of each of the selected electronic databases. The filters for all databases and a full search strategy for PubMed can be found in online supplementary annex 1 (Electronic search strategies).

Bibliographic information (title, journal, year of publication, authors’ names, the organisation to which the first author was affiliated and the country in which the first author's organisation was located) was derived from citations of publications and used for the analysis presented. The source (origin) country of the publication was defined on the basis of the first author’s affiliation.

We used the online research management tool Endnote X5 for data processing (Thomson Reuters EndNote X5, http://www.endnote.com). Data abstracted from all databases searched were merged and duplicates deleted from the data set used for analysis. The data abstraction process was entirely conducted by computer. Given the very large number of publications identified through electronic searches, it would not have been feasible to have authors review the abstracts or titles to determine if the publications had been correctly classified as ‘HIV/AIDS’ publications or to verify that the organisational affiliations of the first authors were correct. For further analysis, the data were exported from Endnote into R (R V.2.15.1, http://www.r-project.org/).

Measures

We used three measures to assess HIV research productivity. First, we derived the number of HIV research studies (not restricted by study design or research area, ie, basic science, clinical, prevention or implementation research) indexed in selected data sources for a specific country. For comparison purposes, total scientific publication rates (per 1 000 000 population) were calculated.

In addition, we hypothesised that if a researcher working in the HIV/AIDS area wants his or her work to be noticed by large numbers of researchers in the HIV/AIDS field, there are two main strategies to use. One is to publish in an HIV/AIDS journal, as other AIDS researchers will be paying attention to the contents of AIDS journals on a regular basis. The second strategy is to publish in a relatively high impact general journal as AIDS researchers will also pay more attention to what is published in high impact medical journals. Therefore, we assessed the numbers and rates of research studies published in all HIV/AIDS journals listed in the Web of Knowledge Journal Citation Reports (JCR)16 for the year 2012 (n=16) and in the selected journals with a 5-year impact factor (IF) ≥3 (n=19; table 1). Acknowledging that key research in the HIV/AIDS arena in the period of observation has been published in non-HIV journals, a set of infectious disease, general medicine and science journals with 5-year IF ≥3 was selected by the authors of the current analysis (journals such as NEJM, Lancet, PLoS Medicine; see table 1). For the last two measures (publications in all AIDS/HIV journals and in journals with IF ≥3), conference publications were excluded.

Table 1

List of HIV/AIDS journals and selected journals with moderate or higher 5-year impact listed in the Web of Knowledge Journal Citation Reports factor in the year 201216

No	Abbreviated journal title	Five-year impact factor (IF)
All HIV/AIDS journals
1	AIDS	6.1
2	AIDS Patient Care STDS	2.9
3	AIDS Res Hum Retrov	2.4
4	AIDS Res Ther
5	AIDS Rev	4
6	Afr J AIDS Res	0.5
7	Curr Opin HIV AIDS	3.2
8	Int J STD AIDS	1.1
9	J Assoc Nurse AIDS C	1.3
10	J Int AIDS Soc
11	Curr HIV Res	1.9
12	Curr Opin HIV AIDS	3.2
13	HIV Clin Trials	2
14	HIV Med	3.3
15	South Afr J HIV Med	0.6
16	J Acquir Immune Defic Syndr	4.6
Selected journals with moderate or higher IF (≥3)
1	New Engl J Med	50.8
2	Lancet	36.4
3	JAMA	29.3
4	Nat Med	27.1
5	Nature	38.2
6	Science	33.6
7	Ann Intern Med	16.3
8	Lancet Infect Dis	18.1
9	J Infect Dis	5.9
10	Clin Infect Dis	9.0
11	Cochrane Database Syst Rev	6.6
12	PLoS Med	16.4
13	Emerg Infect Dis	6.3
14	Am J Public Health	4.8
15	Curr Opin Infect Dis	4.6
16	Clin Microbiol Infect	4.5
17	Antivir Ther	3.4
18	Am J Infect Control	3.3
19	BMC Infect Dis	3.1

Information on journal IFs was obtained from the Web of Knowledge JCR.16 17

Further, we evaluated the structural-level factors related to variation in HIV research productivity in EU countries. Data were obtained from reports published by the World Bank,18 WHO19 and Joint United Nations Program on HIV/AIDS.20 The following indicators were selected and data collected on explanatory factors for HIV research productivity on a structural level:

Demographic indicator: population, total (mid-year);

Economic indicator: gross domestic product (GDP) per capita (value, in US$);

Education indicators:

Public spending on education per capita,

Public spending on education, total (% of GDP);

Research and development (R&D) indicators:

R&D expenditure per capita,

R&D expenditure, per cent of GDP,

Researchers in R&D (per million population),

Overall research productivity (publications total per million population);

Population health indicators:

Health expenditure per capita (value in US$),

Life expectancy at birth, total (years),

Physician density (number per 1000 people);

HIV epidemic indicators:

Estimated number of HIV-infected persons, adults (15+),

HIV prevalence, adult population (15–49 years).

Three of the indicators were calculated using World Bank data—research productivity per million of population (using the number of scientific and technical journal articles and national population), public spending on education per capita (using public spending on education (% of GDP), GDP per capita) and R&D expenditure per capita (using R&D expenditure (% of GDP), GDP per capita). For the analysis, data were averaged over the years 2002–2011. If it was not possible to find data for all the years, the data were averaged over the years where it was available.

Statistical analysis

For each EU member country, the total publication rate (per million population in 10 years) as well as that of articles in selected journals with IF greater than 3 and all HIV/AIDS journals articles were supplied nominal 95% CIs, which were computed based on assuming Poisson distribution for the numbers of publications and applying the mid-P principle.21 Likewise, 95% CIs for the proportion of IF ≥3 articles out of all publications for all countries were computed based on binomial distribution and applying the mid-P principle here, too. Function pois.exact in package epitools and function midPci in package PropCIs, both contained in R environment for statistical computing,22 were used in these computations. To quantify the degree of dependence between the three publication productivity measures, Spearman's rank correlation coefficients were calculated.

A negative binomial regression model was used to assess the impact of selected structural factors associated with variation in the overall publication rate describing HIV research productivity per million population in 10 years across the EU countries. This model was preferred over conventional Poisson regression because substantial over-dispersion on nominal Poisson variance was present, as indicated by the estimated value 9.8 and SE 3.4 of the dispersion parameter in the final negative binomial model. Owing to the strong correlations (r>0.9) observed between variables describing general economic conditions, spending on education and health indicators, only GDP per capita from those indicators was included in the final regression model. Research productivity as measured by the mean annual number of scientific publications per million population was selected to represent the overall level of research and development activity in the country. To describe the burden of the disease, the model included the estimated prevalence of HIV in adults in the year 2009. Finally, the size of the population was added to the model with the idea that in bigger countries there is more ‘critical mass’ in the scientific community as a whole than in smaller countries, and it is conceivable that this would be positively associated with the research output in any specific field, like HIV/AIDS research.

Since each of the selected explanatory variables had a wide range and skew distribution, all of them were transformed to the natural logarithmic scale. The negative binomial regression model was fitted using the function glm.nb in the package MASS, also contained in the R environment. When presenting the results, the estimated regression coefficients (β) and their SEs were transformed to rate ratios (RR) and 95% CIs such that an estimated RR (RR=exp(β×log 2)) associated with a particular variable describes the multiplicative effect of doubling the value of this variable on the publication rate assuming ceteris paribus, that is, no change in the other variables.

Results

A total of 90 563 articles indexed by MEDLINE (PubMed, Ovid), EMBASE and Web of Science (WOS; selected conferences) from January 2002 to December 2011 for the selected 27 European countries were retrieved for this analysis. Six per cent of the entries where the origin country assignment was not possible were excluded from the analysis. The total numbers of HIV/AIDS publications in selected databases (with the numbers of publications from EU 27 retained for the analysis in parentheses) in the period January 2001 to December 2011 were as follows: PubMed 126 977 (28 231), EMBASE 119 278 (32 803), Ovid 89 160 (29 000), and WOS (ISI-Thomson Web of Science) 1646 (529).

After deleting duplicate entries (duplicate entries of the same article by different databases), 39 645 publications from 27 EU countries were included in the analysis.

The total numbers of HIV/AIDS research publications in 2002–2011 by country ranged from 7 to 9128 (median 319). The median publication rate (per million population in 10 years) was 45 (range 5–150) for all publications (see figures 1 and 2 and online supplementary annex 2).

Figure 1

Publication rates (per million population in 10 years, with 95% CIs) of all research publications on HIV/AIDS in member countries of the European Union during 2002–2011.

Figure 2

HIV/AIDS research publication rates (per million population in 10 years, with 95% CIs) in HIV/AIDS journals (JCR 2012), and of articles published in selected moderate or high impact factor journals (IF ≥3) in member countries of the European Union during 2002–2011 (absolute numbers by country are given in parentheses: publications in HIV/AIDS journals on the left, and publications in journals with IF ≥3 on the right).

Across all countries and the period of observation, the median publication rate for articles in HIV/AIDS journals was 4.5 (0–36), and the proportion of total HIV/AIDS research published in these journals was 16% (from 0% in Latvia, Lithuania and Malta to 38% in Cyprus).

For articles in journals with moderate or higher IFs (≥3), the median publication rate was 2.3 (0–13.2). In the graphical presentation of the ‘league table’ according to the total publication rate, the 27 European countries can be divided into three broad groups: (1) ‘top 10 countries’: UK to Ireland, in which the total rate (per million in 10 years) ranged from 75 to 150, and the rate of articles in journals with IF ≥3 varied from 4.4 to 13.2; (2) ‘intermediate six’: Portugal to Finland, the total rate varied between 35 and 50; and the rate of IF ≥3 journal articles from 1.8 to 3.7; and (3) ‘remaining 11 countries’, in which the total rate ranged from 5 to 25, and the rate of articles in IF ≥3 journals being at most one (per million population in 10 years). Across all countries and the period of observation, 7% of the publications were published in journals with a moderate or higher IF (≥3).

The three measures of research productivity used—the total publication rate, publication rate in HIV/AIDS journals and publication rate in selected journals with moderate or higher 5-year IF—were highly correlated (for all pairwise correlations, the Spearman r>0.88).

Table 2 summarises the impact of selected structural factors associated with variation in HIV research productivity in EU member countries, as estimated by a negative binomial regression model. GDP per capita had the strongest effect on publication rate, followed by HIV prevalence, size of the population and total scientific activity level in the country.

Table 2

Impact of structural-level factors associated with HIV research productivity (number of HIV publications per million population in 10 years) in EU countries as estimated by a negative binomial regression model

Structural-level characteristics	β	SE	RR	95% CI
Baseline	3.68	0.09	39.6*	33.2 to 47.5
GDP per capita	0.97	0.15	1.97	1.62 to 2.39
Population size	0.16	0.05	1.11	1.03 to 1.20
Scientific productivity	0.26	0.13	1.19	1.00 to 1.42
HIV prevalence in 2009	0.35	0.10	1.27	1.12 to 1.46

The entries are regression coefficients (β) and their SE associated with the original model with log-transformed covariates, as well as the baseline rate* and RR associated with doubling the values of the covariates.

*The baseline rate=exp(3.68) per million population in 10 years refers to a country with GDP per capita=US$25 000, population size=10 million, scientific productivity=400 articles per million population annually and adult HIV prevalence=0.1%.

EU, European Union; GDP, gross domestic product; RR, rate ratio.

Discussion

Research is considered incomplete and cannot contribute to advancing knowledge unless it is published. To the best of our knowledge, this is the only study to date that used bibliometric methods to evaluate the output of peer-reviewed publications on HIV/AIDS from Europe for the past decade.

Over the period of 2002–2011, a total of 40 000 publications on HIV/AIDS originated from 27 EU countries (in comparison to 53 500 publications from the USA (data not shown)). According to our results, HIV research productivity differs significantly across the European countries, being substantially higher in the majority of the old EU countries (pre 2004) in Western and Northern Europe than in the new members (since 2004), mostly from Central and Eastern Europe.

In addition to significant differences in overall HIV/AIDS research productivity, there are observable differences also in the quantity and proportion of published research in journals with a higher IF between the countries of EU. It is noteworthy, however, that the IF (the median number of citations for a given period of the articles published in a journal17) of journals in which the work was published is a useful but highly controversial indicator of performance. While not a direct measure of quality, IF , as a citation measure, can be indicative of the influence journals have within the literature of a discipline (same subject area).23 24

The structural level factors associated with HIV/AIDS research productivity in our analysis—economic well-being (GDP per capita) and disease epidemiology (adult HIV prevalence) in the countries—are intuitively understandable. Similar findings have been reported previously for other regions or areas of research.3 25 26 Interestingly, while GDP is a commonly used economic wealth indicator, scientific productivity is a much better predictor of economic wealth and human development according to the recent research by Jaffe et al.27

The observation that HIV/AIDS research productivity is positively associated with the adult HIV prevalence in countries is in agreement with the ‘burden of disease’ approach to research priorities.28 Our analysis suggesting a ‘critical mass’ type effect (the size of total population contributing to the number of research publications per million population) may be indicative of the need to explore whether this holds for other areas of research, and if so, to then conduct research to understand the mechanisms through which it occurs (whether this relies on the critical mass of researchers, or patients, study subjects to be enrolled). There are other structural factors that could explain differences between countries such as wide variation in data protection laws restricting data linkage in public health and epidemiological research, especially for a sensitive diagnosis like HIV, but not for basic science.

Factors in addition to the structural factors explored can be related to the differences in research output between high-income countries and medium-income/low-income countries in the EU. The results of randomised controlled trials of new treatments from pharmaceutical studies are more likely to be published in higher impact journals and conduced in more developed countries. Further, a subset of HIV research occurring within Europe is in the context of clinical cohorts such as COHERE, Cascade and EuroCoord. Although these are, by design, cross countries they often have a first author from the more developed countries as partner (ie, the UK).

We would note that defining research by publication in peer-reviewed journals and presentations (abstracts) at the leading international conferences of the field underestimates the total HIV research productivity (by omitting works in non-indexed journals, and non-journal formats such as books, dissertations, reports, etc.). Further, it is important to note that the databases searched consist largely of English-language journals, thereby possibly contributing to selection bias due to language barriers. We tried to mitigate this effect by not setting language filters for the searches.

We used a purposive list of journals to assess the volume and proportion of HIV research published in journals with IF ≥3. Notably, this might have omitted high impact research published in other journals. In addition, for journals’ IFs there is variation among subject areas; basic science journals tend to have higher IFs than journals in specialised or applied areas. Still, our analysis was limited to the specified field (HIV/AIDS) not restricted by the study design or research area (ie, basic science or clinical research, etc); basic science journals (Nature, Science), general medicine (Lancet, NEJM) and specialised journals (AIDS, JAIDS, BMC Infectious Diseases) were specified for IF categorisation. Of note, one could argue that given the relative availability of antiretroviral treatment in Europe, functioning health infrastructure and systems, the underlying factors for continued transmission are mostly behavioural and social factors and not biomedical. Thus, one would perhaps expect the emergence of more social and behavioural science studies of relevance to the European HIV context.

By using the author addresses listed in the bylines of research articles, one can only identify countries (and organisations) where the authors were employed when the research was done or where the article was written, or both. This might lead to some misclassification of the ‘country of origin’ of the work. Also, we suspect that this misclassification is more likely to undercount research in low publication countries. In addition, since we assigned each publication to only one country, we de-emphasised studies with researchers and data from multiple countries. A special analysis of multinational research projects would be of interest for further research.

For many countries, the numbers of publications in journals with a moderate or higher IF were very small; consequently, the associated error margins for both the pertinent publication rate and the proportion of these articles out of all publications are quite wide and should therefore be interpreted with caution. Finally, we studied European countries only, and thus cannot generalise our findings straightforwardly about the association between structural factors and research productivity outside of Europe, although the relationships would appear to be plausible for other countries, too.

We presented data on three different measures of research productivity. However, the measures used were highly correlated, and indicating on measurement on the same underlying phenomenon, and that choice among the three sets of journals would not affect any conclusions to be drawn from the data.

While these limitations are important, we feel it is extremely unlikely that they created the patterns that we observed in the data. Rather, we believe that these patterns were sufficiently strong enough to be observed in the data despite the limitations. Also, we would note that the analysis focuses on the research production from the first decade of the 2000s, and additional research to examine possible changes in research productivity would be useful.

Our findings have several important implications. We believe that evaluating the publication output from HIV research in Europe sets the basis for analysing HIV research and setting strategies to increase research output from all EU countries and its visibility in the scientific community. Highly skewed distribution of productivity among individual scientists29 or universities30 has been explained by a process of accumulative advantage, and the accumulative advantage results in increasing inequality.29 The same model might also be applicable for countries: owing to feedback through recognition and resources and facilities, scientifically productive countries maintain or increase their productivity. Recognising the structural factors (economic well-being, size of the population) of research productivity stresses the significance of strategic decisions in research development. Johnston31 surveyed research productivity studies and found strong evidence from the existing literature that the scale and continuity of funding helped higher level research activity. The weight of the factors suggests recognising the need to further develop research capacity in countries that are currently not highly productive but face public health problems where research might contribute to successfully addressing the problems. This would probably involve both national and international (eg, European Union) resource commitments. The commitments would need to be sustained over time and not limited to a single public health problem. As the amount of scientific research proliferates and science becomes more costly to produce, scientific strategy and funding agencies are increasingly interested in objectively assessing the research output and factors associated with it.32 Despite the limitations, bibliometric analyses such as ours can be used to describe patterns of performance of scientific research across countries. Second, bibliometric analysis as a tool has the potential to assist decision-makers in understanding science and investing in science. This method can be used as an evaluation instrument in research together with other methods, such as expert opinion and panels.