Stability and change in public health studies in Colombia and Mexico: an exploratory approach based on co-word analysis.

OBJECTIVE: To determine the level of stability or change in topic areas published by public health journals in Latin America and the Caribbean, using keywords and co-word analysis, in order to support evidence-based research planning.
METHODS: Keywords were extracted from papers indexed in Scopus® that were published by the Revista de Salud Pública (RSP; Colombia), the Salud Pública de México (SPM; Mexico), and the Revista Peruana de Medicina Experimental y Salud Pública (RPMESP; Peru) for three periods: 2005 - 2007, 2008 - 2010, and 2011 - 2013. Co-word analysis was used to examine keywords extracted. Textual information was analyzed using centrality measures (inbetweenness and closeness). The hypothesis of stability/change of thematic coverage was tested using the Spearman's rho correlation coefficient. VOSviewer was used to visualize the co-word maps.
RESULTS: A moderate level of change in thematic coverage was observed in 2005 - 2010, as evidenced by the correlation coefficients for two of the 3-year periods, 2005 - 2007 and 2008 - 2010: 0.545 for RSP and 0.593 for SPM. However, in 2008 - 2013, more keywords remained constant from one period to the next, given the size of the correlation coefficients for the last 3-year periods: 2008 - 2010 and 2011 - 2013: 0.727 for RSP and 0.605 for SPM.
CONCLUSION: The research hypothesis was partially accepted given that just two consecutive 3-year periods showed a statistically-significant degree of stability in thematic coverage in public health studies. In that sense, this study provides compelling evidence of the effectiveness of using a combined approach for examining the dynamics of thematic coverage: centrality measures for identifying the main keywords and visual inspection for detecting the structure of textual information.

The World Health Organization defines the concept of public health as, “all organized measures (whether public or private) to prevent disease, promote health, and prolong life among the population as a whole…to provide conditions in which people can be healthy, and focus on entire populations, not on individual patients or diseases”(1). Countries of Latin America and the Caribbean have undertaken significant advances in reducing mortality and morbidity rates by applying effective strategies for improving public health (2 – 4). However, significant challenges remain due to the increasing prevalence of psychological (e.g., stress and depression) and lifestyle-related (e.g., obesity and coronary diseases) concerns. Therefore, current public health policies must enact effective advances against new challenges that are relevant to national needs, rather than to international research agendas with limited relevance for health problems at the national level (5 – 7).

Bibliometric studies on public health

Bibliometric studies on public health traditionally apply a descriptive approach using raw and adjusted indicators, such as total output, citation counts, or output adjusted per population (8 – 12). For example, in two studies, Soteriades and Falagas (10, 11) examined scientific output from 1994 – 2004 in biomedical areas (including public health and clinical medicine) for European Union countries and the United States. The research team found that even though the United States led scientific production in this area, European countries showed remarkable advances when adjusted indicators were considered, e.g., papers per 1 000 population, citations per billions of US$, and total product/gross domestic product (GDP). Another study (8) analyzed papers that were published from 1995 – 2004 and indexed in the Web of Science® (Clarivate Analytics, Philadelphia, Pennsylvania, United States; WoS). Authors of that study reported comparable results, and according to multiple regression analysis, GDP was a modest predictor (r2 = 0.53; P < 0.02) of publications for European countries.

The first bibliometric studies on public health in Latin American and the Caribbean (LAC) also applied a descriptive approach, whereas more recent studies identify factors associated with production and collaboration at the country or institutional level. Macías-Chapula (8, 13 – 15), a prolific bibliometric studies researcher, examined visibility and online access to public health research results in Argentina, Chile, and Mexico using data extracted from the LILACS (Latin American and Caribbean Center on Health Sciences Information, PAHO/WHO, São Paulo, Brazil) and MEDLINE® (U.S. National Library of Medicine, Bethesda, Maryland, United States) databases (15). He found that only 6 of 30 health science descriptors emerged as relevant for the analyzed countries in both databases: health services accessibility, health care reform, decentralization, health systems, consumer participation, and financing. Other researchers have analyzed Latin American scientific output compared to world production and combined bibliometric analysis with socioeconomic and health indicators (16, 17). Other advances in the field include analysis of manuscript quality; for example, those published by journals that are indexed in world-class citation databases, such as MEDLINE,® WoS,® and Scopus® (Elsevier, Paris, France) provide a greater level of certainty concerning validity and reliability.

Most of the studies mentioned thus far have a common feature: bibliometric analysis was carried out at the national, and in some cases, regional level. No similar studies focused on the contents of papers published by public health journals were found. Even when bibliometric studies based on citation analysis used a different approach than this study, a general overview was nevertheless included because they were the only ones that employed bibliometric techniques for analyzing the contents of Latin American public health journals. The following section describes a complementary approach for analyzing the contents of scientific production.

Co-word analytical approach

The co-word analytical approach for textual analysis relies on a central premise: simultaneous occurrences of signal words reflect the main topics of a documental body. This differs from co-citation analysis that provides a distribution of clusters grouping relevant authors, studies, and academic publications. French researchers Rip and Courtial (18) were the first to carry out an analysis based on co-occurrence of significant words and terms. Later, their approach was replicated by various fields, such as polymer chemistry, plant proteins, stem cells, patient adherence, intelligence network, and complementary medicine, among others (19 – 23).

An application of co-word analysis in the field of integrative and complementary medicine identified the level of content similarity in 1 329 articles published from 2007 – 2012 in specialized journals by using cluster techniques and a visualization software (20). A previous work with this textual method examined the knowledge structure and evolution of academic literature on patient adherence (24). In that study, researchers analyzed 2 308 articles (788 for the period 2000 – 2005, and 1 520 for 2006 – 2011). They found that in the first period, research themes were more general; while in the second, some thematic changes were observed as evidenced by the appearance of new terms. Despite its acceptance within the academic community, almost 20 years ago an eminent researcher on advanced methods for co-word analysis warned that because words change from one period to the next—in frequency of relations with other words and in terms of positional meaning—it is risky to assume conceptual stability of terms in texts over time (25).

To explore a useful tool for analyzing advances in public health research, this study focused on the main topics covered in the academic literature and any changes to these over time. Its objective was to determine the level of stability or change in topic areas—expressed by the keywords of each paper—published by journals in Latin America and the Caribbean in 2005 – 2013 and indexed by Scopus.® Ultimately, this information may serve as a useful tool for public health research managers and policymakers striving to make evidence-based policies that align the research agenda with country needs.

MATERIALS AND METHODS

This was a bibliometric study using the co-word approach to analyze any change in keywords in public health papers indexed in the Scopus® database in 2005 – 2013. To have a comparable documental body, public health studies were selected from three LAC countries with similar investments in health care as percentage of the GDP.

Documental body

The Scopus® database was chosen as the sole information source because of its broad coverage of academic literature produced in developing countries, especially those in Latin America and the Caribbean. In fact, this database recently added 59 Colombian academic publications to its index (26). Moreover, previous studies have documented its advantages over WoS® for carrying out bibliometric analysis (27, 28).

The countries selected for the study had 4% – 8% of their GDP assigned to health care; established public health research institutes or equivalents; and public health journals (per the journal's scope statement). These three criteria made it possible to select countries with comparable characteristics, and to exclude countries, such as Chile and Cuba, whose percentage of GDP for health care was outside the specified range.

Data collection and matrix generation

After applying the inclusion criteria, three countries were selected: Colombia, Mexico, and Peru. Despite the differences in public health agendas, the three countries share similar characteristics in terms of economic development, public health programs, demographic characteristics, and investment in improving public health coverage. Because of this commonality on contextual issues, it was considered appropriate to make comparisons among them.

The next step was to define the time period to carry out the co-word analysis using textual information from the public health journals of the selected countries: Revista de Salud Pública (RSP), Colombia; Salud Pública de México (SPM), Mexico; and Revista Peruana de Salud Pública y Medicina Experimental (RPMESP), Peru. Even though these journals are published in Spanish—and therefore, have lower citation rates than comparable journals published in English—two are in the second quartile of public health publications (SPM = 0.49 and RPMESP = 0.25), according to SCImago Journal & Country Rank (SCImago Research Group, Consejo Superior de Investigaciones Científicas, University of Granada, Extremadura, Carlos III [Madrid] and Alcalá de Henares, Spain). However, given that the study purpose was to analyze stability and change over time, RPMESP was excluded because the textual information was available only for one period (2011 – 2013) making it impossible to analyze variations from one period to the next.

Studies published from 2005 – 2013 were included in the analysis. The contents of the “keywords” field was extracted for each study (989 papers for RSP and 1 313 for SPM) and were entered into three Microsoft Excel™ (Microsoft Corp., Redmond, Washington, United States) worksheets, one for each 3-year period: 2005 – 2007; 2008 – 2010; and 2011 – 2013. These time periods were used because RSP and SPM are indexed in SciELO (Latin American and Caribbean Center on Health Sciences Information, São Paulo, Brazil), which uses 3-year periods for calculating impact factor.

Visualization of co-word maps

The Microsoft Excel™ Comma-Separated-Values (CSV) files were exported to Table 2 Net tool (SciencesPo Médialab, Paris, France) using the following settings: Normal for “type of network;” Index Keywords/Semicolon-separated for “nodes;” and one expression per cell for the “Links row number” option. The resulting .gefx files were opened in Gephi (Gephi Consortium, Paris, France) visualization software, version 0.8.2, for creating .net (Pajek format) files. To obtain centrality measures for primary keywords (betweenness and closeness), the Force Atlas algorithm was used with appropriate settings (repulsion strength: 10 000; attraction strength: 10; gravity: 10). Centrality measures are indicators that provide an understanding of the identified elements within a text body. “Betweenness” refers to a position property by which a node acts as a bridge or connector between other nodes in a social network, whereas “closeness” represents the average distance a node is from other nodes. Higher values of betweenness and closeness let us identify significant nodes (terms, in this study) within a social network, or a co-words-based map.

To identify words with semantic value, multi-term keywords were considered for obtaining rankings with the highest centrality measures. For visualizing the co-word maps, VOSviewer® (Nees Jan van Eck and Ludo Waltman, Leiden, Netherlands) software was used, with 50 for the “minimum total link strength” of an item field and the default settings for the remaining options.

At the end of the generation of co-word maps, authors constructed six datasets listing multi-term keywords with highest centrality values.

Data analysis

Given that keywords were ranked according to their centrality values, Spearman's rho was used to order ranks to determine if the rank in one 3-year period was associated with the next one. For each correlation coefficient, in addition to its magnitud, the statistical significance and the 95% Confidence Interval were computed based on Fisher's transformation. These computations were performed using IBM SPSS Statistics software, version 15.0 for Windows (SPSS Inc., an IBM company, Chicago, Illinois, United States) and Stata® 13.1 (StataCorp LP, College Station, Texas, United States). If two rankings were correlated, there was assumed to be some degree of stability; and if not correlated, the assumption was that there was some degree of change. Since the aim was to measure the degree of stability or change in thematic coverage and because there were no predefined thresholds, three ranges distributed proportionally were used for understanding the coefficient obtained with the correlation analysis:

0.01 – 0.29 = low stability / high change

0.30 – 0.69 = medium stability

0.70 – 1.00 = high stability / low change

It must be noted that though Spearman's rho ranged from −1 to +1, only positive values were considered because of the assumed minimum level of stability between one 3-year period and the following. Pearson's coefficient correlation was not used given that the rank order—of multi-word keywords with higher values of centrality—was measured at the ordinal level, rather than the interval or ratio levels. Regarding bivariate analysis, a low level of stability among periods in the journals was hypothesized.

RESULTS

Textual information from a total of 2 395 papers was analyzed. From the RSP, 689 papers; and from the SPM, 1 313 papers. Mexico's SPM had a higher publication rate than the Colombian RSP.

Centrality measures

As seen in Tables 1 and 2, the second step of the analysis included describing the 20 terms with the highest betweenness values (a complete listing of the 100 multi-term keywords with the highest centrality values for RSP and SPM is available upon request from the corresponding author). The first 20 keywords of Colombian thematic coverage showed some degree of stability during the period 2005 – 2010 given that at least 14 important multi-term keywords remained during 2005 – 2007 and 2008 – 2010. For the period 2008 – 2013, only 12 multi-term keywords remained among the central ones for both 3-year periods analyzed: 2008 – 2010 and 2010 – 2013 (Table 1). Most of the terms with higher centrality values dealt with demographics (middle-aged, preschool child, socioeconomic factors, etc.) and methodological (cross-sectional study or comparative study) issues. The same trend was observed for the Mexican thematic coverage.

TABLE 1

Rank order of terms with higher centrality values for measuring stability and change in the Revista de Salud Pública (Colombia), 2005 – 2013

Keywordsa	2005 – 2007 (n = 142)	2008 – 2010 (n = 293)	Keywordsa	2008 – 2010 (n = 293)	2011 – 2013 (n = 254)
Comparative study	1	5	Middle-aged	1	1
Middle aged	2	1	Young adult	2	6
Child, preschool	3	11	Cross-sectional studies	3	2
Preschool child	4	12	Cross-sectional study	4	3
Public health	5	29	Comparative study	5	23
Socioeconomic factors	6	7	Psychological aspect	6	7
Cross-sectional studies	7	3	Socioeconomic factors	7	8
Cross-sectional study	8	4	Urban population	8	4
Isolation and purification	9	22	Organization and management	9	5
Infant, newborn	10	34	Health care delivery	10	70
Risk factor	11	14	Child, preschool	11	9
Risk factors	12	15	Preschool child	12	10
Catchment area (health)	13	81	Health service	13	27
Residential care	14	82	Risk factor	14	12
Urban population	15	8	Risk factors	15	13
Psychological aspect	16	6	Aged, 80 and over	16	20
Drug effect	17		Urban health	17	32
Organization and management	18	9	Health services accessibility	18
Clinical trial	19	63	Models, theoretical	19	21
Aged, 80 and over	20	16	Theoretical model	20	22

Centrality values for keywords were computed with the software Gephi (Gephi Consortium, Paris, France).

TABLE 2

Rank order of terms with higher centrality values for measuring stability and change in the Salud Pública de México, 2005 – 2013

Keywordsa	2005 – 2007 (n = 432)	2008 – 2010 (n = 468)	Keywordsa	2008 – 2010 (n = 468)	2011 – 2013 (n = 413)
Controlled study	1		Young adult	1	2
Middle aged	2	2	Middle aged	2	1
Major clinical study	3		Comparative study	3	3
Cross-sectional studies	4	5	Organization and management	4	8
Comparative study	5	3	Cross-sectional studies	5	6
Risk factors	6	18	Cross-sectional study	6	7
Age factors	7	31	Psychological aspect	7	21
Child, preschool	8	8	Child, preschool	8	4
Health survey	9	27	Preschool child	9	5
Public health	10	12	Isolation and purification	10	14
Statistical analysis	11		Urban population	11	25
Risk factor	12	17	Public health	12	10
Urban population	13	11	Clinical trial	13	58
Data interpretation, statistical	14		Health care policy	14	24
Time factors	15	46	Retrospective studies	15	70
Preschool child	16	9	Retrospective study	16	71
Isolation and purification	17	10	Risk factor	17	15
Aged, 80 and over	18	21	Risk factors	18	16
Antibiotic resistance	19	38	Utilization review	19	26
Sex factors	20	74	Health service	20	11

Centrality values for keywords were computed with the software Gephi (Gephi Consortium, Paris, France).

Regarding keywords with higher centrality values for the Mexican public health journal (SPM), an inverse pattern was observed compared to the Colombian journal's thematic coverage; that is, the period 2008 – 2013 had a higher level of stability compared to the previous 6-year period. From 2005 – 2010, just 10 multi-term keywords remained among the terms with the highest centrality values, whereas during 2008 – 2013, 12 multi-term keywords appeared in the top 20 for both 2008 – 2010 and 2011 – 2013 (Table 2).

Co-word analysis-based mapping

For a different perspective on the results of the co-word analysis, Figures 1 and 2 show RSP's network sociograms for the periods 2005 – 2007 and 2008 – 2010. The network map for keywords in RSP (2005 – 2007) showed a low level of thematic diversity because they appeared in clearly delimited clusters and with a low degree of overlap. For example, the left-side cluster includes terms related to health studies (e.g., clinical trials, cohort analysis, disease transmission, genetics, and health survey) and health care infrastructure (e.g., hospitals, information, service, intensive care units, and urban health). The cluster located at the center of the network map includes terms related to demographics (e.g., adult, female, male, middle aged) and health service (e.g., health care quality, health promotion, health service, public health).

FIGURE 1

Network map of co-word analysis for keywords in the Revista de Salud Pública (Colombia), 2005 – 2007

FIGURE 2

Network map of co-word analysis for keywords in the Revista de Salud Pública (Colombia), 2008 – 2010

A different picture emerged from the network map for keywords in RSP (2008 – 2010) that showed a high level of thematic diversity. Clusters were not clearly delimited and there was a high level of overlap among them, especially at the center of the map where five clusters were located. Nevertheless, in the peripheral areas, two clusters were clearly defined. In the upper left area, there was a cluster of keywords associated with health services (e.g., consumer participation, health care delivery, health care quality, public hospital), and in the bottom right, another, but related to heart disease and associated factors (e.g., alcohol drinking, body mass, cardiovascular disease, fitness, hypertension, obesity). For the other areas of the network map, the high level of overlap made it difficult to identify clusters with clearly defined zones.

Bivariate analysis of stability and change

Correlation analysis showed mixed results due to a high level of stability in both journals for the period 2008 – 2013, but a medium level of thematic stability for 2005 – 2010.

The research hypothesis was partially accepted because in the first period (2005 – 2010), the level of thematic change showed moderate variability, as evidenced by the correlation coefficients: 0.545 and 0.593. That means that between the years 2005 – 2007 and 2008 – 2010, the keywords assigned to each paper appeared in a different order, and in some cases, they were different. However, in the second period (2008 – 2013), especially for RSP, thematic coverage was more stable than in the previous, suggesting that the keywords assigned in 2008 – 2010 and 2011 – 2013 appeared in almost the same order and were the same in most cases.

DISCUSSION

Previous co-word analysis of thematic coverage in public health used cluster techniques and social network analysis for examining co-word matrices, and one of them worked with 6-year periods, 2000 – 2005 and 2006 – 2011 (21, 24). For this study, we utilized Spearman's rho correlation coefficient and considered 3-year periods (2005 – 2007, 2008 – 2010, 2011 – 2013) for measuring stability and change of thematic coverage. The research hypothesis was partially accepted given that just one period showed a significant degree of stability in thematic coverage. We found that most of the keywords were visible in both periods, even while the frequency and word relations changed. For example, for the group “diseases and conditions,” the new terms were “chronic disease, clinical practice, cardiovascular disease, apnea syndrome, etc.;” while in the group “research methods and techniques,” the new terms were “RCT, meta-analysis, cohort, double-blind, follow-up, pharmacy records, reliability, and validity, etc.”

The clusters of keywords discussed earlier are based on associations between nodes, which allow identification of significant nodes within the map of co-words. Although those words are compiled manually from each publication, the analysis is carried out by visualization software. Even when we did not analyze the full text of a document and only keywords were considered, the extracted terms provide a general overview of the topic.

This study provides compelling evidence of the effectiveness of using a combined approach for examining the dynamics of thematic coverage: centrality measures for identifying the main keywords and visual inspection for detecting the structure. In that sense, it is a useful approach for monitoring research themes in public health studies, not only in Latin America, but also in other geographic areas. For example, if a health ministry were interested in advances to address the Sustainable Development Goals, it could apply a similar approach to find specific studies focused on prevention programs for malaria or tuberculosis, and compare thematic coverage with prevalence or incidence ratios for those diseases.

We are not suggesting that there is a direct relationship between bibliometric indicators and advances in disease-prevention programs, but we are introducing a complementary perspective that goes beyond the descriptive level and explores the association with other demographic or public health indicators. One example of this is a recent study that examined the relationship between a set of bibliometric indicators and an index for health inequalities in 10 countries of Latin America (16). Even when the authors reported that they did not find a significant correlation among those indicators as measured by Spearman's rho, they offered a plausible explanation: research results do not necessarily influence the decisionmaking process at the ground level. This apparent lack of connection between theory and practice presents another challenge: how to translate research findings in a way that they can be adopted and incorporated at different levels of public health policy.

Limitations. There were three limitations to this study. First, there is a known publication bias against studies in non-English journals. To address this coverage bias, we decided to include all contributions, but focused on the keywords assigned to each paper. Given that the keywords appear in English as well as Spanish, there were no translation issues. In that sense, our findings can be generalized to English and Spanish content collected for the study. Secondly, regarding time periods, we found that some previous studies used 5-year periods for analysis (29, 30). However, because our analysis focused on stability and change, we decided to work with 3-year periods. Lastly, we did not work with the Medical Subject Heading (U.S. National Library of Medicine, Bethesda, United States; MeSH) for co-word analysis because we extracted data from Scopus,® a database that does not have a standardized list of keywords. Therefore, these results can only be generalized to non-MeSH keywords. Results using keywords from a normalized and structured list of subject headings may have differed from those presented here. In order to provide evidence for convergent validity, a future study could use data from MEDLINE® to compare results between Scopus and MeSH keywords.

Conclusions

This study shows the effectiveness of a co-word analysis approach for analyzing stability and change in thematic coverage by two public health journals in Latin America. LAC health ministries and public health research institutes can replicate the procedure described in this review to verify whether or not the topics being published by their domestic or regional public health journals reflect their country's priorities, programs, and policies. Also, regarding the output of research funding, policymakers can use this procedure to provide additional indicators of whether resources allocated for public health programs have met their established evaluation criteria.

It is known that the research agenda of developed countries does not always correspond with that of developing countries, especially for matters such as immunization and disease surveillance. Future work should focus on validating this co-word approach by carrying out contextualized studies for public health issues relevant to each country.

TABLE 3

Correlation matrixes for measuring stability and change of keywords in the Revista de Salud Pública (Colombia; RSP) and the Salud Pública de México (SPM), 2005 – 2013

Journal name	Spearman's correlations, significance, and 95% Confidence Intervals (95%CI)
	RSP 2008 – 2010	RSP 2011 – 2013
RSP 2005 – 2007	0.545a
RSP 2008 – 2010	95% CI: 0.325 – 0.710	0.727a
	n = 54	95% CI: 0.565 – 0.835
		n = 51
	SPM 2008 – 2010	SPM 2011 – 2013
SPM 2005 – 2007	0.593a
SPM 2008 – 2010	95% CI: 0.349 – 0.762	0.605a
	n = 41	95% CI: 0.409 – 0.748
		n = 57

P < 0.01.