A text mining approach for adapting a school-based sexual health promotion program in Colombia.

A common practice among clinical psychologists and other health professionals is the use of school-based sexual health promotion programs as a means for preventing sexually transmitted infections. A fundamental criterion for the designing and adaptation of these programs is the age of their target populations because limited education and language are the most relevant factors that limit the efficacy of these programs. The contribution of this paper consists of assessing both the readability of the written materials that accompany the contents of a Spanish-written school-based sexual health promotion program used in Colombia, as well as the words co-occurrence network structure of its contents. The readability of the evaluated program corresponded to its intended target population aged between 14 and 19, with the schooling of 9-13 years of education. The resulting words co-occurrence network structure of the COMPAS program also mirrored its theoretical content. These results all together are deemed as empirical evidence of the adequacy of the program.

Introduction

School-based intervention programs are frequent. Interventions for autism (Luxford et al., 2017), overweight (Mahmood et al., 2014), depression and anxiety (Werner-Seidler et al., 2017), violence in teen dating (De La Rue et al., 2017), emotional learning (Taylor et al., 2017), drugs and alcohol abuse (Newton et al., 2017), STI and HIV prevention (Mirzazadeh et al., 2018) or unintended pregnancies (Rabeea’h et al., 2017) among others, have been performed. Most of these programs impact the well-being and health of adolescents and young people. Thus, the economic, social, and health impact of these programs is of paramount relevance for governments. School-based psychological interventions seem to be cost-effective for most of these problems (Ekpu and Brown, 2015, Fonner et al., 2014, Lee et al., 2017, Moessner et al., 2016, Morales et al., 2018a, Stallard et al., 2013). However, realizing gains is ultimately dependent on the program efficacy (Lee et al., 2017).

One of the most relevant variables affecting efficacy is the limited education and language of the target population (CDC, 2014). Nonetheless, recommendations for evaluating the linguistic adequacy of the programs are scarce. Professionals only count on general guiding principles such as: applying the program in the language of the cultural group that will receive it (Eldredge et al., 2016, p. 391), conducting focus groups (Resnicow et al., 1999, p. 15) or ensuring that the vocabulary used fits the rules of plain language (Eldredge et al., 2016, p. 443). As far as we know, the only method for evaluating the reading grade level, writing style or vocabulary specifically for health-related issues is the Suitability Assessment of Material (SAM; Doak et al., 1996).

SAM is an assessing scale for health-related information. One of its subscales, literacy demand, evaluate the reading grade level with a straightforward item: Superior = 5th grade or level or lower, Adequate = 6th to 8th grade, and Not Suitable = 9th grade or above. Assessment of comprehension programs is essential and usually neglected. Many health and health-care materials are written at levels that people cannot understand (Foster and Rhoney, 2002, Hill-Briggs and Smith, 2008).

This latter circumstance leads to clinical psychologists to look for methodological alternatives that allow them to evaluate the language used for written materials empirically. Here, computational linguistics play a crucial role for these purposes. Roughly speaking, computational linguistics is an interdisciplinary field concerned with the statistical or rule-based modeling of natural language, which includes the processing of languages and text analysis (Feldman and Sanger, 2007). According to Correa et al. (2018), the so-called “readability measures” are particularly useful for text analyses. The scientific foundation of these formulas relies on the evident fact that the use of different words in written language is more abundant in adults than in children. In other words, the higher the schooling years of a person, the broader his written language, quantified by a metric called lexical diversity (Durán et al., 2004). Lexical diversity is traditionally calculated through the “Type-Token Ratio”; that is, the number of different words (types) divided by the total number of words (tokens) in a document. By considering these ideas, a readability index quantifies the ease with which a reader can understand written texts. Some of these indexes are the SMOG (Mc Laughlin, 1969), the Fry Readability Graph (Fry, 1977), or the Flesch Reading Ease (Flesch, 1948). Although popular word-processing software such as Microsoft Word implements these formulas, their use is scarce and not widespread among health program developers (Fitzsimmons et al., 2010). More useful implementations of these formulas exist in the R environment (R Core Team, 2017), where the analyst not only is allowed to obtain a readability measure of a text but he also gets an estimation of the minimum age that a person should have to understand the text.

The content of the program is another component that threats its efficacy. Usually, and always based on theories with sufficient support, it is common to start incorporating critical components using open or qualitative methods (e.g., interviews or focus groups) that serve as input for confirmation studies with quantitative methods (Bartholomew et al., 2011). Thus, while the use of qualitative methods is common for adapting the contents of programs, quantitative approaches focus on assessing the efficacy of them. Another approach to evaluating program efficacy relies on the so-called network science (Barabási, 2003), which provides a robust analytical framework to evaluate the contents of programs by estimating rankings of words importance through word co-occurrence networks. The co-occurrence network depicts the set of words that appear together quite often in complete sentences. In these networks, it is also usual to detect communities of words or clusters that show subsets of words whose similarity is estimated through statistical indexes such as the Jaccard index (Vijaymeena and Kavitha, 2016). Furthermore, the estimation of these networks by employing a betweenness centrality criterion allows the researcher to quantify the statistical importance of each word, represented as a node within a network that shows the connection between words. The betweenness centrality was devised as a general measure of centrality of a network, and in the context of text-mining can be regarded as an index that quantifies the relative semantic importance of a given word that belongs to a set of words.

Both the readability measures and the words co-occurrence network can be deemed as applications related to text mining analyses that are convenient for the evaluation of intervention programs like Competencies for adolescents with healthy sexuality (COMPAS), a school-based sexual health promotion program. Although in the e-health realm, the application of text mining techniques has been already proposed (Chih-Ping et al., 2005), we are not aware of any previous study that has illustrated how readability measures and words co-occurrence network analysis can be applied to evaluate the efficacy of COMPAS.

Our aim in this work is to provide a text mining approach for adapting a school-based sexual health promotion program to be implemented in a new context rather than the one in which it was initially designed and evaluated. Thus, we are expecting to find hidden to human eye patterns within the COMPAS, key components, how sessions are linked between them, and if 14 to 19 years old adolescents can understand the program. Success in this purpose will guide future research in promotion programs, including a new quantitative step into adaptation and validation of intervention programs, which will save time and resources in implementation.

Method

Materials

COMPAS (Espada et al., 2018) is an evidence-based intervention to promote sexual health and prevent sexual risk behaviors. The primary goal of COMPAS is to reduce unintended pregnancies and STIs in adolescents aged 14 to 19 years old. COMPAS lasts for 5 h within five sessions, and its theoretical foundations are the models of health beliefs, specifically the Social Learning Theory (Bandura, 1977), the Theory of Planned Behavior (Ajzen, 1991), and the Information-Motivation-Behavioral skills model (IBM; Fisher et al., 2009).

The components of the intervention are the transmission of information, training in social skills, training in problem-solving, and maintenance strategies. The components mentioned above include training in self-control and concealed/imagination training using a participatory methodology. Activities such as role-play for skills training, debates, and games to modify false beliefs are carried out. The application of the intervention is in groups (25–30 participants) in high schools. COMPAS is a protocolized program that includes the facilitator's manual, the participants’ notebook, and the materials to guarantee the fidelity of the implementation. These protocols have been used in this paper as .txt or .csv (see Procedure).

The COMPAS program has proven to be valid to increase HIV and STIs knowledge, risk perception related to having unprotected sex to get an STI or unplanned pregnancy, and self-efficacy. This intervention also promotes a more favorable attitude towards protection methods (such as consistent condom use, even when there are obstacles for its use), towards HIV testing and people living with HIV/AIDS. In the long term, the COMPAS program maintains most of the short-term effects and increases subjective norms related to peers' condom use and delay the age of the vaginal penetration onset.

The efficacy of the COMPAS program has been tested in cluster-randomized controlled trials at short term (Espada et al., 2012, Espada et al., 2015, Morales et al., 2014), and its effects have been followed 12-month (Morales et al., 2016) and 24-month post-intervention (Espada et al., 2016). In Spain, COMPAS is the only school-based sexual health promotion that has proven to be as effective as an evidence-based intervention (Cuídate!), according to the Center for Disease Control and Prevention (CDC) (Espada et al., 2015, Espada et al., 2016, Morales et al., 2016). Compared to the control group, COMPAS has proven effective in promoting healthy sexuality, in terms of knowledge and attitudes about sexual health, risk perception, and condom use intention. In the long term, the short-term effects were maintained, and had an impact on the subjective norms and achieved to delay the age of onset of vaginal penetration. Mediators of the efficacy of COMPAS to promote condom use 24-month post-intervention have been explored (Escribano et al., 2015). Additionally, the efficacy of the program based on the degree of fidelity of implementation (high vs. low) has been studied (Escribano et al., 2016).

Procedure

We took the Colombian version (Morales et al., 2019) of the COMPAS program in its original written text format, and we converted into UTF-8 plain text (.txt). As this program consists of five sessions, we then split the complete text into five subdocuments. Each subdocument contained the written information for each session. We put all these subdocuments in one single folder, and then we developed an ad-hoc R script to scrutinize the text difficulty of all texts. To this end, we used the R package “koRpus” (Michalke, 2017) to estimate the text difficulty of these texts with the SMOG formula. The SMOG formula, conceived initially by Mc Laughlin (1969), can be regarded as an adequate quantitative estimator of the linguistic difficulty of Spanish-written texts (Correa et al., 2018).

Furthermore, we used KH Coder (Higuchi, 2015) to obtain the COMPAS co-occurrence Network. The co-occurrence network depicts the set of words that appear together quite often. We created an Excel file with five rows (one per each COMPAS session) as the input for the co-occurrence network. Our pre-processing procedure was as follows. Firstly, we removed Spanish stopwords (e.g., articles, prepositions, etc.) as well as frequent verbs such as: “can,” “to be,” “to have,” and non-informative words such as “case,” “question” and “answer.” Secondly, we removed regular expressions (i.e., “hello,” “welcome,” etc.) and ordinal adjectives (“firstly,” “second,” etc.) from the corpus analysis (i.e., a total of 8 new irrelevant words for the program). Personal names (as “Juan” or “Pedro”) were also excluded. We set the sentences as the unit of analysis and employed a minimum term frequency of 10 as a threshold to depict words co-occurrence, along with a Jaccard similarity index as the filter for connexions between words. We initially ran our description with a lower threshold (5 words), but we obtained a large net with much irrelevant information. We have also increased the threshold to 15 words, but the results were too vague. We colored our resulting networks following a betweenness centrality criterion, which stands for a metric that quantifies the statistical importance of a word, represented as a node within a network that shows the connexion between words. The R script is available under request to the corresponding author.

Results

The approach for adapting the COMPAS program entails the analysis of three interrelated properties. The first property of this program relates to its target population in terms of chronological age and schooling. In this part of the analysis, it should be verified if the readability of the written materials that accompany the COMPAS program corresponds to the target population it was designed for. The second property corresponds to the quantification of the relative importance of tackled topics within the COMPAS program. In this part of the analysis, the tackled topics of the program should match with the emerging clusters of words that result from the text-mining analysis. The third property of the COMPAS program relates to the correspondence of the topics with different numbers of sessions. The description of our results follows this order.

As shown in Fig. 1, the maximum age for understanding a session (session 5) is 19.46 years old, and the maximum schooling is 14.46. The rest of the sessions proved to be within the range of the program target (14 to 19 years old) also for schooling (9 to 13 years of education). These results can be regarded as evidence of the adequacy of the program for being applied to populations of these characteristics.

Fig. 1

Age needed to understand the sessions and schooling as “years of education” needed to understand the sessions.

COMPAS' primary goals are the reduction of both transmissions of STIs and adolescent pregnancies. A convenient way to validate whether or not these topics are reflected in the program is to depict the word co-occurrence network, as shown in Fig. 2, where the terms with higher betweenness centrality were STIs, pregnancies, transmission, followed by HIV, relation, sexual, risk, and contract.

Fig. 2

Words co-occurrence network of the COMPAS program. Color represents communities, 13 in total out of 80 nodes. Edges on network = 101. The density of the network = 0.032. Minimum word frequency to be shown = 10. Jaccard filter (top 100) was used. Units of analysis are sentences.

From the words co-occurrence network of the COMPAS program, the main component among the 13 identified refers to the prevention of STIs and unplanned pregnancies (Fig. 2). The most frequently used terms in the COMPAS manual are: “condom” (linked to safe sex), “sexual” (linked to maintaining a sexual relationship, sexual partner and sexual risk), “pregnancy” (linked to “risk”, “avoid”, “desire” and “protect”), “HIV” (linked to “contract”, “transmit”, “virus”, “infect” and “person”). An interesting relationship was found among the following terms: “desire,” “avoid,” and “prevent,” “transmission,” “infection,” and “pregnancy,” which represents the main objective of this program.

The adequacy of the COMPAS program can also be estimated through the analysis of the emerging clusters, also depicted in Fig. 2. Because the goal of the COMPAS program is the prevention of ITS and non-intended pregnancies, the largest community of words with blue nodes shows the set of words semantically and statistically associated with such a goal. The second community with orange nodes shows another essential component of the COMPAS program: healthy sexuality as an individual activity based on responsible decision-making. The rest of the word communities in the network can be regarded as particular subthemes of the COMPAS program. In particular, a limitation observed here regarding community number 09, is the focus on vaginal sex as the primary means of transmission for STIs, which excludes anal sex. This analysis highlights the need to review the program to reinforce the anal and oral transmission components.

Fig. 3 shows that the sequence of the sessions is in the correct order, it highlights particular and unique components of the program, and establish the connection between each session and with the program as a whole. This result points out the consistency of the COMPAS program with its goal and some peripheric components.

Fig. 3

Bipartite Words co-occurrence network of the COMPAS intervention program. The square nodes are the number of the COMPAS session, and circled nodes represent the most relevant words for each session. The importance of these words is depicted in colors ranging from light orange (low importance) to blue (high importance). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Discussion

Despite a large number of studies focusing on evaluating the efficacy of school-based interventions, the availability of methods for such purposes remains rarely exposed. We tackled this gap by describing a suitable method for testing the adequacy of the contents of a sexual health promotion program for its target population. The results of our method are easy to interpret. They express the minimum age of the participants to understand the contents, and the word co-occurrence network structure represents the topics and sub-topics covered in the program, as well as the years of formal education of participants to have such a comprehension.

The relevance of our contribution is evident when it comes to adapting an evidence-based intervention for a new context (e.g., other country or culture) –instead of the one in which it was initially designed and evaluated. Our work aimed at providing an example of how to empirically adapt and validate a school-based sexual health promotion program through a text mining approach that combines the readability analysis of its written contents and the graph analysis of its word co-occurrence network structure that reflects the relationship between the topics covered in the COMPAS program. Although programs of this sort have been studied elsewhere (e.g., Mirzazadeh et al., 2018, Morales et al., 2018a), the application of text mining techniques for empirical analyses of these materials remains scarce.

Our work provided a convenient approach to estimate if a written material fits the reading comprehension of a group of a specific range of age. Such a contribution should be seen in perspective from previous approaches such as the suitability assessment of materials (SAM) developed by Doak et al. (1996). As we mentioned earlier, one of the subscales of SAM tries to evaluate the reading grade level with a straightforward item that classifies peoples' scores in three categories: Superior = 5th grade or level or lower, Adequate = 6th to 8th grade, and Not Suitable = 9th grade or above. Results indicated that COMPAS materials are appropriate for adolescents aged 14 to 19, which coincides with the age of the participants who have received the program so far (e.g., Espada et al., 2012, Espada et al., 2015). The level of comprehension required by the participants is similar for all sessions, except for the last one, in which a little higher level of maturity is recommended. The last session includes the practice of how to correctly use male condoms, adolescents are trained to use self-instruction technique to guide their behavior, the perception of vulnerability to contracting an STI is addressed, and the implications of adolescent pregnancy are discussed to encourage participants making healthy choices about sex. The high cognitive load of some of these activities and the skills training may explain this result. The COMPAS program was designed to be implemented in schools, between ages 14 to 18, in line with the results obtained by the readability analysis of its written contents. According to recent systematic review and meta-analysis (Fonner et al., 2014, Mirzazadeh et al., 2018, Morales et al., 2018a), most sexual risk reduction interventions for STIs and pregnancy prevention interventions for adolescents were designed to be implemented in schools for at least three reasons: its availability, it is a learning setting, and the large number of adolescents who can benefit from the intervention at once.

Other relevant components of the program are related to healthy decisions making, so that adolescents do not take part in any sexual activity that may put them at risk of getting a STIs or an unplanned pregnancy (Fig. 2: community 2), the evaluation of the positive and negative consequences of their behaviors not only in the short time, but also in the long term (Fig. 2: community 7), and the promotion of the screening test for STIs antibodies in general, and HIV in particular (Fig. 2: community 3). All components are addressed transversally throughout the program (Espada et al., 2018), but particularly the first (decision making) and second (consequences) ones are presented in session 3 “Making decisions” and reinforced in session 5, “Maintaining your decision.” Adolescents are trained to make healthy choices based on the evaluation of the consequences of their sexual behaviors (e.g., having unprotected sex or not). The third component (HIV and STIs test) is addressed more specifically in session 1, in the activity 1D “More risk-less risk,” in which adolescents must recommend the STIs test to young people who have engaged in risky sexual practices. This component is reinforced in the last session, in the activity 5C “Vulnerability to Sexual Infections and Pregnancies,” and information about available resources to be tested for HIV and other STIs is provided to them so that in case of engaging in sexual risk practices they know what to do to avoid sexual risks.

Other small nodes capture important pieces of the program, such as transmission routes of STIs (community 9), heterosexual practices and pregnancy (community 5), other STIs (no HIV) such as gonorrhea, syphilis, and hepatitis (communities 8 and 11), etc. A limitation of the program was identified in the community 9. Although the program explains all the transmission routes of STIs, it gives higher weight to blood and vaginal intercourse. This could be related to the fact that COMPAS was designed from a primary prevention perspective, in which it is expected that most of the participants consider themselves as heterosexual, and anal intercourse is not as frequent as vaginal intercourse in this group, as reported in studies on sexual behaviors in adolescents (e.g., Morales et al., 2018b). It is essential to highlight that sexual diversity is addressed transversally during the program through the discussion of cases that involve young men who have sex with men. COMPAS participants learn about sexual diversity in the Sexual health Trivia, and the proposed role-playing activities allow them to perform any sexually-oriented couple (heterosexual and non-heterosexual), and in our network results, this is missing. Compared to programs targeted to adolescents in general – e.g., COMPAS (Espada et al., 2018) or ¡Cuídate! (Villarruel et al., 2009) –, it would be expected that others specifically targeted towards men who have sex with men (e.g., Eke et al., 2019, Martinez et al., 2018) emphasize more in unprotected anal intercourse as a mode of STIs transmission, especially for HIV. This exciting result suggests the need to revise the COMPAS program to provide a broader approach and an equal weight to the modes of STIs transmission.

The sequence and consistency of the COMPAS program are appropriate, as captured by the bipartite words co-occurrence network (Fig. 3). The first session, “Sexuality and Health,” includes an introduction and welcomes the participants. It is discussed with the participants what healthy sexuality is, and what is not. Besides, it is discussed how to identify the routes of STIs, and adolescents' evaluation of sexual risk assumed. Session 2 “Knowing better the sexual risks” is aimed at changing false beliefs and providing information on STIs transmission and methods of protection. In session 3, “Making decisions,” adolescents learn to make healthy decisions about sex by evaluating the positive and negative consequences of their sexual behavior. In session 4, “Improving your communication about sex,” the “COMPAS” negotiation technique is presented to help adolescents communicate and engage in healthy sexual behaviors. Finally, in session 5, “Maintaining your decisions,” adolescents learn to use the condom correctly and new strategies to maintain healthy sexuality, and they also develop a critical attitude towards the possibility of getting an STI or teenage pregnancy. Sessions presented in this order offer a smooth transition from one session to the next one sharing a common meaning. Thus, “vaginal,” “get,” and “infection,” are keywords for both sessions 1 and 2 and helped in the transition from the first session to the second. “Transmission” is halfway between sessions 2 and 3, “keep” and “relationship” are nexus form sessions 3 and 4, and finally, “situation,” “know,” and “moment” are transitional terms from session 4 to 5. We can also observe how terms as “pregnancy,” “condom,” “sexual,” “risk,” or “partner” (between others) are common to three or more sessions, becoming, as such, core words. This figure helped us to choose the correct order for sessions. All key terms of each session are clustered in communities that represent the expected structure of the program, as shown in Fig. 3. Transversal themes addressed centrally in the program include sexual risk, the relationship with the sexual partner, having sex, pregnancy, STIs (and especially HIV), and the use of condoms as a method of protection.

The number of school-based health programs is increasing at a considerable rate. Text mining methods, including the use of the SMOG formula and the words co-occurrence network, are becoming an essential tool for researchers to quantify the suitability and appropriateness of a program (written text) for a target population. These methods are also adequate for evaluating the validity of a school-based health promotion intervention, like the one we used here. The current study helped us to identify the main components of COMPAS program, analyze the sequence of its sessions, and estimate the age of participants. These results are hard to obtain from a regular visual review. Despite our efforts to give the same importance to vaginal, blood, and anal infection pathways, the results pinpoint that anal contagion is not as named as the vaginal or blood means of infection. As finding the best order of presentation for the sessions is related to its content, we were also able to estimate this particular feature of the program. The first session can be chosen as the most basic one, but from here, the order of sessions is not clear. This article helped us to figure out what is the best order for presenting the sessions. Thus results of this study were also relevant to identify aspects susceptible to improvement, which will be incorporated in future versions of this program to meet the initial objectives and participants’ needs. Future studies should focus on evaluating the efficacy of the Colombian version of the COMPAS program with adolescents in the short- and long-term.

Authors contributions

PVM: Conceptualized the manuscript, Conducted the literature review, Wrote the original paper,

Analyzed the data

JCC: Analyzed and cured the data, Conducted the literature review, Wrote original and reviewed

versions of the manuscript

MGL: Applied the intervention program, conducted literature review

DASR: Applied the intervention program, conducted literature review

EGM: Applied the intervention program, conducted literature review

DPP: Applied the intervention program, conducted literature review

JNC: Applied the intervention program, conducted literature review

PGR: Applied the intervention program, conducted literature review

DAL: Applied the intervention program, conducted literature review

JPE: Supervised and validated the writing of the manuscript

AM: Conducted the literature review and investigation, wrote original and reviewed versions of the

manuscript.