Realist evaluation of intersectoral oral health promotion interventions for schoolchildren living in rural Andean communities: a research protocol.

BACKGROUND: Intersectoral collaboration, known to promote more sustainable change within communities, will be examined in an oral health promotion program (OHPP). In Peru, an OHPP was implemented by the Ministry of Health, to reduce the incidence of caries in schoolchildren. In rural Andean communities, however, these initiatives achieved limited success. The objectives of this project are: (1) to understand the context and the underlying mechanisms associated with Peruvian OHPP's current effects among school children living in rural Andean communities and (2) to validate a theory explaining how and under which circumstances OHP intersectoral interventions on schoolchildren living in rural Andean communities produce their effects. METHODS AND ANALYSIS: Through a realist evaluation, the context, underlying mechanisms and programme outcomes will be identified. This process will involve five different steps. In the first and second steps, a logic model and an initial theory are developed. In the third step, data collection will permit measurement of the OHHP's outcomes with quantitative data, and exploration of the elements of context and the mechanisms with qualitative data. In the fourth and fifth steps, iterative data analysis and a validation process will allow the identification of Context-Mechanism-Outcome configuration, and validate or refine the initial theory. ETHICS AND DISSEMINATION: This research project has received approval from the Comité d'éthique de la recherche en santé chez l'humain du Centre hospitalier universitaire de Sherbrooke. The initial theory and research results will be published in relevant journals in public health and oral health. They will also be presented at realist evaluation and health promotion international conferences. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

A realist evaluation is especially useful to explore and understand the influences of context and underlying mechanisms on intervention effects in complex situations such as deployment of intersectoral collaboration.

This study will underpin the development of an emerging theory regarding the implementation of intersectoral collaboration in health promotion programmes, including oral health promotion.

Language and cultural barriers will be present throughout the research project and may influence data collection and the process of analysis.

Introduction

Oral health (OH) is an essential health component1–3 as it is linked to well-being and quality of life.2 4 Of all oral diseases, dental caries are the most important OH problem worldwide.1 Dental caries represent the third global disease plague,5 and prevalence is increasing among schoolchildren in developing countries,6 7 particularly in Latin America.6–8 This problem affects primarily children of the poorest and marginalised populations.7 Furthermore, dental caries could be considered a complex problem because many factors (individual, familial, social and environmental) contribute to its development.7 9 10 The children experience multiple consequences,7 9 11–13 as do their parents1 11–15 and the local health system.1 11–14 Considering the complexity of the problem of dental caries, and major impacts at different levels, this is a major public health problem.16 Since the prevalence of dental caries is increasing in developing countries, a resolution was adopted by WHO to target OH problems in developing countries.17 To tackle a complex problem such as dental caries, it is necessary to develop intersectoral actions.13

In Peru, as in other developing countries, dental caries are a concern among schoolchildren, particularly in rural Andean communities.18 Despite mixed evidence,19–21 several authors indicate that an oral health promotion program (OHPP) with schoolchildren can counter this problem.4 19 20 To ensure effective, optimal and sustainable change, OHPP should be implemented in collaboration with different local stakeholders (including parents)13 19 20 22–24 and grounded on theoretical basis.19 20 To adapt Peruvian OHPP to the realities of rural Andean communities, it is important to understand how OHPP works in depth19 for those communities including the different elements influencing the development of collaboration between local stakeholders involved25 in OHPP.

OHPP interventions

To reduce dental caries impacts, it is necessary to invest in oral health promotion (OHP) interventions.1 3 Individual OH education programmes may improve OH,2 but without further actions engaging different stakeholders, effects of programmes are barely significant or not sustainable in the long term.12 26 27 An OHPP programme with a comprehensive approach including stakeholders' participation could have a long-term effect.12 22 Health promotion programmes (including OHPP) can be defined as collective actions to produce change, that is, deemed to be desirable in a population.28 For OHPP targeting schoolchildren, school is an appropriate environment to conduct OHP interventions,22 29 since children are usually more receptive and it also facilitates access to community members—the families and teachers.22

The Peruvian OHPP was developed by the Ministry of Health in 200730 and was improved up to 2013 when health promotion interventions (including OHP) were implemented in primary schools.31 On an annual basis, OH professionals perform four types of OHP activities in schools: (1) training on oral hygiene and eating habits related to OH; (2) tooth brushing and flossing demonstrations; (3) distribution of toothbrushes and fluoride toothpaste and (4) application of fluoride or sealants on children's teeth. OH professionals working in public health centres must achieve goals related to those OH activities. Usually, teachers are also involved in OHP. They perform daily promotion of tooth brushing at school and ensure that children have a toothbrush and fluoride toothpaste available at school.31–34

Insufficient cooperation with local stakeholders during the implementation process has been shown to explain poor performance of several health promotion programmes.23 35 Therefore, intersectoral collaboration is necessary12 22–26 35–37 to optimise the effectiveness and efficiency of programmes, and to ensure their sustainability.23 24 35 38

Intersectoral collaboration is defined as a collective action where several stakeholders with a common goal take on different roles.23 This process facilitates networking and the sharing of information, resources, activities and capabilities among different involved stakeholders.39 Unfortunately, intersectoral collaboration is not always present in the implementation process of health promotion programmes.23 35

Currently in Peru, OHPP is a responsibility of dentists working in health centres. In the face of a shortage of dentists and low involvement in OH by other health professionals,40 current deployment of OHPPs is rather uneven in the different regions of the country.41 In some places, non-governmental organisations are working with health authorities to ensure programme deployment.42 Indeed, it is not possible to know the extent of collaboration with different stakeholders with the Peruvian OHPP.

In some parts of Peru, OHP interventions show promising results such as increased tooth brushing by children.42–44 Favourable outcomes, however, emerge slowly in some Andean regions.45 Why are the current dental caries prevention activities conducted among schoolchildren not resulting in improved behaviours and OH? How is collaboration with stakeholders and organisations being deployed, or perhaps thwarted? Can collaboration between stakeholders and organisations optimise dental caries prevention? Very few models and theories have been developed in conjunction with community OHP and intersectoral collaboration.46 To adapt the Peruvian OHPP adequately to local circumstances, and to optimise collaboration with different stakeholders in rural communities, it is crucial to understand elements of context and underlying mechanisms47 that may explain the continuing high prevalence and incidence of dental caries among schoolchildren in rural Andean communities.

Methods and analysis

Objectives

The objectives of this research are: (1) to understand the context and the underlying mechanisms associated with Peruvian OHPP's current effects among schoolchildren living in rural Andean communities and (2) to validate a theory explaining how and under which circumstances OHP intersectoral interventions on schoolchildren living in rural Andean communities produce their effects.

Study design

In order to understand better the factors influencing OHPP implementation, it is essential to consider local stakeholders' and various professionals' views on the programme, and to foster their active participation in the evaluation process.46 A realist evaluation will include different types of participants, in order to achieve the research objectives. This research methodology has been developed from a postpositivist perspective to explore and understand influence of context and underlying mechanisms on intervention effects.47–52 This design is often used to assess complex situations such as deployment of intersectoral collaboration53 as it facilitates the connection of different elements that may influence interventions' effectiveness in specific environments.54

In realist evaluation, the context, underlying mechanisms and programme outcomes are delineated and inter-relations are explored. The context includes pre-existing elements of the environment where interventions occur.48 50 55 56 Underlying mechanisms are elements of stakeholders' reasoning and reactions, depending on the context and resources deployed when interventions are performed.57 Frequently multiple mechanisms can explain interventions' outcomes,50 55 56 but they may be obscured and difficult to identify.52 According to realist evaluation, interventions' outcomes are the result of multiple causes and depend on mobilised mechanisms and elements of the context where interventions are performed.48 50 58 Finally, Context-Mechanism-Outcome (CMO) chains are a conceptual tool that connects elements of context, mechanisms and intervention outcomes.48 50

This research design encourages complementary use of qualitative and quantitative methods in order to understand a range of phenomena that can explain programme outcomes. Quantitative methods will document intervention outcomes descriptively and allow stakeholders to explore, by inference, the mechanisms involved.59 60 Elements of context and mechanisms will be explored with stakeholders using qualitative methods such as semistructured interviews and focus groups.51 59–61

Realist evaluation is usually divided into four distinct steps; however, in this study we have added an additional step: the development of a programme logic model. This addition helps to explain further the programme implementation process and to understand better the mobilised resources and activities carried out.58 Thus, the five steps are as follows: (1) development of a logic model; (2) development of an initial theory; (3) data collection; (4) data analysis, to highlight regularities and relationships between elements of context, mechanisms and outcomes, and to develop CMO chains and (5) initial theory validation and refinement. This process, used iteratively, will generate a final theory explaining the programme CMO configurations.49–51 56 61

First step: development of a logic model

We conducted a review of various policies on OHP with schoolchildren, adopted by the Peruvian government between 2007 and 2013. Following this review, we developed a logic model for Peruvian OHPP directed at schoolchildren (table 1).

Table 1

Logic model for Peruvian OHPP directed at schoolchildren

Program target	Schoolchildren (including preschoolers)
Program context	Program developed by the Ministry of Health in 2007. Development of a collaboration agreement with the Ministry of Education in 2008 and with the Peruvian College of Dentists in 2009. Improvement of the programme in 2012 by the Ministry of Health and in 2013 by the President of the Republic, which requires the implementation of health promotion interventions (including oral health) in all primary schools (including preschool) in Peru
Levels of intervention	Schoolchildren (individual level), Parents (interpersonal level), Schools (organisational level), Rural communities (community level)
Objective	Decreased oral health problems for schoolchildren to improve chewing ability, self-esteem and quality of life
	▾
Componentscarried out in schools	Promotion of daily tooth brushing with fluoride toothpaste in schools Education session and demonstration on oral health in schools Application of sealant or fluorine (varnish or gel) Involvement of parents and community authorities
	▾
Resources	Health centre dentists Preschool and primary school teachers Oral health promotion materials distributed by the Ministry of Health (poster, presentation, toothbrush, fluoridated toothpaste, dental products)
	▾
Output	Application of fluorine (gel or varnish) or sealant Education session with children two or three times per year Daily tooth brushing with fluoridated toothpaste in schools
	▾
Short-term outcomes (knowledge and attitudes)	Increased knowledge on oral health among schoolchildren Increased positive attitudes toward oral health among schoolchildren
	▾
Medium-term outcomes (behaviours)	Increased daily tooth brushing with fluoridated toothpaste among schoolchildren Increased oral hygiene among schoolchildren
	▾
Long-term outcomes (Conditions)	Decreased dental plaque among schoolchildren Decreased dental caries among schoolchildren Increased quality of life related with oral health among schoolchildren

OHPP, oral health promotion program.

Second step: development of initial theory

To explore the actual effects of the Peruvian OHPP, we elaborated an initial theory. This process was carried out following a literature review on the deployment of community health promotion programmes; intersectoral collaboration during the implementation process of health promotion programmes and contextual elements and underlying mechanisms explaining such programme outcomes. Considering the importance of social, physical and environmental factors on OH,10 62 this initial theory is based on an ecological approach,63 and combines OHPP components, stakeholders involved and OHPP's expected effects on schoolchildren. Following the literature review, four probable elements of context and 12 probable mechanisms that may explain OHPP's effects were included in the initial theory. Two types of mechanisms could explain OHHP's effects. First, situational mechanisms would help to develop intersectoral collaboration between stakeholders involved and thus foster their joint and individual involvement. Second, transformational mechanisms would explain every stakeholder contribution to OHPP's effects.64 A director of the Regional Health Authority as well as a director of a non-governmental organisation working in targeted rural communities validated this process. This initial theory will be published separately.

Third step: data collection

Setting

To validate the initial theory, three rural communities in Cusco region were selected. These communities were selected because all stakeholders included in the initial theory are represented; and OHPP interventions have been conducted there previously. Before data collection, there will be a meeting with community stakeholders to confirm their interest in participating in the research project.

Participants

Two types of participants will be enrolled in this study: (1) children attending primary school, to measure OHPP's effects; and (2) various stakeholders (health workers, teachers, parents and other community stakeholders) present in rural communities, to explore and validate different underlying mechanisms that may explain OHPP's effects.

A meeting with the members of each community being studied will be organised to introduce the research project. At the end of this meeting, parents who want their children to participate in the research project will be invited to meet one of the research team members, who will present the consent form for their children. Other stakeholders targeted for this study will be contacted after the meeting to gather their consent to participate. Consent forms for stakeholders and children's parents are available in Quechua and Spanish.

Children will be recruited with a census in the selected communities of all children whose parents have given their consent. Children from 9 to 13 years old will be chosen because the majority of permanent teeth (except third molars) are present at this age, which facilitates international comparisons and epidemiological surveillance.65 The tools used in this study were validated with children of similar age to those who will participate in this research.66 67 According to school lists in the three communities, the potential child population 9–13 years of age is ∼80 children.

Before data collection with children, a member of the research team will introduce them to the research project and will obtain the child's assent to participate. Child assent will be recorded along with the child's parental signed consent form.

Stakeholders will be recruited according to a theoretical sampling68 so that all categories of stakeholders identified in the initial theory are represented: (1) health workers; (2) teachers; (3) parents and (4) other community stakeholders (members of municipal assemblies, members of school parent associations, local health promoters).51 69 They will be identified and approached by research team members. To validate the elements of context and the mechanisms, a minimum of two participants per community will be recruited for each category of stakeholders.

All participants must meet the following inclusion criteria: (1) speak Spanish or Quechua and (2) live or work within the selected communities. The exclusion criterion for this research project is the presence of any disease, or cognitive or sensory deficit limiting social interactions.

Data collection process

The data collection process will validate and enhance elements of context, mechanisms and outcomes described in the initial theory, summarised in table 2. A research team will assist with data collection. Research team members were chosen for their knowledge of the environment in which the study is being carried out and their fluency in Quechua and Spanish. Before data collection, objectives, logistics, techniques and instruments to collect quantitative and qualitative data will be reviewed by the research team70 71 to ensure inter-rater reliability and internal validity of the quantitative scales. Examiners will be calibrated, and will proceed to data collection when a Kappa score of at least 0.8 has been reached.65

Table 2

Data collection process to measure OHPP outcomes, elements of context and mechanisms

Measures		Instruments
Information session with rural communities members and recruitment
OHPP's outcomes on children (quantitative measures)	1 OH knowledge	Questionnaire on knowledge, attitudes and behaviours related to OH
	2 Attitudes toward OH
	3 OH behaviours
	4 Oral hygiene	Dental examination with the Community Plaque Index DMFT
	5 Dental caries
	6 OH-related quality of life	Child version of the Oral Impacts on Daily Performance
Elements of contextMechanisms (qualitative measures)		Focus groups with each type of stakeholders Individual interviews with stakeholders

DMFT, decayed, missing and filled teeth; OH, oral health; OHPP, oral health promotion program

Measures of OHPP's outcomes

Measurement of the OHPP's outcomes will be carried out with child participants, using tools and questionnaires presented in table 1. A pretest of all these instruments will be made with a group of children having similar characteristics, but not participating in the research project.

To measure OH knowledge, attitudes and behaviours, the Questionnaire on knowledge, attitudes and behaviours related to OH developed and validated by Poutanen Lahti66 was selected following a review of different questionnaires. Using Likert scales to measure children's knowledges, attitudes and behaviours, this self-administered questionnaire was used in several studies including studies selected to develop the present initial theory.72 An initial version of this questionnaire was validated with 53 children. Cronbach's alphas were 0.70 for knowledge, 0.75 for attitudes and 0.85 for behaviours related to OH. The clinical presence of plaque and gingivitis has been found to correlate with low self-reported OH behaviours in children who have completed the questionnaire.73

For this research, the Questionnaire on knowledge, attitudes and behaviours related to OH was translated from English to Spanish following a reverse parallel translation process.74 In order to use it with participants whose native language is Quechua, the questionnaire was translated from Spanish into Quechua by an anthropologist speaking Spanish and Quechua. Later, Spanish and Quechua versions of the questionnaire were adapted and validated. The adapted version use in this study includes 26 items on behaviour, 14 items on attitudes and four items on knowledge related to OH. The translation, adaptation and validation process in Spanish of the Questionnaire on knowledge, attitudes and behaviours related to OH will be published subsequently.

To measure the presence of dental cavities and the level of OH, dental examinations will be conducted by OH professionals, according to the WHO recommendations. To ensure standardisation of dental examinations, a pretest will be done to measure interjudge reliability. The presence of dental caries will be reported using the decayed, missing and filled teeth (DMFT) index.65

The presence of dental plaque, a good indicator of a child's oral hygiene,75 76 will be measured using the Community Plaque Index. This index was developed by Corchuelo75 to simplify oral hygiene evaluation, and was validated with 83 children. Following the validation process, the index was found to be strongly correlated with the O'Leary index (sensitivity of 96.3 and specificity of 75.0), which is recognised as the gold standard to measure oral hygiene and dental plaque.75

The Peruvian Spanish version of Child Oral Impacts on Daily Performance (Child-OIDP) will be used to measure children's OH-related quality of life. This questionnaire was developed by Gherunpong et al77 and has been translated into Spanish following a reverse parallel translation process and adapted for Peru by Bernabé, et al67 Child-OIDP assesses the OH impact in connection with eight daily activities of children. For each activity, the questionnaire assesses the presence of an impact, its frequency and its severity. The process of validation of the Peruvian Spanish version of the Child-OIDP took place with 865 children aged 11–12 years old. Cronbach's α for internal consistency is 0.62, and test–retest reliability with an intraclass correlation coefficient is 0.85.67 In order to use this questionnaire with children whose native language is Quechua, the Peruvian Spanish version of Child-OIDP was translated from Spanish into Quechua by an anthropologist speaking Spanish and Quechua and validated qualitatively with two community workers and four children speaking Quechua.

Measures of elements of context and mechanisms

To identify and confirm different elements of context and mechanisms that may explain OHPP's effects, focus groups will be conducted in Spanish or Quechua with an open interview grid71 developed using the initial theory and in accordance with the principles of the realist interview.78 During the data collection process, the interview grid will be augmented to incorporate emerging themes.68 To ensure group homogeneity,70 focus groups will be conducted with each of the types of actors from both communities targeted, for a total of four focus group discussions. Groups will have a maximum of eight participants, to facilitate interactions among participants.70

To further explore and validate various themes that emerge during focus groups, semistructured interviews will be conducted with a minimum of two persons per stakeholder types (a minimum of eight interviews). The interview grid developed with the initial theory and in accordance with the principles of realist interview78 will be enhanced if additional themes emerge.68 Interviews will be conducted by a member of the research team speaking Quechua and Spanish. Focus groups and interviews will be recorded71 and transcribed verbatim68 in Spanish by the research team. Coanalysis will be performed from the Spanish version.79

Fourth and fifth steps: data analysis and theory refinement

Quantitative analysis of OHPP effects

Continuous variables will be analysed for each community using descriptive statistics, including frequency tables, central tendency and dispersion measures. The effects of the OHPP will be summarised for stakeholders at the beginning of each focus group. This process will thus allow stakeholders to explore by inference, mechanisms that may explain these effects.59 60

Qualitative analysis

Qualitative data collection and analysis will be done iteratively and non-linearly.80 Content analysis will permit the identification of emerging and recurring themes from the data. A semistructured analysis grid developed with the initial theory and realist evaluation core concepts will allow causal data coding from a confirmatory perspective.78 This process will facilitate identification of elements of context and mechanisms influencing the OHPP's effects.68 NVivo 11 will be used to code qualitative data. This software allows coanalysis and integration of quantitative data, to facilitate pattern identification.81

Coanalysis of the results will entail independent analyses of transcripts, and differences of opinion will be discussed and validated by the team. Data interpretation will be supported by verbatim transcripts.68 The principal investigator will record events and thoughts in a diary throughout the data collection process.68 82 This will reduce subjectivity and improve credibility of the qualitative results.68

The variety of sources, methods, data types and researchers will allow data triangulation.68 70 Detailed descriptions of elements of context and the participation of different types of stakeholders involved in programme deployment will highlight different perspectives and rival explanations.69 These procedures will ensure credibility and transferability.68

Integration of qualitative and quantitative data

Qualitative and quantitative data will be integrated in two stages. Interpretation of the data from each community80 will highlight CMO configurations, then, an intercommunity analysis will compare those different CMO configurations. This second analysis will foster the development of transferable explanations,68 and will be used to refine and modify the initial theory.50 51 56 61

The final theory will be validated with local stakeholders using two focus groups.51 One group will consist of health professionals and primary school teachers. The second group will include parents and other community stakeholders. This approach will ensure that community members can express themselves freely during the validation process.

Ethics and dissemination

For this study, formal consent will be obtained. Research data will be rendered anonymous, analysed in pooled groups rather than at the individual level, and communities will not be identified. Thus, identities of communities and project participants will remain confidential.

Dissemination

This research project is the first in OH to use such a research design. It is also one of the first studies in OH focusing on development of intersectoral collaboration in the process of implementation of OHP interventions. The main strengths of this research project are the use of multiple cases, mixed methods, validated quantitative tools61 and realist interviewing principles.78

An important limitation is the presence of language and cultural barriers that will be present throughout the research project.83 The presence of a local research team and adaptation of tools by an anthropologist working with those populations should decrease the effects of those barriers. Social desirability and elite bias could also be present, potentially affecting the credibility of the results.68 84 Triangulation of sources and methods, multiple cases, exploration of rival explanations and final theory validation with two different groups of stakeholders should reduce those biases.68

The benefits from this research project could be numerous. From a research perspective, this study will allow the development of an emerging theory about the implementation of intersectoral health promotion programmes, including OHP. This study will allow, moreover, the translation and validation of questionnaires in Spanish and Quechua, which can be used for subsequent studies. For clinical practice, this research will provide recommendations to improve OHP interventions and will propose strategies focusing on intersectoral collaboration. These results could also be transferable to other contexts, particularly for disadvantaged or rural populations. For local people, this study will foster the deployment of OHP interventions in rural communities. This project may also provide a better understanding of specific elements of context in rural communities of developing countries, and may help to identify different mechanisms promoting adoption of healthy behaviour by people living in those communities.