Comprehensive evaluation of Salud Escolar a health school program in Mexico: Rationale, design and methods.

The prevalence of obesity and overweight in Mexican children and adolescents is high (greater than 30%) and lifestyle behaviors are far from achieving health recommendations. Salud Escolar is a complex cross-sectoral multi-level policy-based program in Mexico aiming to support schoolchildren healthy behaviors. We describe the rationale, design and methods for the comprehensive evaluation of Salud Escolar during its first phase of implementation. Using a mixed-methods approach and the logic model of Salud Escolar as a guide, a comprehensive evaluation involving 3 types of evaluations was designed: 1) A design evaluation before program implementation, to determine the consistency between the design of Salud Escolar and the problem to be addressed (i.e., childhood obesity), 2) An implementation evaluation to assess potential execution bottlenecks, and 3) An outcomes evaluation, to measure short-term (i.e., knowledge, attitudes and practices related to healthy eating, drinking plain water and doing regular physical activity) and intermediate outcomes (i.e., fruit and vegetable intake, water consumption and daily moderate to vigorous physical activity). This evaluation will provide essential knowledge about program design and implementation processes, which are vital for drawing robust conclusions about the effectiveness of the program. Results and lessons learned from this comprehensive evaluation will provide evidence to improve Salud Escolar program and facilitate its upscaling process and may provide relevant information for school-based programs in other places sharing socio-contextual conditions.

Introduction

Obesity and overweight in Mexico are present in more than 35% of school-aged children and adolescents (Shamah et al., 2016). Unhealthy lifestyle behaviors commonly associated with higher rates of overweight and obesity, including frequent intake of high energy-dense foods and beverages and high levels of physical inactivity (<60 daily minutes of moderate to vigorous physical activity), are also common among Mexican children (Medina et al., 2018, Shamah et al., 2016). In addition, the childrens' proximal environment seems to be unsupportive of healthy lifestyles (Barquera et al., 2018, Argumedo et al., 2020, Aceves-Martins et al., 2016), underscoring the urgent need to address the obesogenic environment in Mexican schools.

The adoption of a coordinated school-based program to prevent and control obesity and overweight in children, can be part of the solution to this major public health problem (Specchia et al. 2018). A recent review of school-based interventions conducted in low- to middle-income countries identified that the most effective interventions were those with combined diet and physical activity, school teacher-delivery, parental involvement, education sessions, school food modifications and with a duration of more than 8 months (Singhal et al., 2020). In Mexico, even though successful efforts have been developed to promote physical activity and healthy diets among schoolchildren (Safdie et al., 2013a, Carriedo et al., 2013), their evaluation or scalability has been limited. (Théodore et al. 2018)

As a response to the high prevalence of childhood obesity in Mexico and taking advantage of previous experiences in the country (Salud En and Escuela, 2017, Safdie et al., 2013a, Carriedo et al., 2013), in 2019 a multidisciplinary and intersectoral group of academics, practitioners and policy-makers were tasked by the Ministry of Education to design a new school-based program aiming to promote healthy eating and drinking, and adequate physical activity among schoolchildren. Experiences from this expert panel provided the groundings to develop Salud Escolar: Escuelas Saludables y Activas (School Health: Active and Healthy Schools in English), referred herein as Salud Escolar (Secretaría de Salud and Secretaría de Educacion Pública, 2019). Salud Escolar is part of a larger national strategy aiming to ensure healthy environments at Mexican public schools. It consists of a government program involving national- and regional-level agencies from multiple sectors, as well as various stakeholders such as school directors, teachers, parents, and students. It considers a three-phase upscaling process. The first phase of the project, the pilot, began in the 2020–2021 school year, in a convenience sample of approximately 60 elementary schools in Mexico City, selected by the Ministry of Education. Currently, due to the COVID-19 pandemic and school closures, the pilot has been suspended until further notice. Once the program is resumed, the second and third phases will involve upscaling of the program to 10 of the 31 states in Mexico, and at the national level, respectively. The comprehensive evaluation of the pilot will provide insights to strengthen the roll out of the program at greater scale.

The evaluation of health promotion programs is essential in order to collect evidence about their effectiveness and identify ways to improve practice (O'Connor-Fleming et al., 2006). In this sense, since Salud Escolar is a new program, it demands a comprehensive evaluation process that allows identifying areas of improvement in its design, as well as any implementation bottlenecks that can affect its effectiveness and that may facilitate and improve the upscaling process (Swinburn et al. 2007). This paper describes the rationale, design and methods for the comprehensive evaluation (i.e., design, implementation and short and medium-term results) of Salud Escolar during the first implementation phase.

Methods

Mexican school context

In Mexico, education plans and programs are designed at the national level by the Ministry of Education and implemented with a certain level of autonomy at the local level, in the 32 Mexican states and Mexico City (DOF 2019). Compulsory or basic education consists of preschool, elementary (grades 1–6) and middle school (grades 7–9) (Supplementary Table 1) (Gobierno de México n.d.). Elementary schools serve children in part-time schools (4 h per day), extended-day schools (6 h per day) and full-time schools (8 h per day). In total, 76.9% of elementary schools in Mexico are part-time. The net coverage for elementary education is very close to 100% (98.5%). (INEE, n.d.) Although the government funds and operates public schools to offer free education, students’ families are required to provide some school supplies and other resources for school activities, including materials and resources (G.-Olvera et al. 2021). Additionally, school meals are not offered in most public schools in the country, except for those located in deprived areas (e.g., indigenous, rural and marginalized urban areas) where hot school breakfasts are offered to all children (DIF 2020). Schools at this level have the presence of the School Technical Council, which is made up of the principal and all teachers, and its main mission is to improve the educational service provided by the school (SEP 2017).

Table 1

Adapted version of the Terms of TRM for the Design Evaluation.

Objective	Exploration area	Reference term (research question)	Answer type
Analyzing the rationale for the creation and design of the program	Justification for program creation and design	1- Is the problem or priority need that the program seeks to solve identified in a document?	Yes/No
		2- Is there a diagnosis for the problem that the program addresses?
		3- Is there a theoretical or empirical justification that supports the type of intervention proposed by the program ?
Identifying its contribution to national goals and strategies	Contribution to national targets and objectives	4- -Is the purpose of the program aligned with the objectives of the sectoral, special or institutional program?
		5- -With what goals, objectives, and transversal strategies of the current National Development Plan, is the sectorial, special or institutional objective aligned with?	Open-ended
		6- -How is the Program Purpose aligned with the Millennium Development Goals, the Sustainable Development Goals or the Post 2015 Development Agenda?
Identifying the target population, objective and accountability mechanismsa	Potential population, objective and eligibility mechanisms	7- Are the target and program populations defined in official documents and/or in the diagnosis of the problem?	Yes/No
		8- -Does the program have systematized information that allows identifying the total demand for supports and the characteristics of applicants?
		9- Does the program have mechanisms in place to identify its target population?	Open-ended
		10- Does the program have a documented coverage strategy to serve its target population?	Yes/No
		11- Do the program selection procedures for beneficiaries and/or projects include standardized and systematized elegibility criteria? Are they publicly disseminated and congruent?
		12- -Are the procedures for receiving, registering and processing support applications adapted, having defined formats, are available for the purpose and attached to the program policy document?
Analyzing how program support mechanisms work	Beneficiary standards and care mechanisms	13- -Is there available information to identify who is supported by the program?
		14- -Are the procedures for granting goods or services to beneficiaries standardized, systematized, publicly disseminated and consistent?
		15- - What is the procedure for collecting socio-economic information of beneficiaries?	Open-ended
Analyzing the consistency between the design of the program and applicable regulations	Results Indicators Matrix (MIR)	16- - Is there one or a group of activities for each of the Components of the program Results Indicator Matrix (MIR, in Spanish) that are clearly specified, orderly, and necessary?	Yes/No
		17- -Do the components identified by the MIR correspond to the services offered by the program?
		18- - Is the purpose of the MIR a direct consequence of the Components of the program?
		19- - Is the goal of the MIR clearly specified, unique and linked to the strategic objectives of the sectoral program?
		20- - Is it possible to identify the narrative summary of the MIR (Goal, Purpose, Components and Activities) in the program policy document?
		21- - Are there clear, relevant, economic, monitored and appropriate indicators for each of the sectors involved at each of the program's objective levels?
		22- Do program indicators outlined in technical data sheets have a name, a definition, a calculation method, a measuremnt unit and a target?
		23- Are indicator targets expressed in terms of measurement units and aimed at boosting performance?
		24- How many of the indicators included in the MIR are official and public?
		25- Do indicators allow the objective to be measured, directly or indirectly, at that level?
		26- What are the suggested modifications or changes for the MIR?	Open-ended
Identifying the budget operations registries and accountability mechanism	Budget and accountability	27- Does the program identify and quantify the costs of generating goods and services?	Yes/No
		28- Does the program have transparency and accountability mechanisms?
		29- Are the procedures for the execution of program strategies and action lines standardized, systematized, publicly disclosed and included in the normative documents of the program?
Identifying potential synergies and/or coincidences with other programs	Complementarities and overlaps with other federal programs	30- With which federal programs and/or social development actions in other government orders and in what respects could the evaluated program have complementarity and/or overlaps?	Open-ended
Identifying whether the program has other essential elements in its design (such as a theory of change, the contribution of formative research to the design of the program , as well as interventions and action lines with a gender, intercultural and inclusive approach)	Other aspects related to the design of the program	31- Does the program have a theory of change and a conceptual framework that guides prioritized approaches or their components, strategies or action lines?
		32- Were the components, strategies and action lines of the program established in accordance with training research?
		33- To what extent are the components, strategies or action lines identified based on evidence? and, to what extent do they address the causes of the problem?
		34- Do the strategies and action lines of the program consider a gender approach?
		35- Are the components, strategies and action lines of the program intended for children and adolescents with disabilities?
		36- Do the components, strategies and action lines of the program include an intercultural approach?

The program: Salud Escolar

Led by the Mexican Ministries of Education and Health, Salud Escolar consists of four main components: healthy eating, healthy beverage consumption, adequate physical activity, and other cross-sectoral interventions. Across these components, a total of 13 strategies and 29 action lines at different levels of influence (individual, environmental, curricular, and at policy level) are considered. The implementation of such activities considers the interplay of schoolteachers and authorities, with local community actors (e.g., informal food vendors and food retailers around schools) and authorities, as well as national institutions from diverse sectors (e.g., health, sanitary risks, infrastructure, and sports). Although there is a budget for public education in Mexico (SHCP 2021), a specific budget line for the program was not specified because it is expected that the various institutions involved in Salud Escolar will be responsible for the implementation of specific activities and will provide the necessary financial and human resources. The implementation process will also require the involvement of a number of school actors, including parents, schooĺs principals, teachers and food vendors. Fig. 1 presents the logic model of Salud Escolar (Ayvar-Gama, Jáuregui, and Pacheco-Miranda 2020). As part of the upscaling process, Salud Escolar will be comprehensively evaluated during the first implementation phase.

Fig. 1

Comprehensive evaluation model

The evaluation of Salud Escolar will be conducted by the National Institute of Public Health of Mexico. Using the logic model of Salud Escolar as a guide(Cooksy, Gill, and Kelly 2001), a comprehensive evaluation involving 3 types of evaluations (i.e., design, implementation and outcomes) at different stages was planned (Fig. 1). Each of these evaluations aims to address different questions, and correspondingly requires a different evaluation approach. The design evaluation will assess the consistency between the design of Salud Escolar and the problem to be addressed (i.e., childhood obesity). The implementation evaluation will allow the clear identification of bottle necks that need to be monitored and addressed along the program’s lifetime. The outcomes evaluation focuses on verifying that the program achieves the expected results and impacts. In this sense, each stage of the program evaluation is an opportunity to identify whether the program is on the right path to achieve its goals and to improve practice. A brief description of methods for each of the evaluations planned is provided below. This study has been reviewed and approved by the Ethics, Research and Biosecurity Committees of the Mexican National Institute of Public Health.

Design evaluation

. A qualitative approach was used to determine the consistency between the components, strategies and action lines of Salud Escolar and the problem to be addressed (i.e., childhood obesity) (Peters et al., 2013a). The design evaluation started during the first implementation phase of Salud Escolar (June and August 2020) before schools were closed due to the COVID-19 pandemic. Necessary updates to this evaluation will be made once the program is resumed. The evaluation was conducted using an adapted version of the Terms of Reference Model (TRM) for the Design Evaluation (Consejo Nacional de Evaluación de la política de Desarrollo Social, 2017), developed by the Mexican National Council for Evaluation of Social Development Policy (CONEVAL in Spanish). This model is a common reference for design evaluations in Mexico and provides recommendations to decision makers to improve the design of a program, in this case Salud Escolar (Bonvecchio-Arenas et al., 2019, Rodríguez-Ramirez et al., 2019). The TRM consists of a set of 30 questions (e.g., yes/no or open-ended questions) in seven areas: (1) rationale for the creation and design of the program, (2) contribution to national goals and strategies, (3) target population, objective and accountability mechanisms, (4) beneficiary groups and support mechanisms, (5) indicators, (6) budget and accountability, and 7) synergies and coincidences with other federal programs. Adaptations to the TRM were made to include other aspects considered as relevant in the design of a program (questions 31–36 in Table 1), including the theory of change (Kim et al. 2011), the contribution of formative research to the design of the program (Bonvecchio et al. 2014), as well as interventions and action lines with a gender, intercultural and inclusive (i.e., of children with disabilities) approach (UNICEF REGIONAL OFFICE FOR SOUTH ASIA, 2018, Who, 2008, United Nations Educational Scientific and Cultural Organization, 2015). Areas and questions evaluated by the instrument are presented in Table 1.

Following this methodology, we first conducted a documentary analysis of the programmatic documents of Salud Escolar (e.g., implementation guidelines) and related policy documents (e.g., the education law in Mexico). Then, interviews with key informants were conducted to complement any information gaps identified in program and policy documents (Figure 2).

The programmatic documents evaluated were provided by the authorities of the Ministry of Health. Additionally, other documents, published and grey literature (e.g., records, databases, internal and/or external evaluations, scientific literature or other public documentation related to school-based interventions or programs) were reviewed.

. Semi-structured interviews were conducted with a purposive sample of 11 key actors involved in the design and implementation of Salud Escolar (Palys 2008). Interviews were recorded and transcribed verbatim. All transcripts were double checked by the qualitative coordinator of the research team against the recordings to ensure accurate reporting.

. Semi-structured interviews data were coded and analysed with an inductive Thematic Analysis (Braun and Clarke 2006). Transcripts were carefully reviewed to identify key themes and codes. Subsequently, the themes and codes initially developed were refined to proceed with the analysis of the entire data set (Liamputtong 2016). A codebook was developed to define the key themes and codes. Reliability was ensured using inter-rater reliability among coders of 80% or above (Morse 2015). A triangulation of sources (i.e., of the documentary analysis and semi-structured interviews) was sought to qualitatively respond each of the evaluation questions (Denzin, 1970, Mathison, 1988).

Based on the results of the documentary analysis and interviews, TRM yes/no questions (question 1–4, 7–8, 10–14, 16–25, 27–29), were answered according to their degree of compliance as “No”, if the program design did not consider the corresponding characteristic, or as “Yes“ if the program did. Further, if the program design considered a specific characteristic (i.e., responding Yes in the previous question), the question was further classified into four levels of compliance (1 lowest and 4 highest compliance), according to the criteria established by the TRM. In any case, explicit arguments were provided to justify the degree of compliance selected. For the open-ended TRM questions (i.e., questions 5–6,9,15,26,30–36), a narrative response was provided, with the information collected through de documentary analysis and the interviews. Based on these results recommendations for improving the design of Salud Escolar were made to the Ministries of Education and Health.

Implementation evaluation

This will be a cross-sectional study using a mixed methods approach (using checklists, focus groups and interviews) and grounded on the principles of implementation science (IS). IS focuses in identifying the larger system of factors or bottlenecks that affect implementation, knowledge production and utilization to improve the quality, impact and sustainability of the implementation (Tumilowicz et al. 2019). Relevant variables for the successful program implementation will be explored (Proctor et al., 2011, Peters et al., 2013b), including implementation fidelity, acceptability, suitability, feasibility, sustainability, and coverage. It will be carried out during the first 6 months of implementation.

The quantitative component (i.e., checklists) of this evaluation will be conducted at the school level, in all intervened schools (n = 60). Meanwhile, the qualitative component (i.e., focus groups and interviews) will be carried out among key informants from a subsample of 6 schools selected by convenience according with their neighborhood socioeconomic level (3 of low and 3 of medium socioeconomic level). A purposive sampling of school principals, teachers, food vendors, parents and children will be selected to participate in the qualitative component (Palys 2008). At least 30 semi-structured interviews will be conducted with school principals, teachers, and food vendors, and 20 focus groups will be carried out with parents and children. Interviews and focus groups will be recorded and transcribed verbatim.

To evaluate implementation fidelity, a check list developed by our research team based on Salud Escolar programmatic documents and the logic model will be used to investigate whether Salud Escolar activities are being implemented as planned. This list will be applied simultaneously by two interviewers, one from the Ministry of Health and another from the research team. The list will consist of items evaluating the presence/ absence or implementation/no implementation of each of the interventions and action lines of Salud Escolar. Each school will be measured twice within 15 days and an average of implementation fidelity score will be estimated.

Additionally, in order to get a deeper understanding of other implementation variables (i.e., acceptability, suitability, feasibility, sustainability, and coverage), interviews and focus groups with the key informants previously mentioned (i.e., school principal, teachers, others) will be conducted.

Data derived from the check list will be analyzed with the statistical package Stata v.15. Means ± standard deviation for continuous variables and n (%) for categorical variables will be estimated. Linear regression models will be constructed to investigate the factors associated with implementation fidelity. The fidelity score will be introduced as the dependant variable, and models will be adjusted for school characteristics (size, socioeconomic level, infrastructure, number of teachers).

For qualitative data, we will use the same analysis process explained above for the design evaluation to analyse interviews and focus groups data for the implementation evaluation. A triangulation of data sources (of interviews and focus groups) and of methods (between qualitative and qualitative techniques) will be conducted to have a broader understanding and different perspectives of the phenomenon of interest, in this case the implementation variables studied (Patton, 1999, Denzin, 1970). Results will be used to prepare a set of recommendations for the Ministries of Education and Health to improve the implementation process of Salud Escolar.

Outcomes evaluation

. A quasi-experimental intervention design will be used to evaluate the outcomes (from baseline to 10 months) of Salud Escolar on short-term (i.e., knowledge, attitudes and practices related to healthy eating, drinking water and doing regular physical activity) and intermediate term (i.e., fruit and vegetable intake, water consumption and daily moderate to vigorous physical activity). Details of the instruments are described in supplementary table 2. Masking will not be possible due to the nature of the intervention.

. A sub-sample of elementary schools participating in Salud Escolar will be randomly selected from the intervention schools and considered as the intervention group. The comparison group will consist of a similar number of schools selected from the wait-list control of schools (i.e., where the program will be upscaled in following implementation phases) in Mexico City without intervention. Only schools with at least 200 students and a morning shift will be eligible. Schools will be selected to match intervention schools based on the following characteristics: neighborhood socioeconomic level, number of students and distance between schools (at least 250 m between schools). All children and their parents enrolled in the selected school (grades 1–6, aged 6 to 11 years) and attending school on the day of recruitment will be invited to participate in the study. Schoolchildren with a physical or mental disability not able to participate in intervention components or measures will not be eligible. Short-term results will be measured in children with parent consent and who agree to participate in the study. For medium-term results randomized sub-samples of children will be drawn from the short-term outcomes sample.

A questionnaire based on the theory of planned behavior and prepared ex profeso by the Ministry of Health will be used among schoolchildren from grades 3 to 6 (i.e., ages 8 to 11) (Safdie et al., 2013b). This instrument measures self-reported knowledge, attitudes, perceived behavioral control (i.e., self-efficacy) and practices related to study outcomes. Self-reported knowledge, attitudes and perceived behavioral control are measured using statements rated on a 4-point Likert scale. Self-reported practices include fruit and vegetable intake (3 statements), consumption of plain water (2 statements), and foods with a high content of sugar, fat or sodium (e.g., sweets, candies, pastries, sugar sweetened beverages) (3 statements), rated on similar 4-point Likert scale, as well as physical activity practices measured through the physical activity question of the Health Behavior in School-aged Children questionnaire (Currie et al. 2009). Other physical activity practices are explored using other questions prepared ex profeso to capture specific activities considered by Salud Escolar (e.g., active breaks or physical education classes). Children from grade 3 will answer a version using an emoji 4-point Likert scale, whereas older children (i.e., grades 4–6) will use a regular Likert scale.

. Healthy eating will be assessed using a 24-hour dietary recall (24HR). Despite 24HR limitations (e.g. misreporting, time consuming),(Teasdale et al., 2018, Krehbiel et al., 2017) it provides reliable data and detailed intake regardless the respondent literacy when guided by trained staff.(Shim, Oh, and Kim 2014) Furthermore, the automated multiple pass method will be applied during the interview of the 24HR to collect more precise information of the reported food and beverage intake.(Conway et al. 2003) The proportion of children consuming any fruits and vegetables in the previous 24 h will be estimated.

An objective assessment of childreńs lunch will be conducted. Trained fieldworkers will record the foods and beverages amounts consumed by the students during school lunch (either brought from home or purchased at school). A SECA 852 scale will be used to weight foods and beverages, and a set of questions will be posed to children to determine the frequency and type of foods and beverages brought from home and purchased at school (Safdie et al., 2013b). The proportion of children consuming fruits and vegetables in their lunch will be estimated.

A beverage diary tool will be used to record the type, brand, quantity (ml), composition and frequency of beverage intake per day during two weekdays and one weekend day. This instrument has been previously used in Mexican children showing acceptable validity and reliability (Carriedo et al. 2013). The volume (ml/day) of water consumed per day will be estimated.

. Accelerometry (Actigraph GT3x and GT3x + ) will be used to measure daily minutes of moderate to vigorous physical activity (MVPA) in a subsample of children. The number of children selected will be based on the largest possible number given logistic and financial restrictions of direct observation methodologies. Despite its limitations (e.g. limited generalizability) (Pedišić and Bauman 2015), accelerometry reduces the common problems of self-reported physical activity among children (e.g. limited validity and reliability) (Shephard 2003). Data will be recorded into 15 s epoch and 30 Hz sampling rate. Participants will be asked to wear the accelerometer on their waist on the right mid-axillary line during 7 consecutive days (plus a familiarization day) and will be advised to remove the device only during water-based activities and sleeping time. Parents and children will receive verbal and written indications of accelerometer care and usage. Compliance enhancing strategies will include daily school visits and small incentives (stickers) for those wearing the device, as well as phone calls and text messages to parents. Students completing ≥ 4 days with ≥ 10 h/day will be included in analysis. Average minutes per day of MVPA will be estimated using Evenson 's cut points (Evenson et al. 2008).

Additionally, the System for Observing Fitness Time (SOFIT) will be used to monitor physical activity during school hours (McKenzie, Sallis, and Nader, 2016). SOFIT consists of a systematic and direct observation of instruction time capturing participants' activity, class context, and teacher behavior and has been used in school settings facilitating a rich description of physical education classes (Smith, McKenzie, and Hammons 2019). For this study SOFIT will be used to record MVPA performed during physical education classes and school recess. Data will be handled using standard protocols (McKenzie, Sallis, and Nader, 2016) and time engaged in MVPA during this school-time periods will be estimated.

Individual level covariates will be measured using a questionnaire applied to the mother, and will include children’s age and sex, as well as mother’s education level, self-reported BMI (through self-reported height and weight), employment status, marital status, and interest in health. The questionnaire will also include the questions of the National Health and Nutrition Survey of Mexico to estimate household socio economic status and family composition (Instituto Nacional de Salud Pública 2016).

Children’s height and weight will be measured twice using a portable stadiometer SECA 213 and a SECA 876 scale (Hamburg, Germany), respectively, by trained staff. Children whose first and second measures have a difference greater than 2% will require a third measurement. The average of the two measures (or the two closest measures in the case of children with three measurements) will be used for analysis.

The ISCOLE school audit tool (ISAT) will be used to objectively measure the school physical activity and food environment (Broyles et al. 2015). ISAT is a standardized audit of the physical environment supporting schoolchildren active transportation, provision of sports and play facilities, aesthetics, perceived suitability of the school grounds for sport, informal games, general play and other facilities (e.g. drinking fountains). ISAT also includes a questionnaire for school directors with items exploring school facilities and policies for PA and healthy eating at school. ISAT has shown good reliability in measuring the school environment in countries that share physical and social conditions similar to the ones in Mexico (e.g., Colombia) (Broyles et al. 2015).

The study will be conducted in 30 elementary schools (15 intervened schools and 15 wait-list controls) with 35 children per school. This number was defined in order to better estimate the effect (with a reduced number of schools the intervention effect might be confused with the school characteristics) (Donner and Klar 2001), the short time available to collect data (not more than 6 months according to the Ministry of Health), as well as available funding. Given an intra-cluster correlation within schools of 0.025 (Donner and Klar 2001), 35 children per school (1050 in total and 525 per group), a bilateral z-test for proportions with a 0.05 significance level, and a prevalence of fruit and vegetable consumption of 15% in the comparison group, the study achieves a statistical power of 80% or higher to detect a difference of at least 10 percentage points between groups (i.e. 15% vs 25%). Under the same general assumptions and a prevalence of unhealthy food consumption around 60% in the comparison group, the study achieves a statistical power of 84% or higher to detect a difference of at least 13 percentage points between groups (i.e. 60% vs 47%).

Analysis plan

We will analyze the changes in the proportion of children consuming fruits and vegetables, water consumption and daily minutes of moderate to vigorous physical activity. We will employ an intent-to-treat analysis and thus include all participants. Generalized estimating equations (GEE) models will be used to examine across group differences in outcome variables and their changes from baseline to endline. The average effect of the program will be estimated by difference in difference, considering the interaction of the treatment variable with time. Standard errors will be adjusted for data dependencies within schools specifying an exchangeable correlation matrix. The distribution family of the outcome will be selected according to whether it is continuous or binary.

Discussion

The purpose of this article is to report the rationale, design and methods for the comprehensive evaluation of Salud Escolar. Previous experiences and lessons learned in Mexico underscoring the importance of program evaluation, as well as to base the design/implementation of the policy on a theoretical framework (Théodore et al. 2018), were drawn on to inform this study. As long as we are aware, this is the first time that a national school program is evaluated since its design, with the three types of evaluation, guided by a logic model and using a mixed methods approach. The evaluation is planned to consider the different levels of influence of health behaviors as well as a number of actors playing an important role in the implementation process, including, students, parents, schools principals, teachers and food vendors.

Evaluation is an essential component of all health promotion programs. However, if not adequately planned, the quality, impact and results of the evaluation may be limited (Kozica et al., 2014). This may be especially true for nation-wide complex health promotion programs, like Salud Escolar (Craig et al. 2019). This comprehensive evaluation recognizes that knowing that an intervention is efficacious may not be sufficient, it is also necessary to generate evidence about the design and implementation that will effectively deliver nutritional interventions in real conditions (Menon et al., 2014, Habicht and Pelto, 2014, Pelletier et al., 2013). This evaluation will provide essential knowledge about program design and implementation/processes, which are vital for drawing robust conclusions about the effectiveness of the program (Kozica et al., 2014).

However, comprehensive evaluations can be challenging (Issel 2014). To date, only few studies have implemented comprehensive evaluations of health promotion programs, with most of them considering implementation and impact evaluations only (Kozica et al., 2014). Additionally, evidence indicates that although public health interventions generally rely on the theory of change to design and evaluate public health interventions (“Comprehensive Evaluation of the Community Health Program in Rwanda. Final Report.”, 2016, Breuer et al., 2016), external evaluations of the formulation or design of a program are seldom used. The design of a public policy or program includes the development of appropriate strategies to solve a problem (Maier 2000). Nonetheless, when a design evaluation of a program is not conducted and areas of opportunity in its formulation are not identified (ideally in its first year of operation), the implementation and subsequent results of the program could be affected (Hudson, Hunter, and Peckham 2019). Likewise, the use of logic models in evaluations, although not always used, is essential to depict the relation between the program’s activities and their intended effects, to determine what aspects of the program need to be evaluated, to define evaluation questions and the necessary information to answer them, as well as to identify the sources, methods and the more adequate instruments for data collection (CDC Division for Heart Disease and Stroke Prevention, n.d., Cooksy et al., 2001).

Our comprehensive evaluation is based on Salud Escolar logic model, and the three different, yet-complementary, evaluations during the first phase of the program will provide essential information to improve the program’s design, operation process and outcomes. It is well recognized that the impact of evidence-based interventions is limited by the lack of evidence on the best operational strategies for scaling up nutrition interventions (Menon et al. 2014). This evaluation will provide insights to ease the upscaling process of Salud Escolar in the country.

This study has some strengths and is not exempt of limitations. The strengths of this study include the use of mixed methods which facilitates the comprehensive assessment of the program (U.S. Department of Health and Human Services Centers for Disease Control and Prevention. Office of the Director, 2011), a large sample size enabling to conduct robust statistical analyses, the use of device-based physical activity measures (e.g. accelerometers) and more accurate forms of diet assessment than self-reports (lunch assessment), and the use of a quasi-experimental design to evaluate the impact of Salud Escolar. In addition, the voice of several actors is considered, from students to stakeholders. The main limitations of this study include the inherent limitations of some instruments used, as misreporting (e.g., 24-hour dietary recalls or diary beverages) or the lack of information provided by the informants in the qualitative components.

Results and lessons learned from this evaluation will build in the state of the art of school-based interventions, provide evidence for further programs in Mexico and other countries with similar socio-contextual conditions, and might facilitate and improve the national scalability of Salud Escolar in the next implementation phases. It will also generate knowledge to contribute to the field of implementation science in nutrition and physical activity.

CRediT authorship contribution statement

Alejandra Jáuregui: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Supervision, Project administration, Funding acquisition. Selene Pacheco-Miranda: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Supervision. Gabriela Argumedo-García: Conceptualization, Writing – original draft, Writing – review & editing. Joaquín A Marrón-Ponce: Conceptualization, Writing – original draft, Writing – review & editing. Armando G.-Olvera: Conceptualization, Writing – original draft, Writing – review & editing. Jorge Vargas-Meza: Conceptualization, Writing – original draft, Writing – review & editing. Ylenia Yatziri Ayvar-Gama: Conceptualization, Writing – original draft, Writing – review & editing. Daniel Velázquez: Writing – review & editing. Amado David Quezada: Conceptualization, Methodology, Writing – original draft, Writing – review & editing. Anabelle Bonvecchio-Arenas: Conceptualization, Methodology, Writing – original draft, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.