Implementation and Outcomes of a National School-Based Mental Health Program for Middle School Students in Chile.

Background: Chile's national school-based mental health program, Skills for Life (SFL), has demonstrated effectiveness in improving behavioral and academic outcomes in first- through third-grade students. The current study assessed the feasibility and outcomes of SFL's program for sixth- through eighth-grade students.
Methods: We assessed the percentage of students who participated in the program and longitudinal changes on teacher-reported Teacher Observation of Classroom Adaptation Re-Revised (TOCA-RR) scores, youth-reported Pediatric Symptom Checklist-Chile (PSC-Y-CL) scores, grade-point average, and school attendance from sixth to eighth grade (2016-2018) for SFL's workshop intervention. Linear mixed effects models analyzed the association between outcome variables and workshop attendance.
Results: Of the 30,649 sixth graders who attended the 754 participating schools in 2016, 28,204 (92.0%) were screened with the TOCA-RR. Of the 1829 students who screened at risk, 1344 had available workshop data for seventh grade, with 86.9% of them participating in most (≥ 7) workshop sessions. Workshop attendance was significantly associated with improvements in school attendance and peer relationships (a TOCA-RR subscale) in eighth grade. Conclusions: With high rates of behavioral health screening and workshop attendance, this study demonstrated the feasibility of implementing SFL's middle school program on a national scale. Higher workshop attendance by at-risk students was associated with better school attendance and peer relationships in eighth grade, as well as better but not significantly different outcomes on other measures (e.g., teacher-rated school performance and aggressive behavior in the classroom). Overall, these findings provide preliminary evidence of the feasibility and benefits of SFL's middle school program.

Introduction

The Importance of School-Based Mental Health (MH) Programs

Worldwide, the majority of children and adolescents with mental health (MH) concerns never receive MH services (World Health Organization, n.d.). To improve access to needed care, in recent decades, there has been a growing emphasis on school-based MH programs. Schools are well positioned to provide MH services given children’s frequent presence at school, the decreased practical barriers associated with school- versus community-based services (e.g., transportation, childcare arrangements), and the potentially lower stigmatization of school- compared with community-based services (Committee on School Health, 2004; Kern et al., 2017). School-based services also provide advantages including the opportunity for students to immediately practice the skills they learn in their everyday environment with the support of their teachers and peers (Volkaert et al., 2022).

However, schools have limited time and financial resources, which can be barriers to broad implementation of school-based MH programs. To advocate for these programs, in light of these challenges, it is important to understand their feasibility and outcomes. Given that schools must focus primarily on education, it is beneficial to evaluate MH programs’ school-related outcomes (e.g., school attendance, classroom functioning, and academic performance) in addition to their changes in MH symptoms and functioning.

Large School-Based MH Programs

A literature review focused on the feasibility of large-scale school-based mental health programs that was conducted by Murphy and colleagues (2017) identified eight large school-based mental health programs worldwide that had collectively reached over 27 million children. Five of these programs were developed in the USA, two were developed in Australia, and one was developed in Chile. A number of studies have demonstrated the positive effects of these programs on students’ emotional, behavioral, and academic outcomes (Murphy et al., 2017). For example, research on the large school-based MH program that is the focus of the current study, Chile’s Habilidades para la Vida (translated to Skills for Life [SFL]) has demonstrated that participation in second-grade workshops for at-risk students predicted improved teacher-rated classroom adaptation, parent-rated MH, and school attendance from first to third grade (Guzmán et al., 2015). Research has also suggested that school-based MH programs are effective for youth and families from diverse backgrounds (Kern et al., 2017), including youth living in both high- and low- and middle-income countries (Fazel et al., 2014a, 2014b; Murphy et al., 2017). In accordance with findings like these, the World Health Organization (WHO) has described schools as “crucial” for identifying children in need of MH care, and it has promoted the delivery of diagnostic and treatment services in schools when possible (World Health Organization, 2003, p. 15).

The Need to Support Middle School Students

Middle school is a challenging time for many youth due to increased experiences of bullying and onset of various MH conditions in late childhood to early adolescence (Hankin et al., 1998; Lijster et al., 2017; Milsom & Gallo, 2006). Rates of death by suicide are significantly higher in adolescents than in school-aged children (Sheftall et al., 2016). Conditions such as adolescent depression predict poorer school outcomes including lower grades, increased absenteeism, more disciplinary problems, and academic underachievement (Bearman et al., 2020). While these consequences highlight the need for accessible MH services for middle school students, research on school-based MH programs has historically focused more on interventions for elementary school students (Murphy et al., 2017; Sanchez et al., 2018). This is understandable given the importance of intervening for MH concerns as early as possible (Davis et al., 2000), but additional programming is likely needed for students who remain at risk or develop MH concerns as they enter adolescence. Research has suggested that younger adolescents feel more positively about seeking school-based MH supports than older adolescents, indicating a potential critical period for intervention (De Luca et al., 2019). While some research has suggested the feasibility and effectiveness of large-scale school-based MH interventions for early adolescents (e.g., the Cognitive Behavioral Intervention for Trauma in Schools (Jaycox et al., 2012), FRIENDS (Barrett et al., 2006), and School-Wide Positive Behavior Interventions and Supports (Childs et al., 2016), these programs have primarily been implemented in English-speaking countries (Murphy et al., 2017) and with English-speaking populations (Barrett et al., 2003).

It is important to determine whether findings from prior studies on school-based MH programs extend to youth from other cultural contexts and backgrounds given the centrality of cultural factors in youth development (Deighton, 2018). As emphasized by Polanczyk (2015), around the world, resources are being dedicated to programs without evidence of effectiveness. Such practices may interfere with efforts to narrow the gap between youths’ MH needs and their opportunities to benefit from treatment, which is especially concerning in places with extremely high unmet MH needs such as low- to middle-income countries (Salamanca-Buentello et al., 2020). Those creating future school-based MH programs should be able to learn from the development and components of effective interventions administered in a range of cultural contexts, not only those conducted with a specific population (i.e., English-speaking youth in English-speaking countries), since practices may not appropriately generalize for youth throughout much of the world.

SFL for Middle School Students

SFL is a school-based MH program in Chile that currently reaches more than two-thirds of a million (676,165) students per year in the three age levels of the program (Fig. 1). It is provided by the Junta Nacional de Auxilio Escolar y Becas (JUNAEB; translated as the National Association of School Assistance and Scholarship), which is a branch of Chile’s national department of education that focuses on supporting students with biopsychosocial vulnerabilities including MH concerns (Junta Nacional de Auxilio Escolar y Becas, n.d.). All public and subsidized private schools in Chile can apply to participate in SFL, with priority given to schools that meet criteria for high risk based on a formula that accounts for school-level indicators such as family income and maternal education (Guzmán et al., 2015).

Fig. 1

Annual student enrollment for SFL I, SFL II, and SFL III from 2010 to 2021

In acknowledgment of the need for ongoing MH supports as students progress through their development and education, SFL includes MH screening and interventions for elementary, middle, and high school students who are attending participating schools across Chile.SFL’s elementary school program was piloted between 1992 and 1997, followed by the middle school program in 2008 to 2012, and then the high school program in 2016 to 2018 (Ministerio de Educación, 2021). All levels use the three-tiered approach to intervention that is recommended by the WHO (Fig. 2) and aligns with the multi-tiered systems of support framework (August et al., 2018; Hendren et al., 1994). Tier 1 involves universal MH promotion for all students, Tier 2 includes preventive workshop interventions for at-risk students (20–30%), and Tier 3 involves referrals to individual MH services for students with more concerning symptoms (3–12%) (Guzmán et al., 2015; Ministerio de Educación, 2021). At the time this manuscript was written, there had been over 20 academic publications on SFL (e.g., Guzmán et al., 2015), with all but two of them (Gallardo et al., 2015; Leiva et al., 2015) focused on the elementary school program due to its longer history than the middle or high school program.

Fig. 2

WHO three-tier intervention model Figure originally found in Hendren et al., 1994. Mental health programmes in schools (No. WHO/MNH/PSF/93.3 Rev. 1. Unpublished). World Health Organization

In 2011, evaluations of the SFL middle school (grades six through eight) pilot program sought to determine whether a moderately revised version of the SFL elementary school program would benefit early adolescents. Two studies examined the pilot implementation of the middle school workshops (the Tier 2 intervention). One study focused on 65 students from eight schools in a single commune (equivalent to school district) of the Santiago Metropolitan Region (Gallardo et al., 2015), and the other followed 212 students from 68 schools in 11 communes across the country (Leiva et al., 2015). Findings from both studies suggested that higher rates of student workshop attendance predicted improved teacher-reported classroom adaptation but, interestingly, not improved student-reported psychosocial functioning (i.e., social, emotional, and behavioral adjustment; Gallardo et al., 2015; Leiva et al., 2015). Leiva et al. (2015) also examined the effects of parent workshop attendance and found that higher parent attendance similarly predicted improved classroom adaptation among students. The authors postulated that the workshops positively affected classroom adaptation, but not psychosocial functioning, because workshops primarily focused on developing school-based competencies, and referrals were made based on teacher observation. In contrast, psychosocial functioning was more likely to be addressed through the Tier 3 intervention, which involved referrals to individual MH services (Leiva et al., 2015). Despite these mixed findings, school officials and families felt the program was important, and the Chilean government proceeded with the national expansion of SFL for middle school students. The program has grown steadily and been renewed yearly ever since. In 2021, more than 200,000 students participated in the program (Fig. 1).

The Current Study

The current study offers a unique contribution to literature on SFL and school-based mental health programs by examining outcomes for the first cohort of youth who participated in the national implementation of SFL’s middle school workshop program in 2016–2018. Analyses examined whether the national middle school program was feasible by looking at the rates of screening and program attendance. Analyses also explored the effects of student workshop attendance on outcome variables related to classroom adaptation, school attendance, academic achievement, and psychosocial functioning. The study aimed to elucidate whether a national school-based MH program for at-risk Chilean middle school students was feasible and whether the program had any significant quantitative benefits based on the data available.

Hypotheses

Encouraging results from the smaller pilot studies on SFL’s middle school workshop program suggested that it could be feasible on a national scale. These findings also indicated that workshop attendance was most likely to predict improved teacher-reported classroom adaptation from sixth to eighth grade, but not improved student-reported psychosocial functioning during that time (Gallardo et al., 2015; Leiva et al., 2015). Although the pilot studies had not examined school-related outcomes, prior work on the elementary school workshop program had shown that school attendance improved significantly more for students who more consistently attended workshops (Guzmán et al., 2015), so the authors hypothesized that this would also occur in the middle school cohort. Analyses examining the workshops’ effects on academic achievement were exploratory since grade-point average (GPA) had not been included as an outcome variable in prior studies.

Methods

Measures

The Teacher Observation of Classroom Adaptation Re-Revised Scale

The Teacher Observation of Classroom Adaptation-Revised (TOCA-R) is a teacher-report measure of psychosocial functioning in the classroom (Werthamer-Larsson et al., 1991). Originally created for use in the USA, this measure was culturally adapted for use in Chile and found to be valid and reliable in its re-revised form (TOCA-RR; George et al., 1994). In 2011, JUNAEB adapted the TOCA-RR to use in adolescent populations and identified four age-appropriate subscales: MRP (“poor peer relationships,” indicating low levels of integration and interaction with classmates), RA (“aggressive responses,” indicating disobedience, low frustration tolerance, and/or aggressive verbal or physical reactions), PDE (“poor school performance,” indicating low motivation to complete schoolwork and high levels of distraction in the classroom), and BA (“low autonomy,” indicating high dependence on teachers and peers for schoolwork and activities, respectively) (Leiva et al., 2015). These four subscales were validated for reliability in Chile and yielded Cronbach’s alpha ranging from 0.81 to 0.95.

The total TOCA-RR is comprised of 31 items, each of which is scored on a Likert scale of “1” (Almost Never) to “6” (Almost Always) and summed for a total score. In the current study, Cronbach’s alpha for the TOCA-RR was 0.92 in both 2016 and 2018, suggesting a very high level of internal consistency. Higher scores indicate higher degrees of problems with classroom adaptation; therefore, positive beta values in regression models indicate increased difficulty with classroom behavior in eighth grade and negative beta values indicate decreased difficulty. Based on years of practical experience, SFL officials operationalize overall risk on the TOCA-RR in sixth and eighth grade as risk on at least three of the four subscales, each of which has its own validated cut-off score that differs by gender and indicates substantial impairment across multiple domains of classroom adaptation. The program aims to identify 20% of students at-risk given constraints in space and resources for the intervention.

Pediatric Symptom Checklist

The Pediatric Symptom Checklist (PSC) is a broadband measure of psychosocial functioning in children and adolescents. The PSC has been widely studied and validated in many different linguistic and cultural groups (Massachusetts General Hospital, 2021). With a Cronbach’s alpha of 0.91 (Murphy & Jellinek, 1988; Murphy et al., 1996) and a high rate of test–retest reliability (r = 0.84–0.94), the measure has been translated into more than three dozen languages and culturally adapted for use in many countries and communities (Ishizaki et al., 2005; Lowenthal et al., 2011; Thun-Hohenstein & Herzog, 2008).

Nearly three decades ago, the Chilean version of the PSC (PSC-CL) was created for use with Chilean children and adolescents (De la Barra et al., 2005; George et al., 1994; George et al.,; 2004). The PSC-CL is made up of 33 items assessing a broad range of psychosocial concerns and is scored on a Likert scale. There are parent-/caregiver-report (PSC-CL) and youth self-report (PSC-Y-CL) versions of the measure. On both versions, reporters rate each item as “1” (Never), “2” (Sometimes), or “3” (Often Present). The PSC-Y-CL, used in SFL II, has been validated in adolescent populations in Chile and yielded a strong Cronbach’s alpha of 0.97 (Leiva, George, Guzmán et al., 2015). In the current study’s data, the PSC-Y-CL yielded a Cronbach’s alpha of 0.82 in both 2016 and 2018, providing support for strong internal consistency.

As with the TOCA-RR, higher PSC-CL scores reflect more psychosocial concerns. Thus, positive beta values in regression models indicate increased psychosocial difficulty in eighth grade and negative beta values indicate decreased difficulty. Several prior studies evaluating the SFL program or using SFL data have used PSC and TOCA-RR scores (Guzmán et al., 2011; Murphy et al., 2015; Guzmán et al., 2015; Gallardo et al., 2015; Leiva et al., 2015; Dutta et al., 2022) to measure outcomes.

SFL II Program Details

All students are screened with the TOCA-RR and PSC-Y-CL in sixth grade to assess who should be referred for Tier 2 and 3 services, respectively (Fig. 2). Students who screen at risk on the TOCA-RR in sixth grade are referred to the workshops in seventh grade (the Tier 2 intervention), and those who screen as high-risk on the PSC-Y-CL in sixth grade are referred to community MH services the same year (the Tier 3 intervention).

MH professionals (usually licensed psychologists or social workers) within the SFL II program run the Tier 2 workshop interventions. The intervention consists of 10, two-hour weekly sessions over the course of six months for students, as well as three sessions for the student’s parents and two for their teacher. The impact of the Tier 3 program could not be evaluated meaningfully due to missing data for most youth meeting criteria and the data that were available showed no effects.

The MH professionals who run the Tier 2 workshops take attendance on a session-by-session basis and forward these data to the national SFL office at the end of seventh grade. The principles and practices for the workshops are outlined in manuals, which provide general guidelines for how to run the workshops. All new MH professionals receive training on how to use the manuals, and they are regularly reviewed by returning MH staff. Within this general framework, local SFL teams implement the program flexibly to meet the diverse needs of their specific group of students (e.g., from large metropolitan areas like Santiago, to rural areas like Putre, to remote areas with small schools like Puerto Toro). Workshop sessions emphasize strengthening interpersonal, social, cognitive, and affective skills that are relevant to positive classroom and psychosocial adaptation (Cefai & Cavioni, 2014; Gallardo et al., 2015; Leiva et al., 2015). Workshops also focus on factors thought to promote resilience, in particular conflict resolution and group integration.

Procedures

In the current study, students from 754 SFL-participating schools were assessed in 2016 (sixth grade) and again in 2018 (eighth grade) using the TOCA-RR and PSC-Y-CL measures. The TOCA-RR was administered to teachers during interviews conducted by SFL staff, while the PSC-Y-CL was completed by students on paper forms. Data analysts from the national SFL office extracted attendance and GPA for each student from official school records.

Per usual procedures, SFL school-based MH staff inputted teacher and student questionnaire data, along with the data on workshop participation into an online database, and immediately received a report of individual and group score results. This report provided the regional teams with the information needed to plan for the workshops. In addition, these data were also sent to the program’s national office. Program officials de-identified the data and sent it to the USA for analysis. The authors analyzed data on: (1) the TOCA-RR and PSC-Y-CL measures; (2) other relevant academic and sociodemographic variables which were drawn from school records; and (3) rates of workshop attendance for at-risk students. School record variables included baseline (sixth grade) age, gender, participation in Chile Solidario (the country’s welfare program), school attended by the student, and sixth- and eighth-grade school attendance percentage, and grade-point average (GPA). No informed consent was required on the part of students or parents as (1) data collection is a standard aspect of the academic year in SFL-participating schools and (2) all data were de-identified before analysis. Students do, however, consent to taking part in the workshop intervention (Gallardo et al., 2015).

This study was approved by the Mass General Brigham Institutional Review Board as secondary-use, non-human subjects research. Appropriate data use agreements were executed by both teams.

Analytic Method

Software

Data from sixth, seventh, and eighth graders (2016–2018) were compiled and cleaned using SAS v. 9.4. All data analyses were conducted using SPSS v. 24 or SPSS v. 26.

Preliminary Analyses

The authors used independent samples t-tests and Chi-square analyses to assess baseline academic and sociodemographic differences between students who were workshop-eligible (at risk on the TOCA-RR in sixth grade) and workshop-ineligible (all other students).

Adjusting for Non-Random Participation in Workshops and Loss to Follow-Up

To adjust for non-random participation in the workshop intervention, we assessed the relationships between five baseline variables and participation in the intervention in individual linear or logistic regression models. We controlled for variables that were significantly associated with workshop attendance in adjusted linear mixed regression models when estimating intervention effects. We also assessed for predictors of loss to follow-up in eighth grade among the sample of students who attended workshops as a means of adjusting for missing data.

Primary Analyses

Paired samples t-tests assessed differences from sixth to eighth grade in TOCA-RR scores, PSC-Y-CL scores, GPA, and attendance in the full, longitudinal sample. Preliminary analyses also assessed outcomes of parent and teacher workshop sessions. Adjusted linear mixed effects models assessed the impact of workshop attendance as a continuous count variable (0–10 sessions) and other academic and demographic predictors on TOCA-RR and PSC-Y-CL scores, GPA, and school attendance in 2018. Workshop attendance and the respective sixth grade screening score, GPA, or attendance were entered as fixed factors. As hundreds of schools were participating in SFL and each school can vary widely in terms of staffing, geography, and resources, we treated schools as nested random factors. In doing so, we were able to assess the effect of workshop attendance while adjusting for statistical dependencies between students who attended the same school.

Results

Sample Size Selection

In 2016, 30,649 sixth-grade students were enrolled in SFL-participating schools. Of this initial sample, 28,204 (92.0%) had complete TOCA-RR data for that year and 23,287 (76.0%) had complete baseline PSC-Y-CL data. Students who were missing data on one or more of the following baseline demographic variables (n = 151) were then excluded from the sample: GPA, school attendance rate, age, and participation in Chile Solidario. The complete sixth-grade sample therefore consisted of 23,136 students. After using student ID number to match data from students in the sixth-grade sample with data from the same students in 2018 (eighth grade), the full, longitudinal sample consisted of 13,895 students (60.1% of students with complete and valid baseline data). As we aimed to assess longitudinal changes in classroom adaptation, psychosocial functioning, and academic performance both overall and as a function of workshop attendance, only students with complete data on these variables were included in analyses. As noted in a previous paper by our group (blinded for review), the high rate of family mobility from year to year in these schools in lower-resource areas is thought to account for most of the observed attrition over 2 years.

Full Sample Characteristics

As shown in Table 1, students in the longitudinal sample had an average baseline (sixth grade) GPA of 5.65 (SD = 0.53) on a scale ranging from 1 = lowest performance to 7 = highest performance, a school attendance rate of 92.95% (SD = 6.27%), and age of 12.20 years (SD = 0.74 years). Just under half of students were male (48.1%), and a little more than one-quarter were enrolled in Chile Solidario (26.5%). Average scores on the PSC-Y-CL and TOCA-RR at baseline were 55.47 (SD = 7.99) and 78.99 (SD = 24.76), respectively.

Table 1

Comparisons between baseline characteristics of full, workshop-eligible, and workshop-ineligible samples

	Workshop eligible (N = 1829)	Workshop ineligible (N = 12,066)	Comparison statistics	Full sample (N = 13,895)
Male (N, %)	986 (53.9%)	5697 (47.2%)	χ2(1) = 28.51, p < .001	6683 (48.1%)
Chile Solidario (N, %)	584 (31.9%)	3098 (25.7%)	χ2(1) = 31.90, p < .001	3682 (26.5%)
Age (M, SD)	12.41 (0.85)	12.17 (0.71)	t = − 11.65, p < .001	12.20 (0.74)
GPA (M, SD)	5.26 (0.44)	5.71 (0.52)	t = 39.21, p < .001	5.65 (0.53)
PSC-Y-CL Score (M, SD)	58.57 (8.04)	55.00 (7.88)	t = − 18.00, p < .001	55.47 (7.99)
TOCA-RR Score (M, SD)	115.45 (15.77)	73.46 (20.88)	t = − 101.22, p < .001	78.99 (24.76)
% School Attendance (M, SD)	91.63 (7.28)	93.15 (6.08)	t = 8.49, p < .001	92.95 (6.27)

As shown in Table 2, average TOCA-RR scores decreased significantly over the 2 years, indicating an improvement in overall classroom behavior (M2016 = 78.99, SD2016 = 24.76; M2018 = 77.76, SD2018 = 23.52; p < 0.001). GPA improved significantly from sixth to eighth grade (M2016 = 5.65. SD2016 = 0.53; M2018 = 5.68, SD2018 = 0.56; p < 0.001), suggesting small improvements in academic achievement while school attendance worsened slightly but not significantly over the same years (M2016 = 92.96; SD2016 = 6.27; M2018 = 92.88, SD2018 = 7.01; p = 0.148). Average PSC-Y-CL scores worsened slightly and significantly over the 2 years (with an increase in scores indicating an increase in psychosocial concerns): M2016 = 55.47, SD2016 = 7.99 versus M2018 = 56.12, SD2018 = 7.95; p < 0.001.

Table 2

Changes in primary outcomes from 2016 to 2018 in full, longitudinal sample (N = 13,895)

	GPA (M, SD)	PSC-Y-CL Score (M, SD)	TOCA-RR Score (M, SD)	% School attendance (M, SD)
2016	5.65 (0.53)	55.47 (7.99)	78.99 (24.76)	92.95 (6.27)
2018	5.68 (0.56)1	56.12 (7.95)2	77.76 (23.52)3	92.88 (7.01)4

1t = 7.75, p < .001

2t = 9.39, p < .001

3t = − 6.17, p < .001

4t = − 1.45, p = .148

Demographic and Psychosocial Differences by Workshop Eligibility

Of the 13,895 students in the longitudinal sample, 1829 (13.2%) screened at risk on the TOCA-RR in 2016 and were eligible for SFL workshops in 2017. The workshop-ineligible sample consisted of the other 12,066 (86.8%) students who were not at risk on the TOCA-RR. The workshop-eligible sample differed significantly from their workshop-ineligible peers in terms of baseline characteristics. As shown in Table 1, compared with their peers, the workshop-eligible sample had significantly lower GPAs and school attendance rates. They were also slightly, but significantly, older. They had significantly higher baseline PSC-Y-CL scores (more psychosocial concerns) and, as expected, higher TOCA-RR scores (poorer classroom adaptation). The workshop-eligible sample additionally had a significantly higher proportion of males and students enrolled in Chile Solidario than the ineligible sample.

Adjusting for Non-Random Participation in Workshops and Loss to Follow-Up

Of the 1829 students at risk on the TOCA-RR, 485 (26.5%) had missing data for each workshop session and were excluded from the analyses. The effect of baseline covariates on workshop participation was subsequently assessed in the sample of students with complete workshop and sixth-grade data (n = 1344). Significant predictors of attending more workshop sessions included higher baseline GPA (B = 0.26, t = 2.00, p = 0.046), greater baseline school attendance (B = 0.05, t = 6.20, p < 0.001), and younger age (B = − 0.19, t = − 2.67, p = 0.008). Among the sample of students who attended at least one workshop session and had complete baseline data (N = 1323), all students had eighth-grade follow-up data. As such, there was no need to assess any significant predictors of loss to follow-up or include additional covariates to adjust for missing data.

Linear Mixed Effects Models

Association Between Workshop Attendance and TOCA-RR Scores

Parent and teacher workshop attendance showed minimal, non-significant effects on overall post-test TOCA-RR scores (parent: B = 0.05, p = 0.934; teacher: B = − 0.73, p = 0.678). On the four subscales of the TOCA-RR, significance ranged from p = 0.474 to p = 0.960 in the direction of worsened functioning associated with higher parent workshop attendance and from p = 0.294 to p = 0.538 in the direction of better student functioning in relation to higher teacher workshop attendance.

Students attended an average of 8.33 workshop sessions (SD = 2.13). As shown in Table 3, student workshop attendance was associated with non-significantly improved TOCA-RR scores in 2018 after controlling for baseline GPA, school attendance, and age in adjusted models (in addition to clustering for random school-level effects). Beyond the effects of the other predictors, students improved on the TOCA-RR by an average of 0.28 points (p = 0.306) with each additional workshop session they attended. Among the four subscales, greater workshop attendance was significantly associated with improved peer relationships (lower scores on the TOCA-RR’s MRP subscale) in eighth grade (B = − 0.14, p = 0.029) (Table 4). Although the associations between workshop attendance and two of the remaining three TOCA-RR subscales (low school performance and aggressive behavior) were non-significant, results were in the hypothesized direction (B− 0.05, p = 0.435; B = − 0.06, p = 0.475, respectively).

Table 3

Linear mixed effects model predicting 2018 TOCA-RR score, PSC-CL score, school attendance, and GPA by student workshop attendance and baseline characteristics (clustered for random school effects)

Fixed-effects variables	2018 TOCA-RR Score		2018 PSC-CL Score		2018 School Attendance		2018 GPA
	B (SE)	t	B (SE)	t	B (SE)	t	B (SE)	t
Continuous student sessions	− 0.28 (0.27)	− 1.02	0.23 (0.09)	2.52*	0.36 (0.10)	3.66***	0.01 (0.01)	0.95
Baseline student GPA	− 12.76 (1.39)	− 9.20***	− 0.78 (0.47)	− 1.66	0.93 (0.48)	1.91	0.67 (0.03)	24.04***
Baseline percent school attendance	0.14 (0.08)	1.65	0.04 (0.03)	1.36	0.47 (0.03)	15.96***	0.001 (0.002)	0.31
Baseline age	1.06 (0.72)	1.47	− 0.25 (0.25)	− 1.00	− 1.08 (0.26)	− 4.18	− 0.01 (0.015)	− 0.49
Baseline TOCA-RR score	0.35 (0.04)	9.38***	–	–	–	–	–	–
Baseline PSC-CL score	–	–	0.38 (0.02)	15.32***	–	–	–	–

*p < .05, **p < .01, ***p < .001

Table 4

Linear mixed effects model predicting 2018 TOCA-RR subscale scores1 by student workshop attendance and baseline characteristics (clustered for random school effects)

Fixed-effects variables	2018 MRP score		2018 PDE score		2018 RA score		2018 BA score
	B (SE)	t	B (SE)	t	B (SE)	t	B (SE)	t
Continuous student sessions	− 0.14 (0.06)	− 2.19*	− 0.05 (0.07)	− 0.78	− 0.06 (0.08)	− 0.72	0.08 (0.07)	1.15
Baseline student GPA	− 1.95 (0.32)	− 6.12***	− 4.33 (0.38)	− 11.52***	− 1.44 (0.42)	− 3.39***	− 0.84 (0.34)	− 2.45*
Baseline percent school attendance	0.04 (0.02)	1.84	0.03 (0.02)	1.35	0.01 (0.03)	0.46	− 0.005 (0.02)	− 0.22
Baseline age	0.07 (0.17)	0.45	0.31 (0.19)	1.67	0.24 (0.23)	1.06	0.17 (0.18)	0.96
Baseline TOCA-RR subscale score2	0.26 (0.03)	10.18***	0.21 (0.04)	5.99***	0.31 (0.03)	10.49***	0.30 (0.03)	9.62***

*p < .05, **p < .01, ***p < .001

1MRP, “poor peer relationships”; RA, “aggressive responses”; PDE, “poor school performance”; BA “low autonomy”

2Baseline score on the respective TOCA-RR subscale (MRP, RA, PDE, or BA) corresponding to each dependent variable

Association between Workshop Attendance, School Attendance, PSC-Y-CL Scores, and GPA

Students who attended more workshop sessions were significantly more likely to have higher school attendance in eighth grade (Table 3; B = 0.36, p < 0.001). Workshop attendance was associated with significantly higher PSC-Y-CL scores (more psychosocial problems) in eighth grade (B = 0.23, p = 0.012). This mean increase, however, was lower than that corresponding to the full longitudinal sample which included students who were not at psychosocial risk (M = 0.65, SD = 8.20). Student workshop attendance was not associated with eighth-grade GPA.

Discussion

The goal of the current study was to explore the feasibility and outcomes (behavioral, psychological, and academic) of a national school-based mental health program for middle school students in Chile. Study findings confirmed that it was feasible to implement all aspects of the program on a national scale.

In terms of outcomes, we found that student workshop attendance was significantly associated with better eighth-grade school attendance and peer relationships (as measured by the TOCA-RR MRP subscale). In addition, student workshop attendance was non-significantly associated with better overall TOCA-RR scores, school performance (TOCA-RR PDE subscale), and level of aggressive behavior in the classroom (TOCA-RR RA subscale) in eighth grade. In contrast, student workshop attendance was significantly associated with poorer psychosocial functioning (as measured by the PSC-Y-CL) in eighth grade.

This study provides preliminary evidence that SFL workshops were associated with positive outcomes in certain aspects of classroom adaptation and school engagement. In terms of Tier 1 (MH promotion for all students), program staff were able to administer the TOCA-RR screen to 28,204 (92%) students in 2016. It is worth noting that the average scores on the TOCA-RR improved from sixth to eighth grade. Not only were program staff able to screen a large number of students, they were able to carry out the promotion activities that, we speculate, may have led to improved TOCA-RR scores and improved GPA in the full sample of eighth graders.

Results from previous studies of the effectiveness of school-based universal prevention programs for adolescents have been rather mixed, with some showing beneficial results and others not (Werner-Seidler et al., 2017). The magnitude of the impacts reported in previous studies has varied (Durlak et al., 2011; Patalay et al., 2015; Volkaert et al., 2022). Thus, our TOCA-RR findings that range from significant to non-significant improvements are consistent with findings in the current literature.

Research findings suggest that contextual factors matter and these should be considered when analyzing MH-related data (Das et al., 2016). Importantly, our paper’s analyses compared the academic and demographic characteristics of students who were reportedly experiencing behavioral risk in sixth grade (and therefore eligible for the Tier 2) versus those who were not. This comparison demonstrates the well-documented interplay between poverty, MH, and educational achievement (Murali & Oyebode, 2004). Those at risk on the TOCA-RR and/or PSC-Y-CL were more likely to be from families receiving welfare (Chile Solidario), have lower average GPAs and rates of school attendance, and be older (suggesting higher rates of earlier grade retention) than their counterparts who were not at risk. Among students eligible for the workshops due to positive TOCA-RR screens, a number of risk factors (e.g., lower GPA, poorer school attendance, higher PSC-Y-CL score) were associated with lower likelihood of attending workshops. This finding reflects the familiar pattern of those at highest risk being least likely to access services (Hodgkinson et al., 2017). This result underscores the importance of tracking and following up on service referrals and outcomes over time and of continuing to search for ways to more effectively engage the highest risk youth. Providing equitable access to effective MH services to those with highest risk continues to be a problem that requires ongoing focus.

It is also worth noting that as part of Tier 1, teachers attend workshops where they learn about self-care and healthy ways to strengthen their bonds with students, parents, and colleagues. It is possible that improved teacher–student relationships increased teachers’ compassion for their students, which influenced their perceptions of students’ classroom behaviors. Research has demonstrated the powerful effects of teachers’ perceptions and expectations on their students’ long-term achievement (Alvidrez & Weinstein, 1999), potentially due to teachers’ perceptions of students influencing the students’ classroom behaviors and levels of motivation. Future research should explore the student–teacher relationship as a potential protective factor for younger adolescent Chilean students.

In interpreting these findings, it is important to remember that criteria for workshop referral was not based on psychosocial risk (PSC-Y-CL), but on classroom adaptation risk (TOCA-RR). Given the observed lack of improvement on PSC-Y-CL scores, the workshop curriculum may need to be expanded to incorporate additional material focused on addressing students’ MH needs to improve their psychosocial functioning. For example, students may benefit from learning strategies for emotion regulation, distress tolerance, and approach-based coping. The impact of the workshops may also be strengthened in this domain by covering more specific MH topics in both teacher and parent workshop sessions. In addition, large changes often take place during adolescence (e.g., increasingly complex social relationships); few studies effectively model these mechanisms of change with younger adolescents (De Luca et al., 2019). It will be important not only to consider the proximal and distal outcomes that should be measured, but also to consider that early adolescence can be a turbulent period in child development (Casey et al., 2010). For this reason, it might be helpful if outcomes were measured much closer to the end of the workshops, rather than or in addition to up to a year after the end of the workshops, as they were for this study. In addition, youth could be invited to complete a measure of workshop satisfaction to gather information about the workshops’ acceptability and subjective effectiveness.

Limitations

This study had many strengths, including its assessment of a national program using a large, longitudinal sample of Chilean youth. However, there were certain limitations that require acknowledgment. First, nearly 40% of students with complete sixth-grade data did not have complete eighth-grade data. However, significant predictors of missing eighth-grade data were therefore included in statistical models examining workshop effects, and they did not affect the association between workshop attendance and improved school attendance and peer relationships.

Moreover, eligible students were not randomized to attend or not attend the workshops, and there were baseline differences in characteristics between those who attended and did not attend them. Thus, these differences had to be controlled for in statistical analyses.

Another limitation involved students not being “blind” to whether they attended workshops when completing the follow-up PSC-Y-CL in eighth grade. It is also possible that certain eighth-grade teachers knew whether their students attended the seventh-grade workshops when they completed the follow-up TOCA-RR.

Conclusion and Future Directions

Education is a civil right, and the majority of children receive their educations through public education. Children in the public education system come from various backgrounds and have wide-ranging needs. However, access to education and resources is frequently inequitable. To address this disparity, Chile’s SFL program has made its mission to provide much-needed supports to students facing social, economic, psychological, and/or biological disadvantage. As part of their work, SFL has focused on addressing youths’ MH needs—with goals of ensuring students with MH risk can access the school curriculum and meet their achievement potential—by creating school-based MH prevention programs that now serve the full range of age groups.

Over the past decade, SFL has developed and piloted their middle school program, rolled it out nationally, begun to evaluate its effectiveness (Gallardo et al., 2015; Leiva et al., 2015), and made revisions based on evaluations and community feedback. This study is the first to demonstrate that, when implemented nationally on a large scale, attendance at the middle school workshops was associated with significantly improved school attendance and social relationships. These findings support the effectiveness of SFL’s thoughtful and responsive approach, which may serve as a model for other school-based MH programs.

Results from this study are already informing new efforts to further enhance the effectiveness of SFL’s prevention and intervention efforts. In 2020, due to COVID-19, the PSC-Y-CL alone was used to screen students for workshop referral. In order to continue providing much-needed services to students and their families, the SFL team quickly developed new protocols for screening online or over the telephone. They also identified workshop material that was appropriate to administering virtually as well as in-person. Ongoing research is examining the effects of workshops when held virtually in the wake of COVID-19. Next steps may involve additional statistical analyses aimed at identifying variables that contributed most significantly to strong workshop outcomes (e.g., based on data from regions/districts that showed the greatest workshop effects). Findings from these endeavors will inform future iterations of the workshops so they may continue to improve and respond to students’, families’, teachers’, and school communities’ evolving needs.

MH services are scarce, and many individuals do not receive the services they need. As students return to the classroom following the COVID-19 pandemic, it is important to consider schools as a space for MH and academic recovery. The SFL program is well positioned to provide MH supports in the school setting, during a time when MH services are needed most.

This graph depicts the evolution of Skills for Life’s middle school program from 2010 to 2021. Line on the graph represents the number of students in the program. There is a stark, upwards trend, resulting in 676,165 students served in 2021. Accompanying table for Fig. 1 provides number of student enrollment in SFLII as well as all levels (SFL I, II, and III).

This pyramid figure depicts the three-tiered model adopted by the World Health Organization, in which promotion of mental well-being represents Tier 1, prevention of further challenges represents Tier 2, and specialized support represents Tier 3.