Implementation of a Dutch school-based integrated approach targeting education, health and poverty-a process evaluation.

This study provides an evaluation of the implementation of a school-based integrated approach to improve academic outcomes by targeting children's education, health, and poverty. A two-year municipal subsidy program was provided to four primary schools in a deprived urban neighborhood in Amsterdam. Schools were put in charge of the implementation and coordination of the program. The municipality and district authorities provided assistance. This study evaluated whether the program functioned as integrated approach, i.e., whether it targeted multiple domains and environments by involving various agencies and actors, and what factors facilitated or hampered this. It also yielded an overview of the initiatives implemented and the facilitators and barriers of successful implementation of initiatives. Principals' perceptions served as the main input for this study. We thematically analyzed seven written customized plans for spending the subsidy (one to two per school), 15 transcripts of interviews with the principals (three to four per school) and the minutes of 16 meetings between principals, policy officers, and researchers. According to the principals, the schools had made great progress in the education domain and in improving the school's pedagogical climate, but in the health and poverty domains less progress had been made. Apart from the municipality, relatively few external agencies and actors had been actively involved in the program, and progress in other environments than the school was hardly achieved. This study shows that functioning of the program as integrated approach was facilitated by connections between initiatives, and that hired, well-trusted third parties may be crucial to establish these connections. Lay summary This study evaluated whether a two-year municipal program to improve academic outcomes by targeting children's education, health, and poverty, provided to primary schools in a deprived urban neighborhood, functioned as intended, and if so why, or if not, why not. The program was intended to function as integrated approach. This means that it was supposed to target the mentioned domains, the school, home, and neighborhood environment, and to involve various agencies and actors, such as school staff, policy officers, parents, children, and external organizations. The school principals could implement multiple, self-chosen, initiatives. According to the principals, on whose perceptions this evaluation study was primarily based, both teaching and the school climate improved during the program. However, improvements in children's health and poverty levels, and outside the school environment in general, were more difficult to achieve. In addition, the program involved mainly school staff and policy officers. The program thus functioned as an integrated approach, but only to a limited extent. The functioning of the program as integrated approach was facilitated by involving hired third parties to stimulate interconnection of initiatives, i.e., initiatives serving the same goals, involving multiple agencies and actors, and/or being implemented in the same location.

INTRODUCTION

Children growing up in poverty and in areas with high crime rates are at risk for behavioral problems (Votruba-Drzal, 2006; Shelleby et al., 2014) and criminal behavior (Damm and Dustmann, 2014). Compared to children from high socioeconomic status (SES) schools, children from low SES schools have poorer academic achievements, more behavioral problems, and lower well-being (Sellstrom, 2006). Behavioral, social and emotional competencies, health, health behaviors, and academic achievement are interrelated (Hill et al., 2004, Needham et al., 2004, Fiscella and Kitzman, 2009, Suhrcke and de Paz Nieves, 2011, Correa-Burrows et al., 2017). This interrelatedness highlights the importance of considering all these domains when improvements in a particular domain are warranted (Needham et al., 2004, Suhrcke and de Paz Nieves, 2011). Therefore, integrated approaches, in which multiple sectors, such as the health, education, and social security sectors, collaborate could be key to improving children’s health (Commission on Social Determinants of Health, 2008, Shankardass et al., 2012), and related outcomes (Murray et al., 2007, Samdal and Rowling, 2015). Such approaches should preferably be implemented in schools (Dooris, 2006). In the Netherlands, public health is organized outside schools, but is assumed to come about via intersectoral collaboration with schools (Maarse et al., 2018).

Compared to other European countries, the percentage of people at risk of poverty or social exclusion in the Netherlands is relatively low (16.7% in 2016) (Eurostat, 2018). However, underaged children are relatively more often part of a household with a low income (Centraal Bureau voor de Statistiek (CBS), 2018), especially in Amsterdam where in some neighborhoods more than one in three children live in a minimum income household (Onderzoek, Informatie en Statistiek (OIS), 2018). The numerous problems in these neighborhoods that were jeopardizing children’s academic achievements and educational trajectories induced the municipality of Amsterdam in 2016 to launch a two-year subsidy program to support schools in these neighborhoods in implementing an integrated approach (Gemeente Amsterdam, 2016). The integrated approach needed to target multiple domains and environments, through collaboration between various agencies and actors. The overall aim of the program was to provide all children with the opportunity to have the most successful educational trajectory possible, regardless of their race/ethnicity, home situation, and living environment. Schools were in charge, meaning that they could spend the subsidy on initiatives they considered suitable, as long as these targeted education, health, and/or poverty. Municipal policy officers were also involved in the program and implemented several initiatives in participating schools. The approach resembled a health promoting school approach, but differed in that the primary aim of the program was not to enhance children’s health, but to improve their academic outcomes (Turunen et al., 2017).

The current article reports on a qualitative evaluation of the integrated approach at four schools involved in the program from the start. As school principals have been shown to be crucial actors for successful implementation of school health promotion initiatives (Dadaczynski and Paulus, 2015, Roberts et al., 2016, Storey et al., 2016), this article builds on the principals’ perceptions. Just as in literature evaluating the health-promoting schools approach, we focus on the process of, and factors influencing, the implementation (Bartelink et al., 2019). The literature on this subject is scarce (Deschesnes et al., 2003, Samdal et al., 2011; Bergeron et al., 2019). Previous studies have identified ownership and empowerment, leadership and management, collaboration, and integration into the school system as facilitators of a health promoting school approach (Inchley et al., 2007, McIsaac et al., 2017, Darlington et al., 2018). Insufficient capacity, limited time, and poor home/school relationships have been identified as barriers to such an approach (McIsaac et al., 2017, Darlington et al., 2018, Hayes et al., 2019). The current paper contributes to this literature by examining the implementation of a school-based integrated approach targeting the domains of education, health, and poverty.

The aim of this study was to evaluate whether the program functioned as an integrated approach and why. To this end, we examined:

which initiatives were implemented and which subdomains they addressed;

how successful these initiatives were and which factors had influenced their implementation;

to what extent and why the approach targeted different domains and different environments, and involved different agencies and actors.

We hypothesized that the program would function as an integrated approach, because key actors would collaborate to implement initiatives and stimulate progress in all domains.

METHODS

Design

In a two-year multiple-case study, we monitored the progress of four primary schools—in the same deprived city district—implementing the municipal subsidy program.

Setting

To be eligible for participation in the program, schools had to have adequate educational quality and internal organization. Participating schools received 100 000 euros per year. At the end of the first year, one of the schools was no longer eligible and therefore did not receive the subsidy for the second year. All four schools additionally received 25 000 euros from a funding body serving vulnerable groups in society, to be spent on health promotion during the two-year program.

For each year that a school participated, it needed to draw up a customized plan, including goals, planned initiatives, and a budget plan. The municipality needed to approve the plans, but schools were allowed to adjust their plans during the year based on their experiences and new insights. The research team set similar requirements for spending the additional subsidy of the funding body serving vulnerable groups in society. A policy officer of the city district was program coordinator for the four schools.

Although located in the same city district, the schools differed in terms of size, student population, ideological basis, and previously implemented initiatives. For reasons of anonymity, no further details are provided about the schools.

Data sources

Primary data sources were: (1) the schools’ customized plans for spending the subsidy (n = 7); (2) transcripts of interviews with principals and/or vice-principals by the municipal policy officers and/or the research team to discuss implementation, facilitators, and barriers (n = 15); (3) minutes of meetings between principals, municipal and district policy officers, researchers and, on occasion, external parties, to discuss progress, planned initiatives, and problems experienced (n = 16). These were supplemented by secondary data sources: observations of researchers during school visits, policy letters from the municipality, program reports of the municipal research department, and informal interviews with other actors, such as the social worker, parent contact person and remedial teacher.

Data analysis

Thematic analysis was performed by one author and intensely discussed with the other authors. Data were coded using MaxQDA (version 2018). In the first phase, we used the schools’ customized plans, the interviews with school principals and the policy letters from the municipality to identify the initiatives implemented at the schools. These were then categorized into subdomains of the education, health, and poverty domains.

In the second phase, we read the interview transcripts and the minutes of meetings to collect all evaluative remarks by the school principals on the implementation or outcomes of the initiatives identified in the first phase and to identify the facilitators and barriers regarding the implementation of these initiatives. Based on whether the majority of the remarks concerning an initiative were positive, negative, or mixed or neutral, we marked each initiative as successful, unsuccessful, or undecided. Initiatives that were mentioned neither in the interview transcripts nor in the minutes of meetings were excluded from further analysis. Facilitators and barriers mentioned by more than one school principal were conceptually aggregated into a generic set. The informal interviews were used to check for deviating opinions.

In the third phase, we examined to what extent and why the program functioned as an integrated approach. This was done by evaluating the extent to which the program targeted multiple domains and environments and the extent to which different agencies and actors were involved, as well as by assessing which generic facilitators and barriers had been influential.

RESULTS

An overview of the domains, subdomains, and concomitant initiatives is presented in Figure 1. The subdomains addressed in the integrated approach were (a) competencies and capacity of the school team, tailored education, and broad education for the education domain, (b) coordination of care, physical health and socio-emotional health for the health domain, and (c) parental involvement for both domains. Below, we describe successful and unsuccessful initiatives for each subdomain, and the facilitators and barriers to successful implementation of initiatives (Figure 2). Finally, we describe to what extent the program functioned as an integrated approach and why.

Fig. 1:

Examples of initiatives (outer layer) implemented at the schools for each subdomain (middle layer) and domain (inner layer).

Fig. 2:

Schematic representation of facilitators of and barriers to successful implementation of initiatives and their interrelationships. Light gray = facilitator, dark gray = barrier, solid arrows = positive associations, dashed arrows = negative associations, bold = key facilitators/barriers. Note: only the most important relationships are shown.

Education

Principals consistently considered the competencies and capacity of the school team to have increased. Teachers’ knowledge and skills had been improved by introducing study materials and teaching methods, and by training and coaching teachers to apply these materials and methods.

I’m sure – and that’s also confirmed by external experts – that the expertise of the teachers has grown a lot, so you see individual children benefiting from that. – School B

Teacher shortage was reported as a major barrier. Schools especially had difficulty increasing the capacity of the school team by hiring more personnel or expanding their working hours. Limited capacity and time, in turn, hampered the implementation of initiatives.

As regards further professionalization, we didn’t really get round to that, as you’re faced all the time with… also because of the shortage of teachers, you have to manage somehow. – School C

But we weren’t fully able to increase the number of staff per class, as of course we’re having major problems finding substitute teachers. – School D

Teacher shortage was also the main barrier for implementing tailored education initiatives, such as providing more individual guidance through remedial teaching and classes for high achievers, and for setting up and coordinating broad education programs, such as design and technology and 21st-century skills classes, as responsible staff was often needed to help out elsewhere.

We’ve often been forced to have her [a remedial teacher] teach a group, as we just haven’t enough teachers available. – School A

Responsible staff being needed elsewhere also impeded the commitment of the school team and volunteers to such initiatives. Yet, where commitment was present, it facilitated the implementation by enabling active, frequent, and/or longer term follow-up of initiatives.

The principals reported that commitment was also hampered by unfavorable school dynamics, i.e., teachers changing jobs and children changing schools, and by initiatives not fitting in with the goals, values, and lives of the school team, parents, and children.

When people came in they had great enthusiasm and were full of good intentions, based upon their own professional expertise and their own experience of working with this target group. But we didn’t always connect properly. – School C

The fact that schools were in charge facilitated the correct tuning of initiatives to the goals, values, and lives of the school team, which increased the support of the team for these initiatives.

A strong and stable school team was mentioned as a facilitator of successful implementation, as such a team is able to take on additional tasks, signal problems, handle unexpected problems, and retain and apply the knowledge and skills they have gained. The temporary nature of the subsidy and the initially unclear conditions were reported as barriers. They caused delays and limited the implementation of longer term initiatives.

Education and health

Most schools experienced difficulties with their attempts to increase parental involvement in education. Nonetheless, one school reported a substantial increase in mothers’ educational as well as practical involvement. This was achieved by hiring a parent contact person for the entire school, assigning parent coordinators per class, undertaking various activities (e.g., a weekly breakfast meeting), and changing school procedures (e.g., scheduling an introduction meeting with parents).

And since they [the parents] are now far more often present in the school, partly because of the parent coordinator system we developed, they also see what happens in classrooms and the difference between parents who help their children at home [with school work] and those who don’t. And so what you can do at home, so it does have a great effect. – School A

Principals also mentioned that connecting with parents, their culture, and the community with an open mindset facilitated parental involvement. It enabled schools to make school procedures fit the parents’ goals, values, and lives.

While one parent prefers regular updates, another doesn’t, so they’ve [the teachers] also captured that: in what way do you want to receive feedback? […] So that worked very well, much more tailored – School A

Another facilitator of parental involvement was if the parent contact person was capable and trusted by both the school team and the parents. Schools that were less successful in increasing parental involvement had invested less, or less systematically in this, or had not been able to employ a suitable parent contact person.

Lack of involvement of parents was in itself considered a barrier to successful implementation of initiatives, as it impeded both children’s educational progress and school-based initiatives requiring parental participation.

Health

At the end of the program, the principals were still not satisfied about the coordination of care, mostly with external care organizations.

That’s where the children get stuck, and so in the end I say that if that’s not solved, then we’re unable to get the best out of the children in terms of education. That’s just impossible. – School B

Nonetheless, principals saw the increase in care provided within the school as a positive feature.

Yes, it’s an improvement that we now have our parent-child adviser [professional who can help parents and children], that’s really, I really think it’s a blessing […]. Now we have two days [a week], we actually have, we really have someone present here [in the school], and who is also available at other times. Well, I’m really, really very happy with that. – School A

Further increases in provided care within the school and better communication and coordination were called for. This was hampered by a lack of time, fragmented care, and/or existing rules and regulations, such as those regarding privacy.

The principals considered the few initiatives on physical health, such as providing daily breakfast and weekly lunch, to be a success. They were also positive about the more extensive investments in children’s socio-emotional health, for instance through school-wide programs to improve the pedagogical climate. The principals reported that their schools were now more peaceful places, there were fewer incidents, and incidents were dealt with more adequately and efficiently.

What you do see is that the measures we’ve taken regarding behavior and youth support services, we now have a whole range of preventive training courses […]. And for the older children we have implemented resilience training developed by [company name] for the more vulnerable boys this time, […]. That works very well, you notice that it’s very quiet in school, there are very few escalations. – School B

In turn, a more peaceful school environment facilitated the implementation of other initiatives, as it created a climate in which children can learn and teachers spend less time on conflict resolution and maintaining peace.

Poverty

The two poverty initiatives implemented by the municipality were considered a success. Many parents paid the voluntary contribution—for children to participate in events and field trips—using the City Pass, which offers various discounts to low-income families.

This morning I spoke to [name], our secretary, who told me that many parents had already used that system to pay, and that new payments are coming in every week. So that’s a great success. – School C

The social worker who handled payments with the City Pass in school was also available to help parents figure out for which financial arrangements they were eligible, and help them apply.

So an added advantage of this City Pass is that it leads to talks with [name of social worker]. – School B

The principals reported that implementing the two initiatives in the domain of poverty in the schools meant that parents became acquainted with the social worker and were able to consult him/her when dropping off or picking up their children from school. The threshold for consulting the social worker was further lowered by the connection to the low-threshold initiative of the City Pass.

The weekly presence of a social worker in the school also helped build up trust. Understanding and accepting that building up trust and achieving change take time was an important facilitator for success.

The consultation hours [of the social worker], that’s great. […] the fact that [the social worker] is now a familiar face, that parents share that with each other, that sense of trust will have to grow, based on the experiences people have with it, or hear about from others. – School A

The connections between the initiatives were perceived to make coordinating the initiatives easier. In general, though, the coordination and the time it required were considered to be important barriers to implementation.

In my view it’s still not enough, as you should be offering more, but we’re reaching the limits of our possibilities as regards the ability to coordinate, to manage and control, while [at the same time] enabling staff to keep up their normal work. – School C

Integrated approach

Overall, the school-based program did reflect an integrated approach, in the sense that the schools managed to implement initiatives in each of the domains of education, health, and poverty. However, there were large between-domain differences in the number of initiatives that were both implemented and successful.

Integration was reported to improve when external agencies and actors implemented initiatives targeting health and poverty inside school, i.e., an environment primarily concerned with education. This facilitated contact between agencies and actors of different domains and familiarized them with each other’s work. In addition, it helped connect and align different initiatives, as the schools kept an overview of the situation and safeguarded the overall goal.

The school principals reported that the local coordination of the municipal program by a policy officer of the city district, and the links with the municipality, also facilitated integration. It improved the principals’ awareness of local initiatives and connected them to key actors, such as policy officers and external agencies.

Well … that’s what I think we need most, just to have a clear view of the possibilities and of what is available, so you can take advantage of that as schools; yes, definitely – School A

That has been intensified by this kind of project where you know you can resort to someone. That you know they are aware of you. That you know you can ask questions. That’s what I liked a lot – School C

The program coordinator also supported meetings between the schools. Although these meetings took up time, they facilitated an integrated approach, as principals got acquainted with each other and became inspired to work together and solve shared problems, e.g., conflicts between their students.

But that’s what I consider to be benefits of this whole situation, that we get to know each other, that we talk to each other, that we’re on the same page. […]. We’ve had very few really serious incidents since this started, so I think we now have very short communication lines, we can access each other immediately – School A

At the same time, the school-based nature of the program also prevented the program from functioning as an integrated approach, in that other environments than the school, such as the children’s home situation or living environment, were less targeted. Also, parents and actors from external agencies were involved to a much lesser extent than school staff.

DISCUSSION

Our multiple-case study evaluated whether a two-year subsidy program by the Amsterdam municipality, intended to aid the educational trajectories of children in a deprived urban area, functioned as an integrated approach. Although initiatives in all domains were implemented, the schools’ progress mainly concerned the domain of education, and they were less successful in realizing an integrated approach in terms of targeting environments other than the school and in terms of involving agencies and actors from outside the school. Integration was facilitated by implementing initiatives targeting other domains than education inside the school. This simplified the establishment of connections between initiatives and the agencies and actors involved, and helped the school safeguard the overall goal. Integration was also facilitated by involving third parties who could interconnect both the schools and the agencies and actors from different domains. As shown previously for the health promoting school approach (Valois et al., 2015), factors facilitating implementation of the approach appeared to work in concert.

Putting schools in charge mainly resulted in progress in the educational domain. By being put in charge, school principals could implement initiatives that fitted the goals, values, and lives of the school team, parents and children. Previous research has identified a sense of ownership (Inchley et al., 2007, Darlington et al., 2018, Bergeron et al., 2019) and a proper fit between initiatives and an organizations’ mission, priorities, and existing practices to be important facilitators for success (Inchley et al., 2007, Durlak and DuPre, 2008, Langford et al., 2015, Pearson et al., 2015, Rasberry et al., 2015, Darlington et al., 2018). In addition, being put in charge allowed principals to connect several initiatives, which helped the program function as an integrated approach. For example, the current as well as previous studies found that the pedagogical climate at the school, and programs to improve this climate, influenced successful implementation of other initiatives (Lewallen et al., 2015, Darlington et al., 2018). Next, putting schools in charge allowed them to take the time needed to implement initiatives and establish change. Allocating enough time has also been identified previously as a facilitator of implementation (Roberts et al., 2016). However, putting schools in charge also hampered the program’s function as an integrated approach, as domains not within direct reach of the schools were less targeted.

Another important facilitator of successful implementation identified in the literature is a program champion, i.e., someone who believes in a particular initiative and who is able to implement it or influence implementation (Durlak and DuPre, 2008, Sanchez et al., 2014, Pearson et al., 2015, Tooher et al., 2017, Hayes et al., 2019). The current and previous studies found that the ability to implement initiatives requires sufficient capacity and time (McIsaac et al., 2017). Moreover, we found that implementers should be trusted by the schools and the other agencies and actors involved. Earlier studies found that new school health coordinators needed to build a relationship of trust with stakeholders involved (Storey et al., 2016) and that trust in the abilities and intentions of stakeholders involved facilitated intersectoral school health promotion interventions (Bergeron et al., 2019). Our findings add to this literature by showing that trust in the person leading an initiative is crucial to involve the school team, parents, and children, and to allow a school program to function as an integrated approach.

Schools had difficulty increasing parental involvement in education. Parental involvement is recognized as one of the most challenging elements in implementing health promoting school activities (Inchley et al., 2007, Langford et al., 2015). However, parental involvement is an important facilitator for health promotion in schools (Storey et al., 2016, Persson and Haraldsson, 2017), and a poor home–school relationship acts as a barrier (Darlington et al., 2018). We found that a lack of involvement of parents acted as a barrier to successful implementation of initiatives. In order to increase parental involvement, the schools in our study emphasized the importance of connecting with parents, their culture and the community, with an open mindset. Previous research also found that an open-minded attitude and commitment in the relationship facilitated the work of partnerships (Marlier et al., 2015). In addition, parental involvement has been shown to be influenced by parents’ perceptions of invitations (Hoover-Dempsey et al., 2005), positive communication by school staff, and cultural and language barriers (Reynolds et al., 2015). We found that a parent contact person, who was trusted by both the school team and parents, aided parents’ connection and their involvement with the school.

A strength of the current study is its variety of methods and its wealth of data to assess the implementation of the school-based approach. Another strength is that the findings provide information on the facilitators and barriers regarding the implementation of a wide range of initiatives, rather than one specific initiative. A final strength is that this study concerned the implementation of a school-based integrated approach. Literature on such approaches is scarce.

A limitation of the current study is that the context of the school (Keshavarz et al., 2010), in terms of student population, ideological background, and previously implemented initiatives, was not taken into account. However, our multiple-case study allowed us to identify a generic set of facilitators and barriers that can guide future implementation of integrated approaches in schools. A second limitation is that, as the schools in our study were located in the same neighborhood and in close contact, the principals may have influenced each other’s opinions and responses. As contacts between the schools were a crucial element of the program, there was no way to overcome this limitation. A final limitation is that our analysis was primarily based on the perceptions of school principals. However, we did not come across deviant opinions in the various secondary data sources.

Conclusion

This study provides a unique evaluation of a primary school-based integrated approach to improve children’s educational trajectories. For school-based integrated approaches to be successful, connection between initiatives seems crucial. We add to the literature that hired third parties, that match well with and are trusted by the school team, can importantly facilitate this connection by stimulating collaboration between schools and external agencies and actors. This allows for external agencies and actors to come into the schools, and facilitates initiatives of external agencies and actors to connect with school initiatives. This brings domains other than education, and environments other than the school, within the school’s reach, allowing for truly integrated approaches.

ETHICAL APPROVAL

According to the Medical Ethics Committee of the Amsterdam UMC—location AMC, this study did not require approval from a medical research ethics committee, because it was not subject to the Dutch Medical Research Involving Human Subjects Act (WMO).

FUNDING

This work was supported by Fonds NutsOhra (FNO) [grant number 101.243].