Evaluation of the implementation of a relationship and sexuality education project in Western Australian schools: protocol of a multiple, embedded case study.

INTRODUCTION: There is recognition of the importance of comprehensive relationships and sexuality education (RSE) throughout the school years worldwide. Interventions have found some positive outcomes; however, the need for a greater focus on positive sexuality and relevant contemporary issues has been identified by teachers and students. The Curtin RSE Project provides training for teachers and preservice teachers and supports schools through training and advice to implement comprehensive school health promotion (CSHP) focusing on RSE allowing schools to develop programmes relevant to their school community. To examine contemporary phenomenon within a real word context, a case study design will be used to measure implementation. This paper will describe the protocol for a multiple, embedded case study to measure the implementation of CSHP focusing on RSE in a purposive sample of Western Australian schools. METHODS AND ANALYSIS: This mixed methods study will include a multiple, embedded case study. Schools (n=3-4) will be purposively selected from within Western Australia based on their capacity to commit to implementing RSE as a case study school. Data will be collected from students (Grade 6 for primary school; Grades 7-12 for secondary school); teachers and other key staff and parents. Methods include school climate and school curriculum audits, documentation (collected with key staff at baseline and annually), interviews (parents and teachers at Year 2), focus groups (students at Year 2) and an online student survey (collected with students baseline and annually). ETHICS AND DISSEMINATION: School principals will provide consent for school participation and staff and parents will provide individual consent. Student assent and parental consent will be obtained for student participants. Results will be disseminated through open-access reports, peer-reviewed journals and conference presentations. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The case study design will allow for evaluation of the implementation of whole school strategies in a ‘real-world’ setting.

Schools will be supported to develop comprehensive school health promotion strategies appropriate to their school allowing the development of unique strategies relevant to each school.

Collection of multiple sources of data (surveys, focus groups, interviews, school climate audit, school curriculum audit and documentation) will allow for triangulation.

Generalisability of the specific strategies is limited; however, the process will be useful to other schools.

Case study design does not allow for randomisation and findings may not be generalisable.

Introduction

Comprehensive relationships and sexuality education (RSE) in schools has been associated with positive preventive outcomes including delayed onset of sexual debut and reduction in the frequency of unprotected sexual activity.1–4 School-based programmes have also reported some success in enhancing relational skills related to reducing prevalence of bullying.5 6 However, there is less evidence of the effectiveness of interventions focusing on broader RSE including positive sexuality.7 There is significant discussion, including from students,7 about the importance of relevant RSE which includes positive aspects of sexuality8 9 and engages students in contemporary issues such as gender diversity,10 11 discrimination, violence, respectful relationships,8 12 pornography13 and use of social media.14 School students have found the delivery of RSE to be limited in addressing the complex issues faced by adolescents.7 15 16

Planning and implementation of school-based programmes can be guided by frameworks. The health promoting schools (HPS) framework focuses on three key domains: Curriculum, Learning and Teaching; School Organisation, Ethos and Environment; and Partnerships and Services.17 18 While classroom activities provide opportunity for RSE,19 a comprehensive approach recognises the importance of socioecological factors in human development20 highlighting the need for whole school-based programmes. Consistent with Bronfenbrenner’s socioecological model,20 comprehensive school health promotion (CSHP) recognises the complex interrelationship between individuals, determinants of health and the broader environment.21 22 A socioecological approach recognises complex interactions between individuals, interpersonal, organisational, community and societal factors23 24 which helps inform planning and implementation of strategies relevant to specific school communities. This is especially salient for RSE considering the interactions and influences between students, friends, family and school staff along with broader organisational, community and societal interactions and influences.

Although CSHP is endorsed by the WHO18 and implemented worldwide,25 focus is rarely on RSE26 with sexual health programmes often confined to classroom activities, with the absence of whole school strategies such as parent engagement and policy implementation.25 Others have found the emphasis of CSHP was more likely to be on curriculum as opposed to a broader HPS focus27 28 with health issues viewed in isolation and schools being most likely to implement programmes targeting physical activity and nutrition.27 29 Despite the recognition of the potential of HPS for over two decades,30 there are a number of barriers to CSHP focusing on RSE including school and parental support; the ad hoc nature of curricular delivery; limited teacher professional development contributing to apprehension; moral and ethical challenges; parental and community concerns; and crowded curriculum.26

Although schools are considered to be good settings for health promotion and there is evidence that skill-based programmes can impact positively, translation of rigorous evidenced-based programmes within schools presents challenges.29 Randomised control trials provide some evidence of programme effectiveness;5 however given the difficulties in replicating ‘ideal’ programmes, it is important to consider other methods of evaluation.31 Implementation is integral to programme effectiveness. Programme effectiveness and fidelity are dependent on whether the programme has been implemented as intended; however, contexts between schools are variable and in many cases the context of implementation often differs from what had been foreseen during programme design.32 Furthermore, there is less understanding of how programmes can be best implemented in schools.33 School-based interventions are complex, involving a range of actors and strategies within a complex system.33 34 A recent systematic review highlights four key areas of focus integral to the implementation of school health promotion programmes: preparing for implementation, initial implementation, embedding into routine practice, and adaptation and evolution. The level of engagement and reciprocity between school staff, students, parents and external stakeholders is likely to vary between schools, highlighting the importance of individual school planning and implementation.33

Case studies allow for the examination of contemporary phenomenon within a real-world context.35 An exploratory multiple case study of six secondary schools in Portugal found improvement in teacher professional competencies. However, despite including teachers from different learning areas, contextual barriers associated with integration of sexuality education into the curricular and timetabling issues were common.36 Case study design has also been employed to evaluate a whole school approach to sexuality education. A capacity building project across five campus colleges (Grades 1–12) in regional Victoria reported positive outcomes over a 5-year project.26 The project aimed to build capacity with professional development provided for teachers and key staff, a mentor programme established, a community engagement/parent liaison position created and parent forums conducted on all campuses. Embedding sexuality education through documented policy and programme planning which included the voices of students, parents and teachers was found to be important. The development of partnerships strengthened the implementation and enhanced community support and commitment.26

Since 2014, the Curtin RSE Project has been funded by the Health Department Western Australia to provide a range of professional development opportunities for preservice and in-service teachers in the area of school-based RSE with a focus on CSHP across all school grades (K-12). This includes provision of specific tertiary units, symposia, workshops and online support. The Project is available to all school staff and preservice education students in Western Australia. Evaluation of the teacher training workshops has highlighted the benefits of professional development with statistically significant improvements in attitudes towards RSE and increased confidence facilitating RSE. However, there is also a need to determine longer-term impact of the Curtin RSE, especially related to school implementation of whole school-based outcomes.37

A case study design was selected to evaluate the implementation of the Curtin RSE Project. Case study design allows for in-depth analysis of specific schools. A strength of CSHP is the ability for schools to focus on issues of specific relevance to their school. A multiple case study design will allow for the inclusion of primary (K-Grade 6) and secondary schools (Grades 7–12) with the intention of literal replication across cases.35 While schools may select developmentally, pedagogically and locally appropriate strategies for their school, it is anticipated that implementation of RSE will result in positive outcomes. The embedded case study design will use holistic data collection strategies for studying each school as a whole case and surveys will be used to collect data from the embedded unit (students) of the analysis.35 The purpose of the case study is not to measure long-term impact of RSE but to collect data regarding what is possible to implement in schools with a realistic level of support. This paper will describe the protocol for a multiple, embedded case study to measure the implementation of CSHP RSE in a purposive sample of Western Australian schools.

Aims and objectives

This study aims to explore how teacher training and whole school support can impact the implementation of CSHP focusing on RSE in a purposive sample of schools.

Study objectives

The study objectives of this project will be to:

Measure the implementation of whole school health promoting RSE strategies in case study schools.

Explore the perceptions of students, school staff and parents towards RSE in case study schools.

Identify enablers and barriers to comprehensive school-based RSE promotion within case study schools.

Methods and analysis

This mixed methods study will include a multiple, embedded case study (including school climate and school curriculum audits, documentation, interviews, focus groups and surveys)38 39 in purposively selected case study schools. Case studies will be implemented over a minimum of two school years. The audits, documentation, interviews, focus groups and student surveys will be used to measure the implementation of the case study in each school. Student perceptions will be explored through data collected by surveys and focus groups. Perception of parents and school staff will be explored through one-on-one interviews. One-on-one interviews with school staff and parents and focus groups with students will explore barriers and enablers to comprehensive RSE (see figure 1).

Figure 1

Case study planning, implementation and evaluation provide an overview of the case study process. Methods of data collection are linked to objectives. In Year 1, one staff from all case study schools will participate in teacher training and schools will select programme strategies specific to their school. In Year 2, two schools will continue strategy implementation. The School RSE committee meets throughout the case study to plan and implement strategies.

Setting

Based on resources, and to capture diversity of schools, the focus of this study will include three to four schools. It is anticipated that at least one primary and one secondary school be recruited from the Perth metropolitan area, in addition to at least one rural school. Further schools will be recruited if additional resources are obtained. Case study literature suggests the sample size for multiple case study design is dependent on time and resources, however as little as two sites may be used.35 Case study schools will plan for developmentally appropriate CSHP strategies within the whole school community. For primary schools, this includes Grade K-6 (ages 3–12 years); secondary schools include Grades 7–12 (ages 11–18 years). Schools from low- to middle-socioeconomic status will be provided preference.

Recruitment

Schools will be purposively selected based on capacity to commit to implementing RSE as a case study school. Invitations will be sent via email to schools using the Curtin RSE Project’s database.

Participants

Consent for the school to participate in the study will be sought from the Principal and participants will be recruited through case study schools. Consent will be sought from all participants; students will provide assent and their parents will consent for their child to participate. Data for the documentation, school climate audit, school curriculum audit and observations will be collected with the support of the school RSE committee. Students from Grade 6 (primary school; aged 10–12 years) and Grades 7–12 (secondary; aged 11–18 years) will be invited to participate in the online survey. Younger primary school students will not be included in the data collection. Sample will be dependent on case study school size; however, metropolitan primary schools are likely to include intakes of at least 50 students (Grade 6) while total secondary school intake (Grades 7–12) will be at least 900 students in metropolitan schools. One-on-one in-depth interviews will be conducted with relevant school staff and parents (at least 10–15 from each school). Focus groups will be conducted with a purposive sample of students from Grade 6, Grade 7, Grade 9 and Grades 11–12 (6–8 students in each group; at least two groups from each grade at each school).

Patient and public involvement

The need to measure the impact of the implementation of comprehensive RSE has been driven from the literature,8 9 from teachers involved in the broader RSE study,20 and through formative discussions with school staff from two potential case study schools. The nature of this case study design will allow for participant interaction in the planning and implementation of school-based strategies and evaluation of the implementation with members of the school community including students, teachers, other relevant school staff, parents and external stakeholders. The reference group, with representatives from the school community and external stakeholders will direct the planning, however planning of specific strategies will include a more diverse range of participants. For example, different school and external community members may be involved in developing and implementing specific strategies in line with their skills and interests. The initial school climate and curriculum audits will drive the development of initial strategies relevant to each school community.

The RSE case study

The 2-year case study forms part of the broader Curtin RSE Project which provides professional development for practising and preservice teachers.37 Case study schools will establish a RSE committee with involvement of senior administrative staff (ideally the Principal), interested teachers and health services staff in addition to relevant school community and broader community stakeholders. Committee membership will be informed by the issues most salient to the school and new members may be invited during the project. At least three to four teachers from each school will attend a 2-day RSE workshop in the first year of their case study (for details of workshop focus and content, see Burns and Hendriks37) and will be encouraged to commit to implementing RSE strategies focusing on CSHP in their school.18 40 Case study schools will receive direct support from the project team to assist planning and implementing strategies specific to their school. Support will vary depending on the strategies implemented by each school. In addition, potential strategies will be discussed at the workshops, other professional development opportunities and via online support.

Data collection

Documentation

Schools will be asked to provide relevant documents that may support their efforts as a HPS. For example, documents could include relevant school-based policies, items in school newsletters or published on school websites, awareness raising materials developed by students, minutes of relevant health education meetings, records of health curriculum including RSE, parent information, information about workshops and seminars held by the school for the school community. Documents will be used to corroborate other data and will be collected at baseline to assist planning and then reviewed at post intervention. Alternatively, where documentation is contradictory to other data, further investigation will be undertaken.35

School-based audit tools

A school climate and school curriculum audit tool will be developed. Tools focusing on school commitment to Sexual Orientation, Gender Identity and Expression discrimination and bullying in the Southeast Asia region41 and bullying in Australian schools which has been informed by a body of research5 6 40 42 43 will be used to inform the development of this tool. Consistent with a HPSs approach,17 18 the school climate audit will include interrelated components which are considered essential for positive whole school change: leadership and commitment (governance and leadership; planning, monitoring and communication; school policies and procedures); school culture and environment (school ethos, vision and values; internal and external communication; school environment and facilities); professional learning; teaching and learning; parent and community partnerships; and evaluation and review.

The development of the school curriculum audit tool will be informed using previous RSE audit tools.38 39 41 The tool will be mapped against the Western Australian School Curriculum and Standards Authority (WASCSA) Health and Physical Education (HPE) curriculum.44 The curriculum scope and sequence provides content descriptors. While most RSE is taught through the HPE curriculum in Western Australia opportunities to include RSE content across learning areas will also be explored. Since 2017, all Western Australian schools have been required to include the mandated HPE WASCSA curriculum (based on the Australian National Curriculum).45 The curriculum audit will include current content, context and learning area of delivery, future plans, resources and support required to enable effective implementation. The school climate and the school curriculum audits will be completed at the beginning of the project, then annually by key teachers from the RSE committee with the support of the project team.

Interviews

One-on-one semi-structured interviews will be conducted with key staff at each school during the final year of the project. Key staff will include teachers, the school nurse and other relevant school-based health services personnel (school-based health services personnel varies between schools46). The numbers will vary dependent on the structure of the school; however, it is anticipated that at least 5–10 school staff and five parents from each school will be interviewed. The interviews will focus on barriers and enablers to RSE and will include questions relating to the implementation of curriculum and other whole school strategies. Interviews will be conducted by a researcher experienced in qualitative interview techniques, will take between 40 and 60 min and will be conducted at a time and venue suitable to each stakeholder. Interviews will be administered face-to-face or by telephone dependent on location and will be audio-recorded.

Focus groups

A purposive sample of students will be sought from each school (6–per group) to participate in the focus groups. To enable a strong representation across age groups, focus groups will be conducted with students from Grade 6 (primary school), Grade 7, Grade 9 and Grades 11–12 (secondary school). At least two groups for each year level will be conducted at each school. Focus groups will be conducted during the second year of the case study and will explore barriers and enablers to RSE, issues of relevance to their age group, attitudes towards RSE in schools and RSE experienced at school. Focus groups will be conducted at school in a quiet room at a time convenient to the school and students and will take approximately 60 min and will be audio-recorded.

Survey

All secondary students (Grades 7–12) and Grade 6 students from primary schools in participating case study schools will be invited to complete an online survey. Students will be invited to identify sources of RSE advice from a list and identify their most trusted sources of advice. Facilitation of RSE at school including prevalence of RSE at schools, subjects in which RSE was included and level of schooling when RSE was taught will be explored. Students will be asked who taught RSE (eg, teacher, school nurse, etc.) and their perceived relevance of RSE content and skills taught. These questions have been previously used and validated with Australian secondary school students (Grades 7–12).15 Level of comfort with RSE will be determined by three questions relating to feeling embarrassed, uncomfortable and annoyed (Likert scale: always, sometimes, never) in addition to open-ended questions allowing students to discuss these issues further. To determine RSE taught at their current school, students will be asked to identify topics and facilitation methods used. Open-ended questions will be provided to determine most important topics and how RSE could be improved at the school.47

The cross-sectional online survey will be administered at baseline and repeated at the end of the 2-year case study. Administration will occur at a time convenient to the school (eg, health education, form class) to allow for minimal disruption and will be administered by class teachers who will receive protocols for administration. Students will be provided a link which can be completed in a computer laboratory, lap top, or tablet. In the event of the students not having access to a computer or tablet at school, the project team will bring a class set of tablets. The survey will be completed via Qualtrics which has the capacity to be administered offline if required and will take between 20 and 30 min to complete.

Data analysis

The use of multiple sources of evidence is a strength of case study design and enables triangulation of data which strengthens the construct validity of the case study.35 To ensure comprehensive reporting of findings Consolidated criteria for Reporting Qualitative research48 and Strengthening the Reporting of Observational Studies in Epidemiology49 checklists will be followed for qualitative and quantitative components of the study, respectively.

Qualitative data analysis

A grounded theory approach to analysis the qualitative data (documentation, interviews, focus groups, open-ended questions from the survey, qualitative data from the audit tools) will be employed. Constant comparison analysis which requires the researcher to continually sort through the data collected, code information to identify key themes and reinforce theory generation50 51 will be used. This method allows the researcher to generate knowledge about common patterns and themes within the human experience.52 Although the intent of this research is not to develop a grounded theory, the rigour associated with the technique will enable the researchers to present detailed and rich data.51

To ensure credibility, the interview process employed in the focus groups and interviews will be consistent with best practice for interviewing children, adolescents and adults.53 54 To reduce bias and enhance conformability, the coding and themes will be analysed by the research group. Transferability will be achieved by ensuring the data are as rich as possible.55 The case study method allows for comparison of data collected through interviews, focus groups, documentation, open-ended questions and school climate and school curriculum audits. Qualitative data will be managed and stored using NVivo.56

Audit tool analysis

The school climate and school curriculum audit tools will be used to help schools plan their RSE promoting school strategies and curriculum. These data will also be collated as part of the case study data. Process data will be collected to determine the number and type of strategies implemented. Qualitative data will be collected to explore barriers and enablers to the implementation of the school climate and curriculum audit tools and will be analysed as described above.

Quantitative data analysis

Data will be initially analysed by gender and school. Univariate analysis will determine the association between the dependent (taught RSE at current school) and independent variables. Regression analysis will explore associations between the dependent variable and independent variables. Due to the cross-sectional nature of this study, two-proportion z-tests and two sample t-tests will be employed to determine statistical significance between the two time periods for categorical and continuous variables, respectively. Levels of statistical significance will be discerned by p values of p<0.05 (moderate) and p<0.001 (high).

Ethics and dissemination

School principals will also provide consent for their school to participate in the study. All participants will be provided information about the study prior to participation. Assent will be obtained from students and consent from parents for student participation in the surveys and focus groups. The school will provide information forms and collect assent and consent from students and parents either electronically or as paper-based documents. Staff and parents involved in the interviews will provide consent. The school and individual participants will be able to withdraw from the research at any time without negative consequence. The school nurse will be available for any participants if they become distressed during any data collection. External referral will also be provided if required. Students will be provided helpline contacts when they complete the survey and focus groups. Staff and parents will be provided appropriate contacts at the interviews. Survey data will be anonymous; focus group and interview data will be de-identified. De-identified reports will be provided to the school RSE committee and a summary provided for dissemination to families and staff. Care will be taken to ensure qualitative data presented is non-identifying. Findings will allow each case study school to plan their ongoing RSE programmes. Results will be disseminated through an open-access report, peer-reviewed journals and conference presentations. Findings will inform subsequent practising and preservice teacher training implemented as part of the RSE and successful strategies will be provided to schools via online sites.