Classroom Promotion of Oral Language (CPOL): protocol for a cluster randomised controlled trial of a school-based intervention to improve children's literacy outcomes at grade 3, oral language and mental health.

INTRODUCTION: Oral language and literacy competence are major influences on children's developmental pathways and life success. Children who do not develop the necessary language and literacy skills in the early years of school then go on to face long-term difficulties. Improving teacher effectiveness may be a critical step in lifting oral language and literacy outcomes. The Classroom Promotion of Oral Language trial aims to determine whether a specifically designed teacher professional learning programme focusing on promoting oral language can lead to improved teacher knowledge and practice, and advance outcomes in oral language and literacy for early years school children, compared with usual practice. METHODS AND ANALYSIS: This is a two-arm cluster multisite randomised controlled trial conducted within Catholic and Government primary schools across Victoria, Australia. The intervention comprises 4 days of face-to-face professional learning for teachers and ongoing implementation support via a specific worker. The primary outcome is reading ability of the students at grade 3, and the secondary outcomes are teacher knowledge and practice, student mental health, reading comprehension and language ability at grade 1; and literacy, writing and numeracy at grade 3. Economic evaluation will compare the incremental costs of the intervention to the measured primary and secondary outcomes. ETHICS AND DISSEMINATION: This trial was approved by the Monash University Human Research Ethics Committee #CF13/2634-2013001403 and later transferred to the University of Melbourne #1545540. The investigators (including Government and Catholic partners) will communicate trial results to stakeholders, collaborators and participating schools and teachers via appropriate presentations and publications. TRIAL REGISTRATION NUMBER: ISRCTN77681972; Pre-results. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

This trial addresses identified evidence gaps in the effectiveness of classroom-level oral language teaching interventions.

This trial is one of the few methodologically rigorous studies evaluating the potential impact of teacher professional learning on student outcomes.

The use of existing data for baseline and outcome data mixed with brief face-to-face classroom assessments presents a potentially efficient model of research in schools that can be implemented at scale.

There is potential bias from only including schools who opted into the study and limited potential generalisability across education systems where school starting age and teacher preservice education may vary.

The teacher and student movement in this effectiveness trial will affect the capacity to deliver the full ‘dose’ and thus may adversely and differentially affect retention rates.

While most outcomes are objectively measured, mental health outcomes remain a subjective measure and in our study are mostly limited to teacher report and may not reflect mental health outside the school setting.

Introduction

The ability to use oral language to communicate effectively is a key foundation for academic success as well as social and economic participation across the life span.1 2 Receptive and expressive oral language encompasses vocabulary (ie, words), and the grammatical rules and complex pragmatic conventions that are intrinsic to the social and contextual aspects of communication.3 The ability to use language effectively impacts on children’s learning, their social behaviour in and out of the classroom and their ability to develop literacy and numeracy skills.4 5 With respect to literacy, oral language skills underpin the ability to decode and understand text, as well as writing and spelling, and the ability to engage with text across the curriculum. While learning to speak is a task for which humans are generally considered to be biologically well-prepared,6 reading and writing skills are not, requiring prolonged and specific instruction in order for proficiency to be achieved.4 Children who do not master the basics of literacy in the early years of school face long-term academic struggles, are often ambivalent towards school and may face a range of behavioural, social, vocational and social-emotional difficulties into adolescence and adulthood.4 7

There is a clear interdependence between the transition to literacy in the early school years and oral language competence. Snowling and Hulme5 observed that ‘literacy is parasitic on language’ (p. 597), meaning that children’s ability to learn in the classroom and develop literacy skills is reliant on their ability to understand and use oral language effectively. Conversely, the literature examining causes of reading difficulties emphasises the influence of difficulties with oral language on literacy attainment.8 Given the central role that oral language competence plays in academic success, it follows that strategic efforts to improve oral language skills in the early years of school should confer gains in literacy skills, social and emotional well-being and academic trajectories taking into account the considerable impact of socioeconomic status (SES) on language.9–12

The Australian Early Development Census (AEDC)13 (a population measure of early childhood development completed by teachers on all children at school entry) shows that at the start of primary school, around the age of 5 years, significant SES-based disparities in language functioning are already evident. The 2015 AEDC results show that children who live in areas characterised by the greatest socioeconomic disadvantage have the highest rates of developmental vulnerability on the language and cognitive skills (school-based) domain (12.4%), which encompasses literacy and numeracy skills such as letter and sound awareness, rhyming, ability to write own name and simple words and sentences, ability to count to 20, recognise shapes and numbers and compare and sort numbers. This level of developmental vulnerability is more than four times the 3% of children living in the most advantaged areas whose language and cognitive skills are vulnerable14 and is consistent with the increasing evidence of relatively poorer language performance in young children from low SES backgrounds, inequities that are also reflected in later schooling.11 12 15 The Industry Skills Council of Australia16 has identified that ‘literally millions of Australians have insufficient language, literacy and numeracy skills to benefit fully from training or to participate effectively at work’ (p. 1), indicating that such difficulties do not spontaneously resolve over time.16

The early years of schooling represent an opportunity to make a substantive difference to educational and life outcomes by addressing the language abilities of whole populations of children.17 18 The reported high rates of developmental vulnerability with respect to language skills at school entry would suggest that this is both an area in need of pressing attention, and an immediate opportunity for improvement.

Evidence arising from a number of recent Australian and international studies suggests that efforts to impact student outcomes and address SES-based disparities must be centred on teaching quality.19–21 The emphasis on improving teaching aligns with international research highlighting the need to invest in teaching.22–24 Improving language and literacy outcomes for school-aged children must therefore explicitly address teaching quality with respect to teacher knowledge and skills, effective formative assessment and instructional decisions. However, the systematic inclusion of language (eg, grammar) together with specific, phonics-based instruction has been limited in schools for decades.4 25–30 This presents an implementation challenge, as current teachers are often never taught these skills, either during their own schooling or in their pre-service teacher education. Teachers’ foundational knowledge, skills and attitudes regarding language and literacy are critical in any effort to target classroom-based approaches to improve student outcomes.31

Despite the clear importance of oral language for academic achievement, there seems to be no published, rigorous trials of oral language teacher professional learning intervention that have demonstrated a sustained change in student outcomes and/or teacher knowledge and practice. Large-scale randomised controlled trials (RCTs) in schools to test education interventions are uncommon but gaining momentum in Australia and internationally, including low-income and middle-income countries. For example, organisations such as the UK-based Education Endowment Fund and Social Ventures Australia have stimulated interest in trials by funding RCTs to test the effectiveness and cost-effectiveness of teaching and learning interventions.32 33 In Australia, the Oral Language Supporting Early Literacy (OLSEL) pilot RCT demonstrated early gains in reading and oral language outcomes in students whose teachers had been exposed to a targeted intervention designed by educators and speech-language pathologists.18 34 OLSEL draws on the theoretical framework developed by Munro,34 with content broadly consistent with the so-called ‘five big ideas’ of early literacy instruction: phonics-based instruction, phonemic awareness, vocabulary development, comprehension at the sentence, paragraph and topic level and fluency35 also including narrative skills and syntactic complexity.

This paper reports the research protocol for the Classroom Promotion of Oral Language (CPOL) trial. Based on OLSEL,18 it aims to advance the early oral language and literacy skills of students considered to be at-risk for low educational attainment, by improving teacher oral language knowledge and practice in their work with students in the first 2 years of school. The primary hypothesis is that by grade 3, students in the intervention group will have significantly improved reading achievement, when compared with students who experienced usual teaching practice. Secondary hypotheses are that by grade 1 students in the intervention group (compared with usual practice) will have improved outcomes in (1) oral language, (2) early literacy and (3) mental health, by grade 3 they will have improved (4) numeracy, (5) literacy and (6) writing skills, and that intervention teachers will have improved knowledge and practice in classroom-based oral language teaching strategies.

This large-scale cluster RCT will help to generate an evidence-base that can inform high-quality early years teaching and learning in schools, addressing early learning inequalities that can persist across the life span.

Methods and analysis

Study design

CPOL is a cluster RCT of a teacher-led whole of classroom oral language promotion intervention, compared with usual teaching practice. Due to the nature of the oral language teacher professional learning intervention at the whole of class level, randomisation within CPOL is at the school level. The primary outcome is reading ability of the students at grade 3. The components of the trial are summarised in table 1.

Table 1

Graphical depiction (‘Perera diagram’) of the components of the trial shared and unique to the intervention and control groups

Trial component	Intervention	Control
Call for expressions of interest
Briefings
Informed consent
Enrolment and baseline data collection
School randomisation
Professional learning
Teacher data collection: time point 1—end of foundation
Professional learning
Teacher data collection: time point 2—end of grade 1
Student data collection: time point 2—end of grade 1
Final student data collection: time point 4—middle of grade 3
	Schools that meet the inclusion criteria for the trial are emailed by the relevant education department inviting them to participate in the study
	Schools interested in participating are invited to one of two face-to-face briefings to hear more about the commitments and process of the trial
	One foundation class is selected from each participating school and a parent letter and consent form is sent home with the student
	Baseline data are collected for every consented child and for the relevant teacher of the class
	Professional learning and support: all foundation, grade 1, grade 1 / 2 composite teachers and leadership from the intervention schools attend the face-to-face professional learning sessions (days 1, 2, 3), access the online resources/forum and participate in school visits and email/phone support from CPOL support workers
	A teacher survey and two teacher audio-recordings are completed electronically and submitted online (secondary outcomes)
	Professional learning and support: all foundation, grade 1, grade 1 / 2 composite teachers and leadership from the intervention schools attend day 4 of the face-to-face professional learning, continue to access the online resources/forum and participate in school visits and email/phone support from CPOL support workers
	A teacher survey and two teacher audio-recordings are completed electronically and submitted online
	Face-to-face follow-up to assess individual student early literacy and language Electronic and paper- based collection of teacher and parent report of mental health
	Students complete grade 3 NAPLAN (primary and secondary outcomes)

CPOL, Classroom Promotion of Oral Language; NAPLAN, National Assessment Program Literacy and Numeracy.

Ethics and trial registration

This trial was granted ethics approval by the Monash University Human Research Ethics Committee on 15 November 2013 (#CF13/2634-2013001403); this was later transferred to the University of Melbourne on 7 October 2015 (#1545540). This trial was registered on 22 January 2014 (ISRCTN77681972).

Setting

This is a multisite trial being conducted in the state of Victoria, Australia. The Victorian school year typically runs from January to December. Participating primary schools are within a geographic radius of approximately 80 km from the centre of the state capital, Melbourne. The schools are from the Victorian Government Department of Education and Training (DET) and the Catholic Education Commission of Victoria (CECV) (22.07% and 67.62% of all primary school students in Victoria, respectively).36 The intervention consists of face-to-face professional learning for teachers, as well as ongoing support via trained teachers and speech-language pathologists. The face-to-face professional learning component of the intervention will be delivered from four venues across metropolitan Melbourne at each time point. The support worker element of the intervention will be delivered in the schools as well as via telephone and online support.

Participants and recruitment

Eligible schools will be those who respond positively to expression of interest invitations and meet the following eligibility criteria:

≥10% of students identified as developmentally vulnerable in the language and cognition skills domain of the 2009 and/or 2012 Australian Early Development Census;ii

Minimum of 15 students in a foundation cohort in the year prior to the start of the trial.

First round expression of interest invitations will be sent to schools in each sector located within approximately 80 km of the Melbourne city centre. A second round of expression of interest invitations will follow if the required sample size is not met. If more schools respond to the expression of interest invitation than are needed, then schools will be randomly selected to participate from each sector.

Class selection

Once a school has agreed to be in the study, one class will be randomly selected as the index class by the project coordinator. Data will only be collected from teachers and students in the index class, however every teacher responsible for a foundation, grade 1 or grade 1 / 2 composite class in schools randomised to the intervention arm of the study will be invited to attend the professional learning sessions and will have access to implementation support via the support workers.

Index classes will be selected using the following eligibility criteria:

Where only one foundation class exists in the school, that class will automatically become the index class.

Where multiple foundation classes exist, the index class will be randomly selected.

Where only composite foundation/grade 1 classes exist in the school, the index class will be selected from the composite foundation/grade 1 classes. Two classes will be combined where necessary to ensure an adequate number of foundation students are recruited for that cluster (school).

Where foundation classes and foundation/grade 1 composite classes exist in the school, the composite classes will be excluded and a class will be randomly selected from the foundation classes.

Student recruitment

Once the index class has been identified, informed consent will be sought from a parent/guardian of the students belonging to the index classes in the form of a hard copy letter sent home via the classroom teacher. Parent Information Statements (PIS) will be translated into relevant languages as required. PIS will include both informed passive consent (opt out) for CECV schools and a combination of informed active (opt in) and passive consent in DET schools. This combination will aim to minimise recruitment bias while subscribing to the relevant consent policies for each sector.

Randomisation

Schools will be randomly assigned in a 1:1 ratio after baseline data collection to receive the intervention (teacher professional learning days, online components and implementation support) or to the control arm which carries out business as usual in the classroom. Computer-generated block-randomisation will be used, with variable block sizes, stratified by school sector (CECV and DET).

Intervention

All teachers of foundation and grade 1 classes in schools randomised to the intervention arm of the study will be invited to attend four days of face-to-face professional learning convened by the research team (table 2). Like OLSEL, the professional learning is based on Munro’s34 ICPALER (Ideas, Conventions, Purposes, Ability to Learn, Expressive and Receptive Language) framework. ICPALER provides an explicit conceptual and pedagogical framework that teachers can use to consider their students’ language learning ability, the purposes for language use and the underlying receptive and expressive language skills (eg, phonological, morphological, semantic and discourse levels) that a child has mastered. Teachers are then able to explore specific classroom teaching strategies to scaffold students’ acquisition of more sophisticated skills. Four language domains from ICPALER are especially targeted in the teacher professional development: phonemic and phonological awareness, vocabulary knowledge, knowledge and application of narrative structure and comprehension of longer and more syntactically complex sentences.

Table 2

Overview of the four face-to-face professional learning sessions

Session	Content summary	Format
Day 1	Introduction to the need for oral language promotion in the early years’ classroom. Detailed overview of the ICPALER framework for describing and teaching language Discussion of the material provided Instructions for between-unit activities (tasks to be completed in schools before next session)	Facilitated discussion Video footage including teaching examples Table and whole group discussion
Day 2	Day 1 refresher and between-unit activity feedback Assessing and profiling for oral language Using ICPALER to plan and implement classroom-based speaking and listening teaching School planning for implementation Instructions for between-unit activities	Facilitated discussion/activities Modelled use of two assessment tools Practice use of a screening tool Facilitated activities Small group planning
Day 3	Day 2 refresher and between-unit activity feedback Assessing and teaching the ‘four language elements’: phonological and phonemic awareness, vocabulary development, developing and using complex sentences and story grammar. School planning for implementation Instructions for day 4 school presentation	Facilitated discussion and activities Small group planning
Day 4	School presentations of implementation School planning for sustained implementation	Peer-to-peer learning Small group planning

*Online professional learning.

ICPALER, Ideas, Conventions, Purposes, Ability to Learn, Expressive and Receptive Language.

Supplementing the formal days of professional learning, teachers will have the opportunity to participate in a self-directed online learning network of teachers from like-schools, and they will liaise with CPOL support workers via intermittent face-to-face, telephone and online contact, in order that questions are addressed and programme fidelity is enhanced.

1. Face-to-face professional learning days

Teachers will attend four face-to-face professional learning days. Two facilitators and a support worker will deliver the professional learning content.

The first three professional learning days will be held 6–8 weeks apart beginning in May of the year the students are in foundation. The final day will be held in February of the following year when the students are in grade 1. Table 2 outlines the content of the four face-to-face sessions.

2. Online professional learning

The online component of the intervention will be available for the duration of the 2-year intervention period. It comprises a secure website which will be accessible only to intervention teachers and will include:

Relevant documents/professional learning notes and teaching resources available for download by teachers;

Additional video footage for use during between-unit activities;

Simple discussion threads in relation to between-unit activities and general support for professional learning days;

Frequently asked questions.

3. CPOL support workers

The CPOL professional learning will be reinforced by the provision of two CPOL support workers (with either an education or speech pathology professional background) each working 1 day per week for the 2-year intervention phase of the trial. The inclusion of support workers was based on feedback from the implementation of OLSEL18 and previous work suggesting that ongoing and collaborative professional development is important for teachers implementing practice change.37 These workers will provide ongoing support to the participating schools including face-to-face, online and telephone communications. Face-to-face school visits will be scheduled throughout the 2-year intervention phase with the goal that every school be visited at least once (in addition to the online and telephone support).

Support workers will adopt a responsive and flexible approach, largely driven by individual school need. They will schedule their visits according to the self-identified learning needs of the early years team, at times that are convenient to that team. It is anticipated that teachers will integrate the visits into the school’s professional learning team meetings. They will provide assistance to intervention school teachers and leaders in-between professional learning sessions and after the fourth professional learning session in making and maintaining changes to classroom practice. This may include refreshing content from the intervention, assisting with team planning, modelling the assessment or teaching strategies described in the professional learning sessions and/or addressing concerns teachers experience throughout implementation. CPOL support workers will also use a private online forum to facilitate question and answer sessions and moderate learner-generated discussion.

Control arm

The schools in the control arm will conduct teaching as usual in the classroom. After the intervention phase of the study is complete, the opportunity to participate in a 1-day workshop will be offered to control schools. The focus and content of this workshop will be carefully tailored to be distinct from the intervention professional learning days and will focus on teaching strategies for current foundation and grade 1 students, rather than targeting the age group of the CPOL study cohort who will be in grade 2 by this time.

Blinding

The research staff (project coordinator and research assistant), CPOL support workers and intervention facilitators will be aware of the allocation of participating schools. All assessments with students will be conducted by researchers blinded to the schools’ randomisation allocation. Schools will be asked not to disclose their trial arm allocation. All investigators, including the study statistician, will be blind to school allocation for the duration of the trial.

Measures

The following measures will be used. A summary of the data collection schedule is presented in table 3.

Table 3

CPOL measures and data collection schedule

	Time point					Instrument
	Baseline: start of foundation	1. End of foundation	2. End of grade 1	3. Start of grade 2	4. Middle of grade 3
	Jan–Mar 2014	Oct–Dec 2014	Oct–Dec 2015	Jan–Mar 2016	May–June 2017
Measure
School demographics	X					Principal questionnaire: developed and administered by CPOL, collected via survey link
Student demographics	X					School census data: routinely collected by school staff and accessed by CPOL via linkage with the education departments
Teacher evaluation of intervention (process evaluation)		O	O			Evaluation surveys: paper-based form developed by CPOL and collected face-to-face at intervention days
Teacher, principal and literacy leader evaluation of intervention (process evaluation)				O		Semi-structured interviews and focus groups: conducted face-to-face by CPOL research assistant
Primary outcome
Reading scale score					X	NAPLAN: reading score: accessed via data linkage from VCAA
Secondary outcomes
Students
Writing, language and numeracy scale scores					X	NAPLAN: writing, language conventions and numeracy scores (see primary outcome above)
Mental health Parent report Teacher report	­ X O		­ O O			The Strengths and Difficulties Questionnaire (SDQ)44; parent report collected via paper-based form and teacher report via email link to online secure survey
Reading comprehension			O			Reading Progress Test48
Language Receptive language Receptive vocabulary Expressive language			­ O O O			CELF 4: concepts and following directions51 NIH Toolbox Picture Vocabulary Test52 Renfrew Language Scales (4th ed) Bus Story Test 50 All administered by CPOL researchers, face-to-face in student’s school
Teachers
Teacher knowledge	O	O	O			Teacher survey: developed by CPOL using a number of published surveys and administered by CPOL via email link to online secure survey
Teacher practice		O	O			Teacher audio-recordings: teaching samples recorded by teacher and submitted via email and DropBox

O, data collected by CPOL researchers; X, routinely collected by schools or education departments (CPOL to access via linkage or provided by project partners).

CELF, Clinical Evaluation of Language Fundamentals; CPOL, Classroom Promotion of Oral Language; NAPLAN, National Assessment Program Literacy and Numeracy; VCAA, Victorian Curriculum and Assessment Authority; NIH, National Institute of Health

Baseline measures (completed prior to randomisation)

School demographics

A brief principal questionnaire regarding school demographics will be distributed via email to all principals at baseline. The questionnaire will ask about staff and class numbers, potential prior exposure to a number of specific oral language initiatives and the types of speech pathology services accessed by the schools.

Student demographics

The study will use schools’ routinely collected information on student demographics. Details include student date of birth, gender, family language backgrounds and Aboriginal and Torres Strait Islander (ATSI) status. Schools will also be asked to provide information about which students are receiving disability support funding during the first 2 years of school (2014 and 2015).

The School Entrant Health Questionnaire

The School Entrant Health Questionnaire (SEHQ) is a parent-report questionnaire distributed and collected by DET school nurses for all students starting school (including those in CECV schools).38 It records parents’ concerns and observations about their child’s health and well-being. The SEHQ includes domains such as: general health, medications, immunisation status, dental health, speech/language, hearing, vision, disabilities, general development, behaviour and emotional well-being and family stress. The SEHQ also includes maternal and paternal highest level of education. The data will be provided to the study when available as an administrative data set.

The English Online Interview

The English Online Interview (EOI) is a teacher-completed measure of language and literacy.39 In DET schools, the EOI is routinely administered to all students entering foundation, but this is not the case for schools in the Catholic sector. In these schools, teachers will conduct the assessment via hard copy and the data will be entered into the secure database by a CPOL research assistant. A range of printed and online materials will be provided to support teachers to become familiar with and administer the EOI including access to the EOI homepage: http://www.education.vic.gov.au/school/teachers/teachingresources/discipline/english/assessment/Pages/default.aspx.

The EOI assesses students across the three modes of English in AusVELS (the Australian Curriculum in Victoria)—reading, writing and speaking and listening.40 For the purposes of this study, the reading, and speaking and listening sections of the EOI (teacher assessment and/or rating of aspects such as oral language and listening comprehension, phonemic awareness and phonics and concepts of print) will be used.

Primary outcome measure

Reading level of the students in grade 3

When the study cohort is in grade 3 we will access their National Assessment Program Literacy and Numeracy (NAPLAN) results (November 2017). NAPLAN is routinely collected in all schools in Australia and is assessed independently, external to the school. NAPLAN is collected when students are in grades 3, 5, 7 and 9 and comprises tests in four areas: reading, writing, language conventions and numeracy. Each test produces a raw score and a scale score (ranging from 0 to 1000). The reading scale score has been chosen as the primary outcome because of the well-established links between oral language competence and reading acquisition.5 The reading score will be used to measure the medium-term impact (>1 year post intervention) of CPOL on students’ reading ability.

Secondary outcome measures

Teacher knowledge

Teacher knowledge will be measured using the CPOL Teacher Survey, which will be sent to index teachers at baseline, end of foundation and at the end of grade 1. The survey collects demographic information about the teacher sample as well as containing a number of items related to teacher experience and practices, knowledge (eg, of language and language structures) and sources of their knowledge and skill. Teachers will also be asked to self-rate their level of confidence with respect to a range of language and reading instruction parameters.

The tool has been developed specifically for CPOL and is comprised predominantly of items drawn from previously published tools.25,41,42 Items on constructs considered relevant to the CPOL intervention that could not be sourced from published literature (eg, an item pertaining to teacher knowledge of narrative structure) were generated by investigators within the CPOL team. Many of these items were previously piloted in a teacher professional learning programme.43

Teacher practice

Index teachers will record a 10 min audio sample of their teaching during a common whole class lesson, for example, a ‘Big Book’ reading.iii Two samples will be requested at each time point (end of foundation and end of grade 1) and one recording per time point will be randomly selected for analysis. The decision to use audio samples (as opposed to direct observation or video samples) aims to reduce participant burden (promoting better completion rates) and minimise observer bias.

Students:

A number of secondary outcome measures will be collected when the students are in grade 1 and in grade 3. These will be analysed after primary outcome data collection.

Mental health

The Strengths and Difficulties Questionnaire (SDQ)44 is a behavioural screening questionnaire for those aged 3–16 years with 25 questions across five scales (emotional symptoms, conduct problems, hyperactivity/inattention, peer-relationship problems and prosocial behaviour). It has been used in many cohort and intervention studies to briefly assess child mental health difficulties.45–47 The SDQ will be completed at baseline, and at the end of grade 1, by both the classroom teacher and the parent.

Reading comprehension

The Reading Progress Test48 will be administered as a whole of class booklet-based literacy test. It will assess prereading and early reading skills including print concepts, word knowledge and comprehension. The group test will be administered by a blinded CPOL research assistant between October and December 2015, when the students are in grade 1. This is a validated tool and has Australian norms based on a national sample of students.49

Language

Expressive language (syntax and narrative)

The Renfrew Language Scales (4th ed) Bus Story Test50 will be used to elicit a narrative sample from the students. The assessment will be administered by a blinded CPOL research assistant between October and December 2015, when the students are in grade 1. It will be administered as per the Bus Story Test protocol, however the student narrative sample will be audio-recorded. The audio files will be transcribed verbatim and coded for narrative macrostructure (story grammar content) and microstructure (syntax) as per the OLSEL pilot RCT.18

Receptive language

The Concepts and Following Directions subtest (comprehension, recall and ability to act on spoken directions) from the Clinical Evaluation of Language Fundamentals-Fourth Edition (Australian standardisation)51 will assess the students’ receptive language. This assessment will be administered by a blinded CPOL research assistant between October and December 2015, when the students are in grade 1.

Receptive vocabulary

The National Institute of Health (NIH) Toolbox Picture Vocabulary Test (TPVT) will be used to measure receptive vocabulary. The TPVT52 was modified, with permission from NIH, to be delivered on an iPad in an Australian accent. This assessment will be administered by a blinded CPOL research assistant between October and December 2015, when the students are in grade 1.

Writing, language conventions and numeracy

In addition to using the students’ reading score from their grade 3 NAPLAN as the primary outcome, the NAPLAN writing, language conventions and numeracy scores will be used as secondary outcome measures.

Data collection procedures

Schedule

Table 3 outlines the measures and schedule for data collection. Many of the data sources used in the CPOL RCT capitalise on routinely collected data sets. The majority of the primary data collection conducted by CPOL researchers occurs when the students are at the end of grade 1, which coincides with the end of the intervention phase. These data will be collected by teams of three to five CPOL research assistants blinded to the schools’ randomisation allocation from October to December 2015.

Process evaluation

At the conclusion of the intervention phase of the trial, a process evaluation will be conducted to evaluate the extent to which the CPOL intervention was implemented as intended. The objectives of the process evaluation will be:

to evaluate the degree to which teachers and school leaders engaged and complied with the CPOL professional learning intervention;

to identify the facilitators and barriers teachers and/or schools faced in implementation;

to assess the extent to which CPOL strategies are judged to have been maintained in classrooms and built into the school curriculum and wider school environment.

The fidelity of the CPOL intervention will be investigated via a mixed-methods approach, using data collected throughout the RCT pertaining to attendance, support worker notes and observations, as well as in-depth interviews and focus groups with teachers and members of school leadership teams from the intervention arm of the study (see table 3 for details of data collection).

Economic evaluation

A cost-consequences analysis of the intervention compared with the control arm will be conducted from the government perspective, that is, it will include costs and outcomes relevant to government (but not those relevant only to individuals, such as additional out-of-pocket expenses). The economic evaluation will compare the incremental costs of the intervention (costs accrued in the intervention compared with costs accrued in the control group) to the measured primary and secondary outcomes, which are expressed in their natural units, such as point change in NAPLAN score. Costs will include the physical resources and staff time (of training staff and participating teachers) invested in providing all aspects of the intervention, recorded prospectively by the research team. Teacher report of all professional development activities over the 2-year intervention period will be used to assess whether CPOL is associated with a reduction (cost-saving) in other professional development activities. The uncertainty of cost and outcome data will be tested in a sensitivity analysis.

Data management and storage

All schools, teachers and students will be assigned unique numerical identifiers (an ID code) for use throughout the study. A single electronic, password protected, database in REDCap53 will record all participant details. This will be hosted on the Murdoch Children’s Research Institute server, which meets security and ethical confidentiality requirements. Members of the study team will have different levels of access depending on their role. Researchers will be able to access the details of participants where necessary but not their randomisation status unless necessary to that investigator. Participant questionnaire data will be identified by ID code only.

Written materials will be immediately scanned and saved within the study database. Paper versions of assessments or forms will be stored in a locked filing cabinet at the Royal Children’s Hospital and will be available only to the relevant research assistant. Aside from the initial consent forms (DET students), all further data collection material will be identified by unique number only, with no identifying information available.

Sample size and power calculations

The primary outcome of this study is the NAPLAN reading scores at grade 3.

The study is powered to identify a difference between the intervention and control groups in the reading scale score of 0.3 SD (equivalent to 22.98 points based on the mean scale score of 434.1 and a SD of 76.6).54 Given that the average gain in NAPLAN reading score over 2 years was approximately 80 points, this would represent a meaningful difference at the population level (equivalent to a 6-month difference in progress).

Randomisation of 561 students per arm is required to provide 90% power to detect a minimum difference of 0.3 SD on the NAPLAN reading scores at grade 3, allowing for an average intraclass correlation coefficient of 0.08 and an average cluster size of 17. Allowing for a potential attrition rate of 20% by the time students are in grade 3, 700 students per arm (1400 in total) will be required in the study.

Data analysis

Analysis will be conducted using the intention-to-treat (ITT) analysis principle, with students and teachers analysed in the study arm to which their school was randomly allocated.

As a sensitivity analysis, results will also be presented from a per-protocol (PP) analysis. The PP population will include students:

who complete foundation in the first year of the intervention and grade 1 in the second year of the intervention;

who have no more than 50 days of absenceiv in foundation55 56;

who have no more than 50 days of absence in grade 1;

whose NAPLAN reading score at grade 3 is available;

who have been exposed to the index teacher during foundation and to a teacher who attended the intervention during grade 1. If an index teacher leaves during foundation, his/her replacement must be an intervention teacher (intervention students only);

whose index teachers have been exposed to at least three of the four intervention days (intervention students only);

whose school has sent at least one teacher to all four intervention days, that is, school was represented at each session (intervention students only);

whose teachers did not work in any of the intervention schools during the 2-year intervention phase (control students only)

whose school has never employed a teacher who previously worked in an intervention school during the intervention phase (control students only).

All data analyses will be conducted using the Stata software package.57

Statistical analysis plan

The baseline characteristics of the students, schools and the teachers will be summarised by group. Categorical variables will be presented as the number and proportion in each category. Continuous variables will be presented as means and SDs, or medians, ranges and IQRs for non-normally distributed data.

Primary outcome analysis

The primary outcome, the NAPLAN reading score at grade 3, will be summarised by study arm as a mean and SD. The mean score will be compared between the groups using a two-level random effects linear regression model. This model will include a random effect for school, a fixed effect for intervention indicator and for type of school (CECV or DET) and a random effect for the interaction between intervention and school. Results will be reported as a mean difference between groups together with a 95% CI and p value.

A secondary analysis will include fixed effects for each of the following factors: student’s gender and age, family SES, ATSI status, student’s language background other than English (LBOTE), student’s special needs (whether or not receiving disability support funding during the intervention phase of the study) and student’s mental health at baseline as potentially important confounding variables, and will again be reported as a mean difference between groups, 95% CI and p value.

Secondary outcome analyses

NAPLAN writing, language conventions and numeracy scores at grade 3 will be summarised by study arm and will be separately analysed using the same two-level random effects linear regression model as for the primary outcome. Unadjusted analyses of the outcomes NAPLAN writing, language conventions and numeracy scores as well as analyses adjusted for student’s gender and age, family SES, ATSI status, student’s LBOTE, student’s special needs (whether or not receiving disability support funding during the intervention phase of the study) and student’s mental health at baseline will be reported.

Secondary outcomes at the teacher and school level will be summarised within each group, and compared between the groups using linear (continuous outcomes) and logistic (binary outcomes) regression models adjusted for type of school. All regression analyses will take account of any effects of the school (clustering), so that accurate effects of the intervention, regardless of the school, are estimated.

Handling of missing values

Prior to analysis, the available data on the primary and secondary outcomes will be explored. If there is a reasonable amount of missing data (>5%) and the summaries suggest that data may be missing depending on the characteristics of the participants, multiple imputation will be used to handle the missing data. In this case, a single imputation model will be used to impute all of the missing outcomes, using baseline characteristics as auxiliary models. Analysis will be repeated using a complete case analysis for comparison. If there is little missing data, complete case analysis will be presented as the primary analysis. The amount of missing data is not the sole criterion by which the missing data problem will be assessed; the missing data mechanisms and the missing data patterns will also be investigated and reported.

Interim analyses

At the end of the intervention phase (end of grade 1), all teacher outcome measures available will be analysed. The study statistician will remain blinded to the randomisation allocation of students by keeping links between teacher ID, school ID and student ID separate during the interim analysis. The interim analysis will include the following teacher outcome measures:

Baseline: teacher knowledge survey

End of foundation: teacher knowledge survey, teacher practice recordings

End of grade 1: teacher knowledge survey, teacher practice recordings, teacher evaluation of intervention (process evaluation)

Ethics and dissemination

This trial was approved by the Monash University Human Research Ethics Committee (#CF13/2634-2013001403) and later transferred to the University of Melbourne (#1545540). Translation of study results will be facilitated by having CECV and DET, who are responsible for the regulation and funding of Victorian schools, as partners in this trial. Findings from this trial will be of national and international significance in health and education sectors. All investigators have extensive national and international research and policymaker networks. This will ensure academic and policy impact via national and international health, education and early childhood development conferences, academic journals, publications targeting practising teachers, targeted use of social and electronic news media and inclusion in strategic policy forums such as national ministerial and senior officer councils (eg, Standing Council on School Education and Early Childhood). The findings from this trial will be reported according to the Consolidated Standards of Reporting Trials (CONSORT) Statement guidelines.58

Discussion

The CPOL trial is an education-based, rigorous evaluation of an oral language teacher professional learning intervention, an area of research in which there are relatively few gold standard trials. It addresses the crucial need to enhance literacy achievements in the early years of schooling by ensuring a rich oral language classroom environment. As a classroom-level, teacher-led intervention, CPOL takes a population approach to improve oral language competencies for all students. However, it targets schools where student’s oral language skills may be impoverished relative to the demands of the early years classroom. The trial’s efficiencies lie in its data collection design, capitalising on existing data sets routinely collected by schools for submission to their education departments. Partnering with CECV and DET will facilitate leveraging of these data for this intervention and tests a potentially replicable approach for future RCTs in schools.

The CPOL trial responds to identified evidence gaps in classroom-level oral language interventions, and in RCTs testing the effectiveness of teacher professional learning on student outcomes over the short to medium term. This trial will therefore be proof of concept and further confirmation that large-scale RCTs evaluating pedagogy can be efficient and robust in education. If effective, the following outcomes are expected:

The best available evidence that improving teacher knowledge and changing teacher practice regarding language and literacy can lead to student oral language development and sustained literacy improvement in the first 3 years of formal schooling.

A cost-efficient and well-tested intervention that could be (a) delivered to students as routine high-quality teaching practice and (b) included in teacher preservice education, going some way to addressing the question of ‘what works?’ in early years language and literacy education.

Schools are highly susceptible to the adoption of poorly evidenced programme.4 While a number of studies have reported the impact of teacher professional development and learning as a before and after design,59–62 this is an undeveloped field of research.63 We will extend this research by potentially demonstrating immediate and medium-term retention of knowledge and change of practice, and by testing this impact compared with control teachers who are being exposed to regular professional development and to ad hoc access to web-based and other sources of information. Support workers who reinforce the professional learning are included as an adjunct intervention accelerator based on feedback from the OLSEL implementation and on work which suggests that schools and teachers benefit from additional ‘hands on’ support to reinforce and adapt learnings to their actual context.59 64

It is commonly acknowledged that a nexus should exist between teacher knowledge and practice and student outcomes, potentially mediated through teacher attitudes.65 Given the lack of evidence in this area to date, this study will make an important contribution to the education and health literature. Indeed, there is increasing interest in rigorous testing of professional learning interventions in Australia and internationally.63 66 Education remains one of the most powerful predictors and social determinants of adult health outcomes.4 In Australia, the converging health and education policy interests in state and federal governments67–69 would suggest it is timely to rigorously evaluate how schools as education platforms can effectively and equitably address important child development outcomes. This is of vital importance and relevance to health and education policy makers and researchers alike.