Protocol for the systematic review of research on professional learning to promote implementation of a multitiered system of support in education.

INTRODUCTION: A multitiered system of supports (MTSS) represents a widely adopted public health approach to education in the USA. Researchers agree professional learning is critical for educators to implement the critical components of MTSS; however, professional learning approaches vary in their designs and targeted outcomes. While researchers increasingly focus their inquiries on professional learning for MTSS, no systematic research review exists.
OBJECTIVES: The primary objectives for this mixed-methods review are to (1) understand how professional learning focused on MTSS has been operationalised (2) determine the impact of professional learning on educator (eg, knowledge) and implementation (eg, data-based decision-making processes) outcomes and (3) understand the contextual variables that influence professional learning in the USA. We aim to determine which elements of professional learning improve educators' capacity to implement MTSS. METHODS AND ANALYSIS: We will include studies that use quantitative and qualitative methods. PsycInfo, PubMed, CIHAHL and ERIC will be the primary research databases used to search for studies published from January 1997 to May 2018. We also will search the US Institute for Educational Sciences and Office of Special Education Programs websites, ProQuest, Google Scholar, Science Watch and MSN. Finally, we will search the proceedings of relevant conferences, examine the reference lists of studies that pass full screening and contact authors for additional work. Data extraction will include participant demographics, intervention details, study design, outcomes, analyses and key findings. We will conduct a quality assessment and analyse the data using effect size and thematic analyses. ETHICS AND DISSEMINATION: Institutional review board or ethics approval is not needed for this review of already published works. We will disseminate the findings through presentations at state, national and international conferences; presentations to stakeholders and agencies; publication in peer-reviewed journals; and posts to organisational and agency websites. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This mixed-methods review will include quantitative studies of professional learning focused on multitiered system of supports (MTSS) (a public health approach to education adopted by many schools in the USA) to synthesise reporting of effects on educator (eg, knowledge, skills) and implementation (eg, assessment and intervention practices across tiers) outcomes as well as qualitative studies to provide rich descriptions of professional learning design, delivery and context.

The design and methodology for the mixed-methods review are described following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols guidelines.

Knowledge from the synthesis will be compiled and reported to a variety of educational stakeholders to inform research, policy and practice regarding professional learning to support MTSS implementation.

Recent emergence of literature on professional learning and MTSS implementation may result in insufficient numbers of studies to conduct some planned analyses (eg, assessment of heterogeneity, subgroup and sensitivity analysis, publication bias).

Limiting the search to studies conducted in the USA may overlook possible research on professional learning and MTSS implementation from other nations.

Introduction

Rationale

Schools and districts across the USA are adopting multitiered systems of support (MTSS) to address the academic, behavioural and socioemotional needs of students.1 2 MTSS involves organising educators’ assessment, instruction and data-based decision-making practices into multiple tiers of services that increase in intensity.3–6 In other words, MTSS includes primary prevention as well as supplemental and tertiary intervention strategies designed to address the prevalence of academic, behaviour and mental health problems in the USA.7 8 This public health approach to improving the overall effectiveness of the educational system, and to identifying and treating students with needs in a variety of domains makes conceptual sense given the universal access to children and youth in school settings in the USA. In fact, US education legislation (eg, Every Student Succeeds Act of 2015; Individuals with Disabilities Education Improvement Act (IDEIA) of 2004) includes provisions that support school districts to implement MTSS to improve student outcomes.9 10

In addition to policy, MTSS also is empirically supported. Previous systematic reviews of MTSS in the USA included studies looking at implementation and student outcomes from response to intervention (RTI) and positive behavioural interventions and supports (PBIS) models—multi-tiered, public health models focused on academics and behaviour, respectively.11–13 Although questions remain regarding the extent to which causal links can be made, research supports the notion that MTSS implementation relates to improved student outcomes.

Professional learning and MTSS

Implementation of MTSS requires substantial changes in educators’ practices. Some researchers have questioned whether educators can implement the critical components (ie, assessment, instruction and data-based decision-making practices across tiers) of this public health approach with fidelity.14 15 On the other hand, proponents of MTSS have argued that school and district leaders must engage educators in ongoing and effective professional learning (ie, professional development) practices for sustainable implementation to occur.6 16–18 Despite advancements in professional learning and widespread recognition of the importance of educator learning for MTSS implementation,19 it is unclear how educators engage in professional learning focused on MTSS and how professional learning relates to implementation of the critical components with fidelity.

The literature indicates that effective professional learning requires leadership, sustained and focused collaboration, allocation of resources to support learning, systematic implementation, evidence-based learning designs to deliver content and the continuous use of data to monitor and refine professional learning efforts.19 20 There are numerous ways to facilitate professional learning (eg, professional learning communities, study groups, technical trainings, workshops, coaching), but the literature points to specific practices associated with positive results (eg, changes in knowledge, skills and dispositions; changes in practices). In a recent evidence synthesis, Muijs et al determined that the greatest outcomes for professional learning came from experiences in which teachers were asked to engage in problem-solving (a critical component of MTSS models) focused on student learning and engagement.4 21 These activities set the stage for further learning through presentation style formats and increased teachers’ willingness to be coached and receive feedback on skills and implementation of programming in the classroom.21 Other studies indicated that ongoing exposure to effectively designed and delivered content has been associated with implementation of new practices and improved student outcomes.22 23

Although a large body of literature on professional learning in education exists, until recently, much of the literature on professional learning relative to MTSS was conceptual rather than empirical.18 24 25 Research on professional learning and how it relates to educators’ capacity to implement MTSS only recently emerged. Reports from training and technical assistance projects designed to provide professional learning to schools and districts in the USA implementing MTSS indicated that their professional learning services related to improvements in educators’ beliefs,26 knowledge and skills,26–29 and implementation of the critical components of MTSS.28–30 Peer-reviewed journal articles provide evidence that professional learning related to increased educator beliefs,31 educator knowledge and perceived skills,32–34 as well as increased implementation of MTSS.17 33 35

Purpose

Extant research on professional learning focused on MTSS used different professional learning designs and methods to investigate impact. To date, we are unaware of any systematic reviews conducted on professional learning focused on MTSS in the USA. Thus, the proposed review is needed to synthesise research on approaches to professional learning, the effects of professional learning on educator outcomes and MTSS implementation, and variables that influence professional learning effectiveness. We conceptualised this project in October of 2017 and anticipate the project being completed by the summer of 2019.

The proposed mixed-methods systematic review will inform policy and practice in a number of ways. School districts and states within the USA dedicate substantial monetary and personnel resources to professional learning that often do not result in improvements in practice and student outcomes.36 Given the federal and state policies driving MTSS implementation in the USA,1 9 10 37 this systematic review will provide policy-makers and educators in the USA with information about professional learning practices that facilitate improved educator capacity and increased implementation of MTSS, and about contextual issues that influence the effectiveness of professional learning. It also will provide policy-makers and educators from other nations with findings that can inform deliberations about their professional learning focused on MTSS given their national context. Although MTSS emerged largely from the USA, other nations have begun exploring applications of this public health approach to their schools.

Objectives

Our primary goal for this mixed-methods systematic review is to synthesise research from the USA on professional learning focused on MTSS to inform efforts to build educators’ capacity to implement the model’s critical components with fidelity. Our specific objectives are to (1) understand different ways in which educators engaged in professional learning focused on MTSS, (2) determine the effects of different professional learning designs on educator and MTSS implementation outcomes and (3) understand the contextual issues and variables that influence professional learning and its effectiveness. The specific questions we will answer from the systematic review include:

How is professional learning focused on MTSS being conceptualised, designed, delivered and evaluated?

To what extent does professional learning improve:

Educators’ knowledge, attitudes, skills, aspirations and behaviours relative to MTSS?

Implementation of the critical components of MTSS?

What variables facilitate or hinder the delivery of professional learning and educator and implementation outcomes?

What are professional learning recipients’ and facilitators’ perspectives of and experiences with MTSS professional learning?

Methods

Eligibility criteria

The design and methodology for the mixed-methods review are described following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols guidelines (see online supplementary file 1).38 39

Population

School-level educators who worked in a K-12 setting in the USA and participated in professional learning for MTSS implementation will be included. Although professional learning occurs at other units of analysis (eg, district-level administrators), we will delimit our review to educators who work in elementary and secondary (middle and high) schools. School-level educators include school administrators, general and special education teachers, interventionists, content specialists (eg, reading specialists, behaviour specialists), student support service personnel (ie, school psychologists, guidance counsellors, school social workers, school nurses) and school leadership team members (ie, educators identified as on a leadership team tasked with facilitating MTSS implementation at their school). These educators will be included because they most commonly participate in professional learning for MTSS implementation and typically are responsible for facilitating and/or implementing practices associated with the model.

Intervention

Studies will be included if they describe and evaluate any professional learning intervention focused on MTSS that aims to improve educator or implementation outcomes. We will review studies with a variety of professional learning designs including, but not limited to, professional learning communities, trainings, workshops, job-embedded coaching and online modules. Educator outcomes will include knowledge, attitudes, skills, aspirations and behaviours. Implementation outcomes will include assessment, instruction and intervention, and problem-solving practices at the individual student, small group, classroom, grade level or school level.

Comparators

Studies will be eligible for inclusion in the review whether or not they include a comparison group.

Outcomes

Selected studies should report outcomes relative to educators’ capacity to implement MTSS or their implementation of the critical components of MTSS. Educator outcomes will include:

Knowledge and conceptual understanding of the theoretical foundations of MTSS, how instruction and intervention are organised in an MTSS, and/or the elements of data-based problem-solving.

Attitudes, beliefs and/or values regarding the critical components of MTSS.

Perceived or demonstrated skills in making data-based decisions, engaging in problem-solving or implementing evidence-based instruction and intervention with fidelity.

Aspirations, desires, motivation and/or goals to implement practices associated with MTSS.

Behaviours in terms of how consistent they are with MTSS-related knowledge and skill application during and immediately following professional learning.

Outcomes associated with MTSS implementation will include:

Using assessment to inform instruction and intervention (eg, universal screening, progress monitoring use).

Implementation of tier 1 (primary prevention), tier 2 (supplemental intervention) and/or tier 3 (tertiary intervention) processes and/or procedures with fidelity.

Data-based problem-solving (problem identification, problem analysis, intervention development and implementation, programme evaluation/RTI) use to make decisions about the effectiveness of and modification to instruction and intervention across tiers.

Establishment and maintenance of school-wide processes and procedures for MTSS.

Study designs

We will include a variety of study designs in our review including quantitative, qualitative and mixed-methods approaches. Quantitative designs will include experimental (eg, randomised control trials), quasi-experimental, causal comparative, correlational, single-subject and programme evaluation methodologies that involve professional learning focused on MTSS as an independent variable. We also will include qualitative and mixed-methods designs in the review. Conducting a mixed studies review will allow us to provide a thorough and rich description of the different approaches to professional learning focused on MTSS, the effects associated with professional learning, and variables that influence professional learning and its intended outcomes.

Time period

We do not expect to find literature regarding MTSS and professional learning prior to 1997. Despite foundational research on the components of MTSS in education and the existence of public health models in other fields, the precursors to MTSS (RTI, PBIS) did not emerge in the literature until the reauthorisations of IDEIA in 1997 and 2004. Therefore, we will investigate literature on MTSS from January 1997 to May 2018.

Setting

Settings will include traditional K-12 public schools in the USA. Studies will include schools that vary in terms of urbanicity (ie, urban, rural, suburban), state and region, size, student demographics (eg, race and ethnicity, socioeconomic status, disability status, English language learner status) and educator demographics (eg, years of experience, race, highest degree earned).

Exclusion criteria

We will exclude any empirical sources that do not include elementary or secondary educators (district administrators, non-instructional staff, family and community members, preschool educators, and postsecondary educators will be excluded) from traditional public schools (charter schools, centre schools, alternative schools, virtual schools, private schools and faith-based schools will be excluded) located in the USA. Studies that explore consultation without an explicit emphasis on professional learning, as well as studies that focus on implementation of processes and procedures that may be part of an MTSS (eg, performance feedback, effective classroom management, instructional routines, intervention programmes), but are not part of a broader effort to implement the model also will be excluded. Finally, we will exclude non-empirical sources (eg, editorials, conceptual articles) and empirical studies published in languages other than English.

Information sources

We will use a combination of research databases and internet search engines to search for and identify relevant works. We will use PsycINFO, PubMed, CIHAHL and ERIC as the primary research databases to identify works published from January 1997 to May 2018. Research librarians at the University of South Florida main and Shimberg Libraries who specialise in systematic research syntheses will assist us with developing the search strategy including search terms (eg, MTSS, RTI, PBIS, professional learning) to find relevant works. We also will search the US Institute for Educational Sciences and Office of Special Education Programs websites for studies. We will search ProQuest for Dissertations and Theses and use Google Scholar, Science Watch and MSN to identify research articles and book chapters focused on our topic. Additionally, we will use the Google search engine to identify studies and reports conducted by centres or projects focused on training and technical assistance for MTSS and will search the proceedings of relevant conferences (eg, American Psychological Association, National Association of School Psychologists, Association of Positive Behavior Intervention Supports). Finally, we will examine the reference lists of studies that pass full screening and contact their authors for additional published and unpublished work. To validate our search strategy, we will conduct a hand search of the table of contents of 5 prominent journals (eg, Journal of School Psychology, School Psychology Review, Journal of Positive Behavior Intervention and Supports). The hand search will be of a random sample of 10% of issues published between January 1997 and May 2018.

Pilot search strategy

Members of the research team, in consultation with a research librarian, used the PsycINFO, PubMed, CIHAHL and ERIC databases to pilot our search strategy (see online supplementary file 2) resulting in 3875 unduplicated articles. The four main components of the search were intervention focus (MTSS model or components), intervention type (professional learning approach), population (educator job titles or roles) and research design. For each component, we selected keywords commonly found in research on MTSS, and more broadly in research on professional learning and education. Our search strategies will be further revised as we identify new components and/or keywords.

Patient and public involvement

The aims of this systematic review and the outcomes to evaluate were informed by our collective experiences designing, delivering and evaluating professional learning for MTSS in public school settings in the USA. Our experiences included collaborating with school-level, district-level and state-level stakeholders who delivered as well as participated in professional learning focused on MTSS. Participants provided written and oral feedback on their learning and on their satisfaction with the experiences that informed the conceptualisation and design of this systematic review.

Study records

Data management

Electronic search results will be downloaded into DistillerSR and duplicates will be removed.40 DistillerSR will be used for all stages of the systematic review including abstract screening, full-text screening and data extraction.

Screening and selection process

The screening process will proceed through three steps: abstract screen, full-text screen, extraction and extraction of effect size (if applicable). After a training and practice process, team members will individually read the abstracts of each paper and determine whether they meet eligibility based on the inclusion and exclusion criteria. Studies that pass the abstract screen will move to full-text screen to confirm the study’s inclusion. Any uncertainties will be discussed and resolved during biweekly team meetings or consultations with the principal investigators (PIs). The study will then move to extraction using the extraction tool, with two reviewers conducting the extraction independently and meeting to resolve differences. Two members of the review team with expertise in meta-analysis will be responsible for extracting the data necessary to calculate effect sizes for identified domains and outcomes. They will use comprehensive meta-analysis and MPlus for all effect size calculations and statistical analyses (eg, sensitivity analyses, assessment of heterogeneity and publication bias). All review team members will meet on a biweekly basis to discuss the screening process and any unanticipated issues.

Data items and data extraction process

We will use a common extraction tool for all studies, with variations depending on the research design. Two research team members will independently extract the data. The extraction will include basic study information (eg, author, publication source, year), participant and school demographics, intervention details (eg, professional learning design, procedures), study design, outcomes investigated, analyses used, and key findings. The two extractors will meet to resolve discrepancies and any remaining differences will be resolved by a third member of the review team (one of the PIs). As part of the extraction process, we will assess each quantitative and qualitative study for methodological rigour (see table 1).

Table 1

Sample elements for data extraction for studies on professional learning focused on MTSS

Study methods	PL conceptualisation	Population and setting	Outcomes
Study design Participants Intervention(s) Comparators Data collection Data analysis Quality appraisal	PL model or approach PL characteristics Focus Learning design Frequency Duration Evaluation	School-level educators and demographics Administrators Teachers Special educators Interventionists Guidance counsellors School psychologists Social workers Coaches Elementary and secondary schools and demographics	Educator Knowledge Attitudes (beliefs, dispositions) Skills Aspirations (desires, motivation, goals) Behaviours Implementation Assessment Instruction/Intervention Problem-solving Systems and processes

MTSS, multi-tiered system of supports; PL, professional learning.

Quality assessment

All studies that meet eligibility criteria will be assessed for quality independently and in duplicate. For group quantitative designs, we will use a tool developed by Kmet et al that includes a 14-item checklist with items for random allocation, blinding and control of confounding variables.41 For single-case experimental designs, we will use the Single-Case Experimental Design Scale,42 which is composed of an 11-item checklist that includes items associated with design, measurement and analysis. For qualitative studies, we will use the 10-item checklist also developed by Kmet et al.41 Mixed-methods studies will be evaluated using an appropriate combination of checklists (eg, both the quantitative and qualitative checklists). Two independent reviewers will compare their ratings to come to consensus and remaining discrepancies will be resolved by another member of the review team (one of the PIs).

Data

Effect sizes

If we identify sufficient studies evaluating MTSS professional learning for teachers, we will perform meta-analysis for the following outcomes: educators’ (1) knowledge, (2) attitudes, (3) skills, (4) aspirations and (5) behaviours, as well as educators’ implementation of the (1) assessment, (2) instruction and intervention, and (3) data-based problem-solving components of MTSS. We also will perform meta-analysis for studies evaluating school-wide implementation of systems and processes. Where possible, we will calculate effect sizes as standardised mean differences (SMDs) for continuous outcome variables and ORs for dichotomous outcome variables. SMDs and ORs will be converted to SMD using appropriate formulas. Where it is not possible to calculate SMDs, we will calculate ORs, which measure the ratio for the odds of success in the intervention group relative to the odds of success in the comparison group. For correlational studies, we will calculate the correlation coefficient as the effect size (r). For single-subject designs, we will calculate SMDs for studies that include multiple cases (eg, multiple baseline design).43 We will consult with a statistician with expertise on single-subject designs regarding calculating effect size should we find any studies with single cases.

Assessment of heterogeneity

If we identify sufficient studies evaluating professional learning interventions focused on MTSS, we will assess heterogeneity. We will evaluate forest plots and calculate the I2 statistic, an estimate of the percentage of variability across studies that is due to heterogeneity rather than chance.44 Following standard conventions, if I2 is >50%, we will assess the source of heterogeneity. Sources of heterogeneity, for example, might be research design or professional learning type.

Subgroup and sensitivity analysis

If we identify sufficient studies on the impact of professional learning, we will perform subgroup analyses based on the FAIR domains: intervention characteristics (eg, whole group versus individual professional learning), outer characteristics (eg, district support, community resources), inner characteristics (eg, school climate, school structure, culture, school type) and characteristics of individuals (eg, years of teaching experience).45 We will also conduct a sensitivity analysis excluding cohort and correlational studies.

Publication bias

If sufficient studies are identified, we will assess for publication bias by constructing a funnel plot of all studies included in any analysis,46 as well as Egger’s regression test.47

Should there be an insufficient number of studies evaluating MTSS professional learning for teachers necessary to conduct a meta-analysis of effect sizes and subgroup and sensitivity analyses, we will summarise the findings of individual studies narratively, including tables or schemas, to convey the range, type and quality of available research.48

Qualitative data analysis

We will conduct a descriptive–interpretive analysis of all qualitative studies following modified procedures described by Timulak and Lucas et al.49 50 At least two qualitative researchers will (1) independently review the qualitative studies according to the research questions, (2) code and identify major themes related to perspectives on MTSS training, drafting an early conceptual framework, (3) meet and come to consensus for each study until all studies are resolved, revising the conceptual framework, (4) consolidate themes across all studies under common dimensions, finalising the conceptual framework, (5) present the final conceptual framework to the broader research team for consensus and (6) make any necessary revisions to finalise the conceptual framework.

Our experience with MTSS research leads us to believe much of the research literature will be quantitative. However, should our search yield insufficient quantitative studies to synthesise narratively, we will adopt the process of transforming quantitative into qualitative findings described by Piat et al.51 This transformation involves analysing the quantitative literature using the qualitative analysis described above.

Data synthesis

Findings of the quantitative and qualitative syntheses will be compared with the overall aim to inform a model of quality MTSS professional learning. Following procedures described by,38 we will build a model of quality MTSS professional learning that reflect educators’ views from the qualitative studies. We will then compare those recommendations to features of the interventions used in the quantitative literature to determine whether they were present and the extent to which interventions of higher quality (by the educators’ accounts) produced better outcomes.

Should we identify an insufficient number of quantitative sources, the qualitative data analysis process describe above will serve to synthesise the quantitative and qualitative data. The aim of this synthesis will be to build a model of quality MTSS professional learning, including facilitators and barriers identified in the quantitative and qualitative literature.

Dissemination

We will disseminate the findings from this systematic review through a variety of outlets. First, we will present the findings to MTSS stakeholders and agencies who participate in or facilitate professional learning focused on MTSS. These collaborations will facilitate the research team communicating emerging findings and receiving feedback. We also will deliver presentations at state, national and international conferences that researchers and practitioners engaged in MTSS practices attend. Finally, we will submit the results of the systematic review for publication in peer-reviewed journals as well as post reports (eg, white papers, technical reports, research briefs) to organisational and agency websites.

Conclusion

Researchers hypothesise that professional learning plays a critical role in building educator capacity to implement the critical components of MTSS with fidelity. However, until recently, the literature on professional learning focused on MTSS largely has been conceptual. Given emerging literature on professional learning for MTSS in the USA, our goal is to conduct a systematic research synthesis to inform quality professional learning for school-level educators implementing MTSS. We aim to establish the current knowledge base regarding how professional learning is being delivered in the USA, the impact it has on educator and implementation outcomes, and issues that influence its delivery and the associated outcomes. Our results will be disseminated to a variety of stakeholders interested in MTSS implementation to inform both the research and practice of professional learning focused on the public health approach to education.