Study protocol for a cluster-randomized trial of a bundle of implementation support strategies to improve the fidelity of implementation of schoolwide Positive Behavioral Interventions and Supports in rural schools.

Background: Improving the implementation of evidence-based interventions is important for population-level impacts. Positive Behavioral Interventions and Supports (PBIS) is effective for improving school climate and students' behavioral outcomes, but rural schools often lag behind urban and suburban schools in implementing such initiatives. Methods/Design: This paper describes a Type 3 hybrid implementation-effectiveness trial of Rural School Support Strategies (RS3), a bundle of implementation support strategies selected to improve implementation outcomes in rural schools. In this two-arm parallel group trial, 40 rural public schools are randomized to receive: 1) a series of trainings about PBIS; or 2) an enhanced condition with training plus RS3. The trial was planned for two years, but due to the pandemic has been extended another year. RS3 draws from the Interactive Systems Framework, with a university-based team (support system) that works with a team at each school (school-based delivery system), increasing engagement through strategies such as: providing technical assistance, facilitating school team functioning, and educating implementers. The primary organizational-level outcome is fidelity of implementation, with additional implementation outcomes of feasibility, acceptability, appropriateness, and cost. Staff-level outcomes include perceived climate and self-reported adoption of PBIS core components. Student-level outcomes include disciplinary referrals, academic achievement, and perceived climate. Mediators being evaluated include organizational readiness, school team functioning, and psychological safety. Discussion: The study tests implementation strategies, with strengths including a theory-based design, mixed methods data collection, and consideration of mediational mechanisms. Results will yield knowledge about how to improve implementation of universal prevention initiatives in rural schools.

Background and rationale

Research in K-12 schools has documented the importance of the institutional environment on students’ learning as well as their health and safety [1]. With children and adolescents spending typically 180 days of the year at school, the social environment in schools acutely impacts students on a daily basis and influences longer-term child and youth development. Evidence-based prevention initiatives can improve school environments and enhance well-being on a population level, if implemented with fidelity. Efforts to create nurturing school environments and prevent problem behaviors are often conceptualized within a three-tiered public health approach for reducing risk factors and increasing protective factors [[2], [3], [4]], with the aim of preventing the progression of problem behaviors, physical or mental health issues, and academic underperformance. At the first tier, efforts involve supporting all students and are often focused on changing the schoolwide environment, commonly described as universal prevention interventions. At the secondary and tertiary tiers, efforts focus on students for whom additional supports are needed..

With regard to student behavior while in school, Positive Behavioral Interventions and Supports (PBIS) [5] is a commonly-used and evidence-based approach that involves establishing a tiered system of supports. At the first (universal) tier, schoolwide efforts include the establishment of disciplinary and classroom management systems, and staff and teacher adoption of practices that prevent problem behaviors among students. Core components at this schoolwide tier include: defining and teaching a small set of behavioral expectations; establishing acknowledgment systems for students when they engage in desired behaviors; establishing predictable consequences when students engage in problem behaviors; and using data-based decision-making. PBIS does not specify what the behavioral expectations should be, nor how they are taught; however, the common feature of schoolwide PBIS fidelity is for schools to establish such expectations and procedures to support them. At the secondary tier, targeted interventions address the needs of students at risk of academic challenges, and/or those with specific behavior needs. Lastly, at the tertiary tier, interventions are individualized to provide specific support to students with more complex emotional and behavioral challenges.

The benefits of PBIS have been well-established through randomized controlled effectiveness trials, documenting reductions in behavior problems and bullying, and improved social-emotional functioning, prosocial behavior, and academic outcomes [[6], [7], [8], [9]]. Estimates suggest that as of 2018, more than 25,000 schools across the United States were using PBIS [10], representing nearly 20% of the 130,000 public and private K-12 schools nationwide [11]. This level of scaling is impressive for any evidence-based intervention (EBI), and has been facilitated by coordinated systems leadership at national and state levels [[12], [13], [14]]. However, across the United States, PBIS is less-frequently used in rural schools relative to schools in urban and suburban areas [15], representing an area of need for implementation research.

Rural disparities in school climate, student outcomes, and implementation fidelity

In the United States, 57% of public school districts and 32% of public schools are located in rural areas [16], but while it is well-documented that rural schools face significant and unique challenges in implementing innovations [17], relatively little research has addressed these challenges by developing and testing strategies to support implementation of EBIs in rural schools. Rural schools often face challenges associated with location (e.g., geographic isolation, professional shortages); limited resources; perceptions such as distrust of outsiders or skepticism of services to be provided; and student characteristics such as poverty. Although the research that first documented these rural challenges is over three decades old [18], very little has changed in terms of the barriers that still impact implementation of innovations in rural schools. While PBIS has been documented as an effective intervention, most studies of PBIS have been conducted in urban settings [17]. Thus far, few projects have studied the scaling of PBIS in rural areas, with none using a fully-powered randomized trial to study the effectiveness of implementation support strategies. Furthermore, there is a need to identify implementation support strategies that have promise for scaling other types of prevention interventions (e.g., social-emotional curricula, health promotion programs, substance use prevention programs) in rural schools.

Approaches to improve EBI scaling

While the extent of PBIS implementation nationwide is substantial, it is also crucial that implementation occurs with fidelity—that is, the degree to which an intervention is delivered or implemented as intended by the developers [[19], [20], [21]]. As others have noted, when school-based prevention-focused EBIs are implemented at scale outside of a research-based trial, often the interventions are not implemented with fidelity [22], or fidelity is not monitored [23], leaving crucial questions as to whether the EBIs are actually being implemented as intended. A key issue for improving scale-up is to consider ways to help more schools attain implementation with fidelity. Doing so requires exploration of determinants of implementation outcomes, as well as consideration of specific barriers known to hinder EBI implementation.

Much evidence shows that for scale-up of evidence-based prevention programs with fidelity, attention must be paid to not only what is implemented, but how it is implemented [24,25]. In other words, “the process of implementation influences the product” [24]. An implementation science approach can improve understanding of methods to effectively scale evidence-based prevention practices in the real world [26]. Rigorous research studies are needed to examine questions about how to improve the fidelity of implementation of school-based prevention approaches such as PBIS, including examination of the feasibility, acceptability, and costs of those implementation supports. In addition, consideration of contextual factors that impact implementation processes and outcomes is needed, as is research on mechanisms through which such strategies may improve EBI implementation and, ultimately, student outcomes.

The gap in literature addressed by this study

This manuscript describes the protocol for a randomized controlled trial of Rural School Support Strategies (RS3), a bundle of implementation support strategies developed to support the implementation of evidence-based prevention initiatives—such as the PBIS tiered framework—in rural schools. A unique aspect of this bundle of strategies is that it was developed with consideration of feasibility and acceptability among rural schools, while maintaining a focus on ensuring cost-effectiveness for providing a support system to schools that are geographically remote and lack specialized staffing capacity and financial resources, as is often the case in rural communities.

Theoretical framework

A crucial element in the study of how to improve implementation is the use of well-developed theories, models, and frameworks [27]. This study of RS3 is based on the Interactive Systems Framework for Dissemination and Implementation (ISF) [28], which is well-aligned with the tiered continuum of supports that underlies the PBIS framework, due to ISF also having origins in the public health model. The ISF identifies three interactive systems that work to support implementation of evidence-based prevention practices: 1) the synthesis and translation system; 2) the delivery system; and 3) the support system [29]. In many prior studies of implementation support strategies—and particularly those that use the ISF to conceptualize how these three systems can work together—strategies such as training, technical assistance, and quality improvement activities are utilized [29,30]. The theoretical rationale for RS3 derives from the ISF, which specifies which actors and systems should provide implementation supports, and the related Quality Implementation Framework [31], which identifies phases for improving implementation, and actions to be taken by each of the three parts of the ISF system. The Quality Implementation Framework specifies four stages in the implementation process: 1) considering the host setting, including assessing context and building capacity; 2) creating a structure for implementation, including structural features such as teams; 3) providing ongoing structure as implementation begins and continues; and 4) improving future applications. Table 1 illustrates the 14 activities of the QIF within these 4 stages, aligned with specific activities of this project, actors, and measures to evaluate the process of implementation.

Table 1

Project activities, data collection, and alignment with guiding theoretical framework (the Quality Implementation Framework).

Phases and Activities of theQuality Implementation Framework	Project Activities (a intervention only)	Data Collection Activities to Assess Process Phase/Activity
Phase One: Initial considerations regarding the host setting
Assessment strategies1. Assess needs and resources2. Assess intervention fit3. Assess organizational capacity/readiness	• collect baseline team survey and schoolwide staff surveys • conduct on-site visits to assess school environment and context	• baseline team survey (school assessment scale) • baseline staff survey (e.g., climate, organizational readiness, capacity)
Decisions about intervention adaptation4. Possibility for adaptation	• staff surveys • conduct on-site visits to assess school environment and context	• baseline staff survey (attitudes about PBIS, attitudes about evidence-based practices)
Capacity-building strategies
5. Obtain buy-in from crucial stakeholders; foster supportive organizational climate6. Build organizational capacity7. Recruit staff8. Provide effective pre-intervention staff training	• memorandum of understanding/commitment • kickoff institute, spring 2019a • coaching institute, spring 2019a	• baseline data collection onsite (ASSIST, SET) • baseline surveys (staff, students, parents)
Phase Two: Creating a structure for implementation
Structural features for implementation9. Create implementation teams10. Develop an implementation plan	• all schools establish PBIS team • complete PBIS action planning in Tier 1 training (summer 2019)	• team function survey • Team Implementation Checklist • coding of action plans
Phase Three: Ongoing structure once implementation begins
Ongoing implementation support strategies11. Technical assistance/coaching/supervision12. Process evaluation13. Supportive feedback mechanism	• ongoing TA/coaching/supervision by project team (including feedback/audit/monitoring/etc)a • feedback on climate surveys annually (all schools)	• project staff's tracking logs, notes, reflections on each TA meeting • interviews to assess perceived feasibility, acceptability, & appropriateness of support strategies
Phase Four: Improving future applications
14. Learn from experience	• reflect on process & refine implementation supports	• interviews post-intervention to identify lessons learned and assess school team perceptions

For schools in the enhanced condition only; activity is part of Rural School Support Strategies (RS3).

Rationale for the intervention: Rural School Support Strategies (RS3)

High-quality training is necessary for improving knowledge and skills, but alone it is often insufficient to enable implementation with fidelity. In this study, training is a strategy that is delivered to both conditions, with the addition of elements in the enhanced condition. Prior studies of efforts to improve fidelity of EBIs have demonstrated the benefits of comprehensive supports relative to training alone. As described below, RS3 includes a variety of strategies, particularly those articulated in the Quality Implementation Framework (i.e., technical assistance, coaching).

In 2017, the Expert Recommendations for Implementing Change project developed a taxonomy of implementation strategies and assessed evidence about the effectiveness of those types of strategies. After the initiation of the current project, that taxonomy was further refined into the SISTER taxonomy (School Implementation Strategies, Translating ERIC Resources); although it was not published at the time the current study was designed, to be consistent with recommendations for specifying and reporting implementation strategies [32], we name our strategies according to these taxonomies in Table 2 (further details in 3.3.2, Detailed Intervention Description). As noted above, these strategies are applied within the Interactive Systems Framework, which specifies which actors and systems should provide implementation supports, and the related Quality Implementation Framework, which identifies phases for improving implementation outcomes. Most of the implementation strategies in the current study occur in Phase Three of the Quality Implementation Framework—the ongoing structure once implementation begins—and are conducted by the project's implementation support practitioners [33]. In addition, RS3 also addresses Phase One considerations about readiness, and Phase Two of the implementation process, through establishing structures for implementation.

Table 2

Rural school support strategies (RS3): Support system strategies in alignment with SISTERa taxonomy.

SISTER #	Category/Strategy	SISTER Definition	This Project's Activities
	USE EVALUATIVE AND ITERATIVE STRATEGIES
1.	Assess for readiness and identify barriers and facilitators.	Assess school context, capacity, barriers to implementation, strengths or facilitators.	Baseline school team assessment, on-site observations, with feedback to schools prior to implementation.
2.	Audit and provide feedback.	Collect and summarize data for administrators and school personnel to monitor, evaluate, and support implementer behavior.	Establishment of data systems for disciplinary incidents, review of acknowledgment systems during monthly meetings with implementation support practitioners.
5.	Develop a detailed implementation plan or blueprint.	Develop a detailed implementation plan that includes goals/outcomes, process, actors, timeframe, strategies, and performance/progress measures.	Development of school team action plans; structured monthly visits to review school team meeting processes and adjust based on action plan and Team Implementation Checklist.
6.	Develop and organize quality monitoring system.*	Monitor implementation and/or student outcomes for quality improvement.	Both conditions receive paid subscription to SWIS data tracking. In the enhanced condition, implementation support practitioners regularly review data with school coaches.
9.	Monitor the progress of the implementation effort.*	Monitor key implementation outcomes (fidelity, reach, acceptability) and adjust for continuous improvement.	Both conditions complete the TFI in the summer. In the enhanced condition, implementation support practitioners regularly review TFI and action plan with school coaches.
	PROVIDE INTERACTIVE ASSISTANCE
11.	Centralize technical assistance.	Obtain technical assistance about implementation issues.	Monthly appointments with implementation support practitioner and school coach.
12.	Facilitation/problem-solving.	Interactive problem-solving and support in a non-evaluative but informative interpersonal relationship.	Monthly appointments with implementation support practitioner and school coach.
	ADAPT AND TAILOR TO CONTEXT
16.	Promote adaptability.	Identify how the innovation can be tailored or adapted to fit the school context; clarify which elements of PBIS must be maintained to preserve fidelity.	Monthly appointments with implementation support practitioner and school coach. Activities include reviewing climate data, implementation checklists, readiness, etc to identify opportunities to refine key components of PBIS.
19.	Use data experts.	Involve experts to use data generated by implementation efforts.	Implementation support practitioners review SWIS data during TA visits and virtual learning sessions.
	DEVELOP STAKEHOLDER INTERRELATIONSHIPS
22.	Capture and share local knowledge.	Capture knowledge from other schools about implementation successes, and share with other sites.	Partnered activities and small-group sharing in breakout rooms during monthly virtual learning sessions.
32.	Organize school personnel implementation team meetings.*	Develop teams of personnel with protected time to reflect on the implementation process.	School implementation teams are a key component of PBIS and are established in both conditions. Enhanced condition includes support from implementation support about ensuring active team meetings with effective teamwork.
34.	Recruit, designate, and train for leadership.	Recruit, designate, and train leaders to engage in behaviors that support others to adopt the innovation.	Kickoff meeting with school administrators and coaches to discuss the importance of leadership in promoting implementation.
	TRAIN AND EDUCATE STAKEHOLDERS
38.	Conduct educational outreach visits.	Have a person with extensive experience implementing the practice meet with school personnel. Educate them about the innovation with the intent to change school practices.	Onsite visits and monthly TA visits by implementation support practitioners.
39.	Conduct ongoing training.*	Conduct ongoing trainings about the new practices.	Both conditions participate in training institutes, over the course of 3 summers.
41.	Develop educational materials.	Develop manuals, toolkits, and other materials in ways to help stakeholders learn about new practices.	Videos, tip sheets, and other resources on the web portal, with regular updates and notices to school coaches by email and newsletter to inform of resources available.
42.	Distribute educational materials.	Distribute educational materials (e.g., guidelines, toolkits).	Monthly newsletter, updates to web portal with resources.
43.	Make training dynamic.	Vary information delivery methods and activities so that training sessions are interactive.	Monthly virtual learning sessions structured to present material; engage discussion, use breakout rooms, polls; structured activities and reflections, chat check-ins, etc.
44.	Provide ongoing consultation/coaching.	Ongoing consultation/coaching by experts in the new practice.	Monthly eetings with implementation support practitioners.
	SUPPORT CLINICIANS (TEACHERS AND SCHOOL STAFF)
52.	Pre-correction prior to implementation.	Proactively reminding implementers about how to deliver key aspects of the innovation prior to delivery.	Development of tip sheets for implementation success; expert-guided action planning during summer team trainings; reminders from implementation support practitioners during monthly meetings particularly at beginning of school year.
54.	Targeting/improving implementer well-being.	Support school personnel to reduce stress and burnout; promote well-being.	Content of virtual learning sessions and educational materials (newsletters) highlighting self-care and educator well-being.
	ENGAGE CONSUMERS
55.	Increase demand and expectations for implementation.	Increase demand and expectations for the innovation by educating key stakeholders about the new practice and its outcomes.	Content of virtual learning session and educational materials (newsletters) addressing strategies for rolling out the intervention with school staff. Videos on web portal about this topic/strategy.
57.	Involve students, family members, and other staff.	Strategies to connect with students, families, and staff who may not directly be involved in delivering the innovation but are impacted by it.	Content of virtual learning session and educational materials (newsletters) highlights community/family involvement.

Note: * indicates that this strategy is used in both the comparison condition as well as the intervention (RS3) condition.

SISTER = School Implementation Strategies, Translating ERIC (Expert Recommendations for Implementing Change) Resources. Cook CR, Lyon AR, Locke J, Waltz T, Powell BJ. (2019). Adapting a compilation of implementation strategies to advance school-based implementation research and practice. Prevention Science, 20 (6), 914–935.

Structural Features for Implementation: Implementation Teams. While many EBIs do not specify how to approach implementation, the PBIS framework has a history of empirical study about implementation, and well-developed blueprints for implementation [11]. Teaming is an effective implementation strategy [34,35], and a key element of PBIS implementation includes the formation of a school-level PBIS leadership team to guide the process. For the current project, participating schools were required to establish a school leadership team, including the principal, and a team coach, who would ideally be a counselor or another staff member with expertise in student behavior. All schools in this study are using a teaming approach, therefore teaming is not unique to RS3, but the RS3 strategies do seek to improve the ways in which school teams function. Based on the ISF, the university-based support system (implementation support practitioners) work with the school-based team to facilitate implementation. Prior work has shown that how well teams function in the early stages of implementation is related to subsequent implementation outcomes, and technical assistance (TA) can improve team functioning [36]. Factors associated with positive results include leadership and culture among implementation teams, as well as the use of teaming strategies such as monitoring team structure, roles, and participation [37]. Thus, an element of RS3 is to improve school team functioning; as described subsequently, measures of team functioning are being collected as potential mediators of the impact of the RS3 intervention.

Ongoing ImplementationSupports. Ongoing and sustained TA plays a key role in improving fidelity of implementation of prevention programs, as well as creating subsequent improvements in student-level outcomes [24,25,38]. TA is generally conceptualized to include formal or informal consultation, support, reinforcement, and encouragement. TA is now recognized to be a key element of effective scale-up efforts [39], although questions remain about how much is needed and how it should be delivered [40]. As articulated in the ISF, the actors who provide these supports should be the “prevention support system.” Collaborative approaches between TA providers and organizational implementers yield better results [41], and effective TA is characterized by elements such as trusting and collaborative relationships and a strengths-based approach; however, more work is needed to elucidate how TA improves outcomes [40]. TA is a central element of RS3; despite evidence about the value of TA, much prior research has assessed in-person delivery, whereas few studies have explored approaches tailored to rural settings, with an emphasis on remote, virtually-delivered TA to support schools in remote locations or during adverse weather that makes onsite visits challenging. As noted in Table 2, ongoing monthly contacts by the implementation support practitioners engage a variety of the SISTER strategies, not only addressing technicalities of implementation, but also facilitating problem-solving, developing relationships, engaging stakeholders, collecting and evaluating data, and supporting adaptations for each school's context.

Project overview and aims

With this project's primary focus on the exploration of how to improve implementation fidelity in rural schools, an implementation effectiveness trial [42] was designed to test a comprehensive bundle of implementation supports as compared to a standard training-only condition. This hybrid type 3 trial focuses on the fidelity of implementation of PBIS in rural schools, with additional exploration of the feasibility, acceptability, appropriateness, and costs of RS3. The primary outcome is organizational (school)-level implementation fidelity. Additional outcomes include staff-level perceptions of school climate and self-reported implementation of behavior management strategies, and student-level outcomes include behavioral outcomes (office disciplinary referrals), academic outcomes, and perceived climate.

This multi-year parallel-arm study uses a cluster randomized design with 40 rural public K-12 schools in Idaho, which were randomized to one of two conditions: 1) a standard series of trainings about PBIS (basic condition), or 2) an enhanced condition that includes the standard training series plus the supports of RS3. The trial was planned to occur over two school years, but due to the COVID-19 pandemic is being continued for a third year to allow extended intervention and data collection time.

Specific aims of this project are:

To test whether standard training plus the supports of RS3 improve PBIS implementation fidelity relative to a training-only control condition.

To explore mediators of the effectiveness of RS3 on implementation fidelity, such as changes in school-level PBIS team functioning, organizational readiness, and psychological safety among school staff.

To examine whether student outcomes (office discipline referrals, academic achievement and perceived climate) differ for students attending schools randomized to RS3 as compared to students at schools in the control condition.

To test whether improvements in student outcomes occur through mediation by schoolwide fidelity of implementation of PBIS.

To assess the feasibility, acceptability, appropriateness, and costs of RS3.

Mediators: how does RS3 improve implementation outcomes?

There is a need for greater attention to mechanisms of action in implementation science, with mechanisms being the processes or events “through which an implementation strategy operates to affect desired implementation outcomes” [43]. Longitudinal and experimentally-manipulated studies such as randomized trials are ideal for testing such mechanisms, and we identified a-priori hypotheses about potential mechanisms of change. Following standards of evidence from the Society of Prevention Research [44], our project includes hypotheses about several causal mechanisms through which the enhanced supports may mediate implementation fidelity, as well as the collection of data about baseline community context, resources, and demographic characteristics as explanatory variables.

Decades of study in K-12 school environments have documented consistent facilitators of improvements in school practices: professional capacity (e.g., teachers’ knowledge, skills, and support); norms; parent-school-community connections; and guidance (e.g., curricula, policies) [10]. Several of these elements pertain to the outer/community context, as well as the proximal inner organizational context (i.e., culture, climate). These factors have also been noted by the developers of PBIS as determinants of implementation fidelity [[13], [14], [15]], scaling [[16], [17], [18], [19], [20]], and sustainability [16,[19], [20], [21], [22]]. In the current project, we explore how baseline contextual characteristics impact implementation fidelity.

Mediational mechanisms are the way that implementation strategies effect changes in implementation outcomes. One characteristic often considered to be responsible for implementation failures is a lack of organizational readiness—and the lack of attention to organizational readiness during implementation processes; prior work exploring the factors associated with successful scale-up efforts has shown that readiness for EBI implementation efforts in university-community partnerships is often low, and related to perceived lack of resource availability [45,46]. In the current work, we hypothesize that the engagement of support and delivery systems will increase perceived readiness and collective efficacy at schools that receive the RS3 supports. Furthermore, factors such as the quality of relationship between school coaches and teachers mediates the effectiveness of interventions to improve the fidelity of behavior management initiatives [47]. We hypothesize that several components of RS3—notably, the coaching and TA from the support system's implementation support practitioners—will improve PBIS fidelity through several mechanisms: improved school team functioning, increased psychological safety, and improvements in school staff attitudes about EBIs (see measures section).

Methods and measures

Data collection is occurring across four school years, with pre-implementation baseline in spring 2019, and follow-up across subsequent school years (2019–20, 2020–21, and 2021–22).

Setting: school eligibility, recruitment, and selection

This project is being conducted in Idaho, a predominantly rural state. Among the state's 115 regular school districts (i.e., not virtual, charter, correctional, etc.), 65% of districts serve rural areas or townships. The National Center for Education Statistics (NCES) urban-centric locale codes [48] were used to assess eligibility for this project: schools in rural areas (NCES codes 41, 42, and 43) were eligible, as were schools in townships (NCES codes 31, 32, and 33). While some schools (∼10%) in these locales are extremely small, serving fewer than 40 students, most serve greater numbers of students. Based on power calculations, we established an inclusion criterion of at least 100 students per school, yielding a total of 198 Idaho schools meeting that criteria; 42 were eliminated due to having received prior training in PBIS. Thus, there were 156 potential Idaho public K-12 schools from which to recruit. Because of the focus on school-level implementation fidelity, schools were not required to serve specific grade levels (i.e., elementary or high school only). This decision was also made due to the varying composition of rural communities, where some schools serve grade ranges that are less common in urban or suburban settings but tend to be prevalent in rural areas (e.g., K-12, K-8, 7–12, etc.). Informational packets and an invitation video were distributed by mail and email to principals of eligible schools, followed by outreach by project staff in fall 2018. A total of 40 schools were recruited; in addition, three elementary schools from one district applied after the deadline and were held on a waitlist.

Randomization

After the recruitment of schools in May–September 2018, randomization occurred in October 2018. The randomization procedure was overseen by an independent doctoral-level educational statistician who was not on the study team. First, demographic data were confirmed for each of the 40 schools that applied to participate, using the NCES Common Core of Data [48] to characterize school size (number of students), and poverty (% of students eligible for free/reduced-priced meals), and school level/grades served. Schools were blocked into pairs based on these demographics. Blocking also accounted for district membership, to avoid potential cross-contamination within district; therefore, districts with multiple schools participating were blocked before randomizing, matching on number of interested schools and randomly allocating one district (and schools therein) to the intervention condition and the other to control.

Once blocks were established, a coin was flipped to assign one school in each block to intervention, and the other to control. In addition to oversight by the independent statistician, the coin flipping procedure was observed by two independent researchers to verify the procedure and confirm the accurate recording of group assignment. One district with three schools was assigned to the intervention condition, but before schools were notified of assignment, the superintendent decided to withdraw from the project. This district group was replaced by the waitlisted district group of three schools. This randomization procedure yielded two groups of schools with baseline demographic characteristics presented in Table 3. Chi-square and t-tests were conducted to test whether the two groups differed on baseline demographics; there were no significant differences between intervention and control schools (Table 3), suggesting similarity at baseline.

Table 3

Demographics of 40 participating schools.

	Schools Randomized to the RS3 Intervention (n = 20)			Schools Randomized to the Comparison Condition (n = 20)
	Mean (SD)	Min	Max	Mean (SD)	Min	Max	t/χ2 (df)	p
Number of students at each school	334.2 (184.9)	94	681	363.4 (173.2)	161	780	0.51	.610
Number of classroom teachers at each school	17.9 (7.9)	6	36	19.6 (6.1)	12	32	0.79	.436
Percentage of students at school eligible for free/reduced-priced meals	46.0 (19.0)	17.1	91.8	51.0 (16.7)	28.3	92.6	0.89	.380
	Number of schools	%		Number of schools	%
Percentage of students at each school eligible for free/reduced-priced meals							0.46 (2)	.796
<40% students eligible	8	40%		6	30%
40–60% students eligible	8	40%		9	45%
>60% students eligible	4	20%		5	25%
Remoteness (all schools within rural/township locale)							0.45 (2)	.798
Fringe	3	15%		4	20%
Distant	8	40%		9	45%
Remote	9	45%		7	35%
School level based on grades served							0.53 (5)	.991
Elementary only (grade 6 or lower)	12	60%		11	55%
Elementary/middle (K to grade 8)	1	5%		0	0%
Middle school (grade 6 to grade 8)	2	10%		4	20%
High school only (grade 9 to grade 12)	2	10%		2	10%
Middle/high (grade 7 to grade 12)	1	5%		1	5%
All grades (K to grade 12)	2	10%		2	10%
Total number of students across all schools	6684			7268
Total number of teachers across all schools	357			392

Note: Data source is the 2018-19 Common Core of Data, National Center for Education Statistics.

With regard to masking of treatment condition, it is not possible to mask the research team, nor is the masking of school personnel possible because school teams are aware of whether they receive training only, or training plus additional supports. However, some data collection procedures use masking to ensure that data collectors do not know the condition assignment for each school, particularly for the measure of implementation fidelity (see Data Collection, below).

Detailed Intervention Description

Standard training: both conditions

All 40 schools participate in the following activities, which occur on the same timeline for schools in both conditions. Trainings are delivered in-person or virtually, with attendees including either the school PBIS implementation team (5–8 individuals), or the team's coach and principal. Due to the large number of schools, training occurs across seven weeks each summer, with groups of 3–8 schools (all in the same condition). The content and delivery of these standard trainings are consistent across groups and conditions.

Tier 1 Training. The four-day Tier 1 training occurred in summer 2019, after collection of baseline data, and prior to the start of the implementation period in the 2019-20 school year. Attendees included each school's full PBIS team, and training was delivered by two of the project's implementation support practitioners, who are experienced K-12 educators with master's-level credentials in education. Didactic content includes the theoretical rationale of PBIS, core features of the framework, and detailed review of implementation steps. Learning activities occur such as reflection, action planning, and focused activities that allow teams to tailor the PBIS framework to their setting. During the training, each school team develops an action plan to guide implementation, providing structure to their monthly team meetings and supporting implementation of Tier 1 PBIS. Teams are asked to share information with school staff during professional development days, which typically occur in the week(s) prior to the start of each school year, and throughout the subsequent school year.

Tier 2 Training. Tier 2 training occurred in the summer of 2020 and built on the foundation of universal prevention in Tier 1 by addressing strategies to support at-risk students. Although the primary endpoint of our study is fidelity of Tier 1 implementation, the addition of Tier 2 selective supports can improve Tier 1 fidelity. This three-day training was planned to be delivered in-person, but due to COVID-19, it was delivered virtually. The in-person training had been piloted with three schools not in the study; however, due to the need to adapt to a virtual delivery, three schools not participating in the study were recruited in May 2020 to pilot and refine the virtual approach. Delivery of Tier 2 content was led by a co-investigator (TL) who has extensive expertise in advanced tiers of PBIS, in collaboration with the project's two implementation support practitioners who had previously led the Tier 1 trainings. The content of this training focuses on interventions and supports for students who may need more targeted socio-emotional skill-building and small group intervention, including Check-In/Check-Out, a targeted intervention at Tier 2. As with Tier 1 training, active learning techniques (breakout groups, reflections, action planning, modeling, and focused activities) are used. Schools attend virtual sessions with the same groupings of schools (separated by condition, grouped by state region) as in the prior summer.

Tier 1 and 2 Refresher. Due to the widespread disruptions to education settings as a result of the COVID-19 pandemic, an additional virtual training occurred in summer 2021. This training reviewed Tier 1 and 2, provided information about Tier 3 approaches, and began to plan for sustainability after the research project ends.

Data Systems. Data-based decision-making is a core component of PBIS, and all schools receive training and paid access to the PBIS School-Wide Information System (SWIS) for the duration of the project. Training occurred in May 2019 and was delivered by two national SWIS data system expert trainers from PBIS Apps at the University of Oregon.

Check-In/Check-Out Training. Training occurred in May 2020, and was delivered by expert trainers from PBIS Apps at the University of Oregon.

Feedback from Climate Surveys. In addition to the series of trainings noted above, during summer training sessions, each school receives feedback from climate surveys. This feedback is specific to each school, provided to help teams develop an implementation plan targeting relevant areas of need. Feedback includes school scores on the School-wide Evaluation Tool, Tiered Fidelity Inventory, and staff climate surveys (plus parent and student surveys at baseline; more details on measures below).

Enhanced condition: Rural School Support Strategies (RS3)

In addition to the training components described above, schools in the enhanced condition participate in the following activities:

DidacticSupports: Kickoff and Coaching Institutes. In spring 2019, the principal and coach from each of the 20 schools in the enhanced condition attended a one-day in-person institute, which introduced PBIS, with the goal of supporting readiness for implementation and building organizational capacity. The principal and coach from each school team also attended two trainings on coaching, with a two-day session in spring 2019, and a one-day session in spring 2020. Training sessions were developed with the objective of increasing coaches’ knowledge about how to guide schools through the process of change, and increasing skills and confidence to use change strategies, based on the principles of cognitive coaching [49]. The institutes were led by a doctorally-trained educator with experience in PBIS implementation.

External Coaching and Technical Assistance (TA). A central element of RS3 is the provision of external coaching and TA, to support schools during Phase Three of the Quality Implementation Framework (providing ongoing structure as implementation occurs). This aspect of the enhanced intervention is provided by two members of the project team (TC and NA) who are experienced K-12 educators who have previously served as school-level PBIS coaches, and who also delivered the Tier 1 trainings and other didactic elements of the current project. Support is tailored to each school based on the school's needs. In the first semester of the implementation period (fall 2019), each of the two implementation support practitioners made on-site visits to each school, including activities such as attending school PBIS team meetings, reviewing data, and presenting additional information to school staff. After November 2019, all visits shifted to a virtual format. Meetings occur at least monthly for each school, plus additional meetings if requested by schools. Tracking logs are used to document dose and activities conducted. Activities utilized during visits include facilitation (guided problem-solving), audit and feedback, reviewing data, reviewing action plans, observing team meetings, and other strategies to facilitate implementation (see Table 2).

Virtual Learning Sessions. Monthly virtual learning sessions are held during each school year (September to May) for school-level coaches. Sessions are led by the three implementation support practitioners. Each session is planned in advance, with didactic content relevant to implementation (e.g., teaming, behavior analysis, data-based decision making). The learning sessions include presentation of material, with guided work time for school coaches to plan, review, and obtain guidance from the specialists.

Online Resource Portal. A password-protected website was developed to allow staff at schools in the enhanced condition to access videos, blueprints, and educational resources, plus recordings of the training institutes and the virtual learning sessions.

Ongoing Proactive Provision of Educational Materials. Beginning in November 2020, a monthly newsletter has been developed and distributed to school coaches. Initially the goal was to assist teams in adjusting to pandemic-related challenges, and due to positive feedback is being continued through the project. Information includes PBIS-relevant implementation tips as well as a focus on leadership and staff and student wellness.

Data collection

Data collection leverages standard measures in education settings (e.g., academic achievement, office disciplinary referrals), measures that are a routine part of PBIS implementation, and custom measures for this project. Measures are being gathered at a variety of levels, to assess school context and setting, process and outcomes of the intervention, and assessments of parent, staff, and student-level outcomes. Primary outcomes include fidelity of implementation, as well as acceptability, appropriateness, feasibility, and cost. Student-level outcomes include disciplinary incidents, academic outcomes, and perceived school climate. Measures are described below.

Implementation outcome measures

Fidelity. Two measures are used to assess fidelity of PBIS implementation, including one that is self-reported by school personnel, and one that is independently assessed by data collectors who can be masked to each school's treatment condition. The Tiered Fidelity Inventory (TFI) [50] is the tool recommended by the PBIS developers for assessing fidelity of implementation of PBIS at each of the three tiers. It has shown good psychometric characteristics, including strong construct validity, inter-rater and test-retest reliability, and convergence with other fidelity measures [51]. The measure is completed by school-level teams and takes approximately 45–60 min to complete through an online data capture system. The measure yields a percentage (range 0%–100%) with: a) an overall score across all three tiers; (b) a score for each tier; and (c) subscale scores for each tier that focus on key domains of teams, implementation, and evaluation. Scores above 70% represent adequate fidelity. It is collected each summer (2019, 2020, 2021, 2022). The Schoolwide Evaluation Tool (SET) [52] is another widely-used measure of fidelity of implementation of PBIS at Tier 1, with good psychometric characteristics. It takes 2–3 h of direct observation in a school by an independent observer. The measure yields a percentage score ranging from 0 to 100%; above 80% represents adequate Tier 1 fidelity. The SET was collected at baseline (April 2019) by observers who were masked to treatment condition. Due to the pandemic, follow-up collection occurs only in Spring 2022.

Acceptability, appropriateness, and feasibility. The acceptability, appropriateness, and feasibility of RS3 are being explored through interviews with key stakeholders at participating schools, namely, the school PBIS coach, and the principal. At the schools in the enhanced condition, annual semi-structured interviews assess perceptions about RS3 implementation support strategies, and at schools in both conditions, interviews assess perceptions about the common training elements. In addition, although the study is not designed to assess the acceptability or appropriateness of the PBIS framework itself, these interviews also explore determinants of implementation such as perceptions about PBIS, resources, capacity, and other aspects of school context. Perceptions of PBIS acceptability and appropriateness are also assessed on staff surveys with items from the Staff Perceptions of Behavior and Discipline scale [53].

Cost. The cost and cost effectiveness of prevention programs are a key element of success in obtaining widespread scale-up [44,54] and both are important priorities in prevention science [55]. We will use an ingredients-based costing approach as has been applied to other prevention programs [56], following approaches to estimating costing for PBIS implementation [57,58]. We will estimate incremental total and unit costs for the basic condition and the enhanced condition, and cost-effectiveness analyses will compare incremental net costs with the benefits in student-level outcomes, such as disciplinary referrals, associated with the treatment condition.

Hypothesized mediators of RS3's impact on fidelity

Improved team functioning and changes in attitudes and perceptions among staff at schools in the enhanced condition are both hypothesized to mediate improvements in implementation fidelity. These constructs are measured with team surveys and all-staff surveys, which are collected confidentially with identification numbers to allow examination of changes over time. Surveys are distributed each spring to all staff at all 40 schools (both conditions).

School PBIS team activities. School PBIS coaches completed the Team Implementation Checklist [59] in December 2019 and December 2021, to assess implementation process. The checklist assesses PBIS team activities such as whether the team meets regularly, uses discipline data for planning, and monitors progress toward the action plan. Prior work demonstrates high internal consistency (α = 0.93) [12].

School PBIS team functioning and readiness. School PBIS team members are asked to complete surveys in spring 2019, 2020, and 2022. Items from the PROSPER (PROmoting School-community-university Partnerships to Enhance Resilience) project surveys are used to assess functioning of the PBIS team, specific to team goals, culture, and leadership [36]. Additionally, the Organizational Readiness to Implement Change scale [60] assesses change commitment and efficacy.

School all-staff perception and attitudes. Each spring, all school staff (instructional, classified, and administrative) are invited to complete a survey, which includes items to assess several constructs hypothesized to mediate change. Attitudes toward EBIs are assessed with items from the Evidence-Based Practices Attitudes Scale [61]. Perceptions of psychological safety are assessed with the Psychological Safety Scale [62].

Baseline covariates

Numerous variables were assessed at baseline, for the purpose of collecting explanatory variables in subsequent analyses exploring intervention outcomes. These included community-level variables, school demographic and physical characteristics, and elements of the social environment, such as climate.

Community context. At baseline, community context was assessed with items from the School Safety Survey [63], which assesses community risk and protective factors; these items were included on the PBIS team survey that was collected in February 2019.

School physical environment. Aspects of the school physical environment can impact student perceptions of safety, thus the School Assessment for Environmental Typology [64] was collected at baseline to characterize aspects of the built environment at each school. Trained observers masked to school condition conducted the observation at each school in spring 2019.

School context. Demographic characteristics such as school size, funding, student composition, staffing, and fiscal characteristics were obtained from NCES [48].

Teacher and student interactions. Interactions between teachers and students were observed using the Assessing School Settings: Interactions of Students and Teachers (ASSIST) protocol [65,66]. After training by an expert and establishment of inter-rater reliability, observers who were masked to condition conducted observations onsite at each school on one day in spring 2019. Approximately twelve 10-min observations were conducted across various non-classroom settings while students were present, including: morning arrival, cafeteria, hallways, stairwells (if present), and departure areas during school dismissal.

Perceived school climate: parents, students, school staff. Climate was assessed at baseline in March 2019, using the validated Maryland Safe and Supportive Schools (MDS3) suite of questionnaires [13]. Surveys were programmed in Qualtrics by the research team, with a unique set of electronic links distributed to each school, for three sets of stakeholders: 1) parent/caregiver; 2) student; and 3) all school staff. Parent/caregiver surveys are gathered anonymously using an online survey link that was distributed to parents with assistance from participating schools, and survey modules were available in English and Spanish versions. Parent surveys are anonymous, gathered with assistance from participating schools, whereby each teacher facilitates survey administration during class either in the computer lab or using tablets in the classroom. Staff surveys are distributed with a customized email to each school staff member (instructional, administrative, support), using school staff lists. These data are identifiable, collected confidentially to allow linkages with multiple waves of survey data. Parent and student surveys are estimated to take approximately 15 min to complete, and staff surveys take approximately 25 min. Informed consent/assent is obtained prior to survey administration. Staff surveys also occur in spring 2020, 2021, and 2022.

Student outcome measures

Although the proximal outcome is fidelity at the school level, we anticipate that the intervention will impact student-level outcomes. Consistent with prior school-based research [19] and conceptual work regarding the mechanisms of implementation, we hypothesize that improvements in student-level outcomes are mediated by better fidelity of PBIS implementation at the school level.

Student disciplinary incidents. All schools use SWIS [67], the data monitoring tool for office disciplinary referrals for PBIS; data are tracked at an individual student level and changes in office referrals will be examined at a school level (number of referrals per 100 students per school day), and at an individual student level. Impacts on this outcome measure due to COVID-19 are acknowledged in Table 4, which describes the SPIRIT-CONSERVE impacts on the trial due to these extenuating circumstances. Namely, very few schools used SWIS and issued office disciplinary referrals to students in 2020 due to social distancing (i.e., fewer students in classrooms) and many schools using virtual or hybrid modes of instruction, for which students were not in regular physical attendance at schools. As a result, office discipline referrals will be examined for the first 7 months of the trial (September 2019 until March 2020), and thereafter analyses will be exploratory.

Table 4

CONSERVE-SPIRIT extension, description of impacts to study as a result of COVID-19 pandemic.

Item	Title	Description
I.	Extenuating Circumstances	Describe the circumstances and how they constitute extenuating circumstances.	To mitigate transmission of COVID-19, many schools closed or restricted visitors.
II.	Important Modifications	a. Describe how the modifications are important modifications.	Several planned data collection activities were impacted, and conducting on-site visits and in-person training events was not possible.
		b. Describe the impacts and mitigating strategies, including their rationale and implications for the trial.	Intervention delivery was shifted to a virtual format with use of teleconferencing. Data collection was not conducted.
		c. Provide a modification timeline.	Impacts began in March 2020 and are continuing as of the current time.
III.	Responsible Parties	State who planned, reviewed and approved the modifications.	The project principal investigator reviewed and approved modifications.
IV.	Interim Data	If modifications were informed by trial data, describe how the interim data were used, including whether they were examined by study group, and whether the individuals reviewing the data were blinded to the treatment allocation.	Modifications were not due to interim data.
SPIRIT Item and Number		For each row, if important modifications occurred, check one or both of “impact” and/or “mitigating strategy” and describe the changes in the protocol. Check “no change” for items that are unaffected in the extenuating circumstance.

The CONSERVE-SPIRIT Checklist is licensed by the CONSERVE Group under the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International license.

*Aspects of the trial that are directly affected or changed by the extenuating circumstance and are not under the control of investigators, sponsor or funder.

**Aspects of the trial that are modified by the study investigators, sponsor or funder to respond to the extenuating circumstance or manage the direct impacts on the trial.

Student academic outcomes. In Idaho, student achievement is measured with the Idaho Standards Achievement Test [68] in grades 3–8 and grade 10, in English language arts and in mathematics. Due to the COVID-19 pandemic, no testing occurred in spring 2020, and by the following year, many of the students enrolled in schools at baseline had been lost to follow-up. Impacts on this outcome measure are acknowledged in Table 4.

Student perceptions of school climate. As noted above, school climate is assessed among multiple stakeholder groups. Climate surveys were administered to students at baseline (spring 2019), but due to the COVID-19 pandemic and to allow schools to focus on other educational needs, schools in this project did not administer student climate surveys in spring 2020 or 2021. Surveys are being administered in spring 2022; they are anonymous. Survey administration is facilitated by administrators and teachers, with the goal of surveying all assenting students in grades 3–12. Analyses will compare aggregate (average) grade-level perceptions over time, within each school.

Process data: dose, fidelity of the delivery of RS3

Process measures include interviews with school coaches and principals, and tracking logs and reflections from the project's implementation support practitioners, to assess dose of the RS3 implementation support strategies and fidelity to the RS3 approach [69,70].

Modifications and impact of COVID-19

We document modifications with the CONSORT and SPIRIT Extension for RCTs Revised in Extenuating Circumstances (CONSERVE) extension [71] for trials impacted by the COVID-19 pandemic (Table 4). One notable modification and one notable impact occurred as a result of the pandemic. First, the training elements of the study--for both arms of the trial—were delivered as virtual trainings rather than in-person trainings from Summer 2020 onward. As a result, the summer 2020 and 2021 institutes occurred virtually, with school teams meeting in the same regional groupings as they had done during the Tier 1 institute (summer 2019). The one notable impact to this study involved missing data due to the inability to conduct on-site data collection after March 2020 (impacting the Schoolwide Evaluation Tool), the statewide cancellation of academic achievement testing in May 2020, and schools’ requests that due to other burdens on students and families that we not collect the student and parent climate surveys in spring 2020 or spring 2021. Many of the other planned data collection activities (e.g., staff surveys, interviews, implementation tracking logs, etc.) are continuing as intended.

Data management and statistical analysis

Data are being compiled with Research Electronic Data Capture (REDCap) and are stored on a secure data server with access limited to research personnel. Prior to conducting analyses, data are cleaned (e.g., variables defined, missingness and reliability reviewed). Descriptive analyses will assess the distribution and categorical frequencies of district, school, and individual-level (staff, PBIS team, parent, student) data at baseline and subsequent waves. Equivalence testing across intervention groups and across waves will include t-tests, correlation, and chi-square analyses, and repeated measures ANCOVA or similar analyses to assess overall unadjusted group differences. Descriptive analyses will be used to assess the type and impact of missing data and apply multiple imputation, if suitable and necessary.

The primary study outcome, implementation fidelity (Aim 1), is being evaluated at the school level with the Tiered Fidelity Inventory, considering percentage scores, as well as the 70% criterion for sufficient fidelity. The study initially was planned with analyses using hierarchical linear growth models (students nested within schools nested within districts over time) and structural equation models to assess the effects of the intervention on implementation fidelity and student outcomes. However, because the COVID-19 pandemic brought changes to data collection, including an additional wave, we modified our approach and in some cases are able to use more complex analytic methods that provide greater flexibility in model specification.

Between-group effects of the RS3 intervention will be examined within a structural equation model framework as a school-level latent growth curve model, adjusting for clustering and school demographic characteristics or other explanatory covariates. The dependent variable—fidelity scores—will first be modeled as a school-level latent growth curve, allowing us to identify the effect of the RS3 intervention on both the intercept and slope of fidelity. Intervention condition will be modeled first as a binary, time-invariant predictor, and then as continuous time-varying variables representing intervention exposure (e.g., attendance at virtual learning sessions) to assess which RS3 intervention components, and in what dosages, are most impactful on fidelity. This approach provides flexibility to assess mediators of the intervention effect (Aim 2) within the same analytic framework, while also allowing precise model specification (e.g., correlating error terms, covarying intercept and slopes of the latent growth model) and testing.

Although Type 3 hybrids focus primarily on implementation outcomes, student-level outcomes are also of interest. To examine the effect of RS3 on student behavior, academic achievement, and perceived climate (Aim 3), we will use multi-level path analyses—with individuals nested within schools—within the structural equation model framework. Analogous to a multi-level regression model, intervention condition will be modeled as a school-level, baseline dichotomous independent variable and will predict subsequent student-level outcomes (i.e., climate, academic achievement) at follow-up waves. To examine the effect of school-level fidelity as a mediator of student outcomes (Aim 3a), this model will be adapted as a time-ordered mediation path model, which will assess the degree to which fidelity mediates the relationship between the RS3 intervention and student outcomes.

Qualitative and process data

Qualitative data are being collected through interviews with school coaches and principals at several points. In spring 2019, all coaches and principals were interviewed; in spring 2020 all coaches were interviewed; in spring 2022 all coaches will again be interviewed. These brief (30 min) interviews follow semi-structured guides, exploring feasibility, acceptability, and appropriateness of RS3 as well as PBIS; COVID-related challenges and adaptations; and factors facilitating implementation. Participants provide informed consent and interviews are conducted via teleconference (Zoom) and are electronically recorded and transcribed verbatim. Transcripts will be coded by multiple project staff. Thematic content analysis [72] is being used, with analysis conducted in Dedoose (SocioCultural Research Consultants, LLC, California) using detailed codebooks. Rigor will be maintained through consistent inter-coder reliability, checking of the derived themes, and triangulation of these themes with content experts.

Sample size justification and power

The study initially was designed with the goal of assessing school-level implementation outcomes, as well as student-level outcomes, requiring power considerations at both the organization and individual level, across three waves. To detect the hypothesized effects on student-level outcomes, the design proposed 20 schools per condition, with a minimum of 100 students per school, based on a 4-level model (students within schools within districts, with three time points). Assumptions were intra-class correlations of 0.10, 0.20, and 0.10 (for student, school, and district levels, respectively), and a 0.05 two-sided Type I error rate. This yields 80% power to detect an effect size of 0.30 when including multiple covariates at school and district levels. However, as very few of the participating schools were actually located within the same district, the need to explicitly model district as a level is minimal. Furthermore, due to the COVID-19 pandemic, student achievement testing was cancelled in 2020, much of the behavioral outcome data was not valid, and tracking of students across waves was not feasible. Thus, analyses of academic achievement will consider this outcome in 2021 and 2022, with indirect baseline adjustment for aggregate school-level variables. Among the 40 schools in the study, baseline climate surveys were gathered from 6610 students, a higher enrollment than anticipated. Presuming the additional wave in 2022 yields a similar sample size of an average of 165 students at each of the 40 schools, a cluster-randomized (2-level) trial is powered at 81% to find a small-to-moderate effect size of 0.3, with a student-level ICC of 0.10 and α = 0.05, per analyses conducted in WebPower v.0.6 [73].

For school-level outcomes, the most statistically conservative approach to detecting significant condition effects on implementation fidelity would be to use repeated measures analysis of variance (ANOVA). With 40 schools, repeated measures at three timepoints yields power at 80% to detect moderate effect sizes of 0.45 and 0.50 for between-group and within-group effects, respectively. Extension of the trial to a fourth time point (2022) will improve the ability to detect effects. Latent growth curve models are less sensitive to small sample sizes and have substantially more power to detect smaller effect sizes, particularly among small samples [74]. This approach will also be utilized, providing substantially more flexibility in model specification. Due to the pandemic, it may be necessary to consider non-linear trends with outcomes declining after spring 2020.

Discussion

This study is testing the effectiveness of a bundle of implementation supports selected to improve the implementation outcomes for PBIS in rural settings. With the proximal and primary outcome of PBIS fidelity, other key aims include examining individual-level impacts (student outcomes), and mediation of student outcomes by implementation fidelity.

Innovation, significance, and strengths

A key issue for understanding how to scale up interventions is to consider crucial implementation outcomes such as feasibility, acceptability, and—importantly—the effectiveness of those strategies for improving implementation fidelity and clinical (i.e., student-level) outcomes. The current study was designed to address known challenges to school-based EBI implementation in rural settings, through a mix of early on-site visits for relationship-building and establishing trust and credibility, followed by lower-cost and more feasible approaches to deploy implementation strategies (i.e., tailored feedback, problem-solving, conducting team meetings) by the university support system. Thus far, few Type 3 hybrid trials have been conducted in the school-based prevention field, but these trials can be very informative about key scientific issues. Although tests of one single implementation support strategy would yield more definitive evidence about the effectiveness of each component of RS3, a bundled design allows us to test a fairly comprehensive collection of implementation supports. Based on the activities and rationale in our grant application, and review by our team, we identify 22 activities that fall into 7 of the 9 subgroups of SISTER implementation strategies. Although we did not engage in a formal process to code these strategies as others have done recently for a similar school-based prevention trial [75], the bundling of multiple strategies is not uncommon for this type of work [75,76], and the collection of detailed activity logs [77] by our project personnel allows for coding types of strategies used, as well as the dose of support provided. Although our study was designed and funded before the SISTER taxonomy was disseminated, using standard terminology to describe strategies will help provide clarity in interpreting our study results.

This project may help to inform evidence not only about PBIS implementation, but also the implementation of other school-based EBIs in remote locations. Furthermore, there is a need for greater attention to mechanisms of action in implementation science, and we have identified a-priori hypotheses about potential mechanisms through which RS3 may improve outcomes. The study also utilizes a partnership-focused approach, which is important for translational research and effectively scaling prevention programs [78]. Quantitative data collection involves the use of psychometrically-strong measures to assess a broad array of constructs previously demonstrated to be associated with subsequent EBI implementation outcomes. The combination of quantitative and qualitative measures is a strength and will allow for a variety of analyses. The collection of onsite observations with expert-trained data collectors who are masked to intervention condition also adds to the rigor of this work.

Limitations and challenges

Despite many strengths, this study also faces several limitations. Notably, the COVID-19 pandemic has impacted several elements of this project; however, steps have been taken to mitigate the impact and to continue to pursue many of the aims of the original trial design. While the RS3 implementation supports being tested include several of the SISTER strategies rated by educators as highest in importance and feasibility [79], such as ‘provide ongoing coaching/consultation’ (SISTER #44) and ‘monitor the progress of the implementation effort (SISTER #9), it is also important to note that the basic condition is receiving several effective strategies ranked highly for importance, such as ‘conduct ongoing training’ (SISTER #39). In other words, the comparison condition is receiving strategies that are also powerful, which will likely minimize differences between control and intervention. While a relevant comparison for RS3 would be to a no-treatment control condition, to do so may potentially be unethical [80,81] given what is known about the effectiveness of universal prevention programs. Thus, the comparison in this study is between standard training, versus training plus enhanced supports.

Furthermore, we note that although a primary outcome of the study is implementation fidelity, the PBIS approach is not a manualized curriculum or a program that is delivered in a scripted way. Rather, it is a framework through which certain practices can be selected and implemented; as the PBIS developers note, the adaptability of PBIS to local contexts is a strength of the approach [11]. Nevertheless, adaptations reduce standardization of interventions, a tension which can sometimes impact fidelity [82,83]. The current study uses two measures of fidelity that assess core components of PBIS that should be present regardless of the surface adaptations (i.e., Tiered Fidelity Inventory, Schoolwide Evaluation Tool). In addition, the focus on other important implementation outcomes (e.g., acceptability, feasibility, cost) will yield valuable information about RS3's promise for scaling prevention interventions.

Conclusion

This study will yield evidence about the effectiveness of a bundle of implementation support strategies for improving fidelity of implementation of a schoolwide evidence-based framework in rural schools. Information gathered will inform knowledge about the type and dose of support that is needed for the implementation of PBIS with high levels of fidelity, and patterns of implementation over time. Cost and cost-effectiveness analyses will provide information that may be of use to leaders at school levels, as well as those at district and state education agencies. Cost information will also help other policymakers to understand the costs of successfully implementing PBIS at each school, and how the costs incurred with providing additional support might yield increases in implementation fidelity and student outcomes. Baseline contextual variables such as community risk and protective factors, and school physical environment, will be considered, and the study will explore mechanisms through which RS3 may improve fidelity, such as improved team functioning, changes in staff attitudes toward EBIs, and aspects of the school social environment such as psychological safety. This study will yield novel information about the mechanisms through which partnerships between prevention support systems may help to improve the implementation of effective practices in ways that promote safe school environments.

Ethics approval and consent to participate

The protocol and measures were approved by the Institutional Review Board at Boise State University, 101-SB17-207, and all schools provided written agreement from the principal and district superintendent to participate in the project. All surveys are optional and begin with the elements of informed consent, with a waiver of signed documentation of consent for electronic surveys.

Availability of data and material

Requests for data and materials may be made by emailing the corresponding author.

Funding

This study is funded by the , #2017-CK-BX-0021.

Authors’ contributions

LT, CS, and HC conceptualized and designed the study. LT, HC, and CF led the development of the manuscript. LT, HC, CS, and TL selected the survey items and tools for assessing fidelity. MM developed the interview guides and conducted interviews. CS and CF conducted the power analyses. TL, AH, TC, and NA lead the development of programming, including the core elements of RS3. All authors critically reviewed the manuscript and contributed text to their respective sections and areas of expertise on the project.

Author declaration

1) We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.

2) We confirm that the manuscript has been read and approved by all named authors and that there are no other persons who satisfied the criteria for authorship but are not listed. We further confirm that the order of authors listed in the manuscript has been approved by all of us.

3) We confirm that neither the entire paper nor any of its content has been submitted, published, or accepted by another journal. The paper will not be submitted elsewhere if accepted for publication in the Journal.

4) We confirm that we have given due consideration to the protection of intellectual property associated with this work and that there are no impediments to publication, including the timing of publication, with respect to intellectual property. In so doing we confirm that we have followed the regulations of our institutions concerning intellectual property.

5) We confirm that any aspect of the work covered in this manuscript that has involved either experimental animals or human patients has been conducted with the ethical approval of all relevant bodies and that such approvals are acknowledged within the manuscript.

6) We understand that the Corresponding Author is the sole contact for the Editorial process (including Editorial Manager and direct communications with the office). He/she is responsible for communicating with the other authors about progress, submissions of revisions and final approval of proofs.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.