Examination of a Tiered Training Model to Increase School Psychology Graduate Students' Behavior Specific Praise.

Several strategies (e.g., performance feedback, video models, tactile prompting) have been found to be effective for improving preservice teachers' use of foundational behavior management skills. However, there is limited research examining these training strategies for promoting preservice clinicians' use of evidence-based behavior management skills. Furthermore, when these strategies are utilized, personnel receiving training often respond differentially. Therefore, the purpose of this study was to evaluate the effectiveness of a tiered training model that incorporated performance feedback, video models, and tactile prompts to increase school psychology graduate students' rates of behavior specific praise during one-to-one sessions with child clients. Results indicated that rates of behavior specific praise increased and maintained across time. Findings, limitations, and directions for future research are discussed. © Association for Behavior Analysis International 2021.

Introduction

Approximately 5%–30% of children experience learning difficulties in reading, writing, and mathematics (Altarac & Saroha, 2007; American Psychiatric Association, 2013; Catts et al., 2012; Pastor & Reuben, 2008). Learning difficulties can be attributable to skill deficits, performance deficits, or both (Duhon et al., 2004). Regardless of the reasons for the difficulties, targeted positive social reinforcement for effort (skill deficit) and improved performance (performance deficit) can result in children’s improved engagement and academic outcomes (Duhon et al., 2004; St. Peter Pipkin et al., 2007; Zhou et al., 2019).

Praise is one strategy that can function as positive social reinforcement and may lead to improved child behavior (LaBrot et al., 2020a) and academic outcomes (e.g., academic engagement; Sallese & Vannest, 2020). Praise is a statement of approval or admiration for a specific behavior or set of desired behaviors (Brophy, 1981). Praise statements can either be general (i.e., does not reference a specific behavior) or behavior specific (i.e., references a specific behavior; Gable et al., 2009). In particular, behavior-specific praise (BSP) involves the provision of a positive statement that is contingent upon appropriate behavior and describes the behavior being praised (e.g., “You did a great job answering that question”; Ennis et al., 2018; Royer et al., 2019). Although both forms of praise are useful for improving child outcomes, BSP is considered more effective than general praise because it allows children to understand which behavior being praised (Brophy, 1981; Gable et al., 2009).

There is a plethora of research that demonstrates the effectiveness of BSP for improving child outcomes. For example, in a systematic review of the literature, Royer et al. (2019) found that BSP was effective for increasing on-task behavior and decreasing disruptive behaviors in classroom students. Additional research has found that BSP also results in decreased off-task behavior (Dufrene et al., 2014; Reinke et al., 2008). However, research demonstrates that professionals tasked with serving children often deliver variable rates of BSP (e.g., Floress & Jenkins, 2015; Reinke et al., 2013). This may be attributable to the fact that preservice professionals often lack adequate training in the implementation of evidence-based behavior management techniques (Floress et al., 2017a, b; Freeman et al., 2014). Fortunately, strategies to improve preservice professionals use of evidence-based behavior management strategies, such as BSP, have been evaluated and found to be effective.

For example, Rathel et al. (2008) evaluated graphed performance feedback sent via email to increase positive communication (i.e., BSP, general praise, and nonverbal gestures [high fives]) and decrease negative communication (i.e., corrective statements and nonverbal disapproval [shaking head no]) for preservice special education teachers. Likewise, McLeod et al. (2019) evaluated the effects of emailed performance feedback with video models for increasing the frequency of two early childhood special education preservice teachers’ use of emotional labeling (i.e., identifying an emotion the adult or child experienced in an interaction), BSP, and providing choices (i.e., adult provided verbal choice between two objects or activities). Results of both studies indicated improved preservice special education teacher outcomes.

In addition to performance feedback and video models, more direct strategies have also been effective for improving preservice professionals use of evidence-based behavior management techniques. For example, Markelz et al. (2018) tested the use of an Apple Watch to send tactile prompts and text messages on a variable schedule to a preservice special education teacher to increase BSP, active questioning (i.e., saying the child’s name and asking a question that resulted in a response), and classroom scanning (i.e., systematically observing students from one side of the classroom to the other). Results demonstrated immediate increases in BSP, active questioning, and classroom scanning following text message tactile prompts. Overall, research suggests that strategies such as performance feedback, video models, and tactile prompting are effective for improving preservice professionals use of evidence-based behavior management skills.

However, the literature examining strategies for training preservice professionals who serve children is largely focused on preservice teachers. Much like preservice teachers, preservice clinicians (i.e., graduate students in psychology) may also benefit from similar training modalities given the scope of services they provide for children at-risk for learning, behavioral, and social/emotional disorders. Although graduate supervisors train evidence-based strategies in the context of implementing behavioral teacher and parent training, there is limited research examining training and supervision to improve preservice clinicians’ use of behavior management techniques during their direct work (i.e., academic and mental health services) with children (e.g., Jones & Eddy, 2017). Furthermore, the supervision of preservice clinician literature, in general, is lacking in terms of studies that systematically collect data on supervisee’s direct outcomes (Newman et al., 2019). Therefore, research that evaluates training and supervision strategies with systematically collected data on supervisee’s outcomes is needed.

Tiered Training

The training literature includes several strategies that are effective for improving preservice professionals’ use of behavior management techniques (e.g., Markelz et al., 2018; McLeod et al., 2019; Rathel et al., 2008). However, these strategies vary in intensity and resources required for implementation. Thus, a continuum of training supports that increase in intensity if trainees fail to respond to less intensive training efforts could be a viable option for providing focused training. This is a relatively new approach to training that is beginning to garner a promising evidence base among school-based research (LaBrot et al., 2020b; Myers et al., 2011; Sanetti & Collier-Meek, 2015; Simonsen et al., 2014; Thompson et al., 2012).

In a tiered training framework, didactic trainings to teach about behavior management techniques serve as Tier 1 (e.g., LaBrot et al., 2020b). If trainees fail to implement evidence-based behavior management techniques taught during Tier 1, they make a transition to Tier 2, which includes more intensive supports (LaBrot et al., 2020b; Sanetti & Collier-Meek, 2015). If trainees fail to respond to Tier 2 training, they make a transition to Tier 3, which consists of the implementation of additional intensive supports (Myers et al., 2011; Thompson et al., 2012). As trainees begin to respond to higher-level tiers, they can begin to make a transition back to less intensive, lower-level tiers with eventual removal of training supports (LaBrot et al., 2020b; Myers et al., 2011).

As an example, LaBrot, Dufrene, Whipple, et al. (2020b) evaluated the effectiveness of a tiered training model to increase two preschool and two elementary teachers’ rates of BSP. In this study, Tier 1 consisted of large-group didactic training, Tier 2 consisted of emailed performance feedback with video models (termed digital performance feedback), and Tier 3 consisted of tactile prompts. Results indicated that three of the four teachers increased their rates of BSP while receiving Tier 2 supports, whereas two of these teachers maintained their use of BSP following the removal of tiered training supports. However, one teacher required Tier 3 supports before she adequately increased her rates of BSP. This study provides some evidence of the effectiveness of a tiered training model for improving trainees’ rates of BSP. However, this study, as well as other tiered training research, contain some limitations that should be addressed.

First, participants have been limited to in-service teachers in classroom settings that included multiple students (LaBrot, Dufrene, Whipple, et al., 2019b; Myers et al., 2011; Sanetti & Collier-Meek, 2015; Simonsen et al., 2014; Thompson et al., 2012). As such, additional research is needed to determine whether a tiered training model is effective for improving preservice clinicians’ behavior management techniques in the context of one-to-one sessions with clients. Second, LaBrot, Dufrene, Whipple, et al. (2020b) and Myers et al. (2011) are the only known studies to demonstrate that trainees maintained their use of behavior management strategies after making a successful transition through tiered training. Additional research is needed to demonstrate that trainees maintain their use of behavior management techniques following removal of a tiered training model. Finally, additional research replicating the existing literature is needed to establish tiered training as an effective training modality.

Current Study

The purpose of this study was to evaluate the effectiveness of a tiered training model for increasing graduate student clinicians’ rates of BSP during ongoing sessions with clients in an outpatient academic assessment and intervention clinic. In particular, this study closely replicated the procedures of LaBrot, Dufrene, Whipple, et al. (2020b) in that it included similar tiered training procedures (i.e., didactic training for Tier 1, digital performance feedback for Tier 2, and tactile prompting for Tier 3) with the same dependent variable (i.e., BSP). Furthermore, this study sought to address limitations of the existing tiered training literature by (1) training school psychology practicum students to increase rates of BSP during one-to-one sessions with clients, (2) successfully transitioning student clinicians to removal of training supports, (3) collecting data on student clinicians’ maintained use of BSP, and (4) replicating the results of previous tiered training research. This study included the following research questions:

Does a tiered training model increase student clinicians’ rate of BSP and do increases maintain following removal of tiered training supports?

Do student clinicians rate tiered training as a socially valid training model?

Do student clinicians rate BSP as a socially valid behavior management strategy?

Method

Setting and Participants

Three specialist-level graduate students in school psychology participated in the current study. Graduate student participants attended a university in a mid-sized city in the Midwest. The graduate students who participated in this study were enrolled in an off-site, 9-month practicum as part of their standard coursework, located at a university-based academic health center. One night a week, the academic health center held an academic evaluation and intervention clinic for struggling learners (i.e., clients) with and without developmental disabilities, ranging in ages from 6 to 17. Common referral concerns addressed in the academic evaluation and intervention clinic included reading, mathematics, writing, and organization difficulties.

The primary roles of the graduate students (hereafter referred to as student clinicians) included assessment of academic referral concerns, brief experimental analysis of potentially effective academic interventions, intervention implementation, training caregivers to implement academic interventions, and progress monitoring. These clinical activities were largely implemented in an explicit instruction format (Archer & Hughes, 2011). Assessment and brief experimental analysis procedures often involved clients verbally responding to academic problems presented on flashcards, reading passages out loud for various periods of time (e.g., 1–3 min), verbally responding to student clinicians’ questions and prompts (e.g., “Five minus 2 equals 3, the answer is 3”), and verbally engaging in error correction procedures (e.g., restating a problem and its answer three times after providing an incorrect response). Intervention implementation involved a student clinician implementing an intervention identified as effective through the brief experimental analysis, which also involved clients reading, responding to flashcards, answering questions, and engaging in verbal error correction. After intervention implementation, clients’ parents were trained by student clinicians to implement specific academic interventions via behavioral skills training. Following parent training, sessions involved intervention implementation with an emphasis on clients continuing to practice skills, ongoing assessment of additional academic concerns, and progress monitoring. Most sessions also involved student clinicians offering a brief break (e.g., 5 min) to their clients, which involved clients playing games with student clinicians, walking in the clinic hallways, and using the restroom.

To fulfil these primary roles, student clinicians met with clients one-to-one in clinic rooms located at the academic health center. Student clinicians provided clinical services to two clients per night for 1 hr each. All study procedures occurred within the context of the 1-hr sessions in which student clinicians provided clinical services to their two clients. That is, data were collected during both sessions held each night while the student clinicians worked with their clients.

University approval for data collection during ongoing clinical activities was obtained prior to the start of the study. The participating student clinicians provided informed consent to participate in this study and were offered extra credit (i.e., 20 points towards the final grade per semester) for participation. At the beginning of the academic semester the three student clinician participants were observed by supervisors to struggle with behavior management during sessions; in particular, they were observed to deliver low rates of praise. To be included in this study, the three student clinicians had to be observed to deliver less than 10 BSP statements (i.e., less than .5 BSP statements per minute) during a 20-min screening observation (e.g., LaBrot, Dufrene, Whipple, et al., 2020b). Student clinicians would have been excluded from the study if they were observed to deliver 10 or more BSP statements during the screening observation. Screening observations were conducted during ongoing clinic sessions with the student clinicians and their clients.

Julia (pseudonyms used throughout) was a 36-year-old Hispanic female. Jessica was a 23-year-old white female. Erica was also a 23-year-old white female. All three participants were in their 2nd year of graduate education. Each student clinician reported less than 1 year of experience implementing behavioral interventions with children in school and/or clinical settings and they had not yet taken any coursework in behavior management or psychotherapeutic interventions.

Materials

Audio Recorder

Digital voice recorders were used to record verbal interactions between the student clinicians and their clients. The recordings were saved on a HIPPA and FERPA compliant server, and data were coded following sessions with clients.

Tactile Interval Timer

A Gym Boss Interval Timer Stopwatch® was used to provide tactile prompts to the student clinicians. This device was set to provide a reoccurring tactile prompt (i.e., brief vibration) once every minute.

Usage Rating Profile-Intervention Revised (URP-IR)

To assess the social validity of the tiered training model and BSP, student clinicians were asked to complete the Usage Rating Profile-Intervention Revised (URP-IR; Chafouleas et al., 2011) at the conclusion of data collection. The UPR-IR consists of 29 items rated on a 6-point Likert scale ranging from 1 (strongly disagree) to 6 (strongly agree) that assesses acceptability, understanding, feasibility, home school collaboration, system support, and system climate (Briesch et al., 2013). Previous research supports the internal consistency reliability of each subscale (alpha coefficients: .85–.96; Briesch et al., 2013). The URP-IR was modified to assess student clinicians’ acceptability, understanding, and feasibility related to tiered training procedures and BSP. To assess perceptions of BSP, the word “intervention” was replaced with “behavior specific praise” or “BSP.” To assess Tier 2 training procedures, the terms “video models and performance feedback sent in weekly emails (digital performance feedback)” and “digital performance feedback” were added. To assess Tier 3 training procedures, the term “digital performance feedback with vibrating prompts” was added.

Dependent Variables and Data Collection Procedures

Behavior-Specific Praise

The primary dependent variable was BSP. BSP was defined as a specific-labeled praise statement that included a description of the behavior being praised (e.g., “I like that you are paying attention!”; LaBrot et al., 2020b). Researchers that collected data for this study included a faculty member who served as the instructor and lead supervisor of the practicum course (first author) and two predoctoral interns who served as secondary supervisors of the practicum (second and third authors). Prior to collecting data, all researchers were trained to record student clinicians’ rate of BSP to a 90% agreement criterion. Data were collected by researchers listening to audio recordings of the student clinicians’ sessions with their clients. To ensure data collection was distributed across the sessions evenly, researchers randomly chose one of three 20-min time slots to listen to for data coding. That is, the 60-min sessions were divided into three 20-min time slots. Before data were coded, the numbers 1, 2, and 3 were entered into a random number generator (True Random Number Service, 2020). The number that was randomly chosen was the portion of the session (i.e., first-, second-, or third-third) that was listened to and coded for data. Student clinicians’ rate of BSP was recorded using an event recording procedure in which the frequency of BSP within 10-s intervals was recorded, converted to a rate-based measure, and reported as number of BSP statements per minute. Data were not collected during parent training sessions, because the focus of these sessions involved student clinicians training parents to implement academic interventions and not necessarily work directly with clients. Likewise, data were not collected during session breaks. Because data were audio recorded, it was sometimes difficult to know if BSP was truly delivered contingent upon appropriate behavior. For this reason, only BSP statements that were clearly contingent upon appropriate behavior based on upon audio data (e.g., providing BSP for verbally reading a 1-min passage, providing BSP for verbally responding to a flashcard) were recorded. Therefore, instances of BSP that were not clearly contingent (e.g., providing BSP for sitting in a chair) were not coded.

Experimental Design, Phase Change Decisions, and Data Analysis

A concurrent multiple baseline design across participants was utilized to experimentally evaluate the effectiveness of the tiered training model. This design included the following phases: (1) Tier 1 (i.e., small-group didactic-instruction) served as baseline because didactic-instruction was provided to all students as a standard clinic training procedure, (2) Tier 2 (i.e., weekly digital performance feedback), (3) Tier 3 (i.e., tactile prompt with weekly digital performance feedback), (4) maintenance, and (5) 1-month follow-up (LaBrot, Dufrene, Whipple, et al., 2020b). This experimental design meets evidence standards for single-case design as delineated by the What Works Clearinghouse (Kratochwill et al., 2010). In particular, (1) the independent variable was systematically manipulated with the primary researcher determining condition changes, (2) each dependent variable was systematically measured by more than one researcher for more than 20% of the data points, (3) this study included three attempts to demonstrate training effects at three different points in time, and (4) there were nine phases with at least 5 data points per phase (Kratochwill et al., 2010).

Phase change decisions were made based upon predetermined transitional criteria and data trend. The baseline phases consisted of a minimum of 4, 6, and 8 data points for each participant (Kratochwill et al., 2010). During baseline, stable low-level rates of BSP below a prespecified criterion of .5 BSP statements per min (LaBrot et al., 2016; LaBrot et al., 2020b) resulted in a phase change to Tier 2. When a participant had 3 or more data points below the prespecified criterion of .5 BSP statements per minute, they made a transition to Tier 3. Transitions from Tier 3 to Tier 2 and from Tier 2 to maintenance occurred when participants provided BSP at or above the prespecified criterion for 3 or more consecutive data points. The prespecified criterion of .5 BSP statements per min was chosen as previous research has utilized this criterion and demonstrated that it was an adequate rate of BSP to improve child outcomes (e.g., LaBrot et al., 2020a, 2020b).

Data were analyzed visually for level, trend, variability, immediacy of effects, proportion of data that overlapped, magnitude of changes in the dependent variable, consistency within data patterns (Horner et al., 2005; Kratochwill et al., 2010), and based upon whether participants met the prespecified BSP criterion. In addition, Tau U effect sizes were calculated to complement visual analysis. Tau U is a nonparametric effect size analysis that examines nonoverlap between phases (Parker et al., 2011) and corrects for undesirable baseline trends (Vannest & Ninci, 2015). Effect size score ranges include small (0–.20), moderate (.20–.60), large (.60–.80), and very large (above .80) (Vannest & Ninci, 2015). Tau U was calculated for student clinicians’ rate of BSP. For this study, Tau U was calculated by comparing baseline data to data collected across all other phases.

Procedures

Tier 1 (Baseline)

At the beginning of the academic year, the student clinicians attended a small-group (i.e., all seven student clinicians enrolled in the practicum course) didactic training. During this training, a researcher (second author) provided a rationale for and described BSP, planned ignoring, effective instruction delivery, and effective clinic space management. In particular, a description of the effective use of BSP was provided, indicating that praise is most effective when it is behavior-specific and delivered to children contingent upon on-task and appropriate behavior (LaBrot et al., 2020b). The researcher leading the training then modeled three accurate instances of BSP with another researcher, with emphasis on delivering BSP contingently based upon appropriate behavior. At the end of didactic training, researchers recommended that student clinicians deliver BSP to their clients for expected client behaviors in the context of one-to-one sessions. For example, researchers recommended delivering BSP to clients when they read aloud 1-min passages, verbally responded to presented flashcards, verbally answered student clinicians’ questions, and verbally engaged in error correction procedures (e.g., restating a problem and its answer three times after providing an incorrect response). Student clinicians were trained to provide BSP in this manner as it is consistent with explicit instruction, in which affirmative and corrective feedback includes praise that is contingently delivered, specific, provided for performance, focused on effort, based upon a client’s individual improvement, positively stated, and unobtrusive (Archer & Hughes, 2011). Following this training, student clinicians’ sessions were audio recorded and data were coded for BSP. No feedback was provided during Tier 1.

Tier 2 (Digital Performance Feedback)

Following Tier 1, a researcher held an individual 10-min meeting with student clinicians to review the effects of BSP on child behavior and to review their baseline data. Student clinicians were reminded that BSP is most effective when it is delivered contingently and based upon a specific appropriate behavior. Student clinicians were also reminded to deliver BSP to their clients for behaviors that were expected of their clients (e.g., reading passages aloud, verbally responding to flashcards, verbally engaging in error correction). Student clinicians were recommended to deliver BSP statements at least once per 2 min (LaBrot et al., 2016, 2020b). Student clinicians were informed that they would receive weekly digital performance feedback (DPF) via email on their rate of BSP. The researcher explained that DPF is a strategy to provide more support to train student clinicians and that DPF would include performance feedback on their rate of BSP and would include a video model to view if they did not meet the prespecified BSP criterion. In the video model, a predoctoral intern (second author) played the role of a student clinician with a confederate child recruited by the research team. In the video, the predoctoral intern and the confederate child engaged in an academic task similar to the tasks conducted with clients in the clinic. The predoctoral intern provided three instances of BSP to the confederate child contingent upon appropriate behavior. Each instance of BSP also appeared in writing on the screen to clearly delineate when BSP was being delivered and what it should consist of. The video model was approximately 1 min in length.

Weekly DPF was provided on the same day each week (i.e., the Friday before the next clinic night on Monday). If a student clinician met the predetermined BSP criterion (i.e., .5 BSP statements per min), the email included a brief description of BSP, performance feedback on their BSP rate, praise for achieving the desired BSP rate, a reminder of the predetermined BSP criterion, instructions to reply to the email indicating feedback was read, and a positive statement. See below for an example:

Hey [name],

One way to effectively prevent disruptive session behavior is to deliver behavior specific praise (i.e., label the behavior that you are praising). The last time we listened to your session’s recording, we observed that you delivered behavior specific praise 12 times (rate of .6 behavior specific praise statements per minute). Great job! Remember, we wish for you to deliver behavior specific praise at least once every 2 minutes (rate of .5 behavior specific praise statements per minute). Because you delivered a high rate of behavior specific praise, there is no need for you to watch a video model. Thank you for all of your hard work and dedication to your clients. After you have read this email, please respond and let us know that you have done so. Thank you for all you do, we are happy to be working with you!

If a student clinician failed to meet the predetermined BSP criterion, the email included a brief description of BSP, performance feedback on their BSP rate, corrective feedback, a reminder of the predetermined BSP criterion, instructions to watch the DPF video model, instructions to reply to the email indicating feedback was read and that the video model was viewed, and a positive statement. See below for an example:

Hey [name],

One way to effectively prevent disruptive session behavior is to deliver behavior specific praise (i.e., label the behavior that you are praising). The last time we listened to your session’s recording, we observed that you delivered behavior specific praise 6 times (rate of .3 behavior specific praise statements per minute). Remember, we wish for you to deliver behavior specific praise at least once every 2 minutes (rate of .5 behavior specific praise statements per minute). Please watch this video before your next session. After you have read this email and watched the video, please respond to this email and let us know that you have done so. Thank you for all you do, we are happy to be working with you!

Tier 3 (Tactile Prompting)

Tier 3 included Tier 2 supports as previously described (i.e., student clinicians continued to receive weekly DPF during Tier 3). In addition, Tier 3 consisted of a student clinician wearing a Gym Boss Interval Timer Stopwatch®. A researcher informed the student clinician that the interval timer served as a tactile prompt to look for opportunities to provide contingent BSP. That is, the student clinician was instructed to provide BSP for the next instance of a clients’ appropriate behavior after the interval timer provided a prompt. The interval timer provided a tactile prompt once every min. The interval was set higher (i.e., one BSP statement every minute) than the predetermined BSP criterion to overtrain the student clinician to deliver BSP, as research demonstrates that BSP often decreases immediately following intensive training procedures (e.g., LaBrot et al., 2020a, 2020b).

Maintenance and 1-Month Follow-Up

Maintenance data collection occurred the first clinic night following removal of Tier 2. One-month follow-up data collection occurred 1 month from the end of the maintenance phase. During these phases, student clinicians did not receive DPF emails or any feedback on their rates of BSP.

Interobserver Agreement (IOA) and Procedural Integrity

IOA data were collected across all conditions (i.e., baseline, Tier 2, Tier 3, maintenance, and 1-month follow-up) for all participants. For each audio recording in which IOA data were collected, one observer was designated as the primary observer and that observer’s observation was graphed and used for decision making purposes (i.e., phase changes). The secondary observer’s audio recording data was used for IOA. IOA for student clinicians’ use of BSP was calculated by dividing the number of agreed upon BSP statements within intervals by the number of agreed and disagreed upon BSP statements within intervals and multiplying by the quotient of 100 (i.e., smaller number of statements divided by the largest number of statements and multiplied by 100).

IOA data were collected for 29.63% of the observations for Julia, with a mean agreement of 97.80% (range: 91.67%–100%) for rate of BSP. IOA data were collected for 30.43% of the observations for Jessica, with a mean agreement of 97.74% (range: 91.67%–100%) for rate of BSP. IOA data were collected for 33.33% of the observations for Erica, with a mean agreement of 96.19% (range: 92.5%–100%).

At the end of each week, a checklist was completed by a researcher who implemented study procedures to assess procedural integrity (see Appendix A). For Tier 1, the procedural-integrity checklist included the following components: (1) student clinicians attended the small-group didactic training, (2) a researcher described and provided a rationale for BSP, and (3) a researcher modeled BSP three times. For Tier 2, the procedural-integrity checklist included the following: (1) researcher sent an email to the student clinician that included data of the student clinicians’ rate of BSP and (2) the student clinician replied to the researcher’s email confirming that the email was read and the video model was viewed (if applicable). Finally, the procedural-integrity checklist for Tier 3 had the same two steps as Tier 2, and included the following: (1) Gym Boss Interval Timer Stopwatch® was on and set to vibrate every minute, (2) the student clinician was provided with the Gym Boss Interval Timer Stopwatch® prior to the start of their session, and (3) the student clinician wore the Gym Boss Interval Timer Stopwatch®. Procedural-integrity data were collected each week study procedures were implemented across all phases (i.e., Tiers 1, 2, and 3) in which a tier was implemented for all participants and was 100% for all participants.

Results

Results for participants’ rate of BSP are displayed in Fig. 1. Social validity findings are provided in Table 1.

Fig. 1

Student clinicians' rate of behavior specific praise per minute

Table 1

Social validity

	Julia	Jessica	Erica	All
Overall
Acceptability	5.85 (0.38)	5.73 (0.47)	5.00 (0.48)	5.54 (0.56)
Understanding	6.00 (0.00)	5.75 (0.50)	5.25 (0.50)	5.69 (0.48)
Feasibility	6.00 (0.00)	6.00 (0.00)	5.00 (0.58)	5.68 (0.57)
Behavior Specific Praise
Acceptability	5.78 (0.44)	5.89 (0.33)	5.11 (0.33)	5.59 (0.51)
Understanding	6.00 (0.00)	6.00 (0.50)	5.33 (0.58)	5.78 (0.52)
Feasibility	6.00 (0.00)	6.00 (0.00)	5.00 (0.63)	5.67 (0.59)
Tier 2
Acceptability	6.00 (0.00)	5.00 (0.00)	4.50 (0.71)	5.17 (0.75)
Understanding	6.00 (0.00)	5.00 (0.00)	5.00 (0.00)	5.33 (0.58)
Feasibility	6.00 (0.00)	6.00 (0.00)	5.00 (0.00)	5.67 (0.58)
Tier 3
Acceptability	6.00 (0.00)	---	---	6.00 (0.00)
Understanding	6.00 (0.00)	---	---	6.00 (0.00)
Feasibility	6.00 (0.00)	---	---	6.00 (0.00)

Julia

During Tier 1 (i.e., baseline), Julia’s mean rate of BSP was low and stable, with a mean of .08 (range: 0–.15). Upon implementation of Tier 2, Julia’s rate of BSP slightly increased, but remained below the predetermined BSP criterion with a mean of .24 (range: .15–.3). During Tier 3, rate of BSP immediately increased to or above the predetermined criterion with some variability with a mean of .73 (range: .5–1.15). Following the transition from Tier 3 back to Tier 2, Julia’s rate of BSP remained above the predetermined criterion, albeit with some slight variability, with a mean of .79 (range: .5–.93). However, the last 3 data points during Tier 2 consistently remained at high levels above the predetermined criterion, and thus a phase change decision was made. In general, during maintenance, Julia’s rate of BSP was stable and above the predetermined BSP criterion, except for a single data point, with a mean of .63 (range: .25–.8). Data on Julia’s rate of BSP from Tier 3 to maintenance included no overlap with baseline data, with only 2 data points falling below the predetermined BSP criterion, suggesting a strong functional relation between tiered training and increased rate of BSP. Further collection of data during the 1-month follow-up for Julia was planned, however, data collection was abruptly terminated due to the global COVID-19 pandemic.

Jessica

During Tier 1, Jessica’s rate of BSP was low and stable, with a mean of .02 (range: 0–.05). During implementation of Tier 2, there was an immediate and stable increase in BSP, with a mean of .64 (range: .5–.9). During maintenance, BSP remained stable with a slight increasing trend and above the predetermined BSP criterion with a mean of .74 (range: .6–.9). At the 1-month follow-up, Jessica’s rate of BSP initially decreased, but eventually increased and stabilized at or above the predetermined BSP criterion, with a mean of .61 (range: .3–.9). At the end of 1-month follow-up, a decreasing trend was observed. Further data collection until stability was obtained would have occurred, however, data collection was abruptly terminated due to the global COVID-19 pandemic. Data on Jessica’s rate of BSP from Tier 2 to 1-month follow-up did not have any overlap with baseline data, with only 1 data point falling below the predetermined BSP criterion, suggesting a strong functional relationship between tiered training and increased rate of BSP.

Erica

During baseline, Erica’s rate of BSP was consistently low and stable, with a mean of .03 (range: 0–.05). During Tier 2, there was an immediate increase in Erica’s rate of BSP, with a mean of .92 (range: .55–1.35), albeit with variable data. However, the last 3 data points during Tier 2 demonstrated an increasing trend above the predetermined criterion, and thus a phase change decision was made. In maintenance, Erica’s rate of BSP remained stable and above the predetermined BSP criterion, with a mean of .71 (range: .6–.8). At the end of maintenance, a decreasing trend was observed. Further data collection until stability was obtained would have occurred, however, data collection was abruptly terminated due to the global COVID-19 pandemic. Data on Erica’s rate of BSP from Tier 2 to maintenance had no overlap with baseline data and no data points fell below the predetermined BSP criterion, suggesting a strong functional relation between tiered training and increased rate of BSP.

Tau U Effect Sizes

Based on effect size criteria established by Vannest and Ninci (2015), Tau U effect size scores indicated that tiered training resulted in very large effects for rate of BSP for Julia (0.99), Jessica (1.00), and Erica (1.00). This provides further evidence of a functional relation between tiered training and student clinicians’ rate of BSP.

Social Validity

Overall, student clinicians responded favorable to most items on the URP-IR. The mean student clinicians’ ratings of Tier 2 was 4.73. Julia was the only student clinician to receive Tier 3 training, and her mean rating of Tier 3 training was 5. These data indicate that student clinicians found procedures within the tiered training process to be highly acceptable, understandable, and feasible. The mean student clinicians’ ratings of BSP was 4.9, indicating strong acceptability of BSP as a behavior-change procedure.

Discussion

Several strategies have been evaluated and found to be effective for improving preservice professionals’ use of evidence-based behavior management techniques (Ennis et al., 2020; Markelz et al., 2018; McLeod et al., 2019; Rathel et al., 2008). However, there is limited research examining these training strategies to improve preservice clinicians’ use of behavior management skills. Providing gradually intensified training to student clinicians’ if they fail to respond to low intensity strategies may be an efficient approach to improving their mastery of foundational behavior management skills. The purpose of this study was to examine the effectiveness of a tiered training model for increasing student clinicians’ rate of BSP during ongoing sessions with child clients.

In regard to the first research question, visual and effect size analyses indicated that all three student clinicians increased their rate of BSP above baseline levels in response to tiered training. Jessica and Erica increased their rate of BSP during Tier 2 (i.e., digital performance feedback) and did not require Tier 3 training supports. Furthermore, in general both participants maintained their rate of BSP during maintenance, with Jessica maintaining her rate of BSP at the 1-month follow-up. Julia did not initially respond to Tier 2, and therefore required Tier 3 training. During Tier 3 training, rate of BSP increased and Julia was transitioned back into Tier 2 and then maintenance. Visual analysis indicated that Julia maintained her rate of BSP after transitioning through the entire tiered training model.

A possible explanation for Julia’s maintained use of BSP is due to a carryover effect when transitioning from Tier 3 to Tier 2. That is, Julia may have continued to deliver BSP at high rates during Tier 2 due to intervention effects of Tier 3 continuing to affect behavior. Although carryover effects are often a threat to internal validity in single-case design research, these effects were intentional for this study. That is, the purpose of transitioning from higher tiers to lowers tiers was to gradually fade training supports in hopes that the effects of higher tiers persisted during tiers with less intensive supports. Results of this study indicated that a carryover effect may have indeed been present, which resulted in Julia maintaining her rates of BSP during the second implementation of Tier 2 and in the absence of supports (i.e., maintenance).

Results of this study are commensurate with previous research, in that participants improved their use of behavior management strategies, whereas some trainees required more intensive training supports (e.g., LaBrot et al., 2020b; Fallon et al., 2018; Myers et al., 2011; Sanetti & Collier-Meek, 2015). LaBrot, Dufrene, Whipple, et al. (2020b) found that teachers who made the transition from Tier 1 to Tier 2 training successfully maintained their rates of BSP following removal of training, but the only teacher who received Tier 3 supports was unable to be sent back to Tier 2 due to the conclusion of data collection procedures. Likewise, Myers et al. (2011) found that two teachers could successfully make the transition from Tier 3 to Tier 2 and maintain their rates of praise following termination of tiered training, albeit with variable results. Results of this study extend the tiered training literature in two ways. First, this study provides additional evidence for the effectiveness of a tiered training model for improving student clinicians’ behavior management skills. Second, this study demonstrates that trainees who experience multiple tiers of training can be successfully transitioned through each tier and maintain their use of behavior management techniques following termination of tiered training.

The third and fourth research questions investigated whether student clinicians rated tiered training as a socially valid training model and behavior change technique, respectively. Data indicated that student clinicians found training procedures in Tiers 2 and 3 (Julia only) to be overall highly acceptable, understandable, and feasible. This is consistent with previous research finding tiered consultation and training to be a socially valid training modality (LaBrot et al., 2020b; Thompson et al., 2012). Furthermore, data indicated that all three student clinicians rated BSP as a socially valid behavior change technique. This is also commensurate with previous research that has found BSP to be a socially valid behavior change procedure (LaBrot et al., 2020a, 2020b).

In addition to addressing the primary research questions, this study offers additional contributions to the supervision and training literature. Namely, this study demonstrates the effectiveness of a tiered training model for improving preservice clinicians’ in-session behavior management techniques. Some research has examined training strategies to improve preservice clinicians’ clinical skills, such as performance feedback to increase marriage and family therapy graduate students’ use of supportive statements (Gallant et al., 1991); immediate and live supervision to teach psychiatry residents to implement dialectical behavior therapy (Carmel et al., 2016); and bug-in-the-ear coaching to train preservice behavior analysts to implement functional communication training (Artman-Meeker et al., 2017). However, the literature is scant of studies examining strategies to train preservice clinicians to implement the same behavior management techniques they are expected to teach parents and teachers (e.g., Jones & Eddy, 2017). Therefore, this study extends the supervision and training literature by demonstrating the effectiveness of a tiered training model to improve student clinicians’ relevant clinical skills.

Training student clinicians to implement behavior management techniques with integrity may be especially useful in preparing them to conduct assessments (Jones & Eddy, 2017) and implement one-to-one clinical services with children who display disruptive behavior in session. A tiered model of training may be a beneficial approach to training and supervision as results of this study indicated that student clinicians increased and maintained their rate of BSP, with only one student requiring all three tiers. This may suggest that a tiered training model also allows for efficiency in supervision and training. That is, student clinicians’ skills in conducting assessments, implementing interventions, and managing in-session behavior often vary from student to student. A tiered training approach may allow supervisors to allocate more intensive training and supervision efforts to student clinicians who demonstrate a greater need for clinical skill development, while allowing for less intensive supervision for students with less training needs.

This study also incorporated the use of technology and systematically collected outcome data on student clinicians’ outcomes. In a review of the supervision of school psychology graduate student literature, Newman et al. (2019) found that there is a critical need for supervision research to evaluate technology as a supervisory support and collect supervisee outcome data. Therefore, this study extends the school psychology training and supervision literature in that it utilized technology (i.e., digital performance feedback via emails and tactile prompts via interval timers) to promote student clinicians’ outcomes and systematically collected outcome data on rate of BSP. Given the results of this study, a tiered training model of supervision aligns well with the Professional Standards of the National Association of School Psychologists’s (2020) guidelines for supervision as well as American Psychological Association’s Guidelines for Clinical Supervision (2014), in that tiered training included regular progress monitoring of student clinicians’ outcomes with clear, timely, and objective performance feedback. As a result, supervisors of preservice clinicians should consider this model of training and supervision given the promising results of this study and its alignment with professional organizations’ standards of supervision.

Finally, this study evaluated a tiered training model for increasing student clinicians’ rates of BSP in the context of one-to-one sessions with clients. Previous research has evaluated tiered training approaches for increasing educators’ rates of BSP in classroom settings (e.g., LaBrot et al., 2020b; Myers et al., 2011). Delivering BSP in a classroom potentially offers more opportunities for a trainee to deliver BSP given that there are multiple targets eligible to be praised. Therefore, the predetermined BSP criterion chosen for this study (i.e., .5 BSP statements per minute; LaBrot et al., 2016, 2020a) that was based on classroom research may not necessarily be an appropriate rate for one-to-one contexts as it has the potential to interfere with ongoing clinical activities. However, .5 BSP statements per min only constitutes the delivery of BSP, on average, once every 2 min, which may not be enough to interfere with clinical activities. Rather, providing BSP for effort and responding may result in clients’ improved engagement with assessment and intervention activities (e.g., Royer et al., 2019). Further, student clinicians were trained to deliver BSP statements to their clients contingent upon appropriate behaviors that were expected to occur during one-to-one sessions (e.g., reading passages out loud, verbally responding to problems presented on flashcards). This is consistent with explicit instruction methodology, in which feedback includes praise that is contingently delivered, specific, provided for performance, focused on effort, based upon a client’s individual improvement, positively stated, and unobtrusive (Archer & Hughes, 2011). Therefore, this study extends the literature in that a tiered training model was used to train student clinicians to increase their rates of BSP in a one-to-one clinical context. However, future research would benefit from examining whether a BSP rate of .5 statements per min is indeed beneficial, or intrusive, in one-to-one contexts.

Limitations and Future Directions

Although this study demonstrated the effectiveness of a tiered training model to increase student clinicians’ rate of BSP, it is not without limitations. First, this study only included three student clinicians seeing clients in the context of one-to-one sessions. So, it is unclear if a tiered training model would generalize to other preservice clinicians’ behavior management skills in other settings (e.g., school-based practicum placements). Future research should seek to replicate this study to determine whether a tiered training model is effective for other student clinicians in other settings. Second, this study did not collect data on child outcomes. Audio recordings used for data collection in this study made it difficult to operationalize and reliably collect data on children’s disruptive and appropriately engaged behavior. All three student clinicians reported that they perceived their client’s behavior improved as a function of increased BSP. Nevertheless, future research should seek to collect child data to determine whether tiered training results in improved child outcomes. Regarding the use of audio recorders to collect data, a possible third limitation is that use of audio recordings may have resulted in collecting BSP that was inappropriately delivered. When listening to audio files, researchers attempted to only code instances of BSP that were clearly contingent upon clients’ appropriate behavior (e.g., reading 1-min passages aloud, verbally responding to flashcard prompts, answering student clinicians’ questions). However, it is possible that student clinicians may have delivered BSP that was not necessarily contingent (e.g., providing BSP to a client for effort when the client was not putting forth their best effort). This is somewhat commensurate with previous research in that procedural-integrity data on whether BSP is delivered contingently or correctly is not always collected (e.g., Duchaine et al., 2011; Eaves et al., 2020; Hawkins & Heflin, 2011; LaBrot et al., 2016). Therefore, given this limitation, future studies should seek to utilize direct observation or visual recordings of clients’ behaviors as well as collect procedural-integrity data on BSP delivery to ensure that BSP is indeed delivered appropriately (i.e., contingently upon a specific appropriate behavior).

A possible fourth limitation involves the delivery of extra credit to student clinicians’ contingent upon study participation. That is, student clinicians may have increased their use of BSP to earn their extra credit. To decrease the possibility of this limitation, we emphasized to student clinicians that extra credit was contingent upon participation and not contingent upon a specific performance. Furthermore, although student clinicians were indeed aware of the need to increase their rates of BSP due to explicit feedback during DPF procedures, they were not made aware that BSP was the dependent variable of interest. In addition, student clinicians received additional feedback in the context of practicum supervision (e.g., on academic assessment and intervention procedures) that likely masked the study’s dependent variable. This may be evident, given that Julia did not initially increase her rates of BSP. Nevertheless, offering extra credit could have motivated student clinicians to increase their rates of BSP. Future research should seek to replicate this study without offering extra credit to minimize the possibility of another variable contributing to increased rates of BSP.

Finally, 1-month follow-up data were only collected for Jessica, so it is difficult to determine whether the other two participants would have maintained their rate of BSP during long-term follow-up. One-month follow-up data were not collected for Julia and Erica because clinic and data collection procedures were abruptly terminated due to the COVID-19 global pandemic. Likewise, Jessica’s 1-month follow-up data and Erica’s maintenance data included decreasing trends that could not be stabilized due to the abrupt conclusion of data collection. Future replications of this research should aim to collect long-term follow-up data (e.g., 1-month, 2-month) with stabilized data paths to determine whether tiered training results in student clinicians’ long-term maintenance of evidence-based behavior management strategies.

Conclusion

This study provided preliminary evidence for the effectiveness of a tiered training model for increasing student clinicians’ rate of BSP during one-to-one clinic sessions with child clients. Tiered training has an emerging literature base that demonstrates promising outcomes. Therefore, we encourage cautious optimism in the adoption and implementation of a tiered training model of training and supervision for preparing preservice student clinicians in evidence-based behavior management techniques. However, future research to replicate the findings of this study is certainly warranted and strongly encouraged.

	Steps: Tier 1	Yes	No
1	Student clinicians attended small-group didactic training.
2	Researcher described and provided rationale for BSP.
3	Researcher modeled BSP three times.
	Number of steps completed:	/3
	Percentage of steps completed:

	Steps: Tier 2	Yes	No	N/A
1	Researcher sent an email to student clinician that included data on rate of BSP.
2	The student clinician replied to the email confirming they read the feedback and viewed the video model.
	Number of steps completed:	/1 or /2
	Percentage of steps completed:

	Steps: Tier 3	Yes	No	N/A
1	Researcher sent an email to student clinician that included data on rate of BSP.
2	The student clinician replied to the email confirming they read the feedback and viewed the video model.
3	Gym Boss Interval Timer Stopwatch was on and set to vibrate every minute.
4	The student clinician was provided with the Gym Boss Interval Timer Stopwatch before their session.
5	The student clinician wore the Gym Boss Interval Timer during their session.
	Number of steps completed:	/4 or /5
	Percentage of steps completed: