A Cultural Generalization: An Effective Training for Staff Integrity on DTT in the Application of the PEAK in Brazil.

Promoting the Emergence of Advanced Knowledge (PEAK) direct training is an evidence-based curriculum with extensive research. Treatment integrity is one of the critical components of an effective treatment. In developing countries where training opportunities are scarce, providing precise training poses a challenge. The current study evaluated the effectiveness of behavioral skills training (BST) role-play to train three staff members to implement the PEAK-DT language curriculum, using discrete trial teaching (DTT) with three learners with autism spectrum disorder (ASD). Not only did the staff members implement the programs with high treatment integrity, but the learners also improved their performance on targeted skills. In addition, this study considered maintenance and generalization of the skills learned. Overall, data show that the BST role-play was successful as the staff members enhanced the skills of three learners with autism. Supplementary Information: The online version contains supplementary material available at 10.1007/s40617-022-00745-1. © Association for Behavior Analysis International 2022, Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Promoting the Emergence of Advanced Knowledge (PEAK) training system is a protocol that can benefit clinicians around the world in the ability to assess and teach the critical language and learning skills necessary for success (Dixon, 2014). The PEAK-Direct Training (PEAK-DT) has 184 specific programs, based on Skinner’s (1957) analysis of language as verbal behavior. The PEAK content is divided into six main skill domains: foundational learning, perceptual learning, verbal comprehension, verbal reasoning, memory, and mathematical skills (Rowsey et al. 2015). Although research has supported the effectiveness of the PEAK (Dixon et al., 2017; Dunkel-Jackson & Dixon, 2018; Dixon, Belisle, Stanley, & Rowsey, 2018), the program’s success rests on staff implementation of the protocol. The PEAK curriculum was designed to be user-friendly for parents, teachers, and direct-care staff, in addition to behavior analysts and other trained professionals, however training is required to establish the proper instructional skills to teach each of the skill domains (Hahs & Jarynowski, 2019).

The Peak-DT provides an assessment of language skills and prescribed teaching programs based on the learner’s needs (Belisle et al., 2016). Accurate implementation of both the PEAK-DT assessment and resulting teaching protocols are priorities to achieve targeted client outcomes. Therefore, specific training is required to establish the proper instructional skills (Belisle et al., 2016).

Behavioral skills training (BST) is an established, evidence-based staff training methodology (Sarokoff & Sturmey, 2004; Jimenez-Gomez, McGarry, Crochet, & Chong, 2019). BST consists of instructions, modeling, rehearsal, and feedback (Himle et al., 2004). Clayton and Headley (2019) targeted the implementation of discrete trial teaching (DTT) by paraprofessionals working with elementary school children with autism. Due to the high turnover of staff members at school, and a lack of experience in teaching strategies such as DTT, an effective training program is highly valued with a short amount of time. With no prior experience implementing DTT, three paraprofessionals increased their accuracy of implementation on DTT from an average of 64% accuracy during baseline to an average of 97% accuracy after BST.

Belisle et al. (2016) trained three direct-care staff via a multiple probe design. Two of the staff had no prior experience in discrete trial training. The setting was a self-contained school for children with autism spectrum disorder (ASD). Staff implementation of PEAK training improved from an average of 64% in baseline (mean range: 52%–78%) to 100% for all staff after in-situ BST and maintained at this level for weeks after. Corresponding student performance across skills varied. Although one student showed increases across all targeted skills, another student only increased accuracy in one skill, and a third student only showed increases in maintenance after decreases in performance during the BST training phase. Belisle et al. (2016) did not include the duration of the maintenance phase, IOA on actual learner performance, and generalization of staff implemented DTT across programs. Hahs and Jarynowski (2019) examined a workshop-based approach in a school setting with six paraprofessionals in a nonconcurrent multiple probe design. This study extended the PEAK literature base by elongating the maintenance period and ensuring interobserver agreement (IOA) data was collected for both students and staff behavior. This study showed increases similar to Belisle et al. (2016) in staff implementation of PEAK curriculum. Increases, from 52% average accuracy in baseline (mean range: 36%–88%) to 89% average accuracy (mean range: 80%–96%) following BST, were documented. In addition, skill maintenance persisted above baseline levels for 8 weeks, and learner performance generally followed the patterns of staff implementation. However, both studies failed to assess the generalization of skills (i.e., would BST on specific PEAK skills generalize to other skills in the curriculum?).

One significant limitation in the PEAK literature is that a vast majority of research conducted on the PEAK curriculum was conducted in the United States. In addition to the importance of replication of studies in general (Tincani & Travers, 2019), many have stressed the importance of international and cross-cultural replications of behavior-analytic studies (Malott, 2004; Higbee et al., 2016; Sivaraman & Fahmie, 2020). According to the PEAK website (www.peak2aba.com), a total of 38 articles have explored the PEAK Curriculum. All of these studies were conducted in the United States, thus, further exploration on the cultural generality of the PEAK curriculum is warranted.

Applied Behavior Analysis in Brazil

Brazil is a country with an interesting and complicated history in behavior analysis. In the 1960s, Fred Keller brought the experimental analysis of behavior as a Fulbright scholar from Columbia College to Brazil’s Universidade de São Paulo (Todorov, 2006). Since then, the experimental analysis of behavior has thrived in university psychology courses but applied behavior analysis as a field of study and practice has not grown equivalently. In Brazil, as of February 20, 2022, there are 34 board certified behavior analysts (BCBA), 3 board certified assistant behavior analysts (BCaBA), and 4 registered behavior technicians (RBT) in a country of over 214 million people (Behavior Analysis Certification Board [BACB], 2022; IBGE, n.d.). Furthermore, ABA is unknown, misconceived, and distrusted by many Brazilians, and competes with psychoanalysis and a myriad of nonevidence-based beliefs and approaches in the area of autism treatment (Higbee et al., 2016; Bueno dos Santos, 2019). Brazilian state governments are beginning to address autism services, and thus the demand for well-trained ABA practitioners providing quality services for children and adults with disabilities is increasing. With few credentialed ABA professionals in Brazil, the need for evidence-based and user-friendly curricula, and efficient and effective teaching methods is needed for training staff members to effectively serve their clients. This study sought to evaluate the effects of BST on implementation of the PEAK training system in an ABA clinic in Brazil, replicating the Belisle et al. (2016) and Hahs and Jarynowski (2019) studies while adding a generalization probe in order to determine if skills would transfer to new protocols.

Method

Three adults staff members were included in the study who had no more than 4 months of ABA practice experience, had not been introduced to PEAK, and did not have previous experience in implementing DTT. The three staff members were each paired with a child diagnosed with ASD between the ages of 6 and 12 years old, referred to as “learners.” The learners had years of prior ABA services based on different assessments and curricula. The behavior analyst supervising each learner’s services had conducted the PEAK-DT assessment and identified a common deficit in targets 3C-Object permanence, 6C-Imitate drawing basic shapes, and 11O-Named money. In addition, staff members and learners were Portuguese speakers, therefore the PEAK-DT (i.e., Portuguese translated), assessment and curriculum were used in this study (Dixon, 2014/n.d.).

Sam, a 25-year-old with a bachelor’s degree in psychology, was paired with Luiz during play 2 weeks before this study. Luiz was a 6-year-old boy diagnosed with ASD whose method of speaking was vocal. He spoke in 5–6-word sentences mainly to request for items and activities. He identified items receptively for several hundred common items and sorted items into various categories and selected items on the basis of their category. He was able to fill-in phrases and answer a small number of “wh” questions (e.g., “what is your name?”).

Anita, an 18-year-old student of psychology, was paired with Jean during play 3 days before this study. Jean was a 6-year-old boy diagnosed with ASD whose method of speaking was also vocal. He spoke in 3–4-word sentences to ask for items and activities, followed a few simple instructions, was able to name common items, imitated gross motor skills, and answered questions using two words.

Pamela, a 24-year-old occupational therapist was paired with Nei during play 3 weeks before this study. Nei was a 12-year-old boy diagnosed with ASD whose method of speaking was vocal, with occasional support of writing what he wants on a notebook. He spoke in 1–2-word sentences, primarily to asking for items and activities. He identified items receptively for common items, was able to follow one-step instructions, imitated gross and fine motor skills. He was able to read and copy text (Table 1).

Table 1

Diagnosis, Age, and Staff and Learner.

Staff Member	Age	Learner	Age	Diagnosis
Sam	25 years	Luiz	6 years	Autism
Anita	18 years	Jean	6 years	Autism
Pamela	24 years	Nei	12 years	Autism

Setting

This study was conducted in a center for autism treatment that provides ABA services in Brazil. All sessions related to training were conducted in therapy rooms (e.g., baseline and staff training). The therapy rooms were approximately 4 m x 5 m and included a table and 2 chairs.

Materials

This study required access to a PEAK curriculum binder in Portuguese for each learner which included the PEAK assessment, paper data sheet, pen, the instructional protocol, and the appropriate stimuli to run DTT. The selected programs were PEAK-DT 3C: Receptive Language of Object Performance, PEAK-DT 6C: Motor Imitation of Basic Shapes, and PEAK-DT 11O: Named Money. Program 3C required two to three plastics cups, and a small item to hide below one of the cups. Program 6C required blank papers and a pencil. Program 11O required money bills as 2, 5, 10, 20, 50, and 100 Brazilian Reais, and US$5 and $10. An iPhone as video recorder on a tripod was used to record all sessions in order to have a permanent product of this study. Also, each program used a data sheet recommended by the PEAK, including the arranging by trial number, and with a score metric 10-point PEAK scale regarding level of prompts used. The data sheet was translated into Portuguese.

Response Measurement and IOA

The dependent variable for staff performance was the percent correct implementation of DTT based on the specific instructional program. For a well-delivered trial, regardless of program, staff were required to implement up to eight DTT steps. The assessment of staff implementation integrity through a standard checklist for integrity (SCI) to eight steps of DTT in Appendix, during 10 trials block per program, scoring implemented correctly or incorrectly. The eight steps of DTT are (1) materials, stimuli, and arrange to teach are prepared; (2) capture learner’s attention and motivation prior to the trial; (3) SD is clear, and objective; (4) allows 3 s for a response; (5) provides reinforcement if correct; (6) if incorrect, represent the SD and provides the appropriate prompt; (7) appropriate magnitude of reinforcement for the trial; and (8) record in data collection. During the trials, some of the steps were not necessary to implementation, e.g., when a correct response occurred the error correction step was skipped and then was not considered into the percent accuracy. Percent accuracy was calculated by totaling the number of trials correctly performed by staff divided by the number of total steps performed and multiplied by 100.

The dependent variable for learner performance on the PEAK program was percent correct responding. The learner’s performance for each targeted program was calculated as percent of correct responses during 10 trials per session, across the PEAK data collection method. This is a discrete categorization system: 10 points means independent response, 8 points for a response with 1 single prompt, 4 points if the learner required 2 prompts, 2 points for multiple prompts or error correction with most intrusive prompt, and 0 when no response occurred after multiple prompts. The total points earned after 10 trials is divided by the maximum points possible during 10 trials (i.e., 100 points), multiplied by 100, to determine the percent correct.

All sessions were video recorded, and interobserver agreement (IOA) data were generated from the recordings. The IOA for staff implementation was assessed by an observer for 33.33% of baseline, treatment, and of maintenance sessions, and 50% of generalization sessions, resulting in a total of 37.50% across all conditions. The trial-by-trial IOA method was used, dividing the number of step agreements by the number of all steps performed, then multiplying by 100, to express as a percentage (Reed & Azulay, 2011). Resulting IOA was 92.15%. Learner performance IOA was evaluated during the same sessions as staff implementation IOA, also calculated based on the data sheet and performance along with 36.50% across all conditions, and 100% agreement was achieved. The number of agreements dived by the number of all trials, then multiplied by 100, as a percentage (Reed & Azulay, 2011).

Experimental Design

A concurrent multiple-probe design evaluated the staff integrity performance on DTT and application of PEAK programs. The multiple probe design included baseline, intervention, maintenance, and generalization conditions (Horner & Baer, 1978).

Procedures

Baseline

The three staff members attended a meeting about general information about PEAK and then asked to read the PEAK-DT module pages 9–29, introducing PEAK. The programs and schedules were delivered to staff, allowing general questions about selected programs. Each learner had their own PEAK binder, containing an initial assessment and history of each learner, and PEAK data collections as well. Therefore, staff members received initial information to conduct the baseline. A session was scheduled for each dyad, to implement PEAK programs starting the first program 3C-object permanence, then the second program 6C-imitate drawing basic shapes, and finally the third 11O-named money in the sequence, containing 10 trials each program for discrete trials, and data performance of staff members and learners were collected. All sessions were recorded. This approach remained constant throughout the baseline and did not include instruction, model, or performance feedback from the trainer. Each program lasted 3–12 min.

Behavior Skills Training

BST role-play was conducted via an individual session for each staff member in the training room. Unfortunately, due to COVID-19, the workshop format similar to Hahs and Jarynowski (2019) was avoided. The staff were encouraged to ask specific questions about the procedures regarding the discrete trials, to be answered by the trainer. BST incorporated four components to ensure the trainee was accurately implementing a skill. The four components were instruction, modeling, role-play, and performance feedback (Miltenberger et al., 2004). Each individual session lasted 25–35 min until trials achieved a mastery criterion of 90% integrity during role-play.

Instruction

Similar to baseline, staff were provided an opportunity to review the PEAK curriculum and then watch at least one recording of themselves implementing the DTT procedures during baseline. For each session, the staff member being trained was provided written and verbal instructions to implement DTT. Eight steps of DTT were described including: (1) materials, stimuli, and arrange to teach are prepared; (2) capture learner’s attention and motivation prior to the trial; (3) SD is clear, and objective; (4) allows 3 s for a response; (5) provides reinforcement if correct; (6) if incorrect, represent the SD and provides the appropriate prompt; (7) appropriate magnitude of reinforcement for the trial; and (8) record in data collection.

Modeling

During the modeling phase staff role-played the learner. The trainer modeled implementation of DTT across the three different proposed trials: (1) the trial with no response; (2) the trial with an incorrect response; and (3) the trial with a correct response. Modeling of each type of trial were provided once for each staff member.

Role-play

During the rehearsal phase, the trainer stepped into the learner role. Staff then practiced the three types of trials (i.e., trial with no response, incorrect response, and correct response). Data of staff’s implementation integrity of DTT were collected until mastery criteria of 90% were met for each proposed trial once.

Performance Feedback

The trainer provided immediate feedback to correct or deliver praise for to the trial implemented. The feedback to an incorrect trial consisted of the trainer delivering the consequence (e.g., “let’s try this trial again”), brief review of the instructions, modeling, and then an opportunity to the staff member to practice again. The positive feedback for a correct response consisted of the trainer delivering a praise (e.g., “Great performance”) then moving to another trial. The participant practiced until mastery criteria was met for all three types of trials.

DTT Implementation with Learners

Once staff members met the mastery criteria of 90% accuracy across all three trial types during BST role-play with the trainer, the staff were asked to implement the procedures with learners. That is, the next day after mastery, they started teaching their learner during DTT sessions. The programming for each learner focused on teaching the first program, then the second program, and finally the third in the sequence, containing 10 trials for each program following the eight steps of DTT. Each program lasted on average 2–9 min.

Maintenance

The maintenance phase was conducted 4 weeks after the last post-BST role-play session. During maintenance sessions, staff members implemented the procedures with learners similar to the DTT Implementation during baseline condition. Staff members did not have contact with any additional structured training during this phase. During the maintenance probes, staff were instructed to implement the programs in the same conditions that were in the baseline phase with 10 trials blocks for each of those three programs. Each program lasted on average 2–9 min.

Generalization

Twelve weeks post-BST role-play (e.g., 5 weeks after last maintenance session) a different program identified for all learners as skill deficient, PEAK-DT 10G: Receptively Label Community Helpers, was presented to the staff members for implementation without any instruction or training. Three probes of 10 trials each were conducted during this phase. Each program lasted on average 2–4 min.

Results

Data regarding the effectiveness of the BST implemented by staff focusing on treatment integrity, and also the learner’s responding performance are displayed in Fig. 1 and Table 2. A concurrent multiple-probe design was used to evaluate the staff percentage accuracy of implementing DTT and the learner performance on three different skill programs across baseline, intervention, maintenance, and generalization conditions.

Fig. 1

Concurrent multiple-probe design across subjects. Closed data points are the percentage of staff’s integrity of response. Open data points are the percentage of skill acquisition by the learner. The shapes represent different programs.

Table 2

Mean Treatment Integrity across Phases

Staff Member	% Baseline	% BST	%Maintenance	%Generalization
Sam	63	93	92	95
Anita	60	92	-	-
Pamela	59	94	95	95

Staff members mean performance across phases

Sam’s integrity of implementing DTT averaged 63% correct responding during baseline, increased to a mean level of 93% correct responding after BST, and maintained at a mean of 92% after 4 weeks of posttraining. After 5 weeks in the generalization phase the integrity maintained at 95% accuracy.

Anita’s integrity of implementing DTT averaged 60% correct responding during baseline, increased to a mean level of 92% correct responding after BST. The maintenance and generalization phases were not conducted, because Anita needed to resign her work contract during the COVID-19 pandemic.

Pamela’s integrity of implementing DTT averaged 59% correct responding during baseline, increased to a mean level of 94% correct responding after BST and maintained at a mean of 95%, after 4 weeks of posttraining. After 5 weeks in the generalization phase the integrity maintained 95% accurate.

The performance of correct responses performed by the learners are also displayed in Fig. 1 and Table 3. Luiz performed at a mean level of 55% correct on the PEAK programs during baseline. After staff correctly implemented DTT, Luiz’s correct responding across PEAK programs increased to 79% and maintained at 78% after 4 weeks of staff posttraining. During the generalization phase on the new program 10G, Luiz’s performance was 98%.

Table 3

Mean Performance across Phases

Learner	% Baseline	% BST	% Maintenance	%Generalization
Luiz	55	79	78	98
Jean	62	88	-	-
Nei	83	91	90	98

Learners mean performance across phases

Jean performed at a mean level of 62% correct responses on the PEAK programs during baseline. After staff correctly implemented DTT, Jean’s correct responding across PEAK programs increased to 88%. Maintenance and generalization phases were not conducted, because Anita resigned.

Finally, Nei performed at a mean level of 83% correct responses on the PEAK programs during baseline. After staff correctly implemented DTT, Nei’s correct responding across PEAK programs increased to 91%, and maintained it at 90%, after 4 weeks of staff posttraining. During the generalization phase on the new program 10G, Nei’s performance was 98%.

Discussion

Overall, the data demonstrated a functional relation across three staff members, using BST to significantly increase the accuracy of implementation for DTT on PEAK programs. Also, this training resulted in an improvement of language skills for all three learners with autism. The current study’s results replicate the previous findings of Belisle et al. (2016) and Hahs and Jarynowski (2019) while adding a generalization probe to transfer staff implementation skills to a new PEAK program.

Due to the recent shift in how behavior analysts are credentialed worldwide, several prominent scholars have called for evaluations of the application of behavior analysis across culturally diverse populations (Fong, Ficklin, & Lee, 2017; Sivaraman & Fahmie, 2020; Wang et al., 2019). Sivaraman and Fahmie (2020) suggested adaptations to evidence-based strategies may be warranted due to cultural values outside of the United States and Canada. Wang et al. (2019) called for practitioners and scientists alike to consider how strategies are implemented across culturally diverse populations. Fong et al. (2017) called for behavior analysts to practice in a culturally competent manner. The present study addressed each of these calls to action. The first author evaluated the need for adaptations of the PEAK curriculum and translations of BST procedures based on the cultural values of the participants (Sivaraman & Fahmie, 2020). Second, we aimed to evaluate the applicability and feasibility of both the PEAK curriculum and BST in Brazil. Thus, both the PEAK curriculum and training procedures, which were well-documented in English-speaking participants, were generalized to Portuguese-speaking participants (Wang et al., 2019). Finally, due to the success of the training procedures, this article may be able to increase some level of cultural competence for practitioners (Fong et al., 2017) by demonstrating the generality of procedures. To our knowledge, Higbee et al. (2016) is one of the only published studies on training strategies in Brazil.

Furthermore, the present study was a direct replication and extension of Hahs and Jarynowski (2019) and extended the comparative analysis between BST in-situ (Belisle et al., 2016) and BST role-play (Hahs & Jarynowski, 2019). However, it is difficult to assess the best approach between those two viable interventions. Across these three studies (present study included) both strategies lead to improved performance for both staff and learners. Our study aimed to close the gap in several limitations noted by these scholars. First, previous evaluations of BST in the implementation of the PEAK assessment (Belisle et al., 2016; Hahs & Jarynowski, 2019) failed to address the generalization of skills. The present study assessed generalization 5 weeks postmaintenance phase, with the results indicating that two staff generalized implementation of DTT to other PEAK skills with fidelity. Another limitation in Belisle et al. (2016) was the absence of IOA in learner performance. Both Hahs and Jarynowski (2019) and this current study addressed IOA of learner performance. The current study used video recordings of sessions for learner performance IOA data collection. This measure allowed for the validity of learning outcomes to be demonstrated. Although this study demonstrated the generality of ABA implementation, more precisely the PEAK curriculum and BST, and replicated previous findings to culturally and linguistically diverse populations, several limitations must be discussed. First, this was an applied project which focused on the treatment integrity of the staff members; however, procedural fidelity data were not collected on the researcher’s behavior. Future research in culturally and linguistically diverse populations must include procedural fidelity. Second, Anita withdrew due to COVID-19, this limitation affected the interpretation of findings in the maintenance and generalization phases due to the already limited number of participants. Future research should aim to include more participants in each study. Third, our study did not include social validity surveys for either the staff or learner participants. Considering socially significant behavior change and behavior change strategies is a foundational aspect of applied behavior analysis (Baer et al., 1968), future studies should include social validity surveys to evaluate the participant’s perspectives regarding training strategies for staff, teaching strategies for learners, and relationship impact for caregivers. Finally, this study was a direct replication and extension of (Hahs & Jarynowski, 2019), a debate to which strategy (BST in-situ and BST role-play) is more effective must continue. Therefore, future research should conduct a direct comparison between these two training methods, BST in-situ and the BST role-play, by using a group design embedded with single subject design. A more complex research design would allow within group and within subject comparisons, as well as a direct comparison across groups.

Finally, the current literature base for the PEAK curriculum was exclusively conducted in the United States (Featured Peer-Reviewed Publications, 2019). The present study’s results are important because they demonstrate the generality of the procedures across culturally and linguistically diverse populations. Furthermore, this study supports previous evidence of the effectiveness of BST to teach and maintain specific skills in both staff and learner skill repertoires (Hahs & Jarynowski, 2019). Our study successfully demonstrated the generality of ABA procedures (e.g., BST & PEAK Curriculum) in Brazil. It is hoped that it can further generate support of ABA in Brazil.