The Use of Instructive Feedback to Promote Emergent Tact and Intraverbal Control: A Replication.

Instructive feedback (IF) involves incorporating additional acquisition targets into skill-acquisition programs. A recent study by Frampton and Shillingsburg (2020) found that IF led to emergent verbal operants with two elementary-aged children diagnosed with autism spectrum disorder (ASD). The current study replicated Frampton and Shillingsburg with two children with ASD. Therapists conducted sessions of mastered listener-by-name trials (e.g., "Show me otter," with pictures of otter, dog, and elephant) with IF statements for features of the target stimuli (e.g., "It lives in rivers.") embedded during the consequence portion of the trial. We evaluated the acquisition of secondary targets and emergent responses using a concurrent multiple baseline across sets design. We observed increased correct responding for secondary targets and emergent responses for all three sets of stimuli with one participant. The other participant emitted correct responses for secondary targets and emergent operants with the first set but not with the other two sets of stimuli. Results suggested that IF can lead to emergent verbal operants, but the extent of emergence may be idiosyncratic. Supplementary Information: The online version contains supplementary material available at 10.1007/s40616-022-00171-y. © Association for Behavior Analysis International 2022.

Behavior-analytic interventions include a wide variety of acquisition targets because individuals with autism spectrum disorder (ASD) often need to learn many skills across multiple domains. Therefore, behavior analysts should design efficacious and efficient instruction (Reichow & Wolery, 2011). Selecting procedures that are both efficacious and efficient should be prioritized so that learners can amass robust repertoires in a given duration of intervention. In addition, employing efficient procedures produces a benefit for behavior analysts, teachers, and therapists because there will be more time to work on other important skills during an individual’s limited treatment time (Wolery et al., 1992).

Discrete-trial instruction (DTI), an effective teaching procedure, is used to teach a variety of skills; DTI comprises learn units typically referred to as trials (Smith, 2001). A typical DTI trial includes a discriminative stimulus (SD), learner response, prompt, and consequence. Instructive feedback (IF) can be incorporated into DTI, which expands a DTI trial by adding an additional instructional target(s) (i.e., secondary targets; the instructional target in a DTI trial is referred to as a primary target). Stimulus pairing is used to present secondary targets (Petursdottir et al., 2020), and presentations can be programmed in the antecedent or consequence portion of a DTI trial (see Nottingham et al., 2017 and Vladescu & Kodak, 2013 for variations in IF placement). Contrary to the requirement of a learner to respond to the primary target, the learner is not required to respond to the secondary target in IF. If the learner emits a response to the secondary target, this response is not reinforced; therefore, secondary targets are not taught directly (Vladescu & Kodak, 2013). In an example DTI trial with IF, the therapist shows the learner pictures of three animals (antecedent stimuli) and says, “Touch bear.” (conditional/sample stimulus). After the learner touches the picture of the bear (primary target, learner’s response), the therapist provides praise and an edible (consequence) and says, “Bears are mammals” (secondary target; IF). The learner is not required to echo “Bears are mammals.” Later, the therapist presents the antecedent verbal stimulus “Bears are…” in a probe to determine whether the learner has acquired the secondary target.

IF can be an effective and potentially efficient way to teach new skills. The inclusion of secondary targets may result in more rapid learning because learners can acquire twice (or more; Nottingham et al., 2017) the number of targets when IF is included (Shillingsburg et al., 2018). Acquisition of secondary targets with IF has been demonstrated in 1:1 and small-group instruction for several populations, including individuals of typical development and individuals with disabilities like intellectual disability, ASD, speech and language impairments, learning disabilities, developmental delays, and Down syndrome (e.g., Carroll & Kodak, 2015; Leaf et al., 2017; Nottingham et al., 2017; Nottingham et al., 2020; Tekin-Iftar, 2003; Tullis et al., 2022; Werts et al., 1995). Secondary target acquisition with IF has also been demonstrated with several procedural variations such as the number, location, and presentation schedule (Carroll & Kodak, 2015; Nottingham et al., 2017; Nottingham et al., 2020; Vladescu & Kodak, 2013).

Another avenue for increasing the number of targets acquired with IF involves arranging and assessing for emergent verbal operants. Emergent learning occurs when one acquires nontarget information for which there are no programmed consequences (Wolery et al., 1992). Specifically, learners may acquire responses to the secondary targets and also emit correct responses to related verbal operants that were not directly taught (e.g., intraverbals when tacts are the secondary target). Verbal operants are maintained by unique antecedent and consequence events, making them functionally independent (Skinner, 1957). As a result, teaching a response under one source of control, such as a tact (i.e., verbal behavior evoked by a nonverbal SD and maintained by generalized conditioned reinforcement; Skinner, 1957), does not necessarily result in the emergence of related verbal operants, such as intraverbals1 (i.e., verbal behavior evoked by a verbal SD without point-to-point correspondence and maintained by generalized conditioned reinforcement; Skinner, 1957). However, instructional arrangements may promote emergent verbal operants (Grow & Kodak, 2010).

Emergent responding following IF was evaluated in a recent study by Frampton and Shillingsburg (2020). The researchers embedded tacts of features (secondary targets) within mastered listener-discrimination trials (primary targets) and assessed whether intraverbals and reverse intraverbals emerged following IF. Frampton and Shillingsburg conducted the evaluation with two male children (aged 7 and 8 years) who were described as advanced Level-Three learners according to their scores on the Verbal Behavior-Milestones Assessment and Placement Program (VB-MAPP; Sundberg, 2008) and demonstrated bidirectional naming. The trials with IF were conducted in the following order: (a) the researcher delivered the primary target conditional stimulus ( e.g., “Show me judge.”) in the presence of a three-picture comparison array, (b) the learner selected the correct picture (i.e., S+; e.g., picture of the judge), (c) the researcher provided a reinforcer (e.g., praise and a token), and (d) the researcher provided the IF statement pertaining to a feature of the item (e.g., “She uses a gavel.”). After three sessions of IF, researchers conducted probes on listener-by-feature (e.g., “Who uses a gavel?” in the presence of the visual array), tact-by-feature (e.g., “What does this person use?” in the presence of the picture of the judge), Wh- intraverbals (e.g., “What does the judge use?” without any visual stimuli), and reverse intraverbals (e.g., “Who uses a gavel?” without any visual stimuli). Compared to baseline levels of responding, both learners emitted more correct responses across all operants following just three sessions with IF, and correct responses maintained when assessed approximately two weeks later.

The results of Frampton and Shillingsburg (2020) demonstrated that embedding IF could lead to the emergence of other verbal operants (i.e., Wh- intraverbals). The purpose of the current study was to systematically replicate Frampton and Shillingsburg with more learners with ASD. Our systematic replication of Frampton and Shillingsburg included several additions as we required attending to the visual stimulus during IF, measured echoics, and included fill-in intraverbals in probes. Increased levels of attending to the visual stimulus coupled with echoing a vocal IF stimulus may promote acquisition of secondary targets during IF instruction (Haq et al., 2017). Attending to the visual stimulus and emitting covert or overt echoic and self-echoic responses may create the conditions necessary for bidirectional naming (responding as a speaker following listener discrimination training and vice versa; Miguel, 2016), which could be important for acquisition of secondary and emergent targets with IF (Dass et al., 2018). Specifically, emergent tact control after IF delivery (i.e., stimulus pairing) may be a special subtype of bidirectional naming (Petursdottir et al., 2020, p. 194). Therefore, we included participants who demonstrated bidirectional naming, required attending to the visual stimulus prior to IF presentation, and measured the occurrence of overt echoics. Finally, we added fill-in intraverbals to probes because these responses typically develop before Wh- and reverse intraverbal statements (Sundberg & Sundberg, 2011).

Method

Participants and Setting

Two children with ASD, diagnosed by medical professionals not affiliated with the study, served as participants. At the time of the study, participants were receiving behavior-analytic services at a university-based autism center. Participants were recruited for the study based on treatment goals related to listener and speaker responses by feature, function, and class. The experimenters collaborated with the children’s Board Certified Behavior Analysts to confirm appropriateness of the goal and identify teaching targets. Parents provided consent for research during service delivery, which was approved by the institution’s review board for human subjects, and the current evaluation was approved by the autism center’s executive director. Research sessions were conducted in the participant’s designated therapy room (3.3 m x 2.4 m) at the center. Each therapy room included a table and two chairs along with instructional materials and toys.

Miguel was a 5-year-old Hispanic male who had been receiving applied behavior analytic intervention for 12 months, not including a 3.5-month interruption due to novel coronavirus (COVID-19) closure at the center when this study began. He attended half-day sessions two times per week. In addition to his in-clinic intervention services, Miguel attended a public school four days per week. His family spoke Spanish and English at home. All intervention services were conducted in English. He obtained standard scores of 79 and 85 on the Peabody Picture Vocabulary Test-Fourth Edition (PPVT-4; Dunn & Dunn, 2007) and the Expressive Vocabulary Test-Second Edition (EVT-2; Williams, 2007), respectively. Miguel’s responding on the VB-MAPP was in the Level Two range in the mand, echoic, tact, and intraverbal domains. He could emit spontaneous mands for items, tact at least 200 nouns or verbs, answer 12 Wh-questions, and echo a variety of sounds and words (a score of 100 out of 100 on the Early Echoics Skills Assessment [EESA]; Esch, 2008). He demonstrated bidirectional naming prior to the study. That is, he emitted tacts following listener discrimination training and engaged in correct selection responses to listener discriminations following tact training. Novel targets were taught as tacts and probed as listener discriminations and vice versa. Miguel emitted correct responses in the untrained modality in probes that did not include differential reinforcement.

Clare was a 4-year-old Eastern European American female who had been receiving applied behavior analytic intervention for 16 months at the center, not including a 3.5-month interruption due to the COVID-19 closure, when this study began. Due to COVID-19 capacity limits, Clare attended half-day sessions three times a week, but she transitioned to full-day sessions five days a week halfway through the study as restrictions were eased. In the home, Clare’s family spoke both their native European language and English. All of Clare’s intervention services were conducted in English. She obtained standard scores of 147 and 106 on PPVT-4 and the EVT-2, respectively. Clare’s responding on the VB-MAPP was in the Level Two range in the mand, echoic, tact, and intraverbal domains. She could emit spontaneous mands for items, tact at least 200 nouns or verbs, answer at least 25 different Wh-questions, and echo a variety of sounds and words (a score of 95 out of 100 on the EESA). She demonstrated bidirectional naming with the procedures described above for Miguel.

Materials and Target Selection

Materials included a 30 cm × 46 cm cardboard divider, printed data sheets, writing utensils, video camera and tripod, participants’ preferred tangibles, and 5 cm x 9 cm stimulus cards. Each set of stimulus cards consisted of three laminated, colored images of community helpers (Miguel; see Table 1) or animals (Clare; see Table 2) on a white background. Images were found via an internet search engine. Targets included stimuli that corresponded with goals of the participants’ clinical programming and were modeled after those selected by Frampton and Shillingsburg (2020). We selected visual stimuli to which the participant could respond to as a speaker and a listener (e.g., “Otter” in response to the SD and antecedent verbal stimulus “What is it?” and selected otter from an array in response to the conditional stimulus “Touch otter.”).

Table 1

Targets for sets 1–3 for Miguel

Listener-by-feature antecedent verbal stimuli were always presented with a three-stimulus array. Tact-by-feature antecedent verbal stimuli were always accompanied with a picture of the target stimulus

Table 2

Targets for sets 1–3 for Clare

Listener-by-feature antecedent verbal stimuli were always presented with a three-stimulus array. Tact-by-feature antecedent verbal stimuli were always accompanied with a picture of the target stimulus

The IF statements were features of each stimulus. We defined features as relative relations to the target picture (e.g., what the target animal ate, where it lived; Cooper et al., 2020). For each stimulus, we identified features that could not be observed in the picture (e.g., we did not include fur color as it could be observed, the picture of the dog did not include kibble; Frampton & Shillingsburg, 2020). One feature was selected per stimulus based on the participant’s responding during probes (described below). Each stimulus in a set had a feature that used a different carrier phrase (e.g., “It eats _.” “It lives in _.” “Its babies are _.”; Tables 1 and 2), and the carrier phrases were repeated across sets (e.g., three total “It eats _.” targets). To arrange stimuli in sets, we used a logical analysis (Cariveau et al., 2020; Wolery et al., 2014). Stimuli were arranged so that target names and IF statements included a similar number of syllables in each set (see Tables 1 and 2). We confirmed that participants could echo the features by conducting echoic probes with each vocal stimulus (Frampton & Shillingsburg, 2020). Visual images selected for each set were arranged similarly across sets (e.g., one animal in each set was facing forward, to the left, and to the right; community helpers holding items, etc.).

Response Measurement and Interobserver Agreement

The main dependent variable was the frequency of correct independent responses emitted during listener, tact, and intraverbal (fill-ins and Wh- questions) probes (Frampton & Shillingsburg, 2020). Across operants, a correct independent response was defined as the participant emitting a specific target response within 5 s of the antecedent verbal stimulus (see Table 3 for specific operational definitions). Correct responses could include repeating any portion of the antecedent verbal stimulus. An incorrect response was defined as the participant engaging in any response other than the target response or not engaging in a response within 5 s. Correct responses were summed for total frequency and divided by total number of opportunities to obtain a percentage. Sets were considered mastered if the participant emitted correct independent responses on at least 55% of trials within a probe session (i.e., at least 5/9 correct) across at least three of the following operants: listener-by-feature, tact-by-feature, intraverbal Wh- questions, and reverse intraverbals (fill-in intraverbals were excluded from the mastery criterion because they were not included in Frampton & Shillingsburg, 2020). The criterion for mastery was based on Frampton and Shillingsburg (2020), and it was designed to account for emergent responses tested under extinction conditions. We continued to collect data on correct independent responses following mastery of each set to assess responding across time.

Table 3

Operational definitions of correct and incorrect responses

Operant	Phase	Stimulus presentation	Operational definitions
			Correct	Incorrect
Identity matching	Pretest	Arrange three stimuli horizontally on the table behind a divider according to data sheet. Lift divider, say “Match.” and hand the sample picture to the participant	Placing the sample stimulus card on top of or in front of the corresponding card (S+) in an array of 3	Placing the sample stimulus card on top or in front of a card other than S+ (i.e., an S−) in the array or not placing the sample stimulus on any card in the array
Echoic	Pretest	No visual stimulus presented. Present antecedent verbal stimulus according to stimulus name or feature. (e.g., “Say, dog.”; “Say, It eats kibble.”)	Emitting a vocal response with point-to-point correspondence with the antecedent vocal stimulus	Emitting a vocal response that did not have point-to-point correspondence with the antecedent vocal stimulus or not engaging in a vocal response
Tact-by-name	Pretest	Hold one stimulus in front of participant at eye level. Say “What is it?”	Emitting a vocal response that corresponds with the name of the stimulus shown	Emitting a vocal response that does not correspond with the name of the stimulus shown or not emitting any vocal response
Listener-by-name	Pretest and Intervention	Arrange three stimuli horizontally on the table behind a divider according to data sheet. Lift divider and say, “Show me [name].” (e.g., “Show me dog.”)	Touching the S+ in an array of three	Touching an S−, touching two or more cards successively or simultaneously, or not touching any card
Listener-by-feature	Probes	Arrange three stimuli horizontally on the table behind a divider according to data sheet. Lift divider and say “Which [feature]?” (e.g., “Which eats kibble?”)	Touching the S+ in an array of three	Touching an S−, touching two or more cards successively or simultaneously, or not touching any card
Tact-by-feature	Probes	Hold one stimulus in front of participant at eye level. Present antecedent verbal stimulus according to feature. (e.g., picture of dog + “It eats ___.”)	Emitting a vocal response that corresponds with the feature of the stimulus shown	Emitting a vocal response that does not correspond with the feature of the stimulus shown or not engaging in a vocal response
Intraverbal fill-in	Probes	No visual stimulus presented. Present antecedent verbal stimulus according to feature as a fill-in statement. (e.g., “Dog eats __.”)	Emitting a vocal response that corresponds with the feature of the target stimulus	Emitting a vocal response that does not correspond with the feature of the stimulus named or not engaging in a vocal response
Intraverbal Wh-	Probes	No visual stimulus presented. Present antecedent verbal stimulus according to feature as a Wh-question. (e.g., “What does a dog eat?”)	Emitting a vocal response that corresponds with the feature of the target stimulus	Emitting a vocal response that does not correspond with the feature of the target stimulus or not engaging in a vocal response
Reverse intraverbal	Probes	No visual stimulus presented. Present antecedent verbal stimulus according to feature-name as a Wh-question (e.g., “Who eats kibble?”)	Emitting a vocal response that corresponds with the name of the target stimulus	Emitting a vocal response that does not correspond with the name of the target stimulus or not engaging in a vocal response

Correct and incorrect responses had to occur within 5 s of the antecedent verbal stimulus

In addition to the frequency of correct independent responses, we collected data on several other responses. Although the primary targets included in intervention were previously mastered, therapists collected data on participants’ responding to mastered targets during the intervention session. Independent correct and incorrect responses were defined similar to the listener-by-feature operant (Table 3); however, the antecedent verbal stimulus was the name of the stimulus (e.g., “Show me otter” rather than “Which lives in rivers?”). Prompted correct responses were defined as the participant imitating the therapist’s model of the correct response within 5 s. Prompted incorrect responses were defined as the participant failing to imitate the therapist’s model of the correct response, either because they selected an incorrect stimulus or because they did not respond, within 5 s. We also collected data on whether the participant echoed the IF statement during intervention trials. An echoic response was defined as a vocalization that had point-to-point correspondence with the antecedent verbal stimulus (Skinner, 1957), and could include all or some of the words in the IF statement. We recorded the occurrence and non-occurrence of echoics on each intervention trial.

A trained research assistant collected data on the participants’ responding from video for 34% of Miguel’s sessions and 40% of Clare’s sessions; data were collected throughout all phases of the study. An agreement was scored if both observers recorded the same response on a trial (e.g., both scored an independent correct response). A disagreement was scored if both observers scored a different response on a trial. We calculated interobserver agreement on a trial-by-trial basis by dividing the total number of agreements by the sum of agreements and disagreements and multiplying by 100 to obtain a percentage. Mean agreement for Miguel’s intervention sessions was 100% for independent correct responses, prompted correct responses, and echoic responses. Mean agreement for Clare’s intervention sessions was 98% (range, 78–100%) for independent correct responses, 91% (range, 0–100%; the lower bound of the range consisted of one session wherein one prompted response occurred, and the data collectors disagreed on whether it was correct) for prompted correct responses, and 97% (range, 89–100%) for echoic responses. Mean agreement for Miguel’s and Clare’s probe sessions was 98% (range, 78–100%) and 97% (range, 67–100%), respectively.

Independent Variable and Procedural Integrity

The independent variable in the current study was the inclusion of IF within mastered listener-by-feature trials. The IF statement included a feature of the target stimulus and did not include the name of the target stimulus (e.g., “It lives in rivers,” see Tables 1 and 2). Four, female graduate-student therapists implemented the procedures. The therapists were in their mid-twenties (M = 25; range, 24–26), all identified as White, three identified as Hispanic, and had 1.5 to 6 (M = 3.5) years of experience implementing behavior-analytic interventions with children with ASD.

A trained observer collected data on the therapist’s implementation of all components of the procedure with a checklist (supplementary information) across probe, pretest, and intervention sessions for 35% and 37% of Miguel’s and Clare’s sessions, respectively. Therapists were trained to run the procedure with integrity using written protocols, video models, and in-person role-play practice opportunities with feedback (i.e., Behavioral Skills Training). We calculated treatment integrity by dividing the total number of correct components implemented by the therapist by the total number of components per session and multiplying by 100 to obtain a percentage. Mean treatment integrity for intervention sessions was 89% (range, 70–100%) for Miguel and 95% (range, 78–100%) for Clare. Mean treatment integrity for probe sessions was 95% (range, 70–100%) for Miguel and 95% (range, 71–100%) for Clare. Two commission errors occurred in delivery of IF in Clare’s sessions wherein the therapist said the name of the stimulus (e.g., “Dog eats kibble” instead of “It eats kibble.”). None of the treatment integrity errors were errors in reinforcer delivery during probes.

We collected reliability data for procedural integrity for 34% and 33% of Miguel and Clare’s sessions across probe, pretest, and intervention. An agreement was scored if both observers recorded the same score for a component in the session. A disagreement was scored if both observers recorded a different score for a component in the session. Agreement was calculated by dividing the total number of agreements by the sum of agreements and disagreements and multiplying by 100. Mean agreement on procedural integrity was 96% (range, 75–100%) for both Miguel and Clare’s sessions.

Design

To evaluate whether instructive feedback led to emergent intraverbal responses, we used a concurrent multiple baseline design across sets. Baseline assessments were conducted with three sets of stimuli, and therapists measured the participants’ responding across operants. Then, therapists implemented one series of intervention with Set 1. Each intervention series consisted of three sessions (i.e., a total of nine exposures to each IF statement). Following one intervention series, therapists conducted probes to assess emergence across operants and sets. If emergence was not observed (i.e., at least 55% correct independent responses emitted for three operants [excluding fill-in intraverbals]), then the therapist conducted another intervention series with Set 1 before conducting more probes. Sets 2 and 3 remained in baseline conditions while Set 1 was in intervention. Once emergence was observed with Set 1, intervention began with Set 2. This process continued until all sets were exposed to intervention sessions (see supplementary information). Intervention was discontinued once twice the amount of intervention series with Set 1 had been conducted with Sets 2 or 3 and there was no increasing trend across any operant.

Procedure

We replicated the procedure by Frampton and Shillingsburg (2020) with one deviation: We provided 20-s access to preferred items rather than tokens because of the participants’ existing behavior-intervention plans. Preferred items were identified using a brief, daily, multiple-stimulus-without-replacement preference assessment (Carr et al., 2000). If the participants vocally selected an alternative item, therapists provided that item. Each session included three presentations of each target stimulus (i.e., nine trials) as well as one or two warm-up trials (described below). Probe and pretest sessions included trials of interspersed tasks resulting in a range of 12 to 14 trials per session.

Choice Trial

Before each session, we conducted a choice trial to identify a tangible item to deliver according to the reinforcement schedule. Therapists presented an array of three to five preferred items, pointed to each one of the items while providing a tact, and instructed the participant to “Pick one.” Once the participant selected one of the items (i.e., vocal mand, point, reach, or touch), the therapist said, “You can play with (item) after you do some work.” and removed all preferred items from the table to initiate warm-up trials.

Warm-up Trials

Each session began with one or two warm-up trials. Warm-up trials included high-probability tasks (e.g., motor imitation, echoics). General praise followed correct responses. If the participant engaged in an incorrect response, the therapist used the error-correction procedure (described below). Regardless of how the participant responded during the warm-up trials, the therapist moved onto the target task.

Interspersed-Task Trials

We interspersed trials of unrelated, high-probability tasks (e.g., motor imitation, listener discrimination with and without pictures, echoics, and intraverbals) approximately every three trials in pretest and probe sessions (described below). Therapists delivered general praise and 20-s access to a tangible for independent and prompted responses on interspersed trials. If the participant engaged in an incorrect response, the therapist used the error-correction procedure.

Error Correction

Error correction followed incorrect responses emitted during warm-up, interspersed-task, and mastered-listener trials (flowchart in supplementary information); no error correction followed incorrect responses on pretest nor probe trials. Following an incorrect response, the therapist re-presented the SD and immediately provided a model of the correct response. The therapist then re-presented the SD without a response prompt to give the participant 5 s to respond independently. Following a correct response, the therapist presented a distractor task (i.e., a high-probability response). If the participant engaged in an incorrect response during the distractor task, the therapist prompted the correct response. After the distractor task, the therapist again re-presented the SD. If the participant responded correctly, the therapist provided either general praise (warm-up trials) or general praise and access to a tangible item (interspersed-task and mastered-listener trials). If the participant engaged in an incorrect response, the therapist restarted and repeated the error-correction sequence until the participant engaged in an independent correct response to the SD.

Pretests

We evaluated prerequisite skills with all stimuli in each set (Frampton & Shillingsburg, 2020; Shillingsburg et al., 2018); skills included identity matching, echoics, listener-by-name discriminations, and tact-by-name responses. Sessions included three trials of each target stimulus, one or two warm-up trials, and three interspersed-task trials. Skills were tested in the following order: identity matching, echoics, listener-by-name, and tact-by-name. One skill was assessed with each set (e.g., identity matching with Sets 1, 2, and 3) before moving onto the next skill. The therapist presented antecedent stimuli according to descriptions in Table 3. The participant had 5 s to respond following each SD. The therapist did not provide any response prompts, and they provided a neutral statement following correct and incorrect responses (see Table 3 for operational definitions). To advance to probes, correct responses needed to occur on at least 89% of trials within a session across all skills with all sets (data available upon request).

Baseline and Emergence Probes

Baseline and probe sessions evaluated responding across listener-by-feature, tact-by-feature, name-feature intraverbal (fill-in statements and Wh- questions), and reverse intraverbal (Wh- questions) operants. Several trial-order versions for each probe type were created so that stimulus-presentation orders were semi-randomized across probe presentations (i.e., no stimulus occurred on more than two consecutive trials). Based on the procedures in Frampton and Shillingsburg (2020), skills were tested in a fixed order: listener-by-feature, tact-by-feature, intraverbal (fill-in statements and Wh-questions), and reverse intraverbals. One probe type was assessed with each set (e.g., listener-by-feature with Sets 1, 2, and 3) before moving onto the next type. No responses were prompted, and neutral statements were provided by the therapist (e.g., “Okay.,” “Alright.”) after each response regardless of whether it was correct or incorrect.

Instructive Feedback Intervention

The intervention sessions included mastered listener-by-name discriminations as the primary targets and feature tacts as the secondary targets (flowchart in supplementary information). These sessions included warm-up trials but no interspersed tasks.

Primary Targets

The procedure was identical to the listener-by name pretest trials (Table 3) except correct responses were reinforced. If the participant engaged in a correct response, the therapist provided general praise and 20-s access to a preferred item. If the participant responded incorrectly, the therapist used the error-correction procedure. Independent and prompted correct responses were followed by praise and 20-s access to preferred items.

Secondary Targets

Once the preferred item was delivered, the therapist presented the SD at the participant’s eye level while pointing to the stimulus. If the participant did not look at the SD within 5 s of its presentation, the therapist said “Look.” If another 5 s elapsed without attending to the picture, the therapist placed the visual stimulus in front of the preferred item until the participant looked at the picture. Once the participant looked at the picture, the therapist delivered the IF statement, which included the target feature of the stimulus (see Tables 1 and 2). The IF statement did not repeat the name of the stimulus (e.g., “It eats shrubs” instead of “Goat eats shrubs”). After the IF statement was delivered, the experimenter removed the picture after 1 s, removed the other pictures in the array, collected data, and engaged with the participant for the remainder of the time with the preferred item.

Procedural Modifications for Clare

Based on the participant’s responding during probe conditions, we modified the procedures to try to evoke correct responses in the presence of antecedent stimuli. These modifications occurred during probes only; the intervention sessions remained unchanged.

Removing Stimuli Following Incorrect Responses

After four intervention series with Set 1 (i.e., 12 sessions), we noticed Clare was engaging in unintelligible vocalizations or saying “okay” with a short latency for most probe trials across operants and sets. We hypothesized that the function of her short latency vocalizations was to remove the trial stimuli and end probe sessions (i.e., putative escape-maintained behavior). Therefore, we modified how therapists responded to incorrect responses during probes so that non-target vocalizations no longer resulted in immediate presentation of the neutral statement and termination of the trial. Specifically, all unintelligible vocalizations and “okay” resulted in the full 5-s response interval before the therapist provided a neutral statement and moved onto the next trial.

After four intervention series with Set 2 (i.e., 12 sessions), Clare’s responses during probes changed to repetitive, unrelated vocalizations (e.g., “Mommies, and daddies, and babies.”) with short latencies. We hypothesized that the change described above shifted responding from unintelligible vocalizations to the intelligible but repetitive vocalizations. Therefore, we modified the therapist’s responding so that every non-target vocalization resulted in the full 5-s response interval for all remaining probe sessions.

Interspersed-Task Ratio

After six intervention series with Set 3 (i.e., 18 sessions), Clare stopped emitting vocal responses on most probe trials. We became concerned that she was no longer responding due to the lean reinforcement schedule in place during probes compared to most of her intervention sessions (i.e., variable ratio 3 for probes compared to a fixed ratio 1 for intervention and other acquisition programs). Additionally, her responding suggested that removal of the instructional stimuli and shorter sessions may have been a more effective reinforcers than tangible items in the moment. Therefore, we increased the number of interspersed tasks, which resulted in access to more preferred tangibles within the session.

Differential Reinforcement

When we did not observe correct responding reach criterion for Sets 2 and 3 after eleven (i.e., 33 sessions) and ten (i.e., 30 sessions) intervention series, respectively, we decided to move from extinction conditions to differential reinforcement (Mitteer et al., 2020). If the responses had come under appropriate sources of control, we hypothesized that Clare was not emitting the target responses in probes because they did not contact differential reinforcement and the interspersed trials were highly discriminable. Therefore, with this modification, the therapist provided praise and 20-s access to a tangible item if Clare emitted a correct response during a probe trial.

Extended Response Interval

Clare was not emitting vocal responses reliably when we added differential reinforcement; therefore, we made a final modification to increase the response interval (Gorgan & Kodak, 2019). We hypothesized that an extended response interval would increase the probability of vocalizations, if they were acquired under relevant sources of control, to end the probe session more quickly. In addition, in other programs implemented in Clare’s comprehensive applied behavior analytic services, an extended response interval increased correct independent responses. With this modification, Clare had 10 s to respond.

Results

Figures 1 and 2 show correct responses emitted by Miguel and Clare, respectively, in probe sessions (see supplementary information for selection responses during mastered listener discrimination trials and echoics of the IF statements).

Fig. 1

Miguel’s responding during probes for emergence. The introduction of intervention is indicated by a solid line. Dotted lines represent an additional series of intervention. The horizontal, dashed line represents criterion levels. This figure is modeled after Tullis et al. (2021); bar graphs similar to those in Frampton and Shillingsburg (2020) are available in supplementary information

Fig. 2

Clare’s responding during probes for emergence. The introduction of intervention is noted by a solid line. Dotted lines represent an additional series of intervention. The horizontal, dashed line represents the criterion level. Modifications are noted with an asterisk. DR indicates differential reinforcement was in effect. This figure is modeled after Tullis et al. (2021); bar graphs similar to those in Frampton and Shillingsburg (2020) are available in supplementary information

Miguel

In the first baseline session (BL in Fig. 1) conducted with each set, Miguel responded correctly on one trial of intraverbal Wh- for Set 1, one trial of intraverbal fill-in and reverse intraverbal for Set 2, emitted correct selection responses at chance levels for listener-by-feature targets, and did not respond correctly to any other tact or intraverbal trials across sets. After introducing intervention with Set 1 (Fig. 1, top panel), Miguel emitted more correct responses for Set 1 across operants. Miguel’s correct responding increased across all operants and met criterion for listener-by-feature, tacts, and reverse intraverbals in two intervention series (i.e., 6 sessions, 18 exposures to each IF statement). Miguel emitted correct responses on an average of 89% of mastered listener-discrimination trials (range, 78–100%; see supplementary information), and he echoed nearly every IF presentation (M = 99%, range 89–100%).

Miguel began to emit more correct responses to listener-by-feature and reverse intraverbal stimuli during Set 2 baseline sessions after IF was introduced with Set 1 stimuli (Fig. 2, middle panel). Mastery-level responding was observed with listener-by-feature and reverse intraverbal stimuli in baseline. We introduced the IF intervention with Set 2. Correct responding to listener-by-feature remained high in probes, and the other operants increased with continued exposure to IF. After six intervention series with Set 2 (Fig. 1; i.e., 18 sessions, 54 exposures to each IF statement), Miguel’s responding met criterion for listener-by-feature, tacts, Wh-intraverbals and reverse intraverbals. Throughout intervention with Set 2, Miguel’s responding for mastered listener-by-name trials was high (M = 93%, range, 78–100%) with Set 2 stimuli. He emitted echoics for 100% of trials.

Miguel emitted some correct responses to tact-by-feature and listener-by-feature probes in baseline for Set 3, but his responses were below mastery level. After introducing intervention with Set 3 (Fig. 1, bottom panel), Miguel’s correct responding for this set met criterion across all five operants following one intervention series (i.e., 3 sessions; 9 exposures to each IF statement). Miguel’s responding for mastered listener-by-name trials for Set 3 averaged 96% (range, 78–100%), and he echoed 100% of the IF statements. Miguel’s correct responding for tact-by-feature and listener-by-feature in Set 1 remained above mastery during all maintenance probes, but his responding to intraverbal Wh-, intraverbal fill-in, and reverse intraverbal stimuli increased with repeated exposure to maintenance probes. His responding for all operants in Set 2 remained high during the one maintenance probe and responding for intraverbal fill-in responses met mastery levels.

Clare

In baseline for Set 1, Clare did not emit correct responses to tact-by-feature, intraverbal Wh-, intraverbal fill-in, or reverse-intraverbal Wh-, and her correct listener-by-feature responses were at chance levels (Fig. 2, top panel). After introducing intervention with Set 1, Clare emitted more correct responses for all operants, but she did not reach the mastery criterion following four intervention series. Following the participant-specific modification to not remove the SD immediately following “okay” and unintelligible vocalizations, Clare’s responding reached the mastery criterion (i.e., at least 55% correct responding across three operants; a total of 15 intervention sessions or 45 exposures to each IF statement) for Set 1’s tact, listener-by-feature, Wh-intraverbal, and reverse intraverbal operants. Clare’s responding for mastered listener-by-name trials was high (M = 93%, range, 78–100%; see supplementary information). She echoed the IF statement on a mean of 94% of presentations (range, 78–100%).

In baseline sessions for Set 2, Clare did not emit any correct intraverbal Wh-, intraverbal fill-in, or reverse intraverbal responses; correct listener-by-feature responses remained at chance levels; and she emitted few correct tact-by-feature responses (Fig. 2, middle panel). After introducing intervention with Set 2 (Fig. 2), Clare’s correct responding increased but did not reach criterion levels even with repeated exposure to IF. We made a participant specific modification following the fifth intervention series wherein the SD remained for the full 5-s response interval following any incorrect vocalizations. Clare’s responding did not meet the mastery criterion with double the series required with Set 1, so we discontinued intervention for Set 2 after 11 intervention series (i.e., 33 sessions, 99 exposures to each IF statement). Clare’s responding on mastered listener-discrimination trials was more variable with Set 2 (M = 86%, range, 55–100%), but she continued to echo the IF statement on most trials (M = 91%, range, 55–100%).

In baseline sessions with Set 3, Clare emitted few correct responses on tact-by-feature, intraverbal Wh-, and reverse intraverbal trials; no correct responses on intraverbal fill-in trials; and chance-level correct responses on listener-by-feature trials. After introducing intervention with Set 3 (Fig. 2, bottom panel), responding for Set 3 probes increased but did not reach criterion levels. We increased the ratio of interspersed tasks following the sixth intervention series, but levels of correct responding remained unchanged. Clare’s responding did not meet the mastery criterion in double the sessions required for Set 1, so we discontinued Set 3 after ten intervention series (i.e., 30 sessions, 90 exposures to each IF statement) to avoid exposing Clare to further ineffective intervention. Throughout intervention with Set 3, Clare’s correct responding on mastered listener-by-name trials was variable (M = 85%, range, 55–100%) as were her echoics of the IF statement (M = 91%, range 55–100%). Clare’s correct responding for Set 1 listener-by-feature and tacts maintained over subsequent probe sessions without additional intervention sessions for Set 1, whereas her responding across intraverbal operants decreased. We assessed correct responding to Set 2 stimuli, and correct responding remained below mastery levels for all operants except a few probe sessions of tact-by-feature and listener-by-feature.

Following intervention with all three sets, we conducted probes with differential reinforcement (Labeled DR probe, Fig. 2). Correct responding for Set 3 increased for listener-by-feature, tacts, Wh-, and reverse intraverbals but not above levels we had seen previously and not to mastery levels. Correct responses in the presence of stimuli in Sets 1 and 2 did not change. We conducted an additional probe series using differential reinforcement with an extended response interval (10 s). Her responding met criterion for listener-by-feature but decreased across all other operants in Set 3. Responding in the other sets remained unchanged. We discontinued Clare’s participation in the study and taught the remaining targets directly.

Discussion

Adding IF to mastered listener-by-name trials led to the acquisition of secondary targets (tact-by-feature) and increased correct responding during probes for emergence with all three sets of stimuli for Miguel and the first set of stimuli for Clare. Emergent responses were observed with both participants, but only Miguel’s responding for Set 3 matched the efficiency of Frampton and Shillingsburg (2020). Contrary to emergence following just one intervention series, our participants required multiple intervention series before we observed emergent responding. That is, repeated stimulus pairings of the SD and the feature tact were required before participants responded correctly during probes (Petursdottir et al., 2020). The current study extended Frampton and Shillingsburg by adding fill-in intraverbal probes because these may emerge before Wh- intraverbals (Sundberg & Sundberg, 2011). Neither Miguel nor Clare emitted correct responses to fill-in intraverbals prior to the Wh- intraverbals or reverse Wh- intraverbals. Compared to a total duration of three weeks (Frampton & Shillingsburg, 2020), our systematic replication required 16 weeks for Miguel and 26 weeks for Clare from stimulus identification to final probes.

Both participants acquired the secondary targets of tact-by-feature and the corresponding listener-by-feature targets with at least one set of stimuli and correct responding for these operants was observed before intraverbals. One of the mechanisms that may account for this is bidirectional naming (i.e., emitting untaught tacts after listener training and vice versa; Miguel, 2016), which both participants demonstrated prior to the study. The IF we provided was a tact-by-feature response (e.g., “It lives in rivers.” in the presence of the otter). Both participants were likely to echo the IF statements (supplementary information). Given that we required looking at the target picture before delivery of the IF statement, participants may have behaved as listeners and speakers when the IF statement was presented. Their robust tact repertoires, history of bidirectional naming, and exposure to the antecedent verbal stimulus during listener-by-feature trials likely accounted for acquisition of these responses.

Echoic and self-echoic responses may promote acquisition of secondary targets (Dass et al., 2018; Haq et al., 2017) and could also facilitate emergent tact, listener discrimination, and intraverbal responses. Both participants emitted echoics on more than 90% of trials on average across the three sets (supplementary information). Miguel consistently echoed the IF statements, and he acquired all secondary targets and emergent operants across sets. However, these outcomes were observed more slowly for Sets 1 and 2 compared to the participants in Frampton and Shillingsburg (2020). Despite emitting echoics following most IF presentations, Clare did not acquire the secondary targets nor emergent intraverbals for Sets 2 and 3. Similar outcomes of a failure to acquire secondary targets despite high levels of echoics has been reported in some other studies (e.g., Kevin; Vladescu & Kodak, 2013). The participants in Frampton and Shillingsburg (2020) did not emit overt echoic responses during the IF intervention, and the authors suggested that covert echoics and self-echoics could have occurred given the participants’ repertoires (p. 10). Recent research suggested that a verbal-mediation blocking procedure did not prevent acquisition of secondary targets (Dressel et al., 2019). The role of covert or overt echoics and self-echoics in the acquisition of IF is not well understood, and future research should continue to investigate the repertoires that lead to the acquisition of secondary targets and emergent responses following IF.

One potential reason for slow or lack of acquisition is the use of tangible preferred items rather than tokens or edibles, which were used in most studies on IF (e.g., Frampton & Shillingsburg, 2020; Loughrey et al., 2014; Nottingham et al., 2017; Nottingham et al., 2020; Vladescu & Kodak, 2013). We deviated from the procedures in Frampton and Shillingsburg (2020) to provide tangible items rather than tokens because the participants did not use token economies in their programming at the time of this study. Compared to tokens and edibles (chewing and swallowing edibles could interfere with overt echoics), tangible items are not consumed as quickly, and it is likely that there are more competing stimuli present in the environment when IF was delivered. These competing stimuli could decrease the likelihood of attending to the SD and IF, but some studies reported acquisition of secondary targets when consequence IF was used with tangible items (e.g., Carroll & Kodak, 2015; Delmolino et al., 2013; Haq et al., 2017). We sought to circumvent this issue by requiring participants to look at the SD before we delivered the IF. Nevertheless, competing stimuli could have decreased attending to the IF. One way to potentially measure competing stimuli in the environment is by examining the emission of echoics. Both participants regularly echoed the IF, but Clare did not echo the IF on occasion. Of the 648 IF presentations across sets, Clare did not echo 54 IF presentations. The presentations without echoics were more likely to occur when she was playing with the iPad during the reinforcement interval (n = 39, 72%) compared to other preferred tangibles (n = 15, 28%). Future research should consider evaluating the efficacy and efficiency of IF with different types of reinforcers to learn more about the conditions that favor secondary target acquisition and emergent responding.

Although Miguel and Clare passed all the pretests outlined in Frampton and Shillingsburg (2020), their intraverbal repertoires were less advanced than the two participants in in the previous study. Differences in their repertoires could account for differential effectiveness of IF on emergent responses. More advanced intraverbal repertoires may be necessary to see emergent intraverbals following a few IF sessions. A more advanced intraverbal repertoire might be important to the acquisition of emergent responses like those explored in the current study because the responses require conditional control (Axe, 2008). That is, as more targets were added, there were overlapping antecedent stimuli. For example, across the three sets, Clare had to respond to multiple “Where does [animal] live?” antecedent verbal stimuli (i.e., the therapist asks, “Where does otter live?” “Where does eagle live?” and “Where does panda live?”). Exposure to multiple stimuli across the sets likely strengthened more responses under the control of “live,” (i.e., divergent control), so conditional discrimination of the intraverbal—wherein the animal’s name modifies the feature, and both control the response—and convergent control were necessary to emit correct responses during probes (Axe, 2008; Michael et al., 2011). An analysis of participants’ responses during probes may suggest that participants’ intraverbal responses were not under conditional control. For example, in the presence of “What does [animal] eat?”, Clare said “kibble” (i.e., the correct response for Set 1) for Set 2 and Set 3 probes. In the presence of “What does [community helper] use?”, Miguel said “clippers” (i.e., a correct response for Set 1), across earlier probes of Sets 2 and 3. It is possible that Clare lacked component skills for responding to intraverbals that required conditional discrimination (DeSouza et al., 2019).

It is also possible that the participants’ histories with instructional stimuli affected responding during probes (degli Espinosa, 2021). We did not assess whether the participants could respond as a speaker or a listener to the features before IF. That is, we did not assess whether Clare selected the picture of the river after hearing “river” or said “river” in the presence of the picture of the river. We probed the features as listener discriminations and tacts with Clare after completion of all probes, and she responded correctly to all presentations of eight of the nine features as a listener and seven of the nine features as a speaker (supplementary information). We do not know if she would have responded similarly before the introduction of the independent variable. Future research may include additional assessments to determine incoming repertoires and histories with feature stimuli to determine for whom this IF intervention is likely to produce emergent operant responses.

A limitation of the current study is that we made several procedural modifications to Clare’s probe conditions. Individual modifications were needed to arrange conditions that could be more likely to evoke responses to assess emergent responding. Based on the responding we observed in probes (i.e., unintelligible vocalizations emitted with short latencies, repetitive responses), we hypothesized that the function was to remove the trial and end probe sessions earlier (i.e., putative escape-maintained behavior). These observations were coupled with some variable and decreasing responding to interspersed trial tasks which could have suggested that negative reinforcement was a more effective reinforcer than tangible items in the moment (Lalli et al., 1999; supplementary information). Therefore, we modified the probe conditions to no longer remove the trial before the end of the 5-s response interval and, later, increased the ratio of interspersed mastered tasks (Ingvarsson et al., 2009). Nevertheless, we did not see increased correct responding during probes save for Set 1. After discontinuing IF, we included differential reinforcement for correct responses (Mitteer et al., 2020). We did this to see whether correct responding increased when responses contacted reinforcement. Although Clare engaged in some correct responses during this probe, her responding did not increase. Therefore, based on research on prompt dependence (Gorgan & Kodak, 2019), we extended the response interval to 10 s. However, she did not emit more correct responses. Clare may have emitted more correct responses had we implemented differential reinforcement earlier, without the long history of probes under extinction conditions (i.e., Brenna in Petursdottir et al., 2020). However, this would have interfered with our ability to detect emergence. Clare’s responding during probes suggested that she did not acquire the verbal operants through IF, and it could be that we did not have an effective reinforcer. We used choice trials (Frampton & Shillingsburg, 2020), but we did not conduct a separate reinforcer assessment. In addition, it could have been beneficial to train some intraverbal relations to see if others emerged rather than continuing to expose Clare to the current procedures without changes in her behavior for an extended period of time (Petursdottir & Oliveira, 2020; Shillingsburg et al., 2018). Future research could consider conducting reinforcer assessments prior to introducing IF and consider teaching some of the relations if emergence is not occurring.

It is also possible that we did not see increases in correct responding during probes because we did not program indiscriminable contingencies (Stokes & Baer, 1977). The unreinforced probe trials could have been highly discriminable from the reinforced interspersed-task trials, and repeated baseline sessions could have exacerbated this issue. All probe trials were conducted with the same sets of visual and vocal stimuli, there were no prompts or error correction, all responses were followed by a neutral statement, and no responses were followed with reinforcement. In contrast, all interspersed tasks were conducted with different stimuli, incorrect responses were followed by prompts and error correction, and correct responses were followed with praise and access to a tangible item. In addition, the number of probes was extensive, and the order was consistent. This arrangement may not be a concern if only a few series are done, as was the case in Frampton and Shillingsburg (2020), but these arrangements may be problematic when probes are conducted more frequently. Similar to Frampton and Shillingsburg, we conducted probe sessions after three intervention sessions, but our participants required multiple exposures to IF (Petursdottir et al., 2020). Therefore, we exposed our participants to repeated baseline sessions, which may have contributed to some of the issues we encountered with Clare’s responding (described above). Future researchers should consider only conducting probes with subsequent sets after emergence with the current set is observed to save time and mitigate potential negative effects of repeated testing (i.e., use a multiple probe across sets design rather than a multiple baseline across sets design). Additionally, probes should be analyzed for sources of stimulus control like in Petursdottir et al. (2020). That is, researchers should determine not only if the target response is emitted during trials where it is correct (e.g., otters in response to which lives in rivers) but also on trials where it is incorrect (e.g., otters in response to which lives in nests and whose babies are fowls). Future studies may explore the number, schedule, type, and consequences provided following mastered tasks interspersed within probes to program indiscriminable contingencies. Additionally, reinforcement thinning may promote indiscriminable contingencies (LeBlanc et al., 2002), and future research may consider evaluating the use of thinning schedules before generalization assessments.

Another limitation of the current study was that Miguel emitted correct responses to listener-by-feature and reverse intraverbals with Set 2 stimuli and tact-by-feature Set 3 stimuli before the IF intervention, and this was similar to Toby’s correct responding in Frampton and Shillingsburg (2020). Given the sequence of our probes, participants had exposure to the reverse intraverbal SD in the presence of three target pictures during listener-by-feature probes (e.g., “Who lives in rivers?” with an array of otter, dog, and elephant present). This exposure may have been enough to transfer control of the feature to one of the stimuli whose name was then emitted during reverse intraverbal probes (e.g., “Who lives in rivers?” “Otter.”) or temporarily strengthen responses. It is also possible that he contacted these stimuli outside of the intervention sessions. Future research should consider varying the order of probes to avoid potential carryover effects. Additionally, probes could not be completed for Set 2 session four because Miguel’s family went on vacation. However, based on his responding for Set 2 on tact-by-feature and intraverbal Wh-, Miguel’s responding would not have met criterion. After he returned from a two-week vacation, we began with another intervention session.

In conclusion, IF can be an effective way to acquire secondary targets (Werts et al., 1995). Additionally, IF can be arranged to promote and evaluate related emergent verbal operants, including more complex intraverbals. Future research should continue to explore the arrangements to program for and assess for generalization with IF and for whom IF may be beneficial. These procedural refinements could lead to improved efficiency in behavior-analytic instruction and help guide practitioners on how and when to incorporate IF in their practice.

(DOCX 940 kb)