Dissemination of evidence-based practice: can we train therapists from a distance?

Although knowledge about the efficacy of behavioral interventions for children with ASD is increasing, studies of effectiveness and transportability to community settings are needed. The current study conducted an effectiveness trial to compare distance learning vs. live instruction for training community-based therapists to implement the Early Start Denver Model. Findings revealed: (a) distance learning and live instruction were equally effective for teaching therapists to both implement the model and to train parents; (b) didactic workshops and team supervision were required to improve therapists' skill use; (c) significant child gains occurred over time and across teaching modalities; and (d) parents implemented the model more skillfully after coaching. Implications are discussed in relation to the economic and clinical utility of distance learning.

Introduction

Autism spectrum disorders (ASD) are a group of disorders characterized by a continuum of impairments in three key areas: verbal and non-verbal communication, social interaction, and repetitive or stereotyped patterns of behavior (American Psychiatric Association 2000). According to recent epidemiological reports in the US (Fombonne 2003a, b; Yeargin-Allsopp et al. 2003), higher prevalence rates have been reported for ASD than in prior years. Currently, the growing number of ASD cases exceeds the services available for these children (Jacobson and Mulick 2000; Mandell and Palmer 2005; Newschaffer and Curran 2003). This increase in service utilization challenges both researchers and service providers to develop systematic and effective dissemination strategies for transporting efficacious intervention procedures from university research to community service programs.

A number of controlled studies have demonstrated that early intensive interventions using the principles and methods of applied behavior analysis (ABA) can address the core deficits and symptoms associated with ASD and that developmental strategies are gaining empirical support (see Rogers and Vismara 2008 for a review). In addition, the use of parent coaching has been demonstrated to be efficacious and an important common element of effective practice for children with ASD (see Lord and McGee 2001). However, at present, these intensive treatment programs typically only occur in research university settings, making them inaccessible to the majority of the population who are more heterogeneous and receive their services from community based providers with more diverse levels of training and demands on their time (Schoenwald and Hoagwood 2001).

The body of dissemination, or effectiveness research involving empirically supported, university-based ASD early intervention approaches carried out in community programs is limited (Stahmer et al. 2005). While some attempts have been made, (Cohen et al. 2006; Howard et al. 2005; Rogers et al. 1987) to translate research based programs to community settings, early results suggest that these programs may result in less improvement in symptoms than the lab-based studies documented. Thus interventions, when validated in controlled, university-based research settings cannot be assumed to be equally effective when implemented under typical public, community situations (Hoagwood et al. 2001).

Several studies have examined the key therapist, organizational and training-related factors that may affect adoption of empirically supported interventions. Several factors have been identified as critical for therapists’ adoption of the model into clinical practice. Examples include: (a) freeing up time from other clinical work for training; (b) access to reading materials; (c) high-quality supervision and comprehensive training; (d) peer-learning working groups; and (e) program evaluation support (Schmidt and Taylor 2002). In addition, several organizational characteristics and processes are crucial for providing the support and resources therapists need to implement practices. These include funding the time and cost of consistent and qualified supervision, organizational support for evidence-based practice, (e.g., flexible scheduling, pay status, travel, reimbursement mechanisms etc.; Brown et al. 1997; Herschell et al. 2004). Thus, effective dissemination procedures must address a range of broad and complex issues.

Growing evidence from other mental health services suggests that laboratory-based empirically supported interventions can be successfully transported to clinical practice with appropriate planning and adaptation (Henggeler et al. 1997; Taylor et al. 1998). Training protocols addressing the identified key factors for enhancing the dissemination of various empirically supported treatments have been described in the literature (Chamberlain 2003; Urquiza et al. 2003). Standard trainings generally involve studying a treatment manual, attending a lengthy didactic presentation (including case examples, theoretical issues and implementation strategies), and treating one (or more) pilot cases with direct supervision (Barlow et al. 1999). Henggeler et al. (1997, 2002) found expert feedback to be an important factor in treatment fidelity and treatment adherence to be an important predictor for key outcomes in treating violent and chronic juvenile offenders. Parent training has also been identified as an important element for successful dissemination of a treatment model to community settings (Chorpita et al. 2002).

Sholomskas et al. (2005) examined the effectiveness of three common dissemination methods of cognitive-behavioral therapy (CBT) among 78 community-based clinicians: (a) review of a CBT manual, (b) the manual plus access to a CBT training Web site, or (c) the manual plus a didactic seminar followed by supervised casework. Only those in the didactic seminar plus supervision group demonstrated treatment adherence. Therapists in the manual only condition demonstrated some small, short-lived improvement. Although the group participating in the Web-based trainings demonstrated intermediate levels of skill and adherence, scores were not significantly higher than the manual only group, validating the use of a didactic seminar plus supervised practice as an effective means of dissemination in the field of CBT.

Technologies offer a range of methodologies such as audiotapes, videotapes, interactive CDs and the Internet to distribute knowledge and information to reduce the likelihood of depleting clinical, administrative, and fiscal resources (Herschell et al. 2004). In the field of health care, the application of communication and information technologies (i.e., telehealth medicine) has allowed specialists to provide a variety of services, including assessment diagnosis, intervention, and consultation (Duffy and Kirkley 2004; Glueckauf et al. 2002; Singh and Pan 2004; Symon 2001) to remote communities. The use of telehealth medicine, or distance learning via teleconferencing, has also been applied to the education of rural service providers on issues related to child abuse, feeding and communication problems, and management and control of asthma (Wasem and Puskin 2000); however, it has yet to be applied to the dissemination of empirically supported interventions for children and families. Thus, an important question is whether telehealth medicine can increase the availability of intervention models for children with ASD in community settings without significantly sacrificing the quality of care (Chamberlain 2003; Farmer et al. 2002; Glueckauf et al. 2004; Hawley and Weisz 2002; Stirman et al. 2004), particularly in this time of increased demand for ASD programs and documented difficulties in families accessing services (Kraus et al. 2003).

The current study represents a preliminary, quasi-experimental assessment of the effectiveness of varying dissemination methods (including telehealth) for teaching community early intervention therapists to implement the Early Start Denver Model (ESDM; Rogers et al. 2009; Vismara et al. 2009), an evidence-based intervention for infants and toddlers with ASD. Therapists participated in two 5-month phases of learning: (a) direct intervention utilizing the ESDM and (b) implementing the parent coaching model. To address some of the barriers of effective dissemination the training included the use of technology to provide innovative self-instruction materials, didactic training in the intervention, and team supervision on specific cases. In order to examine the application of telehealth for this population, training was delivered via distance education technology for half the group and via live instruction for the other half.

The study was designed to provide preliminary data for four main questions: (a) Is telehealth technology as effective a training medium as live instruction; (b) which training condition(s) are most important for dissemination of the model; (c) is there preliminary evidence of positive changes in child social-communicative behaviors and parent skill level as therapists become proficient in ESDM; and (d) what are experienced community therapists’ responses to the ESDM? Our goal was to learn how to disseminate this model to therapists effectively with the hope of optimizing community-based service delivery to this vulnerable population of very young children with ASD and their families.

Method

Participants

Four early intervention community sites were selected to participate in the study. Inclusion criteria included: (a) an expressed interest in learning the intervention model; (b) enrollment of two or more full-time therapists who had at least a Bachelor’s degree, 2 years of experience treating children with autistic spectrum disorders (ASD), ages 12–60 months; (c) completion of written informed consent; (d) participation in all training activities; and (e) submission of the required data as specified in the study timelines. The sites chosen represented diverse treatment centers, including a children’s hospital setting, a clinical-research setting at a University, a private intervention agency, and a public school.

Each site recruited two to three therapists from various professions (e.g., speech language pathologist, occupational therapist, psychologist, behavior analyst, special education teacher). A total of ten therapists participated, all of whom had been providing some type of educational intervention service to children with ASD for a minimum of 2 years. None of the therapists had any prior training or exposure to the ESDM model, apart from reading an overview article on the model’s origins (see Table 1 for individual therapist characteristics).

Table 1

Individual therapist characteristics

Therapist	Position	Degree(s)	Number of years providing tx; (range of child’s CA in months)	Prior training (other txs)
A1	Psychologist & asst. professor	Ph.D.	13 (24–60)	DTT, VB
A2	Speech language pathologist	M.S., CCC-SLP	30 (0–48)	AAC, PECS
B1	Autism specialist	M.A., BCBA	10 (24–72)	DTT, PECS, PRT
B2	Occupational therapist	M.A.	3 (0–36)	PECS
B3	Speech language pathologist	M.A., CCC-SLP	2.5 (18–60)	PECS
C1	Case manager	M.A.	3 (18–72)	DIR, DTT, PECS, SIT
C2	Case manager	B.A.	3 (24–60)	DIR, PECS
C3	Program director	M.Ed.	3 (18–72)	DIR, DTT, PECS
D1	Early childhood Special educator	B.A.	3 (36–60)	DTT
D2	Behavior specialist	M.A., BCBA	3 (36–120)	DTT

AAC augmentative and alternative communication; DIR developmental, individual-difference, relationship-based model; DTT discrete trial training; PECS picture exchange communication system; PRT pivotal response training; SIT sensory integration therapy; VB verbal behavior

Inclusion criteria for the children included the following: (a) age of 12–60 months; (b) a diagnosis of ASD; (c) no significant health concerns (e.g., lack of functional hand use, lack of ambulation via crawling or walking); (d) no serious or specific medical, genetic, neurological or sensory condition (e.g., Fragile X, Down syndrome, cerebral palsy); (e) parental consent for videotaping; (f) at least 75% parental attendance if the family continued onto the parent training phase; and (g) no participation in additional services exceeding 10 h per week during the study. A total of 32 children were recruited for both the direct intervention and parent coaching phases. Three chose not to participate because of child health concerns or difficulty attending the sessions (e.g., time conflicts, siblings, work schedule etc.), resulting in 29 children completing both phases of the study, ranging in age from 24–51 months. For each of the ten therapists, data related to only one of the children seen by them was randomly selected for analyses. Two children were included in both phase 1 and 2 analyses.

Children were recruited by the therapists based upon referrals to their agencies on a first come, first serve basis and the inclusion criteria. The majority of children were receiving minimal hours of additional services, such as approximately 1 h each of speech, occupational, and/or playgroup therapy per week; no child was enrolled in an intensive, in-home or center-based program. All children’s diagnoses of ASD were initially provided by community clinicians independent of the research project and were confirmed by a trained physician or psychologist at each study site. All children met the standard DSM-IV criteria (American Psychiatric Association 2000) for autistic disorder and exceeded the autism cutoff on the Autism Diagnostic Observation Schedule (ADOS; Lord et al. 1999). In addition, all sites administered the Mullen Scales of Early Learning (Mullen 1995) at the start of the study (see Table 2 for pre-intervention child characteristics).

Table 2

Pre-intervention child characteristics

		CAa M (SD)	RLb M (SD)	ELc M (SD)	ADOSd M (SD)
Phase I	Live	33 (7.3)	22 (13)	23 (10)	13 (4.4)
	Telehealth	33 (7.7)	14 (9.2)	16 (6.9)	15 (4.9)
Phase II	Live	33 (4.4)	15 (12.3)	13 (11)	17 (3.9)
	Telehealth	31 (6)	15 (12.8)	14 (12.5)	15 (5.3)

aChronological age in months

bReceptive language age equivalence in months from Mullen scales of early learning

cExpressive language age equivalence in months from Mullen scales of early learning

dAutism diagnostic observation scales, module 1

Design and Procedure

All sites participated together in two phases of learning, each lasting approximately 5 months: phase 1: direct intervention of the ESDM (Vismara et al. 2009); and phase 2: parent coaching of the ESDM. The ESDM’s curriculum and teaching practices are manualized and draw extensively from previous work in two well-known, empirically supported models: (a) the Denver Model, a relationship and play based, developmental intervention relying on affective exchanges, shown to accelerate learning across a variety of developmental domains (Rogers and DiLalla 1991; Rogers et al. 1986, 1987; Rogers and Lewis 1989); and (b) Pivotal Response Training (Koegel et al. 1987; Schreibman 1988), the naturalistic application of applied behavior analysis aimed at optimizing child motivation to increase communication, language and play skills under natural conditions that more closely resemble the way typically developing children acquire developmentally appropriate skills (Koegel et al. 1999a, b).

During the first training phase, lasting approximately 5 months, therapists learned to use the ESDM in 1:1 treatment with children. This phase included manualized instruction in the teaching principles, intervention techniques, goal development and data collection methods, and fidelity system of ESDM. The second 5 month phase, focused on parent coaching, taught the therapists to educate parents on how to carry out the teaching techniques based on ESDM principles. The therapists were also instructed on two fidelity systems, one on implementation of technique with the child and the second on clinical adherence to the session protocol for coaching parents (see Vismara et al. 2009, for description).

All ten therapists participated in training activities during the same period of time. For the didactic and supervisory phases, five therapists in two distant sites participated only via telehealth technology while the other five participated in the same activities in live, face to face training. Both training phases, direct treatment and parent coaching, consisted of an initial baseline session followed by three training conditions, each lasting approximately 5–6 weeks. These involved: (a) self-instruction with the training materials using print and video materials provided on a DVD; (b) a 10-h didactic training seminar for direct treatment and a 3-h didactic training seminar for parent coaching; and (c) four hours of team supervision for specific discussion of each site’s training cases. These three training conditions (described below) were provided sequentially to all therapists within the same timeline (see Table 3 for timeline of phases).

Table 3

Early start Denver model project timeline

Each training condition began with the specified training activity, which was followed by a 5–6 week period in which the therapists practiced the ESDM with their clients in individual, weekly, 1-h treatment sessions. At the end of each period, therapists submitted the following materials to the investigators: (a) a videotaped probe, approximately 10 min in length, of the most recent treatment session with each enrolled client; (b) a self-rated fidelity score sheet of the taped session for each client; and (c) scored data sheets reflecting the child’s performance on their intervention objectives occurring during the videotaped session. Following each training condition, therapists were also asked to fill out satisfaction surveys to evaluate their perception of the usefulness of the materials and method of distribution, to assess their comfort level in implementing the ESMD, and to receive feedback for improving the teaching content or delivery method used. At the completion of each training condition in the parent coaching phase, therapists submitted the following materials in addition to those listed above: (a) a videotaped session, approximately 1-h in length, of a parent coaching session; (b) self-rated fidelity scores of treatment adherence; (c) self-rated fidelity scores of their adherence to the ESDM parent coaching protocol (more detailed information on the protocol is available from the first author); (d) completed fidelity scores of parents’ implementation of the ESDM (also collected during the videotaped session); and (e) scored data sheets reflecting the child’s performance on their intervention objectives occurring during the videotaped session.

Training Conditions for Direct Intervention (Phase 1)

Baseline Prior to receiving any teaching materials, each therapist was asked to submit a videotaped probe of approximately 10 min involving their own therapeutic interactions with a child. The only instructions provided involved videotaping practices (angles, distance, etc.).

Self-instruction Therapists received a copy of an ESDM training DVD that contained: (a) the treatment manual describing the origins, rationale, assessment practices, teaching strategies, and data system; (b) the ESDM Curriculum Checklist to use for child assessment and development of teaching objectives; (c) the ESDM Fidelity System for determining intervention adherence; and (d) 16 video examples of intervention technique use. They were asked to review all materials and begin to practice the model and fidelity scoring system with the target child. No other instruction was provided.

Didactic seminar The didactic seminar consisted of a total of 10 h of instruction across 2 days, taught by the first and last authors. Five therapists from two sites attended the seminar in person while five therapists from the remaining two sites were connected to the training via teleconferencing. The presenters’ seminar room in a telehealth facility contained a podium, LCD projector and screen, and two television monitors at the front of the room on which the remote participants were televised. The remote participants also sat in telehealth equipped facilities, in Birmingham, Alabama and San Diego, California, where they faced a large television screen on which was displayed the presenter, the slides and videos, and other participants who were speaking. All therapists were connected in “real time” and could see and hear each other and the workshop participants and interact through the television monitors.

The training topics included a detailed review of the treatment model’s origin and principles, curriculum assessment procedures, teaching practices, data collection and fidelity systems. Videotaped examples of technique use and assessment procedures were shown and discussed, and small group exercises were carried out to practice developing child treatment objectives, recording in vivo data, and rating intervention fidelity.

Team supervision This final training condition involved a 2-h meeting among the first and last author and each individual team, discussing and viewing together submitted videos. Therapists attended live or through teleconferencing, as they had in the didactic training. The therapists selected certain topics for discussion and the researchers provided feedback and addressed areas that needed further training based on their a-priori review of the fidelity tapes. This marked the end of the training. A final 1-h telephone conference call was provided by the first author to each team to provide final feedback and closure.

Training Conditions for Parent Coaching (Phase 2)

The therapists continued on to the parent coaching phase following the final feedback session for the initial ESDM model training.

Baseline Parents were asked to engage their child in a 10-min play period and to interact as they would normally do so at home. Following this probe, therapists conducted a parent coaching session, approximately 1-h in length, in which they explained the ESDM’s teaching practices to the parent and demonstrated the techniques during a 10-min play activity with the child. No other instructions were given. The videotape of the parent-child play activity and therapist–parent coaching was submitted prior to receiving any instructional materials.

Self-instruction Therapists were provided an ESDM parent training DVD, which contained: (a) the parent manual, with ten chapters each describing one of the ten ESDM teaching techniques; (b) the parent-child ESDM fidelity rating tool; (c) the therapist–parent coaching ESDM fidelity rating tool; (d) and video examples of parent coaching sessions.

Didactic seminar Three hours of direct instruction on the parent coaching model was provided by the first and last authors, delivered as in the previous didactic training. Topics included a detailed review of the parent training manual topics, the parent fidelity system, the session protocol for therapists to follow in teaching the content (i.e., therapist–parent coaching fidelity), and data management. Videotaped examples of different problem areas that might emerge during parent coaching sessions were shown and discussed. Group exercises involved rating therapists’ parent coaching fidelity.

Team supervision This phase was carried out exactly as described above, involving a 2-h team supervision by the first and last authors focused on observing videos, discussing teaching points, and answering questions. A final 1-h telephone conference was then provided for final feedback and closure.

Dependent Measures

Three measures of training effectiveness were collected: (a) therapists and parents’ fidelity of implementation of the ESDM, (b) frequency of child social communicative behaviors, and (c) therapist satisfaction.

Fidelity of implementation Three types of implementation fidelity were assessed via videotape: (a) therapist–child fidelity involving direct implementation of ESDM, (b) therapist–parent fidelity involving procedures for coaching parents in ESDM, and (c) parent-child fidelity involving parents’ implementation of ESDM with their child. Therapist–child fidelity was assessed using the ESDM Fidelity Scale, evaluating 15 core treatment practices on a 5-point Likert based scale (more detailed information on the protocol is available from the first author). Fidelity, defined by a total score of 85% or greater, consisted of the following practices: (a) management of child attention; (b) quality of behavioral teaching; (c) modulation of child affect and arousal; (d) management of unwanted behavior; (e) dyadic engagement; (f) choice-making opportunities; (g) child motivation; (h) naturalistic materials and age-appropriate activities; (i) adult affect; (j) adult sensitivity and responsivity; (k) communicative opportunities and functions; (l) appropriateness of adult’s language; (m) elaboration of activities; (n) transitions between activities; and (o) child engagement during unstructured times.

Parent fidelity was similarly evaluated via videotape by rating 13 of the above practices used by parents in their first joint activity with the child, which occurred at the start of each session, before any coaching or demonstration was provided. Two practices, ‘naturalistic materials and age-appropriate activities’ and ‘child engagement during unstructured items’ were not included for evaluation because the nature of the parent teaching allowed for no variation on these items. For purposes of analysis, the total percentage score was used to determine whether or not participants met fidelity, and the mean overall rating (i.e., total score/13) was used as a continuous dependent variable for all other analyses.

Therapist–parent fidelity was assessed using a similar, behaviorally anchored 5-point rating scale of 13 parent-coaching behaviors:(a) review of progress; (b) explanation of the target technique; (c) demonstration of the technique; (d) coaching the parent; (e) discussing generalization; (f) management of physical space; (g) management of data; (h) quality of relationship with the parent; (i) management of parent concerns; (j) sensitive, reciprocal communication; (k) time management; (l) management of transitions; and (m) ethical principles.

Inter-rater reliability was established prior to scoring and maintained throughout the study by having two coders independently rate 25% of the tapes. The primary coders were research assistants who were blind to hypotheses and blind to the training delivery group (distance vs. live) and the training condition. They were trained by the first author, a Board Certified Behavior analyst with a Ph.D. in Educational Psychology, who also served as the reliability coder. The first author was trained by the last author, a primary developer of the ESDM model, to 85% agreement on all Fidelity of Implementation measures. For each dependent variable, inter-rater agreement was calculated for 40% of the sessions across all therapists, parents, and children.

Formal analysis of inter-rater reliability was conducted using intra-class correlation coefficients (ICC), which ranged from .88 for therapist fidelity to .99 for therapist–parent fidelity. Internal consistencies for each of the scales—calculated for all across each training condition and each child—were .94 for the parent fidelity scale, .92 for the therapist–parent scale, and .95 for the therapist fidelity scale. Mean fidelity scores for each scale showed adequate variability across conditions, with no significant deviations from normality using the Kolmogorov–Smirnov test. Fidelity scale scores were also transformed into dichotomies indicating whether fidelity was at or above the threshold of 85% (mean score of 4.25) at each training phase. The use of a cut-off criterion of 85% was based on previous clinical practice with the ESDM.

Child social communication behaviors To assess changes in the children’s social communication, data were collected on two behaviors: (a) number of spontaneous functional verbal utterances, and (b) number of imitative behaviors. Spontaneous functional verbal utterances were defined as verbalizations: (a) initiated by the child without an adult model; (b) relevant to the interaction (i.e., no out-of-context responses, stereotypic, or echolalic responses); (c) combined with body and facial orientation toward the adult and/or relevant stimulus materials; and (d) containing a phonetically correct approximation of the word or word combination (adapted from R. L. Koegel et al. 1988; Symon 2005). Imitative behaviors included: (a) imitation of actions on objects; (b) imitation of manual acts without objects; and (c) imitation of vocalizations and words. These had to occur immediately after the model and to not have preceded the model’s behavior in order to be counted.

All child behaviors were transcribed by two independent raters. Intra-class correlation coefficients were calculated across therapist training phases and children in order to maximize the available data for assessing coding reliability. For spontaneous verbal utterances, the intra-class correlation was .98; for imitation, the intra-class correlation was .97.

Observation ratings of child engagement The Child Behavior Rating Scale (CBRS; Mahoney and Wheeden 1998) was used to assess children’s engagement across a 5-point Likert rating scale ranging from 1 (very low) to 5 (very high). The CBRS evaluates the child across seven items organized into two main categories of behavior: (a) child attention consisting of attending, problem-solving, cooperation, and enjoyment; and (b) social initiations involving behaviors such as initiating new play ideas, sharing enjoyment, and affect. All taped episodes were coded by two independent raters. Coding reliability was assessed using intra-class correlations. For child attention, the intra-class correlation was .95; for child social initiations the intra-class correlation was .92.

Therapist satisfaction survey At the end of each training phase, therapists completed a satisfaction survey divided into three sections. The first section, therapist satisfaction, asked them to respond to statements about each training phase using a 6-point Likert-type rating scale (0 = not applicable; 1 = strongly disagree; 3 = neither agree or disagree; 5 = strongly agree). Questions related to whether the teaching activities in each training phase contributed to: (a) understanding the ESDM; (b) increased comfort with the ESDM procedures; and (c) usefulness of the fidelity measure.

The second section focused on therapists’ understanding of specific ESDM techniques. Therapists were asked to rate the techniques they understood best with a plus (+), techniques understood the least with a minus (−), or techniques understood partially but requiring more instruction with a (±). For purposes of analysis, responses were recoded to a 3-point scale (“−” = 1; “±” = 2; “+” = 3). The information gathered was used in the next training phase to provide additional instruction.

The final section, therapist feedback, asked therapists to answer the same three open-ended questions: (a) What were the most and least helpful parts of this method of learning; (b) how could this phase of the program have been improved to help you more; and (c) can you provide additional feedback about the methodology or the training? Likewise, their responses were taken into account when developing the teaching material for the next training phase.

Results

Direct Intervention Phase

Therapist fidelity The fidelity of treatment implementation was examined as both a function of training condition following baseline (i.e., self-instruction, didactic seminar, and team supervision), as well as method of delivery (i.e., telehealth or live). Specifically, therapists’ treatment fidelity scores (using the likert rating scale) were examined using a repeated measures mixed model analysis wherein training condition was treated as a 4-level within-subjects variable and delivery method was treated as a 2-level between subjects variable. Planned comparisons as a follow-up to significant omnibus tests of main effects involved comparing each training condition except baseline with the previous conditions (i.e., difference contrasts). Additional post-hoc comparisons compared the baseline to the mean of all other conditions, baseline to the mean of both the didactic and individual conditions, and the self-instruction condition to the mean of both the didactic and individual conditions. To control for family-wise error rates, we used the Holm sequential Bonferroni method (Holm 1979).

Results revealed a significant main effect for training condition (F (3, 24) = 8.85, p < .001), but no effect for delivery method or for the interaction between method and training condition. As can be seen in Fig. 1, therapist fidelity increased over time from baseline to the final training condition. Examination of planned comparisons within the training condition main effect revealed that treatment fidelity significantly improved between the baseline and self-instruction training conditions (t = 2.60, df = 24, p < .05). Moreover, additional post-hoc comparisons revealed that fidelity at the didactic and individual training conditions in combination were significantly higher than the self-instruction training condition (t = 2.04, df = 24, p = .05). There was no difference between group didactic and team supervision training conditions with respect to therapist fidelity.

Fig. 1

Therapist treatment fidelity scores across training conditions

Therapist fidelity was also analyzed as a dichotomous variable at each training phase indicating whether a therapist was at or above the standard fidelity threshold of 85%. The analysis utilized generalized estimating equations to assess change in this binary fidelity variable over time. Results indicated no significant effects for either condition or delivery method. Although a trend toward increasing fidelity over time was observed, only 50% of the sample was at the 85% fidelity threshold by the final training phase. When the fidelity threshold was lowered to 80%, however, results of the same longitudinal logistic regression revealed a significant effect for condition (Wald X 2 = 10.04, df = 3, p < .05), with 90% of the therapists (nine out of ten) demonstrating fidelity by the team supervision phase (the 10th therapist demonstrated 79% fidelity, just shy of the 80% cut-off). Planned comparisons of the number of therapists at or above fidelity at each successive condition revealed a significant increase from baseline (10%) to self-instruction (40%; X 2 = 4.29, df = 1, p < .05), and from didactic (60%) to team supervision (90%; X 2 = 4.29, df = 1, p < .05).

Therapist satisfaction survey Analysis of therapist reported satisfaction at each training condition (not including baseline) was analyzed using a repeated measures mixed model analysis. Overall therapist satisfaction ratings from each of the two subscales (i.e., satisfaction and understanding) were analyzed separately. Planned comparisons again used difference contrasts, with additional post-hoc comparisons as detailed above. Results revealed a significant main effect for condition for both the satisfaction subscale (F (2, 14.80) = 16.17, p < .001), and the understanding subscale (F (2, 5.59) = 9.16, p < .05). No main effect for delivery method and no interaction effect between training condition and delivery method were found for either subscale. Estimated marginal means for the two satisfaction subscales are shown in Fig. 2. Examination of planned comparisons between training condition levels revealed that satisfaction ratings significantly increased from self-instruction to didactic training phases for both the satisfaction subscale (t = 3.89, df = 13.63, p < .01), and the understanding subscale (t = 4.25, df = 7.63, p < .01). Ratings were not significantly different between the didactic and team supervision conditions for either subscale.

Fig. 2

Therapist satisfaction ratings across training conditions

We examined the degree to which satisfaction scale scores—especially technique understanding scores—might be related to treatment fidelity scores. To this end, we included satisfaction subscale scores as time-varying covariates in the prediction model of therapist fidelity scores over time. Results revealed that neither satisfaction subscale was related to therapist fidelity scores.

All of the therapists answered the survey’s open-ended questions soliciting comments for improving each training condition related to the direct intervention and parent coaching phases. A number of themes emerged from these comments. The most frequently occurring themes for most helpful training activities (regardless of delivery method) related to the inclusion of video examples for demonstrating direct implementation of the ESDM and of group exercises for practicing fidelity coding across measurement systems. There was consensus amongst the therapists that having opportunities to ask questions and group discussion were most useful for learning how to correctly implement the teaching procedures. The majority of comments expressing specific concerns or feedback for improvement related to the request for more group discussion and supervision feedback, varied video demonstrations, and additional time in schedules to complete the research requirements (i.e., sending in paperwork and videotapes).

Change in children’s behavior Change in children’s behavior was examined for each of the four variables measured at each of the four Phase 1 training periods: functional verbal utterances, imitation, attention, and social initiations. Analyses were conducted first by employing a repeated measures mixed model using only training condition as the repeated measures factor. Delivery method was not retained in any of the models given that it was not significant in any prior analyses of therapist fidelity and was not a significant predictor in any subsequent models of change in child behavior over time. Subsequent to examining changes in child behavior as a function of training condition, we examined therapist fidelity as a time-varying covariate in order to assess the degree to which therapist fidelity scores were related to any observed improvements in child behavior over time. Planned comparisons as a follow-up to significant omnibus tests again focused on change between successive training conditions, with the same post-hoc comparisons and error correction as detailed above.

Results of analyses of each of the four child behaviors revealed significant change over time for the number of functional verbal utterances (F (3, 27) = 4.59, p < .05), child attention (F (3, 27) = 6.17, p < .01), and social initiations (F (3, 27) = 7.87, p < .001). There was no significant change for imitation behaviors over time. For the number of functional verbal utterances, planned comparisons revealed that the observed frequency of utterances was significantly greater during the didactic phase (M = 21.40, SEM = 4.63) than at both the team supervision phase (M = 13.5, SEM = 4.63; t = 2.20, p < .05) as well as the self-instruction phase (M = 10.3, SEM = 4.63; t = 3.10, p < .01). There was no difference between the self-instruction phase and baseline (M = 9.5, SEM = 4.63). Estimated marginal means for both attention and social initiations are shown in Fig. 3. Planned and post-hoc comparisons revealed that both attention and social initiation behaviors significantly increased from baseline and from self-instruction to the mean of didactic and team supervision. There was no significant incremental improvement from didactic to team supervision conditions for either attention or social initiation behaviors.

Fig. 3

Child attention and social initiations across training conditions

The inclusion of therapist fidelity scores as a time-varying covariate in the models revealed significant relationships between therapist treatment fidelity and social initiations (F (1, 30.84) = 8.61, p < .01) and between therapist treatment fidelity and attention (F (1, 30.30) = 20.97, p < .001). These analyses revealed that as therapist fidelity increases, child attention and social initiations also generally increase. No relationships between number of functional verbal utterances or imitation were found in analyses examining fidelity scores over time.

Parent Coaching Phase

Therapist–parent fidelity Analysis of therapist–parent fidelity by training conditions in Phase 2 revealed a significant main effect for training condition (F (3,16.86) = 21.88, p < .001). The main effect for delivery method and the interaction between delivery method and training condition were not significant. Planned comparisons of training conditions revealed significant improvements in fidelity scores between baseline and self-instruction (t = 2.75, df = 16.54, p < .05), and between self-instruction and didactic training (t = 3.95, df = 16.79, p < .001). There was no difference between the didactic and team supervision phases. Figure 4 displays the means for therapist–parent fidelity over time. Analysis of these same data with respect to a fidelity threshold of 85% revealed that only one therapist achieved fidelity by the last two training conditions. Lowering the threshold to 80% did not alter these results.

Fig. 4

Therapist–parent coaching fidelity scores across training conditions

Parent fidelity Analysis of parent fidelity scores also revealed a significant main effect for training condition (F (3, 17.87) = 3.19, p < .05). The main effect for delivery method and the interaction between delivery method and training condition were not significant. Figure 5 displays the means for parent fidelity over time. Although planned comparisons did not reveal any differences between training conditions, post-hoc tests revealed that there was significant improvement in parent fidelity scores from baseline to the combined fidelity scores averaged across all subsequent training phases (t = 2.84, df = 16.15, p < .05); however, there were no differences between any of the specific last three training phases (e.g., self-instruction vs. didactic, didactic vs. team supervision).

Fig. 5

Parent fidelity scores across training conditions

Analysis of parent fidelity as a dichotomy, indicating whether any given parent was above or below the threshold of 85% fidelity, revealed that no parents were at fidelity by the self-instruction phase, and that five out of nine parents were at or above this threshold by the didactic and individual training phases. Despite this apparent increase, a longitudinal analysis of this dichotomous variable revealed only a non-significant effect for condition. Re-analysis using an 80% fidelity threshold, revealed a similar trend toward increasing fidelity over time (Wald X 2 = 4.65, df = 2, p = .09), from 1 out of 9 at baseline (11%) to 5 out of 9 by the didactic instruction phase (56%).

We also examined the degree to which changes in parent fidelity scores over time might be a function of therapist–parent fidelity by entering therapist–parent fidelity into the longitudinal model as a time varying covariate. Results of this analysis revealed no relationship between therapist–parent fidelity and parent fidelity scores.

Changes in child behavior The same four child behaviors as in Phase 1—functional verbal utterances, imitation, attention, and social initiations—were again analyzed for change over time. Means and standard deviations for each training phase are shown in Table 4. For each of the child variables in Phase 2, there was no significant change over time between any of the training conditions. Analyses of parent fidelity as a time varying covariate, however, did indicate significant positive relationships between parent fidelity and attention (F (1, 26.53) = 37.31, p < .001), imitations (F (1, 22.60) = 9.31, p < .01), and social initiations (F (1, 24.10) = 27.09, p < .001).

Table 4

Child behaviors means and standard deviations

	Parent coaching phase mean (SD)
	Baseline		Self-instruction		Didactic		Team supervision
Number of words	22.29	(27.13)	15.50	(10.99)	20.89	(16.10)	28.22	(23.42)
Number of imitations	14.57	(10.67)	11.00	(8.38)	13.56	(9.81)	8.11	(9.83)
Attention score	3.00	(.75)	3.26	(.66)	3.46	(.63)	3.54	(.79)
Social initiations	2.84	(.45)	3.07	(.51)	3.14	(.60)	3.13	(.68)

Discussion

The early intervention in ASD literature is currently dominated by studies of efficacy—the examination of research-based intervention strategies carried out in highly controlled conditions, by experts on the model. From this literature, several models have demonstrated efficacy through strong scientific designs and replications (see Rogers and Vismara 2008 for a review). The next step in examining the usefulness of an empirically supported intervention is to move to effectiveness studies—the examination of the effects of an intervention when delivered in community settings by typical community interventionists to children who typically are seen in that setting. The first step in conducting effectiveness trials and increasing access to care involves training providers in community settings to fidelity in a model being implemented.

The purpose of this study was to provide preliminary evidence of the effectiveness of a training paradigm that has been developed to teach the ESDM. The paradigm was designed to address some of the barriers to dissemination such as access to reading/training materials, cost of training, provision supervision over time, and peer and administrative support for training. We compared the effectiveness of distance learning procedures with live instruction. In addition, we provided a hierarchy of training activities and gathered performance data at baseline and after each training activity: self instruction through print and video, didactic instruction through a 2-day conference, and individual supervision of teams. Furthermore, we examined learning of two different interventions—one focused on children, and the other focused on parent education.

A primary question was the importance of live instruction. Our findings clearly demonstrated that teaching via distance learning technology was as effective as teaching using live interaction. There were no differences in therapist performance on the ESDM fidelity tool for those taught via distance learning and those who participated in live training in their ability to use the model with children or to use the methods of parent coaching. There were also no differences in their rate of progress after each learning activity, their final skill level with children and with parents or their satisfaction with the training. Nor did delivery method affect the skill level of the parents who were trained in ESDM during the parent coaching phase. Our data are quite clear: our distance learning activities were as effective as live instruction at teaching others to use the ESDM.

A second primary question was whether the training package we assembled was successful for teaching the model to community therapists, which in our hypothesis we defined as therapist performance at a fidelity level of 85% by the end of training. Prior to this study though, this criterion level had yet to be empirically examined and was established before final coding descriptions in the fidelity system had been finalized. At the end of the direct intervention training phase, only 50% of the group achieved an 85% level of performance on the fidelity measure. This indicates that this particular training package is not sufficient to teach the majority of participants to performance at an 85% level. However, it may well be that an 85% level is an overly ambitious goal. Given the scoring system, a mean score of 4.0 (80%) or higher on the fidelity instrument could be argued as reflective of a fully competent performance, since a score of 4 on any item represents a competent display of the target behaviors without any significant weaknesses. Scores of 5 represent expert performance, above and beyond the target behavior. Indeed, our results revealed that 90% of the group achieved a fidelity score of 80% or higher by the last training phase, and thus the majority of therapists did in fact learn to deliver the model fully competently by the end of the training program.

A third question was the importance of each of the training steps. Due to the quasi-experimental nature of this community project, it is not possible to examine the specific importance of each of the training steps, because they were additive. In addition, although a great of time was provided to these experienced therapists between training steps, it is possible that given additional time with each step the results may have differed. However, we feel we can make some preliminary recommendations for effective training based on what we learned from this process. It would certainly represent a training dream if a significant number in the group could achieve fidelity through the use of the self-instructional materials only. Our examination of the changes in mean fidelity after each training activity revealed that, while the self-instructional materials were helpful and resulted in significant improvement in use of ESDM compared to baseline, only 40% reached fidelity on therapist–child interactions, when defined as 80%, after the self-instructional phase. In addition, the therapists themselves reported little comfort with the method after the self-instruction phase alone. The majority of therapists required further training. The didactic training phase by itself did not add significantly to skill levels in therapist–child intervention, but the combination of the didactic training and the team supervision did result in a significant improvement in therapist fidelity skills. Whether one of these two activities could be dropped without affecting skill development is not known. Again, while additional time to practice after the didactic phase may have led to additional progress, most community organizations are likely to want providers to become proficient in new methodologies within a relatively brief time period, and for training to be brief and cost effective. The team supervision phase is certainly the most expensive in terms of training time; however, it was also considered a main strength of the training program in the trainee evaluation, and even more individual supervision time was requested by therapists. Indeed, this last phase of training resulted in a significant increase in the number of therapists meeting the fidelity threshold of 80%, from 6 out of 10 to 9 out of 10, with only one therapist falling just below fidelity at the last training phase with a score of 79%. Therefore, including team supervision and didactic training together early in training may increase fidelity of implementation more quickly and improve therapist confidence with the methodology.

In learning the parent coaching model, the patterns for the therapists were similar to that for the child based interventions. Again, there was no difference in learning the therapist–parent coaching model between the distance learning and live instruction groups. There were significant increases in therapist skill from baseline across the three levels of instruction. The increase in therapist skill from baseline to self-instructional phase was significant, as was the increase from self-instructional phase and the didactic learning format. Nevertheless, only one out seven therapists achieved fidelity by the didactic and team supervision phases, whether using the 85 or 80% threshold. There may be several reasons for this. Anecdotally, the therapists felt that their sessions (1 h) may not have been long enough to complete all of the parent coaching techniques. They also consistently commented that additional time to practice the ESDM techniques before training parents would be helpful.

Analysis of parent fidelity scores revealed a significant increase from baseline to the end of treatment, with no differences in learning for those whose therapists were being trained via distance learning and those who were being trained live. Parents showed a significant increase in their skills when their therapists moved from the baseline to instructional phases, although there was no difference in parent fidelity between any of the instructional phases. Moreover, change in therapist fidelity in parent training over time was not related to change in parent fidelity ratings over time. Although only one therapist met fidelity of implementation of the parent coaching techniques, over 50% of parents achieved an 85% level of fidelity by the end of the didactic phase of training. Several of the therapists in the program were already conducting parent training using other methods. It may be that they did not master the specific parent coaching techniques outlined in the ESDM manual, but did use other effective methods of training parents. Additionally, parents do not have the level of experience with intervention as the therapists do, therefore additional time in treatment may be necessary to achieve fidelity of implementation. These results are consistent with other community parent training studies showing that while short-term parent training leads to skill improvement in a majority of families, fidelity of implementation is achieved by about half of participating parents (Stahmer and Gist 2001).

How did the changes in therapist skill use affect children over the 5 months of training? We examined four behaviors that our own studies have previously shown to be affected by this intervention: number of functional verbal utterances, imitation, attention to the adult, and number of social initiations to the adult. Two of these behaviors, number of social initiations and amount of attention to the adult, demonstrated significant increases from the baseline period to the team supervision phase. Number of spontaneous functional verbal utterances spoken in a 10-min period also increased significantly over time, but the main increase occurred from self-instruction to the didactic phase. The number of imitative acts did not increase significantly over training. There were no differences in changes in these child behaviors due to distance learning vs. live instruction. Of course the relationship between therapist skill level and child change in behavior are confounded by time, and the children may have improved simply based on time in intervention. However, we also found a significant relationship between changes in therapist skill in using ESDM, as reflected in fidelity scores, and changes in child performance in both attention and initiations. Thus, as the therapists increased their skilled use of ESDM techniques, children demonstrated increases in attention to the adult and number of social initiations directed to the adult. Similarly, the effects of increases in parent use of ESDM techniques were significantly associated with increases in child imitations, social initiations to the parent, and attention to the parent. While this cannot be interpreted causally, it does suggest the possibility that changes in therapists’ use of ESDM therapy techniques mediates improvements in child initiated engagement.

Several main points stand out. The first is that the ESDM intervention approach is teachable, to experienced professionals and paraprofessionals, and also to parents. The teaching package we used, with many hours of self-instruction, 13 h of didactic instruction either live or via telehealth technology, and 4 h of team supervision, was successful in developing competence in the model for the majority of persons participating, regardless of their professional discipline or level of terminal degree. In this small quasi-experimental study, the ESDM was demonstrated to be an effective model when transported to communities and carried out by community therapists just learning the model, as well as by the parents whom they taught. Additionally, most of our results suggest that although the entire training package lasted 10 months, the greatest difference in therapist skill acquisition was achieved by 6 weeks, after the didactic phase. Although the gains did not reflect the highest fidelity criterion of 85%, therapist skill level still remained above satisfactory levels and resulted in child behavioral gains across areas of development. This is very important in terms of providing a relatively low-cost intervention that includes supervision and can lead to improvements even for therapists serving children in remote locations.

The second main point is that the quality of treatment in a relationship-based, developmentally based intervention model can be quantitatively defined and assessed. The fact that therapist fidelity of treatment use was related to changes in children’s behavior suggests the possibility that therapist use of ESDM skills is directly related to outcome measures in children’s behavior. Although the possibility remains that such a relationship is not causal, the finding is consistent with the idea that specific treatment techniques have specific effects on child behavior. Future research that systematically controls therapist acquisition of use of specific treatment techniques may help to further illuminate these possible relationships.

A third main point is that this study has followed a systematic and recommended set of guidelines for conducting effectiveness trials on psychosocial interventions for autism (Smith et al. 2007). First, the study has replicated findings from earlier research (Rogers and DiLalla 1991; Rogers et al. 1986, 1987; Rogers and Lewis 1989) supporting the efficacy of the ESDM model. In the present study, significant gains over baseline rates in child behaviors were demonstrated in a number of domains when assessed after therapists had been trained and met fidelity. Second, the parent coaching intervention techniques and curriculum taught to the therapists developed out of prior research evaluating the step-by-step instructions, decision-making protocol and overall content of the ESDM parent coaching manual (Vismara et al. 2009), which allowed a manualized and research-based plan to be implemented and evaluated across the participating sites. Finally, the model has been tested in a randomized controlled trial at the University of Washington (G. Dawson, PI) and we are now replicating and extending that work in a large scale multisite randomized clinical trial on the direct implementation and parent coaching models, which will examine a multitude of parent, child, and family outcome measures and variables thought to influence outcomes (mediators, moderators). Future research will need to expand these findings further by using attention-control or alternative treatment comparison groups and by employing standardized behavioral measures that allow for the assessment of clinical significance with respect to any significant increases in outcome. Nonetheless, the study represents a large scale effort at developing, validating, and disseminating manualized interventions that have shown efficacy for children with autism.

One clear limitation to the study is the confound between time and training step. It is unclear what aspects of the training curriculum were necessary to develop skills in therapists and families naïve to ESDM. It is clear that having well-written materials and videos that clearly displayed the skills and narrated their use were not sufficient for teaching the model. While these materials taught therapists some skills, only four therapists achieved competence in the model from self instruction. The use of both large and small group instruction added to learner competence, and the learners themselves found the direct supervision and the extensive use of video to demonstrate and to supervise extremely helpful. Thus, these more time intensive methods appear necessary to teach this type of complex intervention. However, the relative contribution of these types of training and whether training would work as well if these aspects were combined must be examined in future controlled studies.

Therapists also had difficulty obtaining fidelity on the parent coaching portion of the methodology. It will be important to examine in more detail which specific aspects of the process were difficult for the therapists and what the specific barriers to the parent coaching they found. Another area for future research would be to examine the relationship between therapist education, training and experience with both autism and parent coaching to their ability to train parents to fidelity in the model. It may be that general parent coaching skills may be important to teaching parents any technique, including ESDM.

Finally, the therapists were quite positive about the training experience and about the ESDM as an intervention model. The developmentally oriented therapists seemed to find that the organization, curriculum, and structure of the intervention were extremely helpful. The behavioral therapists from a Discrete Trial Training background expressed surprise at how quickly the children learned in a play-based intervention and how much fun both child and adult were having throughout teaching episodes.

While the therapists varied in terms of educational and experience characteristics, these findings still reflect effects of training participants who volunteered to partake in an evaluation of training conditions. It is not known whether these findings generalize to other, possibly less motivated, groups of community therapists. Research should also continue to investigate the training effects of large vs. small group instruction. Group supervision enables several therapists to receive instruction simultaneously; however, it can be differentially stressful for therapists who are markedly less experienced or successful than other therapists in implementing the intervention and/or in basic therapeutic skills. Small group instruction allows for more specific information to be passed; however, it requires more time and expense.

In summary, as the number of children diagnosed with ASD grows, the need to make effective treatments broadly available in the community becomes increasingly pressing. In addition to providing further support for the use of the ESDM, an evidence-based treatment in ASD, in direct intervention and in a parent coaching model, information was obtained regarding the dissemination of this type of intervention to community early intervention therapists. The use of distance learning technologies was quite successful. Distance learning options reduce the time required of therapists to attend seminars and receive supervision by eliminating travel time and associate costs, as well as potentially allowing more flexible scheduling options. Although it appears that the rigors of training of therapists to specified adherence criteria, followed by regular monitoring of therapist skills and feedback aimed at optimizing treatment fidelity cannot be eliminated, questions were raised regarding how time intensive these efforts will truly have to be. Fidelity was achieved in the direct intervention phase by the clinicians following only a 2-day didactic and a 2-h team (not individual) supervision session. Although the groups trained in this study were volunteers, it is encouraging that four different types of agencies were willing to allow their therapists release time in order to participate in this project.