Caregiver Satisfaction with Delivery of Telehealth Autism Services.

Objective: The purpose of this study was to assess satisfaction with telehealth interventions for a large nonprofit organization that transitioned interventions for individuals with autism spectrum disorder (ASD) to telehealth during a pandemic. Services provided via telehealth included applied behavior analysis (ABA), speech and language, and occupational therapies. A secondary survey evaluated reasons for declining telehealth services.
Methods: A survey was administered to 10,567 families who were receiving autism interventions. A total of 440 respondents answered all the questions on the survey, and their results were included in this study. A secondary survey was administered to 223 individuals who declined to have telehealth autism interventions.
Results: There was not a clinically meaningful difference in satisfaction across service types. Although all ratings were in the high range, caregivers ranked speech therapists as more dependable than ABA therapists, and this difference was statistically significant. The findings suggested that the majority of caregivers were generally satisfied with services provided in a telehealth format. For those who declined services, the majority indicated a discomfort with the use of technology. Conclusions: The participants of telehealth autism interventions reported high general satisfaction and indicated an improvement in their quality of life. Results provide suggestive evidence that increased satisfaction of telehealth services may allow for further acceptability and access for participants. Future research should evaluate participant and clinician satisfaction with telehealth versus in-person interventions.

Autism spectrum disorder (autism) is a neurodevelopmental disorder characterized by deficits in social communication and the presence of restricted interests and repetitive behaviors (Hodges et al., 2020). According to the Centers for Disease Control and Prevention (2021), 1 in 44 individuals met the criteria for an autism diagnosis. There are many interventions for autism that consist of behavioral, developmental, educational, social-relational, pharmacological, psychological, and complementary and alternative therapies (Centers for Disease Control and Prevention [CDC], 2022). Among the interventions for autism, behavioral interventions are considered the most evidence based (CDC, 2022; Hyman et al., 2020). Some behavioral interventions take on a developmental and naturalistic approach and are referred to as naturalistic developmental behavioral interventions (NDBIs) (Sandbank et al., 2020; Schreibman et al., 2015). Applied behavior analysis (ABA) is a type of behavioral intervention and has been defined as “the process of systematically applying interventions based upon the principles of learning theory to improve socially significant behaviors to a meaningful degree, and to demonstrate that the interventions employed are responsible for the improvement in behavior” (Baer et al., 1968).

Speech and language therapies are the most commonly used developmental therapy (Hyman et al., 2020). Speech and language therapy “helps to improve the person’s understanding and use of speech and language. Some people with autism communicate verbally. Others may communicate through the use of signs, gestures, pictures, or an electronic communication device” (CDC, 2022). Other common interventions can include occupational therapy, physical therapy, and social skills groups. The interventions are commonly provided face to face in the child’s natural environment (e.g., home), but in recent years, remote telehealth interventions have gained popularity, particularly due to the mandated stay-at-home orders during the COVID-19 pandemic.

Telehealth is defined as the use of telecommunication technology (e.g., computers, mobile devices) to provide remote clinical healthcare services and education (Turner, 2003). Telehealth interventions are cost-effective (Gros et al., 2013) and can be provided at any place, which is particularly advantageous for individuals in rural areas and for individuals who cannot access interventions due to ethnic disparities (Zuckerman et al., 2017). An additional benefit of telehealth interventions is that they can occur at any time synchronously (i.e., in real time) or asynchronously (Totten et al., 2016). There are a variety of technologies that comprise telehealth services, including, but not limited to, videoconferencing, text messaging, telephone, and the electronic exchange of images or videos. Given the circumstances of the COVID-19 pandemic, telehealth has shown to be an accessible option for many families as the need for services continues during this time (de Nocker & Toolan, 2021).

Several studies have evaluated telehealth autism interventions. In certain instances, caregivers preferred telehealth services to no services at all (Sutherland et al., 2018). Many studies have shown clinically meaningful gains when low dosage ABA interventions were provided and facilitated by a caregiver (Johnsson et al., 2016; Vismara et al., 2009). The interventions also yielded high consumer satisfaction (Heitzman-Powell et al., 2014; Tsami et al., 2019). Telehealth is considered a feasible platform to deliver certain ABA interventions for autism (Bearss et al., 2018; Ferguson et al., 2019; Machalicek et al., 2016) and has been deemed acceptable for autism assessments (Boisvert & Hall, 2014; Heitzman-Powell et al., 2014; Schutte et al., 2015). Similarly, telehealth speech and occupational therapy interventions have also yielded high satisfaction (Gibbs & Toth-Cohen, 2011; Molini-Avejonas et al., 2015). There is less published research surrounding satisfaction with services when the majority of autism services are provided remotely via telehealth during a national pandemic.

Typically, ABA interventions consist of many therapy hours provided across multiple months with a few behavior interventionists physically present in the child’s environment (Lovaas & Smith, 2003). The absence of an interventionist during telehealth sessions poses unique challenges for caregivers and interventionists because caregivers may need to facilitate and/or implement interventions with little experience. In addition, telehealth interventions may pose unique technical challenges that may prevent the provision of interventions consistently (Johnsson et al., 2016; Tsami et al., 2019; Vismara et al., 2012). Traditionally, speech, occupational, and physical therapy are provided in person and in center for 1 to 2 h per week. During the COVID-19 pandemic, the shift to telehealth ABA, speech, occupational, and physical therapy interventions for autism services provided an opportunity to evaluate satisfaction with telehealth services. The purpose of the present archival study was to assess satisfaction with telehealth interventions for a large nonprofit organization that transitioned interventions for over 10,000 individuals to telehealth during a pandemic.

Method

Participants

Data from a subsample of 10,567 participants who were receiving autism interventions created the survey pool. The participants were receiving one or a combination of the following services: ABA assessment (n = 664), ABA interventions (n = 6305), occupational therapy (n = 4748), physical therapy (n = 575), ABA/social skills group therapy (n = 572), individual speech therapy (n = 6279), and group speech therapy (n = 230). The service recipients were children as young as 3 years of age, and most were in the 6- to 9-year age range (30%). Most of the participants are identified as Hispanic or Latino (50%) (see Table 1 for additional demographic information). These demographics are for the entire population from which the subsample is pulled. Specific demographic for the subsample was not available because the survey was anonymous.

Table 1

NPO participant demographics

Characteristic	Participants		n (%)
	Male	Female
Age
0–3	1,592	484	2,076 (20)
4–5	1,987	591	2,578 (24)
6–9	2,538	602	3,140 (30)
10–17	1,891	478	2,369 (22)
18+	303	101	404 (4)
Total N	8,311	2256	10,567
Service type
ABA assessment			664 (3)
Behavior			6,305 (33)
Group speech therapy			230 (1)
Occupational therapy			4,748 (25)
Physical therapy			575 (3)
Social skills groups			572 (3)
Speech therapy			6,279 (32)
Total N*			19,373
Ethnicity
Asian			1,247 (12)
Black			906 (9)
Hispanic			5,250 (50)
Native American/Alaskan			18 (< 1)
Pacific Islander			64 (1)
White			2,363 (22)
Multiple races			122 (1)
Other			259 (2)
Unknown			339 (3)
Total N			10,567

*Total includes participants who may be receiving more one service

Procedure

Setting

All interventions during telehealth were provided remotely using web-conference platforms. The telehealth sessions were conducted in the participant’s home, with the caregiver present or in the location where the participant spent the majority of their day. The interventionists or therapists were in a different location, commonly in their own homes in a HIPAA compliant room with no other people present.

Technology

Telehealth sessions were conducted via videoconference using a HIPAA compliant Fuze®, Microsoft Teams®, or Zoom® platform. Additional licenses for video platforms were purchased for clinical teams to engage in sessions without the constraint of time limits posed by free videoconferencing services. These licenses provided by the organization allowed access to video platforms in the above manner to both employees and caregivers alike. All employees were provided with the necessary equipment to conduct telehealth sessions. The materials included laptop computers, tablets, and cellular phones, or desktop computers with videoconferencing capabilities. Caregivers who did not have equipment or Wi-Fi access were provided with loaner equipment, which included access to cellular data for use during their telehealth sessions.

Organization

The organization that participated in this study was a nonprofit organization (NPO) that provided interventions to individuals diagnosed primarily with autism. The autism services consisted of in-home and center-based ABA interventions, center-based speech therapy, occupational therapy, and physical therapy.

For ABA services, a three-tier service delivery model was implemented. A Board Certified Behavior Analyst (BCBA®) supervised master’s level clinicians. Each master’s level clinician supervised a number of behavior interventionists who were responsible for implementing interventions directly. Each interventionist had a bachelor’s degree in psychology or related field. The interventionists were trained on how to implement various behavioral procedures during a 10-day training that consisted of in-person and computer-based instructions with role-play scenarios. The therapists also received feedback and additional instructions in the field for at least 1 more week. Supplementary booster training was provided annually, and the master’s level clinician regularly provided feedback and training during session overlaps. Training could occur in sessions in many formats, including, but not limited to, modeling and role-playing new programming being introduced or practice runs of data collection methods that were not regularly used. All training during this time was conducted via telehealth. Prior to March of 2020, all training was provided in person, both during and outside of sessions.

Speech services were delivered via telehealth by a state licensed speech language pathologist (SLP) with a minimum of a master’s level education. In some cases, the speech services were delivered by a state licensed speech language pathologist assistant (SLPA) under the supervision of an SLP. All services were monitored utilizing data collection to ensure participants were making progress on objectives within the individualized treatment plan. Caregivers were encouraged to participate, ask questions, and conduct carryover activities in the home to assist with progress.

Occupational therapy services were delivered via a state licensed occupational therapist (OT) with a minimum of a master’s level education. In some cases, the occupational therapy services were delivered by a state licensed occupational therapy assistant (OTA) under the supervision of an OT. All services were monitored utilizing data collection to ensure participants were making progress on objectives within the individualized treatment plan. Caregivers were encouraged to participate, ask questions, and conduct carry over activities in the home to assist with progress.

Physical therapy services were also delivered via telehealth by state licensed physical therapist (PT) with a minimum of a master’s level education. All services were monitored utilizing data collection to ensure participants were making progress on objectives within the individualized treatment plan. Caregivers were encouraged to participate in session, ask questions and were presecribed relevant activities in the home to support the participant’s progress.

Telehealth services were provided with the therapist in their home or office environment in front of their computer with the participant and their caregiver at their home or in a different location in front of their computer. Depending on the age and skill level of the participants, caregivers assisted to implement the therapy, if needed.

Measures

Surveys

Two surveys were administered to caregivers. One survey was an intervention satisfaction survey for those who opted to receive telehealth interventions, and the second was a survey for those who declined telehealth interventions. The first survey was based on a survey that was administered to recipients of interventions prior to the pandemic for all service types. This survey was administered quarterly for in-person autism therapy sessions prior to the pandemic. During the pandemic, the survey’s questions were revised to solicit a response based on telehealth sessions and administered during the telehealth phase. The survey consisted of a series of questions related to different interventions that the participants may have been receiving. Depending on the number of interventions the participant was receiving, the number of questions varied. There were up to 91 possible questions to answer if someone was receiving all of the services provided by the organization. The questions consisted of basic demographic information, the caregiver’s relationship to the participant, and the total number of therapy hours provided per service type (including consultation, direct 1:1 service, and supervision). The survey was divided into three sections surrounding nine questions in the areas of access, coordination of care, and quality of the provider.

The questions of relevance for the present analysis consisted of those pertaining to the quality of the therapists for each service received including the caregiver’s perception of the therapist’s dependability (question 1), knowledge (question 2), professionalism (question 3), clarity of communication (question 4), and assistance in implementing skills taught to participants via telehealth (question 5). Two additional questions, one about quality of life and one about the participants’ progress, were also presented. Caregivers were asked to rate these areas on a 5-point Likert scale ranging from “strongly agree” to “strongly disagree.” A space allotted for “additional comments” allowed caregivers to provide additional information in an open-ended manner regarding their experiences with telehealth services, but the qualitative data were not included in the present analysis. Figure 1 displays the questions asked in the survey used in this study.

Fig. 1

Telehealth survey questions

When services were declined, a second survey was administered to caregivers for the same period of time during the COVID-19 pandemic. The second survey consisted of five questions that inquired about who completed the survey, prior experience with telehealth/videoconference, reasons for the decision to decline telehealth services, the type of services they were declining, and if they would return to in-person services in the future.

Figure 2 displays the questions asked in the survey of participants who declined telehealth services.

Fig. 2

Decline telehealth survey questions

Satisfaction Survey Administration

The satisfaction survey link was shared via email with all caregivers of individuals who were receiving autism interventions and had an active email and agreed to accept emails (n = 7254). The second survey was administered only to caregivers who declined telehealth interventions (n = 223). The surveys were administered via SurveyMonkey® and were available in both English and Spanish. Caregivers also had the option to request a hard copy to be mailed to them. The survey was administered during a 3-month window during the COVID-19 pandemic from April 2020 to June 2020. Completion of all surveys was voluntary and anonymous. If the participant was not receiving telehealth, their caregiver was led to the survey for participants who declined services.

Data Analyses

The sample of participants who received telehealth from Table 2 was further filtered to only include individuals who received one service type. The purpose of this was to form independent groups whose responses were not influenced by other telehealth autism services and to better isolate perceptions of each service independent of other services. We removed respondents who did not respond to all of the six questions (19 participants were removed, yielding 421 individuals). We then selected individuals who had received only one service (this eliminated 178 individuals, resulting in 243 individuals). Finally, we removed individuals who only received physical therapy due to the small sample size (which eliminated 3 participants, resulting in 240 individuals). It should be noted that descriptive statistics were presented for the entire sample, and inferential statistics were used for this subset of respondents who only received one service type, which excluded physical therapy services altogether. A one-way ANOVA was conducted to assess differences in response patterns as a function of service type, with service type consisting of three levels: ABA (n = 114), ST (n = 91), and OT (n = 34). Bonferroni post hoc tests were conducted for statistically significant results. For questions that violated the assumption of homogeneity of variance, we conducted nonparametric Kruskal-Wallis tests and used Mann-Whitney U-tests when there was a statistically significant result to identify which service lines differed in responses. For all tests, alpha was set at 0.05.

Table 2

Survey participant demographics

Service	Participants	Hours		Other services (n)
	n	Age M (SD)	Mode	ABA	ST	OT	PT	SS
ABA	156	6.4 (4.1)	5–10	0	43	25	1	0
Speech therapy	165	6.1 (3.5)	1–2	65	0	58	3	8
Occupational therapy	107	5.4 (3.2)	1–2	50	58	0	6	42
Physical therapy	12	3.5 (2.4)	1–2	7	3	6	0	5
Total N	440

Results

Of the 10,567 participants in the system, 7254 of the caregivers had an active email address and consented to receive emails from the company. All survey respondents were caregivers for individuals diagnosed with autism who were receiving autism interventions. Caregivers were defined as any adult above the age of 18 who was noted as a primary source of care for the individual receiving services (e.g., parent, grandparent, sibling, stepparent). Approximately, 8% of the caregivers responded to the survey (n = 840). Data from a subsample of 440 caregivers who were participating in telehealth interventions and responded to all the relevant questions was included. Out of the 440 caregiver respondents, 156 of their children were receiving ABA interventions, 165 were receiving speech therapy, 107 were receiving occupational therapy, and 12 were receiving physical therapy. The mean age of survey respondents’ children was 5.4 years (SD = 1.3). Most children received between 5 to 10 h of direct ABA interventions per week and 1 to 2 h per week of speech therapy, occupational therapy, or physical therapy (Table 2). There were 223 survey respondents who declined telehealth interventions.

For the 440 caregivers who received telehealth interventions and completed the survey, the majority of survey respondents selected “strongly agree” for all survey questions. The responses to the questions were separated by service line and are displayed in Fig. 3. Visual inspection of the response distributions indicated the same pattern for all service lines across all of the questions. For the last two questions, the question was presented once, regardless of service type. The majority of survey respondents selected “strongly agree” with the statement that the service provider improved quality of life and that the child made progress during telehealth.

Fig. 3

Responses to first five questions of the satisfaction survey. The bottom graphs represents the responses to questions pertaining to quality of life and progress made for all service types

The mean Likert score responses for each of the six questions per service line are outlined in Table 3. A one-way ANOVA was conducted to compare scores as a function of service type. Prior to the ANOVA, a Levene’s test indicated that the assumption of homogeneity of variance was violated for questions 1, 2, and 3. A Kruskal-Wallis test was conducted for these questions instead, and the results indicated that the service type significantly affected response scores for question 1, H (2) = 7.786, p = 0.02, but there was not a statistically significant difference in responses to question 2, H (2) = 4.96, p = 0.084, or question 3, H (2) = 4.658, p = 0.097. For question 4, there was not a statistically significant difference in means across the service types [F (2,236) = 0.804, p = 0.449. For question 5, there was not a statistically significant difference in means as a function of service type [F (2,236) = 0.844, p = 0.431]. For question 6, there was not a statistically significant difference in means as a function of service type [F (2,236) = 1.812, p = 0.166].

Table 3

Response means (SD) by service type for individuals in receiving only one service

Question	Service type
	ABA	ST	OT
1	4.46 (0.73)	4.7 (0.55)	4.68 (0.53)
2	4.54 (0.67)	4.73 (0.52)	4.48 (0.56)
3	4.58 (0.65)	4.73 (0.63)	4.71 (0.46)
4	4.61 (0.69)	4.71 (0.54)	4.62 (0.65)
5	4.6 (0.67)	4.68 (0.59)	4.53 (0.61)
6	4.31 (0.81)	4.31 (0.84)	4.59 (0.61)

A post hoc Mann-Whitney test using Bonferroni-adjusted alpha level of 0.16 (0.05/3) was conducted to compare all pairs of service types for question 1. The difference in responses for the ABA and ST group was statistically significant, U(NABA = 114, NST = 91) = 4245.5, z = −2.659, p = 0.008. None of the other comparisons was statistically significant (ABA vs. OT, U(NABA = 114, NOT = 34) = 1648, z = −1.528, p = 0.126; ST vs OT U(NST = 91, NOT = 34) = 1491.5, z = −0.401, p = 0.689).

For those who declined telehealth, 61% had not used videoconferencing in the past, and 98% reported that they planned on resuming in-person services when available again. When asked to select amongst the reasons why they declined telehealth interventions, 64% indicated that they did not feel comfortable with videoconferencing, 20% did not have time, 9% did not have a computer or connectivity issues, 4% did not have a location for videoconference sessions, and 3% reported that staff did not explain how to use videoconferencing.

Discussion

Prior studies have indicated that telehealth autism interventions can effectively change a specific behavior if caregivers facilitate the interventions (Ferguson et al., 2019; Ferguson et al., 2020). However, there are few studies that we are aware of that have evaluated the satisfaction of telehealth interventions for individuals with autism for direct skill development, especially during a pandemic when the majority of services and education is being provided via a web-based format. Most prior studies focus on caregiver or staff training. During the COVID-19 pandemic, we had the opportunity to examine general satisfaction using telehealth interventions. Similar to past reviews of implementer satisfaction or treatment acceptability (de Nocker & Toolan, 2021), these results suggest high social satisfaction with clinical staff for telehealth services. For those who declined telehealth, the primary reason indicated was discomfort with the use of videoconferencing.

For those who accepted telehealth interventions, the survey results indicated that the participants were generally satisfied with the intervention outcomes across all autism interventions (i.e., ABA, speech, occupational therapy, and physical therapy), and telehealth improved the participant’s quality of life. The participants rated the service providers as dependable, knowledgeable, professional, and clear in their communications, and that they assisted in understanding their roles. Although all ratings were in the high range, speech therapists rated slightly higher on dependability compared to ABA therapists. One potential explanation for this could be that there were fewer cancellations by the salaried speech therapists versus the ABA service providers, who are paid on an hourly basis. Future studies should evaluate participant satisfaction with ABA interventions for salaried versus hourly staff.

Satisfaction with telehealth services may result in more access to therapies, particularly to those who face barriers due to geography, transportation, cancellations, and other commitments. Providing services in a natural environment versus a center-based model may also open up opportunities that were not considered previously. Telehealth may also provide opportunities to access interventions for caregivers who may not otherwise have an option to participate in in-person sessions. For therapists who traditionally work in center-based settings, telehealth provided an opportunity to see the natural environment, determine ways to best teach skills based on environmental and physical factors and to become more creative in ways to teach a skill.

For caregivers who declined telehealth ABA interventions, the majority reported that they were not comfortable with the telehealth format. The second most common reason why they declined telehealth interventions was limited time. An implication of this information includes enhancing training and support around the use of telehealth for caregivers to increase their comfort level. For those who initially declined and then attempted to use telehealth, satisfaction and confidence increased after just a couple of attempts.

Limitations and Future Research

One limitation with the survey was that the questions were established prior to telehealth interventions rather than being specific to telehealth. This limitation was in part due to the funding source’s requirement to assess general satisfaction unrelated to telehealth. Future telehealth studies are warranted to better identify the barriers that may hinder progress in longitudinal telehealth ABA interventions. Another limitation to consider for future studies is potential reliability and validity issues associated with the current survey. For example, the number of survey questions presented ranged depending on the number of services received. As a result, survey fatigue could have occurred, and the responses may not have been representative of actual perceptions. Randomization of the order of service type presented could alleviate this potential limitation in the future. In addition, within direct ABA services, some clients worked on individual social skills programming while others were engaged in a group format that consisted of a completely different amount of session time. In the survey results, social skills groups were not included due to inability to determine which format services were received and to avoid confusion of social skills goals with social skills services. This limitation can be addressed in any further studies by separately analyzing the responses of social skills group participants.

Based on the present findings, the caregivers of participants who received telehealth autism interventions reported high general satisfaction and indicated an improvement in their quality of life. However, future studies are warranted to evaluate barriers that mitigate telehealth clinical outcomes and to evaluate the participant’s perception of services. Comparing services via telehealth versus in-person services for caregivers, participants and clinicians during the same time period will also make for a more direct comparison of both satisfaction and outcomes.