Clinician Perspectives on Telehealth Assessment of Autism Spectrum Disorder During the COVID-19 Pandemic.

This study examined clinician insights into telehealth assessment services for autism spectrum disorder implemented during the COVID-19 pandemic. 35 clinicians from multiple disciplines across 17 sites in the Autism Care Network were interviewed. Themes identified through qualitative analyses included factors related to confidence in diagnosis (impressions of in-home observation; child and family factors that affected diagnostic confidence; changes in rapport); patient and family factors related to telehealth (perceived family benefits of and barriers to telehealth; factors related to healthcare disparities; factors specific to non-native English speakers); and institutional and workplace factors related to transitioning to telehealth (institutional support; changes to efficacy, attendance, and work satisfaction). Results suggest that telehealth has potential to be an effective tool in autism assessment practice.

The COVID-19 pandemic was an extreme stressor to healthcare systems worldwide. Hospitals administering developmental assessment programs faced unique challenges: while centers were suddenly limited by mandatory shutdowns of outpatient services, families still awaited assessments needed to access intervention. Children with autism spectrum disorder (ASD) are a vulnerable population that benefits from early diagnosis, access to medical care, and behavioral treatment (Fuller & Kaiser, 2019). Prior to the COVID-19 pandemic, children with ASD and their families were facing difficulties accessing much needed assessments and therapies related to multiple factors, including long waitlists and difficulties accessing providers outside of metropolitan areas (Gordon-Lipkin et al., 2016; Zuckerman et al., 2016). Because of restrictions put in place during the COVID-19 pandemic, ASD service providers were faced with the challenge of how to continue providing diagnostic assessments safely to children and families without compromising quality. This challenge created a unique space for innovation in the use of telehealth for ASD assessment. This study describes how telehealth was utilized during this time and provider opinions about the initial implementation of telehealth for ASD assessment by interviewing services providers who are part of the Autism Care Network, a multicenter collaboration of families, providers and researchers working together to drive innovation, quality, safety, and value in healthcare for individuals with ASD.

Telehealth, the practice of using telecommunication technology to provide health services over geographical distances, has been developed and studied over the last several decades (Nickelson, 1998). Telehealth has the potential to allow a wider group of individuals to access healthcare services by overcoming barriers related to transportation, time off from work, childcare, and geographic location. Reviews examining the use of telehealth to provide both psychological assessment and treatment indicate that it appears to be a feasible modality for providing a range of services to diverse populations, and is generally acceptable to both patients and clinicians (Backhaus et al., 2012; Gros et al., 2013). There are several factors that have prevented mainstream adoption of telehealth including variabilities in regulations and reimbursement of telehealth services, no equivalent standard of care for telehealth compared to in-clinic assessment, and variable technology and internet access (Ramtekkar et al., 2020). Adapting autism diagnostic assessment to telehealth also presents a unique set of challenges.

Gold standard assessment models for ASD include a caregiver interview to gather historical information including information on early development, observation of the child’s social communication skills and behavior, direct assessment of cognitive, academic, and language functioning (as appropriate to the child’s age), and parent and teacher questionnaires assessing symptoms related to ASD, general socio-emotional functioning, and adaptive functioning (Zwaigenbaum et al., 2015). Several of these processes would likely translate easily to telehealth (clinical interview, standardized interview measures, electronic parent and teacher report forms), whereas obtaining standardized direct observation and assessment of skills may be more challenging due to a lack of control over the family’s home environment and the need to rely on family members to help administer assessments and manage behavior. A recent review of telehealth measures designed to assess early signs of ASD highlighted the development of several promising behavior observation tools for early screening, although more research is needed to demonstrate their reliability and best practices for use (Dahiya et al., 2020). A scoping review of telehealth use in autism assessment and diagnosis found high agreement between telehealth and in-clinic evaluation in clear-cut cases of ASD, as well as high levels of satisfaction amongst families and clinicians (Alfuraydan et al., 2020). Assessment by telehealth was found to be less effective for more complex presentations. Small sample sizes and other methodological concerns indicated a need for further research. There is also initial support indicating similar findings between telehealth versus in-clinic administration of standardized cognitive and language measures (Hodge et al., 2019; Waite et al., 2010). Despite this support, telehealth is still not widely used for ASD diagnostic services. Potential barriers include technology needs, training in the use of technology, adaptations to ethical standards by regulatory boards, and changes to insurance codes to allow for reimbursement. During the COVID-19 pandemic, telehealth regulations were loosened, temporarily mitigating many of these barriers and presenting an opportunity to use telehealth more broadly (Centers for Medicare and Medicaid Services, 2020).

COVID-19 related restrictions caused significant barriers to administering traditional models of ASD assessment, but also provided unique opportunity for healthcare centers across the nation to begin providing services through telehealth. The current study examines how service providers addressed these challenges by interviewing ASD specialty providers across 17 sites participating in the Autism Care Network. We used the data collected to understand how satisfied clinicians were with the telehealth assessment services implemented and what influenced their satisfaction. We also explored clinician insight in three areas: factors that influenced confidence in diagnostic outcomes; patient and family factors related to successful use of telehealth assessment; and institutional and workplace factors.

Methods

Participants

Between September 2020 and May 2021, we enrolled and interviewed 35 clinicians from 17 (of 20) Autism Care Network sites across the U.S. and Canada. Interviewees included clinical psychologists, developmental behavioral pediatricians, psychiatrists, occupational therapists, nurse practitioners, and speech-language pathologists, all of whom participate in diagnostic assessments for ASD.

Measures

We used a semi-structured interview guide that included open-ended questions and probes. The interview guide was developed by the study team and reviewed by the Autism Care Network committee of behavioral scientists, which includes a variety of Autism Care Network clinicians and family advisors (who have children with ASD). Participants used Research Electronic Data Capture (REDCap) to complete a brief online pre-interview questionnaire assessing clinical approaches before and during the COVID-19 shutdown. The full pre-interview survey is included in Appendix A. Only responses from question 1 were used in the analyses.

Procedure

The study protocol was determined as exempt human subjects research by the Mass General Brigham IRB. The study was approved by the IRB at Nationwide Children's Hospital and a waiver of documented consent was granted as the study involved no greater than minimal risk. A data sharing agreement was executed to share data between researchers at these two institutions. We recruited subjects through the Autism Care Network. The Autism Care Network includes 20 sites across the United States and Canada. Sites are typically located at large academic medical centers located in cites, although most sites serve a diverse population. Sites vary widely in the number of families they serve from rural locations (10–52%) and who use public insurance (3–68%). Of the 20 Network sites, 15 serve a population that is mostly 12 years old or under (62–89% of the total population served), three sites report that 27–48% of their ASD patients are 12 and under, and two sites did not report data. The number of “providers” (i.e., developmental pediatricians, gastroenterologists, neurologists, psychiatrists, nurse practitioners, psychologists, speech therapists, social workers, geneticists and clinical dieticians) that made up each site varies significantly, ranging from 3 to 380 providers (Mean: 57; Median 25). Each site involved in the Autism Care Network has a principal investigator (PI) that we contacted initially to share IRB-approved advertisements at their site to clinicians who were involved in assessment of ASD. Trained members of the study team then sent advertisements and the fact sheet to site contacts, who contacted the study team if interested. We contacted all 20 Autism Care Network sites and conducted interviews at all sites where clinicians expressed interest and were available to schedule an interview (N = 17). Clinicians self-reported eligibility. In order to be eligible, they had to be 18 years or older, speak English, have access to the internet or a smartphone, and participate in assessments and/or treatment for ASD at an Autism Care Network site or similar autism specialty clinic. Interviews were scheduled and a link to the pre-interview survey was sent to interested clinicians. Implied consent was obtained by confirming receipt of the study fact sheet and completion of the pre-interview survey. Individual interviews were conducted by trained members of the study team via a HIPPA compliant videoconferencing platform. Interviews were audio-recorded and transcribed. Limited data sets were shared across institutions containing interview and survey data. The data did not contain any personal identifiers that could be used to link the data with individual subjects.

Four members of the study team, authors 1, 2, 3 & 5, were responsible for data collection in the virtual interviews. Authors 1 and 2 are clinical psychologists with expertise in ASD assessment and treatment. Author 3 is a senior study coordinator trained in qualitative interviewing and methodology. Senior author 4 is a clinical psychologist with significant expertise in assessment and treatment of ASD and hospital administration, who was primarily responsible for the study design, and senior author 5 is an expert qualitative researcher.

Data Analysis

Qualitative Analysis

Qualitative analysis was conducted using a framework approach, a method for interpreting themes through the continuous comparison of codes (Ritchie & Lewis, 2003). The framework approach involves the creation of a thematic framework (or coding structure) for use in categorizing data by themes and subthemes, which we completed using Dedoose v.9.0.15 (Dedoose, 2019). Information was initially elicited from interviewees using open-ended questions and follow up probes in nine domains: (1) Differences in assessment services pre-COVID versus post-COVID, (2) Comfort with tele-assessment, (3) Child and family factors effecting success of telehealth assessment, (4) What worked less well about assessment services post-COVID, (5) Clinician satisfaction or dissatisfaction with tele-assessment, (6) Changes in efficacy and costs of tele-assessment, (7) How did telehealth assessment influence healthcare disparities, (8) Changes to clinical rapport, and (9) How will services change in the future. After all interviews were completed, the study team reviewed several transcripts prior to coding to discuss initial impressions. A coding framework was then developed by reviewing interview domains and refined iteratively through coding a random sample of transcripts. Intercoder consensus was then established between Authors 1, 2, 3 and 5 by coding together another random sample of transcripts and comparing and resolving discrepancies in coding. Once reliable coding was established, the remaining transcripts were split up between Authors 1, 2, 3, and 5 and independently coded. Authors 1 and 2 then reviewed the coded data to identify overarching themes and relationships between themes. An iterative process was used to organize and combine themes as appropriate and refine the statements describing each theme. Participant quotes were chosen to further illustrate these themes.

Quantitative Analysis

Quantitative data from the pre-interview survey was conducted in Microsoft Excel Version 2002. We used descriptive statistics (frequencies, percentages) to explore participant responses describing the level of COVID-related shutdowns at their clinics.

Results

Quantitative Results

Of 20 Autism Care Network sites contacted, 35 clinicians from 17 sites expressed interest in the study and were available to schedule interviews. Clinicians represented sites from 12 different states in the U.S. and 2 Canadian provinces. All 35 clinicians completed the pre-interview survey. When asked to describe the level of shutdown of in-person clinic services, 63% participants reported complete shutdowns (100% of in-person services were stopped), 34% of participants reported partial shutdowns (closed for some services), and 3% of participants reported no shutdown (clinic never shut down, 100% of in-person services remained open). Of the 12 participants who reported a partial shutdown, we asked them to estimate what percentages of services remained in person. Reported percentages ranged from 5 to 75% with a median of 28%.

Qualitative Results

General Satisfaction with Telehealth Services Implemented During the COVID Pandemic

In general, most clinicians felt satisfied with the telehealth services they were implementing, although the responses ranged from very satisfied to unsatisfied (see Table 1). Level of satisfaction was related to numerous factors. Clinicians who found new challenges rewarding and enjoyable, who felt as though they had support from their colleagues and leadership, and those who had access to strong technological infrastructure tended to be more satisfied with the transition to telehealth. During their interview, one clinician commented, “I think we are doing a good job, and I think we are doing right by families, and so that leaves me very satisfied.”

Table 1

Sample quotes illustrating general clinician satisfaction

General clinician satisfaction with telehealth services during the COVID pandemic
High satisfaction	“But clinically, I've been really pleasantly surprised, and I found it rewarding and enjoyable. And it's been a good challenge. It's kept us all on our toes and had the opportunity now to train other people in virtual assessments, interns and people in supervised practice. And I've enjoyed that. I've enjoyed working with them.” “How satisfied am I with the actual clinical work? I think we're doing a good job, and I think we're doing right by families. And so that leaves me very satisfied.” “I think that the other psychologists, at least in my center, are really satisfied with it. I think overall we really like it, and we really hope that a lot of pieces of it are going to stick around.” “I like them. Honestly, I feel like I've become a bit of a—I've become a bit of a convert and a bit of the champion of the why do we want to do this virtually.”
Mixed satisfaction	“I think we are all fairly satisfied, more so than we expected” “It's really hard to say…it's hard to give you just a binary answer like I'm really satisfied versus not. It's really in that range because there are some definite pros and cons.”
Low satisfaction	“We are doing some video assessments, but I have not liked those… There have just been a few where I've done it by video and felt confident in the result” “And I think maybe I started to get resistant to that. I don't think it was a conscious, "I'm not learning this." But it's like, I don't want to get good at this. If this is the new wave of mental health, then I don't want to be part of the new wave of mental health”

Confidence in Assessments Completed by Telehealth

Most clinicians interviewed agreed that telehealth worked well enough or was even preferred for clinical interviews and feedbacks. However, there were mixed responses related to clinician confidence in using telehealth to conduct behavior observations needed to make a final diagnosis. This section outlines the strengths and weaknesses shared regarding virtual behavior observations, as well as the clinical presentations and child/family factors that lead to variance in clinician confidence. Difficulties related to completing other standardized testing by telehealth are also reviewed (See Table 2 for additional quotes).

Table 2

Sample quotes illustrating other themes related to telehealth assessment

Confidence in assessments completed by telehealth
Clinician confidence
Higher confidence	“And now, I feel really confident that the data I get is reliable and supportive of an appropriate diagnosis.” "I think, in general, for me, I think what I'm doing is working well, so most of the time, I feel like I can do a good valid assessment and that that works well through telehealth."
Lower confidence	“I mean, we're definitely recommending that all these kids need reevaluation…Everything is an estimate right now. But there's certain kids where we feel very comfortable with the estimate.” “We had not done telehealth at all. So to take something that we were so used to in-person and just shifted to telehealth with measures like the tele-ASD-peds that was even still under research was really uncomfortable for me.”
Behavior observation by telehealth
Strengths	“But I feel like I'm getting a really good sample of the behaviors that I want to get” “So clinically, I feel like I am so much more informed when I'm watching children playing with their own toys in their own home as opposed to come into my office and getting mad because we tried to check their blood pressure, and getting frustrated with the traffic or sensing their parents stressed about getting to the appointment…” “So I think, with some of these kids, we're actually getting like a truer picture of who they are than we would if we saw them.” “And I actually think, with some of the little kids, we're getting a better picture than we would have gotten when they come into clinic because so often, when they come into clinic, it's a situation where parents are like, ‘They're really inhibited. They're not talking as much.’” “I really liked what that added to the parent’s role in the evaluation. I think having the parent do some of those presses allowed the parent to see their child’s interactions from another perspective at times, could buy in differently because of their role in that and as well as just provided also some opportunities to kind of touch into things that could be later brought up in intervention”
Weaknesses	“The thing that I'm not comfortable with is the confidence that I have that I'm getting the same quality of information from behavioral observations that I have in the past.” “Again, it gives me something I can score, that I can show to a family. It gives me some activities that I can use to structure a remote visit. I don't feel they are as rich as the ADOS in terms of helping diagnosis … ADOS usually gives me a more secure feeling about the impressions I develop.” “Compared to much of the rest of how we do remotely, I don't think that the remote ADOS is– it didn't compare. The in person ADOS gleans much more reliable information, in my opinion.” “Because they don't have those benchmarks, it was harder to know, well, is this really autism or is it just normal Zoom behavior or Zoom etiquette for a three-year-old.”
Clinical presentations and child/family factors effecting confidence	“Yeah, I think everybody's starting to feel like, look, you know, when I see video of kids, and I see them live and face to face, I can see them in the background stimming during a telehealth visit and they're not talking and they're not making eye contact, I kind of feel like, for somebody who's seen kids with autism, I'm not so sure I need an ADOS to tell me that this kid has autism.” “There are a few kids where it’s impossible to do the evaluation because they can’t sit still.” “I think what's hard is– eye contact is hard, but you can tell they get it. You can see if they're making eye contact with their parents. You just can't see if it's subtle, where they're maybe looking at your mouth instead of your eyes. And things like their– you don't always get their sensory stuff because their house environment might not elicit some of their sensory issues, whereas, some of the toys did.” “The one’s who I struggle with, we struggle with as a team, are the higher functioning children, like they’re five, and their verbal. Because then I want to see their face. I want to be right in front of them to catch the more subtle errors and omissions, and I’m not seeing it on video.” “We haven't seen a huge increase in physical abuse, but we also aren't seeing these kids in-person so we can't be totally certain…So that's been something we've talked about as a division that we're at least somewhat worried about but don't know what to do about it.” “I will say I identified potential sexual abuse in one of my patients, and that did not go– that was hard. That was damn near impossible because I couldn't see who was in her room– I couldn't see who was there, but it had to be done.” “So I guess with teenagers, I worry about suicidality, and I worry that I won't get– I worry that the parent is listening in when I'm talking to them via telehealth.” “The only time I didn't like it is when you had parents who were skeptical of the process. So if your confident in a diagnosis based on what a parent had reported and what you had seen, and then they're questioning, "Is this a legitimate way to get a developmental diagnosis?" Is this evaluation as robust as it needs to be? And I get that, for a diagnosis that you will have for life.” “Those were the only visits that I was not overly comfortable in, when you felt like you were getting a lot of pushback from a parent. And the only thing I can say is, "This is just what I'm seeing. I'm happy to evaluate in person, and we can defer any diagnoses or anything until that time," and that's all you can do. So I think my comfort level was at a good point unless a patient was not.”
Building and introducing telehealth models of care	“And I think when this first hit, we were all trying to put feelers out and get a sense of what the autism community as a whole was doing so that we weren't just a solo project over here doing our own thing, and trying to get a feel for what everybody else was doing, what was going to kind of become the standard during COVID times, and trying to have some consistency across other sites as well.” “I think trying it. Honestly, people were a bit reluctant, like how is this going to work? Am I even going to see what I need to see? And then I think– so trying it and that kind of peer support– we had lots of meetings quite frequently about like, how's it going for you? I've done this. What works? And just sharing ideas.” “I'm learning through my colleagues because they are very good at it. And it's like, "Oh. I'm going to watch you." And now, I can do it on my own.” “I think it's all dependent on how comfortable the professional is with, frankly, using a computer and being worried that they're going to break something.” “So I had the ability to draw on the experience where I've asked– I've been coaching parents to do things. And I think, for lots of people, that's the part that's really difficult, is being directive with parents.”
Other testing by telehealth	“And then, yeah, the really oppositional kids that don’t want to do anything, it’s easier when we’re in-person to give them reasons why they might want to.” “And they have been trying to do three hours of remote testing in a block[…]And I've watched, and the kids are like, "Ugh." They hate it. So I've been trying to get them to be able to break it up. […] the kids are doing the best they can, but they haven't even tried the big kids with autism yet.” “We use subtests from the WISC, and you can get a nonmotor kind of IQ score. And so I think the tricky thing there, again, is getting a child to sit and attend to a computer in their home. You want the parent nearby, but you don't want them giving hints or help or adding to your instructions or changing your instructions. And sometimes, that can be tough to navigate.” “Personally, I think that I would really value doing cognitive assessments again.” “So I think the drawback is not being able to do some direct assessment to get some standardized scores for kids.” “I felt more confident with some of them in potentially doing telehealth if we had additional information.” “It was really hard to get vetting information from another caretaker or therapist or anything like that because mostly everyone was virtual as well.”
Establishing rapport	“I feel like I can do it. I think it goes okay with families. I think getting the kid to sit there and engage remains harder.” “So I think when it’s a video appointment, it has gone pretty well. It’s not ideal. It’s not the same as having them in person. But I have felt that was pretty manageable.” “Initially I was like, ‘Oh we’re not going to be able to establish rapport and whatever.’ That sort of fear went away quite quickly. I feel like you could truly establish rapport this way. I really do. And I’ve seen it over and over again.”

Clinician confidence level—Confidence in accuracy of diagnoses made by telehealth assessment varied significantly from very confident in many cases to not at all confident. Many clinicians felt that, in most cases, their observation of the child’s behavior was sufficient to make an accurate diagnosis. As one clinician noted: “Now, I feel really confident that the data I get is reliable and supportive of an appropriate diagnosis.” A minority of clinicians noted that they did not feel comfortable making any final diagnosis by telehealth or expressed strong reservations about the diagnoses being made: “we had not done telehealth at all. So to take something that we were used to in person and shift it to telehealth…was really uncomfortable for me.”

Strengths and weakness of observation by telehealth—The primary identified strength of completing an observation by telehealth appeared to be the opportunity to see children and families in their home environment. This allowed for a more natural observation, getting to see children play with toys they most typically engaged with around those with whom they most typically interact. Some clinicians also stated that parent participation in telehealth assessment provided an opportunity for parents to see their child’s behavior from a different perspective, which was beneficial to parent engagement in the evaluation process: “I really liked what that added to the parent’s role in the evaluation. I think having the parent do some of those presses allowed the parent to see their child’s interactions from another perspective at times, could buy in differently because of their role in that and as well as just provided also some opportunities to kind of touch into things that could be later brought up in intervention.” One clinician mentioned, “I actually think, with some of the little kids, we’re getting a better picture than we would have gotten when they come into clinic.” Clinicians reported that this was particularly true for children who present with significant anxiety or behavior that was inhibited for other reasons in a clinic setting. The major concern reported was not getting the same quality or reliability of observation when it was completed by telehealth. Clinicians often contrasted the various observation protocols they were currently using with the Autism Diagnostic Observation Schedule (ADOS; Lord et al., 2012), which is considered to be a gold standard tool for assessing ASD symptoms. They often stated that they missed the consistency of opportunities for observation presented by the standardized and controlled administration of the ADOS, contrasting this with the opportunity by video for a more varied and naturalistic observation in the home. There was also concern that social behavior may be just different when interacting by video and we do not yet know how to best interpret this.

Clinical presentations and child/family factors that effected confidence—Clinicians outlined different clinical presentations that affected confidence in diagnosis. In general, symptom severity, age, and developmental impairment impacted the ability to make ASD diagnostic decisions by telehealth. More specifically, younger children and children who were more developmentally impaired were easier to assess, whereas older children with more subtle impairment were more difficult to assess over video conferencing. There were also child and family characteristics that could make it challenging to conduct a satisfactory virtual behavior observation. These included children with high activity levels where it was difficult to keep them in the camera view, children who actively refused to interact over video, parent/child interaction difficulties which effected the child’s cooperation, highly anxious children who were inhibited by being on camera, and children with variable social skills where telehealth made it difficult to notice more nuanced deficits. While not directly related to making an ASD diagnosis, it was noted that certain safety issues are difficult to assess for or address by telehealth such as suicidal ideation and behavior, abuse concerns, and significant aggression. Finally, it was noted that clinicians felt less confident making a diagnosis solely through telehealth assessment when families were more resistant to the diagnosis due to this modality being less widely accepted.

Building and introducing telehealth models of care—To develop telehealth models, many sought out trainings and webinars in both general telehealth practice and on specific telehealth assessment measures, while others consulted with partner agencies. Many clinicians reported feeling high levels of initial doubt and anxiety until they actually tried assessment by telehealth. This experience led most who attempted telehealth assessment to realize that this modality was not so different from their in-clinic practice. It was also noted that working as part of a team to problem solve difficulties regarding virtual behavior observation and being able to observe peers first improved confidence. Having experience with any sort of parent coaching protocol (e.g., Parent Child Interaction Therapy) in clinic or by telehealth before the pandemic helped a few clinicians feel more confident in directing parents through a telehealth observation. Comfort with learning new technology was also a factor that some clinicians noted as playing a role in how quickly they adapted.

Other testing by telehealth—Only a few sites attempted to complete standardized, direct assessment with children via telehealth (e.g., cognitive, developmental, speech and language, etc.). Those that did conduct cognitive testing reported being able to collect good information, but that there were limitations, such as not being able to manage behavior effectively, having to rely more on parents for behavior management without interfering with accurate administration, and challenges in maintaining children’s attention over longer testing sessions. Tests available by telehealth were also limited to children age 6 and older and generally with milder cognitive impairments. Clinicians reported missing having direct testing results of cognitive and developmental level to add additional context to the diagnostic picture. Spending more time tracking down information from additional sources, such as school reports or outside evaluations, was stated to be helpful in increasing diagnostic confidence, although this information was often more difficult to access during the pandemic as school and other services were also remote.

Establishing Rapport—Establishing rapport was noted to be somewhat different over telehealth, but the majority of clinicians indicated that establishing rapport was far less difficult than they were expecting, especially with parents and caregivers. Building rapport with children was more variable. A few clinicians found establishing rapport over telehealth more challenging, particularly when video conferencing was not possible and only audio conferencing was used.

“And so far from the families, they seem to be satisfied with it.”

“I think a lot of parents are satisfied with it. […] so many of the families feel like we've gotten an accurate picture of their child. […] they feel that it's been an accurate assessment and a timely assessment.”

“I mean, there are the parents who really only want the in-person […] But in general, I think it worked for more than it didn't.”

“And I've had some highly-educated, well-resourced families that will just say, “No, we want to come in person.” That's the experience that they need as part of what they feel is medical care.”

“So just the normal demographics that you'd expect, right? The ones that had more resources and better stable internet connections had an easier time with telemedicine.”

“But I would say that would be the other big barrier is that there were a small, but certainly significant group of people that either couldn’t use the technology or the technology wasn’t available.”

“Then you have the technology barriers, where some parents are tech-savvy, and some are not. They find it intimidating and some have refused to use zoom, even though we set it up for them.”

“That whenever you have a parent that may already not be taking the process seriously—so they're trying to drive or they're in the drive-through with their family and I don't have their attention—those parents are the ones where I want to bring in.”

“I’ve had a few where, 'oh, well, we’re on our way driving somewhere. Can we do the visit right now?' Not a new diagnostic like that. 'Well, I’ll hold the phone and you can talk to him when he’s sitting in his car seat. 'I don’t think that’s going to work. I’ve had a few like kind of really like, 'oh, my goodness, I can’t believe you thought that was going to be a good idea.'”

“And so I picture these families with four kids and toddlers and screaming kids on buses and just actually getting to the hospital. And what a stressor that is for some families. I love this for them. I love the fact that they don't have to do that.”

“I've been able to see patients as far as a five-hour drive away, […] in some ways—for certain families cut out stress because you don't have to drive to our center in traffic, wait for the hour, then drive home, right, because that disrupts all of the other siblings' schedules and days. It disrupts parents' work schedules.”

“And those who were already working from home, I think really appreciate it because they didn't have to take a lot of time off work to come in.”

“And I imagine there might be value in being in the comfort of your home, in your safest space, going through a difficult process. There might just be a different feel or a different sense of safety or emotional safety. I don't know. I'm guessing that it could also have a positive benefit in that way.”

“Reflecting on how a parent might feel after receiving a diagnosis, and then they're like in their home, they can have a cup of tea, they can just relax rather than driving through disgusting city traffic for an hour and a half. It's just a different– it's just a different feel. And I wonder whether that's actually a nice thing to preserve for some families.”

“The other thing that I think that I think it works really well for are split families. I got a chance to meet so many dads that I'd never seen, grandmas, grandpas, teachers, parents showed up. I was like, "This is awesome.” Recently had a mom share the link with the– there was a bunch of stuff going on at school. So she shared the link with the assistant principal who then called me and was like, "Are you sure it's okay?" I'm like, "Oh, heck yeah. Come on." And so fantastic opportunity. Faster for everybody, got everybody on the same page.”

“I think that's the question that we don't have the answer to because we were only able to collect information from those people that participated, but I do suspect that they probably increased disparities or that there were individuals who were not able or chose not to participate in telehealth that would represent that.”

“That's a very hard question to answer. […] What I don't know, is whether there are patients who don't have access. […] I don't know the answer to that because I only know who I'm seeing. I don't know who I'm not seeing.”

“I think the same folks that were going to have a really hard time getting into the clinic for transportation or just general life chaos reasons were still having trouble getting in. I don't think that changed. I think that the availability of telehealth for the majority of our patients, though it was very few that had poor access to internet, I think that that increased their access to the service and actually reduced some of the barriers related to coming into a doctor's office and allowed me to establish some trust and rapport before the family had to physically come in and see me […].”

“I feel like it's probably like the same families who have this set of factors that are going to make them probably more likely to be late for your appointment, right. Like they're taking public transit to get there and all of those things, right. And then that same thing– that those same factors just apply to their home environment, like their Internet connection is not the strongest, perhaps, or they've got multiple kids to sort of wrangle in this situation.”

“If anything, it's made the evaluation process more accessible to people, because they don't have to try to make it in with medical assistance rideshare or trying to manage getting childcare for their kids while they come in. So I just think it really has improved accessibility for people.”

“And then I think, broadly, it also reduces disparities of no transportation, time out of school, time off work. […] And so I think that that's where we see huge opportunity to reduce disparities through telehealth care.”

“I think the disparities are there are patients and families that don't have the resource to have the devices that they need and the Wi-Fi and lots of technology pieces that were happening.”

“I think there is a very big aspect of social determinants of health that can come into play in terms of who these assessments work well for and who they don't.”

“And so we learn a lot about the social determinants of health from these interactions. So what does the child's play space look like? How good is or how able is the parent to kind of engage and play with them, or are there other barriers to that? So there's a very rich information.”

“On one hand it's helpful I think for clinicians in creating understanding of, "Wow. This is your world." I get it a little bit more than I could have understood in the clinic or– when you're seeing somebody's home, I think it really helps to provide perspective, or gives you empathy for the fact that this parent is trying so, so hard. And so I think I can provide more empathy in that way.”

“The truth is that the families for whom English is not a strength are really being probably further disenfranchised.”

“The non-English -peaking families had a little bit more difficulty. And I think they were also the ones that were less comfortable doing virtual visits. Generally speaking, they're the ones who have wanted to come in in person. They've also been the ones that have been more hard hit with COVID.”

“So our predominantly Spanish-speaking […] families are not coming in to the center as much. […] It's because I would imagine they're experiencing a lot more hardship and barriers, or phone number's disconnected or unresponsive. And so I think actually those children, just probably they haven't gotten any services.”

“Doing virtual assessments with an interpreter has just proven to be very clunky because you're trying to coach in the moment. And if it has to sort of be relayed through an interpreter, that moment might be gone. And it just gets a lot more challenging.”

“I've always not enjoyed using a phone interpreter. But the times that I do have to use one, not only it's like I can't see the interpreter, I can't see the family. And those are my most dreaded visits over the phone.”

“I am doing many evals over telehealth that are inclusive of a interpreter. And might be some to my style but I feel like, now that that person is an equal sized box on the screen, the interpreter is really in the appointment.”

“100%. Whatever we need, whenever we need it, and whatever they could provide, they provided it.”

"Because I did not feel that there was a ton of support”

“I think one thing that really helps is that our hospital is very supportive of new initiatives in general.”

“I just wish we had a bit more vision that didn't have to come from the clinicians because we were feeling the pressure and all of the exhaustion that just comes from changing our whole practice”

“I feel like they did what they could do in this situation based on how prepared or unprepared they were as an institution. So I think the only thing that could have been different is if they were more ahead of the game in terms of telehealth technology to begin with.”

“Some of the barriers really had to do with our own technology related to the healthcare system”

“And then I think the last thing is that each team was only given one camera. […] So just scrambling to help everyone figure out how they could access a computer should they need to work from home”

“And good communication. I've really appreciated all the communication that comes out about decisions being made and changes being made, and things like that.”

“Overall leadership did a really nice job of putting out some really clear guidelines […] so there wasn't a guessing game, or it wasn't like, well, this division or this area is pushing harder for you to ramp up than this area.”

“So giving people like that the flexibility to manage their own time, I think, that was hard to watch that not happen and people be like, “I've never worked harder for you than I am right now. Please, have the respect to know that I can solve this problem.”

“So I feel like there was maybe six weeks of tolerance for do your best, do what you can, get caught up on reports. And then I feel like when that shifted about six weeks in, it almost became like now you have to do catch up. […] And it just felt like that sensitivity, understandably, just couldn't be sort of extended as much anymore. So I think that was a tough change.”

“What continues to be an ongoing issue is more administrative support for emailing the Zoom links to families. […] we all are required to send Zoom links to all of the patients we see that day.”

“And we did a lot of hand-holding. I had phenomenal support. My administrative front desk staff was on it.”

“No. I think speed is always an issue. But, hey, you can only develop this process so fast… That was just a reality of– we shut so quickly, like a door shut. What do we do now? And then we all realized we have to go virtual. And then we were all scrambling together and doing our best. But it didn't take very long to get up and running virtually.”

“So I think early on, there was definitely some efficiency cost in terms of just getting families up and running with the portal and how it works.”

“So in some ways, it sounds like the switch to telehealth actually provides an opportunity for the entire process to be more efficient, potentially.”

“I feel like the interviewing time is shorter

We don't have to wait for the family to be roomed, and we don't have to wait for them to go to the bathroom before the visit starts, and we don't have to wait for them to be checked in.”

“And it's also good for space utilization. Because we don't have– I don't know, maybe other people have tons and tons of space. We never have enough clinic space.”

“[…] a great boon to my quality of life is that I can type as I talk to people. And so I can generate a fairly decent report like during my interview or at least kind of the bones of it.”

“And also with the online, doing it by telehealth, it allows us to just type everything as we're going to. It feels less intrusive.”

“I think I still go through the same steps. I think in general, the telehealth piece can just take longer because we're waiting to connect … So oftentimes, by the time we get going, we're already like into the appointment timing.”

“I will say that this assessment model is less efficient. […] I would say time as well as documentation takes just a little bit longer because you're writing up your observations probably more […] because you want to make sure that data is there very clear as to "these symptoms are present and these symptoms weren't.”

“I think the cost in the assessment in terms of the resources is a cost in terms of us as a department not really adopting these virtual assessments in terms of– there's usually two of us on an assessment. And that's, again, because the diagnostician have felt that they sort of want that extra set of eyes.”

“I definitely feel like I work more. But again that's not all just because of my assessment, per se. It's also just the added responsibility of doing my own scheduling and calling people. And that piece can be overwhelming.”

“So we still occasionally get no-shows and cancellations. For sure, cancellations. No-shows though, we do call right away. And sometimes they're able to jump on the call. So I think that that's improved somewhat.”

“I think for some of the families that would have normally cancelled or just no-showed– I feel like the rate of no-shows have gone down, because it's really easy to just be like, ‘Oh. I'll just click this button.’ Or, ‘I'll just answer my phone.’ In fact, most of the time they have forgotten. And it's like, but they've answered their phone. So they're like, 'Sure. Sure. Sure. Go ahead.’”

“We have a really strong show rate with telehealth, and I think a part of that is, it's a fairly intensive process to work with our clinic to get telehealth signed up and to get the appointment converted… So I think just the timely touchpoints from our support staff really helped the show rate.”

“At the beginning, I think we did have data to demonstrate a reduction in no-shows, but I'd say around August, September, that started to level out, and then we started to see an approaching rate to our previous no-show rate.”

“With some of our no-show rate, I feel, ended up stabilizing. But the earlier part, it was definitely a higher rate of showing.”

“I would say in the second six months, I've had a lot more no-shows and cancellations.”

"So I am fairly certain that the rate of no-shows is the same as it was when we were doing the vast majority of our appointments in person.”

“I think it’s also socially isolating for the provider. In the middle of the telehealth time, it was kind of weird going all day without seeing humans in person.”

“Zoom fatigue is real.”

“I mean, my work-life balance is, I guess as challenging as ever because it’s all sort of happening in the same pot right now.”

“One of the things that I feel like is a little more lacking is the teaming that we used to have on the fly. Because we were all in clinic we were all there, we were together, so you would have more opportunity for consultation.”

“[…] in terms of work efficiency, I don’t have to spend that time commuting. I can work on patient paperwork. I can prepare. I have more time to prepare for my visit. So I really like that.”

“I think that being able to toss dinner in the crockpot and then see your next patient has been so amazing for a work-life balance perspective.”

It was acknowledged that completing a diagnostic assessment by telehealth was not ideal for everyone. Most sites had a plan for bringing patients in clinic for an assessment when symptoms were less clear, or a good observation could not be made by telehealth. Several clinicians, however, stated that few patients needed to come back for an in-clinic appointment, as they were able to confidently make a final diagnosis in the majority of cases. One clinician commented “There was like 5% of kids where I was like, ‘I think we should not [make a telehealth diagnosis].” Multiple clinicians made similar estimates that between 80 and 95% of children who were seen were able to be diagnosed by telehealth assessment. Clinicians also noticed that these patients continued to be difficult to diagnose even at an in-clinic appointment. In other words, “A clear kid is a clear kid, and they're clear in the office and they were clear on video. And the tricky kid is a tricky kid, and they're tricky in the office and they're tricky on video.”

Patient and Family Factors Related to Telehealth

While many clinicians were feeling confident in the assessment activities being completed by telehealth, it is also important to understand the possible impact of patient and family factors on telehealth services. The following section outlines provider perception of families' satisfaction with and willingness to try assessment by telehealth, possible barriers to accessing telehealth services, and possible benefits to telehealth services for families served. Possible benefits and costs of completing services by telehealth to those experiencing health disparities and for non-native English speakers are also outlined (See Table 2 for additional quotes).

General family willingness and satisfaction—Clinicians reported that many families were very satisfied with their telehealth assessment experience. Providers reported that families were often initially skeptical about telehealth evaluations but were pleasantly surprised with how they went. Often times some pre-work (e.g., a phone call or letter sent before the evaluation appointment) was needed to help families understand how an evaluation could be completed by telehealth. Even with this pre-work, some families refused to do the evaluation by telehealth preferring to wait for an in-clinic evaluation. There was not necessarily a clear pattern observed as to who refused to do telehealth, although concerns about the accuracy of telehealth evaluation were often reported.

Possible Barriers to telehealth—Clinicians noted that higher socioeconomic status (SES) families with greater access to and understanding of telehealth had the easiest time completing virtual evaluations. Providers reported that one of the biggest potential barriers to effective telehealth for any given family is having adequate technology. Not having appropriate hardware or sufficiently strong internet connection or difficulty using technology were the biggest barriers to families not being able to participate in telehealth assessments, although providers noted that fewer families had technology issues than expected. Another potential barrier was identifying a safe and confidential place to complete a telehealth assessment. Without the clear boundaries of an office, some families attempted to do appointments in a space that was inadequate for the assessment or would not provide appropriate confidentiality (in a moving car, at a store, etc.).

Benefits to families of having a telehealth option—Several possible benefits were noted for families including overcoming transportation barriers (e.g., more access to families living in rural areas; easier access for those without a car), saving time and money (e.g., less time missed from work or spent traveling to appointments; no need to hire child care), and flexibility with where families could do the appointment (e.g., their office or car when on break from work). One clinician commented, “It's hard to transport these kids to appointments, take time off work, do different things. And so then being able to access us and these appointments easier has actually been a really good thing for families.” It was also noted that telehealth allowed families to complete what is often a very stressful experience from the comfort of their own homes. Using video conferencing allowed for additional family members and other care providers to attend evaluations and feedbacks when they might not otherwise be able to do so (e.g., family members who may not be able to travel or take off work; parents who are co-parenting; therapists, caseworkers, school personnel, etc.).

Factors related to general health disparities—Several clinicians indicated some hesitancy in commenting on how the pandemic and telehealth may have specifically affected health disparities due to lack of data, especially related to who was not seen. Many clinicians reported general concerns about health disparities that did not change whether patients were in clinic or being seen by telehealth. Telehealth was noted to be likely to decrease health disparities for some families due to overcoming transportation barriers and decreasing other barriers, such as time needed off of work or accessing additional childcare. The most often cited barrier to accessing telehealth related to disparities was lack of access to technology and/or internet. Other possible barriers noted included more chaos in the house, more children around to distract, and fewer caregivers to help with environmental management and camera work. Clinicians also reported that telehealth allowed greater insight into families’ home environment that could allow for more individualized treatment recommendations.

Access concerns for non-native-English speaking families—Almost all respondents voiced concerns for non-native-English speaking families having difficulties accessing services both by telehealth and in person during COVID. A few cited data they had that showed an actual decrease in non-native-English speaking families attending appointments or choosing to wait for in clinic appointments at a higher rate. Most centers appeared to continue to have access to interpreters by telehealth. Many indicated that doing telehealth assessment with an interpreter was difficult, although similar to difficulties seen in person (e.g., increased time for appointments; difficulties translating some concepts). Coaching a family through a play-based behavior observation by telehealth through an interpreter was noted to be a particularly challenging activity. Respondents from sites with telephone-only interpreters tended to indicate more dissatisfaction than those with access to interpreters with video. A few clinicians noted that it was easier to talk with the family and the interpreter at the same time by telehealth due to how the video platform was set up.

Institutional and Workplace Factors Related to Transitioning to a Telehealth Model

Most clinicians reported that their sites had done little to no work by telehealth before the pandemic. Institutional support was identified as a large factor in how successful a site was in transitioning to telehealth. The significant pivot that most sites made to transition their clinical work to telehealth also had significant effects on efficiency and workflow, rate of no-shows and cancelations, and workplace satisfaction (See Table 2 for additional quotes).

Institutional support—Clinicians varied widely on how much support they felt they received from their institutions to transition to telehealth services. A number of factors played a role in whether or not clinicians felt supported including leadership and support for innovating telehealth systems of care, level of pressure to maintain productivity, and institutional infrastructure needed for telehealth. Other important factors included clear communication from leadership, as well as support to help alleviate the pressures that the pandemic put on clinicians in their lives outside of work. Good administrative and technological support was also cited as key to helping families effectively access telehealth platforms and technology and feel comfortable with scheduling assessments by telehealth.

Efficiency—Some clinicians noted that there was a significant decrease in efficiency at the beginning of the pandemic due to having to start many new processes at once. As the pandemic went on, some clinicians noted that telehealth increased overall efficiency by decreasing time taken for certain aspects of the process, particularly the interview, checking in and rooming processes, and how clinicians could schedule their time. Additionally, it was noted that telehealth and working remotely could help with space issues, such as not having enough offices or exam rooms to see families in clinic. Telehealth also allowed for greater ease in typing during the clinical appointment, which decreased documentation time outside of the appointment. Others noted that the telehealth model led to a less efficient workflow, with more time needed for preparing for the evaluation, completing assessment tasks, and documentation after the evaluation. Some models also used personnel less efficiently, as more parts of the evaluation needed to be done back-to-back rather than simultaneously or it was deemed important to have multiple people watch the behavior observation when completed remotely. A few reported having additional administrative responsibilities that made their workflow less efficient, such as having to schedule their own appointments, as administrative staff were occupied assisting families in connecting to the telehealth platforms.

Effects on no shows—Most clinicians noticed a decrease in families not showing up to appointments, often attributed to two factors. First, telehealth was easier to access and clinicians could often call to remind families who forgot in the moment and then immediately start the visit. One clinician stated, “I feel like the rate of no-shows have gone down, because it's really easy to just be like, ‘Oh. I'll just click this button.’ Or, ‘I'll just answer my phone.’” Second, because this process was new, appointments were often being scheduled closer to their actual date and there tended to be more contact with families to help them understand the new processes involved in telehealth. Some clinicians noticed an initial decrease in no show rates, but then a leveling out as the pandemic went on, while others noticed no change in no-show rates.

Changes to workplace satisfaction related to remote and telehealth work—Some observed experiencing an increased level of work related stress, particularly in the beginning of the pandemic, given the uncertainty, lack of preparedness, and novelty of teleassessment. Increased stress was also related to feeling isolated and having more difficulty connecting with colleagues, having less opportunity for vacation/breaks, omnipresence of video access and increased fatigue related to increased screen time, and an increase in stress at home (e.g., managing own children doing school at home). One clinician commented, “I think it’s also socially isolating for the provider. In the middle of the telehealth time, it was kind of weird going all day without seeing humans in person.” Telehealth services also meant that nearly all clinicians were working from home. Many expressed increased satisfaction from working at home based on several factors including not having to commute, more flexibility during the work day to balance work and other domestic duties such as household chores and child care, and getting to work in a more comfortable environment.

Discussion

The COVID-19 pandemic temporarily halted clinicians’ ability to conduct traditional autism evaluations. This did not stop clinical teams from providing care in new ways through creativity and critical thinking. This was expressed best by one of the clinicians who was interviewed for this study:

I think I've learned that people are very resilient. And we have done remarkably well as a team and our families have done remarkably well, and I think the institution. You can get very bogged down in a lot of some of the discouraging things going on in the world right now, but you can also look around and be amazed at the resiliency, determination, and team spirit. And so I think part of me has been just blown away at how well we have done and how dedicated we all are. Not just in our clinic, but the hospital as a whole, and making sure we are still showing up and still providing services to our families.

The pivot to telehealth services during the pandemic created space for clinicians to reconsider how assessment services are provided. Most of those interviewed consistently indicated that telehealth is a viable tool for assessment that has potential beyond the emergency protocols of the pandemic. Assessment through telehealth likely could play a role as one part of a larger assessment model. There are clear opportunities to utilize telehealth to provide services to those who have difficulty accessing in person evaluations, “I think I would use it mostly as an option for families who have difficulty getting here, getting to an in person appointment for whatever reason that is. And I think I would build in a potential, kind of a fail-safe in person appointment if the virtual appointment [didn’t work].” At the very least, telehealth was seen as helpful for intake and feedback services to cut down family barriers to service, and increase clinic and clinician efficiency for many if not all patients.

Clinicians were more split on using telehealth for direct assessment of ASD symptoms. It was noted by many that, while not a good fit for everyone, assessment through telehealth was a good fit for many families and clinicians. Clinicians who conducted telehealth assessments overwhelmingly used some form of behavior observation system to evaluate symptoms of ASD. Observation systems were often structured similarly to established in-person assessment tools, such as the ADOS-2. Clinicians who conducted these observations indicated that they were valid tools for observing symptoms of ASD in some populations. Clinicians consistently reported that observation tools were most reliable when assessing the symptoms of young children. Clinicians seemed to have more questions about the reliability of observation tools as they were being used to evaluate the symptoms of older children and adolescents.

Although challenging, there are also possible benefits for direct observation and assessment of ASD symptoms conducted through telehealth. For example, the ability to assess a child’s behavior in their home environment, “So clinically, I feel like I am so much more informed when I'm watching children playing with their own toys in their own home as opposed to come into my office.” Telehealth assessment also provides opportunities to engage parents as partners in the evaluation process in ways that have not been able to be leveraged in traditional clinic settings. Recent results show that caregivers felt comfortable partnering with clinicians to complete a telehealth observation measure of ASD symptoms and that they appreciated playing this role in the evaluation process (Corona et al., 2021). Although still new in practice, further research could improve and refine these practices and demonstrate that they meet or exceed current standard of care for ASD assessment and could be important alternatives to in clinic assessment practices.

There continues to be significant barriers to the future practice of telehealth in ASD assessment. Many clinicians voiced that financial concerns, such as equal insurance reimbursement for telehealth versus in clinic services, would be the primary reason telehealth service options would not continue. Concerns about cost and reimbursement have previously been shown to be the primary barrier to large-scale telehealth for medical centers in both rural and urban areas (Lin et al., 2018). Many respondents in the current sample stated the need to continue to advocate for equal insurance reimbursement, “I think it’s still just the unknown of what we’ll be able to do based on insurance reimbursement. But I think if we are able to continue this model of a hybrid approach, then that would definitely be our preference.” Gaining a better understanding of whether and when telehealth diagnostic visits are efficient and effective may help insurers feel confident in continuing reimbursement. It is also clear that telehealth assessment is particularly challenging for some children and families, particularly children with externalizing behavior challenges, children with more subtle social deficits, families who have difficulty accessing and using needed technology, and children and families who do not speak the same language as the evaluation team. It has been shown that families with limited English proficiency are less likely to access general telehealth services, indicating that further research is necessary before telehealth can be an effective tool in reducing healthcare disparities in this population (Rodriguez et al., 2021). Finally, significant research still needs to be done to establish the diagnostic validity of telehealth assessments outside of the emergency orders of the pandemic. It will also be beneficial to consider whether hybrid models of assessment that incorporate traditional in person as well as telehealth components could be used to increase accuracy of ASD evaluations.

There are limitations to this study to be considered. All of the participating sites were academic medical centers, so these perspectives may not completely reflect the experiences of those who are practicing in other types of assessment settings. Participants also self-selected into the study, so it is unclear how representative this sample is of all the different viewpoints on telehealth services. In addition, although these interviews provided rich insight into clinician experiences, the qualitative nature of the study may limit the ability to generalize these results. Further quantitative research will be necessary to establish the effectiveness of diagnostic decision making over telehealth, as well as to establish parameters of telehealth practice. A significant strength of the study was the inclusion of a wide variety of clinicians from a large sample of sites who are on the forefront of ASD assessment.

The results of this study suggest that, although the COVID-19 pandemic stressed ASD assessment teams, it also provided the space for many teams to rethink how they do ASD assessment and for institutions to accelerate building the infrastructure needed to support large scale implementation of telehealth services. The work that was done to build and execute these assessment processes provided a strong foundation for further investigation and expansion of telehealth services within ASD assessment.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 24 kb)