Identifying Patient Characteristics to Understand Which Children May Receive Diagnostic Clarity in a Virtual Autism Spectrum Disorder Evaluation.

Given long waitlists for autism spectrum disorder (ASD) evaluation coupled with the COVID-19 pandemic, it is crucial to triage patients to services they are likely to receive diagnostic clarity (i.e., virtual, in-person evaluation). Participants attended a virtual ASD assessment. A subset also attended in-person evaluation. Results suggest younger children with educational services for ASD may benefit from virtual assessment while older patients with a history of psychiatric conditions may benefit from in-person evaluation. An ASD symptom severity tool related to virtual and in-person diagnostic clarity. Family history of ASD related to in-person diagnosis while other variables (e.g., age, educational services) did not. The study suggests patient characteristics may be used to determine for whom virtual ASD assessment may be appropriate.

Autism Spectrum Disorder (ASD) is characterized by deficits in social communication and restricted behaviors, interests, or activities (American Psychiatric Association, 2013). The number of individuals with ASD diagnoses in the United States is growing; 1 in 54 children had a diagnosis of ASD in 2016, 10% higher than the number of children with diagnoses in 2014 (Bent et al., 2017; Maenner et al., 2020). It is unclear whether these higher rates reflect changes in diagnostic criteria, improved diagnosis of individuals with less severe or apparent symptoms, increased awareness, increased access to diagnostic services, changes in differential diagnosing, or a true increase in the frequency of ASD (American Psychiatric Association, 2013; Bent et al., 2017; Cardinal et al., 2021). This increased rate corresponds with an increase in demand for ASD diagnostic services (Bent et al., 2017).

According to identification and evaluation guidelines provided by Hyman et al. (2020) a diagnostic autism evaluation should include a thorough history, behavioral observations, and formal assessment of language, cognitive, and adaptive abilities. An individual’s developmental and social history is typically obtained via interview with the patient and caregivers. In addition, reports from collateral informants in different settings, such as teachers, provide further context. Questionnaires can supplement developmental and symptom histories. Behavioral observations, made by clinicians trained to identify behavioral symptoms of autism, are also critical for the diagnostic process. There are several structured observation tools (e.g., Autism Diagnostic Observation Schedule, Second Edition (ADOS-2) and Childhood Autism Rating Scale, Second Edition (CARS-2)) used by clinicians to generate opportunities to observe behaviors associated with autism. Language disorders, intellectual disabilities, ADHD, anxiety, mood disorders and learning disabilities are common differential diagnosis for ASD in addition to also being frequent comorbid diagnoses with ASD (American Psychiatric Association, 2013; Cardinal et al., 2021; Hyman et al., 2020) Therefore, formal assessments provide a clearer picture of whether autism is a diagnosis that best fits a person’s level of functioning and symptom presentation.

The coronavirus (COVID-19) global pandemic has resulted in unprecedented changes in and challenges to ASD diagnostic evaluations, most significantly the shift to virtual evaluations (Berger et al., 2021). Virtual evaluations may utilize asynchronous or synchronous technology. Asynchronous visits typically involve caregivers filming videos of behaviors of concern and completing questionnaires to be forwarded to providers. Videoconferencing is the primary modality used for synchronous visits across healthcare settings (including for virtual ASD assessment) and involves real-time, interactions between a patient, their family, and a health care provider (Alfuraydan et al., 2020).

While virtual autism assessments existed prior to COVID-19, the pandemic and social distancing has accelerated the use, development, and refinement of these assessments. Berger et al. (2021) identified the following as the most widely used virtual autism diagnostic assessments: Autism Spectrum Disorder-Diagnostic Interview and Activities Lifespan (ASD-DIAL; Miller, 2020), Adapted Virtual Autism Behavior Observations (A-VABO; Kryszak & Albright, 2020), Observation of Play Screener: Home Edition (OOPS:HE; Nickel, 2020), Brief Observation of Symptoms of Autism (BOSA; Lord et al., 2020), and TELE-ASD-PEDS (Wagner et al., 2020). These measures are all characterized by synchronous virtual interactions between a clinician and caregiver and involve the clinician guiding the caregiver through a series of activities in the home setting, while the clinician observes. In the rapidly evolving field of autism telehealth and virtual assessments, it is critical to note that at the time of this writing, emerging research supports the use of the TELE-ASD-PEDS (Wagner et al., 2020, 2021), but no psychometric data is available for the other assessments discussed. As more data becomes available it is expected that psychometric properties will be forthcoming and these measures may be modified.

Growing evidence suggests virtual evaluations may address numerous barriers to obtaining diagnosis, particularly those that affect underserved, rural, and/or families of color. Specifically, virtual evaluations may decrease provider and patient costs (e.g., travel time, transportation expenses, missed work) and increase the coverage area of services (Alfuraydan et al., 2020; Berger et al., 2021; Corona et al., 2021). Research also suggests clinicians and families, at large, are comfortable and accepting of virtual evaluations (Corona et al., 2021; Wagner et al., 2020, 2021). For example, when compared to in-person evaluations, caregivers reported virtual assessments were shorter, instructions were easy to follow, and both the caregivers and children were more comfortable (Corona et al., 2021). Virtual assessments also have the potential for reducing wait times for evaluations, increasing availability, and reaching more individuals at an earlier age (Berger et al., 2021).

While there may be many benefits of virtual autism evaluations, there are also cases where an in-person visit is indicated. Alfuraydan et al. (2020) noted that while many families and providers support the use of virtual assessments, virtual visits may not be appropriate for more complicated cases. In addition, Corona et al. (2021) found an 86% overall diagnostic agreement for virtual vs. in-person autism evaluations. Of the 14% of children who did not have diagnostic agreement (misdiagnosed with or without ASD during the virtual visit), all had a history of developmental delays. Therefore, there may be a higher chance of virtual visit misdiagnosis amongst children with neurodevelopmental differences or delays. Additional research suggests that children with early communication concerns were identified with ASD sooner than children without identified communication concerns (Zablotsky et al., 2017). Pediatricians have also been shown to use a “watch and wait” approach with children who display subtler ASD symptoms (Sices et al., 2004). It is possible that children referred for ASD later in life present with subtler symptoms than children referred for evaluation in toddlerhood or early childhood.

Further, many children begin receiving school-based educational services prior to receiving an ASD evaluation or diagnosis. A recent study investigated similarities and differences between children with a medical diagnosis of ASD and children receiving special education services for ASD without a medical diagnosis of ASD (Safer-Lichtenstein & McIntyre, 2020). Although the study was conducted within one state, results suggested that children receiving educational services for ASD without a medical diagnosis were identified as having less severe clinician-rate ASD symptomatology than children who received a medical diagnosis of ASD (Safer-Lichtenstein & McIntyre, 2020). It was also noted that groups did not differ based on adaptive or behavioral functioning. While results suggest that receipt of ASD educational services may indicate a lower diagnostic threshold than an ASD medical diagnosis (Safer-Lichtenstein & McIntyre, 2020)., children with ASD educational services are likely to have been flagged for having functionally impairing symptoms of ASD more broadly that impact their learning. This may subsequently provide diagnostic clinicians with information regarding consistency across settings and possibly increase confidence in observations during virtual assessment. Results may also suggest that children who receive school-based services for ASD as well as a medical diagnosis of ASD may present with clearer ASD symptomatology.

This study highlights a vital question associated with the use of virtual ASD evaluations: for whom are virtual visits sufficient and who should be referred for further in person testing? The current study aims to address this question by identifying characteristics of children who received a diagnosis of ASD during an ASD virtual evaluation, and children who were referred on for further in-person testing. It is expected that younger age and those receiving school-based supports for ASD will be more likely to receive diagnostic clarity during a virtual ASD assessment consistent with previous research indicating adolescents may present with additional comorbidities that require comprehensive differential diagnosis (e.g., Barlow, 2021; Simonoff et al., 2008). It is also hypothesized that patients who receive higher scores on an ASD assessment tool will be more likely to receive diagnostic clarity during a virtual ASD assessment. Further, it is expected that patients with a higher number of historical psychiatric diagnoses will be more likely to be referred for in-person ASD testing as differential diagnosis may require more comprehensive evaluation.

Finally, a subset of the group receiving an initial virtual visit completed the recommended in person follow-up and the current study aims to explore relations between data collected during the virtual visit with in-person ASD diagnostic outcomes. It is likely that receiving school services for ASD and the scores from the ASD assessment tool administered during the virtual visit will positively relate to an in-person diagnostic outcome (i.e., ASD confirmed vs. ASD ruled-out), whereas age, historical psychiatric diagnoses, and parent-reported measures of adaptive, emotional, and behavioral will not relate to diagnostic outcomes.

Methods

Participants and Procedures

All procedures were approved by the Institutional Review Board (IRB). Participants included 87 youth patients (33.3% female) aged 23 months to 17 years (M = 9.14; SD = 3.89) who received a virtual autism spectrum disorder evaluation by a licensed clinical psychologist working with a pre-doctoral psychology intern in an outpatient autism diagnostic clinic at an academic medical center in the United States. Visits were completed collaboratively and live supervision was provided throughout the entirety of the virtual assessment. Diagnostic consensus between the psychologist and intern was conducted near the end of the virtual visit prior to the diagnostic feedback. In-person evaluation was completed by a licensed clinical psychologist/neuropsychologist and/or a psychologist resident in combination with an occupational therapist. Diagnostic consensus between the licensed psychologist, psychologist resident, and occupational therapist was conducted at the end of each visit prior to feedback. Patient race/ethnicities are detailed as follows: 74.7% Caucasian, 4.6% African American, 16.1% Hispanic, 1.1% Asian/Pacific Islander, and 3.4% identified as “Other”. Eight (9.20%) patients reported having a first degree relative with ASD. Patients and their caregivers presented to the clinic with concerns for ASD as well as other developmental, cognitive, or psychiatric concerns. Evaluations consisted of caregiver interview to capture early developmental differences and current functioning, behavioral observation, and caregiver-report measures to assess adaptive functioning, emotional, and behavioral functioning. The current sample only includes patients who attended a virtual visit as their first ASD evaluation through this clinic. The presence of psychiatric disorders was determined, and historical diagnoses were confirmed based on DSM-5 criteria. Diagnostic feedback was provided to the family in a single, same-day feedback session and comprehensive diagnostic reports were written by the psychology team. Patient data from these virtual evaluations were retroactively entered into a database.

Eighty-seven participants completed a virtual assessment and, thus, had a virtual visit outcome (i.e., diagnostic clarity or referral for in-person evaluation). The virtual assessment was each patient’s first ASD assessment, and participants had not previously participated in in-person or virtual diagnostic ASD evaluation. Virtual assessments were conducted secure, synchronous audio and video technology. Assessment measures provided during the virtual assessment were determined by clinic standard operating procedures and age. That is, all virtual assessments included the following: a social-emotional functioning measure (i.e., Behavior Assessment System for Children, Third Edition), adaptive functioning measure (i.e., Adaptive Behavior Assessment System, Third Edition), semi-structured autism diagnostic interview (i.e., Childhood Autism Rating Scales, Second Edition), and behavioral observations. The version of questionnaire measures received was based on patient age. Questionnaires were administered via online platform rather than paper-and-pencil completion. Behavioral observations were generally unstructured for virtual evaluation. That is, parents were encouraged to interact with and/or play with their child like they normally would, in addition to some direct prompts that a clinician may typically do during in-person evaluation (e.g., parents encouraged at times to put forth their own play agenda to see how their child might respond). Only 4 participants were under the age of 3 years and, thus, able to be administered a standardized telemedicine behavioral observation measure (i.e., the TELE-ASD-PEDS). Due to limited data, the TELE-ASD-PEDS was not investigated in the current dataset. In-person evaluation assessment measures were also chosen based on standard operating procedures and additional presenting concerns (e.g., BOSA, self- and caregiver-report measures regarding emotional and behavioral functioning, differential diagnostic clinical interviewing, and cognitive assessment when indicated). Thirty-eight patients received diagnostic clarity (31 received ASD diagnosis; 7 had ASD definitively ruled out) within the virtual visit, whereas 49 participants were referred for in-person assessment. Twenty-two participants of the 49 that were referred for further testing have completed in-person testing thus far, whereas 27 participants have not yet attended an in-person appointment and were waiting to be scheduled at the time of data analysis. Of the 22 participants who have completed a follow-up in-person appointment, 15 participants received a diagnosis of ASD while 7 participants did not receive an ASD diagnosis. Please see Fig. 1 for procedural clarity.

Fig. 1

Flow chart outlining participant diagnostic outcomes

Measures

Demographic Information

Patient medical records were screened for demographic information including age, biological sex, first degree family members diagnosed with ASD, academic accommodations (e.g., individualized education plan [IEP]; 504 Accommodation Plan), number and type of historical psychiatric and developmental diagnoses, and previous cognitive testing prior to the virtual visit. Due to the small sample size, psychiatric and developmental disorders were grouped into larger categories (i.e., anxiety and related disorders [i.e., generalized anxiety disorder, social anxiety disorder, unspecified anxiety disorder, specific phobia, obsessive–compulsive disorder, adjustment disorder, and posttraumatic stress disorder; n = 12], attention deficit/hyperactivity disorder and behavioral disorders [unspecified disruptive behavior, impulse-control, and conduct disorder and oppositional defiant disorder; n = 21], mood and related disorders [i.e., major depressive disorder, disruptive mood dysregulation disorder; n = 5], intellectual disability and related disorders [i.e., borderline intellectual functioning, global developmental delay, other neurodevelopmental disorder; n = 5], speech/language disorders [i.e., language disorder, articulation disorder, phonological disorder, social communication disorder; n = 16], and other [i.e., sensory processing disorder, somatic symptom disorder, developmental coordination disorder, Tourette’s disorder; n = 8]. Sixteen participants had previous cognitive testing available at the time of virtual visit. Four of 31 (12.90%) participants who received an ASD diagnosis after at the virtual visit provided previous historical record of cognitive testing, while whereas 16 participants (32.65%) of the patients referred for in person assessment sample provided previous historical record of cognitive scores. Information was confirmed during caregiver interview. The requirement of an interpreter during the virtual visit was also collected at the time of virtual assessment.

Adaptive Functioning

Prior to the virtual evaluation, caregivers of patients completed the Adaptive Behavior Assessment System, Third Edition (ABAS-3; Harrison & Oakland, 2015) to obtain the Global Adaptive Composite. The Global Adaptive Composite was reported as a standard score (M = 100, SD = 15) and used to represent current adaptive functioning.

Behavioral and Emotional Functioning

Prior to the virtual evaluation, caregivers were also administered the Behavior Assessment System for Children, Third Edition (BASC-3; Reynolds & Kamphaus, 2015) to characterize the patient’s level of internalizing and externalizing difficulties. Internalizing and externalizing composite scores were reported as t-scores (M = 50, SD = 10).

Autism Specific Symptoms

The psychologist also administered the Childhood Autism Rating Scales, Second Edition (CARS-2; Schopler et al., 2010) to quantify the severity of patient ASD symptoms. This measure was scored based on parent report, as well as brief virtual behavioral observations when possible. Severity scores of ASD symptoms were included in analyses. Sixty-four patients were administered the CARS-2 standard functioning version while 23 patients were administered the CARS-2 high functioning version. Consistent with previous literature (e.g., Sanchez & Constantino, 2020), the CARS-2 standard functioning and CARS-2 high functioning versions were combined to create a composite of severity scores of ASD symptoms.

Virtual Visit Diagnostic Outcome

Patients were provided with diagnostic impressions at the end of the virtual visit. Diagnostic impressions included confirmed ASD diagnosis, ASD definitively ruled-out, or referred for further in-person evaluation. Binary codes for diagnostic clarity (i.e., ASD confirmed vs. referred for further evaluation) were created.

Brief Observation of Symptoms of Autism (BOSA)

For children who completed a follow-up in person visit, patients and their caregiver participants in a brief (i.e., 15 min) observational assessment to identify symptoms of ASD (Lord et al., 2020). Specifically, a psychologist and/or psychologist resident and an occupational therapist observed and co-scored a caregiver-patient interaction through a 2-way mirror due to the COVID-19 pandemic. Caregivers and patients removed their masks and subsequently engaged in a series of tasks meant to engage the caregiver and patient in social interaction, reciprocal conversation, and pretend play. Observations were scored using the corresponding ADOS-2 module protocol. Due to the large age range and verbal ability of the participants, multiple modules of the BOSA were administered. Given limited data available, the module administered most frequently was included in analyses (BOSA-F1; n = 10).

Cognitive Functioning

Estimates of patient intellectual functioning was obtained during the in-person assessment visit. Cognitive assessments administered included the Wechsler Intelligence Scale for Children, Fifth Edition (WISC-V; Wechsler, 2014), Wechsler Abbreviated Scale Intelligence, Second Edition (WASI-II; Wechsler, 2011), Stanford Binet, Fifth Edition (SB-5; Roid & Pomplun, 2012), Wechsler Adult Intelligence Scale, Fourth Edition (WAIS-IV; Wechsler, 2008), and Differential Ability Scales, Second Edition (DAS-2; Elliott et al., 2007). Given all scores were computed as standard scores, a composite of full-scale intellectual quotient (FSIQ) was created.

In-Person Diagnostic Outcome

Patients were provided with diagnostic impressions at the end of the in-person visit. Impressions included confirmed ASD, no ASD diagnosis given, and/or other psychiatric conditions other than ASD. Binary codes for ASD confirmed and no ASD diagnosis provided at in-person evaluation were created.

Data Analysis

Independent samples t-tests and chi-square analyses were conducted to examiner group differences in demographic, functional, behavioral, and psychological characteristics. Logistical regression analysis was conducted to determine which demographic, functional, behavioral, and psychological characteristics predicted diagnostic clarity at virtual evaluation or referral for in-person testing.

Missing Data

Given the nature of collecting clinical, longitudinal data, missing data occurred within the current dataset. While most variables had no missing data, completion of online-administered caregiver-reported measures was poor, leading to 65.52% of caregivers completing a measure assessing adaptive skills and 45.98% of caregivers completing measures of behavioral and emotional functioning. Due to large amounts of missing data, pairwise deletion was used when investigating caregiver-report measures.

Results

Virtual ASD Evaluation T-tests and Chi-Square Tests

ASD Diagnosis Confirmed at Virtual Visit vs. ASD Ruled Out at Virtual Visit

In children who received diagnostic clarification at the virtual visits and were not sent on for in-person follow-up, there were no group differences (i.e., ASD diagnosis confirmed at virtual visit vs. ASD definitively ruled out at virtual visit) in biological sex, race/ethnicity, educational services for ASD, or presence of interpreter during virtual visits. There were also no group differences in caregiver-reported adaptive functioning, internalizing symptoms, or externalizing symptoms. There were also no group differences between participants who had ASD ruled out and those that were referred for in-person testing in the aforementioned variables. Having a first degree relative with ASD trended toward significantly predicting group membership (χ2 = 2.94, p = .086). It is notable that no participants who had ASD ruled out at the virtual visit had a first degree relative with ASD. The CARS-2 severity score significantly differentiated group membership, such that those who had ASD ruled out had a lower severity score than those with ASD confirmed at virtual visits (t =  − 16.85, p < .001; see Table 1). It is crucial that these findings are interpreted with the utmost caution as only 7 participants had ASD ruled out at the time of virtual visit. Given the small sample size, participants who had ASD ruled out at virtual visit were not considered further.

Table 1

Descriptive demographic information: ASD diagnosis confirmed at virtual visit vs. ASD ruled out at virtual visit

	ASD ruled out at virtual visit (n = 7)	ASD diagnosed virtual visit (n = 31)	Group differences	Cohen’s D	Cramer’s V
Age	7.73 (3.04)	8.32 (4.18)	t =  − 0.32, p = .753	0.16	–
Sex (% female)	14.29%	38.71%	χ2 = 1.39, p = .238	–	0.19
Race/ethnicity (%)	–	–	χ2 = 1.20, p = .878	–	0.18
Caucasian	71.43%	74.19%	–	–	–
Black	0.00%	3.23%	–	–	–
Hispanic/Latinx	28.57%	16.13%	–	–	–
Asian/Pacific Islander	0.00%	3.23%	–	–	–
Other	0.00%	3.23%	–	–	–
1st degree relative with ASD (% yes)	0.00%	12.90%	χ2 = 2.94, p = .086†	–	− 0.28
Interpreter present (% yes)	28.57%	19.35%	χ2 = 0.34, p = .560	–	0.09
Educational services for ASD (% yes)	14.29%	48.39%	χ2 = 2.52, p = .112	–	0.25
Previous cognitive functioning (% yes)	14.29%	12.90%	χ2 = 0.16, p = .898	–	0.02
Number of historical psychiatric and developmental diagnoses	0.71 (1.11)	0.65 (1.11)	t = 0.13, p = .900	0.05	–
General adaptive composite∇	72.50 (17.17)	66.62 (13.12)	t = 0.89, p = .385	0.38	–
Internalizing symptoms∇	55.00 (7.94)	53.77 (10.69)	t = 0.19, p = .855	0.13	–
Externalizing symptoms∇	68.00 (9.17)	61.85 (12.57)	t = 0.79, p = .442	0.56	–
CARS-2 severity score	1.00 (0.00)	2.90 (0.30)	t =  − 16.85, p < .001**	–	–

†p < .10; *p < .05; **p < .01

∇Caregiver-reported symptoms. All figures reported as mean (standard deviation) unless otherwise noted

ASD Confirmed at Virtual Visit vs. Referred for Further Testing

Group differences were investigated between individuals who had ASD confirmed at virtual visit and those who were referred for further in-person testing. There were no group differences (i.e., ASD confirmed at virtual visit vs. Referred for further testing) in biological sex, race/ethnicity, or caregiver-reported adaptive functioning, internalizing symptoms, or externalizing symptoms. There were no group differences for the presence of an interpreter or having a first degree relative with ASD. These results suggest that adaptive functioning, emotional and behavioral functioning, use of an interpreter, and having a first degree relative with ASD did not differentiate whether a patient may receive diagnostic clarity during a virtual ASD evaluation or be referred on for further in-person testing. However, there were significant group differences in age (t =  − 1.71, p = .046), receiving educational services for ASD (χ2 = 12.81, p < .001), number of historical psychiatric diagnoses (t =  − 2.08, p = .004) and CARS-2 severity score (t = 6.27, p < .001; see Tables 2, 3). Specifically, participants who received diagnostic clarity were younger. Participants who received diagnostic clarity through the virtual visit were more likely to be receiving school services for ASD and reported fewer historical psychiatric diagnoses. Finally, participants who received diagnostic clarity received higher scores on the CARS-2. Given that the number of historical psychiatric diagnoses significantly differentiated groups that received an ASD diagnosis and those who were referred for further evaluation, exploratory analyses were conducted to determine if the type of historical psychiatric and/or developmental condition may be pertinent for understanding group differences. Anxiety and related disorders significantly differentiated groups, such that those with an anxiety disorder were more likely to be referred for in-person evaluation (χ2 = 4.73, p = .030). Mood disorders were found to marginally differentiate group members, such that those with a previously diagnosed mood or related disorder were marginally more likely to be referred for further in-person testing (χ2 = 3.37, p = .066); however, only 5 people had existing mood and related disorders. No other group differences emerged related to historical psychiatric and developmental disorders (all p’s > 0.35). Group differences were used to inform logistic regression analysis for understanding odds ratios related to receiving diagnostic clarity during virtual evaluation.

Table 2

Descriptive demographic information: ASD confirmed at virtual visit vs. referred for further testing

	ASD diagnosed at virtual visit (n = 31)	Referred for in-person evaluation (n = 49)	Group differences	Cohen’s D	Cramer’s V
Age	8.32 (4.18)	9.85 (3.71)	t =  − 1.71, p = .046*	0.39	–
Sex (% female)	38.71%	32.65%	χ2 = 0.31, p = .580	–	− 0.06
Race/ethnicity (%)	–	–	χ2 = 1.98, p = .739	–	0.16
Caucasian	74.19%	75.51%	–	–	–
Black	3.23%	6.12%	–	–	–
Hispanic/Latinx	16.13%	14.29%	–	–	–
Asian/Pacific Islander	3.23%	0.00%	–	–	–
Other	3.23%	4.08%	–	–	–
1st degree relative with ASD (% yes)	12.90%	8.16%	χ2 = 0.474, p = .491	–	− 0.08
Interpreter present (% yes)	19.35%	14.29%	χ2 = 0.36, p = .549	–	− 0.07
Educational services for ASD (% yes)	48.39%	12.24%	χ2 = 12.81, p < .001**	–	0.40
Previous cognitive functioning (% yes)	12.90%	20.41%	χ2 = 0.74, p = .389	–	0.10
Number of historical psychiatric and developmental diagnoses	0.65 (1.11)	1.20 (1.21)	t =  − 2.08, p = .04*	0.50	–
General adaptive composite∇	66.62 (13.12)	71.71 (13.12)	t =  − 1.30, p = .199	0.39	–
Internalizing symptoms∇	53.77 (10.69)	67.08 (16.37)	t =  − 1.72, p = .094	0.63	–
Externalizing symptoms∇	61.85 (12.57)	67.08 (16.37)	t =  − 1.01, p = .321	0.36	–
CARS-2 severity score	2.90 (0.30)	2.24 (0.63)	t = 6.27, p < .001**	1.34	–

*p < .05; **p < .01

∇Caregiver-reported symptoms. All figures reported as mean (standard deviation) unless otherwise noted

Table 3

Hierarchical logistic regression analysis predicting virtual visit outcome

	β	SE β	Wald’s χ2	eβ (odds ratio)	p	95% CI
Step 1
Age (years)	0.12	0.07	2.72	1.13	.099†	0.978–1.296
Educational services for ASD	− 1.98	0.60	10.84	0.14	.001**	0.042–0.0449
Number of historical psychiatric diagnoses	0.30	0.25	1.42	1.35	.233	0.824–2.208
Step 2
Age (years)	0.09	0.09	1.17	1.10	.279	0.928–1.295
Educational services for ASD	− 2.14	0.75	8.20	0.12	.004**	0.027–0.510
Number of historical psychiatric diagnoses	0.18	0.30	0.36	1.20	.548	0.663–2.173
CARS-2 severity score	− 2.83	0.79	12.98	0.06	.001**	0.013–0.275

The dependent variable, virtual visit outcome, was coded such that diagnostic clarity = 0 and referral for in-person evaluation = 1

ASD autism spectrum disorder, CARS-2 childhood autism rating scale, second edition, CI confidence interval

†p < .10; **p < .01

Hierarchical Logistic Regression of Demographic, Educational, and Severity of ASD Symptoms Predicting Virtual ASD Evaluation Outcome

A Hosmer and Lemeshow goodness of fit test revealed the model was an appropriate fit for the current dataset (step 1: χ2 = 8.35, p = .400; step 2: χ2 = 8.10, p = .424). The first step of the logistical regression included patient age, education services for ASD, and number of historical psychiatric diagnoses. Results indicate that patient age trended toward significance and receiving educational services for ASD was a significant predictor of diagnostic clarity vs. referral for in-person testing (see Tables 2, 3). Specifically, age was marginally related to in-person evaluation referral, such that older children were 1.13 times more likely to be referred for further assessment (β = 0.12, SE = 0.07, Wald’s χ2 = 2.72, odds ratio = 1.13, p = .099). Additionally, children receiving educational services for ASD were 14% more likely to received diagnostic clarity at the virtual visit and not be referred for in-person evaluation than those who were not receiving educational services for ASD (β =  − 1.98, SE = 0.06, Wald’s χ2 = 10.84, odds ratio = 0.14, p = 0.001). However, the number of historical psychiatric diagnoses a patient carried (β = 0.30, SE = 0.25, Wald’s χ2 = 1.42, odds ratio = 1.35, p = 0.233) did not uniquely significantly predict diagnostic outcome at the virtual visit.

The second step of the logistical regression included the previously mentioned variables as well as the CARS-2 severity score. Results suggest that age no longer significantly predicted diagnostic outcome (β = 0.09, SE = 0.09, Wald’s χ2 = 1.17, odds ratio = 1.10, p = .279; see Tables 2, 3). However, having educational services for ASD remained a significant predictor of diagnostic outcome (β =  − 2.14, SE = 0.75, Wald’s χ2 = 8.20, odds ratio = 0.12, p = .004). Specifically, children receiving educational services for ASD were 12% less likely to be referred for in-person testing (i.e., more likely to receive an ASD diagnosis at virtual visit). The number of historical psychiatric diagnoses a patient carried continued to not uniquely predict diagnostic outcome (β = 0.18, SE = 0.30, Wald’s χ2 = 0.36, odds ratio = 1.20, p = .548). The clinician-mediated ASD symptom assessment (i.e., CARS-2) significantly predicted diagnostic outcome (β =  − 2.83, SE = 0.79, Wald’s χ2 = 12.98, odds ratio = 0.06, p = .001), such that individuals with higher severity scores were more likely to receive an ASD diagnosis at the time of virtual visit while participants with lower severity scores were more likely to be referred on for in-person ASD evaluation.

Bivariate Correlations of Demographic, Educational, and Severity of ASD Symptoms Predicting In-Person ASD Evaluation Outcome

Correlations of demographic, educational eligibility, severity of ASD symptoms, and in-person evaluation outcomes were investigated. The current dataset did not yield significant bivariate relations amongst data collected prior to or during the virtual visit (i.e., age, sex, race/ethnicity, first degree relative with ASD, interpreter present, educational services for ASD, number of existing psychiatric diagnoses, caregiver-reported adaptive, emotional, and behavior functioning) and the BOSA. Notably, the clinician-scored ASD symptoms measure (i.e., CARS-2) did not relate to scores on in-person behavioral observations (i.e., BOSA). Patient demographics information (i.e., age, sex, race, educational services, psychiatric history), caregiver-reported adaptive functioning, and emotional and behavioral functioning did not relate to in-person diagnostic outcome (see Table 4). Having previous cognitive testing available at the virtual visit also did not relate to in-person diagnostic outcome. However, the CARS-2 did correlate with final in-person diagnostic outcomes (r = 0.34, p = .016). Additionally, having a first degree relative with ASD was significantly related to receiving an ASD diagnosis during the in-person evaluation (r =  − 0.46, p = .030). Patient cognitive functioning measured at the in-person evaluation did not relate to ASD diagnostic outcome. The scores on the BOSA did significantly relate to diagnostic outcomes following the in person visit (r = 0.85, p = .004).

Table 4

Bivariate correlations of demographic, educational eligibility, severity of ASD symptoms, and in-person evaluation outcomes

	n	1	2	3	4	5	6	7	8	9	10	11	12	13	14
1. Age	87	–
2. Sex	87	0.02	–
3. Race/ethnicity	87	0.02	− 0.04	–
4. 1st Degree relative with ASD	87	− 0.12	− 0.06	− 0.06	–
5. Interpreter present	87	0.15	− 0.19†	0.35**	0.01	–
6. Educational services for ASD	87	0.05	− 0.24*	0.08	− 0.07	0.23*	–
7. Reported cognitive functioning	87	0.30**	− 0.20	0.13	0.37**	0.13	0.03	–
8. # of existing psychiatric diagnoses	87	0.24*	− 0.01	− 0.08	0.01	− 0.08	− 0.16	0.12	–
9. Adaptive functioning∇	57	0.16	− 0.03	0.13	− 0.07	0.02	0.04	0.42**	− 0.26	–
10. Internalizing problems∇	41	0.02	0.11	0.23	− 0.69	0.03	0.06	− 0.36*	0.31	0.06	–
11. externalizing problems∇	41	− 0.29†	0.17	0.05	− 0.24	− 0.11	− 0.11	− 0.34*	0.10	− 0.16	0.49**	–
12. CARS-2 severity score	87	− 0.09	0.00	0.03	− 0.47**	− 0.03	0.21†	− 0.33**	− 0.18	− 0.14	0.45**	0.01	–
13. In-person visit cognitive functioning	22	0.06	0.03	0.20	0.15	0.15	− 0.14	0.16	0.13	0.59*	0.17	− 0.31	0.26	–
14. BOSA overall total	10	− 0.32	− 0.29	− 0.03	− 0.54	0.30	0.01	− 0.49	− 0.05	− 0.04	0.48	0.37	0.11	0.39	–
15. In-person diagnostic outcome	22	− 0.10	− 0.02	0.14	− 0.46*	0.32	0.20	− 0.28	− 0.10	− 0.16	0.28	0.22	0.34*	− 0.01	0.85**

ASD autism spectrum disorder, CARS-2 childhood autism rating scale, second edition, BOSA brief observation of symptoms of autism

†p < .10, *p < .05, **p < .001

∇Caregiver-reported symptoms

Discussion

First, it is important to note that the current sample size was relatively small, and results must be interpreted with caution. The lack of group differences in demographics and adaptive and emotional and behavioral functioning between individuals who had ASD ruled out and those who had ASD confirmed or were referred for in-person evaluation may be due to lack of power. Unsurprisingly, the CARS-2, a clinician-mediated ASD specific measure, indicated that individuals with less severe symptoms of ASD were more likely to have ASD ruled out at the time of virtual visit. This suggests that individuals with fewer ASD symptoms may be suitable for virtual evaluation.

Additional results from the current study suggest that patient characteristics known prior to the evaluation may be used to determine the likely diagnostic utility of virtual ASD assessment. ASD assessment completed by the psychologist during the virtual visit is also critical for understanding ultimate diagnosis or need for further testing. While information gathered during the virtual visit did not relate to in-person diagnostic outcomes, this aim should continue to be investigated.

Generally, group differences were found based on age, educational services for ASD, and number of historical psychiatric diagnoses. When placed in a model in which unique variance above and beyond other variables was assessed, results revealed that patients receiving educational services for ASD were more likely to receive diagnostic clarity during the virtual ASD evaluation. Many participants with existing educational services for ASD provided educational evaluation records to the clinic prior to their evaluation, and it is possible that access to records of previous educational testing is related to this finding. That is, patients may have had record of behavioral observations in the school setting, an Autism Diagnostic Observation Schedule (ADOS) conducted by a school psychologist, or autism-specific clinical caregiver interviewing; however, details about this level of information was not included in the current study. Only 20 participants provided the clinic with previous cognitive testing scores which did not differentiate diagnostic outcomes. It is probable that qualitative data (e.g., school psychologist observations) provided additional clarity regarding ASD symptoms across contexts which contributed to diagnostic clarity at the time of virtual evaluation. These results may also simply indicate that ASD symptoms may be more apparent for children who receive services at the school level for ASD symptoms, they may have clearer symptoms and/or greater functional impairment from their ASD related symptoms and, thus, be more readily identified via virtual visit. However, this conclusion warrants further investigations given previous research suggesting symptom severity differences between children who qualify for educational services for ASD and do not receive a medical diagnosis compared to those who do receive a medical diagnosis (Safer-Lichtenstein & McIntyre, 2020).

Furthermore, in the context of the COVID-19 pandemic, it is possible that clinicians may rely more heavily on school-based assessment and eligibility classification. A relatively small snapshot of child behavior and social engagement is captured during virtual evaluation compared to in-person evaluation, which may push clinicians to consider data more heavily from a variety of other sources. For example, one key component of ASD evaluation is understanding consistency in presentation across a variety of contexts. When a child receives education services for ASD, this indicates the child demonstrates behavioral and social-emotional difficulties in that setting as well. Clinicians are highly encouraged to consciously consider how they may rely on school eligibility for ASD diagnostic information. Educational eligibility and medical diagnostic criteria for ASD do not align perfectly. Further, educational requirements to receive services for ASD differs by state, with 34 states not including DSM-5 criteria in their eligibility requirements. Notably, the state in which the current data were collected includes DSM-5 criteria in their educational eligibility guidelines. Recent research indicates that across the United States, children receiving educational services for ASD had less severe symptoms than those with a medical ASD diagnosis (Safer-Lichtenstein & McIntyre, 2020). Ultimately, it is critical for clinicians conducting ASD evaluations to understand their state’s special education edibility requirements when determining how heavily to weigh information received from the school.

Group differences in age suggested that older children and adolescents were significantly more likely to be referred for in-person testing. Intuitively and conceptually, this finding reinforces the notion that those presenting for an ASD evaluation in middle childhood and adolescence were not previously identified, and, thus, may present with subtler ASD symptoms. Additionally, older patients have the potential to have a higher number of comorbidities (e.g., mood or anxiety disorders) that may require more comprehensive differential diagnostic testing (e.g., Simonoff et al., 2008). This conclusion is supported by the independent samples t-test finding that diagnostic outcome (i.e., diagnosis vs. referral) differed based on number of historical psychiatric and developmental diagnoses. Specifically, patients with more historically reported psychiatric diagnoses were more likely to be referred on for in-person testing. However, when the number of historical psychiatric and developmental diagnoses was included in a model that accounts for unique variance across predictors, psychiatric history did not significantly predict diagnostic outcome. The type of historical psychiatric or development condition is also important to consider. The current data suggest anxiety and mood disorders are particularly important to understanding who may require more comprehensive in-person testing. Research suggests that ASD shares many symptoms with other psychiatric conditions, including anxiety and depression, making the differential diagnosis challenging (e.g., Greene et al., 2021; Wittkopf et al., 2021). Specifically, it can be difficult to accurately assess social communication deficits that may differentiate ASD and anxiety and mood disorders (Wittkopf et al., 2021). This differential diagnosis may be further complicated via virtual technology and further necessitate the need for an in-person behavioral observation. Decidedly, the psychiatric history is important for diagnostic clarity; however, may not adequately predict who may ultimately need additional testing in the presence of additional variables (e.g., educational services for ASD).

The lack of group differences based on caregiver-reported adaptive and emotional and behavioral functioning is consistent with a previous study suggesting that individuals with concerns regarding ASD may have generally elevated scores on such measures (Safer-Lichtenstein & McIntyre, 2020). Specifically, adaptive and emotional and behavioral functioning measures are unlikely to differentiate between patients who have presented for an ASD evaluation and receive confirmation of ASD compared to those and who do not receive an ASD diagnosis as most patients presenting for evaluation likely have elevated adaptive, emotional, and behavioral concerns.

Results also indicated that patients who received higher severity scores on the CARS-2 were more likely to receive diagnostic clarity at the virtual visit. This finding is unsurprising as it is expected that higher severity level of ASD symptoms would be related to clearer ASD presentation. While this finding highlights the sensitivity of the CARS-2 clinically, this finding cannot be implemented from a scheduling perspective (i.e., cannot be collected at time of scheduling).

The overarching purpose of the current study is to understand which patients may benefit most from a virtual ASD evaluation. Due to historically lengthy waitlists for ASD evaluation coupled with the COVID-19 pandemic, it is crucial for the field to be able to triage patients to a service where they are most likely to receive diagnostic clarity while increasing access to services. The current study indicates that unique patient characteristics, that are available at time of scheduling, may serve as estimates for probability of diagnostic outcomes. Specifically, it appears that younger children with educational services for ASD may be best suited for virtual ASD assessment, while older youth may be best suited for in-person evaluation.

Finally, while results indicated that, largely, variables from the virtual visit evaluation were unrelated to in-person diagnostic outcome (i.e., ASD confirmed vs. ASD ruled out), it is important to note that only 22 participants of 49 who were referred had completed an in-person evaluation by the time these analyses were conducted. That is, with a larger sample size, data collected within a virtual visit (e.g., CARS-2) may relate to in-person evaluation diagnostic outcomes. Therefore, future studies should investigate this with a larger dataset. Also notable is having an immediate family history of ASD positively related to receiving an ASD diagnosis at the time of the virtual visit. While this is an interesting finding in the presence of symptom assessment, it highlights heritable factors such that individuals with an immediate family history are more likely to also have ASD (e.g., Hallmayer et al., 2011).

Regarding variables collected during the in-person evaluation, the BOSA, a newly created behavioral observation method for assessing ASD symptoms, significantly related to diagnostic outcome. This is promising for future methodology in diagnostic clinics. Specifically, the BOSA appears to capture ASD specific symptoms that correlate with a confirmed ASD diagnosis. However, the fact that CARS-2 scores did not correlate with the BOSA scores deserves further evaluation. ASD diagnostic assessment is an ever-changing environment, particularly in the context of the COVID-19 pandemic. Current best practices recommend that the BOSA is part of a multimethod comprehensive assessment (e.g., administered alongside the CARS-2); however, it is possible that the ADOS-2 will come back into favor during evaluation. It is possible that the CARS-2 and BOSA did not correlate due to differences in direct in-person observation versus reliance on virtual observation and parent report. It is also important to consider the BOSA as an assessment regarding the interactions between a youth and their caregiver, which is inherently different than peer or other relationships. While both the BOSA and the CARS-2 are meant to assess both social communication and restricted and repetitive behaviors, it remains unclear how these two measures correlate across other samples. Therefore, it continues to be recommended that each of these assessment measures should be used as tools within a comprehensive ASD evaluation.

This study should be used alongside other studies to understand more holistically for whom virtual ASD diagnostics may be most appropriate. There are many possible factors that may impact the lack of relations found between virtual visit variables and diagnostic outcomes. For example, behavioral observations tend to be challenging to collect during virtual visits. While it is helpful to understand how a child interacts in their home environment, obtaining a true estimate of social communication and reciprocity is challenging via virtual technology. Additionally, psychological assessments, generally, aim to understand the skills that a person is able to reliably demonstrate across environments, and information regarding consistency across settings may be more challenging to acquire. It is also notable that the CARS-2 collected at the time of virtual evaluation was correlated with in-person diagnostic outcomes (i.e., higher CARS-2 severity score related to ASD diagnosis confirmed). This correlational data should be interpreted with caution as the CARS-2 may not be predictive of diagnostic outcome in this small sample. In addition to having a small in-person sample size, CARS-2 scores tended to be high across children in this sample as parents reported elevated concerns and behavioral observation opportunities were limited.

Strengths and Limitations

The current study has many strengths, including the use of a clinical sample of children who were assessed for ASD by highly trained psychologists at a major medical center. The use of a clinical sample increases the generalization and clinical use of the findings. Due to the nature of the current study, all patients participated in a virtual ASD assessment, which is a relatively novel approach to ASD assessment. Additionally, some of the patients also participated in an in-person evaluation which yielded ultimate diagnostic outcomes that are more representative of the traditional diagnostic model. The inclusion of both virtual and in-person evaluation outcomes assists in understanding of critical components of evaluation.

The current study is not without its limitations. Additionally, data collection continues to be ongoing, though dissemination of results was expedited due to the time sensitive nature of shifts in ASD assessment and need for guidance in scheduling. Only 44.90% of the participants referred for in-person testing had completed their in-person appointment by the time of data analysis. This inherently limits the amount of data available for analysis and could result in statistical differences that are not well represented in the current dataset.

While representative of the geographic region of the outpatient ASD diagnostic clinic, over 70% of the participants identified as Caucasian, which limits our understanding of the impact of racial/ethnic differences in virtual ASD evaluation. Therefore, future studies should investigate how racial disparities in ASD diagnosis may differ during virtual ASD assessment. Additionally, the current study did not collect a measure of socioeconomic status (SES) or income for participants. The lack of this important information limits the generalizability of the current study and should be considered as a future direction. It is also important to note that there are individuals for which a virtual visit may not be feasible (e.g., individuals without access to reliable internet, electronic devices, etc.). The current study did not assess those for whom a virtual visit was not possible due to accessibility issues. Future directions should include investigating the frequency of such accessibility difficulties, and alternative options for these patients.

Although the analyses aimed to understand group differences associated with caregiver-report measures of adaptive, emotional, and behavioral functioning, the amount of missingness significantly limited analyses in which measures could be included and confidence in findings. While it is unlikely that caregiver-reported measures alone distinguish likelihood of diagnostic clarity, it is possible that the current study does not fully capture important nuance that only caregivers can provide.

Finally, it is important to note that the current sample included a large age range which included many adolescents and school-aged children. There were only four participants under the age of three which were eligible for administration of structured virtual behavioral observations (i.e., TELE-ASD-PEDS). Therefore, we were unable to understand group differences attributed to the TELE-ASD-PEDS, which warrants its own investigations as a new measure to assist with virtual ASD evaluation.

Conclusions

In short, the field of psychology broadly has been required to adapt quickly over the past 19 months. The subspecialty of ASD evaluation is no different. It is possible that the changes to evaluation made during the COVID-19 pandemic are not for loss, and, rather, may impact the way in which we attempt to decrease waitlists and increase access to ASD evaluation and treatment. Findings from the current study indicate that patient characteristics including age, receipt of educational supports for ASD, and number of and type of historical psychiatric and developmental diagnoses may differentiate patients who are likely to receive diagnostic clarity or be referred for more comprehensive testing. Additionally, a clinician-mediated tool for assessing ASD (i.e., CARS-2) also significantly differentiated diagnostic outcome at the virtual visit. At this time, the study suggests family history of ASD may be important for understanding who may receive an ASD diagnosis at an in-person evaluation. Although the current study does not indicate important relations between virtual visit data and in-person diagnostic outcomes, this is largely attributed to the smaller sample size of children who currently have in-person evaluation and should be evaluated further. In conclusion, ASD diagnostic clinics may consider prioritizing younger patients with existing educational services under ASD for virtual evaluation while prioritizing older patients with a psychiatric history for in-person evaluation. While further research needs to be conducted to best inform scheduling and patient routing decisions, these data indicate that older patients with a psychiatric history (i.e., multiple historical diagnoses or anxiety and mood disorders) may benefit from being directly referred for in-person evaluation. As many younger children may receive diagnostic clarity during a virtual evaluation; it may be appropriate to have young children prioritized to receive a virtual ASD evaluation, as an in-person evaluation may not be required for diagnostic clarity. Ultimately, these results aim to inform patient scheduling to, in turn, yield benefit for patient and family experiences and clinic waitlists alike.