A Comparison of the Autism Treatment Evaluation Checklist (ATEC) and the Childhood Autism Rating Scale (CARS) for the Quantitative Evaluation of Autism.

The purpose of this study was to evaluate scores generated from the Autism Treatment Evaluation Checklist (ATEC), a parent-rated measure, and those derived from professionally completed Childhood Autism Rating Scale (CARS) evaluations. A cohort of 56 participants diagnosed with an autism spectrum disorder was used for the study, and each child was evaluated independently by the parent using the ATEC and a health care professional using the CARS. The Spearman's rank correlation statistic ρ was used to evaluate the correlation between ATEC and CARS scores. It was observed that there was a significant correlation between total ATEC and CARS scores (ρ = .71). Specific domains in the ATEC evaluation significantly correlated with CARS scores. Sensitivity, specificity, and receiver operating characteristic confirmed the association between CARS and ATEC domains. The results help to validate the utility of the parentally completed ATEC in comparison with an established, professional-related measure of autism.

The Diagnostic and Statistical Manual of Mental Disorders (4th ed., text revision; DSM-IV-TR), published by the American Psychiatric Association (APA; 2000) for the classification of mental disorders, is used to diagnose autistic disorder. The DSM-IV-TR criteria for autistic disorder includes (a) qualitative impairment in social interaction; (b) qualitative impairments in communication; and (c) restricted repetitive and stereotyped patterns of behavior, interests, and activities. Although an autism spectrum disorder (ASD) diagnosis is defined by these three core features (APA, 2000), other features, more physical or systemic in nature, are associated with an ASD diagnosis. These features include, but are not limited to, motor deficits (Piek & Dyck, 2004), fine and gross motor problems (Provost, Heimerl, & Lopez, 2007), balance problems (Minshew, Sung, Jones, & Furman, 2004), muscle weakness (Hardan, Kilpatrick, Keshavan, & Minshew, 2003), sleep problems (Gail Williams, Sears, & Allard, 2004), gastrointestinal disturbances (Nikolov et al., 2009), eating problems (Critchfield, van Hemert, Ash, Mulder, & Ashwood, 2011)), and hyperactivity or lethargy (Banaschewski, Poustka, & Holtmann, 2011). Research suggests that children with ASD are more likely to have health problems and illnesses in general (Atladóttir et al., 2010; Schieve et al., 2011) than typically developing children.

Although there are several measures available, such as the Autism Diagnostic Observation Schedule (Lord et al., 2000), the Autism Behavior Checklist (Krug, Arick, & Almond, 1980), and the Gilliam Autism Rating Scale (South et al., 2002), to be used in the evaluation of ASD, there is a paucity of measures that address these aforementioned features that include more physical or systemic issues in ASD. Noting these issues in autism has the potential to aid in our understanding of the disorder and in possible treatments. One measure, the Autism Treatment Evaluation Checklist (ATEC; Rimland & Edelson, 1999) is unique in that it does address these issues. However, little research has been conducted to compare the ATEC with more well-established ASD measures.

Another issue in ASD is that an ASD diagnosis is marked by a spectrum that displays a wide range of severity from mildly affected to severely affected. Mildly affected children and adults with ASD may develop life skills at an early age and be able to function with minimal assistance, whereas more severely affected children and adults with ASD may be unable to function in almost any setting and require constant supervision. A current and frequently used measure for assessing the severity of ASD in research studies is the Childhood Autism Rating Scale (CARS; Schopler, Reichler, & Renner, 1994). The CARS is a well-established, professional-rated measure in that it is widely used and well validated. The CARS is designed to identify autism as well as to quantitatively describe the severity of the disorder.

The purpose of this study was to compare the scores of the ATEC, a parent-rated measure, with the CARS, a well-established measure that is a recognized way to measure autism severity. Further, this study was designed to evaluate the potential correlation between CARS scores and the specific diagnostic domains in the ATEC evaluation.

METHOD

The study protocol received Institutional Review Board (IRB) approval from Liberty IRB, Inc. (Deland, FL). The study complied with the American Psychological Association ethical standards in the treatment of participants and in obtaining informed consent. All parents signed a consent and a Health Insurance Portability and Accountability Act form and all received a copy.

Participants

Children with a diagnosis of autism, ASD, and pervasive developmental disorder-not otherwise specified (PDD-NOS) were prospectively recruited from the community. After explaining the study and obtaining informed consent from the parent(s), each child was evaluated by a trained professional using the CARS and then the parents completed the ATEC. Neither the professional nor the parent was aware of the scores generated from their respective completed tests. Statistical analysis was then conducted to examine the relationship between the CARS and the ATEC.

A total of 56 participants diagnosed with an ASD were prospectively recruited from the community by using fliers and word of mouth. Each child in the ASD group had been previously diagnosed by a professional. In the state of Texas, the only professionals who are allowed to diagnose ASD are either licensed clinical psychologists or medical doctors. To further evaluate each child's diagnostic accuracy, the diagnosis was confirmed by one of the authors. In addition, the CARS was completed on each child. The study was designed to exclude children with Fragile X disorder, tuberous sclerosis, phenylketonuria, Lesch-Nyhan syndrome, seizure disorder, cerebral palsy, fetal alcohol syndrome, or any history of maternal illicit drug use. Detailed information was collected on each participant regarding age, race, gender, and year of birth.

The children who participated in the study were 2 to 16 years of age (M = 6.5, SD = 3) with a diagnosis of either autism (n = 34) or ASD (n = 22). There were 49 males and 7 females, and regression was reported in 36 (64%) of the children. The children were Caucasian (n = 41) or of Hispanic, Black, Asian, or Mixed Ancestry (n = 15). Table 1 summarizes the scores from the ATEC and the CARS.

TABLE 1

Autistic Disorder Characteristics (n = 56)

Scores	M ± SD	Median	Range	Percentile severity
CARS	36.9 ± 6.1	36.3	24 to 51
ATEC
Total ATEC	57.4 ± 24.3	57.0	8 to 106	30–39
Speech/Language/Communication	10.0 ± 6.6	9.0	0 to 24	30–39
Sociability	12.9 ± 7.0	14.0	0 to 26	40–49
Sensory/Cognitive Awareness	13.6 ± 6.9	14.5	0 to 27	40–49
Health/Physical Behavior	21.0 ± 9.7	20.0	5 to 52	40–49

CARS = Childhood Autism Rating Scale; ATEC = Autism Treatment Evaluation Checklist.

Instruments

Childhood Autism Rating Scale (CARS). Study participants were evaluated using a CARS conducted only by a single study investigator (JKK) who observed the participants and interviewed the parent(s). The investigator who completed the CARS has been formally trained in the administration of the CARS. The CARS is suitable for use with any child over 2 years of age and studies show that it is a reliable and stable indicator of autism in adolescents (Garfin, McCallon & Cox, 1988). The CARS was completed prior to the parents completing the ATEC.

The CARS is a 15-item behavioral rating scale developed to identify autism as well as to quantitatively describe the severity of the disorder (Schopler et al., 1994). The items are as follows: I. Relating to People; II. Imitation; III. Emotional Response; IV. Body Use; V. Object Use; VI. Adaptation to Change; VII. Visual Response; VIII. Listening Response; IX. Taste, Smell, and Touch Response and Use; X. Fear or Nervousness; XI. Verbal Communication; XII. Nonverbal Communication; XIII. Activity Level; XIV. Level and Consistency of Intellectual Response; and XV. General Impressions. Each item is scored from 1 (no pathology) to 4 (severe pathology) in 0.5 intervals. A total score of 15–29.5 is considered nonautistic; a score of 30–36.5 is considered mild to moderate autism; a score of 37–60 is considered moderate to severe autism (these are based on raw scores). These cutoff scores were determined in a study of 537 children enrolled in the Treatment and Education of Autistic and related Communication Handicapped Children (TEACCH) program over a 10-year period (Schopler, Reichler, DeVellis, & Daly, 1980).

The CARS is a well-established, commonly used measure with good psychometrics. The internal consistency reliability alpha coefficient is .94, the interrater reliability correlation coefficient is .71, and the test-retest correlation coefficient is .88 (Schopler et al., 1994). CARS scores have high criterion-related validity when compared to clinical ratings during the same diagnostic sessions, with a correlation of .84 (p < .001; Schopler et al., 1994). As with the cutoff scores, these psychometrics were determined in a study of 537 children enrolled in the TEACCH program over a 10-year period by Schopler et al. (1980). Other comparisons, based on information from records, parent interviews, and nonstructured clinical interviews with the child, report a correlation coefficient of .80 (p < .001). Independent reports on the validity of the CARS indicate that it has high validity. Eaves and Milner (1993) found that it correctly identified 98% of participants with autism and correlated (r = .67) with the Autism Behavior Checklist (Krug et al., 1980). In a similar study by Sevin, Matson, Coe, Fee, and Sevin (1991), 92% of participants were correctly classified, and the CARS correlated with the Real Life Rating Scale (Freeman, Ritvo, Yokota, & Ritvo, 1986). Pereira, Riesgo, and Wagner (2008) found that the convergent validity, in comparison with the Autistic Traits Assessment Scale (Ballabriga, Escude, & Llaberia, 1994), exhibited a Pearson's correlation coefficient of r = .89. Perry, Condillac, Freeman, Dunn-Geier, and Belair (2005), for example, studied a sample of 274 preschool children (ages 2–6 years) clinically diagnosed as falling in one of five groups—Autistic Disorder, PDD-NOS, mental retardation (MR), Delayed, and Other—and found that the CARS had a high concordance with clinical diagnosis using DSM-IV (including excellent sensitivity and specificity). Similarly, Rellini, Tortolani, Trillo, Carbone, & Montecchi (2004) found complete agreement between the DSM-IV and CARS. They also found that the number of false negatives in distinguishing individuals with autistic disorders from other cases of developmental disorders was 0% with CARS. The CARS was shown to have good sensitivity and specificity in distinguishing children with autism from trainable children with mental retardation (Teal & Wiebe, 1986), distinguishing autistic disorder from PDD-NOS, and distinguishing ASD from other developmental disorders and typical development (Chlebowski, Green, Barton, & Fein, 2010).

Autism Treatment Evaluation Checklist (ATEC). Among qualifying participants, the participant's parent completed an ATEC form (Rimland & Edelson, 1999). The ATEC is a one-page form designed to be completed by parents, teachers, or others who see the individual's behavior on a regular basis (Rimland & Edelson, 1999). The age range for the ATEC is 2 years of age and older. The ATEC consists of four subtest scales: Scale I. Speech/Language/Communication (14 items—scores can range from 0 to 28), Scale II. Sociability (20 items—scores can range from 0 to 40), Scale III. Sensory/Cognitive Awareness (18 items—scores can range from 0 to 36), and Scale IV. Health/Physical Behavior (25 items—scores can range from 0 to 75). The four subscale scores can be used to calculate a total score (total scores can range from 0 to 180). The scores are weighted according to the response and the corresponding subscale. The higher the subscale and total score, the more impaired the participant. The lower the subscale and total score, the less impaired the participant. The overall scores in each subscale and the total score can be extrapolated to determine the percentile of severity of the participant in comparison with score distributions provided by the Autism Research Institute.

The ATEC form was developed by the Autism Research Institute (Rimland & Edelson, 1999). Pearson split-half (internal consistency) coefficients provided by the Autism Research Institute based upon evaluation of 1,358 participants revealed uncorrected r values as follows: Scale I. Speech/Language/Communication (r = .92), Scale II. Sociability (r = .84), Scale III. Sensory/Cognitive Awareness (r = .88), Scale IV. Health/Physical Behavior (r = .82), and total score (r = .94). The uncorrected correlation coefficients represent a simple correlation without correcting for other factors between the variables. The internal consistency reliability of the measure is high (r = –.94 for the total score). The ATEC has been successfully used to measure treatment effects and progress over time in several studies in ASD (Jarusiewicz, 2002; Lonsdale, Shamberger, & Audhya, 2002; Magiati, Moss, Yates, Charman, & Howlin, 2011). Moreover, the ATEC has been found to correlate with physical symptoms (Adams, Johansen, Powell, Quig, & Rubin, 2011) and biomarkers in ASD (Kern, Geier, Adams, & Geier, 2010). However, the ATEC is not nationally normed and the reliability and validity evidence is limited.

Statistical Analyses

This study used the statistical package SAS JMP 9. The nonparametric Spearman's rank correlation test statistic (ρ) was utilized to evaluate the relationship between the nonnormally distributed CARS and ATEC scores. receiver operating characteristic (ROC) was used through logistic regression to determine the optimal cutoff point for ATEC total and each domain and also to evaluate the Sensitivity, Specificity and Accuracy between the CARS and ATEC. The ROC curve is a graphical representation of the relationship between 1 – specificity (false-positive) and sensitivity (true-positive rates). A standard way to evaluate the relationship is with the area under the curve (AUC), shown below the plot. In the plot, a line is drawn at a 45-degree angle tangent to the ROC curve. This marks a good cutoff point under the assumption that false negatives and false positives have similar costs. The AUC closer to 1 presents the best relationship. Sensitivity is defined as the proportion of participants rated severe by the CARS and the ATEC to the number of participants rated severe by CARS. Similarly, specificity is defined as the proportion of participants rated mild-moderate by the CARS and the ATEC to the number of participants rated mild-moderate by CARS.

RESULTS

Table 1 presents the autism characteristics of the sample evaluated in this study. Table 2 summarizes the relationship between CARS and ATEC scores. It was observed that there was a significant (p < .0001) correlation between the total ATEC and CARS scores (ρ = .71, p < .0001). Specific domains in the ATEC evaluation significantly correlated with CARS scores as follows: Sensory/Cognitive Awareness (ρ = .74, p < .0001) > Speech/Language/Communication (ρ = .72, p < .0001) > Sociability (ρ = .55, p < .0001) > Health/Physical Behavior (ρ = .31, p < .01). These results suggest that the highest level of correlation between the CARS and the ATEC was in the Sensory/Cognitive Awareness domain, followed by the Speech/Language/Communication domain, then the Sociability domain. The lowest correlation coefficient was in the Health/Physical Behavior domain.

TABLE 2

Spearman's Rank Correlations Between CARS Scores and ATEC Domain Scores (n = 56)

ATEC domain	Rho	p value
Total ATEC score	.71	<.0001
Speech/Language/Communication ATEC score	.72	<.0001
Sociability ATEC score	.55	<.0001
Sensory/Cognitive Awareness ATEC score	.74	<.0001
Health/Physical Behavior ATEC score	.31	<.0219

CARS = Childhood Autism Rating Scale; ATEC = Autism Treatment Evaluation Checklist.

Table 3 presents the cutoff point, sensitivity, specificity, and accuracy between CARS and total ATEC and each one of the four ATEC domains. Sensory/Cognitive Awareness is the best predictor of CARS with cutoff point equal to 12, which means that if the score is greater than 12 the child is considered severely affected, sensitivity equal to 1.00, specificity is .67, AUC is .89, and accuracy is .84. Figure 1 shows the ROC analysis of the relationship between sensitivity (true-positive) and 1 – specificity (false-positive).

TABLE 3

Autistic Disorder: Sensitivity, Specificity, and Accuracy Between CARS Scores and ATEC Domain Scores (n = 55[a])

Scores	Cutoff point	Sensitivity	Specificity	Area under curve	Accuracy
ATEC
Total ATEC	49	0.96	0.67	0.87	0.82
Speech/Language/Communication	7	1.00	0.63	0.87	0.82
Sociability	11	0.89	0.67	0.79	0.78
Sensory/Cognitive Awareness	12	1.00	0.67	0.89	0.84
Health/Physical Behavior	26	0.39	0.89	0.64	0.64

CARS = Childhood Autism Rating Scale; ATEC = Autism Treatment Evaluation Checklist.

One participant was rated nonautistic by CARS; this case was excluded from the sensitivity, specificity, receiver operating characteristic, and accuracy analysis.

FIGURE 1

Receiver operating characteristics. ATEC = Autism Treatment Evaluation Checklist.

DISCUSSION

The results of this study showed a significant association between professionally derived CARS scores and parentally derived ATEC scores using Spearman correlation and AUC analysis. It was further observed that there was significant association between specific domains from the ATEC evaluation and CARS scores, supported with the two statistical methods.

As mentioned in the introduction, many children with autism show in addition to the psychiatric diagnostic triad of (a) qualitative impairment in social interaction; (b) qualitative impairments in communication; and (c) restricted repetitive and stereotyped patterns of behavior, interests, and activities, other features that are more physical or systemic in nature. Although there are several measures available to be used in the evaluation of children and adults with ASD, there is a paucity of measures that address these associated features of a more physical or systemic nature. However, ATEC is unique in that it does address these issues, and this study begins the process of comparing the ATEC with more well-established ASD measures.

As mentioned, the highest AUC and correlation between the CARS and the ATEC was in the Sensory/Cognitive Awareness domain, followed by the Speech/Language/Communication domain, then the Sociability domain. The lowest correlation was in the ATEC Health/Physical Behavior domain. These results would be expected because, as mentioned earlier, the CARS and most ASD measures do not specifically address health and physical issues.

Because recent research suggests that many children with ASD have more physical and health related issues (Schieve et al., 2011), the ATEC has useful attributes to help us study and better understand the physical issues in ASD, such as sleep problems, seizures, gastrointestinal issues, and so on. Further, the ATEC can quantitatively assess these features. Other advantages of the ATEC include simple administration, easy to understand, and that it can be completed by a parent usually within 15 min and thus does not create an undue burden upon participants. The ATEC, in addition to providing a quantitative overall score of severity, provides quantitative domain-specific scores, and ATEC scores can be translated to percentiles of severity. Finally, the ATEC is not copyrighted and may be used free of charge by any parent, researcher, or health care provider. Copies are available on request from the Autism Research Institute (ARI) or the ARI website (http://www.autism.com/ind_atec_survey.asp). The ATEC form can be instantly scored by entering the responses on the ARI website.

Among the strengths of this study was the fact that data was collected prospectively on a cohort of children diagnosed with an ASD, and so unknown potential biases/confounding regarding noncontinuously collected cohorts of participants should have minimally impacted the present study. Raters of the ATEC of CARS were unaware of the results of the other instrument. Further, because a single professional undertook CARS evaluations of study participants, subjective differences in scoring methods between different professionals were minimized, and this helped to significantly reduce potential unknown confounding in the data examined. Also, CARS and ATEC scores, despite being collected for study purposes, were collected in a similar fashion as would be expected to occur in a clinical setting where patients are examined, so that data collection methods should not be unique to the study methods employed but should yield similar results in a standard clinical setting where patients are examined.

Another strength of this study was the demographics of the cohort of participants diagnosed with an ASD examined in the study appear to be similar to the recognized demographics of the general population diagnosed with an ASD, so that the results observed should be expected to be extendible beyond the cohort of participants diagnosed with an ASD examined in this study. In addition, because the participants diagnosed with an ASD examined in this study were wide-ranging with respect to age, gender, racial composition, and severity, potential outlier skewing of the data should not have significantly impacted the results observed.

An additional strength of this study was the consistency and magnitude of the correlations observed between ATEC and CARS scores and the limited number of statistical tests performed. As a result, it is unlikely that the results observed in this study were the result of statistical chance.

Among the limitations of the study is that participants examined were assumed to be on the autism spectrum based upon the fact that they were previously diagnosed with an ASD and a subsequent professional CARS evaluation. It is possible that other tests such as Autism Diagnostic Observation Schedule (Lord et al., 2000) could have influenced whether the study participants were considered on the autism spectrum. Further, other tests metrics may have revealed different potential relationships with the parent-completed ATEC evaluations examined in this study. Despite this potential limitation, CARS evaluations are a well-recognized metric of helping to establish an ASD diagnosis, provide important quantitative measurements of ASD severity, and yield significant correlations with parent-completed ATEC evaluations. Future studies should further explore the potential correlations between other metrics of ASD symptoms with ATEC testing. Another potential limitation of this study was that participants were examined a single point in time using ATEC and CARS evaluations. It is not clear how the presently observed correlations between ATEC and CARS evaluations would vary over the course of multiple examinations at multiple time points. Future studies should further the consistency of the correlation between ATEC and CARS over multiple examinations at different times. Another study limitation is that the measures were not standardized.

In conclusion, the results of this study revealed a significant correlation between total ATEC and CARS scores. It was also observed that there were significant correlations between the ATEC domains and the CARS total score. The findings from the study validate the parent-completed ATEC in comparison with the CARS, an established, professional-related measure of autism. Furthermore, integrating ATEC testing into health care practice in the quantitative evaluation of participants diagnosed with an ASD would allow for the examination and quantification of the symptoms in ASD that are more physical and health related in nature.