Comparison of the Autism Diagnostic Observation Schedule and Childhood Autism Rating Scale in the Diagnosis of Autism Spectrum Disorder: A Preliminary Study.

OBJECTIVES: We examined the agreement between the Autism Diagnostic Observation Schedule (ADOS) and the Childhood Autism Rating Scale (CARS) in the diagnosis of autism spectrum disorder.
METHODS: The ADOS and CARS scores of 78 children were retrospectively collected from a chart review. A correlation analysis was performed to examine the concurrent validity between the two measures. Using the receiver operating characteristic (ROC) curve, we determined the optimal cut-off score of the CARS for identifying autism spectrum disorder.
RESULTS: The CARS score was significantly correlated with the ADOS score (r=0.808, p<0.001). Taking ADOS as the ideal standard, the optimal cut-off scores of CARS for identifying autism and autism spectrum were 30 and 24.5, respectively.
CONCLUSION: We determined the optimal cut-off scores of CARS for screening and diagnosing autism spectrum disorder. Copyright: © Journal of the Korean Academy of Child and Adolescent Psychiatry.

INTRODUCTION

Autism spectrum disorder (ASD) is characterized by deficits in social communication and interaction, along with restricted and repetitive patterns of behavior [1]. It is manifested early in life, and early intervention is reported to improve prognosis [2]. Accordingly, sufficiently early identification of children requiring therapeutic interventions is important. However, there is currently no reliable biomarker to serve this purpose, and diagnosis of the disorder is based on criteria such as the Diagnostic and Statistical Manual of Mental Disorders (DSM) [3].

In clinical practice, rating instruments such as the Childhood Autism Rating Scale (CARS) or the Autism Diagnostic Observation Schedule (ADOS) are used to help clinicians determine the diagnosis. In Korea, the CARS is one of the most widely used tools for assessing ASD. Based on parent reports, the CARS can be applied in a relatively short time, making the instrument a useful screening tool in the community [4,5]. However, the ADOS, a semi-structured instrument recognized as one of the most reliable tools in the diagnosis of ASD, requires a long time to administer [6]. In addition, a qualified examiner (rather than the parent) with sufficient clinical experience requires direct observation of the child to complete the instrument; therefore, the ADOS is not widely used.

A series of Korean studies have been conducted thus far using either the CARS or the ADOS [7-10], but comparisons of these tools are lacking. Instead, studies conducted outside of Korea have reported that the CARS and ADOS scores are significantly correlated [11,12]. However, in terms of their agreement with clinical diagnosis, the ADOS exhibited better agreement than did the CARS, and it has been suggested that the ADOS may be more useful than CARS in patients with mild symptoms [5,11].

The reported differences between the CARS and the ADOS, together with the strengths and limitations of each instrument, warrant a systematic comparison of these tools. Findings of such a study would help to determine whether it would be possible to diagnose, or at least screen for, ASD with the CARS alone (which is relatively easy to administer and thus could be widely used in the community), or an ongoing effort should be made to recommend that all institutions should use the ADOS despite the increased implementation time and cost.

METHODS

Subjects

The study subjects comprised children aged 2–10 years (24–120 months) who were suspected to have developmental delay and evaluated with both the CARS and ADOS, among those seen by the principal investigator (S.-B.H.) at the child and adolescent psychiatry outpatient clinic under Seoul National University Children’s Hospital between January 2013 and December 2014. The subjects were assessed with the ADOS and CARS on the same day by two professionally trained examiners who had either completed a master’s course or received a doctoral degree in special education. Both examiners were trained in and licensed to implement the ADOS in the US and had many years of clinical experience at the study institution. The subjects’ ADOS and CARS scores, as well as two demographic variables (age and sex), were retrospectively collected from the medical chart. This study was approved by Seoul National University Hospital Institutional Review Board (IRB No. 1803-103-931).

Assessment tools

ADOS

The ADOS is a semi-structured diagnostic tool for autistic disorder, developed by Lord et al. in 1999 [6]. The instrument assesses communication, social interaction, play or imaginative use of objects, and stereotypic behavior, among other factors, by means of direct observation of the examinee. It is based on the DSM-IV and the International Classification of Diseases, Tenth Revision (ICD-10), and its utility has been demonstrated in numerous validity and reliability studies conducted in Europe and North America. Together with the Autism Diagnostic Interview-Revised, the ADOS is one of the major tools recognized as the standard in research on autistic disorder. The score is used to classify the examinee’s condition into autism, autism spectrum, or non-autism. In the ADOS, one of four modules is administered according to the examinee’s language level and age. Module 1 is used for children who are nonverbal or whose vocabulary is limited to a few words, Module 2 is used for children who use phrase speech, Module 3 is used for verbally fluent children, and Module 4 is used for verbally fluent adolescents or adults. In the present study, Modules 1 and 2 were used. In Module 1, the examinee’s condition is classified as autism spectrum if the combined score across two domains of social interaction and communication is ≥7, and as autism if the combined score is ≥12. In Module 2, a combined score of ≥8 corresponds to autism spectrum, and a combined score of ≥12 corresponds to autism [13].

CARS

Developed by Schopler et al. in 1988 [4], the CARS is a 15-item behavioral rating scale designed to differentiate autistic disorder from other developmental disorders and assess the severity of autism. The items are reflective of the definition of autistic disorder from a wide range of references and various diagnostic criteria. The scale has been refined based on its use in more than 1500 children in the US over 10 years, and has demonstrated high reliability, validity, and inter-rater agreement [7]. The CARS can be used for children of all ages including preschoolers [7]. The scale’s clinical utility has been demonstrated in a variety of studies, including those conducted in children as young as 2 years old [12,14-16]. In addition, the scale is objective and quantitative, as the rating is based on direct behavioral observation and an interview rather than subjective clinical judgment. Each item is rated from 1 (age-appropriate) to 4 (severe impairment), and the total score is calculated by summing the individual item scores. The total scores range from 15 to 60, and the cut-off score to determine autism is 30. More specifically, a score of <30 is classified as non-autism, a score of 30–36 is classified as mild to moderate autism, and a score of ≥37 is classified as severe autism. In a standardization study conducted in 97 subjects in Korea in 1998, a score of 28 was proposed to be the autism diagnostic cut-off [7]. In the present study, however, we used the original cut-off score (i.e., 30) that was established at the time of its development, and discussed our findings in comparison with those of the previous study conducted in Korea.

Data analysis

Pearson’s correlation analysis was performed to test the concurrent validity of the CARS and ADOS. Correlations between age and CARS score, and between age and ADOS score were also examined via the same analytical method. Age-specific differences between CARS and ADOS scores were tested using the Mann-Whitney U test. To determine the optimal CARS diagnostic cut-off score for ASD diagnosed based on ADOS, a receiver-operating characteristic (ROC) curve was used. All data analyses were performed using SPSS version 22.0 (IBM Corp., Armonk, NY, USA). Statistical significance was determined as a p value of <0.05.

RESULTS

Demographic and clinical characteristics of the subjects

The total number of study subjects was 78, and their mean age was 49 months. There were 68 boys (87.2%) and 10 girls (12.8%). Age distribution is presented in detail in Supplementary Table 1 in the online-only Data Supplement. ADOS Module 1 was used for 64 subjects, and Module 2 was used for 14 subjects. The mean total CARS score was 30.58, which corresponds to mild to moderate autism. The mean total ADOS score was 12.73, which corresponds to autism. The mean score of ADOS Module 1 was 13.08, and the mean score of Module 2 was 11.14 (Table 1). Age was not significantly correlated with either the CARS total score or the ADOS total score. There was no significant difference in the CARS or ADOS total scores by sex.

Table 1

Demographic and clinical characteristics of the subjects

	N	Mean (SD)	Range
Age (month)	78	49.01 (17.15)	25.0-103.0
CARS total score	78	30.58 (4.31)	20.0-42.5
ADOS total score	78	12.73 (3.97)	3.0-23.0
ADOS Module 1 total score	64	13.08 (4.21)	3.0-23.0
ADOS Module 2 total score	14	11.14 (1.99)	8.0-15.0

ADOS: Autism Diagnostic Observation Schedule, CARS: Childhood Autism Rating Scale, SD: standard deviation

Concurrent validity

The CARS and ADOS total scores were significantly correlated with each other (r=0.808, p<0.001). Additionally, the CARS total score was significantly correlated with both the ADOS Module 1 (r=0.805, p<0.001) and Module 2 scores (r=0.712, p=0.004) (Table 2).

Table 2

Bivariate correlations between CARS and ADOS scores

	ADOS total score	ADOS Module 1	ADOS Module 2
CARS total score	0.808 (p<0.001)	0.805 (p<0.001)	0.712 (p=0.004)

ADOS: Autism Diagnostic Observation Schedule, CARS: Childhood Autism Rating Scale

Diagnosis and severity classification

The association between classifications based on the ADOS and CARS was examined (Table 3). All children classified as non-autism on the ADOS were classified as such on the CARS. Of the 23 children classified as being within the autism spectrum on the ADOS, 21 (91.3%) were classified as non-autism on the CARS. Among the 50 children classified as having autism on the ADOS, 44 (88.0%) were classified as such on the CARS.

Table 3

Number of subjects according to diagnostic classification by ADOS and CARS

	CARS
	Negative	Mild/moderate	Severe	Total
ADOS (%)
Negative	5 (6.4)	0	0	5 (6.4)
Autism spectrum	21 (26.9)	2 (2.6)	0	23 (29.5)
Autism	6 (7.7)	42 (53.8)	2 (2.6)	50 (64.1)
Total	32 (41.0)	44 (56.4)	2 (2.6)	78 (100)

ADOS: Autism Diagnostic Observation Schedule, CARS: Childhood Autism Rating Scale

Optimal diagnostic cut-off in the CARS

Regarding the diagnosis of autism based on the ADOS, the sensitivity of the CARS was 88.0%, the specificity was 92.9%, the positive predictive value was 95.7%, and the negative predictive value was 81.3%. Regarding the diagnosis of either autism or autism spectrum on the ADOS, the sensitivity of the CARS was 63.0%, the specificity was 100%, the positive predictive value was 100%, and the negative predictive value was 15.6%.

The overall test accuracy was estimated by calculating the area under the curve (AUC) using an ROC graph. If we diagnosed only autism on the ADOS, the AUC value of the CARS was 0.931 and the highest levels of sensitivity (88.0%) and specificity (92.9%) were observed at a diagnostic cut-off score of 29.75 (Table 4). If we diagnosed both autism and autism spectrum on the ADOS, the AUC value of the CARS was 0.953 and the highest levels of sensitivity (98.6%) and specificity (80.0%) were observed at a diagnostic cut-off score of 24.25 (Table 5).

Table 4

Sensitivity and specificity for different cut-off scores in the CARS for a diagnosis of autism (not including the autism spectrum) by ADOS

Cut off score	Sensitivity	Specificity
CARS 27.75	0.920	0.643
CARS 28.25	0.920	0.679
CARS 28.75	0.920	0.857
CARS 29.25	0.880	0.857
CARS 29.75*	0.880	0.929
CARS 30.25	0.800	0.929
CARS 30.75	0.760	0.929
CARS 31.25	0.760	0.964

the best cut-off score. ADOS: Autism Diagnostic Observation Schedule, CARS: Childhood Autism Rating Scale

Table 5

Sensitivity and specificity for different cut-off scores in the CARS for a diagnosis of autism spectrum and autism by ADOS

Cut off score	Sensitivity	Specificity
CARS 21.25	1.000	0.400
CARS 22.25	1.000	0.600
CARS 23.25	0.986	0.600
CARS 24.25*	0.986	0.800
CARS 25.25	0.945	0.800
CARS 25.75	0.918	0.800
CARS 26.25	0.904	0.800
CARS 26.75	0.877	0.800
CARS 27.25	0.795	0.800
CARS 27.75	0.767	1.000

the best cut-off score. ADOS: Autism Diagnostic Observation Schedule, CARS: Childhood Autism Rating Scale

Utility of the CARS as a screening tool

Based on the newly-derived diagnostic cut-off scores, the feasibility of using the CARS as a screening tool for ASD was investigated. If the aforementioned conventional diagnostic cut-off score of 30 was applied, of the 32 children classified as non-autism on the CARS, 21 were classified as being within the autism spectrum and 6 were classified as having autism on the ADOS (Table 3). In contrast, if the cut-off score was lowered to 24.25, of the 5 children classified as non-autism on the CARS, 1 child was classified as being within the autism spectrum (23 on CARS but 7 on ADOS) and none were classified as having autism on the ADOS.

Sensitivity analysis in boys only

A secondary analysis conducted in boys alone revealed similar results. A significant correlation was found between the CARS and ADOS scores. Among the 43 boys classified as having autism on the ADOS, 38 (88.4%) were classified as such on the CARS. Among the 20 classified as being within the autism spectrum on the ADOS, 19 (95.0%) were classified as non-autism on the CARS. If we diagnosed only autism on the ADOS, the optimal CARS cut-off score was 28.75 if sensitivity was prioritized (sensitivity 93.0% and specificity 88.0%), and 29.75 if specificity was prioritized (sensitivity 88.4% and specificity 96.0%). If we diagnosed both autism and autism spectrum on the ADOS, the optimal CARS cut-off score was 24.25, and this finding was identical to the result from the total sample.

DISCUSSION

The present study aimed to compare two widely used instruments in the diagnosis of autism, namely, the CARS and the ADOS. An additional objective of the study was to identify the optimal CARS diagnostic cut-off score, in relation to the diagnoses made with the ADOS.

The results were as follows: first, the correlation between the total CARS and ADOS scores was highly positive, and the total CARS score was also highly correlated with both of the ADOS Module 1 and 2 scores. Among the 46 children classified as having autism on the CARS, 44 (96%) were classified as such on the ADOS, and of the 50 children classified as having autism on the ADOS, 44 (88%) were classified as such on the CARS. Hence, the two tests were highly agreeable with each other. These findings are consistent with previous reports indicating that the tests exhibit a statistically significant correlation [11] and agreement for the diagnosis of ASD [12]. The present study further investigated the optimal CARS cut-off score for the screening of children diagnosed with autism on the ADOS, and estimated the diagnostic cut-off score to be 29.75, which is close to the existing cut-off (i.e., 30). Given that the CARS is scored in increments of 0.5, the finding that autism is indicated at a score of ≥29.75 is congruent with the existing diagnostic cut-off score of 30.

In contrast, 21 of 23 children classified as being within the autism spectrum on the ADOS were in the non-autism range on the CARS. Therefore, using the current CARS criterion alone, we may fail to screen for children who would be classified as being within the autism spectrum on the ADOS. Previous studies have also reported that the diagnostic rate was lower with the CARS than with the ADOS in children with relatively mild autistic symptoms [5,11]. These findings are in line with the hypothesis that the CARS is more useful in diagnosing children of a low functional level because it includes such developmental items as intelligence level, verbal skill, and the ability to imitate [5,17], whereas the ADOS is useful in diagnosing patients with relatively unclear symptoms as well [3,18].

Accordingly, we examined the optimal CARS diagnostic cut-off score in the screening of children with relatively mild symptoms who would be diagnosed as being within the autism spectrum on the ADOS, and estimated the cut-off score to be 24.25. Thus, if the CARS score is ≥24.5, the possibility of autism spectrum according to the ADOS should be considered.

The diagnostic cut-off score proposed in the present study is similar to the findings of studies conducted outside of Korea. A Japanese study has suggested a CARS score of 25.5/26 as the diagnostic cut-off score to differentiate children with autistic disorder, Asperger’s syndrome, childhood disintegrative disorder, or pervasive developmental disorder-not otherwise specified (PDD-NOS) from those with intellectual disability [14]. Another study suggested that the CARS cut-off score to differentiate children with autistic disorder from those with PDD-NOS is ≥30, whereas the CARS score to discriminate groups with and without ASD is 25.5 [16]. Since the items of CARS include symptoms observed in typical severe autism, such as abnormal movements (e.g., shaking, spinning, and toe-walking) and abnormal responses to visual, olfactory, or tactile stimuli, concerns have been raised for CARS regarding possible failure to diagnose Asperger’s syndrome or high functioning autism [17]. Moreover, the need to lower the CARS diagnostic cut-off score has been suggested to accurately diagnose patients with relatively higher intelligence or higher function [19]. These concerns may be due to the absence of item categories in the CARS that are reflective of the characteristics of high functioning ASD patients, such as inability to empathize (e.g., problems in socioemotional understanding) and qualitative difficulty in verbal and non-verbal communication.

The present study additionally conducted a sensitivity analysis exclusively in boys. Although the clinical presentation of autism may not significantly differ between boys and girls [20], the prevalence of autism is higher in males, and a great majority of the subjects in the current study were boys. Thus, an additional sensitivity analysis was conducted to test whether the study findings would be replicated in boys only, and the results of this subgroup analysis were similar to the findings of the analysis of the entire study sample, thus confirming the reliability of the study.

This study has a few limitations. First, the study subjects were patients who visited a particular healthcare institution (i.e., Seoul National University Children’s Hospital), hence, the study findings cannot be directly applied to another clinical setting or to the general population. Second, the clinical diagnosis was not included in the analysis. We determined that the clinical diagnosis recorded in the medical chart may not be highly reliable, considering that the parents of the patients who visit a hospital with language delay as a chief complaint generally have strong resistance toward accepting the diagnosis of autism, and that sometimes insufficient time is allowed for a thorough examination depending on the outpatient clinical situation. Therefore, the clinical diagnosis in the medical chart is often produced by combination with the outcomes of tests such as the CARS and ADOS, which led us to determine that the clinical diagnosis may not be highly valuable as an independent datum. In addition, the diagnosis based on the ICD-10 provides information regarding the presence or absence of the disorder, but not information regarding the severity. Accordingly, there is a limitation in comparing the clinical diagnosis to the CARS and ADOS scores (i.e., the data primarily based on continuous variables), which is another reason why the clinical diagnosis was excluded from the analysis in the present study. To compare the CARS and ADOS outcomes to the clinical diagnosis, a separate study should be designed in the future to address this specific objective. Third, in the present study, the ADOS based on the DSM-IV was applied, but a recent study found that when patients who were diagnosed with ASD on the ADOS were diagnosed again using the DSM-5, 62% did not meet the diagnostic criteria [21]. Therefore, in the current situation where the DSM-5 is already being used, it cannot be assumed that the ADOS is an ideal test for diagnosing autism. Future research should be conducted using the ADOS-2, which is based on the DSM-5. Fourth, this study was a preliminary study conducted in outpatients who visited our clinic over a period of 2 years only; therefore, the sample size was small. Future studies should include markedly larger samples. Specifically, the number of children classified as non-autism on the CARS when the diagnostic cut-off score was lowered to 24.5 was only five. This can be considered to reflect both the small sample size and the limitation of the sample characteristics of the patients visiting a medical institution.

CONCLUSION

Two CARS diagnostic cut-off scores were proposed in the present study. One was the existing cut-off score of 30, and the diagnosis based on this cut-off exhibited a high level of agreement with the diagnosis of autism on the ADOS. The other cut-off score was 24.5, and the diagnosis based on this cut-off score exhibited a high level of agreement with the diagnosis of autism spectrum on the ADOS. Although the CARS may not replace the ADOS, in a situation where it is difficult to conduct the ADOS, the two CARS diagnostic cut-off scores may be utilized; for the purpose of diagnosing autism, patients with CARS scores of ≥30 are likely to be also diagnosed with autism on the ADOS, and it can be anticipated that they should be treated. For the purpose of screening of children who are suspected to exhibit autistic symptoms, an efficient approach may be to first conduct the easy-to-administer CARS, and to then conduct the ADOS as an additional measure if the CARS score is ≥24.5.

Supplementary Materials

The online-only Data Supplement is available with this article at https://doi.org/10.5765/jkacap.180015.