Measurement of autism: a review of four screening measures.

Using a key word search on electronic databases, two relevant journals and relevant review references, four measures of autistic symptoms (Childhood Asperger Syndrome Test, Autism Behavior Checklist, Social Communication Questionnaire, Social Responsiveness Scale) were reviewed with reference to their ability to discriminate the three major components of autism and measure across the whole spectrum of autism. None of the reviewed measures were able to effectively tap into and differentiate between all points on the spectrum. Further work is required to assess the sensitivity of the measures to independence between domains. The development of a measure sensitive to symptom change/continuity across development may be useful.

INTRODUCTION

The concept of autism has undergone transition since Kanner first delineated it in 1943. The boundaries of the disorder have widened with time, and it is increasingly conceptualized as a spectrum of related diagnostic categories as opposed to an all-or-none disorder.[1] Consequently, it is now believed that what is transmitted genetically is not classic autism but a more varied phenotype of social, communication and/or behavioral difficulties.[2] To screen for these difficulties there is a need for a valid and reliable measure that has a broader conceptualization of the spectrum and its domains. Prior to the development of such a measure, however, it is important that existing screening measures, which have played facilitatory roles in this transition of autism are reviewed; and the requirement, or alternatively the non-requirement, for developing further measures that tap into and discriminate between the entire range of the spectrum and its domains be identified. Hence this review aims to provide a synthesis of published research on four measures of autistic symptoms, with emphasis on:

Ability to discriminate the three major components of autism

Ability to measure across the whole spectrum of autism

MATERIALS AND METHODS

Although the lines delineating existing measures are vague, the following criteria were adhered to in selecting those for review:

Inclusion criteria

The review was restricted to journal-published validation studies for rating-scales/checklists/questionnaires designed to measure autistic symptoms in clinical and educational settings. Only measurements completed by parents/teachers were selected.

Exclusion criteria

Since the focus was on measures targeting autism spectrum disorders (ASD), those not specifically designed for the spectrum and those specifically predesigned for a specific point (e.g. only asperger syndrome) on the spectrum were excluded as the objective was to identify tools with the ability to discriminate across the spectrum. Measures focusing purely on adults/adolescents/infants were also excluded as the measures should have the potential to measure continuity/change of symptoms over time. Interview and observation-based measures and measures targeting response to treatment were excluded on the basis that they are not efficient or suitable for screening purposes. A list of non-selected measures is presented in Table 1.

Table 1

Measures that did not meet the inclusion criteria of this review

Search method

A literature search was conducted incorporating electronic databases, a hand search, and reference lists.

Electronic databases

Initially, a search was conducted in the following databases:

PsycINFO

Ovid MEDLINE

Web of Knowledge

Cochrane Library

The key words used were:

Autism; Autistic disorders; Asperger Syndrome (AS)

Measurement; Questionnaire; Scale; Test; Assessment; Screening

Hand search

Acknowledging their focus on ASD the ‘Journal of Autism and Developmental Disorders’ and ‘Autism’ were hand searched (incorporating keywords used for the electronic search) from the year 2000 to date.

Reference lists

The reference lists of recent reviews and articles identified in the electronic/hand search were examined.

RESULTS

The measures studied within the context of this review are:

Social responsiveness scale/Social reciprocity scale

The social responsiveness scale (SRS)[3] is a questionnaire for teachers/parents measuring children's social impairments in naturalistic social settings. The SRS ascertains social awareness, social information processing, capacity for reciprocal social communication, social anxiety/avoidance, and autistic preoccupations and traits.

Social communication questionnaire/Autism screening questionnaire

The social communication questionnaire (SCQ)[4] was developed in response to the need for a reliable and valid screening instrument based on current diagnostic criteria for autism diagnostic interview – revised (ADI-R).

Autism behavior checklist

The autism behavior checklist (ABC)[5] was developed to establish a scale which could be reliably used in schools to measure levels of autistic behavior.

Childhood asperger syndrome test

The childhood asperger syndrome test (CAST) is a parental questionnaire to screen school-age populations for ASD. Although by name it targets the asperger syndrome (AS) by scope it targets the full spectrum, including typical autism and pervasive developmental disorder-not otherwise specified (PDD-NOS);[6] and thus falls within the boundaries of this review.

Table 2 provides a description of the measures reviewed.

Table 2

Description of measures reviewed

DISCUSSION

The results of the reviewed studies will be discussed with focus on the abilities of the four measures to encompass the entire spectrum of autistic disorders and its components.

Ability to discriminate the three major components of autism

Traditionally, autism has been characterized by severe difficulties in three domains; reciprocal social interaction, communication and repetitive stereotyped behavior. Yet, are impairments in all three domains as intrinsic as once believed? Data suggests differently. In fact the genetic influences on individual domains seem independent of one another.[7] Therefore, it is important that each item on a scale/questionnaire should relate to one domain only, allowing the quantification of domain independence.

Although the SRS consists of items individually relating to no more than one domain, it is also based on the belief that difficulties in all three domains are attributable to a singular underlying deficit in reciprocal social behavior (RSB). Hence, whilst the other domains are included independently it is emphasizing their impact on RSB.[3] This focus on RSB almost to a point of exclusion of other domains confounds the SRS's ability to effectively explore domain independence and identify empirically if RSB is in reality the core deficit. The SCQ was split into four factors: Social interaction, communication, abnormal language and stereotyped behavior, together encompassing the three domains of autism.[4] Individual questionnaire items are linked to one of the three domains. The ABC items were grouped into sensory, relating, body and object use, language, and social and self-help.[5] However, subsequent work grouped ABC items as non-responsive, infant-like, aggressive and stereotypical behaviors, and echolalic speech.[8] The latter study was more empirically based; hence the factors arguably more objective. Nevertheless, the relationship between these factors and the domains remains unclear. For the CAST no factor analytic work was available. Such work may be useful, in view of the CAST's promise as a screening questionnaire in high-risk populations. Information on domain independence of items is necessary for both the CAST and ABC. The SCQ and ABC should also be administered on community samples and the scores on the different factors/domains examined with regard to their independence/interdependence. By definition and diagnosis a clinical autistic sample will include individuals with difficulties in all three domains. Therefore, to explore independence of domains community samples should be studied. Furthermore, providing separate scores and cutoff points per domain may be useful. It may also be useful to determine if different ASD differ with regard to factor profiles. Such information may lead to clearly defined diagnostic criteria for PDD-NOS.

Ability to measure across the whole spectrum of autism

Recent studies have suggested that autistic symptoms are continuously distributed in the population forming a broad spectrum.[9] Hence, the disorder may take many forms from mild-moderate-severe with a variety of impairments. This increasing heterogeneity of the disorder can cause difficulty in measurement.[10] Measures should be able to quantify subtle differences in the degree of impairment across the spectrum and alert clinicians to the severity of symptoms, including those that may be sub-threshold. Such measures may also facilitate a clearer concept of PDD-NOS. With reference to the measures reviewed, although they purported to target the full spectrum, it may be that they only ‘claim’ to do so. Therefore, an important question to focus upon is, ‘Are the reviewed measures able to dimensionally tap into and contrast between all points on this continuous spectrum?’

The SRS, which conceptualizes RSB as a continuous variable, will probably be sensitive to any degree of RSB impairment. In fact, it has been observed that SRS scores were continuously distributed in clinical and school samples,[9] supporting the idea of autism as a spectrum condition rather than an all-or-none diagnosis. Nevertheless, it is prudent to be aware that this may merely imply that deficiencies in RSB are continuously distributed, as the SRS cannot justifiably be called a scale that measures the severity of deficiency in all three domains of impairment. Further, it has been observed that the SRS did not clearly distinguish individuals diagnosed with autism from those with PDD-NOS, especially amongst higher functioning autistic cases.[3] The SCQ and ABC are more categorical measures, in that they are generally designed to assign ‘caseness’ as opposed to severity. Hence, they may not tap into the milder cases on the spectrum. For instance, it has been observed that the SCQ may not be suitable for identifying less clear-cut ASD cases.[11] Similarly, the ABC may not reliably screen for high-functioning autistic individuals.[10] The CAST, like the SRS assumes that social and communication difficulties fall on a continuous distribution; and is based on a dimensional conceptualization of the spectrum. The advantage of this dimensionality is that arbitrary cutoff points can be imposed upon the continuum to delineate categories of behavior qualitatively different from ‘normalcy’.[6] However, while the CAST may be sensitive to the less extremely impaired on the spectrum, it also shows a high proportion of false positives. Hence, it may also be tapping onto difficulties outside the clinical spectrum.

Regardless of their ability to tap into the entire spectrum of autism and provide independent ratings per domain, these measures would be redundant with weak psychometric properties. Hence, the reliability and validity data on the measures are briefly overviewed.

Reliability

Test-retest reliability

As presented in Table 3, long-term measurement stability is indicated for the SRS and the German version of the SCQ, albeit in a small sample of 17 subjects[12] No test-retest data was available for the ABC but given the nature of the scale, such data may be redundant.[13] The CAST seems to have moderately good temporal validity.[6]

Table 3

Reliability of measures reviewed

Internal consistency

Internal consistency for the SRS in data from a school sample was high. Alpha reliability coefficients for the SCQ were also high for both the English and German versions. Split-half reliability data for the ABC were discrepant when two separate studies were compared.[510] This may be because the original study used the Spearman-Brown prophesy formula.[5] The overall classification rate on the ABC was virtually unchanged when only the more highly weighted items on the scale were utilized, suggesting that it could be shortened.[14]

Interrater reliability

High interrater reliability has been obtained for the SRS,[3] implying universality in the expression of symptoms, and the utility of the measure across multiple informants. ABC data seems discrepant, with recent studies observing lower rates than the original validation data. However, the original study did not incorporate corrections for chance agreement. Further, subsequent studies[1015] modified the presentation of the scale to blind raters to its aim. Overall, the reliability of the ABC seems unclear and for further reliability data [Table 3].

Validity

Most of the validation studies focused on the discriminative ability of the measures. Further, the measures were validated against established diagnostic tools for autism to identify construct-validity. The SRS is capable of distinguishing children with ASD from those with other child psychiatric disorders, including mental retardation.[3] Between autism and the other disorders within the spectrum, although significant differentiation was observed, it was less clear-cut and there was substantial overlap. The SRS compares favorably with the ADI-R.[3] Overall, the SRS seems to be a valid instrument for the measurement of clinically significant autistic traits. Understandably, the discriminatory value of the SCQ was significantly correlated with the ADI upon which it was based.[4] Further, the SCQ showed statistically significant differentiation between ASD and other disorders. Interestingly, a relatively weak diagnostic differentiation was provided by the factor representing repetitive stereotyped behaviors.[4] Thus a challenge for researchers is to identify an effective means of measuring this domain. There is also some questionability regarding the SCQ's ability to differentiate autism from mental retardation.[4] In addition, the SCQ may not be able to differentiate between different ‘points’ on the autism spectrum, and may not be an effective screening tool in a non-clinical population.[11] Results from studies on the discriminative validity of the ABC are wide-ranging. Data indicates that total scores in autistics are consistently higher than in severely mentally retarded, emotionally disturbed, deaf-blind, learning impaired or normal.[51416] However, when the more current, broader criteria was used to define autism, a high negative predictive value was obtained[14]; suggesting a need to validate a new cutoff score for the ABC defined by current diagnostic criteria. Similarly, the ABC has been found to be less discriminative with a high number of false-negatives, particularly in mild-moderate autism.[17] With high-functioning autism, when retrospective ratings were used, the ABC was sensitive to its diagnosis.[18] A pilot studydemonstrated that the CAST discriminates well between children with AS/autism and normally developing children.[11] However, there is a lack of clarity as to whether the discrimination was between those with autism/AS and typical children, or simply between AS and typical children.[19] A subsequent pilot study on mainstream school samples showed that the CAST has good specificity (.98) on a cutoff point of 15. However, positive predictive value was poor (.64). In addition, the sample studied may have been biased, as there was a poor response rate (17%). Hence, researchers were unable to reliably calculate the sensitivity of the scale.[11] In a subsequent study the CAST demonstrated good accuracy for use as a screening test in a high-risk population, with high sensitivity and specificity.[6] However, once again, it had a low positive predictive value, leading to a high number of false-positives, which could have detrimental implications and for further validity data [Table 4].

Table 4

Validity of measures reviewed

CONCLUSIONS AND FUTURE DIRECTIONS

All the measures reviewed within the limits of this analysis are relatively brief [Table 2], and efficient for use in busy clinical and educational settings. Responses are obtained firsthand from informants, allowing greater diagnostic input from primary care-providers who have direct observation of the child's behavior.

None of the measures reviewed seem potentially useful in differentiating between disorders within the autistic spectrum. The usefulness of the SCQ and ABC may be limited to screening for the more clinically defined ends of the spectrum. In contrast, the CAST and SRS may be more effective to identify individuals with difficulties which may/may not be indicative of autism. More information regarding the specificity, sensitivity and positive and negative predictive values of the measurements at varying cutoff points (possibly providing within-spectrum discrimination) will be useful.

It may also be useful to study the distribution of scores in a population, especially in a non-clinical one. The SRS provides limited data in this regard. Further work will enhance current understanding on continuities or discontinuities within the spectrum. An important question requiring further exploration is: ‘Are the differences within this spectrum qualitative or merely quantitative?’.

If autism is indeed a spectrum of continuity, developing dimensional and continuous measures will be advantageous, especially with regard to genetic causation studies. However, if these dimensions measure the severity of symptoms, how should severity be defined? Is it merely the number of symptoms? Is not continuity of symptoms over time an important aspect of severity? Hence, a tool that can measure symptom change/continuity over development would be useful. From the scales reviewed, the SCQ emphasizes qualitative deviance rather than developmental delay/impairment, focusing on lifetime manifestations. The ABC takes into account the ‘age factor’ and provides separate profile charts for different chronological ages. However, none of the measures focus on symptom change/continuity over the life course.

With regard to this review, there is a possibility of a publication bias, especially as examiners’ manuals/catalogues were not reviewed in detail. Further, measures pre-focusing on a defined point on the autistic spectrum were excluded. However, there have been reviews on measures targeting such groups.[19]

In conclusion, therefore, none of the reviewed measures were able to singularly tap into and differentiate between all points of the autistic spectrum. With regard to their sensitivity to measuring the three domains of autism as possibly existing independently from one another, further research is necessary.