Autistic traits and autism spectrum disorders: the clinical validity of two measures presuming a continuum of social communication skills.

Research indicates that autism is the extreme end of a continuously distributed trait. The Social Responsiveness Scale (SRS) and the Social and Communication Disorders Checklist (SCDC) aim to assess autistic traits. The objective of this study was to compare their clinical validity. The SRS showed sensitivities of .74 to .80 and specificities of .69 to 1.00 for autism. Sensitivities were .85 to .90 and specificities .28 to.82 for the SCDC. Correlations with the ADI-R, ADOS and SCQ were higher for the SRS than for the SCDC. The SCDC seems superior to the SRS to screen for unspecific social and communicative deficits including autism. The SRS appears more suitable than the SCDC in clinical settings and for specific autism screening.

Introduction

Autism, Asperger’s syndrome, and atypical autism/PDD-NOS are behaviourally defined by compositions of impairments in three domains: reciprocal social interaction, mutual verbal and nonverbal communication alongside with inflexible behaviour patterns, interests and activities (DSM-IV-TR/ICD-10). The diagnoses are clinically increasingly summarized under the label of autism spectrum disorders (ASD). Indeed, a multitude of research shows that the diagnoses are hard to differentiate and probably form one clinical spectrum of behaviour problems, not circumscribed entities (Macintosh and Dissanayake 2004; Witwer and Lecavalier 2008). DSM-V will probably dissolve the single disorders in favour of the autism spectrum concept, indicating differing grades of severity of one diagnosis (Lord 2009).

Studies on the phenotype in first degree relatives of people with autism have shown that a substantial part of parents and siblings tend to show normative (subclinical) variants of full-blown ASD, called the broader or extended autism phenotype, perhaps reflecting a vulnerability for the clinical picture of ASD (Dawson et al. 2002). Thus, autism might not be solely a clinical phenomenon. Recent epidemiological research indicates that ASD rather present the extreme of a trait (Spiker et al. 2002; Constantino and Todd 2003; Skuse et al. 2009). Traits reflect habitual patterns of behavior, thought, and emotion, which are stable over time and exist in all individuals to a varying degree; often normal distribution of these features is postulated. As opposed to disorders, traits are viewed to be quantitatively (dimensional), not qualitatively distributed in the general population.

Some disorders, particularly learning disabilities and mental retardation are already dimensionally defined to a large degree in DSM-IV-TR/ICD-10. Grounding a disorder on either categorical or dimensional views may appear insignificant in the first place, as a line is drawn somewhat arbitrarily between normality and abnormality in both cases and there is always an additional clinical (categorical) decision to be made, whether a persons needs care or not. Nevertheless, a paradigm shift from categorical to dimensional in ASD would have major scientific and clinical impact on the exploration of its genetic and neurobiological roots, on intervention studies, statistical power and the clarification of the phenomenon to patients and their parents (Constantino 2009). There would also be a certain move of autism from psychiatry and clinical psychology to differential and personal psychology, because ASD assessment would preferably be done by using psychometric tools designed for and standardized in the general population. It is therefore of pivotal importance for research and clinical purposes to carefully evaluate the clinical validity of diagnostic instruments claiming to be appropriate in assessing autism as a trait.

Aside from the Autism-Spectrum Quotient (Baron-Cohen et al. 2006; Auyeung et al. 2008) and the Childhood Autism Spectrum Test (Williams et al. 2008), the Social Responsiveness Scale (SRS; Constantino and Gruber 2005) and the Social and Communication Disorders Checklist (SCDC; Skuse et al. 2005) explicitly aim to measure autistic traits in children and adolescents. They have both demonstrated good psychometric properties, continuously distributed scores in large general population samples, single factor solutions and high heritability of the features assessed (Constantino and Todd 2003; Constantino et al. 2004; Scourfield et al. 1999; Skuse et al. 2005; Skuse et al. 2009). The SRS has also demonstrated cross-cultural validity (Bölte et al. 2008b).

The SRS is a 65 item (Likert scale 0 to 3) parent/teacher report form, and generates a singular scale with a maximum score of 195 for behaviour shown in last six months. In the original US American standardization, the mean raw score for parent reported autistic traits in 1,081 typically developed school children aged 4–18 years from Missouri and California was 33.7 (SD = 20.9) in boys and 27.6 (SD = 18.1) in girls. The expected score in ASD lies between 101.5 (SD = 23.6) for PDD-NOS and 117.1 (SD = 22.2) for core autism. Other mental disorders have shown to yield intermediate scores, e.g. attention-deficit hyperactivity disorder (ADHD) (51.1, SD = 32.9) (Constantino and Gruber 2005). Correlations with gold standard clinical ASD instruments are consistent with expectations, e.g. r = .52 to .74 with subscales of the Autism Diagnostic Interview-Revised (ADI-R). Receiver operating characteristics (ROC) analyses in 133 cases of ASD vs. 126 non-ASD cases (ADHD, unspecific developmental disorder, other child psychiatric diagnoses) showed that a SRS-total score of 75 had a sensitivity of .85 and a specificity of .75 for any ASD, while a total score of .85 had a sensitivity of .70 and a specificity of .90.

The SCDC is a time and cost effective parent questionnaire including 12 items (Likert scale 0 to 2) with a maximum score of 24 for current behaviour. The mean value in 118 typically developed children and adolescents from mainstream schools in the UK was found to be 3.25 (SD = 4.15) for boys and 2.39 (SD = 3.14) for girls. In ASD, reported values were 16.6 (SD = 5.7) in 208 clinically diagnosed autism and atypical autism and 13.0 (SD = 6.1) in 76 clinically diagnosed conduct disorder, ADHD, pragmatic language disorder, Tourette’s, and obsessive–compulsive disorder (Skuse et al. 2005). The SCDC has shown correlations with ADI-R algorithm output equivalents (generated by the 3di interview; Skuse et al. 2004) in the order of r = .21 to .41. ROC analyses in the initial standardization study showed that a SCDC-total score of 9 reached a sensitivity of .90 and a specificity of .69 for ASD. A recent study by Skuse et al. (2009) in a cohort of 8,094 eight year olds including 31 register-based educational service diagnoses of ASD, 70 specific learning disability, 92 moderate learning disability, 15 severe learning disability, 2 profound and multiple learning, 57 emotional and behavioural maladjustment, 25 speech, language and communication needs, and 22 physical and sensory disabilities reported a sensitivity of .88 and a specificity of .91 for a SCDC-total score of 8. The latter is in the area of the discriminative power identified for the revised algorithms of the Autism Diagnostic Observation Schedule (ADOS) (Gotham et al. 2008).

Overall, these results convincingly endorse the clinical validity of the SRS and SCDC for usage in research and clinical practice of ASD. Nevertheless, the SRS and SCDC were standardized using quite different samples and procedures, so their clinical validity in an identical sample is unknown. In addition, the clinical samples used to establish their diagnostic validity for ASD were somewhat limited in terms of phenotypic characterization, clinical heterogeneity and syndrome severity. The objective of the current study was therefore to compare the convergent and diagnostic validity of the SRS and SCDC for ASD using exactly the same methods in a single well characterized child and adolescent psychiatric sample (CLIN), and typically developing control mainstream school children (TYP).

Method

Participants

The total sample consisted of N = 480 participants: n = 148 with idiopathic ASD, n = 255 CLIN and n = 77 TYP. ASD and CLIN had been collected in the years 2005 to 2008 within the clinical routine at the Department of Child and Adolescent Psychiatry, Frankfurt/M. University Hospital, and within a long-term research project on the genetic fundamentals of autism (well.ox.ac.uk/monaco/autism/IMGSAC.html). The ASD sample included 98 individuals with autism, 34 with atypical autism/PDD-NOS and 16 with Asperger’s syndrome. For inference statistics, autism, Asperger syndrome and atypical autism/PDD-NOS participants were pooled, as, owing to a rich body of evidence, the spectrum concept will replace the aforementioned single diagnosis in the upcoming DSM-V (see www.dsm5.org, for details). There were 121 males and 27 females aged 4 to 18 years (M = 11.2, SD = 4.1) with an average IQ of 88.95 (SD = 25.5).

In CLIN there were 98 cases of ADHD, 41 with anxiety disorders, 37 with combined ADHD and CD, 28 CD, and 41 had other diagnoses (mental retardation without ASD, affective disorder, tics, learning disability, eating disorder, psychosis, mutism, attachment disorder, enuresis, pica). There were 185 males and 70 females in CLIN aged 4 to 18 years (M = 9.9, SD = 3.7), with an average IQ of 98.5 (SD = 15.9).

Clinical diagnoses were based on ICD-10 research criteria. They were consensus diagnoses of the author’s department’s experienced clinicians, corroborated by findings from standardized diagnostic scales, such as the ADI-R and ADOS for ASD or the Kiddie-Schedule for Affective Disorders and Schizophrenia (Kaufman et al. 1997) for other clinical diagnoses. A diagnosis of Asperger syndrome was given, if the ADI-R autism algorithm cut-off was met, but no history of language delay was apparent on the items first words/first phrases, and ADOS autism spectrum cut-off was met. A diagnosis of atypical autism/PDD-NOS was assigned, if two domain cut-offs were met in the ADI-R (one of which was the social interaction domain) and ADOS autism spectrum cut-off was met.

The TYP were recruited from regular local preschools, primary and secondary schools as well as from authors’ private personal contacts, and had no history or current indication of psychiatric or school problems according to parent information. TYP consisted of 41 males and 36 females with a mean age of 9.3 years (SD = 3.2). Sample characteristics are summarized in Table 1.

Table 1

Sample characteristics

	Autism spectrum disorders (ASD)	Clinical controls (CLIN)	Typical controls (TYP)
N	148	255	77
Age, years: M (SD)	11.2 (4.1)	9.9 (3.7)	9.3 (3.2)
Sex: male/female	121/27	185/70	41/36
IQ: M (SD)	88.95 (25.5)	98.5 (15.9)
SRS scorea: M (SD)	101.1 (31.8)	62.7 (30.9)	26.0 (13.6) (boys)
	106.7 (33.9)	62.9 (27.3)	20.9 (9.6) (girls)
SCDC score: M (SD)	15.6 (5.8)	12.6 (6.7)	5.9 (4.6) (boys)
	15.3 (4.1)	11.4 (6.7)	5.1 (3.0) (girls)

aASD > CLIN > TYP (p < .0001), girls = boys (p > .26)

Questionnaires and Procedure

The German adaptations of the SRS and SCDC were compared in this study. Consistent with the English original, the German version of the SRS (Bölte and Poustka 2008) has demonstrated a single factor structure and good to excellent psychometric properties (Bölte et al. 2008b). However, mean SRS total scores for typically developed girls and boys as well as values for ASD were lower in the German sample, while scores for conduct disorder (CD) and ADHD/CD combined were higher. The SCDC has not been standardized in a German population before. The SCDC was translated into German, independently retranslated into English, adjusted by a native English speaking clinician and tested in practice for understandability before using it in this study.

In order to establish the convergent validity of the SRS and SCDC, the German adaptations of gold standard ASD instruments, namely the ADI-R and the ADOS, as well as the Social Communication Questionnaire (SCQ) were also assessed in the ASD sample. Comparable to the originals, all have demonstrated good to excellent psychometric properties (Poustka et al. 1996; Bölte and Poustka 2004; Bölte et al. 2006, 2008a). The SRS, SCDC and SCQ were either send out for completion by parents as a part of pre-appointment assessment or completed during a clinical visit of their child. They were given one after the other in random order. In most cases, completion of the SRS, SCDC and SCQ preceded the ADI-R and ADOS or other in depths clinical examinations. In CLIN the SRS and SCDC were collected as part of the diagnostic routine at admission. SRS and SCDC data in TYP were obtained at personal contacts of the authors with the parents or by teachers handing-out and re-collecting the questionnaires. IQ was assessed in the ASD and CLIN participants using either Wechsler intelligence batteries or the Raven’s matrices, which mostly result in qualitatively comparable scores in higher functioning ASD (IQ > 85) and CLIN (Bölte et al. 2009).

Data Analysis

SPSS/Win. 16.0 was used for data analysis. Partial correlation (controlled for IQ and age) was computed in ASD to determine convergent validity (correlation with measures aiming to assess similar constructs) of the SRS and the SCDC with the algorithm domain scores of the ADI-R (deficits in social interaction, deficits in communication, stereotypic behaviors; reflecting behavior at age 4 to 5 years or lifetime) and ADOS (deficits in social interaction, deficits in communication; reflecting current behavior) as well as the total score of the SCQ (reflecting behavior at age 4 to 5 years and lifetime). Here, to avoid underestimation of true associations, correlations were corrected for direct range restrictions applying the case II formula by Thorndike (1949), because ADI-R, ADOS and SCQ are categorical scales, not intended to generate variability. The formula uses (a) the correlation of the restricted sample, (b) the standard deviation of the independent variable in the restricted sample and (c) in the unrestricted sample to provide an estimate of the correlation in the population.

Diagnostic validity (value for diagnostic classification), was analyzed by ROC-analyses for ASD vs. TYP and ASD vs. CLIN, as well as for ASD vs. each of the largest CLIN subgroups, ADHD, and anxiety disorders. Area under the curve (AUC) with p-values, sensitivities and specificities for recommended SRS and SCDC cut-offs were calculated. Group and sex differences on the SRS and SCDC were examined using ANCOVAs and post-hoc Tukey tests. Internal consistencies (Cronbach’s Alpha) for the SRS and SCDC are were computed within ASD, CLIN, and TYP.

Results

SRS and SCDC scores are summarized in Table 1. The correlation between SRS and SCDC scores was r = .49 (p < .01). Correlations of the SRS and the SCDC with age in ASD, CLIN and TYP (r = −.01 to .08) and IQ in ASD and CLIN (r = −.10 to -.21) were low (n.s). Cronbach’s Alpha for the SRS was .96 in ASD, .94 in CLIN and .91 in TYP. For the SCDC alphas were .78 in ASD, .91 in CLIN and .80 in TYP.

In ASD, mean SRS score was 102.2 (SD = 32.1) [girls: 106.7 (SD = 33.9); boys: 101.1 (SD = 31.8)] and mean SCDC score was 15.6 (SD = 5.6) [girls: 15.3 (SD = 4.1); boys: 15.6 (SD = 5.8)]. Mean SRS scores were 105.4 (SD = 30.1) in autism, 100.1 (SD = 32.0) in Asperger’s syndrome and 88.5 (SD = 35.1) in atypical autism/PDD-NOS. Mean SCDC scores were 15.9 (SD = 4.9) in autism, 15.5 (SD = 5.2) in Asperger’s syndrome and 14.9 (SD = 6.0) in atypical autism/PDD-NOS.

In CLIN, mean SRS score was 62.8 (SD = 29.9) [girls: 62.9 (SD = 27.3); boys: 62.7 (SD = 30.9)] and mean SCDC score was 12.2 (SD = 6.7) [girls: 11.4 (SD = 6.7); boys: 12.6 (SD = 6.7)]. Within CLIN, mean score for ADHD was 54.7 (SD = 27.8) on the SRS and 12.1 (SD = 6.6) on the SCDC. Mean score for anxiety disorders was 61.5 (SD = 26.4) on the SRS and 10.5 (SD = 7.3) on the SCDC.

Mean score for TYP on the SRS was 26.0 (SD = 13.6) for boys and 20.9 (SD = 9.6) for girls. For the SCDC, figures in TYP were 5.9 (SD = 4.6) for boys and 5.1 (SD = 3.0) for girls.

The ROC analysis for the SRS yielded an AUC of .98 (p < .0001) for ASD vs. TYP (Fig. 1). A total SRS score of 75 had a sensitivity of .80 and a specificity of 1.0 for ASD, a score of 85 had a sensitivity of .74 and a specificity of 1.0 for ASD. ROC analysis for ASD vs. CLIN yielded an AUC of .81 (p < .0001) (Fig. 2). A total SRS score of 75 had a sensitivity of .80 and a specificity of .69 for ASD, a score of 85 had a sensitivity of .74 and a specificity of .79 for ASD. ROC analysis for ASD vs. ADHD showed an AUC of .86 (p < .0001), with sensitivities of .80/.74 and specificities of .78/.83 for ASD using SRS cut-offs. ROC analysis for ASD vs. anxiety disorders showed an AUC of .82 (p < .0001), with sensitivities of .81/.65 and specificities of .74/.80 ASD using the recommended SRS cut-offs.

Fig. 1

Receiver Operating Characteristics (ROC) curve of the Social Responsiveness Scale (SRS) for the condition autism spectrum disorder (n = 148) versus typical development (n = 77); area under the curve = .98 (p < .0001)

Fig. 2

Receiver Operating Characteristics (ROC) curve of the Social Responsiveness Scale (SRS) for the condition autism spectrum disorder (n = 148) versus clinical controls (n = 255); area under the curve = .81 (p < .0001)

The ROC analysis for the SCDC yielded an AUC of .93 (p < .0001) for ASD vs. TYP (Fig. 3). A total SCDC score of 8 had a sensitivity of .90 and a specificity of .75 for ASD, a score of 9 had a sensitivity of .87 and a specificity of .82 for ASD. ROC analysis for ASD vs. CLIN yielded an AUC of .64 (p < .0001) (Fig. 4). A total SCDC score of 8 had a sensitivity of .89 and a specificity of .28 for ASD, a score of 9 had a sensitivity of .85 and a specificity of .39 for ASD. ROC analysis for ASD vs. ADHD showed an AUC of .65 (p < .0001), with sensitivities of .90/.85 and specificities of .29/.43 for ASD using SCDC cut-offs. ROC analysis for ASD vs. anxiety disorders showed an AUC of .68 (p < .0001), with sensitivities of .90/.87 and specificities of .34/.44 for ASD using the recommended SCDC cut-offs.

Fig. 3

Receiver Operating Characteristics (ROC) curve of the Social and Communication Disorders Checklist (SCDC) for the condition autism spectrum disorder (n = 148) versus typical development (n = 77); area under the curve = .93 (p < .0001)

Fig. 4

Receiver Operating Characteristics (ROC) curve of the Social and Communication Disorders Checklist (SCDC) for the condition autism spectrum disorder (n = 148) versus clinical controls (n = 255); area under the curve = .64 (p < .0001)

SRS and SCDC total scores differed highly significant between groups (F 2/469 > 54.9, p < .0001, eta 2 > .19), with ASD scoring higher than CLIN, and CLIN scoring higher than TYP (p < .0001) in both scales. There were no effects of sex (F 1/469 < 1.2, p > .26, eta 2 < .0001) or an interaction of sex by group (F 2/469 < .79, p > .45, eta 2 < .003).

Convergent validity findings for ASD are shown in Table 2. The SRS total score correlated r = .45 with the ADI-R social domain, r = .39 with the ADI-R communication domain and r = .31 with the ADI-R domain for stereotypies (p < .01). The same correlations for the SCDC were r = .35, r = .23 and r = .24 (p < .05). Correlations of the SRS with the ADOS social domain were r = .35 and for the communication domain r = .32 (p < .01). The same correlations for the SCDC were r = .18 and r = .17 (n.s.). Convergent validity with the SCQ was r = .50 for the SRS and r = .36 for the SCDC (p < .01).

Table 2

Convergent validities of the Social Responsiveness Scale (SRS) and the Social and Communication Disorders Checklist (SCDC) in the autism spectrum disorders sample

	SRS	SCDC
ADI-R
Social domain	.45a	.35a
Communication domain	.39a	.23b
Stereotypies domain	.31a	.24b
ADOS
Social domain	.35a	.18c
Communication domain	.32a	.17c
SCQ
Total score	.50a	.36a

a p < .01, b  p < .05, c n.s.

Discussion

This study compared the diagnostic validity for ASD of two questionnaires aiming at assessing autistic traits: the SRS and SCDC. In the same sample, both scales yielded moderate to good sensitivities to screen for ASD, with sensitivity values being higher for the SCDC than for the SRS for both typical and clinical controls. SRS specificities for ASD were higher than its sensitivities and considerably higher than SCDC specificities. Therefore, it appears that the SCDC generates somewhat less ASD false negatives than the SRS, while the SRS generates far less ASD false positives. The SCDC might therefore be viewed superior to the SRS to generate suspicion of ASD. However, it also frequently generates suspicion of ASD in non-ASD individuals. Thus, the SCDC seems indeed sensitive to clinically relevant social and communication problems, but less specific to social and communication problems being characteristic of ASD. The SRS appears more balanced regarding its sensitivity and specificity for social and communication deficits typical for ASD. The latter is endorsed by higher convergent validities of the SRS than the SCDC with gold standard clinical ASD instruments, namely the ADI-R, ADOS and SCQ, although correlations of the SRS with the ADI-R were lower than reported previously (Constantino et al. 2004).

While sensitivities and specificities found here for the SRS, as well as sensitivities for the SCDC, are consistent with previous reports (Constantino and Todd 2003; Constantino et al. 2004; Bölte et al. 2008a; Skuse et al. 2005; Skuse et al. 2009), the specificities identified in this study for SCDC are substantially lower than reported before. The reason might be that compared to earlier research the current study included a more heterogeneous and possibly more severely affected child and adolescent psychiatric clinical control sample. This might have resulted in higher SCDC scores even in non-ASD clinical controls, and in lower discriminative power of the SCDC to differentiate these subjects from autistic individuals. The SCDC authors themselves found markedly lower SCDC specificity in a clinical study (.69) (Skuse et al. 2005) compared to a population-based study (.91) (Skuse et al. 2009). In the population-based study clinical controls had been identified on the basis of being provided with special education, whereas in the clinical study diagnoses where explicit expert classifications. In their clinical study, the control comprised N = 76 participants. The current study examined N = 255 psychiatric controls, in the majority exhibiting childhood psychopathologies known to be associated with high scores on social problem scales (ADHD, conduct disorder).

Overall, owing to purely psychometric prerequisites, our findings are not surprising. The SRS comprises 65 items scored on a 0 to 3 basis (max. score 195), while the SCDC only contains 12 items scored 0 to 2 (max. score 24). Therefore, the SRS’s a priori likelihood being able to adequately reflect an empirical relative in a numerical relative and hence detect individual differences is considerably higher than for the SCDC. It is also technically immanent that there is a higher probability for better indices of reliability and validity, due to the higher number of items (scale lengths). The SCDC’s brevity is a definite quality in terms of time and cost economy as well as expected compliance in parents and teachers to fill-out the questionnaire. However, there is not much space to clarify the psychological construct and test’s intention. Thus, there is a certain risk that the core sensitivity of the SCDC is for more general social-communicative psychopathology. In fact, Skuse et al. (2009) reported a robust correlation of the SCDC with the total score of the Strengths and Difficulties Questionnaire (SDQ), a scale to screen for general psychopathology. Additionally, the correlations of the SCDC with the SDQ scales for “Peer problems” and “Prosocial Behavior” were lower than for “Hyperactivity”. The latter is rather unexpected for a dimensional autism scale. Undoubtedly, comparable problems are also part of the SRS. Such problems arise from the fact that items for population-based scales often have to be worded as concise, simple and understandable as possible to fit all children and adolescents, but particularly typical developed children. The consequence is that they can be perceived in different ways and clinically might seem to fit very different disorders. Clinical ASD scales, such as the SCQ, allow a more specific and complex questioning, but they are of limited value to tap autism outside a clinical setting. Similar to the SCDC, the SRS has shown high correlations to general psychopathology as measured by the Child Behavior Checklist (CBCL). However, it has been demonstrated that the SRS also covers deficits in reciprocal social behavior and communication, which are substantially independent from other domains of child psychopathology, and highest SRS correlations with a CBCL-subscale have consistently emerged for “social problems” (Constantino et al. 2003; Bölte et al. 2009).

More evaluation is needed to determine the clinical validity of the SRS and SCDC. A limitation of the present study is that the TYP group is likely not to be representative of the general population, owing to the methods used to recruit this group. The current study would also have benefited from general psychopathology assessment and IQ testing in TYP. Future studies should also carefully collect data on functional behavior level, which is a pivotal parameter for the designation of any mental disorder.

In summary, owing to its brevity and good sensitivity the SCDC might be suited better than the SRS to screen for unspecific social and communicative deficits including autistic problems in the general population. The SRS shows better specificity and construct validity for ASD and seems therefore more valuable than the SCDC for specific ASD screening and in clinical settings. The latter is supported by the fact that the SRS also offers population-based norms and standard errors of measurement for individual diagnostic assessments and follow-ups. As research on the phenomenology, epidemiology, neurobiology and genetics of autism as a disorder and a trait continue, both the SRS and SCDC should be viewed as scales in progress, prone to revisions.