Validity of the Children's Social Behavior Questionnaire (CSBQ) in children with intellectual disability: comparing the CSBQ with ADI-R, ADOS, and clinical DSM-IV-TR classification.

The Children's Social Behavior Questionnaire (CSBQ) was compared with the Autism Diagnostic Interview-Revised (ADI-R), Autism Diagnostic Observation Schedule (ADOS), and clinical classification in children with mild and moderate intellectual disability (ID), to investigate its criterion related validity. The contribution of the CSBQ to a classification of Autism Spectrum Disorder (ASD) was most specific for the subscales 'contact' and 'stereotyped', with high coherence with all three classification methods. The CSBQ may be used as a signaling, screening, or describing instrument for children with ASD and ID, as it complements other methods by adding unique information about the clinical presentation.

Introduction

A clinical diagnosis of Pervasive Developmental Disorder (PDD; DSM-IV-TR, American Psychiatric Association 2000) or Autism Spectrum Disorder (ASD) requires a detailed developmental interview with the parents, and a direct examination of the child. Additionally, for screening and/or research purposes, specific parent-based questionnaires may be used to get a description of the number and type of currently present ASD problems (Constantino and Todd 2005). One such parent questionnaire is the Children’s Social Behavior Questionnaire (CSBQ; Hartman et al. 2006, 2008; Luteijn et al. 2000a, 2002).

The CSBQ has specifically been developed for assessing social behavior problems within the whole autism spectrum. Development of the CSBQ was based on the idea that existing instruments did not suffice for children with PDD-not otherwise specified (NOS), because items were directly extracted from the DSM-criteria for autistic disorder (AD). With that, the more subtle social problems of children with PDD-NOS are not represented in these instruments. The CSBQ aims to be sufficiently sensitive with respect to the milder end of the autism spectrum, based on the dimensional view of ASDs. This perspective has led to a pool of items representing both core symptoms of AD and more subtle symptoms associated with ASDs, thus trying to capture the behavioral variety from normality to a full diagnosis of AD. The psychometric qualities of the CSBQ were reported to be good (Hartman et al. 2006; Luteijn et al. 2000a) and the CSBQ has been shown to be valuable in measuring (subtreshold) autistic symptomatology in children with normal intelligence and with mental retardation (MR) or intellectual disability (ID; Hartman et al. 2006; de Bildt et al. 2005a; Luteijn et al. 2000a, b).

A revision of the original CSBQ resulted in more refined subscales with good psychometric properties (Hartman et al. 2006, 2008).The CSBQ has now 49 items, composing 6 subscales all with good internal consistency (α = .76–.94), inter-rater reliability (ICC = .75–.89), and test-retest reliability (r = .80–.90). Convergent, divergent, and criterion-related validity were also reported as good. So far, however, criterion related validity has only been investigated in relation to clinical A(S)D classifications, and not to standardized instruments for A(S)D such as the Autism Diagnostic Interview-Revised (ADI-R; Rutter et al. 2003) and Autism Diagnostic Observation Schedule (ADOS; Lord et al. 1999).

The current paper aims to further investigate the criterion-related validity of the CSBQ in children with mild and moderate ID. ID occurs very frequently in children with autism (estimates between 70 and 90%; DeMyer et al. 1974; Steffenburg and Gillberg 1986; Fombonne 2005), but also when the whole spectrum is taken into account, still one in every four children is reported to have ID (26% reported by Chakrabarti and Fombonne 2001). Not only does the group with such dual diagnosis differ from children with ASD with normal intelligence in behavior (e.g., more self-injurious behavior, stereotyped behavior, deviant social responses, delayed social development, fears, etc.; de Bildt et al. 2005b), yet also in outcome, and amount, duration and specificity of needed care (Kraijer 1997). The presentation of ASDs in children with ID, however, is not merely characterized by more severe social, communicative, or flexibility problems indicative of the presence of AD. Rather, the whole spectrum of ASDs is present amongst children with ID, and the differentiation between AD and PDD-NOS is even more complicated than in children with normal intelligence (Kraijer 1997). The validation of the CSBQ in specifically this population aims to add to identifying ASD in children with ID and to describing ASD related problem behavior currently present.

The study is conducted by comparing the CSBQ to the ADI-R and ADOS classifications, the current standards for instrumental classification of A(S)D, and to the clinical DSM-IV-TR-based ASD classifications.

Methods

Participants

The 136 participants in this study were part of a genetic study of ASDs in the North of the Netherlands. Participants were recruited through an epidemiological survey (see for the exact procedure de Bildt et al. 2005a, b) and through an Autism Outpatient Clinic (same procedure, see also Van Lang et al. 2006; Mulder et al. 2004). For the current study, participants were included when they were 4–18 years old and had mild or moderate ID, as defined by the DSM-IV: mild ID, IQ 51–70; moderate ID, IQ 36–50, based on standardized intelligence tests. These included the Wechsler Intelligence Scale for Children-Revised, WISC-R (Wechsler 1974; Vander Steene et al. 1986), the Wechsler Preschool and Primary Scale for Intelligence-Revised (Wechsler 1989; Vander Steene and Bos 1997), the Snijders-Oomen Niet-verbale intelligentie test-Revisie (SON-R; Snijders et al. 1996), and the Bayley Scales of Infant Development (Bayley 1969; Van der Meulen and Smrkovsky 1983). The participants from the current study were also included in the study of Hartman et al. (2006). However, the focus of that study was not on the ADI-R and ADOS. See Table 1 for characteristics of the participants. There were no significant differences between the groups recruited by survey or outpatient clinic for any of the variables in the table.

Table 1

Characteristics of the participants

	N	Age	Sex	ADI-R AD	ADOS ASD	DSM-IV-TR ASD
		Mean ± SD (range; y.m)	Boy N (%)	N (%)	N (%)	N (%)
Moderate ID	49	11.01 ± 3.35 (6.00–18.11)	37 (75.5)	17 (36.2)	32 (65.3)	22 (44.9)
Mild ID	87	10.31 ± 3.79 (4.08–18.11)	67 (77.0)	44 (50.6)	54 (62.1)	44 (50.6)
Total	136	10.57 ± 3.64 (4.08–18.11)	104 (76.5)	61 (45.5)	86 (63.2)	66 (48.5)

ID = Intellectual disability; ADI-R AD = AD classification of the ADI-R; ADOS ASD = ADOS classification of ASD, including AD and non-autism ASD; DSMIV-TR ASD = clinical classification based on DSM-IV-TR of ASD, including AD and non-autism ASD

Instruments

Children’s Social Behavior Questionnaire (CSBQ)

Children with an ASD form a heterogeneous group. The CSBQ is a 49 item parent questionnaire that aims to quantify the different behavioral dimensions along which children with ASD vary. In order to capture this variance, the CSBQ specifies six problem dimensions: ‘not optimally tuned to the social situation (Not tuned)’, ‘reduced contact and social interest (Contact)’, ‘difficulties in understanding social information (Understanding)’, ‘orientation problems in time, place or activity (Orientation)’, ‘stereotyped behavior (Stereotyped)’ and ‘fear of and resistance to changes (Changes)’. Children’s problems are represented in a score profile of these six dimensions. The CSBQ does not evaluate the presence or absence of a disorder. Instead, CSBQ scores are interpreted in relation to norm-groups (Hartman et al. 2006, 2008).

The CSBQ is not exclusively based on the DSM-IV-criteria. It contains the more stringent criteria for ASD, yet also represents less severe variations of these criteria and other associated problems, such as problems in language pragmatics, executive functioning, disruptive behavior, and resistance to change. The CSBQ is rated on a three point scale in order to reflect variation in degree of problem behavior. With this design the CSBQ aims to also capture behavioral variation at the lower end of the spectrum such as seen in children with PDD-NOS. Parents report on their child’s behavior of the last 2 months.

Psychometric qualities, with respect to test-retest, inter-rater, and internal consistency reliability of the subscales, and convergent and divergent validity were reported to be good (Hartman et al. 2006; Luteijn et al. 2000a, 2002). When various groups of children were studied (based on clinical classifications of PDD-NOS, attention deficit hyperactivity disorder (ADHD), comorbid ADHD and PDD-NOS, High Functioning Autism (HFA), and clinical controls), the CSBQ proved to be able to discriminate between these groups on the total scale as well as the subscales ‘contact’, ‘understanding’, ‘stereotyped’, and ‘changes’ (Hartman et al. 2006). The subscales ‘orientation’ and ‘not tuned’ were not specific for ASD, i.e., similar scores were obtained for children with ASD and children with ADHD (Hartman et al. 2006). Although the CSBQ was originally developed for children with normal intelligence, the psychometric qualities of the CSBQ in children with ID were found to be good and norm-groups could be created for mild and moderate ID (Hartman et al. 2006, 2008). In the group with ID, the subscales ‘contact’, ‘orientation’, ‘stereotyped’, and ‘changes’, and the CSBQ total score differed between children with and without ASD (Hartman et al. 2006).

Autism Diagnostic Interview-Revised (ADI-R)

The ADI-R is a standardized investigator-based interview that aims to provide data on the behavior of a child or young adult to discriminate between AD and non-AD (Rutter et al. 2003). The ADI-R focuses on the three domains of autism, based on the DSM-IV (American Psychiatric Association 1994) and ICD-10 (World Health Organization 1992). The ADI-R is conducted in an interview with parents or caregivers and is applicable for mental ages from about 24 months. The classification of the ADI-R is based on the age of 4–5 years. On the basis of the ADI-R, 45.5% of the children received an AD diagnosis (Table 1). We found only fair agreement on caseness between the ADI-R and the ADOS (Landis and Koch 1977; Cohen’s kappa .36) and moderate agreement between the ADI-R and the DSM-IV-TR classification (Cohen’s kappa .51).

Autism Diagnostic Observation Schedule (ADOS)

The ADOS is a semi-structured observational instrument, developed for children, adolescents, and adults referred for ASD, based on the DSM-IV (Lord et al. 1999). The assessment consists of various standardized situations, in which specific social, communicative, play, or stereotyped behavior is expected to be elicited. The ADOS consists of four modules, each applicable for children, adolescents, or adults of different levels of language and development. The ADOS classification includes AD and non-autism AD, and is based on the observation only. On the basis of the ADOS, 63.2% of the children received an ASD diagnosis (Table 1). Agreement on caseness with the other measures was fair (Landis and Koch 1977). Here, we found a Cohen’s kappa of .33 of the ADOS with the DSM-IV TR classification.

Clinical DSM-IV-TR classification

The clinical classification was based on DSM-IV-TR criteria, and was assigned by four experienced clinicians. For classification, information of the child on video during the ADOS was combined with information from parents as reported during the ADI-R. The clinicians were blind for the outcomes of the algorithms of ADI-R or ADOS, in order to assign a clinical classification as independent from the classifications on the instruments as possible. With this method, the clinical classification could be assigned based on the combination of information about current behavior as directly observed and about developmental history as reported by parents. DSM-IV-TR criteria were applied for AD. When a child did not fulfill these, the clinician decided on PDD-NOS or no PDD. In order to measure the level of agreement of the diagnostic classification (AD, PDD-NOS, non-PDD) between clinicians, a weighted kappa was calculated over children independently classified by two raters (10 in each combination of two clinicians). The weights we used were 1 for exact agreement, .5 if one rater scored autism and the other PDD-NOS and 0 in all other cases. The percentage of agreement found was 81.2% and the weighted kappa coefficient was .66 (sd .13). Both the percentage of agreement and the weighted kappa values are considered good according to the criteria of Cicchetti (2001), that combine the criteria reported earlier by Cicchetti and Sparrow (1981) for weighted kappa values and the criteria reported by Cicchetti et al. (1995) for percentages of agreement. Table 1 shows a clinical ASD classification in 48.5% of the children.

Statistics

Associations between the continuous ADI-R and ADOS domain and total scores and the CSBQ subscale and total scores were analyzed with Pearson r correlations. With logistic regressions (separately for each classification method and with each subscale and the total CSBQ, controlled for age and sex) the relations between the scores on the CSBQ and the ADI-R, ADOS, or clinical DSM-IV-TR classifications were evaluated.

Results

As presented in Table 2, correlations of the various subscales of the CSBQ were higher with the ADI-R than with the ADOS, and higher when related to current behavior on the ADI-R than to behavior at age 4 through 5.

Table 2

Pearson r correlations between CSBQ subscale and total scores and ADI-R and ADOS domain and total scores

	Not tuned	Contact	Orientation	Understanding	Stereotyped	Changes	CSBQ total
ADOS
Communication	.02	.39**	.13	−.07	.41**	.07	.21*
Social	−.04	.38**	.10	−.09	.33**	.03	.15
Imagination	.02	.31**	.15	−.12	.21*	−.04	.13
Repetitive restricted	−.09	.13	.13	−.08	.27**	−.03	.06
Total	−.02	.42**	.12	−.09	.38**	.05	.19*
ADI-R age 4–5
Communication	.08	.29**	.12	.30**	.21*	.28**	.27**
Social	.07	.48**	.22**	.28**	.29**	.23**	.35**
Repetitive restricted	.20*	.36**	.23**	.34**	.48**	.35**	.42**
Total behavioral domains	.11	.43**	.21*	.33**	.33**	.30**	.37**
ADI-R current
Communication	.26**	.46**	.26**	.31**	.38**	.43**	.45**
Social	.17*	.60**	.30**	.20*	.35**	.33**	.43**
Repetitive restricted	.27**	.39**	.30**	.42**	.52**	.41**	.50**
Total behavioral domains	.26**	.59**	.33**	.32**	.45**	.43**	.52**

Not tuned = ‘not optimally tuned to the social situation’; Contact = ‘reduced contact and social interest’; Understanding = ‘difficulties in understanding social information’; Orientation = ‘orientation problems in time, place or activity’; Stereotyped = ‘stereotyped behavior’; Changes = ‘fear of and resistance to changes’; CSBQ total = CSBQ total score; Total = total scores of respectively ADOS and ADI-R; Total behavioral domains = total of the current ADI-R scores on the three domains, excluding age of onset; mod = moderate intellectual disability; mild = mild intellectual disability. Pearson r between ADOS total and ADI-R age 4–5: .394 (p = .000); between ADOS total and ADI-R current: .431 (p = .000)

* p < .05; ** p < .01

The subscales ‘contact’ and ‘stereotyped’ were most clearly correlated to the ADOS domains and totals. With respect to the ADI-R age 4–5 almost all subscales (except ‘not tuned’) and the total CSBQ score showed significant correlations. With current behavior, the subscale ‘not tuned’ was also significantly correlated to the ADI-R, like all other subscales and the CSBQ total score. The correlations between CSBQ and ADI-R were all higher for ADI-R current behavior, except for the subscale ‘understanding’ in relation to the social domain as well as the total of the three behavioral domains of the ADI-R.

To investigate how the scores on the CSBQ are related to a classification of A(S)D on the ADI-R, ADOS, or clinical DSM-IV-TR classification, logistic regressions were applied (Table 3) for all CSBQ subscales and the total score, controlled for age and sex.

Table 3

Contribution of the CSBQ to an AD/ASD-classification on the ADI-R and ADOS and to a clinical DSM-IV-TR ASD classification, controlled for age and sex

	ADOS ASD classification	ADI-RAD classification	DSM-IV-TRASD classification
	Odds ratio (95% CI)	Odds ratio (95% CI)	Odds ratio (95% CI)
CSBQ
Not tuned	1.00 (.94–1.16)	1.05 (.98–1.19)	1.01 (.96–1.07)
Contact	1.18*** (1.08–1.29)	1.30*** (1.17–1.44)	1.21*** (1.11–1.32)
Orientation	1.04 (.95–1.13)	1.09 (1.00–1.19)	1.15** (1.05–1.27)
Understanding	.98 (.89–1.06)	1.10* (1.01–1.20)	1.16** (1.06–1.27)
Stereotyped	1.23** (1.08–1.40)	1.30*** (1.14–1.48)	1.25*** (1.10–1.42)
Changes	1.14 (.94–1.39)	1.55*** (1.24–1.92)	1.41** (1.15–1.73)
Total	1.02 (1.00–1.04)	1.05*** (1.02–1.07)	1.04*** (1.02–1.07)

ADOS ASD classification includes AD and non-autism ASD; DSM-IV-TR ASD classification = clinical classification based on DSM-IV-TR of ASD including AD and non-autism AD; Not tuned = ‘not optimally tuned to the social situation’; Contact = ‘reduced contact and social interest’; Understanding = ‘difficulties in understanding social information’; Orientation = ‘orientation problems in time, place or activity’; Stereotyped = ‘stereotyped behavior’; Changes = ‘fear of and resistance to changes’; Total = CSBQ total score

* p < .05; ** p < .01; *** p ≤ .001

The odds ratios express the increase or decrease in the probability of a classification of AD on the ADI-R or ASD (including AD and non-autism ASD) on the ADOS or clinical DSM-IV-TR classification, with increasing scores on the CSBQ subscales or total. For example, with an increase of one point on the total score of the CSBQ, the probability that a child was classified as AD by the ADI-R was elevated with 5% (or was 1.05 times higher).

For the ADOS, only the CSBQ subscales ‘contact’ and ‘stereotyped’ elevated the probability of an ASD-classification, with 18 and 23% respectively for each additional point on the subscale score. For the ADI-R, almost all subscales, except ‘not tuned’ and ‘orientation’ increased the probability of an AD classification with 10–55% (see Table 3). Even more subscales added to the clinical DSM-IV-TR classification of ASD, by 15–41% per point. Only the subscale ‘not tuned’ did not contribute. The CSBQ total score had approximately the same contribution to the clinical DSM-IV-TR classification as to the ADI-R, i.e., 4% per additional point, which is substantially lower than individual subscales. The results in the total group did not differ from the results in the separate levels of ID (data not shown). Additionally, the results did not change when correction for age and sex was eliminated (data not shown).

Discussion

The current paper aimed to investigate the criterion-related validity of the CSBQ in children with mild and moderate ID, by investigating how the CSBQ is related to the ADI-R and ADOS and how the CSBQ contributes to a clinical DSM-IV-TR ASD classification (including AD and non-autism ASD).

The results of our study indicate a satisfactory relationship between the CSBQ and the ADI-R. Specifically, four CSBQ subscales, i.e., ‘contact’, ‘stereotyped’, ‘changes’, and ‘understanding’ are most related and contribute significantly to both ADI-R and clinician-based classifications. A number of important differences between the CSBQ and the ADI-R make a more perfect coherence between both instruments less likely. That is, in contrast to the ADI-R, the CSBQ explicitly tries to capture the behavioral variety from normality to a full diagnosis of AD. Additionally, the CSBQ is a parent questionnaire rather than a clinician based interview, it is dimensional instead of categorical and it reports on the last 2 months without developmental focus. The two CSBQ subscales ‘contact’ and ‘stereotyped’ also compared well with the ADOS. Although the subscales ‘not tuned’ and ‘orientation’ seem less specific for ASD, the latter is important when compared to the clinical DSM-IV-TR classification.

The finding that the CSBQ subscales ‘contact’ and ‘stereotyped’ are the most contributing subscales of the CSBQ with respect to autistic behavior in mild and moderate ID is in line with a former study with regard to the CSBQ in mild and moderate ID that showed a relation between these subscales and autistic symptomatology as measured with the Autism Behavior Checklist (ABC, Krug et al. 1980; de Bildt et al. 2005a, b). These subscales measure the most salient autistic like behavior.

The items of the subscale ‘changes’ measure behavior when confronted with changes, expressed as fear, panicking, resistance, and freezing. Although not included in DSM-IV-TR diagnostic criteria or the ADI-R algorithm, these behaviors are nonetheless more related to the ADI-R and clinical DSM-IV-TR classification than to the ADOS. This is probably due to the fact that these behaviors are less likely to occur at full in a one-to-one test situation like the ADOS, than can be described by parents over time in the ADI-R or CSBQ. From our previous study with the CSBQ in mild and moderate ID it is known that the subscale ‘changes’ is also related to level of ID, with higher scores for children with mild ID as opposed to moderate ID (de Bildt et al. 2005a, b).

Children with high scores on the subscale ‘understanding’ have difficulties in understanding the rules of communication and the social use of language. Like with the subscale ‘changes’, these pragmatic language problems are not specifically captured by the ADI-R algorithm nor are they part of the DSM-IV-TR diagnostic criteria, yet they are clearly associated with the clinical DSM-IV-TR classification and the ADI-R. When comparing the communication domain items of the ADOS to this CSBQ subscale, they are found to have a slightly different focus. Again, the setting of the ADOS is very different from the CSBQ, more so than that of the ADI-R.

The remaining subscales of the CSBQ (‘orientation’ and ‘not tuned’) seem to measure behavior beyond the scope of the ADI-R and ADOS. From the comparison between clinical groups (Hartman et al. 2006, 2008) it is already known that these subscales are rather unspecific for ASD. Behaviors from these subscales also occur in children with ADHD and oppositional defiant disorder (ODD). The finding that these subscales do not relate to the ADI-R or ADOS is therefore not surprising. However, the subscale ‘orientation’ contributes to the clinical DSM-IV-TR classification of ASD in the current study, indicating that more subtle behaviors like little overview of activities with respect to time and place have importance for the clinician in classifying ASD. Additionally, this subscale has also shown to be related to level of functioning, with higher scores for children with moderate ID (de Bildt et al. 2005a, b). The subscale ‘not tuned’ does not contribute to any of the classifications of A(S)D as measured in this study, even though the lack of attuning to a social situation may be viewed as an important component of ASD. However, as mentioned, this subscale is not specific for ASD (Hartman et al. 2006) and therefore may not have a great importance in classifying the disorder. Nevertheless, descriptively, this subscale may be valuable, since the behaviors tapped provide additional information on the nature and severity of problems associated with ASD.

Full interpretation of the current findings is limited by the sample of participants. First of all, the results would have been more clear-cut had the study been conducted in children with normal intelligence, the group for whom the CSBQ originally was developed. Such a study is currently being carried out, however, data are not complete yet. Nevertheless, the currently presented results are important on their own, since the CSBQ is also used for children with ID. Additionally it would have been helpful when more children and adolescents without ASD could have been included, especially with classified other types of behavior problems, e.g., ADHD. Since scores on the CSBQ ‘not tuned’ and ‘orientation’ subscales are reported to be higher in children with PDD-NOS and comorbid ADHD (Hartman et al. 2006), it would have been interesting to further investigate this.

To conclude, the interrelationship between the CSBQ and the ADI-R is stronger than between the CSBQ and the ADOS, even though the ADI-R has a more narrow focus of AD, and takes into consideration a longer time period (developmental history). The ADOS would appear more closely linked to the CSBQ as it includes non-autism ASD, yet its scoring is based on a relatively short time period (i.e., the duration of the assessment only) and a different setting. The source of information seems of great importance in explaining our findings: the CSBQ and ADI-R are based on parent information, the ADOS on the judgment of an external observer. In sum, the contribution of the CSBQ to a classification of ASD in children and adolescents with mild and moderate ID is most specific for the subscales ‘contact’ and ‘stereotyped’ that are correlated to and contribute to the classifications on all three measures. Additionally, the subscales ‘understanding’ and ‘changes’ and the CSBQ total score are valid for predicting an ADI-R or DSM-IV-TR classification in this population. Finally, the subscale ‘orientation’ contributes to a clinical DSM-IV-TR ASD classification only.

The CSBQ can be considered as a helpful tool to describe ASD-related problems of children with mild and moderate ID. The CSBQ subscales add to the clinical DSM-IV-TR classification, and can therefore be valuable in an orienting stage of a diagnostic procedure. Using the score profile of the six subscales, the CSBQ may contribute to identifying whether or not the problem behavior as experienced is suggestive of ASD and whether further diagnostic assessments should be focused on ASD. Additionally, it may be of value in the diagnostic process, clearly not in diagnosing ASD which should be based on a more extensive diagnostic procedure preferably including ADOS and ADI-R, but in complementing other methods by adding unique additional information about the clinical presentation of a child. Even in this group, where differentiating ASD is more complicated, the current study shows that the CSBQ may be used as a signaling, screening, or describing instrument for those with ASD.