Concurrent Validity of the Child Behavior Checklist DSM-Oriented Scales: Correspondence with DSM Diagnoses and Comparison to Syndrome Scales.

This study used receiver operating characteristic (ROC) methodology and discriminative analyses to examine the correspondence of the Child Behavior Checklist (CBCL) rationally-derived DSM-oriented scales and empirically-derived syndrome scales with clinical diagnoses in a clinic-referred sample of children and adolescents (N = 476). Although results demonstrated that the CBCL Anxiety, Affective, Attention Deficit/Hyperactivity, Oppositional and Conduct Problems DSM-oriented scales corresponded significantly with related clinical diagnoses derived from parent-based structured interviews, these DSM-oriented scales did not evidence significantly greater correspondence with clinical diagnoses than the syndrome scales in all cases but one. The DSM-oriented Anxiety Problems scale was the only scale that evidenced significantly greater correspondence with diagnoses above its syndrome scale counterpart -the Anxious/Depressed scale. The recently developed and rationally-derived DSM-oriented scales thus generally do not add incremental clinical utility above that already afforded by the syndrome scales with respect to corresponding with diagnoses. Implications of these findings are discussed.

The Child Behavior Checklist (CBCL; Achenbach and Rescorla 2001) is among the most widely used parent-report measures of youth emotional and behavioral problems in both clinical and research settings. Since the development of the original empirically-based CBCL scales (Achenbach and Edelbrock 1983), Achenbach and colleagues have added narrow-band syndrome scales, competence and adaptive scales, as well as broad-band internalizing, externalizing, and total problems scales to better organize the information obtained from parents on their children’s competences and behavioral problems. With well-established normative data and standardized clinical cutoffs for its various syndrome and broad-band scales, the CBCL syndrome, internalizing, externalizing, and total problems scales have demonstrated strong psychometric properties within clinical settings for discriminating between referred and nonreferred populations (Achenbach 1991; Chen et al. 1994; Drotar et al., 1995). For instance, the CBCL internalizing scale has demonstrated the ability to discriminate between youths with and without anxiety, as well as youths with anxiety disorders and youths with externalizing disorders (Seligman et al. 2004). Significant associations with both general (e.g., anxiety and affective disorders groups) and more specific (e.g., attention-deficit/hyperactivity disorder and conduct disorder) diagnostic groups have also been established (Eiraldi et al. 2000; Kasius et al. 1997), supporting the utility of the CBCL syndrome and broad-band scales across multiple settings.

CBCL DSM-Oriented Scales

Although the CBCL’s empirically-derived syndrome and broad-band scales have evidenced particular strengths, it has been argued that a weakness is their lack of concordance with nosology from the Diagnostic and Statistical Manual of Mental Disorders (DSM; American Psychiatric Association 2000). For example, Jensen et al. (1993) and Kasius et al. (1997) found that the CBCL syndrome scales appear to be sensitive to measuring general mental illness/psychopathology, but not very sensitive to measuring specific DSM diagnoses, particularly for internalizing disorders. This may not be surprising given the factor analytically-derived syndrome scales measure dimensions not entirely consistent with DSM nosology. For instance, items that appear specific to DSM categories of anxiety or depression are scored on the same Anxious/Depressed syndrome scale (e.g., “too fearful or anxious”, “nervous, highstrung, or tense “, “talks about killing self”, “feels worthless or inferior”).

Scales that map closely to DSM nosology may be useful for several reasons. First, DSM-oriented scales have the potential to better screen for and inform subsequently administered diagnostic interviews than other scales that were designed to measure multi-disorder syndromes or other constructs not aligned with DSM categories. Relatedly, reimbursement for mental health services in clinical settings in America relies heavily on the documentation of DSM-based disorders (Achenbach and Dumenci 2001; Doucette 2002). The psychopathology and treatment research literature are also largely based upon the DSM classification system. Therefore, to provide an additional perspective with closer linkage to the prevailing DSM nosology, Achenbach et al. (2003) developed the CBCL DSM-oriented scales to supplement the traditional CBCL syndrome scales. Unlike the syndrome scales, the DSM-oriented scales were not derived via factor analytic methods, but were constructed through agreement in ratings among 22 highly experienced child psychiatrists and psychologists from 16 cultures. These experts rated each pre-existing CBCL item for the degree to which each was consistent with criteria for a particular DSM-IV diagnostic category. Items were then matched with a given diagnostic category if rated as “very consistent with the DSM category” by at least 14 of the 22 experts (Achenbach et al. 2003), and a DSM-oriented scale was developed for categories matching 6 or more items. The six DSM-oriented scales (and the diagnoses they are meant to represent) are: (a) Affective Problems [Dysthymic (DYS) and Major Depressive Disorders (MDD)], (b) Anxiety Problems [Generalized Anxiety Disorder (GAD), Separation Anxiety Disorder (SAD), and Specific Phobia (SPEC)], (c) Attention Deficit/Hyperactivity Problems [Primarily Hyperactive (ADHD-PH), Primarily Inattentive (ADHD-PI) and Combined (ADHD-C) subtypes], (d) Conduct Problems [Conduct Disorder (CD)], (e) Oppositional Defiant Problems [Oppositional Defiant Disorder (ODD)], and (f) Somatic Problems (Somatization and Somatoform Disorders).

Although research supporting the psychometric properties of these CBCL DSM-oriented scales is growing, less is known about the clinical utility of these scales with respect to their correspondence with youth clinical diagnoses. Their comparative performance with related syndrome scales regarding correspondence with DSM diagnoses is also unknown. Knowing the degree to which the rationally-derived DSM-oriented scales correspond with DSM diagnoses relative to the empirically-derived syndrome scales could provide insight into the incremental utility offered by the newly derived DSM-oriented scales as well as whether the DSM-oriented scales achieved their intended aims (i.e., greater correspondence with DSM categories). Among research conducted to date on the CBCL DSM-oriented scales, Achenbach et al. (2003) reported that the CBCL DSM-oriented scales, compared to the syndrome scales, evidenced a similar degree of internal consistency, test-retest reliability, and cross-informant agreement. They also reported that both scales revealed similar associations between classifications of youths’ scores as “deviant” (defined as T-score ≥ 65). The factor structure of the CBCL DSM-oriented scales was also found to be supported in a community sample (Achenbach et al. 2003). Nakamura et al. (2009) also found support for scale reliability, as well as convergent and discriminative validity, for all six CBCL DSM-oriented scales using a large and ethnically diverse clinic-referred sample of 673 children and adolescents diagnosed with structured interviews.

Ferdinand (2008) also explored the clinical utility of the CBCL DSM-oriented Anxiety Problems and Affective Problems scales for predicting pertinent DSM-IV internalizing diagnoses (based on either parent/child impairment ratings or clinician rated clinical severity ratings) via receiver operating characteristic (ROC) curve methodology, in a clinic-referred sample of 277 youths, ages 6–18. Ferdinand found that the CBCL DSM-oriented Anxiety Problems scale could predict SAD, GAD and SPEC when diagnoses were based on parent/child impairment ratings, but could not predict SAD and GAD when diagnoses were based on clinician severity ratings. Similarly, Ferdinand found that the CBCL DSM-oriented Affective Problems scale could predict MDD and DYS when diagnoses were based on parent/child impairment ratings, but could not predict DYS when diagnoses were based on clinician severity ratings.

Although studies have begun to examine the psychometric properties of some of the CBCL DSM-oriented scales, numerous questions remain regarding the clinical utility of these scales on this widely used measure. Particularly unexplored are the DSM-oriented externalizing scales (i.e., Oppositional Problems and Conduct Problems scales) as well as the Attention Deficit/Hyperactivity (ADH) Problems scale. Regarding the DSM-oriented ADH Problems scale, although the developers indicated that this scale was designed to measure both ADHD-PI and ADHD-PH, it may be useful to test empirically whether this scale performs well at identifying ADHD-PI, ADHD-PH, and ADHD-C youths as well as discriminating between ADHD and disruptive behavior related disorders. Similarly, regarding the DSM-oriented Oppositional and Conduct Problems scales, further analysis of performance in clinical settings may provide insight into questions of how best use these scales to assist in diagnostic formulations of ODD, CD and their differential diagnosis. The CBCL’s broad use in diagnostic settings makes informing answers to these questions of particular importance.

For this reason, the present study used ROC curve methodology and discriminative ANOVAs to examine the concurrent validity and correspondence of the internalizing- and externalizing-related CBCL DSM-oriented scales with related DSM diagnoses using a clinic-referred sample of children and adolescents. In general, it was predicted that the DSM-oriented scales would evidence significantly greater correspondence with DSM diagnoses than the syndrome scales given their purported closer linkage with DSM categories. Specific hypotheses for each scale are indicated below, in the Data Analytic Approach section.

Method

Participants

Youths in the present sample were 476 of 557 consecutively referred children and adolescents to two mental health clinics (the Center for Cognitive Behavioral Therapy, in Honolulu, Hawaii; and Judge Baker’s Children Hospital, in Boston, Massachusetts) for diagnostic intake evaluations. Eligibility for youths in the present study included being 6–18 years old and having a CBCL1 and diagnostic data available. Of the 557 consecutively referred youths, three children (0.5%) were younger than 6 years old, two youths (0.4%) had missing diagnostic data and 75 youths (13.5%) did not have an available CBCL (due to their caretakers choosing to not fill out the assessment measures). These 80 youths were thus not included in the study. To help ensure that all CBCLs contained valid responses, inclusion into the study also required each CBCL form to have 90% or more completed items (i.e., fewer than 13 items missing). One participant was excluded due to having 17 items missing, leaving a final sample size of 476 youths. The final sample consisted of 320 (67.2%) boys and 156 (32.8%) girls. Information about the total number of diagnoses present in the sample (primary and anywhere in the diagnostic profile) appears in Table 1. Youths’ ages ranged from 6.55 to 18.9 years (M = 11.4, SD = 2.5), and primary caregivers’ ages ranged from 21 to 78 (M = 41.4; SD = 9.6). Additional youth and primary caregiver demographic information appears in Table 2. All children and parents were fluent in English.

Table 1

Number of diagnoses anywhere and primary in youths’ diagnostic profile (N = 476)

Diagnoses	Specifier
	Principal	Anywhere
Anxiety Disorders
Generalized anxiety disorder	26	75
Separation anxiety disorder	26	74
Specific phobia	8	75
Social phobia	10	41
Obsessive-compulsive disorder	7	18
PTSD	8	22
Panic disorder	0	1
Anxiety NOS	1	1
Affective Disorders
Major depressive disorder	27	69
Dysthymic disorder	11	33
Depressive disorder NOS	4	4
ADHD Disorders
ADHD-combined type	32	110
ADHD-predominantly inattentive type	38	83
ADHD-predominantly hyperactive-impulsive type	4	6
ADHD-NOS	21	48
Disruptive Behavior Disorders
Oppositional defiant disorder	118	174
Conduct disorder	57	97
Disruptive behavior disorder NOS	5	6
Schizophrenia	4	10
Bipolar	1	4
PDD	0	0
Other	19	39
No Diagnosis	65	65

ADHD attention-deficit/hyperactivity disorder; Anywhere a diagnosis that appears anywhere in a child’s diagnostic profile (i.e., principal, secondary, etc.); NOS not otherwise specified; PDD pervasive developmental disorder; Principal a child’s primary diagnosis; PTSD post-traumatic stress disorder; Other includes substance abuse, substance dependence, enuresis, trichotillomania, body dysmorphic disorder, anorexia nervosa, parent-child relational problem, and sibling relational problem. The total number principal diagnoses sum to slightly more than the number of youths in the present study due to a small portion of youths receiving co-principal diagnoses

Table 2

Youth and caregiver demographic information

	n	Percentage
Youth Ethnicity
Multiethnic	182	38.2
White	169	35.5
African American	35	7.4
Asian American	28	5.9
Latino/Hispanic	25	5.3
Other	27	5.7
Missing	10	2.1
Caregiver Type
Biological Mother	268	56.3
Biological Father	83	17.4
Adoptive Mother	22	4.6
Adoptive Father	6	1.3
Grandmother	21	4.4
Grandfather	10	2.1
Other	40	8.3
Missing	26	5.5
Caregiver Marital Status
Married	220	46.2
Divorced, separated	121	25.5
Widowed	18	3.8
Single	84	17.7
Missing	33	6.9
Caregiver Highest Level of Education
No high school	41	8.6
High school	163	34.2
College	192	40.3
Graduate School	35	7.4
Missing	45	9.5
Family Income
$0–$39,000	243	51.1
$40,000 – $79,000	117	24.6
$80,000 – $119,000	52	10.9
$120,000 or more	25	5.3
Missing	39	8.2

Caregiver Type Other includes maternal aunt (n = 1), guardian (n = 1), health professional (n = 1), step father (n = 11), foster mother (n = 14), and “other” (n = 12)

Measures

Child Behavior Checklist for Ages 6–18 (CBCL/6–18; Achenbach and Rescorla 2001)

The 120 items on the CBCL are rated as Not True (0), Somewhat or Sometimes True (1), or Very True or Often True (2). Items are summed to yield (a) competence and adaptive scale scores, (b) eight syndrome scale scores (comprised of 103 of the total 120 items), (c) six DSM-oriented scale scores (comprised of 55 of the total 120 items), and (d) broad-band scale scores (including internalizing and externalizing total scores). Validity and reliability of the syndrome and DSM-oriented scales have been documented (Achenbach and Rescorla 2001; Achenbach et al. 2003), and extensive normative data are available for children ranging from 6 to 18. All analyses were conducted using raw CBCL scale scores, as Achenbach and Rescorla’s (2001) ASEBA manual recommends using raw scores in order to account for the full range of variation.

Children's Interview for Psychiatric Syndromes, Parent Version (P-ChIPS; Weller et al. 1999)

The P-ChIPS is a semi-structured interview designed to be administered to the parents of youths (aged 6–18 years) by trained interviewers. The interviews screen for 20 different mood, anxiety, behavioral, substance, eating, and elimination Axis I disorders as well as psychosocial stressors, and are based on the DSM-IV classification criteria. Content and concurrent validity, as well as inter-rater reliability of the P-ChIPS have been demonstrated. Fristad et al. (1998) showed that the P-ChIPS demonstrated moderate levels of agreement with the ChIPS child interview (mean kappa = 0.41) as well as with clinician diagnoses (mean kappa = 0.49). Across diagnostic categories, sensitivity averaged 87% and specificity averaged 76%, suggesting that the P-ChIPS is similar to other structured interviews administered to parents regarding the accuracy of both positive and negative findings.

Procedure

Prior to any data collection, all participants and their legal guardians underwent standardized Institutional Review Board-approved notice of privacy and consent procedures. The youths’ primary caregivers completed the CBCL in English and also participated in structured diagnostic interviews (i.e., the P-ChIPS). Assessors consisted of Ph.D. level clinical child psychologists and doctoral students in clinical psychology2. A small portion of diagnoses were for problem areas not assessed by the P-ChIPS diagnostic interviews (e.g., trichotillomania). In order to make such diagnostic determinations, assessors acquired information from the parents regarding these problems areas according to DSM diagnostic criteria (APA, 2000) and also discussed diagnostic formulation with available supervisors.

Data preparation

Although missing data levels were low in the present sample [i.e., 369 (77.5%) participants of the full sample had no missing CBCL items, 69 (14.5%) had only 1 missing item, 19 (4.0%) had 2 missing items, 18 (3.8%) had 3–7 missing items and 1 (0.2%) had 12 missing items], we imputed missing data using the Missing Value Analysis (MVA) module of SPSS 15.0 (SPSS, 2006). The SPSS MVA module examines missing data patterns and imputes missing values through a maximum likelihood method based on expectation-maximization algorithms (Little and Rubin 1987). To help ensure that only valid CBCL subscales were included in analyses, we calculated each subscale only if it had less than 20% missing items. Twenty percent instead of 10% was used as the cut-off for subscale inclusion to allow the subscales with low item counts (e.g., the DSM-oriented Anxiety and ADH Problems scales) to have one item missing and still be calculated. Among the DSM-oriented scales, three Anxiety, three Somatic, one Affective, one ADH, one Oppositional and one Conduct Problems scales scores were not calculated due to having more than 20% missing data. Among the syndrome scales, two Anxious/Depressed, two Somatic Complaints, two Thought Problems, one Withdrawn/Depressed, one Social Problems, one Attention Problems, one Rule Breaking Behavior and one Aggressive Behavior scale scores were not calculated due to having more than 20% missing data.

Data Analytic Approach

The correspondence of the DSM-oriented scales and syndrome scales with related diagnoses was examined via ANOVA and receiver operating characteristic (ROC) analyses, using Analyze-It for Microsoft Excel version 2.12 (Analyze-It Software Ltd., 2008). ROC analyses result in Area Under the Curve (AUC) values, which indicate the degree to which a scale predicts a binary classification (e.g., presence or absence of a clinical diagnosis). AUC values significantly greater than .50 indicate that the scale can perform the binary classification better than chance level. AUC values may also be interpreted according to the following: 50–.70, poor; .70–.80, fair; .80–.90, good; .90–1.00, excellent (c.f. Ferdinand 2008). The comparative performance of the DSM-oriented and syndrome scales with respect to corresponding with relevant diagnoses was evaluated via z-test comparisons of AUC values (DeLong et al. 1988). Larger AUC values are indicative of better prediction of diagnostic status. Given the number of ANOVAs and AUC z-tests conducted, consideration of the Bonferroni correction (i.e., .05/37) warranted setting the significance level to .001 to help correct for Type-I error rates in the present study. The DSM-oriented Somatic Problems scale was not examined because no youths in the present sample had a Somatization Disorder.

Using discriminative ANOVA and ROC methodology, we applied a general analytic approach to each DSM-oriented scale. (a) First, if the scale was designed to target a cluster of related diagnoses, we used ANOVA and AUC analyses to evaluate whether the DSM-oriented scale could discriminate youths with each individual diagnosis targeted by the scale from all youths without diagnoses of the related disorders (e.g., evaluating whether the DSM-oriented Anxiety Problems scale — designed to measure SAD, GAD and SPEC — can discriminate youths with SAD from youths without any anxiety disorder). We performed this test to demonstrate that the DSM-oriented scales corresponded significantly with each disorder targeted by the scale. This test was applicable to the DSM-oriented Anxiety, Affective and ADH Problems scales, as these scales were designed to target clusters of related disorders. We predicted that these DSM-oriented scales would correspond significantly (i.e., evidence significant ANOVA and AUC values significantly greater than chance) with all individual diagnostic groups. (b) Second, we used ANOVA and AUC analyses to evaluate whether each DSM-oriented scale could discriminate between any youths with the scale’s targeted disorder group from all youths without any of those disorders (e.g., evaluating whether the DSM-oriented Anxiety Problems scale — designed to measure SAD, GAD and SPEC — can discriminate youths with SAD, GAD and/or SPEC from youths without any of these disorders), and whether each DSM-oriented scale could perform this discrimination significantly better than its related syndrome scale (i.e., significantly greater AUC). These tests were applicable to all DSM-oriented scales. We predicted that all DSM-oriented scales would be able to discriminate the targeted diagnostic groups significantly better than the syndrome scales given that the DSM-oriented scales were developed to more closely correspond with current DSM categories. (c) Third, we used ANOVA and AUC analyses to evaluate whether each DSM-oriented scale could discriminate youths with the scale’s targeted disorder group from youths with a related disorder (but without any disorder related to the targeted disorders), so as to aid in differential diagnostic formulations between related disorders3. For example, one test of this type evaluated whether the DSM-oriented Anxiety Problems scale — designed to measure SAD, GAD and SPEC — can discriminate youths with SAD, GAD and/or SPEC from youths with any affective disorder (and without any anxiety disorder). This type of analysis was applicable to all DSM-oriented scales. Again, we predicted that all DSM-oriented scales would be able to significantly discriminate pertinent diagnostic groups and would perform significantly better than the related syndrome scales (i.e., significantly greater AUC values). The one exception was the DSM-oriented Oppositional Problems scale. We predicted that this scale would not be able to discriminate between ODD and CD youths, given that all the features of ODD are usually present in CD. Notably, we predicted the null hypothesis in this particular case. However, instead of omitting this analysis due to the prediction of the null hypothesis, we thought it was meaningful to include this analysis in the present study. That is, if significant differences were found between ODD and CD youths on the DSM-oriented Oppositional Problems scale, this would suggest that this scale does not specifically measure ODD symptomatology given that both ODD and CD youths should have elevated scores on this scale.

Results

CBCL DSM-Oriented Anxiety Problems Scale

Individual diagnoses of the targeted diagnostic group

The ANOVA and ROC results evaluating the various tests of the DSM-oriented Anxiety Problems scale — including the test of correspondence with individual diagnoses of the targeted diagnostic group — appear in Table 3. As predicted, the DSM-oriented Anxiety Problems scale corresponded significantly with the individual diagnoses targeted by the scale (i.e., SAD, GAD, SPEC; Table 3, row 1, 2 and 3).

Table 3

CBCL scale means and ROC Area Under the Curve (AUC) values for children with and without selected anxiety related diagnoses anywhere in their diagnostic profile

CBCL Scale	Diagnostic Groups	ANOVA					ROC
		n	M	SD	F	p	AUC	z	p
DSM-Oriented Anxiety Problems	SAD	74	6.7	2.6
	No Anxiety	290	2.6	2.3	180.3	< .001	.88
DSM-Oriented Anxiety Problems	GAD	75	6.6	2.6
	No Anxiety	290	2.5	2.3	175.1	< .001	.88
DSM-Oriented Anxiety Problems	SPEC	75	6.4	2.7
	No Anxiety	290	2.5	2.3	153.2	< .001	.86
DSM-Oriented Anxiety Problems	SAD/GAD/SPEC	148	6.3	2.6
	No SAD/GAD/SPEC	325	2.7	2.4	215.0	< .001	.84
Syndrome Anxious/Depressed	SAD/GAD/SPEC	148	10.9	4.9
	No SAD/GAD/SPEC	326	5.6	4.6	130.0	< .001	.80	3.92	< .001
DSM-Oriented Anxiety Problems	SAD/GAD/SPEC	148	6.3	2.6
	Mood (No Anxiety)	37	3.0	2.3	47.2	< .001	.82
Syndrome Anxious/Depressed	SAD/GAD/SPEC	148	10.9	4.9
	Mood (No Anxiety)	37	6.9	4.4	20.8	< .001	.72	3.27	.001

CBCL Child Behavior Checklist; SAD separation anxiety disorder; GAD generalized anxiety disorder; SPEC specific phobia; SAD/GAD/SPEC youths with separation anxiety disorder, generalized anxiety disorder; and/or specific phobia; Mood (No Anxiety) youths with major depressive disorder, dysthymic disorder or depressive disorder not otherwise specified, and no anxiety disorder (any type, including separation anxiety disorder, generalized anxiety disorder, specific phobia, obsessive-compulsive disorder, post traumatic stress disorder, panic disorder, social phobia, and anxiety disorder not otherwise specified).

All AUC values are significantly greater than .50, p < .01

Targeted diagnostic group

As predicted, the DSM-oriented Anxiety Problems scale was able to discriminate the anxiety group targeted by the scale (i.e., SAD/GAD/SPEC youths) from youths without these diagnoses. The Anxious/Depressed syndrome scale also significantly discriminated these groups, and consistent with prediction, the AUC of the DSM-oriented Anxiety Problems scale was significantly greater than the AUC of the Anxious/Depressed syndrome scale (Table 3, row 4 and 5). Both AUC values, however, fell in the “good” range.

Differential diagnoses

As predicted, the DSM-oriented Anxiety Problems scale significantly discriminated youths with SAD, GAD and/or SPEC from affective youths without anxiety disorders (i.e., youths with major depressive disorder, dysthymic disorder, or depressive disorder not otherwise specified, but without any anxiety disorders4). The Anxious/Depressed syndrome scale also significantly discriminated these groups (as evidenced by a significant ANOVA), and consistent with prediction, the AUC of the DSM-oriented Anxiety Problems scale (falling in the “good” range) was significantly greater than the AUC of the Anxious/Depressed syndrome scale (falling in the “fair” range; Table 3, row 6 and 7).

CBCL DSM-Oriented Affective Problems Scale

The ANOVA and ROC results evaluating the various tests of the DSM-oriented Affective Problems scale — including the test of correspondence with individual diagnoses of the targeted diagnostic group — appear in Table 4. As predicted, the DSM-oriented Affective Problems scale corresponded significantly with the individual diagnoses targeted by the scale (i.e., MDD, DYS; Table 4, row 1 and 2).

Table 4

CBCL scale means and ROC Area Under the Curve (AUC) values for children with and without selected affective related diagnoses anywhere in their diagnostic profile

CBCL Scale	Diagnostic Groups	ANOVA					ROC
		n	M	SD	F	p	AUC	z	p
DSM-Oriented Affective Problems	MDD	69	9.1	5.0
	No Mood	369	4.6	3.9	68.9	< .001	.76
DSM-Oriented Affective Problems	DYS	33	7.2	3.2
	No Mood	369	4.6	3.9	13.6	< .001	.72
DSM-Oriented Affective Problems	MDD/DYS	102	8.5	4.6
	No MDD/DYS	373	4.6	3.9	71.3	< .001	.75
Syndrome Withdrawn/Depressed	MDD/DYS	102	6.8	3.6
	No MDD/DYS	373	4.0	3.2	54.9	< .001	.72	1.35	.178
DSM-Oriented Affective Problems	MDD/DYS	102	8.5	4.6
	Anxiety (No Mood)	115	6.1	4.0	16.2	< .001	.65
Syndrome Withdrawn/Depressed	MDD/DYS	102	6.8	3.6
	Anxiety (No Mood)	115	5.0	3.5	13.1	< .001	.64	0.52	.605

CBCL Child Behavior Checklist; MDD major depressive disorder; DYS dysthymic disorder; SPEC specific phobia; MDD/DYS youths with major depressive disorder or dysthymic disorder; Anxiety (No Mood) youths with separation anxiety disorder, generalized anxiety disorder, specific phobia, obsessive-compulsive disorder, post traumatic stress disorder, panic disorder, social phobia, and/or anxiety disorder not otherwise specified, and no affective disorder, including major depressive disorder, dysthymic disorder and depressive disorder not otherwise specified All AUC values are significantly greater than .50, p < .01

As predicted, the DSM-oriented Affective Problems scale was able to discriminate youths with MDD or DYS (i.e., the youths targeted by the scale) from youths without these diagnoses. The Withdrawn/Depressed syndrome scale also significantly discriminated these groups. Contrary to prediction, however, the AUC of the DSM-oriented Affective Problems scale was not significantly different than the AUC of the Withdrawn/Depressed syndrome scale (Table 4, row 3 and 4). Both AUC values also fell in the “fair” range.

As predicted, the DSM-oriented Affective Problems scale significantly discriminated youths with MDD or DYS from anxious youths without any affective disorder (i.e., youths with any anxiety disorder, but without any affective disorder5). The Withdrawn/Depressed syndrome scale was also able to significantly discriminate these groups. Contrary to prediction, however, the AUC of the DSM-oriented Affective Problems scale was not significantly different than the AUC of the Withdrawn/Depressed syndrome scale (Table 4, row 5 and 6). Both AUC values also fell in the “poor” range.

CBCL DSM-Oriented Attention Deficit/Hyperactivity Problems Scale

The ANOVA and ROC results evaluating the various tests of the DSM-oriented Affective Problems scale — including the test of correspondence with individual diagnoses of the targeted diagnostic group — appear in Table 5. As predicted, the DSM-oriented ADH Problems scale corresponded significantly with the individual diagnoses targeted by the scale (i.e., ADHD-PI, ADHD-C; Table 5, row 1 and 2). Correspondence with ADHD-PH could not be examined due to an insufficient number of youths with ADHD-PH in the current sample.

Table 5

CBCL scale means and ROC Area Under the Curve (AUC) values for children with and without selected ADHD related diagnoses anywhere in their diagnostic profile

CBCL Scale	Diagnostic Groups	ANOVA					ROC
		N	M	SD	F	p	AUC	z	p
DSM-Oriented ADH	ADHD-PI	83	6.9	2.6
	No ADHD	229	4.8	3.3	26.8	< .001	.69
DSM-Oriented ADH	ADHD-C	110	9.9	2.7
	No ADHD	229	4.8	3.3	194.4	< .001	.87
DSM-Oriented ADH	ADHD-PI/PH/C	199	8.6	3.1
	No ADHD-PI/PH/C	276	5.3	3.5	112.2	< .001	.75
Syndrome Attention	ADHD-PI/PH/C	199	10.8	4.0
	No ADHD-PI/PH/C	276	6.6	4.5	110.0	< .001	.76	.17	.865
DSM-Oriented ADH	ADHD-PI/PH/C	199	8.6	3.1
	Disruptive Disorder (No ADHD)	88	6.1	3.0	41.4	< .001	.71
Syndrome Attention	ADHD-PI/PH/C	199	10.8	4.0
	Disruptive Disorder (No ADHD)	88	7.2	4.0	49.2	< .001	.74	1.19	.234

CBCL Child Behavior Checklist; DSM-Oriented ADH DSM-oriented ADH Problems scale; Syndrome Attention Attention Problems syndrome scale; ADHD-C attention deficit/hyperactivity disorder, combined type; ADHD-PI attention deficit/hyperactivity disorder, primarily inattentive type; ADHD-PI/PH/C youths with attention deficit/hyperactivity disorder, primarily inattentive type, primarily hyperactive type, or combined type; Disruptive Disorder (No ADHD) youths with oppositional defiant disorder, conduct disorder or disruptive behavior disorder not otherwise specified, and no attention deficit/hyperactivity disorder (any type). All AUC values are significantly greater than .50, p < .01

As predicted, the DSM-oriented ADH Problems scale was able to discriminate youths with ADHD-PI, ADHD-PH, and ADHD-C (i.e., the youths targeted by the scale) from youths without these diagnoses. The Attention Problems syndrome scale was also able to significantly discriminate these groups. Contrary to prediction, however, the AUC of the DSM-oriented ADH Problems scale was not significantly different than the AUC of the Attention Problems syndrome scale (Table 5, row 3 and 4). Both AUC values also fell in the “fair” range.

As predicted, the DSM-oriented ADH Problems scale significantly discriminated youths with ADHD-PI, ADHD-PH, or ADHD-C from disruptive youths without any ADHD disorder (i.e., youths with ODD, CD, or disruptive behavior disorder not otherwise specified, but without any ADHD disorder). The Attention Problems syndrome scale also significantly discriminated these groups. Contrary to prediction, however, the AUC of the DSM-oriented ADH Problems scale was not significantly different than the AUC of the Attention Problems syndrome scale (Table 5, row 5 and 6). Again, both AUC values fell in the “fair” range.

CBCL DSM-Oriented Oppositional and Conduct Problems Scales

The ANOVA and ROC results evaluating the various tests of the DSM-oriented Oppositional and Conduct Problems scales appear in Table 6. As predicted, the DSM-oriented Oppositional Problems scale was able to discriminate youths with ODD (the targeted diagnostic group of this scale) from youths without ODD. The Aggressive Behavior syndrome scale was also able to significantly discriminate these groups. Contrary to prediction, however, the AUC of the DSM-oriented Oppositional Problems scale was not significantly different than the AUC of the Aggressive Behavior syndrome scale (Table 6, row 1 and 2), and both AUC values fell in the “fair” range. With respect to the DSM-oriented Conduct Problems scale, this scale was able to discriminate youths with CD (i.e., the targeted diagnostic group of this scale) from youths without CD. The Rule-Breaking Behavior syndrome scale was also able to significantly discriminate these groups. Contrary to prediction, however, the AUC of the DSM-oriented Conduct Problems scale was not significantly different than the AUC of the Rule-Breaking Behavior syndrome scale (Table 6, row 3 and 4), and both AUC values fell in the “good” range.

Table 6

CBCL scale means and ROC Area Under the Curve (AUC) values for children with and without selected disruptive behavior diagnoses anywhere in their diagnostic profile

CBCL Scale	Diagnostic Groups	ANOVA					ROC
		N	M	SD	F	p	AUC	z	p
DSM Oppositional Problems	ODD	174	6.3	2.3
	No ODD	301	4.3	2.8	68.6	< .001	.71*
Syndrome Aggressive Behavior	ODD	174	15.0	6.7
	No ODD	301	10.1	7.7	49.2	< .001	.71*	0.81	.419
DSM Conduct Problems	CD	97	11.7	5.4
	No CD	378	4.9	5.1	132.0	< .001	.84*
Syndrome Rule-Breaking	CD	97	10.0	4.9
	No CD	378	4.2	4.0	149.3	< .001	.85*	0.75	.451
DSM Oppositional Problems	ODD	174	6.3	2.3
	CD	97	6.3	2.4	0.03	.869	.50
Syndrome Aggressive Behavior	ODD	174	15.0	6.7
	CD	97	15.7	7.5	0.59	.445	.52	0.22	.827
DSM Conduct Problems	CD	97	11.7	5.4
	ODD	174	7.0	4.7	54.7	< .001	.75*
Syndrome Rule-Breaking	CD	97	10.0	4.9
	ODD	174	5.6	3.4	78.5	< .001	.78*	1.81	.071

CBCL Child Behavior Checklist; DSM Oppositional Problems DSM-oriented Oppositional Problems scale; Syndrome Aggressive Behavior Aggressive Behavior syndrome scale; DSM Conduct Problems DSM-oriented Conduct Problems scale; Syndrome Rule-Breaking Rule-Breaking syndrome scale; ODD oppositional defiant disorder; CD conduct disorder

*AUC values significantly greater than .50, p < .01

As predicted, the DSM-oriented Oppositional Problems scale was not able to discriminate youths with ODD from youths with CD. The Aggressive Behavior syndrome scale also could not perform this discrimination. As noted above, this was expected given that both ODD and CD youths exhibit ODD-related symptomatology. With respect to the DSM-oriented Conduct Problems scale, as predicted, this scale was able to significantly discriminate youths with CD from youths with ODD. The Rule-Breaking Behavior syndrome scale was also able to significantly discriminate CD from ODD youths. Contrary to prediction, however, the AUC of the DSM-oriented Conduct Problems scale was not significantly different than the AUC of the Rule-Breaking Behavior syndrome scale (Table 6, row 7 and 8), and both AUC values fell in the “fair” range.

Mothers-only analyses

Given that mothers and fathers have reported differently on the CBCL (e.g., Kendall et al. 2007), such as mothers reporting more youth problems than fathers (e.g., Christensen et al. 1991), we repeated all analyses using mothers-only data (n = 290). Results were almost completely identical as the results based on the full sample. Across all analyses, the only differences between the results based on the mothers-only subsample and full sample were the following: (a) the AUC of the DSM-oriented Anxiety Problems scale was significantly greater than the AUC of the Anxious/Depressed syndrome scale with respect to discriminating youths with SAD, GAD, or SPEC from youths with any affective disorder (but no anxiety disorder) at the p  <  .05 level with the mothers-only sample, but at the p < .001 with the full sample; this difference was likely due to reduced power in the mothers-only analyses as the sample size of the affective disorder group reduced from 37 to 26 affective youths when using the mothers-only data; and (b) the Withdrawn/Depressed syndrome scale was able to discriminate youths with MDD or DYS from youths with any anxiety disorder (and without any affective disorders) at the p < .01 level with the mothers-only sample, but at the p < .001 with the full sample. The patterns of results of all other analyses were identical between the full and mothers-only samples6. The results of the present study are thus likely generalizable to and interpretable in relationship with the literature on mothers’ reports of youth problems.

Gender specific analyses

Given that differences have been found between parental ratings of boys’ and girls’ behavior/emotional problems (e.g., Thurber and Osborn 1993), we repeated all analyses using boys-only data (n = 320) and girls-only data (n = 156). The boys-only and girls-only results were nearly identical as the results based on the full sample. Across all analyses, the only difference of the gender-specific results compared to the full-sample results was the following: for the girls-only analyses, unlike the full-sample results, the Withdrawn/Depressed syndrome scale was not able to discriminate girls with depression (n = 44) from girls with anxiety (n = 46; p = .117); however, consistent with the full-sample results, the DSM-oriented Affective Problems scale was able to somewhat discriminate girls with depression from girls with anxiety (p = .002). Unlike the full-sample results, the associated AUC value of the DSM-oriented Affective Problems scale (AUC = .70) was somewhat larger than the AUC value of the Withdrawn/Depressed syndrome scale (AUC = .61; p = .038), with respect to discriminating between depressed and anxious girls. The patterns of results of all other gender-specific analyses7 were identical to the full-sample results.

Discussion

The present study evaluated and compared the degree to which the rationally-derived CBCL DSM-oriented scales and empirically-derived syndrome scales corresponded with parent-based youth DSM diagnoses. Given that expert child psychiatrists and psychologists assisted in deriving the DSM-oriented scales to be more consistent with DSM nosology, it was predicted that these scales would evidence significantly better correspondence with clinical DSM diagnoses than the syndrome scales. The present findings, however, revealed that the DSM-oriented scales generally did not evidence performance superior to that of the syndrome scales with respect to correspondence with clinical diagnoses. One explanation for this lack of increased correspondence with DSM diagnoses by the DSM-oriented scales is that these scales were derived from the limited pool of 120 items already comprising the CBCL. This likely placed a limit on the ability of the developers to create scales that align more with DSM constructs. In fact, among the six CBCL DSM-oriented scales, the Oppositional and Conduct Problems scales are the only two scales that target specific DSM disorders (i.e., ODD and CD, respectively). The remaining four DSM-oriented scales were developed to target clusters of related disorders (e.g., the DSM-oriented Anxiety Problems scale targets the cluster of GAD, SAD and SPEC anxiety disorders, as opposed to any single anxiety disorder), likely due to a limited item pool.

Interestingly, the DSM-oriented Anxiety Problems scale was the only DSM-oriented scale that evidenced a slight advantage over the syndrome scales. Despite being comprised of only 6-items, a concern raised by previous investigators (e.g., Ferdinand 2008; Kendall et al. 2007), the DSM-oriented Anxiety Problems scale evidenced significantly greater AUC values than the Anxious/Depressed syndrome scale with respect to discriminating (a) the anxiety disorders targeted by the scale (i.e., SAD, GAD, and SPEC) from all other disorders, as well as (b) the anxiety disorders targeted by the scale from affective disorders. These findings are consistent with the notion that the Anxious/Depressed syndrome scale is somewhat less specific to anxiety than the DSM-oriented Anxiety Problems scale, given that the Anxious/Depressed syndrome scale contains items also related to affective problems (e.g., “feels too guilty,” “talks about killing self,” “feels worthless and inferior”). Notably, however, the Anxious/Depressed syndrome scale also evidenced significant ANOVAs and its AUC values fell in the “fair” and “good” range, supporting this scale’s ability to also correspond significantly with anxiety problems.

Regarding assessment of disruptive behaviors, the present study demonstrated that the DSM-oriented Oppositional Problems and Aggressive Behavior syndrome scales performed similarly with respect to correspondence to ODD diagnostic status. The present study also demonstrated that the DSM-oriented Conduct Problems and the Rule-Breaking Behavior syndrome scales performed similarly with respect to corresponding with CD diagnostic status. Notably, the DSM-oriented Oppositional Problems could not discriminate between ODD and CD youths, whereas the DSM-oriented Conduct Problems scale could discriminate between these youths. This finding is consistent with the mutually exclusive diagnostic rule of ODD and CD (i.e., that a youth cannot receive both ODD and CD), as well as the notion that all the features of ODD are usually present in CD. This finding has implications pertaining to the application and interpretation of the DSM-oriented Oppositional Problems and Conduct Problems scales. Before interpreting elevations on the DSM-oriented Oppositional Problems scale as suggestive of an ODD diagnosis (or the need for further testing to determine ODD status), it is necessary to also inspect the DSM-oriented Conduct Problems scale to verify that elevations are not also present on this scale. If the DSM-oriented Conduct Problems scale is not elevated, a diagnosis of ODD may be considered; however, if the DSM-oriented Conduct Problems scale is elevated, a diagnosis of CD should instead be considered.

The gender-specific analyses revealed that the present study’s findings are generally robust to youth gender. The one potential gender-related difference is that the DSM-oriented Affective Problems scale and the Withdrawn/Depressed syndrome scale performed equally well with respect to discriminating boys with depression from boys with anxiety, whilst the DSM-oriented Affective Problems scale performed somewhat better than the Withdrawn/Depressed syndrome scale with respect to discriminating girls with depression from girls with anxiety.

Despite the DSM-oriented scales not evidencing significant advantages over the syndrome scales with respect to correspondence with clinical DSM diagnoses (except for the few instances noted above), it is notable that the 6 DSM-oriented scales are comprised of significantly fewer items than the 8 syndrome scales (i.e., 55 items versus 103 items, respectively). As public mental health systems and provider agencies move towards mandating the use of standardized measurement of youth outcomes, a shortened version of the CBCL — comprised of only the 55-itemed DSM-oriented scales — could be useful in clinical (and research) contexts, particularly when assessment battery length is a concern. More research however is needed to ensure that the DSM-oriented scales perform at least as well as the syndrome scales with respect to measurement accuracy across additional parameters and sample characteristics, such as factor structure, reliability, and age.

There were also limitations of the present study that offer directions for future research. First, there were no youths in the present sample with a somatization disorder. As a result, the DSM-oriented Somatic Problems scale could not be evaluated. Future research evaluating the concurrent validity of the DSM-oriented Somatic Problems scale is thus needed. There were also very few youths in the sample with a diagnosis of ADHD-PH, limiting the ability to evaluate the concurrent validity of the DSM-oriented ADH Problems scale with respect to this diagnosis. Further, only the CBCL, as opposed to also the Teacher Report Form (TRF) and/or Youth Self-Report (YSR), was examined in the present study. Findings related to both the TRF and YSR DSM-oriented scales would provide further understanding of the performance of these DSM-oriented scales in clinical settings, such as their correspondence with DSM diagnoses. Other questions pertaining to the applicability of the DSM-oriented scales could also be examined. For instance, knowing the degree to which the DSM-oriented Anxiety Problems scale corresponds with other anxiety disorders beyond the anxiety disorders targeted by the scale (i.e., SAD, GAD, and SPEC) would help inform its usage. Further, knowing the degree to which other parent-report measures (e.g., the Revised Child Anxiety and Depression Scale — Parent Version; Ebesutani et al. 2009) designed to map onto specific DSM constructs correspond with clinical diagnoses compared to the CBCL DSM-oriented scales may also inform whether the CBCL DSM-oriented scales achieved their aims.

Overall, although the present findings support the concurrent validity of the recently derived CBCL DSM-oriented scales, the present study did not identify significant advantages of the DSM-oriented scales over the syndrome scales with respect to correspondence with DSM diagnoses. Given that in creating the DSM-oriented scales, the developers likely sought to increase correspondence with DSM diagnoses relative to that of the syndrome scales, additional attention may be needed to explore why increased correspondence was not achieved and whether increased correspondence may be obtained via modifications to the DSM-oriented scales. Kendall et al. (2007), for example, recently attempted to improve the 6-item DSM-oriented Anxiety Problems scale by adding an additional 10 anxiety-related CBCL items to the scale. Although this did not lead to clear improvement of the scale (i.e., their new 16-item anxiety scale predicted anxiety disorders better than the original 6-item CBCL DSM-oriented Anxiety Problems scale when based on mother reports, but predicted anxiety disorders worse when based on father reports), these findings are promising with respect to the potential for enhancing the clinical utility and concurrent validity of the CBCL DSM-oriented scales via scale/item modifications. Additional similar efforts would be valuable. In the meantime, findings of the present study demonstrated that the CBCL DSM-oriented scales evidenced significant correspondence with related DSM diagnoses and are thus clinically useful alongside the syndrome and broad-band scales to incorporate into the assessment of youth emotional and behavior problems. It is encouraged that future efforts be made to increase correspondence of the DSM-oriented scales with DSM diagnoses for enhanced utility in clinical and research settings.