Further Evidence of the Diagnostic Utility of the Child Behavior Checklist for Identifying Pediatric Bipolar I Disorder.

BACKGROUND: Pediatric bipolar (BP) disorder is a prevalent and highly morbid disorder. While structured diagnostic interviews have been developed to aide in the diagnosis of pediatric BP disorder, these tools are lengthy, costly, and not widely available. One possible diagnostic aid is the Child Behavior Checklist (CBCL).
OBJECTIVE: To assess the diagnostic utility of the Child Behavior Checklist (CBCL)-Bipolar (BP) profile to identify children with a diagnosis of BP-I disorder.
METHOD: Subjects were derived from four independent datasets of children and adolescents with and without attention deficit hyperactivity disorder (ADHD) and BP-I. Subjects were recruited from pediatric and psychiatric clinics and the community. All subjects had structured clinical interviews with raters blinded to subject ascertainment status. We used an empirically-derived profile from the CBCL consisting of an aggregate t-score from the Attention, Anxiety/Depression, and Aggression subscales (CBCL-BP profile) to operationalize the presence or absence of bipolar symptoms. Receiver operating characteristic (ROC) curves were used to examine the ability of the CBCL-BP profile to identify children with and without a structured interview diagnosis of BP-I disorder.
RESULTS: The sample consisted of 661 subjects (mean age: 11.7 ± 3.3 years, 57% male, and 94% Caucasian). Twenty percent of participants (N=130) met structured interview criteria for a full diagnosis of BP-I disorder. The ROC analysis of the CBCL-BP profile yielded an area under the curve of 0.91. A t-score of ≥195 on the CBCL-BP profile correctly classified 86% of subjects with BP-I disorder with 80% sensitivity, 87% specificity, 61% positive predictive value, 95% negative predictive value.
CONCLUSION: The CBCL-BP profile efficiently discriminated pediatric subjects with and without a structured interview diagnosis of BP-I disorder. Findings suggest that the CBCL-BP profile may be an efficient tool to help identify children who are very likely to suffer from BP-I disorder.

Introduction

Pediatric bipolar (BP) disorder is a prevalent and highly morbid disorder estimated to afflict up to 2% of youth worldwide (1, 2). Youth afflicted with BP disorder are at high risk for a wide range of adverse outcomes including psychiatric hospitalizations, substance use disorders and suicidality (3–7). However, because the diagnosis of pediatric BP disorder requires a level of clinical expertise that is not readily available, improved efforts to help identify children who may have BP disorder could greatly facilitate the identification of a sizeable group of children at high risk for adverse outcomes.

While structured diagnostic interviews have been developed for research studies to aid in the diagnosis of pediatric BP disorder, these tools are not practical to use in clinical practice since they are lengthy, costly and not widely available. Consequently, simpler diagnostic aids are needed to help identify children who may have pediatric BP disorder.

One possible diagnostic aid is the Child Behavior Checklist (CBCL) (8). The CBCL is an empirically derived broadband assessment tool of psychopathology with excellent psychometric properties that has been translated into over 100 languages and is an easy to use paper and pencil instrument. A body of research (9–12) and a meta-analysis (13) have shown very high correspondence between a unique profile of the CBCL consisting of elevations in the Attention, Anxiety/Depression and Aggression subscales greater than two standard deviations (SDs) above the norm (≥ 210 combined) with structured diagnostic interview derived clinical diagnosis of pediatric BP disorder (hence termed the CBCL-BP profile). However, because some studies failed to find an association between the CBCL-BP profile and a diagnosis of pediatric BP disorder, (14–18), there is a clear need for further evaluation of the utility of the CBCL to help identify children who may have BP disorder.

Further evidence as to whether the CBCL-BP profile can help identify children who may have pediatric BP disorder has important implications. Considering the high morbidity and disability associated with BP disorder and its unique therapeutic needs, evidence supporting the CBCL-BP profile’s diagnostic utility to help identify children suspected of suffering from BP disorder could be very useful for mental health practitioners and primary care physicians worldwide. Whether a child is affected with BP disorder is particularly relevant in the differential diagnosis of a child presenting with mood instability. If a child suffers from BP disorder and is started on an antidepressant or stimulant medication instead of an anti-manic treatment, they are at risk for worsened symptoms including suicidality and psychiatric hospitalization (19–22).

The main aim of this study was to re-examine the diagnostic efficiency of the CBCL-BP profile for identifying pediatric BP-I disorder in a large sample of children with and without BP-I disorder. To this end, we applied conditional probability analysis and a receiver operator curve (ROC) analysis to four large data sets of children with ADHD, BP-I disorder and controls of both sexes that included close to 700 youth. We also sought to determine the best cutoff point for the CBCL-BP profile for predicting a structured diagnostic interview derived diagnosis of pediatric BP-I disorder. Based on the literature and our previous work, we hypothesized that the CBCL-BP profile would be highly predictive of a clinical diagnosis of pediatric BP-I disorder.

Patients and methods

Sample

The sample was derived from four independent studies using identical assessment methodology: 1) and 2) were prospective controlled family studies of boys and girls 6 to 17 years of age with and without DSM-III-R ADHD (Boys Study: n = 140 ADHD and n = 120 Controls; Girls Study: n = 140 ADHD and n = 122 Controls) (23, 24); 3) was a prospective controlled family study of youth 10 to 18 years of age with (n = 105) and without (n = 98) DSM-IV pediatric BP-I disorder (25); and 4) was a prospective family study of youth 6 to 17 years of age of both sexes with active symptoms of DSM-IV BP-I disorder (n = 105) (4). The ADHD studies recruited subjects from pediatric and psychiatric clinics. The BP-I disorder studies recruited subjects from referrals to the Clinical and Research Programs in Pediatric Psychopharmacology at the Massachusetts General Hospital and through advertisements in the community. Controls were recruited from pediatric clinics, advertisements to hospital personnel and community newspapers and internet postings. All studies excluded adopted subjects, subjects where the nuclear family was not available, subjects with major sensorimotor handicaps, autism, inadequate command of the English language or full scale IQ < 70 (< 80 for ADHD studies). Potential subjects were also excluded from the ADHD studies if they had a primary psychotic disorder and from the BP-I disorder studies if their BP-I disorder was due solely to a medication reaction. For all four studies, parents provided written informed consent to participate. Children and adolescents provided written assent to participate. The Partners Human Research Committee approved these studies.

Assessment procedures

In all four studies, psychiatric assessments of subjects were made with the Kiddie Schedule for Affective Disorders – Epidemiologic Version (KSADS-E) (26–28). The K-SADS-E is a semi-structured interview tool used in research that assesses for the presence or absence of past and current symptoms within diagnostic categories. Extensively trained and supervised psychometricians with undergraduate degrees in psychology conducted all interviews. For the ADHD studies and the controlled BP disorder study, raters were blind to the ascertainment status of the families. For the BP disorder family study, raters were blind to the study assignment and whether the subject was a proband or sibling.

Diagnoses were based on independent interviews with parents and direct interviews with children older than 12 years of age. Data were combined such that endorsement of a diagnosis by either reporter resulted in a positive diagnosis. For the sample used in this study, 46.9% (n = 310) of the assessments were based on the parent interview only. For the assessments where parents and children were both interviewed there was agreement between reporters in 30.4% of subjects (n = 201), disagreement between reporters in 4.8% of subjects (n = 32), and no information available on the presence or absence of agreement between reporters for 17.9% of subjects (n = 118). In the subjects where there was disagreement between parents and children, 28 diagnoses of BP disorder were made based on parent report, and four diagnoses of BP disorder were made based on child report.

To assess the reliability of our overall diagnostic procedures, we computed kappa coefficients of agreement by having experienced, blinded, board-certified child and adult psychiatrists and licensed experienced clinical psychologists diagnose subjects from audiotaped interviews made by the assessment staff. Based on 500 assessments from interviews of children and adults, the median kappa coefficient was 0.98 for the ADHD studies and the controlled BP disorder study and 0.99 for the BP disorder Family study.

Socioeconomic status (SES) was measured using the 5-point Hollingshead scale. A higher score indicates being of lower socioeconomic status (29).

CBCL

The parent of each participant completed the 1991 version of the CBCL for ages 4 to 18 years. The CBCL queries the parent about the child’s behavior in the past six months and aggregates this data into behavioral problem t-scores (8, 30). A computer program calculates the t-scores for each scale. Raw scores are converted to gender and age standardized scores (t-scores having a mean of 50 and SD of 10). A minimum t-score of 50 is assigned to scores that fall at percentiles of ≤ 50 on the syndrome scales to permit comparison of standardized scores across scales. t-scores above 70 (2 SD) indicate clinical disorder. Subscales include Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints, Social Problems, Thought Problems, Attention Problems, Rule-Breaking Behavior and Aggressive Behavior.

Statistical analysis

We first compared demographic characteristics between subjects with and without BP-I disorder in each of the studies separately and then in the combined sample using Student’s t-test for continuous outcomes, Pearson’s χ2 test or Fisher’s exact test (for expected counts < 5) for binary outcomes and Wilcoxon rank-sum test for ordinal outcomes. Next, we calculated conditional probabilities for each of the studies separately using a conservative cut-off point of ≥ 180 for the CBCL-BP profile. We subsequently combined the data from the four studies and used receiver operating characteristic (ROC) curves to examine the ability of the CBCL-BP profile to identify those with and without a structured interview diagnosis of BP-I disorder. ROC analysis uses each value across the entire range of scores for the CBCL-BP profile as the cutoff for defining a case and compares this classification to the “true” diagnosis, as defined by the clinical interview. The ROC analysis then plots the false positive rate (1-specificity) and the true positive rate (sensitivity) for each value of the CBCL-BP profile on the x- and y-axis, respectively, to create the ROC curve. Starting in the lower right-hand corner of the plot, each successive point corresponds to an increase in one point of the CBCL-BP profile. ROC analysis summarizes diagnostic efficiency with the AUC statistic. An AUC of 0.5 means the test does not predict the disorder in any way and an AUC of 1.0 means the test predicts the disorder perfectly. We used conditional probabilities to examine the diagnostic utility of various cutoff points. For each cutoff, we calculated sensitivity, specificity, PPV, NPV and the percent correctly classified.

Furthermore, using the same methods as described above, we performed a sensitivity analysis in a restricted sample consisting of subjects for whom there was no disagreement regarding the presence or absence of BP-I disorder (n = 511). The restricted sample included subjects where the K-SADS-E assessment was based on parent interview only and subjects where there was agreement between parent and child report.

All analyses were performed using Stata® (Version 14).

Results

Demographic characteristics of the sample

Subjects from the four original independent samples were only included in this sample if a CBCL was completed. Table 1 shows the demographic details from the four contributing samples. In examining differences between children with and without BP-I disorder in the controlled family study of BP-I disorder, children with BP-I disorder were more likely to be Caucasian (p ≤ 0.001). There were no meaningful differences between children with and without BP-I disorder in the ADHD studies. In the combined group that included the four contributing samples, children with BP-I disorder were more likely to be male (p = 0.002). No other meaningful differences were found within the combined sample in age, socioeconomic status or race.

TABLE 1.

Demographic characteristics of those with and without BP-I disorder from the individual studies and all studies combined

	Boys ADHD study		Girls ADHD study		BP disorder controlled study		BP disorder family study		All studies combined
	No BP disorder	BP disorder	No BP disorder	BP disorder	No BP disorder	BP disorder	No BP disorder	BP disorder	No BP disorder	BP disorder
	n = 238	n = 13	n = 211	n = 9	n = 82	n = 64	n = n/a	n = 44	n = 531	n = 130	Test statistic	p-value
	Mean ± SD	Mean ± SD	Mean ± SD	Mean ± SD	Mean ± SD	Mean ± SD	Mean ± SD	Mean ± SD	Mean ± SD	Mean ± SD
Age	11.1 ± 3.3	10.3 ± 3.3	11.7 ± 3.2	11.0 ± 3.3	13.5 ± 2.1	13.7 ± 2.4	n/a	9.2 ± 3.4	11.7 ± 3.2	11.7 ± 3.6	t659 = 0.08	0.94
Socioeconomic status[a]	1.7 ± 0.9	2.1 ± 1.0	1.8 ± 0.9	1.8 ± 0.7	1.6 ± 1.0	1.9 ± 1.2	n/a	1.8 ± 1.0	1.7 ± 0.9	1.8 ± 1.1	z = -1.29	0.20

Note. ADHD = attention deficit hyperactivity disorder; BP = Bipolar

Socioeconomic status (SES) was measured using the 5-point Hollingshead scale. A higher score indicates being of lower SES. Not everyone has SES reported. Smaler sample sizes for this measure were: boys ADHD study: 251; girls ADHD study: 219: BP controlled study: 108; all studies combined: 622.

Not everyone has race reported. Smaller sample sizes for this measure were: girls ADHD study: 215, BP disorder controlled study: 139; all studies combined: 649.

Significant difference in percent Caucasian between those with and without bipolar disorder in the BP-I disorder controlled study, p < 0.001

Conditional probability analysis

As shown in Table 2, similar values of sensitivity, specificity and percent correctly classified with BP-I disorder were observed in each individual study using the conservative cut-off point of ≥ 180 for the CBCL-BP profile (Table 2).

TABLE 2.

Sensitivity, specificity and percent correctly classified using the Child Behavior Checklist-Bipolar (BP) profile with ≥ 180 to identify youth with BP-I disorder in each study

Study	Sensitivity (%)	Specificity (%)	Correctly classified (%)
Boys ADHD study (n = 251)	85	74	74
Girls ADHD study (n = 220)	89	74	75
BP disorder controlled study (n = 146)	89	93	91
BP disorder family study (n = 44)	91	n/a[a]	91

Note. ADHD = attention deficit hyperactivity disorder

Specificity could only be calculated for studies that included control subjects

ROC analysis

Since all the studies used nearly identical methodology and assessments and had similar conditional probability analysis results, we combined data from the four samples for this analysis to improve statistical power. Thus, our combined sample consisted of 661 subjects, of which 130 (19.7%) had BP-I disorder. Figure 1 depicts the combined t-scores from the four studies for the CBCL-BP profile that yielded an AUC of 0.91.

FIGURE 1.

Receiver operating characteristic (ROC) curve of the Child Behavior Checklist-Bipolar Profile T-scores in subjects from the total sample with and without Bipolar I disorder (n = 661)

Examination of the performance of specific cut off t-scores that correspond to 0.5 SD increases on the CBCL-BP profile to correctly identify subjects with a diagnosis of BP-I disorder showed that a cut-point of 195 had the best properties as determined by the AUC with 80% sensitivity, 87% specificity, 61% PPV, 95% NPV, and 86% correctly classified with BP-I disorder (Table 3).

TABLE 3.

Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and percent correctly classified in the use of the Child Behavior Checklist-Bipolar (BP) profile t-scores to identify youth with BP-I disorder in the total sample from all four studies (n = 661)

CBCL-BP profile Cut-point	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Correctly classified (%)
≥ 165	95	63	39	98	69
≥ 180	89	77	49	97	79
≥ 195[*]	80	87	61	95	86
≥ 210	61	94	72	91	88
≥ 225	39	97	79	87	86
≥ 240	15	99	86	83	83

A cut-point of 195 for the CBCL-BP profile had the best properties to correctly identify subjects with a diagnosis of BP-I disorder as determined by the AUC

Sensitivity analysis using a restricted sample

We found similar findings in our sensitivity analysis using the restricted sample as we did in the full sample. In the combined sample there were 87 subjects (17%) who had BP-I disorder. There was no significant difference (p = 0.06) in mean age between subjects with BP-I disorder (10.3 ± 3.5 years) and those without BP-I disorder (11.0 ± 3.2 years). The ROC analysis of the CBCL-BP profile yielded an AUC of 0.93. A t-score of ≥ 195 on the CBCL-BP profile was the best cut-off score and correctly classified 87% of subjects with BP-I disorder with 86% sensitivity, 87% specificity, 58% PPV and 97% NPV.

Discussion

ROC analysis using data from four independent large data sets of children with and without a structured interview derived diagnosis of BP-I disorder showed that a combined t-score of 195 on the CBCL-BP profile efficiently identified children with pediatric BP-I disorder. These results support and extend previous results (9–13) and provide strong evidence that the CBCL-BP profile is a useful tool to help identify youth who may have BP-I disorder.

Although our results demonstrating the very high efficiency of the CBCL-BP profile in identifying children with pediatric BP-I disorder are consistent with previous research in a sample of youth with and without ADHD and their siblings (10), in a sample of Brazilian children (12), and in a meta-analysis (13), data from two community samples of treatment-seeking youth did not find an association between the CBCL-BP profile with pediatric BP spectrum disorder (18, 31). Although the reasons for these discrepancies are not entirely clear, several explanations are plausible. Subjects in these community samples were assessed by community clinicians with varied clinical skills and diagnostic traditions without the benefit of a structured diagnostic interview (18, 31) raising the possibility that differences in assessment methodology may have accounted for the negative findings. It is also possible that CBCL-BP profile may be less accurate in children with pediatric BP spectrum disorders relative to children with more narrowly defined BP-I disorder. For example, Diler et al. reported more modest ROC results in a sample with a broader BP disorder phenotype in which the AUC for the CBCL-BP profile was moderately accurate (0.72–0.78) (16).

It is also noteworthy that while Volk et al. failed to find an association between a ≥ 210 cut-off on the CBCL-BP profile and a diagnosis of pediatric BP disorder in a large population sample of twins (14), this profile was associated with significant morbidity including social and school problems as well as increased suicidal behaviors. As suggested by the findings of our analysis showing that the optimal cut-off score was ≥ 195, it is possible that the absence of an association between the CBCL-BP profile and pediatric BP disorder in the Volk et al. study may be related to their use of a higher cut-off score of 210 for the profile. More work is needed to further evaluate these issues.

While the AUC for the CBCL-BP profile was very high in our analysis, the modest PPV suggests some children with an elevated CBCL-BP profile will not meet criteria for BP-I disorder when evaluated clinically. On the other hand, the high NPV value indicates that a clinician can be confident that a child does not have BP-I disorder if the CBCL-BP profile is not elevated. Nevertheless, it is important to stress that the CBCL-BP profile should not be used as a diagnostic tool but rather as a diagnostic aid.

Our findings documenting the very high efficiency of the CBCL-BP profile to identify children with a structured interview derived diagnosis of BP-I disorder has important clinical implications. With the limited behavioral health resources available worldwide, the CBCL-BP profile can greatly facilitate the identification of children at very high risk for BP disorder in community mental health clinics and within the primary care setting. The early identification of children at-risk for BP disorder can lead to the implementation of early intervention strategies that could mitigate the poor outcomes associated with pediatric BP disorder. Furthermore, the wide availability of the CBCL allows for cross-cultural identification of children with the CBCL-BP profile, which could be used to improve our understanding of this phenotype across the world.

Our findings need to be viewed with consideration of some methodological limitations. Although the raters who administered the structured interview were highly trained and supervised, they were not clinicians. Despite this, there were strong kappa coefficients of agreement between the lay interviewers and expert clinicians. Moreover, in one of the source studies (the family study of BP disorder), the diagnosis of BP-I disorder relied on a clinical assessment by an experienced clinician with expertise in pediatric BP-I disorder (JW). Finally, since part of the sample was referred and mostly Caucasian, our findings may not generalize to community samples and other ethnic groups. Despite these limitations, our work suggests the CBCL-BP profile is an efficient, simple to use tool that can help identify children who may have pediatric BP-I disorder who may benefit from further clinical assessment.