Reliability and Validity of a New Comprehensive Tool for Assessing Challenging Behaviors in Autism Spectrum Disorder.

OBJECTIVE: The purpose of this study was to examine the validity and reliability of the Korean Comprehensive Scale for the Assessment of Challenging Behavior in Developmental Disorders (K-CSCB).
METHODS: In total, the parents of 189 patients with autism spectrum disorder (ASD) and 168 controls completed the K-CSCB, the Behavior Problems Inventory (BPI) and Child Behavior Checklist (CBCL). The reliability and validity of the K-CSCB was investigated.
RESULTS: The K-CSCB was found to be a reliable instrument (Cronbach's α=0.97). There was a significant difference between the ASD and control groups in all subscale scores. Scores on the K-CSCB subscales were significantly correlated with those on the BPI and CBCL. The diagnostic validity was 97.7%, and the cut-off score with the highest sensitivity and specificity was 12.5 points.
CONCLUSION: The K-CSCB is the first tool in Korean to assess problematic behavior in individuals with ASD, and this study shows that it is a valid and reliable instrument. We expect the K-CSCB to be widely used in clinical and research settings.

INTRODUCTION

The number of individuals diagnosed with autism spectrum disorder (ASD) has been increasing, with estimates ranging from 0.07% to 2.64%.1234 ASD is a developmental disorder characterized by impairment in social communication and restricted/repetitive behaviors.5 Although not included in the core symptoms of ASD, behavioral problems, including aggression, temper tantrums, non-compliance, and self-injurious behaviors, are burdensome to caregivers and commonly require intervention.6 Their presence can be a major impediment to activities, socialization, and other learning opportunities and can persist into adolescence and adulthood, affecting the long-term prognosis.7 A previous study reported that one-third of children aged 1.5 to 5.8 years showed problematic behaviors in the clinically significant range.8 Some studies have reported that over 90% of patients with ASD exhibit some form of challenging behavior.9

The need for the early identification and accurate assessment of these problem behaviors has led to the development of various measures designed to assess behavioral problems in ASD. The most widely used instruments include the Aberrant Behavior Checklist (ABC),10 Behavior Problems Inventory (BPI),11 Nisonger Child Behavior Rating Scales (NCBRF),12 Behavior Assessment System for Children (BASC),13 Home Situations Questionnaire-Pervasive Developmental Disorder version (HSQ-PDD),14 and Child Behavior Checklist (CBCL).15 These instruments vary in scope, length, targeted age and population. The ABC and NCBRF were initially designed to measure problem behaviors in individuals with intellectual disabilities, and the BASC and CBCL are tools that do not target a specific psychiatric diagnosis. Most tools mentioned are broad-band instruments and, as they encompass an extensive range of psychopathology, only a few items target problem behaviors.11 Moreover, evidence on the validity and reliability of these tools for individuals with ASD is limited. A recent review of the properties of tools that measure behavior problems in children with autism reported that, among 12 tools, there was evidence available for only 6, and most showed mixed results.16 The CBCL and HSQ-PDD had the most robust measurement properties. However, the CBCL was developed to be used with typically developing children, and there was no evidence of its construct validity in children with ASD, whereas the HSQ-PDD was developed recently, and the developers are continuing to explore the most effective groupings of items.17

In South Korea, there has been increasing concern about managing the problem behaviors of patients with ASD, and the “Developmental Disabilities Assistance and Bill of Rights Act,” enacted in November 2015, announced a plan to establish nationwide behavior development enhancement centers for the management of problem behavior in patients with developmental disorders. However, despite the increasing social awareness of the strong likelihood of problem behaviors in ASD, there is no instrument in South Korea with well-established validity and reliability that can accurately assess problematic behavior in individuals with ASD. With the exception of the CBCL and BASC, none of the pre-existing instruments has been standardized in Korean,1819 and all were developed in Western countries, thus limiting their widespread use in South Korea.

Kim et al. (unpublished) developed a tool to assess challenging behavior in patients with ASD, the Korean Comprehensive Scale for the Assessment of Challenging Behavior in Developmental Disorders (K-CSCB). We pooled items from tools that were considered valid and reliable, collected information from caregivers and therapists about additional symptoms not included in the pooled items, and allowed a group of psychiatrists, educational professionals, clinical psychologists, and behavior therapists to determine face validity. The purpose of this study was to establish the reliability and validity of the K-CSCB. We recruited patients with ASD from both clinical and community settings to widen the generalizability of our study results to patients with ASD with various characteristics.

METHODS

Participants

Participants with ASD, aged 5–22 years, were recruited from hospitals and special education schools during the period of February 2016 to July 2016. The clinical group was recruited from the Seoul National University Children's Hospital and the Seoul National University Bundang Hospital. Diagnoses of ASD were confirmed by board-certified child and adolescent psychiatrists according to the Diagnostic and Statistical Manual of Mental Disorders, 4th edition (DSM-IV). Individuals with ASD who were receiving special education were recruited from six special education schools in Seoul and Incheon. Exclusion criteria were as follows: 1) schizophrenia or any other childhood-onset psychotic disorder, 2) mood disorders, 3) conduct disorder or oppositional defiant disorder, and 4) inability of the parents to understand or cooperate. The control group was defined as community-dwelling children who were free of any psychiatric diagnoses, and the exclusion criteria were the same as those applied to the ASD group, with the addition of an ASD diagnosis. We only included patients who completed more than 95% (84 out of 89 items) of the K-CSCB.

After a detailed explanation of the study, written informed consent was obtained from the participants' legal guardians, and the study protocol was approved by the Institutional Review Board of Seoul National University Hospital (1406-016-585).

Assessment

The Korean Comprehensive Scale for assessment of Challenging Behavior in Developmental Disorders (K-CSCB)

This scale was originally developed by Kim et al. (unpublished) to assess problematic behavior during the past 2 months in individuals with developmental disorders, particularly ASD. The items of this scale are listed in Supplementary Table 1 (in the online-only Data Supplement). It initially consisted of 89 items and six subcategories. There were 13 items addressing self-harm, 17 addressing aggressive behavior, 22 addressing stereotyped behavior, 13 addressing unresponsiveness and negativism, 12 addressing attention deficits and hyperactivity, and 12 addressing inappropriate words and behavior. It investigates various behaviors that may appear during the childhood, adolescence, and early adulthood of patients with developmental disorders and was designed to be completed by the primary caretaker of the patient (e.g., parent, teacher, or behavioral therapist). Each item assesses frequency on a five-point scale (from 0 to 4, with 4 being the most frequent) and severity on a four-point scale (from 0 to 3, with 3 being the most severe).

Behavior Problems Inventory

The Behavior Problems Inventory (BPI) is a 52-item narrow-band behavior rating scale for behavior problems in individuals with developmental disorders.11 It assesses the severity and frequency of problem behaviors and includes 14 items related to self-injurious behaviors, 24 items related to stereotypic behaviors, and 11 items related to aggressive/destructive behaviors. The reliability and validity of the BPI have been well established, with a full-scale Cronbach's α of 0.83, test-retest reliability of 0.76, inter-rater reliability of 0.92, and fair criterion validity as assessed by diagnoses of pervasive developmental disorder (PDD) or stereotyped movement disorders.

Child Behavior Checklist 1.5-5

The Child Behavior Checklist (CBCL) 1.5-5 is a 100-item parent-report measure that is widely used to screen for problematic behaviors; however, this instrument does not target any specific psychiatric diagnosis.20 The CBCL 1.5-5 is known to have good reliability and validity.20 The CBCL 1.5-5 is known to have good reliability and validity.2021 The raw scores on seven syndrome scales and five DSM-oriented scales are converted to normalized T-scores (mean=50, standard deviation=10); we used scores on the internalizing, externalizing, aggressive behaviors, and total problems sections to evaluate the concurrent validity of the K-CSCB. The parents of school-aged children with ASD (aged 6–12 years) completed the CBCL 1.5-5. The CBCL 1.5-5 has been found to reliable in measuring problem behaviors in ASD youth over 5 years old.22

Statistical analyses

The demographic and clinical characteristics of the ASD and control groups were compared using independent t-tests for continuous variables and chi-square tests for categorical variables. In regard of missing data, the subscales scores of individuals that were missing more than 1 item were excluded from the analyses and only subscale scores that had not missing items were included. The reliability and validity analyses were based on the severity scores of each subscale. Cronbach's α was calculated to evaluate the internal consistency of the scale and each subscale. The test-retest reliability, correlation between subscales, and correlation between the frequency and severity score of each subscale were calculated using Pearson's correlation analyses. Concurrent validity was assessed using Pearson's correlation coefficients for scores on scales widely used for the assessment of problematic behavior, the BPI and CBCL. Discriminant validity was evaluated by comparing scores in the ASD and control groups via independent t-tests. As there was a significant difference between the ASD and control groups in age and gender, we further compared the scores between the two groups using analysis of covariance (ANCOVA), with age and gender as covariates. To test the diagnostic utility and determine the cut-off score that best discriminates the ASD from the control group, we analyzed the receiver-operating-characteristic (ROC) curve and selected the score with the highest sensitivity and specificity.

All statistical analyses were performed using SPSS ver. 22.0 software (IBM Corp., Armonk, NY, USA), and statistical significance was set at a two-tailed p-value<0.05.

RESULTS

The demographic characteristics of the ASD and control groups are shown in Table 1. We recruited 215 patients with ASD; demographic data were lacking for 1, and 25 completed less than 95% of the K-CSCB. Of the 214 controls recruited, basic demographic data were lacking for 5, and 41 completed less than half the K-CSCB. In total, data from 189 patients with ASD and 168 controls were included in the analyses. The mean age was significantly higher and there were more male participants in the ASD than in the control group. Among the patients with ASD, 89.3% had a nationally registered disability grade, and 55.1% had the highest grade (grade 1), whereas only 1.1% had the lowest grade (grade 3).

Table 1

Demographic characteristics of participants

ASD: autism spectrum disorder, SD: standard deviation

The local prevalence of each item in the ASD population is presented in Table 2. All patients with ASD reported at least one problem behavior, and the subscale showing the highest prevalence was “inappropriate words or behavior” (96.4%, ranging from 13.2% to 68.3%), followed by attention deficit/hyperactivity (94.3%) and stereotypy (94.2%). The item with the highest prevalence was “short attention span” (84.7%), and the item with the lowest prevalence was “pulling body parts (eyeballs, tongue, genitalia, etc.)” (3.2%).

Table 2

Local prevalence of each item: ASD group

ASD: autism spectrum disorder

The internal consistency of the items was calculated, and six items from the self-harm category and eight items from the aggressive behavior category were deleted due to low item-total correlation (less than 0.3). As a result, the full-scale Cronbach's α was 0.97, and all subscales showed fair internal consistency, ranging from 0.73 to 0.93 (Table 3). We found significant correlations between the frequency and the severity scores of each subscale. Thirty-two participants completed the retest after an interval of 4 weeks, and all subscales except that addressing self-harm showed statistically significant test-retest reliability (p<0.05) (Table 3). There were significant correlations between each pair of subscales (p<0.01) (Table 4).

Table 3

Internal consistency, test-retest reliability, and frequency-severity correlation of K-CSCB subscales

*p<0.05, **p<0.01. K-CSCB: Korean Comprehensive Scale for the Assessment of Challenging Behavior in Developmental Disorders, F-S: frequency-severity correlation

Table 4

Correlation between subscales of the K-CSCB

*p<0.01. K-CSCB: Korean Comprehensive Scale for the Assessment of Challenging Behavior in Developmental Disorders

The K-CSCB total and subscale scores were significantly higher in the ASD than in the control group (p<0.001) (Table 5). Even after including age and gender as covariates, all results remained significant (p<0.001).

Table 5

Comparison of the K-CSCB subscales score of ASD and control participants

K-CSCB: Korean Comprehensive Scale for the Assessment of Challenging Behavior in Developmental Disorders, ASD: autism spectrum disorder

The corresponding subscales of the K-CSCB were significantly correlated with the BPI scales addressing self-injury, stereotypic behaviors, and aggressive/destructive behaviors, and the total K-CSCB score was correlated with the total BPI score (Table 6). All K-CSCB subscales, except inappropriate words and behaviors and unresponsiveness, were significantly correlated with the corresponding CBCL subscales for internalizing, externalizing, aggressive behavior, and total behavior scores. The unresponsiveness scores showed significant correlation with internalizing and total CBCL scores.

Table 6

Correlation of K-CSCB subscales with BPI and CBCL subscales

*p<0.05, **p<0.01. K-CSCB: Korean Comprehensive Scale for the Assessment of Challenging Behavior in Developmental Disorders, BPI: Behavior Problems Inventory, CBCL: Child Behavior Checklist

According to the ROC analysis, the diagnostic utility of the full instrument was significant at 97.7% (p<0.001), and the diagnostic utility of each subscale was also significant (Table 7). The cut-off score of the full instrument with the highest sensitivity and specificity was 12.5 points. At this score, the sensitivity was 90.4%, the specificity was 94.7%, the positive-predictive value was 93.4%, and the negative-predictive value was 92.2%.

Table 7

Area under ROC curve for each subscale

*p<0.001. ROC: receiver operating characteristic, AUC: area under curve, SE: standard error

DISCUSSION

The K-CSCB is the first tool developed in Korean designed to evaluate behavioral problems in ASD. It is the only instrument that includes such important items as “repeatedly opens and closes zippers or buttons on clothes,” “repeatedly pursues a particular tactile sensation (e.g., stickers, toothpaste, bubble wrap),” and “overly sensitive to auditory sensations (e.g., echoing sound of auditorium, firecrackers, speaker noise).” All of these items showed a high local prevalence in the ASD population (34.4, 28.6, and 62.0%, respectively). Another strength of this study is that we recruited participants from both clinics and special education schools. As interventions for problematic behavior can be applied not only in the clinical setting (e.g., pharmacotherapy) but also in the school setting (e.g., behavioral therapy), it is important that this instrument was validated in populations from both settings.

This study provides evidence that the K-CSCB is valid and reliable. It showed excellent internal consistency (Cronbach's α=0.97), which exceeded those of the previous aforementioned instruments (BPI=0.83, ABC=0.68–0.90, HSQ-PDD=0.80 and 0.90, CBCL 1.5–50.93).11142123 The test-retest reliability was significant for most subscales; the only exceptions, the self-harm subscale, may be attributable to the small number of participants who completed the retest, causing increased probability of type II errors. Moreover, as self-harm behavior requires more immediate intervention compared to other behaviors, the stability of this construct could vary over time. Further studies with a larger sample size are needed to confirm the test-retest reliability of the K-CSCB. The K-CSCB also showed good discriminant validity and concurrent validity, especially with the BPI (r=0.77).

Whereas most scales measure only the severity of a problem behavior, the K-CSCB simultaneously measures severity and frequency. This may be useful in clinical situations where the severity and frequency of a problem behavior do not match, as a severe problem may be infrequent or a frequent problem may not be severe. However, as the frequency and severity scales showed a strong correlation with each other, little may be gained from retaining both scales, and it may be possible to eliminate one of them. We suggest the flexible use of both scales according to the purpose of data collection.

Total scores on the K-CSCB range from 0 to 225; in the ASD population in our study, the lowest score was 3 and the highest score was 120. All participants in this group reported more than one problematic behavior, which is a higher rate than found in previous studies. A study using the BPI found that three-quarters of the sample endorsed at least one item; the highest prevalence rate among items was 22.2%,11 whereas the highest in this study was 84.8%. The high prevalence of problem behaviors in this study may have been due to the fact that the ASD population was over-represented by individuals with high severity, as more than half had a disability grade of 1 (highest grade). However, comparison with the results of other studies may be difficult due to differences in study populations, sampling strategies, and study designs. As the prevalence of problematic behavior among patients with ASD has not been studied in South Korea, the K-CSCB could be a valuable tool in future epidemiological research.

Kalb and Loeber reported that 25% to 65% of children aged 2–16 years of age have problems with noncompliance.24 Among children with autism, compliance with instructions may be particularly important because of the many requests to comply with adult directions that are issued during early behavioral interventions.25 Despite the high rate of noncompliance in patients with ASD, the HSQ-PDD is the only accepted tool that measures behavioral non-compliance in children with ASD.14 Although the items are not as specific as those in the HSQ-PDD, some of the items on the “unresponsiveness/negativism” subscale of the K-CSCB may reflect non-compliance in patients with ASD (e.g., no response to verbal instruction, no response to structured activities). As the BPI and CBCL do not measure noncompliance per se, further research investigating concurrent validity with HSQ-PDD are needed.

ASD is associated with high rates of comorbidity with attention-deficit/hyperactivity disorder (ADHD), and 30–80% of those with ASD meet the criteria for ADHD.25 Owing to the overlapping characteristics and etiological factors, Rommelse has convincingly argued that the two disorders should be systematically studied together.2627 There was also a high rate of ADHD traits in the ASD population in this study. The most prevalent item was “short attention span,” and 94.3% of the children with ASD had affirmative responses to at least one item in the “attention deficit/hyperactivity” subscale. Thus, we propose that the K-CSCB may have a promising role in future research involving comorbid ASD and ADHD.

There were some noteworthy limitations to this study. First, although the patients with ASD recruited by hospitals were diagnosed by child and adolescent psychiatrists, we relied on parent reports for confirmation of the ASD diagnosis of the participants recruited by special education schools. There was also a significant difference between the ASD and control groups in age and gender. We tried to compensate for this by adding age and gender as covariates, and this did not alter the significance of the results. We used the CBCL 1.5–5 version in school-aged ASD children. However, as the developmental age of ASD children are lower compared to typically developed children, the use of the CBCL 1.5–5 in ASD may capture the behavioral problems in school-age ASD individuals well. Finally, the participants consisted primarily of children and adolescents, which limits the generalizability of our findings to adults with ASD.

The K-CSCB is a new tool for the assessment of problem behavior in patients with ASD, and this study provides evidence for its validity and reliability. As this is the first tool in Korean, we expect that this tool will be widely used in clinical and research settings in South Korea.