The validities and efficiencies of korean ADHD rating scale and korean child behavior checklist for screening children with ADHD in the community.

OBJECTIVE: The purpose of this study is to examine the validity of primary screening tools for attention deficit hyperactivity disorder (ADHD) in a community-based sample of children using the Korean version of the Child Behavior Checklist (K-CBCL) and the Korean version of the ADHD Rating Scale (K-ARS).
METHODS: A large-scale community-based study for ADHD screening was conducted in the Jeollabuk province in the Republic of Korea. In 2010-2011, we surveyed a total of 49,088 first- and fourth-grade elementary school students. All of the participants in this study were assessed by the K-ARS-Parent version (K-ARS-P) and the K-ARS-Teacher version (K-ARS-T) as the primary screening instruments. The Diagnostic Interview Schedule for Children Version IV (DISC-IV) was used for confirming the diagnosis of ADHD. DISC-IV was administered to subjects who received top 10% scores in the K-ARS-P or K-ARS-T tests.
RESULTS: Of the 3,085 subjects who completed the DISC-IV, 1,215 were diagnosed as having ADHD. A reasonable level of sensitivity, specificity, and negative predictive value were obtained when the total K-ARS-P scores were ≥90th percentile. The positive predictive value and specificity increased significantly when the total K-ARS-P scores were ≥90th percentile, T scores were ≥60 in the attention problems of K-CBCL, and T scores were ≥63 in the total problems of K-CBCL.
CONCLUSION: These results suggested that the K-ARS-P could effectively serve as a primary screening tool to identify elementary school children with ADHD in the community. Also, there might be some increment in the effectiveness of K-ARS-P when combined with K-CBCL-A and K-CBCL-T as a secondary screening tool.

INTRODUCTION

Attention-deficit hyperactivity disorder (ADHD) is characterized by inattentiveness, hyperactivity, impulsivity, or a combination of these features. According to the Diagnostic and Statistical Manual of Mental Disorders, fourth edition (DSM-IV), approximately 3-5% of school-aged children are affected by ADHD;1 however, recent studies suggest that the prevalence of ADHD is between 2.2 to 17.8%.2 A similar prevalence has been reported in a recent research conducted by the Seoul Metropolitan Child and Adolescent Mental Health Center.3 The DSM-IV identifies four subtypes of ADHD: combined type, predominantly inattentive type, predominantly hyperactive/impulsive type, and not otherwise specified type.1 ADHD is the most common behavioral disorder in school-age children, and it is frequently accompanied by many serious complications, such as conduct disorder and depression.4

A follow-up study of children with ADHD showed that children with ADHD, compared to other children, are more likely to have a significant impairment in social, academic, or occupational functioning.5,6 Recent studies suggest that onethird of children with ADHD have co-morbid anxiety neurosis or depressive disorder and even develop alcohol or drug addiction.4,6,7 Due to their inattentive and impulsive behavior, children with ADHD have difficulty in following rules and having good relations with peers at school. Hence, if these problems in the classroom occur frequently, they can become highly burdensome for teachers.8,9 Consequently, ADHD leads to severe problems in life, including job failure and social dysfunction in adulthood.10,11,12 Based on previous studies, it is estimated that 30-70% of individuals diagnosed with ADHD during childhood continue to have symptoms of ADHD in their adult life.5,13,14,15,16,17 Furthermore, the quality of life of not only the children with ADHD but also that of their families is negatively affected.5,18,19 Therefore, early screening and proper management of children with ADHD, such as intervention by experts in the field, can contribute to controlling many of the psychological and social problems that accompany ADHD, and improve the quality of life.

The most fundamental step towards improving child and adolescent mental health should be the recognition of the importance of mental health in children as well as the implementation of early-screening of high-risk children, and proper intervention and treatment. Fortunately, the Republic of Korea has developed an extensive project for improving child and adolescent mental health to solve mental health problems in children at the national level. However, projects for early screening of children with ADHD have been implemented in Seoul and the metropolitan area, whereas there is a lack of proper implementation in rural areas, including cities around the Jeollabuk province.

It has been reported that the Child Behavior Checklist (CBCL), 20,21 a rating scale for assessing children who have behavioral and emotional problems, is a useful and valid tool to evaluate children with ADHD.22,23 In addition, the parent version or the teacher version of the ADHD rating scale (ARS),24 based on the DSM-IV diagnostic criteria for ADHD, is considered to be an economical and effective tool for ADHD screening.25,26

The purpose of this study is to examine the validity of primary screening tools for attention deficit hyperactivity disorder (ADHD) in a community-based sample of children using the Korean version of the Child Behavior Checklist (K-CBCL) and the Korean version of the ADHD Rating Scale (K-ARS).

METHODS

Subjects

With support from 14 different school administrations, a total of 49,088 students in the Jeollabuk province participated in this study. The study participants consisted of three distinct groups: first graders who were surveyed during 2010 (1st group, n=15,725), first graders who were surveyed during 2011 (2nd group, n=15,165), and fourth graders who were surveyed during 2011 (3rd group, n=18,198). At first, we only recruited 1st grade students. Since there could be differences in ADHD characteristics and cut-off values for the K-ARS, we recruited students from all grades. However, there were problems in conducting the study due to an extremely large sample size. Therefore, we decided to include only 4th grade students instead. Informed consent was obtained from the parents, and assent was obtained from the participating children prior to their inclusion in the study. This study was approved by the Chonbuk National University Hospital Review Board.

Instruments

Korean ADHD Rating Scale for teachers or parents

In order to assess the severity of ADHD symptoms in school-aged children, the ADHD Rating Scale-IV (ARS), which is based on the DSM-IV diagnostic criteria and consists of 18 items, was designed.24 The Korean version of ADHD Rating Scale-IV was developed, and standards were established. The content of the Korean version was divided into odd-numbered items that reflected the symptoms of inattention, and even-numbered items that reflected the symptoms of hyperactivity and impulsivity, with nine items in each category. All of the scores were included in the total score.26 Either parents or teachers were allowed to mark the scale. In a previous Korean study, the K-ARS and the K-CBCL showed high inter-rater reliability and concurrent validity.26,27

Korean Version of Child Behavior Checklist

The children were evaluated using the Korean version of the Child Behavior Checklist (K-CBCL).28 The CBCL crafted by Achenbach20,21 was redesigned to develop a Korean version, the K-CBCL. This is a frequently-used questionnaire with 121 items, and provides parent-reported data on behavioral problems in children. A total problem behavior score was computed by summing the scores obtained for each item. The raw score of K-CBCL can be converted to a T-score, which has a mean of 50 points and a standard deviation of 10 points. The K-CBCL has total 13 sections, and lists externalizing behavior problems, which are composed of attention problems and aggressive and delinquent behavior, and internalizing behavior problems which comprise withdrawal, depressed behavior, and somatic complaints. The social functioning scale includes socialization and academic functioning in the K-CBCL. The K-CBCL showed a high inter-rater reliability and concurrent validity in the Republic of Korea.28

Diagnostic Interview Schedule for Children Version IV

The Diagnostic Interview Schedule for Children (DISC) is a highly structured diagnostic interview that was designed to be used by clinicians and non-clinicians. Its first incarnation was created by the National Institute of Mental Health (NIMH) in 1979, and the current version, the DISC-IV,29 has been available since 1997. Revisions in the latest version were designed to aid in the diagnosis of more than 30 psychological disorders found in children and adolescents, based on the criteria in the DSM-IV and the International Classification of Disease-tenth edition (ICD-10),30 The DISC-IV contains 6 modules assessing psychiatric disorders: anxiety disorders, mood disorders, schizophrenia, disruptive behavioral disorders, alcohol and substance abuse, and other disorders (e.g., eating disorders, tic disorders and trichotillomania). Recent domestic studies found that the Korean version of the DISC-IV shows good reliability and excellent validity for the diagnosis of ADHD in children of the Republic of Korea.31

Procedures

Orientation for ADHD survey

The overall orientation for our study was provided to nurse teachers or head teachers of elementary schools in the Jeollabuk province. The outline of our study and general information about ADHD, including the prevalence, symptoms, proper management, screening methods, were included in the orientation.

Primary screening or phase I

The primary screening was carried out by using the ARS for teachers or parents from May to June in 2010 and 2011. The K-ARS for teachers and parents, a total of 55,445 questionnaires, along with instructions was distributed to the class teachers of subjects. The class teachers completed the teacher version of the K-ARS (K-ARS-T). With help from the class teachers, the parent version of the K-ARS (K-ARS-P) along with instructions was distributed to the parents of subjects. We obtained a total 49,088 completed questionnaires from the subjects (response rate, 88.5%). The subjects, in whom the total parent or teacher K-ARS scores were at or above the 90th percentile cut-off point, were identified as potential candidates for participation in the next phase of the study.

Secondary screening or phase II

Selection of subjects

After the primary screening, we selected 8,546 subjects as participants for the secondary screening test. Among these potential candidates, we obtained the consent from the parents of 3,085 subjects (1st group, n=1,221; 2nd group, n=961; 3rd group, n=903) for their participation in this phase.

Selection of the DISC-IV interviewer

Before conducting the secondary screening, we selected the interviewers for conducting the DISC-IV, the secondary screening tool. The interviewers were selected from graduate students of the counselling department of Jeonju University, and the nursing and psychology departments of Chonbuk National University. The raters (psychiatric specialists with clinical experience of DISC-IV and experts who had previously participated in the study by Yang and colleagues32) participated in a consensus meeting for obtaining favorable concordant validity. The consensus meeting consisted of an observation of the administration of the evaluation by an experienced supervisory psychiatrist and an actual administration via videotapes.

Implementation of the Secondary screening

The parents of the subjects completed the DISC-IV through an interview in school or at the education office of the region. For the second time they completed the K-ARS-P, with the help of the same form that was used for primary screening and the K-CBCL. In addition, K-ARS and K-CBCL that were completed by the same people during secondary screening were used to examine validity and reliability as well as to investigate symptoms other than those related to ADHD. The secondary screening was carried out from July to August in 2010 and 2011.

Data analysis

We used descriptive statistics to investigate the demographic characteristics. The t-test was used to compare the K-ARS or K-CBCL scores between the ADHD group and non-ADHD group. Pearson's correlation analysis of K-ARS with K-CBCL was performed. To evaluate the discriminant validity of K-ARS and K-CBCL (K-CBCL-A; Korean Child Behavior Checklist attention problems, K-CBCL-E; Korean Child Behavior Checklist externalizing problems, K-CBCL-T; Korean Child Behavior Checklist total problems) in the diagnosis of ADHD, we examined the sensitivity, specificity, positive predictive value, and negative predictive value by analyzing the receiver operating characteristic (ROC) curves. All analyses were conducted with SPSS version 12.0 (p<0.05).

RESULTS

Three thousand and eighty-five subjects (36.1%) agreed to participate in the secondary screening. The K-ARS-T and the K-ARS-P scores in participants were significantly higher than those in non-participants. Although there was no gender difference between participants and non-participants in group 1, more number of boys participated in the secondary screening. Of the 3,085 subjects, 1,215 subjects (39.4%) were diagnosed as having ADHD. There was no gender difference in the diagnosis of ADHD. Most of the information was obtained from the subject's mother (81.8%), followed by the subject's father (10.7%), the subject's grandparents (5.1%), and others (2.4%). Socioeconomic status of ADHD subjects was lower than that of non-ADHD subjects. When compared to non-ADHD subjects, the K-ARS-T, K-ARS-P, K-CBCL-A, K-CBCL-E, and K-CBCL-T scores were significantly higher in the ADHD subjects. We generated ROC curves and calculated the area under the curve (AUC) because of the significant correlations of these subscales with K-ARS. Figure 1 shows the ROC curves of K-ARS and K-CBCL. Table 1 demonstrates the AUC of K-ARS and K-CBCL for the 1st, 2nd and 3rd group. In all groups, the AUC of all scales was more than 0.5. The AUC of K-ARS-P was the highest, whereas the AUC of K-ARS-T was low compared to that of other scales. These results were similar to those in the study by Yang and colleagues, in which the AUC of K-ARS-T was also low (AUC<0.5).32

Figure 1

ROC curves of K-ARS-T, K-ARS-P, K-CBCL-A, K-CBCL-E, K-CBCL-T. 1st group: first graders surveyed during 2010, 2nd group: first graders surveyed during 2011, 3rd group: fourth graders surveyed during 2011. K-ARS-T: Korean ADHD Rating Scale Teacher version, K-ARS-P: Korean ADHD Rating Scale Parent version, K-CBCL-A: Korean Child Behavior Checklist attention problems, K-CBCL-E: Korean Child Behavior Checklist externalizing problems, K-CBCL-T: Korean Child Behavior Checklist total problems.

Table 1

AUC in K-ARS and K-CBCL in the study sub-groups

1st group: first graders surveyed during 2010, 2nd group: first graders surveyed during 2011, 3rd group: fourth graders surveyed during 2011. AUC: area under the curve, K-ARS-T: Korean ADHD Rating Scale Teacher version, K-ARS-P: Korean ADHD Rating Scale Parent version, K-CBCL-A: Korean Child Behavior Checklist attention problems, K-CBCL-E: Korean Child Behavior Checklist externalizing problems, K-CBCL-T: Korean Child Behavior Checklist total problems

To estimate the discriminant validity of K-ARS and K-CBCL as screening tests for ADHD, the sensitivity, specificity, positive predictive value and negative predictive value were obtained and comprehensively assessed (Table 2). The 90th percentile cut-off scores of K-ARS-T were 15, 16, and 13 in the 1st, 2nd, and 3rd group, and those of K-ARS-P were 16, 17, and 16 in the 1st, 2nd, and 3rd group, respectively. We selected only a few subscales for the screening tests for efficient evaluation. Since it is important to study attention problems during ADHD evaluation,33,34 K-CBCL-A was included in the analysis. Although it is well-known that an externalizing disorder such as oppositional defiant disorder or conduct disorder coexists with ADHD, an internalizing disorder such as anxiety disorder has also been found to co-exist with ADHD.35,36 During ADHD evaluation, both externalizing symptoms and internalizing symptoms should be considered. Although K-CBCL-E scores were significantly correlated with K-ARS scores, there existed a significant overlap between K-CBCL-A and K-CBCL-E and a marginal overlap between K-CBCL-A and K-CBCL-T. Therefore, similar to previous studies,5,32 K-CBCL-T was preferred over the K-CBCL-E for the analysis.

Table 2

Sensitivity, specificity, positive predictive value, negative predictive value of K-CBCL and K-ARS for ADHD diagnosis in the study sub-groups

1st group: first graders surveyed during 2010, 2nd group: first graders surveyed during 2011, 3rd group: fourth graders surveyed during 2011. ADHD: attention-deficit hyperactivity disorder, AUC: area under the curve, K-ARS-T: Korean ADHD Rating Scale Teacher version, K-ARS-P: Korean ADHD Rating Scale Parent version, K-CBCL-A: Korean Child Behavior Checklist attention problems, K-CBCL-E: Korean Child Behavior Checklist externalizing problems, K-CBCL-T: Korean Child Behavior Checklist total problems, NPV: negative predictive value, PPV: positive predictive value

Discriminant validity showed a similar tendency in all groups. The AUC of K-ARS-P was the highest among the measures evaluated; 0.845, 0.802, and 0.796 in the 1st group, 2nd group, and 3rd group, respectively. The 90th percentile cut-off score of K-ARS-P resulted in a reasonable level of sensitivity, specificity, and negative predictive values in all groups; 1st group (sensitivity: 0.783, specificity: 0.730, negative predictive value: 0.847), 2nd group (sensitivity: 0.761, specificity: 0.695, negative predictive value: 0.820), and 3rd group (sensitivity: 0.710, specificity: 0.715, negative predictive value: 0.748). Sensitivity and negative predictive value were the highest among those for other conditions, and the specificity was also reasonably high. This suggested that the K-ARS-P is a valid evaluation scale with a diagnostic value. The optimal cut-off score of the K-ARS-P was between 14.5/15.5, 16.5/17.5, and 14.5/15.5 in the 1st, 2nd, and 3rd group, respectively. In all groups, the 90th percentile valid cut-off scores of the K-ARS-P (16, 17, and 16 in 1st, 2nd, and 3rd group) had a high specificity and balanced sensitivity, which was similar to that of the optimal cut-off scores.

The sensitivity, specificity, and negative predictive value of the K-ARS-T scores that were more than the 90th percentile were relatively low compared to those of the K-ARS-P scores that were more than the 90th percentile. This finding indicated that compared to the K-ARS-P, the K-ARS-T is a less useful screening tool due to its low sensitivity.

In the 1st group, the sensitivity and the specificity of the T scores that were more than 60 in the K-CBCL-A were 0.376 and 0.867, respectively, while those of the T scores that were more than 63 in the K-CBCL-T were 0.297 and 0.924, respectively. In the 2nd group, the sensitivity and the specificity of the T scores that were more than 60 in the K-CBCL-A were 0.392 and 0.867, respectively, while those of the T scores that were more than 63 in the K-CBCL-T were 0.261 and 0.921, respectively. In the 3rd group, the sensitivity and the specificity of the T scores that were more than 60 in the K-CBCL-A were 0.429 and 0.837, respectively, while those of the T scores that were more than 63 in the K-CBCL-T were 0.276 and 0.939, respectively. In all groups, compared to the more than 90th percentile K-ARS-P scores, T scores more than 60 in the K-CBCL-A and more than 63 in the K-CBCL-T yielded a high level of specificity and low level of sensitivity.

By analyzing cases in the clinical range of the K-CBCL with scores equal to or higher than the 90th percentile in the K-ARS-P, a high level of specificity and low level of sensitivity was obtained in all of the groups. Furthermore, the positive predictive value and specificity of the K-ARS-P and the two subscales of the K-CBCL were significantly high in all of the groups. In the 1st group, the highest positive predictive value (0.822) and specificity (0.976) were obtained when the K-ARS-P score was at the 90th percentile or higher, and the T scores of the two K-CBCL subscales were in the clinical range. Under the same conditions, the highest positive predictive value and specificity were also observed in the 2nd group and 3rd group.

DISCUSSION

A large-scale community-based study of ADHD screening was conducted in the Jeollabuk province of the Republic of Korea for two consecutive years, 2010 and 2011. In 2010, we surveyed all first graders in the elementary school. In 2011, we surveyed all the first and fourth graders in the elementary school. ADHD was detected by multiple comprehensive methods, the principal ones being the observation of parents and teachers, direct observation and interviews by experts, behavior-rating scales, and clinical examination. ADHD, characterized by the co-existence of attention problems and impulsive behaviors, can cause severe difficulties in studies and personal relationships in life. Due to these problems, children with ADHD can be easily identified by teachers in school or parents at home. Therefore, interviews or rating scales of parents or teachers are the most useful to diagnose ADHD clinically. Likewise, the evaluation of children during ADHD screening studies in the community is mainly performed by teachers or parents rather than by the self-reports of the children. In this study, we used K-CBCL and K-ARS as screening tools for ADHD. Furthermore, we suggested more effective screening tools for ADHD through the analysis of K-CBCL and K-ARS.

In this study of screening tools for ADHD using K-ARS and K-CBCL, the K-ARS-P was determined to be the primary screening tool. In addition, we suggest that the K-ARS-P and K-CBCL tests be used together as a secondary screening test for ADHD children who need treatment. In the 1st group, when total K-ARS-P scores were higher than the 90th percentile, we obtained the highest sensitivity (0.783) and a negative predictive value (0.847) with a relatively high specificity (0.730). Also, under the same conditions, highest sensitivity and negative predictive values with a relatively high specificity (0.644) were also obtained in the 2nd and 3rd groups. This result indicates that the K-ARS-P is a useful primary screening tool. The highest sensitivity was for K-ARS-P scores that were in the 90th percentile or above, thereby showing that our methods of primary screening for ADHD (≥90th percentile points in the K-ARS-P) were quite effective. In addition, we carried out the K-CBCL test only in children with K-ARS-P scores corresponding to the 90th percentile or above. When T scores of both K-CBCL-A and K-CBCL-T were in the clinical range, we obtained the highest specificity (0.976) and positive predictive value (0.822) in the first group. Under the same conditions, the highest specificity and positive predictive value were obtained in the 2nd and 3rd groups. A minor improvement in the PPV with a decrease in the NPV was achieved when K-CBCL-A and K-CBCL-T were administered to those children who were in the K-ARS-P screen (+). Therefore, the addition of the K-CBCL does not seem to be an effective primary screening tool. It is especially true when considering that the K-ARS is a short 18-item survey, whereas the K-CBCL is a much lengthier and a more time-consuming survey. However, since high specificity corresponds to high positive predictive value and is the ideal property of a "rulein" test,37 the K-CBCL-A and the K-CBCL-T might be effective secondary screening tools for the detection of ADHD in children when considering referrals to the hospital for further management.

From the analysis of ROC curves in our study, it was found that the K-ARS-T was a less effective screening tool than the other evaluation methods. This result was the same as that obtained in a study by Yang et al.;32 however, it was different from those in other studies.5,24,25,26,27 In these studies it was reported that the evaluation of children by teachers was more valid than that by parents. In the study by Yang et al.,32 like our study, evaluation by teachers of first graders was conducted at the beginning of the school term. On the contrary, the study by Kim et al.5 was carried out during the second semester, and the efficiency of the K-ARS-T was more favorable than that of the K-ARS-P. Therefore, the timing of administration might be the reason why the teacher rating scale was less valuable in this study.

There are some limitations of this study. Because the study focused on screening for ADHD, we excluded behavioral and emotional problems as well as psychiatric disorders other than ADHD. We presented the cut-off values for each group. However, we could not determine a definite cut-off value for all subjects. Since we used the 90th percentile as the cut-off point for both K-ARS-T and K-ARS-P that were used as secondary screening tests for ADHD, it is possible that we could have excluded ADHD-affected children who could not be identified as potential candidates for participation in the secondary screening. Also, it is possible that the cut-off values may differ with age. Since the K-ARS-T and K-ARS-P scores in participants were significantly higher than those in non-participants, this sampling bias could have potentially affected our results, which could possibly increase the sensitivity and the specificity of K-ARS-T and K-ARS-P. Despite these limitations, a total of 49,088 children in elementary schools in the Jeollabuk province participated in the primary screening phase of our study. In addition, 3,085 children took part in the secondary screening after written consent from their parents. Compared to other studies, our study had a large sample size and this makes the results of our study more reliable. Furthermore, we carried out both K-ARS and K-CBCL evaluation in the same people as well as DISC-IV at the same time to diagnose ADHD. Therefore, our study is important for establishing effective screening tools for ADHD in the community.

In this study, we used the K-CBCL and the K-ARS as screening tools for children with ADHD who were later diagnosed as ADHD based on the DISC-IV criteria, and we then examining their validity as screening tools in ADHD children. Reasonable levels of sensitivity, specificity, and negative predictive value were obtained when the total K-ARS-P scores were ≥90th percentile. Using K-ARS-P and K-CBCL together for assessment, the positive predictive value and specificity were significantly high when the total K-ARS-P scores were ≥90th percentile, T scores ≥60 in K-CBCL-A and T scores ≥63 in K-CBCL-T. Our results suggested that the K-ARS-P could effectively serve as a primary screening tool to identify elementary school children with ADHD in the community. Also, there might be some increment in the effectiveness of K-ARS-P when combined with K-CBCL-A and K-CBCL-T as a secondary screening tool.