Evaluation of estimating missed answers in conners adult ADHD rating scale (screening version).

OBJECTIVE: Conners Adult ADHD Rating Scale (CAARS) is among the valid questionnaires for evaluating Attention-Deficit/Hyperactivity Disorder in adults. The aim of this paper is to evaluate the validity of the estimation of missed answers in scoring the screening version of the Conners questionnaire, and to extract its principal components.
METHOD: This study was performed on 400 participants. Answer estimation was calculated for each question (assuming the answer was missed), and then a Kruskal-Wallis test was performed to evaluate the difference between the original answer and its estimation. In the next step, principal components of the questionnaire were extracted by means of Principal Component Analysis (PCA). Finally the evaluation of differences in the whole groups was provided using the Multiple Comparison Procedure (MCP).
RESULTS: Findings indicated that a significant difference existed between the original and estimated answers for some particular questions. However, the results of MCP showed that this estimation, when evaluated in the whole group, did not show a significant difference with the original value in neither of the questionnaire subscales. The results of PCA revealed that there are eight principal components in the CAARS questionnaire.
CONCLUSION: The obtained results can emphasize the fact that this questionnaire is mainly designed for screening purposes, and this estimation does not change the results of groups when a question is missed randomly. Notwithstanding this finding, more considerations should be paid when the missed question is a critical one.

Attention Deficit / Hyperactivity Disorder (ADHD) is characterized as having inattentiveness, hyperactivity, and impulsivity. This disorder was primarily thought to be a problem limited to youth. However, in recent years, researchers have found that ADHD is often a chronic condition which persists into adulthood (1, 2). The available data suggest that between 30 and 70 percent of children with ADHD continue to manifest symptoms in adulthood (2-5). It is estimated that between 1 and 7 percent of the adult population experience ADHD symptoms (2-4).

The four most commonly used self-report measures (3) for ADHD are the Conners Adult ADHD Rating Scale (CAARS) (1, 6); the Wender Rating Scale (7); the Copeland Symptom Checklist (8); and the Brown Scale (9). These screening tools are not generally used for diagnostic purposes, as inattention, impulsivity, and volatile mood are features of several other psychiatric conditions. The screening forms are useful when a quick screen for ADHD symptoms is required.

The CAARS is a set of easily administered self-report questionnaires and observer-rated instruments which has been designed to assess symptoms related to ADHD in adults.

A main problem in self-report measures is the high probability of missing some questions by ADHD patients which can interfere with scoring the questionnaire. In this case, to score the questionnaire, the operator should complete the missed answer(s) through averaging the answers of other questions in that subscale and putting the result for the missed answer (1).

In this study, we evaluated the validity of the estimated missed answers to find out whether this estimation was valid and on which questions it could be used. We also extracted principal components of the screening version of the CAARS questionnaire.

Material and method

Instrument

The CAARS is a suitable instrument for evaluating ADHD symptoms in adults. It utilizes a 4-point format in which respondents are asked to rate items pertaining to their problems.

The self-report screening form (CAARS-Self report: Screening Version, CAARS-S: SV) which was used in this study, has 30 items that assess ADHD symptoms according to the 4th edition of Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) (3). The subscales derived from the questionnaire are:

Inattentive (9 items),

Hyperactive-Impulsive (9 items),

Total ADHD symptoms,

ADHD Index (12 items).

We prepared and used the computerized version of the questionnaire to benefit from rapid scoring, increased accuracy, easy distribution of the questionnaire to the participants by E-mail, and making a database for statistical analysis more easily.

Participants

We sent the questionnaire to 600 students at four universities via E-mail. We also distributed 120 paper-forms of the questionnaire among the students. A total number of 400 questionnaires were filled out and returned. A total number of 530 questionnaires were returned among which 130 participants’ answers were incomplete or scratched, and 400 questionnaires were filled out completely. This indicates the importance and need of proper estimation of missed or scratched answers in the questionnaire. Figure 1 demonstrates the characteristics of the participants. In generalization of the results, it should be taken into account that the study was performed on a population of university students.

Figure 1

The participants distribution of age, gender and education

Statistical Analysis

We performed all statistical tests and calculations using the “Matlab 2009a” software. Lilliefors test is used to evaluate whether the distribution of the sample is normal. The Lilliefors test is a 2-sided goodness-of-fit test suitable when a fully-specified null distribution is unknown and its parameters must be estimated, i.e. it is a Kolmogorov-Smirnov test with unknown null distribution. The statistical distribution of the subscales is shown in figure 2. The raw scores of the CAARS are converted to t-scores to eliminate the gender and age effects. The t-scores used in CAARS are linear ones which do not change the actual distributions of the variables (1); therefore, the variables which are not normally distributed in the raw data will continue to be distributed in this way after being converted to t-scores.

Figure 2

Distribution of participants’ t-scores in subscales of CAARS. (CAARS: Conners Adult ADHD Rating Scale)

Validity Evaluation

To investigate the validity of the estimated missed or scratched answers, we assumed each question is missed individually, and estimated its answer using the average of other answers in its group. Then, we compared the original answers with their calculated estimations for each question to find any significant difference. For example, the first question belongs to inattention subscale which contains 9 questions. We considered the average of the other 8 questions as the estimation of the answer to the first question and evaluated if there is a significant difference between this estimation and real answers to the first question. To investigate the veracity of this approach, we used Kruskal-Wallis test which is a non-parametric version of ANOVA.

In the next step, we used the Multiple Comparison Procedure (MCP) to evaluate the significance of the differences between the original and estimated answers in each group. Since the chance of incorrectly finding a significant difference would increase with the number of comparisons, using the MCP can provide an upper bound on the probability that any comparison could be incorrectly found significant.

Principal Components

In some questionnaires, the components (questions) are highly correlated (redundant). In such cases, it is useful to reduce the dimension of the components. An effective procedure for performing this operation is Principal Components Analysis, PCA (10). PCA involves transforming a number of possibly correlated variables into a smaller number of uncorrelated variables called principal components. This technique which is closely related to factor analysis is used for extracting principal components of the CAARS questionnaire.

Results

The characteristics of the participants in age, gender and education are demonstrated in figure 1. The mean age of the participants was 24.4 ± 5.9 years. 47.75% of the participants were male and 66.5% of them were educated.

Figure 2 demonstrates the distribution of t-scores. It shows that their distribution and consequently the distribution of raw scores is not normal; thus, non-parametric tests are used for statistical evaluations.

The results of Kruskal-Wallis tests are demonstrated in Table 1. This table shows each question belongs to which group (9 in inattention, A; 9 in impulsivity, B; 18 in ADHD, C; and 12 in ADHD index, D). P-values less than 0.01 indicate significant difference between original answer and its estimation. Results indicate that a significant difference (df =399, P<0.01) exists between the original and estimated answers for some questions; therefore, the substitution method is not appropriate for their estimation.

Table 1

The results of Kruskal-Wallis test for comparison between original and estimated answers in CAARS questionnaire

Q No.	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
Subscale	A	B	D	B	D	B	D	B	A	D	D	A	A	D	B
P-Value	0.151	0.416	0*	0*	0.001*	0.724	0*	0.1787	0.726	0*	0.773	0.726	0*	0*	0*
Subscale	C	C		C		C		C	C			C	C		C
P-Value	0.919	0.154		0*		0.2231		0.003*	0.366			0.533	0*		0*
Q No.	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30
Subscale	D	B	A	D	A	B	D	D	B	A	B	D	A	A	D
P-Value	0*	0.786	0.092	0.647	0*	0.56	0*	0*	0.067	0.098	0*	0*	0*	0.001*	0*
P-Value		0.019	0.012		0*	0.015			0*	0.006*	0*		0*	0*

P-values less than 0.01 are considered to have significant difference.

As Table 1 demonstrates, this estimation is significantly different from the original value for questions 13, 20, 28, 29 (Inattentive subscale), 4, 15, 26 (Hyperactive/ Impulsive subscale), 4, 13, 15, 18, 20, 26, 28, 29 (ADHD subscale) and for 3, 7, 10, 14, 16, 22, 23, 27, 30 (ADHD Index).

The results of MCP shows that the estimation, when evaluated in the whole group, is not significantly different (α=0.001) from the original value in each subscale.

Figure 3 indicates the results of PCA which transforms a number of possibly correlated variables into a smaller number of uncorrelated variables. This figure expresses the obtained Eigen values for each component. Eight components have Eigen values greater than one and can be extracted.

Figure 3

The Eigen values for all components drived from CAARS questionnaire based on Principal Component Analysis (PCA).

The distribution of t-scores in the statistical population for each subscale is demonstrated in Figure 4. The t-scores used with CAARS have a mean of 50 and a standard deviation of 10. Values around 50 indicate that the participant is in the average range in that subscale whereas higher t-scores represent a problem; lower t-scores suggest that the participant does not present particular symptoms.

Figure 4

The t-scores distribution diagram of participants’ in four subscales of CAARS (inattention, hyperactivity, ADHD and ADHD index).

T-scores distribution in the statistical population indicates that in the “Inattention” group (A), 44% of the participants were in the average range (regarding their age and gender); 10 % were very much above the average having high levels of symptoms that could meet the diagnostic criteria for attention-deficit/hyperactivity disorder; and 8% were much below the average. In the “Impulsivity” group (B), the average range contains 38 % of the participants. 6 % were very much above the average; and 11% were much below the average. In the “ADHD” group (C), 43% of the statistical population was in the average range; 13% was very much above the average; and 7% was quite below the average. In the “ADHD Index” group (D), 56% of the statistical population was in the average range; only 1% were very much above the average; and 6% much below the average.

Discussion

ADHD is a common problem in adults which can be evaluated using some self-report questionnaires including the Conners Adult ADHD Rating Scale (CAARS). A main problem in self-report measures is the high probability of missing some questions by ADHD patients which can interfere with scoring of the questionnaire. In this case, to score the questionnaire, the operator should complete the missed answer(s). In this study, we evaluated the estimation of the missed answers in the self-report screening version of CAARS.

Four hundred university students participated in this study. The Kruskal-Wallis results revealed that the differences between original and estimated values were significant for some particular questions. Four of these questions are in the “Inattention” subscale, 3 in “Hyperactive/Impulsive”, 8 in “ADHD” and 9 in “ADHD Index”. It is noteworthy that the estimation of a particular question in A or B subscale is obtained by averaging 8 other questions in that subscale while its estimation in C subscale is obtained by averaging 17 other questions.

Individual evaluation of each question shows that only the estimation of the answers in 11 questions can be valid. Therefore, when a high percentage of the participants in one study have missed a particular question, it should be noted that the estimation should not be used for the critical questions mentioned in Table 1 and the incomplete questionnaires should be removed from the study. However, the results of MCP, when evaluated in the whole groups, indicate that the estimations did not differ significantly from their original values in four subscales, i.e. the result of the whole groups will not change with this estimation.

Eight principal components were extracted by means of PCA which transforms a number of possibly correlated variables into a smaller number of uncorrelated variables called principal components. The first component accounts for as much of the variability in the data as possible, and each succeeding component accounts for as much of the remaining variability as possible.

The novelty of this study is in the evaluation method of estimation and taking into account the Iranian population. Regarding semi-similar studies, PCA is used as a factor analysis method in translated versions of ADHD questionnaires (11-13). In the French version of ASRS, self-report adult questionnaire is considered and factor analyses retrieved the original two-factor structure although impulsivity items clearly load on a specific third factor in students only(11). Also German and Chinese children are respectively investigated (12,13). Psychometric properties of CAARS and FBB-HKS questionnaires are compared for the former and both instruments’ scores showed reliability as well as factorial and convergent / discriminant validity (12). For the latter, mixed findings with respect to the psychometric properties of the Chinese translation of the SDQ are reported (13). Considering adult Iranian population, one study have reported the prevalence of 3.7% for ADHD in a group of 244 university students, but no estimation or validity evaluation were investigated (14).

In summary, although the Kruskal-Wallis results revealed that the difference between the original and estimated values is significant for some particular questions individually, the results of MCP, when evaluated in the whole group, indicate that the estimations did not differ significantly from their original values in any of the four subscales, i.e. the result of the whole group will not change with this estimation. Nevertheless, more considerations should be paid for critical questions.

The obtained results can emphasize the fact that this questionnaire is mainly designed for screening purposes, and this estimation does not change the results of groups when a question is missed randomly. Nevertheless, if many of the participants missed a critical question, this substitution would not be valid and more considerations should be paid in such cases.