Examining the relation between ratings of executive functioning and academic achievement: findings from a cross-cultural study.

The present study investigated the relation between academic performance and ratings of executive functioning in children aged 6-11 from four countries: Sweden, Spain, Iran, and China. Ratings of executive functioning were made by both parents and teachers using the Childhood Executive Functioning Inventory (CHEXI). The results showed that the Chinese sample was generally rated as having more executive deficits compared to the other samples. The finding that executive functioning deficits are exacerbated in China is most likely the result of cultural biases. Boys were generally rated as having poorer executive functioning compared to girls, except in Iran where parents, but not teachers, rated girls as having poorer executive functioning compared to boys. However, this opposite pattern of results found for Iran is not likely to reflect true gender differences in executive functioning. Despite some differences in the levels of executive functioning across countries, both the inhibition and working memory subscales of the CHEXI were related to academic achievement in all four countries, except for CHEXI parent ratings in China. Altogether, the results indicate that the CHEXI may be used as a screening measure for early academic difficulties, although cultural biases clearly have to be taken into consideration.

Executive functioning refers to higher mental processes that allow for flexible and complex goal-directed behavior (e.g., Welsh, 2002). It includes a set of interrelated constructs, of which inhibition and working memory appear to be the most central (e.g., Miyake, Friedman, Emerson, Witzki, & Howerter, 2000), and it is of central importance to many aspects of daily life. Previous research has for example shown that poor executive functioning is a major deficit in attention deficit/hyperactivity disorder (ADHD; e.g., Willcutt, Doyle, Nigg, Faraone, & Pennington, 2005) and executive impairments are also related to poor academic achievement (e.g., Biederman et al., 2004).

Executive functioning is most often assessed through laboratory tests. However, in recent years, there has been an increased interest in the development of rating instruments for executive functioning. Previous research has shown that EF laboratory measures and EF ratings are often not very strongly correlated (e.g., Catale, Lejeune, Merbah, & Meulemans, in press; Mahone et al., 2002; Mcauley, Chen, Goos, Schachar, & Crosbie, 2010; Thorell, Eninger, Brocki, & Bohlin, 2010). These findings suggest that tests and ratings capture different aspects of EF, and, for this reason, ratings should not replace laboratory tests. Ratings have the advantage of being easy to administer to a large number of individuals and may therefore be valuable as screening instruments. In addition, ratings measure behavior over an extended period of time and may also be more likely to capture executive functions as they are used in daily life. However, relatively few previous studies have examined to what extent ratings of executive functioning can be used to predict functional impairments such as academic achievement. One exception is the study by Barkley and Fischer (2011), which showed that, among adults with ADHD, impairment in major life activities and occupational functioning were better predicted by ratings of executive functioning than by laboratory tests. In addition, a few studies of children have revealed a relation between ratings on the Behavior Rating Inventory of Executive Functions (BRIEF) and academic achievement (e.g., Clark, Pritchard, & Woodward, 2010; Mahone et al., 2002; Mcauley et al., 2010).

To our knowledge, almost all previous studies examining the relation between ratings of executive functioning and functional impairments have used the BRIEF. As repeatedly argued (e.g., Mcauley et al., 2010; Thorell & Nyberg, 2008), this rating scale does not assess executive functioning alone but also ADHD symptoms, as items such as “is impulsive” and “has a short attention span” are included. This does not of course preclude that the BRIEF can be a fairly good screening measure of academic achievement as children with ADHD often have difficulties in school. However, it is not possible to separate the effects of executive difficulties and ADHD symptoms if using the BRIEF and this is important, especially as children with poor executive functioning are at increased risk of academic failure even though they do not have ADHD (e.g., Diamantopoulou, Rydell, Thorell, & Bohlin, 2007).

An alternative rating instrument for assessing executive functioning in children is the Childhood Executive Functioning Inventory (CHEXI; Thorell & Nyberg, 2008), which is freely available for download on the Internet (http://www.chexi.se). In contrast to the BRIEF, the CHEXI was developed to focus specifically on different types of executive control rather than using more general statements or items included in the symptom criteria for ADHD. The CHEXI includes four subscales, which were created based on Barkley's (1997) hybrid model in which inhibition, working memory, regulation, and planning are seen as constituting the major EF deficits in children with ADHD. However, a previous factor analysis conducted using a preschool sample yielded two broad factors referred to as inhibition and working memory (Thorell & Nyberg, 2008). This factor structure has been replicated in Belgian samples of both preschoolers (Catale et al., in press) and school-aged children (Catale, Meulemans, & Thorell, 2012). The CHEXI has also been shown to be able to discriminate between children with ADHD and controls (Catale et al., 2012; Thorell et al., 2010).

With regard to the linkage between the CHEXI and functional impairments, one previous study has found associations to both language skills and mathematics in a kindergarten sample (Thorell & Nyberg, 2008). The importance of executive functioning deficits as an early predictor of poor academic achievement has been emphasized by several previous studies. More specifically, executive skills such as working memory have been argued to be necessary for complex cognitive operations, which in turn predict learning (e.g., Gathercole, Pickering, Knight, & Stegman, 2004). However, using laboratory measures of executive functioning as a screening measure for learning difficulties is seldom possible in large samples. In addition, one previous study has found independent effects of laboratory tests and ratings of executive functioning (Thorell & Nyberg, 2008). Thus, EF rating may provide very valuable information, proving of course that a link between these ratings and school performance can be clearly established.

In the present study, we aimed to investigate the relation between the CHEXI and academic achievement in children ages 6–11. Because ratings have the disadvantage of being influenced by cultural biases, data from four countries were included. This allowed us to examine to what extent the relation between ratings of executive functioning and academic achievement varies across the countries under study.

METHOD

The present study included participants from four different countries: Sweden (n = 141), Spain (n = 219) China (n = 72,) and Iran (n = 49). The Swedish data were collected in the greater municipality of Stockholm, the country's capital and largest city. The Spanish data were collected in A Coruña and the Chinese data in Hong Kong. Finally, the Iranian data were collected in the city of Kermanshah, which is located in western Iran. Thus, the data were mainly collected from urban areas. Within the different areas, data were collected from randomly selected schools. We collected data on parental education (i.e., highest level of completed education) as a measure of socioeconomic status (SES) in all countries except Spain using a 3-point scale (1 = only has mandatory education, 2 = has completed secondary education, 3 = university/college studies). This scale was unfortunately not directly comparable due to large differences in educational systems between the different countries. However, it could be concluded that the actual levels of parental education corresponded well with national statistics in all countries. In the text below, the national statistics are presented within parentheses to allow for an easy comparison between the levels of education in our sample and national statistics. For Sweden 10% (14%) had mandatory schooling as their highest level of education, 50% (42%) had secondary education, and 40% (44%) had studied at a college/university (Statistics Sweden, 2011). In China, 71% (74%) had mandatory schooling, 12% (7%) had secondary schooling, and 17% (18%) had studied at a college/university (Census and Statistics Department, Hong Kong, 2011). Finally, in Iran, 57% (58%) had mandatory schooling, 29% (30%) had secondary education, and 14 (12%) had studied at a college/university (Statistical Centre of Iran, 2012). Thus, the samples were representative at least with regard to SES.

The age of the children ranged from 6 to 11 years (6–10 years for Spain) with the following mean ages for the four countries: 102 months for Sweden, 99 months for Spain, 106 months for China, and 104 months for Iran (see Table 1). There was a significant group difference with regard to age, F = 7.02, p < .001. Post hoc analyses revealed that the Spanish sample was significantly younger compared to the samples from the remaining three countries. Age was therefore used as a covariate when examining differences in CHEXI ratings across countries. CHEXI ratings were made by both parents and teachers, and the results are presented separately for the two subscales: working memory (e.g., “Has difficulties with tasks or activities that involve several steps”) and inhibition (e.g., “Has difficulty holding back his/her activity despite being told to do so”). The CHEXI was translated into the different languages used in the study. All translations (except the Iranian translation) were performed using translation and back-translation by bilingual translators. In all countries, researchers with expertise within this area of research carefully reviewed all translations.

Table 1

Means and Standard Deviations of CHEXI Ratings for Each Country.

	SWEDEN		SPAIN		CHINA		IRAN
	Boys	Girls	Boys	Girls	Boys	Girls	Boys	Girls
	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)
BACKGROUND VARIABLES
Sex (n)	70	71	106	113	38	34	24	25
Age (months)	103 (15.75)	102 (15.86)	98 (8.38)	99 (9.41)	103 (16.36)	109 (14.87)	106 (15.13)	102 (15.60)
CHEXI – WORKING MEMORY
Parents	1.87 (0.57)	1.69 (0.56)	2.27 (0.75)	2.12 (0.73)	2.91 (0.69)	2.20 (0.68)	1.97 (0.68)	2.07 (0.72)
Teachers	1.86 (0.77)	1.74 (0.85)	2.03 (0.95)	1.71 (0.89)	2.69 (0.64)	2.25 (0.58)	2.42 (0.89)	1.97 (0.74)
CHEXI – INHIBITION
Parents	2.25 (0.78)	1.98 (0.74)	2.97 (0.71)	2.65 (0.72)	3.31 (0.65)	2.62 (0.62)	2.87 (0.81)	3.01 (0.66)
Teachers	2.03 (0.95)	1.62 (0.73)	2.19 (1.05)	1.65 (0.78)	2.99 (0.74)	2.33 (0.61)	3.05 (0.96)	2.44 (0.87)

Reliability was calculated using Spearman-Brown's split-half coefficient and the results showed adequate reliability for both CHEXI subscales in all four countries using either parent (rs ranging between .77–.94) or teacher ratings (rs ranging between .86-.96). Internal consistency, measured as Cronbach's alpha, was also satisfactory for both the inhibition (αs ranging between .85-.96) and working memory subscales (αs ranging between .89-.96). Parent and teacher ratings on the CHEXI were significantly correlated for both inhibition (r = .42, p < .001) and working memory (r = .43, p < .001). Please refer to Thorell and Nyberg (2008) for more information about this instrument, including, for example, test-retest reliability, validity, and a description of this instrument in relation to other EF-rating instruments.

Ratings of academic performance (mathematics and language skills) were made by teachers on a 5-point rating scale: 1 (much below average), 2 (below average), 3 (average), 4 (above average), and 5 (much above average). Previous studies have shown that these types of ratings correlate very highly with results on national tests of academic achievement (e.g., Henricsson & Rydell, 2006).

Finally, we included the block design subtest from Wechsler Intelligence Scale for Children (WISC; Wechsler, 1991) in order to control for nonverbal intelligence. This subtest has been shown to correlate very highly with full-scale IQ (r = .93; Groth-Marnat, 1997). Unfortunately, the measure of intelligence was only available in the Swedish and Iranian samples.

RESULTS AND DISCUSSION

Ratings of Executive Functioning in the Four Countries

First, two-way analyses of covariance (ANCOVAs) were computed with country and gender as the two factors and age as the covariate. With regard to the CHEXI variables, the results showed that there were significant effects of country with regard to all variables, all Fs > 15.03, ps < .001 (see Table 1). Post hoc analyses revealed that, with regard to teacher ratings, the CHEXI scores were significantly lower (i.e., showing better executive functioning) in Sweden and Spain compared to China and Iran. For the working memory subscale, the Iranian sample was also found to have significantly lower scores compared to the Chinese sample. For the parent ratings, the children in the Swedish sample were again rated as having significantly better executive functioning compared to the samples from the three other countries on both subscales. In addition, the children in the Iranian sample were rated as having significantly better working memory compared to the children in the Chinese sample.

The ANCOVAs also showed a significant effect of gender, all Fs > 10.27, ps < .001, indicating that boys were generally more impaired in executive functioning compared to girls. However, for the parent ratings, there was also a significant interaction effect of country and gender for both CHEXI subscales. These significant interactions were a result of the fact that, in the Iranian sample, parents rated girls as having greater executive deficits compared to boys, whereas the opposite was true for the samples from Sweden, Spain, and China. For the teacher ratings, no such interaction effect was found.

Unfortunately, there was very little data available to investigate whether the cross-country differences as measured by the CHEXI would also hold for laboratory measures of executive functioning. However, data for one working memory test, the backward condition of the digit span subtest from WISC (Weschler, 1991), were available in the Swedish and Iranian samples, and the results showed no significant differences between the two countries and the girls in both countries were rated as having slightly better executive functioning compared to boys. Girls have also been shown to have less problems in previous studies using other rating instruments of executive functioning (e.g., Gioia, Espy, & Isquith, 2003; Klenberg, Jämsä, Häyrinen, Lahti-Nuuttila, & Korkman, 2010). Conclusively, most of our data suggest that boys have poorer executive functioning compared to girls and, as the inconsistent gender difference found for Iran could not be replicated using a laboratory measure of working memory, this finding is most likely a result of the patriarchal view that is still dominant in some parts of Iran.

With regard to cross-country differences in executive functioning, it should be noted that the present study did not primarily aim to address this issue. Rather, data from different countries were included in order to examine the relation between executive functioning and academic achievement across cultures. Without several different types of measures of executive functioning (i.e., ratings as well as laboratory measures), it is not possible to draw any certain conclusions. However, it is interesting to note that the Chinese children in the present study were rated as having more executive deficits compared to the children in the other countries, whereas previous studies using laboratory measures have shown that Chinese children are more skilled in executive functioning compared to U.S. children (e.g., Lan, Legare, Ponitz, Li, & Morrison, 2011; Sabbagh, Xu, Carlson, Moses, & Lee, 2006). This may be taken to indicate that the present cross-country differences do not reflect true differences but rather cultural biases. Perhaps executive functioning deficits are exacerbated in China because the Chinese culture has such a strong emphasize on self-regulatory skills. A similar conclusion was drawn by Hinshaw and colleagues (2011) with regard to ADHD symptoms, and two previous studies have demonstrated that, when asked to rate the same children from videotapes, Chinese teachers rated the children as displaying higher levels of disruptive behaviors than the American teachers did (Mann et al., 1992; Mueller et al., 1995). Conclusively, the potentially biased nature of EF ratings, especially from a cross-cultural perspective, points to the need for continued research on the validation of EF-rating measures such as the CHEXI.

Relations Between Executive Functioning and Academic Achievement

Correlations between the two CHEXI subscales and academic achievement for the four countries are presented in Table 2. The results showed that with regard to teacher ratings, the working memory subscale was significantly related to both mathematics and language skills in all countries. With regard to the inhibition subscales, significant relations were found to mathematics in all countries except Sweden and to language skills in all countries except China. For the parent ratings, no significant relations were found between executive functioning and academic performance in the Chinese sample. In the other three countries, parent ratings on both CHEXI subscales were significantly related to both mathematics and language skills, except for the relation between working memory and language skills in the Iranian sample. All relations between the CHEXI subscales and academic achievement remained significant when controlling for nonverbal intelligence.

Table 2

Correlations Between the CHEXI and Academic Achievement (One-Tailed Tests).

	CHEXI –Working Memory Subscale		CHEXI – Inhibition Subscale
	Parents	Teachers	Parents	Teachers
	r	r	r	r
SWEDEN
Mathematics	−.28∗∗∗	−.42∗∗∗	−.19∗	−.13
Language skills	−.33∗∗∗	−.41∗∗∗	−.29∗∗∗	−.22∗∗
SPAIN
Mathematics	−.51∗∗∗	−.59∗∗∗	−.35∗∗∗	−.48∗∗∗
Language skills	−.54∗∗∗	−.59∗∗∗	−.38∗∗∗	−.46∗∗∗
CHINA
Mathematics	−.13	−.59∗∗∗	−.04	−.25∗
Language skills	−.01	−.38∗∗∗	.07	−.04
IRAN
Mathematics	−.35∗∗	−.36∗∗	−.25∗	−.29∗
Language skills	−.23	−.51∗∗∗	−.25∗	−.24∗

p < .05.

p < .01.

p < .001.

In summary, despite the main differences in ratings of executive functioning across the four countries, relations between the CHEXI and academic performance were found in all countries, except for the CHEXI parent ratings in the Chinese sample. This inconsistency is difficult to explain, especially as it was only found for the parent ratings. If these results reflect true differences in the relation between executive functioning and academic achievement in China compared to the other countries, this may be explained by differences with regard to the abilities required for academic success. China still has a traditional view on education. The educational system in Europe, on the other hand, has a strong emphasis on generic skills (e.g., reasoning abilities, critical mind), and children are expected to solve problems independently and to think for themselves—abilities that require high executive skills. However, this does not explain the relatively strong relations between executive functioning and academic achievement in Iran, which also has a traditional educational system. More studies are clearly needed before any conclusions can be drawn regarding this issue. Another important issue for future research would be to examine to what extent the CHEXI can be used as a screening measure for children with other executive-functioning-related learning problems such as dyslexia.

Finally, it may be worth mentioning that academic achievement was found to be more strongly related to working memory compared to inhibition, which is in line with one previous study using the CHEXI (Thorell & Nyberg, 2008) as well as with some studies using laboratory measures (Monette, Bigras, & Guay, 2011; Thorell, 2007).

Altogether, the present findings indicate that CHEXI ratings, especially the working memory subscale, may be a valuable screening measure for predicting early academic difficulties. However, cultural biases have to be taken into consideration, and there is a need to establish culturally adapted norms.