Ages & Stages Questionnaire-Brazil-2011: Adjustments on an Early Childhood Development Screening Measure.

Introduction. Professionals who assess early childhood development highly benefit from reliable development screening measures. The Ages & Stages Questionnaire was adapted Brazil in 2010 and named ASQ-BR. Modifications in some items were required to improve the instrument's psychometric properties. The present study modified the ASQ-BR to verify if those changes increase its characteristics. Method. This study researched 67 522 children from 972 public day care centers and preschools. Changes in items were made considering Cronbach's α and item-to-total correlations. Reliability, dimensionality, and item-to-total correlations were calculated. Results. Regarding dimensionality, 86.2% of the scales in ASQ-BR-2011 were unidimensional. Internal consistency showed improvement from 2010 to 2011: 53.8% of the scales increased the α statistics against 41.2% that decreased, and 5.0% remained the same. Finally, 65.2% of the modified items showed improvement. Conclusions. Overall, the instrument's psychometrics improved from 2010 to 2011, especially in the personal/social domain. However, it still leaves room for improvement in future studies.

Introduction

Well-designed screening measures to assess early childhood development are valuable tools for families, pediatricians, and the entire set of professionals whose work involves dealing with children. There are several instruments worldwide in the field, such as the Denver Developmental Screening Test[1] and the Child Behavior Checklist.[2] However, the most widespread and cross-culturally adapted measure among screening tests focusing on children early development is the Ages & Stages Questionnaire (ASQ).[3]

The ASQ, especially its third edition (ASQ-3), is a parent-completed questionnaire that evaluates development of children from 1 to 66 months of age. The ASQ-3 consists of 21 questionnaires that correspond to age interval categories based on Piaget’s and Gesell’s theories. The intervals are named according to the first month of the age interval it refers to: children aged 2, 4, 6, 8, 9, 10, 12, 14, 16, 18, 20, 22, 24, 27, 30, 33, 36, 42, 48, 54, and 60 months.[3] Since children tend to present more developmental hallmarks during early life stages and fewer differences in later stages,[3,4] the questionnaires inquiring about older children involve larger age intervals as compared to questionnaires of younger children. Questionnaires assess 5 developmental domains: (a) Communication, (b) Gross and (c) Fine motor skills, (d) Problem solving, and (e) Personal/social abilities.

The ASQ-3 has been translated into more than 20 different languages, and its feasibility as an international tool for screening development of children is supported by several researcher studie.[5-11] The Brazilian Portuguese version of the ASQ-3 is called the ASQ-BR. It was first used as a research tool in 2010. The results of that research were published by Filgueiras et al[5] in 2013 and showed that the Brazilian version had good psychometric characteristics. Among the 21 scales, 19 had their psychometric properties analyzed—exceptions were 2- and 4-month-old questionnaires. Regarding dimensionality, only 3 of 95 scales showed bidimensionality (10, 54, and 60 months of Personal/social domain). Diversely, some of ASQ-BR’s scales presented unreliable data based on the Cronbach’s α coefficient. For example, 67% of the scales of the personal/social domain presented scales with α< 0.65. A similar phenomenon was observed in 3 scales of the Problem solving domain as well as in 2 scales of both Gross and Fine motor skill domains. Only the Communication domain had all scales with αs above 0.65.

Two different hypotheses have been proposed by Filgueiras et al[5] when the above-mentioned results were discussed. First, the ASQ-3 was developed for parents to apply on their children, while the development of the ASQ-BR focused on a sample of teachers and caregivers in child day care centers. This difference between types of respondents might have caused some impairment of the internal consistency. Second, the Personal/social domain is, by definition, 2 domains in one. This is due to the premises that personal skills are associated with children’s ability to be independent and look for adult help only when needed and that social skills are usually defined as the adequacy of behaviors and emotional control during and throughout social interactions.

The purpose of the present work is to improve the ASQ-BR by proposing changes based on Filgueiras et al[5] study in order to try to solve the problems pointed out in the first adaptation of the ASQ-3 to Brazilian Portuguese for a public day care center sample. For that, items that might show either of the problems mentioned above were altered in accordance with item-to-total correlation and ability to assess day care centers activities. Examples of such activities are the use of forks by children (for safety reasons) and playing with teddy bears; to avoid allergic reactions, teddy bears were replaced by rag dolls, among others.

Methods

Participants

This study researched 67 522 children enrolled in 972 public day care centers and public preschools in Rio de Janeiro, Brazil. Data from these children were assessed by 11 664 teachers and caregivers as respondents for a second version of the ASQ-BR. Children within the age range of 9 to 66 months were distributed according to the 16 age intervals defined by the ASQ-3. Questionnaires 2, 4, 6, and 8 months were excluded due to minimal or no participation. The 9-month questionnaire was incorporated into the 10-month questionnaire due to an overlap of age intervals. The project was approved by the Ethical Committee of the Pontifical Catholic University of Rio de Janeiro, Brazil.

Changes on ASQ-BR

The ASQ-3[5] was cross-culturally adapted to Brazilian Portuguese, thus generating the ASQ-BR. For that, back-translation of the original material was performed according to the guidelines of the International Test Commission.[12] The results obtained with the ASQ-BR were evaluated psychometrically, and the items with statistical problems, for example, item-to-total correlation lower than 0.30, were listed to be modified in the present work. A panel of multidisciplinary specialists in early development consisting of psychologists (n = 4), educationalists (n = 2), health scientists (n = 1), and economists (n = 3) were gathered in order to evaluate the list of problematic items in the ASQ-BR. The panel’s conclusions were incorporated into the ASQ-BR.

A pilot study with 120 children in 10 public child care centers was conducted to figure out whether the modifications mentioned above were clear and if the items were adapted to Brazilian public child day care centers. Teachers’ and caregivers’ suggestions were also incorporated into the ASQ-BR after approval of the multidisciplinary panel. Afterwards, one of the authors of the ASQ-3 (the original American instrument) was consulted about the Brazilian version of the ASQ-3, who gave her approval. These procedures generated a final version of Ages & Stages Questionnaire–Brazil–2011 (ASQ-BR-2011). Finally, in order to make terms clear for the reader, the vocabulary employed in this study will be as follows: (a) scale—the 6-item set of questions measuring one development domain within one age interval; and (b) questionnaire—the entire set of scales within a specific age interval.

Procedures

The directors of the 972 public child day care centers and preschools surveyed were invited to participate in a full day (8-hour) training previously scheduled by the Secretary of Education of Rio de Janeiro. Each meeting had approximately 30 day care directors, and the 16 ASQ-BR-2011 questionnaires were presented by 1 of the 10 professionals (9 psychologists and 1 educationalist) previously trained on the ASQ-BR-2011 by the main author of the ASQ-BR-2011. All of the directors were responsible for taking the ASQ-BR-2011 questionnaires to their preschools or day care centers so that each child could be evaluated by his/her teacher or caregiver. The directors then passed on their ASQ-BR-2011 training to the teachers and caregivers in their schools or day care centers and the latter would be responsible for directly administering the ASQ-BR-2011 to the children. The tests were applied according to the children’s age and classroom—approximately 25 children per classroom.

Data collection occurred between November 11 and December 23, 2011. After the application, the whole paper-based material used to administer the ASQ-BR-2011 was collected by the directors and delivered to the Secretary of Education of Rio de Janeiro. At that point, the Secretary of Education of Rio de Janeiro entered all the data regarding the ASQ-BR-2011 into a website developed specifically for the present study.

Statistical Analyses

Descriptive statistics were initially performed in order to understand the sample’s characteristics in terms of age and sex across the 16 age intervals. Average (mean) and standard deviation (SD) of the 5 scales—Communication, Gross motor, Fine motor, Problem solving, and Personal/social—were used to compare the results from the ASQ-BR and the ASQ-BR-2011. An independent sample t test was also employed to compare the ASQ-BR and ASQ-BR-2011 in each scale within each questionnaire. A factor analysis was made for each scale separately. Factors were extracted using the maximum likelihood technique employing the eigenvalue above 1.0 criteria.[13,14] Unidimensionality was expected in all ASQ-BR-2011 scales. However, an oblique rotation—Promax (κ = 4.0)—was adopted in cases of multidimensionality because the same domain should be evaluated in a 6-item set[4]; thus, high-to-moderate correlations were expected among factors, which would justify oblique rotation.[15]

The internal consistency index employed in the present study was Cronbach’s α. Values above 0.65 were expected because of the low number of items present in the ASQ-BR-2011 scales. That would tend to impair the α statistics.[15-17] Finally, item-to-total correlations for each one of the 480 items of the ASQ-BR-2011 were calculated and compared with the ASQ-BR correlations in order to figure out if modified items met the goal of the present study.

Results

The sample’s statistics depicted very similar data among boys and girls. The sex distribution of the children was close to chance, with exception of the 12-month interval that showed a concentration of boys of above 55%. Sample’s characteristics are presented in Table 1.

Table 1.

Descriptive Statistics of the ASQ-BR-2011: Sample Size, Sample Distribution, Mean Age, and Age Standard Deviation per Sex and Age Interval.

Age Interval (in Months)	n	Boys			Girls
		Percentage	Mean Age	SD	Percentage	Mean Age	SD
10	73	54%	9.93	0.35	46%	10.00	0.42
12	141	57%	11.97	0.60	43%	11.93	0.61
14	216	47%	14.08	0.53	53%	14.14	0.52
16	545	53%	16.20	0.50	47%	16.18	0.52
18	972	54%	18.04	0.58	46%	18.02	0.57
20	1143	53%	20.05	0.57	47%	20.04	0.58
22	1259	54%	22.01	0.56	46%	22.00	0.57
24	1637	55%	24.27	0.73	45%	24.24	0.72
27	2390	53%	27.02	0.89	47%	27.01	0.86
30	3264	52%	30.02	0.89	48%	30.01	0.87
33	3375	53%	32.99	0.86	47%	32.99	0.86
36	4689	53%	36.80	1.30	47%	36.79	1.34
42	7703	53%	42.04	1.70	47%	42.09	1.69
48	6921	52%	47.72	1.60	48%	47.70	1.61
54	9511	51%	53.87	1.87	49%	53.81	1.85
60	23683	49%	61.73	1.94	51%	61.83	1.99
Total	67522	51%	44.83	12.67	49%	44.39	13.48

Abbreviations: ASQ-BR-2011, Ages & Stages Questionnaire–Brazil–2011; SD, standard deviation.

Since the ASQ-BR data were collected in 2010 and the ASQ-BR-2011 data were collected in 2011, arithmetic mean (M) and standard deviation (SD) for each scale of each year were calculated. The comparison between the 2 years of application showed several significant differences (P< .05), which will be discussed later. The Communication domain showed significant differences only on the 60-month scale, with higher mean in 2011 as compared to 2010. Regarding the Gross motor domain, no scale had significant differences. The Fine motor domain had 7 of the 16 age intervals with significantly higher mean on 2011 if compared to 2010: 10, 22, 24, 27, 30, 33, and 36 months. On Problem solving, 4 scales showed significantly higher mean on 2011: 16, 20, 22, and 60 months. The Personal/social domain had 5 scales with significantly higher mean in 2011 than in 2010: 10, 20, 22, 24, and 27 months. Table 2 depicts mean, SD, and the P value of the independent sample t test for each age interval and domain.

Table 2.

Mean and Standard Deviation of the Sum and Cronbach’s α of Each Developmental Domain of the ASQ-BR in Both 2010 and 2011 (Mean Comparison Between Years Using t Test Statistics).

Age Interval (in Months)	Year of Assessment	n	Communication			Gross Motor			Fine Motor			Problem Solving			Personal/Social
			Mean (SD)	t Test (P Value)	α	Mean (SD)	t Test (P Value)	α	Mean (SD)	t Test (P Value)	α	Mean (SD)	t Test (P Value)	α	Mean (SD)	t Test (P Value)	α
10	2010	236	26.7 (17.2)	P = .45	0.76	38.5 (16.9)	P = .06	0.79	37.6 (17.2)	P< .05*	0.78	35.5 (16.6)	P = .12	0.72	35.8 (13.8)	P< .05*	0.58
	2011	73	30.4 (18.1)		0.81	44.6 (14.8)		0.79	42.7 (16.6)		0.81	40.1 (20.3)		0.89	41.4 (16.4)		0.80
12	2010	434	36.4 (16.6)	P = .13	0.75	43.3 (17.9)	P = .13	0.85	38.1 (17.5)	P = .47	0.77	35.7 (17.6)	P = .19	0.77	33.8 (17.1)	P = .13	0.76
	2011	141	39.8 (16.9)		0.78	44.9 (16.6)		0.83	40.1 (19.0)		0.84	40.2 (18.4)		0.83	36.1 (17.3)		0.79
14	2010	752	31.3 (16.7)	P = .19	0.75	47.5 (17.9)	P = .16	0.85	35.8 (17.4)	P = .29	0.77	34.1 (15.5)	P = .26	0.77	30.6 (16.5)	P = .08	0.76
	2011	216	34.6 (17.9)		0.81	49.0 (17.6)		0.90	38.7 (18.0)		0.83	39.1 (18.8)		0.85	35.0 (17.9)		0.79
16	2010	987	29.9 (14.6)	P = .29	0.75	52.2 (13.7)	P = .28	0.89	43.2 (16.7)	P = .31	0.75	36.7 (18.2)	P< .05*	0.80	34.0 (15.5)	P = .09	0.71
	2011	545	32.0 (15.8)		0.79	53.7 (11.7)		0.83	46.7 (15.8)		0.82	42.6 (17.6)		0.77	38.3 (16.1)		0.74
18	2010	1103	33.5 (16.3)	P = .35	0.73	55.8 (7.9)	P = .36	0.85	44.7 (14.6)	P = .19	0.79	37.3 (15.3)	P = .18	0.79	41.6 (14.5)	P = .29	0.64
	2011	972	33.4 (16.4)		0.79	54.9 (10.5)		0.82	47.0 (14.0)		0.76	40.1 (15.0)		0.74	43.8 (15.0)		0.75
20	2010	1031	33.1 (19.2)	P = .43	0.77	52.3 (11.5)	P = .33	0.64	41.2 (14.8)	P = .10	0.72	38.2 (13.9)	P< .05*	0.72	36.2 (12.9)	P< .05*	0.64
	2011	1143	36.7 (18.8)		0.83	54.8 (9.4)		0.72	45.3 (13.5)		0.71	43.1 (11.9)		0.59	42.7 (13.7)		0.70
22	2010	955	35.4 (18.4)	P = .54	0.82	48.2 (13.0)	P = .21	0.74	40.4 (13.6)	P< .05*	0.67	39.3 (13.9)	P< .05*	0.63	38.1 (12.6)	P< .05*	0.57
	2011	1259	39.2 (17.3)		0.68	52.0 (10.5)		0.64	46.2 (11.9)		0.64	45.1 (12.1)		0.63	47.0 (12.1)		0.63
24	2010	1454	44.1 (17.5)	P = .16	0.81	52.1 (10.6)	P = .10	0.71	42.8 (12.6)	P< .05*	0.64	41.1 (13.4)	P = .22	0.65	35.7 (13.1)	P< .05*	0.59
	2011	1637	48.1 (15.5)		0.82	54.1 (.9.4)		0.65	48.7 (11.3)		0.67	45.9 (11.9)		0.63	45.2 (12.7)		0.65
27	2010	2222	48.1 (14.3)	P = .17	0.84	50.9 (11.9)	P = .09	0.67	34.1 (15.2)	P< .05*	0.62	47.8 (12.5)	P = .14	0.63	34.3 (11.4)	P< .05*	0.63
	2011	2390	50.7 (12.9)		0.75	54.4 (9.2)		0.65	42.8 (14.3)		0.71	51.9 (10.3)		0.62	44.1 (24.2)		0.18
30	2010	2814	50.3 (13.0)	P = .31	0.77	53.5 (10.1)	P = .13	0.71	34.2 (17.7)	P< .05*	0.70	45.0 (14.8)	P = .32	0.62	46.2 (11.7)	P = .16	0.53
	2011	2786	49.9 (13.1)		0.74	53.9 (9.9)		0.67	40.5 (17.6)		0.80	47.2 (14.0)		0.71	48.4 (14.6)		0.35
33	2010	3316	47.8 (14.3)	P = .24	0.75	52.2 (11.2)	P = .15	0.67	36.4 (18.7)	P< .05*	0.78	46.8 (14.1)	P = .23	0.69	46.8 (12.9)	P = .31	0.61
	2011	3002	51.6 (12.3)		0.70	56.1 (12.3)		0.13	44.9 (31.0)		0.24	51.3 (22.0)		0.22	50.3 (11.2)		0.66
36	2010	5291	46.8 (12.4)	P = .25	0.74	53.8 (10.4)	P = .11	0.69	42.2 (17.9)	P< .05*	0.79	48.4 (13.4)	P = .17	0.67	49.5 (10.7)	P = .14	0.66
	2011	4416	49.3 (11.1)		0.65	56.4 (17.3)		0.18	48.6 (21.5)		0.40	51.3 (12.0)		0.68	51.7 (9.8)		0.59
42	2010	8859	46.9 (12.8)	P = .47	0.66	54.8 (8.7)	P = .29	0.71	41.3 (15.5)	P = .07	0.79	48.7 (13.2)	P = .15	0.67	48.0 (10.4)	P = .18	0.57
	2011	6631	48.3 (12.4)		0.69	55.9 (8.2)		0.67	45.1 (14.3)		0.71	49.9 (14.3)		0.67	50.8 (10.3)		0.57
48	2010	8528	48.2 (13.9)	P = .21	0.66	53.3 (10.4)	P = .31	0.64	39.0 (16.4)	P = .06	0.71	44.1 (14.5)	P = .51	0.67	47.6 (11.2)	P = .17	0.52
	2011	6347	50.6 (12.7)		0.77	55.1 (9.3)		0.71	44.0 (15.6)		0.75	46.0 (14.1)		0.70	50.6 (10.7)		0.57
54	2010	7452	52.4 (11.7)	P = .11	0.78	54.3 (9.8)	P = .13	0.70	44.4 (14.5)	P = .08	0.72	40.6 (14.5)	P = .78	0.68	49.3 (10.5)	P = .12	0.53
	2011	9511	48.8 (14.0)		0.80	51.7 (12.1)		0.78	41.9 (15.3)		0.75	39.5 (21.8)		0.35	47.6 (11.5)		0.58
60	2010	47	47.9 (15.2)	P< .05*	0.78	48.5 (17.0)	P = .23	0.71	46.8 (15.4)	P = .49	0.72	46.7 (15.3)	P< .05*	0.70	51.6 (11.3)	P = .26	0.52
	2011	23 683	38.4 (13.2)		0.44	51.9 (16.2)		0.39	45.5 (14.8)		0.77	41.2 (16.5)		0.76	49.5 (15.3)		0.26

Abbreviations: ASQ-BR, Ages & Stages Questionnaire–Brazil; SD, standard deviation.

Mean considered significantly different when P< .05.

The factor analysis presented unidimensionality in 69 of the 80 scales (86.2%)—more than the ASQ-BR, which had 92 of 95 (96.8%) unidimensional scales. The 11 exceptions of the ASQ-BR-2011 (13.8%) were the following: 27-month Personal/social scale (2 dimensions); 30-month Personal/social scale (2 dimensions); 33-month Gross motor (3 dimensions), Fine motor (2 dimensions), and Problem solving (3 dimensions); 36-month Gross motor (3 dimensions) and Fine motor (2 dimensions); 54-month Problem solving (2 dimensions); and 60-months Communication (2 dimensions), Gross motor (2 dimensions), and Personal/social (2 dimensions).

Among the 480 items of the ASQ-BR, 69 (14.4%) were modified. Regarding the internal consistency, Cronbach’s α was below 0.65 in 22 of the 80 scales (27.5%), a result similar when compared to 2010 ASQ-BR’s statistics (25.0%). The scales that presented α< 0.65 divided by domain were as follows: in Communication, only the 60-month scale; in Gross motor, the 22-, 33-, 36-, and 60-month scales; in Fine motor, the 22-, 33-, and 36-month scales; in Problem solving, the 20-, 22-, 24-, 27-, 33-, and 54-month scales; and in Personal/social, the 22-, 27-, 30-, 36-, 42-, 48-, 54-, and 60-month scales. Comparing the α of the 80 scales in each year of application (2010 and 2011), it increased in 43 scales (53.8%), decreased in 33 scales (41.2%), and remained the same in 4 of the 80 scales (5.0%). Cronbach’s α for the ASQ-BR and the ASQ-BR-2011 are depicted in Table 2.

Item-to-total correlations between 2010 and 2011 were also compared. First, regarding the ASQ-BR-2011, 81 of the 480 items (16.9%) presented r< 0.30—somewhat more than the double of the value found previously (8.3%) by Filgueiras et al.[5] However, it does not necessarily mean that the scale has worsened, as will be later discussed. For instance, 287 of the 480 items (59.8%) improved their item-to-total correlation between years. In contrast, 185 of the 480 items (38.5%) presented lower results in 2011 when compared to 2010; and 8 of the 480 items (1.7%) remained the same.

Table 3 deepens our comprehension of the Cronbach’s α results by showing their increment between 2010 and 2011. A few phenomena can be observed. The first is the expectation of item-to-total improvement after item modifications, which should lead to increased α values. That happened to 19 of the 80 scales (23.7%). The second phenomenon consists of changes in Cronbach’s α despite the absence of item modifications. That happens in 40 of the 80 scales (50.0%). Such type of change across years of application allow raising a few hypotheses, which will be discussed later, despite a small oscillation in Cronbach’s α in response to differences between samples seems acceptable. The third phenomenon happens when the modifications seem to impair the reliability of the scale, that is, the α is lower in 2011 than in 2010. That happens in 17 of the 80 scales (21.3%). Finally, the fourth phenomenon is the α remaining the same, which happens in 4 of the 80 scales (5%). Table 4 shows which items improved or worsened among the modified items of the ASQ-BR-2011.

Table 3.

Internal Consistency (Cronbach’s α) Difference Between Years[a].

Age Interval (in Months)	Cronbach’s α Increment (2011 minus 2010)
	Communication	Gross Motor	Fine Motor	Problem Solving	Personal/Social
10	0.05	0.00	0.03	0.17	0.22
12	0.03	−0.02	0.07	0.06	0.03
14	0.06	0.05	0.06	0.08	0.03
16	0.04	−0.06	0.07	−0.03	0.03
18	0.06	−0.03	−0.03	−0.05	0.11
20	0.06	0.08	−0.01	−0.13	0.06
22	−0.14	−0.10	−0.03	0.00	0.06
24	0.01	−0.06	0.03	−0.02	0.06
27	−0.09	−0.02	0.09	−0.01	−0.45
30	−0.03	−0.04	0.10	0.09	−0.18
33	−0.05	−0.54	−0.54	−0.47	0.05
36	−0.09	−0.51	−0.39	0.01	−0.07
42	0.03	−0.04	−0.08	0.00	0.00
48	0.11	0.07	0.04	0.03	0.05
54	0.02	0.08	0.03	−0.33	0.05
60	−0.34	−0.32	0.05	0.06	−0.26

Positive values mean increment of Cronbach’s α, and negative values correspond to decrement of internal consistency.

Table 4.

Item-to-Total Correlation Changes Using the Difference Between the Results of 2011 and 2010[a].

Modified Items by Domain (Item Number in the Scale, in Age Interval Order)	Delta (Δ) of Item-to-Total Correlation Among the Modified Items (2011 minus 2010) in Each Questionnaire (Age Intervals in Months)
	10	12	14	16	18	20	22	24	27	30	33	36	42	48	54	60
Communication
6/4	0.09	0.00	—	—	—	—	—	—	—	—	—	—	—	—	—	—
6/1/1	—	—	—	—	—	0.03	−0.03	—	—	−0.02	—	—	—	—	—	—
Gross Motor
1	—	—	—	—	—	—	—	0.01	—	—	—	—	—	—	—	—
3/1	—	—	—	—	—	—	—	0.06	−0.28	—	—	—	—	—	—	—
6/5/4/3	—	—	—	—	—	—	—	—	0.06	−0.01	−0.27	−0.25	—	—	—	—
Fine Motor
5	—	—	—	−0.01	—	—	—	—	—	—	—	—	—	—	—	—
4/2	—	-	—	—	—	—	—	0.10	0.00	—	—	—	—	—	—	—
3	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.05	—
Problem Solving
4	0.15	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—
5	0.26	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—
5/3/4/5	—	—	—	—	0.05	0.01	0.03	−0.02	—	—	—	—	—	—	—	—
2	—	—	—	—	—	−0.05	—	—	—	—	—	—	—	—	—	—
4/3	—	—	—	—	—	—	—	—	−0.09	0.00	—	—	—	—	—	—
4/2/3	—	—	—	—	—	—	—	—	—	−0.01	−0.19	0.06	—	—	—	—
4/4/1	—	—	—	—	—	—	—	—	—	—	−0.21	0.01	0.02	—	—	—
Personal/Social
2	0.37	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—
3	0.47	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—
2	—	—	−0.15	—	—	—	—	—	—	—	—	—	—	—	—	—
2	—	—	—	0.10	—	—	—	—	—	—	—	—	—	—	—	—
3/3/1	—	—	0.05	0.14	—	0.10	—	—	—	—	—	—	—	—	—	—
4	—	—	−0.07	—	—	—	—	—	—	—	—	—	—	—	—	—
1	—	—	—	—	0.09	—	—	—	—	—	—	—	—	—	—	—
5/3/1/1	—	—	—	—	0.13	0.09	0.08	0.00	—	—	—	—	—	—	—	—
4/2	—	—	—	—	—	0.08	0.00	—	—	—	—	—	—	—	—	—
5/3	—	—	—	—	—	0.18	0.15	—	—	—	—	—	—	—	—	—
6/4/5/1/3/2	—	—	—	—	—	0.26	0.05	0.04	−0.16	−0.12	−0.04	—	—	—	—	—
2/2/1/3	—	—	—	—	—	—	—	—	—	—	—	—	0.06	0.01	0.12	−0.26
4/1/4/5	—	—	—	—	—	—	—	—	−0.12	−0.12	—	0.08	0.05	—	—	—
3/3	—	—	—	—	—	—	—	—	—	—	—	—	—	0.06	0.04	—
5	—	—	—	—	—	—	—	—	—	—	—	—	—	0.07	—	—
6/6/5	—	—	—	—	—	—	—	—	—	—	—	—	0.07	0.06	0.03	—

Positive values mean increment of item-to-total correlation, and negative values correspond to decrement of item-to-total correlation. Bold faces represent significant (p<0.05) improvements of item-to-total correlation using r-to-z transformation.

Discussion

The objective of the present study is improving the ASQ-BR by trying to understand and modifying the problematic items highlighted by Filgueiras et al[5] in the first adaptation of the ASQ-3 to the Brazilian context. Two criteria were established to modify ASQ-BR’s items: (a) item-to-total correlation below 0.30 and (b) items related to activities that do not take place in Brazilian public day care centers. The second criterion was indeed controversial since the ASQ-BR was developed to assess children in a public day care center environment and there was no statistical evidence to support changes in the work by Filgueiras et al.[5] However, teachers and educationalists from Rio de Janeiro and Brazil urged to question through press and social media if ASQ items’ contents were really adequate to the Rio de Janeiro’s public day care system.[18,19] That led ultimately to strong social demands for alterations in several ASQ-BR-2011 items, despite there being no statistical problems reported by Filgueiras et al.[5]

Children’s development in either ASQ-BR or ASQ-BR-2011 seems to be similar in a few domains and different in other. Those results need to be carefully discussed because different causes can lead to significant differences between groups. Regardless of that, the expectation was no statistical differences, since the sampling method and the large size of the sample respected Fisher’s law of large numbers.[20] The significant differences between years may have been caused either by modifications in the test application process or on certain items.

Initial results showed a balanced sample with boys and girls divided in nearly a 50/50 distribution. This is similar to the sex distribution in the Brazilian population—51% of boys and 49% of girls[21]—and can be regarded as evidence that the research in fact collected reliable data. The present research data derive from public day care centers, as did the ASQ-BR. Nevertheless, the ASQ-BR-2011 diversely includes data from public preschools, thus highly enlarging the variety of the sample. Indeed, the municipal educational structure of Rio de Janeiro enrolls children from 6 to 54 months old in day care centers. However, children who are older than 60 months are not enrolled in day care centers, but in preschools—with very few exceptions. That should explain the difference between Filgueiras et al’s[5] and the present study’s sample sizes in the 60-month questionnaire.

Communication Scales

The Communication domain had only one significant difference, the 60-month scale. Alpha statistics show a decrease of 0.34 between 2010 and 2011. Besides, the ASQ-BR-2011 scale is bidimensional against an unidimensional scale in 2010. Those differences lead to the conclusion that 2010 and 2011 60-month scales measure communication differently, although no modifications were made in their items. The inclusion of children from public preschools in the ASQ-BR-2011 suggests that the instrument was not well adapted to the school environment, though proper to day care centers. Another possibility is higher effectiveness of the training for the subjects of 2010 as compared to those in 2011, since only directors of day care centers were trained in years. Diversely, the directors of preschools were only included in the training in 2011 and consequently did not receive as much training as the latter.

Despite it remaining unclear whether either of the above-mentioned situation occurred, there is a third and more likely possibility. As explained before, the municipal educational structure in Rio de Janeiro enrolls children older than 60 months in preschools instead of day care centers. That could explain the difference in sample sizes and lead to the fact that the children surveyed through the 60-month questionnaire in the 2011 sample are older than the 60-month questionnaire children in the 2010 sample. After careful consideration, it can be stated that the ASQ-BR sample’s characteristics show that the mean age was 57.39 (SD = 0.95) for boys and 58.23 (SD = 1.13) for girls.[5] In contrast, the children from the ASQ-BR-2011 sample seem to be 4 months older, with an expected increase of the standard deviation (see Table 1).

Regarding language, several developmental hallmarks take place when the child is 5 years old. For example, grammar knowledge regarding relative and conditional clauses is apparently not fully developed in children less than 5 years old. Nevertheless, sentence comprehension shows significant improvement in main clauses and pseudo-clauses between 4 and 5 years.[22] There is evidence that conceptual knowledge develops especially at the age of 5 because children this age tend to perform systematically well in metalinguistic tasks, but not before.[23] The Secretary of Education of Rio de Janeiro provides teachers and caregivers with a list of recommended tasks and activities according to the children’s age,[24] and such guidelines have significantly more activities regarding linguistics in preschools as compared to day care centers. Perhaps the 4-month difference in the mean age between the ASQ-BR and ASQ-BR-2011 samples combined with the fact that children in preschools practice more linguistic tasks than children in child day care centers lead to the statistical difference between 2010 and 2011.

Regarding reliability, only 6 of the 16 Communication scales presented a decrease of Cronbach’s α. Among those, only the 60-month scale crossed the critical value of 0.65. The remaining 10 of the 16 scales actually improved and the item modifications apparently helped. Altogether, the only recommendation with respect to ASQ-BR’s Communication scales is changing problematic items of the 60-month scale, since it is the only scale above the established criterion of Cronbach’s α.

Gross Motor Scales

The Gross motor domain seems to be the most stable in terms of statistical differences between years. No significant differences were found in any age interval. That suggests that the Gross motor domain is assessed similarly in both ASQ-BR and ASQ-BR-2011. Only 4 of the 16 scales had α below 0.65. The difference (Δ) between the 2010 and 2011 scales was lower than 0.10. Nevertheless, considering the reliability indicated by Cronbach’s α, the previous statement seems not to be fully accurate. Among the same 4 scales—22, 33, 36, and 60 months—2 scales (33 and 36 months) had items modified unsuccessfully.

The considerations regarding the 60-month age interval in the Communication domain seem to apply to the Gross motor domain as well, especially with respect to differences between samples. Besides, the content of the items were not fully modified. The only noticeable alteration was regarding where the caregiver or teacher should observe the behavior—“( . . . ) at home, at the house of a friend or in a park” was changed to “( . . . ) at school, at the child day care center, at home, at the house of a friend or at a park.” Essentially, the multidisciplinary panel tried to bring the item content closer to respondents’ daily activities. Such attitude was synergic with the International Test Commission guidelines for cross-cultural adaptations.[12] Even so, the modifications proposed seemed empirically ineffective. Based on that outcome, at least one of the following 2 measures should be adopted to improve reliability: (a) change the item back to the ASQ-BR version or (b) try to modify the item once again so that it really assesses the content the domain to which it belongs is supposed to assess.

Fine Motor Scales

The fine motor scales presented 7 age intervals with statistical differences between years of assessment. A few hypotheses can be raised to explain these differences. The first possibility—the modification of an item disturbing the way the scale measures the domain—was excluded since the delta between 2010’s and 2011’s item-to-total correlation was positive or very close to zero (see Table 4). Another possible explanation is the effect of time on teachers’ and caregivers’ evaluations. Time is indeed a variable usually associated with validity and reliability decrease.[25] There are several reasons for that: cultural and social changes, existence/absence of training on the instrument, professional expertise, and so on.[26] Two possibilities seem plausible in this issue: (a) teachers and caregivers did not receive proper training on the ASQ-BR-2011 or (b) children in this age range did improve their fine motor skills from 2010 to 2011.

At the beginning of 2011, the Secretary of Education of Rio de Janeiro hired 1500 teachers for child day care centers through a public selection,[27] which meant an increase of over 10% of professionals using the ASQ-BR-2011. Despite the efforts for adequate training given to the directors of municipal schools and day care centers, it is not entirely possible to guarantee the quality of the training passed on to the teachers by the directors. The novelty of the profession for the recently hired teachers as well as some lack of quality on directors’ training may be raised as causes to the difference on the average. The second possible explanation is the development of new activities in child day care centers in Rio due to 2010’s assessment. The Secretary of Education had developed a free adaptation of the ASQ-3’s learning activities book.[28] The book gives teachers and caregivers guidelines of how to improve children performance in classroom regarding each ASQ domain. There are no official reports or information about the effect of those activities in municipal children day care centers in Rio de Janeiro but the initiative might be an explanation to the difference of averages in the fine motor skills between 2010 and 2011.

Problem Solving Scales

The Problem solving scales presented only 4 age intervals with significant differences: 16, 20, 22, and 60 months. Each case seems to imply different explanations. First, the 16-month problem solving scale had no item modification, thus eliminating this explanation. Despite of the α decrease—α = 0.80 in 2010 against α = 0.77 in 2011—it is still a reliable scale and the decreased α value does not seem to have changed enough to justify the mean difference. A similar situation happened with the 22-month scale. Regardless of having one item altered, the item-to-total correlation delta of the modified item between years is 0.05. That means that the item improved along years. Even so, the α statistics remained the same, which suggests the scale as a whole has counterbalanced the modification. As discussed in the Fine motor domain section, the policies of the Secretary of Education of Rio de Janeiro are the most plausible explanation for the 16- and 22-month Problem solving cases.

The 20-month scale had 2 items changed. One of them clearly impaired the rest of the scale. The same happened with Cronbach’s α on the 33-month scale. Based on these results, either one or a group of changed items may have deviated from the domain assessment, modifying the way the scale measures problem solving skills and, consequently, affecting average. Nunnally and Bernstein[15] argue that 2 aspects influence Cronbach’s α results: number of subjects in the sample and number of items. The ASQ-BR has indeed only 6 items per domain, thus being quite sensitive to changes. This is due to a set of items with balanced structure that is confirmed empirically along its development and psychometric analyses. A cross-cultural adaptation must be done carefully enough to be as close as possible to the original scale and, at the same time, deeply adapted to the cultural context of the target population.[12] The balance between both conditions is pivotal to a successful adaptation. On the 33-month scale, the average did not significantly change between 2010 and 2011, though that does not seem to be the case for the 20-month scale. Besides, there is the possibility that this difference is explained by Rio de Janeiro’s public policies. Nevertheless, changing the structure of the scale can lead to reliability loss, which occurred on both scales.

Finally, the 60-month Problem solving scale showed statistical difference of averages between years without changing items and significant modifications of the α statistics. The same argument used to explain this age interval regarding the Communication domain might be applied here, if there is evidence that 5 year olds is also a hallmark to problem solving skills. Executive functions, the foundation of problem solving, indeed present a gap in performance of 4-year-old children when compared to 5-year-old children.[29,30] For example, task switching and inhibitory control are 2 domains of the executive functions that are assessed by the Dimensional Change Card Sort Task. When this task is administered in both age intervals, their performance is statistically different between the ages of 4 and 5 years.[29] Thus, it makes sense to conclude that children in Rio de Janeiro’s preschools perform better in problem solving tasks than children in day care centers.

Personal/Social Scales

This domain of scales presented the largest number of modified items: 18 of the 96 items (18.8%) of the Personal/social domain. Among the age intervals of the ASQ-BR, only the 12-month scale did not have alterations. That was due to the fact that the personal/social scales were the most impaired ones in terms of internal consistency in the study of Filgueiras et al[5]—67% of the scales had α below 0.65. The Personal/social domain showed the highest improvement among all the scales from 2010 to 2011. The comparison between years regarding Cronbach’s α shows that 11 of the 16 scales (68.8%) increased their internal consistency, and among those, 4 (25.0%) crossed the critical value of 0.65.

The results showed that there was no statistical difference between the averages of 2010 and 2011, with the exception of the 20-, 22-, 24-, and 27-month age intervals. Regarding the first 3 scales—20, 22, and 24 months—there was no item with item-to-total correlation higher or lower enough to justify such discrepancy on the mean. One of the possibilities for that is, as mentioned before, that the new teachers’ performance or their training were somehow compromised, and that led to some statistical difference. On the 27-month Personal/social scale, one item was changed and apparently the internal consistency fell significantly when compared to the ASQ-BR’s α for the same age interval—from 0.63 to 0.18, a difference of −0.47. The item change probably impaired the whole scale and definitely jeopardized the reliability. A compromised reliability can be the explanation for the difference on the averages between 2010 and 2011.

Regarding internal consistency, Cronbach’s α is below 0.65 in 8 of the 16 scales (50.0%)—22, 27, 30, 36, 42, 48, 54, and 60. That is better than the results in ASQ-BR for this statistics (67%). Despite improving, the ASQ-BR-2011 still needs several adjustments to be considered acceptable in terms of reliability. Two different situations occurred and should be considered: (a) the item-to-total correlation increased though the scale remained below 0.65 and (b) the item-to-total correlation decreased and it impaired the consistency of the scale. A necessary step relative to the first situation is to examine the other items and verify which ones lower the scale’s reliability. Personal/social scales 22, 36, 42, 48, and 54 are included in this situation. We recommend that further studies evaluate those scales carefully and propose changes in the problematic items in order to improve them.

In the second situation—scales 27, 30, and 60—5 different items were diagnosed with item-to-total correlation decrease. The personal/social 27- and 30-month scales presented the same items with decreased item-to-total correlation if children imitate adults in at least 1 of the 4 proposed behaviors or if they push a cart while avoiding corners and blockades. Among the 4 behaviors listed on the ASQ-BR for the first item, 2 were altered: (a) “to tap on the chicks” was changed to “to send kisses”; and (b) “to pull his/her own ear” was changed to “to put a hand on the top of the head.” Both changes were made for the reasons mentioned before, that is, to improve internal consistency and respect the opinion of teachers and caregivers, who thought both behaviors somehow incited violence. Those changes did not seem to produce the desired effects, indeed tended to raise more questions than answers. In the item the child is asked to imitate specific adult behavior but the requested behavior cannot not be easily imitated by Brazilian children. That means there was some maladaptation of the original item. Perhaps some other imitation game, such as the Brazilian version of “monkey see, monkey do,” would more appropriate and should replace the item.

In the second item, children were asked to avoid corners and blockades by pushing a cart. However, in the ASQ-BR-2011, children were also allowed to do the same thing by pushing car toys. Despite the main behavior focused by the item was avoiding obstacles, pushing a cart seems to have a different course of action than pushing a car toy—those are considerably smaller and do not require that the child moves his/her entire body. Possibly, pushing a real cart or a toy consists of 2 significantly different tasks and teachers and caregivers use different strategies for children’s responses. Some consider there was task completion if the child does either one of the tasks while others consider task completion was achieved only if the child does both tasks. This peculiarity of the interpretation of the item might have caused some confusion among respondents, which may have led to poor statistics. The present study recommends adjusting the item to just 1 of the 2 behaviors so that the item-to-total correlation may improve.

On the 60-month scale, only one item was modified; nonetheless, it got worse. The ASQ-BR item inquires if the child is capable of serving herself by serving food from a bigger recipient to a smaller one. This kind of task is not common in child day care centers and preschools in Brazil since children’s plates are ready when they are served, which makes this example of independent behavior impossible to be observed. The item was modified so that a behavior closer to the Brazilian reality could be found. The new version “Does the child know how to eat dessert from a pot and discard unwanted food such as fruit peel or seeds?” was fully changed, despite the target domain—personal independence—being kept. Filgueiras et al[5] suggest that the ability of doing things without asking adults is an important hallmark in development of the Personal/social domain, which leads to full independence. Despite maintaining the same idea, the items seem to measure the Personal/social domain differently. The distortion regarding the assessment of the domain may be the cause for the problematic statistics in the item. Future studies should study the ASQ-BR-2011 in order to make it clearer and closer to the context of Brazilian day care centers and preschool activities.

Conclusions

The present study seems to have partially achieved its objective. Several scales were improved due to item modification and presented a higher reliability index—Cronbach’s α—than the first year of assessment. The ASQ-BR-2011 showed better psychometric properties when compared with the 2010 version, ASQ-BR. Regardless, a few limitations in this study can be addressed for future research in the area. The study of item difficulty and item information is lacking is this study and interesting statistics could arise from it. For example, if an item is too difficult for a group of children, it might not help the scale. The same reasoning is valid for when an item is too easy: it would not discriminate children accurately. Either situation is a possible cause for the lack of balance of a few scales of the ASQ-BR-2011.

The modifications on the 60-month questionnaire were followed by the inclusion of preschool children in the sample. That may be considered a problem due to the combination of factors that could confound variables. It is impossible to state for sure if reliability problems with the 60-month scales—especially personal/social—are due to item modifications or differences in sampling. It is strongly recommended that future studies evaluate ASQ-BR-2011 item modifications without preschool children.

Author Contributions

CMTS contributed to conception and design; drafted manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring itegrity and accuracy.

AF contributed to design; contributed to acquisition, analysis, and interpretation; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring itegrity and accuracy.

JLF contributed to conception and design; contributed to interpretation; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring itegrity and accuracy.