Feasibility, reliability, and validity of adolescent health status measurement by the Child Health Questionnaire Child Form (CHQ-CF): internet administration compared with the standard paper version.

AIMS: In this study we evaluated indicators of the feasibility, reliability, and validity of the Child Health Questionnaire-Child Form (CHQ-CF). We compared the results in a subgroup of adolescents who completed the standard paper version of the CHQ-CF with the results in another subgroup of adolescents who completed an internet version, i.e., an online, web-based CHQ-CF questionnaire.
METHODS: Under supervision at school, 1,071 adolescents were randomized to complete the CHQ-CF and items on chronic conditions by a paper questionnaire or by an internet administered questionnaire.
RESULTS: The participation rate was 87%; age range 13-17 years. The internet administration resulted in fewer missing answers. All but one multi-item scale showed internal consistency reliability (Cronbach's alpha > 0.70). All scales clearly discriminated between adolescents with no, a few, or many self-reported chronic conditions. The paper administration resulted in statistically significant, higher scores on 4 of 10 CHQ-CF scales compared with the internet administration (P < 0.05), but Cohen's effect sizes d were </= 0.21. Mode of administration interacted significantly with age (P < 0.05) on four CHQ-CF scales, but Cohen's effect sizes for these differences were also </= 0.21.
CONCLUSION: This study supports the feasibility, internal consistency reliability of the scales, and construct validity of the CHQ-CF administered by either a paper questionnaire or online questionnaire. Given Cohen's suggested guidelines for the interpretation of effect sizes, i.e., 0.20-0.50 indicates a small effect, differences in CHQ-CF scale scores between paper and internet administration can be considered as negligible or small.

RCT Entities:   Population Interventions Outcomes

Introduction

During the past two decades, several measures have become available to describe generic health-related quality of life in pediatrics, but adolescent self-report questionnaires received relatively little attention until now [1, 2]. The Child Health Questionnaire (CHQ) is one of the most widely used pediatric health-related quality of life measures and has been translated into 21 languages (32 countries). There is a form for parents and also a self-report form for adolescents, the Child Health Questionnaire-Child Form (CHQ-CF) [2-7]. The CHQ covers physical and psychosocial aspects of health, and includes the impact of child health problems or handicaps on family life [3]. This study focuses on the evaluation of missing answers at the item level, distribution of the scale scores, reliability, and validity of the CHQ-CF in an adolescent population.

We expect the commonly used paper format of the CHQ and other health questionnaires to be increasingly replaced by internet versions, especially in adolescent populations that are accustomed to the use of computers and the internet [8]. From the perspective of clinicians and researchers, the advantages of using the internet include avoiding paper work, on-line data-entry, and procedures designed to reduce the number of missing answers and the length of questionnaires [9, 10].

In general, the mode of questionnaire administration (e.g., written questionnaire, face to face interview, telephone interview, computer questionnaire) may affect the participation rate, number of missing answers, psychometric properties, and actual scores [11-14]. With regard to health questionnaires, several studies demonstrated some differences between the commonly used paper versions and computer versions of the same questionnaires [15-17]. Especially in studies comparing paper and computer questionnaires on sensitive topics, administration via computer was found to increase reporting of e.g., drug use or unsafe sexual behaviors, as this medium is apparently perceived as providing more privacy than a paper form [18-20].

With regard to online, i.e., internet or web-based administration of health questionnaires, several studies have demonstrated that online health questionnaires are feasible in various settings, especially among adolescents [21, 22]. However, very few randomized studies have evaluated whether psychometric properties and scores differ between the paper and the internet mode of administration of the same health questionnaire [23-26].

In this study, we compared indicators of the feasibility, reliability, and validity of the CHQ-CF in a subgroup of adolescents who completed the standard paper version of the CHQ-CF with the same indicators in another subgroup of adolescents who completed a newly developed internet version of the questionnaire. Additionally, we compared the mean CHQ-CF scores and distributions of the scale scores between both subgroups. A randomized parallel group design was applied in a large adolescent population (13–7 years old), ensuring that both subgroups were comparable.

The study assessed and compared the paper and internet mode of CHQ-CF administration with regard to the following indicators:

the number of missing answers (indicator of feasibility),

the distribution of the scale scores including mean scale scores in the whole sample and in gender and age specific subgroups,

the internal consistency reliability of multi-item scales (indicator of reliability),

the ability of the CHQ-CF to discriminate between subgroups with and without self-reported chronic conditions (indicator of construct validity).

Methods

Study population

In 2003, 1,071 students in 55 classes of various educational levels in the 3rd year of seven secondary schools (13–7 years old) in the area of Vlaardingen (metropolitan area) and Harderwijk (rural area), The Netherlands, were invited to complete the Child Health Questionnaire Child Form (CHQ-CF). The parents and students each received written information about the study several weeks before data collection; parents could refuse their child’s participation, and participation by the students was voluntary.

Data collection

The CHQ-CF consists of 87 items with 4, 5, or 6 response options divided over 10 multi-item scales and two single-item scales (Table 1) [3]. To reduce respondent burden, the item “change-in-health–was not fielded in this study, and the CHQ-CF scales “role functioning-emotional–and “role functioning-behavioral–were combined into a single scale. The combination of the two role functioning scales is a departure from the CHQ-CF instructions that makes the test analogous to the parent form of the CHQ in this regard [3]. For each scale, items were summed up (some recoded/recalibrated) and transformed into a 0 (worst possible score) to 100 (best possible score) scale [3]. Items on standard socio-demographic variables and the prevalence of seven chronic conditions were included in the questionnaire. From the conventional paper format, using the same wording of the items and instructions, an internet version of the questionnaire was developed through a generic internet tool using PHP (4.0.1), MySQL (3.22), and JavaScript (1.3) [27]. The internet version of the questionnaire listed the items of each CHQ-CF scale on a separate web-page. The internet version did not allow the respondent to select more than one answer to each item of the CHQ-CF and it checked the questionnaire for missing answers before the respondent could “logout– If one or more of the items were not answered, the internet version prompted the respondent to go back to complete those items; but, if the user failed to “logout–properly, missing answers would remain.

Table 1

CHQ-CF scales, items per scale, and interpretation of low and high scores

CHQ-CF Scales	Number of items	Description low score	Description high score
Physical functioning (PF)	9	Child is greatly limited in performing all physical activities, including self-care, due to health	Child performs all types of physical activities, including the most vigorous without limitations due to health
Role functioning: Emotional (RE)b	3	Child is greatly limited in school work or activities with friends as a result of emotional problems	Child has no limitations in schoolwork or activities with friends as a result of emotional problem
Role functioning: Behavioral (RB)b	3	Child is greatly limited in school work or activities with friends as a result of behavior problems	Child has no limitations in schoolwork or activities with friends as a result of behavior problems
Role functioning: Physical (RP)	3	Child is greatly limited in school work or activities with friends as a result of physical health	Child has no limitations in schoolwork or activities with friends as a result of physical health
Bodily pain (BP)	2	Child has extremely severe, frequent and limiting bodily pain	Child has no pain or limitations due to pain
General behavior (BE)	17	Child very often exhibits aggressive, immature, delinquent behavior	Child never exhibits aggressive, immature, delinquent behavior
Mental health (MH)	16	Child has feelings of anxiety and depression all of the time	Child feels peaceful, happy, and calm all of the time
Self esteem (SE)	14	Child is very dissatisfied with abilities, looks, family/peer relationships and life overall	Child is very satisfied with looks, family/peer relationships and life abilities, overall
General health perceptions (GH)	12	Child believes his/her health is poor and likely to get worse	Child believes his/her health is excellent and will continue to be so
Change in health (CH)c	1	Child’s health is much worse now than 1 year ago	Child’s health is much better now than 1 year ago
Family activities (FA)	6	The child’s health very often limits and interrupts family activities or is a source of family tension.	The child’s health never limits or interrupts family activities nor is family a source of tension
Family cohesion (FC)	1	Family’s ability to get along is rated “poor–	Family’s ability to get along is rated “excellent–

Reproduced with permission [3]

The CHQ-CF scales “Role functioning-emotional–and “Role functioning-behavioral–were merged into a single scale “Role functioning-emotional/behavioral–(REB) in this study

This single-item scale was not fielded in this study

Randomization

Within each school class, students were randomly assigned to either the paper or the internet mode of administration using SPSS-generated random numbers. Students completed the questionnaires, either on paper or online in a classroom with computers linked to the internet, under the supervision of a research assistant; the students were allowed adequate privacy.

Analysis

Preparatory secondary vocational education was labelled as “lower secondary education”; secondary schools that prepare students for higher professional training as "intermediate secondary education", and university preparatory secondary education as "higher secondary education". Differences between the characteristics of the participants allocated to the paper versus the internet versions of the questionnaires were tested by Student’s t-test and the χ2 test. We assessed the frequency of missing answers to CHQ-CF items; the difference in the number of missing answers between the two formats was assessed by the Mann-Whitney U test. We assessed the distributions of the CHQ-CF scale scores to evaluate floor and ceiling effects (≥25% of the respondents having the lowest/highest score) for both modes of administration. Differences between CHQ-CF scale scores by format in the total sample were assessed by Mann-Whitney U tests. Additionally, after transforming the scale scores into ranks, ANOVA was applied to test whether the mode of questionnaire administration interacted with the variables gender (male n = 432; female n = 501) and age (13–4 year olds, n = 399; 15–7 year olds, n = 534). Cohen’s effect sizes, defined as d = [Mean(a) Mean(b)]/SD, where the denominator was the square root of [(n-1)SD2 + (n-1)SD2] / [(n-1) + (n-1)], were applied to indicate the relative magnitude of score differences between modes of administration. Here, the letters "a" and "b" refer to the subgroups administered the paper and internet forms of the test, respectively [28]. Following Cohen’s suggested guidelines, 0.20 ≤d < 0.50 indicated a “small effect– 0.50 ≤d < 0.80 a “medium effect– and d ≥0.80 a “large effect–[28]; Norman et al. have suggested that, in general, d = 0.50 can be considered as threshold for a “minimally important difference–(MID) [29]. Cronbach’s α was applied to evaluate the internal consistency reliability of CHQ-CF multi-item scales by format; α of 0.70 or higher was considered to indicate sufficient internal consistency reliability [30]. We applied statistical tests of the hypothesis that the Cronbach’s α reliability coefficients of CHQ multi-item scales in the sample administered the test on paper were equal to those administered the test online [31]. We applied item-level discriminant tests to evaluate whether the CHQ-CF items represent separate domains. For each mode of questionnaire administration, we evaluated whether (on average) correlation coefficients (Pearson-r correlation coefficients) between the items and their own scale score (without the item under consideration) were higher than the correlation coefficients between these items and any other scale. The average Pearson-r correlation coefficients were calculated by applying Fisher’s z transformations [32]; we tested whether the differences between the average Pearson-r correlation coefficients in the subgroup administered the paper form and in the subgroup administered the test online were statistically significant [33]. We assessed the CHQ-CF’s ability to discriminate between subgroups with 0, 1 or 2, and 3 or more chronic conditions, after having transformed the CHQ-CF scale scores into ranks, by ANOVA with the independent variables “number of chronic conditions– "mode of questionnaire administration– and the interaction term “number of chronic conditions–“mode of questionnaire administration– Cohen’s effect sizes d = [Mean(a) Mean(b)]/SD in the condition subgroup were calculated for 1 or 2 versus 0 conditions, and for ≥3 versus 0 conditions. The designations "a" and "b" refer to the subgroups without chronic conditions and those with chronic conditions, respectively [28].

All analyses were done using SPSS, Version 11.0.1. The medical ethical committee of the Erasmus MC-University Medical Center Rotterdam, approved the study.

Results

Participants and randomization

The participation rate was 87%. The age range of the participants was 13–7 years (mean age 14.7 years; SD 0.68), 54% were female, 93% were born in the Netherlands, and the majority attended lower secondary education (Table 2). The prevalence of self-reported chronic conditions was as follows: asthma, 8%; allergies, 25%; hearing problems, 7%; visual problems, 8%; headaches or migraine, 17%; chronic lower back pain, 17%; and depression or anxiety attacks, 8% (Table 2). These characteristics were equally distributed in the groups assigned to the paper and internet versions of the questionnaires (P ≥0.05; Table 2). The demographic characteristics of the participants (age, gender, country of birth, and educational level) reflected those of the general population of Dutch adolescents [34].

Table 2

Characteristics of study participants (total sample: n = 933; participants assigned to the paper questionnaire:n = 475; participants assigned to the internet questionnaire: n = 458)

	Total study group (n = 933)			Group with paper mode of administration (n = 458)			Group with Internet administration (n = 475)			P-value
	Mean (SD) or Range	n	% of Participants	Mean (SD) or Range	n	% of Participants	Mean (SD) or Range	n	% of Participants	Internet versus paper mode
Demographic characteristics
Age (years)
Mean (SD)	14.7 (0.68)			14.7 (0.68)			14.7 (0.68)			0.61a
Range	13–7			13–7			13–7
Gender
Women		501	54%		244	51%		257	56%	0.17b
Born in the Netherlands
Yes		866	93%		441	93%		425	93%	0.90b
Educational level of the school
Lower secondary education		545	58%		274	58%		271	59%
Intermediate secondary education		179	19%		94	20%		85	19%	0.88c
Higher secondary education		209	22%		107	23%		102	22%
Chronic conditions:
Asthma
Yes		76	8%		40	8%		36	8%	0.81b
Allergies
Yes		229	25%	118	25%			111	24%	0.88b
Problems with hearing
Yes		62	7%	30	6%			32	7%	0.70b
Problems with seeing
Yes		78	8%	39	8%			39	9%	0.91b
Headaches or migraine
Yes		159	17%	81	17%			78	17%	1.00b
Chronic low back pain
Yes		159	17%	88	19%			71	16%	0.22b
Depression and/or anxiety attacks
Yes		74	8%	36	8%			38	8%	0.72b

Student’s t-test

Chi square test df = 1

Chi square test df = 2

Difference in the number of missing answers between different modes of CHQ-CF administration

At the item level, use of the paper version of the CHQ-CF resulted in more missing answers (0–.89% per item; mean 0.54%) compared with the internet version (0–.22% per item; mean 0.04%; P < 0.01).

CHQ-CF scores by mode of administration

A ceiling effect was observed for four CHQ-CF scales in the subgroup that completed the paper questionnaire, and 3 CHQ-CF scales in the subgroup that completed the internet questionnaire (Table 3). Four CHQ-CF scales, i.e., “general behavior– “role functioning-physical– “mental health– and “family activities– resulted in statistically significant, higher scores for paper versus internet administration (P < 0.05), but the effect sizes (d) were ≤0.21 (Table 3). The mode of questionnaire administration did not interact significantly with gender (P ≥0.05 regarding all scales), nor with age (P ≥0.05 regarding six scales), except for the CHQ-CF scales “role functioning-emotional/behavioral–(P < 0.05), “mental health–(P < 0.05), “self esteem–(P < 0.05), and “general health–(P < 0.01). Regarding these 4 CHQ-CF scales, administration of the paper version resulted in lower scores than online administration (or nearly equal scores in the case of “mental health– in the subgroup of 13–4 year olds, while in the subgroup of 15–7 year olds, paper administration resulted in higher scores compared with internet administration; the Cohen’s effect sizes (d) for these differences, regardless of sign, were ≤0.21 (data not shown).

Table 3

Comparison of mean scores, distributions of the scale scores, and other psychometric properties of CHQ-CF scales in subgroups with paper (n= 475) and internet modes (n = 458) of questionnaire administration

CHQ-CF scalesa (range 0–00)	Mode of admini stration	Mean (SD)	Paper versus internet mode of administration		Range of scores	% maxd	% mine	25th %tile	50th%tile	75th%tile	Cronbach’s alpha	Average item-own scale correlation f	Average item-other scale correlation
			P-value (MWU)b	Effect size dc
Physical Functioning	Paper	96.0 (6.9)	0.37	0.01	44–00	56	0	96	100	100	0.69	0.40	0.19
	Internet	95.8 (7.2)			37–00	53	0	93	100	100	0.72	0.44	0.18
Role funct.-Emo/beh	Paper	89.4 (17.2)	0.14	0.00	0–00	60	0	78	100	100	0.81	0.65	0.35
	Internet	89.5 (15.1)			0–00	54	0	86	100	100	0.70 g	0.51 g	0.28
Role funct.-Physical	Paper	95.0 (12.9)	0.02	0.05	22–00	81	0	100	100	100	0.86	0.74	0.30
	Internet	94.4 (11.7)			22–00	75	0	89	100	100	0.76 g	0.60 g	0.25
Bodily Pain	Paper	73.5 (22.7)	0.56	–0.05	0–00	25	0	60	80	100	0.88	0.80	0.35
	Internet	74.7 (21.4)			0–00	24	1	60	80	90	0.89	0.81	0.30
General Behavior	Paper	80.9 (10.6)	0.00	0.21	25–00	1	0	75	82	88	0.83	0.44	0.26
	Internet	78.6 (11.4)			35–00	0	0	72	79	87	0.85	0.46	0.26
Mental Health	Paper	76.5 (15.4)	0.01	0.12	14–00	2	0	69	80	89	0.92	0.63	0.35
	Internet	74.6 (14.8)			14–00	0	0	67	77	84	0.90 h	0.60	0.31
Self Esteem	Paper	74.7 (12.2)	0.99	0.02	20–00	1	0	68	75	82	0.88	0.56	0.30
	Internet	74.4 (11.6)			25–00	0	0	70	76	82	0.87	0.55	0.28
General Health	Paper	73.5 (16.5)	0.59	0.05	19–00	3	0	63	76	86	0.82	0.50	0.28
	Internet	72.6 (16.7)			20–00	1	0	63	75	86	0.81	0.58	0.28
Family Activities	Paper	80.0 (17.7)	0.03	0.11	8–00	17	0	67	83	96	0.81	0.58	0.24
	Internet	78.1 (16.9)			25–00	13	0	67	79	92	0.78	0.54	0.24
Family Cohesion	Paper	70.6 (23.5)	0.91	0.00	0–00	18	2	60	85	85	na	na	0.36
	Internet	70.7 (23.0)			0–00	16	2	60	85	85	na	na	0.33

The CHQ-CF scales ‘Role functioning-emotional–and ‘Role functioning-behavioral–were merged into a single scale in this study

Two-sided Mann-Whitney U test of scale scores of the group using paper, and the group using the internet version of the questionnaire

Difference of mean scores divided by weighted average of SDs in groups given the paper and internet version of the questionnaires [28, 29]

% of respondents with best/worst possible score (ceiling/floor)

Each item was correlated with the applicable ad-hoc scale without the item under consideration

Statistically significant differences of Cronbach’s alpha/average Pearson-r correlation coefficients in subgroups given paper or internet questionnaires (P ≤0.01; P ≤0.05) [31, 33].

na Not applicable; Role funct.-Emo/beh - Role functioning-emotional behavioral; Role funct.-Physical - Role functioning-physical

Internal consistency reliability of scales by mode of administration

Cronbach’s αs for the two formats were adequate for all CHQ-CF scales, except “physical functioning–in the subgroup administered the paper version of the questionnaire (α = 0.69). The two “role functioning–scales and “mental health–showed statistically significant, higher Cronbach’s αs in the subgroup administered the paper version of the questionnaire compared with the alphas in the subgroup administered the internet version (P < 0.01, respectively P < 0.05) (Table 3). All multi-item scales, regarding both modes of administration, showed higher average (corrected) item-own scale correlation coefficients than average item-other-scale correlation coefficients. The two “role functioning–scales showed statistically significant, higher average item-own scale correlation coefficients in the subgroup administered the paper version of the questionnaire compared with the item-own scale correlation coefficients in the subgroup administered the internet version (P < 0.01) (Table 3).

Construct validity by mode of administration

All mean CHQ-CF scale scores were lower in the subgroup with one or two reported conditions and in the subgroup with three or more reported conditions when either was compared with the subgroup with no reported conditions. For both modes of questionnaire administration, and for all CHQ-CF scales, the more chronic conditions that were reported, the higher the effect sizes compared with the subgroup with no chronic conditions. ANOVA showed statistically significant CHQ-CF score differences by “number of chronic conditions–for all scales (P < 0.01) (Table 4). The mode of questionnaire administration did not interact significantly with the variable “number of chronic conditions–(P ≥0.05 for all scales) (Table 4).

Table 4

Ability of the CHQ-CF scales to discriminate between subgroups differing in the participants–number of chronic conditions, for the group that was assigned to complete the paper version (n = 475) and for the group that was assigned to complete the internet version (n = 458)

		Number of chronic conditions per participant
CHQ-CF scalesa (range 0–00):	Mode of admini- stration	0 conditions	1 or 2 conditions	≥ conditions	1 or 2 versus 0 conditions	≥ versus 0 conditions	ANOVA P-value Number of chronic conditions	ANOVA P-value Internet versus paper mode	ANOVA P-value Interaction term Number of chronic conditions by mode of questionnaire administration
		n = 224 (Paper)n = 206 (Internet) Mean (SD)	n = 210 (Paper)n = 217 (Internet) Mean (SD)	n = 41 (Paper)n = 35 (Internet) Mean (SD)	Effect size db	Effect size db
Physical functioning	Paper	97 (5)	96 (7)	90 (10)	0.24	0.80*
	Internet	97 (5)	95 (7)	89 (12)	0.28	0.64*	0.00	0.95	0.52
Role funct.-emo/behav	Paper	93 (13)	88 (18)	77 (24)	0.25	0.64*
	Internet	91 (14)	89 (15)	79 (18)	0.15	0.66*	0.00	0.20	0.72
Role funct.-physical	Paper	97 (11)	94 (13)	91 (17)	0.19	0.34
	Internet	96 (10)	94 (13)	91 (13)	0.14	0.34	0.00	0.07	0.69
Bodily pain	Paper	82 (18)	70 (22)	47 (24)	0.52*	1.45*
	Internet	80 (18)	72 (21)	55 (24)	0.39	1.03*	0.00	0.20	0.32
General behavior	Paper	84 (9)	80 (10)	71 (13)	0.40	0.98*
	Internet	81 (10)	78 (11)	69 (14)	0.26	0.90*	0.00	0.03	0.65
Mental health	Paper	83 (10)	74 (16)	57 (16)	0.56*	1.58*
	Internet	80 (11)	72 (15)	59 (18)	0.50	1.20*	0.00	0.26	0.29
Self esteem	Paper	79 (10)	73 (12)	61 (12)	0.52*	1.44*
	Internet	77 (9)	73 (12)	66 (14)	0.33	0.80*	0.00	0.26	0.06
General health	Paper	80 (14)	71 (15)	54 (14)	0.57*	1.82*
	Internet	79 (13)	70 (17)	55 (18)	0.54*	1.29*	0.00	0.74	0.58
Family activities	Paper	83 (16)	79 (18)	70 (20)	0.26	0.67*
	Internet	81 (16)	78 (17)	66 (15)	0.17	0.97*	0.00	0.03	0.51
Family cohesion	Paper	77 (21)	68 (23)	53 (26)	0.38	0.89*
	Internet	74 (20)	68 (24)	64 (28)	0.24	0.38	0.00	0.18	0.05

The CHQ-CF scales ‘Role functioning-emotional–and ‘Role functioning-behavioral–were merged into a single scale in this study

Difference of the means divided by SD in the subgroup with chronic condition(s), where * indicates at least a “minimally important difference–(d ≥0.50) [28, 29]

Role funct.-Emo/beh - Role functioning-emotional behavioral; Role funct.-Physical - Role functioning-physical

Discussion and conclusions

In this study we applied a randomized design to compare the results of the Child Health Questionnaire-Child Form (CHQ-CF) administered by a paper questionnaire and by an online questionnaire. The results provided support for the feasibility, internal consistency reliability, and construct validity of the CHQ-CF scales. Both modes of questionnaire administration yielded comparable scale scores and showed comparable psychometric properties. Additionally, the study provided reference/norm scores for clinical studies (general population of 13–7 year olds).

Strengths of the current study

The participation rate was high. Study group characteristics (age, gender, country of birth, and educational levels) were representative of those of the general population of Dutch adolescents [34]. Randomization to either the paper or internet mode of administration of the CHQ-CF was successful with respect to the evaluated characteristics.

Limitations

We applied a randomized parallel group design that allows for the comparison of indicators of feasibility, reliability, and validity at the group level between a subgroup that completed a paper version and a subgroup that completed an internet version of the CHQ-CF. However, this did not allow an evaluation of whether the same person would provide equivalent or different answers to the same CHQ-CF questionnaire administered by the alternative mode, which would require a randomized crossover design [25, 35]. Such an evaluation at the individual level requires the respondent to forget all previously provided answers at the second assessment, e.g., by waiting 1 or 2 weeks between the two measurements. It also requires that there is no effect from having previously completed a CHQ-CF questionnaire by any mode, paper or internet, at the second assessment, and that scores by the same mode of administration after a relatively short interval, in the absence of changes in health status, are exactly the same. However, in an evaluation of retesting with the same paper version of the CHQ-CF after 2 weeks, 5 out of 10 CHQ-CF scales showed statistically significant, higher scores at the second measurement with Cohen’s effect sizes ranging from 0.25 to 0.40, while intraclass correlation coefficients between the first and second measurement ranged from 0.06 thru 0.84 [7]. Furthermore, in a randomized crossover design, “carry-over–effects may be present, i.e., completing an internet version before a paper version may have a different effect on the second assessment, than does completing a paper version before an internet version [35]. Despite the logistical and the above-mentioned methodological challenges, we recommend future studies comparing the paper and internet versions of the CHQ-CF applying a randomized crossover design to evaluate congruency of answers to CHQ-CF items at the individual level.

In this study, internet and paper questionnaires were completed in a controlled environment with adequate privacy and supervision. This may not be the case during future applications. We are unaware of the impact less privacy during completion of the questionnaires may have, but this would apply to both the paper and the internet versions of the questionnaire.

For both modes of questionnaire administration, we did not evaluate correlations between CHQ-CF scores and a relevant parent-rated questionnaire such as the CHQ-PF50 [2, 3]. Test-retest reliability of the CHQ-CF and its responsiveness and sensitivity to changes in health were not evaluated in the current study. The CHQ-CF has a relatively large number of items; therefore, we recommend developing a shorter version in the future.

Psychometric properties

The psychometric properties, with only a few exceptions, were equal between the two modes of questionnaire administration. The Cronbach’s α of the scale “physical functioning–in the subgroup administered the paper version of the questionnaire was just under 0.70, and the difference with the alpha in the subgroup administered the internet version was not statistically significant.

Missing values

Compared with the paper version, the internet version was successful in reducing the quantity of missing data. Theoretically, differences in selective partial non-response between formats might have contributed to differences in scores that were reported in this study. In our study, in the subgroup (n = 86) that had at least one missing answer to a paper CHQ-CF item, all scale means were somewhat lower than in the subgroup (n = 389) with no missing answers, but these differences were not significant (P ≥0.05). Thus, missing answers are not a source of the observed score differences.

Score differences between modes of questionnaire administration

Recently, Ritter et al. found no statistically significant score differences between internet and paper modes of administration for 16 health-related measures, but the study was conducted in an opportunity sample retrieved from the internet, which limits its generalizability [23]. In a randomized internet-paper comparison among adolescents concerning various health measures other than the CHQ-CF, only one statistically significant score difference was reported among 21 topics [24]. In another randomized adolescent study, a medical consumption index and 11 indicators of fruit consumption and determinants of fruit consumption did not show statistically significant score differences between internet and paper administration of the questionnaire, except for one measure that showed small score differences between modes of administration [25]. The International Study of Asthma and Allergies in Childhood (ISAAC) questionnaire did not show statistically significant score differences between internet and paper administration in two randomized adolescent studies [25, 26].

In our study, in the whole sample, the paper version resulted in slightly, yet statistically significant, higher scores for 4 of 10 CHQ-CF scales compared with the internet version. One plausible explanation is chance, since it should be considered that given multiple comparisons, there is a 1-in-20 chance of a false association for each comparison (Type I error at α = 0.05) [36]. A commonly used Bonferroni correction for 10 comparisons would imply an adjusted α = 0.05/10 = 0.005 [36]; at α = 0.005, only one score difference (regarding the scale “general behavior– was significant. Furthermore, given Cohen’s suggested guidelines for the interpretation of effect sizes, three of the four statistically significant differences between modes of administration can be considered as negligible (d ≤0.12), and one difference regarding the CHQ-scale “general behavior–(d = 0.21) can be considered as small [28]; all effect sizes were far below d = 0.50 that was suggested as an approximate threshold for “minimally important differences–by Norman et al. [29]. This study provides no explanations for the established small score differences between paper and internet administration, or for the established statistically significant, but small interaction effects of administration mode with age in the case of four CHQ-CF scales.

Conclusions

With increasing application of online health questionnaires rather than questionnaires on paper, especially in adolescent populations, it should be noted that comparison of results requires that the scores between these modes of administration do not show meaningful statistically significant differences. This study showed that, overall, paper and internet versions of the CHQ-CF yielded only a few, negligible or small, differences. Paper and internet modes of CHQ-CF administration may be combined in a single study, although researchers should consider the possibility of minor score differences depending on the mode of administration for some scales. We recommend repeated studies in other populations, including clinical populations, to confirm or reject our results.