The Ukrainian version of the pediatric Canadian acute respiratory illness and flu scale: a linguistic validation study.

BACKGROUND: There is no internationally recognized outcome measure for the assessment of acute respiratory tract infections (ARTIs) in children. The only identifiable scale initially developed for pediatric application has been the Canadian acute respiratory illness and flu scale (CARIFS). The aim of our trial was to adapt the English version of the CARIFS to the Ukrainian language.
MATERIALS AND METHODS: We performed forward and backward translation of the original version of the CARIFS according to the recommended standard. Then, the final CARIFS-based Ukrainian questionnaires were given to 149 caregivers whose 3-12 years old children suffered from ARTI. The questionnaires were completed twice by a caregiver 3-6 hours apart and once by a physician just after the second completion by a caregiver. The database was analyzed to assess the consistency (the Cronbach's α coefficient), sensitivity (the standardized response mean; the effect size), reliability (test-retest analysis), and validity (Pearson's correlation) of the CARIFS in the Ukrainian language.
RESULTS: The backward translation of the Ukrainian version of the CARIFS demonstrated its good correspondence to the English version. The Cronbach's α coefficient was 0.805, and item to total correlation coefficients varied from 0.185 to 0.665. The standardized response mean was 1.73, and the effect size was 2.50 suggesting good sensitivity of the scale. In the test-retest reliability analysis of 99 questionnaires, the median CARIFS score for the first and the second measurement was 19.0 (interquartile range [IQR]: 14.5-25.0) and 19.0 (IQR: 15.0-25.0), respectively, with a median change of 0.0 (IQR: -1.0 to 0.0, P=0.996). The Pearson's correlation coefficient between the CARIFS score completed by a responder and a physician was 0.832 (P=0.004).
CONCLUSION: The Ukrainian version of the CARIFS-based English questionnaire proved to be a consistent, sensitive, reliable, and valid instrument in the assessment of ARTI in preschool and elementary school children in the Ukrainian population.

Background

Due to universal prevalence, acute respiratory tract infections (ARTIs) require considerable health care and social resources that include medical visits, hospitalizations, a sick person/caregiver working day missed, and costs for medication. However, the assessment of ARTIs is difficult because it is based on the subjective perceptions of patients and their caregivers.

Until recently, there were no validated patient-related surveys for ARTIs either in adults or in children. Nowadays, the Wisconsin upper respiratory symptom survey is a recognized instrument for the assessment of ARTIs in adults, while others are emerging.1,2 Despite ARTIs in adults sharing common features with pediatric illness, the above-mentioned surveys include many items inconsistent with pediatric applications.

The only identifiable scale initially developed and validated as an instrument for the assessment of pediatric ARTIs has been the Canadian acute respiratory illness and flu scale (CARIFS).3 The CARIFS was successfully used in clinical trials conducted in native English-speaking countries.4,5 Meanwhile, foreign language versions of the CARIFS are not available.

The objective of our trial was to adapt the English version of the CARIFS to the Ukrainian language. The present study describes the translation process and evaluates the translated version in a Ukrainian pediatric population in terms of consistency, sensitivity, reliability, and validity.

Materials and methods

CARIFS questionnaire

The original CARIFS included the perception of 18 items describing the child’s ARTI (in order as they appeared in the questionnaire): 1) “poor appetite”; 2) “not sleeping well”; 3) “irritable, cranky, fussy”; 4) “feels unwell”; 5) “low energy, tired”; 6) “not playing well”; 7) “crying more than usual”; 8) “needing extra care”; 9) “clinginess”; 10) “headache”; 11)“sore throat”; 12) “muscle aches or pains”; 13) “fever”; 14) “cough”; 15) “nasal congestion/runny nose”; 16) “vomiting”; 17) “not interested in what’s going on”; and 18) “unable to get out of bed”. Symptoms were rated as “none”, “minor”, “moderate”, “major” problem or “do not know/not applicable”, with corresponding values of “0”, “1”, “2”, “3”, or “missing value”. The maximum score indicated the worst disease was 54 units (18×3) (4). The items were classified into three domains: activity (1–7, 17, and 18), parental impact (8 and 9), and symptoms (10–16).

Translation

The translation of the CARIFS was performed according to the recommendation of Guillemin et al.6 Two bilingual pediatricians with Ukrainian as their first language translated the English version into Ukrainian. In the translation process, priority was given to the meaning rather than equality of words. During a primary meeting, consensus was achieved on the initial Ukrainian version based on those two translations. Subsequently, six bilingual speakers with English as their native language performed backward translation into English. Neither of these translators had any relevant medical knowledge or knew anything about the CARIFS. Where there was good correspondence between the source and backward translations, the respective Ukrainian item was approved for the final Ukrainian version.

Patients and assessments

The patients were otherwise healthy Ukrainian children of both sexes aged 3–12 years with symptoms and signs corresponding to ARTI. This infection was defined as the manifestation of rhinitis, cough, sore throat, and fever or a combination of these.

Children were excluded from the study in the case of chronic adenotonsillar pathology, recurrent otitis, sinusitis, respiratory allergy, gastroesophageal reflux disease, moderate-to-severe or severe disease of any organ systems, or families with cognitive or linguistic impairment that compromised their ability to complete the questionnaires.

The CARIFS questionnaire was administered four times. On the first day, the questionnaire was completed three times: 1) by a caregiver; 2) by a caregiver in 3–6 hours after the initial completion; 3) by a physician just after the second attempt was made by a caregiver. To evaluate the sensitivity and effect size (ES), the questionnaire was additionally completed by a caregiver on the day 5.

Statistical analysis

The descriptive statistics included median, interquartile range (IQR), maximum and minimum values, number and percent, skewness and kurtosis of the frequency distribution, and standard error. The nonparametric Mann–Whitney U test was used to assess the difference between the CARIFS scores recorded on the first and the second attempts, and the difference between the baseline and the fifth day scores.

The content validity assessed whether the items measured the full range of ARTI. We used the “floor and ceiling effect” to assess this. The distribution of the results were evaluated with the “floor” (the best) score to be between 0 and 3 and the “ceiling” (the worst) score to be between 51 and 54. The floor and the ceiling scores were calculated as the 5% tails of the full range of the scale (54×0.05=2.6 rounded 3).

Internal consistency was evaluated using correlation analysis between the items of the CARIFS and assessed using Cronbach’s α.7 This described the extent to which all the items in the CARIFS measured the same concept of ARTI in children. Acceptable values of the Cronbach’s α coefficient ranged from 0.70 to 0.95, with lower figures signifying insufficient number of questions or poor relationship between items, and higher figures suggesting excessive number of questions or their similarity.8 Cronbach’s α was recalculated, when an individual CARIFS item was deleted to evaluate the contribution of each to the internal consistency. Item–total correlation coefficient was calculated to assess the independence of the items in the CARIFS.

Means and standard deviations (SDs) were calculated from a square root of the CARIFS score as data were skewed. The sensitivity of the CARIFS was analyzed using the standardized response mean (SRM), calculated as the difference between the baseline mean score and the mean score on the day 5, divided by the SD of the difference. The ES was calculated as the difference between the mean score on the day 5 and the baseline mean score divided by the baseline SD.9

The test–retest reliability was measured as the agreement between two measurements taken 3–6 hours apart and expressed as the intra-class correlation coefficient (ICC). The ICC was measured for the total score and for three domains. Pearson’s correlation coefficient was used to correlate the responder’s and physician’s measurements of the CARIFS score and to assess the construct validity.

The analysis was performed with the use of STATISTICA 8 (StatSoft, Inc., Tulsa, OK, USA). The level of statistical significance was set at 0.05.

Ethics

The study received approval of the institutional review board at Lviv National Medical University. Parents signed the informed consent form before their children entered the study.

Results

Most items of the CARIFS questionnaire met their core meanings as it was seen from the backward translation process (Table 1).

Table 1

Backward translation of the CARIFS questionnaire into Ukrainian language

No	Source item	Backward translation (number of translators)
1	Poor appetite	Poor appetite (3); bad appetite (2); lack of appetite (1)
2	Not sleeping well	Not sleeping well (2); disturbed sleeping (2); bad sleeping (1); poor sleeping (1)
3	Irritable, cranky, fussy	Irritable (3); cranky (1); grumpy (1); irritable, cranky, fussy (1)
4	Feels unwell	Feels unwell (4); bad mood (1); feels uncomfortable (1)
5	Low energy, tired	Fatigue (3); tiredness (1); tiredness, malaise (1); tiredness, fatigue (1)
6	Not playing well	Not playing well (2); not playing lots (1); plays less than usual (1); lack of desire to play (1); low desire to play (1)
7	Crying more than usual	Tearfulness (3); repetitive crying (1); excessive crying (1); whining (1)
8	Needing extra care	Needing more care (2); needs extra care (1); requires more attention (1); needs for extra care (1); takes more care (1)
9	Clinginess	Clinginess (4); clingy (1); holds close near himself (1)
10	Headache	Headache (6)
11	Sore throat	Sore throat (6)
12	Muscle aches or pains	Muscle pain(s) (5); aching muscles (1)
13	Fever	Fever (6)
14	Cough	Cough (6)
15	Nasal congestion/runny nose	Nasal congestion/runny nose (4); stuffiness in nose/runny nose (1); stuffy/congested nose/runny nose (1)
16	Vomiting	Vomiting (6)
17	Not interested in what is going on	Not interested in surrounding (4); not interested in what is going on (1); lack of interest (1)
18	Unable to get out of the bed	Difficulty to get up from bed (5); wants to be in bed (1)

Abbreviation: CARIFS, Canadian acute respiratory illness and flu scale.

Excellent agreement was achieved between translators for the backward translation of the illness-specific symptoms.

One hundred forty-nine Ukrainian questionnaires were dispensed to children. The typical patient participated in the study was a 5–6-year-old child (Table 2).

Table 2

Baseline characteristics of children with ARTI

Characteristic	Patients (n=149)
Age (years), median (IQR)	5.5 (4.0–8.0)
Sex (male), n (%)	72 (48)
BMI (kg/m2), median (IQR)	15.7 (14.9–16.3)
Duration of ARTI (days), median (IQR)	2.0 (1.0–3.0)
Temperature (°C), median (IQR)	37.6 (37.1–38.1)

Note: Duration of ARTI before enrollment.

Abbreviations: ARTI, acute respiratory tract infection; IQR, interquartile range; BMI, body mass index.

Children had an equal sex presentation and normal BMI. The questionnaires were mainly completed on the second day of ARTI, when body temperature, as measured in the armpit, was mildly hyperthermic.

All responders had completed the CARIFS questionnaire initially. On the first time, the median duration for completion of the questionnaire was 4.5 (IQR: 3.5–5.5) minutes. Seventy-three percent of the responders who filled in the questionnaire were mothers (n=109), 15% fathers (n=22), 10% grandfather or grandmother (n=15), and 2% other guardians (n=3). Only 66% (n=99) of families completed the questionnaire again in 3–6 hours. Fifty-three percent of CARIFS questionnaires (n=79) were completed by a physician.

The baseline CARIFS median score was 18.0 (IQR: 13.0–24.0). The skewness and kurtosis in the score distribution analysis were 0.641 and 0.225, respectively (Figure 1).

Figure 1

Distribution histogram of the CARIFS score in children.

Abbreviation: CARIFS, Canadian acute respiratory illness and flu scale.

Taking a square root from the CARIFS score resulted in reduction of the skewness to 0.089, making the distribution of the score more normal. Minimum baseline CARIFS score was 5 and the maximum was 44, and none of the responders reported the ceiling or the floor scores. Most of the scores occupied 24%–44% of the full scale.

The disease-specific symptoms were the most reportable items in the CARIFS (Table 3).

Table 3

Baseline scores of the constituent items and Cronbach’s α coefficient for internal consistency of the Ukrainian version of the CARIFS

No	CARIFS item	Reported, % (n)	Median (IQR)	Cronbach’s α coefficient	Item–total correlation (95% CI)
1	Poor appetite	70.5 (105)	2 (0–2)	0.783	0.550 (0.369–0.612)
2	Not sleeping well	67.1 (100)	1 (0–2)	0.795	0.389 (0.223–0.501)
3	Irritable, cranky, fussy	64.4 (96)	1 (0–2)	0.783	0.580 (0.395–0.630)
4	Feels unwell	73.2 (109)	1 (0–2)	0.790	0.458 (0.287–0.551)
5	Low energy, tired	76.5 (114)	1 (1–2)	0.785	0.545 (0.365–0.609)
6	Not playing well	63.0 (95)	1 (0–2)	0.776	0.665 (0.465–0.680)
7	Crying more than usual	61.7 (92)	1 (0–2)	0.790	0.463 (0.292–0.555)
8	Needing extra care	62.4 (93)	1 (0–2)	0.797	0.359 (0.194–0.479)
9	Clinginess	66.4 (99)	1 (0–2)	0.791	0.445 (0.275–0.542)
10	Headache	29.5 (44)	0 (0–1)	0.809	0.195 (0.167–0.252)
11	Sore throat	73.2 (109)	1 (0–2)	0.799	0.328 (0.164–0.454)
12	Muscle aches or pains	20.8 (31)	0 (0–1)	0.802	0.241 (0.079–0.382)
13	Fever	84.6 (126)	2 (1–3)	0.798	0.346 (0.182–0.469)
14	Cough	87.2 (130)	2 (2–3)	0.815	0.185 (0.177–0.242)
15	Nasal congestion/runny nose	80.5 (120)	2 (1–2)	0.807	0.200 (0.038–0.347)
16	Vomiting	24.2 (36)	0 (0–1)	0.801	0.277 (0.114–0.413)
17	Not interested in what is going on	19.5 (29)	0 (0–1)	0.794	0.422 (0.254–0.526)
18	Unable to get out of bed	32.9 (49)	0 (0–1)	0.787	0.512 (0.336–0.588)

Notes: Cronbach’s α, if the specific item is deleted; bold type numbers denote significant coefficient; CARIFS items present as they appear in the English version, but data are given for the Ukrainian version.

Abbreviations: CARIFS, Canadian acute respiratory illness and flu scale; IQR, interquartile range; CI, confidence interval.

Fewer responders rated constitutional items, with “muscle pains” and “not interested in what’s going on” being the least mentioned. The median rate of individual scores ranged from “no” to “minor problem”, while the impact of fever, cough, and nasal symptoms was frequently recorded as a moderate problem.

The Cronbach’s α coefficient for the Ukrainian CARIFS was 0.805, and inter-item correlation coefficient was 0.191. The Cronbach’s α varied if the specific item was deleted. The minimal loss of the internal consistency of the CARIFS questionnaire took place, if the “cough” item was deleted, and the maximal loss occurred, when “not playing well” was deleted. Item to total correlation coefficients varied from 0.185 for the “cough” to 0.665 “for not playing well”.

One hundred percent of responders (n=149) reported a decrease in the CARIFS score on the day 5 with a median change of 12.00 (IQR: 8.0–18.0, P<0.0001). The SRM and ES for the squared root CARIFS score and its domains were high (Table 4).

Table 4

Sensitivity of the Ukrainian version of the CARIFS questionnaire as compared between the days 1 and 5 in children with ARTI

Domains	SRM	ES
Activity	1.58	1.96
Symptoms	1.49	2.58
Parental impact	1.03	0.96
Total	1.73	2.50

Abbreviations: CARIFS, Canadian acute respiratory illness and flu scale; ARTI, acute respiratory tract infection; SRM, standardized response mean; ES, effect size.

In the test–retest reliability analysis of 99 questionnaires, the median CARIFS score for the first and the second measurement was 19.0 (IQR: 14.5–25.0) and 19.0 (IQR: 15.0–25.0), respectively, with a median change of 0.0 (IQR: −1.0 to 0.0, P=0.996). The ICC for the total CARIFS was 0.992, for the domain activity was 0.988, for symptoms was 0.984, and for parental impact was 0.981. All values were highly statistically significant (P<0.001). The Pearson’s correlation coefficient between the CARIFS scores completed by a responder and by a physician was 0.832 (P=0.004).

Discussion

This was the first attempt to evaluate the CARIFS questionnaire in a foreign language. The Ukrainian version was tested in a standard approach to translation, using a statistical analysis that explored consistency, sensitivity, reliability, and validity.

During forward translation, particular difficulties were encountered with “clinginess”, which had no single-word synonym in Ukrainian, and, therefore, it was translated as a phrase “holds close nearby him or her”. Another concern rose up with the item “unable to get out of bed”. In the Ukrainian translation, it meant physical disability rather than a voluntary inability caused by fatigue. Therefore, we replaced “unable to get out” with Ukrainian “difficulty to get out”, as a child was able to get out. Further, “low energy, tired” was changed to Ukrainian “easy fatigability”, and “irritable, cranky, fussy” met the Ukrainian word that embraced all these features. Backward translation demonstrated good correspondence of the sense of the Ukrainian version. Convergence was high both for the activity and the parental impact domains. There was a complete correspondence in interpretation of the disease-specific items.

Most responders experienced no difficulties while filling in the questionnaire. Time taken for completion of the questionnaire was relatively short, and the absence of missing values suggested good comprehensibility of the CARIFS items. Some misunderstandings appeared with interpretation for items 3, 6, 7, and 9 in application to older children and for items 4, 10–12 to younger children. Indeed, assessment of feelings or pains requires interviewing a good responder, which does not include a small child. Conversely, “crying more than usual” and “clinginess” may not be appropriate for the elementary school children, whose advancing socialization replaces more primitive reactions.

The baseline median score of the Ukrainian version of the CARIFS was 18, while in the original English version, it was 28 or around 25 (approximation from the chart).3,4 In another study, the CARIFS median score was reported between 15 and 20 (approximation from the chart) depending on the age of children.10 Such a discrepancy could not be explained by differences in duration of illness before enrollment, a major determinant of the disease severity, as it was similar in the referenced studies. Variables that affected the baseline score might be symptomatic medication and season-dependent etiologies of ARTI. Indeed, the common cold caused by rhinoviruses and manifested with nasal discharge and obstruction with mild or absent systemic symptoms peaks in early autumn.11 In temperate climates, more aggressive influenza infection causing prominent local and systemic symptoms occurs in winter.12 So that, the study of ARTIs attempted in these different seasons may significantly differ in terms of the baseline CARIFS score. Finally, over- or underestimation of the symptoms might be due to differences in personality of the caregivers, while cultural differences cannot also be excluded.

The analysis of the frequency histogram of the CARIFS score exhibited moderate deviation from the normal distribution both for symmetry and amplitude. Skewness of the score was 0.641 vs 0 (shifted to the right) and kurtosis was 0.225 vs 3 (flattened peak) in the normal distribution. Skewness of the CARIFS data was also reported in a study by Butler et al.13 In a case of moderate right skewness, taking a log or square root of a data set is often enough to make data suitable for the analysis based on the assumption of the normality. In our study, taking a square root from CARIFS score considerably reduced skewness from 0.641 to 0.089. In comparison analysis, an alternative approach may be the use of non-parametric methods based on ranks without the assumption of the normality. Among such methods, the Mann–Whitney U-test, Wilcoxon signed, or rank sum tests can be suitable.

In our study, Cronbach’s α of the Ukrainian CARIFS items was relatively higher than the one originally reported for the English version.3 This finding may be at least partially explained by the two facts that: 1) different language versions will never be ideally the same; 2) alpha is a property of the scores for a test from a specific sample of responders, meaning that the variation in alpha may be expected even within the same language versions of the test. Therefore, it was proposed that investigators should not rely on published alpha estimates and should measure alpha each time the test was administered.14

Every CARIFS item contributed almost equally to the Cronbach’s α coefficient, suggesting that the questionnaire is constructed of equally important terms. However, individual item to total correlation coefficients varied considerably. The best correlation was documented for “not playing well” with some weaker relationships for other items that described activities of a child. Surprisingly, the disease-specific items, such as cough, nasal congestion, and sore throat, correlated less with the total CARIFS score. This could be partly explained by the fact that these local symptoms constituted a smaller part of the questionnaire, which was mostly “occupied” by the “systemic” attributes of the disease. Furthermore, the CAFIRS scores were measured on the second day of the disease, when systemic presentation of ARTI is profound, especially in younger children, who dominated the study.

The sensitivity of the Ukrainian version of the CARIFS was excellent with total and domain-specific SRM and ES varying within magnitudes of the large effect.8 The SRM was not calculated in the source study but ES for the total CARIFS was 0.99 in contrast to 2.50 in the present research, suggesting the higher difference in the CARIFS scores between the baseline and follow-up observation measurements.3 The latter could be explained by the fact that reassessment of the CARIFS score was virtually made on the third day of the disease in the Canadian study, while our responders completed the questionnaire on the fifth day, when the disease could be less intense. Additionally, as it is shown, the CARIFS scores demonstrated a fair extent of skewness, which might corrupt calculations based on untransformed means and SDs.

The test–retest reliability was high with ICC for the total CAFIFS score, activity, symptom, and parental impact domains being greater than 0.98. Such a good correspondence of the paired data was due to the relatively short time between the first and second measurements ranging from 3 to 6 hours. The description of the CARIFS items in such a setting was rather due to the effective recall of the previously recorded scores than to a true independent re-assessment of the score. Increasing the time between the tests, instead, might mirror the course of the disease or the efficacy of the symptomatic treatment making test–retest issues problematic in the fast changing acute illness.

We were not able to evaluate the construct validity of the CARIFS questionnaire as there was no “gold standard” instrument with which the data could be compared. In the original work, the surrogate gold standard measures were parent global assessments with a visual analogue scale and body temperature.3 We did not adopt these measures as the former would have a generalized level of subjectivity for attitudes, beliefs, or psychological characteristics, and the latter would measure partial and not necessarily inherent aspect of ARTI. In this study, the “gold standard” measure was a questionnaire completed by the physician. Almost immediate rescoring demonstrated the high and significant Pearson’s correlation coefficient between responder’s and physician’s scores.

Thus, the Ukrainian version of the CARIFS-based English questionnaire proved to be a consistent, sensitive, reliable, and valid instrument in assessment of ARTI in the preschool and elementary school children in the Ukrainian population.