Validity and Reliability of an Expanded Vegetable Questionnaire Among Elementary School Children.

BACKGROUND: The purpose of this study was to expand the School Physical Activity and Nutrition questionnaire to include a greater variety of vegetables and to evaluate the relative validity and reliability of these revised items.
OBJECTIVES: This study utilized 2 convenience samples of third to fifth graders for an analysis: validity (n = 70) and reliability (n = 76). Validity was assessed by comparing questionnaire items with vegetable intake reported from a 24-hour dietary recall covering the same reference period. Reliability estimates were assessed via same-day test-retest.
RESULTS: Agreement correlations ranged from 0.35 to 0.71. Kappa statistics varied from 0.16 to 0.66. Percentage agreements ranged from 57% to 87%. Test-retest Spearman coefficients were greater than 0.50 for 6 items, weighted Kappa values were greater than 0.40 for all 7 items, and percentage agreement exceeded 75% for 5 items.
CONCLUSIONS: Results suggest that the questionnaire is a valid and reliable measure of the previous day's vegetable intake in third- to fifth-grade students. This trial was registered at ClinicalTrials.gov as NCT02668744.

Introduction

The field of nutrition continues to struggle with a lack of dietary measures for children that are short, comprehensive, easy to administer, and have robust validity and reliability associated with their use (1). Food frequency questionnaires (FFQs) are regularly used to estimate children's dietary intakes in school-based settings, as they serve as effective alternatives to other methods of dietary assessment, which can be time-consuming and expensive, require extensive staffing requirements, or be burdensome to participants (2).

Numerous FFQs exist that assess the entire diet or intake of particular food groups; however, the field is limited in instruments that measure a variety of vegetables and are designed specifically for elementary-aged, multiethnic populations. There are some valid and reliable measures frequently used in school-based studies, such as the Block Food Screener (3) or the School Physical Activity and Nutrition (SPAN) survey (4, 5). However, these instruments, like most FFQs, are limited in the breadth of items assessing intake of vegetables. Most FFQs have been designed to include only 1 or 2 questions assessing vegetable intake, and this often reflects the intake of vegetables collectively, rather than different varieties or types of vegetables. A lack of scope in variety measurements of vegetable intake can limit evaluations of public health programs and interventions focusing on vegetable consumption. The development of dietary assessment tools that can accurately and reliably measure a variety of vegetable intake in youth is warranted. Therefore, the goal of this study was to develop and test the validity and reliability of a more comprehensive screener for assessing the intake of vegetables in elementary-aged children.

Methods

TX sprouts

Cross-sectional baseline data from TX Sprouts, a cluster-randomized, school-based gardening, cooking, and nutrition intervention, were used. TX Sprouts targeted third- to fifth-grade students and their parents from 16 elementary schools in the Austin area. Schools included in the trial had to meet the following inclusion criteria: 1) have a high proportion of Hispanic children (>50%); 2) have a high proportion of children participating in the free and reduced lunch program (>50%); 3) be located within 60 miles of (The University of Texas at Austin) campus; and 4) have no previous or existing gardening program. Based on these criteria, 73 schools were invited to participate, and 20 schools from 5 different independent school districts agreed to participate. The first 16 out of the 20 schools to provide letters of support were randomly assigned to either the intervention (n = 8 schools) or control group (delayed intervention; n = 8 schools). The remaining four schools were placed on a contingency list, in case any of the 16 randomly assigned schools dropped out of the study. Full methods of the ongoing TX Sprouts intervention will be published elsewhere. For this analysis, only data from the first 2 waves of data collection (12 schools) were used. All procedures involving human participants were approved by the Institutional Review Boards of The University of Texas at Austin. Written informed consent was obtained from all participant's parents and assent was obtained from participants prior to testing.

Questionnaire development

The SPAN questionnaire was created and tested by The University of Texas Health Science Center at Houston’s School of Public Health as a surveillance system to monitor the prevalence of overweight/obesity in school-aged children in Texas. The SPAN questionnaire measures physical activity, nutrition attitudes and knowledge, and dietary behaviors. The questionnaire has 2 versions: 1 each for elementary and middle/high school students.

The relative validity of the elementary version of the SPAN 2000 food behavior questions were evaluated by Thiagarajah et al. (5). Agreement between responses on the SPAN and a 24-hour dietary recall (24hDR) resulted in Spearman correlations of 0.34 and 0.66 for vegetable and bean items, respectively. Kappa values for vegetable and bean items were 0.17 and 0.60, respectively. Total percentage agreement for vegetable items was low (27%), while the bean item was higher (88% agreement between recall and questionnaire responses). The validation study was limited, in that it was tested in a primarily non-Hispanic, White sample (n = 121) of fourth-grade students in Indiana.

The reliability of the elementary SPAN 2000 food intake questions was assessed in a separate sample (n = 322) of multiethnic, fourth-grade students by Penkilo et al. (4). In particular, bean and vegetable items showed acceptable test-retest correlations (0.65 and 0.79, respectively). Acceptable kappa statistic values for beans and vegetables (0.66 and 0.65, respectively) were also reported. Percent agreements between test-retest administrations for beans and vegetables (91% and 74.3%, respectively) showed the items were reproducible.

Since its first administration in 2000, the SPAN questionnaire has undergone 3 revisions (2004, 2009, and 2015) to include additional dietary items and health behavior questions. However, subsequent versions have remained limited in vegetable intake questions. The most recent iteration of the SPAN questionnaire (2015) contains 5 questions about vegetable intake (starchy vegetables, orange vegetables, green vegetables, other vegetables, and beans).

Expansion of the questionnaire

TX Sprouts staff expanded the 2015 version of the SPAN questionnaire. New vegetables added to the screener include radishes, garbanzo beans, black beans, collard greens, Swiss chard, and the specification of lettuce as “romaine lettuce.” Vegetables were added based on prior observations that these vegetables are consumed in moderate to large quantities by children. Vegetables were also added on the basis that they are common vegetables used in school-based interventions. These vegetables may be consumed in low quantities initially; however, interventions expose children to these vegetables. Vegetables removed in the design of the revised screener included mushrooms, eggplant, celery, zucchini, asparagus, artichokes, baked beans, refried beans, and pork and beans. These vegetables were removed because previous efforts in refinement have found these vegetables are not consumed on a usual basis by children. Attempts were made to align vegetables within an item, based on matching the food groupings used with those of the University of Minnesota's Nutrition Coordinating Center or vegetables with similar micronutrient compositions (6). Expanded questionnaire items were evaluated for content validity by a panel of 8 experts in the fields of nutrition and public health who had previous experience with collecting data on vegetable intake in elementary-aged populations. Revisions were made based on comments from the review, until consensus was achieved among the group.

The original SPAN and the expanded vegetable questionnaires both utilize the same, non-quantified food frequency approach to assess the previous day's food intake. The frequency of consumption of food items is recorded without portion size. Others have acknowledged the superiority of non-quantitative versus quantitative questionnaires in school-aged children, as children are more accurately able to rank intake of a food item compared with providing serving sizes (1, 7). Students are provided with 4 potential responses to select from (“no, I didn't eat any of these vegetables yesterday”; “yes, I ate at least 1 of these vegetables 1 time yesterday”; “yes, I ate at least 1 of these vegetables 2 times yesterday”; or “yes, I ate at least 1 of these vegetables 3 or more times yesterday”).

Questionnaire administration

Students completed questionnaires during the school day at their respective schools as part of a larger data collection effort for TX Sprouts. Questionnaires were administered throughout the school day by research staff, following a standardized protocol. The expanded vegetable questionnaire was included as part of the baseline TX Sprouts Student Questionnaire, which asked students questions about demographics, eating behaviors, cooking and gardening behaviors, and nutrition knowledge. For same-day test-retest reliability, students were given the questionnaire in the morning before lunch and were given the questionnaire again in the afternoon following lunch (on average, about 2 hours later).

24-hour dietary recalls

Within each wave of TX Sprouts, two 24hDRs were collected on a random subsample. A total of 547 students had two 24hDRs within the first 2 waves of TX Sprouts. Recalls were collected via telephone by trained staff, who were supervised by a Registered Dietitian Nutritionist and used Nutrition Data System for Research Version 2014, which is a computer-based software application that facilitates the collection of recalls in a standardized fashion (8). Parents and/or guardians of students assisted with recalls. Recalls were collected all 7 days of the week. Students received a $10 incentive for completing the recall. A panel of 3 trained nutrition professionals confirmed that the recalled vegetable groupings matched those of the expanded questionnaire.

Sample size

A student's 24hDR food group servings data and questionnaire responses were used for analysis only when both instruments reflected the same 24-hour period of intake. For the validity assessment, the sample consisted of 70 students. A separate convenience study sample of 76 students completed the same-day test-retest reliability assessment. Demographic information for both validity and reliability samples are provided in .

TABLE 1

Demographics of Students Participating in the Test-Retest Reliability and Relative Validity Analyses of the 7-Item Expanded Vegetable Questionnaire

	Reliability	Relative Validity
Total sample	76	70
Sex
Male	38 (50%)	37 (52.9%)
Female	38 (50%)	33 (47.1%)
Ethnicity
Hispanic	31 (40.8%)	35 (50.0%)
Non-Hispanic White	19 (25.0%)	16 (22.9%)
Non-Hispanic Black	14 (18.4%)	11 (15.7%)
Othera	12 (15.8%)	8 (11.5%)
Age, mean ± SD	10.10 ± 0.31	9.20 ± 0.94

Data are shown as n (% of total sample), unless otherwise indicated.

Other includes Native American, Asian/Pacific Islander, non-listed other ethnicity, or undisclosed ethnicity.

Statistical analyses

Descriptive statistics (means, SDs, and percentages) were calculated to describe sample demographic characteristics. To examine criterion validity, frequencies of “times per day” from the expanded questionnaire were compared to servings from the 24hDR. For each item, 3 different analyses were used to assess validity: Spearman rank-ordered correlations, the Kappa statistic, and percentage agreement. Spearman correlations were computed because data were not normally distributed. To assess the same-day test-retest reliability of each item at time 1 with that of time 2, 3 different analyses were used: Spearman rank order correlations, the weighted Kappa statistic, and percentage agreement. Weighted Kappas were used, as opposed to unweighted, as they provide a single value to represent agreement and adjust for chance agreement and the degree of disagreement (9). Multiple measures were used in assessing validity and reliability, in order to reduce bias in interpreting results (4, 10). All data were analyzed using SPSS Statistics, Version 25.0.

Results

Validity

Spearman correlations, Kappa statistic values, and percentage agreements between vegetable items on the 24hDR and questionnaire responses are reported in . Correlation coefficients ranged from 0.35 for squash, green beans, or cucumbers to 0.71 for beans. The Kappa statistics ranged from 0.16 for squash, green beans, or cucumbers to 0.66 for beans. Only the item pertaining to the consumption of beans showed agreement above 0.50. Items showed a range of percentage agreements, from 57% for tomatoes or peppers to 87% for beans.

TABLE 2

Relative Validity and Test-Retest Reliability of the Expanded Vegetable Questionnaire

	Relative Validity Compared to 24-Hour Dietary Recalls					Test-Retest Reliability
Vegetable Items	Spearman Correlation, rho	Spearman Correlation, 95% CI	Kappa, κ	Kappa, 95% CI	Percent Agreement	Spearman Correlation, rho	Weighted Kappa, κ	Percent Agreement
Carrots, beets, sweet potatoes, or radishes	0.49	0.28–0.65	0.34	0.12–0.57	70.0%	0.47	0.49	75.0%
Squash, green beans, or cucumbers	0.35	0.13–0.05	0.16	0.02–0.30	64.3%	0.61	0.60	76.3%
Tomatoes or peppers	0.43	0.22–0.61	0.21	0.02–0.40	57.1%	0.55	0.54	78.9%
Broccoli, cauliflower, or cabbage	0.48	0.28–0.64	0.46	0.23–0.70	80.0%	0.68	0.60	80.3%
Leafy green vegetables (like spinach, collard greens, Swiss chard, or romaine lettuce)	0.59	0.41–0.72	0.46	0.23–0.68	78.6%	0.50	0.45	73.7%
Starchy vegetables: potatoes, corn, or peas (does not count French fries or chips)	0.53	0.33–0.68	0.39	0.18–0.61	71.4%	0.57	0.53	71.1%
Beans (like pinto, garbanzo, kidney, or black beans)	0.71	0.58–0.81	0.66	0.45–0.86	87.1%	0.61	0.55	84.2%

Reliability

Spearman correlations, weighted Kappa statistics, and percentage agreements based on test-retests of questionnaire administration are reported in Table 2. Test-retests of all items resulted in correlations above 0.50, with the exception of item pertaining to carrots, sweet potatoes, or radishes, which produced a 0.47 correlation for agreement. Values for weighted κ ranged from 0.45 for leafy green vegetables to 0.60 for squash, green beans, or cucumber and for broccoli, cauliflower, or cabbage items. Percentage agreements ranged from 71% for starchy vegetables to 84% for beans.

Discussion

The purpose of this study was to expand the SPAN screener to include a larger variety of vegetables and test the relative validity and reliability of this screener in elementary school children. Overall, the expanded 7-item questionnaire, compared to a 24hDR, resulted in higher Spearman correlations, improved Kappa statistics, and better percentage agreements than previous psychometric evaluations of the SPAN screener in primarily non-Hispanic, White children (5). Mean correlations, weighted Kappa statistics, and percentage agreements for same day test-retest reliability were slightly lower than previous assessments of reproducibility for the SPAN questionnaire in multiethnic elementary children (4).

Correlations for items assessing vegetable intake are generally lower than other food items on a FFQ (10). Previous research has found that vegetables may be forgotten or not recognized if they're consumed as part of mixed dishes, resulting in lower agreement between a FFQ and the reference assessment method (5, 11, 12). This study's findings suggest that breaking apart vegetables into multiple items based on variety may require children to think in greater detail about mixed dishes. FFQs may ask children about vegetable intake in “servings” (13), as opposed to “times,” as used in this study. Vegetables consumed as part of mixed dishes may be eaten in smaller portions than what is perceived as a serving (11). Asking for intake in terms of times may be easier for child respondents to report those vegetables consumed as part of mixed dishes.

Test-retest correlations, weighted Kappa statistics, and percentage agreements were within acceptable limits for reproducibility; however, they were lower than previous assessments of reproducibility for the SPAN questionnaire (4). Weighted Kappa statistic values for all vegetable items were below 0.60; however, this statistic is only 1 metric of reliability. Weighted Kappa values in this study were lower than Spearman correlation coefficients, similar to what has been previously reported (9). The percentage agreement values may be more intuitively meaningful. The acceptability of reproducibility criteria depends on the final purpose of the measure (14). Higher reliabilities are recommended for measuring individual differences; however, the expanded questionnaire is intended to measure group differences.

There are several strengths of this study. Sample sizes for psychometric assessment in this study were comparable to other validation and reproducibility studies in children and adolescents (10, 15). However, the current study was conducted in multiethnic, third- to fifth-grade students. Use of a 24hDR, considered to be a gold-standard measure of self-reported dietary intake, to assess the relative validity of the expanded questionnaire is seen as a strength of this study (2).

There are several limitations of this study that need to be mentioned. Both the expanded questionnaire and 24hDR are subject to measurement error, bias, and social desirability (2, 16–18). Dietary assessments in children pose unique challenges, including potentially limitations in children's concepts of time, food recognition and knowledge of preparation methods, ability to quantify estimated portion sizes, motivation, literacy, memory capabilities, and concentration spans (1, 19, 20). Objective measures of dietary assessments, such as plate-waste studies (21, 22), remote food photography (23), or the use of biospecimens (24, 25) (i.e., serum plasma carotenoids), may have lower potential for reporting biases, compared to subjective measures such as FFQs and 24hDR. However, objective methodologies have their limitations, often being burdensome, time-consuming, or unfeasible in large-scale or school-based data collection.

Another potential limitation of the expanded questionnaire, as with the original SPAN questionnaire items, is that the reference time interval of the prior day may not represent an individual's habitual food intake. Others have utilized this reference time interval in validation and reproducibility studies, as research has found that younger children are able to more accurately recall intakes from shorter time periods, compared with extended time periods. The expanded questionnaire does provide an estimate of a population's dietary patterns (4, 5, 15). Inferences made from the expanded vegetable questionnaire should be made at the group intake level, as opposed to the individual level.

The researchers attempted to group vegetables that are nutritionally similar. However, a limitation of the groupings is that it is possible that 1 vegetable within an item was predominantly responsible for the observed correlation. Groupings of vegetables were used rather than asking about individual items to reduce the total number of items, therefore decreasing the respondent burden. Additionally, as some vegetables tend to be episodically consumed, the high agreement garnered from the vegetable items may be a result of a high prevalence of non-consumers.

There is a need for short, non-quantitative questionnaires that comprehensively and accurately measure the dietary intake of vegetables in elementary-aged children. Criterion validity results from this study indicate that this questionnaire is a valid tool for assessing vegetable intake among third- to fifth-grade children. Test-retest reliability results from this study were slightly lower than past psychometric evaluations, although they were still within acceptable limits for a FFQ. The accuracy of vegetable intake information generated by the expanded vegetable screener is dependent on the validity and reproducibility of this screener in the population it is intended to be used for. Researchers electing to use the expanded vegetable questionnaire should assess reproducibility within their population. The expanded vegetable items reflect a greater variety of vegetables than previous versions, allowing for greater capacity of use in conducting needs assessments, evaluating school-based interventions, or assessing population-based surveillance. Further research can include continued refinement and testing in other populations. Technological innovations have led to changes in how dietary assessments are collected (26, 27). Future work can focus on the continued improvement of self-reported methods and the incorporation of biomarkers or technology-based methods to capture intakes (19).