Non-response and external validity in a school-based quasi-experimental study 'The Healthy Primary School of the Future': A cross-sectional assessment.

Limited evidence is available about (non)-representativeness of participants in health-promoting interventions. The Dutch Healthy Primary School of the Future (HPSF)-study is a school-based study aiming to improve health through altering physical activity and dietary behaviour, that started in 2015 (registered in ClinicalTrials.gov on 14-06-2016, NCT02800616). The study has a response rate of 60%. A comprehensive non-responder analysis was carried out, and responders were compared with schoolchildren from the region and the Netherlands using a cross-sectional design. External sources were consulted to collect non-responder, regional, and national data regarding relevant characteristics including sex, demographics, health, and lifestyle. The Chi-square test, Mann-Whitney U test, or Student's t-test were used to analyse differences. The analyses showed that responders (n = 494) were comparable with non-responders (n = 348) and regional data (n = 6172) with regard to sex and health. Responders did not significantly differ from regional data with regard to lifestyle. Responders had significantly higher educated parents compared to non-responders and were more often of autochthonous ethnicity compared to regional data. Major differences were observed between responders and schoolchildren in the Netherlands, regarding, among others sex, ethnicity, and parental employment rates. We conclude that a potential healthy-volunteer effect in the HPSF-sample is limited. External validity is high when compared to the regional population but low when compared to the national sample. For future intervention studies, we advise to evaluate outcome measures according to regional/national standards and to cooperate with external parties in early stages of research to be able to assess and enhance generalisability.

Introduction

Selective (non)-response is a common issue that can jeopardize the external validity of studies, especially in those that involve lifestyle interventions (Groeneveld et al., 2009). Over the past years, participation rates in scientific studies have decreased, and selective participation was more prominent. This increases the risk of differences between responders and non-responders (Nohr and Liew, 2018; Drivsholm et al., 2006). The decision to participate in a study is not a random process, but depends on a number of factors (Wennlöf et al., 2003). Several studies have indicated that compared to non-responders, responders often have a higher socio-economic status (SES) (Van Loon et al., 2003; Martikainen et al., 2007; Plachta-Danielzik et al., 2008), report better subjective health and healthier lifestyles (Van Loon et al., 2003), are less likely to be overweight or obese (Plachta-Danielzik et al., 2008), are more likely to be unemployed (Korkeila et al., 2001), and may be more health conscious (Brussaard et al., 1997). Furthermore, women are often more likely to respond (Van Loon et al., 2003; Martikainen et al., 2007; Korkeila et al., 2001). Non-responders generally have an increased mortality and morbidity (Lindsted et al., 1996), which might be explained by the healthy volunteer bias. This bias, which is related to non-response bias, indicates that a person with existing health problems or a low SES is less likely to voluntarily participate in a study (Lindsted et al., 1996). Considering school-based studies, it has been demonstrated that the requirement of active parental consent may cause lower participation rates compared to studies that do not require active parental consent or in which passive parental consent is possible (Esbensen et al., 2008).

Despite a large number of intervention studies on health behaviour, only a handful of studies has assessed differences between non-responders and responders, mainly because information on non-responders is often absent (de Winter et al., 2005). This might especially be the case in studies with specific study populations, such as children. Moreover, because of the discouragement to store personal data due to the new General Data Protection Regulation (GDPR) in the European Union, it will become more challenging to perform epidemiological studies and representativeness studies in the future (Chassang, 2017). This research gap is a major concern, since the introduction of error due to systematic differences between those selected and those not selected can have a large impact on the generalisability of the results. The objective of the current study is to assess differences between non-responders and responders of the large Dutch school-based quasi-experimental Healthy Primary School of the Future (HPSF)-study (Willeboordse et al., 2016), and to assess the external validity of the study sample, by comparing responders with schoolchildren in the region of the study setting and in the Netherlands.

Methods

This study has a cross-sectional design and compares the baseline study population of the HPSF (i.e. responders) with a) non-responders within the participating schools; b) schoolchildren in the region; and c) schoolchildren in the Netherlands. Non-responders include all children from the participating schools who did not hand in a positive informed consent form for the baseline measurements. Because external sources, who have access to data on an individual level, were aware of the participation status of the school children, they were able to provide data for non-responders and responders separately. Data of external sources were aggregated to ensure that they could not be traced back to individuals.

Study setting and recruitment HPSF-participants

The HPSF-study started in 2015 and aims to normalize children's body mass index (BMI) through a school intervention which focuses on physical activity and dietary behaviour. A detailed description of the study has been published elsewhere (Willeboordse et al., 2016). Briefly, the study includes four intervention schools and four control schools (using a regular school approach) in the Parkstad region in the province of Limburg, the Netherlands. No distinction between intervention or control schools was made in the current study. The study population consists of a dynamic cohort of children aged 4–12 years (group 1–8 in the Netherlands). However, the current study only examined children that were enrolled in the study at baseline. Within the eight participating schools, children were recruited by means of information brochures, informative meetings for parents and reminders by the school staff. Additionally, the research team informed children during classroom visits. In order to participate, children needed to hand in a positive informed consent form signed by their parents or caretakers. All children in the schools were eligible to participate in the study. At baseline, 1403 out of 2326 children (60.3%) in the HPSF-study handed in a positive informed consent form [this response rate differs from the response rate mentioned by Vermeiren et al. (Vermeiren et al., 2018), because a different date of pupil-counting was used and because Vermeiren et al. also included children from a school that were not approached to participate at baseline]. Participation rates per school ranged between 46.7% and 69.5%. Unwillingness by parents to fill in questionnaires and no interest in scientific studies were the main reasons to decline participation.

Outcomes

An overview of outcomes, including definitions, and suppliers of the data for the current cross-sectional study are presented in Additional files A-B. Outcomes were selected based on relevance for the HPSF-study, and availability of reliable external datasets for non-responders, schoolchildren in the region, or schoolchildren in the Netherlands. Outcomes include: sex (boy/girl), total Strength and Difficulties Questionnaire (SDQ) score (Van Widenfelt et al., 2003) (continuous; 0–40 points), weight (continuous; kilogram), height (continuous; meter), overweight or obesity (percentage; International Obesity Task Force (IOTF)-values (Cole and Lobstein, 2012)), obesity (percentage; IOTF-values (Cole and Lobstein, 2012)), self-reported physical complaints (yes/no), self-reported mental health problems (yes/no), medicine use (yes/no), ethnicity (native/Western foreigner/non-Western foreigner), family composition (living with both parents/not living with both parents), educational level mother (low/middle/high), use of youth healthcare (yes/no), concerns regarding the upbringing of the child (yes/no), self-reported high impact events in family (yes/no), self-reported high impact events in school (yes/no), parental employment rates (paid/unpaid), hidden poverty at baseline (continuous; 0–4 points), sports club membership (yes/no), active transport to school (yes/no), sedentary behaviour (continuous; min/day), moderate-to-vigorous physical activity (continuous; min/day), breakfast consumption (continuous; days/week), and fruit consumption (continuous; days/weeks). Outcomes that were analysed on a school-level include: central exam results according to Dutch standards (continuous; 501–550), parental educational level (0/0.3/1.2) (DUO, 2015), school advice (percentage of pupils referred to preparatory secondary vocational education or lower), schoolyard size (continuous; m2 per child), and satisfaction with the school (continuous; 0–10 points). School-level outcomes were compared to identify major differences between schools that participated in the HPSF-study and schools that did not participate.

Data collection

External sources were consulted to collect information about non-responders, schoolchildren in the region, and schoolchildren in the Netherlands (Additional file A). The region was defined as ‘Parkstad Limburg’, which includes eight municipalities in the south of the Netherlands. Schools in the HPSF-study are located in four of the eight municipalities of ‘Parkstad’. This region is known to have a relatively low SES (Steenbakkers et al., 2014; Jansen, 2015). External data suppliers for non-responders are the Regional Public Health Services South-Limburg (GGD-ZL) and Educational Monitor Limburg (OML). Data suppliers for regional data are the GGD-ZL, OML, and the Active Living-study (Van Kann et al., 2015). National data were collected from the Dutch National Food Consumption Survey (VCP) (Van Rossum et al., 2011), and Statistics Netherlands (CBS). Data for the school-level analyses were collected from the Dutch Education Executive Agency (DUO), the Active Living-study (Van Kann et al., 2015), and OML. The GGD-ZL and OML also enriched the HPSF-dataset with additional data. Youth Healthcare (JGZ) routinely collects data of children growing up in the region. The JGZ is part of the GGD-ZL. In order to increase comparability between the groups, data from the same source were compared if available, i.e. responders-data from external sources were used instead of HPSF-data in some cases (Additional file B). For all other analyses, baseline data of the schools in the HPSF-study were used for responder analyses.

Data analysis

For each analysis, only children of the HPSF with a comparable age as those of the external source were included (Additional file B). Data from responders were excluded from the data at the regional level to avoid duplication. Unfortunately, it was not possible to exclude responders from the data at a national level. Therefore, two analyses with extremes were performed, i.e. one analysis assuming that no responders were included in the national data, and one analysis assuming that all responders were included in the national data. Because the responders are only a small section of the national data, we do not expect any major differences between the two analyses. Missing data were deleted according to the listwise-deletion strategy (Allison, 2001). It was not possible to perform data-imputation, as most external datasets were aggregated. Chi-square tests were used for categorical outcomes. Continuous outcomes were tested using Student's t-tests and Mann-Whitney U tests for parametric and non-parametric data respectively. Data were analysed using Statistical Package for the Social Sciences (SPSS) 21.0 software. Differences were considered statistically significant at P < 0.05 for two-sided testing.

Results

Non-response bias in relation to sex, demographics, and health at baseline had not occurred to any appreciable degree. However, responders had significantly higher educated parents compared to non-responders (Table 1). With regard to responders and schoolchildren in the region, no differences were observed for sex, demographics, health, and lifestyle (physical activity and diet) at baseline (Table 1, Table 2). However, responders were significantly more often of autochthonous ethnicity compared to regional data (Table 1). Compared to the average Dutch population, parents of responders were more often unemployed, and responders were less often male, and of a non-Western immigration background. Responders more often had a sports club membership, but they less frequently used active transport to school as compared to schoolchildren in the Netherlands (Table 2). No significant difference was observed for educational level of the mother, breakfast consumption, and fruit consumption between responders and Dutch schoolchildren (Table 1, Table 3).

Table 1

Demographics of responders and non-responders of the Healthy Primary School of the Future Study (2015) compared to regional and national data.

Subject	Responders HPSF (percentage or mean (SD))	Non-responders HPSF (percentage or mean (SD))	P-value	Parkstad region (percentage or mean (SD))	P-value	Netherlands (percentage or mean (SD))	P-value
Sex (male; %)	48.18/46.43a	51.72	0.31	50.18	0.39	51.16	0.03b
	N = 494/560a	N = 348		N = 6172		N = 1,154,908
SDQ score	6.14 (4.57)	6.02 (5.00)	0.72	6.01 (4.88)	0.57	NAc
	N = 492	N = 345		N = 6122
Weight boys (kg)	22.24 (3.68)	21.61 (3.92)	0.09	21.76 (3.98)	0.07	NAc
	N = 236	N = 180		N = 2678
Height boys (m)	1.19 (0.05)	1.18 (0.06)	0.08	1.19 (0.06)	0.23	NAc
	N = 236	N = 180		N = 2675
Weight girls (kg)	21.76 (4.10)	22.14 (4.47)	0.37	21.67 (4.10)	0.74	NAc
	N = 254	N = 168		N = 2693
Height girls (m)	1.18 (0.06)	1.18 (0.06)	1.00	1.18 (0.06)	0.86	NAc
	N = 254	N = 168		N = 2691
Overweight and obese (%)	12.83	13.06	0.92	12.19	0.68	⁎
	N = 491	N = 268		N = 5373
Obese (%)	3.05	3.36	0.82	2.75	0.70	NAc
	N = 491	N = 268		N = 5373
Self-reported physical complaints (% yes)	10.25N = 478	10.59N = 340	0.89	11.54N = 6016	0.40	NAc
Self-reported mental health problems (% yes)	2.17N = 461	3.96N = 328	0.14	3.15N = 5722	0.24	NAc
Medicine use (% yes)	8.64	8.70	1.00	8.44	0.89	NAc
	N = 486	N = 345		N = 6087
Ethnicity (% autochthonous/Western/non-Western)	81/12/7 or 84/12/3a	76/15/9	0.21	78/11/11	0.02d	76/7/17	<0.001b, d
	N = 484/516 a	N = 343		N = 6070		N = 1,154,908
Family composition (% living with both parents)	79.59	73.55	0.05	77.72	0.34	NAc
	N = 490	N = 344		N = 6112
Educational level mother (% Low/middle/high)	25/45/30 or 14/46/40a	30/50/20	0.003d	27/42/31	0.37	17/43/40	0.18b
	N = 473/506a	N = 331		N = 5930		N = 1,054,000
Use of youth healthcare (%)	4.44	4.01	0.76	4.45	1.00	NAc
	N = 495	N = 349		N = 6181
Concerns regarding the upbringing of the child (% often/always)	3.74	6.29	0.23	4.51	0.26	NAc
	N = 481	N = 334		N = 6010
Self-reported high impact family event (%)	28.60N = 486	N = 338	0.19	27.76N = 6033	0.69	NAc
Self-reported high impact school event (%)	9.68N = 475	10.88N = 331	0.58	9.11N = 5860	0.68	NAc
Employment rates (%)	58.76/87.27a	NAc		61.92	0.09	94.42	<0.001b
	N = 708/1163a			N = 15,036		N = 3,674,000
Hidden poverty (%)	19.51	NAc		18.42	0.41	NAc
	N = 226			N = 4991

All significant values are in bold.

Percentages of responders vary among analyses because a different selection of responders has been used for the comparison with non-responders and the Netherlands.

Assumption that all responders were included in the national data or assumption that none of the responders were included in the national data did not alter the results.

Data not available.

Pairwise tests results are displayed in additional file C.

No quantitative comparable data available. Statistics Netherlands (CBS) demonstrated that in 2015, 12.2% of all children aged 4–12 years were overweight or obese, among which 3.9% were obese (CBS, 2015). This number was slightly higher in the current study, which showed that 15.9% was overweight or obese, and that 4.1% suffered from obesity (HPSF group 2–7, n = 1110).

Table 2

Physical activity of responders and non-responders of the Healthy Primary School of the Future Study (2015) compared to regional and national data.

Subject	Responders HPSF (percentage or mean (SD))	Non-responders HPSF (percentage or mean (SD))	P-value	Parkstad region (percentage or mean (SD))	P-value	Netherlands (percentage or mean (SD))	P-value
Sports club membership (%)	83.94/84.10a	NAc		84.10	1.00	73.52	<0.001b
	N = 436/497a			N = 1289		N = 725
Active transport to school (%)	85.22/80.36a	NAc		85.08	0.42	88.98	<0.001d
	N = 433/504a			N = 1106		N = 726
Sedentary behaviour (min/day)	499.61 (70.63)	NAc		495.55 (62.64)	0.32	NAc
	N = 387			N = 797
Moderate to vigorous physical activity (min/day)	52.93 (17.93)	NAc		50.97 (17.88)	0.08	NAc
	N = 387			N = 797

All significant values are in bold.

Percentages of responders vary among analyses because a different selection of responders has been used for the comparison with non-responders and the Netherlands.

Assumption that all responders were included in the national data or assumption that none of the responders were included in the national data did not alter the results.

Data not available.

Assumed that none of the responders were included in the national data, as sample size is too small to perform additional analyses.

Table 3

Dietary intake of responders and non-responders of the Healthy Primary School of the Future Study (2015) compared to regional and national data.

Subject	Responders HPSF (percentage or mean (SD))	Non-responders HPSF (percentage or mean (SD))	P-value	Parkstad region (percentage or mean (SD))	P-value	Netherlands (percentage or mean (SD))	P-value
Breakfast (days/week)	6.82 (0.77)	NAb		NAb		6.74 (1.06)	0.17a
	N = 485					N = 726
Fruit consumption (days/week)	5.06 (1.81)	NAb		NAb		5.07 (1.97)	0.91a
	N = 481					N = 726

Assumed that none of the responders were included in the national data.

Data not available.

Based on a school level analysis, no differences were observed between schools included in the HPSF-study and schools in the region (Table 4) at baseline, although parents and children of the HPSF were significantly less satisfied with their school. For HPSF and schools in the Netherlands, no differences were observed for number of pupils, secondary school advice, and central exam results, but parents of children in the HPSF-study were more often lower educated.

Table 4

School level analysis of the Healthy Primary Schools of the Future (2015) compared to regional and national schools.

Subjecte	HPSF (percentage or mean (SD))	Parkstad region (percentage or mean (SD))	P-value	Netherlands (percentage or mean (SD))	P-value
Number of pupils	259 (26)	232 (12)	0.38	217 (2)	0.09
	N = 9a	N = 60		N = 6634
Central exam results	534.13 (1.35)	535.40 (0.51)	0.42	534.94 (0.06)	0.54
	N = 9a	N = 56		N = 5332
Parental educational level (% 0.3 or 1.2)b	0.19 (0.04)	0.14 (0.02)	0.21	0.11 (0.00)	0.01
	N = 9a	N = 60		N = 6637
School advice (% of pupils referred to preparatory secondary vocational education or lower)c	29.90 (13.86)N = 9a	24.57 (13.61)N = 58	0.39	N = 22.97 (14.91)N = 6542	0.15
Schoolyard size (m2/child)	7.38 (0.05)N = 8	6.38 (0.04)N = 17	0.57		NAd
Satisfaction with the school	7.58 (1.25)N = 464	7.73 (1.15)N = 9486	0.004		NAd

All significant values are in bold.

Data were collected for nine HPSF schools, because two schools merged directly after baseline.

In Dutch: leerlingengewicht with categories 0, 0.3 and 1.2; calculated as the number of children with category 0.3 + 1.2/total amount of pupils.

In Dutch: VMBO or lower; calculated as the number of children referred to VMBO or lower/total amount of pupils which received a secondary school advice.

Data not available.

No quantitative data on physical education were available for either the regional or national level. The Dutch Inspectorate for Education reported that most primary schools gave physical education twice a week, with an average of 52 min per lesson (active sporting time) in 2016–2017 (Peil.Bewegingsonderwijs, 2018). In 2015, five out of nine HPSF gave physical education twice a week, and four schools gave physical education once a week. The average duration per lesson was 58 min.

Discussion

As the overall participation rate of children in the HPSF-study was 60%, concerns could be raised about the representativeness of the sample. However, we observed no statistical difference between responders and non-responders on the majority of outcomes including obesity, physical and mental health, and family composition. The so-called ‘healthy-volunteer’ bias in the HPSF-study is limited to a higher educational level of parents. As only parental non-Western immigration background differed between HPSF-responders and regional data, we argue that the external validity of HPSF-responders with the region was high. The external validity of our study population with the Dutch population is low, as differences with Dutch children have been found on a diverse palette of characteristics including sex, parental employment, parental education, ethnicity, and physical activity behaviours.

Previous studies reported that participants with a high SES are more likely to participate in scientific studies (Van Loon et al., 2003; Martikainen et al., 2007; Plachta-Danielzik et al., 2008; Struijk et al., 2014; Pinsky et al., 2007), which is consistent with our findings. However, no other differences between non-responders and responders were observed in this study, while several other studies report differences in terms of health, lifestyle, obesity, and unemployment (Van Loon et al., 2003; Martikainen et al., 2007; Plachta-Danielzik et al., 2008; Korkeila et al., 2001; Brussaard et al., 1997). This might be partially attributed to the methods by which participants were recruited in the HPSF-study. First of all, time and resources were reserved for personalised contact in order to prevent the healthy volunteer bias (Jordan et al., 2013). Secondly, in the intervention schools of HPSF, participation in the intervention was not linked to participation in the study. This might have reduced the chance that participants refused to participate. All children who completed the baseline survey will automatically be participating in the intervention study.

The finding that pupils with higher educated parents are more likely to participate in this sample is not only relevant when reporting on generalisability, but possibly also relevant when reporting on the effectiveness of the intervention. Generally, higher-educated persons tend to have a better health and a healthier lifestyle, as previously confirmed in the HPSF-cohort (Vermeiren et al., 2018). Moreover, low parental education is a major determinant of overweight and obesity in children (Danielzik et al., 2004). Consequently, there could be less room for improvement in studies involving lifestyle interventions. There is a risk that small effects remain undetected and the study does not achieve the health outcomes it attempted to achieve. However, as the healthy volunteer bias in our sample is limited to parental educational level, we estimate that this risk is fairly small.

The low external validity of our study sample with the Netherlands was expected, since the region is known not to be representative of the nation. Several national reports have shown that the area of Parkstad is a relatively poor region with a low SES, low employment rates, and a different composition of the population as compared to the rest of the Netherlands (Steenbakkers et al., 2014; Jansen, 2015). This might be due to the history of the region, as South-Limburg and Parkstad are a former mining area. Several health markers and healthy lifestyle habits differed between HPSF-respondents and the Dutch average, including less frequent use of active transport to school, and more frequent sports club membership. A previous report showed that children in the Parkstad region have a higher prevalence of unhealthy lifestyles, but this has not been uniformly confirmed by our study (Steenbakkers et al., 2014; Jansen, 2015). The low external validity of the current study should be put into perspective. Although our outcomes differ with the average in the Netherlands, we should acknowledge that several areas in the Netherlands are comparable with the Parkstad region with regard to SES-status, employment rates and ethnicity. Therefore, our study might have a high external validity if these regions are being studied.

The current study has a few limitations and methodological considerations that should be mentioned. In the current study, all children from the eight different HPSF-schools were analysed as one group, neglecting the cluster design of the HPSF-study. Unfortunately, it was not possible to provide an accurate and complete estimate of the intra-class correlation in schools. However, other studies have shown a relatively low ICC in school-based cluster RCTs (Waters et al., 2018; Viggiano et al., 2015; Amorim et al., 2007). If the ICC would be taken into account, it is likely that the P-value would increase. This would indicate that the current results are the most conservative. Some of the measurement methods differed slightly between data sources (Additional file A). Where possible, these methodological issues were minimised, for example by recoding divergent answer options into new comparable categories. The HPSF-study is a longitudinal open cohort study in which participants enrol between 2015 and 2019, and drop-out when they graduate, leave school and/or decide to stop participating in the study. In the current study, a cross-sectional analysis of baseline participants has been performed. Therefore, our conclusions cannot be extrapolated to the longitudinal HPSF-population. We hypothesize that the dynamic study cohort does not create new sources of selection bias. However, it is likely that the HPSF-intervention attracts families with a healthy lifestyle, which consequently reinforces the differences between those exposed and those not exposed to the intervention. We could not correct for multiple testing in the current study, whereby the risk for false positive outcomes increases. However, as this study has an explorative purpose, we attach more importance to clinical relevance rather than statistical significance. Some analyses showed relatively large differences between groups, while these differences were not statistically significant. These could nevertheless be taken into account when drawing conclusions. We can rule out potential non-response bias in terms of the variables that were observed. However, the risk remains that non-response bias still exists along unobserved dimensions. Another limitation of our study is the quality of the external datasets used for comparison. The external datasets in this study are most likely, to an unknown extent, subject to either the healthy volunteer effect or low external validity. However, it is methodologically complex to assess these effects in large population-based screenings. Moreover, some external data sources collected data from 2007 and onwards, which is before the baseline measurement of the HPSF-study in 2015. Lastly, only children of HPSF with a comparable age as those of the external source were included. Therefore, in most analyses, we were not able to perform an analysis with all responders included.

If the long-term effects of the HPSF-study were to be modelled to the Dutch population, adjustments could be made to the higher participation-rate of households with a high SES, and the socio-economic and health-related differences of the Parkstad region compared to the Netherlands. The data presented here can be used to determine the adjustments that are needed.

Although our study shows similar response rates compared to other health-related intervention studies (Van Loon et al., 2003; Plachta-Danielzik et al., 2008; Lindsted et al., 1996), the healthy volunteer effect has rarely been studied using both non-responder data and population databases. This is mainly due to limited available data on non-respondents. The majority of studies investigating non-response are population-based screenings on disease prevalence in adults (Struijk et al., 2014; Pinsky et al., 2007). The current study gives a unique view of non-response and external validity of a school-based cohort study at both subject and group-level. We were able to select well-matching up-to-date databases at non-responder, regional, and national level covering a wide area of relevant outcome measures. Therefore, we can take a healthy volunteer effect into consideration when drawing conclusions. This is a major advantage as compared to studies that did not assess external validity, as they can face difficulties solidifying their conclusions.

In conclusion, a possible healthy-volunteer effect in the HPSF-sample is limited. We therefore do not foresee problems regarding bias due to low response rates. We showed a high external validity for the regional population but a low external validity for the national sample. We argue that assessing generalisability should always be considered to be incorporated in the design of a study protocol, for which the current study can be used as an example. Obtaining information on non-responders will most likely become more complicated in the future due to the new GDPR of the EU (Chassang, 2017). Therefore, we advise researchers to evaluate their outcome measures according to regional and national standards to allow outcome comparability, and, where possible, to cooperate with other parties in early stages of their research to obtain information on non-responders (e.g. regional health care settings or national databanks).

Declarations

Ethics approval and consent to participate

The need for ethical approval of the HPSF-study was waived by the Medical Ethics Committee Zuyderland, Heerlen (MEC 14-N-142). All HPSF-responders did provide written parental informed consent. Data from the GGD-ZL of non-responders were provided on a group level and could therefore not be traced back to individuals. All other datasets did not include data that could be traced back to individuals.

Availability of data and material

The data that support the findings of this study are available from the external data suppliers mentioned in the article, but restrictions apply to the availability of these data. The data were used under license for the current study, and are therefore not publicly available. Data are however available from the authors upon reasonable request and with permission of the external data suppliers.

Competing interests

None.

Funding

This work was supported by the Limburg Provincial Authorities (200130003); Friesland Campina (LLMV00); and by Maastricht University.

Authors' contributions

EAB performed the analyses, and has written the introduction, methods, and results section of the manuscript. JJP collected the data from the external data suppliers. KK provided data from the GGD-ZL. DvK provided the data from the Active Living study. CPvS has reviewed the manuscript. MW supervised the various stages of the study and has written the discussion section of the manuscript. All authors have read and approved the final version of the manuscript.