Measurement of Health and Social Behaviors in Schoolchildren: Randomized Study Comparing Paper Versus Electronic Mode.

INTRODUCTION: Electronic survey mode has become a more common tool of research than it used to be previously. This is strongly associated with the overall digitization of modern society. However, the evidence on the possible mode effect on study results has been scarce. Therefore, the aim of this study is to investigate the comparability of findings on health and behaviours using a paper-versus-electronic mode of survey with randomization design among schoolchildren.
METHODS: A randomized study was conducted using a mandatory questionnaire on international Health Behaviour in School-aged Children (HBSC) study in Lithuania, enrolling 531 schoolchildren aged 11-15 years. The questionnaire included health and social topics about physical activity, risk behaviours, self-reported health and symptoms, life satisfaction, bullying, fighting, family and school environment, peer relationships, electronic media communication, sociodemographic indicators, etc. The schoolchildren within classes were randomly selected for electronic or paper mode.
RESULTS: It was found that by study mode differences are inconsistent and in the majority of cases do not exceed 5%-point difference between the modes. The only significant difference was that in the paper survey the participants reported more exercise than in the electronic survey (OR=8.08, P<.001). Other trends were nonsignificant and did not show a consistent pattern - in certain behaviours the paper mode was related to healthier choices, while in others - the electronic.
CONCLUSIONS: The use of electronic questionnaires in surveys of schoolchildren may provide findings that are comparable with concurrent or previously conducted paper surveys.

Introduction

Information and communication technology has become an ever more demanded working tool to enhance the management, efficiency, and quality of surveys on health and social phenomena. There are several kinds of electronic questionnaires – online access, mobile device administered by the researcher, or computer/device handled by respondent. The responses can be collected by participant, researcher or a proxy (if a participant is minor). Overall digitization of social life and communication suggests ever-increasing pressure to conduct digital surveys and, therefore, it is essential to assess how reliable and valid the digital methods are and, if they replace paper-and-pencil method, are the findings comparable?

The online mode reduces the study costs by saving on the costs of paper and printing as well as from transportation (1). Besides, it ensures quick data with virtually no errors and suggests fewer no-response answers (2). Another important point is that these devices permit automatic checking of responses and complex skip patterns. However, in the digital survey mode, it is essential to ensure who is filling in the questionnaire, which is not always feasible.

The literature on the effects of digital-based and computer-adaptive testing suggests that digitization of standardized tests is a precise and appropriate research mode both from a scientific and logistic point of view (3, 4). Nonetheless, some researchers propose that the reliability of data obtained by the web-based approach should be determined (5). There is also a potential for selection bias, where a particular type of participant may be more prone to a particular survey mode (e.g. preference for digital mode among younger, more affluent or educated people). Moreover, in online mode, the participants can be unknown, not meet eligibility criteria or make double entries. Therefore, due to the potential for selection bias a randomized controlled design could be regarded as the main choice in studies on potential mode effects.

Even though many studies analysing the issue of mode effect on study results use randomization, quite a lot of them address the issue of response rate foremost, while content-specific comparison receives less attention. Also, such studies rarely investigate younger groups and the majority of them do not use randomization. For example, in the international Health Behaviour in School-aged Children (HBSC) study some countries use mixed mode design for more than a decade, e.g. Belgium (6), but they usually do not randomize the schools or children, leaving the choice of mode up to the school’s or child’s preference – which may be a subject to bias.

Thus, even though research on the validity and reliability of digital versus paper mode is quite extensive, such assessment in adolescents is rarely addressed. Moreover, the randomized approach in the research of mode effect is not always applicable, leaving the findings with a potential for self-selection or school-specific bias. In addition, the health perceptions and behaviours have also been under-investigated from this perspective. Therefore, the objective of our study is to compare the findings from paper and electronic mode using a randomized controlled design among schoolchildren.

Methods

Study Process and Sample

The randomized controlled study was conducted in May 2017 at five secondary schools in Lithuania. All study subjects were informed about the details of the study and that the return of the filled questionnaire will be treated as the informed consent. The anonymity of study participants and confidentiality of the data was ensured. The study was conducted as a pilot project for an oncoming 2018 Health Behaviour in School-aged Children study in Lithuania. The schools were randomly selected from the national schools’ list, by choosing the first five schools who agreed to participate in the study. The schools were from the second-largest city, other cities, and one town. In every school, the questionnaire was administered to 5th, 7th, and 9th grades (predominant age of children 11, 13 and 15 years, respectively). Then, the randomization was applied for every class in the school, with one-half of students filling the questionnaire in paper mode and the other half in electronic mode. Every class was randomized to define which half of the students’ list filled the online and which the paper version of the questionnaire.

Questionnaires (both electronic and paper mode) were administered in school classrooms by trained researchers who complied with written instructions. The electronic version of the questionnaire was uploaded to Google Forms, which was available only to the researchers. During the survey, the researchers shared the web link to study participants. The online questionnaire was filled in on desktop or tablet computers. The places for survey were usually classrooms, computer rooms or libraries. In some cases, the survey of paper and online mode was conducted simultaneously in the same room. Every researcher wrote the notes about the procedure of survey.

Measurements

The tool for the study was based on the then-current version of the standardized international HBSC research protocol (7). The HBSC questionnaire covers a wide range of health and social topics about schoolchildren’s physical activity, risk behaviours, self-reported health and symptoms, life satisfaction, bullying, fighting, family, school environment, peer relationships, electronic media communication, sociodemographic indicators, etc. Only the mandatory items were included. The sequence, formulation, and overall visualization of items did not differ by mode.

Some items of the questionnaire were used from particular scales or subscales:

HBSC symptom checklist, 8 items (7),

Family Affluence Scale, 6 items (8),

Multidimensional Scale of Perceived Social Support: Family, 4 items (9),

Multidimensional Scale of Perceived Social Support: Friends, 4 items (adapted from (9)),

Teacher and Classmate Support Scale: Classmates, 3 items (adapted from (10)),

Teacher and Classmate Support Scale: Teachers, 3 items (adapted from (10)),

Online contact with friends and others, 4 items (11),

Preference for online communication, 3 items (12),

Social media addiction, 9 items (13).

Data Analysis

Data was processed using MS Excel 2010 and analysed using IBM SPSS Statistics, version 20. The descriptive analysis included the calculation of the prevalence of different health behaviours (%). The items were dichotomized based on the cut-offs used in the 2014 Health Behaviour in School-aged Children study report (14). The main purpose of the analysis was to estimate whether various health-related items are similarly distributed among study groups in schoolchildren that filled in the questionnaire in paper-versus-electronic mode. For this, the percentage point differences were calculated, and logistic regression was used with the calculation of certain behaviours’ risk when comparing the modes. The differences between the modes were estimated using percentage point difference and odds ratios with the reference group being electronic mode (OR=1.00). Given that despite randomization there were some imbalances between the study groups by gender, grade, and school, these indicators were adjusted for in the multivariate logistic regression model.

Due to multiple comparisons of different indicators, the Bonferroni correction was used: in total, 78 variables were compared, so the conventional significance level of P<0.05 was decreased to P<0.001 (0.05/78=0.00064). The P-values between 0.001 and 0.05 were reported as trends.

Results

The study sample comprised 531 schoolchildren – 261 filled the electronic questionnaire and 270 the paper version. The overall response rate was 83.0% with higher rates among girls and elder schoolchildren. A detailed comparison of study groups by gender, grade, and school are presented in Table 1. Regardless of randomization, there were some differences observed between study groups and since they were definitely random (by design of the study) their statistical significance was not calculated.

Table 1

The main characteristics of study sample.

Characteristic	Electronic mode	Paper mode	n	Response rate
Gender
Boys	51.4	48.6	255	77.0
Girls	47.3	52.7	275	89.0
Grade
5th	49.7	50.3	187	77.9
7th	48.8	51.2	201	84.1
9th	49.0	51.0	143	88.8
School
#1 (large city)	47.3	52.7	74	89.2
#2 (large city)	48.7	51.3	224	94.5
#3 (city)	50.5	49.5	103	60.6
#4 (city)	50.0	50.0	48	80.0
#5 (town)	50.0	50.0	82	91.1

In this study, the internal consistency of scales and subscales was acceptable and the difference between the modes was not more than .07 points – with no consistent superiority of either mode (Table 2).

Table 2

Internal consistency of study scales and subscales by survey mode.

Scale	Number of items	Internal consistency (α)
		Electronic mode	Paper mode
HBSC symptom checklist	8	.78	.79
Family Affluence Scale	6	.52	.58
Multidimensional Scale of Perceived Social Support: Family	4	.76	.69
Multidimensional Scale of Perceived Social Support: Friends	4	.90	.85
Teacher and Classmate Support Scale: Classmates	3	.77	.70
Teacher and Classmate Support Scale: Teachers	3	.75	.74
Online contact with friends and others	4	.54	.54
Preference for online communication	3	.84	.81
Social media addiction	9	.75	.76

Health Behaviours

In the field of health behaviours (Table 3), the largest difference depending on survey mode was observed in extensive physical activity – in paper mode, the schoolchildren more frequently reported daily exercise until getting out of breath or sweating (OR=8.08, P<.001). Other indicators had no differences except the trends that students in paper mode more frequently reported, such as having a regular breakfast on weekends (OR=1.93, P=.009). Almost all aspects of health behaviours differed between the survey modes by no more than 5% points.

Table 3

Health behaviours of schoolchildren by survey mode.

Characteristic		Prevalence, %		% difference	OR	P
		Electronic	Paper
Eating habits
Having breakfast during the weekdays	Every day	58.8	62.7	3.9	1.18	.366
Having breakfast during the weekends	Every day	79.6	87.7	8.1	1.93	.009
Having breakfast with parents	Every day	41.0	40.7	-.3	1.00	.982
Having dinner with parents	Every day	47.1	45.6	-1.5	.96	.816
Eating fruits	Every day	41.8	38.5	-3.3	.87	.446
Eating vegetables	Every day	32.6	34.2	1.6	1.07	.707
Eating sweets	Every day	16.1	13.8	-2.3	.83	.453
Drinking soft drinks	Every day	5.4	6.3	.9	1.23	.593
Drinking energy drinks	Every day	2.3	.4	-1.9	.16	.097
Health and well-being
Subjective health assessment	Good	88.5	91.8	3.3	1.59	.132
Life satisfaction	6–10 (10 pts scale)	87.7	85.8	-1.9	.84	.510
Headache	Rarely	84.3	82.5	-1.8	.91	.707
Stomach ache	Rarely	93.5	93.7	.2	1.04	.912
Backache	Rarely	91.6	92.1	.5	1.09	.794
Feeling low	Rarely	80.1	81.3	1.2	1.14	.577
Irritability or bad temper	Rarely	72.0	76.5	4.5	1.39	.115
Feeling nervous	Rarely	70.5	69.7	-.8	1.01	.946
Difficulties in getting to sleep	Rarely	79.7	83.2	3.5	1.33	.214
Feeling dizzy	Rarely	89.7	89.9	.2	1.08	.798
Brushing the teeth	More than once a day	61.3	62.8	1.5	1.05	.809
Body image	A bit too thin	11.9	15.0	3.1	.71	.200
	A bit too fat	29.1	30.7	1.6	.91	.654
	About the right size	59.0	54.3	-4.7	1.00	-
Physical activity
Physical activity at least 60 minutes per day (last week)	7 days	18.9	20.6	1.7	1.16	.518
Exercise in free time until getting out of breath or sweating	Every day	3.1	19.7	16.6	8.08	<.001
Risk behaviour
Cigarette smoking (lifetime)	Never	73.2	77.8	4.6	1.37	.187
Cigarette smoking (last month)	Never	88.1	92.3	4.2	1.75	.097
Alcohol drinking (lifetime)	Never	62.8	68.4	5.6	1.37	.129
Alcohol drinking (last month)	Never	88.9	88.0	-.9	.89	.706
Cannabis taking (lifetime)	Never	94.3	97.0	2.7	2.22	.101
Cannabis taking (last month)	Never	98.1	99.2	1.1	2.13	.376
Sexual intercourse	No	95.0	93.3	-1.7	.81	.626

Social Behaviours and School

The selected indicators of social behaviours under study showed slightly bigger differences than health behaviours, though they were inconsistent and nonsignificant (Table 4). Here the trend in paper mode was that the children were more likely to report having friends to share joys and sorrows, but also more cyber-bullying and more treatment-needed injuries (.001

Table 4

School behaviours of schoolchildren by survey mode.

Characteristic		Prevalence, %		% difference	OR	P
		Electronic	Paper
Friends support
Friends help	Agree	76.6	76.4	-.2	.97	.870
Counting on friends	Agree	75.1	78.3	3.2	1.18	.440
Having friends to share joys and sorrows	Agree	78.2	87.7	9.5	1.98	.005
Being able to talk about problems with friends	Agree	70.5	76.4	5.9	1.35	.138
Classmate support
Enjoy being together with students	Agree	59.8	54.1	-5.7	.81	.243
Students in class are kind and helpful	Agree	53.3	52.8	-.5	1.00	.982
Students accepting one as he/she is	Agree	67.4	61.8	-5.6	.80	.231
Teacher support
Teachers accepting one as he/she is	Agree	75.1	78.1	3.0	1.23	.348
Teachers caring	Agree	49.8	52.6	2.8	1.15	.426
Feeling a lot of trust in teachers	Agree	65.1	67.5	2.4	1.16	.462
School perception
Feeling about school	I like it a lot	81.1	82.9	1.8	1.14	.586
Pressure by schoolwork	Some or a lot	72.2	72.7	.5	.99	.947
Bullying
Taking part in bullying another student	Yes	44.2	42.9	-1.3	.97	.889
at school, last two months
Being bullied at school, last two months	Yes	50.2	46.5	-3.7	.86	.402
Taking part in cyber-bullying, last two months	Yes	20.3	18.7	-1.6	.92	.708
Being cyber-bullied, last two months	Yes	14.1	22.3	8.2	1.82	.011
Physical fighting
Having a physical fight, last year	Yes	31.7	29.4	-2.3	.92	.688
Injuries
Being injured with treatment needed, last year	Yes	48.8	57.6	8.8	1.45	.036

Family Environment

The evaluation of schoolchildren’s family environment revealed that there were almost no differences depending on survey mode (Table 5). The children in paper mode reported slightly better family communication and support, but this was nonsignificant (P=0.068). All other indicators did not reach a 5%-point difference and, regarding items on family affluence, the differences by paper mode were also minor.

Table 5

Family-related perceptions of schoolchildren by survey mode.

Characteristic		Prevalence, %		% difference	OR	P
		Electronic	Paper
Communication
Ease to talk about things that really bother: to father	Easy	63.2	69.7	6.5	1.43	.068
Ease to talk about things that really bother: to stepfather	Easy	11.5	15.5	4.0	1.48	.221
Ease to talk about things that really bother: to mother	Easy	79.7	82.0	2.3	1.15	.526
Ease to talk about things that really bother: to stepmother	Easy	9.6	10.1	.5	1.08	.848
Support
Family really tries helping	Agree	92.0	93.7	1.7	1.29	.461
Getting emotional help and support from family	Agree	83.9	88.0	4.1	1.43	.164
Being able to talk about problems with family	Agree	69.3	71.2	1.9	1.13	.537
Family is willing to help in making decisions	Agree	86.6	90.3	3.7	1.45	.183
Affluence
Own bedroom	Yes	81.2	81.4	.2	1.02	.941
Dishwasher at home	Yes	62.5	59.7	-2.8	.87	.496
Bathrooms at home	One or more	97.7	97.8	.1	1.03	.958
Family car	One or more	95.0	93.7	-1.3	.78	.525
Computers at home	One or more	97.3	97.7	.4	1.15	.806
Family travel abroad for vacation, last year	Once or more	86.5	81.6	-4.9	.68	.129

Electronic Media Communication

The survey included three main aspects of electronic communication – online contact with friends, preference for online communication, and social media addiction (Table 6). Here there were two trends: in electronic mode, children reported using social media more as a way to escape from negative feelings and having conflicts with family members because of social media use (.001

Table 6

Electronic media communication of schoolchildren by survey mode.

Characteristic		Prevalence, %		% difference	OR	P
		Electronic	Paper
Online contact with friends and others
Close friend(s)	Every day	66.7	63.3	-3.4	.81	.277
Friends from a larger friend group	Every day	37.9	40.6	2.7	.12	.535
Friends that you got to know through the internet but didn’t know before	Every day	11.5	14.7	3.2	1.32	.302
Other people than friends	Every day	44.8	47.7	2.9	1.11	.563
Preference for online communication
On the internet, I talk more easily about secrets than in a face-to-face encounter	Agree	27.3	25.7	-1.6	.93	.725
On the internet, I talk more easily about my inner feelings than in a face-to-face encounter	Agree	26.8	22.4	-4.4	.78	.238
On the internet, I talk more easily about my concerns than in a face-to-face encounter	Agree	27.6	20.1	-7.5	.67	.054
Social media addiction
Regularly felt dissatisfied because you wanted to spend more time on social media	Yes	16.9	18.0	1.1	1.03	.912
Often felt bad when you could not use social media	Yes	27.2	25.7	-1.5	.88	.528
Tried to spend less time on social media, but failed	Yes	28.7	25.3	-3.4	.81	.284
Regularly neglected other activities (e.g. hobbies, sport) because you wanted to use social media	Yes	13.8	12.4	-1.4	.90	.700
Regularly had arguments with others because of your social media use	Yes	14.9	12.0	-2.9	.76	.290
Regularly lied to your parents or friends about the amount of time you spend on social media	Yes	17.2	16.1	-1.1	.89	.628
Often used social media to escape from negative feelings	Yes	30.3	22.5	-7.8	.64	.030
Had serious conflict with your parents, brother(s) or sister(s) because of your social media use	Yes	21.8	13.5	-8.3	.49	.004
Regularly found that you can’t think of anything else but the moment that you will be able to use social media again	Yes	40.6	39.9	-.7	.93	.704

Procedure-Specific Findings

In this study, the environment and circumstances of the survey were also documented in order to depict the procedure-specific findings. So, during the survey and especially in online mode some participants were able to see the adjacent participants’ responses, thus infringing the privacy of other responders. In addition, the teachers sometimes refused to leave the classroom even when asked.

It was also observed that some schoolchildren were not content with the assigned mode of the survey. However, this was not mode-specific: some adolescents expressed the wish to move from paper mode to electronic, while others vice versa. The former ones were keener to choose the electronic device (computer or tablet) instead of paper, while the latter preferred more privacy. Of note, some students were concerned about the split of the class into different modes as if treated unequally.

Discussion

Electronic research mode is very convenient for large-scale studies. In order to address the possible effect of survey mode on its results, we conducted a randomized study to eliminate the potential for selection bias within the study sample. This is the main strength of our study since the previous research has quite frequently neglected the issue of self-selection bias that arises in non-randomized studies. This is especially relevant across different social conditions such as schools, where some of them may have better resources to prefer online mode, either through better financing or through higher quality of educational services. So, by the study design, our study avoided the possible self-selection bias or school-specific differences by randomizing the schoolchildren within classes. We also adjusted the calculations by main sociodemographic indicators that could affect the differences. Besides, the inclusion of different size schools from bigger and smaller urban areas also increased the diversity of schoolchildren.

However, when discussing our study weaknesses, we had a limited sample size, which potentially led to an underestimating of the statistical significance of differences, especially when controlling for multiplicity. Nevertheless, we found that in the majority of cases the differences between the modes were small and did not exceed a 5% point. For such five percent differences to detect as statistically significant at P<0.05 level, we would have needed the sample from 431 to 1,559 per arm – and this without the multiplicity correction that was applied in our study (for P<0.01 640 and 2,315 participants would be needed (15), respectively, and for P<0.001 even more). Compared to previous studies, our sample size was rather medium, and we had no intention to find minor differences as statistically significant. After all, the fact that absolute differences between the modes were inconsistent (i.e. not showing better health behaviours in either mode) suggests the likely absence of substantial differences.

Another limitation of our study was the lack of replicability since our study participants had an opportunity to fill in the questionnaire only in one of the modes. Therefore, the assessment of the consistency of results within individuals was not possible. This occurred because we raised no question regarding the particular subject’s replicability of responses – rather, we had an interest in comparing the population (i.e. study sample) estimates.

Overall, our study findings revealed that differences by study mode are virtually absent and in the majority of cases do not exceed five percent difference between the modes. These findings do not have many studies to compare with since the schoolchildren’s health behaviours have rarely been addressed in previous research on survey modes. The study on the HBSC sample was previously reported by Vereecken and Maes (6) in Belgium. Their findings showed some differences by mode, but our results did not support them. We did not detect those differences not only due to a smaller sample size (i.e. lower power of the study) but rather due to the absence of absolute difference.

That same study (6) noted that for several questions about feelings and affective states more socially desirable responses were found in paper format. However, in our case, this was not observed. Even though we saw some larger differences when assessing social support measures, this was also inconsistent. The fact that the adolescents provide equivalent responses in paper and computer formats was also found elsewhere (16).

It was found that adolescents were more likely to report substance use and less desirable aspects of psychological well-being using a digital format (17). However, we found that subjective health was reported as slightly better in paper mode (like Smith et al. (18) in the military sample), while higher life satisfaction was reported as better in electronic mode, which does not suggest the consistency of mode effect.

The issue of different responses by survey mode has been addressed with other samples as well. For example, patients after knee surgery reported similar levels of daily functioning, quality of life, pain intensity as well as symptoms (19). Similarly, for college students in education and psychology, the survey mode did not have strong differential effects on data quality regarding the learning environment and perceptions (20). One study on military participants found some differences in health behaviours by mode, though like our study they did not exceed five percent (18).

We also compared the internal consistency of scales. Previous studies demonstrated that electronic mode is likely to show higher internal consistency compared to the paper, with differences by up to .30 (16), however, in our study there was no superiority of either mode (differences did not exceed .07). Some other studies also showed no relevant differences in psychometric properties by mode (21).

Previous research suggests that young people are keener to choose the electronic than the paper version (22), while the studies of other samples are rather ambivalent: for instance, the study on people who take supplements and vitamins found the electronic version as more acceptable (1, 23), while HIV patients preferred the paper version (24). Interestingly, our procedure-specific findings also indicate ambiguity, since some children preferred to move to electronic, while others to paper mode. This was rather unexpected due to the hypothesized preference of digital natives toward electronic mode. It could be explained by the fact that maintaining privacy was an issue during this study, especially when filling in electronic questionnaires on desktop computers: schoolchildren were able to see the answers of adjacent classmates, which could have made them feel insecure. Additionally, in some classes, the teachers refused to leave the room, which may interfere with the confidentiality perceptions of children and the sincerity of their responses. The fact that some schoolchildren complained about having different survey modes across the class implies that, in the future, a class as an entity should preferably be investigated using the same mode.

Regarding the cost-effectiveness of the shift from paper mode to an electronic mode in our study, the main difference was related to expenses for paper and printing the questionnaire as well as typing in the responses from paper to database. In addition, the probability of data typing errors in case of the electronic mode is virtually zero. Nevertheless, the shift toward electronic mode should be approached carefully: if the survey is going to be uploaded online with a non-restricted access, the study participants cannot be controlled. This may further result in a situation where some subjects submit several questionnaires, or they do not meet the eligibility criteria for age or other relevant characteristics. It should be emphasized that in online surveys the basic concern is related to the problem of who really fills in the questionnaire and if they meet the eligibility criteria of the study. This should be controlled whenever possible.

Conclusions

Summarizing our study, it can be stated that the comparison of electronic and paper mode in the research of health and social behaviours among schoolchildren revealed no consistent differences between the modes. There were some items or questions that had larger differences between the survey modes, however, they did not have a trend to be healthier or more socially desirable in one particular mode. This suggests that, in the future, the use of electronic questionnaires in surveys of schoolchildren may provide findings that are comparable with concurrent or previously conducted paper surveys. However, this does not relieve the concerns related to electronic surveys where the study participants are not controlled in terms of eligibility criteria. Thus, when the electronic survey responders are unknown, this still threatens the validity of study findings.