Validation of Self-Reported Smartphone Usage Against Objectively-Measured Smartphone Usage in Hong Kong Chinese Adolescents and Young Adults.

OBJECTIVE: This study evaluated the validity of self-reported smartphone usage data against objectively-measured smartphone usage data by directly tracking the activities in the participants' smartphone among Chinese adolescents and young adults in Hong Kong.
METHODS: A total of 187 participants were recruited (mean age 19.4, 71.7% female) between 2017 and 2018. A smartphone usage tracking app was installed on all participants' smartphone for 7 consecutive days. After the 7-day monitoring period, they completed a selfadministered questionnaire on smartphone usage habits.
RESULTS: Although the correlation between self-reported and objectively-measured total smartphone usage time was insignificant (ρ=-0.10, p=0.18), in three out of the four usage domains were positively and significantly correlated, namely social network (ρ=0.21, p=0.005), instant messaging (ρ=0.27, p<0.001), and games (ρ=0.64, p<0.001). Participants' self-report of the total time spent on smartphones exceeded the objective data by around 760 min per week (self-reported 1,930.3 min/wk vs. objectively-measured 1,170.7 min/wk, p<0.001). Most of the over-reporting was contributed by the web browsing domain (self-reported 447.8 min/wk vs. objectively-measured 33.3 min/wk, p<0.001).
CONCLUSION: Our results showed large discrepancies between self-reported smartphone and objectively-measured smartphone usage except for self-reported usage on game apps.

INTRODUCTION

Smartphone use is wide-spread and its penetration rate is higher than 50% in many countries [1]. The extant literature had examined the adverse impact of smartphone use on health outcomes, including bodily pain [2-4], poor mental health [5], and sleep problems [6-8]. Smartphone usage level is also correlated with and could clinically assist the diagnosis of smartphone addiction [9]. While smartphone addiction has a clear definition and there exist several validated scales for measuring smartphone addiction and its components [10-13], most existing questionnaires that measure smartphone usage level were not validated, thus the validity and reliability of self-report data were questionable. This is due to the nature of smartphone usage that is sometimes difficult to recall; smartphone usage pattern is typically frequent with short durations [9]. Among Korean University students, in the most extreme cases of smartphone users, objective data showed that they spent nearly 300 sessions per day with each session lasting on an average for two minutes [9]. Therefore, it is natural to hypothesize that self-reported smartphone usage might be subjected to recall bias. In fact, objectively-measured smartphone usage data can be passively-collected by directly tracking the activities in the smartphone [9]. However, such kind of direct smartphone usage tracking requires a lot of fieldwork resources and therefore direct tracking may not be practical. Therefore, self-reported smartphone usage data are still being widely used in epidemiological studies, and the internal validity of these studies heavily relied on the validity of the self-reported smartphone usage data. For example, a study on the association between smartphone usage, anxiety, and depressive symptoms asked the respondents to self-reported 11 types of smartphone usage in a 6-level frequency scale ranging from “never” to “very often.” [14] In the present study, we aim to evaluate the validity of self-reported smartphone usage data against objectively-measured data among Chinese adolescents and young adults in Hong Kong. For both objective and subjective smartphone usage, we classified the usage for four purposes, namely social networking, instant messaging, web browsing, and gaming, as they had different impacts on health [15,16].

METHODS

Participants

This study was conducted from March 2017 to May 2018. The participants were recruited in one secondary school (corresponds to Year 7 to 12 in the US education system) and two tertiary institutions in Hong Kong. In secondary school, invitation letters were sent to parents of all students, and the list of potential participants was prepared by the school. In the two tertiary institutions, invitation emails were sent to all undergraduate students. Only those aged 11–25, able to speak and read Chinese, and owned a smartphone with the Android operating system were recruited in this study. Phones with Apple iOS operating system were excluded as Apple iOS app programmers were not allowed to collect smartphone usage data in a continuous manner. Written consent was obtained from all participants. For participants under 18 years old, parental written consent was also obtained. As a token of appreciation, a supermarket cash coupon worth HK $100 (approximately US $13) was given to the participants after data collection. This research was approved by the Human Subjects Ethics Sub-Committee of the Hong Kong Polytechnic University (HSEARS20160425004).

Data collection

Objectively-measured smartphone usage

We have installed a smartphone usage tracking app (created by our team) on all participants’ smartphone for 7 consecutive days. It was found that smartphone usage pattern has strong within-subject correlation [17] and 7-day monitoring would be adequate to capture the habitual smartphone usage behavior of a participant. This app tracked the opening and closing of all apps in the smartphone. No private and personal data, such as the content of instant messaging and web pages browsed, were collected. For participants who possessed more than one smartphones, we installed the app in the most commonly-used one. All participants installed this app by downloading the application package (APK) via the QR code in the information sheet. We define that an app was being used when it was running in the graphical interface, and an opening of an app will trigger the closure of the app currently being used, if any. Due to the limitation of the Android operating system, all system default built-in apps could not be captured. The usage time was tracked by seconds. The name of the apps used, their opening time, and closing time were stored on the smartphone and upload to a remote server (Firebase) once internet connection was available, and all uploaded data were removed from the smartphone.

Self-reported smartphone usage

After the 7-day monitoring period, all participants completed a self-administered questionnaire about the information on smartphone usage habit, including the number of smartphones and tablets possessed and time spent on smartphone. The time spent on social network apps (such as Instagram, Twitter, and Facebook), instant messaging apps (such as WhatsApp, Skype, Wechat), web browsing apps (including web browsers such as Firefox and Opera and website-specific apps), games and comics apps (grouped together as they referred to the entertainment of similar nature and some games and comics shared the same story), and total time spent was reported, and they were separated by school-days and holidays. Other types of apps (multimedia apps such as YouTube, tools such as dictionary, health-related apps such as pedometer, and camera apps) were grouped into the “others” category and excluded in the current validation study. Upon completion of the questionnaire, the smartphone usage tracking app was removed from the participants’ smartphone.

Data processing

All recorded smartphone applications were categorized into social network (Facebook, Twitter, Instagram, Weibo, etc.), instant messaging (Whatsapp, Skype, Line, Wechat, etc.), web browsing (including browser and apps developed for browsing specific websites), games (except games plug-ins and guides), and other types of apps. Regardless of the types of usage that could be performed by an app, one app would be assigned to one single category. Time spent on each usage session was computed as the difference between closing time and opening time. Time spent on sessions of less than 1 second was discarded as it was likely to be wrong presses.

The total self-reported time spent on the smartphone was computed as five times school-day usage plus two times holiday usage. As all participants were full-time students who spent an average of 6–8 hours per day on lessons where smartphone usage was prohibited, the upper limit of smartphone usage would be around 12 hours per day and all self-reported usage of 84 hours or more per week (i.e., 12 hours per day) were regarded as outliers. As we believed that the participants reporting these non-feasible usage amount indeed were reflecting that they were having a large volume of usage, we decided not to remove these outliers and replaced these values by a feasible usage time of 84 hours.

Statistical analysis

Spearman correlation was used to assess the association between self-reported and objectively-measured smartphone usage. Wilcoxon signed-rank test was used to compare the mean difference between self-reported and objectively-measured smartphone usage. Bland-Altman plots were used to visualize the agreement between self-reported and objectively-measured smartphone usage.

RESULTS

Table 1 shows the description of the 187 participants. The mean age (SD) of the sample was 19.4 (2.7). Most of them were females (71.7%) and were from the two tertiary institutions (82.9%). Nearly all of them (90.9%) only own one smartphone and more than half of them (55.1%) own at least one tablet.

Table 1.

Descriptive statistics (N=187)

	Mean	SD
Age	19.4	2.7
	Frequency	Percentage
School
Secondary school	32	17.1
Tertiary institutions	155	82.9
Gender
Male	53	28.3
Female	134	71.7
Number of smartphone owned
1	170	90.9
2	16	8.6
≥3	1	0.5
Number of smartphone shared with family members or friends
0	161	86.1
1	23	23.3
≥2	3	1.6
Number of tablet owned
0	84	44.9
1	97	51.9
≥2	6	3.2
Number of tablet shared with family members or friends
0	135	72.2
1	47	25.1
≥2	5	2.7

Figure 1 shows the scatter plots and the correlations between self-reported and objectively-measured smartphone usage. Although the correlation between self-reported and objectively-measured total smartphone usage time was insignificant (ρ=0.10, p=0.18), in three out of the four usage domains, namely social network (ρ=0.21, p=0.005), instant messaging (ρ=0.27, p<0.001), and games (ρ=0.64, p<0.001), the correlations were positively significant.

Figure 1.

Scatter plots between self-reported and objectively-measured smartphone usage.

Table 2 shows the comparison of self-reported and objectively-measured time spent on smartphone usage. The participants over-reported the total time spent on smartphone for around 760 min per week (self-reported 1,930.3 min/wk vs. objectively-measured 1,170.7 min/wk, p<0.001), or an over-report of 64.9%. Most of the over-reporting was contributed by the web browsing domain (self-reported 447.8 min/wk vs. objectively-measured 33.3 min/wk, p<0.001). Note that the self-reported time spent on the four domains (2,124.5 min/wk) did not add up to the self-reported total time spent (1,930.3 min/wk), as the participants may own more than one smartphone.

Table 2.

Mean comparison between self-reported and objectivelymeasured smartphone usage (N=187)

Domain	Self-reported, minute per week (SD)	Objectivelymeasured, minute per week (SD)	Wilcoxon signed-rank test Z-value
Social network	502.6 (472.6)	155.4 (240.4)	9.12
Instant messaging	842.3 (881.8)	521.0 (521.6)	4.87
Web browsing	447.8 (515.9)	33.3 (132.4)	10.65
Games and comics	331.8 (554.5)	199.6 (377.0)	4.69
Total	1930.3 (970.5)	1170.7 (763.7)	7.02

All comparisons were significant at 0.1% level

Figure 2 shows the Bland-Altman plots of the agreement between self-reported and objectively-measured smartphone usage, with the latter as the reference. For the total minute spent on smartphone per week, the mean difference and its 95% limit of the agreement were 760 and (-1,774, 3,293), respectively, indicating a large difference between the self-reported and objective-measured time spent. In addition, the variation of the differences increased with the mean measurement. For the agreement in domain-specific usages, results were similar to that of the total minute spent, with wide 95% limits of agreement.

Figure 2.

Bland-Altman plots of the agreement between self-reported and objectively-measured smartphone usage.

DISCUSSION

Our results showed that self-reported smartphone usage had large discrepancies with objectively-measured smartphone usage. Participants tended to have over-reported the time spent on smartphone, and self-reports were weakly correlated with objective data except for time spent on games. Our results agreed with a validation study that self-reported phone usage (including calls and messaging) over-reported objectively-measured phone usage and they were weakly correlated [18]. Therefore, researchers should be cautious when using self-reported smartphone usage, except for those aiming at identifying the correlates of time spent on smartphone games. For measuring smartphone usage on the social network, instant messaging, and web browsing, objective tracking is suggested. While finegrained usage tracking requires installation of tracking apps, for overall usage statistics, researchers can now use recently released usage tracking apps by Google and Apple such as Digital Wellbeing and Screen Time, respectively.

Many of our results did not agree with a similar study conducted among Korean college students [19]. In that study, their participants under-reported their time spent on smartphone usage by about 20%, while in the current study, our participants over-reported their time spent on smartphone usage by about 65%. There are many reasons behind such a discrepancy, one of the possible explanation was the difference in smartphone usage pattern across the two regions. According to objective data, Korean college students on average spent 4–5 hours per day on smartphone [9,19], and around 30% (80 minutes) were on instant messaging. On the other hand participants in our sample spent about 3 hours on smartphone, and around 45% (75 minutes) were on instant messaging. The different time-spent patterns, in particular the higher portion of time spent on instant messaging for our participants, led to lower criterion validity in our sample. The reason will be explained in details below.

The participants had better accuracy in recalling time spent on games than time spent on other types of usage, and this finding can be explained by the nature of smartphone usage. Our objectively-measured data showed that our participants on average used social network apps 75.7 times and instant messaging app 363.7 times a day, while they only used games apps 45.6 times a day. In addition, the average usage duration for social network apps, instant messaging app, and games app were 2.3, 1.7, and 4.9 min, respectively. The frequent and short usage nature of the social network and instant messaging apps made the recalling of time spent on them challenging. In some extreme cases, participants used their instant messaging app for >1,000 times per day and each usage lasted for only a few seconds. We can expect that recalling the total time spent was impossible. On the other hand, the usage of games apps was either last for minutes (and sometimes hours) or with only a few seconds (that the users launched the games apps to collect rewards), and the usage of games apps was infrequent. These properties made the recalling of time spent on games apps possible.

Our previous study showed a positive correlation between objectively-measured smartphone usage and smartphone addiction [9], and a similar result was also demonstrated in our sample that objectively-measured smartphone usage and smartphone addiction measured by Smartphone Addiction ScaleShort Version [10] was positively-correlated (ρ=0.30, p<0.001). However, the correlation became insignificant when objectively-measured smartphone usage was replaced by self-report level (ρ=0.09, p=0.22). These findings also reflect the invalidity of self-reported smartphone usage level.

The major limitation of this study lied on the sampling and data collection. Only Android users were recruited in this study and our results could not and should not be generalized to users of other smartphones. This also explained the small number of adolescent participants in our sample; most students in our recruited secondary schools were using an iPhone which is not supported by our tracking app. Our sample consisted of more females as the male-to-female ratio was approximately 1:3 in one university in our study (Education University of Hong Kong, https://www.eduhk.hk/sao/?p=221), and therefore males were under-represented. In our sample, about 10% of the participants owned more than one smartphone and more than half of them owned at least one tablet. It was possible that the participants reported the usage on all devices due to recall bias, but we could not capture the usage on the devices other than the smartphone with our tracking app installed. Also, we assumed that all smartphone usage recorded by the tracking app belonged to the participants, but it was possible that another person was using the smartphone. Another limitation was that our smartphone tracking app could not record the usage of Android default system apps. We suspected that this limitation could partially explain the large discrepancy between self-reported and objectively-measured time spent on web browsing, as the usage of Google Chrome, the default web browser of the Android system, could not be recorded. This speculation was evident by our data that 112 (59.9%) participants logged zero seconds of web browsing by our tracking app. Therefore, future studies should also incorporate the usage time of such system-default applications. One minor limitation was that games and comics apps were collapsed together and inferences for games alone could not be made.

To conclude, one self-reported time spent on smartphone games apps was correlated with objective data. A valid and reliable smartphone usage questionnaire should be developed in the future for large-scale epidemiological studies.