Cross-cultural validation of the Social Media Disorder scale.

BACKGROUND: With the popularity of social networking sites, there is an urgency to devise instruments to evaluate social media addiction in different cultural context. This paper evaluates the psychometric properties and validation of the Social Media Disorder (SMD) scale in People's Republic of China.
METHODS: A total of 903 Chinese university students were recruited to participate in this cross-sectional study. The internal consistency, criterion validity and construct validity of the SMD scale were examined.
RESULTS: The results suggested that the 9-item SMD scale had good psychometric properties. Its internal consistency was good, with a Cronbach's alpha of 0.753. The results showed weak and moderate correlations with other validation constructs, such as self-efficacy and other disorder symptoms suggested in the original scale. The Chinese version of SMD demonstrated a good model fit for a two-factor structure in confirmatory factor analysis, with χ2 (44.085)/26=1.700, SRMR=0.059, CFI=0.995, TLI=0.993 and RMSEA=0.028.
CONCLUSION: The SMD scale is suitable for researchers and practitioners to measure problematic usage of social media in a different context, especially on the Chinese population.

Introduction

Since the 1990s, increasingly widespread usage of the Internet and smartphones has led people to spend less time interacting with others in the non-virtual world.1 Internet usage and social networking sites (SNSs) have become increasingly popular in the digital world. The global Internet usage rate is 45.8%, and high-income countries have usage as high as 81.7%.2 Conceptualizing the information communication technology (ICT) addictive behaviors and developing relevant measurements are challenging, and researchers have had difficulty reaching consensus for two major reasons. First, Internet functions and applications have developed rapidly over the past two decades. The Internet no longer serves primarily to connect terminals, as in its initial stage: it now includes many elements that have fundamentally changed users’ lifestyles, such as online media, discussion forums, social networking, business-related activities (including business-to-business, business-to-consumer and peer-to-peer platforms), instant communication, live streaming and cloud-related applications. Second, the number of devices and media that can access Internet-related functions has increased dramatically. Desktop computers are no longer the only devices for Internet connections: smartphones and portal devices have drastically increased the mobility of Internet-related applications, further integrating them into everyday life.

Previous ICT-related addictions research mainly focus on mobile phone, Internet and gaming addictions. There were attempts to develop a scale to study addiction proneness among adolescents on mobile phone usage.3–7 Internet addiction (IA) test has been developed by Young8 to detect Internet dependency. Also, Internet gaming disorder symptoms were listed in the Diagnostic and Statistical Manual of Mental Disorders (DSM-5).9

With the upsurge of Internet usage, there has been controversy about the impact of the use of smartphones and SNS applications. Excessive smartphone usage may lead to depression and anxiety.10 Li, Lau, Mo, Su, Tang, Qin, Gross11 found that IA and problematic usage of social networking may be associated with a higher likelihood of depression and insomnia. People with more SNS use also have a higher tendency for other types of addiction.12 Lee et al13 reported that smartphone and SNS use was associated with higher stress and social anxiety and lower self-control. In recent years, with the help of big data and data mining technology, online social media such as Facebook have been used to screen for major depressive disorders among university students.14 In contrast, one study suggested that there is no significant relationship between social media use and depression.15 A clinical report also suggested that Facebook usage may provide positive interaction and social improvement for psychiatric patients.16 Other positive impacts of smartphone and SNS use have also been found.17–19

With popularity of SNSs, there is an urgency to study this phenomenon. Scales have also been used to specifically address problematic SNS use. Caplan20 developed the Problematic Facebook Use Scale, which was recently validated among Italian adolescents and young adults.21 Bergen Facebook Addiction Scale was developed to examine addition, including salience, mood modification, tolerance, withdrawal, conflict and relapse, based on six items using a 5-point Likert scale.22,23 The scale was validated in Italy, Hungary and Hong Kong with a focus on Facebook, Twitter and Instagram.24–26

However, the above scales mainly focused on SNSs that are only available in Western societies. Facebook, Twitter, Instagram, YouTube and Google-related services are inaccessible or extremely limited by the Great Firewall of China for the sake of maintaining social order and protecting national security.27 SNSs are in high demand in mainland People's Republic of China, and this demand has been met by People's Republic of China-based IT conglomerates such as Alibaba Group Holding Ltd. and Tencent Holdings Ltd., which developed applications with Chinese characteristics such as WeChat and Weibo. Recently, a study validated the Social Networking Activity Intensity Scale among junior school students in People's Republic of China,28 but few studies have focused on Internet-related addiction, particularly social media addictive behavior, in mainland People's Republic of China. The latest survey report of the China Internet Network Information Center29 showed that the Internet penetration rate in People's Republic of China reached 55.8% in 2017, 753 million people, of whom 97.5% accessed the Internet via mobile phones. The majority of Chinese Internet users (73%) are aged 10–39, and over 85% use social networking applications. However, government authorities have restricted the use of particular types of SNSs and Internet services. For example, Facebook, Instagram and Twitter cannot be accessed within People's Republic of China. Chinese versions of these restricted applications have been developed, such as WeChat, RenRen and Weibo. With the popularity of SNSs in People's Republic of China, the major stakeholders, including policy analysts, academics and medical practitioners, have called for a validated Chinese version of an instrument with which to study social media usage.

Aim

A promising scale was recently developed in Europe based on a large national sample in the Netherlands to measure SNS-related disorders. Social Media Disorder (SMD) scale is based on DSM criteria and includes nine criteria: preoccupation, tolerance, withdrawal, displacement, escape, problems, deception, displace and conflict. The current short 9-item scale with proposed cutoff criteria for scholars to measure SNSs-related disorders symptoms.30 Hence, the SMD scale is more comprehensive than the existing scales and fits the context of People's Republic of China better. The 9-item SMD was validated by the scale developers with over 2000 Dutch adolescents and further validated among Turkish adolescents.31,32 The purpose of this paper is to validate and evaluate the psychometric properties of this scale in People's Republic of China.

Methods

Participants and measures

The participants of this cross-sectional study were 903 college students from a tertiary institution located in Guangdong, southern China: 111 were male and 792 were female, with an average age of 20.56 years (SD=2.753; range from 18 to 23). The recruited sample matches the gender ratio of the official school record.

According to van den Eijnden et al,30 the SMD scale comprised with nine items with dichotomous answers (No=0; Yes=1). The scale was translated into Simplified Chinese by two translators with proficiency in both Chinese and English, who used standard translation and back-translation procedures.33 To ensure that the scale is suitable for the context of People’s Republic of China, the social media defined as the most popular SNSs applications, such as WeChat, Sina Weibo, Tencent Weibo, Baidu Tieba, Zhihu, Momo, Meituan-Dianping, Douban, Meipai, Tianya and Douyin. Also, to avoid the translated versions with any potential cultural biases within the context of People’s Republic of China,34,35 two pilot studies were conducted in Guangzhou and Xi’an to ensure that native Chinese speakers could fully understand the translated questions. Each pilot study involved five university students with major ranging from arts, social sciences, business, education and sport sciences. In the debriefing sessions, the pilot participants reported that they could fully understand the meaning of the questions. The data from the pilot studies were not included in the analysis.

Procedure

The questionnaire was circulated on the university’s student intranet system from June to July 2018. To replicate the data collection method of the original scale developers, the respondents were invited to use their unique student accounts to complete the survey online.30 The system did not allow multiple submissions and the completed data stored on the system were completely anonymous. The study was also strictly adherence to the ethical standards with informed consent from all of the respondents. Each respondent spent around 10 mins to complete this self-administered questionnaire.

Several psychometric testing tools were used to analyze the data. Confirmatory factor analysis (CFA) with diagonally weighted least squares (DWLS) estimator was used to examine the factor structure of SMD.36,37 The DWLS is considered as less biased and more optimal fit for scale with items estimated by different categories ranging from two to seven.38–40 The cutoff criteria for good model fit were evaluated as follows. Both the comparative fit index (CFI) and the Tucker–Lewis index (TLI) were over 0.950, the standardized root mean square residual (SRMR) was under 0.08 and the root mean square error of approximation (RMSEA) was under 0.06.41–44 Due to the effect of large sample size, an acceptable model should be indicated by χ2/df≤3 on top of the above criteria.45–48 The internal consistency of the SMD scale was assessed by Cronbach’s alpha.49 An evaluation on all of the nine items corrected item-total correlation.44 Criterion validity was assessed with other addiction-related scales reported in the literature. The SMD scale and other IA-related scales generally had weak negative correlations with self-esteem and moderate positive correlations with depression, loneliness and impulsiveness.30–32,50 Hence, the SMD scale was analyzed with the following well-established scales, including the 12-item General Health Questionnaire (GHQ-12),51,52 the Short Warwick-Edinburgh Mental Well-being Scale (SWEMWBS),53–55 the Satisfaction with Life Scale (SWLS),56–59 the Subjective Happiness Scale (SHS),60 the Personal Wellbeing Index (PWI),61 the Rosenberg Self-esteem Scale (RSE)62–64 and the Brief Self-Control Scale (BSCS).65,66 The above analyses were conducted with IBM SPSS 25.0 and R computing environment with lavaan package version 3.5.2.67

Results

Confirmatory factor analysis

The results of the CFA of the 9-item SMD scale are shown in Table 1. Model 1 evaluated the 9-item SMD with single-factor structure. Yet, the results suggested that the scale did not fit the model well, with χ2 (181.044)/27=6.705, p<0.001, SRMR=0.109, CFI=0.957, TLI=0.943 and RMSEA=0.080. Model 2 still evaluated the SMD with one latent factor structure and based on the modification indices to separating out the covariance between the errors. The results indicate adequate model fit, with χ2 (70.974)/25=2.839, p<0.001, SRMR=0.072, CFI=0.987, TLI=0.982 and RMSEA=0.045.

Table 1

Confirmatory factor analysis of the 9-item SMD scale with different factor structure

Model	χ2	df	χ2/df	RMSEA [90% CI]	CFI	TLI	SRMR
1	181.044***	27	6.705	0.080 [0.069–0.091]	0.957	0.943	0.109
2a	70.974***	25	2.839	0.045 [0.033–0.058]	0.987	0.982	0.072
3	44.085*	26	1.700	0.028 [0.012–0.042]	0.995	0.993	0.059

Notes: aIncludes covariance between the error terms for items SMD6 and SMD7, and SMD7 and SMD9. *p<0.05, ***p<0.001.

Abbreviations: RMSEA, root mean square error of approximation; CFI, comparative fit index; TLI, Tucker–Lewis index; SRMR, standardized root mean square residual.

Model 3 provided the CFA analysis of the SMD with two-factor construct. The first latent factor with items 1, 2, 3, 4, 5 and 8; the second one with items 6, 7 and 9. The results indicated good model fit, as χ2 (44.085)/26=1.700, p=0.015, SRMR=0.059, CFI=0.995, TLI=0.993 and RMSEA=0.028. The final standardized factor structure of Model 3 is presented in Figure 1. Overall, the results suggested that the 9-item SMD had a generally good fit for two underlying factor structure.

Figure 1

Final standardized model for the 9-item SMD scale (Model 3).

Internal consistency

Table 2 shows the means, standard deviations, skewness, kurtosis, corrected item-total correlations and Cronbach’s alpha values if items were deleted for the 9-item SMD scale. The mean score of the 9-item SMD scale was 4.124 (SD=2.353; N=903), and the corrected item-to-total correlations of the scale ranged from 0.301 to 0.513, which are within the desired levels. The Cronbach’s alpha for the 9-item SMD scale (0.753) is above the acceptable range. No significant differences and relationship were observed in the scale scores on sex and age of the respondents, according to the independent-sample t-test and correlation results.

Table 2

Descriptive statistics for the 9-item SMD scale

Item	Criterion	Mean	SD	Skewness	Kurtosis	rit	αiid
SMD1	Preoccupation	0.83	0.371	−1.808	1.272	0.301	0.748
SMD2	Tolerance	0.45	0.498	0.185	−1.970	0.452	0.726
SMD3	Withdrawal	0.57	0.496	−0.266	−1.933	0.452	0.726
SMD4	Persistence	0.53	0.500	−0.104	−1.994	0.513	0.715
SMD5	Displacement	0.55	0.497	−0.216	−1.957	0.473	0.722
SMD6	Problem	0.15	0.360	1.933	1.740	0.412	0.734
SMD7	Deception	0.18	0.385	1.664	0.771	0.451	0.728
SMD8	Escape	0.61	0.489	−0.434	−1.816	0.368	0.741
SMD9	Conflict	0.25	0.433	1.155	−0.668	0.459	0.725

Notes: rit=corrected item-total correlations; αiid=Cronbach’s alpha, if item deleted.

Abbreviation: SMD, Social Media Disorder.

Criterion validity

The Chinese version of the SMD scale demonstrated good criterion validity with other construct-related measures. Table 3 shows weak and moderate correlations between the SMD scale with psychological disorder symptoms and well-being-related measures, including GHQ-12 (r=0.273, p<0.001), SWEMWBS (r=−0.170, p<0.001), SWLS (r=−0.150, p<0.001) and PWI (r=−0.213, p<0.001). The scale also holds significant moderate negative correlation with happiness (SHS; r=−0.209, p<0.001), self-esteem (RSE; r=−0.278, p<0.001) and self-control (BSCS; r=−0.417, p<0.001).

Table 3

Correlations between the 9-item SMD and other related scales

Construal-related scales	9-item SMD
12-item General Health Questionnaire (GHQ-12)	0.273***
Short Warwick-Edinburgh Mental Well-being Scale (SWEMWBS)	−0.170***
Satisfaction with Life Scale (SWLS)	−0.150***
Subjective Happiness Scale (SHS)	−0.209***
Personal Wellbeing Index (PWI)	−0.213***
Rosenberg Self-esteem Scale (RSE)	−0.278***
Brief Self-Control Scale (BSCS)	−0.417***

Note: ***p<0.001.

Abbreviation: Abbreviation: SMD, Social Media Disorder.

Discussion

This study has been validated the Chinese version of the SMD scale. The scale has demonstrated good psychometric properties. The results for internal consistency, criterion validity and construct validity were generally in line with those of prior studies. The CFA results of this study, especially Models 2 and 3 also reported good model fit and possess similar findings in various goodness-of-fit indices, such as χ2/df, RMSEA and CFI.30 The Cronbach’s alpha in this study (0.75) replicated the findings (in the range of 0.75–0.82) of the original scale developers and related studies.30,31 With regarded to the criterion validity, this study replicated the findings of social media usage with low self-esteem (RSE)50 and self-control (BSCS).13 The results also suggested that SMD scale is associated with the psychological disorder symptoms as reported in the literature. In particular, GHQ-12 (r=0.273) is used for screening minor psychiatric disorders, the higher the score signifies more severe the condition.52 In contrast, the interpretation of SWEMWBS score (r=−0.170) is the other way round, lower score means lower mental well-being.55 Hence, the problematic usage of social media has an adverse impact not only to the mental well-being of individual, but also their quality of life.30–32

The data of this study suggested that the 9-item SMD with two latent factor structure rather than unidimensional as advocated by the scale proposers.30 The first factor with items 1, 2, 3, 4, 5 and 8 (α=0.70). The three items comprised the second factors (α=0.73) are item 6 (regularly had arguments with others because of your social media use?), item 7 (regularly lied to your parents or friends about the amount of time you spend on social media?) and item 9 (had serious conflict with your parents, brother(s) or sister(s) because of your social media use?) – were related to how the respondents interacted with their parents, siblings and others. These three items have factor loadings ranging from 0.829 to 0.915. The majority of respondents answered “No” for item 6 (x̅=0.15), item 7 (x̅=0.18) and item 9 (x̅=0.25), and these mean scores are significantly different from those of other items, which ranged from 0.45 to 0.83. This compelling finding is mainly due to two factors that are commonly found among the university students in People’s Republic of China. The first is related to the impact of the one-child policy, which limited the likelihood of the respondents having siblings to interact with. The latest official census statistics showed that the average family size in People’s Republic of China decreased from 5.3 in the 1950s to merely 3.02 in 2012.68 Second, People’s Republic of China’s university system requires nearly all students to live in on-campus dormitories, a rather self-contained and self-sufficient setting. The students were thus not living with their parents.69 Nevertheless, this paper suggests keeping items 6, 7 and 9 in the scale. The 9-item SMD scale still possesses good psychometric properties and validity, as indicated by the results. More data from different Chinese samples are needed in future studies to further reconfirm the applicability of the SMD scale.

Given to the following grounds for correlating the error terms in Model 2 analysis, Model 3 with two-factor structure is more preferred. SMD scale items 6, 7 and 9 are related to problem, deception and conflict criterion, respectively, by asking the respondents whether they had any problems with their parents, siblings and peers. As discussed above, the sample of this study is university students in People’s Republic of China, who are subject to the influence of the government and educational policies. The result of the CFA analysis in Model 2 with post hoc modification generally replicates the findings of prior studies.30,31 However, according to van den Eijnden et al30

researchers sometimes correlate error terms... to improve model fit, this should be avoided if possible, since it means that there is some other issue that is not specified within the model that is causing the covariation. (p. 481)

Hence, Model 3 is more appropriate as the CFA results with better model fit than Model 2. In short, the results for the 9-item SMD scale show that the model generally has a good fit for construct validity with two-factor structure in the context of People’s Republic of China.

Another contribution of this study demonstrated that the SMD scale was appropriate for use on the Chinese population. With the rapid development of SNSs and related applications in People’s Republic of China in the past decade, it is necessary to validate a Chinese instrument to further study its impact. According to social cognitive theory, reciprocal causation is at work as positive outcomes and rewards reinforce the human behavior.12,70,71 SNS applications in People’s Republic of China include many positive outcomes and reward services, including food ordering, delivery, car hiring, broadcasting, e-business and e-commerce that significantly influence people’s lifestyles and relationships. As reviewed in the literature, SNSs are a double-edged sword. The increasing usage of SNSs in People’s Republic of China raise new research questions regarding its impacts on social trust, well-being and the quality of life of the netizen population; these topics are still largely understudied due to the lack of validated scales.

This study may have the following potential limitations. First, respondents were recruited from one Chinese university only, it may limit the generalizability of the findings to other context or age group. However, the relatively large sample size in this study may compensate this limitation. Second, this study only collected limited measurements to evaluate the criterion validity of SMD. Due to the availability of validated Chinese-translated scales related to problematic usage of social media and concerns of lengthy questionnaire that may discourage participation, this study adopted other reliable construct-related scales, such as the GHQ-12, SWEMWBS, SWLS, SHS, PWI, RSE and BSCS that are commonly used or discussed in social media addiction validation studies and the literature on ICT-related addiction.30,31,50 Third, one may have concerns about the use of dichotomous answers in the SMD scale. However, this study only validated the Chinese version of the scale without altering its original meaning and format. Further study may attempt to compare the scale with various number of categories, ranging from two, five and seven. Lastly, this study only used the traditional approach to evaluate the psychometric properties of the scale. There is an emerging clinically based validation procedure to evaluate the scale with clinimetric approach.72–75 Future studies may consider using this innovative approach to analyze the clinical validity of the SMD scale.

Conclusion

To conclude, the findings show that the 9-item SMD is applicable to Chinese college populations. The scale possesses good psychometric properties in internal consistency, criterion validity and construct validity. As numerous studies have suggested that Internet-related addictions are common in the world. It is urgent to derive a reliable and validated problematic usage of SNS instrument to study this growing and rapidly changing phenomenon. However, current measurements are largely based on English and non-Chinese SNS applications that limit their application in the context of People’s Republic of China. Hence, validating a Simplified Chinese (the written language of China) version of the SMD scale can enable scholars and medical practitioners to more accurately study the Chinese speaking population.