Determination the cut-off point for the Bergen social media addiction (BSMAS): Diagnostic contribution of the six criteria of the components model of addiction for social media disorder.

OBJECTIVE: Social media disorder (SMD) is an increasing problem, especially in adolescents. The lack of a consensual classification for SMD hinders the further development of the research field. The six components of Griffiths' biopsychosocial model of addiction have been the most widely used criteria to assess and diagnosis SMD. The Bergen social media addiction scale (BSMAS) based on Griffiths' six criteria is a widely used instrument to assess the symptoms and prevalence of SMD in populations. This study aims to: (1) determine the optimal cut-off point for the BSMAS to identify SMD among Chinese adolescents, and (2) evaluate the contribution of specific criteria to the diagnosis of SMD.
METHOD: Structured diagnostic interviews in a clinical sample (n = 252) were performed to determine the optimal clinical cut-off point for the BSMAS. The BSMAS was further used to investigate SMD in a community sample of 21,375 adolescents.
RESULTS: The BSMAS score of 24 was determined as the best cut-off score based on the gold standards of clinical diagnosis. The estimated 12-month prevalence of SMD among Chinese adolescents was 3.5%. According to conditional inference trees analysis, the criteria "mood modification", "conflict", "withdrawal", and "relapse" showed the higher predictive power for SMD diagnosis.
CONCLUSIONS: Results suggest that a BSMAS score of 24 is the optimal clinical cut-off score for future research that measure SMD and its impact on health among adolescents. Furthermore, criteria of "mood modification", "conflict", "withdrawal", and "relapse" are the most relevant to the diagnosis of SMA in Chinese adolescents.

Introduction

Social media defined as online applications which allow the interaction with others, maintenance of relationships, formation interest groups, and development of individual’s presence (Kietzmann, Hermkens, McCarthy, & Silvestre, 2011). Due to the availability of mobile devices, the use of social media has become widespread in people’s daily lives. According to a report of the China Internet Network Information Center (CINIC, 2020), there are 0.904 billion social media users in China. Although social media may provide new forms of social connection and broaden relationships (Baker & Moore, 2008), its use can have adverse effects on the physical and mental health, family life, and social life of individuals, especially adolescents and young people (Frost & Rickwood, 2017; Pantic, 2014; van den Eijnden, Koning, Doornwaard, van Gurp, & Ter Bogt, 2018). Many studies have demonstrated that using social media problematically may be related to psychopathology (e.g., anxiety, depression, self-injurious behaviour, suicide risk and suicidal ideation), personality aspects (e.g., low self-esteem, and high impulsivity), and academic (e.g., low school connectedness, and poor academic performance) (Bányai et al., 2017; Espinoza & Juvonen, 2011; Keles, Mccrae, & Grealish, 2020; Sampasa-Kanyinga, Chaput, & Hamilton, 2019; Shafi et al., 2019; Shensa et al., 2017).

Therefore, an increasing number of scholars have suggested that problematic social media use should be viewed as a behavioural addiction (Andreassen, Pallesen, & Griffiths, 2017; Den Eijnden, Lemmens, & Valkenburg, 2016; Ryan, Chester, Reece, & Xenos, 2014). The prevalence rates of social media disorder (SMD) among adults range from 1.6 to 47% (Alabi, 2013; Andreassen et al., 2017; Jafarkarimi, Tze, Sim, & Hee, 2016), and among adolescents, the rate is 4.5% (Bányai et al., 2017). The lack of a consensual diagnosis and classification for SMD is the key factor leading to these high ranges of prevalence rates. Hence, to correctly distinguish between disordered and non-disordered social media users and accurately estimate the prevalence of SMD, it is vital to determination and validation of an optimal cut-off score of a maturity scale, and to develop a set of diagnostic criteria based on a solid set of existing diagnoses of behavioural addiction.

According to the component model of addiction proposed by Griffiths, SMD comprises a core set of criteria (Mark D Griffiths, 2005). Specifically, these criteria are salience (i.e., social media use becomes the single most important activity), tolerance (i.e., requiring increased amounts of time spent on social media use to achieve the former effects), mood modification (i.e., using social media as a coping strategy to deal with mood problems), relapse (i.e., loss of control of social media use and repeated reversions to problematic use), withdrawal symptoms (i.e., unpleasant feelings and physical effects when unable to use social media), and conflict (i.e., conflicts with other persons, other activities, and within the individual themselves caused by social media use). Based on Griffiths’ six criteria, the DSM-5 proposed nine diagnostic criteria for internet game disorder (IGD) (Association, 2013). Specifically, these criteria are: preoccupation, withdrawal, tolerance, loss of control, give up other activities, continue despite problems, deception, escape, and impaired function (Association, 2013).

The Bergen social media addiction scale (BSMAS) developed by Andreassen and colleagues is based on Griffiths’ six criteria for addiction, and it enjoys widespread usage (Andreassen et al., 2017). Many studies have demonstrated the appropriate psychometric properties of this scale in different cultural contexts (Bányai et al., 2017; Lin, Broström, Nilsen, Griffiths, & Pakpour, 2017), including in Chinese cultures (Leung et al., 2020). A clear cut-off score for the BSMAS plays a key role in distinguishing disordered users from non-disordered users and correctly estimating the prevalence of SMD. However, only one study has suggested an optimal empirical cut-off of 19 points for this scale for use with Hungarian populations (Bányai et al., 2017). There is no research on the cut-off point of the BSMAS in the Chinese context.

Strategies to determine the cut-off scores include the epidemiological approaches and clinical interviews. Clinical diagnosis is believed to be the gold standard for selection of cut-off points. However, they are sometimes unavailable. Thus, epidemiological approaches, such as: latent profile analysis (LPA), have been performed to identify cut-off points (van Smeden, Naaktgeboren, Reitsma, Moons, & de Groot, 2014). The LPA group representing the most severe level of health problem was considered as the “case” to determine the cut-off point of an instrument. As far as we know, there is no research to evaluate the on the sensitivity and specificity of the empirical cut-off point based on LPA.

The present study aimed to (1) examine the clinical cut-off point for the BSMAS, (2) evaluate the diagnostic performance of the empirical cut-off point based on LPA approach, (3) estimate the 12-month prevalence of SMD among Chinese adolescents, (4) estimate rates of endorsement of the Griffiths’ six criteria in a representative community sample of adolescents, (5) evaluate the discriminative validity of Griffiths’ six criteria for diagnosis of SMD.

Methods

Participants and procedure

The clinical sample

The clinical sample (n = 252) was collected from treatment facilities that specialize in internet addiction in Yingtan City, Jiangxi, China, between 1 September 2019 and 15 September 2020. The sample was composed of 252 participants aged 12–18 years (mean = 14.53; SD = 1.21; male = 87.7%) (Table 1). After participants completed the BSMAS, two certified psychiatrists conducted blind structured clinical interviews.

Table 1.

Sample characteristics

	Clinical sample (n = 252) n (%)/M (SD)	Community sample (n = 21,735) n (%)/M (SD)
Age	14.53 (1.21)	16.23 (1.94)
Gender
Male	221 (87.7)	10,321 (47.49)
Female	31 (12.3)	11,414 (52.51)
Middle school stage
Junior middle school	176 (69.8)	10,215 (47.00)
Senior middle school	76 (30.2)	11,520 (53.00)
Social media sites
WeChat	97 (38.5)	6,954 (32.00)
QQ	152 (60.3)	13,895 (63.93)
Sina Weibo	3 (1.2)	395 (1.82)
Others	0 (0.0)	491 (2.26)
Weekly social media use (hours)	26.24 (11.34)	14.87 (15.90)

The community sample

The sample were collected between November 2019 and January 2020 among adolescents in Yingtan city of Jiangxin Province in southern China and Weifang city of Shandong Province in northern China. We adopted the following formula to calculate the sample size (Hu, Tang, Chen, Kaminga, & Xu, 2020). where Z1-α/2 = 1.96 when α = 0.05, p was the prevalence of social media disorder (SMD) among adolescents (which was 4.5% according to a previous study), and d was the admissible error (which was 0.3% here). The sample size was calculated to be 18,366 according to the formula. Considering a non-response rate of 10%, the theoretical sample size was 20,172.

In order to select a representative sample of adolescents, a multistage cluster randomized sampling method was used. School type in China was classified into key middle schools, general middle schools, and vocational middle schools based on academic levels. Thus, in the first stage, 1 key middle school, 3 general schools, and 2 vocational middle schools in Yingtan city and 2 key middle schools, 5 general schools, and 2 vocational schools in Weifang city were randomly selected. Then, we randomly selected 400 classes from these schools, and invited all students from these classes to participate in the study.

Informed consent was required from all students and their parents for participation in the study. The students and their parents were assured that they were free to refuse participation, because this is an anonymous study, the researcher, teacher, and principal will not know which student refuse to participate the study. All students and their parents were informed about the purpose, procedures and measurements, potential risks and possible benefits of the study before the survey.

The students completed the questionnaire in a classroom under the supervision of the interviewer. Of the 23,549 students approached, 1,343 declined to participate. On the survey day, 255 students were absent because of truancy or illness. Among them, 198 completed the survey at home under the supervision of the interviewers via video, and 57 refused. The valid data set included 22,149 students, and the overall response rate was 94.05%. Among the participants, 414 (1.87%) adolescents did not complete all items, and their data were excluded from the analysis because of missing BSMAS values, age, and/or other variables.

Ultimately, the total sample comprised 21,735 adolescents. More than half of the sample was female (52.5%). Age ranged between 12 and 19 years (average = 16.23 years; standard deviation = 1.94). Sample characteristics are shown in Table 1.

Measures

Sociodemographics and social-media-use-related behaviours

Information regarding gender, age, and middle school stage was collected. The social media sites variable was used to evaluate the sites most used by respondents for social purposes (i.e., 1 = WeChat, 2 = QQ, 3 = Sina Weibo, 4 = others).

SMD

The Chinese version of the BSMAS was administered to assess SMD within a 12-month period (Leung et al., 2020); the scale covers all six criteria for addiction proposed by Griffiths. Each criterion was reflected by an item, resulting in a 6-item scale. Students were asked to rate each item on a 5-point Likert scale, ranging from “1 =very rarely” to “5 = very often”. If the item was rated as “very often”, then the criterion was considered endorsed for the analyses presented in this paper (Halley M. Pontes & Griffiths, 2015). Cronbach’s alpha was 0.91 in this study.

Clinical criteria for SMD

A structured clinical interview schedule was developed based on nine DSM-5 criteria for IGD. According to Griffiths’ international recommendations (M. D. Griffiths et al., 2016), a set of questions were asked to examine these nine criteria, such as “Do you feel restless, irritable, moody, angry, anxious, or sad when attempting to cut down or stop using social media or when you are unable to use it (i.e., withdrawal)?”. According to the DSM-5 criteria for IGD, if five or more items were endorsed, then the participant was diagnosed with SMD.

Validation variables

Given that impulsivity traits have widely been identified as constituting the hallmarks of behavioural addictions (Brand, Young, Laier, Wölfling, & Potenza, 2016; Wang et al., 2017), we adopted impulsivity as an external criterion to test the clinical cut-off score of the BSMAS and the validity of the LPA classification. Self-esteem were also adopted as external criteria because impaired self-esteem has long been identified in addictive states (Bányai et al., 2017). Specifically, the Brief Barratt Impulsiveness Scale (BBIS) (Luo, Chen, Ouyang, & Xiao, 2020; Morean et al., 2014), and Rosenberg’s Self-Esteem Scale (RSES) (Ji & Yu, 1993; Rosenberg, 1965) were applied. In this study, Cronbach’s alpha was 0.86 for the BBIS, and 0.83 for the RSES. Self-reported academic performance was adopted as an index of impairment and assessed by asking participants to define their academic performance as “1 = very bad, 2 = bad, 3 = medium, 4 = good, or 5 = very good” according to their grades from the last school report card in the total score for all subjects. Weekly social media use was chosen to assess participants’ time invested in using social media per week on smartphones and/or other devices. Weekly social media use was calculated as (daily use time on a school day × 5) + (daily use time on a weekend day × 2).

Statistical methods

Receiver operating characteristic (ROC) curve analysis was adopted to determine the optimal clinical cut-off point based on the gold standards for clinical diagnosis. The diagnostic efficacy of the BSMAS was assessed through the area under the ROC curve (AUC). The sensitivity analysis was adopted to determine the optimal clinical cut-off point of the BSMAS. Sensitivity (the proportion of true positives), specificity (the proportion of true negatives), positive prediction rate (PPR; the proportion of correctly diagnosed positive cases), negative prediction rate (NPR; the proportion of correctly diagnosed negative cases), and the diagnostic accuracy (the proportion of true positive + true negatives) were calculated.

The validity of this clinical cut-off score was confirmed in the community sample by comparing external criteria (e.g., the BBIS and RSES) and impairment (e.g., academic performance) between SMD and non-SMD groups.

Latent profile analysis (LPA) was conducted in the community sample to identify the groups of adolescents with higher risk of SMD in Mplus 7.4 (Collins & Lanza, 2013). Models with 2–6 latent profiles were estimated based on the scores of the six items of the BSMAS. The Akaike information criterion (AIC), Bayesian information criterion (BIC), sample-size-adjusted Bayesian information criterion (SSABIC), and entropy of each model were examined. Finally, the Lo-Mendell-Rubin adjusted likelihood ratio test (LMRT) was used to determine the best class solution.

The sensitivity analysis was performed to determine the optimal empirical cut-off point of the BSMAS, and the group of “disordered social media users” according to the LPA was considered as the “gold standard”. The validity of the LPA identification of “disordered social media users” was assessed by comparing external criteria (e.g., the BBIS and RSES), impairment (e.g., academic performance), and relevant variables of SMD (e.g., weekly social media use) between the LPA classes. Considering the probabilistic nature of the LPA classes, the Wald’s Chi-square test of mean equality for latent class predictors in mixture modelling was used for these comparisons (see www.statmodel.com/download/meantest2.pdf).

Using sensitivity analysis, the diagnostic performance of the empirical cut-off point based on LPA was assessed, and the community adolescents diagnosed with SMD according to the clinical cut-off score was used as the “gold standard”.

To explore the contributions of specific criteria to the diagnosis of SMA, the non-parametric conditional inference tree (C-Tree) (Hothorn, Hornik, & Zeileis, 2006; Rehbein, Kliem, Baier, Mößle, & Petry, 2015) was performed. The C-Tree analysis can determine the strongest association between any predictor (e.g., six addiction criteria, weekly social media using time, age, and gender) and the response variable (SMD diagnosis, which according to BSMAS cut-off score of 24) via a permutation test framework. The analysis was not pre-registered, and the results should be considered exploratory.

Ethics

Informed consent was obtained from all participants, while parents’ permission was also obtained for those less than 18 years of age. The procedures were carried out in accordance with the Declaration of Helsinki. The ethical approval for this study was also obtained from the ethics committee of Jiangxi Mental Hospital (No. 20190113).

Results

Determination of the clinical cut-off point for the BSMAS

According to structured clinical interviews, 28 of the 252 participants were diagnosed with SMD. This clinically diagnosed group was used as the gold standard. The diagnostic efficiency was demonstrated by the high AUC (0.998, 95% CI: 0.95, 1.00, P < 0.001).

Table 2 shows the sensitivity, specificity, diagnostic accuracy, PPV, and NPV of possible cut-off scores of the BSMAS. At a score of 24, the sensitivity was 96.4%, the specificity was 99.1%, and the Youden Index achieved its maximum value (95.5%). Thus, clinically, only 3.6% of truly addicted social media users were not identified, and fewer than 1% of non-addicted users were considered addicted. Additionally, at this cut-off score, the PPV is 93.1%, and the NPV is 100%. That is, more than 90% of the users diagnosed with SMD were identified correctly, while no users diagnosed without SMD were mistakenly identified. The diagnostic accuracy was 98.8%. At a cut-off score of 19, as recommended by Bányai and colleagues, the PPR was quite low, at 43.7%.

Table 2.

Cut-Off Points for the BSMAS based on the clinical diagnostic interviews (n = 252)

Cut-off points	True positive	True negative	False positive	False negative	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Accuracy (%)	Youden's index (%)
19	28	188	36	0	100	83.9	43.7	100	85.7	83.0
20	28	197	27	0	100	87.9	50.9	100	89.3	87.0
21	28	205	19	0	100	91.5	59.6	100	92.5	91.0
22	28	214	10	0	100	95.5	73.7	100	96.0	95.0
23	27	221	3	1	96.4	98.6	90.0	100	98.4	94.4
24	27	222	2	1	96.4	99.1	93.1	100	98.8	95.5
25	24	223	1	4	85.7	99.6	96.0	98.2	98.0	85.2
26	21	224	0	7	75.0	100	100	97.0	97.2	75.0
27	18	224	0	10	64.3	100	100	95.7	96.0	64.0
28	14	224	0	14	50.0	100	100	94.1	94.4	50.0
29	11	224	0	17	39.3	100	100	92.9	93.2	39.3

Note. BSMAS: Bergen social media addiction scale; specificity (true positive/true positive and false negative), sensitivity (true negative/true negative and false positive); accuracy (true positive and true negative/all); PPR: positive predictive rate (true positive/true positive and false positive); NPR: negative predictive rate (true negative/true negative and false negative); Youden's index: defined as sensitivity + specificity – 1.

Confirmation of the validity of the clinical cut-off point

According to the cut-off score of 24 for diagnosing SMD, the prevalence of SMD estimated among community adolescents, boys, and girls was 3.5, 4.9, and 2.2%, respectively. As shown in Table 3, compared with adolescents without SMD, those with SMD tended to (1) be male, (2) use social media more than 30 hours weekly, (3) have worse academic performance, and (4) have higher impulsivity and lower self-esteem.

Table 3.

Comparison between the SMD and non-SMA groups according to cut-off point of 24 in the BSMAS (n = 21,735)

	SMD (n = 758)	non-SMD (n = 20,977)	P
	n (%)	n (%)
Gender
Male	503 (66.4)	9,818 (46.8)	<0.001*
Female	255 (33.6)	11,159 (53.2)

Note. SMD: social media disorder; BSMAS: Bergen social media addiction scale; BBIS: brief Barratt impulsiveness scale; RSES: Rosenberg's self-esteem scale.

*: P value was obtained by χ2 test;

**: P values were obtained by Mann-Whitney U test.

Latent profile analysis

The results of the LPA in the community sample were shown in Table 4. The AIC, the BIC, and the SSABIC values continued to decrease, as the number of groups increased. However, the best loglikelihood value of the six-group solution was not replicated. The entropy was adequate for two-group solution to five-group solution. Finally, the five-group solution was adopted based on the L-M-R test.

Table 4.

Results obtained from the Latent Profile Analysis

Model	Log-likelihood	Replicated log-likelihood	Nr. Of free parameters	AIC	BIC	SSABIC	Entropy	LMR-LRT test	P
2 classes	−193046.30	YES	19	325965.88	326117.62	326057.24	0.93	5,836.54	<0.001
3 classes	−162963.94	YES	26	312760.48	312968.13	312885.51	0.92	13,032.96	<0.001
4 classes	−156354.24	YES	33	302153.70	302417.26	302312.39	0.88	10,470.99	<0.001
5 classes	−151043.85	YES	40	296247.67	296567.14	296440.21	0.91	5,836.54	<0.001
6 classes	–	NO	–	–	–	–	–	–	–

Note. AIC: Akaike information criterion; BIC: Bayesian information criterion; SSABIC: sample-size-adjusted Bayesian information criterion; LMRT: Lo-Mendell-Rubin adjusted likelihood ratio test.

The features of the five classes are shown in Fig. 1. The first class, named “casual users” (42.9%), and the second class, named “regular users” (21.1%), represent adolescents who generally selected “very rarely” or “rarely” on the BSMAS for all six items. The third class, named “low-risk high-engagement users” (10.2%), and the fourth class, named “at-risk high-engagement users” (21.7%), scored similarly higher on “salience” and “tolerance”, while the third class scored much lower on “mood modification”, “relapse”, “withdrawal”, and “conflict” compared to the fourth class. The fifth class, named “disordered users” (4.2%), represents adolescents who generally rated their social media use as “very often” or “often” on the scale for all six items.

Fig. 1.

The five classes obtained from the latent profile analysis

Determination of the empirical cut-off point for the BSMAS

As shown in Table 5, The adolescents in the “disordered users” group had significantly higher BBIS scores, lower BSES scores, worse academic performance, and longer social media use time than those in the other four groups.

Table 5.

Comparison of the five latent classes: Testing Equality for Latent Class Predictors (n = 21,735)

	Casual users (N = 9,319)	Regular users (N = 4,576)	Low risk high-engagement users (N = 2,222)	At risk high-engagement users (N = 4,711)	Disordered Users (N = 907)	Over test
						Waldx 2	P value
Gender (Male %)	44.81a	45.35b	42.62c	54.15d	63.07e	197.42	<0.001
Age (years), Mean (SE)	15.89 (0.02)a	16.29 (0.03)b	16.63 (0.04)c	16.60 (0.03)c	16.42 (0.06)d	442.99	<0.001
Weekly social media use (min 0.5, max 72, mean 14.87, SD 15.90), Mean (SE)	9.91 (0.14)a	14.33 (0.25)b	18.31 (0.40)c	25.12 (0.27)d	31.41 (0.71)e	701.73	<0.001
Academic performance (min1, max 5, mean 2.93, SD 1.07), Mean (SE)	3.06 (0.01)a	2.86 (0.02)b	2.96 (0.02)c	2.75 (0.02)d	2.58 (0.04)e	263.95	<0.001
BBIS (min1, max 4, mean 2.10, SD 0.46), Mean (SE)	2.01 (0.01)a	2.23 (0.01)b	2.11 (0.01)c	2.34 (0.01)d	2.44 (0.02)e	527.67	<0.001
RSES (min 1, max4, mean 3.00, SD 0.63), Mean (SE)	3.14 (0.01)a	2.89 (0.01)b	3.04 (0.01)c	2.78 (0.01)d	2.71 (0.03)e	358.92	<0.001

Note. Different subscript letters (a, b, c) in the same row reflect significant (P < 0.05) difference between the means while same subscript letters in one row reflect non-significant difference between the means according to pair wised Wald χ2 test of mean equality for latent class predictors in mixture modeling (www.statmodel.com/download/meantest2.pdf).

Based on the sensitivity analysis, a cut-off score of 23 is suggested to be an ideal empirical cut-off to distinguish addicted social media users from non-addicted users. At this point, highest diagnostic accuracy (99.9%) was achieved, with the sensitivity of 99.1%, specificity of 99.9%, PPV of 99.7%, NPV of 99.8%.

Evaluation of the diagnostic performance of the empirical cut-off point

The community adolescents diagnosed with SMD according to the clinical cut-off score of 24 was used as the “gold standard” to evaluate of the diagnostic performance of the empirical cut-off point of 23. According to the sensitivity analysis, diagnostic accuracy of the empirical cut-off point of 23 was 97.8%, with a sensitivity of 100%, specificity of 99.5%, PPV of 87.2%, and NPV of 100%.

Endorsement of Griffiths’ six criteria and predictive power of specific criteria for SMD

As shown in Table 6, “salience” and “tolerance” were the most endorsed criteria among the general adolescents (both at 3.4%). The least endorsed criterion was “mood modification” (1.79%). Then, the endorsement of specific criteria corresponding to diagnosis of SMD was evaluated using Cohen’s kappa coefficients. Comparatively, the criterion “mood modification”, “relapse”, “withdrawal”, and “conflict” corresponded well to the diagnosis (Cohen’s κ > 0.50), while “salience” (Cohen’s κ = 0.44) and “tolerance” (Cohen’s κ = 0.46) corresponded relatively poorly with the overall classification of SMD.

Table 6.

Endorsement of diagnostic criteria overall and for disordered users, and non-disordered users (n = 21,735)

Criterion	Cohen's κ	Overall n (%)	Disordered users (n = 758), n (%)	Non-disordered users (n = 20,977), n (%)	P
Salience	0.44	738 (3.4)	344 (45.4)	394 (1.9)	<0.001*
Tolerance	0.52	743 (3.4)	362 (47.8)	381 (1.8)	<0.001*
Mood modification	0.46	410 (1.9)	316 (41.7)	94 (0.4)	<0.001*
Relapse	0.53	453 (2.1)	327 (43.1)	126 (0.6)	<0.001*
Withdrawal	0.53	550 (2.5)	350 (46.2)	200 (1.0)	<0.001*
Conflict	0.51	470 (2.2)	323 (42.6)	147 (0.7)	<0.001*

Note.*: P value was obtained by χ2 test.

To further explore the contributions of specific criteria to the classification with SMD, a C-Tree was constructed (Fig. 2) in the community sample. The criterion “mood modification” and “withdrawal” showed the higher predictive value for the diagnosis of SMD. Endorsement of both “mood modification” and “withdrawal” was associated with a high probability of classification of SMD, 95.5% (group 6), while denied both these criteria led to a very lower probability of SMD of 1.76% (group 1). For those adolescents who endorsed “mood modification” but denied “withdrawal”, endorsed “conflict” gave the most subsequent information: If adolescents endorsed “conflict”, the probability of classification of SMD was 71.4% (Subgroup 5); If they denied “conflict”, the probability of classification of SMD decreased to 23.5% (Subgroup 4). For those adolescents who denied “mood modification” but endorsed “withdrawal”, endorsed “relapse” gave the most subsequent information: If adolescents endorsed “relapse”, the probability of classification of SMD was 63.3% (Subgroup 3); If they denied “relapse”, the probability of classification of SMD decreased to 17.1% (Subgroup 2).

Fig. 2.

Conditional inference tree plot predicting social media disorder by Griffiths’ six criteria, age, gender and social media using time (n = 21,375)

Discussion

Using clinically diagnosed disordered social media users as the gold standard, the current study suggested a clinically optimal cut-off score of 24 for diagnosing SMD with the BSMAS. The highest diagnostic accuracy (98.8%) and the best balance between sensitivity (96.4%) and specificity (99.1%) were achieved at this cut-off score. The previous recommendation for a cut-off score was 19 based on empirical research (Bányai et al., 2017). In this study, the PPV were relatively low at this cut-off score (43.7%). The lower PPV indicates the more cases with a positive diagnosis would be identified incorrectly. In an epidemiological screening context, this may lead to an overestimation of the prevalence of SMD and may be considered unacceptable.

The LPA approach has been performed to identify cut-off points in the condition of a gold standard based on clinical interviews are unavailable. The adequate diagnostic accuracy of the empirical cut-off point based on LPA suggest that this approach can be a reliable method to identify cut-off points. The LPA approach can facilitate epidemiological studies and interventions, especially when “positive” is defined as an increased likelihood of behavioural problem and its related harms.

The estimated 12-month prevalence of SMD among Chinese adolescents was 3.5%. The results indicated a relatively conservative estimated prevalence of SMD. Although they used the same instruments as the current study, it is not surprising that Tang and colleagues reported a higher prevalence of 44.9% because of the difference in cut-off scores (Tang et al., 2018). The study also revealed that adolescents diagnosed with SMD spent more time on social media use, showed worse academic performance, and presented higher impulsivity and the lower self-esteem, which is in line with previous studies (Bányai et al., 2017; Savci, Ercengiz, & Aysan, 2018). However, different from Bányai’s study in Hungarian adolescent sample (Bányai et al., 2017), our research shows that adolescents with SMD are more likely to be male. This finding is consistent with studies of gender differences on IGD (Rehbein et al., 2015).

Several scholars in the field of behavioural addiction consider the “salience”, “relapse”, “mood modification”, and “conflict” criteria of Griffiths’ components model as being parallel with the “preoccupation”, “loss of control”, “escape”, and “continue despite problems” criteria of the DSM-5 (Den Eijnden et al., 2016; Lemmens, Valkenburg, & Gentile, 2015). Disputes have occurred on the distinguishing power of DSM criteria of “preoccupation” and “tolerance” between disordered and highly engaged players (Kardefelt-Winther, 2014; Kardefeltwinther, 2015). In accordance with the findings from studies regarding IGD (H. M. Pontes, Kiraly, Demetrovics, & Griffiths, 2014), we found that the “salience” and “tolerance” criteria were endorsed at high rates, but weak in distinguishing “disordered” and “highly engaged” social media users and predicting SMD. These results suggest that thinking about or even increasingly using social media may represent high engagement but not necessarily pathology (Kardefelt-Winther, 2014; Kardefeltwinther, 2015; H. M. Pontes et al., 2014). Future research should examine the validity of these criteria to diagnose SMD.

In contrast to findings from studies regarding IGD, The “mood modification” criteria showed the lowest endorsement rate and the highest predictive power in comparison to the “mood modification” criteria of IGD (Rehbein et al., 2015). As a daily communication tool, social media users may be less aware than gamers that their purpose for using social media is to escape adverse moods or personal problems. Additionally, most non-disordered users may use social media as a way to connect with others or pass time rather than coping with something. In this study, only 0.4% of non-disordered users using social media as a coping strategy to deal with mood problems, while in disordered users, the rate is 41.7%.

The criteria “conflict”, “withdrawal”, and “relapse” have been widely recognized as core components of addiction (Charlton & Danforth, 2007). Our findings revealed that these criteria provided fairly high distinguishing power between “disordered” and “highly engaged” social media users, and strong predictive power of SMD independently. Our results further demonstrated that SMD is an addictive behaviour with the same overarching structure and core components as other forms of addictive behaviours (e.g., IGD) (Den Eijnden et al., 2016).

Although this study provides unique information on the identification of SMD and the prevalence of SMD among Chinese university students, some limitations exist. First, the analyses were limited to the six criteria based on Griffiths’ component model of addiction. A wider set of criteria as proposed for IGD in DSM-5 and most importantly significant clinical impairment as highlighted in the the International Classification of Diseases, 11th Revision (ICD-11) strongly require further investigation. Second, the community sample only included adolescents in two cities of China as participants. Therefore, the findings need to be cautiously interpreted in terms of generalizability. Third, self-report questionnaires among community adolescents may lead to various response biases, including memory recall bias, social desirability bias, and response style bias.

Conclusions

Using ROC analysis, the current study determined the BSMAS cut-off point of 24 based on the gold standards for clinical diagnosis. The diagnostic performance of the empirical cut-off point based on LPA was evaluated, and the high diagnostic accuracy provided support for the LPA approach to identify cut-off point when gold standards are unavailable. Furthermore, the current study assessed diagnostic contribution of the six criteria of the components model of addiction for social media disorder. This study found that the most relevant criteria to the diagnosis of SMD in Chinese adolescents are “mood modification”, “conflict”, “withdrawal”, and “relapse”.

Funding sources

This study was supported by the Natural Science Foundation of Jiangxi Province of China (No. 20192BAB205037), and the “Hundred Talents Program” funding from Zhejiang University.

Authors’ contribution

TL, YL, BW: contributed in conceptualizing and designing the study, analysis and interpretation of data, drafting and revising the article, and final approval of the version to be published. LQ, LC, SW, ZZ, JX, HC, QL, MH, JT, WH: contributed in collecting, analysis, and interpretation of data, drafting the article, and final approval of the version to be published.

Conflict of interest

The authors declare no conflict of interest.