Validation of the Internet Gaming Disorder Scale - Short-Form (IGDS9-SF) in an Italian-speaking sample.

Background and aims The inclusion of Internet Gaming Disorder (IGD) in Section III of the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders has increased the interest of researchers in the development of new standardized psychometric tools for the assessment of such a disorder. To date, the nine-item Internet Gaming Disorder Scale - Short-Form (IGDS9-SF) has only been validated in English, Portuguese, and Slovenian languages. Therefore, the aim of this investigation was to examine the psychometric properties of the IGDS9-SF in an Italian-speaking sample. Methods A total of 757 participants were recruited to the present study. Confirmatory factor analysis and multi-group analyses were applied to assess the construct validity. Reliability analyses comprised the average variance extracted, the standard error of measurement, and the factor determinacy coefficient. Convergent and criterion validities were established through the associations with other related constructs. The receiver operating characteristic curve analysis was used to determine an empirical cut-off point. Results Findings confirmed the single-factor structure of the instrument, its measurement invariance at the configural level, and the convergent and criterion validities. Satisfactory levels of reliability and a cut-off point of 21 were obtained. Discussion and conclusions The present study provides validity evidence for the use of the Italian version of the IGDS9-SF and may foster research into gaming addiction in the Italian context.

Introduction

The recent inclusion of Internet Gaming Disorder (IGD) in Section III (“Emerging Measures and Models”) of the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5; American Psychiatric Association, 2013) appears to have increased the interest of researchers in the development of new standardized psychometric tools for the assessment of various online addictions. IGD has been characterized by a “persistent and recurrent use of the Internet to engage in games, often with other players, leading to clinically significant impairment or distress” (APA, 2013, p. 795). The DSM-5 asserts that further empirical evidence is needed to confirm the nine criteria proposed for the clinical diagnosis of IGD, and to formally define IGD as a mental disorder in future editions of the DSM. Of the nine criteria, seven criteria are identical to those of gambling disorder and five criteria to substance use disorder (Petry et al., 2014), and refer to preoccupation with Internet games, withdrawal symptoms, tolerance, unsuccessful attempts to control participation in Internet games, loss of interest in previous hobbies, continued excessive use of Internet games, deceiving family members, use Internet games to escape, and losing a significant relationship, job or education, or career opportunity. To be diagnosed as a disordered gamer, five (or more) out of these criteria need to be endorsed over a period of 12 months (APA, 2013).

The nine IGD criteria directly map onto the six criteria of Griffiths’ components model of addiction, and which have been used to conceptualize a number of technological addictions (Griffiths, 1995, 2005). According to Griffiths, by “determining whether non-chemical […] addictions are addictive in a non-metaphorical sense” other potentially addictive behavior should be compared “against clinical criteria for other established drug-ingested addictions” (Griffiths, 2005, p. 192). The six criteria comprise salience, mood modification, tolerance, withdrawal symptoms, conflict, and relapse. Salience occurs when addictive activities dominate a person’s thinking, feelings, and behavior; mood modification occurs when a person uses substances or is engaged in activities to change their mood state; tolerance refers to the need to increase (over time) the amounts of engagement in the addictive behavior to achieve the initial mood-modifying effects; withdrawal symptoms refer to the unpleasant feeling states and/or physical effects that occur when the individual decreases or suddenly reduces their addictive activities; conflict indicates both the intrapsychic and interpersonal problems that arise as a consequence of addictive activities; and relapse refers to the unsuccessful efforts to stop engaging in the addictive behavior if the individual is trying to cease.

Another frequently discussed issue concerns the different terminologies present in the scientific literature to define more or less the same phenomenon (Widyanto & Griffiths, 2006), such as computer game dependence (Griffiths & Hunt, 1998), video game addiction (Griffiths & Davies, 2005), Internet gaming addiction (Kuss & Griffiths, 2012a), excessive game engagement (Brockmyer et al., 2009), and problematic online gaming use (Kim & Kim, 2010). A shared nomenclature of the IGD concept and the relative standardized psychometric tool have been proposed by Griffiths, King, and Demetrovics (2014) and Pontes and Griffiths (2015, p. 138) to foster both the consensual view of the phenomenon from the scientific standpoint and the unification of the different approaches into a singular one. To achieve these goals, some psychometric tools based on the DSM-5 criteria of behavioral addictions have been developed including the Internet Gaming Disorder Test (IGD-20 Test; Pontes, Király, Demetrovics, & Griffiths, 2014), the nine-item Internet Gaming Disorder Scale – Short-Form (IGDS9-SF; Pontes & Griffiths, 2015), the 10-item Internet Gaming Disorder Test (Király et al., 2017), the Internet Gaming Disorder Scale (Lemmens, Valkenburg, & Gentile, 2015), the updated Clinical Video game Addiction Test (Van Rooij, Schoenmakers, & Van de Mheen, 2017), and the Video Game Dependency Scale (Rehbein, Kliem, Baier, Mößle, & Petry, 2015).

The present study employs Griffiths’ model and validates the IGDS9-SF in a different national context (i.e., Italian). The IGD-20 Test comprises six dimensions corresponding to the six components of the model, whereas the IGDS9-SF contains nine items as indicators of a single latent factor, thus providing a briefer assessment of the IGD. The IGDS9-SF was developed to be utilized in large-scale surveys. The validity of both versions has been confirmed in the Spanish-, Portuguese-, and Slovenian-speaking samples (Fuster, Carbonell, Pontes, & Griffiths, 2016; Pontes & Griffiths, 2016; Pontes, Macur, & Griffiths, 2016).

In the Italian context, research on technological addictions has mainly focused on Internet Addiction Disorder (Coniglio, Sidoti, Pignato, Giammanco, & Marranzano, 2012; Ferraro, Caci, D’Amico, & Di Blasi, 2007; Servidio, 2014). To the best of the authors’ knowledge, IGD and its assessment tools have not been explored in Italy. In light of this, the main goal of the present study was to examine the psychometric properties of an Italian version of the IGDS9-SF. Consequently, the factorial structure and the measurement invariance (MI) of the IGDS9-SF across age and sex groups were evaluated. Moreover, the associations between the translated version and other related measures were examined to evaluate the convergent and criterion validities of scores on the IGDS9-SF. Finally, an empirical cut-off point for the Italian version of the instrument was further provided to distinguish disordered and non-disordered gamers.

Methods

Participants and procedure

Participants were recruited from Italian schools, universities, and gaming halls. The schools were chosen on the basis of their availability, and the students of the schools were selected by randomly sampling the pool of classes with students aged over 16 years. Participants were voluntary invited to take part in the study by completing a self-report questionnaire, which took approximately 15 min to complete. The period of the data collection spanned from February to June 2016. Potential order effects were controlled by presenting the questionnaires in three randomized orders. A total of 757 questionnaires were collected. Of these, 47 questionnaires were not fully completed and were excluded from the subsequent analyses. In addition, 23 questionnaires were removed after cleaning the data set. The final sample comprised 687 participants (375 males and 312 females; mean age = 21.62 years, SD = 3.90). Of these, 93.4% of the participants were unmarried, 80.5% of the participants were high school graduates, and 72.8% of the participants were students. The sample was split into two age categories: those aged 16–19 years were classed as adolescents (N = 254) and those aged over 20 years were classed as young adults (N = 433).

The scales were translated from English into Italian separately by the Italian authors of the present study following the recommendations by Merenda (2006). After the measures were translated into Italian, they were back-translated into English by a native speaker to establish their comparability. The resulting Italian version was subjected to a pilot study with a sample of 30 students to capture eventual problems concerning items content.

Measures

Socio-demographics

The questionnaire included questions concerning sex, age, relationship status, educational level, and employment to obtain a profile of the respondents’ demographic features.

Internet Addiction Test (IAT) – Italian version

The Italian version of the IAT (Fioravanti & Casale, 2015; original English version by Young, 1998) is a 20-item scale that assesses the severity of self-reported compulsive use of the Internet for adults and adolescents. Each item is responded to on a 5-point Likert scale that ranges from 1 (never) to 5 (always). The total score is computed by averaging the scores obtained in each item. In the present study, the internal reliability of the IAT was excellent (Cronbach’s α = .95).

Gaming Addiction Scale (GAS)

The Italian back-translated version of the GAS – Short-Form (original English version by Lemmens, Valkenburg, & Peter, 2009) was used to assess the levels of gaming addiction. The scale comprises seven items rated on a 5-point Likert scale from 1 (never) to 5 (very often) assessing the feelings and behaviors of the gamers and their relationships with other people or things. Each item refers to the seven DSM-based criteria for game addiction, i.e., salience, tolerance, mood modification, withdrawal symptoms, relapse, conflict, and problems (e.g., Griffiths, 2005; Griffiths & Davies, 2005). The GAS was found to have very good levels of internal consistency in the present study (Cronbach’s α = .89).

Bergen Social Networking Addiction Scale (BSNAS)

The Italian back-translated version of the BSNAS (original English version by Andreassen et al., 2016) assesses the experiences in the use of social media over the past year. It contains six items reflecting core addiction elements (Griffiths, 2005). Each item is answered on a 5-point Likert scale ranging from 1 (very rarely) to 5 (very often). In the present study, the internal consistency of the BSNAS was very good (Cronbach’s α = .88).

IGDS9-SF

The Italian version of the IGDS9-SF (original English version by Pontes & Griffiths, 2015) assesses the severity of IGD and its detrimental effects by examining both online and/or offline gaming activities occurring over a 12-month period. The scale comprises nine items corresponding to the nine core criteria defined by the DSM-5. They are answered on a 5-point Likert scale ranging from 1 (never) to 5 (very often). Higher scores indicate higher degree of gaming disorder. In the present study, the IGDS9-SF had excellent reliability with an internal consistency coefficient (Cronbach’s α) of .96, and is comparable with the coefficients reported in other studies (Fuster et al., 2016; Pontes & Griffiths, 2015, 2016).

Statistical analysis

Before performing the data analysis, the cleaning of the data set was conducted by the inspection of cases with missing values in the instruments. The univariate normality of all items of the IGDS9-SF was checked by following the standard guidelines of Kim (2013). More specifically, “[f] or sample sizes greater than 300 […] either an absolute skew value larger than 2 or an absolute kurtosis (proper) larger than 7 may be used as reference values for determining substantial non-normality” (p. 53). In addition, the univariate outliers were identified using the graphic approach (inspection of Boxplot), whereas the multivariate outliers were inspected using Mahalanobis distances and the critical value for each case based on the chi-square (χ2) distribution values. In total, 23 cases were removed, thus yielding a final data set of 687 valid cases eligible for subsequent analyses.

First, statistical analyses comprised an independent samples t-test to verify sex and age effects on the scores of the variables taken into account.

Second, data were submitted to confirmatory factor analysis (CFA) to assess the construct validity of the IGDS9-SF, as well as to multi-group analyses across sex and age to assess its MI. For CFA, the χ2 and its degree of freedom (df), the comparative-of-fit index (CFI), the root mean square error of approximation (RMSEA) and its 90% confidence interval (CI), and the standardized root mean square residuals (SRMR) were used. For χ2, test values associated with p > .05 were considered good-fitting models. However, since the p value of the χ2 test is sensitive to large sample sizes (meaning the p is <.01 when sample sizes are large, regardless of the quality of model fit), it is recommended that multiple indices are used such as the CFI and the RMSEA in addition to the χ2 statistic. For CFI, values greater than or equal to .90 were accepted as indicators of good fit (Bentler & Bonett, 1980). Hu and Bentler (1999) demonstrated that RMSEA is one of the most informative criteria and recommended a value close to .06 in conjunction with an SRMR value of .08 or less.

Furthermore, in line with Vandenberg and Lance’s (2000) recommendations, MI across age and sex was evaluated through the following steps: (a) testing for the invariance of number of factors (configural invariance); (b) testing for the equality of factor loadings (weak or metric invariance); and (c) testing for the equality of indicator intercepts (strong or scalar invariance). The classical approach based on the χ2 difference (Δχ2) test was used. As this method is sensitive to the model’s complexity and large sample size, it is recommended to compare two nested models using cut-off values of ΔCFI < .01 and ΔRMSEA < .015 for metric and scalar invariances (Chen, 2007; Cheung & Rensvold, 2002). As Bollen (1989) suggested, metric invariance is an important prerequisite for meaningful cross-group comparison.

Third, the scale reliability was examined using: (a) the average variance extracted (AVE) that assesses the extent to which the items of a specific factor converge or share a high proportion of variance (Hair, Black, Babin, & Anderson, 2010); values greater than .50 are considered adequate; (b) the standard error of measurement (SEM) that assesses the degree to which the observed scores fluctuate as a result of the measurement errors (Morrow, Jackson, Disch, & Mood, 2011). The criterion of acceptable precision was SEM ≤ SD/2 (Wuang, Su, & Huang, 2012); (c) the factor determinacy coefficient of the internal consistency (Tabachnick & Fidell, 2013). As noted by Brown (2003), this coefficient represents an important result of factor analysis. In particular, a high degree of determinacy indicates that “the factor score estimates could serve as suitable substitutes for the factor itself” (Brown, 2003, p. 1418). Factor score determinacy represents the correlation between the estimated and true factor scores. It ranges from 0 to 1 and describes how well the factor is measured, with 1 being the best value (Muthén & Muthén, 1998–2012). The larger the coefficient (e.g., ≥.70, Tabachnick & Fidell, 2013), the more stable the factors, in the sense that the observed variables account for substantial variance in the factor scores, whereas low values mean that the factors are poorly defined by the observed variables. Fourth, convergent and criterion validities were established through the analysis of the correlation patterns between the construct of interest and other related constructs.

Finally, the receiver operating characteristic (ROC) curve analysis was used to assess the discriminate ability of the IGDS9-SF at varying cut-off points, according to the GAS cut-off score (GAS 21+ criterion; Lemmens et al., 2009, pp. 87–88) as standard. All statistical analyses were performed using Mplus 7.2 and IBM SPSS Statistics 20.

Ethics

The study procedures were carried out in accordance with the Declaration of Helsinki. The investigation was approved by the research team of Department of Human Sciences Ethics Committee (December 2015). Permission was required from heads and deans to conduct the research study at the school/institution. Written informed consent was obtained from students over 18 years of age, whereas parents or legal guardians provided written consent for students under 18 years of age to participate.

Results

Independent sample t-test

Significant sex differences (t(648.267) = 10.03, p < .001) and age differences (t(676.317) = 6.61, p < .001) emerged in IGDS9-SF scores. More specifically, males and young adults obtained higher IGDSF-9 scores (Table 1).

Table 1.

Mean and standard deviations for IGDS9-SF scores in sex and age groups

Groups	N	Mean	SD
Males	375	18.75	9.796
Females	312	12.53	6.349
Adolescents	254	13.33	6.461
Young Adults	433	17.45	9.831

CFA

To test the original single-factor structure of the IGDS9-SF, CFA was conducted with the mean and variance adjusted maximum likelihood (MLMV) method. The fit indices were acceptable: χ2 = 182.132, df = 27, p < .001; RMSEA = .091, 90% CI = .07–.10; CFI = .958; SRMR = .03. As the RMSEA value was high, a careful inspection of the modification indices (MIs) suggested adding a covariance path between the error terms of Items 6 and 7 (MI = 42.819). After carrying out a second CFA, the indices showed a better degree of fit: χ2 = 138.030, df = 26, p < .001; RMSEA = .072, 90% CI = .06–.09; CFI = .970; SRMR = .02. All factor loadings were significant and ranged from .72 to .94 (Figure 1).

Figure 1.

Graphical summary of the confirmatory factor analysis results obtained from the nine items of the Internet Gaming Disorder Scale – Short-Form (IGDS9-SF) with two error terms correlated (N = 687)

MI across sex and age groups

To evaluate the generalizability of the model across males and females, adolescents and young adults, two multi-group CFAs using MLMV estimation were performed. For each analysis, an unconstrained model with factor loadings free to vary between subgroups was compared with a constrained model, in which the factor loadings were held constant across subgroups. Before conducting multi-group analyses, separate CFAs were performed for age and sex subgroups. Results indicated a good fit of the data for each subgroup; the MI of the single-factor solution was supported at all three levels (configural, metric, and scalar) across sex and age groups (Table 2).

Table 2.

Measurement invariance by sex and age groups

Model	χ2	DF	Δ χ2	ΔDF	Sig.	CFI	ΔCFI	RMSEA	ΔRMSEA
Sex
Females	63.111	26	–	–		.958	–	.068	–
Males	96.741	26	–	–		.970	–	.081	–
Configural	144.709	52	–	–		.964	–	.072	–
Metric	175.066	60	30.357	8	.001	.956	.008	.075	.003
Scalar	192.307	68	17.241	8	.001	.952	.004	.073	.002
Age
Adolescents	32.096	26	–	–		.992	–	.030	–
Young adults	133.261	26	–	–		.957	–	.092	–
Configural	166.225	52	–	–		.964	–	.080	–
Metric	183.846	60	17.621	8	.001	.961	.003	.073	.007
Scalar	197.619	68	13.773	8	.001	.960	.001	.074	.001

Reliability analyses

Once the single-factor solution was confirmed, the extent to which the items of the specific factor converged or shared a high proportion of variance was assessed through the AVE method. The result provided a good value (AVE = .76). In addition, the SEM was calculated to assess the degree to which the observed scores fluctuated as a result of the measurement errors. As expected, the value met the criterion (SEM = 1.79 ≤ SD/2 = 4.48). Finally, the factor score determinacy coefficient was .99, showing an excellent degree of internal consistency.

Convergent and criterion validities

The convergent validity was assessed by correlating the IGDS9-SF scores with the scores of two similar scales (i.e., the GAS and the IAT), and the criterion validity was evaluated through patterns of correlations between the IGDS9-SF and the BSNAS scores. The BSNAS was chosen because it utilizes the same six behavioral addiction criteria used for the IGDS9-SF (Griffiths, Kuss, & Demetrovics, 2014, p. 121). Results clearly demonstrated high correlations among the variables of interest, thus confirming the hypothesized validities (Table 3).

Table 3.

Bivariate correlations between IGDS9-SF, GAS, IAT, and BSMAS scores

	GAS	IAT	BSNAS
IGDS9-SF	.809**	.827**	.764**

**p < .001.

Cut-off point

The ROC curve analysis was carried out to determine the optimal cut-off value (Figure 2). Following Charlton and Danforth’s (2007) recommendations and incorporating Lemmens et al.’s (2009) methodological approach, the monothetic format was applied to determine whether some respondents were classifiable as addicted gamers, given that this format (which requires endorsement of all of the criteria) provides a stricter and more realistic estimate of addicted gamers, whereas the polythetic format (which requires addicts to endorse half or more of the proposed criteria) is likely to lead to an overestimation of the frequency of addicted gamers. The discriminating ability of the IGDS9-SF at varying of cut-off scores using the GAS 21+ criterion (Lemmens et al., 2009) as gold standard was examined. The ROC analysis resulted in a cut-off point of 21 in determining IGD (Table 4), and the area under the curve was of .935.

Figure 2.

The receiver operating characteristic (ROC) curve of the Italian version of the IGDS9-SF

Table 4.

Cut-off point and characteristics for the IGDS9-SF

Cut-off point	TP	TN	FP	FN	PV+	PV−	Acc	Sensitivity	Specificity
15	105	433	139	10	97.74%	43.03%	78.31%	.757	.913
16	105	450	122	10	97.83%	46.26%	80.78%	.787	.913
17	104	461	111	11	97.67%	48.37%	82.24%	.806	.904
18	102	468	104	13	97.30%	49.51%	82.96%	.818	.887
19	102	481	91	13	97.37%	52.85%	84.86%	.841	.887
20	101	487	85	14	97.21%	54.30%	85.58%	.851	.878
21	99	492	80	16	96.85%	55.31%	86.02%	.861	.860
22	98	495	77	17	96.68%	56.00%	86.31%	.865	.852
23	98	503	69	17	96.73%	58.68%	87.41%	.879	.852
24	97	508	64	18	96.58%	60.25%	88.06%	.888	.843
25	94	523	49	21	96.14%	65.73%	91.12%	.914	.817
26	90	537	35	21	95.55%	72.00%	91.26%	.939	.783
27	86	546	29	26	94.96%	76.79%	91.99%	.955	.748
28	87	555	31	17	94.71%	83.17%	93.94%	.970	.730

Note. TP = true positives; TN = true negatives; FP = false positives; FN = false negatives; PV+ = positive predictive value; PV− = negative predictive value; Acc = accuracy; Bold value = cut-off point.

Discussion

The primary purpose of the present study was to examine the psychometric properties of the IGDS9-SF in an Italian-speaking sample. To this aim, the instrument was subjected to construct, convergent, and criterion validities, as well as to the identification of an empirical cut-off point for determining a binary classification between disordered and non-disordered gamers. Overall, the instrument was deemed to be valid. More specifically, results from CFA confirmed the single-factor solution as the model achieved an acceptable fit to the data. All factor loadings were statistically significant and relatively high, demonstrating that all items were good construct indicators. These results generally corroborated prior validity studies (Fuster et al., 2016; Pontes & Griffiths, 2015, 2016; Pontes et al., 2016). However, the unexpected covariance between the residual errors for Items 6 and 7 was theoretically justifiable as the two items might be referred to the same criterion (i.e., conflict).

The MI across sex and age groups was also tested to verify whether the components of the Italian translation of the brief screening tool operated equivalently across different groups. Evidence of configural, metric, and strict invariances was found. More specifically, the factorial structure resulted invariance across the different groups, and the meaning of the construct was equivalent (as assessed by the instrument) because both ΔCFI and ΔRMSEA indices were below the cut-off values. Future research should replicate the multi-group analyses in other countries, given the lack of empirical studies assessing age and sex subgroup invariances. In terms of reliability, data supported the internal consistency of the IGDS9-SF as assessed by several indicators, such as the Cronbach’s α, AVE, SEM, and factor determinacy, whose values were found to be high. This demonstrates that the measure is reliable and accurate in assessing IGD.

In addition to these results, criterion-related and convergent validities were warranted by the expected positive pattern of correlations emerged between the IGDS9-SF and all the related measures. The high associations lend support for the assumption that IGD represents “a part of the postulated construct of Internet addiction,” which, in turn, “comprises a heterogeneous spectrum of Internet activities […], such as gaming, shopping, gambling, or social networking” (Kuss & Griffiths, 2012b, p. 348).

Regarding the ROC curve analysis, a first empirically optimal cut-off of 21 points was yielded for diagnosing IGD with the brief version of the scale. It should be noted that the moderate positive predictive value may be explained by the low prevalence of disordered gamers in the Italian sample (as noted in a previous paper by Maraz, Király, & Demetrovics, 2015). Future investigations should be conducted to assess whether such a cut-off point has an empirical and clinical validity, as suggested by Pontes and Griffiths (2015, p. 141).

Finally, in line with the data reported in the previous research (Griffiths, Davies, & Chappell, 2003; Ko, Yen, Chen, Chen, & Yen, 2005; Lee, Ko, & Chou, 2015), gender and age differences were found. More specifically, males and young adults seemed to be more engaged in gaming activities. Alongside the socio-demographic characteristics, personality-related aspects should be further examined in terms of protective/risk factors to give support to the existing empirical research, which has already demonstrated relationships between some personality traits (such as narcissism, neuroticism, consciousness, trait aggression, sensation seeking, state and trait anxiety, etc.) and gaming addiction (Mehroof & Griffiths, 2010; Müller, Beutel, Egloff, & Wölfling, 2014; Stopfer, Braun, Müller, & Egloff, 2015), thus helping to identify a more detailed profile of disordered gamers.

The present study provides validity evidence for the use of the Italian version of the IGDS9-SF, and also contributes to and extends the body of the literature on the topic. However, some limitations need to be highlighted. A more representative sample of the population is required to generalize the findings. Future replication research should strive to employ also a clinically diagnosed sample to consider the instrument as a valid diagnostic tool. The study is also limited by the fact that all the data were self-report and are subject to well known associated biases, such as social desirability biases, short-term recall biases, etc. Taken as a whole, the present study will hopefully foster research into gaming addiction in the Italian context, thus expanding the investigation into culture-specific factors and, at the same time, facilitating a general and international consensus for defining the criteria of IGD.

Authors’ contribution

LM and VdP: study concept and design, analysis and interpretation of data. MDG and MS: study supervision.

Conflict of interest

The authors declare no conflict of interest associated with this publication.