Face validity evaluation of screening tools for gaming disorder: Scope, language, and overpathologizing issues.

AIM: Critics of gaming disorder (GD; i.e., Internet gaming disorder in the DSM-5; Gaming disorder in the ICD-11) have expressed concerns about the potential risks of misclassification (e.g., false positives). An important consideration of relevance to this discussion is the extent to which commonly used screening instruments contain appropriate, sensible, and relevant items. The aim of this review was to evaluate the face validity of items within current tools for GD.
METHODS: A systematic review of databases identified 29 instruments. An item bank (n = 417 items) was independently evaluated by three professional raters (i.e., a senior academic in clinical psychology, a senior psychometrician, and an academic/clinical psychologist) according to guidelines for defining and measuring addiction and gaming disorder.
FINDINGS: Evaluation of the item bank identified issues related to: scope (i.e., "scope creep" or items of questionable relevance); language (i.e., confusing language, unusual wording or syntax); and overpathologizing (i.e., pathologizing typical and/or beneficial aspects or consequences of gaming). A total of 71 items across 23 tools had at least one face validity issue.
CONCLUSIONS: Most items (83%) demonstrated satisfactory face validity and were consistent with either the DSM-5 or ICD-11 GD classification. However, many tests contain at least one item that may pathologize normal gaming behaviors. Such items refer to basic changes in mood when gaming, a desire to play or continue playing games, and experiencing immersion when gaming. This analysis highlights the challenges of screening for problematic behaviors that are thought to arise within the context of normal recreational activities.

Introduction

Gaming disorder (GD) has been officially recognized in the eleventh edition of the International Classification of Diseases (ICD-11; WHO, 2019), following the recognition of “Internet gaming disorder” as a condition for further study in Section III of the DSM-5 (American Psychiatric Association, 2013). These developments have followed the publication of numerous studies of this condition, including epidemiological studies, neurobiological research, and interventions (Brand, Young, Laier, Wölfling, & Potenza, 2016; Feng et al., 2017; Saunders et al., 2017; Stevens, King, Dorstyn, & Delfabbro, 2019). In this literature, research participants have often been classified or assigned to groups of “addicted” or “disordered” gamers using one of the many available screening instruments that have been created over the last two decades (King et al., 2020). Some commentators have criticized the evidence as “weak” (van Rooij et al. 2018, p. 6), and raised concerns about the risk of misclassification (Starcevic, 2017). These concerns often cite the notion that the disorder would “stigmatise” and create “moral panic” (Aarseth et al., 2017; Bean, Nielsen, Van Rooij, & Ferguson, 2017) about a common class of behavior that may not be inherently harmful for the majority of participants.

Along similar lines, researchers who support GD have noted that the bar should be set reasonably high to avoid viewing “regular” or recreational behavior as a problem (King, Delfabbro, Potenza, Demetrovics, Billieux, & Brand, 2018b, King, Herd, & Delfabbro, 2018a). This distinction is particularly relevant to a condition such as GD given that there are many people who would identify as “highly engaged” in the activity (Billieux, Flayelle, Rumpf, & Stein, 2019). These individuals may report spending a lot of time thinking about games and seeking out new gaming opportunities but do not experience negative consequences related to their recreational participation in gaming activities (King & Delfabbro, 2014). For such players, the positives of gaming may largely outweigh any negatives (which may be minor). Thus, it is important, from a research, clinical, and public health perspective, that screening measures are not overly sensitive and do not misclassify these individuals as problematically involved or “addicted” to games (Ko, Lin, Lin, & Yen, 2019; Rumpf et al., 2018).

Previous reviews have noted some inconsistencies and psychometric weaknesses in instrumentation (King, Haagsma, Delfabbro, Gradisar, & Griffiths, 2013; Petry et al., 2014). These reviews have tended to focus on important psychometric considerations, such as factorial structure, reliability, and correlates (convergent and divergent validity). Such evidence is critical to determining the psychometric performance of the test, including whether these measures can reliably differentiate between clinical and normal or recreational behavior (what is sometimes referred to as criterion validity). If such validation does not exist, then the scores obtained on these scales will merely reflect the nature of the items included which may ultimately produce inflated prevalence rates of harm. In support of this view, there have been multiple studies of adolescents that report gaming disorder figures that exceed 20% which seem to defy common sense or logic, and which likely include a high number of false positives (Sosso et al., 2020; Seok & DaCosta, 2012; Wang, Chan, Mak, Ho, Wong, & Ho, 2014; Xin, Xing, Pengfei, Houru, Mengcheng, & Hong, 2018).

One way to evaluate new instruments is to consider the extent to which the content of test items matches the clinical description of the condition (i.e., face validity). While face validity is not considered to be the optimal indicator of a test's quality, it can nevertheless be beneficial as a preliminary check. A face validity evaluation of test items is usually undertaken in the early stages of test construction, typically by a small group of independent raters (Holloway et al., 2014). Although this approach is subjective and depends largely on the views and interpretation of each panel member and the panel composition itself (Campbell, Hann, Roland, Quayle, & Shekelle, 1999), face validity checks may have beneficial outcomes (e.g., identify potential problems with test items that can then be reviewed psychometrically). This is particularly so when undertaken by a panel of experts or individuals with personal experience, especially for tests of new conditions or tests containing previously unexamined scale items (Hardesty & Bearden, 2004).

This review examines the extent to which the items included in prevailing GD tests appear to be consistent with current criteria and descriptions of the condition. This paper employs the acronym GD to encompass both the DSM-5 and ICD-11 classifications, including the symptoms of the condition. This review involved a systematic search to identify all currently used tests and then a panel evaluation of all test items to identify any potential face validity issues. This evaluation was primarily intended to identify items that may benefit from further examination and should not be viewed as objective evidence of the strengths or weaknesses of any given test highlighted in this analysis. However, as the field of behavioral addictions appears to be expanding its scope, it is important to identify and avoid potentially problematic items being transposed to other proposed conditions that have similarities to gaming behaviors. Many of the first tools for problematic gaming and/or GD were created by adapting measures of pathological gambling and substance use disorders (King & Delfabbro, 2019); similarly, measurement approaches to other problematic online behaviors (e.g., social media use) have been guided by those used for GD (e.g., Besser, Loerbroks, Bischof, Bischof, & Rumpf, 2019).

Methods

Identification of Screening Tools

This review aimed to evaluate all available screening tools for GD published in English. A search of Google Scholar and academic databases, including PsychINFO, Academic Search Complete, ScienceDirect, PubMed and Web of Science (search date: 4/4/2019), yielded 3,499 results (including duplicate results). The following keywords and protocol were used: (video game OR gam*) AND (tool OR measure OR test OR validation OR psychometric OR screening OR diagnos* OR item OR instrument). A list of tools was generated by identifying tests in the methodology section of empirical studies. Excluded tests included: composite tests (i.e., two or more tests combined into one); established tests or criteria with modified or added items; and tests without a citation or full text. Internet use-related tests (e.g., Young Internet Addiction Test) were included if they had been employed in studies to measure GD symptoms. Other relevant reviews (King et al., 2013; Lortie & Guitton, 2013; Petry et al., 2014) were also consulted to assist in identifying relevant tools. The Google Scholar citation lists of all original sources (N = 12,996 citations) were checked for any additional tests, which included 320 empirical studies. In total, 29 tools were identified. From these tools, an item bank of 417 items was produced for evaluation.

Item Evaluation

All 417 items were evaluated by three professional raters (i.e., a senior academic in clinical psychology, a senior psychometrician, and an academic/clinical psychologist). The objective of this analysis was to identify items that appeared problematic on face value, and to then reach a panel consensus rating on these items. A scoring sheet was assigned to each rater who completed the sheet blinded to the other two raters' responses.

The aim of the face validity evaluation of each item was to consider whether: (1) the item matches its respective criterion: whether the item's wording was consistent with the construct and concepts of addiction, such as impaired control (Goodman, 1990; Griffiths, 2017; West, 2001), and/or descriptions of gaming disorder or addiction according to the DSM-5 and/or ICD-11 (Grant and Chamberlain, 2016; King & Delfabbro, 2019); (2) the item refers to a harmful or distressing experience; whether the item referred to a gaming behavior that is distressing, risky or harmful to the user and/or others (Stein, Phillips, Bolton, Fulford, Sadler, & Kendler, 2010), and excluded behaviors or consequences that would generally be considered normal or adaptive (Billieux et al., 2017; Kardefelt-Winther et al., 2017); and (3) the item is clear and easily understood: whether the item employed clear and unambiguous language (Willis, 2004). The above considerations informed the development of a scoring sheet (Excel spreadsheet) with five categories: scope, language, overpathologizing, lacking clinical relevance, and other.

Initial ratings of the item bank (i.e., 5 categories × 417 items = 2,085 individual items) were collated, with the following rater discrepancies noted: 219 problematic items (10.5%) according to one rater (with 43 or 2.1% of these items rated as having another unanimously rated validity issue), and 88 problematic items (4.2%) according to two raters. A second evaluation round involved all three raters re-evaluating the 88 items with 2 positive ratings, with the option to provide reasoning for each decision. The re-evaluation led to 41 items endorsed by two raters changed to unanimous agreement, and 47 items unchanged. There was no clear pattern to non-unanimous ratings according to the five categories, but one particular two-rater combination (DLK and JB) was more inclined to critique items across all five categories (n = 30 were left non-unanimous). Items with single rater-identified issues were not re-evaluated. The most common issue that was identified by a single rater related to “lacking clinical relevance” (n = 79 or 36%), and n = 25 of these same items were unanimously endorsed as “overpathologizing”, suggesting that there was some overlap in the perception of these two categories. Internal discussion suggested that some of the inconsistent agreement on “lacking clinical relevance” related to differing views of the utility of some items for gathering useful information about the severity of gaming-related problems.

Only items with total agreement from all three raters were included in the analysis. It was reasoned that, given the subjectivity inherent to face validity evaluation (as highlighted by the discrepancies reported above), it was optimal to exclude items that did not elicit unanimous agreement in a small professional group. For a review of this kind, the objective was to present the stronger examples of potential face validity issues. Included items were discussed by the panel after their individual evaluations to check and confirm scores and reasoning. A table summarizing the group's reasoning was created. The evaluation was conducted without referring to other test information that might influence judgment (e.g., practical considerations and psychometric properties, supporting evidence base). For example, whether an item was internally consistent with other items was not relevant. The problematic aspects of items have been underlined, where necessary, for ease of reporting results. All results are presented in complete detail in tables for transparency.

Results

Table 1 presents the item evaluation for 23 tests with face validity issues, including the test name, item wording, and rationale for inclusion. In total, face validity issues were identified across 71 items. Items were grouped into one of three categories that referred to its primary limitation. These limitations included: scope (i.e., “scope creep” or items of questionable relevance); language (i.e., confusing language, unusual wording or syntax); overpathologizing (i.e., pathologizing typical and/or beneficial aspects or consequences of gaming), and other issues (i.e., low level harms and “discordant” items that do not match the symptom area).

Table 1.

Summary of face validity issues identified in gaming disorder (GD) screening tools (n = 23)

Tool	Item*	Reason
A-EQ (Charlton & Danforth, 2007)	I never miss meals because of Internet gaming.	Confusing: Negative phrasing creates a generalization. Skipping one meal yields an affirmative answer.
	I have never used Internet gaming as an escape from socializing.	Confusing: Negative phrasing creates a generalization. Overpathologizing: Escape is not problematic if not associated with negative consequences.
AICA-S (Wölfling et al., 2012)	How strongly are your thoughts involved with playing computer games?	Confusing: The meaning of “thoughts involved” may be unclear.
	How often do you play computer games, although you resolved not to do so or did you play more often or, respectively, longer than you had intended?	Confusing: Multiple parts and complicated logic/syntax may create some uncertainty.
CSAS (Rehbein et al., nd)	Games are on my mind, even at times when I'm not playing.	Overpathologizing: Not unusual to think about hobbies even not involved in them.
	I feel that games are becoming more and more important to me.	Overpathologizing: May refer to becoming more interested in a new hobby.
	Playing games is the best way for me to forget about my problems.	Overpathologizing: Gaming to forget problems is not inherently a problem.
DIA (Ryu et al., 2019)	Do you have a strong desire to do activities such as internet/ Games/SNS? If you want to play internet/Games/SNS, is it hard to tolerate?	Confusing: Multiple parts and complicated logic/syntax may create some uncertainty.
	Do you want to spend more Internet/Gaming/SNS time, find more interesting things, or use better equipment such as cell phones, computers to make you feel as fun as before?	Confusing: Multiple parts and complicated logic/syntax may create some uncertainty.
GAIT (Vadlin et al., 2015)	I often think about games or about gaming, even when I am not playing.	Overpathologizing: Not unusual to think about hobbies even not involved in them.
GAIA (Wong & Hodgins, 2014)	I miss my game character when I am unable to play video games.	Scope: Not clear if distressing or generates low mood. Missing a character may create positive self-reflection. Not all games have distinct or persistent in-game characters (e.g., mobile games, strategy games).
	I often play video games to feel better.	Overpathologizing: Refers to a positive gaming experience.
	I often play video games to release stress.	Overpathologizing: Releasing stress may be a positive or adaptive experience.
	I often play video games to change my mood, relax tension or feel more excited.	Overpathologizing: These experiences may be positive or adaptive.
	I often play video games to forget about my life outside of gaming.	Overpathologizing: Games as entertainment involve escaping from reality as part of the experience.
GAS-7 (Lemmens et al., 2009)	Did you play games to forget about real life?	Overpathologizing: Games as entertainment involve escaping from reality as part of the experience.
GAS-21 (Lemmens et al., 2009)	Do you spend much free time on games?	Overpathologizing: Free time on gaming is unlikely to cause interference.
	Did you play longer than intended?	Overpathologizing: Playing longer than intended may indicate having fun and/or may occur without being harmful.
	Did you play games to forget about real life?	Overpathologizing: Games as entertainment involve escaping from reality as part of the experience.
	Have you played games to release stress?	Overpathologizing: Releasing stress may be a positive or adaptive experience.
	Have you played games to feel better?	Overpathologizing: Refers to a positive gaming experience.
IGD-20 (Pontes et al., 2014)	I never play games in order to feel better.	Confusing: Negative phrasing creates an awkward generalization.
	I have significantly increased the amount of time I play games over last year.	Overpathologizing: May be endorsed by people who have come to enjoy playing games more recently, or rediscovering gaming as a hobby.
	I usually think about my next gaming session when I am not playing.	Overpathologizing: Not unusual to think about hobbies even not involved in them. Lacking clinical relevance: Might refer to required planning for a complex game.
	I play games to help me cope with any bad feelings I might have.	Overpathologizing: May refer to experiences of gaming as a positive coping strategy.
IGDS9-SF (Pontes & Griffiths, 2015)	Do you systematically fail when trying to control or cease your gaming activity?	Confusing: “Systematically” is a complex word that may be unclear when referring to an individual's personal experiences, especially for adolescents.
IGDT-10 (Király et al. 2017)	Have you played to relieve a negative mood (for instance helplessness, guilt, or anxiety)?	Overpathologizing: Relieving negative mood may be positive or adaptive.
IGUESS (Jo et al., 2018)	I play Internet games to remove or avoid negative feelings.	Overpathologizing: Relieving negative mood may be positive or adaptive. The alternative (playing games to experience negative feelings) is unlikely to be endorsed.
Lemmens IGD-9 (Lemmens et al., 2015)	Have you felt unsatisfied because you wanted to play more?	Overpathologizing: It is not uncommon or inherently problematic to desire more time involved in hobbies.
	Have you played games so that you would not have to think about annoying things?	Overpathologizing: May capture temporary distraction or avoidance.
Lemmens IGD-27 (Lemmens et al., 2015)	Have you felt unsatisfied because you wanted to play more?	Overpathologizing: Wanting more may not necessarily be a negative experience.
	Have you played games to forget about your problems?	Overpathologizing: May be adaptive for some users to play games occasionally to forget problems.
	Have you played games so that you would not have to think about annoying things?	Overpathologizing: May be adaptive for some users to play games occasionally to redirect attention.
POGU (Kim & Kim, 2010)	I feel unrestricted when playing online game.	Scope: Refers to a psychological experience (feeling “unrestricted”) outside of known problematic gaming processes. Confusing: Unrestricted has multiple meanings and interpretations. Overpathologizing: May refer to feeling uninhibited which may be positive.
	I feel good and very interested while I play online game.	Overpathologizing: Refers to a positive experience.
	I experience a buzz of excitement while I play online game.	Overpathologizing: These experiences may be positive or adaptive.
	Playing online game is when I most feel pleasure.	Overpathologizing: Refers to a positive experience.
	I find myself saying ‘‘just a few more minutes” when playing online game.	Overpathologizing: Wanting more may not necessarily be a negative experience.
	I feel intimate with people who I know from online game than people in reality.	Scope: May reflect relationship issues that extend beyond problematic gaming.
	I feel that many people playing online game acknowledge my skills than in real life.	Scope: May reflect relationship issues that extend beyond problematic gaming. Overpathologizing: Social reinforcement in large online games may be common for experienced players.
	People I met in online game are easier to understand than real life people.	Scope: May reflect social anxiety, relationship issues or other developmental issues that are more primary than problematic gaming symptoms.
POGQ (Demetrovics et al., 2012)	How often do you play longer than originally planned?	Low level harm (Other): Playing longer than intended may indicate having fun and/or may occur without being harmful.
	How often do you daydream about gaming?	Overpathologizing: Daydreaming may be a positive experience that does not interfere with daily activities.
	How often do you lose track of time when gaming?	Overpathologizing: Immersion in entertainment may be positive.
	How often do you feel time stops while gaming?	Overpathologizing: This may be a positive experience of immersion.
POGQ-SF (Pápay et al., 2013)	How often do you play longer than originally planned?	Low level harm: Playing longer than intended may indicate having fun and/or may occur without being harmful.
	How often do you daydream about gaming?	Overpathologizing: Daydreaming may be a positive experience that does not interfere with daily activities.
	How often do you lose track of time when gaming?	Overpathologizing: Immersion in entertainment may be positive.
PVP Scale (Salguero & Morán, 2002)	When I lose in a game or I have not obtained the desired results, I need to play again to achieve my target.	Overpathologizing: The desire to replay a game may be part of the enjoyment and not inherently problematic.
sIATgaming (Pawlikowski et al., 2013)	How often do you find yourself saying ‘‘just a few more minutes’’ when gaming?	Overpathologizing: The desire to keep playing may be normative and positive.
VASC (Yılmaz et al., 2017)	I forget my problems while playing videogames.	Overpathologizing: Games as entertainment involve escaping from reality as part of the experience.
	In videogames, defeating my enemies/leaping up a level give me pleasure.	Overpathologizing: Refers to a positive gaming experience.
	In videogames, defeating my enemies/leaping up a level makes me feel stronger than my enemies.	Overpathologizing: Refers to a positive gaming experience.
	I think playing videogames is very enjoyable activity.	Overpathologizing: Refers to a positive gaming experience.
	In videogames, defeating my enemies/leaping up a level increase my self-esteem.	Overpathologizing: Refers to a positive gaming experience.
	I do not feel bored when I play videogames.	Overpathologizing: Refers to a positive gaming experience.
	I feel happy when I play videogames.	Overpathologizing: Refers to a positive gaming experience.
	I always talk about videogames with my friends.	Overpathologizing: Refers to a positive social experience related to gaming.
	I make friends via online videogames.	Overpathologizing: Refers to a positive social experience related to gaming.
	I see my videogames/game characters in my dreams.	Overpathologizing: Dreaming about games may be a normal experience that does not cause distress or interference.
	I act like videogame characters in my daily life activities.	Scope: Not all games have distinct characters for player to relate to and imitate. Overpathologizing: Role-playing may be a positive form of identity expression.
VAT (van Rooij, Schoenmakers, Van den Eijnden, Vermulst, & van de Mheen, 2012)	Do you think about gaming, even when you're not online?	Overpathologizing: Not unusual to think about hobbies even not involved in them.
	Do you look forward to the next time you can game?	Overpathologizing: May refer to positive experience of anticipation.
	Do you think you should be gaming less often?	Discordant (Other): Item refers to “preoccupation” but appears to measure personal insight. Also assumes that the behavior is at a sufficiently high level to be problematic, i.e., thinking one should spend less time gaming does not necessarily mean current use is excessive.
CIUS (Meerkerk et al., 2009)	Do you think about the Internet, even when not online?	Overpathologizing: Not unusual to think about hobbies even not involved in them.
	Do you look forward to your next Internet session?	Overpathologizing: May refer to positive experience of anticipation.
	Do you think you should use the Internet less often?	Discordant (Other): Item refers to “preoccupation” but appears to measure personal insight. Also assumes that the behavior is at a sufficiently high level to be problematic, i.e., thinking one should spend less time online does not necessarily mean current use is excessive.
YIAT (Young, 1998a,b)	How often do you prefer the excitement of the Internet to intimacy with your partner?	Scope: May refer to relationship issues that extend beyond problematic gaming. Most activities (not just gaming) may be preferred over intimacy. Some respondents do not have partners.
	How often do you form new relationships with fellow online user?	Overpathologizing: May refer to a positive social gaming experience.
	How often do you check your email before something else that you need to do?	Scope: Refers to email which may not be relevant to gaming activities.
	How often do you find yourself anticipating when you will go online again?	Overpathologizing: May refer to positive experience of anticipation.
	How often do you find yourself saying “just a few more minutes” when online?	Overpathologizing: The desire to keep playing may be normative and positive, and may be socially determined.

Items may take the form of a statement or question. A-EQ: Addiction-Engagement Questionnaire; AICA-Sgaming: Assessment of Internet and Computer Addiction Scale-Gaming; CSAS: Video Game Dependency Scale; DIA: Diagnostic Interview for Internet Addiction; GAIT: Game Addiction Identification Test; GAIA: Game Addiction Inventory for Adults; GAS-7: Game Addiction Scale-7 items; GAS-21: Game Addiction Scale-21 items; IGD-20: Internet Gaming Disorder-20 Test; IGDS9-SF: Internet Gaming Disorder Scale-9 Short Form; IGDT-10; Internet Gaming Disorder Test-10 items; IGUESS: Internet Game Use-Elicited Symptom Screen; Lemmens IGD-9: Internet Gaming Disorder Scale-9 items; Lemmens IGD-27: Internet Gaming Disorder Scale-27 items; POGU: Problematic Online Game Use; POGQ: Problematic Online Gaming Questionnaire; POGQ-SF: Problematic Online Gaming Questionnaire-Short Form; PVP Scale: Problematic Video game Playing Scale; sIATgaming: Short Internet Addiction Test-Gaming; VASC: Video Game Addiction Scale for Children; VAT: Video Game Addiction Test; CIUS-14: Compulsive Internet Use Scale-14 items; YIAT: Young Internet Addiction Test; YDQ: Young Diagnostic Questionnaire.

Table 2 presents a summary of the total face validity issues for each test. Overall, most tests had very few items with face validity issues. The four tests with the highest proportion of face validity issues were the VASC (52%) (Yilmaz, Griffiths, & Kan, 2017), POGU (45%) (Kim & Kim, 2010), POGQ-SF (25%) (Papay et al., 2013), and the YIAT (25%) (Young, 1998a,b). It should be noted that these percentages encompass items with different concerns and thus should not be considered as comparable ratings. Further, certain issues (e.g., overpathologizing) may be more serious than others (e.g., questionable clarity) and therefore these ratings are unweighted.

Table 2.

Overall evaluation of the face validity of gaming disorder (GD) screening tools (n = 29)

Face validity issues	A-EQ	AICA-Sgaming	BAM-VG	C-VAT2.0	CSAS	DIA	GAIT	GAIA	GAS-7	GAS-21	IGD-20	IGDS9-SF	IGDT-10	IGUESS	Lemmens IGD-9	Lemmens IGD-27	Petry IGD	PIE-9	POGU	POGQ	POGQ-SF	PVP	SCI-IGD	sIATgaming	VASC	VAT	CIUS-14	YIAT	YDQ	Total
Scope	–	–	–	–	–	–	–	1	–	–	–	–	–	–	–	–	–	–	4	–	–	–	–	–	1	–	-	2	–	8
Language	2	2	–	–	–	2	–	–	–	–	1	1	–	–	–	–	–	–	1	–	–	–	–	–	–	–	–	–	–	9
Overpathologizing	1	–	–	–	3	–	1	4	1	5	3	–	1	1	2	3	–	–	6	3	2	1	–	1	11	2	2	3	–	56
Other	–	–	–	–	–	–	–	–	–	–	1	–	–	–	–	–	–	–	–	1	1	-	–	–	-	1	1	–	–	5
Total issues	3	2	–	–	3	2	1	4	1	5	5	1	1	1	2	3	–	–	10	4	3	1	–	1	12	3	3	5	–	76
Total items affected	2	2	–	–	3	2	1	4	1	5	4	1	1	1	2	3	–	–	8	4	3	1	–	1	11	3	3	5	–	71
Proportion of test* (%)	16	13	–	–	17	20	6	15	14	18	20	11	10	11	22	11	–	–	40	22	25	11	–	8	52	14	14	25	–	17%a

Number of items with face validity issues/Total test items.

Total Face validity issues/Total items (i.e., 71/417). Total A-EQ: Addiction-Engagement Questionnaire; AICA-Sgaming: Assessment of Internet and Computer Addiction Scale-Gaming; BAM-VG: Behavioral Addiction Measure for Video Gaming (Sanders and Williams, 2016); CSAS: Video Game Dependency Scale; C-VAT 2.0; Clinical – Video Game Addiction Test 2.0; DIA: Diagnostic Interview for Internet Addiction; GAIT: Game Addiction Identification Test; GAIA: Game Addiction Inventory for Adults; GAS-7: Game Addiction Scale-7 items; GAS-21: Game Addiction Scale-21 items; IGD-20: Internet Gaming Disorder-20 Test; IGDS9-SF: Internet Gaming Disorder Scale-9 Short Form; IGDT-10; Internet Gaming Disorder Test-10 items; IGUESS: Internet Game Use-Elicited Symptom Screen; Lemmens IGD-9: Internet Gaming Disorder Scale-9 items; Lemmens IGD-27: Internet Gaming Disorder Scale-27 items; Petry IGD: Petry et al. consensus statement on IGD criteria; PIE-9: Personal Internet Gaming Disorder Evaluation-9 items (Pearcy et al., 2016); POGU: Problematic Online Game Use; POGQ: Problematic Online Gaming Questionnaire; POGQ-SF: Problematic Online Gaming Questionnaire-Short Form; PVP Scale: Problematic Video game Playing Scale; SCI-IGD: Structured Clinical Interview-Internet Gaming Disorder (Koo et al., 2017); sIATgaming: Short Internet Addiction Test-Gaming; VASC: Video Game Addiction Scale for Children; VAT: Video Game Addiction Test; CIUS-14: Compulsive Internet Use Scale-14 items; YIAT: Young Internet Addiction Test; YDQ: Young Diagnostic Questionnaire.

Based on citation records, these four tests have been employed in 71 empirical studies (or, 21% of empirical studies that have employed the 29 tests). This literature primarily involved the YIAT (n = 62 studies), which includes 31 studies that have used the tool to assist in identifying clinical samples, followed by the POGQ-SF (6 studies), POGU (2 studies), and VASC (1 study). The authors of these studies have generally not expressed any concerns about these tests' validity, or their capability in identifying cases of GD, such as any inconsistency between test score and psychiatric evaluation or other adjunctive measure. Overall, the majority of available tests in the literature appear to be considered valid and reliable by the researchers who employed them, with the exception of Snodgrass et al. (2018) who reported that IGDS9-SF scores were not associated with CTRA, a leukocyte gene expression profile activated by chronic stress. The authors stated that “this may reflect the ineffectiveness of this measure…to adequately capture the experience of problematic gaming” (p. 11).

Scope Issues

Four tests (8 items) were identified in this category (i.e., GAIA, POGU, VASC, and YIAT). For example, the POGU refers to feeling intimate with other online players and feeling acknowledged by online players. Similarly, the YIAT refers to preferring gaming instead of intimacy with a partner, which is a forced choice question that may not necessarily centrally relate to gaming as the problem (i.e., this may relate to problems in the relationship unrelated to gaming). These items could be referring to psychological issues that fall outside the general category of addictive behavior and often may not be relevant. Similarly, the VASC refers to acting like a video game character in real life which may not be relevant to problematic gaming and/or to games that do not involve distinct playable characters (e.g., strategy games).

Suggested modifications

These tests may be improved by modifying items about relationship difficulties to be specifically related to gaming. Some tests could exclude items that refer to behaviors that are not included in the nomenclatural definitions of GD or, alternatively, score or weight these items separately (e.g., the C-VAT 2.0 and AICA-Sgaming do not count their items on “craving” and “daily time spent gaming”, respectively, toward total addiction scores).

Language Issues

Another important issue was the use of items that employed unclear or unusual wording (9 items, 6 tests). Unclear and vague phrasing was evident in items with multiple parts (e.g., DIA: “Do you want to spend more gaming time, find more interesting things, or use better equipment such as cell phones, computers to make you feel as fun as before?”) and in short items with ambiguous language (e.g., POGU: “I feel unrestricted when playing online game”; A-EQ: “I feel a sense of power when I am Internet gaming”). The internal logic of some items was affected by unnecessary negative phrasing (e.g., IGD-20: “I never play games in order to feel better”, “I know my main daily activity has not been negatively affected by my gaming”). The IGDS9-SF uses the adverb “systematically [fail]” which refers to a fixed plan to fail, usually in relation to a group or organization, which may be inappropriate for capturing unregulated behavior among individuals. The word “systematically” may also be less comprehensible for adolescents, a population that is often invited to complete gaming-related measures.

The wording of some tests/items might potentially be improved by reducing unnecessary or inconsistent words, including terms that refer to abstractions (e.g., “sense of power”). Replacing negative adverbs of frequency (“I never play…”) with positive phrasing (“I play”) may improve item sensitivity and provide a clearer measure of frequency.

Overpathologizing Gaming

A major problem identified across most tests (56 items, 20 tests), and which has been raised by researchers who criticize gaming disorder as a mental disorder (Aarseth et al., 2017; Bean, Nielsen, Van Rooij, & Ferguson, 2017; Billieux, Schimmenti, Khazaal, Maurage, & Heeren, 2015; Przybylski, Weinstein, & Murayama, 2016; van Rooij et al., 2018), is the potential for tests to “overpathologize” normal and adaptive gaming behaviors and consequences. The VASC was found to include 11 items that refer to the benefits of gaming (e.g., “I think playing video games is a very enjoyable activity”; “I feel happy when I play video games”; “I always talk about video games with my friends”). These items are counted toward a total score, based on 21 items, to indicate “possible addiction” (Yılmaz et al., 2017, p. 873). Further, the VASC has a cut-off score of 90 out of a possible 105, and therefore the test requires that individuals endorse items referring to benefits of gaming in order to be considered “addicted” (i.e., endorsing only the harm-related items [e.g., “I cannot resist playing videogames even if it negatively affects my life”; “I have sleeping problems due to playing videogames”] is insufficient to be considered a problematic gamer). Similar problems of pathologizing the benefits of gaming arise in other tests to a lesser extent (i.e., these items are not required to meet the cut-off but they would nevertheless inflate total scores in the distribution). Some examples include: GAIA: “I often play video games to release stress”; POGU: “I feel good and very interested while I play online game”; VAT: “Do you look forward to the next time you can game?”.

Some items appear to be based on an assumption that having any level of interest in games, or valuing games as a hobby, is inherently risky (e.g., CSAS: “I feel that games are becoming more and more important to me”; GAS-21: “Do you spend much free time on games?”). Immersion effects and losing track of time while engaged in gaming, which may be a consequence of normal enjoyment or “flow” (Csikszentmihalyi, 1997), is similarly considered problematic (e.g., POGQ: “How often do you feel time stops while gaming?”, “How often do you lose track of time when gaming?; POGU: “I feel a buzz of excitement while I play online game”). The IGD-20 conflates a factual statement about the time requirement associated with goals in games with a symptom of addiction (i.e., “I often think that a whole day is not enough to do everything I need to do in-game”), which would be true of many modern video games, particularly open-ended and service-based games with unlimited goals and opportunities.

Many of these tests may be improved by simply removing certain items, or instead using such items in a new tool that measures the benefits of gaming. Some items may be modified effectively by including an appropriate qualifier that specifies distress or functional impairment (e.g., “Do you prioritize and spend almost all of your time on video games?” rather than “Do you spend much free time on games”).

Other Issues

Another limitation of the reviewed tests was items that refer to relatively mild versions of symptoms or capture behaviors that may be unlikely to reflect true harm. These items may be useful in identifying emerging problematic behaviors (i.e., ensuring the test has sufficient sensitivity to capture a range of problematic behaviors) but may have less utility in a brief test designed to capture GD only. These “low-risk” items may be problematic for a test, in the same way as the “overpathologizing” items, when they are weighted equivalently to items that refer to severe types of behaviors or consequences (e.g., failing school, relationship breakdown). In addition, some items appear to have low thresholds for endorsement (i.e., low item difficulty) due to their wording or scoring approach, such as when individuals who have experienced any instance of the behavior (e.g., once within the last 12 months) would be considered the same as more severe user profiles. For example, the A-EQ refers to “never miss meals” and being “sometimes late” due to gaming. Similarly, the VAT and CIUS refer to “rush through homework”, which may be of minor consequence even when occurring frequently, thus this item may capture an opportunity cost rather than true harm (see Delfabbro & King, 2019).

Some items refer to a component of addiction (e.g., preoccupation, withdrawal, escape) but lack the essential aspects of the description that specify clinical severity. For example, the CSAS, GAIT, IGD-20 and POGU refer to thinking about games at times when “not playing” (NB: response categories of these tests refer to agreement not frequency), which may not capture the phenomena of having constant thoughts or predominantly thinking about games (DSM-5 concept: preoccupation) or the precedence of gaming-related thoughts (e.g., planning, reliving, anticipating) over other non-gaming cognition (ICD-11 concept: escalating priority). Similarly, the POGQ and POGQ-SF refer to “daydreaming” about games, which may not be synonymous with preoccupation.

Along similar lines, the IGD-9/27 refers to feeling “unsatisfied” and wanting to “play more” and gaming to not “think about annoying things”. These terms may potentially capture gaming-related withdrawal (Kaptsis, King, Delfabbro, & Gradisar, 2016), feelings of inadequacy (King et al., 2018a,b), and a reliance on games for mood relief (Petry et al., 2014). However, these items are scored dichotomously, which may make them too sensitive or easy to endorse (i.e., generate false positives) by capturing gaming experiences typically reported by regular gamers when reflecting on the past year of activities (Colder Carras, Porter, van Rooij, King, Lange, & Labrique, 2018; Delfabbro & King, 2019). Another example of oversensitivity is an item that refers to thinking “just a few more minutes” when playing games, which is used in the POGU, sIATgaming, and YIAT20. This item may potentially capture loss of control in some cases, but it may also refer to poor time management skills or procrastination.

These tests may be improved by rewording items that refer to GD symptoms to better capture the severity, frequency and duration of behaviors (e.g., see Ko et al. 2019). Minor or temporary negative consequences of gaming could be weighted lower for scoring purposes or relegated to a measure of “emerging/low risk” problematic gaming instead of addiction.

Discussion

This evaluation found that most items in GD tools demonstrated satisfactory face validity and were consistent with either/both the DSM-5 or ICD-11 classifications. However, many tools contained at least one item that may pathologize normal gaming behaviors. Such items refer to basic changes in mood when gaming, a desire to play or continue playing games, and experiencing immersion when gaming. These issues may relate to the difficulty in adapting concepts from substance-based addictions (e.g., urges, craving, and tolerance) to activities like gaming where changes in hedonic balance are a normal part of the activity. These potential test limitations have implications for scale development in the field of behavioral addictions, which appears likely to expand into new areas following calls for a wider range of repetitive behaviors to be studied (Potenza, Higuchi, & Brand, 2018). As already flagged by some authors (Billieux et al., 2015; Starcevic, Billieux, & Schimmenti, 2018), researchers may be inclined to adapt existing measures by substituting keywords to study these conditions, thereby perpetuating these face validity issues in new tests for other behaviors.

These face validity issues may be useful to consider for brief evaluation of other measures, particularly in the field of behavioral addictions where there has been debate on whether some proposed conditions (e.g., online social networking addiction, smartphone addiction) may be overextending or misappropriating the concept of addiction (Billieux et al., 2015). It is noteworthy that this evaluation provided support for the items contained in Petry et al. (2014) IGD measure, which was developed by an international group of 14 experts. These items had also been translated into 10 languages. It may be speculated that this additional attention may have assisted in ensuring that items were adequately aligned with the DSM-5 criteria.

This review has some noteworthy limitations. It should be emphasized that any items identified in this evaluation may not be problematic psychometrically (i.e., an item identified as “overpathologizing” may actually still perform adequately). This evaluation was based on a small panel of qualified experts; however, these ratings should be considered as justified opinions (e.g., akin to the peer review process). Other raters may not necessarily agree with the panel's determination. For this reason, we have provided all items and our corresponding reasoning in Table 1 for transparency and to enable others to consider and re-evaluate. Other limitations of this evaluation were the lack of comparison with available psychometric data for each test (to reduce introducing bias into ratings) and the lack of examination of non-English versions of items. This evaluation also did not account for the typical context and purpose for each test (e.g., use in clinical settings or prevalence studies), which may have introduced some bias, and thus did not account for the constraints and requirements of each context.

On a final note, this review shows that the study of GD has led to a proliferation of tools over time. Many of these instruments are the “same but different” (i.e., similar concepts with slightly different practical execution). Every year since the release of the DSM-5 in 2013, between 2 and 5 new measures for GD have been created that are mostly similar in length and content to existing ones. Many research teams are working in “silos” by continuing to use and create tools that may not be entirely comparable to each other in some respects. Thus, while the DSM-5 formulation of GD has received strong support and research investment, there is also an isolationist quality to the field's collective epistemic uniformity. It seems difficult in this current situation with so many competing tools to make progress or to turn back. A gold standard tool may help to unify the field and enable, for example, more consistent estimates of prevalence across countries. It is hoped that this review provides some practical avenues to improve current measurement approaches and prompts critical reflection on the essential elements of the formulation of gaming as a potential disorder (King, Koster, & Billieux, 2019). Finally, in demonstrating that many tests have items that appear to pathologize normal gaming behaviors, this evaluation highlights the challenges of screening effectively for a mental disorder that is thought to arise within the context of a normal recreational activity.

Authors' contributions

The first author designed the review and conducted literature searches and provided summaries of the identified content. The second and third authors contributed to the face validity analysis. The first author wrote the first draft, and all authors contributed to and approved the final manuscript.

Funding

This work received financial support from a Discovery Early Career Researcher Award DE170101198 funded by the Australian Research Council.

Declaration of interest

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.