Serious games for smoking prevention and cessation: A systematic review of game elements and game effects.

OBJECTIVE: Serious health games might have the potential to prevent tobacco smoking and its health consequences, depending on the inclusion of specific game elements. This review aimed to assess the composition of serious games and their effects on smoking initiation prevention and cessation and behavioral determinants.
MATERIALS AND METHODS: We systematically searched MEDLINE, Embase, PsycINFO, and Web of Science for publications that evaluated serious games aimed at changing smoking behavior or behavioral determinants. A taxonomy by King et al was used to classify game elements.
RESULTS: We identified 15 studies, evaluating 14 unique serious games. All games combined multiple game elements (mean 5.5; range, 3-10). Most frequently used were general and intermittent rewards, theme and genre features, and punishments. Six studies on smoking prevention together assessed 20 determinants and found statistically significant positive effects for 8 determinants (eg, attitude, knowledge, intention). Of 7 studies on smoking cessation, 5 found positive, statistically significant effects on smoking cessation or status. These studies found statistically significant positive effects for 6 of 12 determinants (eg, self-efficacy, attitude, intention). The majority of included studies had poor or fair methodological quality, lacked follow-up measures, and had fixed (as opposed to free, on-demand) play sessions.
CONCLUSIONS: Serious games included multiple types of game elements. The evidence from a number of studies suggests that games may have positive effects on smoking-related outcomes, particularly smoking cessation. However, as most studies had important methodological limitations, stronger designs are needed to demonstrate, quantify, and understand the effects of serious games.

INTRODUCTION

Serious games, defined as games with a primary purpose other than pure entertainment, evolved rapidly in the last decade. Such games can be used to educate, communicate, and improve health. The latter can target disease prevention, diagnosis, management, and treatment in a great variety of health behaviors and diseases. Previous reviews suggested that serious games had small positive effects on health-related behaviors and their determinants. For example, exergaming, which requires rigorous physical exercise, was effective in increasing or substituting physical activity in adults in the short term, and to decrease childhood obesity. Similarly, serious games were found to be effective in psychotherapy and showed positive effects on determinants of sexual health behavior.

While many studies have assessed the overall effectiveness of serious games, the composition of serious games (ie, inclusion of game elements) is mostly unattended. Serious games commonly combine a series of game elements, which are features that are “found in most (but not necessarily all) games, readily associated with games, and found to play a significant role in gameplay.” Many of such elements bear correspondence with behavior change techniques or learning mechanisms,, such as feedback, rewards, and social influences. Specific game elements may strengthen players’ engagement, feelings of progress, and motivation to finish.

Serious games have been used increasingly to prevent initiation among adolescents or to motivate and support adult smokers to quit smoking. Yet, no review has systematically assessed the cumulated evidence regarding the effects of serious games on smoking-related outcomes, including the strength of such evidence. In addition, there is no systematic overview of which game elements are included in serious games. This review addressed the following 2 specific objectives: (1) assess what game elements are included in serious games aimed at smoking prevention and cessation and (2) assess the effects of those serious games on smoking-related outcomes.

MATERIALS and METHODS

We registered our systematic review protocol a priori in the PROSPERO register (ID CRD42018097430) and we followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines for reporting our systematic review.

Inclusion and exclusion criteria

We searched for original research articles that evaluated serious games targeting tobacco smoking prevention, cessation, or behavioral determinants. We structured our inclusion and exclusion criteria according to the PICOS (Population, Interventions, Comparisons, Outcomes, Study designs) model. Regarding population, we included all populations; thus, no specific populations were excluded in this review. Regarding interventions, we included serious games (ie, games with a primary purpose other than pure entertainment) and excluded other games, as well as interventions lacking game elements. We included studies with a serious game as intervention targeting tobacco smoking (eg, cigarettes, cigars, pipes), whereas studies on smokeless tobacco (eg, sucking, chewing, and snuffing tobacco) or on substances other than tobacco (eg, cannabis) were excluded. We included both digital and analog games. Regarding comparisons, we included all comparisons; thus, no specific comparisons were excluded in this review. Regarding outcomes, we included studies that evaluated effects of a serious game on smoking-related outcomes and excluded studies only reporting usability evaluations and player experiences. Regarding study designs, we included studies with the following research designs: randomized controlled trials (RCTs), nonrandomized trials, before-after studies without a control group, case-control studies, and cohort studies.

Moreover, we included only 1 study (most comprehensive one) when multiple studies on the same serious game in the same population were found. We included studies in English or Dutch and excluded studies in other languages. We did not apply restrictions on publication year.

Search strategy

Two reviewers (M.E.D., S.v.S.) and a librarian (J.G.D.) systematically searched MEDLINE, Embase, PsycINFO, and Web of Science from inception for relevant published articles. Web of Science includes all ACM Digital Library and IEEE Xplore publications. The World Health Organization International Clinical Trials Registry Platform and Cochrane Library intervention databases were searched to identify (un)published trials, articles, interventions, or reviews. The search was complemented with hand-searching studies in relevant reviews as well as snowballing by means of searching for relevant studies in the reference list of included articles. The search query mainly combined the concept of serious games with the concepts of smoking and health promotion (see Supplementary Appendix A for the full search query). Conceptually, the systematic search strategy can be summarized as follows: (([serious gaming] and ([health promotion] OR [effect measures])) NOT [animal studies]) OR [trials]. In order to enhance precision of search results, we used VOSviewer to visually identify potentially irrelevant terms eligible for NOTing out in our search strategy. This led to disambiguation (eg, excluding irrelevant records on “game” as in “wild game”)., Other than these terms, we have been highly conservative with narrowing our search strategy based on terms that showed up in the VOSviewer word cloud. See Supplementary Appendix B for the corresponding VOS network visualization.

Study selection

After removing duplicates, 2 reviewers (S.v.S., M.E.D.) independently screened studies based on title and abstract and assessed the remaining articles full text for eligibility. Rayyan QCRI (Rayyan, Doha, Qatar) and EndNote (Clarivate Analytics, Philadelphia, PA) were used for screening and data management. Interrater reliability was assessed before a consensus meeting to discuss any disagreements. If no consensus was reached, a third reviewer (M.P.F.) was consulted. Interrater reliability of the screening based on the title and abstract resulted in moderate agreement (Cohen’s kappa = 0.44) and the interrater reliability of the screening based on the full text resulted in moderate agreement (Cohen’s kappa = 0.5).

Quality appraisal

Two reviewers (M.E.D, M.P.F.) independently appraised the quality of each included study by using the most suitable tools of the Study Quality Assessment Tools of the National Institutes of Health. Studies were not excluded based on the quality assessment. Quality appraisal yielded a substantial agreement (Cohen’s kappa = 0.67). Interrater reliability was assessed before a consensus meeting in which reviewers discussed, determined, and reported the methodological strength of each study. If no consensus was reached, a third reviewer (S.v.S.) was consulted.

Data extraction and synthesis

A data extraction form was developed based on the Cochrane data collection form for RCTs and non-RCTs. The form was pilot-tested on the first 5 studies and adjusted where needed. Two authors (M.E.D., S.v.S.) extracted data in parallel and then checked results against each other. Any disagreements were discussed, and consensus was reached without the need of a third reviewer (M.P.F.). The following items were extracted: full reference, country, study design, setting, participant characteristics, interventions (serious game) characteristics, included game elements, and measures and effects on smoking-related outcomes. We classified the effects as positive (ie, outcomes improved), negative (ie, outcomes worsened), no difference (ie, outcomes did not change or barely changed), or mixed (ie, outcomes both improved and worsened), as well as the statistical significance for each of these effects (if reported).

Game elements were extracted using a taxonomy of game elements by King et al, after using 3 different taxonomies,, for the first 5 studies. This taxonomy was selected as the most relevant taxonomy since all game elements could be categorized in 5 categories: (1) narrative and identity features (eg, interactive storytelling), (2) social features (eg, social aspects of game playing), (3) manipulation and control features (eg, user input influencing in-game outcomes), (4) reward and punishment features (eg, the ways in which players win and lose in video games), and (5) presentation features (eg, the visual and auditory presentation of video games). Every category consists of multiple subfeatures.

Corresponding authors of included studies were contacted twice with a request for additional materials detailing the intervention content when no published materials were available. When not available or no response was received, the intervention description in the published included article was used.

RESULTS

Study inclusion

Our PRISMA flow diagram (see Figure 1) shows that the database search yielded 4003 results, including 1018 duplicates. After removal of duplicates, 2985 articles were screened based on title and abstract. We excluded 2865 studies in this screening, mainly because the studies did not report on a serious game as intervention for smoking prevention or cessation. The remaining 120 studies were evaluated as full text. Hereafter, we excluded 105 studies, and once more, the main reason for exclusion was that the studies did not report on a serious game. No studies were added from other sources, resulting in the inclusion of 15 studies in total, evaluating 14 unique serious games.

Figure 1.

PRISMA flow diagram. ICTRP: International Clinical Trials Registry Platform; WHO: World Health Organization.

Study characteristics and quality

Table 1 presents details on study characteristics. All studies were in English and published from 2008 to 2018, except 1 study published in 1999. The majority of studies were conducted in North America (n = 11), with the other studies conducted in Europe (n = 4). Seven studies were (randomized) controlled trials, 5 were pre-post studies, 2 studies had a between- or within-subjects factorial design,, and 1 was a cross-sectional study. If reported, the clinical and laboratory setting were most common, followed by “not bound to a setting,” and school setting. The target populations were children, adolescents or young adults in 40% of the studies, and the remaining studies covered a broad age range. Sample sizes ranged from 25 to 1698, in which two-thirds of the studies had 25-91 participants. Except for 1 study that only studied women all studies included both women and men, with 1 study not reporting on gender. Six of 7 studies that reported race and ethnicity included a mixture of races and ethnicities. The 5 studies that reported on educational level included both participants that attended high school or lower education and participants that attended college or university. The mean smoked cigarettes per day was 17.76 among the 7 studies that reported this. Last, study quality scores (good, fair, poor) were evenly distributed over studies (see Supplementary Tables 1-3). Most studies scored fair or poor due to unreported information that is crucial to determine methodological quality, sample sizes that would not detect effects with enough power, high dropout or loss to follow-up, and unblinded stakeholders.

Table 1.

Study characteristics

Study	Quality scorea	Country	Design	Setting	Participants	Sex	Age (y)	Ethnicity/race	Educational level	Cigarettes smoked/d
An 201319	Good	United States	Randomized controlled trial	Not bound to a setting	Young adult smokers (n = 1698)	Female 72.44% Male 27.56%	24.07 (NR), NR	Hispanic/Latino 10.95%; Non-Hispanic/Latino 89.04% White 73.91%; black or African American 10.36%; other 8.60%; multiple 7.13%	High school or less 32.63% Some college + 2-year degree 50.35% 4-year degree or more 17.02%	19.82 (NR)
Bordnick 201220	Fair	United States	Randomized controlled trial	Clinic	Nicotine-dependent treatment-seeking cigarette smokers (n = 86)	Female 47.7% Male 52.2%	Experimental group 47.9 (10.4), NR Control group 46.2 (8.4), NR	Experimental group: Caucasian 5%; African American 90%; Hispanic 5% Control group: Caucasian 16%; African American 80%; Hispanic 4%	NR	Control group 26.4 (10.0) Experimental group 24.5 (6.3)
Brinker 201621	Poor	Germany	Cross sectional study	Secondary schools	Grade 7 students (n = 125)	Female 39.2% Male 60.8%	12.75 (NR), NR	NR	NR	NR
DeLaughter 201622	Fair	United States	Pre-post study	Laboratory	In- or outpatient smokers (n = 30)	Female 67% Male 33%	NR (NR), 19-65+	White/Non-Hispanic or Latino 100%	Grades 9-11 3% Grade 12/GED 27% College 1-3 years 40% College 4 years or more 30%	13.8 (10.0)
Duncan 201823	Good	United States	Pre-post study	NR	Adolescents never smokers (n = 25)	Female 36% Male 64%	11.56 (0.77), NR	African American 72%; American Indian/Native American 12%; Hispanic 12%; other 4%	NR	0
Girard 200924	Poor	Canada	Randomized controlled trial	Clinic	Adult smokers (n = 91)	Female 57% Male 43%	44 (11), NR	NR	NR	>21 (45% of population)
Gordon 201725	Fair	United States	Pre-post study	Not bound to a setting	Female smokers (n = 73)	Female 100%	39.1 (13.1), NR	African American 16.4%; Asian 1.4%; Caucasian 72.6%; multirace/other 9.6%; other 5.5%; Hispanic/Latino 8.2%	<High school 4.1% High school graduate or equivalent 30.1% Some college 37.0% College graduate 21.9% Post-college 6.8%	16.0 (13.2)
Khazaal 200826	Good	Switzerland	Pre-post study	Psychiatric hospital	Adult smokers that are inpatients of a psychiatric hospital (n = 51)	Female 46% Male 54%	43.7 (12.8), NR	NR	NR	23 (14)
Khazaal 201327	Poor	Switzerland	Randomized controlled trial	NR	Adult smokers in early stages of change (n = 240)	Female 65% Male 35%	31.5 (12.1), NR	NR	High school or lower 10.9% Professional school 31.35% University degree 57.75%	15 (7.9)
Metcalf 201828	Poor	United States	(stratified, wait-list) Controlled Trial	Game setup location (ie, small office)	Adults recently quit smoking (n = 61)	Female 45.90% Male 52.45%	NR (NR), NR	Asian 3.3%; African American 39.3%; white 45.9%; other 3.3%; multiracial 3.3%; prefer not to answer 3.3%	NR	NR
Nemire 199929	Poor	United States	Randomized controlled trial	NR	Grade 7 students (n = 72)	Female ≈ Male	NR (NR), NR	NR	NR	NR
Parisod 201830	Good	Finland	Randomized controlled trial	Schools and not bound to a setting	Early adolescents (n = 151)	Female 52.32% Male 47.68%	11, 10-13	NR	NR	0.67% is current smoker
Rath 201531	Good	United States	Randomized controlled trial	Not bound to a setting	Youth and young adults (n = 689)	Female 48% Male 52%	NR (NR), 13-24	Non-Hispanic white 59.8%; Non-Hispanic black 14.2%; other non-Hispanic 7.8%; Hispanic 18.2%	Some high school or less 56.8% Some college or more 43.2%	58% has never tried cigarettes
Song 201332	Fair	United States	2 × 2 between-subjects factorial design	Laboratory	Social smokers (n = 62)	Female 38.7% Male 61.3%	22.05 (3.28), NR	NR	NR	Frequency of smoking: once a week (median)
Xu 201433	Fair	United States	2 × 2 within-subjects factorial design	Laboratory	Couples in long term relationships of which at least 1 is a smoker (n = 40)	NR	24.10 (6.17), 19-42	NR	NR	11.56 (6.01)

Values are mean (SD), range; mean (SD); or median, range, unless otherwise indicated.

NR: not reported.

See Supplementary Tables 1-3 for detail.

Intervention characteristics

Table 2 presents details on intervention characteristics. Of the 15 included studies, 14 studies evaluated unique serious games. All games were digital games, with the exception of 2 board games., Khazaal et al, reported on the same game in different populations. Most serious games were aimed at smoking cessation (n = 9), whereas 6 aimed at smoking initiation prevention. About a quarter of studies did not report on the theoretical basis of their serious game. If reported, cognitive behavioral therapy and theory of planned behavior were most often the basis of the game. A third of the serious games were played once, for a range of—if reported—2-90 minutes. In addition, more than half of the serious games were played multiple times in a range of—if reported—2-10 weeks. Two serious games had free usage,, for 1 of these games, it was reported that the serious game was played twice as much as the nongamified website. Three serious games were assessed alongside a cointervention.,,

Table 2.

Intervention characteristics

Study ID	Health objective	Intervention	Availabilitya	Type	Theoretical basis	Intervention group(s)	Game play	Control group(s)
An 201319	Smoking cessation	“RealU2” (trial name) aimed at abstinence from cigarette use among young adult smokers.		Digital	Social cognitive theory; theory of reasoned action and planned behavior; self-determination theory	Receiving individually tailored health messages Receiving individually tailored health message + online peer support	Once per week, for 6 wk	Untailored general interest messages
Bordnick 201220	Smoking cessation	“Virtual Reality Skills Training” aimed at development and practice of coping skills for smoking cessation.		Digital	Cognitive behavioral therapy	Using virtual reality skills training Cointervention: nicotine replacement therapy (nicotine patch)	Once per week (1 h), for 10 wk	Nicotine replacement therapy (nicotine patch) only
Brinker 201621	Smoking prevention	“Smokerface” aimed at smoking prevention.	Google Play; App Store	Digital	Theory of planned behavior	Using photo-aging app	2 min, once	Not applicable
DeLaughter 201622	Smoking cessation	“Crave-Out” aimed at managing cravings during a quit attempt.		Digital	NR	Playing a distraction/motivation game (Crave-Out)	10 min, once	Not applicable
Duncan 201823	Smoking prevention	“smokescreen” aimed at smoking prevention.	App Store	Digital	A logic model of cognitive and motivational variables	Playing a role-playing videogame (smokeSCREEN)	4 sessions (1 h), in 2 wk	Not applicable
Girard 200924	Smoking cessation	“Virtual arm” (VR therapy) aimed at reducing tobacco addiction.		Digital	Cue exposure therapy	Playing in a virtual environment crushing cigarettes. Cointervention: psychosocial smoking cessation treatment program	Once per week (30 min), for 4 wk	Balls instead of cigarettes Cointervention: psychosocial smoking cessation treatment program
Gordon 201725	Smoking cessation	“See Me Smoke-Free” aimed at smoking cessation among women.		Digital	NR	Using “See Me Smoke-Free” application	For 1 mo	Not applicable
Khazaal 200826	Smoking cessation	“Pick-Klop” aimed at changing the attitude toward tobacco addiction among smokers.		Analogue	Cognitive behavioral therapy	Playing the board game “Pick-Klop,” a group smoking cessation game	Once, 1-1.5 h	Not applicable
Khazaal 201327	Smoking cessation	“Pick-Klop” aimed at changing the attitude toward tobacco addiction among smokers.		Analogue	Cognitive behavioral therapy	Playing the board game “Pick-Klop,” a group smoking cessation game	Once per week (1 h 30 min), for 2 wk	Psycho-education Waiting list
Metcalf 201828	Smoking cessation	“Take control” aimed at providing support for those in recovery from tobacco addiction.		Digital	Cue exposure therapy, extinction therapy, virtual reality therapy, cognitive behavioral therapy	Playing the virtual reality cue refusal video game “Take Control”	3 sessions (15-30 min), in 4 wk	Waiting list
Nemire 199929	Smoking prevention	“Virtual Environment technology” aimed at smoking prevention among adolescents.		Digital	NR	Using virtual environment technology Cointervention: Discussing personal goals + workbook material	Once per week (50 min), for 8 wk	Life skills training
Parisod 201830	Smoking prevention	“Fume” aimed at improving tobacco related health literacy.	Google Play; App Store	Digital	Health literacy determinants	Playing mobile health game “Fume” Using nongamified website	1 session of guided training (20 min), free usage in 2 wk	No intervention
Rath 201531	Smoking prevention	“Flavour Monsters” aimed at improving tobacco related attitudes and beliefs		Digital	Theory of planned behavior	Playing “Flavour Monsters” with tobacco related information	±4.5 h in several sessions	Not applicable
Song 201332	Smoking prevention	“Super Smoky” aimed at educating about the risk of social smoking.		Digital	Possible selves	Playing with a self-avatar and future face condition Playing with a self-avatar and current face condition Playing with a someone else’s avatar and future face condition Playing with a someone else’s avatar and current face condition	Once	Not applicable
Xu 201433	Smoking cessation	Cooperative games (not named) aimed at undermining nicotine cravings.		Digital	NR	Playing games with self-expanding and cigarette cue present Playing games with self -expanding and cigarette cue absent Playing games without self-expanding and cigarette cue present Playing games without self-expanding and cigarette cue absent	Once (11 min)	Not applicable

NR: not reported.

Availability on October 1, 2019.

Game elements

Table 3 presents the results on included game elements. None of the studies explicitly reported any game elements, and consequently, we extracted game elements from the game description in the articles. All serious games included a combination of game elements from multiple categories (mean 5.5 per serious game; range, 3-10). Most game elements were in the category “reward and punishment features” (n = 32), next was “narrative and identity features” (n = 20), followed by “manipulation and control features” (n = 12) and “presentation features” (n = 11), and last was “social features” (n = 8). On game element level, the most frequently included game elements were general rewards types (n = 11). The next most included game elements were theme and genre features (n = 9) (eg, interacting with virtual lungs). The next most included game elements (n = 9) were punishments (eg, score reduction). Next, graphic and sound features (eg, playing within fictional medieval castle) (n = 8) and intermittent rewards (n = 7) (eg, varying difficulty of scenarios) were the most included game elements. Game elements that were not included were support networks (eg, internet forums to assist playing), negative rewards (eg, repairing items), pay-out intervals (eg, rewarded instantly for playing), franchise (eg, trademarked names), and explicit content (eg, violence).

Table 3.

Game elements categorized according to taxonomy of King et al

	Categories	Narrative and identity features			Social features				Manipulation and control features				Reward and punishment features									Presentation features
Studies	Game elements	Avatar creation (eg, animation of 3D selfie21)	Storytelling device (eg, health messages dialog19)	Theme and genre (eg, virtual lungs29)	Social Utility (eg, group comments on questions26)	Social formation (eg, cooperative play33)	Leader board (eg, scores30)	Support network	User input (eg, pick up and crush cigarette24)	Save (eg, checkpoint at each level31)	Player management (eg, use skills to react to cues21)	Noncontrollable (eg, periodic messages25)	General reward type (eg, earning points27)	Punishment (eg, score reduction32)	Meta-game reward (eg, success academically and socially23)	Intermittent reward (eg, difficulty of scenarios20)	Negative reward	Near miss (eg, difficult end boss31)	Event frequency (eg, unlimited replay of game28)	Event duration (eg, no end until button pressed20)	Pay-out interval	Graphics and sound (eg, fictional medieval castle24)	In-game advertising (eg, links to other resources21)	Franchise	Explicit content	Total
An 201319		x	x	x		x																				4
Bordnick 201220				x		x			x		x			x		x						x	x			8
Brinker 201621		x							x					x								x				4
DeLaughter 201622												x	x	x		x				x			x			6
Duncan 201823		x	x	x									x	x	x							x				7
Girard 200924				x					x													x				3
Gordon 201725					x							x	x										x			4
Khazaal 200826		x		x	x								x			x										5
Khazaal 201327		x			x								x	x		x		x								6
Metcalf 201828				x					x				x						x			x				5
Nemire 199929				x					x				x			x						x				5
Parisod 201830			x	x	x		x						x	x		x						x				8
Rath 201531			x	x						x	x	x	x	x		x		x				x				10
Song 201332		x	x									x	x	x												5
Xu 201433						x							x	x												3
Total per game element		6	5	9	4	3	1	0	5	1	2	4	11	9	1	7	0	2	1	1	0	8	3	0	0
Total per category		20			8				12				32									11

Effects on smoking prevention and behavioral determinants

Table 4 presents the results regarding effects on smoking-related outcomes. Six studies were aimed at smoking initiation prevention. Of these, only 1 study assessed prevention by measuring use of cigarettes, in which no statistically significant difference was found (participants were nonsmokers before and after their study). These 6 studies together assessed 20 behavioral determinants of smoking initiation prevention. Statistically significant effects were observed for 8 of 20 of these determinants (ie, attitude, knowledge, intention, perceived risk, perceived susceptibility, and perceptions). Three studies found statistically significant positive effects on attitudes toward smoking, albeit not statistically significant in all conditions. One study found statistically significant positive effects for smoking-related knowledge from pre- to postintervention, but not at follow up. One study found a statistically significant positive effect in behavioral intention to not involve in social smoking in 1 of the conditions in 1 study (ie, future face vs current face), and positive but not statistically significant effects in the other condition (ie, self-avatar vs someone else’s avatar). Studies that assessed perceived risk, perceived susceptibility, and perceptions about outcomes of cigarettes smoking also reported statistically significant positive effects.

Table 4.

Effects on smoking-related outcomes

Study ID	Measures	Outcomes	Direction of difference	Statistical significance
An 201319	1. Self-reported 30-d abstinence of cigarette smoking	1. Smoking abstinence at 12-wk evaluation was 11% in the control group, 23% in the intervention group with tailored health message, and 31% in the intervention group with tailored health plus peer coaching.	Increase	1. Significant
Bordnick 201220	Cigarettes smokeda (timeline follow back) Cravingsa (Questionnaire of Smoking Urge-Brief; scale 1-7 ]low to higher cravings]) Self-efficacya (Smoking Abstinence Self-Efficacy, scale 1-5 [not at all tempted to extremely temped to extremely tempted]) Coping skillsa (Condiotte and Lichtenstein Confidence Questionnaire; scale 1-5 [no confidence to high confidence])	Postintervention, the intervention group smoked less cigarettes (M = 0.65, SD = 1.2) in the previous 7 d compared with the control group (M = 2.4, SD = 3.2). The intervention group smoked less cigarettes at follow-up [at 1, 2, and 6 mo postintervention]. At 6 mo postintervention, the intervention group had a mean of smoked cigarettes of 0.41 (SD = 0.86) compared with the control group (M = 7.4, SD = 7.3). Postintervention, the intervention group had lower cravings (M = 1.3, SD = 0.5) compared with the control group (M = 2.1, SD = 1.1). The intervention group had postintervention more self-efficacy (M = 1.7, SD = 0.72) than the control group (M = 2.5, SD = 1.2). At all follow-up time points, the intervention group had greater confidence to avoid smoking than the control group.	Decrease Decrease Increase Increase	Significant Significant Significant Significant
Brinker 201621	Motivation (“The animation of my 3D-selfie motivates me not to smoke”; scale 1-5 [fully agree to fully disagree]) Knowledge (“I learned new benefits of non-smoking”; same scale used as above)	Over 60% of the participants (fully) agreed (ie, felt motivated not to smoke), whereas 14.4% of the participants (fully) disagreed. Over 60% of the participants (fully) agreed (ie, learned new benefits of nonsmoking), whereas 12.8% of the participants (fully) disagreed.	Positive Positive	Not reported Not reported
DeLaughter 201622	1. Cravingsa (Questionnaire of Smoking Urge-Brief; scale 1-7 [low to higher cravings])	1. Mean cravings were 3.24 preintervention and 2.99 postintervention.	1. Decrease	1. Not significant.
Duncan 201823	Knowledge (scale 1-0 [correct or incorrect/not sure]; maximum value 9.0) Self-efficacy to refuse offers (scale 1-4 [not sure at all to definitely sure]) Attitude (scale 1-4 [more negative attitudes to more positive attitudes]) Perceived norms [scale 1-4 (strongly disagree to strongly agree) Intentions to use cigarettes (scale 1-5 [not at all likely to definitely likely])	Knowledge improved from preintervention (M = 4.16, SD = 2.53) to postintervention (M = 5.48, SD = 2.40), there were no differences at follow-up (M = 4.8, SD = 2.09). Self-efficacy improved from preintervention (M = 2.98, SD = 1.18), to postintervention (M = 3.40, SD = 0.87), to follow-up (M = 3.44, SD = 0.96). Attitude hardly changed from preintervention (M = 1.36, SD = 0.59), to postintervention (M = 1.37, SD = 0.47), to follow-up (M = 1.38, SD = 0.68). Perceived norms resulted in lower scores from preintervention (M = 1.81, SD = 0.69), to postintervention (M = 1.70, SD = 0.63), to follow-up (M = 1.67, SD = 0.67). Intentions resulted in lower scores from preintervention (M = 1.51, SD = 1.21), to postintervention (M = 1.07, SD = 0.15), to follow-up (M = 1.10, SD = 0.24).	Increase Increase No difference Decrease Decrease	Significant pre- to postintervention; not significant at follow up Not significant Not significant Not significant Not significant
Girard 200924	Nicotine dependencea [Fagerström; scale 0-10 (no to high level of addiction)] Abstinence ratea (daily smoker’s journal and exhaled carbon monoxide)	The intervention reduced nicotine addiction, and crushing cigarettes in VR had a better impact than grasping balls. After the fourth week, 2% of participants in the control group and 9% in the intervention group were abstinent. In the twelfth week, 2% of participants in the control group was abstinent, against 15% in the intervention group.	Decrease Increase	Significant Not significant postintervention; statistically significant at follow up
Gordon 201725	Number of cigarettes smoked per day 7-d abstinence (percentages) Self-efficacy to quit smokinga (Condiotte and Lichtenstein Confidence Questionnaire; scale 1-5 [no confidence to high confidence]) Prolonged abstinence (percentages)	The number of cigarettes smoked per day decreased from baseline17 to follow-up at 30 d12 and at 90 d.11 Self-reported 7-d abstinence increased from baseline (4%) to follow-up at 30 d (37%) and at 90 d (47%). Confidence in quitting decreased from baseline (3.62) to follow-up at 30 d (3.35) and at 90 d (3.13). Prolonged abstinence increased from 21% at the 30-d assessment and 32% at the 90-d assessment.	Decrease Increase Decrease Increase	Significant Significant Significant Not significant
Khazaal 200826	Intention to quit smokinga [scale 0-100 (not at all to absolutely)] Attitude to smoking prohibition in hospitals [scale 0-100 (unacceptable to completely justifiable)]	The intention to quit smoking increased from preintervention (M = 71.5, SD = 28.8) to postintervention (M = 75, SD = 25.4). Attitude regarding the smoking did improve from preintervention (M = 61.4, SD = 34.3) to postintervention (M = 74.2, SD = 27.2).	Increase Increase	Significant Not significant
Khazaal 201327	Stage of change (“yes” or “no” per stage) Self-efficacya (Smoking Self-Efficacy Questionaire-12; scale 1-5 [not at all sure to absolutely sure to refrain from smoking when facing stimuli]) Attitudes toward smoking scale-18a (scale 1-5 [totally disagree to fully agree]) Attitudes toward Nicotine Replacement Therapy scale-12a (scale 1-5 [totally disagree to fully agree]) Smoking status Occurrence of new smoking cessation attempts Number of cigarettes smoked per day	Smokers in the intervention groups were more likely to be in stage 2 (contemplation, preparation, and action) rather than stage 1 (precontemplation) compared with participants in the control group. Self-efficacy scores had an overall time effect to external [F(2, 233) = 6.0, P = .003] and internal stimuli [F(2, 234) = 6.2, P = .002], scores increased. Attitude scores of the intervention groups related to the pleasure of smoking, decreased over time [F(2, 234) = 8.0, P < .0005] and compared with the control group [F(2, 235)= 5.7, P = .004]. Attitude scores on the psychoactive benefits of smoking decreased and had an overall time effect [F(2, 234) = 6.6, P = .002]. Attitude scores on adverse effects of smoking increased and had an overall time effect [F(2, 227) = 9.5, P < .0005]. Overall, participants of the intervention groups increased scores over time, compared with the control group [F(2,235) = 16.7, P < .0005]. Smoking status had a positive effect over time (P = .001; OR = 4.27; 95% CI, 1.87-9.77), and participants in the Pick-Klop group were less likely to be smokers compared with the control group (P = .04; OR = 0.32; 95% CI, 0.11-0.96). The mean number of quit attempts increased over time [F(2, 233) = 31.0, P < .0005]. A decrease of cigarettes smoked was observed over time [F(2, 229) = 24.8, P < .0005] and between the Pick-Klop group and the other groups [F(2, 230) = 3.5, P = .03].	Forward Increase Decrease and increase Increase Decrease Increase Decease	Significant Significant Significant Significant Significant Significant Significant
Metcalf 201828	Substance use (tobacco or alcohol) (fill-in-the-blank) Self-efficacy (scale 1-5 [low to high]) Behavioral intention (same scale used as above) Attitude (same scale used as above)	There was an overall improvement in the intervention group of 50% less substance use at 1-wk follow-up compared with baseline. Though, after completion of the study, 27% of the participants increased their substance use. The rate of participants continuing using tobacco after entering the study decreased by 4%. Tobacco substance users had a mean self-efficacy score of 4.07 at baseline, 4.24 after the study, and 4.01 at 1-wk follow-up. While participants had a relatively high behavioral intention to quit or reduce their substance use, these scores decreased from baseline to follow-up. Participants’ score in intention to quit using problematic substances (alcohol/drugs/tobacco) was 4.27 at baseline, 4.00 after the study, and 3.93 at 1-wk follow-up. Participants’ score in intention to reduce the use of problematic substances was 4.56 at baseline, 4.43 after the study, and 4.15 at 1-wk follow-up. Average scores on attitude were at baseline 3.88, 3.99 after the study, and 3.67 at 1-wk follow-up. It was found that 2-4 participants that reported tobacco use as their problem substance had an average score on self-efficacy, attitude, and intended behavior of 3.99 at baseline; after the study, their score was 3.97.	1. Decrease 2. No difference 3. Decrease 4. Decrease 2-4 No difference	1. Not reported 2, Not reported 3. Not reported 4, Not reported 2-4 Not significant
Nemire 199929	Use of cigarettes (and other drugs) (score 1: “never used,” score 2: “use a few times but not in the past year”) Intention to use cigarettes (and other drugs) (score 1: “definitely no intent to use,” score 2: “probably no intent to use”) Attitude toward smoking (score 1: “Strongly agree with negative statements about smoking,” score 2: “Agree”)	The mean pre- and postintervention ratings for all 3 groups ranged from 1.09 to 1.28, indicating that participants were nonsmokers/users both before and after the intervention. The mean pre- and postintervention ratings for all 3 groups ranged from 1.11 to 1.44, indicating that participants had no intention to smoke/use drugs both before and after the intervention. There was no main effect on group or item level, yet improvements were found in 2 items. The mean pre- and postintervention ratings for all 3 groups ranged from ranged from 1.69 to 2.07, indicating that in general participants had negative attitudes toward smoking both before and after the intervention.	No difference No difference No difference	Not significant Not significant Not significant
Parisod 201830	Anti-smoking self-efficacy scalea (scale 15-60 [lowest to highest level of self-efficacy]) Perceptions about outcomes of cigarette smokinga (Smoking outcome Expectation scale [positive outcomes of smoking; scale 3-12 (most to least favorable result) / negative outcomes of smoking; scale 3-12 (least to most favorable result)]) Attitudes toward tobacco use [scale 1-4 (most negative to most positive attitude)] Tobacco-use motivesa (scale 3-12 [lowest to highest motives to smoke]) Motivation to decline tobacco use in the future (scale 1-4 [highest to lowest motivation]) Knowledge (mentioning any consequences of smoking; sum)	Antismoking self-efficacy was not different within and between groups (intervention group: median = 55 at baseline, median = 57 postintervention). Outcome expectations showed favorable changes in the intervention group (for positive outcomes of smoking: median = 3 at baseline, median = 3 postintervention; for negative outcomes of smoking median = 11 at baseline, median = 12 postintervention). Within the other groups and between groups, no differences were found. Attitudes toward cigarette smoking showed favorable changes in the intervention group (median = 1 at baseline, median=1 postintervention). Within the other groups and between groups, no differences were found. Motives to use tobacco was not different within and between groups (intervention group: median = 3 at baseline, median = 3 postintervention). Motivation to decline tobacco in the future was not different within and between groups (intervention group: median = 1 at baseline, median = 1 postintervention). Knowledge about tobacco was not different within and between groups.	No difference Increase Increase No difference No difference No difference	Not significant Significant (intervention group) Significant (intervention group) Not significant Not significant Not significant
Rath 201531	1. Attitudes about tobacco products and the tobacco industry (Anti-Tobacco Industry Index; scale 1 [positive attitude] to 10 [negative attitude])	1.1 In the group that did not play, or played very little of Level 1, no differences were found in attitude toward tobacco products or tobacco industry from recruitment (M = 7.45, SD = 1.81) to 3 mo postintervention (M = 7.40, SD = 1.83). 1.2 In the group that mastered level 1 and 2, their score rose from recruitment (M = 7.70, SD = 1.70) to 3 mo postintervention (M = 8.01, SD = 1.63). 1.3 In the group that mastered level 3 or won the game, their score rose from recruitment (M = 7.39, SD = 1.63) to 3 mo postintervention (M = 7.99, SD = 1.69).	1.1 No difference 1.2 Increase 1.3 Increase	1.1 Not significant 1.2 Significant 1.3 Significant
Song 201332	Perceived risk (scale 1 [strongly disagree] to 10 [strongly agree]) Attitude toward social smoking (same scale used as above) Intention to quit smoking (same scale used as above) Perceived susceptibility (same scale used as above)	1.1 Perceived risks were higher in the future face condition (M = 7.55, SD = 1.93) than in the current face condition (M = 6.92, SD = 2.00). 1.2 In the self-avatar condition, perceived risks in the future face condition (M = 8.14, SD = 1.53) were higher than in in the current face condition (M = 6.84, SD = 2.00). 1.3 Perceived risks were higher in the self-avatar condition (M = 7.49, SD = 1.87) than in the someone else’s avatar condition (M = 7.00, SD = 2.07). 2.1 In the future face condition (M = 7.94, SD = 1.26), participants had stronger negative attitudes toward social smoking than did participants in the current face condition (M = 6.81, SD = 1.80). 2.2 Negative attitudes toward social smoking were slightly higher in the self-avatar condition (M = 7.46, SD = 1.89) than in the someone else’s avatar condition (M = 7.31, SD = 1.40). 3.1 A stronger intention to quit smoking was found in participants in the future face condition (M = 8.37, SD = 1.62) compared with the current face condition (M = 7.24, SD = 1.99). 3.2 Intention to quit smoking was higher in the self-avatar condition (M = 8.21, SD = 1.58) than in someone else’s avatar condition (M = 7.43, SD = 2.09). 4. Perceived susceptibility to the consequences of smoking was higher in the self-avatar condition (M = 7.77, SD = 1.26) than in the someone else’s avatar condition (M = 5.96, SD = 2.20).	1.1 Increase 1.2 Increase 1.3 Increase 2.1 Increase 2.2 Increase 3.1 Increase 3.2 Increase 4. Increase	1.1 Not significant 1.2 Significant 1.3. Not significant 2.1 Significant 2.2 Not significant 3.1 Significant 3.2 Not significant 4. Significant
Xu 201433	Smoking statusa (Bedfont Smokerlyzer; scale 1 [nonsmoker] to 7 [heavy smoker]) Self-expansion substituting the reward effects of nicotine	Smokerlyzer scores decreased from the initial session (M = 3.31, SD = 1.25) to the scan session (M = 1.25, SD = 0.58). Self-expansion (novel, exciting, and interesting/challenging activity with a partner) can act as a reward substitute of nicotine, as it weakens brain reactivity to cigarette cues (ie, undermine nicotine cravings) among nicotine-deprived smokers.	Decrease Decrease	Significant Significant

Outcome measure valid and reliable.

CI: confidence interval; OR: odds ratio.

Cigarettes smoked (timeline follow back)

Cravings (Questionnaire of Smoking Urge-Brief; scale 1-7 ]low to higher cravings])

Self-efficacy (Smoking Abstinence Self-Efficacy, scale 1-5 [not at all tempted to extremely temped to extremely tempted])

Coping skills (Condiotte and Lichtenstein Confidence Questionnaire; scale 1-5 [no confidence to high confidence])

Nicotine dependence [Fagerström; scale 0-10 (no to high level of addiction)]

Abstinence rate (daily smoker’s journal and exhaled carbon monoxide)

Self-efficacy to quit smoking (Condiotte and Lichtenstein Confidence Questionnaire; scale 1-5 [no confidence to high confidence])

The number of cigarettes smoked per day decreased from baseline to follow-up at 30 d and at 90 d.

Intention to quit smoking [scale 0-100 (not at all to absolutely)]

Self-efficacy (Smoking Self-Efficacy Questionaire-12; scale 1-5 [not at all sure to absolutely sure to refrain from smoking when facing stimuli])

Attitudes toward smoking scale-18 (scale 1-5 [totally disagree to fully agree])

Attitudes toward Nicotine Replacement Therapy scale-12 (scale 1-5 [totally disagree to fully agree])

Anti-smoking self-efficacy scale (scale 15-60 [lowest to highest level of self-efficacy])

Perceptions about outcomes of cigarette smoking (Smoking outcome Expectation scale [positive outcomes of smoking; scale 3-12 (most to least favorable result) / negative outcomes of smoking; scale 3-12 (least to most favorable result)])

Tobacco-use motives (scale 3-12 [lowest to highest motives to smoke])

Smoking status (Bedfont Smokerlyzer; scale 1 [nonsmoker] to 7 [heavy smoker])

Effects on smoking cessation and behavioral determinants

The effects of playing a serious game on smoking status and cessation were assessed in 7 studies. Four studies found statistically significant, positive effects on smoking status,,,, and 1 study found positive effects that were significant at one time point but insignificant at another time point. Moreover, the 3 studies that assessed the effect on the number of smoked cigarettes all found positive, statistically significant effects.,, Last, 1 study found statistically significant, positive effects on cravings.

Six studies assessed the effects of serious games on behavioral determinants of smoking cessation among adult smokers.[20,25-28,33] These 6 studies assessed a total of 12 behavioral determinants of smoking cessation. Of these determinants, 6 had statistically significant positive effects (ie, attitude, self-efficacy, intention, self-expansion, and coping skills). One study found statistically significant, positive effects on attitude toward smoking. Self-efficacy effects were inconclusive with statistically significant positive effects, and statistically significant negative effects. One study found statistically significant, positive effects on behavioral intention to quit smoking. Studies that assessed self-expansion and coping skills also reported statistically significant, positive effects.

Other findings

Three studies evaluated a serious game that was delivered in combination with another intervention. Two studies reported statistically significant, positive effects on smoking cessation., Although most studies reported on outcomes observed shortly after the intervention,,,,,, 5 studies reported on results after a slightly longer follow-up (eg, at 12 weeks).,,,, Of the latter studies, 2 found statistically significant, positive effects on the number of cigarettes smoked and cravings, and cessation rates. Rath et al found statistically significant, positive effects at follow-up on antismoking attitudes. The outcome measures used in the studies were all validated,,, partly validated,,, or none of the measures were validated.,,,,,, Only 2 studies assessed carbon monoxide levels when measuring smoking status., The majority of the studies with statistically significant effects had a fair or good study quality score. Owing to the heterogeneity of the included studies, it was impossible to perform a pooled meta-analysis.

DISCUSSION

This systematic review aimed to gain insight into the composition and effects of serious games on smoking-related outcomes. We found that serious games included multiple game elements of different categories. The evidence from a number of studies suggests that games may have positive effects on smoking-related outcomes, particularly on smoking cessation. Six studies on smoking prevention together assessed 20 determinants and found statistically significant positive effects for 8 determinants (ie, attitude, knowledge, perceived risk, perceived susceptibility, intention, and perceptions). Nine studies were aimed at smoking cessation, of which 7 assessed smoking status or cessation. Of these studies, 4 found statistically significant, positive effects on smoking status, and 3 studies assessed the effect on the number of smoked cigarettes with all finding positive, statistically significant effects. Six studies assessed the effects of serious games on 12 determinants of smoking cessation, 7 of which found statistically significant positive effects (ie, self-efficacy, self-expansion, attitude, coping skills, and intention).

This review analyzed which game elements are included in serious games aimed at smoking prevention and cessation. Similar to our findings, a review of serious games for sexual health promotion found that “feedback” and “rewards” were most frequently included game elements. These game elements are known behavior change techniques or learning mechanisms., The extracted game elements of the serious games in this review mainly correspond to the “10 ingredients of great games” (eg, avatars, 3-dimensional environments, narrative context). Most serious games in this review did not include elements in the category “social features.” The game element “support networks” was not included in any of the studies. Social features, which include support networks, appear thus underrepresented in serious games for smoking initiation prevention and cessation. This underrepresentation suggests that there is a window of opportunity to incorporate these features in these serious games, which could contribute to their effectiveness to prevent or stop smoking. This is because serious games with social features included in this systematic review all reported positive significant effects on smoking cessation and determinants.,, Moreover, the literature points out that smoking cessation and relapse are strongly associated to smoking behavior in social networks, and smoking behavior is impacted by social influence. For many people, identification with and social norms of important reference groups play an important role in increasing a smokers’ self-efficacy to quit. Social support networks can further offer platforms for increasing motivation or interest in quitting smoking and preventing or sustaining abstinence, and have provided preliminary effectiveness in smoking cessation.

Our finding that a number of studies suggested that games may have positive effects on smoking-related outcomes is in line with other reviews on serious health games. Sexual health games positively affected behavioral determinants (eg, intention), yet no effect on behavior was found (eg, delayed sexual initiation). Exergaming was found to result in positive effects on physical activity–related outcomes, while determinants (eg, self-efficacy) were not assessed. Serious games for healthy lifestyle promotion yielded small positive effects on health behaviors (eg, healthy diet) and determinants (eg, knowledge). Serious games for chronic disease self-management are effective in improving knowledge and self-management. Our review shows that some serious games have the potential to stop smoking and positively influence behavioral determinants of smoking cessation. Considering the effects of serious games on smoking initiation prevention, the results were more ambiguous and will likely require a timespan covering years rather than weeks. Also, some studies may have lacked power to detect statically significant effects.

Our review exposes various limitations in the included studies and raises serious questions. The heterogeneity of studies, intervention characteristics, and outcomes inhibits an unambiguously, conclusive evaluation of the effects of serious games on smoking-related outcomes. Moreover, the reporting of the studies did not allow an analysis of the effects of specific or combinations of game elements on smoking-related outcomes. Further, many studies were classified as poor or fair (methodological) quality, raising questions about the validity of results. One of the main issues was unreported information. This prevents a distinction between studies that reported incompletely but may have in fact good methodological quality and studies that actually had poor methodological quality. Lack of information on characteristics of study also creates uncertainty about the generalizability of the results. Moreover, about a quarter of the studies did not report any theoretical basis of their game or intervention; however, this is crucial for effectiveness of serious games on behavioral determinants. This is underlined by a review that places theoretical basis in the first stage of a framework for developing serious games. Another limitation of the studies in our review is that only 5 studies reported follow-up outcomes up until 12 weeks. As it is known that most smokers have multiple quit attempts before achieving maintained smoking cessation, long-term results are of great importance to assess whether serious games have lasting effects on smoking on smoking-related outcomes. To illustrate, 1 study showed that at follow-up the positive effects on self-efficacy and intention to change health behavior can substantially be reduced. In accordance with a Cochrane review, we suggest that for research follow-up of at least 6 months of abstinence should be used as a benchmark for sustained smoking cessation. Another limitation is that most studies did not use or partly used validated and reliable measures. In addition, smoking status was only in 2 studies assessed by carbon monoxide; other studies used self-reports. Self-reports tend to underestimate smoking behavior. Furthermore, for studies assessing effects of a serious game combined with another intervention, it remains unclear if the effects can be attributed to the serious game, or are a result of the cointervention. Last, the fixed playtime of the serious games is a limitation. The amount of playtime (ie, frequency and duration) of serious games and settings of studies varied greatly among studies in our review, and only 2 studies allowed free usage. Effects assessed based on free usage are likely to reflect effects close to real-life effects in contrast to fixed play sessions. The fixed play of most serious games might thus generate effects that are not (fully) generalizable to real-life settings.

Limitations

Our systematic review might be affected by publication bias, which may have resulted in an overestimation of the potentially positive effects of serious games on smoking-related outcomes. Another limitation of our review is the quality appraisal of included studies. Despite the independent appraisal by 2 reviewers and the use of quality appraisal tools, there was room for subjectivity in the appraisal. Another limitation is that we were dependent on the descriptions of a game in the included articles for the extraction of game elements, which were often not detailed. Therefore, we might have misclassified game elements. Last, the results of this review might be influenced by the taxonomy of King et al, which we used for extraction of game elements. Although we used the most relevant taxonomy according to our assessment of 3 taxonomies, game elements might have been missed due to the specific taxonomy used.

CONCLUSION

In this review, we gained insight into the composition and effects of serious games on smoking-related outcomes. As a next step, it would be valuable to assess the contribution of single or combinations of game elements on the effects of these serious games on prevention and cessation of smoking. Based on the results of our review, we suggest to include game elements with reward and social features and subsequently assess their contribution to effects. These features are the most and least included game elements, respectively, and a contribution assessment would show whether this is justified.

This study stresses the need for health informaticians to base serious game design on theories and to apply multiple game elements, among which game elements from the category “social features.” Health informaticians should further consider setting up social networks in relation to game use. They should evaluate these games on their effectiveness, with standardized, valid evaluation measures and measurements. Furthermore, we urge future evaluations to be based on voluntary playtime and to assess long-term results (at least 6 months postintervention). Still, the intended intensity of interaction (ie, intermittent interaction or time-intervals) between the player and the game should be taking into account when determining the length of the follow-up period. Moreover, we highly recommend that future evaluations of serious games be more methodologically homogenous and of high quality. We advocate for detailed descriptions of study populations to shed light on external validity, studies’ methodologies, intervention components, and protocols.

To conclude, serious games included multiple game elements of different categories and seemed to positively affect smoking-related outcomes, particularly smoking cessation. However, owing to the various limitations of the literature, we cannot be conclusive about the effects of serious games on smoking prevention, cessation, or behavioral determinants. At this point, we have insufficient knowledge on the effects and working mechanisms of serious games to make well-founded recommendations to game designers. Once again, future evaluations should be more homogenous, comprehensive, and of high quality.

FUNDING

MED and MPF were supported by FNO-Healthy Future Nearby (102253).

AUTHOR CONTRIBUTIONS

MED and MPF designed the study and JGD, MED, and SvS built the search. MED and SvS carried out the data extraction and drafted the article. All authors (MED, SvS, AEK, JGD, MWMJ, and MPF) were involved in data analysis, contributed to the writing of the final article, and approved the final article.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

Click here for additional data file.