Effectiveness of serious games and impact of design elements on engagement and educational outcomes in healthcare professionals and students: a systematic review and meta-analysis protocol.

INTRODUCTION: Serious games (SGs) are interactive and entertaining digital software with an educational purpose. They engage the learner by proposing challenges and through various design elements (DEs; eg, points, difficulty adaptation, story). Recent reviews suggest the effectiveness of SGs in healthcare professionals' and students' education is mixed. This could be explained by the variability in their DEs, which has been shown to be highly variable across studies. The aim of this systematic review is to identify, appraise and synthesise the best available evidence regarding the effectiveness of SGs and the impact of DEs on engagement and educational outcomes of healthcare professionals and students. METHODS AND ANALYSIS: A systematic search of the literature will be conducted using a combination of medical subject headings terms and keywords in Cumulative Index of Nursing and Allied Health, Embase, Education Resources Information Center, PsycInFO, PubMed and Web of Science. Studies assessing SGs on engagement and educational outcomes will be included. Two independent reviewers will conduct the screening as well as the data extraction process. The risk of bias of included studies will also be assessed by two reviewers using the Effective Practice and Organisation of Care criteria. Data regarding DEs in SGs will first be synthesised qualitatively. A meta-analysis will then be performed, if the data allow it. Finally, the quality of the evidence regarding the effectiveness of SGs on each outcome will be assessed using the Grading of Recommendations Assessment, Development and Evaluation approach. ETHICS AND DISSEMINATION: As this systematic review only uses already collected data, no Institutional Review Board approval is required. Its results will be submitted in a peer-reviewed journal by the end of 2018. PROSPERO REGISTRATION NUMBER: CRD42017077424. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

A comprehensive and prospectively registered (International Prospective Register of Systematic Reviews #CRD42017077424) systematic literature review protocol reported according to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols statement.

An assessment of the overall quality of the evidence regarding effectiveness of serious games (SGs) on each outcome, using the Grading of Recommendations Assessment, Development and Evaluation approach

The definition of ‘SG’ in this systematic review is limited by the fact that no consensus prevails in the literature.

Introduction

Education of healthcare professionals and students is of the utmost importance to promote the adoption of best practices and to improve patient safety. In 2013, the WHO underlined a need to train healthcare professionals to face population health needs and health service delivery adapted to epidemiological and demographic realities.1 Engagement, defined as a learner’s involvement and interest towards an educational intervention, has been positively correlated with educational outcomes such as learning and behaviour change.2 3 Designing educational interventions to sustain the engagement of healthcare professionals and students is therefore critical to their effectiveness. As such, the last decade has seen a growing interest from researchers about the use of serious games (SGs) in healthcare professionals’ and students’ education.4

SGs are designed as entertaining and active learning environments which can be made available on any digital platform (eg, smartphone, computer). Learning in SGs typically occurs through a gameplay that engages the learner in challenges adapted to his in-game skills. Challenges are defined as subjective experiences that solicit the learners’ skills.5 For example, challenges can require the learner to explore, experiment, compete or cooperate with other learners.6 7 Gameplay is defined as a combination of challenges and design elements (DEs).8

DEs, which can be seen as building blocks or shared features of games (eg, points, difficulty adaptation, story), are suggested to be instrumental in improving the learner’s engagement in SGs.4 9 10 Therefore, a SG may be composed of several DEs and these DEs may be similar across SGs.11 A list of DEs, based on a literature review by the authors, is presented in table 1.10 12–15 Based on Csikszentmihalyi’s flow theory and theoretical propositions of other authors in game design, Pavlas suggests that DEs operate by influencing antecedents of engagement such as: the learner’s ability to concentrate on task, his sense of control, the feedback he receives and a deep but effortless involvement.16 Lameras et al underline the difficulty in linking DEs and engagement to specific learning processes in SGs. Nonetheless, it is believed that higher engagement will lead the learner to become deeply involved and to repeatedly take on the challenges offered to improve his in-game performance and, consequently, his educational outcomes.17 However, while supporting engagement, there are also concerns that DEs can become a source of distraction from the learning content and negatively affect educational outcomes.16 18 19

Table 1

Design elements in serious games to be assessed

Avatars	The learner chooses or creates a graphical representation of himself in the serious game.
Badges for achievement	A visual representation that serves as a symbol for the learner’s achievements
Content unlocking	Access to new aspects of the serious game (eg, higher levels) when certain tasks have been accomplished
Difficulty adaptation	Levels of difficulty of the challenges can be adjusted by the learner (eg, easy, medium and hard) or automatically adjusted to the learner’s performance.
Hints	A suggestion or an indication given by the serious game to help the learner achieve a challenge. A learner may choose to receive a hint or the serious game can give hints based on his performance.
Leaderboard	A table or a graph that ranks the learners according to their success based on specific criteria
Performance tables or graphs	A table or a graph that provides information to the learner about the progression of his performance over time
Plot	A narrative discourse that serves to organise the events of a story in a logical or temporal order
Points	Points are awarded or subtracted depending on the learner’s performance and serve to numerically represent the progression. Points can therefore serve as a reward and as an immediate feedback tool.
Teams	Learners who work together to achieve a common goal. Teams will be broadly regarded here to encompass either cooperative or versus playing.
Time pressure	A time limit that is allowed for the learner to achieve a specific challenge. A time limit can be illustrated by a countdown timer that indicates to the learner the time remaining.
Virtual goods	Game assets that have a certain in-game value. Virtual goods can sometimes be bought or exchanged using real-world currency.

Previous reviews have highlighted the mixed effectiveness of SGs when compared with other educational interventions (eg, classroom learning) in improving healthcare professionals’ and students’ engagement or educational outcomes.4 20 21 As previous authors pointed out, the integration of an educational purpose to the gameplay is a delicate process that should be considered throughout SGs development, right up to the initial designing phase of the SGs.22 23 As such, the mixed effectiveness of SGs in supporting engagement and improving educational outcomes could be explained by the highly variable DEs found in SGs.24 Moreover, the optimal integration of DEs in SGs remains to be discovered.22 25 Findings from a recent meta-analysis investigating the effectiveness of SGs on healthy lifestyle promotion underlined the necessity of strong theoretical foundations in designing SGs and the need to further explore which DEs are the most impactful.26 As the development of SGs can be quite expensive and time consuming, describing which DEs have been integrated in the development of SGs for healthcare professionals and students, as well as their impact on engagement and educational outcomes, could help in the efficient development of future SGs.20 27 However, a systematic review of the impact of DEs of SGs on engagement and educational outcomes has yet to be published.

Therefore, our objectives are twofold: (1) to systematically identify, appraise and synthesise the best available evidence regarding the effectiveness of SGs on engagement and educational outcomes of healthcare professionals and students and (2) to describe which DEs have been integrated in the development of SGs for healthcare professionals and students and their impact on engagement and educational outcomes.

Methods

The protocol for this systematic review was developed according to the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P; see online supplementary file 1) and follows the guidance provided by the Cochrane Handbook for Systematic Reviews of Interventions.28 29 The protocol was also registered prospectively on the International Prospective Register of Systematic Reviews (# CRD42017077424).

Eligibility criteria

The eligibility of studies will be assessed based on study characteristics, participants, interventions assessed, comparators and the outcome measures for inclusion in the systematic review and the meta-analysis.

Study characteristics

Primary studies published in English or in French from 1 January 2005 to 31 December 2017 will be considered. Previous reviews on SGs have identified a growth in published literature starting mid-2000s,24 30 and no experimental study assessing a SG for healthcare professionals’ and students’ education was found before that date.4 As suggested, Effective Practice and Organisation of Care (EPOC) Cochrane Review Group, randomised controlled trials and cluster randomised controlled trials will be included in the current review.31

Types of participants

Studies with healthcare professionals and healthcare students from undergraduate to postgraduate education, either in a clinical or an academic setting will be considered. Clinical settings include all environments in which healthcare is provided. Studies conducted exclusively among patients or students receiving non-healthcare-related education (eg, high school students) are beyond the scope of this review as the objectives of the games evaluated in these populations (eg, illness self-management in patients, the learning of mathematical concepts in students) differ from those developed for healthcare professionals and healthcare students.

Types of interventions

Studies assessing a SG as a stand-alone intervention or as part of multicomponent intervention (eg, combined with workshop, classroom or digital-based learning) will be considered. For this systematic review, we define SGs as interactive and entertaining digital software with a primary educational purpose that engage the learner through various challenges.5 8 32 33

Types of comparators

We will consider studies where the comparator is any type of educational intervention.

Types of outcome measures

Studies reporting at least one measure of engagement or educational outcomes will be considered.

We will retain the definition and the indicators of engagement reported by Perski et al, who defined engagement as: (1) the extent of the learner’s involvement and (2) a subjective experience characterised by affect, attention and interest.3 These two dimensions will be considered individually. Regarding involvement, we will look individually at the duration and the frequency of SGs usage, either self-reported or electronically measured by the SGs. Regarding subjective experience, we will consider individually all self-reported measures of the learner’s experience while using the SG.

We will define the educational outcomes, which are learning and behaviour change, according to the levels of evaluation proposed by Kirkpatrick.34 Learning represents the knowledge that was acquired (ie, factual knowledge; eg, knowledge about the physiopathology of a specific disease), the skills that were improved (ie, how to perform a certain procedure; eg, ultrasound needle placement) or the attitudes that were changed (ie, how worthwhile the learner believes it is to apply the learning in his day-to-day role; eg, attitudes towards pain management) after playing the SG. These three dimensions will be considered separately (ie, knowledge, skills and attitude). Behaviour change represents a change in the day-to-day role of the healthcare professionals or the students after playing the SG (eg, coronary heart disease patients’ referral to cardiac rehabilitation by healthcare professionals). For this review, we will consider studies reporting both subjective (eg, self-reported evaluation) and objective measures (eg, quiz) of learning and behaviour change.

Information sources

Bibliographical databases

Eligible primary studies will be identified through a comprehensive search of six bibliographical databases: Cumulative Index of Nursing and Allied Health (EBSCO), Embase (Ovid), Education Resources Information Center (ProQuest), PsycInFO (APA PsycNET), PubMed National Center for Biotechnology Information and Web of Science—Science Citation Index (SCI) and Social Sciences Citation Index (SSCI) (Institute for Scientific Information (ISI)—Thomson Scientific).

Handsearching

Relevant journals will be handsearched for additional articles. Such journals include, but are not limited to: Games for Health Journal, Games, G|A|M|E The Italian Journal of Game Studies, International Journal of Computer Games Technology, International Journal of Serious Games and JMIR Serious Games. We will also handsearch for additional articles the reference lists of identified studies and previous systematic reviews related to the use of serious games in healthcare care professionals and healthcare students.

Search strategy for bibliographical databases

The search strategy was collaboratively and iteratively developed with the assistance of a librarian. We initially developed the search strategy for PubMed using a combination of medical subject headings and keyword related to the following key concepts: serious games, healthcare professionals/healthcare students and effect on educational outcomes. The search strategy was then translated for the other bibliographical databases (see online supplementary file 2).

Study records

Data management

All collected references will be imported and managed in EndNote V.X8 (Clarivate Analytics). Based on the eligibility criteria, we will manage and categorise references in specific files and subfiles inside the software. Full texts will be uploaded in the software at the corresponding stage of the selection process.

Selection process

Three reviewers will independently perform the selection process of the collected references. The titles and the abstracts will be screened by applying the previously stated eligibility criteria. A full-text assessment of the remaining references will then be performed. We will resolve all disagreements through discussion and consensus. Reasons for excluding references will be documented and reported in a PRISMA flow diagram at the full-text assessment stage of the selection process.35

Data extraction process

Two independent reviewers will independently perform the data extraction process using an adapted electronic format of the EPOC data extraction template. The original template was developed by the EPOC editorial team to serve as a guide in extracting data from primary studies.36 The reviewers will pilot the form by individually extracting data from a study and by comparing completed forms. Precisions will be added, and wording will be revised, if needed. Notwithstanding data regarding the studies eligibility, we will extract data regarding the impact of the DEs of the SGs on the previously stated outcomes. Data related to the DEs will be extracted during actual gameplay if the SG is publicly available. When it is not possible, data related to the DEs will be extracted based on the information provided in the article. A Kappa statistic will serve to illustrate agreement between the two independent reviewers on the extraction of DEs due to the challenge that could represent this step. Data needed to assess the risk of bias (eg, allocation concealment, blinding) will also be extracted. All data items to be collected are listed below. When data adjusted for baseline differences between groups are available, we will use them to compute effect sizes. When adjusted data are not available in the article, we will use the unadjusted data. All corresponding authors will be contacted in the case of unclear or missing data in the published articles. We will enter all data in RevMan V.5.3 (The Cochrane Collaboration), a software which allows data management and analysis during the process of a systematic review and a meta-analysis.37

Data items

To guide the extraction of data items related to DEs, a list of DEs was identified through a literature search by the authors (see table 1).10 12–14 Accordingly, we will extract the following data items from the included studies:

To a descriptive purpose: study setting; participants’ inclusion and exclusion criteria; study aim; design employed; unit of allocation (eg, individual, group); start and end dates of the study; duration of participation; unit of measurement; time points measured; tool used to measure the outcome; validity of the tools; name of the SG evaluated; theoretical framework used for the SG development; cost and duration of the SG development; clinical topic addressed in the SG; method of delivery of the comparator intervention (eg, e-learning, face to face), duration and frequency of use of the comparator intervention, clinical topic addressed in the comparator intervention;

To a meta-analytic purpose: study population; sample size; statistical differences at baseline between groups; DEs in the SG evaluated (see table 1); duration of SG usage; frequency of SG usage; outcomes related to the learner’s subjective experience while using the SG (eg, interest, flow), knowledge acquisition, skill improvement and attitude and behaviour change; risk of bias data (as outlined below).

Assessment of risk of bias

Two authors will independently assess the risk of bias of included studies using the EPOC criteria.38 Nine criteria are used and, for each one of them, the study can be judged at ‘low risk’, ‘high risk’ or ‘unclear risk’ of bias. A high risk of bias diminishes the reliability of the study’s results. The criteria account for aspects regarding the allocation sequence and concealment, measurements and characteristics of baseline outcome, incomplete outcome data, selective outcome reporting, contamination, blinding, and other risk of bias. Specific guidance to evaluate the risk of bias individually for each criterion is provided by EPOC. Each criterion will be considered independently and no attempt at assigning an overall score at each study will be made, as suggested by EPOC.38 We will resolve all disagreements about the risk of bias assessment of the two authors through discussion and, if needed, with the help of a third review author.

Assessment of selective reporting of outcomes

We will assess the selective reporting of outcomes by comparing the outcomes reported in the primary study with the ones stated in the published research protocol. If no published research protocol is available, we will check if the study was prospectively registered and compare the outcomes reported in the primary study with the ones in the registration form. If the study was not prospectively registered, we will compare the outcomes presented in the Methods section with the ones reported in the results section. In all cases, the corresponding authors of studies for which there are discrepancies in the outcomes reported will be contacted by email to obtain relevant unreported data.

Assessment of publication bias

If more than 10 studies are included in the meta-analysis, we will assess publication bias by constructing a funnel plot with RevMan V.5.3 (The Cochrane Collaboration).37 The interpretation of the funnel plot will follow the guidance included in Cochrane Handbook for Systematic Reviews of Interventions.28 If not, the assessment of publication bias will be done qualitatively.

Data synthesis

Quantitative data synthesis

The meta-analysis will evaluate the effectiveness of SGs on healthcare professionals’ and students’ engagement (ie, duration and frequency of SGs usage; outcomes related to the learner’s subjective experience while using the SG) and educational outcomes (ie, knowledge acquisition, skill improvement and attitude and behaviour change). The meta-analysis will only include low risk of bias studies based on the assessment using the EPOC criteria.38 In the case of serious reporting or publication bias, no meta-analysis will be performed. This decision will be made by consensus between review authors while also considering clinical and methodological diversity.28

When multiple trial arms are reported in a single trial, we will include only the relevant arms according to the intervention received and the comparison made (ie, SG vs any type of educational intervention). If multiple independent comparisons are relevant in a single study (ie, no common group between comparisons), we’ll include individually all comparisons in the meta-analysis. If one or more groups are shared between comparisons, we’ll first try to combine several groups to create a single pair-wise comparison. If it’s not possible, we’ll split the participants in the ‘shared group’ into multiple subgroups, with smaller sample size, to allow pair-wise comparisons. No minimal number of participants per arm will be required for a study to be initially included in the meta-analysis.

Random-effects models will be used to pool weighted effect sizes. It is assumed, by using random-effects models, that intervention effect across studies follow a distribution and are not all giving an estimate of the same intervention effect.28 The decision to use random-effect models was made due to the expected variability between SGs (notably on DEs) and in study design. Pooled weighted effect sizes will be calculated with 95% CI for each outcome. Continuous outcomes that were obtained by using different measures will be grouped by using standardised mean differences. The significance of the effect sizes will be determined using Cohen’s classification (<0.2= negligible; 0.2–0.49=small; 0.5–0.8=moderate;>0.8= large).39 We will define a statistically significant result by a two-sided alpha of 0.05. All quantitative data synthesis will be realised using RevMan V.5.3 (The Cochrane Collaboration).37

Assessment of heterogeneity

Statistical heterogeneity represents the inconsistency of the studies’ results and the percentage of variation across studies that is not due to chance.28 We will assess statistical heterogeneity using the χ2 test and the I2 statistic within RevMan V.5.3 (The Cochrane Collaboration).37 An I2 value superior to 50% will be considered as a high level of heterogeneity.

Subgroup analysis

We will try to explain the source of statistical heterogeneity by exploring clinical and methodological diversity through subgroup analyses. Following the guidance of the Cochrane Handbook for Systematic Reviews of Interventions, caution is required in comparing the magnitude of effect between subgroups as these results remain observational. Therefore, these subgroup analyses will serve primarily to investigate heterogeneity and will be conducted when at least two studies can be included in an individual subgroup.

Population: whether current healthcare professionals or students are the focus of the study.

Intervention: whether the serious game was delivered as a stand-alone intervention or as part of a multicomponent intervention.

Publication year: whether the study was published before or after 2014. As SG development is intrinsically linked to the technological state at a given moment, it is suggested that the overall quality of SGs should have improved in the last couple of years, and similarly their effectiveness in supporting engagement and improving educational outcomes.30 In parallel, the New Media Consortium declared in 2014 that the use of SGs was to be significantly experimented by educational institutions in the next 2 to 3 years.40

Sensitivity analyses

Sensitivity analyses will be performed to evaluate if the exclusion of small studies would affect the findings of the meta-analysis and the statistical heterogeneity for each previously stated outcome. If, after exclusion of small studies (if appropriate), at least three studies included in the meta-analysis evaluated SGs containing a specific DE (see table 1), we will perform sensitivity analyses to evaluate the impact of this DE on engagement and educational outcomes. This minimal number of studies (three) is necessary to minimise, during analysis, the potential risk of homogeneity in the DEs integrated to the SGs.

Assessment of the overall quality of evidence

The quality of the evidence regarding the overall effectiveness of SGs and each outcome will be assessed by using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach.41 GRADE formalises the process of evaluating the overall quality of evidence and formulating recommendations. For each individual outcome, there are four levels of quality of evidence (very low, low, moderate, high) which represent our confidence in the estimate of effect. Quality of evidence depends on factors such as risk of bias, unexplained heterogeneity and indirectness of the evidence. Two authors will independently assess and justify the quality of evidence regarding the overall effectiveness of SGs on each selected outcome. All disagreements will be resolved through discussion and consensus.

Qualitative data synthesis

A qualitative data synthesis will be presented whether or not a meta-analysis will have been performed. Descriptive data (eg, year published, country of origin, population, sample size, study design, name of the SG assessed, clinical topic addressed, outcomes measured) of each included study will be presented in a tabular form. A summary table containing an overview of all DEs included in the SGs assessed will also be presented. The results according the overall effectiveness of SGs and the impact of DEs on engagement and educational outcomes will be presented narratively.

Discussion

This systematic review will fulfil important needs in the designing of SGs for healthcare professionals’ and students’ education. As previous systematic reviews focused on a narrative evaluation of the overall effectiveness of SGs in the healthcare professions,4 21 42 43 to our knowledge, this will be the first systematic review to describe and evaluate of the impact of the prespecified DEs on engagement and educational outcomes. Previous systematic reviews were able to quantify the impact of several DEs in simulation (eg, range of task difficulty, deliberate practice)44 45 and internet-based education (eg, integration of an online discussion forum and audio files)46 by adopting a similar approach to the one we propose. Depending on the quantity of data available, planned sensitivity analyses will allow the quantitative evaluation of the impact of the DEs on selected outcomes. However, we acknowledge that possible homogeneity in DEs integrated to SGs could represent a limit to planned sensitivity analysis.

The protocol of this systematic review is prospectively registered (PROSPERO #CRD42017077424) and is reported according to the PRISMA-P guidelines.29 Our research strategy was collaboratively designed with the assistance of a librarian and will extensively cover several relevant databases. However, as clear criteria have yet to be established in the literature regarding what can be considered a SG, we will adopt an inclusive definition that does not discriminate based on the SG genre (eg, quiz, adventure).5 8 32 33 In the case of ambiguity regarding the inclusion of specific studies due to the nature of the of intervention, review authors will assess their eligibility through discussion and consensus. Nonetheless, this limitation will be addressed in the reporting of the full systematic review and considered in the formulation of recommendations. Moreover, as the integration of the selected DEs in SGs developed for healthcare professionals and students is currently unknown, it is currently planned to describe the integration of these DEs in developed SGs and their alignment with theoretical foundations, as stated by the authors.