Preparing Residents Effectively in Emergency Skills Training With a Serious Game.

INTRODUCTION: Training emergency care skills is critical for patient safety but cost intensive. Serious games have been proposed as an engaging self-directed learning tool for complex skills. The objective of this study was to compare the cognitive skills and motivation of medical residents who only used a course manual as preparation for classroom training on emergency care with residents who used an additional serious game.
METHODS: This was a quasi-experimental study with residents preparing for a rotation in the emergency department. The "reading" group received a course manual before classroom training; the "reading and game" group received this manual plus the game as preparation for the same training. Emergency skills were assessed before training (with residents who agreed to participate in an extra pretraining assessment), using validated competency scales and a global performance scale. We also measured motivation.
RESULTS: All groups had comparable important characteristics (eg, experience with acute care). Before training, the reading and game group felt motivated to play the game and spent more self-study time (+2.5 hours) than the reading group. Game-playing residents showed higher scores on objectively measured and self-assessed clinical competencies but equal scores on the global performance scale and were equally motivated for training, compared with the reading group. After the 2-week training, no differences between groups existed.
CONCLUSIONS: After preparing training with an additional serious game, residents showed improved clinical competencies, compared with residents who only studied course material. After a 2-week training, this advantage disappeared. Future research should study the retention of game effects in blended designs.

RCT Entities:   Population Interventions Outcomes

Health care, with its exponential growth in knowledge and increasing demands on the competencies required of doctors, has a need for new and more cost-effective training models.[1] Online learning, together with an instructor-led course or as a fully online course, can be used to improve the efficiency and flexibility of medical training.[1,2] Technology-enhanced simulation programs provide learning opportunities for controlled skills practice, without risk to the patient.[3] The effectiveness of simulation programs is well established. Their use in medical education is consistently associated with large improvements in knowledge, skills, and attitudes when compared with no intervention.[3-5]

However, full-scale computer-based simulators are often expensive, both in terms of initial purchase price and running costs.[6] Furthermore, attention to student motivation is often neglected in simulation programs.[7] As a result, learners tend to use them to reach certain learning outcomes but avoid continuous practice.[8] Serious games offer a challenging learning environment in which video game characteristics are attuned to educational goals.[9-13] They offer “situated meaning and learning,” whereby players interact with the game world through probing, seeing, and experiencing things in a specific context.[14] An important and common type of serious games are simulation games; they offer learning tasks in a realistic, engaging online environment, where learners directly experience the consequences of their decisions.[9,12,15,16] More specifically, they offer a mix of pedagogical (eg, cases, multimodal representations, scaffolding), game (eg, competition, context, multiple levels), and simulation (representations of situations or objects, user interaction, and feedback)[17] elements. Experiential or task-centered learning in games can be associated with guided discovery learning.[9] The rationale for making real-world tasks the basis of a learning environment is to promote the application of knowledge and transfer to practice.[18] Because of their scalability, simulation games have the potential to teach knowledge and skills that are typically acquired in a simulation center and at a fraction of the cost.[6] The integration of fun and challenge in games can reduce stress and enhance motivation and effectiveness.[19,20] In addition, self-directed learning seems to enhance intrinsic motivation, because it affords a greater sense of autonomy.[21] If students are intrinsically motivated to learn, they will likely spend more time learning and feel more positive about what they learn.[22] Furthermore, the element of competition in games is expected to stimulate sustained play and learning.

Several studies have examined the effects of serious games on learning outcomes and motivation. Most reviews of these studies, however, find that they have mixed and ambiguous outcomes, largely because of methodological flaws.[11,15,16,23,24] Examples of such flaws are: the use of pre-to-post comparisons leading to overestimation of the game effect, internal validity threats (history, selection),[15] and a lack of studies with a suitable control condition or randomised controlled trial.[16] In short, serious games offer promising learning tools for engaged, complex learning, but more research is needed to evaluate their efficacy in health care.

Training in emergency care skills is critical for patient safety and is an essential part of undergraduate and postgraduate medical education. Each year, worldwide more than 1.5 million health care professionals attend a variety of emergency care courses, using the internationally standardized “ABCDE” approach.[25] This method prioritizes initial resuscitation of critically ill patients. The mnemonic stands for Airway, Breathing, Circulation, Disability, and Exposure. Substantial resources are invested in teaching these complex cognitive skills[26] with the aim of integrating knowledge, numerous skills, and a professional attitude within tight time constraints.[27] Considering the training challenges and tight budgets in health care organizations, more efficient training models are required. In 2010, the International Liaison Committee of Resuscitation recommended combining online self-directed training with instructor-led skills training.[28]

As a preparation for instructor-led emergency skills training, we have developed a serious game (abcdeSIM), in which medical residents can stabilize patients in a virtual emergency department. In a previous study of fourth-year medical students, we found that abcdeSIM and text-based cases were no better than an e-module (used by the control group) at improving their cognitive emergency care skills. However, the game-based group was found to be more motivated than the text-based group.[29] Because these students were inexperienced in emergency care, we suspect that working on such a limited number of simulated cases, although engaging, was too complex for them. In the current study, we will investigate the effectiveness of the same game for residents (i.e, the original target group), in a blended design. Should game-based learning prove to be effective as a preparation for face-to-face training of complex acute care skills, then training time may be reduced. Once simulation games have been developed, they can be used for skills training for large numbers of trainees, with no extra costs for instructors or simulated patients, in contrast to simulation centers. Blended learning, combining e-learning with classroom training, has been shown to make emergency care training more efficient for different learning goals, while maintaining learning outcomes.[25,30]

The research questions of this study are:

(1) Do residents, who use the abcdeSIM game in addition to a course manual as a preparation for classroom training, show better emergency care skills (before training) than residents who exclusively use a course manual? (2) Are they motivated to play the game? (3) Are game-playing residents more motivated to learn the course content than residents who only used the course material? (4) Is there a difference in skills level between groups after 2 weeks of training?

Our hypothesis is that residents who use the game will show improved skills at the start of face-to-face training compared with those who exclusively used the course manual (1). We expect residents to be motivated to play the game (2) and also expect game-playing residents to be more motivated for the course content than residents who only used the course manual (3). We have no specific expectations of the skills level after 2 weeks of training (4).

METHODS

Participants, Setting, and Study Design

We performed this study with second-year family-practice residents. All family-practice residents in The Netherlands are required to do a 6-month traineeship in an emergency department of a hospital; before the start, residents must complete a 2-week general emergency care course. After passing this course, they are allowed to start their traineeship under the supervision of certified attending physicians. This course includes emergency care subjects based on the ABCDE approach to emergency resuscitation. Each year, 500 to 600 Dutch family-practice residents are trained and assessed in this 2-week certified course. They are assigned in training groups of 70 to 90 residents by the academic training organization; there are no systematic differences between groups.

Most residents have limited experience with emergency care. Because the residents of a particular training group usually know each other (they are following the same courses), random assignment to both experimental groups was not feasible; game accounts could easily be exchanged between groups. In addition, different resident groups are quite homogeneous in age and experience. Therefore, we used a quasi-experimental design with a historical control; residents from the December training group were treated as the control (reading) group and received (only) the course manual 6 weeks before the 2-week classroom training. Residents from the next March and September groups were treated as the intervention (reading and game) group and in addition received an account for the abcdeSIM game 6 weeks before training. They were advised to first study the course manual and then play the game. The game group completed an online evaluation questionnaire after playing the game. Emergency care skills were assessed before the training and on the last training day. Participants completed a pretraining questionnaire during the first training day, and their data on posttraining assessment were recorded.

Selection of Participants

Six weeks before the start of the face-to-face training, residents were asked to participate in the extra pretraining assessment; this took place 2 hours before the start of the face-to-face-training. This assessment was not part of the regular program and had no consequences for the participants. On the first training day, all residents were in addition asked to participate in the rest of the study (questionnaires, posttraining assessment). Participants signed a consent form for both study parts; the study was approved by the Dutch ethical board for research in medical education (NVMO, Ned. Vereniging voor Medisch Onderwijs, no 210).

MATERIALS

Preparatory Course Manual

All residents received a course manual on emergency care skills as a preparation for classroom training. The manual contained instructional material (no cases) on the ABCDE approach and the essentials of medical emergency and trauma care. Subjects that were covered in the manual were impaired critical functions, disturbances of consciousness, etc. (see Table 1).

TABLE 1

Overview of Preparation of Classroom Training for the 2 Experimental Groups

Preparatory Serious Game

The abcdeSIM simulation game provides an online-simulated emergency department, where residents can apply and exercise their emergency care skills with virtual patients.[31] Its design was based on analyzing the task demands of the ABCDE approach for stabilizing acutely ill patients. The learning objectives, resulting from this analysis, were the basis for the choice and development of the cases including the feedback. The contents of the cases were validated by an expert panel. The game was designed by an experienced game design company and pilot tested with the target group before implementation. All relevant tools for assessment and stabilization of acutely ill patients are virtually available (stethoscope, laboratory tests, infusion fluids, blood, medication). A high-fidelity mathematical model of human physiology for respiration, circulation, and neurological functioning was implemented on the virtual patients to create realism and give immediate feedback on the patients' condition. Players started with doing a tutorial, explaining the game interface. The game contained 6 regular adult patient cases, presented as animated photos (cases in Table 1). The game was primarily aimed at training clinical emergency care skills; communication skills were not addressed in the game. Each case had to be solved within 15 minutes (a timer was presented) and could be done as often as desired. During playing, players received direct feedback on their actions through a monitor with data on the patient's condition and from the assisting virtual nurse. After playing a case, residents got a score and narrative feedback. The game score was dependent on how many correct decisions were made according to the ABCDE method and how efficiently this was done (less minutes resulted in more points). Peers' scores per case and a high score list were presented to stimulate competition between players. The simulation game contained only cases, no additional instructions (Fig. 1 for screenshots of the game) (see document, Supplemental Digital Content 1, http://links.lww.com/SIH/A306, more detailed game description). The reading and game group studied the course material as a preparation for game play.

FIGURE 1

Screenshots of the abcdeSIM game.

Classroom Training

The 2-week training course was aimed at training emergency care skills, using the ABCDE approach and basic and advanced life support techniques. The design and assessment of this course are comparable with Advanced Trauma Life Support and Advanced Life Support courses. The training provided a combination of lectures, scenario training in small groups with standardized patients and mannequins, and skills training. Each participant acted as the treating physician in 3 scenarios with a simulated patient, on 1 scenario with a pediatric mannequin and assisted as a nurse in 6 other scenarios. In addition, one observed and critiqued peers in 15 scenarios. Each 20-minute scenario play was followed by a 20-minute feedback session. To summarize, during training, residents played an active role in 10 scenarios and reviewed another 15 scenarios.

Assessment and Evaluation Instruments

Skill Assessment

Posttraining assessment was part of the regular certified emergency care course. Fifteen assessors with different medical specialty backgrounds, all qualified instructors in internationally certified emergency medicine courses and trained according to international standards (a standardized Generic Instructor Course), assessed the residents, 1 rater per candidate. Assessments consisted of a 1-case scenario test (15 minutes) with a standardized, trained simulation patient. Because of the design, the assessors could not be blinded for the condition; they may have heard about the game as a preparation for training, but they were not involved in the study, nor had they any interest in a specific outcome. Before assessments, raters were briefed on the scenarios and instrument. During assessments, a course director was available for questions. This is a high-stakes assessment for the residents; if they fail, 1 resit is offered with another scenario and 2 raters (including the course director). If residents fail again, they are not allowed to start the emergency department traineeship. Participants for the pretraining and posttraining assessments were assigned at random to different assessment scenarios. The same scenarios were used for the reading group and the reading and game group. The skill assessment scenarios were different from the scenarios in the game. The pretraining assessment followed the same procedure (1-case scenario test with a standardized simulation patient).

The assessment instrument aimed to measure the ability to assess and treat seriously ill or injured patients and consisted of a clinical competency scale (6 items on the ABCDE method and diagnostics, eg, “uses ABCDE approach on initial assessment”), a communication competency scale (3 items on communication, eg, “communicates with patient effectively”), both rated on a 7-point scale (7 = excellent). In addition, the assessor judged the candidate on a global performance scale (using a single 10-point scale to rate “independent functioning in caring for acutely ill patients in the emergency department” (10 = perfect)) (see document, Supplemental Digital Content 2, http://links.lww.com/SIH/A307, assessment instrument). The pass/fail cut point was based on the global performance scale (fail if <6). The assessment instrument was validated in a separate study; the competency scales showed good construct validity and internal consistency. The clinical competency scale and global performance scale showed moderate interrater reliability (intra class correlation = 0.49 resp. 0.50); the communication competency scale had poor interrater reliability (intra class correlation = 0.27).[26] Although communication skills were not addressed in the game, we do report them, because they were a part of the regular course and its assessment. These assessment results also provide insight into the competencies on which the game does and does not have an impact.

Motivation Questionnaire on the Game

After working on all cases in the game, residents who participated in the reading and game group completed an online questionnaire evaluating the game and the motivation to play. The questionnaire consisted of 9 statements, including items such as “I felt actively involved with the patient cases,” to be scored on a 5-point Likert scale (5 = fully agree) (see document, Supplemental Digital Content 3, http://links.lww.com/SIH/A308, motivation questionnaire).

Pretraining Questionnaire on Task Value, Self-efficacy, and Self-assessment

During the first training day, a questionnaire was completed by all participants on motivation toward the course, self-efficacy, and self-assessment. The Motivated Strategies for Learning Questionnaire has been used extensively in educational research projects to measure students' motivational orientations.[32] We used 2 subscales: task value (9 items, eg, “I think that what I am learning in this course is useful for me to know”) and self-efficacy (9 items, eg, “I'm certain I can understand the ideas taught in this course.”) All items were rated on a 7-point Likert scale (7 = very true of me). In addition, participants completed a self-assessment instrument, on the basis of the raters' assessment instrument.

Self-study Time

On the last training day (before assessment), all residents completed an evaluation form on the course, including a question on the hours of preparation for the course. The questionnaire was anonymous; data on self-study time was available on group-level only.

Statistical Analysis

A reliability analysis (Cronbach α) was calculated for the questionnaires and assessment instruments. Independent t tests were performed to compare group characteristics, assessment data, and motivation data from the reading and reading and game group. Unless the distribution of scores is severely skewed, data from rating scales can be analyzed as if they were interval without introducing bias.[33] Effect sizes (ESs) were calculated using Glass’s δ. Practical significance of research results can be quantified from being small (ES ≈ 0.20) to moderate (ES ≈ 0.50) to large (ES ≈ 0.80).[34] Associations between game data were calculated using Pearson coefficient. We treated missing data with pairwise deletion and used SPSS for the statistical analysis.

RESULTS

Characteristics of Participants

Of 210 eligible residents (60 from the reading group, 150 from reading and game group), 159 (76%) consented to participate in the study, 52 in the reading and 107 in the game and reading group. A subgroup of 18 (reading) and 24 (reading and game) residents agreed to participate in the extra pretraining skill assessment. There were no differences between the reading group and reading and game group on main characteristics, such as experience with acute patients and score on national test for family-practice residents (Table 2). The pretraining assessment group (n = 42) did not differ from the rest of the research participants (n = 117, data not presented).

TABLE 2

Characteristics of Reading Group and Reading and Game Group Before and After Training

Assessment Results Emergency Care Skills

Reliability of the scales (Cronbach α) was 0.92 for the 6-item clinical competency scale and 0.81 for the 3-item communication competency scale. Before training (after the game), the reading and game group performed better on clinical competency skills (P = 0.03, Table 3) with a medium-large effect size (Glass’s d = 0.62) than the reading group. Improvements occurred particularly in the items on initial assessment (d = 0.82), treatment (d = 0.72), and requests for additional diagnostics (d = 0.50). The reading and game group also showed less variability in competency levels (more homogeneity, measured as smaller standard deviation scores; P = 0.02). There were no differences in communication competency skills or on global performance scores between groups before training.

TABLE 3

Assessment Results Before Training and After Training

There was an association between assessment scores on the global performance scale with the clinical competencies scale (r = 0.74, P < 0.001) and with the communication competencies scale (r = 0.42, P = 0.006).

At the end of the 2-week training, scores on the competency and global performance scales were similar for both groups (Table 3). There was also an association between assessment scores on the global performance scale with the clinical competencies scale (r = 0.80, P < 0.001) and with the communication competencies scale (r = 0.60, P < 0.001).

Self-study Time and Game Data

Mean self-study time (self-reported) before the course for the reading group was 9.9 hours (SD = 5.9 hours), and for the reading and game group, it was 12.4 hours (SD = 5.7 hours); hence, the game group spent 2.5 hours extra self-study time (P = 0.007). Mean game-playing time (logged) for the game group was 2.2 hours per resident (SD = 0.95 hours), which is consistent with the self-reported study time. The mean playing time per case (excluding the tutorial) was 17 minutes. The longest playing time was spent on the first case (internal bleeding with shock, mean = 28 minutes) and on what seemed to be the most difficult case (subarachnoid hemorrhage leading to seizures, mean = 27 minutes). The maximum playing time in the game was 15 minutes per case indicating that (on average) cases were completed more than once. There was an association between playing time and game score (r = 0.49, P < 0.001), indicating a learning effect within the game.

Evaluation of the Game

Reliability (Cronbach α) of the 9-item game motivation scale was 0.80 (n = 90). Mean score on the 5-point scale was 3.9 (SD = 0.41). Above average scoring items were “I felt actively involved with the cases” (mean = 4.2, SD = 0.53), “My attention was completely drawn to the cases” (mean = 4.2, SD = 0.63), and “I liked playing the game” (mean = 4.2, SD = 0.69). Below average scoring item was “I regularly felt stressed during playing the cases” (mean = 3.3, SD = 0.87) (See supplemental Digital Content 3, http://links.lww.com/SIH/A308, for detailed evaluation data). Responses from the residents that indicated strong points of the game included the following: “it's a very good way to develop experience with this approach, as a preparation for seeing patients”; “it feels very ‘real’”; “it is very instructive”; and “I could feel the stress.” Points for improvement of the game were: “it took me some time to figure out how to select and put back the tools in the game”; “I would have liked more feedback on my actions”; and “I would like more information on how scores are composed and may be improved.” A few remarks were made on limitations in game play because of a slow computer or internet access.

Task Value, Self-efficacy, and Self-assessment

Reliability (α)of the task value scale was 0.76 (n = 159), for the self-efficacy scale α was 0.83 (n = 159), and for the self-assessment scales on clinical and communication competencies α was 0.88 and 0.71 (n = 150), respectively. On comparison, both groups showed the same scores on task value (motivation for the course, mean = 6.2/6.2; SD = 0.43/0.39 for the reading and reading and game group resp. on the 7-point scale) and on self-efficacy before training (mean = 4.8/4.8; SD = 0.66/0.61 for the reading and reading and game group resp.). The self-assessment scores on clinical competencies before training were higher for the reading and game group (n = 104, mean = 4.7, SD = 0.54) than the reading group (n = 46, mean = 4.4, SD = 0.74, P = 0.01). There were no differences in self-assessed communication competences or global performance before training. Compared with the assessors' rating, both groups overestimated their skills.

DISCUSSION

This study compared the cognitive skills and motivation of residents who only used a standard course manual as a preparation for classroom training on emergency care (the “reading group”) with residents who used an additional serious game (the “reading and game group”). In terms of our research questions, we found that residents who combined a course manual and a simulation game as a preparation for classroom training improved their emergency care skills, compared with residents who only used a course manual (1), which was in line with our hypothesis. The game had no effect on communication competencies or on global performance in caring for acutely ill patients. As expected, residents also were motivated to use the game (2). Contrary to our expectation, the motivation before training for the training content was not different between groups (3). After 2 weeks of face-to-face training, there no longer was a difference between groups in skills level (4).

With limited extra self-study time (+2.5 hours, compared with 10 hours spent on the course manual by both groups), the serious game had a positive effect on clinical competencies at the start of the training, with a moderate-large effect size, and also reduced the variability in clinical performance. For teachers, a more homogeneous student group is easier to train. The same communication competency levels were measured in both groups, which can be explained by the fact that the game did not address these competencies. Nor was there an effect of the game on global performance. Apparently, this more holistic, perhaps personal notion of global performance captures for assessors more than only clinical competencies. The fact that both skills are not perceived as the same by assessors was supported by the (only) partial correlation of the global performance and clinical competency scores. Moreover, the self-assessment of the reading and game group only improved for the clinical competencies (not for global performance), compared with the reading group.

Anecdotal reports from teachers indicated that the extended (game-based) training preparation had a positive impact on the training level. Before, the group showed considerable heterogeneity; some trainees had experience with scenario-based ABCDE training, whereas most started with only the knowledge from the course manual.

Our study basically compared reading and reading in adjunct with game play. As the reading and game group spent more time on self-study, it remains questionable how important the game characteristics were for skills learning. The simulated cases, being at the center of the game, provided an opportunity to practice the skill with a variety of patient problems. The finding that this had a positive impact on residents' performance is consistent with research on task-centered learning, showing that learning with a variety of real-world tasks facilitates skills development and transfer to clinical practice.[18,27] In addition, reviews on technology-enhanced simulation in health care show that in comparison with no intervention, simulation programs are associated with large effects on knowledge and skills.[35,36] Future design-based research that controls for time on task should determine which game features enhance performance and motivation. These may include, for example, the narrative line, the scoring system, multiple sources of feedback,[37] and animated cases.[38]

Considering the issue of games and motivation, we found that residents were motivated to learn with the game and felt actively involved and immersed with the game cases (as was illustrated by remarks as “I could feel the stress,” “It's a very good way to develop experience with this approach”). This was supported by the fact that they played the patient cases several times and spent 2.5 hours extra self-study time on the game. In a previous study with the same game, used by medical students, the group working on the game felt more motivated compared with a group working on the same cases in a text-based format.[29] These are important results, because in self-guided, online training programs, motivated trainees will put more effort into learning. It also suggests that the simulation game could be used as a skills maintenance tool after training. How do games motivate learners to spend extra time on task? Choice and the opportunity for self-directed learning seem to enhance intrinsic motivation, because they afford a greater sense of autonomy.[21] The opportunities for self-directed learning and interest in the subject of the game probably created intrinsic motivation. More research is needed on the specific features that make games engaging for learning, compared with simulations and interactive cases.

Residents who used the game as an additional preparation were just as motivated for the course content as residents who only used the course material (as preparation). A possible explanation for this is the high level of motivation for the course among all residents, as shown by the high task value scores. When residents know they are going to need certain skills in practice, they usually are quite motivated for the course. Our results show that motivation to engage with an instructional format should be distinguished from motivation to learn a specific task.

Both groups had the same (general) self-efficacy levels, but the reading and game group correctly self-assessed their clinical skills as superior to that of the reading group (although both groups slightly overestimated their skills relative to the assessor's rating). Apparently, the game does not easily change the residents' general sense of self-efficacy during the course, whereas it did make them more aware of their improved emergency skills.

After 2 weeks of training, we no longer found a positive effect of the preparatory game. An obvious explanation for this is learning time; the effect of the 2.5-hour game time was overshadowed by the 2-week classroom training. In addition, in terms of the number of different scenarios discussed, classroom training offered much more opportunities for learning than game training (25 vs. 6 cases).[39] Considering the relatively high assessment scores at the end of training, there probably also was a ceiling effect in the residents' competencies after the 2-week training.

Given the higher starting level in clinical competencies, a relevant question is whether classroom training can be shortened in combination with the game, maintaining learning outcomes. This would make the blended training design more cost-effective because online games are scalable to large numbers of health care professionals without extra costs (in contrast to simulation centers). A study on Advanced Life Support training, comparing assessment results at the end of a conventional 2-day course with a 1-day course supplemented with online interactive simulation cases, showed similar knowledge and skills.[25] Instead of the course manual, we currently use an e-module, including exercises on the emergency approach and a demonstration video with a simulated patient. More worked cases of patient problems could be added to this e-module. To further enhance the preparatory skills level of residents, a number of cases could be moved from the training to the game. This enables residents to practice with a larger variety of virtual patients and focus on their personal learning needs. We know that for training complex cognitive skills, offering a high variety of learning tasks is important to allow transfer to new tasks.[39] Training can possibly be shortened, reducing direct and indirect costs. Future research should confirm the effectiveness of this new blended design.

In the current study, we found a positive effect of the game on emergency care skills with residents. In a previous study with medical students, however, the game group did not profit from this game, compared with a group working on text-based cases or the control group working on only an e-module.[29] This indicates an “expertise reversal effect,” where a rich learning environment benefits experts, but is ineffective or even counterproductive for novice learners.[40] The effects of game design choices for different user's expertise levels are an interesting field for further study.

One limitation of our study is the fact that groups were not randomized; thereby, confounders may potentially have played a role. Our research groups were very homogeneous (second-year family-practice residents), and we believe that we measured the most important possible confounders, such as experience with acute patients and general knowledge on national tests, and found no differences between groups.

Another limitation is the relatively small number of participants for the pretraining assessment (n = 42). Although the number of residents was limited, they do not differ from the total group on important characteristics and thus may be considered as representative for the groups they came from. In addition, self-assessment data from the total group (n = 150) supports the measured improvement in clinical skills from the smaller group. Despite the small sample, we did find (practical) significant differences in skills between groups. Nevertheless, the small numbers limited our possibilities to analyze relationships between game time, performance, and motivation, more specifically. It would be interesting to replicate the study with larger groups of residents and a shorter (blended) training design.

Thirdly, we assessed residents' competencies in a single patient scenario. Extensive evidence in assessment research shows that content specificity is the main cause of unreliability and outweighs other sources of bias.[41] However, we have analyzed an internationally representative emergency care assessment situation where single scenario assessment is commonly used[42,43] and our assessment instruments were validated in a separate study.[26] Furthermore, we did not assess the effects on patient outcomes. Few emergency care courses have had patient outcomes as an end point,[44] but it would be worthwhile to investigate the transfer to clinical practice.

Finally, this study is not conclusive for the question of whether the learning time or the game format was responsible for the improved skills, because the game group spent more time on task. Many studies on serious games discuss proposed elements that are important for motivation and learning, but relatively few describe empirical evidence.[45-48] Comparative design-based research will have to show which features of simulation cases enhance learning outcomes.[49,50] This “value-added” research approach (Mayer) will be an important next step in the young field of game-based learning.[51]

In summary, this study showed that serious games can be used as an effective, motivating preparation for instructor-led emergency care courses to teach medical residents' clinical competencies. Learning from doing, with a variety of realistic, interactive patient cases, through error and without harming patients is an important potential benefit of games and simulation programs. Future research is needed to show how this effect can be sustained and whether training time can be reduced in combination with online-simulated cases, maintaining learning outcomes. More research is also needed on the question of which game features are critical for engaged skills learning.