Gamification for the Win in Internal Medicine Residency: A Longitudinal, Innovative, Team-Based, Gamified Approach to Internal Medicine Board-Review.

BACKGROUND: Game-based learning is an engaging and effective educational strategy in medical education. The Internal Medicine resident board review at our institution was considered dull and poorly attended by trainees. We hypothesized that a gamified, longitudinal, team-based approach to board review would rejuvenate board review and improve learner perception of quality and attendance.
METHODS: We sought to improve the resident perception of and participation in board review through an innovative longitudinal, team-based, game-based intervention, the "Cohort Cup". The "Cohort Cup" was developed and implemented over a 22-week intervention period from November 2017 to May 2018. Teams (cohorts) competed in real-time against one another. Evaluation methods include a pre/post attitudes survey on a 5-point Likert scale (1 - strongly disagree, 5 - strongly agree) and attendance data.  Findings: Of 105 residents eligible to participate, 82 completed the pre-intervention survey, and 74 completed the post-intervention survey. We observed statistically significant increases in self-perceptions of engagement, the perceived value of the sessions, and preferences for game-based learning. Self-perceptions of learner engagement improved from 2.74 to 3.8. The value of the educational experience increased from 2.68 to 3.95. Preferences for game-based learning improved from 3.77 to 4.32. Board review attendance doubled. Residents commented the intervention improved class bonding. Board passage rate increased from 86% to 97%.
CONCLUSIONS: Our game-based intervention successfully rejuvenated our board review. We observed more joy in the learning environment and improvements in resident engagement, and in their attitudes regarding board review. Game-based learning can be a valuable educational tool and can be a positive facet of educational communities.

Introduction

Board review at our urban, academic Internal Medicine residency program was poorly attended with poor resident perceptions of utility and effectiveness. There was a general malaise regarding board review that negatively affected attendance, attitudes, participation, and, consequently, its perceived effectiveness. Many sessions were somewhat passive and unengaging. Residents did not view the board review as a coherent educational activity. Measurement of resident success was only assessed by passage on the American Board of Internal Medicine (ABIM). Most fundamentally, board review did not effectively utilize key features of learning. This report outlines our innovative approach to rejuvenating board review utilizing principles of adult learning, self-motivation, and game-based learning.

Within the realm of game-based learning, "gamification" involves the application of game design elements (conceptual building blocks integral to building successful games) to traditionally nongame contexts [1]. Friedlander et al., in 2011, described direct links between the neurobiology of learning and medical education [2]. They articulate concepts such as repetition, reward, reinforcement, active engagement, and stress, all of which can be seen during a game-based activity [1,2]. With respect to the learning environment, Rutledge et al., in 2018, described gamification as a practical consideration for medical educators [1]. They qualified the concept of gamification with respect to medical education as a valuable learning tool, facilitating and motivating the learner’s progression to self-determination and intrinsic motivation [1].

Given the basis for gamification as a useful learning technique, the value and perception of game-based learning have been described with an emphasis on engagement, learning climate, and learner perspectives [3-11]. Studies have shown modest improvements in knowledge and surgical skills with respect to gamified educational systems; most of this work has been done in subspecialties within Internal Medicine and Surgery [12-26]. These studies use primarily electronic platforms and do not commonly employ real-time directly competitive games, other than escape rooms. To date, there are no studies on the benefits and effectiveness of game-based learning within Internal Medicine residency programs pertaining specifically to resident board review and longitudinal didactics.

With these concepts in mind, we designed, implemented, and evaluated a longitudinal, team-based, game-based board review competition, the “Cohort Cup”. The “Cohort Cup” was intended to facilitate a learning environment rich in the aforementioned principles of medical education. We hypothesized that our “Cohort Cup” would improve resident attendance at board review sessions, resident perceptions of the value of board review, and motivation to prepare and attend.

Materials and methods

A junior faculty member with a passion for games developed the “Cohort Cup”. He proposed an outline of the longitudinal competition to the residency leadership and which was subsequently approved by the residency operations committee. The faculty champion worked closely with chief residents and used the American College of Physicians Internal Medicine in-training exam (ITE) score report to identify key topics that traditionally had been problematic for residents. The game-based intervention occurred weekly from November 2017 through May 2018, excluding holiday weeks. It was held in the central academic conference room for one hour during the weekly academic half-day educational didactics. Our residency program comprised 105 eligible (categorical Internal Medicine) residents and assigned residents to one of five cohorts for scheduling purposes; this preassigned structure of cohorts formed the basis of our team-based “cohort” competition. Each team comprised an equal number of PGY-1, PGY-2, and PGY-3 residents.

To begin, each cohort received supplies to create a coat-of-arms poster to symbolize their cohort’s sense of comradery within the residency; these were prominently displayed in the conference room throughout the competition. Inspired by popular game shows and childhood games, we adapted classic games that facilitated group learning. A variety of games were used, including a Jeopardy-like game, celebrity password, virtual scavenger hunt, and others (see Appendix 2 for templates and rules for each game). Each game-based session lasted for one hour, and all games utilized high yield board-review information from the American College of Physicians' Medical Knowledge Self-Assessment Program (MKSAP). The faculty champion was present for each learning game with a referee’s jersey, a whistle, and a stopwatch to facilitate control over the learning environment. Penalties were assigned and flags tossed into the gaming field whenever an “unsafe learning” event or situation presented itself (e.g., residents verbally berating a fellow teammate for losing a point). This ensured that a supportive, competitive spirit of gamesmanship was maintained and protected against a potentially unsafe learning environment. At the end of the competition, a “Cohort Cup” trophy was presented to the winning cohort, and they were taken out to dinner by program leadership, including the Program Director, Associate Program Directors, and the Vice-Chair for Education.

We used two leaderboards (visual representations of team scores) throughout the Cohort Cup. The first leaderboard consisted of five glass vases lined up in the conference room where the Cohort Cup was held. Each team’s vase held colored beads corresponding to their cohort’s color that reflected the number of total points they had accumulated thus far in the competition. Additionally, the hand-crafted Cohort Cup trophy itself served as a leaderboard. It featured a multicolor LED light that would shine the color of the cohort that was currently in the lead and was stationed in the central conference room (where most residents would see it daily).

Pre- and post-surveys were developed by physician educators with experience in survey design, modified iteratively, and piloted for usability. Surveys measured resident attitudes regarding self-perceptions of engagement, quality, value, and senses of peer bonding. Our survey was created internally to directly measure the above attitudes. (see Appendix 1 for pre/post surveys). The surveys utilized a 5-point Likert scale: 1 = strongly disagree, 2= disagree, 3= neutral, 4=agree, 5= strongly agree. Analysis comparing mean pre/post response value was conducted, and our results were statistically significant with 95% confidence and p-value ≤.01 for all questions reported. The surveys also contained a field for “Additional Comments” where participants would provide their own thoughts and opinions regarding the intervention.

The Wayne State University IRB granted us an educational innovation exemption. Residents were made fully aware of the optional nature to participate.

Results

Throughout the seven-month intervention period, 22 “Cohort Cup” sessions occurred, including a three-hour grand finale.

Of 105 eligible residents, 82 completed the pre-intervention survey, and 74 completed the post-intervention survey (78% and 70% response rate, respectively). Results are shown in Table 1. We found statistically significant improvements in resident self-perceptions of engagement during board review, perceived educational value of board review, and preferences for game-based learning in board review after the intervention. Self-perceptions of learner engagement improved from 2.74 to 3.8. (p=1.307e-8). The value of the educational experience increased from 2.68 to 3.95. (p=1.97e-12). Preferences for game-based learning improved from 3.77 to 4.32. (p=5.4e-4). These data represent resident averages pre- and post-intervention.

Table 1

Cohort Cup Pre/Post Survey Results

Pre-intervention and post-intervention survey results.  As denoted, 82 pre-surveys collected and 74 post-surveys were collected.  The number of residents eligible to take the survey = 105. (78% pre-intervention response rate, and 70% post-intervention response rate).  Surveys were collected electronically with no incentive for voluntary participation.  Analysis was conducted via t-test with 95% significance. (Last four survey items were not assessed on the pre-intervention survey)

1 - Strongly Disagree, 2 - Disagree, 3 - Neither Disagree nor Agree, 4 - Agree, 5 - Strongly Agree

N/A=Not applicable.

Survey Item	Pre-intervention Average (SD) (N = 82)	Post-intervention Average (SD) (N=74)	P-Value
I am engaged in the academic half-day as they are currently designed.	2.74 (1.195)	3.8 (1.007)	P = 1.307e-8
The current structure of the academic half-day contributes to my education.	2.68  (1.121)	3.95 (.949)	P = 1.97e-12
I prefer games and active learning as opposed to standard lecture-based didactics for my learning.	3.77 (1.087)	4.32 (.853)	P = 5.4 e-4
I prefer the “Cohort Cup” activity as opposed to standard lecture-based didactics for my learning.	-	4.24 (.957)	N/A
I feel that I learn more from the “Cohort Cup” activity than I do from standard lecture-based didactics.	-	4.03 (.986)	N/A
I prefer to have the “Cohort Cup” remain part of my didactics.	-	4.42 (.725)	N/A
I feel that the “Cohort Cup” activity has helped me bond with others in my residency	-	4.36 (.769)	N/A

Additionally, we investigated post-intervention preferences for the cohort-cup to remain as a core component of the resident board-review curriculum and also investigated the effect of the cohort-cup on resident bonding. We observed an average score of 4.42 for preferences for the “Cohort Cup” to remain present in resident board review and an average score of 4.36 regarding positive effects on resident bonding. Average attendance doubled from 12 residents per session during the previous traditional didactic board review to 25 residents per session for the “Cohort Cup” session. We also received comments on the survey reflecting residents’ increased enjoyment, engagement, and sense of comradery. These perceptions were also articulated via informal meetings regarding feedback after the intervention.

One faculty was present for each game, and, during our intervention, we observed that residents began asking the faculty host for topics ahead of time so that they could prepare for the sessions. It was noted that one resident came in on their day off to participate and help their team win that day. Our board passage rate increased from 86% the year before the “Cohort Cup” to 97% the year of the “Cohort Cup”.

Discussion

The response to the “Cohort Cup” was promising; our game-based, team-based, longitudinal intervention yielded improvement in perceptions of multiple metrics, including perceived effectiveness, engagement, and value of learning games. We also improved the board review attendance. The faculty noted that the gamified sessions resulted in numerous positive interactions amongst and between the cohort members. Gamified systems have the potential to significantly positively impact learning climate, learner engagement, and relatedness [1-11]. We believe the "Cohort Cup" accomplished this mission by appropriately facilitating a competitive environment to harness the potential of game-based learning.

Aside from the quantitative survey data presented above, we observed palpable changes in energy regarding the gamified board review. The "Cohort Cup" became a regular topic of conversation within our department and served as a vehicle to untie residents and facilitate teamwork and comradery. People identified by their "Cohort Cup" color and even came to morning reports dressed in their team colors! They developed cheers that were heard while waiting for conferences to begin. Literature supports that game-based learning positively impacts team building and comradery, and we noticed this with our program [1,2,6-7,9]. We observed that residents would attend in higher numbers, even when they were on busy inpatient services. Residents were fully invested in helping their cohort win, and were doing so with positive energy and outward demonstrations of joy during the games. Additionally, residents demonstrated on the survey that the activity improved their sense of bonding, a finding likely related to the intrinsic interactive game-based and team-based design and indicative of the principle of relatedness concerning the value of the learning environment [1,2]. The “Cohort Cup” brought an educational zest back to the educational paradigm within the program. With respect to competition, it also demonstrated the effective use of safe competition to facilitate engagement and motivation. This analysis shows that such a gamified educational intervention, rooted in real-time direct competition, can contribute positively to the educational culture within a residency program.

Several elements contributed to the success of the “Cohort Cup”. First, a faculty champion was essential. Analogous to a board review director, the champion identified relevant board review material, created games, and held a constant presence during each session. Additionally, the champion facilitated a lively, competitive atmosphere while preserving the integrity and safety of the learning environment for all learners. Second, the use of leaderboards both during and in-between “Cohort Cup” sessions contributed to a sense of competition that was essential to the success of the “Cohort Cup”. Their pervasiveness throughout the year also contributed to the motivation and structural permanence of the “Cohort Cup”. Finally, program leadership was fully supportive of the project and willing to experiment with a new paradigm.

There are some limitations to our intervention and analysis. Firstly, we present the results of a single program. We do not know if our residents are more game-oriented than other residents at other programs. Second, there may have been some selection bias. Residents who are competitive and game-oriented may not attend a traditional board review and be more likely to participate in a competitive board review. Finally, the champion was vested in the success of the project. Any board-review structure led by a committed and engaging facilitator would result in a more engaging and satisfying board review, even without having a gamified basis. Another major determining factor, and possible limitation, is that the intervention is only as strong as the residents’ willingness to participate, given the key principle of learner autonomy as an integral component of autonomy within the concept of self-determination [1]. As such, the voluntary nature of the “Cohort Cup” and the spirit of the moderator/environment becomes paramount to garnering participation.

Given the success of the “Cohort Cup”, we have set out to further quantify and qualify the effects of gamified learning environments. Currently, we are amidst analysis of a focus group-based formal qualitative analysis of a similar implementation of gamified learning. We are assessing a longitudinal game-based board review curriculum at a second institution, the University of Pittsburgh in the Department of Pediatrics, to further assess the learning environment of such an innovative curricular paradigm. Additionally, we have connected with a few institutions nationwide to continue to build a platform for game-based education within medical education. We hope that the attached templates serve as a basis for programs to engage in such a worthwhile endeavor. Given the recent shift to virtual education, as many programs begin to resume in-person didactics, we hope that programs will choose to incorporate innovative game-based sessions within their curricular paradigms.

Conclusions

Our longitudinal team-based, game-based board review paradigm, the “Cohort Cup”, successfully improved resident perceptions of board review quality and also improved resident attendance. Board review has become an engaging environment that residents seem to enjoy more. Game-based learning has become a mainstay for our board-review curriculum delivery and could potentially be adopted by other residency programs.