ImmuneQuest: Assessment of a Video Game as a Supplement to an Undergraduate Immunology Course.

The study of immunology, particularly in this day and age, is an integral aspect of the training of future biologists, especially health professionals. Unfortunately, many students lose interest in or lack true comprehension of immunology due to the jargon of the field, preventing them from gaining a true conceptual understanding that is essential to all biological learning. To that end, a new video game, ImmuneQuest, has been developed that allows undergraduate students to "be" cells in the immune system, finding and attacking pathogens, while answering questions to earn additional abilities. The ultimate goal of ImmuneQuest is to allow students to understand how the major cells in the immune system work together to fight disease, rather than focusing on them as separate entities as is more commonly done in lecture material. This work provides the first assessment of ImmuneQuest in an upper-level immunology course. Students had significant gains in learning of information presented in ImmuneQuest compared with information discussed in lecture only. Furthermore, while students found the game "frustrating" at times, they agreed that the game aided their learning and recommended it for future courses. Taken together, these results suggest that ImmuneQuest appears to be a useful tool to supplement lecture material and increase student learning and comprehension.

INTRODUCTION

Immunology has always been a field of study important to biology, especially for those interested in the health professions. However, with the more recent media attention given to vaccinations and global epidemics, it is a field that has garnered interest even from those who are not as scientifically minded. While a thorough understanding of immunology has become increasingly important in this day and age, and scientific knowledge of the field has grown significantly in recent years, students often struggle and lose interest due to a large amount of jargon in the field and a lack of good active-learning exercises to emphasize major concepts (1, 8).

It is well documented that students learn best in active-learning environments, and this is especially true for science, technology, engineering, and mathematics (STEM) majors, who are accustomed to lab experiences as a part of their course work (5). Indeed, lab exercises are often the most tangible aspect of science, allowing students to see for themselves the principles and techniques they hear about in lecture. Unfortunately, in the field of immunology, it is often difficult to translate major concepts, as well as new findings, into lab experiments suitable for undergraduates. A review of the literature as well as a nationwide survey of available course syllabi shows that the primary laboratory exercises students complete in immunology courses are technique-based and focus on antigen-antibody binding (2–4, 6, 7, 10, 11). Unfortunately, these experiments are unable to illustrate major concepts of immunology such as how the innate and adaptive immune systems work independently and together to fight a pathogen. The American Society for Cell Biology has developed some lab exercises that attempt to teach larger immunological concepts, but these experiments require the use of live animals, which may not be possible for faculty and students at most colleges and universities (2–4, 6). As a result, many undergraduate institutions teach immunology courses without an associated lab. This is unfortunate for all students interested in immunology, as the lack of good active-learning exercises, such as labs, means that the students have no additional methods by which to learn the material other than lecture and reading of the text. For students who are more tactile and visual learners, this could certainly affect their interest and ability to succeed in the field.

Recently, a video game called ImmuneQuest was developed to be used in undergraduate immunology courses to aid in student learning (http://immunequest.com). The overall goal of ImmuneQuest is for students to build and control a virtual immune system to protect a host from increasingly challenging microbial threats. In the process, they learn how individual cells and molecules work together in a “living” system, to better conceptualize and visualize discrete information provided in lecture. Specifically, students begin their game controlling a single macrophage searching for and recognizing pathogens colonizing the body. Students must answer questions about macrophages and innate immunity correctly in order to gain more abilities (i.e., activate the macrophage) so that they can destroy the pathogen while causing minimal damage to the host. As the infection continues, students “upgrade” their immune system’s capabilities and control more immune cells to rid the body of more challenging pathogens. Students must use the skills and knowledge gained in early levels in order to complete later levels, earning stars and points based on how well they answered questions, killed the pathogen, and saved the host. While the primary focus of ImmuneQuest at this time is on innate immunity, additional levels are being developed to include the adaptive immune system in the fight against the pathogen.

Because ImmuneQuest is a video game that can be installed on a student’s computer, it is a tool that can be used by faculty and students at any type of undergraduate institution regardless of available facilities or resources. Therefore, it was hypothesized that ImmuneQuest would be an effective active-learning tool in an undergraduate immunology course to improve student learning compared with lecture alone.

METHODS

Participants and ImmuneQuest

Biology majors in an Immunology course (BIO341; n = 20), were required to complete Part 1 of ImmuneQuest as part of their final course grade. Students ranged in age from sophomores to seniors and all students had completed the minimum core biology curriculum which includes one year of General Biology followed by a semester of Genetics. The required first part of ImmuneQuest is free to all users, with three levels to complete, focusing on the early responses of the innate immune system including macrophage activation and complement fixation. The instructor set up a group for the students in the course prior to the beginning of the semester, and all students enrolled in that group within ImmuneQuest, allowing the instructor to track student progress and scores. Because the game was downloaded as an app, students worked on ImmuneQuest independently on their computers, rather than as part of an organized laboratory during class hours. Students were able to earn up to 40 points total on the assignment, equal to 5% of the total course grade. Points were given for successful completion of each level (10 points per level) and the final 10 points were evaluated based on how well individual students performed in competition with the rest of the students in the course. All students were informed of the point breakdown at the start of the semester and could see their ranking against other students on the ImmuneQuest course page. As part of the learning process of ImmuneQuest, students are allowed to replay levels as many times as they choose to improve their scores; only the final score was seen by the instructor and other participants.

In-class assessment tool

Students were given an in-class pre-assessment quiz on immunological concepts in the first week of class prior to any lecture on innate or adaptive immunity. The assessment tool contained 11 questions on topics covered in lecture only, topics that would be covered solely by ImmuneQuest, or topics that were covered in both lecture and ImmuneQuest. All assessment questions were obtained from available test banks from the textbook publisher or from a list of questions supplied by ImmuneQuest. Students were unaware of which questions were covered by which delivery method. After the first unit on innate immunity was completed, students were asked to begin their work on ImmuneQuest and were given six weeks to complete all three levels of Part 1 independently, in addition to the regular course requirements. At the end of the semester (three weeks after the completion of ImmuneQuest and the last unit exam on lecture material covered on the assessment), students were given the post-assessment tool (Table 1), which contained the same questions on immunological concepts as the pre-assessment tool, but also contained additional questions on student perceptions of ImmuneQuest. The timing of the post-assessment was chosen to ensure that none of the material from either ImmuneQuest or lecture would be “fresh” in students’ minds to potentially bias results.

TABLE 1

Delivery format of all questions from the pre- and post-assessment tool.

Question	Delivery Format
1. True/False – Individual cells in the immune system have specific mechanisms for destroying pathogens and tend to work independently of other immune cells/proteins.	Lecture and ImmuneQuest
2. True/False – Activated macrophages lead to host cell damage when functioning.	ImmuneQuest
3. This immunoglobulin leads to the release of histamine from mast cells. IgM IgD IgA IgE	Lecture
4. Which of the following pattern-recognition receptors are used to detect microbes? TLR CDK CCR	Lecture and ImmuneQuest
5. Which of the following statements is true? Macrophages must first recognize specific parts of debris or microbes with receptors in order to phagocytose them. Macrophages can phagocytose anything lying around in tissue.	Lecture and ImmuneQuest
6. When immune cells become unresponsive and enter developmental arrest this is called: apoptosis anergy allelic exclusion	Lecture
7. Which of the following genes is involved in lymphoid gene rearrangements? FLT3 RAG Kit	Lecture
8. Which of the following is NOT involved in macrophage activation? IFN-g Red blood cell TNF-a T helper cells PAMP	ImmuneQuest
9. Which of the following is NOT part of the B cell co-receptor? CD19 CD81 CD3 CD21	Lecture
10. Detection of this by macrophages indicates that microbes may be present: CXCR PAMP TNF-a	ImmuneQuest
11. In humans, this protein promotes transcription of genes not normally expressed in the thymus to aid in negative selection of T cells: NF-kB AIRE AP-1 c-Jun	Lecture
12. On a scale of 1–10 with 10 being very much and 1 being not at all, how much did you enjoy the ImmuneQuest simulation?
13. Do you think that ImmuneQuest helped you understand concepts in Immunology better? Why or why not?
14. Would you recommend ImmuneQuest be used in future immunology courses?
15. If you had to use one word to describe ImmuneQuest, what would it be?

Ig = immunoglobulin; TLR = toll-like receptors; CDK = cyclin-dependent kinase; CCR = chemokine receptors; FLT3 = Fms-related tyrosine kinase 3; RAG = recombination activating gene; IFN = interferon; TNF = tumor necrosis factor; PAMP = pathogen-associated molecular pattern; AIRE = autoimmune regulator; AP = activator protein.

Assessment methods

Both the pre- and post-assessment tools were blind scored by the author. Enjoyability of ImmuneQuest was scored on a scale of one to ten, with ten indicating very much and one indicating not at all (Table 1). Pre- and post-assessment scores were subsequently matched for students and compared with ImmuneQuest scores and overall class grades for comparison and analysis. All statistical analyses were conducted using SigmaPlot 12 running a Student’s t-test or an ANOVA on Ranks with a Tukey Test (Systat Software, Inc.).

Informed consent and institutional review board protocols

All students signed an informed consent form prior to completing the assessment tool and ImmuneQuest. All assessments were coded and anonymous so that the scorer did not know whose work was being evaluated until all grading was completed. Approval to conduct this study was granted by the Elmhurst College Institutional Review Board, which determined that the protocol fulfilled the necessary requirements for human subject research.

RESULTS

ImmuneQuest improves student learning in an immunology course

As part of the laboratory component of an undergraduate immunology course, students were asked to complete all three levels of part 1 of ImmuneQuest. Not surprisingly, some students showed excitement at the idea of playing a video game as part of the course grade while others were not as thrilled with the idea. However, all of the students did participate in the video game during the allotted time. Students were asked at the beginning and end of the semester to complete a short assessment (Table 1) answering questions on immunological topics as well as their opinions on ImmuneQuest in the post-assessment. Figure 1A demonstrates that overall scores improved between the pre-assessment (4.68/11; n = 19) and post-assessment (9.65/11; n = 20) with average growth of 5/11 points for all students (p < 0.001). However, improved scores alone do not indicate that ImmuneQuest, itself, aided in student learning. Therefore, the questions on the assessment tool were then divided into topics covered only in lecture, both in lecture and in ImmuneQuest, or only in ImmuneQuest (Table 1). Figure 1B indicates that topics covered in lecture improved from 25% correct in the pre-assessment to 83% correct in the post-assessment, a 58% change. Furthermore, topics covered in lecture and ImmuneQuest improved from 70% to 89% correct, a 19% increase. Finally, topics covered by ImmuneQuest improved from 32% to 98%, a 66% increase. While student learning improved with all delivery methods, students scored significantly better on topics covered by ImmuneQuest compared with lecture alone (p < 0.05).

FIGURE 1

Student learning improves with ImmuneQuest. A) Pre- and post-assessment scores of all students in the course. * denotes statistical significance p < 0.001, Student’s t-test. B) Pre-and post-assessment scores were sub-divided into material covered in lecture only, both lecture and ImmuneQuest, or ImmuneQuest only. Significant improvements were seen in all cases between pre and post scores. Students scored significantly higher on post-assessment information covered in ImmuneQuest only compared with lecture alone. * denotes statistical significance p < 0.05 compared to lecture only post-assessment scores, ANOVA on Ranks with a Tukey Test.

ImmuneQuest aids learning in students of all abilities

As students completed levels on ImmuneQuest, they received points and stars for how well they answered questions, killed the pathogen, and prevented host damage in a timely manner. The number of stars obtained was directly proportional to points earned in the game and, for simplicity, all assessment was based on stars earned. At the end of the game, the number of stars students received throughout the three assigned levels was determined; this is shown in Figure 2A. Perhaps not surprisingly, 14/20 students earned 5 stars on the first, easiest, level. Interestingly, by the final level (level 3), only 6/20 students earned 5 stars and 3 students were unable to complete the level, earning 0 stars. In order to succeed on subsequent levels, students must apply skills and knowledge obtained in earlier levels. Therefore, it is not surprising that the three students who earned 1 star on level 1 were the same three students who could not complete level 3. However, it should also be noted that students who earned 1 star on a level could improve on subsequent levels and earn, for example, 4 stars later on. Therefore, failure to do well on one level did not preclude the chance to do well ever again, but it is clear that students needed to hone their skills to continue successfully.

FIGURE 2

Students of all abilities succeed and fail while playing ImmuneQuest. A) The number of stars (out of 5) students earned on each of the three levels. 0 stars indicates the student was unable to complete the level. B) Average overall score in the course (out of 100%) for students who received 5 stars or 1 star on each level of ImmuneQuest. C–D) Average (C) pre- and (D) post-assessment scores (out of 11) for students who received 5 stars or 1 star on each level of ImmuneQuest. No significant difference in scores was observed in B–D.

Interestingly, while one might assume that students who did well on ImmuneQuest were also the strongest students in the course, based on overall course grade, this was actually untrue. When the average class grade of students who earned 5 stars versus 1 star on each level was compared, there was no significant difference between the students (Fig. 2B). Indeed, better grades were not a predictor of game success, as students who earned a C in the course were just as likely to do well on the game as students who earned an A. Nor was game success a predictor of success in the course: students who earned 1 or 0 stars on the game were as likely to earn an A in the course as they were a C, indicating that the ability to do well on an exam did not necessarily translate to the ability to successfully complete ImmuneQuest or vice versa. Similar results were observed when comparing pre- and post-assessment scores for students who earned 5 stars versus 1 star (Figs. 2C and 2D, respectively).

Students can be grouped into “types”

Upon completion of the game, a detailed analytics report was provided to the author by ImmuneQuest indicating the amount of time students spent completing the levels as well as the success or failure of their attempts. Upon further analysis, it was clear that students could be grouped into four “types”: 1) The Perfectionist – students who spent an above-average amount of time completing the game and had a large number of aborted attempts when they realized they would not earn a higher score; 2) The Minimalist – students who spent a below-average amount of time completing the game, with few overall attempts, but usually completed all levels; 3) The Strugglers – students who spent an above-average amount of time with many losses and the inability to complete all levels; and 4) The Average students. The number of hours spent on the game, as well as number of wins, losses, and aborted attempts is shown in Figure 3A for all of the student types. Overall, the average student spent approximately five hours completing all three levels of the game with eight wins, three losses, and seven aborts. Furthermore, examination of final course grades indicated that there was no significant difference in grades between students in any of the groups (Fig. 3B).

FIGURE 3

Students playing ImmuneQuest could be grouped into “types” and had varying opinions on the game. A) Students were divided into four “types” – Average, Perfectionist, Minimalist, or Struggler – and hours spent playing the game as well as number of wins, losses, and aborted attempts were analyzed. B) Average overall course grade (out of 100%) for each “type” of student. No significant difference was observed between each “type.” C) A Wordle was created to illustrate student responses to a question asking them to use one word to describe ImmuneQuest. “Frustrating” was used by 7/20 students while all other words were written once.

Students recommend ImmuneQuest, even if it is “frustrating”

While the data indicate that student learning improved with the use of ImmuneQuest there is always a question of whether or not students appreciate this new learning modality. At the end of the post-assessment tool, students were asked to rank on a scale of 1 to 10 how much they enjoyed ImmuneQuest with 10 being equivalent to “very much” and 1 being “not at all.” Overall, students scored ImmuneQuest as a 6.6/10 for enjoyability. Students were also asked if they believed ImmuneQuest helped them to understand concepts in Immunology and 15/20 responded in the affirmative (Table 2). As part of the post-assessment, students were also asked to write one word to describe ImmuneQuest and a Wordle was designed to integrate all student responses (Fig. 3C). Interestingly, while many students (7/20) considered the game “frustrating,” including all members of the Minimalist and Struggler groups, the Perfectionist group had other terms to describe the game including “different” and “insightful,” as well as “time-consuming.” The same was true for the Average group; a small number of students (2/10) described the game as “frustrating” while the rest of the students used terms such as “challenging,” “fun,” “nerdy,” and “interactive.” Finally, when students were asked if they would recommend ImmuneQuest for future classes, 16/20 responded yes, indicating that although students found the game modestly enjoyable, yet frustrating, they still realized that the game aided their learning.

TABLE 2

Student responses to the post-assessment question “Do you think that ImmuneQuest helped you understand concepts in Immunology better? Why or why not?”

Affirmative Responses	Negative Responses
“Yes, it helped with solidifying topics and concepts that were taught earlier as well as helping remember them.”	“Not really – I was focusing more on beating the levels than learning about Immunology.”
“Yes, it tricked me into learning and was way easier than journal articles or some other assignment.”	“No – I like working with things hands-on. For example, laboratory experiments.”
“Yes, it did help me recognize the damage macrophages do to the host and how many steps are required to defeat infections.”	“No. I felt it may have reiterated them a bit, but it was mostly just frustrating.”
“Yeah, I suppose.”	“Not really, I just wanted to win.”
“Yes for the first 2 levels. The third level was hard to beat because there were too many upgrades.”	“Not really it was frustrating to play so I didn’t want to pay attention to the info.”
“Yes, it gives a visual representation of what occurs during an immune response.”
“It helped put things in perspective sometimes.”
“With the context from class, yes. Less restrictions on movement would be better/realistic.”
“Yes, visualization of learned concepts.”
“It helped me understand some concepts better because of being able to see certain components at work.”
“Some concepts, but maybe if we were able to play more levels.”
“Somewhat it didn’t go into much detail but some stuff was helpful”
“Yes, because I am a visual learner and it helped me to see the process.”
“Kind of. Seemed a little basic and then suddenly challenging.”
“It reinforced some of the stuff learned in class; but its other points were things not covered in class – it was helpful with the questions and powerups.”

DISCUSSION

Immunology has always been a field incredibly important to health professionals, but with the recent surge in media attention, it is increasingly relevant to everyone. Unfortunately, the nature of the immune system is such that it requires a study of living organisms, which is difficult, if not impossible, to do at most undergraduate institutions, meaning that most immunology students leave their courses without a significant active-learning component beyond lecture alone. The development of ImmuneQuest provides a mechanism by which students at any undergraduate institution can play a role in the immune system, learning how the cells and molecules work together as a whole to fight pathogens, in order to understand immunological concepts.

The results presented here indicate that, in addition to being a useful teaching tool that helps students visualize the function of the immune system, ImmuneQuest also increases student learning compared with lecture alone (Fig. 1). While the sample size of this study was relatively small (n = 20), the fact that significant gains were observed shows that these findings are relevant to classes of all sizes. However, it should be noted that the smaller sample size may compromise the statistical data leading to larger p values than would be expected. The fact that the highest learning gains were seen in topics covered by ImmuneQuest alone indicates that when students were required to learn the material on their own, they retained the information better than from the traditional lecture format. These data are not surprising based on research showing that flipped classrooms lead to increased student learning (9) and the use of a video game may provide a mechanism by which faculty can flip the classroom in a relatively enjoyable way for students. Interestingly, only modest gains were observed in assessment questions that were covered in both lecture and ImmuneQuest. After a thorough review of the survey questions, it was noted that many of the questions covered in both lecture and ImmuneQuest were more conceptual in nature, rather than detail/jargon-oriented. Although these questions were specific to immunology and none of the students had learned the information in a prior course, it is likely that students were able to utilize their critical reasoning skills developed in introductory biology courses to deduce the correct answer. Because of that, smaller learning gains than expected were observed in material covered in both lecture and ImmuneQuest (19%, Fig. 1B). If the conceptual questions were not included in the analysis, then we observe a 50% increase from pretest (40%) to posttest (90%), much closer to what was expected at the beginning of this analysis. These results support the idea that students struggle most with the jargon of immunology rather than the larger concepts. In the future, the assessment tool will be modified to better ensure that there are questions of equal difficulty in all three categories of delivery method, with a specific focus on the language of immunology in addition to major concepts. Furthermore, while the survey questions in this study focused primarily on knowledge and comprehension (Bloom’s Level 1 and 2), future studies will assess the ability of ImmuneQuest to improve higher-order learning including analysis and synthesis.

For educators, what is perhaps most important in the analysis of ImmuneQuest is that it aids learning for students of all abilities, not just the “best” students (Fig. 2). Indeed, high scores on each level of ImmuneQuest did not necessarily correlate with a high overall grade in the course. This is likely due to the fact that the ability to succeed in a video game is based not only on learning, but also additional skills that are not necessary for academic success. In the future, it would be interesting to determine whether the addition of ImmuneQuest to an immunology course specifically improves all grades in a course or whether it has a larger impact on weaker students.

It is important to note that the students were directed to start their ImmuneQuest experience after they had completed the unit exam on innate immunity in the course. While this was initially done to ensure that students would be able to answer most of the questions on ImmuneQuest because of their knowledge from lecture, increasing their chances of success in the game, it is unknown what the effect would have been on their exam scores and/or their opinion of ImmuneQuest if they had played the game concomitantly while learning the material in lecture. Furthermore, because students were only required to complete part 1 of ImmuneQuest, they did not take part in future levels, which include the addition of reinforcements (i.e., neutrophils) to innate immunity, nor were they able to visualize adaptive immunity, as these levels are still in development. Based on the learning gains already observed here, one can assume that ImmuneQuest will lead to increased student understanding as more levels are played. Student responses clearly indicated that students recommend ImmuneQuest for future classes and, specifically, recommend additional levels being played. Based on these data, it is expected that future immunology students will be required to complete all levels of ImmuneQuest for further assessment.

However, the requirement of further levels leads to an additional question: What happens to the students who were unable to complete level 3 in part 1? Three out of 20 students were unable to complete that level, even though they made several attempts, including the two Strugglers who spent a combined 17 hours on the game (Fig. 3A). Clearly they had not learned all of the skills necessary to succeed, but if they were unable to complete the first part, this would potentially preclude any success in later activities. If that is the case, then there is a concern that the addition of more levels may mean that some students do not obtain the learning gains because they cannot complete a significant portion of the game. This is something that educators will need to be mindful of when using ImmuneQuest in the classroom. If students are struggling with the material in the game and a large number are unable to successfully complete parts, faculty may need to be willing to adjust their expectations mid-course if the need arises. These results may also indicate that ImmuneQuest is testing students’ higher-order learning abilities to apply and synthesize material from previous levels in order to succeed in subsequent levels of the game. As stated above, future studies will examine the role of ImmuneQuest in improving all levels of Bloom’s Taxonomy rather than focusing on the lower levels of knowledge and comprehension.

Finally, it is not surprising that the students playing the game could be divided into certain archetypes that educators commonly see throughout their classrooms (Fig. 3). The fact that students at all grade levels fell into each category indicates that the Perfectionists were not always the top students in the class. Indeed, some of the weakest students in the class spent a great deal of time on the game in an attempt to improve their final course grades. Furthermore, student perceptions of the game, while not affected by their overall course grade, did appear to be influenced by the amount of time and effort they put into the game. The fact that all of the Strugglers and Minimalists found the game “frustrating” but only two Average students and no Perfectionists used the same terminology indicates that success in the game had a large influence on perceptions, which is to be expected. These data are important as the ImmuneQuest team considers ways to modify the game so that all students eventually succeed. It is quite likely that student perceptions of the game may change if all students are able to complete all levels.

Taken together, it is clear that ImmuneQuest was a useful learning tool suitable for students in an immunology classroom at large or small undergraduate institutions, and it is recommended for future courses, both by the student users themselves and as a result of this analysis. As alternative pedagogies, including video games and flipped classrooms, continue to be developed and a focus remains on students understanding major concepts rather than minutiae to succeed in the field, it is hoped that more opportunities like ImmuneQuest will be available in the future for educators and students alike.