Teaching the biological consequences of alcohol abuse through an online game: impacts among secondary students.

A multimedia game was designed to serve as a dual-purpose intervention that aligned with National Science Content Standards, while also conveying knowledge about the consequences of alcohol consumption for a secondary school audience. A tertiary goal was to positively impact adolescents' attitudes toward science through career role-play experiences within the game. In a pretest/delayed posttest design, middle and high school students, both male and female, demonstrated significant gains on measures of content knowledge and attitudes toward science. The best predictors of these outcomes were the players' ratings of the game's usability and satisfaction with the game. The outcomes suggest that game interventions can successfully teach standards-based science content, target age-appropriate health messages, and impact students' attitudes toward science.

INTRODUCTION

The yearly national study, Monitoring the Future, presents a detailed analysis of drug use among American youth (Johnston ). In 2009, the proportions of eighth, 10th, and 12th graders who admitted drinking an alcoholic beverage in just the 30-d period prior to the survey were 15%, 30%, and 44%, respectively (Johnston ). Alcohol use by teenagers is a continuing and omnipresent issue in most communities. Teen drinking has been related to higher levels of depression, anxiety, and behavioral problems (for a review, see Cerdá ). Given that an early-age onset of drinking is predictive of future problems of substance abuse, a strategy to teach about its consequences should be within the domain of schools, especially given how crucial adolescence is for determining future healthy behaviors (Grant, 1997; Hawkins ; Grant and Dawson, 1998). But how can science educators reinforce health content within an already overcrowded curriculum? Given the demise in many states of school-based health education, the need for innovative ways to maintain drug education is all the more crucial. The Web Adventure, N-Squad (http://nsquad.rice.edu), targeting middle school students, was designed to respond to these dual needs.

The goal of N-Squad was to integrate middle school science standards (National Research Council, 1996) into a digital game, while simultaneously presenting students with an age-appropriate view of alcohol's effects on the human body (see Klisch , for a description of the rationale and development of the game). Framing the issue of alcohol consumption in terms of its impact on the digestive, circulatory, and nervous systems enables teaching about alcohol to accomplish a dual purpose. The standards-based approach recommends more emphasis on “learning subject matter disciplines in the context of inquiry, technology, science in personal and social perspectives, and history and nature of science” (p. 113). The consequences of alcohol abuse offer an excellent social context for learning about body systems. For example, questions about how alcohol is handled by the circulatory system afford a practical look at the process of elimination and the concept of blood alcohol concentration (BAC). Relating alcohol consumption to the digestive system opens inquiry into the digestive pathway and the role of the alcohol dehydrogenase (ADH) enzyme in alcohol's metabolism. The nervous system's interaction with alcohol involves understanding brain plasticity and why alcohol, when consumed early in life, can produce detrimental long-term consequences.

It was with this backdrop that the N-Squad Web Adventure, funded by the National Institute on Alcohol Abuse and Alcoholism (NIAAA), was designed to address adolescent drinking from a science education perspective. An underlying assumption is that knowledge about the science of alcohol abuse contributes to better decision making regarding alcohol. Assessments of similar online games, The Reconstructors (http://reconstructors.rice.edu/), covering drugs of abuse; MedMyst (http://medmyst.rice.edu/), covering infectious diseases; and CSI—The Experience: Web Adventures (http://forensics.rice.edu), covering forensic science, resulted in significant gains in content knowledge (Miller , 2004, 2006, 2011; Klisch ). A similar goal, cognitive gains, was one of the intended outcomes for the alcohol-focused game.

One obstacle educators face is the decrease of interest in science that seems to occur with the transition from elementary to middle school. Many researchers have observed a steady decline of attitudes toward science in middle and high school, resulting in a decreasing number of students who pursue science at the college level (Simpson and Oliver, 1990; Gibson and Chase, 2002; Tai ; Martin ). This development provides the basis for growing concerns that the United States will not be able to provide a sufficient number of science professionals in the future (National Science Board, 2007; Kuenzi, 2008). Consequently, the promotion of favorable attitudes toward science at this critical time was another desired outcome of the N-Squad Web Adventure.

There is some disagreement in the literature about the correlation of attitudes toward science and science achievement. While affective attitudinal factors have long been recognized as possible predictors of individual differences in school achievement in general (Evans, 1965; Bloom, 1976), most recent studies show only a moderate or no relationship between attitudes toward science and science achievement (Simpson and Oliver, 1990; Houtz, 1995; Osborne ; Martin ). However, attitudes toward science have been reported to be a major influence when it comes to deciding whether to pursue science at the high school level or in college and as a future career choice (Simpson and Oliver, 1990; Tilleczek and Lewko, 2001; Gibson and Chase, 2002).

Attitudes toward science are shaped by many factors outside the school curriculum, such as media or parents and peers. However, two school-related factors influence students' attitudes toward science: 1) difficulties in learning the subject at school and 2) the content of the school curriculum (Osborne ). Previous studies have shown that activity-based science instruction, including computer-based activities, can enhance positive attitudes toward science (Ebenezer and Zoller, 1993; Gibson and Chase, 2002; Siegel and Ranney, 2003). Therefore, our working hypothesis assumed that if we provided students with an engaging learning experience through a role-play science game, knowledge about the consequences of alcohol would increase and more positive attitudes toward science would be the result.

NATURE OF THE INTERVENTION

In line with previously developed games (The Reconstructors and MedMyst), the N-Squad series was developed using narrative as a means of posing a problem and guiding inquiry toward a solution. The story line is presented episodically, with each episode requiring 30–40 min to play. Episode one and episode two end with a “cliff-hanger,” which is intended to serve as an incentive for the player to continue to the problem's final conclusion in episode three. N-Squad can be played on any Flash-enabled browser and is freely available online.

In the Web Adventure, the player enters a futuristic world in which he or she becomes a member of a team of forensic scientists (Figure 1) in the fictitious city of Neuropolis (hence the N-Squad name). Forensics was selected as a way to frame the alcohol content, based on input from student focus groups. Approaching alcohol abuse through the lens of forensic science provided a realistic role-playing perspective to engage players. In this instance, the player takes on the role of a forensic investigator arriving at the scene of an auto accident. The current interest in forensic science may be driven in large part by television media. Research by Malone (1980) and others (Malone and Lepper, 1987; Asgari and Kaufman, 2005; Foster, 2008) suggests that an educational game's effectiveness is related to how well the learning content is integrated into the fantasy content. Based on prior research, another dimension that influences learning outcomes is enjoyment/satisfaction with a gaming environment (Malone, 1980; Crook , 2008; Foster, 2008). Enjoyment/satisfaction is usually positively correlated with knowledge gain. Therefore, by making an engaging and satisfying game, we enhance the odds that the game will also be an effective teaching tool.

Figure 1.

Splash page of the N-Squad Web Adventure.

The problem presented in N-Squad centers on solving a hit-and-run incident that led to the death of a 19-yr-old driver. Was it an accident or foul play? The player must resolve this issue. During the investigation, the player is actively involved in the forensic process as an N-Squad team member. The career role-play activities require the player to examine the crime scene for evidence, take fluids from the body to test for the presence of ethanol, run basic toxicology tests and interpret the results, compare BACs, interview suspects, talk with experts in the treatment of alcohol abuse, and present conclusions based upon the evidence.

To avoid cognitive overload and disorientation, guided discovery was chosen to best promote constructive learning (Mayer, 2004). Following this concept, characters guide the player. Throughout the adventure, on-screen characters or pop-ups offer instructions and provide feedback and explain why a certain method is applied, providing a deeper insight into the rationale behind a particular scientific method (Figure 2). Questions or “checks for understanding” are embedded throughout the game to ensure a player does not continue with a misconception and to test the comprehension of what has been explained or deduced (Figure 3).

Figure 2.

Example of guided inquiry in the N-Squad Web Adventure. An on-screen character guides the player in performing a headspace gas chromatography analysis.

Figure 3.

Example of a check for understanding in the N-Squad Web Adventure. Feedback is given after the player chooses the wrong answer.

N-Squad departs significantly from “life skills” or “drug prevention” approaches to alcohol education. It is anchored in science content and meant to be complementary to these types of drug education programs by providing answers to why early and excessive alcohol consumption is detrimental. Learning objectives that incorporate secondary school science curricula were developed together with our panel of subject matter experts (middle and high school teachers). The learning objectives were then used in the game design process to align the different interactive components in the game to the intended learning. For example, one learning objective was to “recognize that alcohol can cause long-term changes in the way the liver functions.” To address this learning objective, we included a virtual histology analysis in episode 1 of N-Squad, in which players are asked to identify changes in the liver caused by alcohol and to conclude how these changes affect liver functions.

Many of the scientific learning objectives were addressed through minigames within the story line. The approach of embedded games has previously been shown to be effective in promoting comprehension of complex science content (Miller ; Miller and Crook, 2008). For example, the process of alcohol metabolism in the liver by the ADH enzyme is conveyed through a game, ADH Attack (Figure 4). In ADH Attack, the player must capture as many alcohol molecules as possible before they reach the brain. With higher levels of alcohol consumption, the player realizes that it is not possible to capture (i.e., metabolize) all of the alcohol molecules.

Figure 4.

Screen capture from ADH Attack. After the player clicks “begin game,” alcohol molecules start to move from the capillary into the liver cell. The player's goal is to “attack” (e.g., inactivate) the alcohol molecules with ADH molecules.

The embedded minigames, such as ADH Attack (Figure 4), Operation: Plastic Brain (Figure 5), and the BAC-O-Meter (Figure 6) provide visualizations of concepts and provide the “stickiness” that encourages players to return to the episodes. An overview of the specific learning objectives of each N-Squad episode and examples of game activities that address these learning objectives are given in Table 1.

Figure 5.

Screen capture from Operation: Plastic Brain. The player is asked to assemble a neuron using parts from a “cell bin.”

Figure 6.

Screen capture from BAC-O-Meter. The player can estimate BAC by entering his or her gender, weight, servings per hour, and metabolic rate.

Table 1.

Learning objectives covered by the N-Squad Web Adventure

	Learning objectives	In-game activity (example)	Question(s)a
Episode 1	Classify alcohol as a depressant based on its effect in the body.	Database search, “Truth or Trash” quiz	5, 11
	Identify organs of the digestive systems and their functions.	“Gut Check”: interactive database and quiz	6, 12
	Trace the path alcohol takes through the digestive system.	Autopsy	2, 7
	Describe the role of the liver in alcohol elimination.	Autopsy, histology analysis	10
	Recognize that alcohol can cause long-term changes in the way the liver functions.	Histology analysis	8, 9, 15
Episode 2	Analyze a blood sample for alcohol (ethanol) using headspace gas chromatography.	Headspace gas chromatographic analysis	16, 17
	Identify organs associated with the absorption, distribution, and elimination of alcohol.	“The Journey”: interactive journey through the body following alcohol molecules	13, 14
	Comprehend how BAC is related to level of intoxication.	“BAC-O-Meter”: interactive estimation of BAC based on influencing factors	18–23, 25, 31
	Examine factors that influence BAC.	“BAC-O-Meter”: interactive estimation of BAC based on influencing factors	1, 26, 27
	Recognize the importance of ADH in alcohol metabolism.	“ADH Attack”: interactive simulation of alcohol metabolism by ADH molecules	24, 28–30
Episode 3	Examine factors that influence alcohol use.	Interactive database search	32, 33, 36
	Investigate the structure and function of a neuron by creating a nerve cell model.	“Operation: Plastic Brain”: neuron reconstruction	34, 35, 37, 38, 44
	Identify the regions and functions of the brain affected by alcohol.	“Mind Storm”: interactive database of the brain and its functions	39–41
	Know how the teenage brain and the adult brain differ in their response to alcohol.	“Operation: Plastic Brain”: interactive simulation of the process of “brain wiring”	3, 42, 43
	Observe how alcohol alters neurotransmission to produce short- and long-term effects.	“Art and Alcoholism”: interactive simulation of the neurotransmission process	44, 45
	Explore the nature of alcoholism.	“Art and Alcoholism”: interactive simulation of alcohol's effects on neurons	4, 45, 46
	Interpret data and draw conclusions from a virtual DNA profiling experiment.	DNA fingerprint analysis	—

aContent knowledge questions that address this learning objective (see Supplemental Material).

METHODS

Participants

Approval of the study design and protection of human subjects was obtained through the Rice University Institutional Review Board. Secondary students were recruited through an online solicitation sent to middle and high school science teachers who had volunteered at various professional development workshops. Although the original instructional design of the website targeted middle school students, through analysis of Web logs and comments received from players and/or teachers, we observed a large audience among high school students enrolled in a variety of science or special forensic science classes. From a design perspective, we were interested in determining whether high school students would have gains in achievement similar to those of our intended audience. This data would also provide us with information about the leveling of concepts appropriate to middle and high school.

Parental consent and student assent were required for participation, and teachers were provided a similar computer game on infectious diseases for students who elected not to participate in the alcohol game study. Alternatively, teachers could allow their students to play the N-Squad game without participation in the study. A $200 stipend was offered to teachers for their adherence to the testing protocol and for collecting both parental consent and student assent. To protect students from coercion, it was made clear that study participation was entirely voluntary and that participation would not impact students' grades or teacher stipends. The final sample drawn from six teachers in six states consisted of 335 students (46% male; 53% female). The student population was composed of 170 middle school students (grades 6–8) and 165 high school students (grades 9–12) and represented 23.9% from urban areas, 29.8% from suburban areas, and 46.3% from rural areas. Students described themselves as Caucasian (44.8%), African American (21.8%), Hispanic/Latino (25.6%), Asian/Pacific Islander (2.7%), Native American (0.6%), or other (4.5%).

Procedure

Five online sessions, ranging from 20 to 40 min in length, were part of the intervention. A coding system blinded the researchers to the identity of the students. Subjects with missing data and those who did not complete all three episodes were eliminated from the data set.

All measures were administered in schools' computer laboratories with students working at individual workstations. Session 1 (pretest) included the administration of the demographic and computer experience measures and the pretest of content knowledge and of attitudes toward science. After an intentional 3-d delay, sessions 2, 3, and 4 occurred—one per day—in which students played the three N-Squad episodes. Session 5 was a posttest occurring 3 d after session 4 and included the same content knowledge and attitudes toward science measures and game satisfaction and usability scales. Teachers were instructed to avoid preteaching the content of the website before the pretest and to refrain from providing recap of content included in the game until after the posttest.

Instruments

Science Content Measures.

The test of content knowledge consisted of 46 multiple-choice items with four response options (see Supplemental Material for the complete scale). The content questions were developed to directly address the learning objectives covered by different activities in each episode of the game (see Table 1). In an effort to target a middle school reading level, middle school teachers not participating in the field test were involved in wording the questions. Construct validity was established by involving subject matter experts (middle and high school teachers) in the development of the items. The content questions included simple recall of facts, as well as more challenging questions that required a deeper understanding of body system processes. For example, a simple recall question about the metabolism of alcohol was “Alcohol is broken down by enzymes in the:” (correct answer: “liver”), whereas a question requiring understanding of the alcohol metabolism process asked “Intoxication can occur if:” (correct answer: “alcohol molecules outnumber ADH molecules”). Most learning objectives were addressed by multiple content questions (see Table 1). Internal consistency reliability estimates were 0.74 for the pretest content knowledge scale and 0.86 for the posttest scale, which included questions identical to the pretest in a different order. This increase in internal consistency reliability estimates between pre- and posttest is expected, since intercorrelations among test items increase when subjects know more about the subject matter.

Attitudes toward Science.

A five-item scale ranging from “strongly disagree” to “strongly agree” was used to evaluate attitudes toward science. The scale was developed based on a 20-item measure by Francis and Greer (1999). We chose this scale because it was shown to be unidimensional and reliable, and its content validity was supported by a positive correlation to the number of science-related subjects studied by the sample used in the study. An example from our scale is: “Science is too difficult to be any fun” (reverse coded; see Supplemental Material for the full instrument). The same science attitudes items were used for pre- and posttests. Internal consistency reliability estimates (Cronbach's alpha) were 0.71 for the pretest attitudes toward science and 0.74 for the posttest.

Game Experience.

Two questions assessed computer game experience: 1) “On average, how much time do you spend PER WEEK on playing games on the computer?” Participants reported the number of hours per week on a scale with 1 = < 1 h, 2 = 1–4 h, 3 = 5–10 h, and 4 = 10 h or more. 2) “How experienced are you with playing games on the computer?” Responses were 1 = not at all, 2 = somewhat, 3 = experienced, and 4 = very experienced. These two items were combined to create a measure of game experience, with a reliability estimate of 0.56 (using the Spearman-Brown prophecy formula on the correlation between these two items, r = 0.39). Although this reliability estimate is low, the measure was positively correlated with the pretest of content knowledge (r = 0.16, p < 0.01) and the pretest of science attitudes (r = 0.12, p < 0.05) and was used as a control variable in subsequent analyses.

Postintervention Measures

In addition to the content knowledge and attitudes toward science items on the pretest, scales to measure satisfaction with the game and the game's usability were part of the postmeasures. The usability scale was developed based on usability problems that occurred in game-play sessions with focus groups and consisted of three items to determine the ease of game play. The satisfaction scale was based on a scale previously used to assess how well participants liked the game (Miller , 2004, 2006; Klisch ) and consisted of six items (see Supplemental Material for the complete instruments). Both scales used a five-item Likert scale ranging from “strongly disagree” to “strongly agree.” Internal consistency reliability estimates (Cronbach's alpha) were 0.92 for satisfaction and 0.66 for usability.

RESULTS AND DISCUSSION

The overall approach involved examining the differences between pretest and delayed posttest results on content knowledge and attitudes toward science using a repeated-measures t test. Following this, we conducted multiple-regression analyses to determine which of the variables contributed to posttest knowledge and posttest attitudes toward science.

Content Knowledge Acquisition

To address the question of whether learning gains resulted from N-Squad exposure, a paired-sample t test was performed across all grades to gauge pre- to posttest knowledge acquisition. The mean on the pretest was 18.36 (SD = 6.20; 39.9% correct). The posttest mean was 22.32 (SD = 8.45; 48.5% correct). Performance on the posttest was significantly better than performance on the pretest (t (334) = 12.04, p < 0.01), and the effect size was large (d = 0.98; Cohen, 1988), indicating that students' content knowledge significantly increased after they played the game. A repeated-measures analysis of variance was conducted using the general linear model (GLM), with pretest and posttest content knowledge entered as two measurement occasions and grade level (middle or high school) entered as a between-subject variable. There was significant interaction between grade and content knowledge acquisition (F (1333) = 9.84, p < 0.01 ). Middle school students' means were 16.86 (SD = 5.46) on the pretest and 19.81 (SD = 7.76) on the posttest, compared with high school students' means of 19.91 (SD = 6.54) on the pretest and 24.90 (SD = 8.38) on the posttest. One can only speculate as to reasons for these differences, but high school students may have had more exposure to some of the basic concepts due to prior interventions or through more years of classroom instruction, which would have provided them with a different baseline of knowledge prior to playing N-Squad. For example, high school students would be better positioned to understand the discussion of enzymes, since this topic is likely to be covered in their high school biology curriculum. One is not nearly so confident that middle school students would have had similar exposure. As stated previously, the inclusion of high school students was intended to provide some feedback on the grade span that would benefit from instructional content originally designed for middle school. Follow-up studies would be needed to tease out the reasons for the apparent differences between grades. Nevertheless, the middle school students' gains were significant, indicating that the game was still efficacious at the middle school level in terms of students acquiring new science knowledge.

Changes of Attitudes toward Science

To determine the intervention's impact on attitude change, we compared the pretest and posttest attitudes toward science scale using the same paired-sample t test approach as was used to assess content knowledge acquisition. Mean science attitudes were slightly more positive on the posttest (mean = 3.67, SD = 0.69) than they were on the pretest (mean = 3.59, SD = 0.69). Although the effect size was relatively small (d = 0.23), the difference was significant, t (334) = 2.91, p < 0.01. A GLM repeated-measures analysis was conducted with pretest and posttest attitudes entered as two measurement occasions and grade entered as a between-subject variable. There was no evidence of interaction between grade and attitude change (F (1333) = 6.12, p = 0.17). In other words, attitudes did not differ significantly between middle school and high school students. As game designers, we would like to maximize the attitude shift in future games. One element that seems to factor into the attitude shift is the degree to which the player engages in role-playing science careers (Miller ). The first-person, forensic investigator approach, while present in N-Squad, was not as well developed as in another of our games (CSI—The Experience: Web Adventures). Further research should investigate whether the degree and authenticity of role-play in science games will increase students' “science identity” by allowing players to view themselves as successful in a science-related job.

Satisfaction, Usability Ratings, and Game Experience

The mean for satisfaction with the game was 3.56 (SD 0.91) and the mean for game usability was 3.39 (SD 0.78) on five-point scales. As such, the usability of the game and satisfaction with the game were rated above average. In terms of time spent on playing games, 60% of all students reported that they spend less than 1 h per week playing games on the computer, 29% reported spending 1–4 h, 8% 5–10 h, and 3% more than 10 h. Assuming that time spent playing games correlates with game experience, these numbers are consistent with students' self-ratings on game experience: 11% rated themselves as being not experienced with playing games on the computer, 48% as somewhat experienced, 27% as experienced, and 14% as very experienced.

Differences between Genders

To analyze whether knowledge gains differed between genders, a further GLM analysis was conducted. Pretest content knowledge and posttest content knowledge were entered as two repeated-measurement occasions, and gender was entered as a between-subject variable. The analysis suggested no difference between genders with regard to knowledge acquisition (F (1333) = 0.94, p = 0.33).

Another GLM analysis was done with pretest and posttest science attitudes entered as two repeated-measurement occasions and gender entered as a between-subject variable. There was no evidence of interaction between gender and change of attitudes toward science (F (1333) = 0.01, p = 0.93). An independent sample t test with female and male students revealed that neither satisfaction nor game usability ratings differed between genders (satisfaction: t (333) = −0.53, p = 0.60; game usability: t (333) = −0.10, p = 0.93). Taken together, female and male students benefited equally from the game with regard to content learning and change of attitudes toward science, and there was no difference between genders in satisfaction and usability ratings. These results provide us with some level of confidence that the game does not have an obvious gender bias that would result in differing impacts; but rather, that N-Squad is equally effective for male and female middle and high school students.

Correlational and Hierarchical Regression Analysis

Correlations among all study variables are shown in Table 2. Posttest performance on content knowledge and science attitudes was correlated with the corresponding pretest variables; however, what is noteworthy is the correlation between usability and posttest performance and the correlation between posttest science attitudes and satisfaction with the game. Posttest performance was highly correlated with game usability but was not correlated with satisfaction with the game. Conversely, posttest science attitudes correlated with satisfaction more strongly than it did with usability.

Table 2.

Correlations between study variablesa

		1.	2.	3.	4.	5.	6.	7.
1.	Pretest	(0.74)
2.	Posttest	0.74	(0.86)
3.	Satisfaction	−0.03	0.05	(0.92)
4.	Usability	0.30	0.31	0.09	(0.66)
5.	Pretest science attitudes	0.21	0.18	0.34	0.16	(0.71)
6.	Posttest science attitudes	0.17	0.19	0.47	0.17	0.73	(0.74)
7.	Game experience	0.16	0.10	-0.02	0.08	0.12	0.05	(0.56)

a Reliabilities (1–6: Cronbach's alpha; 7: Spearman-Brown prophecy) on the diagonal. N = 335. Values over r = 0.11 significant at the p < 0.05 level; values over 0.15 significant at the p < 0.01 level.

Although zero-order correlations provide information about the relations among the constructs of interest, they do not take into account the multi-colinearity among predictors. To further examine the independent contribution of these predictors on learning and science attitudes change, we conducted a series of hierarchical regressions.

Determinants of Content Knowledge Learning

In the first multiple-regression analysis, posttest content knowledge performance was the criterion variable (Table 3). We entered pretest content knowledge performance, pretest science attitudes, and game experience as the first and second steps to control for prior knowledge before playing the game, prior science attitudes, and individuals' game experience. In a third step, posttest science attitudes, usability rating of the game, and satisfaction with the game were entered. As shown in Table 3, after controlling for gaming experience, pretest performance, and pretest science attitudes, only usability was a significant predictor of posttest performance. This result indicated that those who found the game features easier to use, game instructions easier to follow, and so on, were more likely to learn from it. In other words, students who reported less difficulty with the game play may have expended less cognitive effort on deciphering the game mechanism and more cognitive effort on the game content, resulting in higher content knowledge gains. Thus, increasing the ease of game play might improve student learning, as long as the game is challenging enough to keep students engaged and excited. Although the effect of usability on posttest performance is small in our study, it may have been attenuated by the relatively low internal consistency of the game usability scale we used, which had only three items. Future research is needed to refine the game usability scale and to further analyze the connection between usability and learning.

Table 3.

Regression of posttest content knowledge performance on attitude variables controlling for pretest content knowledge and game experience

Predictor	ΔR2	β
Step 1	0.497**
Pretest		0.70**
Game experience		−0.05
Step 2	0.002
Pretest		0.69**
Game experience		−0.06
Pretest science attitudes		0.05
Step 3	0.013*
Pretest		0.67**
Game experience		−0.06
Pretest science attitudes		−0.01
Posttest science attitudes		0.06
Usability		0.10*
Satisfaction		0.03
Total R2	0.512**

* p < 0.05, ** p < 0.01; n = 335.

Table 4.

Predicting posttest science attitudes with other attitude variables (controlling for pretest science attitudes and game experience)

Predictor	ΔR2	β
Step 1	0.53**
Pretest science attitudes		0.73**
Game experience		−0.04
Step 2	0.06**
Pretest science attitudes		0.63**
Game experience		−0.02
Posttest science attitudes		0.05
Usability		0.04
Satisfaction		0.25**
Total R2	0.59**

* p < 0.05, ** p < 0.01; n = 335.

Determinants of Science Attitude Changes

In the second hierarchical multiple-regression analysis, posttest science attitudes were the criterion variable. Game experience and pretest science attitudes were entered in the first step as control variables; posttest content knowledge performance, satisfaction with the game, and game usability rating were entered in the second step. We found that after controlling for game experience and pretest science attitudes, only satisfaction with the game was a significant predictor of posttest science attitudes (Table 4). This finding is consistent with other research showing that a positive learning experience, which is reflected in the game satisfaction ratings, can positively influence attitudes toward science (Ebenezer and Zoller, 1993; Gibson and Chase, 2002; Siegel and Ranney, 2003).

CONCLUSIONS

The success of the three-episode N-Squad game offers insights into one effective method for achieving dual purposes—using a standards-based approach to deliver age-appropriate health messages, while also fostering more positive attitudes toward science. The gains in science knowledge are not totally without precedent. Prior Web Adventures on other topics that focus upon specific learning objectives and incorporate solid instructional design have proven equally effective (Miller , 2004, 2006, 2011; Klisch ). Retention of knowledge after a 3-d delay between exposure and testing is extremely positive. Repeating the measures with an even longer delay would strengthen claims of lasting learning outcomes. There is some concern that the gain in scores reflects only an average of 48.5% on the posttest. However, given that the total intervention was less than 3 h and that we intentionally requested that teachers not intercede with any instruction until after the posttest, this gain seems reasonable. If N-Squad were used in a typical classroom, one would expect teacher-led discussions and an alignment with current curriculum. Within this context, student retention would potentially be even greater; however, for the purposes of our study we intentionally elected not to bias the results with variations in teacher instruction.

Although all science content questions were aligned with the learning objectives and corresponding game activities, the questions reflected a recall or comprehension level of understanding in Bloom's taxonomy (Bloom, 1956). Further research that might include open-ended questions, interviews, or classroom observations would be needed to assess whether N-Squad enables students to transfer and apply concepts learned in the game.

With regard to the appropriateness of the intervention beyond the intended middle school audience, the data suggest that the topics covered are not totally redundant with high school students' existing knowledge of alcohol effects on the body, and N-Squad could be used effectively for grades 6–12.

Another limitation of this study is the lack of a control condition. A comparison of students who played N-Squad with students who either did not receive any intervention or who played a different, unrelated game would allow a stronger argument about the efficacy of the intervention. In the interest of avoiding excessive testing that would encroach upon instructional time, this design was not selected.

While knowledge gain about the effects of alcohol on the body were significant, it is equally noteworthy that the game intervention improved attitudes toward science. Given the longitudinal data of Tai regarding the importance of students' early career expectations as indicators of ultimate science, technology, engineering, and mathematics (STEM) career choices, there is the opportunity to use games as one of the many tools that can shape attitudes toward science. One can imagine how adolescents might “try out” a variety of science careers through authentic games as a means of spurring STEM career decisions.

Two additional lessons learned from this study revolve around the importance of player satisfaction and usability as factors that contribute to knowledge and attitudinal change. Though it is not surprising that satisfaction and usability play important roles in game design, it reminds developers of educational games how significant the attributes of satisfaction and usability are in producing knowledge gains and attitudinal shifts.