Development of a malaria knowledge test for student travelers.

This paper describes a malaria knowledge test (MKT) developed to evaluate a web-based game for students who increasingly travel to malaria-risk regions of the world. The 18-item MKT was structured according to the dimensions of the self-regulation model (SRM) to measure the accuracy of students' beliefs about malaria. An experimental design was used to compare three game versions. Students (N=482) participated in 2010 by completing a pre-test, playing a Web-based game simulating student travel to malaria-endemic destinations, and completing a post-test. Study data support the validity and reliability of the MKT for the evaluation of malaria education interventions and for student self-assessment. Use of the MKT to evaluate an educational game about malaria revealed a strong overall learning effect and discrimination by game version, travel experience, and SRM dimension. This 5-min test may also be adapted for educational outreach purposes among health care providers globally, residents of malaria-endemic regions, and other high risk travel groups (e.g., elderly, chronic health conditions, pregnant, or returning to malaria-endemic regions to visit friends/relatives).

Introduction

As student travel to malaria-risk regions increases,[1] malaria education becomes a more pressing need because infection can rapidly lead to permanent disability or death.[2] Many travelers have limited knowledge of malaria despite it being a leading cause of death worldwide, with 200-300 million infections and nearly one million deaths each year.[2,3] In the absence of a malaria vaccine, risk-reduction depends upon the consistent use of personal protection measures and immediate response to the development of symptoms.[4] Students who become ill while abroad disrupt activities and unnecessarily burden local health care systems. When symptoms develop after returning home, clinicians may be less familiar with the complexities of malaria diagnosis and treatment, and malaria parasites may be introduced into areas previously cleared of transmission.[3] Factors favoring malaria transmission include a low knowledge of malaria and the discomfort, inconvenience, and cost of personal protection measures.

This paper describes a malaria knowledge test (MKT) developed to test the efficacy of an educational game (a web-based travel-simulation that informs students about malaria through scenario-based decision-making and outcomes that include health, illness, disability or death).[5,6] MKT development was undertaken when a suitable knowledge test was not identified in the literature. Youthful travelers are uniquely vulnerable because they have less experience with the atypical characteristics of many tropical diseases,[2,7-11] and they are susceptible to the developmentally normative risk-taking behavior characteristic of adolescents and young adults.[12,13]

Materials and Methods

Although a more detailed description of the overall experimental design, participant sample and MKT items is provided elsewhere,[5,6] the highlights of MKT development and evaluation are reviewed in this report. Following iterative testing of the MKT and an associated game, a summative evaluation was conducted using an experimental design that compared three game versions that varied player access to feedback in response to game decisions. Feedback was chosen as the factor of interest because there is limited evidence about which feedback strategies are most effective in the context of brief-exposure educational games, particularly those addressing complex topics with potentially serious consequences if content is misinterpreted. Each treatment group was exposed to one of three types of explanatory feedback: i) a rationale for why a game action had a positive or negative effect, provided automatically as a pop-up text box after each player action (in-game decision); ii) a rationale accessed at player-discretion by clicking on a link associated with that action; and iii) no access to a pop-up rationale. Students were recruited from seven Midwestern U.S. campuses. Study abroad program administrators distributed information about the study using internal e-mail lists; study co-investigators had no direct access to these email lists.

Procedure

Approval was obtained from the Institutional Review Board on each campus prior to study deployment. A link within the invitation letter was used to access the study protocol after participants read the informed consent information. Consent was implied when students clicked on the study link, and participation could be terminated at any time. Participants were linked sequentially from the pre-game survey to the game and then to the post-game survey. Following post-game survey submission, participants were directed to a summary site where they were provided with their pre/post MKT scores and additional malaria resources.

Development of the malaria knowledge test

The MKT was structured using the dimensions of the self-regulation model (SRM)[5,14] and content consistent with US and international standards for malaria education.[2-4,7] The SRM is centered on the personal construction of meaning that is processed both cognitively and emotionally.[14] The SRM proposes that emotions influence behaviors differently than cognitive representations, and are a crucial component of theories to predict how individuals respond to perceived health risks.[15,16] Cognitive representations are assessed within the following dimensions in response to a perceived health threat: identity (symptoms and labels), cause, timeline, consequences (seriousness), control and/or cure, and coherence (overall understanding). Emotional representations are examined through the measurement of an individual’s degree of fear, anxiety, or worry related to a health threat.

Individual responses to health challenges are determined by personal beliefs and expectations.[14] Coping activities are undertaken in response to health beliefs and the worry or anxiety elicited by these beliefs.[16] In applying the SRM to malaria, an accurate understanding of malaria risk and prevention options is needed to modulate an appropriate emotional response to malaria as a health threat. This emotional processing may then prompt risk reduction actions, such as a rapid response to potential malaria symptoms. It is important to improve representational accuracy for a complex disease such as malaria that has serious consequences and is prevalent in many areas of the world, yet is largely unfamiliar to individuals from non-endemic countries.

The process of enhancing the accuracy of personal representations takes time and often repeated exposures because health beliefs can be resistant to change.[14,15] According to the SRM, a student who believes that malaria is no big deal (i.e., unlikely and not worrisome) will have minimal motivation to engage in prevention behaviors. When this student experiences debilitating symptoms secondary to blood stage malaria infection, or learns of a tragic malaria outcome for a peer, this new information impacts the student’s representation for malaria and may increase motivation to engage in prevention behaviors. Conversely, returning from travel to a malaria-endemic country with no apparent symptoms can reinforce inaccurate information about malaria, such as transmission being rare and/or not serious, and may reduce motivation to seek testing for potential malaria symptoms during the year following risk exposure. Learning primarily from personal experience may be dangerous in the case of malaria. However simulated experience offers a safe approach for the refinement of cognitive and emotional representations through trial and error, with feedback provided that is specific to the actions taken.

The MKT was constructed to gauge the accuracy of malaria knowledge in a study abroad student population. Further, there is evidence that learning experiences that increase an individual’s emotional engagement and perception of risk are more likely to change beliefs and behavior than abstract information.[17,18] To measure the emotional representation for malaria, students were asked to rate their degree of worry about being infected during travel to a malaria-endemic region. This item was part of a risk perception assessment that also included perceived likelihood of being infected with malaria.

A pilot test of the initial 12-item MKT was administered on paper with 13 students who were close to departure for a field experience in a malaria-endemic location. The mean age was 22 years, with about half (54%) reporting graduate student status and all recently receiving classroom education about malaria prevention. The MKT items were scored 0 for incorrect and 1 for correct; the overall mean score was 91% correct. Revisions to increase consistency and flow included adding six questions to achieve three per SRM dimension and consistently offering five response options in a multiple choice format (three true/false questions were restructured).

A second pilot test was conducted over the Internet using Zoomerang survey software (MarketTools, San Francisco, CA, USA) with 36 pre-nursing undergraduate students who had no prior classroom exposure to malaria education. Participants were primarily female (97%) and had a mean age of 19 years (range 18-21). Eighty percent planned to study abroad, 11% had already returned from a study abroad experience, and 73% had previously traveled outside the United States unrelated to a study abroad experience. The mean MKT score was 50%. Following this second pilot test, three cognitive interviews were conducted with students to further explore their interpretation of the meaning of question stems and response options, and what they liked most and least about content and formatting. Following minor text revisions, face validity was tested with five study abroad student reviewers. Content validity was tested for items within each dimension of the SRM with five travel health specialists, and a survey research specialist suggested formatting edits to enhance online comprehension.

The final pre-game survey consisted of an 18-item multiple-choice MKT with five response options (Appendix A), and two items that assessed perceptions about the likelihood and degree of worry related to malaria infection that could be acquired during travel to a malaria-endemic region (Appendix B). The post-game survey additionally included nine demographic items, one player satisfaction item, and an open-ended text box for comments.

Results

First, select findings from the comparison of game versions are provided because the MKT capacity to detect knowledge differences between groups (by pre/post knowledge gain, game version, past travel, SRM dimension) is central to the evaluation of MKT efficacy. Second, an SRM dimensional analysis and examination of responses to individual MKT items are reported in order to inform future malaria risk reduction interventions.

Demographic data is summarized in Table 1 for the 482 participants who completed the study protocol. MKT mean completion time was approximately 5 min. The Kuder-Richardson 20 (KR-20) coefficient of reliability for measures with dichotomous choices (correct/incorrect) was 0.82 pre-game and 0.73 post-game. The pre-game MKT mean score was 6.6 (37% correct, SD=4.2) and the post-game mean score was 13.8 (77% correct, SD=2.8). The pre/post comparison (t=39.38, df=481, P<0.001) and large effect size (0.767) demonstrated a strong overall learning effect.

Table 1

Participants’ characteristics.

Mean age in years=21 (SD=4.6)	n*	%
Gender: female	352	73
Student status
Undergraduate	421	92
Graduate	35	7
Racial/ethnic background
Asian	30	7
Black or African American	12	3
Hispanic or Latino/a	10	2
Native American or Alaskan Native	10	2
Native Hawaiian or Pacific Islander	2	0.4
White or Caucasian	393	86
Study abroad status
Interested, but no specific plans	36	8
Planning	152	34
Currently abroad	76	17
Returned	187	35
Prior travel to a malaria-risk region
Yes	155	32
No	299	63
Don’t know	24	5
Future travel to a malaria-risk region
Yes	309	64
No	129	27
Don’t know	40	8
Computer/video game play frequency
Never or rarely	239	51
At least once a month	111	23
At least once a week	62	13
Several times a week	40	8
Daily	24	5

n, may vary due to missing cases.

Pre/post difference scores on the MKT were also compared by game version. The group that automatically received explanatory feedback after each game action showed the greatest knowledge gain (F(2482)=6.5, P<0.005). There were no differences in knowledge gain by study abroad status or by frequency of playing computer games. Perceptions of worry about malaria infection increased from pre- to post-game (t=3.54, df=477, P<0.001, 2-tailed) and there were differences for the variables of pre/post knowledge scores and worry ratings when examined by travel experience. Participants who responded yes to an item about past travel to a malaria-risk region had higher mean knowledge scores both pre-game F(2478)=39.85, P<0.0001) and post-game (F(2478)=7.54, P<0.001) compared to those who responded no or don’t know, although the malaria-experienced group still answered only 49% of the pre-game knowledge questions correctly. Participants reporting past travel to a malaria-risk region had lower worry ratings pre-game F(2478)=17.90, P<0.01 and post-game F(2478)=21.44, P<0.01 as compared to those without this reported risk exposure. However worry ratings increased from pre- to post- for both the malaria-experienced (t=3.098, df=154, P<0.01) and the malaria-naïve (t=2.483, df=298, P<0.01) groups.

A self-regulation model dimensional analysis revealed that knowledge increased in every dimension, although the timeline and consequences dimensions had the lowest mean scores at both time-points (Table 2). The pre-test items were examined to determine common errors in baseline malaria knowledge. For timeline, 92% were unable to identify the shortest time period from infection to symptoms and 67% did not recognize the possibility of a delayed symptom onset. For consequences, 86% were unaware that malaria infection may result in severe anemia, kidney failure, and brain effects, and 80% were unaware that malaria might be rapidly fatal. Additionally, 68% did not know that the infectious agent is a parasite (cause) and 82% failed to recognize that the peak biting time for malaria transmission is dusk to dawn (coherence). When participants were asked to choose the best prevention measure from a list of four actions and one don’t know response (control), 49% opted for a non-existent vaccine and 25% selected don’t know. In contrast, identity was the dimension with the greatest gain in representational accuracy after participants completed game play; nearly all correctly identified malaria symptoms (97%), the type of diagnostic test (99%), and the degree of risk globally (92%).

Table 2

Malaria knowledge test scores by self-regulation model (SRM) dimension (N=482, 3 items per SRM dimension).

Dimension	Pre-test mean (SD)	Post-test
Identity	1.5 (1.13)	2.9 (0.43)
Cause	1.4 (0.87)	2.5 (0.64)
Timeline	0.5 (0.76)	1.8 (1.02)
Consequences	0.8 (0.88)	1.7 (0.85)
Control	1.3 (0.95)	2.6 (0.66)
Coherence of understanding	1.0 (1.00)	2.3 (0.82)

Among respondents reporting past travel to a malaria-risk region on the pre-test, 47% opted for a non-existent vaccine and 11% selected don’t know when asked to choose the best prevention measure. The post-test responses for this group were examined as a best-case scenario for malaria comprehension since these students had experienced both actual and simulated exposure to malaria transmission. The lowest MKT scores persisted within the SRM dimensions of timeline and consequences. The three items with the most incorrect responses on the post-test were: malaria can be rapidly fatal (63%); malaria infection may result in severe anemia, kidney failure, and brain effects (46%); symptoms may first be noticed months after being infected (44%).

Discussion

This section discusses what the game evaluation data reveal about the MKT as an assessment instrument, and how the MKT is linked to the larger objective of reducing risk among travelers to malaria endemic destinations. The MKT was constructed along the dimensions of the SRM to measure the accuracy of malaria beliefs in a study abroad student population. Study findings demonstrated acceptable internal consistency (pre and post KR-20) and pre/post sensitivity to change for knowledge and worry about malaria. The MKT items also successfully discriminated between groups by the game version, by travel experience, and by SRM dimension. Pre-game MKT scores indicated that malaria knowledge deficits exist, while post-game MKT scores improved across all SRM dimensions in both the malaria-experienced and malaria-naive groups.

This substantial pre/post learning effect is a key precursor for competent adherence to travel recommendations. Findings also revealed an increase in the perception of worry about malaria after playing the game. The potential for increased worry to motivate risk-reduction actions is supported by prior data indicating that risk examined as a feeling was a better predictor of risk-reduction behavior than a purely cognitive probability judgment.[17,18] Participants showed evidence of persistent health belief inaccuracies in the two representational dimensions with the lowest pre/post knowledge scores: timeline and consequences. These dimensions may be more complex to comprehend, such as the potential for a delayed symptom onset or for a rapid progression to permanent disability or death once symptoms develop. These illness characteristics of malaria deviate from illnesses most familiar to this study population.

The significant pre/post increase in MKT scores among participants reporting past travel to a malaria-endemic region indicates that even malaria-experienced travelers can benefit from this type of intervention. The finding that participants with past travel to a risk region had significantly higher pre-and post-MKT scores as compared to those without this travel experience supports the SRM precept that concrete experiences impact mental representations. However, the group with past travel to a risk region still answered only 49% of the pre-game knowledge questions correctly. This is consistent with data that has identified knowledge gaps in travelers irrespective of travel history.[9,19-21]

The pre/post scores were lowest in the SRM dimensions of timeline and consequences for all participants irrespective of travel history. Most surprising on the pre-test was the enduring belief among nearly half of the malaria-experienced participants that the best malaria prevention measure is a non-existent vaccine. Because malaria is a communicable disease with non-specific symptoms that can progress rapidly into a life-threatening illness, knowledge assessment and education about risk-reduction is of the utmost importance.

A malaria knowledge test for travelers was not identified in the literature. However a 34-item survey was conducted with Taiwanese physicians and nurses in 2008 on the broader topic of mosquito-transmitted disease (malaria, yellow fever, and dengue).[22] The mean percentage correct for the six items about malaria was 67%. The malaria item with the most erroneous responses (83.2%) addressed the potential for a delayed symptom onset, a timeline dimension similar to the findings of this study. The complicated nature of parasite physiology in the human body and varying risk by geographic region present challenges to health care providers who must provide pre-travel assessment and education; individualized risk reduction plans that include vaccines, medication, and chronic care/emergency management; and diagnosis and treatment across a spectrum of acuity. The authors of this Taiwanese study conclude that clinician knowledge deficits might be remedied through increased exposure to international standards for the delivery of travel health services.

The present study demonstrates how unique features of a health threat may impact the measurement of the accuracy of knowledge. Malaria has a complex timeline dimension (acute, cyclic, relapsing), and consequences that can range from complete resolution with treatment to permanent disability or death if not recognized and treated promptly. Few health conditions have similar features, and even fewer have the global prevalence and historic significance of malaria. Identifying the malaria content with the greatest likelihood for being misunderstood is important when designing educational approaches. Further, the comparison of game versions showed that automatically providing explanatory feedback in a brief-exposure educational game supports learning better than having learners seek feedback as desired or not providing this feedback. These findings provide valuable information for health educators and clinicians about the malaria content most likely to be misunderstood or under-appreciated, and how to use feedback most effectively in games and simulations to increase the accuracy of representations for malaria.

Study limitations include a recruitment pool originating from one region of the United States and a low participation rate (34% of those who accessed the 30-min online protocol completed it without incentive). Participants who were female (73%) and white (86%) were overrepresented, although this is consistent with national demographic data for study abroad students who over the past decade have been on average 65% female and 82% white.[23] This study did not test the degree of knowledge retention beyond one post-test, or the impact of increased knowledge and worry on subsequent travel-related behavior.

Conclusions

The growing population of students who travel to malaria-endemic regions would benefit from more accessible and engaging approaches to travel health education, and from a test to evaluate malaria knowledge. One such test, the MKT, required about 5 min to complete when administered online to students participating in study abroad programs on seven Midwestern U.S. campuses. The MKT demonstrated acceptable internal consistency and sensitivity to change in response to an educational intervention. Pre-game MKT scores revealed malaria knowledge deficits among study abroad students. Game exposure produced significant improvements in MKT scores across all dimensions of the SRM in both the malaria-experienced and malaria-naive groups.

Greater understanding of traveler cognitive and emotional representations about malaria will enhance the creation of more tailored interventions. The SRM representational dimensions most likely to yield inaccurate responses were timeline and consequences. This presents a hazard when students do not understand, for example, how quickly malaria can progress to death or disability, how parasite drug-resistance or counterfeit drugs can increase risk for a tragic outcome even with a rapid diagnosis, or why symptoms might develop after returning home. This open-access knowledge test may also be adapted to evaluate other malaria education interventions, and for educational outreach among health care providers globally, residents of malaria-endemic regions, and other high risk travel groups (i.e., elderly, chronic health conditions, pregnant, or returning to a malaria-endemic region to visit friends/relatives).