Evaluation of an active learning module to teach hazard and risk in Hazard Analysis and Critical Control Points (HACCP) classes.

The terms hazard and risk are significant building blocks for the organization of risk-based food safety plans. Unfortunately, these terms are not clear for some personnel working in food manufacturing facilities. In addition, there are few examples of active learning modules for teaching adult participants the principles of hazard analysis and critical control points (HACCP). In this study, we evaluated the effectiveness of an active learning module to teach hazard and risk to participants of HACCP classes provided by the University of Vermont Extension in 2015 and 2016. This interactive module is comprised of a questionnaire; group playing of a dice game that we have previously introduced in the teaching of HACCP; the discussion of the terms hazard and risk; and a self-assessment questionnaire to evaluate the teaching of hazard and risk. From 71 adult participants that completed this module, 40 participants (56%) provided the most appropriate definition of hazard, 19 participants (27%) provided the most appropriate definition of risk, 14 participants (20%) provided the most appropriate definitions of both hazard and risk, and 23 participants (32%) did not provide an appropriate definition for hazard or risk. Self-assessment data showed an improvement in the understanding of these terms (P < 0.05). Thirty participants (42%) stated that the most valuable thing they learned with this interactive module was the difference between hazard and risk, and 40 participants (65%) responded that they did not attend similar presentations in the past. The fact that less than one third of the participants answered properly to the definitions of hazard and risk at baseline is not surprising. However, these results highlight the need for the incorporation of modules to discuss these important food safety terms and include more active learning modules to teach food safety classes. This study suggests that active learning helps food personnel better understand important food safety terms that serve as building blocks for the understanding of more complex food safety topics.

Introduction

In the mid 1990s, the introduction of HACCP (Hazard Analysis and Critical Control Points) represented the first major risk-based regulation implemented in the USA to reduce foodborne illness in meat and poultry, juice and juice products, and fish and fishery products (Anonymous, 1995, 1996, 2001). HACCP has also been voluntarily incorporated in the manufacturing of other food commodities, and has been adopted at the international level as the system of choice to organize food safety plans. However, after many years of implementation the effectiveness of HACCP in controlling biological hazards is not clear (Kafetzopoulos et al., 2013; Wallace et al., 2011; Wallace et al., 2014).

Despite this fact, teaching HACCP does persist and several curricula have been organized to teach HACCP classes. Yet, the most widely accepted curriculum is the one organized by the International HACCP Alliance, which is based on a 2-day (16 h) training. This curriculum has become the standard curriculum to teach introductory HACCP classes to food industry personnel working with meat and poultry, or other food commodities (Gravani et al., 2006; Jackson et al., 1996).

Most of the HACCP training is done through workshops based on providing handout materials and extensive lectures (Wallace and Powell, 2005). Previous studies that examined efficacy of HACCP concluded that there are limitations in the knowledge of HACCP team members in the long term that impact the effectiveness of HACCP plans (Taylor and Taylor, 2004), and that the HACCP team knowledge is not better than the knowledge of individual team members (Wallace et al., 2011). In addition, the difficulties in performing an effective hazard analysis appear to remain underestimated (Wallace et al., 2014). A related area of study is the evaluation of barriers for implementation of HACCP. For instance, we know that financial resources (Herath and Henson, 2010), cultural differences (Azanza and Zamora-Luna, 2005; Bas et al., 2007; Panisello and Quantick, 2001; Vela and Fernandez, 2003), and lack or insufficient application of pre-requisite programs, such as good manufacture practices, affect the quality of HACCP plans. In addition, some food manufacturing personnel may not even get appropriate training to understand how to apply HACCP principles, especially those working for small companies (Karipidis et al., 2009; Yapp and Fairman, 2006).

There are very few studies addressing the efficacy of the teaching of HACCP to ensure food personnel understand and implement HACCP principles. For instance, the incorporation of an active assessment, based on primary trait analysis and pedagogical reasoning, has been proposed to improve the teaching effectiveness for HACCP classes (IHA, 2016). It is important to remember that participants of HACCP classes are comprised primarily of adult learners (Lo et al., 2004). Another example is the evaluation of the Grade “A” Dairy HACCP core curriculum (Murphy, 2013). However, to our knowledge, there are no publications examining efficacy of interactive modules to teach key HACCP principles. The lack of clear understanding of the definitions of hazard and risk is a consistent feature among participants of HACCP classes (Oyarzabal, 2015). There is a substantial body of research in health sciences highlighting the positive impact of interactive, engaging modules when teaching basic health concepts to health science students. For instance, card games and board games have been specifically developed for medical education, and they are very useful in teaching basic concepts (Bochennek et al., 2007). However, we are not aware of any use of games as active learning modules to teach the concepts of hazard and risk in food safety classes. For this reason, we incorporated an active leaning module that included self-assessment of knowledge and the use of a dice game to demonstrate differences in the meaning of the words hazard and risk in the context of food safety (Oyarzabal, 2015).

The terms hazard and risk are significant building blocks for the organization of risk-based food safety plans. Unfortunately, these terms are not clear for some personnel working in food manufacturing facilities. The National Advisory Committee for Microbiological Criteria for Foods has defined hazard as “a biological, chemical, or physical agent that is reasonably likely to cause illness or injury in the absence of its control” (NACMCF, 1998). Therefore, we have had a definition of hazard in food safety regulations for more than 20 years, but there is no definition of risk. To better explain these terms, we developed an interactive training module (Oyarzabal, 2015) that has been employed during the teaching of HACCP classes. In this study, we evaluated an interactive module based on a dice game to teach hazard and risk during the teaching of HACCP classes provided by the University of Vermont (UVM) Extension to adult personnel working in the food manufacturing industry in 2015 and 2016. The objectives included the assessment of an interactive module on change in comprehension of hazard and risk, two key terms in HACCP.

Materials and methods

A total of 72 participants of the HACCP classes taught by the UVM Extension in 2015 and 2016 were invited to complete questionnaires to asses an active learning module with a dice game to illustrate concepts of risk and hazard. These HACCP classes have been accredited by the International HACCP Alliance and the curriculum involves a 2-day (16 h) training of adult personnel working in food manufacturing facilities in the USA. Participants worked in different food industries, including dairy products; eggs; meat and poultry; produce and fruits; processed foods; seafood and others (Table 1). Providing the responses to the questionnaires, playing the game and discussing the terms hazard and risk took approximately 25–30 min. All participants received a verbal explanation emphasizing that the filling out the pen-and-pencil questionnaires was voluntary, and that the information collected was confidential.

Table 1

Self-assessment of the understanding of the definition of hazard and risk among participants of HACCP classes.

Definitions from Participants		Number (%)	Average Self-Score (1 through 6)
Correct for:	Hazard	40 (56)	3.5
	Risk	19 (27)	3.5
	Both	14 (20)	3.5
Incorrect for:	Hazard	31 (44)	3.3
	Risk	52 (73)	3.4
	Both	23 (32)	3.3

Participants of the HACCP classes were divided in groups of two to no more than six individuals. The overall goal of the interactive module was to discuss the terms hazard and risk with a dice game. Participants were first asked to complete Questionnaire #1 to perform a baseline assessment of their knowledge of the terms hazard and risk. Questionnaire #1 included questions about gender, age, highest education level attained, years of work with foods, food commodities participants work with, the writing of the participants’ definitions of hazard and risk, and a self-assessment of the understanding of the differences between hazard and risk (1 = not really, though 6 = very well).

Participants were then taught a dice game and allowed to play the game for 10–15 min. Although the dice game has been described elsewhere (Oyarzabal, 2015), it is important to provide a brief description of the game. The objective of the game is to accumulate the most points possible by rolling two dice, and the winner is the person with the most points at the end of the game. Players self-select when to continue or stop rolling the dice in any given turn. Points continue to accumulate with each successive turn. However, players lose all points accumulated in a turn when rolling a seven (7) and forfeit the turn. They also lose all points accumulated on all previous turns when rolling two consecutive doubles. Additionally, players are obliged to roll the dice again after rolling doubles (Oyarzabal, 2015). The data collected by individuals on the results from different rolling of the dice is used to discuss the concepts of hazard and risk.

During the analysis of the responses in Questionnaire #1, we accepted as correct any definition of hazard that include the term “harm” and/or “dangerous” but we did not include as correct the definitions that include the terms “contaminant” or “adulteration” because these terms have been clearly and consistently defined in food safety regulations, and in many different countries, to include, in addition of agents that produce illnesses, agents that do not cause illness, harm or injury, such as filth (FSMA, 2015). If the terms “contaminant” or “adulteration” were associated with the words “harm” and/or “dangerous,” we considered these definitions as correct. In addition, we accepted as correct the definitions of risk that included the terms “likelihood,” “chance,” “possibility,” but we did not accept the term “potential” because this term is used to characterize hazards (potential to cause illness or diseases) and not to characterize the occurrence of the hazards.

After playing the game, the class was encouraged to discuss the terms hazard and risk using the dice game as an example, and a review of the application of these terms in food safety was presented. The most appropriate definition of hazard (cause harm) was reviewed with the class, along with the definition of hazard according to the National Advisory Committee for Microbiological Criteria for Foods (NACMCF, 1998)─a biological, chemical, or physical agent that is reasonably likely to cause illness or injury in the absence of its control. The most appropriate definition of risk (probability, possibility, likelihood) was also briefly discussed with the class with the clarification that there is no a precise definition of risk in food safety regulations in the US.

At the end of the module, participants were asked to complete Questionnaire #2. This questionnaire included a retrospective pretest and self-report on new understanding of the terms hazard and risk. This questionnaire included a series of statement columns to assess understanding “Now” versus “Before” with the possible answers for Now and Before as: A little (1); Somewhat (2); A good bit (3); Got it (4) (values given to each answer):

Understand the definition of hazard and risk

Recognize the differences between these terms

Recognize the importance of these terms in food safety

There were also two open-ended questions, and two programmatic rating questions:

What was the most valuable thing(s) you learned by playing the dice game?

Any additional comments?

How would you rate the organization of the hazard/risk module? (1 = poorly organized, though 6 = very well organized)

If you have attended similar presentations, how does this presentation compare with others you have attended? (1 = poorly organized, though 6 = very well organized; NA = not applicable)

The written answers provided by the participants to the definitions of hazard (a harmful, dangerous event) and risk (probability or chance of a hazard occurring) were analyzed to calculate the number of participants that provided an acceptable answer to hazard, risk or both. The data from answers to questions presented in Questionnaire #2 were analyzed for normality with the Shapiro-Wilk Normality Test, which tests for normality within a dataset, with the assumption that the dataset is normally distributed (null hypothesis). The mean scores and standard deviation of the retrospective test results “before” and “now” were calculated (Pratt et al., 2000) and then compared using the Fisher’s exact test, a test used to analyze categorical data with small sample sizes. Statistical analyses were performed with the open-source statistical package R (R Core Team, 2016).

Results

We collected 71 responses for questionnaire #1 and #2. Among respondents, 65% were 40 years old or younger, 49% had a bachelor’s degree and 62% had 10 years or less of experience working with foods (Table 2). At baseline, 40 participants (56%) provided a correct definition of hazard, 19 participants (27%) provided a correct definition of risk, 14 participants (20%) provided a correct definition of both hazard and risk, and 23 participants (32%) did not provide a correct definition for hazard or risk. There were no statistical differences in the number of females or males that provided the correct response for both hazard and risk (analysis based on percentages: P > 0.05). There were no statistical differences among the categories within years of work with food (p > 0.05) (Table 2).

Table 2

Baseline demographics of the study population and of the participants who provided the correct definitions for both hazards and risk.

		All Participants (n = 71)	Participants (n = 14) Who Provided the Correct Definitions of Hazard and Risk
Parameter	Variable	No. Participants (%)	No. Participants (%)1
Sex	Female	29 (41)	7 (24)
	Male	42 (59)	7 (17)
Age	18–30	26 (38)	7 (27)
	31–40	19 (28)	2 (11)
	41–50	13 (19)	3 (23)
	51–60	8 (12)	1 (13)
	61 and over	3 (4)	1 (33)
Highest education level attained	HS2 diploma	23 (33)	3 (13)
	Associate degree	5 (7)	0
	BS/BA3 degree	34 (49)	10 (29)
	Post grad work	2 (3)	0
	Master’s degree	5 (7)	1 (20)
	Beyond MS/MA4	1 (1)	0
Years of work with foods	Less than 1 year	7 (10)	3 (43)
	1–5 years	22 (31)	4 (18)
	6–10 years	15 (21)	1 (7)
	11–20 years	18 (25)	4 (22)
	21–35 years	9 (13)	2 (22)
	36 years and over	0	0
Food commodities I work with	Produce and fruits	30	5
	Eggs	16	2
	Meat and poultry	25	2
	Processed foods	33	7
	Dairy products	42	6
	Seafood	9	2
	Other	18	4

The numbers under column All Participants represent 100% for each row.

HS = high school.

BS/BA = Bachelor of Science/Bachelor of Art.

MS/MA = Master of Science/Master of Art.

The average self-assessment score for the knowledge of these terms was 3.4 out of possible 6. There were no differences in the average self-assessment scores among those who provided correct answers compared to those who provided incorrect answers to the terms hazard and/or risk (Table 1). Only 14 participants (20%) had a self-assessment of knowing these terms very well or well. However, only 4 individuals (6%) from those that provided the correct definition to hazard and risk had a self-assessment of knowing these terms well (Table 3). From those individuals who had a self-assessment of knowing the terms very well, none provided a proper definition of the term risk.

Table 3

Responses to the question “I do understand the differences between hazard and risk” presented in Questionnaire #1. Possible vales ranged from 1 to 6.

	Number of Participates Who
Response Level	Did Not Respond Correctly to Any Question (n = 57)	Responded Correctly to both Questions (n = 14)
6 (very well)	2	0
5	2	4
4	8	2
3	4	5
2	4	1
1 (not really)	3	1

Results from the Shapiro-Wilk Normality test revealed that data from questions presented in Questionnaire #2 were not normally distributed (p < 0.05). A statistical difference (p <0.05) was observed in the self-assessment (Now versus Before) when respondents were asked to compare their previous understanding of the terms hazard and risk; the application of these terms in food safety; and their new understanding after completing the training module (Table 4).

Table 4

Statistical analysis between Now and Before for all questions (refer to Table 1). Statistical differences set at p ≤ 0.001. Possible values ranged from 1 to 4. A larger value on a row signifies a better understanding of the statement asked.

	Means Scores (St Dev)		p Values1
Statement	Before	Now
Understand the definition of hazard	2.62 (0.86)	3.82 (0.49)	p < 0.001
Understand the definition of risk	2.74 (0.75)	3.76 (0.57)	p < 0.001
Recognize the differences between these terms	2.52 (0.98)	3.79 (0.56)	p < 0.001
Recognize the importance of these terms in food safety	2.62 (0.86)	3.72 (0.61)	p < 0.001

p values were generated using the Wilcoxon Signed Rank test.

Thirty participants (42%) stated that the most valuable thing they learned with this interactive module was the difference between hazard and risk. Forty-six participants (65%) responded NA (not- applicable) to the question if they have attended similar presentations, meaning that they have not. Sixteen participants (23%) responded that this presentation compared high (5) or very high (6) with similar, previous presentations they attended, and only seven participants (7%) chose 4 (5), 3 (1) or 2 (1) as the response to this question.

Discussion

When constructing the questions for Questionnaire #2, we used a pretest/posttest approach, which allows participants to serve as their own baseline for comparison. The pretest-posttest approach has several limitations when using self-report measures. The most important limitation appears to be the overestimation of knowledge by participants at the beginning, an overestimation that could make participants realize that they knew much less than what they originally may have reported on a pretest. This phenomenon is known as response-shift bias. To avoid response-shift bias, we decided to collect both contemporary (Now) and retrospective (Before) information after the conclusion of the module as part of our program assessment.

The terms hazard and risk are significant building blocks for the organization of risk-based food safety plans. Unfortunately, these terms are not clear for some personnel working in food manufacturing facilities. The definitions of hazard and risk are complex and food regulatory agencies in the USA have included the definition of “hazard” in their regulations for more than 20 years. An important working definition of hazard was provided by the National Advisory Committee for Microbiological Criteria for Foods (NACMCF, 1998): A biological, chemical, or physical agent that is reasonably likely to cause illness or injury in the absence of its control. This is the definition that we emphasize in HACCP training. We chose the definition from NACMCF because we evaluated HACCP training only. NACMCF’s definition of hazard is used in HACCP training for consistency. In addition, some of the training that we evaluated was done in early 2015, when FSMA training was not available yet. However, the Food Safety Modernization Act of 2011 (FSMA, 2015) provides an updated definition of the word hazard:

Any biological, chemical (including radiological), or physical agent that has the potential to cause illness or injury.

In this new definition, the phrase “that is reasonably likely to cause illness or injury” is replaced by the phrase “that has the potential to cause illness or injury” to distinguish between “hazard” and “known or reasonably foreseeable hazard.” Thus, the term “significant hazard,” which has been used for almost two decades when teaching HACCP classes, has been replaced by “hazard requiring preventive controls,” which is a new term established by FSMA. Another phrase that has been deleted from the NACMCF definition is the phrase “in the absence of its control.” The removal of this phrase makes for a better alignment with the hazard definitions by Codex HACCP Annex, and the current seafood and meat and poultry HACCP. Therefore, the new definition of hazard is simpler but is based on a tiered approached when performing a hazard analysis. From the myriad of hazards that can be found in a food product, the focus moves towards the identified “known or reasonably foreseeable” hazards that require preventive controls for that food product in a food manufacturing facility. It is known that the hazard analysis step is difficult to perform and there are still significant gaps in the knowledge of hazard identification (Wallace et al., 2014) in complex processes, such as food manufacturing (Liu et al., 2015). These knowledge gaps, or lack of understanding of the meaning of some fundamental terms, can result in the underestimation or overestimation of the most relevant hazards in a food manufacturing facility.

The definition of hazard in HACCP principles includes the words “reasonably likely to cause illness,” which tend to create confusions between the terms hazard and risk. It is important to highlight that the hazard analysis step, when building a food safety plan, includes the evaluation of the hazards based on “likelihood of occurrence.” The definition of risk per se has been studied many times and these discussions in Europe have culminated with the Food and Agriculture Organization of the World Health Organization defining risk as “a function of the probability of an adverse health effect and the severity of that effect, consequential to a hazard(s) in food” (FAO/WHO, 2013), and the standards series that include ISO 31,000, 27,000 and 28,001 (Security Risk). The different interpretations on the definition of “risk” highlight the variability in the use of this term by different risk management communities (Ale et al., 2015). Nevertheless, it appears that any teaching of the word “risk” in food safety may need to be further described within the current knowledge on the psychology of risk perception, which is very complex. Some studies suggest that we tend to have “optimistic bias” (people’s beliefs that they are less likely than other people to experience negative events) and “illusions” of control when it comes to food safety hazards (Frewer et al., 1994; Sargeant et al., 2010).

The fact that less than one third of the participants answered correctly to the definitions of hazard and risk at baseline is not surprising. It is common for food processors to have difficulties understanding critical concepts related to food safety. In a survey of personnel from food businesses, 85% were not able to correctly identify good hygiene practices as a prerequisite program to HACCP (Trafialek and Kolanowski, 2014), while another study revealed that misunderstandings of what product description means result in extensive technical description of the overall process without focusing on key food safety issues (Dzwolak, 2014). These previous studies and our date emphasize the need to further train food personnel, including food handlers and food manufacturers, in the importance of understanding the term hazards and how to assess the likelihood of these hazards contaminating or being present in foods. It is important to highlight that after 20 years of training in HACCP, there have been very few attempts to incorporate and evaluate interactive learning modules to teach adults important terms that serve as building blocks for the understanding of more complex food safety topics.

By providing a retrospective pretest, we captured the description of the change as experienced subjectively by program participants (Hill, 2005; Howard et al., 1981). The evaluation showed that participants of HACCP classes where a dice game was used as an active learning module did improve their understanding of the terms hazard and risk, which are critical terms in building food safety programs. Table 4 shows the results from self-reflexing questions, where participants had to score the level of knowledge they had before the module and after the module. The comparisons show statistical differences for all questions, meaning that participants did feel that they had a better understanding of these terms after the interactive module. Thus, participants acknowledged through self-assessment that the dice game helped them gain knowledge on the understanding of the terms hazard and risk. As food safety regulations moves toward the incorporation of risk-based approaches (Anonymous, 2015) and more information is generated to calculate risk, individuals trained in these principles will be able to better incorporate new data when performing hazard analysis in their own processing facilities. Then, the concept of prevention will be better understood when developing risk-based food safety programs.

The results from this study suggest that the incorporation of an interactive game for active learning in HACCP classes, or other type of food safety training, may help food manufacturing personnel understand basic concepts to then build more complex food safety plans to cope with food safety challenges that may arise in the future. These more elaborated food safety plans, which are risk-based approaches to enhance food safety with emphasis on prevention and active monitoring, are now required by new regulations in the USA (Anonymous, 2015). Thus, individuals that understand the definitions of hazard and risk will be more prepared to focus on the most important hazards in their facilities. Further research is needed to see how a better understanding of these important food safety terms impacts food safety practices and results in safer foods.

Although several differences (e.g., responses by females versus males) were not significant, further studies with large sample sizes may be warranted to confirm whether these differences are indeed not significant or are one of the limitations of our study. The pre-post design is an efficient study design for pilot testing, however outcomes of the intervention should be tested against a control group. Future studies should evaluate individuals who went through interactive training modules with other individuals who are taught HACCP classes without interactive module to determine the effects of interactive training to teach hazard and risk to individuals working in food facilities.

Declarations

Author contribution statement

Omar A. Oyarzabal: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Wrote the paper.

Ellen Rowe: Conceived and designed the experiments; Analyzed and interpreted the data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.