Discrepancies Between Implementation and Perceived Effectiveness of Leading Safety Indicators in the US Dairy Product Manufacturing Industry.

Background: In the United States, the dairy product manufacturing industry has consistently had higher rates of work-related nonfatal injuries and illnesses compared to the national average for industries in all sectors. The selection and implementation of appropriate safety performance indicators are important aspect of reducing risk within safety management systems. This study examined the leading safety indicators implemented in the dairy product-manufacturing sector (NAICS 3115) and their perceived effectiveness in reducing work-related injuries.
Methods: Perceptions were collected from individuals with safety responsibilities in the dairy product manufacturing facilities. OSHA Incident Rate (OIR) and Days away, restricted and transferred (DART) rates from 2013 to 2018 were analyzed.
Results: The perceived most effective leading were safety observations, stop work authority, near miss reporting, safety audits, preventative maintenance, safety inspections, safety training attendance, and job hazard analysis/safety analysis, respectively. The 6-year trend analysis showed that those implementing all eight top indicators had a slightly lower rates than those that did not implement all eight. Production focused mentality, poor training, and lack of management commitment were perceived as the leading causes of injuries in this industry.
Conclusion: Collecting leading indicators with the unique interest to meet the regulatory requirements and to document the management system without the actual goal of using them as input to improve the system most probably will not lead to an effective reduction of negative safety outcomes. For leading indicators to be effective, they should be properly selected, executed, periodically evaluated and actions are taken when necessary.

Introduction

The dairy product manufacturing industry traditionally focuses on challenges related to assuring food safety. However, the singularities of the production process and work environment may create conditions that negatively impact their workforce’s safety and health [1]. In the United States, the dairy industry has consistently had higher rates of work-related nonfatal injuries and illnesses compared to the national average for industries in all sectors. In 2019, dairy workers (NAICS 3115-Dairy Product Manufacturing) experienced a rate of 4.1 nonfatal incidents per 100 full-time workers, compared to 2.6 for all private industries, and 3.6 for food manufacturing, while the rate of cases resulting in Days Away, Restricted, or Transferred (DART) was almost double the national average at 3.0 compared to 1.6 [2].

In the efforts to prevent work-related injuries and illness, implementing an occupational safety and health management system (OSHMS) is an important approach. However, safety improvements are more likely to be successful when organizations have a timely review of the effectiveness of the OSHMS. Systematic measurement of safety performance supports a proactive safety culture and the evaluation of the impact of safety initiatives [3]. The identification of appropriate safety performance indicators is an important part of the OSHMS planning process [4] that allows measuring its overall impact within an organization. Although different types of indicators can evaluate performance, the occupational safety and health field has traditionally relied on lagging indicators that measure success based on things that have already gone wrong [4]. These lagging indicators (i.e., injury rate, workers’ compensation claims, and injuries resulting in days away from work) provide consistency and comparability among organizations regardless of their size [5]. However, they do little to predict performance and represent only one dimension to describe how safe an organization truly is [[6], [7], [8], [9]], especially since zero injuries do not necessarily embody a safer place [10,11].

In contrast, the concept of leading indicators has grown in popularity since it has been reasonably proposed that focusing on input or process can be more effective in anticipating changes in safety conditions, evaluating the functionality of the system, and identifying early signs of poor safety performance. Leading indicators are intended to monitor activities that are implemented to prevent negative safety outcomes [4,12]. Therefore, their importance lies in identifying potential causes and anticipating safety outcomes [13]. Safety audits, behavioral observations, job hazard/safety analysis (JHA/JSA), trainings, and safety climate measurements are examples of common leading safety indicators reported in the literature [9,14]. While the list of safety indicators is long, the literature is scarce on how to select reliable ones and their limitations. The selection of leading indicators should respond to the organization’s needs and be used as a tool for decision-makers in steering OSHMS based on reliable safety performance information [13,15]. Guo and Yiu (2016) define leading safety indicators as a set of quantitative and/or qualitative measurements that can describe and monitor validly and reliably safety conditions. The point where the safety conditions are measured across the injury pathway defines the conceptual difference between leading and lagging indicators since it reflects whether the metric is viewed as an antecedent or outcome of safety [3]. Proper selection of leading and lagging indicators is critical to evaluate the impact of OSHMS since they should provide relevant information to address actions and contributions to the continuous improvement of the system. While research is still needed to identify the link between leading indicators and their effect on injury rates, studies have shown there is an increasing interest in the safety profession to measure them because of the distinct possibility that a link does exist [16]. This study aimed to examine the leading safety indicators used in the dairy product-manufacturing sector and evaluate their perceived effectiveness in reducing work-related injuries from the perspective of employees with safety responsibilities.

Materials and methods

Participants and sample selection

The target population consisted of individuals with responsibilities for occupational safety and health currently employed in dairy product-manufacturing facilities with 11 or more employees throughout the United States. Contact information from potential participants was obtained from the International Dairy Foods Association (IDFA), the Dairy Farmers of America (DFA), and the Northeast Dairy Association (NDA). This research study was reviewed and approved by the local Institutional Review Board (IRB log No. 18-178). All information gathered was anonymous, and thus no answers could be traced back to any individual or organization. No compensation was offered for participation.

Survey instrument

The survey instrument was constructed with the attention to question sequencing to ensure that individuals participating in the survey would utilize their personal knowledge of what constitutes a leading indicator. The questionnaire comprised three separate sections: (1) participants’ demographic information (5 questions - job title, education, work experience, and safety certifications held); (2) company information (4 questions - size, occupational incident rate (OIR), and days away, restricted, and transferred (DART) rate for a 6-year period); and (3) perception and implementation of leading indicators (2 questions); and two open-ended questions about the causes of high injury rate in this industry. No personal identifiers were collected. Twelve of the most implemented leading indicators identified in the literature [3,12,[17], [18], [19]] were included in the survey (safety audits, preventative maintenance, safety training attendance, safety observations, safety inspections, near-miss reporting, stop work authority, JHA/JSA, safety meeting attendance, corrective action completion rate, worker perception survey, and attendance tracking). This information was also used to assess any perception gaps between indicators implemented and those perceived as effective regardless of being implemented in the workplace. The survey also gave the respondent the ability to write in any other indicators that may not have been provided. While completing the survey, participants had the option to read the indicator definition by clicking on it.

The survey also included questions about the indicator’s length of implementation. Last, the respondents were asked to provide the top three perceived reasons associated with high incident rates in this industry and what they believed could be done to mitigate them. The survey was designed to prevent going back to previous sections to avoid making any changes to initial responses based on acquired knowledge as participants progressed through the survey.

Survey instrument pilot-testing and administration

The survey instrument was pilot tested for understandability of the cover letter and survey instructions, readability, length, and easiness of use in five different facilities. The survey was distributed to both Safety and Human Resources (HR) Professionals. Feedback from the group resulted in minor modifications such as question formatting and wording.

The survey was created using QualtricsXM software, and an anonymous link was generated and distributed directly via email to 312 dairy product manufacturing facilities with 11 or more employees. The survey link was accompanied by an introductory cover letter, which provided the purpose, description, expected outcomes, and anonymity of the study.

Data analysis

Each survey was reviewed to ensure all questions were completed according to the instructions given. Surveys with repetitive response patterns and/or inconsistent responses were removed from the dataset. All data from the surveys were analyzed using IBM’s Statistical Package for Social Science (SPSS) version 28. Prior to data analysis, each data point entered was reviewed three times to ensure the information was recorded accurately to ensure proper coding.

A descriptive analysis was performed to summarize the demographic data and identify trends in the participants’ responses. Participants ranked the 12 leading indicators from “1” to “12” based on perceived effectiveness with “1” being the most effective and “12” the least effective. Kendall’s Coefficient of Concordance (W) was used to measure agreement among respondents [20] on (1) perceived effectiveness of the 12 leading indicators regardless of implementation in the facility and (2) perceived effectiveness of those who implemented at least the top eight leading indicators (Fig. 1). Values of Kendall’s Coefficient of Concordance (W) close to zero indicate low agreement while those close to 1 indicate strong agreement” [21]. The Chi-Square (χ2) test was used to test the significance of the coefficient of concordance at an alpha level .05.

Fig. 1

Data collection and analysis flowchart.

For selecting the top leading indicators that combined both perceived effectiveness and implementation, a matrix was created to identify those with the highest cross-product value and cluster the sample into implemented and not implemented (Fig. 1). By using the matrix results, a 6-year trend analysis and Mann–Whitney statistical test were conducted to compare the performance of OIR and DART rates between those implemented and those who did not implement. Data from the most recent years were not included due to the potential impact of the COVID-19 pandemic.

Results

A total of 84 surveys were completed, which equated to a 27% response rate of the 312 surveyed facilities. Two surveys were excluded due to incomplete data (n = 82). Most of the participants (78%) came from facilities with 100 or more employees, while 16% of all responses were from facilities that did not hire a safety professional (Table 1).

Table 1

Demographics of participants and characteristics of the facilities variables

	Participants (n = 82)	Percentage
Length of safety responsibilities
>1 year	5	6.1%
1–2 years	11	13.4%
3–5 years	24	29.3%
6–10 years	26	31.7%
>11 years	14	17.1%
No information	2	2.4%
Safety and Health Certifications held
None	57	69.5%
Certified Safety Professional (CSP)	13	15.9%
Graduate Safety Practitioner (GSP)	4	4.9%
Associate Safety Professional (ASP)	3	3.7%
Others	3	3.7%
Occupational Hygiene and Safety Technician (OHST)	1	1.2%
Safety Management Specialist (SMS)	1	1.2%
Certified Safety Director (CSD)	1	1.2%
Multiple Certifications	1	1.2%
Facilities size by number of employees
Small size facilities (11–19 employees)	5	6.1%
Medium size (20–99 employees)	13	15.9%
Large size (>100 employees)	64	78.0%
Facilities with dedicated Safety Professional	69	84.1%

Leading indicators implemented and perceived effectiveness

From the list of 12 leading indicators, participants identified those being implemented at their facility. Safety audits, preventative maintenance, and safety training attendance were all implemented in 90% of the facilities surveyed. Respondents then ranked the 12 leading indicators based on their perceived effectiveness. Survey results showed eight of the leading indicators were perceived as the most effective by more than half of the facilities, with safety audits, preventative maintenance, and safety training attendance being perceived as the most effective by 90% of respondents (Fig. 2). Participants were found to have a weak agreement (W = 0.279; p < 0.0001) [21] on the top-ranked leading indicators perceived as the most effective in reducing the incident rate in the dairy product manufacturing sector regardless of their implementation. The perceived top six leading indicators agreed on were safety observations (3.9), stop work authority (4.7), safety audit (4.8), near-miss reporting (4.9), preventative maintenance (5.5), and safety inspections (5.6).

Fig. 2

Comparison of the ranking of leading safety indicators based on their perceived effectiveness (regardless of implementation status) and percentage of implementation in the dairy manufacturing (n = 82). 1 Lower ranking indicate more effectiveness- Kendall’s W = 0.279, χ2 = 251.317, p < 0.000, df = 11).

A total of nine facilities implemented the 12 leading indicators, while about half of the facilities implemented at least eight of the leading indicators. Thus, the sample size for the subsequent analysis of agreement on the implemented leading indicators based on their perceived effectiveness was defined using the median value (n = 40). Accordingly, there was a weak agreement among participants (W = 0.232; p < 0.000). The perceived most effective categories of the implemented leading indicators were safety observations (3.7), stop work authority (4.0), near-miss reporting (4.3), safety audits (4.9), and preventative maintenance (5.5), while job hazard analysis/job safety analysis (JHA/JSA) was perceived as the least effective implemented indicator (Table 2). On average, the length of implementation of each indicator was 4.6 years (SD = 0.77) and 1.5 years (SD = .71) for companies implementing all eight indicators and those who did not implement all eight indicators, respectively.

Table 2

Ranking of perceived effectiveness of the top implemented eight leading indicators (n = 40)

Leading indicator	Mean rank∗
Safety observations	3.7
Stop work authority	4
Near Miss reporting	4.3
Safety audits	5.9
Preventative maintenance	5.9
Safety inspections	6.1
Safety Training attendance	7.4
Job hazard analysis/safety analysis	8.3

Kendall’s W = 0.232, χ2 = 64.950, p < 0.000, df = 7.

Lower rankings indicate more effective perceived indicator.

Trends of incident rates over a 6-year period

A total of 79 companies self-reported OIR and DART rates. Those were compared year by year for companies implementing all eight top leading indicators and those that did not implement all eight. The OIRs were slightly higher in those companies that did not implement all eight indicators in comparison to those that did implement all eight indicators (p > 0.05) except for 2016 and 2018 data (p < 0.05) (Fig. 3a). The OIRs remained almost steady during the study period for companies that did not implement the eight indicators, while a slight decrease over time was observed for those who implemented at least all eight. In turn, DART rates did not show a consistent trend or statistical differences between these two groups (p > 0.05) (Fig. 3b). Overall, the OIRs and DARTs were higher than the average private industry rates over the same period except for the DART rates in the last two year’s data.

Fig. 3

Trends of occupational injury rates (OIR) and days away, restricted and transfer rate (DART), over a 6-year period for companies who implemented the eight top indicators (n = 40) and those who did not implement (n = 39).

Perceived top causes for incidents and recommendations

A total of 76 participants responded to the perceived reasons for the high incident rates reported in the dairy product manufacturing sector. While there were many causes provided, the four major reasons that were given by 20% or more of the survey respondents were production mentality (30%), lack of safety support by upper management (30%), lack of training/knowledge (24%), and unsafe acts by employees (21%) (Table 3). As to their effectiveness in reducing incident rates, the top four included more effective training/education (44%), increased management support/buy-in for safety (33%), increased employee involvement in safety (27%), and a developed ergonomics assessment program (19%) (Table 4).

Table 3

Perceived major reasons for high incident rates in the dairy product manufacturing industry (n = 76)

Description	Number of respondents	Percentage (%)
Production-first Mentality	23	30
Lack of Safety Support by Upper Management	23	30
Lack of Training/Knowledge	18	24
Unsafe Acts by Employees	16	21

Table 4

Recommendations given by respondents to effectively reduce incident rates

Measure	Number of respondents	Percentage (%)
More Effective Training/Education	33	44
Increase Management Support/Buy-in for Safety	25	33
Increase Employee Involvement in Safety	20	27
Develop an Ergonomics Assessment Program	14	19

Discussion

The dairy product manufacturing sector consistently reports higher injury rates when compared to the private industry national average. This study aimed to identify implemented leading indicators, assess safety responsible personnel’s perceptions of the effectiveness of leading indicators on reducing safety outcomes (OIR and DART), and compare safety outcomes based on the leading indicator’s implementation status. The results from this study indicated that at least one leading safety indicator from the initially selected 12 was being implemented in the dairy facilities participating in this study. Regardless of the implementation, safety observations, stop work authority, safety audits, near-miss reporting, preventative maintenance, and safety inspections, respectively, were perceived as the most effective leading indicators in reducing the OIR and DART rates in this industry. According to the published literature, these indicators are among the most implemented metrics that companies track at the corporate level [16,[22], [23], [24]]. Safety audits, safety observations, and preventive maintenance have been previously suggested in other industries as key metrics for effective management systems [13,16,25].

Safety training attendance was reported as one of most frequently implemented leading indicators; however, it was not in the top six of those perceived as being the most effective in reducing incident rates. Having safety training attendance in place as an indicator may be explained by its requisite for achieving regulatory compliance, which also highlights the critical role statutory requirements play in selecting indicators to evaluate the safety performance [24]. The disagreement between implementation and perceived effectiveness might also reflect a flaw in the conceptualization of the indicator. Leading indicators are intended to monitor changes in elements of a safety management system that could have the potential to modify risk levels before people get hurt; therefore, to be effective, they should measure specific practices that anticipate variations in risk levels [3,26]. In fact, participants identified lack of training/knowledge as one of the causes of negative safety outcomes in this industry, and it is also one of the most preferred safety interventions [27]. Using safety training attendance as a leading indicator fails to identify changes in the training outcomes, such as knowledge, skills, and attitudes, which have a demonstrated association with changes in risk levels and improvement in safety culture [[28], [29], [30], [31]].

Employees’ authority to stop an operation or an action, which they considered to be not controlled or would potentially expose them beyond an acceptable risk, was also evaluated as a leading indicator in this study. Stop Work Authority (SWA) was perceived as the second most effective leading indicator, although it was placed at the bottom in implementation. To possess an effective role as a leading indicator, SWA requires a strong safety culture where employees feel empowered to halt a dangerous activity if they have a concern regarding their safety without being intimidated by the effect of that action on aspects such as production [32]. However, production-based incentives can minimize the support of SWA policies from middle and upper management. Paradoxically, the low level of implementation of SWA may be an indicator of the safety culture in the sector, and it is also aligned with the perceived causes of high injury rates in the industry, where the production-first mentality was placed as number one by participants. Successful implementation of SWA is dependent on situational and organizational factors [33], meaning employees may be unsure or reluctant about exercising this option when competing priorities such as productivity and safety are in play. The implementation of SWA policy as a leading indicator requires a clear allocation of responsibility [34] and accountability across all levels of the organization, as well as visible management commitment to genuinely support employee involvement in safety.

In general, discrepancies between implemented and perceived effectiveness of the leading indicators identified in this study can be explained by the driven nature of these indicators since the ones being implemented are considered basic safety measures recommended by regulatory agencies (e.g., OSHA) and insurance companies, commonly used in the industry, or easily measurable. Additionally, a misunderstanding in the actual conceptualization of leading indicators may lead safety personnel to record activities (e.g., safety training attendance or attendance tracking) rather than tracking practices that ensure workers’ competencies, hazard recognition and control, and/or lowering risk levels. Collecting indicators with the unique interest to document the activities within a management system but without the actual goal of using them as input to implement preventive actions is not going to lead to an effective reduction of negative safety outcomes [24,35].

Participating dairy facilities that implemented at least all top eight leading indicators selected in this study were found to have a slightly better safety performance (OIR and DART rates) than those that did not implement all top eight leading indicators (p > 0.05). Due to the nature of this study design, it is not possible to state that minor differences identified are associated with the leading indicators in place. It is likely that companies implementing more leading indicators might have a better safety culture, and therefore, more effective practices to control hazards and minimize the risk level, which is directly tied to favorable safety outcomes. A critical aspect of the potential effectiveness of leading indicators is associated with the ability of the indicator to capture gaps in the safety management system. If the selection and implementation of the leading indicator are not aligned with proven causality [26,36], then it will be difficult to quantify cause-and-effect relationships directly related to their implementation. For instance, worker perception surveys should be oriented to identify trends in organizational practices and policies that prioritize production over safety, which can progressively act detrimental to the safety culture [37].

This study must be interpreted considering its limitations. First, the sample size reduces the opportunity to generalize the findings. Second, self-reporting OIR and DART rates from the past 6 years could have affected the participation rate. Future research should consider capturing the perceived effectiveness of leading indicators by both management and line employees in this industry and other industry sectors to determine if they have the desired effect. In addition to perceived effectiveness, longitudinal studies should be designed to evaluate the impact of safety indicators on safety performance over time and their intersection with the production over safety mentality. Conducting qualitative studies to examine in depth the reasoning behind the lack of management support, production over safety mentality, and lack of knowledge would contribute to a better understanding of the relationship between leading safety indicators and safety outcomes.

Conclusion

Implementing leading safety indicators may have a positive impact on injury rates within an organization, but they will only be effective if they address the root causes of the poor safety performance and go far beyond regulatory compliance. There does not seem to be a simplistic formula for determining the leading indicators that will be the most effective at reducing injury and illness rates. The identification of a set of effective leading indicators would be a great benefit for the sector since it would allow the industry to focus its attention on meaningful safety metrics. While it may be impossible to establish a specific set of leading indicators for the dairy industry, the measures chosen may need to be linked with specific risks at individual dairy establishments. Finally, the overall maturity of an organization’s safety culture is an important consideration in the effective implementation of leading indicators.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Conflicts of interest

The authors declare that there are no conflicts of interest.