Evaluation of the Quality of Occupational Health and Safety Management Systems Based on Key Performance Indicators in Certified Organizations.

BACKGROUND: Occupational Health and Safety Management Systems are becoming more widespread in organizations. Consequently, their effectiveness has become a core topic for researchers. This paper evaluates the performance of the Occupational Health and Safety Assessment Series 18001 specification in certified companies in Iran.
METHODS: The evaluation is based on a comparison of specific criteria and indictors related to occupational health and safety management practices in three certified and three noncertified companies.
RESULTS: Findings indicate that the performance of certified companies with respect to occupational health and safety management practices is significantly better than that of noncertified companies.
CONCLUSION: Occupational Health and Safety Assessment Series 18001-certified companies have a better level of occupational health and safety; this supports the argument that Occupational Health and Safety Management Systems play an important strategic role in health and safety in the workplace.

Introduction

Despite major advances in occupational health and safety (OHS), which have led to measures that aim to prevent injuries and illness in the working environment, challenges persist in many organizations [1]. Risks endanger the workforce, equipment, the working environment, and impact the competiveness and economic performance of both industries and communities. Occupational accidents and diseases have profound adverse consequences; workers are injured, equipment is destroyed, the quantity and quality of production falls, there are economic losses due to early retirement and staff absence, all of which adversely affect the organization's reputation and competitiveness [2]. According to statistics, such incidents result in nearly 2.3 million deaths every year and incur costs over 2.8 trillion dollars globally [3]. These findings clearly show that occupational accidents and diseases are a major concern and must be properly managed.

Increasing awareness of the adverse effects of occupational accidents and diseases on workers and workplaces has led to the increasing enforcement of preventive measures to combat risks [4]. Industrial countries initially attempted to address the issue through the introduction of health and safety regulations. But catastrophic accidents such as Bhopal (India), Chernobyl (Ukraine), and Piper Alpha (UK) resulted in a view of OHS that was more focused on technical and human factors [1], [5]. Since then, a number of OHS management systems (OHSMSs) have been introduced by national and international bodies. These systems are comprehensive tools that can take into account many of the facets of OHS [6].

Recent research shows that the OHSMSs play a fundamental role in tackling OHS challenges, improving worker safety, reducing workplace risks, and creating better, safer working conditions [7]. The most reputable OHSMSs, which are increasingly popular in organizations, include the Occupational Health and Safety Assessment Series (OHSAS 18000), the Occupational Safety and Health Administration's Voluntary Protection Program, and International Labor Organization guidelines (ILO-OHS 2001) [8]. Over the years, the OHSAS 180001 British Standard [6] has emerged as the most popular system. The standard has been implemented systematically in workplaces to help managers identify and address OHS risks. The 2006 report by the British Standard Institute stated that nearly 26,222 companies in 116 countries (including Iran) had been certified as meeting the OHSAS 18001 standard; this figure had reached 56,251 by the end of 2009 [9].

Despite the widespread implementation of OHSMSs in the workplace, there is a lack of comprehensive, robust evidence to demonstrate their effectiveness [10]. This may be due to the fact that their effectiveness is evaluated by retrospective performance indicators, such as time lost due to occupational illness or accidents, measured as Lost Time Injuries, and the Injury Severity Rate [7], [11]. These historical indicators focus on past events. Therefore, in many working environments, they suggest a good level of health and safety as accidents do not happen, although workers are exposed to hazardous conditions. Conversely, they can indicate a poor level of health and safety when unfortunate accidents happen—even if workers are not exposed to hazardous conditions. Therefore, these lagging indicators cannot be used in isolation as they can fail to provide detailed information about complex, multifaceted OHS situations [7], [11].

According to Oztaş et al [12], the performance of a system must be monitored with appropriate indicators, otherwise the investment is wasted. Research into OHSMSs has resulted in systems that include proactive instruments, which can provide timely information and help to predict potential health and safety problems [6]. There is a clear need for a comprehensive, informative approach to evaluating the quality of OHSMSs based on both lagging and leading indicators [7], which will enable managers and professionals to assess the success of their OHSMSs [7], [13].

In this context, the main aims of this study are as follows: (1) to develop appropriate criteria and performance indicators for OHSMSs; and (2) to compare OHSMSs performance criteria in OHSAS 18001-certified and noncertified organizations in Iran.

Literature review

OHSMSs are systematic instruments and powerful tools that enable organizations to manage their occupational risks, and help managers to control health and safety challenges in the workplace [14]. Their most important role is to support and promote good practice, and identify significant social and economic issues in the area of OHS [15].

Most new research into OHSMSs has been restricted to specific topics such as the certification process, the benefits of OHSMS implementation, the impact on company performance and employees' attitudes toward unsafe acts, and its effects on the occupational accident rate. For example, Santos et al [16] investigated the advantages of OHSMSs in small and medium-sized companies in Portugal. Fernandenz-Muniz et al [2] addressed the relationship between occupational safety management and performance in 455 Spanish companies. The study found that safety management systems had a positive effect on both safety, and financial, economic and competitive performance. Remawi et al [17] examined the relationship between safety management systems and employees' attitudes toward unsafe acts. Vinokumar and Bhasi [18] studied the effect of safety management system certification on safety performance in the chemical industries. The study investigated workers' perceptions of six safety management practices in OHSAS 18001-certified and noncertified companies. Finally, Abad et al [6] assessed the correlation between OHSAS 18001 adoption, and objective measures of safety performance and productivity.

However, few researchers have investigated the performance of OHSMSs in certified organizations based on OHSMS-related criteria and indicators. Bottani et al [19] compared the performance of safety management systems in certified and noncertified organizations. The study evaluated safety variables correlated with OHSMSs, such as risk assessment, corrective action, training, communicating safety goals, and updating risk data. However, the method relied on questionnaires that were distributed to workers and managers. It was therefore subjective and insufficient to draw any robust conclusions about OHSMS performance. A more recent study took a multicriterion decision-making approach to evaluating the effectiveness of OHSMSs [20]. The quantitative model that was developed was used to evaluate performance of OHSMSs in certified organizations.

Despite the extensive research into OHSMSs, there appears to be no systematic evaluation of the performance of OHSMSs based on appropriate key performance indicators in certified and noncertified organizations. Given the importance of evaluating OHSMS performance, this study attempts to fill the gap in the literature.

Material and methods

Population and sample

This study evaluates the effectiveness of OHSMSs on the management of health and safety. The sample consists of six companies in Iran that are involved in large-scale industrial projects such as the design and construction of power, oil, and gas facilities. Three of the companies were OHSAS 18001 certified, and the other three were not. The three certified companies had at least 3 years' experience of health and safety management.

Survey instrument

OHSMSs take a systems approach. This is based on the “Plan, Do, Check, Act” management model, which is composed of items such as policy, objectives, strategies, practices, procedures, functions, and roles. Therefore, the five core activities of most OHSMSs (and especially OHSAS 18001) are policy, planning, implementation, checking, and management review. Each activity is associated with a set of criteria that have certain effects on system performance. As the aim of this study was to investigate health and safety practices, a set of criteria and related indicators was developed for each of the five OHS activities (Table 1).

Table 1

Occupational health and safety management systems (OHSMS) areas of activity, associated criteria, and key performance indicators (KPI)

OHSMS activity	Code	Criteria	KPIs
Policy	PO1	A. Top management commitment	A1: The no. of OHS meetings in which top managers participate
	PO2	B. Communicating OHS policy & availability at workstations	B1: Percentage of employees informed about OHS policy
	PO3	C. Reviewing & updating OHS policy	C1: The no. of OHS policies that have been reviewed
	PO4	D. Consistency with other organizations' policies	D1: Percentage of OHS regulations & standards applicable to workstations
	PO5	E. Workers' participation in developing OHS policy	E1: The no. of OHS hazards reported by workers
	PO6	F. Simplicity & understandability of OHS policy	F1: The no. of workers who have a good understanding of OHS policy
	PO7	G. Preliminary risk assessment for developing OHS policy	G1: The no. of risk assessments carried out in units
	PO8	H. Supervision of OHS policy implementation	H1: The no. of managerial meetings to discuss OHS issues
Planning	PL1	I. Workers' participation in workstation risk assessments	I1: The no. of near-miss reports by workers
	PL2	J. Encouraging workers to participate in risk assessments	J1: The no. of rewards given to workers for OHS hazard reports
	PL3	K. Recording & reporting OHS activities for risk assessment planning	K1: The no. of units in which OHS report & record-keeping systems exist
	PL4	L. Communicating OHS activities	L1: The no. of OHS brochures distributed to workers
	PL5	M. Reviewing & updating risk assessment policies	M1: The no. of risk assessments updated
	PL6	N. Using units' OHS data during OHS program development	N1: The no. of near misses
	PL7	O. Deadline for OHS programs	O1: The no. of OHS programs carried out in a defined period
	PL8	P. Announcing OHS programs & objectives	P1: The no. of OHS events for employees
	PL9	Q. Allocating financial resources to OHS programs	Q1: Financial resources allocated for OHS/y ($)
Implementation and operation	IM1	R. Training workers in OHS to ensure competence	R1: The no. of h allocated for OHS training per person
	IM2	S. Using risk assessment results during OHS training plan development	S1: The no. of workstations for which a risk assessment exists & corrective action or changes have been made
	IM3	T. Announcing OHS activities & issues to workers	T1: The no. of OHS posters, bulletins, or newsletters published
	IM4	U. Workers' participation in OHS activities	U1: The no. of accidents due to a lack of PPE
	IM5	V. Incentive for workers to participate in OHS activities	V1: The no. of rewards for participating in OHS activities
	IM6	W. OHS documentation & regulation	W1: The no. of tasks that have OHS procedures
	IM7	X. Allocating financial resources to ERP	X1: The no. of ERP training course completed
	IM8	Y. Emergency response drills based on risk assessment results	Y1: The no. of workstations that have an ERP procedure
	IM9	Z. Practical emergency response drills based on procedures	Z1: The no. of emergency response drills performed
	IM10	AA. Provision of emergency response equipment & regular inspection & testing	AA1: The no. of verified OHS procedures applied during purchase or use
	IM11	AB. Establishing an organizational structure for OHS	AB1: The no. of units that have an OHS reporting system
Checking	CH1	AC. Measuring & monitoring based on risk assessment	AC1: The no. of units where OHS performance has been evaluated
	CH2	AD. Measuring & monitoring based on lagging indicators	AD1: The no. of OHS violations, & no. of sanctions
	CH3	AE. Record & control systems for OHS activities	AE1: The no. of units that have OHS reporting systems
	CH4	AF. Announcing results of OHS audits to workers	AF1: The no. of meetings held with workers on OHS issues
	CH5	AG. Deadline for OHS audits	AG1: The no. of audits performed in a given period
	CH6	AH. Continuous review of OHS audits	AH1: The no. of audits that have been reviewed
	CH7	AI. Worker involvement in accident investigations	AI1: The no. of accident investigations carried out with worker participation
	CH8	AJ. Reviewing & updating accident investigations	AJ1: The no. of training courses on accident investigation
	CH9	AK. Announcing accident investigation results to employees	AK1: The no. of accident reports sent to units
	CH10	AL. Announcing corrective & preventive actions	AL1: The no. of meetings carried out to discuss corrective & preventive actions
	CH11	AM. Presence of a recording, reporting & analysis system for accidents	AM1: The no. of accidents, reported near misses
Management review	MA1	AN. Having a timeframe to review meetings	AN1: The no. of review meetings carried out
	MA2	AO. Results of OHS activities available for review	AO1: The no. of OHS performance reports from units
	MA3	AP. OHS indicators included in reviews	AP1: The no. of recommendations for continual improvement
	MA4	AQ. Presence of a manager during review meetings	AQ1: The no. of managers of units attending review meetings

ERP, emergency response procedures; PPE, personal protective equipment.

The method for the selection of criteria and indicators has been described in detail elsewhere [20]. In brief, in this study it consisted of: (1) a comprehensive review of the literature and guidelines on the effectiveness of OHSMSs; (2) the development of an initial list of criteria related to the five main activities in these systems; (3) an examination of the relevance and appropriateness of the criteria by five university professors; (4) the elimination of redundancy through simplification and replacement; (5) the design of a questionnaire to assess performance with respect to: policy (8 criteria), planning (9 criteria), implementation (11 criteria), checking (11 criteria), and management review (4 criteria). The final step was an evaluation of the validity and reliability of criteria by a selected sample of 30 OHS managers from different petrochemical and refinery industries.

The data that was collected for each of the five activities was coded and analyzed using SPSS 16.0 software (SPSS Inc., Chicago, IL, USA). Differences in OHS performance between the two groups (certified and noncertified companies) were tested using the Mann–Whitney U test.

Results

Table 2 shows the results of the comparative analysis of the eight performance indicators for the policy component of OHSAS 18001 for certified and noncertified companies. It presents the minimum, maximum, median, and range for all criteria, and the results of the Mann–Whitney U test. Similar results are presented for planning (Table 3), implementation (Table 4), checking (Table 5), and management review (Table 6) activities. The Mann–Whitney U test reveals that there are significant differences between OHSAS 18001-certified and noncertified companies with respect to most OHS criteria. This suggests that OHS performance in OHSAS 18001-certified companies is higher than in noncertified companies.

Table 2

Comparison of policy criteria for Occupational Health and Safety Assessment Series 18001 certified and noncertified companies

Criteria	Certified companies				Noncertified companies				p
	Min	Max	Median	Range	Min	Max	Median	Range
PO1	0.25	1.29	0.92	1.04	0.02	0.32	0.025	0.3	0.000
PO2	0.7	1.3	0.75	0.6	0	0.05	0.025	0.05	0.000
PO3	0.12	2	1	1.88	0	0	0	0	0.000
PO4	1	1	1	0	0	0.3	0.1	0.3	0.000
PO5	0.3	2	0.75	1.7	0.08	0.15	0.1	0.07	0.000
PO6	0.3	0.6	0.5	0.3	0.04	0.1	0.07	0.06	0.000
PO7	0.3	0.85	0.6	0.55	0.08	0.1	0.1	0.02	0.000
PO8	0.2	1.29	0.95	1.09	0.04	0.09	0.05	0.05	0.000

Max, maximum; Min, minimum.

Table 3

Comparison of planning criteria for Occupational Health and Safety Assessment Series 18001 certified and noncertified companies

Criteria	Certified companies				Noncertified companies				p
	Min	Max	Median	Range	Min	Max	Median	Range
PL1	0.12	0.9	0.66	0.78	0	0.03	0.003	0.03	0.000
PL2	0	1	0.1	1	0	0.5	0.01	0.5	0.56
PL3	0.1	0.9	0.5	0.8	0	0	0	0	0.000
PL4	0.23	0.96	0.4	0.73	0.01	0.24	0.023	0.23	0.001
PL5	0.2	0.5	0.28	0.3	0	0.01	0.005	0.01	0.000
PL6	0.35	0.73	0.59	0.38	0.48	1.35	0.82	0.87	0.03
PL7	0.38	0.9	0.7	0.3	0.01	0.1	0.02	0.09	0.000
PL8	0.6	1	0.8	0.4	0.005	0.01	0.005	0.005	0.000
PL9	0.001	0.9	0.09	0.89	0.0005	0.005	0.0005	0	0.000

Max, maximum; Min, minimum.

Table 4

Comparison of implementation criteria for Occupational Health and Safety Assessment Series 18001 certified and noncertified companies

Criteria	Certified companies				Noncertified companies				p
	Min	Max	Median	Range	Min	Max	Median	Range
IM1	0.2	0.7	0.55	0.5	0.04	0.07	0.07	0.03	0.000
IM2	0.23	0.9	0.8	0.67	0.07	0.1	0.08	0.03	0.000
IM3	0.2	0.4	0.2	0.2	0.01	0.01	0.01	0	0.000
IM4	0.1	0.65	0.1	0.55	0.4	0.5	0.45	0.1	0.225
IM5	0.1	0.78	0.56	0.68	0	0.5	0.01	0.5	0.007
IM6	0.2	0.73	0.61	0.53	0.06	0.07	0.063	0.01	0.000
IM7	0.1	024	0.18	0.14	0.003	0.02	0.003	0.017	0.000
IM8	1	1	1	0	0	0	0	0	0.000
IM9	0.1	0.35	0.27	0.25	0.01	0.05	0.01	0.04	0.000
IM10	0.7	1	0.8	0.3	0	0	0	0	0.000
IM11	0.75	0.9	0.85	0.15	0.03	0.26	0.03	0.23	0.000

Max, maximum; Min, minimum.

Table 5

Comparison of checking criteria for Occupational Health and Safety Assessment Series 18001 certified and noncertified companies

Criteria	Certified companies				Noncertified companies				p
	Min	Max	Median	Range	Min	Max	Median	Range
CH1	0.1	1	1	0.9	0.1	0.1	0.1	0	0.000
CH2	0.25	0.8	0.5	0.55	0.4	1.1	0.68	0.7	0.102
CH3	0.1	1	1	0.9	0.1	0.1	0.1	0	0.004
CH4	0.4	0.6	0.5	0.2	0.005	0.01	0.01	0.005	0.000
CH5	0.1	3	0.45	2.9	0	0	0	0	0.000
CH6	0.45	0.65	0.55	0.2	0.005	0.01	0.005	0.005	0.000
CH7	0.02	0.35	0.08	0.33	0.005	0.005	0.005	0	0.000
CH8	0.01	0.09	0.06	0.08	0	0.005	0	0.005	0.000
CH9	0.05	1	0.285	0.95	0	0.005	0	0.005	0.000
CH10	0.45	0.55	0.5	0.1	0.05	0.05	0.05	0	0.000
CH11	0.4	1	0.9	0.6	0.4	0.6	0.5	0.2	0.009

Max, maximum; Min, minimum.

Table 6

Comparison of management review criteria for Occupational Health and Safety Assessment Series 18001 certified and noncertified companies

Criteria	Certified companies				Noncertified companies				p
	Min	Max	Median	Range	Min	Max	Median	Range
MA1	0.3	0.9	0.7	0.6	0	0.2	0.01	0.2	0.000
MA2	0.02	1	0.9	0.98	0.1	0.1	0.1	0	0.2
MA3	0.1	0.7	0.5	0.6	0.01	0.1	0.1	0.09	0.001
MA4	0.2	0.5	0.4	0.3	0	0.1	0.01	0.1	0.000

Max, maximum; Min, minimum.

An important finding is the lack of difference for some criteria, namely: encouraging workers to participate in risk assessments (PL2); using OHS data to prepare units' OHS programs (PL6); workers' involvement in OHS activities (IM4); performance measurement using lagging indicators (CH2); and the presentation of OHS results during the development and review of OHS programs and plans (MA2).

Discussion

Many occupational injuries and associated costs can be prevented or reduced through investment in health and safety [21], [22]. The most effective way to limit occupational accidents is to improve OHS performance [23]. In turn, the best way to improve OHS performance is to establish OHS management as a long-term strategy [6]. This is why many organizations and industries have implemented OHSMSs. The proactive criteria that are integrated into these systems not only decrease OHS risks, but also provide solutions for controlling them and improving organizational OHS performance. However, despite innovative trends in the implementation of OHSMSs, particularly OHSAS 1800l, their effectiveness has been a subject of controversy, causing many managers to think of them as no more than an expensive bureaucratic exercise [9].

To alleviate these concerns, OHS performance should be evaluated on the basis of realistic and appropriate indictors. This is because every problem or failure in OHS has its own particular characteristics, and the success of the OHSMS cannot be judged by simply calculating the number of accidents [24]. Nevertheless, most organizations continue to use lagging indicators, such as Frequency Rate, Lost Time Injuries, and Injury Severity, to evaluate their OHSMS [7], [25]. This can be compared to an individual with a broken leg, whose vital signs (temperature, blood pressure, and respiration) are normal. If the physician only assesses the patient's condition on the basis of these signs, he/she will be considered to be healthy, although in fact he/she has a broken leg [26]. This is an example of Drucker's statement “What gets measured, gets managed” [1].

OHS is complex, which is why the present study used both proactive and reactive indicators to evaluate the effectiveness of OHSMSs. This approach makes it possible to predict OHS performance, and identify weaknesses and defects in the OHSMS. In addition, strategic goals and programs can be defined for planning and resource allocation. The selected criteria and indicators are specific to OHS management practices and play a role in creating safe environments. The OHS management practices evaluated in this study include: management commitment, workers' involvement in OHS activities, employee training, hazard communication, safety briefings, accident investigations, OHS inspections, incentives and rewards system, corrective actions, safety managers' participation in OHS meetings, well-documented OHS rules and procedures, OHS promotion policies, risk assessment, etc.

Our findings are consistent with the work of Fernandez-Muniz et al [27], Wachter and Yorio [28], and Vinodkumar and Bhasi [18]. These authors reveal that their selected criteria are crucial factors for predicting the performance of OHSMSs, and have a fundamental role in improving OHS conditions in the workplace. They suggest that management commitment is extremely influential in the OHS performance of an organization. However, the success of OHS activities and programs also depends on the availability of appropriate resources and adequate support. Therefore, to achieve their aims, managers must have a firm commitment to OHS and prioritize it over productivity.

The findings of this study also indicate that certified companies are most likely to enforce OHS rules and procedures. This is consistent with Vinodkumar and Bhasi [18] who point out that these factors are more important in certified, than noncertified organizations. Rules and procedures have an important role in improving OHS performance. Their enforcement can improve the safety behavior of workers, which may, in turn, prevent accidents. Another result of this study is that the level of employee training is higher in certified, than non-certified companies. This also confirms earlier work [29]. It is important to understand that OHS training is fundamental for safety behavior. Appropriate training not only includes workers in OHS programs and activities; it also helps them to acquire the knowledge and skills required for their tasks, and informs them about potential workplace hazards. Such training is very effective in reducing the number of unsafe acts.

Our results show that the difference between certified and noncertified companies concerning risk assessment and corrective action is statistically significant. Risk assessment is a systematic approach that uses available information to identify hazards and to calculate risk. Our findings suggest that OHSMSs provide the necessary tools to carry out such assessments. This is consistent with Bottani et al [19]. Regarding communication, we found that levels of communication and dissemination of information about OHS issues are higher in certified than non-certified companies. This result is in line with Fernandez-Munize et al [27], who reported that communication and the transmission of information about OHS issues are significantly higher in certified companies.

With respect to incident investigation systems, the findings of this study are in line with the study by Frazier et al [21], who noted that an incident reporting system is a primary factor in safety management. In the same vein, Yoon et al [30] found that accident rates decrease when a company is OHSMS certified. Therefore, it can be concluded that an incident reporting system enhances the safety performance of organizations.

In our study, all differences between certified and noncertified companies were statistically significant, except for a small number of criteria that include: (1) encouraging workers to participate in risk assessment activities (PL2); (2) using OHS data to prepare units' OHS programs (PL6); (3) workers' involvement in OHS activities (IM4); (4) measuring performance using lagging indicators (CH2); and (5) presenting OHS results when developing OHS programs and reviewing plans (MA2). This may be because traditional safety management methods are followed in noncertified companies. These methods use prizes and incentives to motivate workers to work safely. However, they do not always lead to significant improvements in safety, as they are focused on technical requirements and achieving short-term objectives [18]. A final point to note is that our findings support the argument that OHSMSs play a strategic role, which can enhance OHS conditions in the workplace.

Limitations

The study has its some limitations. The main goal was to assess the performance of OHSMSs using appropriate indicators and criteria, which were based on the views of, and suggestions from, only a few OHS experts. Therefore, advice could be taken from a broader range of experts in future studies. Moreover, the influence of OHSMSs on OHS management practices was evaluated in only six companies. Further assessments should be carried out in a broader range of industries.

Conclusion

This study assessed the effectiveness of OHSMSs in improving OHS conditions. The results revealed that the safety performance of OHSAS 18001-certified companies is better than that of noncertified companies. Therefore, it can be concluded that OHSMSs improve OHS conditions and support healthy and safe workplaces. However, establishing and implementing an OHSMS is only the first step in the structured management of health and safety systems in the working environment. To consolidate their role, and increase their acceptance by employees and other beneficiaries, their performance must be assessed using appropriate indicators.

Conflict of interests

The authors declare no conflict of interest.