Development and Validation of a Practical Instrument for Injury Prevention: The Occupational Safety and Health Monitoring and Assessment Tool (OSH-MAT).

BACKGROUND: The Occupational Safety and Health Monitoring and Assessment Tool (OSH-MAT) is a practical instrument that is currently used in the German woodworking and metalworking industries to monitor safety conditions at workplaces. The 12-item scoring system has three subscales rating technical, organizational, and personnel-related conditions in a company. Each item has a rating value ranging from 1 to 9, with higher values indicating higher standard of safety conditions.
METHODS: The reliability of this instrument was evaluated in a cross-sectional survey among 128 companies and its validity among 30,514 companies. The inter-rater reliability of the instrument was examined independently and simultaneously by two well-trained safety engineers. Agreement between the double ratings was quantified by the intraclass correlation coefficient and absolute agreement of the rating values. The content validity of the OSH-MAT was evaluated by quantifying the association between OSH-MAT values and 5-year average injury rates by Poisson regression analysis adjusted for the size of the companies and industrial sectors. The construct validity of OSH-MAT was examined by principle component factor analysis.
RESULTS: Our analysis indicated good to very good inter-rater reliability (intraclass correlation coefficient = 0.64-0.74) of OSH-MAT values with an absolute agreement of between 72% and 81%. Factor analysis identified three component subscales that met exactly the structure theory of this instrument. The Poisson regression analysis demonstrated a statistically significant exposure-response relationship between OSH-MAT values and the 5-year average injury rates.
CONCLUSION: These analyses indicate that OSH-MAT is a valid and reliable instrument that can be used effectively to monitor safety conditions at workplaces.

Introduction

Injury is a major occupational health problem. Although the occurrence of occupational injury has decreased remarkably over the past few decades, preventable injuries and deaths still occur. In the USA, the Bureau of Labor Statistics estimated that approximately 8.6 million nonfatal work-related injuries and about 5,600 fatalities occurred in 2007 [1]. The direct and indirect cost of the occupational deaths and injuries amounted to about US$192 billion per annum [1]. In Germany, nearly 900,000 compensable occupational injuries (injuries requiring at least 3 days off work) and about 35,000 occupational diseases occurred in 2012 [2]. This led to medical and rehabilitation costs alone of about €3.9 billion [2].

Many factors can lead to accident or injury in the workplace. Poor physical fitness, poor lifting and carrying techniques, excessive force, low awareness of dangers, stress, and even the nature of workplace design may make important contributions to the occurrence of injuries at workplaces. A wide range of professional injury prevention programs have therefore been carried out globally, especially in industrialized countries, besides the common legal framework and technical guidelines. Most successful injury prevention programs are based on a common set of key elements [3], [4]. These include, e.g., technical improvement, management leadership, worker participation, hazard identification, hazard prevention and control, education, and training.

Due to great interest in comprehensive monitoring and assessment of safety conditions at workplaces, a practical measurement tool was developed by a group of safety professionals at the German Social Accident Insurance Institution for the woodworking and metalworking industries. Key elements in work safety were integrated into a quantitative assessment index, which is currently used by about 500 labor inspectors in the German woodworking and metalworking industries. This article describes the development and validation of the Occupational Safety and Health Monitoring and Assessment Tool (OSH-MAT).

Methods

Development of the initial version of OSH-MAT

A professional injury prevention program, the German Occupational Safety and Health Act, has been carried out for some years in Germany by the German Social Accident Insurance institutions. One important outcome of this program has been the development of a practical assessment tool for safety conditions, termed the “Approach to Classify Safety Measures for Preventing Occupational Injuries and Diseases.” This is also termed the “TOP” approach, since it covers primarily three sets of key elements.

Technical measures (T) include, e.g., safety equipment of machinery, ergonomic design of work materials and workplaces, the design of vehicles and roads, and the substitution of hazardous substances.

Organizational measures (O) are, e.g., the execution of risk assessments, safety tests, instruction of employees, preparation of an emergency plan, and efficiency monitoring of the measures taken.

Personnel measures (P) include, e.g., wearing personal protective equipment and basic and advanced training of employees in carrying out safety instructions and observing occupational health-related communication structures.

The aim of the TOP approach is to provide a comprehensive description of the state of the art of occupational safety and hygiene conditions by evaluating key elements of these conditions. This may permit early identification of practical safety deficits and ensure long-term success in injury prevention.

OSH-MAT is a standardized scoring index of the TOP approach. It covers the same sets of key elements (T, O, and P) with 12 carefully selected scoring items. Each item has a rating value ranging from 1 to 9, with higher values indicating higher levels of safety conditions. A detailed description of OSH-MAT is given in Table 1.

Table 1

Results of inter-rater reliability assessment

Subscale	No	Items	% agreement∗	ICC
Technology	T1	Are work equipment and protective measures in adequate condition and appropriate?	76.2	0.67
	T2	Are the workplaces ergonomic?	74.6
	T3	Are safe transport, handling and storage procedures possible?	81.2
	T4	Are physical, biological and chemical pollutants minimized?	77.9
Organization	O1	Is organization adapted to the work task and to the staff?	74.6	0.74
	O2	Is the risk assessment constructive and comprehensive and are appropriate measures derived and implemented?	81.2
	O3	Is the entrepreneur's attitude towards occupational safety and health positive?	72.1
	O4	Is the operation well prepared for predictable emergencies and accidents?	77.1
Personnel	P1	Are the staff qualified according to their work tasks?	78.7	0.64
	P2	Do the staff know and accept their responsibilities in the field of occupational safety and health?	75.4
	P3	Is the communication within the company appropriate?	72.1
	P4	Do the employees work safely?	79.5

ICC, interclass correlation coefficient.

Agreement with maximum deviation of ± 1 point.

Validation of OSH-MAT

Design and study sample

Validation was carried out by examination of the inter-rater reliability, content validity, and construct validity. Inter-rater reliability was examined in a cross-sectional survey among 128 companies in the German metalworking industry between December 2011 and July 2012. OSH-MAT was used to quantify work safety conditions at the 128 companies by 30 well-trained labor inspectors from the German Social Accident Insurance Institution for the woodworking and metalworking industries. Each company was evaluated twice independently and simultaneously by two labor inspectors. Content and construct validity were evaluated by use of the routinely documented OSH-MAT values and injury rates at 30,514 companies in the German woodworking and metalworking industries.

Collection of occupational injury data

The German Social Accident Insurance institutions are the only institutions that are responsible for the compensation and rehabilitation of occupational diseases and injuries in Germany. In accordance with the German legislation (German Social Code book 7), all compensable occupational injuries (involving at least 3 days' absence from work) in Germany were recorded routinely by the German Social Accident Insurance Institutions. Injuries occurred at work or on the way (to or from work) were recorded separately in two different databases. In this analysis, all formally reported compensable occupational injuries at work in the woodworking and metalworking industries were considered, regardless of the number of working days lost. The linkage between injury data and OSH-MAT assessment results was carried out based on an individual identification number of each company, which was generated routinely by the German Social Accident Insurance institution for the woodworking and metalworking industries.

Statistical analysis

To investigate the inter-rater reliability, agreement between the independent double ratings of each company was examined by quantification of the intraclass correlation coefficient (ICC) and the absolute agreement of the rating values (with a maximum deviation of 1 point). Content validity was evaluated by quantification of the association between the OSH-MAT values and the 5-year average injury rates (injury rates were quantified as , where is the number of cases in year, and is the number of total working hours of the employees in year) of the corresponding companies. Poisson regression analysis was performed to assess this association adjusted for the size of company and industrial sectors.

Construct validity was examined by principal component factor analysis with varimax rotation. All analyses were conducted with the SAS 9.4 statistical software package (SAS Institute, Cary, NC, USA).

Results

The inter-rater reliability examination of OSH-MAT was carried out in 128 small and medium-sized companies (Table 1). The absolute agreement between double ratings of the 12 OSH-MAT items varied from 72% to 81% (with a maximum deviation of 1 point in a 9-point rating system). The ICC values for the three subscales of T, O, and P were 0.67, 0.74, and 0.64, respectively. The ICC values indicated good to very good agreement between the double ratings.

To date, OSH-MAT has been used routinely by about 500 labor inspectors for safety evaluation of 30,514 companies from 47 different industrial sectors in the German woodworking and metalworking industries. Eighty-six percent of these companies are small, with <50 employees; 12% are medium-sized (50–499 employees) and 2% are large companies (> 500 employees).

Fig. 1 describes the 5-year average injury rates in the 30,514 companies in relation to the OSH-MAT values. An inverse exposure–response relationship between OSH-MAT values and the 5-year average injury rates was observed. For comprehensive evaluation of the content validity of OSH-MAT, Poisson regression analysis was carried out (Table 2). The analysis demonstrated that company size was a strong indicator of injury rates. Compared to large companies, small companies had an injury rate that was on average 2.5 times as high. OSH-MAT was also a strong and independent indicator of injury rates. Even after adjustment for relevant confounders, OSH-MAT values showed significant inverse exposure–response relationships with the 5-year average injury rates. Compared to companies with higher OSH-MAT values (≥ 7), companies with lower OSH-MAT values (< 4) had on average about 50% more injuries.

Fig. 1

The 5-year average injury rates of 30,514 companies in the German woodworking and metalworking industries depending on OSH-MAT subscale values. OSH-MAT, Occupational Safety and Health Monitoring and Assessment Tool.

Table 2

Poisson regression analysis of the association between OSH-MAT values and 5-year average injury rates

	Crude		Adjusted∗
	RR	95% CI	RR	95% CI
Size of company
≥500 workers	1	—	1	—
200–499 workers	1.67	1.65–1.68	1.58	1.56–1.60
100–199 workers	2.06	2.03–2.08	1.95	1.92–1.97
50–99 workers	2.29	2.25–2.32	2.16	2.13–2.19
<50 workers	2.63	2.60–2065	2.54	2.52–2.56
OSH-MAT values
OSH-MATTechnology: ≥ 7	1	—	1	—
OSH-MATTechnology: 4 – <7	1.49	1.48–1.50	1.29	1.28–1.30
OSH-MATTechnology: < 4	1.93	1.90–1.97	1.42	1.40–1.45
OSH-MATOrganization: ≥ 7	1	—	1	—
OSH-MATOrganization: 4 – <7	1.38	1.37–1.39	1.19	1.18–1.20
OSH-MATOrganization: < 4	1.97	1.94–2.0	1.38	1.35–1.40
OSH-MATPersonnel: ≥ 7	1	—	1	—
OSH-MATPersonnel: 4 – <7	1.41	1.40–1.42	1.23	1.22–1.24
OSH-MATPersonnel: < 4	1.94	1.90–1.97	1.44	1.41–1.47
OSH-MATTOP: ≥ 7	1	—	1	—
OSH-MATTOP: 4 – <7	1.47	1.46–1.49	1.24	1.23–1.25
OSH-MATTOP < 4	2.10	2.06–2.14	1.46	1.43–1.48

CI, confidence interval; OSH-MAT, Occupational Safety and Health Monitoring and Assessment Tool; RR, relative risk.

Adjusted for industrial sectors and size of company.

The construct validity of OSH-MAT was examined by the use of routinely documented OSH-MAT values of the 30,514 companies in a factor analysis (Table 3). The analysis showed that all four key questions from the “T(echnology)” field loaded on Factor 1, all four key questions from “O(rganization)” loaded on Factor 2, and all four key questions from “P(ersonnel)” loaded on Factor 3. They measure the internal conditions relating to the “Technical”, “Organizational” and “Personnel” fields, respectively.

Table 3

Factor analysis of routinely documented OSH-MAT values of 30,514 companies

Items	Factors loading∗
	Factor 1	Factor 2	Factor 3
T1	0.80	0.39	0.40
T2	0.82	0.38	0.39
T3	0.81	0.38	0.39
T4	0.80	0.40	0.38
O5	0.39	0.80	0.40
O6	0.38	0.80	0.35
O7	0.37	0.78	0.41
O8	0.40	0.79	0.40
P9	0.41	0.41	0.79
P10	0.40	0.42	0.78
P11	0.38	0.39	0.80
P12	0.43	0.40	0.78

OSH-MAT, Occupational Safety and Health Monitoring and Assessment Tool.

Spearman correlation, varimax rotation, eigenvalues > 0.5.

Discussion

The aim of the present analyses was to validate a routinely used injury prevention measurement tool, and to evaluate whether this instrument can be used in a standardized way to monitor safety conditions at workplaces.

Injury is an important public health problem worldwide. According to the global data of the United States National Center for Injury Prevention and Control, injury is the leading cause of death for persons aged 1–44 years and the third leading cause of death for persons aged 45–54 years [5]. If the impact of injury is measured in years of potential life lost, injury is ranked as the highest burden of disease worldwide, followed by cancer and heart disease [6].

Modern injury prevention in the workplace has the aim not only of preventing injuries, but also of early recognition and elimination (or reduction) of all risks to workers' lives and health at work. For this aim to be achieved, all potential work-related risks need to be identified and assessed in advance. In many countries, this process is laid down in OSH legislation. However, without clear definition of its key elements and detailed procedures, some small and medium-sized companies still experience difficulties in identifying such hazards and in applying suitable measures to prevent work-related health risks. This problem appears to be demonstrated by empirical data collected in this study among the 30,154 companies. They show that the size of company is a strong indicator for injury rate, and is positively associated with OSH-MAT values.

As a practical and technical supply of current OSH legislation in Germany, OSH-MAT serves as a useful tool for systematic evaluation of safety conditions at workplaces. It may help to identify early relevant occupational hazards or safety deficits, and to promote effective prevention of work-related injuries and diseases. Due to its higher specificity and technical requirements, OSH-MAT can be applied only by well-trained safety engineers with many years' work experience. That means that, in order to ensure valid and reliable use of OSH-MAT, this instrument should be applied by well-trained safety personnel who are familiar with the working process and technical standards in an industry for which OSH-MAT is designated. For this reason, the generalizability of this instrument is limited. Without well-trained safety professionals and reversion of its key items, this instrument is not recommended for use by other industry sectors.

Currently, some 500 well-trained labor inspectors of the German Social Accident Insurance Institution in the woodworking and metalworking industries are eligible to use OSH-MAT for safety measurements. The excellent inter-rater reliability agreement of OSH-MAT values among 30 different labor inspectors indicates reliable routine use of OSH-MAT among the 30,514 companies in the German woodworking and metalworking industries.

The theoretical construct of OSH-MAT was examined in this study by factor analysis. The factor loadings demonstrated that the three subscales T, O, and P of OSH-MAT measure three different safety aspects, while the four items within each subscale measure exactly the same aspects. These results correspond exactly to the structural measure theory behind this instrument.

Poisson regression analysis demonstrated that OSH-MAT is a strong and independent indicator for injury rates. This instrument can therefore be used effectively to monitor safety conditions and to provide useful guidelines for future prevention priorities or strategies. Here we must emphasize that OSH-MAT is a measurement tool and not an intervention. The use of OSH-MAT may not automatically lead to an improvement of injury rates. However, it can give warnings of safety deficit; provide priorities in safety activities; and push for a company to implement interventions to reduce possible future risks.

Theoretically, companies with poor OSH-MAT values may have higher potential of improving injury rates than companies with higher OSH-MAT values. At which level of OSH-MAT is an intervention program more effective, and to which extent of changing OSH-MAT values may largely improve the injury rates still need to be investigated in longitudinal studies with randomized experimental design.

Overall, the present analyses indicate that OSH-MAT is an evidence-based practical instrument that can be used effectively and in a standardized form for the monitoring of safety conditions at workplaces.

Conflict of interest

We declare that there is no potential conflict of interest that could raise questions about the credibility of this paper.