Determinants of the accuracy of occupational hygiene expert judgment.

An experimental study was performed to determine the applicability and accuracy of occupational hygienist's expert judgment in occupational exposure assessment. The effect of tier 1 model application on improvement of expert judgments were also realized. Hygienists were asked to evaluate inhalation exposure intensity in seven operating units in a tile factory before and after an exposure training session. Participants' judgments were compared to air sampling data in the units; then after relative errors for judgments were calculated. Stepwise regressions were performed to investigate the defining variables. In all situations there were almost a perfect agreement (ICC >0.80) among raters. Correlations between estimated mean exposure and relative percentage error of participants before and after training were significant at 0.01 (correlation coefficients were -0.462 and -0.443, respectively). Results showed that actual concentration and experience resulted in 22.4% prediction variance for expert error as an independent variable. Exposure rating by hygienists was susceptible to error from several sources. Experienced subjects had a better ability to predict the exposures intensity. In lower concentrations, the rating error increased significantly. Leading causes of judgment error should be taken into account in epidemiological studies.

Introduction

Exposure assessment is a core element of all health risk management plans. There are several well recognized strategies for assessing and managing occupational inhalational exposures; which are mostly based on the air sampling as a gold standard1, 2). However, under different circumstances, only limited numbers of these strategies are applicable3). Lack of experienced hygienists, instrumental limitations, budget deficit and variable nature of the exposure are among the most limiting factors in application of routine air sampling strategies4). These problems are highlighted in developing and under-developed countries with financial crisis and limitations in access to expert workforce. Reconstruction of past exposures in the absence of adequate sampling data is another challenge in the exposure assessment arena.

Use of experts, as a source of information in exposure assessments, gained popularity in epidemiological studies since decades ago2). Expert judgment, as a cost-effective method, in this fashion could enhance the quality of exposure data5). Most studies in this area used expert opinions in reconstruction of exposure for case-control studies6,7,8,9). Generally, in the absence of measurement data, expert rating could be the best available technique for retrospective exposure assessment. However, there are different levels of agreement between results obtained by the expert judgments and actual measurements. Despite wide application of expert judgment in the past exposure reconstruction, its application as an instrument in screening and preliminary evaluation of inhalational exposures is not common. Expert data in combination with other uncertain sources of information such as mathematical models, could lead to better understanding of the problems10,11,12). Application of expert decision as a prior in Bayesian data analysis (BDA) framework is a unique state of the art application of expert judgment in occupational hygiene5, 12).

Reliability of expert opinions in predicting the intensity of inhalational exposure is questionable. It depends on such environmental, demographical and psychological factors as gender, educational level, experience and the odor of the contaminant13,14,15). This implies that further work should be conducted to compare available inhalation exposure assessment methods, including expert judgment16). The objectives of this study were: 1) to investigate the role of the experiences and personal factors of the occupational hygienists in expert judgment correctness; 2) to investigate the usefulness of training on the reduction of exposure rating errors; and 3) to determine the applicability of experts rating under different exposure scenarios in a tile and ceramic industry.

Subjects and Methods

Study design

This is a “cross sectional, before-after study” conducted in a tile and ceramic production factory in Yazd, Iran. Tile and ceramic production is the main industrial activity in Yazd province with about 9,900 workers. The factory had six production lines with a capacity of 19,000 m2/day for production of wall and floor ceramic and tile. Raw materials consisted of silicates, feldspars and carbonates which were used in the form of powder to make pastes and coatings. Background data about production processes, tasks, and dust concentration in ceramic and tile industries were gathered from the last three years measurements in fifteen similar factories and were used for preparation of educational materials for hygienists. For gathering general information and demographic data a researcher-made questionnaire was also distributed to the hygienists. Need for cognition scale (NFC) questionnaire was used as a measure of participants’ tendency in problem solving. Validated Persian version of 18 items NFC questionnaire (Cronbach’s alpha=0.7) was also distributed and filled by the participants. A separate questionnaire was developed to elicit the hygienists’ ratings.

Measurements

Since there was no change in processes and work practices in the factory, the annual measurement records of the last three years of the factory were used as a gold standard for comparison with hygienists’ predictions. All historical measurements were gathered from gravimetric sampling and analyses of respirable dusts based on national institute of occupational health (NIOSH) method 060017) and were reported in mg/m. All samples were taken by a personal sampling pump at 1.7 liter per minute for at least 30 min on a fiberglass filter.

Hygienists

Two groups of occupational hygienists (5 experienced and 8 inexperienced) with at least bachelor degree in occupational hygiene engineering were asked to participate in this study. Experienced hygienists had at least 3 yr of experience in the hazardous agent’s measurements, health, safety and environment (HSE) consultation and teaching occupational hygiene principles.

Both groups were relatively homogenous in term of educational background in occupational hygiene. Inexperienced hygienists had no field experience and were recently graduated from the university. At the first step, hygienists were asked to visit the production units and predict the minimum, maximum and mean respirable dust concentration. During their visit, a competent ceramic technician gave them the necessary information about the process. Each participant had one working day tour in the factory.

After that, participants attended in a training session which covered general concepts about industrial air pollution generation and principles of prevention, some basics about the control of substances hazardous to health (COSHH) and control banding. Then, the hygienists were asked to re-visit the factory units and assess the minimum, maximum and mean respirable dust concentration.

Statistics

Agreement reliabilities among raters were calculated by the Fleiss kappa statistic in R software with irr package18). Inter-class correlations were also computed for rater’s predictions. All raters were asked to categorize their estimations based on American industrial hygiene association (AIHA) exposure rating group. Fleiss Kappa test was used to calculate inter-rater agreement.

Absolute (E) and relative error (E) of predictions were calculated as criteria of rating validity (Equations 1 and 2).

C and C refer to sampling data and hygienists predicted concentrations, respectively.

Correlation between NFC score and predictions were also investigated. Effect of training and other parameters on prediction precision was calculated in terms of absolute and percentage of relative error. To determine the significant parameters on raters’ error reduction, stepwise linear regression was performed with age, sex, NFC score, experience and actual concentration as the predictors of rating error.

Results

Descriptive characteristics

The mean age of the experienced and inexperienced participants was 29.3 (SD=1.55) and 26.4 (SD=1.32) yr, respectively, with no significant difference regarding age between experienced and inexperienced participants (p=0.131). Mean tenure in experienced group was 4.9 yr (SD=0.94). NFC scores were also not significantly different in both groups (p=0.559).

The overview of measurements from last three years—as a gold standard—is shown in Table 1. The highest dust concentration was observed in crushing, press and coating preparation units, respectively. Meanwhile, there was no significant difference in mean dust concentration in different units (f= 1.299, p=0.313).

Table 1.

Descriptive statistics for historical measurements

Unit	N	Minimum	Maximum	Mean	SD
Crushing	2	13.30	19.20	16.25	4.17
Ball mill	5	1.92	19.06	7.75	6.94
Coating preparation	4	0.96	18.63	9.17	7.25
Press	2	4.84	14.90	9.87	7.11
Coating line	2	1.99	8.67	5.33	4.72
Furnace	4	5.96	10.60	8.39	2.25
Packing	4	0.99	3.90	2.45	2.39

Data acquired from 3 yr consecutive measurements from production units. All numbers in mg/m3.

Table 2 shows the predicted concentration before and after training course. Similar as those observed in sampling data, both groups of hygienists also ranked crushing unit as a most polluted unit. However, this consistency was not observed for the rest of units.

Table 2.

The hygienist’s concentration prediction before and after training session

Raters	Factory Unit	Mean (SD)		p-value
		Before	After
Non-experienced	Crushing	11.10 (2.22)	13.79 (4.40)	0.080
	Ball mill	7.06 (2.41)	7.44 (1.72)	0.068
	Press	5.00 (1.20)	7.25 (1.49)	0.216
	Coating preparation	5.59 (1.74)	6.44 (1.76)	0.273
	Coating line	3.28 (0.92)	4.41 (1.36)	0.461
	Furnace	3.16 (0.9)	4.34 (0.81)	0.225
	Packing	2.34 (0.83)	2.88 (0.74)	0.465
Experienced	Crushing	11.70 (5.97)	21.20 (7.92)	0.075
	Ball mill	7.50 (4.57)	11.00 (3.81)	0.522
	Press	4.30 (1.86)	6.10 (2.41)	0.011
	Coating preparation	4.60 (4.83)	5.90 (3.29)	0.088
	Coating line	2.30 (1.89)	2.60 (1.64)	0.028
	Furnace	2.04 (1.39)	3.30 (1.60)	0.017
	Packing	2.46 (2.64)	3.74 (2.82)	0.016

There were also gender specific differences in some measurements (Table 3). Females in all situations, except for coating line predicted a lower level of dust in comparison to males. However, only in a part of measurements these differences were significant.

Table 3.

Mean reported concentration according to gender for experienced subjects (results of Mann-Whitney U test)

Factory Unit		Male (n=4)	Female (n=3)	U value	p- value
Crushing	Before	11.60 (0.55)	9.67 (1.76)	4	0.476
	After	17.07 (1.15)	9.00 (1.00)	0.00	0.032
Ball mill	Before	8.25 (1.45)	5.33 (0.33)	0.5	0.048
	After	8.37 (0.94)	6.67 (0.67)	2	0.138
Coating preparation	Before	6.56 (0.36)	3.67 (0.44)	0.00	0.032
	After	6.62 (0.24)	5.00 (0.58)	0.5	0.048
Coating line	Before	3.05 (0.41)	3.67 (0.73)	4	0.48
	After	3.57 (0.22)	5.00 (1.00)	3.5	0.368
Press	Before	5.75 (0.32)	4.00 (0.76)	0.5	0.05
	After	8.25 (0.75)	6.33 (0.33)	1	0.064
Furnace	Before	3.44 (0.48)	2.50 (0.29)	2	0.150
	After	4.37 (0.24)	4.08 (0.74)	4	0.476
Packing	Before	2.67 (0.45)	1.67 (1.67)	0.5	0.048
	After	3.12 (0.37)	2.33 (0.33)	2	0.150

The ICC analysis shows almost perfect agreement (ICC>0.80) among raters (Table 4). Only the ICC for the mean values reported before training by inexperienced raters was marginal (ICC=0.807). Generally, the ICCs in experienced group were higher than inexperienced group; but these differences were decreased after training. Nearly in all cases the lowest ICC was for minimum exposure ratings (except that in inexperienced rates before training which the ICC for minimum values reported was the lowest). Training session had a positive effect on ICCs for inexperienced raters. However, it had only slight positive effect on ICCs in experienced raters.

Table 4.

Raters ICCs based on experience category, rated parameter and rating round

Raters	Parameter	ICC (95% CI)
		Before	After
All	Min	0.945 (0.852–0.989)	0.919 (0.795–0.983)
	Max	0.960 (0.894–0.992)	0.982 (0.954–0.996)
	Mean	0.955 (0.877–0.991)	0.960 (0.897–0.992)
Experienced	Min	0.941 (0.823–0.988)	0.871 (0.662–0.974)
	Max	0.944 (0.847–0.989)	0.965 (0.907–0.993)
	Mean	0.966 (0.906–0.993)	0.950 (0.866–0.990)
Non-experienced	Min	0.833 (0.501–0.967)	0.874 (0.629–0.975)
	Max	0.888 (0.603–0978)	0.965 (0.876–0.993)
	Mean	0.807 (0.415–0.961)	0.931 (0.791–0.986)

As raters were asked to categorize their estimations based on AIHA exposure rating, Fleiss Kappa test was used to calculate inter-rater agreement. It was higher after training in all experienced and inexperienced group ratings. However, in all cases the degree of agreement between raters was fair (0.2–0.4).

Correlation analysis

Correlation was made between mean percentage of relative error of participants and their NFC scale. There was no significant relation between reported NFC score and percentage of relative error in participants (Table 5). Correlations between estimated mean exposure and relative percentage error of participants before and after training were significant at 0.01 level (correlation coefficients were −0.462 and −0.443, respectively). In other word, hygienists had a higher percentage of error in lower concentrations. There was also a strong positive correlation between relative error before and after training session (correlation coefficient = 0.597). In contrast to inexperienced subjects, relative errors were decreased after training session in all units. Training session led to 22.85, 14.67 and 12.35% reduction in experienced hygienists’ error for press, furnace, and coating line, respectively. Relative errors were decreased after training session in all units except for packing unit. Training did not have a positive effect on error reduction in packing unit for experienced hygienists. Among inexperienced hygienists the error in packing unit raised from 73.82 to 104.84 after training session as well. However, this order is not the same in inexperienced subjects; among which training led to 22.88, 18.22 and 15.04% reduction in slurry preparation, press and furnace, respectively.

Table 5.

Effect of training on prediction error in experienced and non-experienced hygienists

Factory unit	Experience
	Yes			No
	Ebefore %	Eafter %	Δ Error	Ebefore %	Eafter %	Δ Error
Crushing	31.65	22.49	–9.16	37.18	33.46	–3.72
Ball mill	24.12	16.89	–7.24	44.46	45.76	1.30
Press	49.30	26.45	–22.85	56.38	38.16	–18.22
Coating preparation	39.00	32.00	–7.00	66.46	43.58	–22.88
Coating line	38.95	26.60	–12.35	56.80	51.16	–5.64
Furnace	62.30	47.62	–14.67	75.62	60.58	–15.04
Packing	26.48	26.47	0.00	73.82	104.84	31.02

Simple correlation for all participants showed that after training, error reduction degree was significantly correlated with participants’ relative error before training (correlation coefficient=−0.303, p=0.003). Experienced stratified analysis showed that in experienced raters it was also significant (correlation coefficient=−0.538, p=0.001), however there was not a statistically significant correlation among inexperienced subjects (cc=−0.3, p=0.08)

Regression analysis

Hygienists’ relative error after training (Eafter) was used as a dependent variable in a stepwise regression. Age, sex, experience, NFC score, and actual concentration entered in the model as the independent variables. NFC score, sex, and age produced no significant prediction as the independent variables to the Eafter. Therefore these three parameters were eliminated from the model. Results showed that actual concentration and experience can describe 22.4% variance prediction for the independent variable (Table 6).

Table 6.

Results of a stepwise multiple regression analysis with experience and concentration as the dependent variables

Model	Cumulative r2	F change	p-value
Constant, experience	0.163	17.39	0.001
Constant, experience concentration	0.224	6.87	0.01

Discussion

The occupational hygienists in this study quantified the exposure intensity based on their field observation and background knowledge. We found a good correlation between ratings and actual measurements similar to other studies5). Our results suggest that expert rating is among the best available techniques for assessment of occupational exposures in the situation that we encounter sparse data. The aim of the most studies in this field is application of rating from retrospective exposure assessments2, 14), however our findings are also applicable in this field but our main goal was to investigate the applicability of rating in primary exposure assessment in industries. Our findings could be used in combination with mathematical models such as BDA framework for occupational hygiene decision making.

We found nearly perfect agreement between subjects’ ratings and measurements in all units. However our findings should be interpreted with caution, because the units in this study were highly polluted and the results showed that dust concentration itself could also affect rating accuracy. In some cases, huge errors were observable. It seems that the raters’ error depended on several factors. It was concluded that the effect of actual concentration on prediction error should not be neglected. Some other researchers used normalized expected concentrations according to the exposure limit for the predictions; but in this study the subjects were asked to report their judgment as a quantitative measure in mg/m3. In this situation it is harder for raters to exactly report a number. Despite this difference, our results about raters’ reliability are in accordance with prior findings.

Correctness of exposure ratings by hygienists is susceptible to error from several sources. These sources should be taken into account in epidemiological studies, preliminary assessments or regulatory judgments. In general, in lower concentrations, the rating error increased significantly. Even in some situations there were significant differences between males and females. But with adequate education this error could be reduced in some extent. Logan et al. found that training workshop could improve the accuracy of raters about 50%19). Another study by Rocheleau et al. found that training had significant improving effect on all raters8). However, some studies found that these trainings had no20) or only little effects21) on quality of assessments. Our findings implied the importance of training session and supporting the raters before conducting expert assessment to increase the correctness of the results. Effect of this training course in inexperienced raters was much more important. Although we found that training led to error reduction in both groups, but this difference in ratings before and after training was only significant in inexperienced raters. However it should be studied which training method and which educational content is most suitable for error reduction in this field.

We used subjects with no intimate familiarity with the ceramic and tile industry and found a good agreement between their ratings and actual measurements. It is in accordance with Ramachandaran et al. study which found that good rating necessarily doesn’t need prior familiarity with desired exposure scenario5). However, it seems that use of experienced hygienists could improve the quality of exposure driven data. However training was also effective in both groups, but experienced subjects error reduction was higher than those without experience. On the other hand the adaptive response from experienced raters who worked for a long time in industry should be taken into account. Experienced hygienists may show adaptive response to their surrounding work abnormalities and exposures. This phenomenon was observed in some other studies on odor and safe behavior15). In cases with low dust concentration, the error was higher in comparison with other units; sensory effect of pollutants could affect the rating of raters. Other studies also found that odor can trigger the rating of subjects15).

We found that NFC is not a predictor for net value of reported exposure intensity and error reduction. However, Vadali et al.13) found that NFC is a good predictor of rater’s error. Among all parameters studied in this project, we found experience and concentration of the pollutant in the workplace as the best error predictors in raters. It seems that experienced raters are the best choice in exposure rating for community based case control studies and so on. However, it should be kept in mind that we should interpret the results of exposure judgment according to dustiness of units. In clean environments, raters are prone to higher errors; however this error is minimal in dusty environments. Between these two factors, it seems that experience is the most important one with 16.2% variance. It seems that more experience gives the subjects this ability to better predict the exposures. Kandlikar et al.22) described two types of expertise as a factor in experts’ predictions. In our raters, substantive expertise which is a kind of knowledge leads to the better estimation of subjects with experience22). Other studies also found that the ability of raters depends on their industrial experience. It implies that characteristic of raters should be considered in case control studies. Our findings support this belief that experienced subjects better predict the exposure. It is also common in other studies in general2). Results showed that experience has superiority in comparison with knowledge in exposure rating. Participants reported a wider range of prediction after analysis; it may be due to being conservative about decision making. Our findings suggest that expert judgment should be used in a systematic matter and with consideration of affecting parameters.

Limitations

Our bank of measurements suffered from measurement limitation. However, we tried to check its numbers with other factories but the confidence interval of the estimated means was large. Exposure measurements in this study were based on long time continuous air sampling, but day to day variation is an important issue in interpretation of occupational exposures. However, the production situations and patterns were exactly same during all work days in the factory. No change was made to the production line or materials. And therefore we accepted exposure situation in the time of study same as those occurred in the measurements. Regarding the number of participants, it was impossible for us to conduct a tour in the factory with more participants. According to our study design we ought to conduct all visits and expert evaluations in one day and in one specific factory. With our crews the team was about 20 persons and it was relatively hard for facility administrative to handle more visitors at same time.

In most situations the concentration was high. Some other studies showed that type of the contaminant could affect the rating of the rater8); therefore our results may be only valuable for mineral industries. Our experienced raters had about 5 yr of experience in occupational hygiene sectors. Other studies should consider the effect of experience in other ranges. Most of units in our study had high concentration of pollution and it may limit the value of our findings. However, it is dominant in most tile and ceramic facilities, but other studies should test the applicability of expert rating in different range of concentrations. We used the raters’ decisions separately; however other studies suggest that the inter-rater reliability would be improved if they work together. Use of Bayesian data analysis could lead to better rating of measurements.