How accurately does the Brief Job Stress Questionnaire identify workers with or without potential psychological distress?

OBJECTIVES: The manual for the Japanese Stress Check Program recommends use of the Brief Job Stress Questionnaire (BJSQ) from among the program's instruments and proposes criteria for defining "high-stress" workers. This study aimed to examine how accurately the BJSQ identifies workers with or without potential psychological distress.
METHODS: We used an online survey to administer the BJSQ with a psychological distress scale (K6) to randomly selected workers (n=1,650). We conducted receiver operating characteristics curve analyses to estimate the screening performance of the cutoff points that the Stress Check Program manual recommends for the BJSQ.
RESULTS: Prevalence of workers with potential psychological distress defined as K6 score ≥13 was 13%. Prevalence of "high-risk" workers defined using criteria recommended by the program manual was 16.7% for the original version of the BJSQ. The estimated values were as follows: sensitivity, 60.5%; specificity, 88.9%; Youden index, 0.504; positive predictive value, 47.3%; negative predictive value, 93.8%; positive likelihood ratio, 6.0; and negative likelihood ratio, 0.4. Analyses based on the simplified BJSQ indicated lower sensitivity compared with the original version, although we expected roughly the same screening performance for the best scenario using the original version.
CONCLUSIONS: Our analyses in which psychological distress measured by K6 was set as the target condition indicate less than half of the identified "high-stress" workers warrant consideration for secondary screening for psychological distress.

Introduction

On December 01, 2015, the Japanese government launched the Stress Check Program, a new occupational health policy to screen workers with high psychosocial stress in the workplace[1)]. The following two components of the program are mandated: (1) decreasing the risk of mental health problems through periodic surveys and feedback on survey results to workers, and (2) prevention of mental health problems by screening high-risk workers and giving them an opportunity to consult with a physician. However, only weak evidence exists for the effectiveness of these components[2)].

Among the instruments of the Stress Check Program, the program manual recommends using the Brief Job Stress Questionnaire (BJSQ)[3)] and proposes criteria for defining "high-stress" workers[1)]. Although screening of workers with apparent mental health disorders is not the program's primary objective, practitioners do need to know the accuracy or limitation of the instrument, as a certain number of workers needing special care by mental health experts will be screened in the process. This study aimed to examine how accurately the BJSQ identifies workers with or without potential psychological distress.

Methods

Procedure

We conducted an online survey in February 2016. Participants had registered themselves with a Japanese online survey company. The company had access to more than 2,000,000 registered members in all prefectures of Japan. The registered members were encouraged to participate to benefit society by answering social research questions with regard to various issues. In total, 581,660 working people (excluding self-employed, unemployed, or students) aged 20-69 years, eliminating sex or age bias, were randomly invited to participate from the potential pool of participants. The sex ratio was 1:1, with an equal number of participants in each age group (20-29, 30-39, 40-49, 50-59, and 60-69 years). If the eligible workers agreed with the terms and conditions of the online survey, they could access the self-report questionnaire. They were instructed that individual privacy would be strictly protected and that both participation in the study and withdrawal from the study were voluntary. Responses from the participants were considered to be their consent for participating in the study. Participants were given modest remuneration equivalent to about $5. For financial reasons, recruitment was stopped when the number of accepted participants exceeded 1,650.

Measures

1) BJSQ

The BJSQ consists of 57 items used to assess job stressors (17 items: e.g., psychological job demands, job control), psychological and physical stress reactions (29 items), and buffering factors, such as social support at work (11 items). The program manual proposes criteria for defining high-stress workers based on the BJSQ[1)]. High-stress is defined as having the highest level of stress reaction (criterion (i) ) or having a moderate level of stress reaction, along with having the highest job stressors (or lowest social support in the workplace) (criterion (ii) ). The criteria were established based on expert consensus, and criterion (ii) was included because the program is aimed at improving the psychosocial work environment and reducing psychosocial stress among the high-stress workers.

In the present analysis, we adopted an approach of simply summing the item score of the four-point Likert scale (1=low stress to 4=high stress) to calculate stress reaction and job stressor scores. For stress reaction and job stressor, the score ranged from 29 to 116 and 26 to 104, respectively. Cronbach's α coefficients were 0.82, 0.62, 0.87, and 0.95 for the job demands, job control, workplace support, and stress reaction scales, respectively. The proposed cutoff points were 77 for the stress reaction score for criterion (i) and 76 for the job stressor score and 63 for the stress reaction score for criterion (ii). Based on these criteria scores, 16.7% of the participants were identified as high-stress workers.

We also examined the performance of the simplified BJSQ. This simplified version was constructed from selected items from the BJSQ: six for job stressors, 11 for stress reactions, and six for buffering factors. Again, we simply summed the item score of the scales to calculate the stress reaction and job stressor scores. The scores for stress reaction and job stressor ranged from 11 to 44 and 12 to 48, respectively. Cronbach's α coefficients were 0.79, 0.76, 0.87, and 0.93 for the job demands, job control, workplace support, and stress reaction scales, respectively. The cutoff point for criterion (i) was set at 31 for the stress reaction score and the cutoff points for criterion (ii) were set at 39 for the job stressor score and 23 for the stress reaction score. Based on these criteria, 15.5% of the participants were identified as high-stress workers.

2) Psychological distress

The K6 scale, developed by Kessler et al.[4)], comprises six items measuring the extent of psychological distress with five possible responses (ranging from 0=none of the time to 4=all of the time). The Japanese translation of the K6 scale has acceptable reliability and validity[5)]. The item scores are summed to calculate a total score (range: 0-24), with higher scores indicating greater psychological distress. In the present study, Cronbach's α coefficient was 0.88 for K6. We constructed a binary variable of psychological distress, defined as K6 ≥13, which has been found to indicate severe mental illness in a Japanese community sample[6)]. Prevalence of workers with potential psychological distress was 13.0%.

3) Demographic variables

We measured sex, age group, educational attainment (>12 years or ≤12), employment status (regular or part-time), occupations, and annual household income to describe the study population.

Statistical analysis

We calculated relevant indices for screening performance. Using an external criterion of K6 ≥13, we conducted receiver operating characteristics curve analyses to estimate the sensitivity, specificity and Youden index with the cutoff points recommended by the Stress Check Program manual for BJSQ. We also evaluated the positive and negative predictive values and likelihood ratios. For reference to choose appropriate criteria in the workplace, we repeated the analyses and compared the above indices for the different cutoff points of criteria (i) and (ii). Assuming that sensitivity and specificity are of equal importance, the Youden index indicates an optimum cutoff point as a linear function of sensitivity and specificity[7)]. As for the likelihood ratios, the following interpretations were used[8)]: >10 or <0.1 indicated "probably rule in or rule out as the post-test probability of disorder largely increase or decrease," 5-10 or 0.1-0.2 indicated "often very informative as the post-test probability of disorder moderately increase or decrease," and 0.2-5 indicated "of little assistance as the post-test probability of disorder slightly increase or decrease."

IBM SPSS Statistics for Windows, Version 23 (IBM Corp., Armonk, NY, USA) was used for the statistical analyses.

The study aims and protocol were reviewed by the Research Ethics Committee of the Kitasato University Medical Ethics Organization (No. B15-132).

Results

Table 1 shows participants' demographic characteristics. Participants were distributed across the entire age range of the working population (range: 20-69 years; mean: 44; standard deviation: 14) and were in diverse occupations.

Table 1.

Demographics of participants (n=1,650)

		n	%
Sex
	Men	825	50.0
	Women	825	50.0
Age (years old)
	18-29	330	20.0
	30-39	330	20.0
	40-49	330	20.0
	50-59	330	20.0
	60-69	330	20.0
Education
	≤12 years	421	25.5
	>12 years	1,229	74.5
Employment contract
	Regular	985	59.7
	Part-time	665	40.3
Occupation
	Managers	164	9.9
	Professionals	198	12.0
	Technicians and associate professionals	101	6.1
	Clerical support workers	520	31.5
	Service and sales workers	317	19.2
	Craft and related trade workers	58	3.5
	Plant and machine operators, and assemblers	58	3.5
	Armed forces occupations	151	9.2
	Others	83	5.0
Annual household income (10,000 yen)
	≤299	315	19.1
	300-499	464	28.1
	500-799	497	30.1
	800-999	193	11.7
	1000-1499	143	8.7
	>1500	38	2.3
Prevalence of the workers with potential psychological distress defined as K6 score ≥13		215	13.0

Examination based on the original BJSQ

We estimated screening performance as follows: sensitivity, 60.5%; specificity, 88.9%; Youden index, 0.504; positive predictive value, 47.3%; negative predictive value, 93.8%; positive likelihood ratio, 6.0; and negative likelihood ratio, 0.4.

Assuming the cutoff point of criterion (i) was set at 65 for the stress reaction score and the other cutoff points were not changed, we expected the best screening performance (i.e., Youden index: 0.570). However, this approach brought increased prevalence of extracted high-stress workers from among the participants (32.3%) and decreased positive predictive value (33.0%). If we set 90 for the stress reaction score for the cutoff point of criterion (i) to extract around 10% of the participants as high-stress workers, we had to overlook >60% of the cases (sensitivity: 39.1%).

Assuming that the cutoff point of criterion (ii) was set at 55 for the job stressor score and the other cutoff points were not changed, here, too, we expected the best screening performance (i.e., Youden index: 0.549). Accordingly, 32.8% of participants were screened as high-stress workers and the positive predictive value dropped to 32.0%.

The positive likelihood ratio increases as cutoffs increase, but no cutoffs produced more than 10 for the positive likelihood ratio. (Table 2; Detailed tables are available from the authors upon request).

Table 2.

Prevalence of the high-risk group defined by Brief Job Stress Questionnaire (BJSQ), and screening performance indices for the different cutoffs of BJSQ using an external criterion of K6 ≥13# (n=1650)

Cutoff		Prevalence of the high-risk group	Sensitivity	Specificity	Youden index	Positive predictive value	Negative predictive value	Positive likelihood ratio	Negative likelihood ratio
*: Cutoffs recommended by the program manual. #: Prevalence of workers with potential psychological distress was 13.0%.
BJSQ
Stress reaction score for criteria (i)
	55	52.8	94.4	53.4	0.478	23.3	98.5	2.0	0.1
	60	41.6	87.4	65.2	0.526	27.4	97.2	2.5	0.2
	65	32.3	81.9	75.1	0.570	33.0	96.5	3.3	0.2
	70	25.1	74.4	82.3	0.567	38.6	95.6	4.2	0.3
	77*	16.7	60.5	89.9	0.504	47.3	93.8	6.0	0.4
	80	14.3	51.6	91.3	0.429	47.0	92.6	5.9	0.5
	85	11.3	44.7	93.7	0.384	51.3	91.9	7.1	0.6
	90	9.7	39.1	94.7	0.338	52.5	91.2	7.4	0.6
	95	8.5	34.4	95.3	0.297	52.5	90.7	7.3	0.7
Job stressor score for criterion (i)
	45	18.7	61.9	87.7	0.496	43.0	93.9	5.0	0.4
	50	18.4	61.4	88.1	0.495	43.6	93.8	5.2	0.4
	55	18.0	61.4	88.5	0.499	44.4	93.9	5.3	0.4
	60	17.5	61.4	89.1	0.505	45.8	93.9	5.6	0.4
	63*	16.7	60.5	89.9	0.504	47.3	93.8	6.0	0.4
	70	15.7	59.5	90.9	0.504	49.4	93.7	6.5	0.4
	75	14.9	58.1	91.6	0.497	50.8	93.6	6.9	0.5
Stress reaction score for criterion (ii)
	55	32.8	80.5	74.4	0.549	32.0	96.2	3.1	0.3
	60	29.9	77.2	77.1	0.543	33.6	95.8	3.4	0.3
	65	26.5	73.5	80.5	0.54	36.1	95.3	3.8	0.3
	70	21.0	65.1	85.6	0.507	40.5	94.2	4.5	0.4
	76*	16.7	60.5	89.9	0.504	47.3	93.8	6.0	0.4
	80	15.8	59.1	90.7	0.498	48.8	93.7	6.4	0.5
	85	14.9	57.7	91.5	0.492	50.4	93.5	6.8	0.5
	90	14.7	57.2	91.6	0.488	50.6	93.5	6.8	0.5
	95	14.7	57.2	91.7	0.489	50.8	93.5	6.9	0.5
Simplified BJSQ
Stress reaction score for criterion (i)
	20	53.6	95.3	52.7	0.48	23.2	98.7	2.0	0.1
	25	32.3	84.2	75.5	0.597	34.0	97.0	3.4	0.2
	31*	15.2	55.3	90.8	0.461	47.4	93.1	6.0	0.5
	35	10.3	41.4	94.4	0.358	52.4	91.5	7.4	0.6
	40	7.6	30.2	95.7	0.259	51.6	90.2	7.0	0.7
Job stressor score for criterion (i)
	15	17.0	56.7	88.9	0.456	43.4	93.2	5.1	0.5
	20	16.2	56.3	89.8	0.461	45.3	93.2	5.5	0.5
	23*	15.2	55.3	90.8	0.461	47.4	93.1	6.0	0.5
	30	13.3	52.6	92.6	0.452	51.6	92.9	7.1	0.5
Stress reaction score for criterion (ii)
	25	37.9	87	69.5	0.565	29.9	97.3	2.9	0.2
	30	31.3	78.1	75.7	0.538	32.6	95.9	3.2	0.3
	35	21.0	63.7	85.4	0.491	39.5	94.0	4.4	0.4
	39*	15.2	55.3	90.8	0.461	47.4	93.1	6.0	0.5
	45	13.3	52.6	92.6	0.452	51.6	92.9	7.1	0.5

Examination based on simplified BJSQ

In total, 55.3% of the cases were included as high-stress as per the program manual's definition. In this case, we estimated screening performance as follows: sensitivity, 55.3%; specificity, 90.8%; Youden index, 0.461; positive predictive value, 47.4%; negative predictive value, 93.1%; positive likelihood ratio, 6.0; and negative likelihood ratio, 0.5. We also found lower sensitivity compared to that of the original version by 5%.

Assuming that the cutoff point of criterion (i) was set at 25 for the stress reaction score and the other cutoff points were not changed, we expected almost the same screening performance as the best scenario using the original version (i.e., Youden index: 0.597). In this scenario, prevalence of high-stress workers would increase to 32.3% of the participants and the positive predictive value would decrease to 34.0%.

Discussion

Usage of BJSQ with the cutoff points recommended by the Stress Check Program manual will moderately increase post-test probability of potential psychological distress. The optimal screening performance based on Youden index will be achieved by lowering the cutoff point of either the job stressor or stress reaction score by 10 points; this procedure will screen almost one-third of the participants and practitioners may apply the BJSQ and adjust the cutoff in their workplaces considering their capacity. Almost the same level of screening performance could be expected for the simplified version as for the original version, but the lower sensitivity for the simplified compared with the original version should be taken into consideration.

Workers with a K6 score ≥13 can be justified in being referred to mental health experts, as the cutoff point for K6 was found to be useful if a screening program targets severe mental illness, has limited resources for secondary screening (such as the labor of health care professionals), and thus seems to indicate a high post-probability of the disorders in the positives[6)]. However, K6 is also a screening tool and has errors. Caution should be taken in interpretation of the findings, as we could not adopt a concrete outcome, such as a physician's diagnosis of a specific disorder. The associations between self-reported symptoms and measures of the BJSQ were probably inflated because of the common method variance[9)] or a possible bias owing to a tendency toward negative affectivity[10)]. Therefore, the screening accuracy of our findings should be interpreted more conservatively. Generalizability of the findings is limited because of the online survey; however, the anonymity provided may have incentivized the participants to answer honestly.

Conclusion

Screening of "high-stress" workers as per the recommendation of the Stress Check Program manual is informative. However, based on our findings using psychological distress defined by K6 as the target condition, practitioners should be aware that less than half of those defined as high-stress workers warrant consideration for secondary screening for psychological distress.

Acknowledgments: The present study was supported by the Health and Labor Sciences Research Grant 2015-2017 (H27-Rodo-Ippan-004) from the Ministry of Health, Labour and Welfare, Japan.

Conflicts of interest: None declared.