Validation study of a distress screener.

OBJECTIVES: A 3-item screening instrument called the Distress Screener was developed for early identification of distress among employees on sick leave. The Distress Screener consists of three items obtained from the distress subscale of the four-dimensional symptom questionnaire (4DSQ). This study assessed an optimal cut-off point and validated the Distress Screener by relating it to the 4DSQ and to medical diagnoses.
METHODS: 171 sick-listed employees filled in the Distress Screener and the 4DSQ (containing four subscales: distress, depression, anxiety and somatisation) and medical diagnoses were obtained from occupational physicians (OPs). The optimal cut-off point was assessed by computing sensitivity and specificity values. Validity was assessed by relating the Distress Screener score to the scores on 4DSQ subscales. In addition scores were compared to mental health medical diagnoses and the degree of similarity between two repeated measurements was obtained.
RESULTS: Using the 4DSQ distress score >10 as reference standard, the optimal cut-off point of the Distress Screener was > or =4. Regarding validity, a high correlation (0.82) existed between the Distress Screener and the 4DSQ distress subscale and it was significantly different from the correlations with the other 4DSQ subscales. Also a high correlation existed for the test-retest reliability (0.83). Furthermore, a high score on the Distress Screener seemed to be related to the medical diagnosis 'Stress-related complaints'. All low scores seemed to be related to the medical diagnosis 'Other complaints'. Sensitivity (0.85) and specificity (0.78) values, and positive and negative predictive values of the screener were comparable to those of the 4DSQ distress subscale.
CONCLUSIONS: The Distress Screener is a valid instrument for use by the OP during consulting time as a quick scan for early identification of distress in employees on sick leave. The cut-off point > or =4 is useful for early identification of distress in employees on sick leave.

Introduction

The prevalence of common mental disorders (CMDs) in employees on sick leave is reported to be over 30% in the Netherlands in 2007. This prevalence is high compared with other countries [12, 13, 19, 20]. Adjustment disorders account for most psychopathology giving rise to inability to work in the Netherlands, whereas psychiatric illnesses, such as major depression, anxiety disorders, psychoses, and personality disorders, account for only a small minority of cases [21, 22]. The societal and financial costs of disfunctioning in terms of (long term) sickness absence due to CMDs are extensive [2, 8]. With the increase of mental workload over the past decades, the fraction of psychological problems related to occupational stress has increased rapidly [4]. Dutch studies revealed that about 20% of patients with CMDs stayed on sick leave for more than 1 year [3, 14]. Within the context of Dutch occupational health care, distress is commonly experienced by employees and frequently related to sickness absence [9]. In case of distress-related sickness absence early identification is important to prevent long-term sick leave and to enable early interventions, because it is known that long-lasting work disability reduces the chance of return to work [7].

Terluin [16] developed the four-dimensional symptom questionnaire (4DSQ) which is a self-report questionnaire of 50 items that measures non-specific general distress, depression, anxiety and somatisation. The 4DSQ is incorporated in the guideline for psychological problems of the Dutch association of occupational medicine and is frequently applied by occupational physicians (OPs) among employees on sick leave. A disadvantage of the 4DSQ is the relatively long time that it takes to be filled in (5–10 min). In that way precious time is lost during a consultation. Consequently, occupational physicians often request their patients to fill it in at home and hand it in at the next consultation. So, there is a need for a short questionnaire, which identifies distress in an early stage of sick leave. In this study a 3-item version of the 4DSQ distress subscale was developed and tested. The purpose of this study was to assess an optimal cut-off point and to validate this so-called Distress Screener.

Methods

Research Population

The research population consisted of employees on sick leave of three Dutch companies: an academic hospital, an university and a steel company. A diversity of job functions were included in this study. The specific inclusion criteria were: full or part-time on sick leave, duration of sick leave shorter than 8 weeks and no period of sick leave with the same reason within 1 month before the current episode to select only incident cases of work disability.

Recruitment Study Population

All employees who were on sick leave for more than 1 week received the Distress Screener together with an explanatory letter from the OP. The respondents who met the inclusion criteria and were willing to participate, were divided in two groups according to the score of the Distress Screener with cut-off point four. This cut-off point was based on a required level of specificity, calculated in a group of non-selected primary care patients (N = 2127) and was used for the intervention study: the ADAPT study. The research population in the current paper is based in part on the population of the ADAPT study. The ADAPT study is a randomized controlled trial evaluating the cost-effectiveness of a participatory workplace intervention compared with usual care for sick-listed employees with distress [23]. The workplace intervention is a stepwise approach in which an employee and supervisor identify and prioritize obstacles and solutions for a return to work guided by a mediator. The intervention is aimed to reach consensus between a sick-listed employee and his or her supervisor about a plan for return to work. In the period between April 2006 and May 2007, respondents who were screened positive according to the Distress Screener were recruited for the ADAPT study [23]. In addition, a sample of non-distressed (screened negative) employees on sick leave were recruited from the same three companies in the period January till May 2007. Permission was obtained from the Medical Ethics committee and all respondents provided informed consent.

Within one till 2 weeks after filling in the Distress Screener, the respondents filled in the 4DSQ. In addition, data of OPs diagnoses were obtained from the medical file of each employee. These diagnoses were a proportional breakdown of the types of health problems that were typical for this population and which led to extended sickness absence. OPs in the Netherlands classify diagnoses according to the international classification of diseases (CAS) [10] which is based on the ICD-10.

Measurement Instruments

The Distress Screener is a short questionnaire which comprises three items of the 4DSQ distress subscale: “During the past week, did you suffer from worry?”, “During the past week, did you suffer from listlessness?” and “During the past week, did you feel tense?”. The selection of items was made a priori from a dataset consisting of 2,127 primary care patients. The three items were chosen based on their factor loadings and ‘difficulties’, so as to maximize the discrimination between subjects with 4DSQ distress-scores ≤10 and subjects with 4DSQ distress-scores >10. The response scale contains three options: “no” (0), “sometimes” (1), and “regularly or more often” (2). A total score was constructed by summing up the answers on the three items. The cut-off point that discriminates between ‘screened positive’ and ‘screened negative’ was established on a score of 4 or higher. A positive score means that the person involved is scored as distressed according to the Distress Screener.

The 4DSQ is a questionnaire comprising 50 items distributed over four scales: the distress scale contains 16 items (range 0–32), the depression scale contains 6 items (range 0–12), the anxiety scale contains 12 items (range 0–24) and the somatisation scale contains 16 items (range 0–32) [16]. The reference period is “the past week”. The response scale contains five categories: “no” (0), “sometimes” (1), “regularly” (2), “often” (2), “very often or constantly” (2). The item scores are summated to scale scores. Discrimination between ‘cases’ and ‘non-cases’ were established for distress score >10, somatisation score >10, depression score >2, and anxiety score >7 [17, 18, 24]. Measuring distress with the 4DSQ has been shown to be a valid and reliable measurement [17]. According to Van Rhenen et al. [24] the distress score of >10 is appropriate for use in studies of distress in working populations and therefore used as reference standard in this study.

OPs diagnoses [CAS-codes based on the ICD-10 and developed by the Dutch Association of Occupational Medicine (NVAB) and the Employed Persons’ Insurance Administration Agency (UWV)] were classified into six categories of CMDs. Four categories of diagnoses were related to the subscales of the 4DSQ, ‘Distress’ (‘Stress-related complaints’), ‘Somatisation’, ‘Depression’ and ‘Anxiety’, plus the categories of ‘Other psychological complaints’ and ‘Other complaints’ (see Table 1).

Table 1

Diagnoses classified in six categories

Categories	Classified diagnoses
Stress-related complaints	Tension, irritated, anger, nervousness, other adaptation disorders, surmenage and burnout
Depression	Depressive episode, depressive personality syndrome and neurotic depression
Anxiety	Panic syndrome, generalised anxiety, agoraphobia, social phobia, other anxiety syndromes and post-traumatic stress syndrome
Somatisation	Somatisation, persistent somatoform pain syndrome, other somatoform syndromes, carcinophobia and nosophobia
Other psychological complaints	Psychosis, other psychiatric syndromes and personality syndromes
Other complaints	Symptoms which were not classified elsewhere; dermatology, bone and muscle system, respiratory system, urogenital system, cardiac system, nervous system, eye system

Analyses

In order to adjust for the stratified sampling procedure, we used weight factors for all analyses (0.666 for screen-positive and 1.308 for screen-negative employees) to ensure that the population reflected the composition of the source population of incident sick-leave cases.

Using the 4DSQ distress score >10 as reference standard, the receiver operator characteristic (ROC) curve of the Distress Screener was obtained for a range of seven cut-off values (0–6). Sensitivity and specificity were calculated for each cut-off value. With regard to this study, identification of distress by the Distress Screener in an early stage of sick leave, it was important to select a high number of true-positively classified distressed persons and restrict the number of false-positively classified healthy persons. Therefore, determination of the optimal cut-off point was based on a high sensitivity value with the most appropriate specificity value. To examine the validity of the Distress Screener the Pearson correlation coefficients were calculated between the total score of the Distress Screener and the total score of each 4DSQ subscale. We compared correlated correlation coefficients, according to the method described by Meng et al. [11], including a Bonferroni correction. The degree of similarity between two repeated measurements, the test–retest (12-day) reliability, was obtained by computing the Pearson correlation coefficient of the total score of the three items of the Distress screener with the total score of the same three items of the 4DSQ distress subscale.

Furthermore, the validity was examined by comparing the outcomes of the Distress Screener (screened negative and screened positive) with OPs diagnoses (categorized CAS-codes). Both relations of the Distress Screener with OPs diagnoses and the 4DSQ distress subscale with OPs diagnoses were compared. Sensitivity and specificity values and positive and negative predictive values were determined from the established outcomes. Outcomes of both the Distress Screener and the 4DSQ distress subscale were dichotomised.

Results

Subjects

A total of 5,845 Distress Screeners were sent to employees on sick leave (see Fig. 1). About 2,562 screeners came back, in which 1,620 employees were still sick-listed and 942 resumed work. Respectively, 517 employees screened positive and 328 employees screened negative were divided into two subgroups based on cut-off point 4 of the Distress Screener. Eventually, 171 employees (resp. 82 and 89) met the inclusion criteria and gave their informed consent. Causes of dropping out were: employees did not meet the inclusion criteria (resp. 31 and 3), employees were not sick-listed anymore (resp. 162 and 51), and employees were eventually not willing to participate (resp. 112 and 61). From 74 screened positive employees (90%) and 62 screened negative employees (70%) the first diagnosis in the medical file was available. The remaining 35 employees were not consulted by the OP and therefore medical diagnoses were not available. On average there were 12 days between filling in the Distress Screener and the 4DSQ (SD = 7.42; median = 10).

Fig. 1

Flow diagram of sick-listed employees including periods of data collection and sample characteristics

Optimal Cut-Off Point

Table 2 shows the weighted sensitivity and specificity values of each cut-off point. From the ROC-curve the score four or higher was derived as the optimal cut-off point of the Distress Screener because, it had the highest sensitivity value with the most appropriate specificity value. The area under the curve (AUC) had a value of 0.762 (95% CI 0.639–0.885).

Table 2

Weighted sensitivity and specificity values over each cut-off point of the Distress Screener (N = 136)

Cut-off point	Sensitivity	Specificity
6	0.463	0.866
≥5	0.721	0.809
≥4	0.798	0.726
≥3	0.899	0.601
≥2	0.949	0.426
≥1	1.000	0.300
0	1.000	0.000

Note: Cut-off point 4 is most optimal

Correlations Between Distress Screener and 4DSQ Subscales

Table 3 shows correlations between the Distress Screener and the four subscales of the 4DSQ. The Distress Screener and the 4DSQ distress subscale had a high correlation of 0.82. The Distress Screener correlated higher with the 4DSQ distress subscale compared with the other 4DSQ subscales. Furthermore, this corrected correlated correlation coefficient was significantly different from the three correlations of the Distress Screener with the other 4DSQ subscales. The test–retest (12-day) reliability had a high correlation of 0.83.

Table 3

Weighted Pearson correlation coefficients between the distress screener and the 4DSQ subscales

	Distress (N = 170)	Somatisation (N = 167)	Anxiety (N = 168)	Depression (N = 170)
Distress screener	0.82*	0.51	0.55	0.60

* P < 0.000

Comparison of Distress Score and OPs Diagnoses

Table 4 presents the prevalence of distress according to the Distress Screener and the prevalence of distress according to the 4DSQ distress subscale compared to the medical diagnoses. ‘Anxiety’ and ‘Somatisation’ were not diagnosed by the OPs and therefore not presented in Table 4. When comparing OPs diagnoses with positive and negative Distress Screener scores it can be seen that 21 out of 26 employees (80.8%) with the diagnosis ‘Stress-related complaints’, were screened positive. All employees with the diagnosis ‘Depression’ or ‘Other psychological complaints’, scored positive on the Distress Screener. Furthermore, 76 out of 98 cases (77.6%) with OPs diagnosis ‘Other complaints’ scored negative on the Distress Screener. This resulted in a sensitivity value of the Distress Screener for detecting any psychological problem as diagnosed by the OPs of 0.85 (28/28 + 5) and a specificity value of 0.78 (76/76 + 22). The positive predictive value (PPV) and negative predictive value (NPV) of the Distress Screener were respectively, 0.56 (28/50) and 0.94 (76/81).

Table 4

Weighted and rounded Distress Screener scores and 4DSQ-distress scores compared to OPs diagnoses (N = 136)

		OPs diagnoses
		Stress related complaints (%)	Depression (%)	Other psychological complaints (%)	Other complaints (%)
Distress screener	Positive	21 (80.8)	5 (100)	2 (100)	22 (22.4)
(cut-off ≥4)	Negative	5 (19.2)	0	0	76 (77.6)
4DSQ-distress score	Positive	20 (80)	5 (100)	2 (100)	28 (28.9)
(cut-off ≥11)	Negative	5 (20)	0	0	69 (71.1)

When comparing OPs diagnoses with the 4DSQ distress subscale scores it can be seen that 20 out of 25 employees (80%) who were diagnosed as having ‘Stress related complaints’ had a positive score (i.e. distressed) on the 4DSQ distress subscale. All employees with the diagnosis ‘Depression’ and ‘Other psychological complaints’ had positive scores (i.e. distressed) and 69 out of 97 cases (71.1%) with the diagnosis ‘Other complaints’ had a negative score (i.e. non-distressed). This resulted in a sensitivity value of 0.84 (27/27 + 5) and a specificity value of 0.71 (69/69 + 28) of the 4DSQ distress subscale. The PPV and NPV of the 4DSQ distress subscale were respectively, 0.49 (27/55) and 0.93 (69/74).

Discussion

Main Findings

The present study examined the validity and the optimal cut-off point of the Distress Screener. According to the results cut-off point ≥4 is most optimal. With regard to early identification of distress in employees on sick leave it is important to select a high number of true-positively classified distressed persons. Thus, in this study and the ADAPT study the optimal cut-off point was correctly chosen.

The results confirmed a high validity of the Distress Screener. In line with our expectations the Distress Screener was only significantly related to the 4DSQ distress subscale and not to the other three 4DSQ subscales. Furthermore the results indicated a high validity of the Distress Screener and the 4DSQ distress subscale when comparing them to OPs diagnoses. Both the Distress Screener and the 4DSQ distress subscale showed high agreement with OPs diagnosis ‘Stress-related complaints’ and other psychological problems as assessed by the OPs. The sensitivity and specificity, and the PPV and NPV of the Distress Screener were comparable to those of the 4DSQ distress subscale.

Explanation of Results

As only five employees were diagnosed with ‘Depression’ by the OPs, it is not possible to draw up conclusions of outcomes of the positive score of the Distress Screener and the 4DSQ distress subscale with OPs diagnosis ‘Depression’. Therefore, it should be cautioned to interpret these results. Literature suggests that psychological distress overlaps with various symptoms of depression and burnout, and contains psycho-physiological and behavioural symptoms that are not specific to a given pathology [6]. In line with the nature of distress as the general, most basic dimension of psychopathology, it is to be expected that distress symptoms also are present when depression is diagnosed.

The relation between OPs diagnosis ‘Other complaints’ and distress can be explained in several ways. First, it is reasonable that the OPs did not recognize the symptoms of distress in some cases. And secondly, it is reasonable that OPs were more focused on diagnosing the somatic complaint, distress was not the main diagnosis at that moment.

‘Anxiety’ and ‘Somatisation’ were not diagnosed by the OPs. It is imaginable that symptoms of anxiety and somatisation are more difficult to recognize and will be recorded often as stress-related mental disorders. Literature confirms that anxiety disorders encompass a combination of distress and depression, or distress and anxiety symptoms [5], along with a variable degree of somatisation symptoms [15]. Furthermore, symptoms like back pain, neck pain and painful muscles were quite often diagnosed according to the medical files. We classified these diagnoses into the category of ‘Other complaints’. It is plausible that these complaints were not immediately recognised by the OPs as somatisation symptoms and were recorded as physical problems.

The PPVs of both the Distress Screener and the 4DSQ distress subscale were relatively low values. These low PPVs were directly proportional to the low prevalence percentages of 25% of ‘any psychological problem’ diagnosed by the OP. ‘Any psychological problem’ was compared with the positive and negative scores of both the Distress Screener and the 4DSQ distress subscale (33/(98 + 33) and 32/(97 + 32)). Furthermore, it is plausible that the OPs overlooked distress and other psychological problems (the employee does not mention it). The PPV also decreases through these false-positives, which becomes not false-positive by a false (positive) outcome of the Distress Screener, but through a false (negative) outcome of the reference standard (the OP).

Study Limitations

The following limitations in this study were present. First, the Dutch registration system of CAS codes is focused on reporting one diagnosis of sickness absence by the OP. This diagnosis is directly linked with the reason of sick leave of the employee. This may explain high numbers of the diagnosis ‘Other complaints’. The second limitation of this study was the average of 12 days between completing the Distress Screener and the 4DSQ. Within these 12 days the psychological state of a person can change, especially in this early stage of sick leave. The longer the time between completing the Distress Screener and the 4DSQ, the higher the chance that the situation or psychological state changes. Distress symptoms can decrease over time or the cause of distress can disappear. Third, selection bias might have occurred while in the screened positive group eight participants and in the screened negative group 27 participants were not diagnosed by the OPs. Probably, participants without a diagnosis recovered and returned to work so quickly, that the OP did not have the opportunity to establish a diagnosis. Within the current analyses of the data these participants were classified as ‘missing’. Fourth, It is debatable to use OPs diagnoses as external criterion as there are differences between diagnoses of the OP/General Practitioner and outcomes of validated questionnaires [1]. Alternatives were the 4DSQ or other validated questionnaires. In the guideline for psychological problems applied by OPs, the 4DSQ is also one of the diagnostic tools. Finally, generalizing the results of this study should be cautioned as only three companies were involved in this study. Moreover, the population was limited to a primary occupational healthcare population of employees on sick leave.

Implications for Practice and Research

The Distress Screener can be used by the OP during consulting time as a quick scan for the early identification of distress in employees on sick leave. In other words, the Distress Screener can be used by the OP for detecting employees on sick leave who need additional attention for their emotional state, whatever the reason of sick leave might be. Filling in the Distress Screener takes considerably less time than filling in the 4DSQ distress subscale and is at the same time as valid as the 4DSQ distress subscale. When physical complaints are present the Distress Screener is able to identify quickly additional distress. But, when using the Distress Screener only, lack of relevant information can occur. Considering obtaining a diagnosis, application of treatments and additional costs, the OPs should decide to fill in the whole 4DSQ (by the employee) during the first consultation if an employee scores four or higher on the Distress Screener. It is relevant and valid for OPs to screen during consulting time. When a psychological problem emerged through screening by the 4DSQ, this will be discussed with the employee. If the employee agrees with the recommended actions the guideline for psychological problems of the Dutch association of occupational medicine will be applied by the OP.

The Distress Screener can also be used for research purposes. The optimal cut-off point should then be adjusted to the objectives of the research. At last, further research is needed within the working population, wherein the Distress Screener can be tested as a tool for predicting absenteeism. However, screening of populations should be performed carefully and an appropriate intervention should be available to justify screening.