The validity of the distress thermometer in prostate cancer populations.

BACKGROUND: The Distress Thermometer (DT) is widely recommended for screening for distress after cancer. However, the validity of the DT in men with prostate cancer and over differing time points from diagnosis has not been well examined.
METHOD: Receiver operating characteristics analyses were used to evaluate the diagnostic accuracy of the DT compared with three commonly used standardised scales in two prospective and one cross-sectional survey of men with prostate cancer (n = 740, 189 and 463, respectively). Comparison scales included the Impact of Event Scale - Revised (IES-R, Study 1), the Hospital Anxiety and Depression Scale (HADS, Study 2) and the Brief Symptom Inventory-18 (BSI-18, Study 3).
RESULTS: Study 1: the DT showed good accuracy against the IES-R at all time points (area under curves (AUCs) ranging from 0.84 to 0.88) and sensitivity was high (>85%). Study 2: the DT performed well against both the anxiety and depression subscales for HADS at baseline (AUC = 0.84 and 0.82, respectively), but sensitivity decreased substantially after 12 months. Study 3: validity was high for the anxiety (AUC = 0.90, sensitivity = 90%) and depression (AUC = 0.85, sensitivity = 74%) subscales of the BSI-18 but was poorer for somatization (AUC = 0.67, sensitivity = 52%). A DT cut-off between ≥3 and ≥6 maximised sensitivity and specificity across analyses.
CONCLUSIONS: The DT is a valid tool to detect cancer-specific distress, anxiety and depression among prostate cancer patients, particularly close to diagnosis. A cut-off of ≥4 may be optimal soon after diagnosis, and for longer-term assessments, ≥3 was supported.

Introduction

Screening cancer populations to detect heightened psychological distress is now well accepted as a desired component of good cancer care 1. This approach aims to efficiently identify those patients in need of in-depth psychological care in order to direct services to where they are most likely to be effective 2. In choosing a screening measure, however, there are trade-offs to be made between cost, brevity and ease of administration and scoring, and most importantly, the effectiveness of the scale in detecting distress with regard to scale specificity and sensitivity. The Distress Thermometer (DT) presents as a single-item distress screening scale that has acceptable validity compared with longer measures 3 and on the basis of brevity is well placed for implementation in practice settings. This scale has been found to perform well in a range of languages 4–8 and across a number of cancer types, for example, breast 9, blood 10, thyroid 11 and colorectal 12 cancer, and in community as well as research settings 13. However, the performance of this scale in men with prostate cancer has not been well evaluated, which is surprising given the high prevalence of this cancer in many countries 14 and problematic given that men may express mental health problems differently than do women 15.

Specifically, traditional notions of masculinity that emphasise the values of being autonomous 16 and unemotional (i.e. ‘boys don't cry’) 17 may lead many men to be reluctant to express emotion or distress, compared with women 18. Consistent with this, it has been proposed that common diagnostic criteria to identify psychological disorders may not be sensitive to men's unique expression of emotion in relation to their mental health 19–23. This issue is then further complicated by findings that the trajectories of adjustment for men with prostate cancer over time is heterogeneous 24, as is their response to intervention. This means that accurately detecting distress in this patient group at both diagnosis and across time is crucial, increasing the importance of having a clear understanding of the performance of the DT in this context.

In this regard, Roth et al. administered the DT to 121 men with prostate cancer and found the scale to be largely acceptable to men with prostate cancer (77% evaluable questionnaires) with the scale performing satisfactorily by comparison with the Hospital Anxiety and Depression Scale (HADS 25) when applying a DT cut-off score of ≥5 26. This study, however, did not specifically assess sensitivity and specificity, and the sample included mostly men with advanced disease. More recently, Lotfi-Jam et al. 27 administered the DT to 332 men with prostate cancer commencing radiation therapy and found a DT cut-off score of ≥4 identified almost all men with HADS scores indicative of anxious or depressive symptomatology. The scale was found to be highly acceptable, although the prevalence of distress was low. Again, sensitivity and specificity were not evaluated. Importantly, according to Lotfi-Jam et al., the DT was able to identify a group of men who were not distressed according to the HADS, but who had unmet supportive care needs and prostate cancer-specific quality of life decrements.

In summary, to date, most DT validation studies have not included men with prostate cancer and have largely applied measures of anxiety and depression as the comparator. In addition, the extent to which the DT is effective along different phases of the illness trajectory is not yet clear. Merport et al. 28 recently suggested that for long-term survivors, the DT was not valid with poor sensitivity as compared with the Brief Symptom Inventory-18 (BSI-18 29), on the basis of the data from a mixed sample of cancer patients who were at least 2 years post-diagnosis (median of 9 years). These authors subsequently argued against its use in the survivor population. We suggest that further data from a less heterogeneous sample are needed to draw such a conclusion.

Accordingly, the present research has three aims. First, we sought to assess the sensitivity and specificity of the DT in prostate cancer patients. Second, we compare the performance of the DT in this patient group for cancer-specific distress and also in relation to two gold standard measures for anxiety and depression. Third, we assess validity of the DT across time in a prospective setting to ascertain how the measure performs in the longer term.

Methods

Participants

Study 1

These data were from a longitudinal study that included 740 (81.7% from total eligible) men diagnosed with prostate cancer in the geographic catchment areas of South-East and North Queensland, Australia. The sample and methods are described in detail elsewhere 30. In brief, participants were on average 63.4 years of age (SD = 7.5 years; range = 43.3–83.6 years) at diagnosis; most were in a relationship (83.9%); over half (52.1%) had tertiary or trade qualifications; approximately equal numbers were employed (43.6%) versus retired (45.8%); and 40% of participants reported a gross annual household income of <$40 000, whereas nearly 25% grossed >$80 000 per annum. At recruitment to the study, the participants' mean time since diagnosis was 25.6 days (SD = 26.9), and their median time since diagnosis was 19 days. In Study 1, participants completed the DT 31 and the Impact of Event Scale – Revised (IES-R) 32,33 at baseline, 12-month (91% response) and 36-month (88% response) follow-up.

Study 2

Participants were 189 (46.9% from total eligible) Queensland men with localised prostate cancer who had chosen or had undergone surgical treatment. The study data were originally from a randomised controlled trial of a couples-based sexuality intervention for prostate cancer patients. The sample is described in detail elsewhere 34,35. In brief, the mean age of the participants in the sample was 62.6 years (SD = 6.8), and approximately two-thirds (65%) of the participants had completed university, college, or a trade or technical certificate. Approximately 28% of participants reported a gross annual household income of <$40 000 per annum, whereas about 26% fell in the $40 000–$80 000 bracket and nearly 40% reported earning more than $80 000 per annum. At recruitment to the study, the patients' mean time since diagnosis was 115.2 days (SD = 110.5); 25% had already undergone surgery for their prostate cancer and their median time since surgery was 77 days (SD = 109.9). In Study 2, participants completed the DT 31 and the HADS 25 at baseline and 12-month (84% response) follow-up.

Study 3

This sample was from a randomised controlled trial comparing usual care to a multimodal supportive care intervention for men recently diagnosed with localised prostate cancer in Queensland, which is described elsewhere 36. Participants were 463 (69.9% from total eligible) men diagnosed with localised prostate cancer between September 2011 and November 2012 who were recruited through the Queensland Cancer Registry, a population-based register of cancer incidence. The mean age of participants was 64.4 years (SD = 7.6), and on average, they were 10.8 months post-diagnosis (SD = 3.0, range 0.6–21.5 months). Approximately 80% of the samples were married, 62% had completed university/college or had a trade/technical certification, and 45% were retired with a further 34% employed full-time. Thirty-four per cent of the sample's gross annual household income was <$40 000, 27% fell in the $40 00 to $80 000 bracket and 36% grossed >$80 00 per annum. In Study 3, participants completed the DT 31 and the BSI-18 29 at baseline.

Measures

Distress thermometer

The DT is widely used as a screening measure of global psychological distress 31 and was used to assess the current level of psychological distress experienced by prostate cancer patients across each of the three studies described in turn. Participants are asked how distressed they feel on a single 11-point scale, ranging from 0 (no distress) to 10 (extreme distress).

Impact of Event Scale – Revised

The IES-R assesses cancer-specific distress and contains three subscales: intrusion, avoidance and hyperarousal 32,33. A cut-off of >33 out of a total possible score of 88 has been recommended to provide diagnostic information for high distress 37. In the current study, the alpha coefficient showed excellent internal reliability for the total scale score across the time points (α = 0.94–0.95).

Hospital Anxiety and Depression Scale

The HADS 25 includes two subscales: Anxiety (HADS-A) and Depression (HADS-D). We used a cut-off of ≥8 to indicate at least mild distress on either of the subscales, as well as a total HADS scores (HADS-T) ≥15 to indicate clinically significant distress. Cronbach's alpha for HADS-A, HADS-D and HADS-T was high at baseline and 12 months (α = 0.80–0.91).

Brief Symptom Inventory-18

The BSI-18 29 assesses current psychological distress at baseline with three subscales: anxiety (α = 0.75), depression (α = 0.85) and somatisation (α = 0.63), as well as a global severity index (GSI) score (α = 0.87). Raw scores for the three scales and the GSI are converted to t-scores. The recommended case rules to identify respondents with clinically significant symptom elevations is a GSI t-score ≥ 63, or if any two subscale t-scores are 63 or higher, according to the BSI-18 scoring manual. However, other studies using cancer patient samples have adopted varying cut-offs 15,38,39. For this study, we examined two classification methods: t-score ≥ 63 and t-score ≥ 57 as an alternative case-rule 38.

Statistical analyses

Frequencies, percentages, means and SDs were calculated to describe each of the study samples in terms of socio-demographic characteristics and psychological distress. Receiver operating characteristics (ROC) analysis was then used to evaluate the diagnostic accuracy of the DT to detect cases identified by the IES-R (Study 1), HADS-A, HADS-D and HADS-T (Study 2), and the BSI-18 (Study 3). This involves plotting the fraction of true positives (sensitivity) against the fraction of false positives (1 – specificity) at various threshold settings for the DT. The area under the ROC curve (AUC) quantifies the ability of the scale to discriminate between participants with and without clinical distress and was expressed along with a 95% confidence interval (95% CI). An AUC of 1 represents perfect agreement between the two scales, whereas a value of 0.5 represents a test with no apparent accuracy relative to the established criterion. Optimal DT thresholds were selected according to the Youden Index 40, which is based on the vertical distance from the line of equality to the ROC curve. Using this method, the cut-off is selected as the point on the ROC curve that is farthest from chance, corresponding to the maximum value of (sensitivity + specificity − 1). In addition to the AUC, sensitivity, specificity, the percentage correctly classified along with the positive and negative likelihood ratios were also presented. Chi-squared tests were used to assess differences in the ROC analyses where applicable.

Results

Study 1

Psychological distress

Participants rated their distress level on the DT at baseline (M = 3.1, SD = 2.5), 12 months (M = 2.3, SD = 2.3) and 36 months (M = 2.3, SD = 2.2). The percentage of participants scoring above the IES-R cut-off (≥34) decreased as the participants moved further away from their diagnosis 24,35. At baseline, 10.0% of the participants were classified as distressed, compared with 2.0% at 12 months and 2.5% at 36 months.

Diagnostic accuracy of the DT

At baseline, the AUC was 0.87 (95% CI = 0.83–0.91) with an optimal DT cut-off of ≥5 (Table 1 and Figure 1). Over time, the optimal DT cut-off decreased to ≥4 at 12 months and ≥3 at 36 months, whereas the corresponding AUCs remained high at 0.88 and 0.84, respectively. Across the three time points, sensitivity for the optimal DT was high (ranging from 85.7% to 92.9%) and the specificity was acceptable (ranging from 67.0% to 77.5%). There were no significant differences in the results between men aged <65 years and 65 years and older at baseline (χ2 = 0.43, df = 1, p = 0.512), 12 months (χ2 = 2.12, df = 1, p = 0.145) or 36 months (χ2 = 0.07, df = 1, p = 0.785).

Table 1

Summary of results from the area under the ROC curve analyses for the distress thermometer

Comparative scalea	Cases/cohort (n)	Area under ROC curve (95% CI)	Optimum cutpoint for DTb	Sensitivity (%)	Specificity (%)	Correctly classified (%)	Positive likelihood ratioc	Negative likelihood ratiod
Impact of Event Scale – Revised
Total at baseline	72/722	0.87 (0.83–0.91)	≥5	86.1	74.9	76.0	3.43	0.19
Total at 1 year	14/637	0.88 (0.81–0.94)	≥4	85.7	77.5	77.7	3.81	0.18
Total at 3 years	14/557	0.84 (0.75–0.94)	≥3	92.9	67.0	67.7	2.82	0.11
Hospital Anxiety and Depression Scale
Total at baseline	45/187	0.83 (0.77–0.90)	≥3	71.1	78.2	76.5	3.26	0.37
Total at 1 year	29/159	0.80 (0.70–0.89)	≥4	48.3	94.6	86.2	8.97	0.55
Anxiety subscale at baseline	44/187	0.84 (0.78–0.90)	≥3	72.7	78.3	77.0	3.35	0.35
Anxiety subscale at 1 year	23/159	0.83 (0.73–0.93)	≥6	56.5	97.8	91.8	25.62	0.44
Depression subscale at baseline	11/188	0.82 (0.68–0.97)	≥4	72.7	81.9	81.4	4.02	0.33
Depression subscale at 1 year	17/159	0.73 (0.58–0.87)	≥4	47.1	90.9	86.2	5.14	0.58
Brief Symptom Inventory 18
Total at baseline	19/463	0.84 (0.74–0.95)	≥5	79.0	84.9	84.7	5.23	0.25
Anxiety subscale at baseline	19/463	0.90 (0.81–0.98)	≥5	89.5	85.4	85.5	6.11	0.12
Depression subscale at baseline	23/463	0.85 (0.77–0.94)	≥5	73.9	85.2	84.7	5.00	0.31
Somatization subscale at baseline	23/463	0.67 (0.54–0.81)	≥5	52.2	84.1	82.5	3.28	0.57

95% CI, 95% confidence interval; DT, distress thermometer; ROC, receiver operating characteristics.

Caseness was defined as ≥34 for Impact of Event Scale – Revised, ≥8 on either the anxiety or depression subscale for Hospital Anxiety and Depression Scale, and t ≥ 63 for Brief Symptom Inventory 18.

The optimal cut-off for the DT is given by the maximum value of the Youden index which measures the vertical distance from the line of equality to the ROC curve.

Positive likelihood ratio = Sensitivity/(1 – Specificity).

Negative likelihood ratio = (1 – Sensitivity)/Specificity.

Figure 1

Receiver operating characteristics curves for the Distress Thermometer versus the Impact of Event Scale – Revised, at baseline, 1 and 3 years after diagnosis

Study 2

At baseline, the average DT score was 2.2 (SD = 2.2) and decreased to 1.6 (SD = 2.2) at 12 months. For the HADS subscales, about a quarter (23.0%) of the participants at baseline and 14.0% at 12 months scored above the anxiety cut-off, whereas 6.0% at baseline and 11.0% at 12 months were identified as reaching caseness for depression. At baseline, 10.1% of the participants reached the HADS-T clinical cut-off of ≥15, with a similar proportion (10.7%) identified as clinically distressed at the 12-month follow-up 35.

When comparing the DT to HADS-T at baseline, the AUC was 0.83 (95% CI = 0.77–0.90) with an optimal DT cut-off of ≥3 and remained fairly stable after 12 months at 0.80, whereas the threshold increased to ≥4 (Table 1 and Figure 2). Similar results to HADS-T were observed for both HADS-A and HADS-D at baseline. However, at 12 months, the optimal DT cut-off was ≥6 compared with HADS-A with an AUC of 0.84 (95% CI = 0.78–0.90), whereas for the depression subscale, the AUC fell to 0.73 (95% CI = 0.58–0.87) with a DT cut-off ≥4. Further, from baseline to 12 months, the sensitivity of the DT compared with the anxiety and depression subscales lowered substantially from 73% to 57% and 73% to 47%, respectively. In contrast, specificity tended to be higher after 12 months compared with baseline for HADS-T and both subscales.

Figure 2

Receiver operating characteristics curves for the Distress Thermometer versus the anxiety and depression subscales, and the total of the Hospital Anxiety and Depression Scale, at baseline and 1 year after diagnosis

Study 3

The average DT score was 1.95 (SD = 2.4), whereas the average standardised BSI global severity index of participants was 45.32 (SD 8.37, range 36–72), with mean subscale scores for somatization 47.41 (SD 6.81, range 42–81), depression 46.05 (SD 7.04, range 42–77) and anxiety 44.99 (SD 7.38, range 39–73). Examining levels of distress using the t-score cut-off of ≥63, 19 participants (4.1%) reached caseness. For the subscales, 4.1% were classified as cases for anxiety, 4.9% for depression and 4.9% for somatisation. By using the alternative classification method of t ≥ 57, 55 participants (11.8%) were identified as a case at baseline.

The AUC was 0.84 (95% CI = 0.74–0.95) for the total score, with a DT cut-off ≥5 maximising sensitivity (78.9%) and specificity (84.9%) (Table 1 and Figure 3). For the subscales, the DT was more accurate in predicting anxiety and depression than somatisation, with AUCs of 0.90, 0.85 and 0.67, respectively. Furthermore, sensitivity ranged from 90% for anxiety down to 52% for somatisation, whereas specificity was high (82–86%) across each of the subscales. The selected threshold value was DT ≥ 5 on each subscale. For the alternative case-rule, the AUC was 0.80 (95% CI = 0.74–0.87) for the total score, and a DT cut-off of ≥4 maximised sensitivity (63.4%) and specificity (82.8%).

Figure 3

Receiver operating characteristics curves for the Distress Thermometer versus the total and the anxiety, depression and somatization subscales of the Brief Symptom Inventory 18 at baseline

Discussion

In these groups of men with prostate cancer, the DT 31 performed well at close to the time of diagnosis in terms of specificity and sensitivity for the detection of cancer-specific distress as well as for anxiety and depression. Optimal cut-off points, however, were somewhat variable for the different scales and component subscales, and time since diagnosis did appear to affect test sensitivity for anxiety and depression. For example, for the HADS 25, ≥3 was optimal for anxiety and ≥4 was better for depression; for the BSI-18 38, ≥5 was optimal for all subscales, although sensitivity was poor for somatisation. As time passed (1 year on), the DT became less sensitive as a marker for anxiety and depression; however, it retained high sensitivity for cancer-specific distress up to 3 years after diagnosis. On the basis of these data, we propose that a cut-off of ≥4 may be the best fit for the detection of psychological distress soon after a prostate cancer diagnosis, but that as time passes and into the longer term (more than 1 year after diagnosis), a cut-off of ≥3 may work most effectively as a one-off screening question that can then be followed up with more detailed diagnostic and needs assessment.

Most men with prostate cancer did not demonstrate high levels of psychological distress, suggesting that many men are psychologically resilient to their cancer experience. First, this raises feasibility and methodological challenges for psychosocial intervention research that seeks to reduce distress. Specifically, low levels of distress in this population group may create a floor effect for interventions that seek to reduce psychological distress 24. From this, it seems prudent to advise that in the future, such research should target high distress men. However, this then creates a practical challenge in terms of recruiting sufficient numbers for studies to be adequately powered such that larger collaborative studies may be needed across jurisdictions. Second, although many men may report low levels of psychological distress, long-term decrements in domain-specific quality of life after prostate cancer are common, particularly for sexual function 41. For example, many men, up to approximately 50%, report ongoing long-term unmet supportive care needs in the areas of psychological and sexual wellbeing 42–44. Further, recent data suggest men's psychological response to prostate cancer is closely related to their masculine self-esteem 35. Treatment for prostate cancer may impact masculine self-esteem not only through sexual changes but also other physical effects such as changes in muscle mass and adiposity 45, urinary and bowel control problems, as well as role and relationship changes 46–48. Interventions that closely articulate with men's symptom experiences, such as sexual dysfunction and bodily changes, and that address masculine self-esteem as a potential mediator of distress, may be most salient and effective 35.

Importantly, the present results support the use of the DT in men with prostate cancer and provide an empirically derived cut-off score that maximises sensitivity and specificity. The addition of the DT to psychosocial care plans for men with prostate cancer is indicated. Men are less likely to seek psychosocial support services after cancer compared with women 49,50 and are underrepresented as users of such services 50. Clearly, the first critical step is to identify those who are distressed and who likely require intervention beyond the standard support provided by front line oncology care staff. From this, men need to be referred to services that match their specific concerns and level of need 51, and in this regard, telephone-delivered nurse and peer counselling and support are highly acceptable to men with prostate cancer as sources of support 24,52. However, use of psychosocial care services by people with cancer is predicted by positive attitudes to help-seeking that includes the extent to which patients believe their doctor is supportive of these services 49,53. Hence, if we are to effectively increase use of psychosocial care by men with prostate cancer, screening and referral to support will likely need the advocacy of the treating health care team.

A limitation of our study was the relatively small number of men who were identified as being clinically distressed, anxious or depressed in each of our three cohorts. This resulted in wide confidence intervals for the estimated AUCs, making it problematic to identify significant differences in the validity of the DT over time and between subscales, because of the level of uncertainty associated with the point values. However, despite the small number of distress cases, results were generally consistent across the three cohorts. Also, distress is a multifactorial construct that has been described as encompassing emotional, psychological, social and spiritual elements 54,55, and this raises challenges in determining the optimal criterion to apply when assessing test sensitivity and specificity. Again, despite this issue, our results were consistent across three different measurement approaches for anxiety, depression and cancer-specific distress. Finally, we note that compared with other scales used in the present study, we found a relatively low reliability coefficient for the BSI-18 29 such that caution may be needed when considering the diagnostic accuracy of the DT for distress expressed through physical symptoms.

In conclusion, the present study presents comprehensive data showing the acceptability of the DT 31 for screening for psychological distress after a diagnosis of prostate cancer and across the illness trajectory. On the basis of this research, we have developed a problem list for the DT 31 that is specific to this patient group and that, similar to previous research 56, includes two questions allowing the patient to prioritise key concerns for intervention. Routine screening for distress in this patient population will allow service providers to better ensure that thoroughly focussed and in-depth psychological care is delivered where it is most needed and at the level most appropriate. We propose that self-management and low intensity approaches are indicated for most men with prostate cancer with clinically stepped-up services for those who report high distress or need and complex problems. The application and evaluation of a tiered or stepped care approach with this patient group is a priority for future research and care planning.