Critical consideration of assessment methods for clinically significant changes of mental distress after psycho-oncological interventions.

OBJECTIVES: Considering the heterogeneity of cancer entities and the associated disease progression, personalized care of patients is increasingly emphasized in psycho-oncology. This individualization makes the use of measurements of individual clinically significant change important when studying the efficacy and effectiveness of psycho-oncological care. Two conceptualizations for the measurement of clinical significance are critically contrasted in this study: the Reliable Change Index (RCI) and the Minimal Important Difference (MID) method.
METHODS: In total, 2,121 cancer patients participated in the study and a subsample of 708 patients was reassessed about 4 months later. Psychological distress was measured using the Hospital Anxiety and Depression Scale. We evaluated two measures of clinical significance (RCI, MID) by comparing the respective numbers of improved, unimproved, and deteriorated patients.
RESULTS: Individually significant changes were observed with both methods; however, determined rates of improvement differed substantially: MID (66.67%) and RCI (48.23%). Most importantly, according to MID, 17.93% of patients were identified as being improved, although their respective improvements were not statistically significant and thus unreliable.
CONCLUSIONS: The benefits of RCI outweigh MID, and therefore, the RCI is recommended as a measure to assess change.

INTRODUCTION

Coping with the diagnosis of cancer, and mastering the associated tasks and changes, can be a significant challenge for patients. Physical and mental distress are often associated with cancer, and can deplete patients' quality of life, disease progression, and survival rates (Chan, Ahmad, Yusof, Ho, & Krupat, 2015; Karakas & Okanli, 2014; Linden, Vodermaier, MacKenzie, & Greig, 2012). The most common psychological consequences are anxiety and depression (Bussmann et al., 2018; Linden et al., 2012). Due to the heterogeneity of cancer entities and the individual disease progression, the focus in recent years has increasingly been on patient‐oriented medicine (Sinaiko et al., 2017). Consequently, it is required of the care system to provide the right patient at the right time with the right care at the right place (Kusch, Labouvie, Gerlach, Hellmich, & Hallek, 2016; Kusch, Labouvie, & Hein‐Nau, 2013). In order to ensure evidence‐based patient‐centered care, care providers are also developing psycho‐oncological programs that can be used to provide individualized quality‐assured patient care, for example, continuous screenings for stress, psychoeducation, and stepped psycho‐oncological treatments (Fann, Ell, & Sharpe, 2012; Forsythe et al., 2013).

Usually, statements of effectiveness are based on the analysis of its statistical significance. On a group level, statistical significance gives information about differences found in terms of a probability level lower than would be expected if occurring by chance (Page, 2014). In controlled and well‐conducted group studies, significance testing provides meaningful and necessary evidence about the impact of specific interventions on a given population (Bothe & Richardson, 2011). However, the disadvantages of analyses on a group level are that very marginal differences can be statistically significant if the sample size is large enough (Hays, Brodsky, Johnston, Spritzer, & Hui, 2005; Hays, Spritzer, Sherbourne, Ryan, & Coulter, 2018). Furthermore, the results yield no information about individual change and, thus, cannot be used as an indicator of clinical significance (Bothe & Richardson, 2011; Lambert & Ogles, 2009). Similarly, when it comes to identifying responders to a particular treatment, the mere use of group‐level significance can lead to misclassification of patients as a responder if they show no change on an individual level (Hays et al., 2018).

Therefore, in addition to statistical significance on a group level, the relevance of clinical significance and related concepts are increasingly being used to improve change measurement and clinical decision‐making. These approaches are also increasingly used for the assessment and improvement of psycho‐oncological care programs (Bedard et al., 2013; Guyatt, Osoba, Wu, Wyrwich, & Norman, 2002; Ogles, Lunnen, & Bonesteel, 2001). The concept of clinical significance represents the assessment of significant change on an individual level. The methods used to accomplish this are either distribution‐based or anchor‐based approaches (Ogles et al., 2001; Page, 2014; Wyrwich, Norquist, Lenderking, Acaster, & The Industry Advisory Committee of International Society for Quality of Life Research, 2013).

Many different concepts have been developed to assess clinical significance. One of the leading concepts is the Reliable Change Index (RCI) (Jacobson & Truax, 1991). The definition of clinical significance is that a patient has returned from a so‐called dysfunctional (clinical) state to a so‐called functional (healthy) state (Jacobson & Truax, 1991). To observe this, it is necessary to combine two criteria, a statistically significant change and a clinically significant change. Only based on both criteria, the individual change can be classified within defined categories. A patient is classified as “recovered,” if the difference between the pre‐ and the posttest value is greater than the RCI (i.e., is statistically reliable), and if the posttest score has passed a predetermined cut‐off point. Put another way, this classification may only take place if there is a statistically and clinically significant change. Accordingly, a patient is classified as “improved,” if there is a statistically change, but the values did not pass the predetermined cut‐off point. Thus, the patient's dysfunctional symptoms are still present subsequent to treatment. Furthermore, there is a category of patients who are classified as “unimproved” or “deteriorated.” Unimproved means that patients revealed no statistical change, regardless of whether the cut‐off point was crossed. Furthermore, patients who report a statistically significant worsening of symptoms are classified as “deteriorated.”

However, besides the RCI, another concept of clinical significance, the minimal differences between two measurement points have been suggested, also known as Minimum/Minimal Important Difference (MID). Within this approach, the minimum significant difference or change for the patient should be represented as a score (Guyatt et al., 2002; Jaeschke, Singer, & Guyatt, 1989; Revicki et al., 2006; Wyrwich et al., 2013). The statistical significance is not a requirement for the calculation (in contrast to the RCI) (Page, 2014). First, the MID was defined as the “smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient's management” (Jaeschke et al., 1989, p. 408). The MID is calculated by the smallest significant difference between pre‐ and posttest value, which represents a “significant” change (Copay, Subach, Glassman, Polly, & Schuler, 2007; Revicki et al., 2006; Revicki, Hays, Cella, & Sloan, 2008). In order to define the significance of this change, the anchor‐based approach compares the results with other measures using an external anchor or criterion, whereas distribution‐based methods calculate the MID by using a measure of variability (Copay et al., 2007; Crosby, Kolotkin, & Williams, 2003). The advantage of the anchor‐based methods is the comparison of results with an external anchor, while the advantage of distribution‐based methods is that changes are presented free of random variations (Crosby et al., 2003). Each MID value for a given instrument may vary with regard to the studied population and the given context (Revicki et al., 2008). The aim of the MID is to provide feedback to the patient, as well as to the clinician, about the benefits and implications for further treatment. To balance the advantages and disadvantages of anchor‐based and distributed‐based approaches within the concept of MID, it is commonly recommended to calculate the MID using a combination of both methods (Bedard et al., 2013; Guyatt et al., 2002; Revicki et al., 2008). However, there is still no agreement which method or combination is the best (Guyatt et al., 2002). Because of the potential relevance of this decision for each individual patient, it is essential to make the right clinical decision. One study has already provided an overview about three different methods of clinical significance including standard error of measurement (SEM), standard error of prediction, and the RCI (Hays, Brodsky, et al., 2005). Note that the SEM is often used to calculate the MID (Ousmen et al., 2018). Whereas in the study of Hays, Brodsky, et al. (2005), the importance of examining the significant change at the individual level for improvement, consistency, or deterioration is emphasized, in many studies, significance on an individual level is often not ensured (Breitbart et al., 2015). Furthermore, no recommendation was made with regard to which of the methods should be used. Because of the high relevance of clinical decision‐making, this study aims to critically contrast the two measures of clinical significance (RCI, MID) based on the change in the symptoms of anxiety and depression by cancer patients due to psycho‐oncological treatment.

METHODS

Participants and procedure

Data collection took part as part of a standardized program of the Clinic I of Internal Medicine (Clinical Psychology) in cancer patients of the Centre of Integrated Oncology Cologne and from the region. All participants provided written informed consent. The data were collected at two measurement time points. The first measurement time point (t1) was at the time of inpatient admission of the cancer patients and the second measurement time point (t2) was 4 months later. The questionnaire was handed out at t1 as part of the standardized care program (Kusch et al., 2014). At t2, the questionnaire was handed out again if patients stayed in the hospital or sent to patients by mail, if they were discharged from the inpatient unit before t2. In total, 2,121 cancer patients (1,643 women [77.5%]) with mean age of 53.02 (SD = 13.50) and mean Hospital Anxiety and Depression Scale total score (HADS‐T) of 16.91 (SD = 8.56) participated in the study at t1 and 708 patients (582 women [82.2%]); mean age of 53.23 (SD = 13.00) and mean HADS‐T of 13.67 (SD = 7.88) filled out the questionnaire a second time at t2. The cancer diagnoses among the participants are presented in Table 1. All procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. The work was approved by the Ethics Commission of Cologne University's Faculty of Medicine (reference number 15‐048).

Table 1

Percentage of cancer diagnoses among participants at t1

Types of cancer	Percentage (%)
Breast	40.9
Lymphoid, hematopoietic, and related tissue	18.2
Female genital organs	9.1
Digestive organs	7.0
Respiratory and intrathoracic organs	3.9
Eye, brain and other parts of central nervous system	3.3
Male genital organs	1.7
Thyroid and other endocrine glands	1.6
Skin	1.5
Urinary tract	1.4
Mesothelial and soft tissue	1.3
Lip, oral cavity and pharynx	1.1
Ill‐defined, secondary and unspecified sites	0.8
Bone and articular cartilage	0.4
Residual category (including different forms of cancer)	7.8

Measures

Self‐reported distress was measured by using the German version of the HADS (Herrmann‐Lingen, Buss, & Snaith, 2011). The scale is an established tool for the assessment of anxiety and depression in cancer patients (Mitchell, Meader, & Symonds, 2010; Vodermaier & Millman, 2011). Furthermore, it is recommended as a screening tool for the measurement of psychological strain according to the S3‐guidelines for the psycho‐oncological management of adult cancer patients (Deutsche Krebsgesellschaft, Deutsche Krebshilfe, & AWMF, 2014). The HADS is a self‐rating questionnaire and consists of 14 items with a total score ranging from 0 to 42. It is also possible to calculate subscales scores, but in psycho‐oncological contexts, patients often show combined and fluctuating manifestations of anxious and depressive symptoms so that a global measure of the HADS‐T can best represent the clinical situation (Herrmann‐Lingen et al., 2011). Based on the HADS‐T, cancer patients suffering from significant distress can be reliably distinguished from cancer patients without distress (Mitchell et al., 2010). Accordingly, the analyses of the present study will focus on the HADS‐T score. There are different cut‐off scores recommended for HADS‐T. Specifically, in cancer patients, a sum score of HADS‐T ≥ 15 can be used as the cut‐off value to identify patients with an increased need for psycho‐oncological care and especially for depression symptoms (Mitchell et al., 2010; Vodermaier & Millman, 2011). The psychometric properties of the HADS indicate a reliable and valid instrument (Bjelland, Dahl, Haug, & Neckelmann, 2002; Herrmann‐Lingen et al., 2011). In the present study, Cronbach's alpha was excellent (HADS‐T: α = 0.91).

Statistical analyses

Patients' characteristics are described by using means and standard deviations. Kolmogorov–Smirnov test results indicated that the data were not normally distributed. Nonetheless, the statistical measure of change on group level was examined by multivariate analysis of variance for repeated measures, since the F‐test is robust in a large sample if the assumption of normal distribution is not met. Tests were calculated two‐tailed with an assumed significance level of p < .05.

For the analysis of clinical significance, only cancer patients who had higher values at the first measurement time point than a critical threshold of HADS‐T ≥ 15 were included. The rationale for this decision was that the present study is focused on the methodological comparison of two concepts of clinical change and whether these two concepts would influence clinical decision‐making differentially. We presumed that individuals with relevant self‐reported distress would more likely improve through psycho‐oncological care allowing for the best possible comparison of the two concepts. Note, however, that we do not want to imply or indicate that patients with values under the cut‐off threshold of ≥15 could not potentially benefit from psycho‐oncological care. The calculation formula for RCI used here is:

According to this formula, the RCI is calculated as the difference between the pre‐ (Y) and posttest (X) values divided by the standard measurement error of the difference (Jacobson & Truax, 1991). As a measure of reliability, α is recommended and will be used for the test instrument and the examined sample (Lambert & Ogles, 2009).

A RCI > 1.96 corresponds to a 95% confidence interval and indicates that the individual change is statistically significant. In the present case, the change between the pre‐ and posttest values had to be greater than or equal to five to indicate a statistically significant change for each individual patient. To assess whether the change is also clinically significant, the postvalue must pass a predetermined cut‐off point, which separates the dysfunctional from the functional population (Jacobson & Truax, 1991). This cut‐off point can be defined in three different ways (Jacobson & Truax, 1991; Lambert & Ogles, 2009). In the present study, the most conservative cut‐off point was used. This cut‐off point is based on information from functional and dysfunctional population and is recommended to determine the significant clinical change if functional standards are known. We used the calculation formula for this cut‐off point: (Lambert & Ogles, 2009). Thus, data from a healthy population were used to define the cut‐off point (HADS‐T: M = 9.45; SD = 6.80) (Hinz & Brähler, 2011). In consequence according to the calculation formula, the cut‐off point of HADS‐T = 16.52 was used in addition to the RCI to determine significant clinical change in the present study.

In order to define clinically significant change employing MID, studies were searched in which an MID was established for the HADS‐T. We found one study that had calculated an MID of 1.5 for the HADS‐T in a sample of patients with chronic obstructive pulmonary disease (Puhan, Frey, Büchi, & Schünemann, 2008). The MID of 1.5 was calculated using an anchor‐based approach employing a linear regression analysis. Two self‐report instruments that assessed the burden of disease (symptomatology and affect) were used as anchors. In the same study, an effect size of 0.5 SD units of change score was additionally used to estimate a MID of 1.17 for the HADS‐T. In this case, the MID was estimated based on a distribution‐based method as an alternative to the anchor‐based approach (Puhan et al., 2008). Investigators regularly consider an effect size of 0.5 SD units as an adequate estimate of clinically significant change (Walters & Brazier, 2003). Given that the HADS‐T results only in positive integers, no actual difference results in using the anchor‐ or distribution‐based method. More specifically, the MID between pre‐ and postmeasurement always had to be ≥2‐point change on the HADS‐T.

RESULTS

At t1 1,251 of all 2,121 patients (59.0%) exceeded a HADS‐T score ≥ 15, indicating relevant distress; the mean HADS‐T score of these patients was 22.61 (SD = 5.85). From all 708 patients who took part at both measurement time points, 396 patients (55.9%), who exceeded a HADS‐T ≥ 15, reported a mean HADS‐T at t1 of 22.08 (SD = 5.59) and at t2 of 17.40 (SD = 7.16). Significant differences between the group who took part at only t1 (group 1; 1,413 patients) and the group who took part at both t1 and t2 (group 2; 708 patients) were found with respect to gender (a higher proportion of women in group 2; χ2[1] = 13.68, p < .001) and their respective HADS‐T scores (higher scores in group 1; t[2119] = 2.75, p = .006) at measurement time point t1. There were no differences between the groups with regard to age (t[2118] = −0.488, p = .626). Overall, there was a statistically significant improvement on the group level (HADS‐T: F[1,707] = 95.35, p < .001; = .119) from t1 to t2 in all patients of group 2. Furthermore, patients who exceeded the critical threshold of the HADS‐T score of ≥15 at the time of pre‐examination showed also a significant change at t2 (F[1,395] = 176.75, p < .001; = .309), illustrating an even stronger effect of treatment in this subgroup of patients of group 2.

For the analysis of statistical and clinical significance on an individual level, first the RCI was calculated. Additionally, an external MID value was used as alternative reference. In order to better illustrate the effects of using the two different types of clinically significant change, in this analysis, only patients who reached or exceeded the critical threshold for psychological strain as measured by HADS‐T scores ≥15 at the time of the pre‐examination were included (i.e., the group of patients for whom psycho‐oncological treatment is indicated).

According to RCI concept, 193 of 396 (48.74%) cancer patients exhibited a statistically reliable change of self‐reported distress (improved; see Table 2). Statistically significant and clinically significant change in symptoms was seen in 191 cancer patients (48.23%) (recovered). In turn, 29 (7.32%) patients showed a reliable worsening of symptoms (deteriorated) and 174 (43.94%) patients showed neither statistical nor clinically significant changes (unimproved). The estimated change of ≥2 points on the HADS‐T was used as a cut‐off point to assess the clinical change using the MID concept. For HADS‐T, 264 of the 396 analyzed patients (66.67%) achieved a change of at least ≥2 points in HADS‐T between t1 and t2. However, among the 264 patients, 71 patients (17.93%) did not meet the criteria of a reliable statistical change and 73 patients (18.43%) of significant clinical change according to the concept of the RCI. Table 2 illustrates this pattern of results using a crosstab to give an overview on patients' changes with regard to depressive and anxiety symptoms as determined by using the RCI and MID concept.

Table 2

Frequency and percentage of different change based on RCI and MID for cancer patients for HADS‐T from pre‐ to postexamination

	MID—deteriorated (pre–post difference ≤ −2)	MID—unimproved (2 > pre–post difference > −2)	MID—improved (pre–post difference ≥ 2)	Total
RCI—deteriorated (pre–post difference ≤ −5)	29 (7.32%)	0	0	29 (7.32%)
RCI—unimproved (5 > pre–post difference > −5)	39 (9.85%)	64 (16.16%)	71 (17.93%)	174 (43.94%)
RCI—improved (pre–post difference ≥ 5)	0	0	193 (48.74%)	193 (48.74%)
Total	68 (17.17%)	64 (16.16%)	264 (66.67%)	396 (100%)

DISCUSSION

Before interpreting the data, it should be called to mind that the aim of this study is not the evaluation of a specific psycho‐oncological treatment package, but rather a comparison of two commonly used methods of determining clinical significance. The efficacy of treatment is not emphasized in the present study, so the relatively high dropout rate and the missing control group are not exceptional given the naturalistic sample. Thus, this study aimed to critically contrast two concepts for the measurement of clinical significance (RCI and MID) within a sample of cancer patients with respect to the identifiable ratio of individual improvement and deterioration in symptoms between the methods.

Overall, the patients with signs of substantial psychological stress before treatment showed significant improvement in symptoms of anxiety and depression. These results are in line with other studies, which previously highlighted the potentially highly positive effects of integrative psycho‐oncological treatment on anxiety and depression in cancer patients (Grassi, Spiegel, & Riba, 2017; Kost et al., 2009). To transfer these group‐level results to measurements of clinical relevance for each individual patient, RCI and MID were calculated and critically compared. Both measurements of clinical change supported the claim of patient‐oriented psycho‐oncological care as being efficacious. Based on the RCI concept, 48.74% of patients exhibited a reliable statistical change (improvement) with regard to symptoms of anxiety and depression. The results of the analysis are in line with other studies focusing on individual case analyses (Grassi et al., 2017; Kost et al., 2009). In addition, 48.23% of patients reported a statistically and clinically significant change (recovered) based on their scores on the HADS‐T. On the downside, 29 (7.32%) patients reported statistically increased levels of depression and anxiety.

The estimation of change and clinical significance was also performed employing the MID. According to the MID concept, 66.67% of patients showed a clinically meaningful improvement. Thus, compared with the RCI, the estimated number of patients with clinically significant improvements was clearly higher when using MID. However, 71 patients (17.93%), who supposedly had improved, did not meet the requirements for a statistically reliable change. Moreover, 73 patients (18.43%) did not meet the criteria for significant clinical change according to the RCI concept (i.e., rather belong to a healthy as compared to a psychologically stressed population). In addition, 39 patients (9.85%) were classified as deteriorated according to MID. Although again, this numerical change was not statistically reliable. Thus, on the one hand, the MID value as a cut‐off point appears less stringent than the RCI's cut‐off points, and on the other hand, it is not ensured that identified changes are indeed reliable changes and consequently there is a substantial risk of overestimating the results when studying the effects of an intervention. Interestingly, using MID did not result in a larger group of individuals being detected whose psychological stress had deteriorated. Thus, the lower threshold of the MID only resulted in a more liberal detection of improvement and not of deterioration.

An important advantage of RCIs compared with the MID is that statistical significance of each individual change is a requirement for determining clinical significance. Or with other words, clinical significance is always statistically safeguarded (Lambert & Ogles, 2009). Also, the calculation of an individual's clinical significance using the RCI concept is quick and efficient (Lambert & Ogles, 2009). A criticism is that the RCI is a rather conservative method and the cut‐off points used are relatively strict criteria. Individuals with low initial symptom severity have little chance of undergoing a clinically relevant change as they have little room for improvement (Lambert & Ogles, 2009). However, we believe that being conservative when determining the benefit of an intervention is a merit rather than being problematic. Before ending treatment, it should be safely ensured that a person indeed has improved rather than discharging a patient who only ostensibly has improved. Another potential problem with the MID was not much of a problem in our sample due to the instrument we used. There are a number of different definitions for the calculation of the MID, which often have different aims (King, 2011). For the approach of the anchor‐based methods as well as for the distribution‐based methods, different authors suggest different types of calculations (King, 2011; Revicki et al., 2008). In a recent structured review, Ousmen et al. (2018) reported that the most commonly used distribution‐based method was the 0.5 SD, followed by the SEM. Note that as the best approximation, a combination of both approaches is recommended (Bedard et al., 2013; Guyatt et al., 2002). However, this approach rarely used in practice (Ousmen et al., 2018). Furthermore, Hays, Farivar, and Liu (2005) highlighted that despite the fact that several measures (e.g., SEM) are related to the MID, these measures do not provide direct information about the MID (e.g., new information about the size of change) and that their use should therefore be discouraged. In our sample, given that only integer numbers are calculated as individual scores on the HADS‐T, these different MID scores resulted always in the same HADS‐T change score of 2. Thus, in the present study, it seemed rather inconsequential, which method to use.

Study limitations

Besides the strength of large sample size, examination on patients' reports, and the benefits for clinical and research practice, the study also has limitations. One limitation is that the difference from pre‐ to postexamination does not necessarily have to be related solely to psycho‐oncological treatment. Frequently, patients respond to cancer diagnosis with signs of anxiety and depression, which often remit spontaneously without the need for psycho‐oncological support (Cook, Salmon, Hayes, Byrne, & Fisher, 2018). To examine this aspect in a study with an additional randomized control group would be interesting and important. Additionally, the data showed a very high drop‐out rate of approximately 66.6% in the period between pre‐ (2,121 cancer patients) and postexamination (708 cancer patients). This can primarily be explained by the fact that the data collection did not take place as part of a research project, but within routine care. It is possible that patients switched to outpatient care were no longer in treatment or even died. This was possibly especially likely because the data were collected in a highly specialized treatment facility, recruiting patients from a relative wide catchment area. Nonetheless, there are differences between the patients who dropped out and who took part at both measurement time points with regard to gender and the initial HADS‐T score. To control these differences would be interesting and important in further studies. Note, however, that the differences on the HADS‐T are significant but that the mean HADS‐T values of the drop‐out group (M = 17.27, SD = 8.69) and the nondrop‐out group (M = 16.18, SD = 8.24) nonetheless had higher values than the critical threshold of HADS‐T ≥ 15. Arguably, the self‐reported psychological distress was clearly relevant in both groups. Moreover, this group difference had no direct influence on the analysis of possible differences between the individual change measurements.

Clinical implications

Concepts to determine the clinical significance of treatments are increasingly being used for adequate change measurement and clinical decision‐making. These approaches are also increasingly used in the assessment and improvement of psycho‐oncological care. Therefore, it is important to critically assess which method can be used for a clinical decision. Although the MID ostensibly shows a higher percentage of improvements, it is statistically unreliable and as a consequence of its use, an overestimation of the effects of this form of intervention is possible. In addition, errors in clinical decision‐making may result if patients' treatments are ended prematurely. Due to these weaknesses of the MID and the substantive advantages of the RCI, the RCI is recommended as a measure of change in the care research of cancer patients.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.