Evaluating the psychometric properties of the Immunotherapy module of the MD Anderson Symptom Inventory.

INTRODUCTION: Immunotherapies have revolutionized the treatment of various cancers, but little is known about their symptomatic toxicity. Assessing these symptoms is best accomplished by asking the patients themselves. However, such reports are subjective and may face challenges as bonafide scientific data. Demonstrating the validity of symptom assessment tools, mainly through the reduction of measurement errors, has the potential to improve patient care if these tools are widely adopted. To that end, we present herein the psychometric properties of the Immunotherapy for Early-Phase Trials module of the MD Anderson Symptom Inventory (MDASI-Immunotherapy EPT) in patients receiving various immunotherapies in early phase trials at a major cancer center.
METHODS: One hundred forty-five patients completed the inventory at baseline, with 85 of them also doing so after 9 weeks of treatment. The mean (±SD) age of the patients was 57.0±12.9 years. Also, 56% of the patients were women, 79% identified as white, and 49% had at least some college education.
RESULTS: The internal consistency reliability of the MDASI-Immunotherapy EPT was excellent, as the Cronbach's alphas for all of its subscales were at least 0.88 (range 0.88-0.95). Known-group validity based on Eastern Cooperative Oncology Group performance status groupings was excellent at 9 weeks after the start of an immunotherapy trial for the MDASI-Immunotherapy EPT severity (effect size, 0.96) and interference (effect size, 0.82) subscales. We found substantial changes in the symptom items difficulty remembering (effect size, -0.85), fever and/or chills (effect size, -0.63), disturbed sleep (effect size, -0.52), diarrhea (effect size, -0.42), and swelling of hands, legs, or feet (effect size, -0.39).
CONCLUSIONS: In conclusion, the MDASI-Immunotherapy EPT is a valid, reliable, and sensitive tool for measuring symptomatic toxicity. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Patients with cancer experience disease and treatment-related symptoms that profoundly impact their quality of life and ability to function.1 Symptoms are further aggravated by newer cancer treatments such as immunotherapies that interfere with the immune system. With this disruption in immune balance, a unique set of toxicities referred to as immune-related adverse events has emerged. These events are classically autoimmune in nature and are often T-cell mediated.2

Toxic effects associated with immunotherapy are generally assessed via tabulation of adverse events, which are graded by clinicians. However, it is generally accepted that clinicians typically underestimate the symptoms of toxicity in patients under their care.3 To more accurately assess such symptoms, a better strategy is to rely on patient-reported outcomes (PRO). The patient symptom experience that is captured through PRO questionnaires administered during oncological clinical trials plays a critical role in how drug-approval agencies such as the US Food and Drug Administration and European Medicines Agency evaluate the overall clinical risks and benefits of new therapeutic agents.

Symptom assessment requires psychometrically validated tools that are easy to use and quick to administer. One such tool, the MD Anderson Symptom Inventory (MDASI),4 was designed to assess the severity of common cancer-related and treatment-related symptoms in a way that better reflects the symptom experience of the patient population with cancer. The MDASI assesses not only the intensity of cancer-related symptoms but also the level of symptom interference with daily functioning. Symptoms specific to a particular cancer, treatment method, or treatment site can be added to the core MDASI. In fact, we have done so, creating MDASI modules that include the 13 symptom and 6 interference items of the core MDASI augmented by additional disease-specific and treatment-specific symptom items. Specifically, we developed MDASI modules for patients with brain tumors,5 head and neck cancer,6 treatment-related heart failure,7 lung cancer,8 and malignant pleural mesothelioma,9 among others. The number of additional module-specific symptom items is minimized to keep the MDASI concise and easy to use in clinical and research settings and to facilitate repeated measurement.

The goal of the present prospective study was to demonstrate the reliability and validity of a MDASI module specific to immunotherapy for early-phase trials (MDASI-Immunotherapy EPT). This module asks patients to describe symptoms related to cancer in general and rate symptoms related specifically to immunotherapy during early-phase trials. We hypothesized that the MDASI-Immunotherapy EPT is a valid and reliable symptom assessment measure.

The MDASI-Immunotherapy EPT was developed to fill a need for an instrument that covered the spectrum of symptomatic adverse events that could be experienced by patients receiving immunotherapy. Although some other PRO measures have been used in immunotherapy, the MDASI-Immunotherapy EPT has several benefits and key advantages, such as that it is based on the core-MDASI that has been validated across cancer types and cancer treatments and available in multiple different languages. The MDASI-Immunotherapy EPT also has all eight symptoms that King-Kallimanis et al have described as critical adverse events related to immunotherapy, namely, fatigue, diarrhea, cough, shortness of breath, musculoskeletal pain, rash, pruritus, and fever.10 In their review of 28 registration trials for FDA-approved anti-programmed-cell-death-1 inhibitors, King-Kallimanis et al reported that 75% of these trials included PRO measures, and most trials used the EuroQol five dimension or the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire. King-Kallimanis et al indicated that no PRO measure reviewed included all eight symptoms that the authors considered as critical adverse events related to immunotherapy, namely fatigue, diarrhea, cough, shortness of breath, musculoskeletal pain, rash, pruritus, and fever.10 We note that all of these eight symptoms are included in the MDASI-Immunotherapy EPT.

Methods

Subjects were recruited from the Investigational Cancer Therapeutics department at The University of Texas MD Anderson Cancer Center. To be eligible for this study, individuals were required to be at least 18 years old, speak English, have a pathological diagnosis of cancer, and be receiving immunotherapy. Patients were excluded if clinical research staff felt that the patient was unable to understand or unwilling to sign a written informed consent document. Patients completed the MDASI-Immunotherapy EPT at baseline (before the initiation of immunotherapy) and after 9 weeks of treatment. All patients provided written consent to participate. This study was approved by the MD Anderson Institutional Review Board (protocol PA15-0315).

Measures

At the time of enrollment, research staff asked study subjects to complete self-administered questionnaires that ask patients about their symptoms and their impact on daily functions. Survey staff answered questions about the study and assisted with completion of survey forms as needed. Patient demographic information (eg, sex, age, marital status, education level, employment status) was collected during the initial clinic visits using a general survey questionnaire. A study-specific clinician checklist was used to collect medical information from hospital records, including treatment, presence of metastases, cancer diagnosis, location, and staging.

Eastern Cooperative Oncology Group (ECOG) performance status (PS) was used to describe the patient’s level of functioning.11 ECOG PS is a physician-rated measure of functional ability using scores ranging from 0 (fully active; able to carry on all predisease performance without restrictions) to 4 (completely disabled; cannot perform self-care; totally confined to bed or chair).

The MDASI-Immunotherapy EPT asks patients to rate the severity of disease-related and treatment-related symptoms over the past 24 hours. Each symptom item (pain; fatigue; nausea; disturbed sleep; distress/feeling upset; shortness of breath; difficulty remembering; lack of appetite; drowsiness; dry mouth; sadness; vomiting; numbness/tingling; rash; diarrhea; pain in the abdomen; swelling of hands, legs, or feet; headache; night sweats; and fever and/or chills) is rated on an 11-point scale ranging from 0 (not present) to 10 (as bad as you can imagine). Patients also rate the degree to which symptoms interfere with various aspects of life over the past 24 hours. In addition, each interference item (general activity, mood, normal work (including both work outside the home and housework), relations with other people, walking ability, and enjoyment of life) is rated on an 11-point scale ranging from 0 (did not interfere) to 10 (interfered completely). To create the MDASI-Immunotherapy EPT, seven symptoms were added to the core MDASI rash; diarrhea; pain in the abdomen; swelling of hands, legs, or feet; headache; night sweats; and fever and/or chills. Selection of these immunotherapy-specific symptom items was based on the literature review and clinician input.

Scoring the MDASI-Immunotherapy EPT

The ratings in the MDASI-Immunotherapy EPT can be averaged into several subscale scores: mean severity score (the 13 core symptom items plus the immunotherapy-specific items), mean core score (the 13 core symptom items only), and mean interference score (the interference items only). The interference items can be broken down further into a mean activity-related interference score (work, general activity, and walking ability) and mean mood-related interference score (relations with people, enjoyment of life, and mood).

A more sensitive characterization of symptoms for a given cohort may use a subset of the most severe symptoms reported by that group. Symptom items may be used individually or in subsets without summary scoring if specified a priori. Specific symptom items can also be used based on the expected (ie, prespecified) outcome. For example, a hypothesis by our group was that cancer patients undergoing immunotherapy who experienced pneumonitis would have worsening of shortness of breath.

Statistical analysis

All statistical analyses were conducted using SPSS (V.24; IBM, Armonk, New York, USA). Means, SDs, ranges, and 95% CIs were computed for all symptoms and subscales.

Reliability of the MDASI-Immunotherapy EPT

Internal consistency reliability refers to the extent to which the items in a scale measure the same concept. Cronbach coefficient alphas were computed to estimate the internal consistency reliability of the MDASI-Immunotherapy EPT core (13 MDASI symptom items), severity (core plus immunotherapy-specific items), and interference (6 interference items) subscales. The criterion for good internal consistency (reliability) is a Cronbach’s alpha value of 0.70 or higher.12

Validity of the MDASI-Immunotherapy EPT

Known-group validity comparisons were made for the MDASI-Immunotherapy EPT subscales relative to ECOG PS scores. Independent t tests were performed to demonstrate ECOG PS group differences in the subscales. The MDASI-Immunotherapy EPT should be able to discriminate between patients with good and poor PS.

Sensitivity of the MDASI-Immunotherapy EPT

Sensitivity is defined as the ability of an instrument to detect a change in outcome using its subscales or items when such a change is expected. Several tests were conducted to assess the MDASI-Immunotherapy EPT’s sensitivity to changes in PS. Effect sizes were calculated to estimate the magnitude of differences in subscale scores and items.13 14

First, whether the MDASI-Immunotherapy EPT could detect worsening of symptoms among patients with deteriorating ECOG PS was evaluated. Also, whether the MDASI-Immunotherapy EPT could detect changes in patients whose PS changed over the course of their disease—particularly, whether symptom severity increased for patients whose PS deteriorated over time—was examined. Patients whose PS improved were expected to have improved symptom scores. Changes in the MDASI-Immunotherapy EPT subscales and individual items were computed and considered to be clinically meaningful at an SD of 0.5 or higher, the level often used to indicate meaningful differences.15

Results

Of the 145 patients who completed the MDASI-Immunotherapy EPT at baseline, 85 also did so at week 9 of treatment. The reduction in the number of patients completing questionnaires at week 9 compared with baseline were due to attrition, or patients having come off the study due to progressive disease, clinical progression, withdrawal of consent or due to toxicities. The mean (±SD) patient age was 57.0±12.9 years. About 56% of the patients were women, 79% identified as white, and 49% had at least some college education. The five most common cancer diagnoses were colorectal (9%), skin (non-melanoma) (9%), ovarian (8%), cervix (7%), and sarcoma (6%).

Baseline/pretreatment symptom severity

The mean baseline core, immunotherapy severity, and interference subscale scores were 2.07, 1.72, and 2.46, respectively; see table 1. The most severe core symptoms reported at baseline were fatigue, pain, disturbed sleep, drowsiness, and distress/feeling upset. The least severe core symptoms were vomiting and nausea. Pain in the abdomen was the most severe Immunotherapy EPT module-specific item reported at baseline.

Table 1

Descriptive statistics for the severity (on a 0–10 rating scale) of the symptom items of the MDASI-Immunotherapy EPT at baseline and week 9 of treatment

Mean (SD)		Mean (SD)
MDASI-Immunotherapy EPT item	Baseline	MDASI-Immunotherapy EPT item	Week 9
Fatigue	3.7 (2.7)	Pain	3.6 (3.3)
Pain	3.1 (3.3)	Fatigue	3.5 (2.5)
Disturbed sleep	2.7 (2.7)	Disturbed sleep	2.6 (2.6)
Drowsiness	2.5 (2.6)	Drowsiness	2.1 (2.1)
Distress/feeling upset	2.2 (2.5)	Pain in the abdomen	2.1 (2.4)
Lack of appetite	1.9 (2.6)	Distress/feeling upset	1.8 (2.1)
Dry mouth	1.9 (2.6)	Difficulty remembering	1.9 (2.3)
Numbness/tingling	2.0 (2.6)	Numbness/tingling	1.7 (2.0)
Pain in the abdomen	2.0 (2.7)	Dry mouth	1.8 (2.5)
Sadness	1.8 (2.3)	Lack of appetite	1.6 (2.1)
Difficulty remembering	1.6 (2.1)	Sadness	1.5 (2.1)
Shortness of breath	1.6 (2.2)	Night sweats	1.5 (2.2)
Nausea	1.3 (2.5)	Shortness of breath	1.4 (2.0)
Diarrhea	1.1 (2.3)	Swelling of hands, legs, or feet	1.3 (2.1)
Swelling of hands, legs, or feet	1.1 (2.3)	Headache	1.0 (1.7)
Night sweats	1.0 (2.0)	Rash	1.0 (2.0)
Vomiting	0.7 (2.0)	Diarrhea	1.3 (2.1)
Headache	0.9 (1.8)	Nausea	1.0 (1.7)
Rash	0.9 (2.1)	Fever and/or chills	0.9 (1.9)
Fever and/or chills	0.6 (1.5)	Vomiting	0.6 (1.5)

MDASI-Immunotherapy EPT, Immunotherapy for Early-Phase Trials module of the MD Anderson Symptom Inventory.

Symptom severity at week 9 of treatment

The mean core, severity, and interference subscale scores at week 9 were 1.92, 1.70, and 2.62, respectively; see table 1. The most severe core symptoms reported were pain, fatigue, disturbed sleep, drowsiness, and difficulty remembering. The least severe core symptoms were vomiting and nausea. Pain in the abdomen was the most severe Immunotherapy EPT module-specific item reported at week 9 of treatment.

Validation of the MDASI-Immunotherapy EPT

Internal consistency reliability. The MDASI-Immunotherapy EPT subscales demonstrated good internal consistency reliability. At baseline, the Cronbach coefficient alpha values were 0.91 for the core subscale, 0.92 for the severity subscale, and 0.93 for the interference subscale. The Cronbach alpha values at week 9 of treatment were 0.90 for the core subscale, 0.92 for the immunotherapy severity subscale, and 0.94 for the immunotherapy interference subscale. The Cronbach alpha values at all assessment time points are presented in table 2.

Table 2

Internal consistency reliability for the MDASI-Immunotherapy EPT at each assessment time

Time	Core	Immunotherapy EPT severity	Immunotherapy EPT interference
Baseline	0.91	0.92	0.93
Week 1	0.91	0.92	0.93
Week 2	0.91	0.92	0.92
Week 3	0.90	0.91	0.92
Week 4	0.88	0.89	0.91
Week 5	0.89	0.91	0.94
Week 6	0.89	0.90	0.93
Week 7	0.89	0.91	0.93
Week 8	0.89	0.91	0.95
Week 9	0.90	0.92	0.94

MDASI-Immunotherapy EPT, Immunotherapy for Early-Phase Trials module of the MD Anderson Symptom Inventory.

Construct (known-group) validity. We performed known-group validity comparisons for the MDASI-Immunotherapy EPT subscales relative to ECOG PS. The MDASI-Immunotherapy EPT discriminated between patients with good and poor PS; see table 3. Specifically, at baseline, patients with an ECOG PS of 0 had lower MDASI-Immunotherapy EPT severity subscale scores than did patients with an ECOG PS of at least 1 (effect size, −0.45). At week 9 of treatment, the mean severity and interference subscale scores were substantially worse in patients with an ECOG PS of at least 1 than in patients with an ECOG PS of 0, as indicated by large effect sizes (−0.96 for the severity subscale and −0.82 for the interference subscale). This confirmed excellent known-group validity for the newly developed MDASI-Immunotherapy EPT severity and interference subscales at week 9.

Table 3

Known-group validity of the MDASI-Immunotherapy EPT by ECOG PS at baseline and week 9

	Patients with ECOG PS available, n	MDASI-Immunotherapy EPT symptom severity score*				MDASI-Immunotherapy EPT interference score†
		ECOG PS=0, mean (SD)	ECOG PS≥1, mean (SD)	95% CI	Effect size	ECOG PS=0, mean (SD)	ECOG PS≥1, mean (SD)	95% CI	Effect size
Baseline	145	1.3 (2.0)	1.8 (1.5)	−1.40 to 0.50	−0.45	2.1 (3.2)	2.5 (2.2)	−2.5 to 1.6	−0.19
Week 9	84	0.5 (0.7)	1.7 (1.4)	−2.60 to 0.08	−0.96	0.8 (1.1)	2.7 (2.3)	−4.1 to 0.4	−0.82

*The average of the 13 core and seven immunotherapy-specific symptom items.

†The average of the six interference items.

ECOG, Eastern Cooperative Oncology Group; MDASI-Immunotherapy EPT, Immunotherapy for Early-Phase Trials module of the MD Anderson Symptom Inventory; PS, performance status.

Sensitivity to change in ECOG PS. We assessed the MDASI-Immunotherapy EPT to determine whether it can detect symptom changes when PS changes over the course of treatment. ECOG PS data were available for 145 patients at baseline and 85 patients at week 9 of treatment; see table 4. Calculation of effect sizes for change scores between baseline and week 9 demonstrated that the differences were clinically meaningful as reflected by the effect sizes for the MDASI-Immunotherapy EPT symptoms difficulty remembering (0.85), fever and/or chills (0.63), disturbed sleep (0.52), diarrhea (0.42), and swelling of hands, legs, or feet (0.39). Based on Cohen’s criteria13 14 for effect sizes, the above would be considered moderate to large effects.

Table 4

Sensitivity of the MDASI-Immunotherapy EPT based on changes in ECOG PS in patients with worsening PS from baseline to week 9 (n=10)

Variable	Baseline, mean (SD)	Week 9, mean (SD)	95% CI of the difference	Effect size
MDASI-Immunotherapy EPT subscale
Symptom severity*	1.5 (1.5)	1.9 (1.4)	−1.2–0.4	−0.29
Symptom interference†	2.9 (3.0)	2.8 (2.5)	−2.0–2.1	0.02
MDASI-Immunotherapy EPT symptom item
Rash‡	0.8 (1.8)	0.9 (1.5)	−1.2–1.0	−0.06
Fever and/or chills‡	0.5 (1.3)	1.3 (2.7)	−2.6–1.0	−0.63
Headache‡	1.2 (1.8)	1.2 (1.6)	−1.4–1.4	0
Swelling of hands, legs, or feet	0.5 (1.3)	1.0 (1.6)	−1.4–0.4	−0.39
Numbness/tingling‡	1.6 (3.0)	1.7 (1.8)	−1.5–1.3	−0.03
Dry mouth‡	1.1 (1.4)	2.6 (2.9)	−3.1–0.6	−1.04
Fatigue	3.4 (3.0)	3.7 (2.8)	−3.0–2.4	−0.10
Sadness	1.9 (3.0)	2.4 (3.3)	−3.0–2.0	−0.17
Night sweats	1.1 (1.6)	1.0 (1.3)	−1.1–1.3	0.06
Lack of appetite	1.6 (2.6)	2.0 (2.7)	−3.4–2.6	−0.15
Distress/feeling upset	2.3 (3.2)	2.4 (2.8)	−1.7–1.5	−0.03
Shortness of breath	0.8 (1.7)	0.8 (1.5)	−0.3–0.3	0
Diarrhea	0.9 (1.7)	1.6 (2.7)	−1.8–0.4	−0.42
Pain	2.0 (2.4)	2.6 (3.1)	−1.4–0.2	−0.24
Drowsiness	2.7 (2.3)	3.2 (2.0)	−1.7–0.7	−0.22
Pain in the abdomen	1.8 (2.2)	1.6 (1.8)	−1.0–1.4	0.09
Difficulty remembering	0.9 (1.3)	2.0 (2.2)	−2.4–0.2	−0.85
Disturbed sleep	2.5 (2.3)	3.7 (2.8)	−2.8–0.4	−0.52
Nausea	1.0 (1.4)	1.4 (2.5)	−1.5–0.7	−0.28
Vomiting	0.4 (1.3)	0.4 (1.3)	--§	0

*The average of the 13 core and seven immunotherapy-specific symptom items.

†The average of the six interference items.

‡Significant at p<0.01 using paired t test.

§Could not be computed because the SE of the difference was 0.

ECOG, Eastern Cooperative Oncology Group; MDASI-Immunotherapy EPT, Immunotherapy for Early-Phase Trials module of the MD Anderson Symptom Inventory; PS, performance status.

Discussion

In this study, we tested the immunotherapy-specific module of the MDASI in patients receiving cancer treatment in an early-phase setting. It is critically important to understand the symptomatic toxicity of cancer therapies in the early-phase setting. The results provide strong psychometric evidence for the use of the MDASI-Immunotherapy EPT. The module’s severity and interference subscales exhibited high internal consistency reliability. These two subscales also exhibited excellent known-group validity at week 9 as evidenced by substantial differences in MDASI-Immunotherapy EPT ratings based on ECOG PS groupings. MDASI-Immunotherapy EPT items such as difficulty remembering, fever and/or chills, disturbed sleep, diarrhea, and swelling of hands, legs, or feet exhibited high sensitivity to changes in ECOG PS.

The MDASI modules offer advantages over other symptom-assessment measures. First, unlike with disease-specific symptom scales, data collected using MDASI modules can be used to compare symptom prevalence and severity across cancer types, which is necessary for epidemiological studies and clinical trials that may include patients with different types of cancer. By rank-ordering the severity of core symptom items for different types of cancer, researchers can identify most of the symptoms that are consistently burdensome for patients with cancer and thus compare symptom burdens across cancers. A similar argument can be made for different cancer treatments where symptom burden due to hematopoietic stem cell transplantation can be compared with immunotherapy. Second, because validation of any symptom assessment measure is costly and requires time and effort, each validation of an MDASI module provides incremental evidence of the validity, reliability, and sensitivity of the original MDASI instrument. The core items can be included in new modules with fewer of the expected psychometric steps typically used in instrument development with the exception of demonstration that the core items are relevant and sensitive to changes in the target patient group. Third, the MDASI is available in several linguistically and psychometrically validated foreign language versions. A study of symptom ratings made by patients with cancer in four countries using four different language versions of the MDASI16 provided evidence of minimal variations in MDASI symptom ratings due to culture and language.

The ability of a PRO instrument to detect change was one of the criteria set forth in the US Food and Drug Administration’s guidance on the use of PRO in labeling claims. In particular, regulatory agencies are interested to see whether changes in the PRO scores are related to changes in patients’ state of functioning. We show herein that the MDASI-Immunotherapy EPT items are sensitive to changes in ECOG PS (related to disease). Furthermore, a recent study demonstrated that MDASI symptom items, such as coughing, fatigue, and shortness of breath, were predictive of survival in patients with lung cancer.17 Another study showed that fatigue, distress/feeling upset, and sadness were significant predictors of how symptoms interfered with daily functioning.18 These studies further establish the ability of the MDASI symptom items to detect changes in a patient’s status.

Our study was limited in that most of the subjects were from a comprehensive cancer center. Therefore, they may not be representative of cancer patients in general. However, our study also had the strength of a fairly large sample of patients receiving immunotherapy alone or in combination with other cancer therapies.

Future research involving the MDASI-Immunotherapy EPT should include at least three directions. First, we developed this module of the MDASI primarily by reviewing the literature and seeking input from clinicians. However, regulatory agencies, particularly the US Food and Drug Administration, recommend incorporating extensive patient input via qualitative interviews in developing symptom assessment tools, especially if a labeling claim is desired. Hence, qualitative patient interviews are needed to confirm the content validity of the symptom items in the MDASI-Immunotherapy EPT. Second, identifying what constitutes meaningful change is needed to facilitate the interpretation of MDASI-Immunotherapy EPT scores. Ideally, an anchor-based approach with several clinical anchors should be considered, complemented by the distribution-based method of determining meaningful changes in the scores. Finally, different immunotherapies may produce different toxicities and that requires further work. For instance, the chimeric antigen receptor T-cell therapy is associated with possible side effects that include cytokine release syndrome and neurologic problems.

Conclusion

The MDASI-Immunotherapy EPT is a valid and reliable instrument for assessing the severity of symptoms and their interference with function in patients with cancer receiving immunotherapy. As more immunotherapy combinations become available, future analysis might include using immunotherapy combination type in the subgroup analyses.