Measuring treatment satisfaction in MS: Is the Treatment Satisfaction Questionnaire for Medication fit for purpose?

BACKGROUND: The Treatment Satisfaction Questionnaire for Medication (TSQM) was designed to assess patient treatment satisfaction in chronic diseases. Its performance has not been examined in multiple sclerosis (MS). The 14 items of the TSQM cover four domains: Effectiveness, Side Effects, Convenience, and Global Satisfaction.
OBJECTIVE: To evaluate performance of the TSQM in patients with relapsing MS, using data collected from the TENERE study (NCT00883337), in which 324 patients received oral teriflunomide or subcutaneous interferon beta-1a for ⩾48 weeks.
METHODS: Five measurement properties were examined using traditional psychometric methods: data completeness, scale-to-sample targeting, scaling assumptions, reliability (including test-retest), and construct validity (internal: item-level scaling success, confirmatory factor analysis, and exploratory factor analysis; external: convergence, discrimination, and group differences).
RESULTS: There were few (<2%) missing item data; domain scores could be computed for all patients. Score distributions were skewed toward higher satisfaction; two domains had marked ceiling effects. Scaling assumptions were supported. Internal consistency reliability was high (Cronbach's α > 0.90). Internal validity tests supported item groupings. Correlations supported convergent and discriminant construct validity; hypothesis testing supported group differences validity.
CONCLUSION: This investigation found the TSQM to be a useful tool, exhibiting good psychometric measurement properties in patients with relapsing MS in the TENERE study.

Introduction

Patient satisfaction with medication, resulting from factors such as the effectiveness, convenience (e.g. route of administration, dosing frequency), or side effects of the medication, is associated with better adherence to, and persistence with, treatment.[1,2] These findings, consistent across many diseases and clinical settings,[2] highlight the ongoing need to evaluate and improve patients’ treatment experience.

Many scales have been used to measure treatment satisfaction. Frequently, they are applied inconsistently and/or have not been evaluated in the specific disease setting being assessed.[1] In their roadmap to patient-focused outcome measurement in clinical trials (Figure 1), the US Food and Drug Administration (FDA) highlight the importance of examining clinical outcome assessments (COAs) in their context of use.[3] This is because COA suitability, as a measure of the concept of interest, is dependent upon the context of use. There is, therefore, no such thing as a “validated instrument.” European Union (EU) and US guidelines recommend that, if a measurement instrument is applied in a new disease setting, it is confirmed as fit for purpose in that context.[4,5]

Figure 1.

FDA roadmap to patient-focused outcome measurement in clinical trials.[3]

Reproduced with permission from the US Food and Drug Administration.

The TSQM was designed as a general measure of treatment satisfaction with medication. An initial pool of 55 candidate items was developed from focus groups of a panel of 500 patients with chronic disease (migraine, arthritis, hypertension, asthma, diabetes, psoriasis, hypercholesterolemia, and depression) and refined to 31 test items. Via a multistep iterative process, these were reduced to 14 final items (Supplementary Appendix 1) covering the majority of the variance in the test population.[6]

The TSQM has been examined, using standard psychometric methods, in several settings,[6,7] although not yet in MS. A study using data from 400 patients with cystic fibrosis treated with inhaled antibiotics concluded that the TSQM had good measurement properties in patients with this condition.[7] Using data from patients with various chronic diseases (see above), Atkinson et al.[6] applied psychometric tests to examine the performance of the TSQM, and also concluded that it possessed good psychometric properties. Some of their findings are noteworthy for patients with MS; they reported significant differences across the TSQM between different methods of treatment administration, with individuals using injectable therapies reporting low satisfaction and convenience.[6]

In patients with relapsing forms of multiple sclerosis (RMS), longer treatment duration has been linked with improved long-term outcomes,[8] so it is important to ensure patient treatment satisfaction in order to maximize persistence with treatment over the long term.[2] Teriflunomide, a once-daily oral immunomodulator approved for the treatment of relapsing-remitting MS, demonstrated consistent efficacy with a well-documented safety profile in randomized, placebo-controlled monotherapy studies in patients with RMS[9-11] and in patients with a first clinical episode suggestive of MS.[12] The phase 3 TENERE study (NCT00883337) compared teriflunomide with subcutaneous interferon beta-1a (scIFNβ-1a) in patients with RMS, and included the 14-item TSQM to measure patient satisfaction with either intervention.[13] The TSQM has been used in many studies of patient satisfaction in MS (reviewed by Ting et al.[14]), but to our knowledge, its measurement performance has yet to be examined comprehensively in the MS context of use.

Here, we examine the performance of the TSQM in patients with RMS using traditional psychometric methods to determine its fitness for purpose in the TENERE sample of patients with RMS.

Methods

Study design and participants

Details of the TENERE study are published elsewhere.[13] Briefly, patients aged 18 years or older with a diagnosis of RMS, an Expanded Disability Status Scale (EDSS) score of ⩽5.5, and no relapse(s) within the 30 days were randomized (1:1:1) to receive once-daily teriflunomide 14 or 7 mg, or scIFNβ-1a 44 µg thrice weekly. The study was designed to end 48 weeks after the last patient was randomized.[13] Patient satisfaction with treatment was assessed using the TSQM version 1.4.[13]

TSQM structure

The TSQM (version 1.4) comprises 14 items across four domains focusing on effectiveness (three items), side effects (five items), convenience (three items), and global satisfaction (three items) of the medication over the previous 2–3 weeks, or since the patient’s last use.[6] With the exception of item 4 (presence of side effects; yes or no), all items have five or seven responses, scored from one (least satisfied) to five or seven (most satisfied). The 7-item scales had a non-neutral midpoint, such that there were more positive response options than negative response options, to allow for precise information to be obtained at the upper end of the score distribution. Item scores are summed to give four domain scores, which are in turn transformed to a scale of 0–100. Item 4 was not included for scoring. If an item score is missing and half of the items in the domain are complete, domain scores may be imputed from the person-specific mean score of completed items.[15]

TSQM administration

The TSQM was administered every 12 weeks from Week 12 to Week 48, and every 24 weeks thereafter up to Week 96.[13] The TSQM was administered in patients’ local languages, using translations of the original questionnaire certified by translation agencies as linguistically equivalent (Supplementary Appendix 2).

TSQM evaluation

Five TSQM measurement properties were evaluated using TENERE data. Week 48 data were used unless otherwise indicated, as Week 48 was the timepoint used for the primary analysis of TENERE.[13]

Data completeness

To assess the extent to which the TSQM could be used successfully in TENERE (i.e. how acceptable the questionnaire is to test subjects), we computed item-level missing data for randomized patients, and the proportions of patients for whom domain scores could be computed. Fewer missing data indicate greater acceptability.[16]

Scaling assumptions

We assessed the legitimacy of summing TSQM item scores from TENERE, without weighting or standardization, to generate domain scores. Summing is considered legitimate when items of a domain are broadly parallel and contribute similarly to the construct being measured. These requirements are considered satisfied when items have similar means and variances,[17] and item-to-domain score correlations, corrected for overlap, exceed 0.30.[18]

Scale-to-sample targeting

To examine the match between the potential range measured by the TSQM and the observed range measured in TENERE, we examined domain score distributions to ascertain the extent to which these met the recommended criteria of, spanning the available scale range,[19] mean scores located near the scale midpoint,[20] not being excessively skewed (skewness < 1.0),[16] and floor and ceiling effects (proportions of patients with minimum and maximum scores, respectively) <20%.[21]

Reliability

Multiple reliability indicators are available to evaluate the extent to which scale scores are free from random error. We examined internal consistency (corrected item-total correlations, Cronbach’s α, and homogeneity coefficients (mean item − item correlations for each domain)), test–retest reproducibility (agreement between scores at separate time points), and standard errors of measurement. Reliability is considered adequate for group comparisons when corrected item-total correlations are >0.30,[22] Cronbach’s α >0.80,[23] and homogeneity coefficients >0.30.[20]

The relatively long measurement interval in TENERE (⩾12 weeks) could allow change over time to confound interpretation of test–retest estimates. Therefore, a conservative estimate of test–retest reproducibility was approximated by comparing TSQM values at Weeks 24 and 48 for patients with stable disease, defined as patients without relapses for the duration of treatment. A random effects model intra-class correlation coefficient was calculated using values generated by a repeated measures analysis of variance (ANOVA), and a score >0.80 was considered acceptable.[23]

Standard errors of measurement, computed as standard deviation × √[1 − reliability coefficient] were used to interpret reliability estimates as confidence intervals (CIs) around scores (95% CI = score ± 1.96 × standard error of measurement), using Cronbach’s α as the reliability coefficient. Low standard errors of measurement demonstrate low measurement error.[24]

Validity

To assess the extent to which the TSQM measures the constructs it purports to measure, we first tested internal construct validity (the extent to which items of the TSQM are grouped correctly into domains) as a prerequisite for interpretation of external construct validity tests (which provide more direct information on the constructs measured). Three examinations of internal construct validity were undertaken. Item-level convergent and discriminant validity were tested by computing scaling success rates. A definite scaling success was scored when an item’s correlation with its own domain (corrected for overlap) was significantly higher (>2 × standard error) than its correlations with another domain. Exploratory factor analysis (EFA), performed as a maximum likelihood factor analysis, was used to identify factors that explain the maximum amount of variance. Confirmatory factor analysis (CFA) was performed as a hypothesis-driven approach to further understand shared variance between variables due to factors. Goodness-of-fit indices were assessed against predefined criteria for good fit: Root Mean Square Error of Approximation <0.08, Normed Fit Index >0.9, Goodness-of-Fit Index >0.9, Adjusted Goodness-of-Fit Index >0.9, and standardized root mean square residual < 0.05.

Two examinations of external construct validity of the TSQM were undertaken. First, scale-level convergent and discriminant construct validity were tested by examining the extent to which the direction, magnitude, and pattern of correlations between variables were consistent with expectation. We examined correlations between TSQM domains and baseline patient characteristics (age, gender, EDSS, and Fatigue Impact Scale (FIS) scores), hypothesizing that these correlations would be lower than the TSQM between-domain correlations. Second, group differences construct validity was tested using score differences between responders and non-responders on a range of clinical outcomes. The outcomes were selected based on measured parameters that we hypothesized would be likely to explain a clinical difference, and are detailed in Supplementary Table 1. Group mean score differences were expressed in terms of statistical (p value from independent samples ANOVA) and clinical significance (Cohen’s d; effect size (ES)). ES was interpreted using Cohen’s criteria: ⩾0.2 to <0.5 for a small difference; ⩾0.5 to ⩽0.8 for a moderate difference; and >0.8 for a large difference.[25]

Results

Study participants

Patient characteristics in the TENERE study (Table 1)[13] were generally similar to those of patients in other phase 3 studies of teriflunomide[9,10] and other oral disease-modifying treatments for RMS,[26-29] albeit with a slightly lower mean EDSS score at baseline in TENERE.

Table 1.

Baseline demographics and disease characteristics.

	sc IFN β-1a (n = 104)	Teriflunomide 7 mg (n = 109)	Teriflunomide 14 mg (n = 111)
Age, years, mean (SD)	37.0 (10.6)	35.2 (9.2)	36.8 (10.3)
Female, n (%)	71 (68.3)	70 (64.2)	78 (70.3)
Caucasian, n (%)	104 (100)	109 (100)	111 (100)
Time since first symptoms of MS, years, mean (SD)	7.7 (7.6)	7.0 (6.9)	6.6 (7.6)
No. of relapses within previous year, mean (SD)	1.2 (1.0)	1.3 (0.8)	1.4 (0.8)
Relapsing-remitting MS, n (%)	104 (100)	109 (100)	108 (97.3)[a]
Use of DMT in previous 2 years, n (%)	25 (24.0)	23 (21.1)	13 (11.7)
Baseline EDSS score, mean (SD)	2.0 (1.2)	2.0 (1.2)	2.3 (1.4)
Baseline FIS score, mean (SD)	34.2 (32.7)	39.5 (34.8)	42.5 (37.8)

DMT: disease-modifying therapy; EDSS: Expanded Disability Status Scale; FIS: Fatigue Impact Scale; IFN: interferon; MS: multiple sclerosis; sc: subcutaneous; SD: standard deviation.

Randomized population (n = 324).

Secondary progressive MS (n = 1); progressive relapsing MS (n = 2).

Data completeness

TSQM data completeness in TENERE was good. Each item was missing a response in fewer than 2% of patients (n = 324; range: 0.3%–1.9%; Table 2). Domain scores could be computed for all participants (Supplementary Table 2).

Table 2.

Item-level analyses of TSQM.

Domain	Item		Response categories, n	Patients with missing data, n (%)[a]	Correlation with domain[b,c]				Scaling success rate, %[d]
					Effectiveness	Side effects	Convenience	Global satisfaction
Effectiveness	Q1	Satisfaction with prevention/treatment	7	3 (0.9)	0.90	0.19	0.27	0.54	100
	Q2	Satisfaction with symptom relief	7	5 (1.5)	0.88	0.24	0.27	0.56
	Q3	Satisfaction with time to start working	7	5 (1.5)	0.89	0.21	0.31	0.56
Side effects	Q4	Side effect presence[e]	2	6 (1.9)	NA	NA	NA	NA	100
	Q5	Bother from side effects	5	4 (1.2)	0.19	0.76	0.44	0.26
	Q6	Side effects interference with physical function	5	4 (1.2)	0.25	0.83	0.46	0.26
	Q7	Side effects interference with mental function	5	2 (0.6)	0.23	0.66	0.42	0.29
	Q8	Impact of side effects on satisfaction	5	2 (0.6)	0.23	0.71	0.50	0.34
Convenience	Q9	Treatment easy to use	7	1 (0.3)	0.24	0.44	0.83	0.39	100
	Q10	Easy planning of use	7	1 (0.3)	0.23	0.42	0.82	0.41
	Q11	Intake convenience	7	2 (0.6)	0.36	0.41	0.82	0.52
Global satisfaction	Q12	Confidence in benefits	5	2 (0.6)	0.51	0.19	0.36	0.81	100
	Q13	Balance between good and bad things	5	2 (0.6)	0.49	0.23	0.38	0.83
	Q14	Global satisfaction	7	3 (0.9)	0.59	0.33	0.55	0.80

NA: not applicable; SE: standard error; TSQM: Treatment Satisfaction Questionnaire for Medication (version 1.4).[6]

Randomized population (n = 324).

Patients from intent-to-treat population with complete TSQM domain information at Week 48 (n = 243–246).

Item-own domain correlations corrected for item overlap (bold values).

Percentage of correlations where item-own domain correlation (corrected for overlap) exceeds item—other domain correlation by more than 2 × SE (where SE = 1/√n).

Dichotomous item, not scored.

Scaling assumptions

Scaling assumptions were satisfied for all four domains. Item mean scores and variances were similar (Supplementary Table 2) and all item total correlations (corrected for overlap) exceeded 0.30 (Table 2). This supports, for each domain, the summing of item scores to generate domain scores without standardization or weighting.

Scale-to-sample targeting

For all domains except effectiveness, scores did not span the whole scale range, demonstrating skewing toward high scores (Table 2). Mean and median scores exceeded the scale midpoint (50). High mean scores accompanied by ceiling effects (defined as maximum scores in >20% of patients) were particularly marked for side effects (mean score: 90.1, 72% of patients with maximum score), and convenience (mean score: 82.2, 38% of patients with maximum score). Both domains had notably higher ceiling effects with oral treatment (teriflunomide) than with injectable treatment (scIFNβ-1a). There were no notable floor effects, with small percentages of patients with minimum scores (minimal satisfaction) in each domain. Together, these high scores suggest good overall treatment satisfaction that was generally higher with teriflunomide than with scIFNβ-1a.[13,30]

Reliability

Internal consistency reliability was high for all domains, with Cronbach’s α >0.90 and all homogeneity coefficients >0.75; corresponding standard errors of measurement were thus relatively small. Test–retest reproducibility coefficients exceeded 0.70 for three domains (side effects, convenience, and global satisfaction) indicating adequate reproducibility given that these were likely conservative estimates.[23] The coefficient for effectiveness was low (0.44).

Validity

Tests of internal construct validity supported the proposed item groupings. Definite scaling success rates for all four domains were 100% (Table 2). EFA grouped the 13 scoring items into four factors with item content equivalent to the four TSQM domains (Supplementary Table 3). CFA (Figure 2) also supported TSQM item groupings; at Week 48, Goodness-of-Fit indices met the predefined criteria. The largest contribution to global satisfaction came from effectiveness (standardized estimate for association was 0.63), followed by convenience (0.54) and side effects (0.32).

Figure 2.

Confirmatory factor analysis of the TSQM.

TSQM: Treatment Satisfaction Questionnaire for Medication (version 1.4).[6]

Ovoids represent unobserved variables (domains); rectangles represent observed variables (items); arrows represent the hypothesized links between the variables; parameters relative to each arrow are standardized estimates of the strength of association between the linked variables.

Root mean square error of approximation, 0.067; Normed Fit Index, 0.958; Goodness-of-Fit Index, 0.925; Adjusted Goodness-of-Fit Index, 0.884; Standardized Root Mean Square Residual, 0.044.

Tests of external construct validity supported the constructs measured by the domains. Correlations among TSQM domains were consistent with expectation, and supported the four domains as measures of related but different constructs (Supplementary Table 2). As in the CFA, perceived effectiveness was linked with global satisfaction (correlation coefficient, 0.69). Correlations between TSQM domains and age, gender, EDSS, and FIS were low (ranging from 0.01 to −0.31), indicating treatment satisfaction was not biased by these variables (Supplementary Table 2).

As hypothesized, there was a statistically significant (p ⩽ 0.05) and clinically meaningful (ES >0.3) relationship between each TSQM domain and the clinical outcomes tested (Table 3). For example, the minimal number of patients with adverse events leading to treatment discontinuation had a statistically (p < 0.0001) and clinically (ES, 3.24) significantly (reduced side effects domain score (31.3; n = 2) compared with patients who did not (90.6; n = 243). There were also highly statistically significant (p < 0.0001) relationships between the convenience domain and relevant clinical outcomes. Treatment received (teriflunomide/scIFNβ-1a, used as a proxy for mode of administration) showed the strongest relationship (ES = 1.74) with convenience.

Table 3.

Relationships between clinical outcomes and TSQM domains at Week 48.

Domain	Clinical outcome	Patients with outcome		Patients without outcome		Effect size, Cohen’s d	p value[a]
		n	Score, mean (SD)	n	Score, mean (SD)
Effectiveness	Treatment failure[b]	51	61.2 (19.5)	192	68.8 (22.4)	0.35	0.028
	Confirmed relapse	50	61.6 (19.6)	193	68.7 (22.4)	0.33	0.041
Side effects	AEs leading to treatment discontinuation	2	31.3 (17.7)	243	90.6 (18.3)	3.24	0.020
	Nervous system disorders	92	86.1 (21.5)	153	92.6 (17.0)	0.38	0.009
	General disorders or administration-site conditions[c]	73	82.0 (24.6)	172	93.6 (15.0)	0.63	<0.0001
Convenience	Treated with sc IFN β-1a[d]	74	63.2 (19.1)	176	89.8 (13.4)	1.74	<0.0001
	General disorders or administration-site conditions[c]	74	74.0 (22.1)	172	85.8 (17.0)	0.63	<0.0001
Global satisfaction	Treatment failure[b]	52	63.2 (21.2)	193	72.2 (20.7)	0.43	0.006
	Confirmed relapse	51	63.6 (21.3)	194	72.1 (20.8)	0.41	0.011

AE: adverse event; ANOVA: analysis of variance; IFN: interferon; sc: subcutaneous; SD: standard deviation; TSQM: Treatment Satisfaction Questionnaire for Medication (version 1.4).[6]

All relationships with p < 0.05 for patients with complete TSQM domain information at Week 48.

p value from ANOVA.

Confirmed relapse or permanent treatment discontinuation for any reason.

General disorders and administration-site conditions were mainly driven by influenza-like illness.

Specific outcomes for convenience are difficult to identify in a randomized-controlled trial, and we observed a relationship with AEs related to mode of administration (injectable sc IFN β-1a vs oral teriflunomide) using treatment received as a proxy.

Discussion

This analysis provided a comprehensive evaluation, using traditional psychometric methods, of the extent to which the 14-item version of the TSQM is a fit-for-purpose measure of treatment satisfaction in the TENERE study of patients with RMS. Overall, we found that the TSQM exhibits good measurement properties and met the requirements of traditional psychometric tests. Specifically, we found that item scores could be summed without weighting or standardization to form total scores that were reliable, and for which evidence supported their validity as measures of different aspects of treatment satisfaction.

Analysis of scale-to-sample targeting identified a potential limitation of the TSQM for the RMS context of use. Marked ceiling effects for the side effects and convenience domains were observed in the teriflunomide-treated group. This may be a reflection of high levels of patient satisfaction with teriflunomide treatment, which is supported by the significant and clinically meaningful improvement in TSQM score for the teriflunomide 14 mg group versus the scIFNβ-1a 44 µg group on the side effects and convenience domains in TENERE.[13,30] Preliminary results from the Teriflunomide Patient-Reported Outcomes (Teri-PRO; NCT01895335) study of real-world teriflunomide use also indicate that patient satisfaction, as measured by the TSQM, increases when patients switch their disease-modifying therapy to teriflunomide.[31] Furthermore, an analysis of the TSQM in patients with chronic diseases found that injectable modes of administration were associated with lower TSQM scores, which could again suggest that scores for teriflunomide-treated patients are expected to be higher than those of patients treated with scIFNβ-1a.[6] The skewed mean scores and high ceiling effects we observed may indicate that the TSQM limited the possible measurement of satisfaction in these patients, with the “true” satisfaction of the teriflunomide-treatment group likely to be higher than that actually measured; the differences between scIFNβ-1a and teriflunomide may, therefore, be larger than measured.

In this analysis, internal consistency indicators (Cronbach’s α and homogeneity coefficients) were very high, particularly given the small numbers of items in each domain. This implies the items in each domain were closely related and may suggest possible item redundancy.[32] However, indicators of internal consistency may also be elevated spuriously by ceiling or floor effects, and we have noted skewed score distributions in our analysis. Reanalysis of reliability could help to determine whether there is true item redundancy. Although traditional psychometric methods are widely used, they do have recognized limitations.[33] In this instance, reliability analyses using the person separation index, generated by the more modern Rasch measurement theory analysis,[34] might be informative.

Although the intervals between TSQM data collection were too long to permit a robust evaluation of test–retest reproducibility, our conservative approximations implied that high reproducibility is to be expected for three domains (global satisfaction, convenience, and side effects). It is difficult to know how best to interpret the value of 0.44 for effectiveness, and this merits further investigation.

CFA implied that global satisfaction with treatment within the TENERE study population was driven primarily by effectiveness, followed by convenience and side effects. This is consistent with studies of treatment adherence in patients with MS, which have identified treatment efficacy as important and lack of efficacy as a key reason for treatment discontinuation,[35,36] and also with findings in other diseases, which showed global satisfaction was most strongly linked with effectiveness.[6] It would be of interest to explore how relapses and disability progression are linked with changes in TSQM, and if these clinical changes in turn affect its measurement properties.

Although the patient-unblinded nature of TENERE may have influenced patient satisfaction ratings,[13] we do not expect it to influence the empirical measurement performance of the TSQM, as analyzed in this study.

An important next step would be to examine the item content of the TSQM, to optimize it for the RMS patient population. Qualitative research might identify new items that extend the measurement range of the TSQM, reduce ceiling effects, and advance measurement of treatment satisfaction in patients with RMS.

To our knowledge, this is the first time the performance of the TSQM has been evaluated in a sample of patients with RMS. While, as noted, evaluation in a single study population does not confirm measurement performance in all contexts, our comprehensive analysis supports the TSQM as a fit-for-purpose measure of treatment satisfaction in TENERE. Based on this, it seems reasonable to conclude that TSQM is likely to be appropriate for use in studies of disease-modifying therapies for patients with RMS. Indeed, the tool is being used as an outcome measure to provide further understanding of patient experiences of teriflunomide treatment in routine clinical practice in ongoing phase 4 studies,[31] and it is our intention to use data from such studies to perform a follow-on evaluation of TSQM performance in the context of use of real-world patients with RMS. However, as with all instruments, detailed analysis demonstrates room for improvement. Here, the suboptimal scale-to-sample targeting implies that treatment satisfaction may be underestimated by the TSQM in this context of use, and modification of the TSQM may overcome this limitation.