Evaluation of multiple sclerosis disability outcome measures using pooled clinical trial data.

OBJECTIVE: We report analyses of a pooled database by the Multiple Sclerosis Outcome Assessments Consortium to evaluate 4 proposed components of a multidimensional test battery.
METHODS: Standardized data on 12,776 participants, comprising demographics, multiple sclerosis disease characteristics, Expanded Disability Status Scale (EDSS) score, performance measures, and Short Form-36 Physical Component Summary (SF-36 PCS), were pooled from control and treatment arms of 14 clinical trials. Analyses of Timed 25-Foot Walk (T25FW), 9-Hole Peg Test (9HPT), Low Contrast Letter Acuity (LCLA), and Symbol Digit Modalities Test (SDMT) included measurement properties; construct, convergent, and known group validity; and longitudinal performance of the measures individually and when combined into a multidimensional test battery relative to the EDSS and SF-36 to determine sensitivity and clinical meaningfulness.
RESULTS: The performance measures had excellent test-retest reliability and showed expected differences between subgroups based on disease duration and EDSS level. Progression rates in detecting time to 3-month confirmed worsening were lower for T25FW and 9HPT compared to EDSS, while progression rates for LCLA and SDMT were similar to EDSS. When the 4 measures were analyzed as a multidimensional measure rather than as individual measures, progression on any one performance measure was more sensitive than the EDSS. Worsening on the performance measures analyzed individually or as a multidimensional test battery was associated with clinically meaningful SF-36 PCS score worsening, supporting clinical meaningfulness of designated performance test score worsening.
CONCLUSION: These results support the use of the 4 proposed performance measures, individually or combined into a multidimensional test battery as study outcome measures.

Recognition of the limitations of existing measures of multiple sclerosis (MS)–related disability[1] led to development of the Multiple Sclerosis Functional Composite (MSFC) as an alternative clinical outcome measure.[2,3] The MSFC integrated 3 quantitative performance measures: Timed 25-Foot Walk (T25FW), 9-Hole Peg Test (9HPT), and Paced Auditory Serial Addition Test (PASAT).[2,3] Component test scores were normalized by conversion to z scores using a reference sample and averaged to create the MSFC score. Subsequent utilization of the MSFC in clinical trials and other studies highlighted several advantages: high reliability; coverage of cognition, which is not adequately captured by other outcome measures; a single score on a continuous scale; ease of use; and capacity to be administered by a trained technician rather than a physician. However, while utilization of z scores offered statistical advantages, it impeded MSFC acceptance and adoption. Both regulatory agencies and clinicians had difficulty interpreting the clinical meaningfulness of the MSFC score.

Subsequent research provided guidance around thresholds of MSFC measure variability and clinically meaningful change.[4-7] In addition, researchers recommended adding Low Contrast Letter Acuity (LCLA), a test of visual impairment validated in MS,[8,9] and replacing the PASAT with the Symbol Digit Modalities Test (SDMT).[10] In this article, we report the initial analyses of a pooled database to assess measurement properties; sensitivity; construct, convergent, and known group validity; and clinical meaningfulness of 4 performance tests (T25FW, 9HPT, LCLA, and SDMT) individually and combined into a multidimensional outcome measure of MS-related disability.

Methods

Study design

The Multiple Sclerosis Outcome Assessments Consortium (MSOAC) was established in 2012 to accelerate development and validation of improved clinical outcome measures of MS-related disability.[11,12] The organization of MSOAC and overall approach to develop and validate clinical outcome measures of MS-related disability have been reported previously.[13] Frequent interactions with the European Medicines Agency and US Food and Drug Administration contributed to MSOAC's research approach.[13] After executing data use agreements, MSOAC obtained prospectively acquired patient-level data from 16 clinical trials including 14,370 participants and combined these into a single database. Fourteen of these trials, comprising 12,776 patients, were used for the analyses reported here.

To allow aggregation of data from distinct datasets, a common data standard was developed through the Coalition for Accelerating Standards and Therapies, and data from each clinical trial were remapped to the Clinical Data Interchange Standards Consortium data standard to create a single pooled data set.[14] The standardized data comprising the control and treatment arms of clinical trials formed the MSOAC database. Data elements included demographics, MS disease characteristics, treatment, relapse data, Expanded Disability Status Scale (EDSS) score, performance measures, and patient-reported outcomes.[13] The MSOAC database does not include the MRI data.

Outcome measures

These analyses focused on 4 performance measures and their relation to other measures in the database: T25FW (short distance walking speed to measure ambulation),[15] 9HPT (dexterity and upper extremity motor function),[16] LCLA (vision),[9] and SDMT (cognitive processing speed and sustained attention).[10] Data on PASAT are presented elsewhere.[17] The EDSS is an ordinal scale ranging from 0 to 10 based on the severity of findings on the neurologic examination, walking ability, and ability to carry out activities of daily living, with higher scores indicating worse disability.[18] The Short Form–36 (SF-36) is a 36-item questionnaire that includes 8 multi-item health concepts (Physical Functioning, Role-Physical, Bodily Pain, General Health, Vitality, Social Functioning, Role-Emotional, and Mental Health).[19] Scores are a mean of subsetted questions and range from 0 to 100; higher scores indicate better health-related quality of life (HRQoL). The SF-36 has 2 summary scales, the Physical Component Summary (PCS) and the Mental Component Summary (MCS), whose calculation produces a T-score, with a mean score of 50 and SD of 10, representing the reference score for the US general population. The Beck Depression Inventory (BDI) is a 21-item self-report measure of depression with scores ranging from 0 to 62 and higher score indicating more severe depression symptoms.[20]

For these analyses, worsening was defined as follows: T25FW (20% increase),[15] 9HPT (20% increase),[16] LCLA with 2.5% contrast (20% or 7-letter decrease),[9] SDMT (4-point decrease),[10] EDSS (baseline score 0: 1.5-point increase, baseline score 1.0–5.5: 1-point increase, baseline score ≥6.0: 0.5-point increase),[21] and SF-36 PCS Score (5-point worsening).[22] For all measures except SF-36 PCS, the worsening had to be sustained for at least 3 months.

Statistical methods

No imputation was done for missing data other than for participants unable to complete the T25FW or 9HPT because of disability. Following convention, imputation for patients unable to perform was 180 seconds for T25FW and 300 seconds for 9HPT.[23] The MSFC administration and scoring manual states that for T25FW testing patients should use their usual assistive devices and an effort should be made to use the same device over the course of the study. Summary scores of the SF-36 MCS and PCS were calculated using standard methods, which provide T-scores for analysis. For the SF-36 8 health concept scores, Quality Metrics Health Outcomes Scoring Software was utilized. The maximum data recovery method was used to handle missing data. If any individual item was missing for the BDI score, the total score was not calculated for that participant and time point.

Test–retest reliability was assessed by intraclass correlation coefficient (ICC) of all administrations of each test (2–6 compared with test 1) based on periods in which patient status on the EDSS did not change and not exceeding 6 months from baseline.[24] Correlations among the EDSS and performance tests were assessed by Spearman rank correlation coefficient. Time to confirmed clinically meaningful worsening was analyzed by Kaplan-Meier methods. Cohen kappa coefficient was used to assess agreement in worsening in different disability measures. Baseline outcome measure scores were compared. The baseline score for each performance measure was compared between the groups of patients based on disease duration and EDSS score using an analysis of variance model adjusting for age in 5-year age bands.

Data availability

Per data use agreements, analyses were done through a contract research organization (Premier Research) under the oversight of the Critical Path Institute. Pooled data were not available to individual sponsors or academic members, although per agreement, placebo data were made available publicly.[25] Consortium members contributed to development of the statistical analysis plan, and had access to results from all analyses. The authors had full access to all the data generated in this fashion.

Results

Table 1 summarizes the data available, baseline demographics, disease characteristics, EDSS score, performance test results, and participant self-reported measures. Overall, the population was relatively young with a recent diagnosis of MS, predominantly relapsing-remitting (RR) course, and mild disability. Although fewer studies included LCLA, SDMT, and self-reported measures, substantial data were available for all outcome measures.

Table 1

Baseline characteristics

The frequency distributions of the T25FW and 9HPT were positively skewed and showed floor effects, with scores tending to be clustered at shorter times (figure 1). Both possessed the ability to distinguish gradations of performance in the middle of the scale. LCLA distribution appeared mildly negatively skewed without floor or ceiling effects. SDMT scores showed no evidence of skewing, or floor or ceiling effects.

Figure 1

Distribution of performance measure scores at baseline

(A) Timed 25-Foot Walk (T25FW) (seconds). (B) 9-Hole Peg Test (9HPT) (seconds). (C) Low Contrast Letter Acuity (LCLA) with 2.5% contrast (number correct). (D) Symbol Digit Modalities Test (SDMT) (number correct).

Table 2 summarizes trends over the first 6 assessments for the performance tests. T25FW, 9HPT, and LCLA tended to worsen over time and showed minimal or no practice effects, while the SDMT demonstrated modest practice effects. Test–retest reliability was estimated by calculating the ICC, accounting for practice effects where needed. All of the measures showed good test–retest reliability, though the ICC for T25FW was somewhat lower (0.71) compared to the other tests (0.84–0.88).

Table 2

Practice effects and test–retest reliability of performance measures with tests 2–6 each compared to test 1

To compare sensitivity to change of the performance measures with EDSS, time from baseline to 3-month confirmed worsening over 24 months was analyzed (figure 2). The study populations available for each comparison differed, leading to differing proportions with 3-month confirmed worsening on EDSS. Using a 20% threshold for T25FW, 6.5% worsened compared to 20.2% on EDSS. Using a 20% threshold for 9HPT, 2.9% worsened compared to 20.2% on EDSS. Using 7-point threshold for LCLA, 13.1% worsened compared to 16.1% on EDSS. Using 4-point threshold for SDMT, 15.0% worsened at 18 months compared to 11.4% on EDSS at 18 months and 14.5% at 24 months. Thus, progression rates were lower for T25FW and 9HPT compared to that of EDSS, while progression rates for LCLA and SDMT were similar to or higher than compared to EDSS. When the performance tests were combined into a multidimensional outcome measure, the proportion of participants worsening on any one performance test was substantially greater than the proportion worsening on EDSS. When worsening on 2 performance tests was required, sensitivity to disability progression was somewhat reduced compared to the EDSS. The progression events defined by the performance tests were weakly associated with or independent of those defined by the EDSS: T25FW (Cohen κ = 0.02, 95% confidence interval [CI] −0.00 to 0.03), 9HPT (κ = 0.00, 95% CI −0.01 to 0.01), LCLA (κ = 0.11, 95% CI 0.08 to 0.14), and SDMT (κ = −0.02, 95% CI −0.06 to 0.02).

Figure 2

Kaplan-Meier graphs of time to 3-month confirmed disability worsening on performance measures compared to Expanded Disability Status Scale

(A) Timed 25-Foot Walk (T25FW). (B) 9-Hole Peg Test (9HPT). (C) Low Contrast Letter Acuity (LCLA) with 2.5% contrast. (D) Symbol Digit Modalities Test (SDMT). (E) Any 1 or any 2 performance measures.

To investigate construct and convergent validity, correlations between the performance measures and EDSS were analyzed (table 3). The T25FW and 9HPT correlated strongly with one another and demonstrated the strongest correlation to the EDSS relative to other performance measures. LCLA and SDMT were weakly correlated to the other performance measures and EDSS. Between the two, the SDMT had somewhat stronger correlation to the EDSS. Cross-sectional correlations among outcomes at baseline were notably stronger than the correlations among changes from baseline to endpoint, which had a similar pattern of correlative strength (T25FW > 9HPT > SDMT > LCLA), but wholly weaker in magnitude.

Table 3

Correlations between outcome measures

Known group validity was assessed as a function of disease duration and disability level (table 4). At baseline, values for all 4 performance measures were better in participants with MS of shorter duration (<10 years since symptom onset) compared to those with disease of longer duration (≥10 years). Similarly, the results on all 4 performance tests were better in participants with lower EDSS scores (0–3.5) vs those with higher EDSS scores (4.0–10).

Table 4

Known group analysis of baseline values based on disease duration and disability level

To explore clinical meaningfulness, correlations were calculated between performance measures and participant self-reported measures of HRQoL and depression (table 3). At baseline, T25FW, 9HPT, and SDMT correlated moderately with SF-36 PCS and significantly but weakly with MCS and BDI. LCLA correlated weakly with SF-36 PCS and MSC, and BDI. Correlations between change baseline to endpoint in the performance measures and change on SF-36 PCS or MCS, or BDI were generally not significant and weak at best (table 3). Among participants with worsening from baseline to endpoint on the T25FW, 9HPT, or SDMT, the mean SF-36 PCS also worsened (p < 0.001, p < 0.001, and p = 0.0308, respectively) (table 5). Similarly, among participants who showed baseline to endpoint worsening on the T25FW, 9HPT, or SDMT, the proportions of participants with 5-point PCS worsening on SF-36 PCS were greater. Nonsignificant trends were seen for mean PCS change and the proportion with 5-point PCS change among participants who did or did not experience baseline to endpoint worsening on LCLA. Mean PCS worsened among participants with baseline to endpoint worsening in each of 2 groups: those with worsening on any one measure and those worsening on 2 or more performance measures. The SF-36 PCS was improved or stable, respectively, among participants who did not worsen on 1 or on 2 or more performance measures. Similarly, the proportions of participants with 5-point SF-36 PCS worsening were greater among participants who showed baseline to endpoint worsening on 1 or on 2 or more performance measures, suggesting that worsening on any single performance measure was clinically relevant.

Table 5

Change in Short Form–36 Physical Component Summary (PCS) in participants with and without worsening on Expanded Disability Status Scale (EDSS) and performance measures

Discussion

We present analyses to characterize the measurement properties; sensitivity; construct, convergent, and known group validity; and clinical meaningfulness of 4 performance measures—T25FW, 9HPT, LCLA, and SDMT—to permit use individually or combined into a multidimensional test battery as a primary or co-primary outcome measure. We have assessed the components of the proposed multidimensional test battery in relation both to the EDSS and self-reported measures of health-related quality of life and depression. These results, based on a database of 14 datasets comprising 12,776 participants, represent the largest pooled analysis of prospectively acquired clinical trial data in MS to date. The demographics of the pooled dataset largely reflect the type of patents historically enrolled in MS clinical trials, for which trials in RRMS have predominated.

The distributions of the T25FW and 9HPT were positively skewed, and both measures demonstrated floor effects. The potential for a high proportion of patients to perform these measure as well as can be performed by a healthy control can result in reduced ability to distinguish gradations of performance at the lower end of the scale (demonstrated by far left peaks in figure 1, A and B). Baseline LCLA and SDMT scores were more normally distributed, without evidence of floor or ceiling effects. The T25FW, 9HPT, and LCLA showed no clearcut evidence of practice effects. As is typical of most cognitive measures, the SDMT exhibited some practice effects, but these appeared not to affect the normality of the SDMT's frequency distribution. All 4 performance measures demonstrated good test–retest reliability, indicating they yield reproducible scores if there is no change in the participant's condition. As a result, changes in a score can be assumed to be due to the participant's condition rather than measurement variability. These results support the advantageous measurement properties of the 4 performance measures.

These results provide a cautionary note regarding the population for which these measures will be most useful. The majority of participants represented in the pooled dataset had RRMS with relatively mild disability. In turn, the T25FW and 9HPT exhibited floor effects, which may explain the decreased sensitivity of 3-month confirmed worsening of T25FW and 9HPT compared to EDSS. More sensitive tests may be needed in studies enrolling participants with mild gait and upper extremity impairments.[21] One might question whether confirmed worsening on EDSS at the low end of the scale in patients with MS with mild impairment represents increasing disability, or simply new signs on the neurologic examination.

At baseline, the T25FW and 9HPT had stronger correlations with the EDSS and with each other than with the other performance measures. These results support the construct validity of the T25FW and 9HPT, as both are measures of physical functions that overlap with the EDSS in its lower range (EDSS 0–4.0) as seen in this population. In comparison, LCLA and SDMT correlated less strongly with EDSS and the other performance measures, supporting their additive value, to assess functions not captured by the other performance measures and EDSS. Compared to correlations at baseline, all the correlations for change from baseline to endpoint were much weaker. Cohen kappa coefficients showed that the confirmed worsening events defined by the 4 performance measures were largely independent of those defined by EDSS. Taken together, these results suggest that the 4 performance measures assess overlapping but somewhat different aspects of disability and disability worsening than does the EDSS.

All 4 performance measures were worse in participants with longer MS disease duration and with worse disability measured by EDSS, supporting known group validity. Exploratory analyses were undertaken to assess the clinical meaningfulness of worsening on the performance measures using the SF-36 PCS as an anchor. SF-36 PCS correlated moderately at baseline with T25FW, 9HPT, and SDMT and weakly with LCLA. SF-36 MCS and BDI correlated weakly with all 4 performance measures at baseline. Group aggregate changes from baseline to endpoint in the performance measures and self-report measures correlated weakly or not at all when directionality was not considered. Importantly, for participants experiencing confirmed worsening from baseline to endpoint on the T25FW, 9HPT, and SDMT, the SF-36 PCS was significantly worse, and the likelihood of a 5-point worsening on SF-36 PCS, considered clinically important,[22] was significantly greater. The LCLA results mirrored these findings but with statistically nonsignificant trends. These results indicate that the T25FW, 9HPT, and SDMT assess clinically meaningful aspects of MS-related disability and that the proposed thresholds for clinically meaningful change for each are reasonable. The nonsignificant trend of concomitant worsening in the LCLA and SF-36 PCS provides some support for the clinical meaningfulness of 7-letter change in LCLA. The clinical meaningfulness of the 7-letter LCLA worsening has been demonstrated previously, using validated visual quality of life scales.[9]

Limitations of this work include the availability of somewhat fewer data for LCLA, SDMT, and self-reported measures. Relatively few datasets contained all 4 performance measures and EDSS, limiting the analyses of their relative sensitivity, including in RR vs progressive MS, as a function of baseline disability level, and to capture worsening disability independent of relapse. In addition, our ability to fully explore clinical meaningfulness of the performance measures using self-report measures also was restricted by lack of self-report measures used across studies beyond SF-36 and BDI. Other measures of self-report have been applied to the MS population, but these analyses were limited by the surveys in the existing dataset, that is, the SF-36. Also, although the dataset included the full range of disability, the majority of participants had RRMS and relatively mild disability, with median EDSS of 2.5, reflecting the over-representation of clinical trials in RRMS in the MS field. This point may limit a full understanding of the performance tests in more disabled, progressive populations. Finally, for these analyses, we pooled treatment groups and focused on 3-month confirmed disability worsening (rather than 6-month), which could have affected the results.

These results confirm the advantageous measurement properties of the T25FW, 9HPT, LCLA, and SDMT and support their construct, convergent, and known group validity, and sensitivity, particularly when combined into a multidimensional test battery. The associations with established measures of disability (EDSS) and HRQoL (SF-36) indicate that they evaluate clinically meaningful aspects of MS-related disability. These findings support the use of these measures either alone or together as a multidimensional test battery as primary or key secondary endpoints in MS studies.