Refining eligibility criteria for amyotrophic lateral sclerosis clinical trials.

OBJECTIVE: To assess the effect of eligibility criteria on exclusion rates, generalizability, and outcome heterogeneity in amyotrophic lateral sclerosis (ALS) clinical trials and to assess the value of a risk-based inclusion criterion.
METHODS: A literature search was performed to summarize the eligibility criteria of clinical trials. The extracted criteria were applied to an incidence cohort of 2,904 consecutive patients with ALS to quantify their effects on generalizability and outcome heterogeneity. We evaluated the effect of a risk-based selection approach on trial design using a personalized survival prediction model.
RESULTS: We identified 38 trials. A large variability exists between trials in all patient characteristics for enrolled patients (p < 0.001), except for the proportion of men (p = 0.21). Exclusion rates varied widely (from 14% to 95%; mean 59.8%; 95% confidence interval 52.6%-66.7%). Stratification of the eligible populations into prognostic subgroups showed that eligibility criteria lead to exclusion of patients in all prognostic groups. Eligibility criteria neither reduce heterogeneity in survival time (from 22.0 to 20.5 months, p = 0.09) nor affect between-patient variability in functional decline (from 0.62 to 0.65, p = 0.25). In none of the 38 trials were the eligibility criteria found to be more efficient than the prediction model in optimizing sample size and eligibility rate.
CONCLUSIONS: The majority of patients with ALS are excluded from trial participation, which questions the generalizability of trial results. Eligibility criteria only minimally improve homogeneity in trial endpoints. An individualized risk-based criterion could be used to balance the gains in trial design and loss in generalizability.

The clinical heterogeneity of amyotrophic lateral sclerosis (ALS) makes conducting clinical trials complex.[1] To increase the potential to demonstrate therapeutic efficacy, investigators apply eligibility criteria to enroll a more homogeneous population, to improve protocol adherence, or to exclude patients who are unlikely to benefit.[2-4] For ALS, this often means excluding those patients with long disease durations or those who are unlikely to survive the follow-up period. Thus, many patients are excluded,[2] which could have important consequences for the generalizability of a trial (i.e., “to whom do the results of this trial apply?”).[4] A low generalizability potentially provides limited information about safety or efficacy for the general population.[2,4,5]

Nevertheless, a pragmatic approach (i.e., no selection) may not be feasible in ALS because this increases endpoint variability and may inflate sample size.[6] It is therefore important to balance the generalizability of a trial (i.e., selecting a population that represents the general population) and its endpoint heterogeneity (i.e., selecting a sensitive population to show efficacy). Prediction models such as the recently validated ALS survival model[7] might improve the selection of patients by using the predicted outcome as an inclusion criterion instead of a set of arbitrary criteria.[8]

Virtually all ALS clinical trials have imposed various sets of eligibility criteria, but little is known about the consequences.[2,9] In this study, we review the current practices of participant selection and assess the effects on trial populations, efficacy endpoints, and generalizability. We then gauge the value of an individualized risk-based selection criterion in balancing outcome heterogeneity and patient exclusion rates.

Methods

The effect of eligibility criteria on population characteristics and trial endpoints was estimated in a 2-step approach. First, we systematically reviewed the literature to compose a list of commonly used criteria and meta-analyzed the baseline characteristics of the included trials. Subsequently, we applied the extracted criteria to an incidence cohort to estimate their effects on survival and functional decline.

Search strategy and trial selection

Two authors (R.P.A.v.E. and I.E.V.) individually searched the PubMed and Embase database for publications dating from January 1, 2000, up to and including November 2017 using the following search terms: amyotrophic lateral sclerosis or motor neuron disease and clinical trial. To harmonize the comparison between clinical trials, we included only randomized, placebo-controlled, clinical trials evaluating the efficacy of a single pharmacologic agent. We excluded clinical trials investigating multiple agents, exclusively aiming to determine safety (phase I), having a nonclinical primary endpoint, or starting enrollment before the approval of riluzole (1996).

Data extraction and harmonization

For each trial, we extracted the eligibility criteria and baseline characteristics. If a trial included multiple dosing groups, groups receiving the experimental agent were collapsed by a fixed-effects within-study meta-analysis. Predicted vital capacity was recorded as either the forced vital capacity (FVC) or slow vital capacity because there is no true difference between the 2 values.[10] When the mean Amyotrophic Lateral Sclerosis Functional Rating Scale (ALSFRS) score was reported instead of the revised ALSFRS (ALSFRS-R), we added 7.76 to the mean and 0.23 to the reported SD because this procedure provided the least systematic error during simulations. The ALSFRS-R slope was calculated as the average monthly change from randomization. Studies reporting the ALSFRS slope were transformed to the ALSFRS-R slope by multiplying the mean ALSFRS rate of decline by 1.1 and its standard error by 1.14. If data were not reported as mean with SD (e.g., median and range), we transformed the reported estimates according to the method of Hozo et al.[11] Finally, if only the mean was reported without SD (6% of the studies), we imputed the natural logarithm of the SD from the natural logarithm of the mean using linear regression.[12]

Exclusion rate and incidence cohort of patients with ALS

Our primary goal was to estimate, per set of eligibility criteria, the number of patients from the general population who would be considered ineligible to participate (i.e., exclusion rate). The exclusion rate per trial was defined as the number of ineligible patients divided by the total number of patients (population size). An exclusion rate of 60%, for example, indicates that only 40% of the general population was potentially evaluated in a given trial; the effectiveness and safety are unknown for 60% of the population. Because eligibility criteria are applied in a step-wise manner, the exclusion rate is a cumulative buildup of several percentages. An example of how this rate is calculated is presented in table 1.

Table 1

Example of the calculation of the exclusion rate for the Pentoxifylline 2006 trial

The denominator is crucial for the accuracy of the exclusion rate. We therefore approximated the population size by including all consecutive patients with ALS in the Netherlands diagnosed between January 2006 and December 2016 in the analysis. A detailed description of the incidence cohort of patients with ALS in the Netherlands, which has a coverage rate of 81%, has been given elsewhere.[13] In short, patients with ALS in the Netherlands diagnosed with possible, probable (laboratory supported), or definite ALS according to the revised El Escorial criteria are registered centrally at The Netherlands ALS Center.[14] Patients are located by annual screening of large medical center registries and by individually contacting Dutch neurologists.[13] For each patient, we determined whether he or she participated in a clinical trial in the same time period. In addition, clinical data were collected to determine an individual's eligibility for trial participation at the day of diagnosis (figure e-1 available from Dryad, doi.org/10.5061/dryad.86f1m6g). Time-varying variables such as ALSFRS-R total score and FVC were, ideally, collected at diagnosis or within a 3-month time interval; otherwise, the score was recorded as missing. Complete survival data (date of death or last follow-up) were obtained by checking the online municipal population register at 3-month intervals. In total, 39 surviving patients (1.3%) were followed up for <6 months and 139 (4.7%) for <12 months.

Standard protocol approvals, registrations, and patient consents

The medical ethics committee and institutional review board of the University Medical Center Utrecht approved this study.

Statistical analysis

Variables necessary to determine eligibility contained, on average, 36% missing values; kidney function was most often missing (in 82% of the cases). Eligibility was based on all variables provided (figure e-1 available from Dryad, doi.org/10.5061/dryad.86f1m6g). Missing data were accounted for by creating multiple imputed datasets (n = 100) using predictive mean matching and bootstrapping, discarding the first 100 iterations (burn-in). The imputation model contained all covariates; survival time was modeled with Nelson-Aalen estimates.[15] Subsequently, the extracted eligibility criteria were applied to each imputed dataset. Using the incidence cohort as reference, we estimated the proportion of patients eligible for each trial individually. Results across imputations were pooled with Rubin rules.[16] Baseline characteristics and eligibility percentages were summarized by meta-analysis. Continuous variables were meta-analyzed using the raw reported means; count variables were recoded to logit-transformed proportions. Subsequently, between-population heterogeneity was evaluated with the Cochrane Q test and the I2 statistic.[17] All meta-analytical models were based on a random-effects model, fitted with restricted maximum-likelihood estimation. Standard errors of coefficients were adjusted according to the methodology given by Knapp and Hartung.[18]

For each patient, we determined his or her risk profile by taking the exponent of the centered linear predictor of the European Network for the Cure of ALS (ENCALS) survival model.[7] An estimated risk score of 0.5 indicates that the risk of dying during follow-up is half the risk of dying for the average patient with ALS, whereas a score of 2 means that this risk is twice the average. Using the quintiles of the individual risk scores, we defined 5 prognostic groups: very long, long, intermediate, short, and very short. To estimate the underlying survival time distribution, we fitted a parametric Royston-Parmar proportional hazard models with 3 internal knots (based on Akaike information criterion) per prognostic subgroup.[19,20] The interquartile range of the survival times was used to define survival time variability. Bootstrapping (n = 10,000) was used to estimate 95% confidence intervals (CIs) around the interquartile range. Variation in ALSFRS-R rates of decline was estimated by linear mixed-effects models, as described previously.[6] Longitudinal sample calculations by Eland (2009) determined the number of patients necessary to detect a 25% reduction in slope after 12 months (monthly follow-up) with 80% power and a 2-sided α of 5%.[6] The sample size was used as an estimate of the sensitivity of the populations to detect a given treatment effect. All meta-analyses were performed with the R metafor package (version 2.0-0, Viechtbauer[21]). Linear mixed-effects models were fitted with the lmer function (lme4, version 1.1-12).[22] Imputation of missing data was performed using the aregImpute function (Hmisc, version 4.0–3).

Data availability statement

All protocols, analyses, and anonymized data will be shared by request from any qualified investigator.

Results

A total of 38 randomized placebo-controlled clinical trials were included in this study (figure e-2, available from Dryad doi.org/10.5061/dryad.86f1m6g); their eligibility criteria were applied to an incidence cohort of 2,904 patients with ALS. Figure 1 summarizes the primary eligibility criteria and exclusion rates per trial. On average, 59.8% (95% CI 52.6%–66.7%) of the patients would be excluded from trial participation on the day of diagnosis in the incidence cohort. Exclusion rates varied widely between trials, ranging from 14% to 95%. Between 2000 and 2010, 53.3% of the patients were excluded, which increased to 65.5% between 2010 and 2017 (p = 0.08). Not meeting a specific El Escorial category is the most important reason for exclusion (23%, 95% CI 18%–28%), followed by FVC (17%, 95% CI 14%–20%) and disease duration (12%, 95% CI 8%–15%) (figure e-1 available from Dryad, doi.org/10.5061/dryad.86f1m6g).

Figure 1

Overview of eligibility criteria and exclusion rates of ALS trials conducted between 2000 and 2017

Caterpillar plot of the exclusion rates at diagnosis per trial (n = 38). Exclusion rates ranged from 14% to 95%; numerical results per trial are given in table e-1 (available from Dryad, doi.org/10.5061/dryad.86f1m6g). ALS = amyotrophic lateral sclerosis; ALSFRS (n) = number of selection items on Amyotrophic Lateral Sclerosis Functional Rating Scale; Def = definite; FALS = familial ALS; FVC = forced vital capacity, percent predicted; G-CSF = granulocyte colony-stimulating factor; IGF-1 = insulin-like growth factor-1; LI = lead-in ALSFRS-R slope; LS = probable, laboratory supported; Prob = probable.

The enrolled populations of the 38 trials are presented in figure 2 and reveal a large variability between trials of all patient characteristics (p < 0.001), except for the proportion of men (p = 0.21). If we select those patients who are eligible for >50% of the trials from our general population (eligible population in table 2) and compare them with actual trial participants at our center (trial participants in table 2), large differences can be seen. Actually enrolled patients (n = 260, 9.0%, table 2, column 4) differ from their eligible population in sex (more men, p = 0.019), age (younger, p < 0.001), progression rate (slower, p < 0.001), ENCALS risk profile (better, p < 0.001), and survival (longer, p < 0.001). This suggests that, despite the already applied eligibility criteria, only a selective subset of the eligible patients will participate in trials. This finding is indicative of an additional latent selection process.

Figure 2

Between-trial variability in population characteristics of enrolled participants

Caterpillar plots of the reported summary data for the experimental and placebo groups (total number of patients 10,489). Solid black diamonds indicate the meta-analyzed average with 95% confidence interval. Study heterogeneity was present in all baseline characteristics except for the proportion of men (I2 = 11%; p = 0.21). (A) Age at randomization; (B) proportion of men; (C) symptom duration; (D) bulbar onset; (E) vital capacity (VC); and (F) Amyotrophic Lateral Sclerosis Functional Rating Scale–Revised (ALSFRS-R).

Table 2

Population characteristics of various populations

Eligible patients are, compared to the general ALS population, 11% less likely to die during the first 24 months after diagnosis (pooled hazard ratio 0.89, 95% CI 0.84–0.93, p < 0.001). When we stratify the eligible population into prognostic subgroups by applying a personalized prediction model, we can seen that defining eligibility leads primarily to a temporary survival difference in the poorest prognostic group (6-month survival increases from 65.5% to 81.6%, figure 3). Large proportions (45%–66%) of other prognostic subgroups are also excluded, while the effect on survival is minimal (an absolute 2.3%–4.6% increase at 12 months and 1.5%–3.6% increase at 18 months). Within the eligible population, 12.7% of patients still have a very poor prognosis and 16.3% are very long survivors. Results were similar for the individual trials (figure e-3 available from Dryad, doi.org/10.5061/dryad.86f1m6g), revealing a rather random exclusion process within all prognostic subgroups. Overall, eligibility criteria reduced heterogeneity in survival time by only 6.9% (95% CI −3.5% to 16.3%, p = 0.09), reducing survival time variability from 22.0 months (95% CI 20.8–23.4) to 20.5 months (95% CI 18.8–22.4). This further underscores that both very short- and long-surviving patients remain in the trial population.

Figure 3

Effect of selection criteria on overall mortality since diagnosis

Mortality since diagnosis for 5 prognostic subgroups defined by the European Network for the Cure of ALS personalized prediction model. Colors represent the 5 prognostic subgroups: very long (green), long (yellow), intermediate (orange), short (red), and very short (black) survival.[7] Solid lines in the Kaplan-Meier plots represent the survival patterns for all patients (n = 2,904; A), whereas the dotted lines are the survival curves for the eligible population (n = 1,194; B), defined by all patients who are eligible for > 50% of the trials. Boxplots provide the variability in survival time per prognostic subgroup (left, all patients; right, eligible patients). Dotted lines in boxplot are the unstratified interquartile ranges (25th and 75th percentile); overall survival time variability was reduced by 6.9% (95% confidence interval −3.5% to 16.3%, p = 0.09). Bar charts provide the number of patients in each subgroup.

Longitudinal ALSFRS-R scores up to 24 months after diagnosis were available for 696 patients. Patients with a poor prognosis are relatively underrepresented in the ALSFRS-R data (bar charts, figure 4A). Between-patient ALSFRS-R variability does not differ between unselected and trial-eligible patients (slope variance 0.62 vs 0.65, p = 0.25). The average ALSFRS-R rate of decline in eligible patients, however, is higher (0.91 vs 0.98, p = 0.14), which could reduce the sample size by 5.3% (from 125 to 118 patients per arm). Figure 5 shows the sample size reductions for the 38 trials relative to their eligibility rate, revealing a strong relationship between both variables (Pearson r 0.58, 95% CI 0.32–0.76, p < 0.001). The sample size estimates for trial populations defined by the ENCALS risk score are shown in black and gray. A sample size reduction of 34%, similar to Edaravone,[23] can be achieved by selecting only those patients with risk scores between 0.55 and 3.3. This single selection criterion would mean that ≈48% of the patients remain eligible, which would lead to an almost 5-fold increase in eligibility rate compared to Edaravone. None of the trials was more efficient than the prediction model in optimizing both the sensitivity and eligibility rate of the populations.

Figure 4

Longitudinal ALSFRS-R patterns in a subset of 696 patients

Colors of the boxplots represent the 5 prognostic subgroups: very long (green), long (yellow), intermediate (orange), short (red), and very short (black) survival.[7] Purple dotted line is the average pattern of decline in the eligible population (n = 356). Solid purple line is the average pattern in the full dataset (n = 696). Boxplots provide the variability in rates of decline over time (points per month), estimated by the best unbiased linear predictors (BLUPs) from an linear mixed-effects model. Dotted lines in boxplots represent the average rate of decline. (A) All patients and (B) eligible patients. ALSFRS-R = Amyotrophic Lateral Sclerosis Functional Rating Scale–Revised.

Figure 5

Effect of trial eligibility criteria and the ENCALS risk scores on sample size

Sample size calculations were performed for various populations, either selected by the trial eligibility criteria (n = 38, orange) or defined by different cutoffs for the European Network for the Cure of ALS (ENCALS) risk scores (n = 1,050 combinations). Results are expressed as sample size inflation factor (IF; x-axis). To exemplify, an IF of 0.79 means that the sample size is 21% smaller compared to the sample size necessary when all patients are included (green triangle, 100% eligibility). Populations defined by the ENCALS risk scores for all possible combinations of cutoff values are shown in black and gray. Black dots are the selected populations that resulted in the largest reduction in sample size and highest eligibility rate. To exemplify, the Edaravone 2017 trial resulted in the largest reduction in sample size (−34%, IF 0.66), with a 10% eligibility rate. The ENCALS model could select a similarly sensitive population with 48% of the patients remaining eligible. Numerical effects of the different trials are provided in table e-1 (available from Dryad, doi.org/10.5061/dryad.86f1m6g).

Discussion

In this study, we show that on average 59.8% of the patients with ALS are found to be ineligible to participate in clinical trials. Although eligibility criteria reduce the number of patients with a poor prognosis, there is an adverse exclusion process that leads to a substantial untargeted exclusion of patients from all prognostic subgroups. Moreover, currently applied eligibility criteria select populations that may still contain a relatively high number of patients who show slow or fast progression rates and do not reduce between-patient variability. These findings raise questions regarding not only the value of currently applied eligibility criteria but also the generalizability of clinical trial results in ALS. Using prediction models could individualize participant selection and optimize the balance between endpoint heterogeneity and the generalizability of trial results.

The concept of generalizability plays a central role in the translation of trial results to medical decision making.[4] Clinical trials with highly selected subgroups are difficult to interpret in real-world settings, and the safety or effectiveness of a drug may be unknown for the majority of the patients. We show that 59.8% of patients are excluded from participation at diagnosis. This percentage is, however, an underestimation because most patients will be enrolled a few months after diagnosis (6–9 months). At that time, 15% to 24% of our patients, who could theoretically be prescribed the drug at diagnosis, are deceased and are thus never evaluated in clinical trials. Moreover, a larger proportion of the remaining patients will fail the criteria due to disease progression. Together, these may lead to exclusion rates in real-world settings that approximate those reported in other fields (80%–96%).[4,5,24,25] These high exclusion rates could result in the indirect removal of patients with specific drug-responsive pathways. For example, patients with ALS–frontotemporal dementia and familial ALS are often excluded; however, because these subtypes are related to the C9orf72 repeat expansion,[26] they are indirectly related to C9orf72 disease pathways.[27] As was shown recently,[28] it is possible that the treatment effect is modified by pharmacogenetic interactions, which could be missed by the indirect exclusion of specific subgroups and may disguise important treatment clues. The same may hold true for the larger exclusion rates among bulbar-onset patients or women.

Because of the heterogeneous nature of ALS,[29] eligibility criteria aim primarily to reduce the amount of between-patient variation and to improve protocol adherence.[2,3] We show, however, that currently defined populations still contain fast- and slow-progressing patients and between-patient variation is virtually unaffected. A likely explanation is that selection criteria are applied in a step-wise, univariate manner (table 1), while progression rate is dictated by prognosis,[30] which is defined by a multivariate combination of predictors.[7] When a trial aims to exclude fast-progressing patients, patients who are on the lower limits of each criterion could still be enrolled. The sum of the lower limits, however, means that these patients have a poor prognosis and fast progression rates. To exemplify, a trial with only 2 criteria (age <75 years and FVC >60%) would enroll a 74-year-old patient with an FVC of 61% but exclude a 76-year-old with an FVC of 104%. The first patient is likely to exhibit a faster rate of decline and is more likely to die during follow-up. More important, the real-world effect is minimal. The dexpramipexole study,[31] for example, excluded patients with a disease duration >24 months and an FVC <65%. However, if none of the criteria had been applied, the statistical power would have been reduced from 90% to 88% (assuming an inflation of the reported SD of 3%). It is doubtful whether a 2% gain in power justifies the exclusion of 25% to 40% of the patients.

This questions the value of currently used eligibility criteria, and a revision would seem to be indicated. There is a need to balance endpoint heterogeneity (or sample size) and the generalizability of trials. This balance could be achieved by using individual risk scores rather than group-level criteria. The risk score can be conceptualized as a summary of all available prognostic information per individual. Replacing sets of several eligibility criteria by a single risk estimate would allow investigators to select only those patients who are the most likely to exhibit the investigator-preferred disease pattern. This could reduce between-patient heterogeneity more effectively while balancing both generalizability and eligibility. In our ALSFRS-R example for Edaravone,[23] instead of excluding 90% of the patients, the same homogeneity effect could be reached using a risk-based selection with 48% of the patients remaining eligible (a nearly 5-fold increase).

Our study has several limitations that should be considered. First, because populations may differ between countries, our results are limited to a specific geographic area. In Italy and Ireland, for example, the age at onset is slightly higher, which would underestimate our current exclusion rate.[32] Second, this work focused primarily on optimizing the balance between endpoint heterogeneity and generalizability of trial results. However, ALS remains a complex disorder, and investigators may apply eligibility criteria with imperatives other than simply selecting the most responsive subgroup (e.g., safety, pharmacodynamics, or hypothesized mechanism or action). It may therefore be insightful to distinguish between biological eligibility criteria (those that are set for the hypothesized drug mechanism) and design criteria (those that are set to optimize the trial design). This is especially important when considering the current developments toward a personalized, genotype-oriented approach in ALS.[28,29] Genotype- or biomarker-oriented trials inherently have high exclusion rates and potentially investigate (ultra) rare subgroups, which compromises their feasibility. The antisense trial in SOD1-related ALS,[33] for example, had an enrollment rate of 1 patient per 4 months per site, which underscores the importance of optimizing the eligibility rate. Therefore, future clinical trials could combine both group-level biological (e.g., genetic marker or disease pathway) and individualized risk-based criteria to optimize both drug responsiveness and trial design.

As final note, our results, supported by previous studies,[2,34] indicate a latent selection of patients in which young male patients with a relatively mild disease are overrepresented in trial populations. This latent selection process is most probably the result of a multifactorial process, one that cannot be estimated in the meta-data of published clinical trials. The reason for eligible patients with ALS declining study participation is in 94% of cases the physical burden.[9] Given the relative underrepresentation of patients in advanced stages, it is plausible that the physical burden is an important latent factor, one that becomes more apparent as the disease progresses. This is supported by the observation that dropout and noncompliance are related to lower ALSFRS-R and FVC scores.[3] Similarly, the physician may deem the patient unfit to undergo the trial and not offer the option of participating.[9] Sex differences may also play a role; female patients are more often reluctant about medical testing or have an inability to cope with the protocol.[35] These latent factors may be overexpressed when 2 trials run simultaneously at the same site. The patient’s choice for a certain trial could be related to a certain characteristic of the patient (e.g., sex or age). Dedicated studies are needed to determine the relevance of each factor and to potentially develop strategies to reduce this latent selection process (e.g., by reducing the physical burden using home-based outcome measures or patient-reported data).[36]

Our results reveal that the majority of patients with ALS are excluded from trial participation at diagnosis, which raises questions regarding the generalizability of current trials. Exclusion of ineligible patients only minimally improves homogeneity in trial endpoints. A risk-based selection criterion could individualize trial participant selection and may improve the balance between endpoint heterogeneity and exclusion rates.