Improved ALS clinical trials through frequent at-home self-assessment: a proof of concept study.

OBJECTIVE: To determine the potential for improving amyotrophic lateral sclerosis (ALS) clinical trials by having patients or caregivers perform frequent self-assessments at home. METHODS AND PARTICIPANTS: We enrolled ALS patients into a nonblinded, longitudinal 9-month study in which patients and caregivers obtained daily data using several different instruments, including a slow-vital capacity device, a hand grip dynamometer, an electrical impedance myography-based fitness device, an activity tracker, a speech app, and the ALS functional rating scale-revised. Questions as to acceptability were asked at two time points.
RESULTS: A total of 113 individuals enrolled, with 61 (43 men, 18 women, mean age 60.1 ± 9.9 years) collecting a minimum of 7 days data and being included in the analysis. Daily measurements resulted in more accurate assessments of the slope of progression of the disease, resulting in smaller sample size estimates for a hypothetical clinical trial. For example, by performing daily slow-vital capacity measurements, calculated sample size was reduced to 182 subjects/study arm from 882/arm for monthly measurements. Similarly, performing the ALS functional rating scale weekly rather than monthly led to a calculated sample size of 73/arm as compared to 274/arm. Participants generally found the procedures acceptable and, for many, improved their sense of control of their disease.
INTERPRETATION: Frequent at-home measurements using standard tools holds the prospect of tracking progression and reducing sample size requirements for clinical trials in ALS while also being acceptable to the patients. Future studies in this and other neurological disorders should consider adopting this approach to data collection.

Introduction

In recent years, the number of potential targets available for therapeutic intervention in ALS has greatly increased. Along with this expanding list, the number of new experimental agents has also enlarged. While this is a welcome development, it creates an imperative to consider ways to increase the efficiency of clinical trials and to lower the barriers to participation so that more experimental therapeutic agents can be tested rapidly and efficiently. Clinical trials in ALS have in fact have evolved to be more efficient, with the most significant change being a transition from studies based on survival as a primary endpoint to measures based on function. , , , In addition, modeling studies and recent clinical trials have suggested that cohort enrichment to select for rapidly progressive patients may reduce sample size and trial duration. However, clinical trials still require many months to complete, and participants must travel to the study center frequently, limiting involvement to those in close proximity to study sites and who are not too debilitated to travel. In addition, even for those patients capable of participating in trials and who live close to sites, data on disease progression are available only during the infrequent visits to the study center, most commonly once every 2 or 3 months.

The ALS At Home study was designed to address two issues in ALS trial design. First, we wished to determine whether participants with ALS or their caregivers could be trained to assess their clinical status by performing a variety of outcome measures at home. Some of these measures are identical to commonly used outcomes routinely obtained at clinical trial sites, while others are less commonly used and may represent extensions of what can usefully be measured in ALS patients. This at‐home approach has the potential to both increase the geographic extent across which patients may be recruited to a study and to reduce the burden of participation for patients.

Second, we sought to determine the potential benefit of increasing the frequency of patient assessment. With measurements typically being taken only once every 2–3 months, the calculated rates of deterioration for each participant are likely negatively impacted by low sampling frequency. This variability in measurement mandates a sample size on the order of 120–150 per treatment arm in order to have sufficient power to appreciate a moderate effect size. , Variability in rates of decline with respect to any measure come from three major sources: 1. “noise” related to inevitable nondisease‐related factors, such as subject motivation or fatigue (for clinical measures); 2. measurement error and inconsistencies due to the tool or technique itself or: 3. deviations from the assumption that ALS progresses in a linear manner. One straightforward way to reduce inaccuracies in the calculated slope is to sample far more frequently. To the extent that variability is produced by any of these factors, frequent sampling may allow for a more accurate estimation of the true rate of progression for any individual patient.

In this study, we sought to study the potential benefit of frequent measurements at home, in a manner analogous to an actual clinical trial. Reducing the burden of trial participation, extending the geographic reach of studies, and reducing error of measurement for clinical trials have the potential to markedly improve clinical trial efficiency, including reducing study duration and sample size requirements. We also hoped to understand individual patient/caregiver attitude and response to collecting data independently at home.

Methods

Many of the basic details of the study design and organization have already been reported.

Here we provide a short overview of these concepts.

Overall structure

Study coordination and oversight, including institutional review board review, occurred at the Barrow Neurological Institute (BNI), Phoenix, AZ. Beth Israel Deaconess Medical Center, Boston, MA was responsible for all information technology, including design and implementation of the ALS AT HOME web portal and most phone apps. The app responsible for recording speech signals was designed by Aural Analytics and Arizona State University (JL, VB, SH).

Recruitment strategy

Our recruitment efforts were mostly web‐based through the Centers for Disease Control (CDC) ALS patient registry (https://www.cdc.gov/als/), foundation websites, advertisements through Facebook and Google Ads, and a consistent social media presence on Facebook® and other sites. Enrollment was also encouraged at the ALS clinics at Beth Israel Deaconess Medical Center, Barrow Neurological Institute, and ALS Clinics associated with the Northeast ALS Clinical Trials Consortium (NEALS). ALS patient inclusion criteria included a diagnosis of ALS within 5 years and no known additional neurological disorder of relevance. A small number of healthy subjects were also recruited alongside these ALS patients; these individuals’ data were not included here and will be reported separately.

Measurement tools and methods

The following data collection tools and methods were used in this study:

Digital hand grip strength. The Camry Handgrip Dynamometer (Camry Scale‐USA, City Industry, CA) was used to measure bilateral grip strength. Participants were asked to maximally squeeze the device with each hand and upload the measured values to the ALS‐at‐home REDCap database via the web portal.

Spirometry. Slow‐vital capacity (SVC) values were acquired with the AirSmart Spirometer (Nuvoair AB, Stockholm, Sweden). Data were automatically uploaded to Air Smart’s health cloud and downloaded to the ALS‐at‐home REDCap database. Participants were asked to repeat the measurement three times for a given session and record the maximal value.

Electrical impedance myography (EIM). The Skulpt® Scanner® (aka Chisel, Myolex, Inc, Boston, MA) fitness device was employed. Participants collected EIM data on bilateral biceps, forearms, calves, and quadriceps. All data were uploaded directly to the Myolex database. Only the 50 kHz phase values, obtained from the widest electrode configuration, were used in this analysis, since this measure effectively assessed disease progression in earlier work using a medical‐grade EIM system.

Speech analysis. A separate ALS AT HOME speech app and database was developed by Aural Analytics (Phoenix, Arizona, USA). These results will be reported separately.

Activity tracker. The Mi Band® (Xiaomi Corp, Beijing, China) was used for this purpose with data directly uploaded to ALS‐at‐home database via Apple Health app or Google Fit app.

ALS functional rating scale‐revised (ALSFRS‐R). An online version of the standard ALSFRS‐R was completed by participants on a weekly basis (this is the only measure that was obtained weekly) and was directly uploaded to our REDCap database.

Patient‐reported experience measures (PREMs). PREM data, which included a series of questions regarding challenges using the devices and obtaining data, as well as questions seeking an understanding of an individual’s emotional response to the entire data collection approach, were collected at 7 days and 3 months via an automatic online survey and stored directly in the REDCap database.

Screening, consent, enrollment, participant education, and measurement tool disbursement

These aspects of the study were previously explained in detail. Briefly, after learning of the study, participants completed prescreening questions online at the ALS‐at‐home website. They were then contacted by the study liaison and medical records were requested and reviewed; if inclusion and exclusion were met, they were invited to participate in the study. The individual then participated in an informed consent webinar, after which they were provided an electronic institutional review board‐approved informed consent form. Once enrolled, participants were shipped a package containing all the products to be used in the data collection described earlier. They were then asked to review a series of instructional videos and complete an online test to ensure they understood their proper use, after which they began data collection.

Data collection schedule

Participants were instructed to obtain all outcome measures daily for 90 days, then twice weekly for an additional 180 days, except for the ALSFRS‐R which participants collected weekly throughout the study. Ongoing feedback was provided to the participants via the study liaison. All participants were provided with email and phone contact information if they had questions or encountered problems during the data collection procedures. Participants who completed 90 and 270 days of study activities were rewarded by small gifts (an ALS‐at‐home t‐shirt and hat, respectively).

Data analysis

We wished to determine the change in sample size estimation for a hypothetical clinical trial at increasingly less frequent time intervals of measurement (daily, twice‐weekly, weekly, biweekly, and monthly). Daily data were used to calculate sample size; we then down‐sampled to less frequent measurement periods, performing a new sample size estimation for each reduced dataset. We based this estimation by establishing a slope of decline for a given measure in a given patient, and then calculating the mean and standard deviation in the slope of decline across the entire group of patients. For our hypothetical clinical trial sample size estimation, we assumed 90% power (1−β), two‐sided P values, with a significance level (α) of 0.05 to detect a 30% mean difference between the treatment groups. For the sample size estimation, we used a one sample model for a continuous outcome given by the following :

Here, Z is the standard normal distribution for the respective level of significance and the power. The effect size is given by the following equation:

For simplicity, we assumed 1:1 randomization and no loss to follow‐up (i.e., 100% retention). To achieve this estimation, the initial dataset was reviewed and processed as follows:

Participants with fewer than seven measurement sessions for all measures were removed from the analysis entirely. Similarly, participants were required to have at last two observations in each of the down‐sampled time periods, after imputation; if these were not available for a given measure (e.g., hand grip strength), a slope of progression could not be calculated and that measure for that patient was excluded.

Individual measures that demonstrated a positive slope on daily measurement (improving values) were removed; because of our wide inclusion criteria, some patients did not change over the course of the study and had nominally positive slopes for some outcome measures.

All daily data points that were more than two standard deviations greater or less than the mean for a given measure across the entire data collection period were removed.

Imputation

Given the nature of the data collection process, with individuals or caregivers requested to obtain daily data, there were many missing values in this dataset, usually without explanation. Nevertheless, we attempted to take a conservative approach toward imputation. No imputation was performed for daily measurement analyses; thus, strictly speaking, the daily measurements are more accurately described as “as frequent as daily,” especially since this metric decreased to twice‐weekly at the 3‐month point by study design. Twice‐weekly measurements were arbitrarily chosen as those taken on day 1 and day 5 (thus leaving a 3‐day and a 2‐day gap between measurements). Thus, when down‐sampling to twice‐weekly measures (only relevant up to the end of month 3), if there was a missing value in either day 1 or day 5, then the last observation carried forward mechanism was used for imputation. If the previous day’s observation was also missing, then the next observation carried backward was used. That is, a missing value on a given day was imputed using the previous day’s data point, but if that was also missing, then the following day’s observation was used for analysis. However, if both the previous‐day and the next‐day observations were missing, then the data would be kept as missing. Outside of imputing values for the twice‐weekly calculation, no additional imputation was performed since all weekly, biweekly, and monthly calculations could be performed after this one imputation procedure.

Results

Demographics and overall enrollment and participation data

Figure 1 and Table 1 provides the overall demographics, including the number screened, consented, and enrolled. They also provide data on the number of people who completed the at the 3‐month time point and at the study conclusion. As can be seen, despite over 100 ALS patients initially enrolling, only 61 actually participated in the trial to the extent that their data were sufficient to include in the analysis. However, the vast majority of those completed at least 3 months of study, with a smaller cohort remaining involved out to 9 months.

Figure 1

CONSORT diagram showing recruitment and attrition information.

Table 1

Demographics and enrollment information.

	ALS participants enrolled	ALS participants performing 1st day of measurements	ALS participants completing a minimum of 7 days	ALS participants completing 3 months of study	ALS participants completing 9 months of study
Total Number	113	72	61	58	17
Sex (M/F)	70/43	50/22	43/18	40/18	17/0
Age (mean ± SD)	60 ± 10.4	60.1 ± 9.9	60.1 ± 9.9	60.7 ± 9.5	60.2 ± 7.5
Means ALSFRS‐R at time of enrollment	N/A	36.1	35.8	35.3	34.0

Impact of frequent measurements on the estimation of actual slope for a given patient

Figure 2 shows representative examples of the how the slope uncertainty (95% confidence intervals) enlarges with decreasing measurement frequency in several individuals even though the mean slope itself remains relatively stable.

Figure 2

Examples of how frequency of sampling impacts the uncertainty in the slope approximation for individual patients over the first 3 months of the study. Data are provided for the individual ALS patient who had the most data collected for that measure, so as to most clearly show the impact of down‐sampling on the confidence intervals. The line represents the least‐squares regression. As frequency decreases, the uncertainty in the slope of the line (gray regions) expand markedly for all measures. N refers to the number of measurements included in the calculation. Note that ALSFRS‐R data were collected on a weekly rather than a daily basis.

Effect of more frequent measurements result in sample size estimates, all other characteristics remaining stable

Table 2 shows the calculated mean and standard deviation in the rate of change based on daily, twice‐weekly, weekly, biweekly, and monthly measurements in all subjects across the 9‐month assessment period. As can be seen, for most measures, there is an increase in effect size when more frequent measurements are made. This results in a marked reduction in sample size, in part, because sample size is inversely related to the square of the effect size. A subset of these results is shown in Figure 3.

Table 2

Mean per‐day slope estimations and associated standard deviations, effect sizes, and resulting sample size estimations

		Daily	Twice‐weekly	Weekly	Biweekly	Monthly
Right handgrip	Mean	−0.044	−0.048	−0.041	−0.037	−0.036
	Std	0.041	0.041	0.056	0.064	0.065
	Effect size	0.32	0.35	0.22	0.17	0.16
	Sample size	101	84	211	346	383
Left handgrip	Mean	−0.048	−0.045	−0.042	−0.041	−0.043
	Std	0.047	0.046	0.044	0.044	0.046
	Effect size	0.304	0.294	0.280	0.277	0.278
	Sample size	114	121	134	137	136
EIM left biceps	Mean	−0.015	−0.014	−0.016	−0.015	−0.016
	Std	0.0095	0.015	0.028	0.031	0.039
	Effect size	0.48	0.28	0.17	0.14	0.12
	Sample size	46	132	337	527	710
EIM right biceps	Mean	−0.013	−0.015	−0.013	−0.009	−0.006
	Std	0.013	0.025	0.028	0.033	0.050
	Effect size	0.30	0.17	0.13	0.08	0.03
	Sample size	116	344	560	1580	8855
EIM left quads	Mean	−0.015	−0.013	−0.012	−0.011	−0.014
	Std	0.014	0.023	0.026	0.040	0.049
	Effect size	0.320	0.176	0.134	0.081	0.084
	Sample size	103	338	582	1577	1470
EIM right quads	Mean	−0.01	−0.006	0.003	0.006	−0.006
	Std	0.065	0.007	0.018	0.014	0.026
	Effect size	0.274	0.258	0.084	0.128	0.071
	Sample size	50	157	1487	636	2090
ALSFRS−R	Mean	–	–	−0.064	−0.025	−0.027
	Std	–	–	0.018	0.029	0.041
	Effect size	–	–	0.379	0.262	0.195
	Sample size	–	–	73	153	274
SVC	Mean	−0.165	−0.078	−0.064	−0.058	−0.054
	Std	0.205	0.097	0.106	0.112	0.149
	Effect size	0.240	0.239	0.181	0.155	0.109
	Sample size	182	182	320	434	882
Activity tracker	Mean	−9.73	−6.35	−9.74	−8.84	−6.53
	Std	12.75	11.97	18.02	21.547	25.78
	effect size	0.228	0.159	0.162	0.123	0.075
	Sample size	158	327	315	548	1436

Figure 3

Examples of increasing sample sizes estimations as frequency of measures decreases (A) Left handgrip strength (B) SVC (C) EIM left biceps (D) ALSFRS‐R. Note: ALSFRS‐R data were only acquired weekly rather than daily.

Participants impression of the study

Participants assessed both their views of the ease of measurement of the outcomes and their impression of how measurement affected their impression of the study and their general status. Figure 4 shows the results for our patient‐related experience measures at two different time points (at the approximately 1‐week and 3‐month time point). Participants generally felt that they had improved in their ability to take measurements over time, with most completing the entire set of measurements within 20 min by 92 days. Most also found participation reassuring and that it gave them a sense of control. Of course, we only obtained data on those who remained in the study and who felt comfortable answering the questions. Nearly all subjects who discontinued did so without informing study staff of their reasons for doing so.

Figure 4

PREMS data. The questions were as follows: (A) Overall, the online training provided the information I needed to conduct the self‐assessments (B) Since the start of the study or the time point of last survey, how easy or difficult was it to conduct self‐assessments with the equipment provided for this research study? (C) How confident were you in conducting self‐assessments with the equipment provided? (D) How easy or difficult was it to enter the data into the study portal? (E) How often did you have sadness, anxiety, regret, or feeling upset about the results of my data? (F) How often did you have thoughts on how the measurement results were affecting work or family life (G) Taking my measurements has made me more in control of my disease (H) Taking my own measurements has made it easier (or harder) to cope with my disease (I) Approximate time taken for measurements on Day 8 and Day 92.

Discussion

This study demonstrates the promise of performing a clinical trial in ALS using an entirely remote data collection approach. First, we were able to show that it is feasible to train patients and caregivers to obtain reliable data using simple tools with remote training conducted entirely online. We also showed that many patients persisted in collecting data very frequently for an extended period, even in the absence of a therapeutic intervention. Additionally, we were able to confirm our hypothesis that more frequent measurements resulted in a reduced sample estimation, due to increasing accuracy in the slope approximation for each subject and an overall reduction in variance in slopes across the population of participants. Finally, we demonstrated that of those participants who remained in the study and responded to our questionnaires, the majority had positive feelings toward participation.

To our knowledge, this is the first study of its kind in ALS and thus we met a number of challenges along the way, some expected and some unexpected. As noted in our initial publication describing the overall set up and baseline data, we encountered technical challenges, including problems with using the devices, ongoing Internet, database, app, and website maintenance‐related issues, and challenges in recruiting patients sight‐unseen. As noted earlier, we lost a great deal of participants at the 3‐month point when the data collection was reduced to twice weekly. It remains unclear as to why this happened, but may reflect the habitual nature of doing an activity on a daily basis as compared to on a twice‐weekly basis.

The ALS participants we enrolled were not identical to those typically recruited to clinical trials; most had a more slowly progressive disease course, as demonstrated by the mean rate of ALSFRS‐R progression being approximately −0.6 points/month (based on our weekly dataset) versus the generally accepted average of about −1.0 points/month rate. This was perhaps one reason we identified a number of measures that nominally increased over time in some subjects.

We experienced a high loss to follow‐up. Of the 112 ALS patients enrolled, only 58 collected data until the 3‐month time point and only 17 to study completion at the 9‐month point. Of those remaining, a number had sparse data collection at various periods throughout the study. This loss to follow‐up weakens our sample size estimations. Of course, if we were testing an actual drug as part of this study, it seems probable that individuals would have persisted in their efforts, and that dropout would be no worse than what is noted in most clinical trials. Indeed, this supposition is supported by other studies that have attempted to do primarily at‐home data collection. For example, one recent study evaluated the potential impact of Lunasin, a soy peptide that may alter histone acetylation, and incorporated a home assessment approach (albeit not at the very high frequency of assessment that we employed). That study demonstrated a superb retention rate of 84%, better than most clinical trials. Nonetheless, this hypothesis, in the context of very frequent self‐assessments, requires testing.

Given the high degree of missingness and inherent variability in the data, we were forced to take a number of steps to process the data for effective analysis. We removed outlying data points that were more than two standard deviations beyond the mean for that measure over the entire period of data collection. This was necessary since in reviewing the data, there were some extreme values that likely represented measurement or recording error. In hand grip, for example, it appeared that a few participants intermittently switched sides in recording of data and then reverted back. Second, we performed imputation to obtain meaningful twice‐weekly measurements from which we could then effectively down sample to less and less frequent intervals. Finally, we simply removed patients with very limited datasets (less than 7 measurements) since that would have added little information and considerable noise to our analyses. A separate and planned challenge to interpretation was that the daily measurements ended at 3 months. While the 9‐month dataset captures that 3‐month daily data and demonstrates a reduced sample size compared to less frequent measures, we do not know what true “daily” measurement out to 9 months would have shown. Redoing these sample size analyses for just the data out to 3‐months would likely overstate the value of the daily measurements since we have very short periods from which to draw, resulting in very small datasets for monthly measures (only 4 data points for any measure).

Of the ALS participants who persisted in measurement, most indicated that the study actually made them feel more in control of their disease. But a significant caveat is that we were unable to obtain input from patients who discontinued participation. From phone discussions with some of those that discontinued, the major reason was related to the complexities of performing multiple measurements and the challenges of interacting with our user portal, although several did express concern and fear about watching their values decline over time. It might be hypothesized that this negative experience could be ameliorated to some extent by the addition of an experimental therapeutic agent. In many cases, however, the generally positive reactions to home assessments could indicate greater engagement and persistence in the hope that some effect may ultimately be observed as well as a greater sense of personal responsibility in seeing the study to conclusion. Of course, this is only conjecture and only through incorporating at‐home measurements into a clinical trial can we hope to learn the nature of people’s true acceptance of such an approach.

This study was conceived, planned, executed, the data analyzed, and the manuscript mostly completed before the COVID‐19 pandemic. The unexpected and unprecedented impact of this crisis changed the apparent need for and value of home‐based clinical trials. Accordingly, it is important to recognize that the broader concept of “virtual trials” was introduced nearly a decade ago in the Pfizer REMOTE trial to assess a therapy for improved bladder control and function. We entirely support these efforts as approaches to pursuing clinical research and therapeutic trials during the COVID pandemic. However, we underscore that while our study embraced and relied upon the at‐home data collection approach, the main element which we sought to assess was the value of frequent versus occasional measurements and its potential effect on sample size requirements.

There were a number of limitations to this study. First, as noted earlier, we had considerable loss to follow‐up and unlike most standard studies, were unable to identify quantitatively the reasons for an individual’s leaving the study. Second, we had ongoing challenges with our complex data collection infrastructure, ranging from the website intermittently crashing for unclear reasons (making it impossible for subjects to upload their data) to updates on the smartphone operating systems that would require our making modifications to the apps. Third, as noted earlier, the choice to reduce to biweekly measurements after 3 months reduced the significance of the daily measurement interpretation and may also have contributed to the large loss to follow‐up, as many people did appear quite engaged up until that point. Fourth, we did not collect data on the large number of people who quit the study very early, limiting our interpretation of the PREMs information.

In summary, this effort shows that frequent, at‐home data collection is feasible in an ALS clinical trial and that it could hold the promise of reducing sample size requirements while keeping individuals engaged, obviating the need to travel to a tertiary care center. We strongly encourage both academic researchers and the pharmaceutical industry to consider such approaches for future trials not only in ALS but also in other neurological diseases.

Conflict of Interest

Dr. Rutkove holds equity in Myolex, Inc, (the company that produces the Skulpt® Scanner), has served on the board of directors, has received consulting income from the company, and is named as an inventor on patents owned or licensed to Myolex, Inc.