Improving clinical trial efficiency: Is technology the answer?

Adjunctive randomized placebo‐controlled trials are the gold standard for determining whether new antiseizure drugs (ASDs) are effective for patients with treatment‐resistant epilepsy. In these studies, patients receiving the novel ASD as an add‐on are compared with patients for whom a placebo has been added. These studies have been quite robust overall. In general, drugs that have succeeded in standard animal models of epilepsy (maximal electroshock, pentylenetetrazol) have also succeeded in randomized controlled trials, suggesting that the trial design is effective for its purpose.1 Moreover, until recently the two “adequate and well‐controlled trials” that are required by regulatory bodies in order for a drug to be approved have shown consistent results. This is perhaps surprising, considering that these trials depend on patient seizure report, which is notoriously poor.2, 3 Many epilepsy patients also have significant memory difficulties. Moreover, seizures are counted by means of a paper diary, which is shockingly low‐tech in the era of seizure detection devices, or at the very least electronic diaries. This low‐tech mechanism for seizure counting should fail for a number of reasons, including failure of patients to be aware that a seizure has occurred, retrograde amnesia for the seizure after the fact, and failure to write down a seizure in a timely fashion, with subsequent forgetting of its occurrence.

In this interesting exercise by Goldenholz et al.,4 a dataset of 10 patients who had intracranial electrodes implanted chronically are used to model patients enrolled in randomized controlled add‐on studies. These patients all had the Seizure Advisory System in place, which differed from other chronically implanted systems (such as the Responsive Neurostimulator) in that it incorporated not only an electrographic seizure detector but also an audio recorder that could capture sounds occurring at the time of detections.5 This allowed the researchers to sort out “true clinical seizures” (those with electroencephalogram and audio confirmation) and determine which “true seizures” were identified by patients in their diaries. The authors compare study results (1) when diary data are used exclusively (reproducing the conditions under which all trials of ASDs have been performed to date); (2) when all clinically verified seizures were counted, whether reported or not (and diary‐reported seizures without electroencephalogram were discarded); and (3) when all electrographic seizures, whether clinically verified or not, were counted. The authors conclude that if the drug does not have a strong effect, counting all seizures, whether clinical or not, resulted in a higher chance of a “successful trial” (defined as a statistically significant outcome with a level of p < 0.05, consistent with regulatory requirements) than counting clinical seizures, verified or not.

Is this a “win” for the high‐tech solution? Actually (and surprisingly), perhaps not, because the only improvement in trial efficiency was seen when subclinical (electrographic) seizures were counted in addition to clinical ones.

So, what is wrong with counting electrographic seizures if this is what would improve trial efficiency? This might work for a phase II or proof‐of‐concept study and should definitely be considered for this purpose. However, likely issues would preclude the use of this type of seizure counting for a regulatory phase III trial. In the eyes of regulatory bodies such as the Food and Drug Administration (FDA), such data would be considered a “surrogate outcome” because it is not (for now at least) coupled with a clinical sign or symptom. The authors make a case for considering this a “true outcome” because some studies suggest that presence of interictal activity is coupled with cognitive dysfunction.6 However, there are no data to support that this is true in all cases.

Mistakes can be made when, owing to expediency, surrogate outcomes replace clinical ones in trials. One dramatic example comes from cardiology. In the past it was common to use reduction in ventricular premature depolarizations (VPDs) as an outcome measure when searching for drugs to treat fatal arrhythmia after myocardial infarction (MI). This made sense, because a higher frequency of VPDs was correlated with a higher risk of cardiac death after MI. The medical community was shocked when a large trial definitively proved that certain antiarrhythmic drugs reduced VPDs but actually increased death due to arrhythmias after MI.7 Although unlikely, one could imagine a scenario in which a new drug, because of its specific mechanism, might markedly reduce subclinical seizures while increasing clinical ones.

The true win would have been an outcome in which counting all clinically verified seizures would provide better power than counting only patient‐reported seizures. After all, this would in theory solve all the seizure‐counting issues described above, such as patients’ lack of awareness of seizures and seizure forgetting. It would pave the way for other seizure‐counting solutions, such as wrist‐worn devices, electroencephalogram (EEG) patches, and electronic diaries. But, surprisingly, it seems in this simulation that these solutions were not helpful in improving trial success. This result is unfortunate. However, as noted by the authors themselves, the current study is not definitive. The simulation derived a million “virtual” subjects from only 10 actual patients. Also, these patients may have become better than average at seizure counting by virtue of learning from the device recordings over time. This type of study should be repeated and continued to determine whether high tech beats low tech for the purposes of performing trials.

Disclosure of Conflict of Interest

J. French receives NYU salary support from the Epilepsy Foundation and for consulting work on behalf of the Epilepsy Study Consortium for Acorda, Adamas, Alexza, Anavex, Axcella Health, Biogen, BioPharm Solutions, Cerecor, Concert Pharmaceuticals, Engage, Eisai, Georgia Regents University, GlaxoSmithKline, GW Pharma, Marinus, Monteris, Nestlé Health Science, Neurelis, Novartis, Pfizer, Pfizer‐Neusentis, Ovid, Roivant, Sage, SciFluor, SK Life Sciences, Sunovion, Takeda, UCB Inc., Upsher Smith, Xenon Pharmaceuticals, Zogenix, and Zynerba. J. French has also received personal compensation for serving as associate editor of Epilepsia and received research grants from Acorda, Alexza, Eisai Medical Research, LCGH, Lundbeck, Pfizer, SK Life Sciences, Sunovion, UCB, Upsher‐Smith, and Vertex as well as grants from the Epilepsy Research Foundation, Epilepsy Study Consortium, Epilepsy Therapy Project, and NINDS. She is on the editorial board of Lancet Neurology, Neurology Today, and Epileptic Disorders. She is scientific officer for the Epilepsy Foundation for which NYU receives salary support. She has received travel reimbursement related to research, advisory meetings, or presentation of results at scientific meetings from the Epilepsy Study Consortium, the Epilepsy Foundation, Eisai, GW Pharma, Marinus, Nestlé Health Science, Pfizer, Sage, SK Life Sciences, Takeda, UCB, Upsher‐Smith, Zogenix, and Zynerba. I confirm that I have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.