EpiNet as a way of involving more physicians and patients in epilepsy research: Validation study and accreditation process.

OBJECTIVE: EpiNet was established to encourage epilepsy research. EpiNet is used for multicenter cohort studies and investigator-led trials. Physicians must be accredited to recruit patients into trials. Here, we describe the accreditation process for the EpiNet-First trials.
METHODS: Physicians with an interest in epilepsy were invited to assess 30 case scenarios to determine the following: whether patients have epilepsy; the nature of the seizures (generalized, focal); and the etiology. Information was presented in two steps for 23 cases. The EpiNet steering committee determined that 21 cases had epilepsy. The steering committee determined by consensus which responses were acceptable for each case. We chose a subset of 18 cases to accredit investigators for the EpiNet-First trials. We initially focused on 12 cases; to be accredited, investigators could not diagnose epilepsy in any case that the steering committee determined did not have epilepsy. If investigators were not accredited after assessing 12 cases, 6 further cases were considered. When assessing the 18 cases, investigators could be accredited if they diagnosed one of six nonepilepsy patients as having possible epilepsy but could make no other false-positive errors and could make only one error regarding seizure classification.
RESULTS: Between December 2013 and December 2014, 189 physicians assessed the 30 cases. Agreement with the steering committee regarding the diagnosis at step 1 ranged from 47% to 100%, and improved when information regarding tests was provided at step 2. One hundred five of the 189 physicians (55%) were accredited for the EpiNet-First trials. The kappa value for diagnosis of epilepsy across all 30 cases for accredited physicians was 0.70. SIGNIFICANCE: We have established criteria for accrediting physicians using EpiNet. New investigators can be accredited by assessing 18 case scenarios. We encourage physicians with an interest in epilepsy to become EpiNet-accredited and to participate in these investigator-led clinical trials.

More physicians and patients need to participate in clinical research so that we can learn more about the optimal treatment of epilepsy

The EpiNet study group has been established to facilitate multicenter collaborative research studies

The diagnosis of epilepsy and specific seizure types often involves uncertainty, and this needs to be acknowledged

Formal accreditation processes should be undertaken to confirm that investigators participating in trials use similar diagnostic strategies

The EpiNet study group has established an accreditation process for investigators participating in the EpiNet‐First trials

Introduction

Epilepsy is a common but complex disease with multiple subtypes. Although it is a global disease, the incidence, etiology, and outcomes of epilepsy vary among countries.1 Much regarding the pathogenesis and management of epilepsy, and detail of geographic variation, is still unknown. The investigation of rare forms of epilepsy requires the involvement of multiple centers to gain sufficiently large cohorts. Despite this, relatively few patients participate in research. Knowledge of epilepsy management is mostly derived from studies conducted in high‐income countries; resource‐poor areas are seldom included, although most people with epilepsy live in these regions.2 There is a great need to involve epilepsy experts from developing countries in research projects.

The efficacy and tolerability profiles of antiepileptic drugs (AEDs) are generally based on trials of short duration performed in highly selected patient groups.3 The relative merits of these drugs have not been well established, and there is little evidence to guide clinicians when the first AED fails, or to direct combination therapy when monotherapy is ineffective. The precise indications for nonpharmaceutical therapy, including diet, vagal nerve and deep brain stimulation, and surgery, still need to be determined.

When doctors do not know the best treatment for a patient, they should endeavor to find out; this involves accessing evidence‐based guidelines, reviewing the literature, and consulting experts. We contend that if there is no evidence to guide therapy, then doctors should encourage patients to participate in trials in which different treatments are compared (comparative effectiveness research).

Barriers to participation in research

Many clinicians do not consider research to be part of their job. They are responsible for the care of the patients in clinics or hospital wards, but they may consider that research is for academics. Even if clinicians are interested in research, they are often too busy to participate. Most health services are under considerable financial pressures, and many physicians work long hours and struggle to cope with the clinical demands. Many health services do not encourage research. The difficulties of conducting clinical research while undertaking routine clinical care have recently been highlighted in the New England Journal of Medicine. Only 4% of 459 general practices in the United Kingdom who were approached to participate in a comparative effectiveness trial comparing two statin drugs ever recruited patients.4, 5

Patients can be suspicious of research; they may be concerned that they will be put at risk, or that researchers will take advantage of them. Time and patience are required to convince patients that it is in everybody's interests—particularly theirs—to find out which treatments are most effective. Even if the risks from the research are no greater than those associated with routine clinical care, participation takes time, and this can leave them financially disadvantaged if they are not reimbursed for their visits.

Regulatory barriers need to be overcome. Institutional review boards and ethics committees have an important role to protect patients and to ensure that research is of a high standard. However, this means that even relatively low‐risk studies may have to overcome major hurdles before they commence. Regulatory hurdles are particularly high for multicenter studies, where multiple ethics committees and research governance committees need to evaluate a project.

Pharmaceutical companies have the resources to overcome the regulatory hurdles. They are also able to make clinical research lucrative for investigators. If researchers are generously reimbursed for their time, then they may sacrifice other activities for research. This is often a mutually beneficial arrangement; the research gets done according to a satisfactory time scale, and the investigators may earn funds that can be used to develop services in their departments. This research model is effective in ensuring that particular types of clinical research are performed; however, the agenda for this research is set by pharmaceutical companies, which are working under a financial imperative; they are interested in research studies that will help them market their particular drug. They are less likely to be interested in funding research that might not increase their sales, and they may be wary of comparative studies that might actually show another drug is more effective than their own. One of the consequences of the success of this model is that it may create a mind‐set in researchers and healthcare providers that they should be generously reimbursed for research. However, we argue strongly that it is in everyone's interests to find out what the best treatments are for the many different types of epilepsy.

The EpiNet project as a means to undertake research in clinical practice

The EpiNet project was established in 2009 under the auspices of the New Zealand chapter of the International League Against Epilepsy (NZLAE) to try to increase the proportion of doctors and patients with epilepsy who participate in comparative effectiveness research.6, 7

EpiNet is an online database that is versatile, easy to use, and comprehensive (http://www.epinet.co.nz). Information is collected for a range of axes, including seizure type, epilepsy syndrome, etiology, investigations, and treatment history. The database can be accessed by approved investigators who log on to a secure, password‐protected website. The information contained within it is owned by the individual investigators participating in the EpiNet collaboration, and access to the information is controlled by a steering committee.

An international pilot study was undertaken from September 2010 to November 2011. Adult and pediatric neurologists from 13 countries participated. By the end of November 2011, more than 1,050 patients had been registered by 55 neurologists or research assistants working in 25 centers in 13 countries.8 All patients gave their informed consent to have their information stored in the database.

As of the end of October 2016, over 10,000 patients from 29 countries had been entered into the database.

Because the goal of EpiNet is to integrate research with clinical practice, the database has also been designed to function as an electronic clinical epilepsy record for the patient. The results of a patient's investigations and the full drug history are available in a single document. Records can be shared with other physicians from the patient's country if the investigator and patient so wish. Investigators have a database of their own patients, and each investigator can download information from their own patients' records and analyze these without any permission required of the EpiNet study group.

The EpiNet steering committee wants the project to be a grassroots collaboration. We want researchers around the world to identify research questions and develop research protocols to answer these questions. Researchers who are not currently participating in EpiNet are welcome to perform trials using this platform. We also want people with epilepsy to have a major role in directing the research. We want to run large, simple, pragmatic, investigator‐led, multinational trials and prospective population‐based cohort studies at low cost. We intend to ask questions that pharmaceutical companies are not asking.

Data quality: EpiNet validation study and accreditation of investigators

To ensure that research is of high‐quality, and that studies are comparable, investigators must have comparable skills and exhibit a satisfactory interrater agreement. Data collection should be performed in a standard way. The EpiNet database has been designed to collect comprehensive information about patients with epilepsy, in a user‐friendly but standardized manner.

The validity of research studies depends on the level of confidence that those included actually had the condition of interest. Epilepsy is often a clinical diagnosis, and it is well recognized that the interrater agreement between clinicians in the diagnosis and syndromic classification of epilepsy is only moderate, even between experienced neurologists.9, 10, 11, 12 Ongoing education is important. Internal rules and checklists can be established within a database to ensure that patients recruited for studies do meet all the entry criteria, and an expert panel can review details of patients who might be eligible for a study before they are recruited. The level of agreement between investigators can also be improved by using predefined diagnostic criteria and panel discussion.

The EpiNet steering committee intends to accredit investigators for particular studies. We envisage multiple levels of accreditation, which are not necessarily arranged in a hierarchical manner. Investigators who have a particular subspecialty interest might be accredited for some studies, but not others. The EpiNet steering committee will oversee the accreditation processes but will not necessarily arrange accreditation for specific studies; these processes may be determined by the particular study coordinators. Accreditation for one study may require assessing a standardized set of case histories or reviewing electroencephalograms (EEGs); accreditation for another study may be given after investigators have attended a workshop or may be based on formal or informal qualifications, such as the International League Against Epilepsy's proposed Competencies for Epileptologists.13

This paper addresses the processes we have developed to accredit investigators for the first of the multicenter pragmatic randomized controlled trials that are being undertaken by the EpiNet study group.

Aims of EpiNet validation study

The EpiNet validation study was undertaken from December 2013 through 2014. It had two aims: (1) to accredit investigators for the EpiNet‐First trials. These are randomized controlled trials in patients with previously untreated epilepsy (details regarding the EpiNet‐First trials are given below and in the discussion); (2) to determine how much variability there is between neurologists and epileptologists in diagnoses.

This paper addresses accreditation for the EpiNet‐First trials. The EpiNet steering committee wanted to ensure that all investigators who participate in these trials diagnose epilepsy and seizure types in a consistent way. For practical purposes, this involves meeting particular criteria determined by the steering committee. The steering committee also wanted to establish a process that would allow future investigators who had not participated in the validation study to get accredited at a later date without having to assess the complete set of 30 cases.

A separate paper analyzing levels of agreement between neurologists and epileptologists is being prepared. This will look at levels of agreement for the entire sample, for subgroups from differing geographic areas, and for subgroups determined by levels of specialization and experience.

Methods

Initially, 40 scenarios of real patients who had paroxysmal attacks of various types were prepared by P.B. The six members of the EpiNet steering committee then independently assessed the 40 cases to determine:

How likely it was that each patient had experienced one or more epileptic seizures; the options presented for level of confidence of a diagnosis of epilepsy were: certain (seizure recorded on EEG); beyond reasonable doubt; probable; possible; unlikely; not epilepsy.

For subsequent analysis, these six options were consolidated into three broader categories: epilepsy (certain, beyond reasonable doubt, and probable); possible epilepsy; and not epilepsy (unlikely; not epilepsy).

The nature of the attacks, with seizures categorized by the ILAE 2010 classification schema14 (focal seizures; generalized seizures; epileptic seizures, but nature uncertain; nature of attacks uncertain [possibly epileptic]; attacks not epileptic).

The etiology of the epilepsy, in broad categories, using the ILAE 2010 classification schema14 (genetic/presumed genetic; structural/metabolic; unknown; not epilepsy).

After discussion within the steering committee, 10 cases were excluded, either because they were too similar to other cases or because more than one of the six members of the steering committee disagreed with the consensus diagnosis. The final set of 30 cases chosen for the validation study included 21 patients with epileptic seizures and 9 with seizure mimics.

The EpiNet steering committee invited neurologists, epileptologists, and pediatricians who had expressed an interest in the EpiNet‐First trials and other physicians with an interest in epilepsy to participate in this validation study. Participants were invited to assess the 30 cases and determine diagnosis, seizure type, and etiology, as detailed above, via the online EpiNet database. Twenty‐three of the cases were presented in two steps, with clinical information in step 1 and results of investigations in step 2. Participants were required to give their opinions regarding epilepsy diagnosis, seizure type, and epilepsy etiology after each step.

The EpiNet steering committee chose a subset of cases that could be used to accredit investigators for the EpiNet‐First trials.15, 16 These trials involve patients with newly diagnosed epilepsy, and the steering committee therefore decided that the most important requirement was that investigators can correctly determine whether patients have epileptic seizures. The second requirement was that investigators can correctly distinguish focal seizures from generalized seizures because there are separate trials for patients with different seizure types.15, 16 Following discussion, the EpiNet steering committee identified 18 cases to be used for accreditation for the EpiNet‐First trials. Some of the patients have epilepsy and some have other common paroxysmal attacks. These cases represent the types of patients who are commonly seen in first seizure clinics but also include several patients with established epilepsy. The steering committee determined by consensus which responses were acceptable for these cases. For most cases, a range of responses was accepted; for instance, if the steering committee thought the patient clearly has epilepsy, then only the options “certain,” “beyond reasonable doubt,” or “probable” were accepted; if any members of the steering committee considered that there was diagnostic uncertainty, then the “possible” option was also considered acceptable. The steering committee took the approach that it is better to err on the side of not including in trials some people who might actually have epilepsy than to include people who do not have epilepsy.

When accrediting individual investigators, we initially focused on a subset of 12 cases. To be accredited, investigators could not diagnose epilepsy (“certain,” “beyond reasonable doubt,” or “probable”) in any of the cases that the steering committee had determined did not have epilepsy, but they could diagnose one of the epilepsy cases as not having epilepsy (“unlikely” or “not epilepsy”). Investigators could not diagnose generalized seizures in cases the steering committee determined had focal seizures, and vice versa; however, the option “epileptic seizures, but nature uncertain” was considered acceptable in some cases. If investigators made an error with 1 of the initial 12 cases, the final 6 of the 18 cases were reviewed. When assessing the full 18 cases, investigators could be accredited if they diagnosed 1 of 6 nonepilepsy patients as having possible epilepsy but made no other false‐positive errors in diagnosis, and they could make no more than one error regarding seizure classification.

Investigators' responses on the etiology form were not considered in the accreditation process.

We have included as supporting information the 12 cases that were not used for the accreditation process; these demonstrate the type of material that is included in the case scenarios. We have not included the 18 cases that were actually used to accredit investigators because the same cases are still being used to accredit new investigators.

We calculated kappa values to look at the levels of agreement among all participants and separately for those who were accredited for the EpiNet‐First trials. The kappa statistic is a measure of agreement that is corrected for the level of agreement that would be expected to occur by chance alone. It is calculated as the proportion of agreement beyond that expected by chance (the achieved beyond chance agreement as a proportion of the attainable beyond chance agreement). Kappa values range from −1 to 1. According to Landis and Koch, kappa values can be classified as poor (less than chance—kappa below 0), slight (0.01–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61– 0.80), and almost perfect agreement (0.81–1.00).17

Results

One hundred eighty‐nine physicians from 36 countries assessed the 30 cases. The distribution of responses for these 189 physicians for each of the cases is shown in Tables 1, 2, 3.

Table 1

Likelihood of epilepsy (189 participants)

	Step 1			Step 2
	Epilepsy	Possible epilepsy	Not epilepsy	Epilepsy	Possible epilepsy	Not epilepsy
Epilepsy cases
1	169 (90.4)	11 (5.9)	7 (3.7)	176 (94.1)	7 (3.7)	4 (2.1)
3	179 (94.7)	6 (3.2)	4 (2.1)	180 (95.2)	5 (2.7)	4 (2.1)
4	189 (100)	0 (0.0)	0 (0.0)	189 (100)	0 (0.0)	0 (0.0)
5	123 (65.4)	38 (20.2)	27 (14.4)	115 (61.2)	40 (21.3)	33 (17.6)
8	186 (99.5)	1 (0.5)	0 (0.0)	187 (100)	0 (0.0)	0 (0.0)
9	88 (46.8)	61 (32.5)	39 (20.7)	145 (77.1)	34 (18.1)	9 (4.8)
10	107 (56.6)	55 (29.1)	27 (14.3)	135 (71.4)	31 (16.4)	23 (12.2)
11	186 (98.9)	2 (1.1)	0 (0.0)	186 (98.9)	2 (1.1)	0 (0.0)
13	165 (87.8)	18 (9.6)	5 (2.7)	179 (95.2)	7 (3.7)	2 (1.1)
14	135 (87.7)	18 (11.7)	1 (0.7)	NA	NA	NA
15	183 (96.8)	4 (2.1)	2 (1.1)	186 (98.4)	2 (1.1)	1 (0.5)
17	186 (98.9)	1 (0.5)	1 (0.5)	186 (98.9)	1 (0.5)	1 (0.5)
18	119 (62.9)	47 (24.9)	23 (12.2)	NA	NA	NA
20	174 (92.1)	14 (7.4)	1 (0.5)	183 (96.8)	5 (2.6)	1 (0.5)
22	135 (72.9)	18 (9.6)	34 (18.2)	174 (93.1)	9 (4.8)	4 (2.1)
23	159 (84.1)	22 (11.6)	8 (4.2)	160 (84.7)	21 (11.1)	8 (4.2)
24	131 (69.3)	25 (13.2)	33 (17.5)	150 (79.4)	16 (8.5)	23 (12.2)
26	117 (61.9)	48 (25.4)	24 (12.7)	171 (90.5)	15 (7.9)	3 (1.6)
28	174 (92.1)	15 (7.9)	0 (0)	186 (98.4)	3 (1.6)	0 (0.0)
29	184 (97.4)	2 (1.1)	3 (1.6)	NA	NA	NA
30	151 (79.9)	31 (16.4)	7 (3.7)	173 (91.5)	14 (7.4)	2 (1.1)
Not epilepsy cases
2	11 (5.8)	25 (13.2)	153 (80.9)	8 (4.2)	24 (12.7)	157 (83.1)
6	32 (17.0)	50 (26.6)	106 (56.4)	14 (7.5)	28 (14.9)	146 (77.7)
7	12 (6.4)	29 (15.3)	148 (78.3)	13 (6.9)	29 (15.3)	147 (77.8)
12	16 (8.5)	26 (13.8)	147 (77.8)	NA	NA	NA
16	23 (12.2)	30 (15.9)	136 (72.0)	20 (10.6)	30 (15.9)	139 (73.5)
19	23 (12.2)	38 (20.1)	128 (67.7)	NA	NA	NA
21	27 (14.4)	73 (38.8)	88 (46.8)	NA	NA	NA
25	23 (12.2)	29 (15.3)	137 (72.5)	10 (5.3)	20 (10.6)	159 (84.1)
27	10 (5.3)	7 (3.7)	174 (92.1)	NA	NA	NA

NA, not available.

Table showing the number of investigators who chose a particular option for the likelihood that each patient had epileptic seizures. Percentages are shown in italics in parentheses. The cases are separated into epilepsy and not‐epilepsy cases as determined by the EpiNet steering committee.

Table 2

Seizure diagnoses (189 participants)

Case		Step 1					Step 2
		Focal	Generalized	Epilepsy‐NU	Attack‐NU	Not epileptic	Focal	Generalized	Epilepsy‐NU	Attack‐NU	Not epileptic
Epilepsy
1	Gen	31 (16.8)	83 (44.9)	53 (28.7)	12 (6.5)	6 (3.2)	37 (20.3)	86 (46.5)	51 (27.6)	8 (4.3)	3 (1.6)
3	Focal	180 (97.3)	0 (0.0)	0 (0.0)	3 (1.6)	2 (1.1)	180 (97.3)	0 (0.0)	0 (0.0)	3 (1.6)	2 (1.1)
4	Gen	15 (8.0)	164 (87.2)	7 (3.7)	2 (1.1)	0 (0.0)	8 (4.3)	174 (92.6)	5 (2.6)	1 (0.7)	0 (0.0)
5	Focal	83 (44.6)	29 (15.6)	17 (9.1)	34 (18.3)	23 (12.4)	81 (43.6)	25 (13.4)	18 (9.7)	33 (17.8)	29 (15.6)
8	Gen	12 (6.4)	165 (88.2)	9 (4.8)	1 (0.5)	0 (0.0)	3 (1.6)	181 (96.8)	3 (1.6)	0 (0.0)	0 (0.0)
9	Focal	105 (56.8)	1 (0.5)	1 (0.5)	44 (23.8)	34 (18.4)	155 (84.8)	1 (0.5)	4 (2.6)	12 (6.5)	13 (7.0)
10	Focal	144 (77.0)	0 (0.0)	0 (0.0)	27 (14.4)	16 (8.6)	157 (84.0)	0 (0.0)	1 (0.5)	16 (8.6)	13 (7.0)
11	Gen	2 (1.1)	163 (87.6)	20 (10.6)	1 (0.5)	0 (0.0)	1 (1.5)	173 (93.0)	11 (5.9)	1 (0.5)	0 (0.0)
13	Focal	134 (71.7)	16 (8.6)	23 (12.3)	9 (4.8)	5 (2.7)	155 (82.9)	15 (8.0)	12 (6.4)	3 (1.6)	2 (1.1)
14	Ep – NU	19 (10.1)	94 (50.0)	66 (35.1)	7 (3.7)	2 (1.1)	NA	NA	NA	NA	NA
15	Gen	6 (3.2)	170 (90.4)	9 (4.8)	2 (1.1)	1 (0.5)	2 (1.1)	180 (95.7)	4 (2.1)	1 (0.5)	1 (0.5)
17	Focal	164 (87.7)	15 (8.0)	6 (3.2)	1 (0.5)	1 (0.5)	163 (87.2)	16 (8.6)	6 (3.2)	1 (0.5)	1 (0.5)
18	Focal	139 (73.9)	2 (1.1)	4 (2.1)	27 (14.4)	16 (8.5)	NA	NA	NA	NA	NA
20	Focal	53 (28.3)	77 (41.2)	43 (23.0)	7 (3.7)	7 (3.7)	109 (58.6)	48 (25.8)	27 (14.5)	1 (0.5)	1 (0.5)
22	Gen	20 (12.8)	110 (59.1)	10 (5.4)	21 (11.3)	25 (13.4)	15 (8.1)	15. (80.7)	10 (5.4)	6 (3.2)	5 (2.7)
23	Gen	8 (4.3)	116 (61.7)	55 (29.3)	6 (3.2)	3 (1.6)	8 (4.3)	116 (61.7)	55 (29.3)	6 (3.2)	3 (1.6)
24	Focal	139 (74.7)	0 (0.0)	3 (1.6)	26 (14.0)	18 (9.7)	155 (83.3)	0 (0)	3 (1.6)	15 (8.1)	13 (7.0)
26	Gen	19 (10.1)	1.4 (55.3)	11 (5.9)	36 (19.2)	18 (9.6)	2 (1.1)	172 (91.5)	6 (3.2)	7 (3.7)	1 (0.5)
28	Focal	50 (26.7)	69 (36.7)	59 (31.4)	10 (5.3)	0 (0.0)	100 (53.2)	42 (22.3)	44 (23.4)	2 (1.1)	0 (0.0)
29	Focal	180 (95.7)	1 (0.5)	3 (1.6)	1 (0.5)	3 (1.6)	NA	NA	NA	NA	NA
30	Gen	85 (45.2)	26 (24.5)	27 (14.7)	21 (11.2)	9 (4.8)	36 (19.2)	121 (64.4)	22 (11.7)	6 (3.2)	3 (1.6)
Not epilepsy
2		5 (2.7)	4 (2.2)	5 (2.7)	27 (14.6)	144 (77.8)	4 (2.2)	3 (1.6)	5 (2.7)	25 (13.5)	148 (80.0)
6		32 (17.2)	5 (2.7)	3 (1.6)	41 (22.0)	105 (56.5)	16 (8.6)	3 (1.6)	1 (0.5)	27 (14.5)	139 (74.3)
7		4 (2.2)	3 (1.6)	2 (1.1)	38 (20.4)	139 (74.7)	5 (2.7)	4 (2.1)	2 (1.1)	37 (19.9)	138 (74.2)
12		17 (9.0)	0 (0.0)	1 (0.5)	32 (17.0)	138 (73.4)	NA	NA	NA	NA	NA
16		20 (10.6)	3 (1.6)	5 (2.7)	28 (14.9)	132 (70.2)	18 (9.6)	2 (1.1)	5 (2.7)	31 (16.5)	132 (70.2)
19		24 (12.9)	2 (1.1)	2 (1.1)	35 (18.8)	123 (66.1)	NA	NA	NA	NA	NA
21		24 (12.8)	5 (2.7)	9 (4.8)	68 (36.4)	81 (43.3)	NA	NA	NA	NA	NA
25		22 (11.7)	4 (2.1)	6 (3.2)	32 (17.0)	124 (66.0)	10 (5.3)	4 (2.1)	4 (2.1)	21 (11.2)	149 (79.3)
27		4 (2.1)	2 (1.1)	2 (1.1)	14 (7.5)	166 (88.3)	NA	NA	NA	NA	NA

NA, not available; NU, nature uncertain.

Table showing the number of investigators who chose a particular option for the nature of each patient's attacks. Percentages are shown in italics in parentheses. The diagnoses in column 2 are the seizure types determined by the EpiNet steering committee.

Table 3

Etiology (189 participants)

Case		Step 1				Step 2
		Genetic	Structural/metabolic	Unknown	Epilepsy unlikely	Genetic	Structural/metabolic	Unknown	Epilepsy unlikely
Epilepsy
1	Genetic	43 (23.4)	22 (12.0)	108 (58.7)	11 (6.0)	58 (31.5)	21 (11.4)	98 (53.3)	7 (3.8)
3	Struct‐ met	0 (0.0)	176 (93.1)	9 (4.8)	4 (2.1)	0 (0.0)	181 (95.6)	5 (2.7)	3 (1.6)
4	Genetic	160 (87.4)	7 (3.8)	16 (8.7)	0 (0.0)	165 (90.2)	4 (2.2)	14 (7.7)	0 (0.0)
5	Unknown	50 (28.1)	11 (6.2)	83 (46.6)	34 (19.1)	46 (25.8)	10 (5.6)	82 (46.1)	40 (22.3)
8	Genetic	155 (87.1)	3 (1.7)	19 (10.7)	1 (0.6)	164 (91.1)	0 (0.0)	16 (8.9)	0 (0.0)
9	Unknown	4 (2.1)	50 (26.7)	81 (43.6)	51 (27.4)	14 (7.5)	53 (28.5)	92 (49.5)	27 (14.5)
10	Struct‐ met	0 (0.0)	106 (57.3)	54 (29.2)	25 (13.5)	0 (0.0)	133 (71.5)	35 (18.8)	18 (9.7)
11	Genetic	163 (88.1)	0 (0.0)	22 (11.9)	0 (0.0)	166 (89.3)	0 (0.0)	20 (10.8)	0 (0.0)
13	Unknown	64 (35.6)	23 (12.8)	86 (47.8)	7 (3.9)	87 (47.8)	17 (9.3)	77 (42.3)	1 (0.6)
14	Unknown	48 (26.0)	24 (13.0)	104 (56.2)	9 (4.9)	NA	NA	NA	NA
15	Genetic	150 (82.0)	3 (1.6)	27 (14.8)	3 (1.6)	158 (85.4)	2 (1.1)	23 (12.4)	2 (1.1)
17	Unknown	10 (5.5)	116 (63.4)	55 (30.1)	2 (1.1)	8 (4.4)	118 (64.5)	55 (30.1)	2 (1.1)
18	Unknown	17 (9.1)	68 (36.6)	69 (37.1)	32 (17.2)	NA	NA	NA	NA
20	Unknown	99 (54.1)	9 (4.9)	70 (38.3)	5 (2.7)	112 (61.2)	14 (7.7)	55 (30.1)	2 (1.1)
22	Genetic	108 (61.4)	1 (0.6)	31 (17.6)	36 (20.5)	139 (78.1)	2 (1.1)	24 (13.5)	13 (7.3)
23	Unknown	45 (24.1)	30 (16.0)	100 (53.5)	12 (6.4)	49 (26.2)	25 (13.4)	103 (55.1)	10 (5.4)
24	Struct‐ met	0 (0.0)	133 (71.5)	22 (11.8)	31 (16.7)	0 (0.0)	159 (85.0)	9 (4.8)	19 (10.2)
26	Genetic	91 (50.0)	9 (5.0)	45 (24.7)	37 (20.3)	154 (84.2)	3 (1.6)	20 (10.9)	6 (3.3)
28	Unknown	129 (70.5)	0 (0.0)	50 (27.3)	4 (2.1)	120 (64.9)	10 (5.4)	53 (28.7)	2 (1.1)
29	Struct‐ met	0 (0.0)	183 (96.8)	1 (0.5)	5 (2.6)	NA	NA	NA	NA
30	Genetic	32 (17.1)	51 (27.3)	83 (44.4)	21 (11.2)	92 (49.7)	29 (15.7)	57 (30.8)	7 (3.8)
Not epilepsy
2		1 (0.6)	11 (6.1)	12 (6.7)	156 (86.7)	1 (0.6)	9 (5.0)	11 (6.2)	158 (88.3)
6		5 (2.8)	19 (10.4)	22 (12.1)	136 (74.7)	1 (0.5)	6 (3.2)	10 (5.4)	167 (90.8)
7		0 (0.0)	7 (3.8)	19 (10.3)	159 (86.0)	0 (0.0)	9 (4.9)	17 (9.2)	159 (86.0)
12		21 (11.4)	7 (3.8)	10 (5.4)	147 (79.5)	NA	NA	NA	NA
16		2 (1.1)	13 (7.3)	20 (11.2)	143 (80.3)	2 (1.1)	11 (6.1)	18 (10.0)	149 (82.8)
19		0 (0.0)	11 (6.1)	25 (13.8)	145 (80.1)	NA	NA	NA	NA
21		1 (0.6)	10 (5.6)	50 (27.9)	118 (65.9)	NA	NA	NA	NA
25		1 (0.5)	25 (13.5)	17 (9.2)	142 (76.7)	1 (0.5)	14 (7.5)	10 (5.4)	161 (86.6)
27		1 (0.6)	5 (2.8)	8 (4.4)	168 (92.3)	NA	NA	NA	NA

NA, not available.

Table showing the number of investigators who chose a particular option for the etiology of each patient's attacks. Percentages are shown in italics in parentheses. Column 2 shows the etiology as determined by the EpiNet steering committee.

Of the 21 cases that the steering committee determined had epilepsy, agreement with the steering committee regarding the diagnosis at step 1 ranged from 47% to 100% (Table 1). The mean percentage of investigators who diagnosed epilepsy for the 21 cases at step 1 was 83%. For the 18 cases where information was provided in two steps, the mean percentage of investigators who diagnosed epilepsy increased from 85% to 90% after step 2.

Of the 9 cases that the steering committee determined did not have epilepsy, agreement among participants with the steering committee at step 1 ranged from 47% to 92%. The mean percentage of investigators who diagnosed “not epilepsy” for the 9 cases at step 1 was 72%. For the 5 cases where information was provided in two steps, the mean percentage of investigators who diagnosed “not epilepsy” increased from 72% to 79% after step 2.

The responses for seizure types are shown in Table 2. The steering committee considered that 9 of the patients had generalized seizures; the mean proportion of participants who diagnosed generalized seizures for these 9 cases at step 1 was 67%, increasing to 80% at step 2. The steering committee considered that 11 patients had focal seizures; the mean proportion of participants who diagnosed focal seizures for these 11 cases at step 1 was 67%. For the 9 cases where information was provided in two steps, the mean percentage of investigators who diagnosed focal seizures at step 1 was 63%, increasing to 75% at step 2.

The responses for etiology are shown in Table 3. Of the 8 cases that the steering committee determined had a genetic etiology for their epilepsy, agreement with the steering committee ranged from 31% to 91% at step 2. The mean proportion of participants who diagnosed a genetic etiology for these 8 cases at step 1 was 62%, and increased to 74.9% at step 2.

Of the 4 cases that the steering committee determined had a structural/metabolic etiology for their epilepsy, agreement with the steering committee ranged from 72% to 96% at step 1. The mean proportion of participants who diagnosed a structural/metabolic etiology for these 4 cases at step 2 was 87%.

The EpiNet steering committee was uncertain of the etiology in 9 of the patients with epilepsy. The mean proportion of participants who selected “unknown” for the etiology for these 8 cases was only 48% at step 1. After step 2, 56% of participants selected “unknown” for the etiology.

Accreditation criteria

When we focused on the 18 cases that the steering committee determined should be used for accreditation, 105 (55.6%) of the 189 physicians who assessed all 30 cases in the EpiNet validation study met the standard for accreditation.

The kappa value for the diagnosis of epilepsy at step 1 for all 189 participants who completed the 30 cases was moderately good, at 0.60, although it was only fair for seizure type (0.39) and etiology (0.38). For the 105 physicians who met the criteria for accreditation when assessing the subset of 18 cases, the kappa value for all 30 cases at step 1 increased to 0.70 for the diagnosis of epilepsy, 0.47 for seizure type, and 0.45 for etiology.

Discussion

The validation study and accreditation process

Many of the patients included in this study had classical presentations, and we expected most participants to find assessment of these cases easy. More than 90% of participants correctly diagnosed epilepsy in 10 of the 21 epilepsy cases, and all 189 participants correctly diagnosed epilepsy in case 4. However, we also deliberately included several “gray cases.” This was done to determine how participants manage diagnostic uncertainty because most physicians see patients where there is genuine uncertainty regarding the diagnosis. Practicing clinicians sometimes decide that it is appropriate to treat these patients, but including people who do not actually have epilepsy in trials of epilepsy treatment will increase the “noise” and reduce the power of the trials. If these patients are randomized, then potential bias will be minimized because similar patients will end up in all arms of a trial. The effect of the added noise can be reduced by increasing the number of patients recruited to the trial; however, this adds considerably to the cost and time taken to complete the study, and trials may still be underpowered if too many patients who do not have epilepsy are included. It is probably impossible to completely exclude all such patients from research studies, but efforts need to be taken to minimize the number of people who do not have epilepsy who are recruited into trials. Patients in whom there is a high level of diagnostic uncertainty should not be recruited into clinical trials.

We have not given details of the specific cases that were used for accreditation, nor the exact criteria. This is because we intend to use these cases for accrediting new investigators. We believe that an accreditation process is required for diagnostic rigor, though we are also aware that this process may be challenging to potential investigators; however, we believe that it is important for the integrity of the trials that a formal accreditation process is established.

We have undertaken the validation study and accreditation process to reassure the EpiNet study group and the wider epilepsy community that investigators who recruit patients for the trials can accurately diagnose epilepsy and the seizure type under investigation. The steering committee comprises adult and pediatric neurologists and epileptologists from five different countries and three continents. We cannot be certain that the diagnoses made by members of the steering committee for the validation study are correct, though we believe that they are. Relatively few of the patients had diagnostic video‐EEG monitoring, which we would consider the gold standard for epilepsy diagnosis. At the very least, the accreditation process ensures that all those participating in the EpiNet‐First trials have a similar approach to diagnosis of epilepsy and epileptic seizures. The physicians accredited for the EpiNet‐First trials had a high level of interrater agreement (k = 0.70). This is a higher level of agreement than the interrater agreement between pediatric epileptologists in a study of the reliability of diagnoses of first seizures in children (k = 0.41),12 and neurologists in adult studies of the diagnosis of first seizures (k = 0.58),9 and the diagnosis of syncope versus seizure (k = 0.4).10 The higher kappa value seen in our validation study may be due to the expertise of assessors and differences in case selection. Although most of our cases had experienced few attacks, there were several cases with a longer history. In the pediatric study,12 although the assessing panels comprised epileptologists, the study was enriched for cases in which there was a high degree of diagnostic uncertainty. Both adult studies cited9, 12 were prospective studies, with an unselected case mix.

Studies under way using EpiNet

Randomized controlled trials within EpiNet

The EpiNet‐First trials commenced in New Zealand in late 2015.15, 16 These are pragmatic, unblinded, randomized controlled trials in patients with previously untreated epilepsy. Patients who have had two or more seizures, and who have not previously received AEDs, are randomized to one of four AEDs. Five separate trials are being run in parallel. The specific drugs included in each trial depend on the patient's seizure type and the suitability of sodium valproate; patients with focal seizures are randomized to levetiracetam, carbamazepine, or lamotrigine (Trial 1). Patients with generalized seizures are randomized to levetiracetam or sodium valproate if valproate is deemed a suitable drug (Trial 2); if valproate is not considered suitable (e.g., in women of child‐bearing age), then patients with generalized seizures are randomized to receive either levetiracetam or lamotrigine (Trial 3). If the investigator is uncertain of the seizure type, then patients are randomized to receive either levetiracetam, sodium valproate, or lamotrigine (Trial 4) or, if valproate is not considered suitable, to either levetiracetam or lamotrigine (Trial 5).

The EpiNet‐First trials are based closely on the SANAD‐ll trials.18 The SANAD‐ll (Standard and New Anti‐Epileptic Drugs) trials are unblinded pragmatic trials being conducted in the United Kingdom in which patients with newly diagnosed epilepsy, who have not previously been treated with AEDs, are randomized to receive either sodium valproate or lamotrigine (“standard AEDs”) or levetiracetam or zonisamide (“newer AEDs”). The standard drugs to which the newer drugs are being compared are the drugs that were deemed most effective in the SANAD‐l studies.19, 20

The EpiNet‐First trials have been registered on the Australia and New Zealand Clinical Trials registry, and patients are currently being recruited. We encourage physicians worldwide to join these trials. Physicians can participate in one, some, or all of the trials. Formal approval from each center's institutional review board or research governance committee and ethics committee is required before physicians can participate. The coordinating center in Auckland is happy to help individual physicians obtain approval for these trials. Investigators are not being paid to enter patients into these trials.

EpiNet registries

Registries have been established within EpiNet to collect prospective data on patients who have had a first seizure or are discontinuing treatment. There is also a registry for patients who start a first AED who do not get recruited into one of the EpiNet‐First trials. Investigators are encouraged to systematically record information in these registries of all eligible patients they see. Proposals for new registries are welcome.

Future directions

The database has evolved as we have received feedback from users. It has now reached a considerable level of sophistication. A large amount of information regarding patients and their seizures can be collected; however, it is not necessary to collect all this information on all patients, and some questions only appear in particular contexts.

New forms can be added to the database to capture information regarding different aspects of epilepsy. A form to collect information relating to status epilepticus has recently been released, and any EpiNet investigator can use it. Anyone with an interest in a particular type of epilepsy or treatment modality is welcome to propose a study and to help with development of the required forms.

Investigators are welcome to enter information on all their patients or to participate in single studies if they wish. Although the platform has been established to encourage multicenter studies, it is also possible to run studies that are restricted to a particular institution or country. EpiNet is suitable to establish a registry to determine the outcome associated with the introduction of a new AED in a particular country or region. The database can be used to collect information required for epidemiological studies.

We believe that using an Internet‐based platform such as EpiNet might be the only way in which sufficient data can be gained regarding outcomes in rare syndromes or when the outcome itself is relatively uncommon, such as sudden unexpected death in epilepsy (SUDEP).

Conclusion

We encourage physicians to participate in comparative effectiveness research studies. EpiNet has been designed as a platform to undertake clinical research in epilepsy at relatively low cost. EpiNet is available to all specialist physicians who treat people with epilepsy. Physicians are invited to participate in the formal registries and record in a systematic manner the treatments they institute and the outcomes of treatment. Physicians are also invited to participate in simple pragmatic clinical trials and to propose new trials that could be undertaken using this platform; the EpiNet‐First trials are now under way, and we are looking for new investigators for these studies. In an effort to ensure appropriate patients are enrolled in the trials, physicians need to be accredited for these trials. We see the accreditation process as a way of raising standards and would therefore encourage physicians to get accredited and to participate in these trials (http://epinet.co.nz/index.cfm?PageID=11).

Neurologists, epileptologists, and people affected by epilepsy who wish to participate in clinical research are invited to visit the EpiNet website (http://www.epinet.co.nz) or to contact the authors.

Disclosure

None of the authors has any conflict of interest to disclose. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.

Appendix S1. The 12 cases that were not used for the accreditation process. These demonstrate the type of material that is included in the case scenarios.

Click here for additional data file.

Epilepsia Open
Argentina
Analía Calle
Claudio Palacios
Australia
Anita Cairns
Patrick Carney
Donald Craig
Deepak Gill
Sachin Gupta
Cecilie Lander
Hanka Laue‐Gizzi
Natalie Hitchens
Michelle Kiley
Nicholas Lawn
Elizabeth Reyneke
Kate Riney
Meng Tan
Michael Tan
Mark Thieban
Emma Whitham
Chong Wong
Belgium
Benjamin Legros
Michel Ossemann
Germain van Rijckevorsel
Brazil
Ana Gabriela Ferrari Strang
Angela Gifoni
Linden Helio
Bruno Monnerat
Canada
Paula Brna
Elizabeth Donner
Stephanie Jacques
Nathalie Jette
Richard McLachlan
Ismail Mohamed
Thi Phuoc Yen Tran
China
Xiao Bo
Song Fan
Yang Guang
Ming Li
Kang Wang
Shouwen Zhang
Colombia
Lady Ladino
Denmark
Jakob Christensen
Margarethe Sophie Kӧlmel
Marina Nikanorova
Finland
Annukka Uusitalo
Paivi Vieira
France
Stephane Auvin
Georgia
Maia Alkhidze
Tamar Ediberidze
Nino Gogatishvili
Tamar Jishkariani
Germany
Dieter Dennig
Anja Grimmer
Rosa Michaelis
Susanne Schubert‐Bast
Caspar Stephani
Stefan Stodieck
Martin Vollbrandt
Andreas Zellner
Greece
Dimitrios Zafeiriou
Hungary
Andras Fogarasi
Peter Halasz
India
Rameshwar Nath Chaurasia
Satish Jain
Raj Nair
Pravar Passi
Surekha Rajadhyaksha
Sita Jayalakshmi Sattaluri
Harshuti Shah
Kavita Srivastava
Vrajesh Udani
Ireland
Daniel Costello
Italy
Umberto Aguglia
Arnaldo Bartocci
Paolo Benna
Simone Beretta
Edoardo Ferlazzo
Daniela Laino
Alberto Spalice
Pasquale Striano
Alberto Verrotti
Clara Zanchi
Jamaica
Amza Ali
Malaysia
Kheng Seang Lim
Hui Jan Tan
Mexico
Alfredo Ramirez
Ildefonso Rodriguez‐Leyva
New Zealand
Neil Anderson
Alan Barber
Pietro Cariga
James Cleland
Nicholas Child
Suzanne Davis
Viswas Dayal
Cameron Dickson
John Doran
Roderick Duncan
Richard Frith
Pratima Giri
Michael Herd
David Hutchinson
Ivan Iniesta
Jayaganth Jayabal
Bethany Jones
Justin Kao
Dean Kilfoyle
Nicole McGrath
John Mottershead
Colette Muir
Melinda Nolan
Jennifer Pereira
Anna Ranta
Ian Rosemergy
Sneha Sadani
Mark Simpson
Claire Spooner
Paul Timmings
Elizabeth Walker
Diana Wei
Ernest Willoughby
Edward Wong
Teddy Wu
Nigeria
Birinus Ezeala‐Adikaibe
Talabi Olusola
Pakistan
Hiba Mahmud
Zarine Mogul
Peru
Julio Espinoza
Jose Hernandez Vizarreta
Portugal
Elia Maria Baeta
RuteTeotónio
Serbia
Bosanka Jocic‐Jakubi
Stevo Lukic
Slovenia
Marko Korošec
Tomaz Zgur
Spain
María Gómez Eguílaz
Sweden
Fredrik Asztely
Thailand
Pasiri Sithinamsuwan
United Kingdom
Joseph Anderson
Pauls Auce
Archana Desurkar
Khalid Hamandi
Andrew Kelso
Violeta Sanchez
Aurangzeb Sidra
Phil Smith
Tim Wehner
Gavin Winston
United States of America
Edgard Andrade
Meriem Bensalem‐Owen
Michelle Boudreau
Tracie Caller
Kevin Chapman
Geetha Chari
Kathryn Davis
Brian Droker
Mirret El‐Hagrassy
Dawn Eliashiv
Chigolum Eze
Christi Heck
Arif Kabir
Dmitriy Kolesnik
Alice Lam
Jonathan Lopez
Tammaa Maamoon
Jennifer Madan Cohen
Rama Maganti
Chinasa Nwankwo
Kristen Park
Simona Proteasa
Evan Sandok
Syndi Seinfield
Julia Toub
Elaine Wirrell
Uraguay
Marcela Arbildi
Vietnam
Truong Tran Thien