IDEAL approach to the evaluation of machine learning technology in epilepsy surgery: protocol for the MAST trial.

Epilepsy and epilepsy surgery lend themselves well to the application of machine learning (ML) and artificial intelligence (AI) technologies. This is evidenced by the plethora of tools developed for applications such as seizure detection and analysis of imaging and electrophysiological data. However, few of these tools have been directly used to guide patient management. In recent years, the Idea, Development, Exploration, Assessment, Long-Term Follow-Up (IDEAL) collaboration has formalised stages for the evaluation of surgical innovation and medical devices, and, in many ways, this pragmatic framework is also applicable to ML/AI technology, balancing innovation and safety. In this protocol paper, we outline the preclinical (IDEAL stage 0) evaluation and the protocol for a prospective (IDEAL stage 1/2a) study to evaluate the utility of an ML lesion detection algorithm designed to detect focal cortical dysplasia from structural MRI, as an adjunct in the planning of stereoelectroencephalography trajectories in children undergoing intracranial evaluation for drug-resistant epilepsy. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

It is clear that the fields of epilepsy and epilepsy surgery lend themselves ideally to the application of machine learning (ML) and artificial intelligence (AI) technologies to directly benefit patient care in areas such as: seizure detection; analysis of clinical, electrophysiological and imaging data; seizure localisation and prediction of medical and surgical treatment outcomes.1 2 Although many of these applications have already been explored, as yet, there have been few reports of ML algorithms directly being used to guide management decisions in patients.1 3–5 The difficulty in proceeding from the development of algorithms to prospective evaluation in clinical care is attributable to issues surrounding transparency, reproducibility, ethics and effectiveness and changes required in traditional clinical trial designs.6–8 In recent years, the Idea, Development, Exploration, Assessment, Long-Term Follow-Up (IDEAL) collaboration has formalised stages for the evaluation of surgical innovation and medical devices, and, in many ways, this pragmatic framework is also applicable to ML/AI technology, balancing innovation and safety (figure 1).9–12 This has been recently supplemented with a stage 0 that centres around a proportionate preclinical evaluation that balances safety while facilitating innovative first-in-human studies.12

Figure 1

Schematic of the IDEAL stages of evaluation for surgical/medical device innovation. IDEA, Idea, Development, Exploration, Assessment, Long-Term Follow-Up.

In this paper, we present the IDEAL stage 0 evaluation and protocol for a combined IDEAL stage 1/2a study for a lesion detection algorithm designed to detect focal cortical dysplasia as part of the planning of stereoelectroencephalography (SEEG) electrode trajectories in patients with drug-resistant epilepsy undergoing invasive presurgical evaluation.13 14

Algorithm background

The multicentre epilepsy lesion detection (MELD) algorithm was developed to detect focal cortical dysplasia using volumetric T1-weighted and fluid-attenuated inversion recovery (FLAIR) MRI sequences. Its utility is in aiding in the presurgical evaluation of drug-resistant epilepsy - to localise MRI lesions that may be responsible for epilepsy that may be amenable to surgical resection.

Its development and retrospective evaluation have been previously presented.13 14 Cortical surface-based features such as cortical thickness, grey-white matter interface contrast and FLAIR intensity from confirmed lesions were used to train a neural network classifier. The classifier was initially validated using a leave-one-out cross-validation approach but has since been validated on external cohorts with high sensitivity (73.7%) specificity (90.0%) and an area under the ROC curve of 0.75.14 15

IDEAL stage 0 evaluation

The IDEAL-D stage 0 evaluation involves classifying the device, assessing the risks using a failure modes and effect analyses (FMEA) and conducting an evaluation proportionate to the potential risks involved.12 Software is classified according to the associated device. Therefore, in the context of being used to plan SEEG trajectories, the MELD algorithm can be viewed as associated with a non-absorbable implant, putting it in tier 3 of the device categorisation. An FMEA shows that the main risk is the additional electrodes causing bleeding, which would be classified as occasional frequency (<1/100) and potentially of serious severity (could result in injury or impairment).16 Proportionately, the preclinical evaluation includes device, clinician and patient perspective studies, with a systems perspective gleaned from the existing literature. The cost of inserting additional electrodes is minimal compared with the cost of the SEEG procedure as a whole and within the normal clinical variation and, therefore, health-economic analyses are not required at this stage (table 1).12

Table 1

Preclinical evaluation of the MELD algorithm, classified by study type according to the IDEAL stage 0 framework

Study classification	Evidence
Device perspective	Safety and efficacy have been formally assessed in the peer-reviewed literature.13–15 From the perspective of this difficult-to-localise cohort of patients undergoing SEEG, the sensitivity of the classifier was 74% with 100% specificity (no lesions detected in controls). In 3 of these 34 patients, a focal SOZ was not identified and the classifier identified additional lesions that were not implanted.13Safety relates to identification of false positive clusters. The risk profile has been limited by sampling a maximum of three clusters per patient.
System perspective	There is clearly an established need for such algorithms as evidenced by the increasing evaluation of ‘MRI negative’ cases for epilepsy surgery, especially through SEEG. Not all patients undergoing epilepsy surgery have a SOZ identified and, even in those who do, a third do not achieve seizure freedom following resective surgery.20
Patient perspective	As part of the MAST Trial planning, a group of parents and children with epilepsy were surveyed at to assess whether the addition of up to three extra electrodes would be acceptable. Of 15 respondents (14 parents and one adolescent with epilepsy), all 15 (100%) agreed that the risk would be acceptable when balanced against the potential benefits and all 15 (100%) would enrol in the trial were they/their child to undergo SEEG implantation at our centre.
Clinician perspective	Clinicians from multiple specialties (neurosurgery, neurology, neurophysiology, neuroradiology) were involved in the initial development and retrospective evaluation studies, that showed promising performance. Specifically, these same clinicians will be involved in this prospective trial, attesting to the clinical acceptability and utility. The main users of the output will be neurosurgeons and the integration of the output into the planning software has been tested (figure 2B), although more formal learning curves will need to be assessed prior to multicentre studies.

IDEA, Idea, Development, Exploration, Assessment, Long-Term Follow-Up.; MELD, multicentre epilepsy lesion detection; SEEG, stereoelectroencephalography; SOZ, Seizure Onset Zone.

Protocol for IDEAL stage 1/2a evaluation

The aim of the MELD as an Adjunct for SEEG Trajectories (MAST) Trial, a single-arm single-centre prospective pilot (IDEAL stage 1/2a) study, is to assess whether MELD is a helpful adjunct in the planning of SEEG electrode trajectories. Patients undergoing SEEG at our centre will be eligible for inclusion. Exclusion criteria include tuberous sclerosis or other large structural abnormality and prior resective surgery. The pilot sample size is 20, chosen based on the principles of the IDEAL stage 2a.9 Fully informed consent will be required for participation, either from the parent or the child, in line with UK law. The patient pathway is shown in figure 2 and summarised below.

Figure 2

Study flowchart in the MAST trial, with routine clinical care pathway (orange) and trial-specific (blue) elements shown. MDT, multidisciplinary team; MELD, multicentre epilepsy lesion detection; SEEG, stereoelectroencephalography; MAST, MELD as an Adjunct for SEEG Trajectories.

Following routine planning of the SEEG electrode trajectories by the multidisciplinary team (MDT) using the non-invasive evaluation (clinical semiology, EEG videotelemetry and MRI±other adjuncts, figure 3A), the MRI scans, which are collected as part of the routine clinical care, will be run through the MELD algorithm. The putative lesion clusters identified by the algorithm will be reviewed by an expert neuroradiologist with expertise in presurgical evaluation of epilepsy to ensure that they are not driven by artefact. Once artefacts have been excluded, the top three putative lesion clusters will then be identified (on the ipsilateral side only if the planned implantation is unilateral). If these clusters are not already sampled by existing electrodes, up to three extra electrodes may be added to sample from each of those top three lesion clusters (figure 3B).

Figure 3

(A) Schematic of current clinical planning of electrode locations using the non-invasive evaluation by the MDT. The data from the pre-surgical evaluation are used to generate hypotheses of the location of the SOZ. The primary hypothesis in this example is the temporal lobe (green) with alternative areas involving orbitofrontal and insular cortices (yellow). Electrode trajectories are then planned to sample these areas of interest and establish important resection boundaries, with red dots and lines showing orthogonal and parallel electrodes respectively. (B) Example of planned electrodes on sagittal, axial, coronal and 3D views of a T1-weighted MRI scan. The MELD-identified lesion (grey, yellow arrow) is not sampled as the nearest electrode (yellow on the 3D panel) traverses postero-inferior to it. In this example, an extra electrode would, therefore, be added to sample the identified lesion in the depths of the supramarginal gyrus. EEG, electroencephalography; MDT, multidisciplinary team; MELD, multicentre epilepsy lesion detection; SOZ, seizure onset zone; PET, positron emission tomography; SPECT, single-photon emission computed tomography.

The primary objective is to assess the proportion of patients that had additional electrode (ie, extra electrodes implanted into unsampled identified clusters) contacts in the clinically defined seizure onset zone (SOZ). Given the small sample size, no statistical testing will be conducted, and the results will be descriptive statistics only. For each patient, we will also assess a number of secondary objectives, including blinded neurophysiological and neurosurgical assessments of whether or not the data from the additional electrodes would have affected subsequent management decisions (table 2). To gain an understanding of the situations in which the algorithm might be particularly helpful, we will also collect data on the indication for SEEG and the MDT members’ confidence in the ability of the SEEG implantation to identify a seizure onset zone. A transparent account of harms, including the number of electrodes implanted into ‘false positive’ lesions, will be reported, allowing a balanced assessment of risks and benefits prior to consideration of further stage testing. Steps taken to optimise the workflow will also be reported.

Table 2

Primary and secondary objectives of the MAST trial

Objectives	Outcome measures/endpoints
Primary objective	Assess the proportion of patients that had additional electrode contacts implanted in the SEEG-defined seizure onset zone.
Secondary objectives	For each case, we will assess: Pre-implantation confidence of the MDT members in identifying a seizure onset zone (prior to MELD information) as a measure of the ‘difficulty’ of the SEEG exploration Number of electrodes added Number of electrodes already in identified lesions Was a MELD-identified cluster part of the SOZ (and if so, how many clusters?) Would the SOZ have been identified without MELD? Blinded neurophysiological assessment of the SOZ contacts with and without additional electrodes Putative resection boundaries with and without the additional electrodes, to be modelled by a neurosurgeon Safety of adding additional electrodes

MDT, multidisciplinary team; MELD, multicentre epilepsy lesion detection; SEEG, stereoelectroencephalography; SOZ, Seizure Onset Zone.

The risks and benefits of such an approach were carefully considered during the design phase. In order to evaluate the incremental benefit of the algorithm, the clinical standard of care could not be altered and, therefore, moving existing electrodes was not considered an option. The risk of complications per additional electrode is small.16 In addition, a group of 14 parents of children with epilepsy and one adolescent with epilepsy were surveyed at the Epilepsy in Childhood: Carers Uniting with Researchers Information Day. All 15 (100%) agreed that the risk would be acceptable when balanced against the potential benefits, and all 15 (100%) would enrol in the trial were they/their child to undergo SEEG implantation.

Ethical approval has been received for this study from the UK Health Research Authority (IRAS ID 275480) and the study began recruitment in September 2020. Further details of the protocol are available at https://clinicaltrialsgov/ct2/show/NCT04383028. The study has been funded by EANS-Stryker Research Grant 2020 & The Rosetrees Trust (A2665), but the funders had no input into the design or conduct of the study. The study will be reported in according with relevant guidelines including IDEAL stage 1/2a, Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) & Standards for reporting diagnostic accuracy (STARD) guidelines.11 17 18

IDEAL stage 2b and beyond

We see the MELD algorithm as an additional part of the presurgical evaluation, which should be available to clinicians before the SEEG electrode trajectories are planned. The current algorithm was developed using single-centre data, with the clinical study being conducted at the same centre. The first step to further generalise the applicability would be to construct a large multicentre data set that accounts for variability in subjects, scan strength and protocols across centres. This is currently being constructed as part of the MELD study.19

If this should progress beyond the current stage 1/2a study, we envision conducting a stage 2b study as a feasibility randomised trial in a small number of centres, which assesses the proportion of patients in whom a SOZ is identified on SEEG with and without the benefit of the algorithm. This would allow us to optimise stability of the algorithm across centres, train other surgeons to integrate it into their clinical workflow, establish estimated effect sizes and assess feasibility of conducting a larger randomised study across multiple centres as an IDEAL stage 3 study.

Analysis of the stage 2b or 3 studies may also result in the additional findings, such as a select group of patients (eg, with interictal positron emission tomography (PET), ictal single-photo emission computed tomography (SPECT) or magnetoencephalography (MEG) concordance with the top identified lesions) in whom it may be debated that SEEG is not necessary prior to resection. If such findings were identified, the subsequent prospective clinical evaluation would have to go through the rigorous process of evaluation from stage 0 again as it would be using the algorithm in a different context to the present evaluation. However, many aspects may overlap with the current evaluation, making the process more streamlined.

Conclusion

ML technologies have much to offer to clinicians and patients in the field of neurosurgery and safe and robust evaluation is crucial to realising these opportunities. In this paper, we have outlined the IDEAL stage 0 evaluation, a protocol for an IDEAL stage 1/2a clinical trial and future directions for the use of the MELD algorithm in assisting the planning of SEEG trajectories. The robust framework balances safety and innovation, ultimately benefitting patients by improving outcomes.