Interrater reliability in visual identification of interictal high-frequency oscillations on electrocorticography and scalp EEG.

High-frequency oscillations (HFOs), including ripples (Rs) and fast ripples (FRs), are promising biomarkers of epileptogenesis, but their clinical utility is limited by the lack of a standardized approach to identification. We set out to determine whether electroencephalographers experienced in HFO analysis can reliably identify and quantify interictal HFOs. Two blinded raters independently reviewed 10 intraoperative electrocorticography (ECoG) samples from epilepsy surgery cases, and 10 scalp EEG samples from epilepsy monitoring unit evaluations. HFOs were visually marked using bandpass filters (R, 80-250 Hz; FR, 250-500 Hz) with a sampling frequency of 2,000 Hz. There was agreement as to the presence or absence of epileptiform discharges (EDs), Rs, and FRs, in 17, 18, and 18 cases, respectively. Interrater reliability (IRR) was favorable with κ = 0.70, 0.80, and 0.80, respectively, and similar for ECoG and scalp electroencephalography (EEG). Furthermore, interclass correlation for rates of Rs (0.99, 95% confidence interval [CI] 0.96-0.99) and FRs (0.77, 95% CI 0.41-0.91) were superior in comparison to EDs (0.37, 95% CI -0.60 to 0.75). Our data suggest that HFO identification and quantification are reliable among experienced electroencephalographers. Our findings support the reliability of utilizing HFO data in both research and clinical arenas.

Interrater reliability (or IRR) of ECoG and scalp EEG interictal HFOs was favorable (κ = 0.80 in ripples, and κ = 0.80 in fast ripples)

Interclass correlation for rates of ripples and fast ripples was superior in comparison to epileptiform discharges

HFO identification and quantification are reliable among experienced electroencephalographers

Given the high prevalence, morbidity, and mortality of intractable epilepsy,1 the identification of a biomarker of the epileptogenic zone—and more generally, intractability—would be of great utility in the evaluation and treatment of epilepsy

Animal and human studies have identified a disproportionate burden of high‐frequency oscillations (HFOs) in the seizure‐onset zones,2, 3 and multiple studies have linked favorable postsurgical seizure outcomes to the resection of cortical sites showing interictal HFOs, especially fast ripples (Rs; ≥250 Hz), on electrocorticography (ECoG).4, 5, 6, 7 In contrast to ECoG, the identification of HFOs with noninvasive scalp electroencephalography (EEG) has proven more challenging. Although several studies have linked beta, gamma, and ripple activity to hypsarrhythmia8 and epileptic spasms9, 10 in West syndrome, and to the seizure‐onset zones in focal epilepsy,11 it is only recently that fast ripples (FRs) have been identified with the use of subdermal scalp electrodes in adults with epilepsy,12 and with the use of standard scalp electrodes in children with epilepsy.13 Nevertheless, it has not yet been established that the visual identification and quantification of scalp HFOs is reliable.

In the only study to evaluate the interrater reliability (IRR) of HFO identification using ECoG data, the authors reported poor IRR, at least among a team with varied experience in HFO analysis.14 As such, the clinical utility of HFOs is presently limited by the lack of a standardized approach to identification and quantification. Accordingly, we set out to evaluate the IRR of both HFOs as well as conventional epileptiform discharges (e.g., spikes), on both ECoG and scalp EEG, by electroencephalographers who are experienced in the identification and quantification of HFOs.

Methods

Standard protocol approvals

The use of human subjects and the analyses presented here were approved by the institutional review board at University of California Los Angeles (UCLA). We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.

EEG selection and sample abstraction

To evaluate IRR agreement, reviewers were presented with 10 ECoG samples and 10 scalp EEG samples. Each sample was 10 s in duration. The samples of ECoG were abstracted from the intraoperative studies of 10 consecutive patients who underwent single‐stage (i.e., no extraoperative ECoG) surgical resection at UCLA Mattel Children's Hospital between July 2016 and April 2017. The samples of scalp EEG were abstracted from overnight video‐EEG studies from 10 children admitted to the UCLA Mattel Children's Hospital Epilepsy Monitoring Unit in the same period, for evaluation of either seizures or seizure‐like events. The reference studies were neither consecutive nor randomly selected. Instead, samples were abstracted from children in an effort to yield an approximately even mix of samples with and without HFOs. To accomplish this, we preferentially selected EEG studies from young children with uncontrolled epilepsy whose etiologies have been associated with HFOs (e.g., West syndrome and tuberous sclerosis complex) to obtain samples likely to harbor HFOs. Conversely, to include cases in which HFOs were less likely to be encountered, we selected EEG recordings from children who were older, and with either well‐controlled or nonexistent epilepsy. It is important to note that each sample was selected without knowledge of HFO prevalence and was reviewed in only standard view (i.e., 10 s/page, 7–15 μV/mm) prior to abstraction. Although epochs with abundant artifact obscuring the EEG studies were explicitly avoided for the purpose of sample selection, transient artifacts (such as those that might accompany “false” HFOs) were not avoided. EEG/ECoG samples were prepared by a board‐certified pediatric electroencephalographer (HN) who was not involved in marking EEG/ECoG for determination of reliability. Clinical and demographic characteristics of patients from whom ECoG and scalp EEG were abstracted are summarized in Table S1.

EEG/ECoG recording

EEG/ECoG recording was obtained using Nihon Kohden (Irvine, CA, U.S.A.) acquisition hardware and software, using at least 21 gold‐plated electrodes placed according to the International 10–20 system for scalp EEG, and grid macroelectrodes as clinically indicated for the ECoG. EEG/ECoG was acquired with a digital sampling frequency of 2,000 Hz, which defaults to a proprietary Nihon Kohden setting of a low‐frequency filter of 0.016 Hz and a high‐frequency filter of 600 Hz at the time of acquisition. For intraoperative ECoG samples, patients were maintained on narcotics and paralytics, and a minimum of 10 min was recorded after sevoflurane and propofol discontinuation to mitigate any anesthetic effects on the ECoG.4, 15 For scalp EEG studies, a minimum of 10 min of non–rapid eye movement (REM) sleep was obtained, without muscle or movement artifact.

Visual analysis of high‐frequency oscillations

All EEG/ECoG samples were reviewed independently by 2 board‐certified pediatric electroencephalographers (JYW and SAH) who are experienced in visual identification of HFOs. The order in which samples were reviewed was randomized and was therefore different for each rater. There was no time limit set for sample review. The reviewers were blinded to all clinical information and did not have access to other epochs of the EEG studies from which samples were abstracted. In addition, the reviewers were unaware of the approach to sample selection, and specifically had no expectation as to how many studies would exhibit HFOs. Using Persyst software version 13 (Persist Development Corporation; Prescott, AZ, U.S.A.) EEG reviewing software, conventional epileptiform discharges (e.g., spikes, paroxysmal fast activity) were marked with standard review settings (30 mm/s; high‐pass filter = 1 Hz; low‐pass filter = 70 Hz; notch filter = 60 Hz). A longitudinal bipolar montage was used for scalp EEG samples, and common average reference was used for ECoG samples. For ECoG samples, electrodes with a significant burden of artifact were excluded from average referencing.

Ripples (or Rs) and fast ripples (or FRs) were reviewed and marked in a side‐by‐side fashion such that standard EEG (settings as above) was displayed on the left side and HFOs were displayed (and time‐locked) on the right side. Rs were viewed with 150 mm/s, band‐pass filter 80–250 Hz, and FRs were viewed with 300 mm/s, band‐pass filter 250–500 Hz, with both Rs and FRs defined as oscillatory events with at least 4 cycles, which are clearly visible above the background signal in the filtered data (see examples in Fig. 1).

Figure 1

Examples of HFO visual analysis. A, Example of visual analysis of ECoG HFOs in case 3 (4‐year‐old girl with tuberous sclerosis complex and medically intractable focal epilepsy). Left: Original ECoG trace. Middle: Temporally expanded filtered ECoG trace for ripples. Right: Temporally expanded filtered ECoG trace for fast ripples. Ripples are marked in green and fast ripples are marked in red. B, Example of visual analysis of scalp EEG HFOs in case 17 (4‐year‐old girl with focal epilepsy due to left temporal focal cortical dysplasia). Left: Original scalp EEG trace with bipolar montage. Right: Temporally expanded filtered EEG traces (upper, ripple; lower, fast ripple). Ripples are marked in green and fast ripples are marked in red.

For each sample, each rater first identified all conventional epileptiform discharges (Eds; including spikes, polyspikes, and paroxysmal fast activity), then Rs, and FRs last. In addition, raters indicated the location (channels) of all identified EDs, Rs, and FRs. The raters were allowed to specify the location as a single channel, multiple channels, or “diffuse” for all events. Each rater made a notation on the actual EEG at the onset of the event. Two events were judged to be the same event if they occurred with spatial overlap (i.e., channels in common) and began within one‐tenth of a second of each other. The “event agreement rate” was determined in each sample as follows: (Total number of events marked by both rater A and B)/(Total number of events marked by either rater A or B).

“Complete agreement” of the location was given when 2 raters specified the location with exactly the same description such as “channel A” or “channel A/B.” “Partial agreement” was given when 2 raters specified the location with partial overlap (e.g., rater 1 described the location as “channel A,” whereas rater 2 indicated “channel A/B”). “No agreement” was given when 2 raters specified the location without any overlap. If one rater marked a given event but the other did not, agreement of location was not determined.

Statistical analysis

The presence/absence and rate of each finding was tabulated. Interrater reliability (or IRR) is defined as the proportion of agreement not accounted for by chance alone with kappa (κ) = (P O − P E)/(1 − P E), where P O is the proportion of observed agreement and P E is the proportion of agreement expected by chance. Intraclass correlation coefficients (ICC, 2‐way mixed‐effects model) were calculated to assess the consistency of reported rates. As a rule of thumb, κ and ICC >0.7 are considered adequate, and κ and ICC >0.9 are considered excellent. Continuous summary data were presented as median and interquartile range based on nonparametric distributions where appropriate. Statistical calculations were accomplished with Stata software (version 14; StataCorp; College Station, TX, U.S.A.).

Results

Prevalence of EDs and HFOs

As expected, there was a substantial burden of EDs and HFOs in the study samples (mean event rate: ED 25.5/min; R 16.8/min; FR 5.7/min). Among all 20 samples, the reviewers agreed that EDs were present in 13, and absent in 4 studies (no consensus in 3 studies). Reviewers agreed that Rs were present in 9 studies, absent in 9, and did not reach consensus in 2 cases. Similarly, reviewers agreed that FRs were present in 5 studies, absent in 13, and again, consensus was not reached in 2 cases. In Table S1, we have listed the age and etiology of patients from whom EEG samples were abstracted and tabulated the presence or absence of EDs, Rs, and FRs, with event rates and event agreement rates.

Reliability across raters

Interrater reliability for identification of each ED and HFO, and interclass correlation (ICC) for reported rates of EDs and HFOs are summarized in Table 1, and stratified by ECoG and scalp EEG. There was agreement as to the presence or absence of EDs, Rr, and FRs in 17, 18, and 18 cases, respectively. IRR was favorable, with κ = 0.70, 0.80, and 0.80, respectively, and similar for ECoG and scalp EEG studies. However, with regard to reported rates, ICC for Rs (0.99, 95% confidence interval [CI] 0.96–0.99) was statistically superior to FRs (0.77, 95% CI 0.41–0.91), and both HFO subtypes were far superior in comparison to EDs (0.37, 95% CI −0.60 to 0.75), for which reliability was deemed poor. Mean event agreement rates of EDs, Rs, FRs were 53.3%, 62.6%, and 55.0%, respectively.

Table 1

A, Interrater reliability for identification of conventional epileptiform discharges, ripples, and fast ripples; B, reliability (intraclass correlation coefficient; ICC) for reported rates of conventional epileptiform discharges, ripples, and fast ripples

(A)	Overall agreement	Kappa (free‐marginal)	(B)	ICC	95% CI
All cases (n = 20)			All cases (n = 20)
Any epileptiform discharge	0.85	0.70	Any epileptiform discharge	0.37	−0.60 to 0.75
Ripples	0.90	0.80	Ripples	0.99	0.96 to 0.99
Fast ripples	0.90	0.80	Fast ripples	0.77	0.41 to 0.91
ECoG only (n = 10)			ECoG only (n = 10)
Any epileptiform discharge	0.90	0.80	Any epileptiform discharge	0.79	0.17 to 0.95
Ripples	0.90	0.80	Ripples	0.99	0.99 to 1.00
Fast ripples	1.00	1.00	Fast ripples	0.98	0.92 to 1.00
Scalp only (n = 10)			Scalp only (n = 10)
Any epileptiform discharge	0.80	0.60	Any epileptiform discharge	0.29	−1.86 to 0.82
Ripples	0.90	0.80	Ripples	0.94	0.77 to 0.99
Fast ripples	0.80	0.60	Fast ripples	0.61	−0.57 to 0.90

Agreement of the location of the events

With regard to the localization of events, we encountered substantial variation. For the 13 cases in which the reviewers agreed that EDs were present, there was complete agreement as to localization in 5 cases, partial agreement in 6 cases, and no agreement in 2 cases. The distribution of agreement was similar for the 6 ECoG studies (2 complete agreement; 4 partial agreement; and 0 no agreement) and the 7 scalp studies (3 complete agreement; 2 partial agreement; and 2no agreement).

Among the 8 studies (2 ECoG, 6 scalp) in which reviewers agreed that Rs were present, there was complete agreement as to localization in 2 cases (one ECoG, one scalp), partial agreement in 5 cases (one ECoG, 4 scalp), and no agreement in one case (scalp only). For the 5 studies (2 ECoG, 3 scalp) in which both raters agreed that FRs were present, there was complete agreement in 4 cases (2 ECoG, 2 scalp) and partial agreement in one case (scalp only).

Discussion

In contrast to a prior study,14 we have demonstrated favorable interrater agreement in the visual identification, localization, and quantification of interictal HFOs on both ECoG and scalp EEG recordings, at least among EEG readers who are experienced in visual HFO analysis. It is notable that this study suggests that interictal HFOs are more reliably identified and quantified than interictal EDs such as spikes. It is especially noteworthy that blinded raters reliably identified HFOs on scalp EEG from highly selected young children (all ≤4 years) with intractable epilepsy. Our results support the findings from recent study demonstrating the presence of FRs observed on scalp EEG in children with tuberous sclerosis and epilepsy.16 The current study extends the identification of scalp Rs and FRs to focal epilepsies, with diverse etiology including tuberous sclerosis complex, hypoxic ischemic encephalopathy, and focal cortical dysplasia.

However, this study is methodologically limited on several fronts. First, the EEG and ECoG samples reviewed in this study were exceptionally short (10 s) for feasibility purposes. Inasmuch as reliability may be favorable in this study paradigm, IRR might be compromised—or perhaps enhanced—in a “real‐world” setting in which electroencephalographers review considerably longer samples. Indeed, prior studies have utilized samples of 16, 8 to 10 min4, 15 duration for HFO analysis. Furthermore, the use of longer samples might have led to a higher overall prevalence of HFOs and Eds, with the assumption that HFOs and EDs with lower event frequency likely escaped detection in our study. Nevertheless, the event rates in this study were high, with Rs and FRs observed at rates of 16.8/min and 5.7/min, respectively. Both of these rates are high in contrast to those of prior studies.5, 13, 17 We expect that the feasibility of rapidly reviewing longer samples of EEG/ECoG will be facilitated with the continued development and validation of contemporary automated HFO detectors.8, 16 In any HFO analysis, it is critical to distinguish HFO from artifact (i.e., false‐positive HFOs), given that paroxysmal stereotyped artifacts may often resemble HFOs after band‐pass filtering, and especially in those cases with low signal‐to‐noise ratio.18 We believe raters were successful in this endeavor given the approach to identification, that is, simultaneous side‐by‐side EEG/ECoG evaluation with standard and HFO‐specific review settings. Accordingly, raters likely rejected false‐positive HFOs with satisfactory IRR. In the future, we hope to implement computer‐aided analyses to better discriminate HFOs from artifact or other false‐positive HFOs.19

Just as we must endeavor to reject false‐positive HFOs generated by artifacts, it is essential that pathologic HFOs are distinguished from physiologic HFOs. However, we cannot conclude that experienced electroencephalographers can reliably identify and distinguish pathologic HFOs from physiologic HFOs in this study. Because our dataset was derived almost exclusively from children with epilepsy (one subject had autism but not epilepsy), further study in normal subjects is needed to better elucidate the detectability of physiologic HFOs on scalp EEG. In short, this study demonstrates favorable identification of HFOs, but not necessarily favorable discrimination of pathologic and physiologic HFOs.

Finally, one may question the degree to which the raters were truly independent. Although the EEG/ECoG reviews were blinded and independent, both raters trained at the same institution and collaborate frequently on a clinical and research basis. It is not clear that favorable IRR among these raters can be extrapolated to similarly experienced electroencephalographers at other institutions. Nevertheless, utilizing raters from the same institution does not guarantee favorable IRR, as was seen in a prior study from our institution that demonstrated poor IRR in the identification of hypsarrhythmia.20 Above all, we speculate that electroencephalographer experience in HFO analysis is the single most important consideration. In evaluating the report of Spring et al.,14 in which IRR was deemed poor, we suspect this might have occurred because of variable experience among raters, especially as 3 of 6 raters had no formal experience in HFO analysis, and one was still in training. Future efforts are needed to clarify what constitutes sufficient experience to infer favorable reliability. An approach such as a web‐based seminar might enhance reliability across electroencephalographers.

To the extent that the report of Spring and colleagues threatens the utility of HFOs in clinical and research arenas,14 we believe that this study may provide some reassurance that the IRR of visual HFO identification is adequate among highly experienced electroencephalographers. Nevertheless, IRR poses a continued challenge, and is especially important in consideration of the multicenter clinical trials presently underway, which seek to establish the significance and prognostic value of HFOs in the surgical management of epilepsy.

Disclosures

The authors have no conflicts of interests to disclose. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.

Table S1. Clinical information of the study cohort.

Click here for additional data file.