Interrater and Intrarater Agreement of Epileptic Encephalopathy Among Electroencephalographers for Children with Infantile Spasms Using the Burden of Amplitudes and Epileptiform Discharges (BASED) EEG Grading Scale: Study Design and Statistical Considerations.

BACKGROUND: Infantile spasms are a serious epilepsy syndrome with a poor prognosis. Electroencephalography (EEG) has been a key component in the prognosis and treatment of infantile spasms. This multi-center study protocol is developed to investigate interrater and intrarater agreement of an electroencephalographic grading scale-the Burden of Amplitudes and Epileptiform Discharges (BASED) score among electroencephalographers.
METHODS: Thirty children, aged 0-2 years, with infantile spasms who were hospitalized in the Chinese PLA General Hospital will be recruited into this study by stratified sampling. Seven electroencephalographers from different Class A tertiary hospitals will select a 5-min epoch with the most severe epileptiform discharge, score the EEG reports, and provide the basis for the scoring. The 420 (30 × 7 × 2) scoring results provided by electroencephalographers in two rounds can be analyzed statistically using weighted kappa (weighted [Formula: see text]) statstic, Fless' kappa (Fless' [Formula: see text]) statistic, and intraclass correlation coefficient (ICC) to calculate the interrater and intrarater agreement. DISCUSSION: We will recruit more electroencephalographers than were included in previous studies to assess the interrater and intrarater agreement in the selection of 5-min EEG epochs, the BASED scores, and the basis for scoring. If the BASED score has an adequate interrater and intrarater agreement, the score will have more significance for guiding the clinical management and for predicting the prognosis of patients with infantile spasms.

Key Summary Points

Introduction

Infantile spasm syndrome is a catastrophic form of childhood epilepsy that often manifests between 3 and 7 months of infancy [1, 2] and is characterized by involuntary, massive motor spasms during early infancy that herald a lifelong disorder of severe seizures and intellectual disability [3-7]. Electroencephalography (EEG) can be used to diagnose infantile spasms and to assess the therapeutic effect [8]. Most patients with infantile spasms show a distinctive hypsarrhythmia pattern on EEG that was first demonstrated by Gibbs [9]. Hypsarrhythmia, including typical hypsarrhythmia and modified hypsarrhythmia, is defined as electroencephalographic (EEG) abnormalities characterized by random bilateral high-voltage slow waves and multifocal spikes, with multiple lesions in different regions of the brain [1, 10, 11]. The remission of hypsarrhythmia is an important indicator of treatment effectiveness [12]. However, the interrater agreement is poor with regard to the determination of hypsarrhythmia [13, 14] and constitutes a significant challenge in the management of infantile spasms patients. Thus, there is a need for the development of a novel method for the electrophysiological evaluation of infantile spasm patients.

Mytinger et al. [14] developed the Burden of Amplitudes and Epileptiform Discharges (BASED) score to better interpret the EEG of infantile spasm patients. The BASED score, which ranges from 0 to 5, comprises a structured rating approach that is based on the EEG waveform, amplitude, and frequency. Higher scores (e.g., a BASED score of 4 and 5) are associated with a higher likelihood of epileptic encephalopathy. Mytinger et al. [14] found that the change in the BASED score highly correlated consistently with clinical remission, and inferred that the patients with a pre-treatment BASED score of 4 or 5 and a post-treatment score of 3 or less could be diagnosed as having achieved electrographic remission. In the Mytinger’s study, of the 22 patients with clinical remission, 19 showed electrographic remission, which proved that clinical remission closely matched the electrographic remission. Among the three raters who blindly assessed the BASED scores, there was moderate to near-perfect interrater agreement in the interpretation of several EEG characteristics (abnormally high amplitude, epileptiform discharge, or 3 spike foci, or 50% of 1-s bins, grouped multifocal spikes, or paroxysmal voltage attenuation) [14, 15].

The BASED score, which was initially proposed in 2015, has high interrater agreement [16-18]. With further research, an advanced version of the BASED score was further developed in 2021 through the modification and verification of the original BASED score [15], mainly in the following aspects: significantly, the background amplitude changes from 200 μV on all channels to 200 μV on most channels and to 300 μV on the temporal lobe with the deletion of montage on the occipital lobe and in the midline; some new EEG characteristics were introduced, such as paroxysmal voltage attenuations and grouped multifocal spikes; in addition, the verification of the interrater agreement of the BASED score in 2015 aimed to determine the presence or absence of hypsarrhythmia whereas, in the modified version of the BASED score, the interrater and intrarater agreement was ascertained based on various EEG features [14, 15]. Though the positive outcomes of interrater agreement in the interpretation of the BASED score were obtained between three blinded reviewers [15], that verification was undertaken in an internal single-center study that did not generate strong evidence to support the results.

Here, we design a multi-center study to re-evaluate the interrater and intrarater agreement in the assessment of the BASED score among the electroencephalographers randomly recruited in seven class A tertiary hospitals in China. This article serves as the protocol of the study and elaborates the details of the study design and statistical considerations of the study.

Methods

Study Design and Procedure

We designed this multi-center study to assess the interrater and intrarater agreement of the BASED scores in clinical practice in the People’s Republic of China. Electroencephalographers from seven hospitals will randomly review the EEG of 30 infantile spasms patients in two replications (rounds) and will assign BASED scores. Statistical analysis for assessing interrater agreement and intrarater agreement between different raters and between two replications will be conducted by study investigators.

Study Objects

We will recruit infantile spasms patients with EEG records who have been hospitalized in the pediatric departments of the First Medical Center of the PLA General Hospital from June 2019 to June 2021. Ethical approval for the study was granted by the Ethics Committee of the First Medical Center of the PLA General Hospital (S2022-209-01). All participants will be assigned an initial BASED score, which will be given by a chief electroencephalographer who has more than 20 years of experience in clinical neuroelectrophysiology. Then, the participants will be divided into six groups by their initial BASED score ranged from 0 to 5. Five participants will be randomly selected from each group to form a cohort of 30 subjects for the study.

All participants should meet all the inclusion criteria and should not have any of the exclusion criteria as defined below. Inclusion criteria are (i) provision of written informed consent by the legal guardian or parent; (ii) diagnosed with infantile spasms based on clinical manifestations and EEG results; (iii) 3–18 months; and (iv) with clinical records of at least 4-h sleep EEG data. Exclusion criteria include (i) unwillingness to participate in the study and (ii) irremovable artefact on many channels of the EEG.

EEG Raters

Seven class A tertiary hospitals will be selected in this study: the First Medical Center of the Chinese PLA General Hospital, Yuquan Hospital of Tsinghua University, Affiliated Hospital of Inner Mongolia Medical University, the First Hospital of Jilin University, Harbin Children's Hospital, The First Affiliated Hospital of Anhui Medical University, and Hainan Women and Children's Medical Center. From each medical institution, we selected one certified electroencephalographer with a master’s degree in medicine and 15–20 years of clinical experience in neuroelectrophysiology to review the interrater and intrarater agreement of the EEG BASED score for 30 children with infantile spasms hospitalized in the Department of Pediatrics at the Chinese PLA General Hospital. These seven electroencephalographers will undergo standardized training conducted by chief electroencephalographer Jian Chen on the BASED scoring at the Chinese PLA General Hospital. All electroencephalographer training will be provided on an online communication platform. One week before the training, an independent researcher will send the detailed grading criteria using the BASED scoring system to the electroencephalographers. The electroencephalographers will be required to understand the scoring system as they can and outline the items that they cannot understand. During the formal training, the chief electroencephalographer, Dr. Jian Chen, will carefully explain each scoring criteria and display examples of EEG segments to educate and train the future raters. The chief electroencephalographer will also address the questions from the future raters. After the training, the chief electroencephalographers will select six typical EEG samples with a BASED score from 0 to 5, and send them to the electroencephalographers. The electroencephalographers, who review and rate five or more EEG segments correctly will be identified as a qualified rater and will be subsequently recruited in the study. We will continue recruiting and training additional qualified electroencephalographers until the prespecified total number is reached.

Electroencephalographers are demanded to correctly interpret the electrographic outcome using the BASED score according to the following rules:

More than 5-min EEG data will be selected from the EEG, including at least 4 h and one sleep cycle recorded by a high-frequency filter 70 Hz, low-frequency filter 1 Hz, 15 s per page and longitudinal bipolar montage. Mainly, the following EEG channels will be assessed: Fp1-F7, F7-T3, T3-T5, Fp1-F3, F3-C3, C3-P3, Fz-Cz, Cz-Pz, Fp2-F4, F4-C4, C4-P4, Fp2-F8, F8-T4, and T4-T6. The most epileptic 5-min sleep epoch will be selected on every page for 15 s. If the EEG characteristics such as waveform and amplitude meet criteria for a score of 3, they will be given corresponding scores; In other cases, scores of 0–2 will be assigned in accordance with the whole EEG data.

The 2021 BASED scoring criteria are shown in the Appendix (supplementary material), with a focus on the following EEG characteristics: normal background amplitude, abnormal high amplitude, epileptiform discharge, or 3 spike foci, or 50% of 1-s bins, grouped multifocal spikes, paroxysmal voltage attenuation, and sleep spindle waves.

Study Procedures

The first round of scoring: seven electroencephalographers will independently complete the 2021 BASED scoring of the 30 participants within 1 month. In the first round, 30 EEG data sets from 30 participants will be obtained and evaluated by the seven electroencephalographers. An independent researcher will collect the results of scoring, mark the results as ‘Round 1 + institution’, and put them in sealed envelopes.

In the second round of scoring: within the second month after the first round of scoring, seven electroencephalographers who have participated in the first round of the study will perform the second round of scoring without any awareness that these data are duplications but with the EEG data rearranged. An independent researcher collects the results of scoring 1 month later, mark the results as ‘Round 2 + institution’, and put them into sealed envelopes.

Finally, the electroencephalographers from the seven participating hospitals will provide 420 (30 7 2) scoring results in the first and second round of the study. From these reports, we will be able to evaluate the interrater and intrarater agreement of the BASED score (Fig. 1).

Fig. 1

Workflow for evaluation of interrater and intrarater agreement among seven EEG raters on the EEG data of 30 patients

Randomization

The chief electroencephalographer will divide the EEG data into six groups according to the clinical and EEG characteristics of patients. Five patients will be randomly selected by the independent researcher from each group to form a 30-patient cohort. For the randomization, random codes generated by computers will ensure the concealment of distribution. In order to ensure that there is no serious imbalance in scores, labels of EEG data that are assigned to raters are randomly and automatically generated by the computer, and each electroencephalographer obtains a different number.

Blinding

All EEG data will be recorded using the Micromed® EEG system (Micromed, Mogliano Veneto, Italy). When recording EEG, it is required for the technicians to input the patient identification number, name, gender, age, and collection time in the electronic medical record system. An independent researcher will export the EEG data to a European Data Format (EDF) format and rename the EEG data from a patient with a random serial number. In order to establish blinding and avoid bias, patient-identifiable information will be removed. Only the independent researcher will have access to the patient information. The participating patients, electroencephalographers, and study investigators will not have access to the any of the information. Electroencephalographers can only examine the deidentified EEG data through the EDF files. Furthermore, different random serial numbers will be generated and allocated in the first and second rounds. The collected data will be sealed and stored with no disclosure. The raters and investigators will be blinded from randomization allocation.

EEG Data Acquisition

Each participant will undergo recording of at least 4-h valid sleep EEG data. First, the chief electroencephalographer Jian Chen will select three 5-min interictal EEG epochs (these will be chosen at will if there is no obvious epileptiform discharge) during sleep, which can be identified by the presence of sleep spindles or eyes closed and absent muscle artifact, give them a random serial number, and then assign them to the raters in European Data Format (EDF) without any of the patient's personal information. The electroencephalographers are asked to determine the most epileptiform 5-min epoch from the three truncations and to score it. The choice of the EEG epoch is a component of the interrater and intrarater agreement.

Study Objectives and Measures

Study Objectives

The objective of this study is to determine the interrater and intrarater agreement of the BASED scores that are assigned by the electroencephalographers to pediatric patients with infantile spasms.

Study Measures

The EEG raters identify one EEG epoch with the most severe epileptiform discharge and fill in the relevant serial number in the form. Thereafter, the electroencephalographers will assign a BASED score to each of the EEG epochs. Each rater will assign one ordinal BASED score to each of the subjects in each round. This will generate two 30 × 7 matrices (30 subjects, seven raters, and two replications) of data on the BASED scores for data analysis.

We will acquire the general information of EEG raters, including age, gender, major, degree, and work experience. The demographic and clinical characteristics of the participants, including age, gender, family history of epilepsy, age of spasm onset, number and type of antiepileptic drugs used, normal or abnormal cranial MRI findings, presence or absence of a definite etiology, and whether spasm-free will be recorded.

Statistical Analysis Plan and Considerations

Interrater Agreement

The interrater agreement of the ordinal BASED scores between any pair (two) of the raters will be evaluated by unweighted weighted kappa (weighted ) [19] with squared weights. The interrater agreement of the scores between all raters will be evaluated by Fless’ kappa (Fless’ ) [19]. The assessment will be repeated for the BASED scores collected from each of the two rounds (Round 1 and Round 2). Furthermore, we will compute an average of two BASED scores for each subject by each rater in the two rounds, and then evaluate the interrater agreement based upon the average BASED scores. A 95% confidence interval will be reported for each of the kappa statistics.

We will the follow guidelines of Landis and Koch [20] for interpreting a kappa statistic: a kappa between 0.00 and 0.20 indicates slight agreement; a kappa between 0.21 and 0.40 represents fair agreement; a kappa between 0.41 and 0.60 characterizes moderate agreement; a kappa between 0.61 and 0.80 defines substantial agreement; a value of greater than 0.80 equates to almost perfect agreement.

In addition, the type (A,1) intraclass correlation coefficient (ICC) described by McGraw and Wong [21] will be estimated to measure the interrater agreement between the seven raters. Here, we will fit a two-way random-effects model without any interactions to assess the absolute agreement of a single score because each of the seven raters will be randomly selected from a larger population of electroencephalographers in their institute. The type (A,1) ICC will be computed for the BASED scores collected from each of the two rounds (Round 1 and Round 2) and for the average BASED scores and will be reported with the 95% confidence interval for each of the ICCs.

The value of an ICC ranges from 0 to 1, with 0 indicating no agreement and 1 indicating perfect agreement among raters. Gisev, Bell, and Chen [19] recommended the interpretation of the estimated value of ICC as follows: an ICC value less than 0.50 are indicative of poor reliability; an ICC value between 0.50 and 0.75 indicate moderate reliability; an ICC value between 0.75 and 0.90 indicate good reliability; an ICC value greater than 0.90 indicate excellent reliability.

Intrarater Agreement

Intrarater agreement of the ordinal BASED scores between the two scores assigned by the raters in the two rounds will be evaluated by the weighted with squared weights for each of the seven raters. The guidelines of Landis and Koch [20] will be applied for interpreting kappa statistics.

The type (A,1) ICC presented in McGraw and Wong [21] will be estimated to measure the intrarater agreement between the two scores that have been assigned by each of the raters in the two rounds. We will fit a two-way mixed-effects model without any interactions to assess the absolute agreement of a single score. We will assign a random effect to the subjects and a fixed effect to the two rounds as the subjects will be randomly selected from a larger population, whereas the two rounds will not be random. A 95% confidence interval will be reported for each of the ICCs.

Percent Agreement and Summary Statistics of Baseline Characteristics

Percent (proportion) agreement, as the basic agreement index, will be reported as a measure of agreement in conjunction with statistics that are chance-corrected, though the effect of chance in achieving agreement between raters is not accounted for in percent agreement. Baseline demographics and clinical characteristics of both participants and raters will be summarized by descriptive statistics (mean and standard deviation or count and percentage).

Power and Sample Size Calculation

We specified the analysis for evaluating intrarater agreement by unweighted kappa statistic as our primary analysis to conduct power and sample size calculation. In our study, a total of seven raters (electroencephalographers) will be scheduled to review the EEG epochs. To conduct hypothesis testing simultaneously on the intrarater agreement kappa statistics derived from two rounds of rating of the seven raters, we applied the Bonferroni adjustment and specified a type I error of 0.05/7 = 0.0071 for each of the seven tests for each rater’s intrarater agreement evaluation from two rounds. In a test for agreement between two rounds of each rater, using the unweighted kappa statistic, a sample size of 30 subjects (EEG epochs from distinct patients) will achieve more than 88% power to detect a true kappa value of 0.60 in a test of : versus : , when six categories of BASED scores have the frequencies that are equal to 0.17, 0.17, 0.17, 0.17, 0.17, and 0.17, respectively. This power calculation was conducted based upon a significance level of 0.05/7 = 0.0071 after Bonferroni adjustment.

Strengths and Limitations

Hypsarrhythmia is a specific EEG pattern that is mainly manifested as diffuse, asynchronous, high-amplitude, slow-wave activity and multifocal spikes [22, 23]. Many researchers considered the absence or presence of hypsarrhythmia as an all-or-none phenomenon—that is, a diagnosis of hypsarrhythmia should satisfy the co-occurrences of multifocal epileptiform discharges and diffuse, asynchronous, high-amplitude, slow-wave background activity. However, in previous studies, the definition of hypsarrhythmia has been inconsistent [24-27]. Moreover, a previous study has shown that the interrater agreement on the absence and presence of hypsarrhythmia was poor [28]. As electrographic remission is the criterion for evaluating treatment efficacy, a standardized evaluation method for EEG data is needed to facilitate consistent interpretation of EEG outcomes.

A small-scale interrater agreement evaluation of the BASED score was undertaken in an early study, but the participants were limited to the personnel at the researcher's institution [15]. Mytinger et al. [14, 15] published their research on the interrater agreement of the BASED score in 2015 and 2021. In the study published in 2015, the authors found that the BASED score demonstrated better interrater agreement than traditional EEG analytical methods in the presence or absence of hypsarrhythmia. Furthermore, the study published in 2021 found high levels of interrater agreement on the EEG characteristics of some EEG channels. Compared with previous studies, theirs is the first study to verify the BASED score that was conducted by researchers other than the inventors of the BASED score; moreover, the researchers selected three familiar raters. This study is a nationwide large-scale multi-center study to evaluate both interrater and intrarater agreement of epileptic encephalopathy among electroencephalographers for children with infantile spasms using the BASED EEG grading scale. In our study, we will choose electroencephalographers from seven different and unrelated hospitals to participate in the experiment; in addition, we will not only study the interrater and intrarater agreement of a certain EEG feature but also evaluate the consistency of the BASED score, scoring basis, and the selection of the 5-min EEG epoch with the most severe epileptiform discharge [14, 15]. We will use this rigorous protocol to evaluate the interrater and intrarater agreement of the BASED score. The purpose of our research is: (i) To evaluate whether the BASED score has adequate consistency between different raters—that is, the interrater agreement; (ii) To assess the stability of the same rater in different times—that is, the intrarater agreement; and (iii) To explore the practical value of the BASED score in clinical practice through the proposed study.

Most studies in the field that calculated the interrater and intrarater agreement have focused on selecting the results from more than half of the raters and using clinical symptoms as the reference points. However, we choose the median BASED score that was reported by all electroencephalographers as a reference point. As multi-category data, the BASED score is unsuited for confirmation by a method where the minority is subordinate to the majority. Moreover, there is no clear relationship between electrographic remission and clinical remission. The influence of outliers has been avoided by not selecting the average as a reference point.

Many guidelines in China and other countries recommend that electrographic remission, similar to clinical remission, can be used as a key point to predict the treatment efficacy and prognosis of infantile spasms patients. Electrographic remission is defined as the disappearance of hypsarrhythmia EEG without clinical symptoms of infantile spasms in sleep stages. Clinical remission refers to a spasm-free state that appears within 2 weeks after effective treatment and lasts for at least 28 days [29, 30]. The EEG assessment of infantile spasms patients mostly depends on the absence or presence of hypsarrhythmia, which is a subjective finding. If the BASED score is proven to have moderate-to-perfect interrater and intrarater agreement, the clinico-electrographic remission of infantile spasms will be confirmable via a unified and standardized method. In some cases, the change in the BASED scores after adrenocorticotropic hormone (ACTH) or other treatments will provide clinicians with more information to adjust and individualize therapy [31]. In another example, though vigabatrin (VGB) is the first-line therapy for infantile spasms, previous studies have reported irreversible visual field defects, abnormal brain structure, and other side effects of this treatment [32-34] that are related to treatment duration. In case an effective clinico-electrographic remission index becomes available, the duration and dosage of VGB could be more precisely adjusted to minimize the occurrence of side effects [33, 35, 36].

Ethics and Dissemination

This article presents a study design protocol and statistical consideration of a multi-center study to determine the interrater and intrarater agreement of the BASED score. In this study, seven randomly selected electroencephalographers will score the EEG data of 30 infantile spasms patients based on the BASED score, and the weighted kappa, Fless’ kappa, and ICC will be used to measure the interrater and intrarater agreement. Ethical approval for the study was granted by the Ethics Committee of the First Medical Center of the PLA General Hospital (S2022-209-01). This study will be conducted in accordance with the ethical principles of the Declaration of Helsinki. All survey respondents provided informed consent and all data were anonymized prior to the analysis.

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 107 KB)

This is a study design to evaluate interrater and intrarater agreement of epileptic encephalopathy among electroencephalographers using the Burden of Amplitudes and Epileptiform Discharges (BASED) score.

Electroencephalography (EEG) is an effective diagnostic tool in the prognosis and treatment of infantile spasms, which is a catastrophic epilepsy syndrome.

An estimated kappa value greater than 0.80 and an intraclass correlation coefficient value greater than 0.90 will indicate excellent agreement.

BASED score with high interrater and intrarater agreement will have guiding significance in clinical practice.