The difficulty of diagnosing NCSE in clinical practice; external validation of the Salzburg criteria.

To improve the diagnostic accuracy of electroencephalography (EEG) criteria for nonconvulsive status epilepticus (NCSE), external validation of the recently proposed Salzburg criteria is paramount. We performed an external, retrospective, diagnostic accuracy study of the Salzburg criteria, using EEG recordings from patients with and without a clinical suspicion of having NCSE. Of the 191 EEG recordings, 12 (12%) was classified as an NCSE according to the reference standard. In the validation cohort, sensitivity was 67% and specificity was 89%. The positive predictive value was 47% and the negative predictive value was 95%. Ten patients in the control group (n = 93) were false positive, resulting in a specificity of 89.2%. The interrater agreement between the reference standards and between the scorers of the Salzburg criteria was moderate; disagreement occurred mainly in patients with an epileptic encephalopathy. The Salzburg criteria showed a lower diagnostic accuracy in our external validation study than in the original design, suggesting that they cannot replace the current practice of careful weighing of both clinical and EEG information on an individual basis.

INTRODUCTION

Nonconvulsive status epilepticus (NCSE) is defined as a continuous state of seizures without convulsions or multiple nonconvulsive seizures for more than 30 minutes without interictal full recovery.1 It is a neurologic emergency with high morbidity and mortality.2 Clinical symptoms of NCSE can be subtle and misleading and therefore diagnosing NCSE relies heavily on electroencephalography (EEG). Different EEG criteria for NCSE exist (for a summary see reference 33) and a global or regional consensus is lacking. Diagnostic criteria with high sensitivity for NCSE would improve diagnostic accuracy and clinical management, whereas global consensus on, and general application of these criteria would greatly benefit communication and research.

The Salzburg 2013 criteria for diagnosing NCSE4, 5 were the first criteria to be clinically validated.6 In this clinical validation study, a very high sensitivity (97.7%, 95% confidence interval [95% CI] 87.9‐99.6) and specificity (86%, 80.8‐94.6,) were found with an overall accuracy of 92.5% (88.3‐97.5). The report represents a praiseworthy effort to improve EEG diagnostic consensus of NCSE. However, the study design has limitations that could hamper external implementation.7, 8 One of the main concerns was the involvement of many of the local study physicians who were attendants of the Salzburg 2013 meeting; inclusion of their bias while scoring the reference standard cannot be ruled out. In addition, 36% of included patients had an NCSE, which is a much higher percentage when compared to literature9 and our own clinical experience and therefore possibly a different patient cohort or setting. These limitations could hamper external validation, which is considered a critical step in prognostic models.10 Here we report an independent clinical validation of the Salzburg criteria for the diagnosis of NCSE in two different tertiary healthcare settings.

METHODS

A retrospective, diagnostic accuracy study was conducted using acute video‐EEG recordings lasting 30‐60 minutes from patients admitted to the Radboud University Medical Center (Radboudumc), a university hospital with a large neurocritical care unit, and Kempenhaeghe, a national tertiary referral center for epilepsy and sleep disorders, both in The Netherlands. In our study, we followed the methods as described in the original Salzburg criteria paper by Leitinger et al.6

Study design and participants

All consecutive EEG recordings from both adult and pediatric patients with a clinical suspicion of NCSE (defined by the referral reason of the EEG being NCSE) were included in the clinically suspected NCSE group. All consecutive EEG recordings without a clinical suspicion but with an abnormal EEG were included in the clinically not “suspected for NCSE” group. We excluded patients with technically insufficient EEG recordings (where interpretation was not possible due to artifacts) and EEG recordings lasting less than 30 minutes. The diagnostic algorithm was validated in a control group consisting of EEG recording with any abnormal findings from patients without a clinically suspected NCSE. Based on the results from previous study, we calculated that 100 patients in both groups would be needed to adequately power the study.

Procedures

As a part of standard care, the university center used the Hirsh criteria,11 while no specific criteria were used in the tertiary epilepsy center. All EEG reviewers were board‐certified clinical neurophysiologists with varying years of experience that reflects clinical practice. None of the authors were familiar with the Salzburg criteria prior to this study. Four authors (I.L. and M.Z. at Kempenhaeghe and N.A. and M.A. at the Radboudumc) thoroughly studied the clinical criteria as stated in the original article4 and retrospectively assessed the EEG recordings while being blinded to clinical data.

Reference standard

As a reference standard, the expert opinion of another four authors was sought, who had access to all clinical information (including laboratory tests, imaging studies, response to treatment, follow‐up, and outcome) and all EEG recordings, and were blinded to the Salzburg scorings. These reference authors (J.A. and C.A. at Kempenhaeghe and J.P. and C.S. at Radboudumc) individually reviewed the cases. If the individual conclusions differed, the cases were reevaluated by both clinicians together to reach a consensus. This consensus scoring was used in the comparisons.

Statistics

Standard calculations for sensitivity, specificity, predictive values, and likelihood ratios were used.

Spearman correlations were calculated for interrater agreement. Data were stored in the data management system Castor12 and for analysis we used Prism 5 software (GraphPad Software Inc., San Diego, CA). A P‐value of < 0.05 was considered significant.

RESULTS

Demographics

A total of 191 EEG studies from 187 patients were reviewed: 50 EEG recordings in the clinical validation group were obtained between May 28, 2015 and Oct 1, 2016 at the epilepsy centre Kempenhaeghe, and 47 in the clinical validation group were collected between Dec 1, 2015 and Nov 22, 2016 at the Radboudumc. For the control groups, 50 consecutive EEG recordings between Apr 1 and Sep 30, 2016 were collected at Kempenhaeghe and 44 were collected between Jan 4, 2016 and Nov 16, 2016 at the Radboudumc (Figure 1). In the clinical validation group (patients with a clinical suspicion of an NCSE), the patients were older, had more frequently prior cerebral hypoxia, and a decreased vigilance compared to the control group (patients without a clinical suspicion of an NCSE). In the patient group at the university hospital, 21% had a prior diagnosis of epilepsy and 15% had a history of cerebral hypoxia, whereas in the patient group at the epilepsy center, 98% had a prior diagnosis of epilepsy and no patients had a history of cerebral hypoxia (see Table 1).

Figure 1

Flowchart of the study

Table 1

Cross tables for both patients groups

Patients with clinically suspected NCSE	NCSE reference	No NCSE reference
NCSE Salzburg; Total (E/A)	8 (4/4)	9 (2/7)	17 (PPV 8/17 = 47.1%)
No NCSE Salzburg; Total (E/A)	4 (4/0)	76 (40/36)	80 (NPV 76/80 = 95.0%)
	12	85	97
	Sens 8/12 = 66.7%	Spec 76/85 = 89.4%

Abbreviations: A, academic center; E, epilepsy center; NCSE, nonconvulsive status epilepticus; NPV, negative predictive value; PPV, positive predictive value; sens, sensitivity; spec; specificity.

Diagnostic accuracy of Salzburg criteria

Of the 97 patients with a clinically suspected NCSE, 12 were classified as being an NCSE by the reference standard, resulting in a pretest probability of NCSE of 11% at the university hospital and 16% at the epilepsy center. The Salzburg criteria in our cohort of patients with a possible NCSE had an overall sensitivity of 67% and a specificity of 89%. The negative predictive value was 95% and the positive predictive value was 47%, with a positive likelihood ratio of 6.3 and a negative likelihood ratio of 0.4 (see Table 1). In the control group, 11 patients fulfilled the Salzburg criteria (specificity 89.2%), of which one was confirmed by the reference standard (true positive) and 10 were not confirmed (false positives). In these false‐positive EEG findings, the criteria were fulfilled during a seizure with postictal recovery without a clinical suspicion of NCSE (Table S1).

Interrater agreement

The interrater agreement between the reference standards was moderate (Spearman r s = 0.54; P < 0.001). The interrater disagreements occurred mainly in patients with a severe epileptic encephalopathy vs an NCSE. The interrater agreement between the scorers of the Salzburg criteria was also moderate (Spearman r s = 0.41; P < 0.001). The reasons for disagreement were the following: two patients with severe epileptic encephalopathy, one metabolic encephalopathy, and one patient with subtle signs of convulsive status epilepticus (nystagmoid eye movements) vs NCSE. In 4 of 191 patients, the consensus rating was different from both the two original ratings.

DISCUSSION

In this external validation study, we found the Salzburg criteria for diagnosing NCSE to have a moderate diagnostic accuracy and interrater consistency. We found a moderate sensitivity and a reasonable to good specificity. Therefore, the Salzburg criteria appear to be possibly useful as a primary screening method for NCSE with a high negative predictive value, but lack precision to objectively diagnose the condition in different clinical practice settings.

The diagnostic accuracy of the Salzburg criteria we found is clearly lower than in the original validation study.6 In the validation group, the lower positive predictive value of the Salzburg criteria can be largely explained by the markedly lower number of true positives (8/97 in the current study compared to 42/120 in the original article). Therefore, following Bayesian reasoning, because the a priori chance of having an NCSE is relatively low, as in our study and as found in literature,9 the positive predictive value and sensitivity of the Salzburg criteria decrease. Unsurprisingly, this will mean that the clinical value of the criteria will depend on the patient case mix and the a priori chance of having NCSE for any given center. The high number of false positives in both the validation (9/97) and in the control group (10/94) may lead to aggressive antiseizure therapy, which is unnecessary and probably deleterious for the patient.

We feel that the main reason for not being able to apply the Salzburg criteria successfully in all patients is that there are inherent pitfalls in applying the criteria to patients with an epileptic encephalopathy such as a hypsarrhythmia or multiple independent spike foci pattern (that may or may not be known prior to the EEG), in whom NCSE is suspected. These patients will have an overall abnormal background recording and usually will show epileptiform discharges for >10 seconds that are often in the 2‐5 cycles/seconds range with some fluctuation. That automatically puts these patients in the “possible NCSE” group, without the need for any additional abnormality that would positively indicate an additional NCSE in this group. We would therefore recommend to expand the criteria with an initial reading of the clinical info and EEG for the diagnosis of epileptic encephalopathy, and, if that is found, to further exclude these patients from applying the criteria as they are currently formulated. In addition, further defining the terminology used, with more precise specification of terms such as discharges, fluctuation, and evolution, and adding a quantitative method that can interpret the dynamics of the EEG, might also help to not overinterpret these patterns.

In addition, patients with diffuse encephalopathies who show generalized or lateralized periodic discharges, such as seen in postaxonic encephalopathy (PAE), will also automatically be classified in the “possible NCSE” group, even though that label is of questionable value in cases with PAE. It is still debatable whether to call a lateralized periodic discharge an epileptic discharge,13 and qualifications such as “relatively uniform morphology” can also lead to subjective interpretation. Therefore, it may also be necessary to exclude this particular pathology from further interpretation using the Salzburg criteria, or further specify which dynamic patterns can be interpreted safely within that framework. Alternatively, one could consider giving a different weight to the “possible NCSE” classification, interpreting that subgroup as “not certain enough to initiate treatment,” instead of grouping it with “definite NCSE” as done in the original study.

Other explanations for the lower diagnostic accuracy compared to the original article may be the variability in the reference standard scorings in our study, and the possibly different duration of the EEG recordings (30‐60 minutes in our study, a minimum of 20 minutes or more in the original paper). In addition, at least some of the authors of the original article may have had a certain level of experience using the criteria, which could have affected their interrater variability, both in the study and reference group.7 The variability in reference standard scorings was high in our cohort, which is known from literature, for example, in hypsarrhythmia patterns.14, 15 This emphasizes how difficult and subjective EEG scoring, including that of an NCSE, currently is, and how useful it would be to have objective, straightforward criteria to help clinicians do that. We would also like to stress, as the original paper did, that the diagnosis of NCSE still begins with a clinical judgment about its likelihood, and that clinical data need to be incorporated in the final decision about NCSE or not.

The results of this study are limited by various factors similar to the original article, such as the absence of a true gold standard and the retrospective design of this study; specifically because the clinical decision to give antiepileptic drugs is a step in the Salzburg criteria that cannot be taken retrospectively. Future validation studies should therefore preferentially be executed prospectively in a real‐time patient care setting.

We conclude that diagnosing NCSE is difficult, and this problem is not yet solved by the Salzburg criteria. The criteria showed a lower diagnostic accuracy in our external validation study than in the original design, suggesting that they cannot be automatically transferred to any clinical setting where an EEG interpretation for suspected NCSE is needed. The main confounders for correct interpretation were patients with an epileptic encephalopathy or an encephalopathy with periodic discharges. If the Salzburg criteria are to be implemented in everyday practice, we suggest adapting the decision rules and interpretation to avoid overinterpreting the EEG results and improve specificity.

CONFLICT OF INTEREST

None of the authors has any conflict of interest to disclose. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.

Click here for additional data file.