Risk, diagnostic error, and the clinical science of consciousness.

In recent years, a number of new neuroimaging techniques have detected covert awareness in some patients previously thought to be in a vegetative state/unresponsive wakefulness syndrome. This raises worries for patients, families, and physicians, as it indicates that the existing diagnostic error rate in this patient group is higher than assumed. Recent research on a subset of these techniques, called active paradigms, suggests that false positive and false negative findings may result from applying different statistical methods to patient data. Due to the nature of this research, these errors may be unavoidable, and may draw into question the use of active paradigms in the clinical setting. We argue that false positive and false negative findings carry particular moral risks, which may bear on investigators' decisions to use certain methods when independent means for estimating their clinical utility are absent. We review and critically analyze this methodological problem as it relates to both fMRI and EEG active paradigms. We conclude by drawing attention to three common clinical scenarios where the risk of diagnostic error may be most pronounced in this patient group.

Introduction

Recent research suggests that functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) offer diagnostic information ancillary to standard clinical examinations of seriously brain-injured patients. A subset of these techniques, called active paradigms (Laureys and Schiff, 2012), utilizes a volitional mental task, such as mental imagery or selective attention, as a proxy for behavioral response to commands. Using these methods, investigators have revealed residual covert awareness in some patients once thought to be in a vegetative state — also referred to as unresponsive wakefulness syndrome (VS/UWS) (Laureys et al., 2010). Given the success of these techniques, clinical application may improve diagnostic accuracy (Giacino et al., 2014; Laureys et al., 2004; Owen, 2013). However, integrating active paradigms into the standard diagnostic protocol poses several difficult questions regarding clinical utility.

Neuroimaging and EEG assessment of brain injury requires powerful statistical tools for data analysis. Whether group-averaged brain activity is tested for statistical significance, individual subject neural anatomy is adapted to standard cortical maps, or regression filters are used to eliminate artifacts, statistical modeling is an invaluable tool for identifying statistically significant findings. However, which statistical methods ought to be used may remain unclear as the majority of techniques used to assess seriously brain-injured patients have not yet been model-tested nor extensively tested against one another (but see Stender et al., 2014).

Given this methodological difficulty, it is possible for different methods to produce conflicting results. Some results may be so heterogeneous that they engender empirically equivalent, yet mutually inconsistent, diagnoses. For example, when working from an identical data set, one task design or method of analysis may detect the presence of awareness in patients previously thought to be in a VS/UWS (Cruse et al., 2011), while another may fail to identify any statistically significant findings (Goldfine et al., 2013). Which results reflect the truth (i.e. whether a patient is aware or not) may remain an open question, as independent confirmation may be technically difficult or precluded on theoretical grounds.

In other domains of medicine, information garnered from clinical assessment may be corroborated with pathophysiological examination. Ideally, multiple sources of diagnostic information converge to form a conciliatory picture of the patient's condition. If diagnostic information is inconsistent, physicians have recourse to further tests that assess the truth of the patient's complaint. Diagnosis after serious brain injury with the aid of active paradigms presents a more complicated puzzle. Because the only accepted clinical procedure for assessing seriously brain-injured patients requires behavioral participation – either through self-report, command following, or functional object use (cf. Giacino et al., 2004) – confirming active paradigm findings by appealing to patients who are already known to be behaviorally unresponsive is problematic. This problem may be resolved by comparing results to other well-established prognostic and diagnostic evidence (Giacino et al., 2014; Young, 2009a). However, as previous research has demonstrated (e.g. Owen et al., 2006; Monti et al., 2010; Cruse et al., 2011; Naci and Owen, 2013; Fernández-Espejo and Owen, 2013; Gibson et al., 2014; Monti et al., 2015), some patients may satisfy all clinical criteria for the VS/UWS while consistently demonstrating awareness through active paradigms. Such patients deviate from standard diagnostic categories (Bruno et al., 2011) and challenge assumptions regarding the attribution of consciousness when conventional evidence is absent (Bayne and Hohwy, 2014; Shea and Bayne, 2010).

Given these methodological obstacles, how can investigators determine which methods ought to be used? Surely the relative accuracy of active paradigms may be determined as this research matures. Promising work on intrinsic cortical networks and their role in modulating consciousness (Demertzi et al., 2014, 2013a; Fernández-Espejo et al., 2012; Soddu et al., 2012; Vanhaudenhuyse et al., 2011, 2010, 2009) may provide a standard experimental benchmark, or gold standard, to validate new methods in the future. However, because this benchmark has not yet been established, investigators may appeal to a different rationale when attempting to identify which methods are most appropriate for clinical practice.

While remaining agnostic to the superiority of any method, we aim to make explicit the normative rationale for utilizing a particular fMRI or EEG active paradigm over others in the absence of a precise estimation of clinical utility. This, we argue, may be an effort to strike a balance between false positive and false negative diagnostic errors. Both false positive and false negative diagnoses in this patient group may have important implications for medical decision-making. These implications may include end-of-life issues (Demertzi et al., 2011), pain management (Boly et al., 2008; Demertzi et al., 2013b, 2009; Laureys et al., 2002; McQuillen et al., 1991; Schnakers and Zasler, 2007), and the concerns of family members (Graham et al., 2014; Jox et al., 2012; Kitzinger and Kitzinger, 2013; Kuehlmeyer et al., 2012). While problems of diagnostic error extend to all neuroimaging methods, we focus our discussion on fMRI and EEG active paradigms that are a source of current disagreement in the disorders of consciousness literature.

Assessing consciousness after serious brain injury

Consciousness, as defined in clinical neurology, consists of two components: wakefulness and awareness (Giacino et al., 2014; Laureys et al., 2004; Multi-society task force on PVS, 1994; Plum and Posner, 1982; Jennet and Plum, 1972). Wakefulness is demonstrated by behavioral or electrophysiological manifestations of arousal, while awareness is demonstrated by sustained, reproducible, voluntary behavior, or evidence of language comprehension and expression (Giacino et al., 2004, 2002; Multi-Society task force on PVS, 1994). For any given healthy conscious individual, wakefulness is often, if not always, accompanied by awareness.

Following a period of coma, some seriously brain injured patients may emerge into a VS/UWS or minimally consciousness state (Giacino et al., 2002; Jennet and Plum, 1972; Plum and Posner, 1982). Individuals diagnosed as VS/UWS exhibit semi-regular circadian rhythms, yet evidence no concomitant awareness of visual, auditory, tactile or noxious stimuli (Multi-society task force on PVS, 1994). The VS/UWS is therefore referred to as, “wakeful unresponsiveness” (Jennet and Plum, 1972). By contrast, minimally conscious patients demonstrate regular circadian rhythms with intermittent but reproducible evidence of awareness (Giacino et al., 2002). The fine-grained sub-categorizations of minimally conscious state+ and minimally conscious state− have recently been introduced to parse out differences in the complexity of functional recovery (Bruno et al., 2013). Minimally conscious+ patients demonstrate higher-order behavioral responses, such as command following or intelligible verbalization, while minimally conscious− patients exhibit lower order responses, such as visual pursuit, localization to noxious stimuli, or stimulus-driven cognition (Demertzi and Laureys, 2014). All such conditions are generally referred to as disorders of consciousness (DoC).

Current methods for diagnosing seriously brain-injured patients utilize a combination of clinical assessment, patient history, structural MRI, and resting state EEG. The primary diagnostic instrument, the clinical examination, probes a patient's preserved awareness through behavior. According to validated behavioral scales, visual fixation, command following, functional object use, localization to noxious stimuli, intelligible verbalization, or intentional communication are evidence of awareness (Shiel et al., 2010) (Giacino et al., 2004; Multi-society task force, 1994; Teasdale and Jennet, 1974). If a patient demonstrates any number of these behaviors in a predictable and task appropriate manner, it is inferred that the patient is, at least minimally, conscious (See Seel et al., 2010 for an extensive review).

While the clinical examination is considered the gold standard for diagnosis of seriously brain-injured patients (Giacino et al., 2014), results may conflict due to variations in scale application or the subjective interpretation of examination findings. Studies investigating the accuracy of these scales consistently suggest that 30–40% of VS/UWS patients may be clinically misdiagnosed (Andrews et al., 1996; Childs et al., 1993; Schnackers et al., 2009). Moreover, in a recent cohort study assessing diagnostic accuracy in seriously brain-injured patients, it was demonstrated that levels of awareness fluctuate over a 6.5-hour time interval (Candilieri et al., 2011). Taken together, these findings suggest that variability in the interpretation of examination findings, as well as the unique clinical presentation of each patient, may hinder diagnostic accuracy. The potential for misdiagnosis that follows raises serious ethical concerns. Indeed, misdiagnosis may have profound implications for medical management, the well being of patients' families, and end-of-life decisions (Fins, 2003).

After accounting for the proportion of patients misdiagnosed as VS/UWS due to clinical errors, there still remain patients who retain residual preserved awareness despite appearing to be entirely behaviorally unresponsive at the bedside. These patients, while unable to demonstrate overt evidence of awareness, may nevertheless be capable of covert command following with the aid of fMRI or EEG. The fMRI mental imagery task developed by Owen and colleagues was the first to identify a VS/UWS patient who retained covert awareness (Owen et al., 2006). In this study, a 23-year old patient, who had been clinically diagnosed as VS/UWS, was instructed to imagine two activities – playing tennis and moving from room to room in her house – while fMRI recorded her brain activity. Previous fMRI research on healthy participants demonstrated that willfully imagining these activities produced unique hemodynamic changes in the supplementary motor area and parahippocampal gyrus, respectively (Boly et al., 2007). Based on these findings, investigators were able to decode which activity the patient was imagining from the imaging evidence alone. When compared to healthy participants, the patient's neural response was statistically indistinguishable. Neural activation was robust, reproducible, task-appropriate, and sustained for precise 30-second intervals, thus precluding the possibility of an automatic, stimulus driven response to verbal cues (Nachev and Husain, 2007; Owen et al., 2007). In 2010, this technique was applied in a heterogeneous sample of clinically diagnosed VS/UWS and minimally conscious patients (Monti et al., 2010). Of those in the VS/UWS subgroup, four (17%) willfully modulated their brain activity to commands.

The application of fMRI active paradigms has been highly effective for identifying covert awareness in some behaviorally unresponsive patients. It has also generated a significant body of work that applies the principles of command following to fMRI in a variety of ways (see Table 1). Despite this success, fMRI poses several technical limitations that may prevent widespread use in clinical practice. fMRI is expensive, requires patients to be transported to the imaging unit, and may be inappropriate for patients who are critically ill, are unable to remain motionless in the scanner, or have metallic implants. In light of these obstacles, investigators have turned toward portable, noninvasive EEG techniques to assess residual awareness in seriously brain-injured patients (see Table 2). Like fMRI active paradigms, these methods investigate whether participants can covertly modulate their brain activity. These studies primarily assess power and wave frequency oscillations in a participant's cortex. Using these methods, investigators have demonstrated, among other findings, the utility of power spectral analysis in detecting covert motor imagery (Goldfine et al., 2011), the detection of covert command following in several patients clinically diagnosed as VS/UWS (Cruse et al., 2011, 2012a), and a significant link between the probability of residual cognition and patient etiology (non-traumatic versus traumatic brain injury) (Cruse et al., 2012b).

Table 1

Examples of fMRI active paradigms.

Study	Participants	Task	Relevant finding(s)
Owen et al. (2006)	1 DoC patient (VS/UWS = 1)	Mental imagery (tennis vs. spatial navigation)	Neural modulation in DoC patient to commands
Boly et al. (2007)	34 healthy participants	Mental imagery (tennis vs. spatial navigation)	Neural modulation in healthy participants to commands
Monti et al. (2010)	54 DoC patients (VS/UWS = 23; MCS = 31)	Mental imagery (tennis vs. spatial navigation)	Neural modulation in DoC patients to commands
Bardin et al. (2011)	6 DoC patients (MCS = 5; LIS = 1)14 healthy participants	Mental imagery (varying motor tasks: tennis; swimming; table tennis; racket ball; pushing legs; volleyball; karate; basketball; rock climbing)	Neural modulation in healthy participants and DoC patients to commands
Monti et al. (2013)	1 DoC patient (MCS = 1) 21 healthy participants	Selectively attending to visual stimulus	Evidence of selective attention to visual stimulus in DoC patient to command
Naci et al. (2013)	15 healthy participants	Selectively attending to auditory stimulus	Evidence of selective attention to auditory stimulus in healthy participants to command
Fernández-Espejo and Owen (2013)	1 DoC patient (VS/UWS = 1)	Mental imagery (tennis vs. spatial navigation)	Evidence of neural modulation in DoC patient to command
Naci and Owen (2013)	3 DoC patients (VS/UWS = 1; MCS = 2)	Selectively attending to auditory stimulus	Evidence of selective attention to auditory stimulus in DoC patients to command

Vegetative State/Unresponsive Wakefulness Syndrome = VS/UWS; Minimally Conscious State = MCS; Locked in Syndrome = LIS; Disorders of Consciousness = DoC.

Table 2

Examples of EEG active paradigms.

Study	Participants	Task	Relevant finding(s)
Goldfine et al. (2011)	3 DoC patients (1 = LIS; 2 = MCS)5 healthy participants	Mental imagery (swimming vs. spatial navigation)	Mental imagery in healthy participants and some DoC patients to command;Differing spectral changes in DoC patients relative to healthy participants
Cruse et al. (2011)	16 DoC patients (VS/UWS = 16)12 healthy participants	Motor imagery (clenching fist vs. wiggling toes)	Motor imagery in DoC patients and healthy participants to command
Cruse et al. (2012a)	1 DoC patient (VS/UWS = 1)6 healthy participants	Motor imagery (bilateral hand movement)	Motor imagery in DoC and healthy participants to command
Cruse et al. (2012b)	23 DoC patients (MCS = 23)	Motor imagery (clenching fist vs. wiggling toes)	Motor imagery to command in patients with traumatic etiologies
Lulé et al. (2013)	18 DoC patients (LIS = 2; MCS = 13; VS/UWS = 3)16 healthy participants	Auditory oddball task	Evidence of command following in some healthy participants and DoC patients

Electroencephalography = EEG; Vegetative State/Unresponsive Wakefulness Syndrome = VS/UWS; Minimally Conscious State = MCS; Locked in Syndrome = LIS; Disorders of Consciousness = DoC.

Clearly, there is a breadth of research strategies for assessing the presence of covert awareness in seriously brain-injured patients with active paradigms. These strategies are driven by the common goal of accurately tracking the presence and recovery of consciousness. Investigators may attempt to estimate the accuracy of these methods, or their clinical utility, in order to identify which is most appropriate for clinical practice. However, this may prove difficult for both practical and principled reasons. This problem may be clarified by laying out the challenges that arise when attempting to estimate clinical utility for both fMRI and EEG active paradigms.

Estimating clinical utility

An estimation of a test's diagnostic accuracy – its clinical utility – is derived from calculating its sensitivity and specificity. Sensitivity and specificity are best understood through a four-by-four matrix, which represents the convergence or divergence of the test results and the true state-of-affairs (see Table 3). A true positive is an instance of the results corresponding with the state-of-affairs. This may be evidence of an effect when the target phenomenon is, in fact, present in the test conditions. Similarly, a true negative may be no record of an effect because the target phenomenon is absent from the test conditions. In either case, the test results faithfully reflect the experimental target. A test that most reliably produces true positives and true negatives is considered a gold standard.

Table 3

Standard formulae for estimating clinical utility.

Test outcome	Experimental target present	Experimental target absent
Test effect measured	True positive	False positive
Test effect not measured	False negative	True negative
Estimation of clinical utility	Sensitivity = [n(true positives) / (n(true positives) + (false negative)]	Specificity = [n(true negatives) / (n(true negatives) + (false positives)]

False positives and false negatives – known as type I and type II errors – are categorically different. These findings represent a divergence in the test results and the true state-of-affairs. A false positive occurs when evidence of an effect is measured, yet the target phenomenon is absent from the test conditions. Conversely, a false negative occurs when an effect is not measured even though the target phenomenon is, in fact, present in the test conditions (see Table 3).

The rate of false positives and false negatives bears directly on the estimation of sensitivity and specificity of a diagnostic test (see Table 3). A test is 100% sensitive if it detects every instance of a target phenomenon, even at the expense of returning false positive results. As false negatives increase, sensitivity is reduced. This relationship is expressed formally as: [n(true positives) / (n(true positives) + n(false negatives)].

Specificity, on the other hand, is a function of a test's ability to uniquely detect a target phenomenon. A test is 100% specific if it accurately discriminates between true positives and false positives. While a highly specific diagnostic test effectively shields out false positives, increases in discriminatory standards may also inadvertently exclude true findings. A false negative may result. This relationship is expressed formally as: [n(true negatives) / (n(true negatives) + n(false positives)].

There is a natural difficulty in estimating the sensitivity and specificity of imaging techniques designed to detect residual awareness in brain-injured patients. This is primarily due to the lack of independent methods for checking the correspondence of test results with the true state-of-affairs. To determine if a positive or negative result is, in fact, true or false, one requires a veridical benchmark of the state-of-affairs, or gold standard, for comparison. However, since the only known method for assessing residual awareness in seriously brain-injured patients appeals to behavior, information derived from patients already known to be behaviorally unresponsive cannot serve this function. Without this benchmark, it may remain unclear whether imaging findings are accurate, even if they fit with the overall clinical profile of the patient. This problem has direct implications for the calculation of clinical utility. If there are no independent methods for confirmation, an accurate calculation of sensitivity and specificity may be difficult or impossible to determine.

Sensitivity

One way to address this problem with respect to sensitivity may be to compare imaging findings in patients to the performance of healthy participants under identical experimental conditions. Because healthy participants are known to be conscious, it stands to reason that a technique capable of reliably producing true positives in a control population will likely produce true positives in the clinical population. Moreover, ensuring the reliability of effects at the single-subject level in healthy participants is an important criterion for successful application of any method used on individual patients. In a control study used to validate the widely deployed fMRI mental imagery task, Boly and colleagues demonstrated that the technique was sensitive in 100% of individual healthy participants (Boly et al., 2007). Group level identification and localization of canonical regions of interest were used in classification models tailored for analysis at the single-subject level. Moreover, the task design – mental imagery for discrete 30-second intervals – combined with instructions to up-regulate specific brain regions, leaves little doubt that the measured effect is the result of willed neural modulation (Fernández-Espejo and Owen, 2013; Owen et al., 2007; Owen, 2013).

Despite the high-level accuracies reported in some control studies of active paradigms, others have drawn into question the single-subject level sensitivity of both fMRI and EEG (Giacino et al., 2014). For example, in the seminal fMRI study by Monti and colleagues, only 1 of 31 patients clinically diagnosed as minimally conscious was able to command follow through mental imagery (Monti et al., 2010). One may presume that this is a clear evidence of false negatives. After all, if a patient can follow commands behaviorally, then intuitively she should also be able to command follow through mental imagery. However, it is also possible that in some patients, deficits in language comprehension, decision-making, working memory or executive function may yield brain activity too weak to interpret or to engage in the experimental task (Naci et al., 2012). This suggests that, while findings in patients may appear to be false negatives, there is no way to independently confirm whether this is a failure of the imaging technique, or a confound inherent to patient response. Alternative explanatory models, such as a diagnostic taxonomy that distinguishes minimally conscious+ from minimally conscious− patients (Bruno et al., 2012), may account for these findings.

Notwithstanding the challenge of accurately identifying false negatives in patients, several studies testing EEG active paradigms in healthy participants have also returned sensitivity rates of less than 100%. In the most extensive EEG active paradigm study to date, Guger and colleagues (2003) detected volitional motor imagery with classification accuracies greater than 70% in roughly 50% of healthy participants. The EEG technique developed by Cruse and colleagues only detected willed cortical oscillations in 75% of healthy participants (Cruse et al., 2011), and Lulé and colleagues' recent EEG study produced variable results in a similar control group (Lulé et al., 2013). These findings highlight that false negatives remain a serious methodological challenge for investigators seeking to develop EEG active paradigms.

To be sure, these studies have demonstrated statistically significant findings in the majority of participants, and have laid a conceptual groundwork to develop more effective methods. In the case of healthy participant studies, investigators have recourse to independently check results by conducting post-scan interviews. However, in behaviorally unresponsive patients this is not possible. In either case, though, what these findings do suggest is that there is no prima facie guarantee that any active paradigm will detect awareness in 100% of participants known to be conscious a priori. The risk of false negatives that follows must be taken into consideration as investigators interpret the findings of future studies.

Specificity

Ideally, diagnostic tests should not produce false positives. However, it is standard to presume that a small proportion of positive results may occur by chance. The well-known demonstration of phantom hemodynamic activity in a deceased salmon (Bennett et al., 2010), or EEG recordings from watermelon (Schafer et al., 2011), remind us that background noise may appear as positive results without proper statistical precautions. It is therefore a common practice to assess neuroimaging and EEG findings against statistical thresholds of varying degrees of conservativeness.

The statistical threshold for any given neuroimaging or EEG experiment determines the level of evidence necessary for rejecting the null-hypothesis. A standard threshold in neuroimaging and EEG research is derived from the assumption that 5% of results will be false positives. From this assumption the common statistical threshold, p < 0.05, is determined. The confidence level of each independent test, or p-value, is compared to this statistical threshold. If a test result's p-value is greater than the statistical threshold, it is assumed that the result may be due to chance.

In any neuroimaging or EEG experiment, it is often necessary to perform multiple independent tests — whether from individual participants, individual trials, individual fMRI voxels, or individual EEG recording sites. As independent tests grow, comparisons between tests also increase. These comparisons precipitate a greater probability of false positives occurring by chance. This problem may be further compounded with the use of machine learning — a common method for developing neuroimaging and EEG active paradigms used to assess residual awareness in behaviorally unresponsive patients. If, for example, the learning phase of a linear classifier is biased due to, among other reasons, neuroimaging or EEG artifacts, the resulting classification model may inadvertently produce false positive results and undermine the specificity of the test.

Evidently, statistical thresholds may require varying degrees of conservativeness. Correction methods, such as Family Wise Error, Bonferroni Correction, or False Discovery Rate reduce significance thresholds to ensure the highest confidence level for positive findings. Bonferroni correction, for example, divides the standard significance threshold of p < 0.05 by the number of independent tests. If an investigation has 20 independent tests, this correction would reduce the significance threshold to p < 0.0025 (5%/20). Employing these methods of analysis allows investigators to set the evidentiary bar high. For linear classifiers, this may produce a highly conservative classification phase that rejects borderline findings. However, such correction methods may also increase the burden of proof to such a degree that true results are erroneously interpreted as false positives.

fMRI versus EEG

In fMRI experiments, standard statistical techniques that correct for multiple comparisons (e.g. Family Wise Error or False Discovery Rate) reduce the probability of false positives. Moreover, in contrast to whole-brain analysis, studies that restrict interpretation to regions of interest vastly reduce the quantity of multiple comparisons and subsequent need for statistical correction. For these reasons, region of interest analysis is a standard approach in fMRI active paradigms used to assess residual awareness in behaviorally unresponsive patients (Fernández-Espejo and Owen, 2013; Monti et al., 2010; Naci and Owen, 2013; Owen et al., 2006). The application of alternative analytic methods, such as independent component analysis (Demertzi et al., 2014) or multivariate pattern analysis (Bardin et al., 2012), may precipitate methodological challenges in need of future critical reflection (Fins, 2012).

Apart from these statistical details, the fMRI mental imagery task design also reduces the chance of false positives. The probability of a participant – patient or healthy volunteer – activating the supplementary motor area merely by chance, at exactly the right point in the experimental task, sustaining that activity for precise 30-second intervals, and repeating the process several times, is incalculably low (Owen, 2013). This gives reason to believe that, provided that the task design is well controlled and implemented faithfully, fMRI active paradigms rarely produce false positive results.

Estimating the clinical utility of EEG active paradigms, on the other hand, is currently more challenging. Because the most extensive EEG active paradigm studies to date have not returned 100% sensitivity ratings in healthy participants, the probability of false negatives remains a problem. Notwithstanding this complication, a far greater obstacle is controlling for movement artifacts. VS/UWS patients, while presumed to be unaware, may still exhibit bodily movement. Without appropriate methods for artifact exclusion, even minor electrical activity in a patient's facial musculature may bias linear classification and lead to false positive results.

Two statistical methods for inferring the presence of covert command following from the output of machine-learning algorithms – binomial distribution and permutation testing – approach this problem differently (Cruse et al., 2011; Cruse et al., 2013; Goldfine et al., 2013; Noirhomme et al., 2014). A binomial test is a parametric method for distributing the probability of positive or negative findings across discrete, independent tests (Noirhomme et al., 2014). Importantly, binomial distribution assumes that the analyzed data are acquired from an experimental design that treats each trial as an independent assay. A permutation test, on the other hand, is a nonparametric method that estimates the distribution of the null-hypothesis from the entire data set (Noirhomme et al., 2014). By randomly permuting individual trials hundreds to thousands of times, this method mitigates the possibility of trial effects that may inadvertently influence the results of adjacent trials. Unlike binomial distribution, assumptions of a permutation model are not built into the design of the experiment. However, a large quantity of data may be required to achieve enough permutations to draw valid conclusions (Cruse et al., 2014b).

To demonstrate the relevant differences between these statistical methods, consider a recent dispute over the use of an EEG active paradigm to detect covert awareness in a cohort of seriously brain-injured patients. Using an EEG active paradigm and binomial distribution model, Cruse and colleagues tested for the presence of covert command following in 12 healthy participants and 16 patients clinically diagnosed as being in a VS/UWS (Cruse et al., 2011). The experiment was organized in a pseudorandomized block design, in which each block was preceded by verbal instructions for toe wiggling or fist clenching mental imagery. Verbal instructions were followed by a series of 15 beeps, or trials, with randomly generated inter-stimulus time intervals of 4.5–9.5 s. 25 electrodes over the motor cortices, extending from electrode C3 to C4, were selected for data acquisition. Filtered and artifact subtracted data were provided to a liner support vector machine (SVM) classifier in matched block pairs. Cruse and colleagues hypothesized that, if event-related desynchronizations and synchronization occurred in motor regions in response to motor imagery commands, it could be inferred that participants were willfully modulating their brain activity, and were therefore aware. The results of the study indicated high classification accuracy for both hand and toe mental imagery in 75% of healthy participants (n = 9) and 19% of VS/UWS patients (n = 3). Decoding accuracy in the patient sample was between 61% and 78%, with p-values ranging from p = 0.0015 to p = 3 × 10−8.

Due to concerns over low frequency artifacts permeating across trials and blocks – hypothesized to be due to participant movement – Goldfine and colleagues reanalyzed a subset of these findings to determine whether the patient results were false positives (Goldfine et al., 2013). Unlike Cruse and colleagues' sequential block-wise pairing, Goldfine and colleagues compared all possible block pairs during classification. This approach, it was argued, eliminated the possibility of artifact contamination across blocks, which may be inaccurately classified as positive findings if the blocks are too close together in time. Reanalysis decreased the classifier accuracy for two of the three patients who returned positive results in Cruse and colleagues' original study to chance — 56% and 59%, respectively. Classification accuracy was also decreased, though still within the boundaries of significance, for three healthy participants.

Goldfine and colleagues then carried out a permutation test on all blocks to calculate significance. It was argued that the block design of the original study violated the assumptions of the binomial distribution model (Goldfine et al., 2013). Since low frequency artifacts appearing in one trial may contaminate the results of adjacent trials, the tests were not functionally independent as binomial distribution assumes. The p-values for two patients – those whose classification accuracies were reduced to chance – were raised above the significance threshold of p < 0.05. However, the value of one remaining patient still fell within the boundaries of significance (p = 0.0286) (Goldfine et al., 2013: supplement).

Finally, to correct for multiple comparisons, Goldfine and colleagues applied a False Discovery Rate method. “…[W]ithout correction for multiple comparisons,” they argued, “a classifier is expected to yield ‘positive’ results in a fraction of patients just by chance, and we wanted to determine whether this phenomenon was a plausible interpretation of our findings” (Goldfine et al., 2013: supplement). After correction, the p-value for the final patient fell outside the boundaries of statistical significance. The completed reanalysis suggested that all patients presumed to generate true positive results in the original study had instead generated false positives.

In response to these criticisms, Cruse and colleagues argued that permutation testing did not satisfy statistical standards for drawing valid conclusions from limited data. Because the significance test was restricted to blocks, as opposed to trials, nearly half of the participants only produced 36 permutations (Cruse et al., 2013: 291). However, the literature on nonparametric statistics suggests that at least 1000 permutations are necessary for this type of analysis (Maris, 2004; Maris and Oostenveld, 2007). Cruse and colleagues also noted that the method proposed by Goldfine and colleagues rendered dramatic changes to the healthy participant data. In the original study, 75% of healthy participants exhibited high classification accuracy and significant p-values (Cruse et al., 2011). By contrast, permutation testing and false discovery rate correction returned significant findings in only 40% of healthy participant data (2 out of 5). Reflecting on this, Cruse and colleagues argued that:

Although there are few known truths when attempting to detect covert awareness, the one thing we can assume to know is that when healthy volunteers are asked to do the imagery tasks described in our original paper, they are doing them […]. Because [Goldfine and colleagues'] method fails to detect command following in 60% of healthy volunteers, it is equally likely to fail to detect command following (where it exists) in most patients. (Cruse et al., 2013: 291–292).

The reduction of sensitivity resulting from permutation testing is not exclusive to the Goldfine–Cruse exchange. In a recent study by Noirhomme and colleagues (2014), findings derived from an EEG active paradigm and binomial test on a similar cohort of seriously brain-injured patients (Lulé et al., 2013) were reanalyzed using permutation testing to determine the relative benefits of different statistical models. Of those patients who initially obtained classification accuracies above chance with a binomial test (n = 3), reanalysis with permutation testing reduced classification accuracies to chance and no statistically significant p-values were found. Noirhomme and colleagues concluded that binomial testing could lead to biased estimations of significance and potential diagnostic errors. However, unlike Cruse and colleagues' investigation, the patient data reanalyzed by Noirhomme and colleagues was derived from patients clinically diagnosed as minimally conscious (Noirhomme et al., 2014: 689). This suggests that, at the very least, the patients were known to be minimally aware prior to EEG testing. Taken together, these findings suggest that, when using EEG in combination with machine-learning to detect residual awareness in seriously brain-injured patients, binomial distribution may bias results toward false positives. However, permutation testing may also bias results toward false negatives.

To be sure, the goal of rigorous peer review and reanalysis is to derive the most accurate neuroimaging and EEG techniques for assessing residual covert awareness in seriously brain-injured patients. While a subset of fMRI active paradigms may be sufficiently refined to estimate clinical utility, it is evident that more work needs to be done on any protocol that utilizes mental imagery as a proxy for behavioral command following. Movement artifacts, statistical noise and other confounds are challenges that both fMRI and EEG active paradigms face. False positive and false negative errors are germane to all such challenges. Ultimately, neuroimaging techniques with high sensitivity and specificity are desired. However, finding the right balance between false positives and false negatives may be the only recourse until a veridical measurement of consciousness is identified (Giacino et al., 2014: 106; Giacino et al., 2009).

Inductive risk and diagnostic error

Philosophers of science may describe the challenge of estimating the clinical utility in this context as a problem of underdetermination. In its most general form, the underdetermination thesis holds that, for any two empirically equivalent hypotheses that have the same class of observation consequences, empirical rationale will not resolve which hypothesis to accept. Rather, one must appeal to extra-empirical reasons – that is, pragmatic or normative reasons – for hypothesis adoption (Biddle, 2013; Lauden, 1990; Lauden and Leplin, 1991; Longino, 2002; Norton, 2008). Indeed, in some cases, this may amount to adjudicating between the relative risks and benefits of accepting a hypothesis on the grounds of its potential social impact. It is well known that seriously brain-injured patients are already vulnerable to high rates of misdiagnosis (Andrews et al., 1996; Candilieri et al., 2011; Childs et al., 1993; Schnackers et al., 2009). Thus, choosing among particular experimental methods, which may be known to bias results toward false positives or false negatives, may compound this diagnostic problem. If error is inevitable, which type of error should investigators make?

One way to resolve this problem is to appeal to a principle of inductive risk (Douglas, 2000). A principle of inductive risk weighs the relative value of accepting an underdetermined hypothesis by calculating the benefits and harms it could generate. This may amount to formulating explicit acceptance rules that assign value to all possible outcomes (Hempel, 1965: 92). For example, it may be argued that the evidential strength for any given hypothesis is, “a function of the importance […] of making a mistake in accepting or rejecting [it].” (Rudner, 1953: 2; original italics). This may lead one to conclude that, “How sure we need to be before we accept a hypothesis will depend upon how serious a mistake [caused by accepting it] would be” (Rudner, 1953: 2; original italics). With respect to assessing seriously brain-injured patients, what risks do we take in attributing a false positive or false negative diagnosis? An answer to this question will depend on the value we place on the possible outcomes.

In the conclusion of their reanalysis, Goldfine and colleagues implicitly gesture toward such an acceptance rule. They state that, “in the diagnostic setting (e.g., the determination of consciousness, genomic diagnosis of cancer), classifier failure can misinform clinical decision making, with major consequences for patients and families” (Goldfine et al., 2013: 290). Given that Goldfine and colleagues' reanalysis brought to light the possibility of false positive results related to binomial testing, we take their conclusion to mean that false positive diagnoses, as opposed to false negatives, pose the greatest problem in this clinical context. Indeed, if seriously brain-injured patients are falsely ascribed awareness as a result of certain analytic methods, decisions that are inconsistent with the patient's actual condition may occur. But is it also possible for equally important consequences to arise if we falsely presume that a patient is in a VS/UWS when, in fact, she retains some dimension of awareness? Building on related work (Jox et al., 2012), we outline three clinical scenarios where this trade-off between false positive and false negative diagnoses may be most pronounced in this patient group (see Table 4).

Table 4

Potential risks of false positive and false negative diagnoses.

Clinical scenario	Diagnostic error	Potential outcomes and risks
Disclosure of results	False positive	Prolonged and unnecessary treatment in the Intensive Care Unit. Unnecessary financial and emotional exhaustion for families.
	False negative	Potentially undermine family's belief that patient is aware, causing emotional suffering. Potentially cause families and health care workers to be less attentive to patient.
Withdrawal of life support	False positive	Missed “window of opportunity” to withdraw life sustaining therapies.
	False negative	Premature withdrawal of life sustaining therapies.
Resource allocation	False positive	Unnecessary allocation of resources to medically futile cases.
	False negative	Insufficient medical resources allocated to patient.

This list is not intended to be exhaustive. See also Jox et al. (2012) for a complimentary outline of risks.

Disclosure of results to patients' families

The disclosure of false diagnostic information to families may have serious implications for clinical decision-making. It is standard practice for a family member to serve as a surrogate decision maker. If surrogates are provided information inconsistent with the patient's actual condition, clinical decisions may not align with the best interests of patients and families. A false positive diagnosis may influence a surrogate to prolong life support in the hopes that a patient will recover. Since it is falsely believed that the patient is covertly aware, financial and emotional resources may be misdirected. Conversely, a false negative diagnosis may cause a family and health care workers to emotionally and financially withdraw from a patient. Since it is falsely believed that there is no evidence of awareness, enrolling the patient in rehabilitation or long-term care facilities may not occur.

Clearly, fMRI and EEG active paradigm findings, whether true or false, will have some measurable impact on clinical decision making. Determining whether if and how such information is disclosed may mitigate the harms that could arise in conjunction with false positive and false negative diagnoses (Graham et al., 2014). Disclosure of results may be framed in various ways. It may be argued that positive results (both true positives and false positives) pose a minimal risk because the effect of disclosure is negligible. Families coping with serious brain injury may already hold deep beliefs about a patient's preserved awareness. In one study investigating family attitudes, 90% of patients diagnosed as VS/UWS were thought by family members to retain some dimension of awareness contrary to their clinical diagnosis (Tresch et al., 1991). However, in a recent multi-center study investigating families' perceptions of preserved consciousness in brain injured patients, 76% of participants estimated the same level of consciousness that diagnostic tests showed (Jox et al., 2015). In the remaining cases, the investigators reported that consciousness was underestimated. The difference in these study findings may be explained by the recent introduction of the minimally conscious state diagnostic category. Indeed, many of the patients included in Tresch and colleagues' (1991) study may have satisfied the diagnostic criteria of the minimally conscious state. In either case, these studies suggest that surrogates may have strong attitudes regarding preserved consciousness in brain injured patients, which may be unaltered by fMRI or EEG findings. Further systematic study of families' attitudes in response to positive results is clearly warranted.

The disclosure of negative findings (both true and false negatives) may be more ethically problematic. Aside from errors generated by statistical modeling, other confounds, including a patient's attention span, ability to understand instructions, and contingent levels of arousal, may contribute to false negative findings. Given that surrogates may not understand the test conditions that produce negative findings – whether true or false – such results may be framed as “uninformative”, if disclosed at all. Whether or not their response is appropriate, families are likely to interpret all information, or lack there of, as diagnostically salient. Hence, disclosure of false negative results without proper counseling is palpably dangerous. Ensuring that surrogates understand the meaning of negative findings should be a central focus of future research.

Withdrawal of life sustaining therapies

To date, the most extensive research on this patient group has been in chronic populations (n > 6 months post-injury). It is in these cases that the problems of estimating clinical utility have emerged. However, it is likely that neuroimaging methods will soon be used in the diagnosis and prognostication of acutely comatose patients. While the clinical benefit of these techniques in acute populations has yet to be conclusively demonstrated (but see Norton et al., 2012; Gofton et al., 2009; and see Young, 2009b for review of outcome after cardiac arrest), it is possible that fMRI or EEG findings may assist surrogate decision makers in weighing the decision to continue, or withdraw, life-sustaining therapy. Thus false positive and false negative findings may have significant implications for acutely comatose patients.

Although the desiderata to withdraw life-sustaining therapies would surely extend beyond fMRI or EEG findings – namely to, among other factors, etiology, structural neuroimaging, and clinical examination (Giacino et al., 2014) – the risk remains that contravening evidence of awareness or favorable prognosis would not be produced if statistical methods biased results toward false negatives. As noted earlier, families may also falsely interpret a lack of measured effect as evidence of unawareness. Without proper counseling, this may lead to premature withdrawal of life sustaining therapies during the acute phase of recovery.

Conversely, imaging techniques that bias results toward false positives may undermine withdrawal of life sustaining therapies. Recent research at Canadian trauma centers demonstrates a 32% mortality rate following serious brain injury with 70% of deaths associated with the withdrawal of life sustaining therapies (Turgeon et al., 2011). Most decisions to withdraw life sustaining therapies occurred very early, with one half of deaths within 72 hours of injury. For surrogates in this situation, there may be a perceived “window of opportunity” for the withdrawal of care to avoid undesirable patient outcomes (Kitzinger and Kitzinger, 2013). If acutely comatose patients are falsely ascribed consciousness or a favorable prognosis, surrogates may forego the withdrawal of life sustaining therapies and miss this opportunity.

It may also be argued that false negatives generate a self-fulfilling prophecy in this research setting. Indeed, if it is already evident that withdrawal of life sustaining therapies occurs early on, false negative results may hasten patient death before sufficient evidence pertaining to long-term outcome is acquired. This may engender methodological complications for any study investigating the diagnostic and prognostic benefit of fMRI or EEG in acutely comatose patients. If, on the other hand, investigators err on the side false positive results, methodological and ethical concerns regarding the premature withdrawal of life sustaining therapies may be avoided. If fMRI or EEG results prompt a closer examination of the patient's condition, even if the results are eventually ruled out as false positives, this “second look” may reveal information that is both ethically and epistemologically salient. What is clear is that more research must be done to determine the possibility of drawing accurate diagnostic or prognostic information from fMRI and EEG in acutely comatose patients. Assuming that this is possible, disclosing this information with proper counseling may help families navigate these difficult decisions.

Equitable distribution of medical resources

False positive and false negative diagnoses of seriously brain injured patients may influence the fair distribution of medical resources. Patients with serious brain-injuries require highly specialized medical care in both the intensive care unit and long term care facilities. In a recent Canadian supreme court case regarding the withdrawal of life sustaining therapies in a patient clinically diagnosed as being in a VS/UWS (Cuthbertson v. Rasouli, 2013), the cost of intensive care was estimated at $3000 per day. In jurisdictions where medical expenses are subsidized by federal, state, or provincial bodies, these costs may be transferred to tax payers. Notwithstanding the financial burden of prolonged treatment, the use of resources for cases perceived to be futile may be difficult to reconcile with the extant needs of other patients. Due to these circumstances, seriously brain-injured patients are highly vulnerable to socio-economic pressures. The reasonable probability of false positive diagnoses resulting from active paradigms, even with the most conservative statistical models, may draw criticism from policy makers at medical and governmental institutions.

On the other hand, false negative diagnoses may have a detrimental effect on treatment outcome if medical resources are removed from patients that, in fact, stand a good chance of recovery. It is well known that after partial recovery from serious brain injury, most patients are transferred to facilities with suboptimal palliative care (cf. Fins, 2003). While it remains to be demonstrated empirically, the decrease in intensive interaction that accompanies the transfer to long-term care may be therapeutically detrimental.

Problems of just distribution of medical resources will surely arise when caring for seriously brain-injured patients. The general risks to the equitable distribution of resources are difficult to quantify. How to balance potential false positive and false negative diagnoses with the demands of health care justice will be an important question as this research matures.

Conclusions

In this article, we have reviewed several fMRI and EEG active paradigms used to assess residual covert awareness in seriously brain-injured patients. We anticipate that, given further work, these techniques will continue to draw attention to ambiguities in prognosis and diagnosis of these patients and further refine our understanding of their conditions. We have also attempted to highlight the unique methodological puzzles that emerge in this research setting. These puzzles, we believe, hinder efforts to provide an accurate calculation of clinical utility. Adjudicating between particular methods of analysis, which, in turn, may bias results toward false positives or false negatives, remains one of the most challenging obstacles in this research setting. In the face of uncertainty, we suggest that the relative risks engendered by false positive or false negative diagnoses may play a role in guiding future research.

To be sure, like many research programs investigating the use of active paradigms for assessment of serious brain-injury (e.g. Giacino et al., 2014; Stender et al., 2014), we do not currently advocate diagnosis or prognostication based exclusively on fMRI, EEG, or any other form of neuroimaging. Nor do we suggest that the evaluation of a technique's clinical utility be completely handed over to risk assessment. When in doubt, the most methodologically sound and ethically responsible way to determine if a patient is truly covertly aware may be to appeal to a variety of sources of evidence (Cruse et al., 2014b; Forgacs et al., 2014; Giacino et al., 2014; Gibson et al., 2014). This approach may incorporate findings from validated clinical examinations, as well as alternative structural and functional techniques, to evaluate the total body of evidence regarding a patient's condition. However, it must also be acknowledged that some patients, whose entire clinical presentation is consistent with the VS/UWS, may nevertheless demonstrate awareness with the use of validated neuroimaging or EEG techniques. An interim solution to this problem may be to develop a rigorous model-based diagnostic decision-tree that guides physicians and researchers through the relative risks of diagnostic error. Future work on such a decision-tree would greatly benefit research programs looking to integrate functional neuroimaging into standard diagnostic protocols (Coleman et al., 2009; Laureys et al., 2004; Owen, 2013).

If the methodological obstacles inherent to fMRI and EEG active paradigms make diagnostic errors difficult to avoid, we ask: what type of error is best to make? Interest in this problem, particularly in relation to the stipulation of statistical thresholds, is now an emerging topic of discussion in the disorders of consciousness literature. We recommend a sustained discussion of this problem from both conceptual and empirical perspectives. In some cases, this may amount to reanalyzing previous neuroimaging findings with alternative statistical methods (Goldfine et al., 2013; Noirhomme et al., 2014). In others, investigators may choose to apply several different correction methods to one data set in order to demonstrate how these alter study findings (Cruse et al., 2014a). Finally, some investigators may choose to apply two or more neuroimaging methods to a patient sample for comparison (Forgacs et al., 2014; Gibson et al., 2014; Stender et al., 2014). In each instance, careful reflection on this problem will strengthen the case for including neuroimaging and EEG in routine diagnostic and prognostic protocols. This, we believe, will ultimately benefit the long-term well-being of seriously brain-injured patients and their families.