Challenges and Pitfalls Associated with Diagnostic and Prognostic Applications of Functional Neuroimaging in Disorders of Consciousness.

The diagnostic assessment of patients with disorder of consciousness is currently based on clinical testing at the bedside and prone to a high error rate in the assessment of the degree of conscious awareness. Investigation of more objective assessment strategies, such as the use of functional magnetic resonance imaging (fMRI) to detect conscious awareness, are becoming increasingly popular in the research community. However, inherent challenges to the use of fMRI threaten its validity as a diagnostic tool and will need to be resolved prior to its integration into the clinical setting. These challenges, which range from the heterogeneity of the patient sample to factors influencing data acquisition and biases in interpretation strategies, are discussed below. Recommendations aimed at mitigating some of the limitations are provided.

INTRODUCTION

Following a severe brain injury, some patients may progress from coma into a vegetative or minimally conscious state (VS; MCS). VS is characterized by reemergence of spontaneous or stimulus-induced eye-opening and MCS by inconsistent but clearly discernible behavioral signs of conscious awareness [1]. Diagnostic assessment is particularly challenging as it is based on behavioral evidence which is often difficult to decipher. There are no definitive tests capable of discerning whether a particular behavior is reflexive, non-reflexive but involuntary, or volitional. The clinical impression is ultimately based on inferences drawn from the circumstances under which the behavior occurs. This scenario opens the door to a variety of influential factors that may bias the diagnostic impression. Such influences stem from the examiner (e.g., tendency to interpret reflexive behaviors as volitional and vice-versa), patient (e.g., fluctuating arousal, underlying sensory impairments), and clinical setting (e.g., sedating medications). Consequently, diagnostic error in patient with disorders of consciousness (DoC) is high (30-40%) [2]. This problem can be mitigated to some degree using a standardized approach to assessment such as the Coma Recovery Scale-Revised (CRS-R) [3], however, a gold standard for detection of conscious awareness does not exist. Failure to detect conscious awareness can lead to dire consequences, including premature withdrawal of care [4].

The last decade witnessed a surge in the development of advanced neuroimaging techniques designed to identify patients capable of following instructions and communicating but unable to execute these behaviors on bedside examination due to speech and motor impairments. Prior fMRI investigations relying on the blood-oxygen level dependent (BOLD) signal have shown that some patients diagnosed with VS or MCS can covertly follow instructions to perform motor imagery and visual cognition tasks [5-7]. The term “functional locked-in syndrome” has been proposed to describe this sub-set of patients [8]. Despite the encouraging results reported in these studies, there are many challenges and pitfalls associated with the application of fMRI techniques to questions surrounding conscious awareness. In the following sections, we discuss these challenges, which are often overlooked in published reports, focusing particularly on issues concerning study design, data collection, data analysis, and interpretation and dissemination of results. Although fMRI is a promising modality that may help decrease the high rate of misdiagnosis in DoC, we contend that these issues will need to be reconciled before diagnostic applications can be incorporated into the clinical setting.

FMRI-BASED APPROACHES TO DIAGNOSTIC ASSESSMENT OF DOC

fMRI relies on the basic principle that active brain regions require increased oxygenation which leads to a contrast in the BOLD signal between more and less active areas. The BOLD signal can be derived in one of three ways: 1) presenting passive language or visual tasks and interrogating regions that respond to the stimuli, 2) presenting active tasks that require effortful cognitive activity on behalf of the participant in response to specific instructions, and 3) presenting no stimuli or tasks and simply asking the participant to rest in the scanner.

Typically, investigators look for differences in BOLD activity between patients with different DoC diagnoses (i.e. VS versus MCS) and between patients and healthy subjects. A clinical diagnosis of VS is questioned when brain activity patterns or connectivity resemble that of patients diagnosed with MCS or healthy subjects.

In the case of passive and resting-state fMRI, conclusions regarding effortful cognitive activity (e.g., decision-making) cannot be made. For example, if upon presentation of spoken sentences activations are observed in brain regions known to be involved in language function, some assumptions can be made regarding semantic processing [6, 9]. This does not provide evidence that the subject can understand language and does not demonstrate purposeful interaction with the environment, a central tenant of preserved consciousness. To avoid this problem, investigators have focused efforts on using active fMRI paradigms to elicit command-following and communication, behaviors that suggest preservation of conscious awareness. Hence, this review focuses primarily on active paradigm fMRI, although the challenges presented here may also apply to passive and resting-state functional neuroimaging.

A seminal study published in 2006 [6] compared BOLD brain activation patterns in response to instructions to perform motor imagery and covert spatial navigation in one patient with severe brain injury deemed to be in a VS and 12 healthy control subjects. The pattern and amplitude of the BOLD signal change in the expected brain regions (i.e., supplementary motor area during imagination of playing tennis, and parahippocampal gyrus, posterior parietal lobe, and lateral premotor cortex during imagination of navigating rooms of a familiar home) approximated that of the healthy control subjects. Results were interpreted as evidence that the participant was able to understand and respond appropriately to spoken instructions despite the lack of behavioral evidence for command-following.

In a follow-up study [5], 54 patients with DoC (VS=23, MCS=31) participated in an fMRI study employing a similar command-following paradigm. Activation maps from one patient diagnosed as MCS and four others diagnosed as VS (based on CRS-R criteria) suggested they were actively following commands via mental imagery. A second task was administered to determine if the basic paradigm could be restructured to assess communication ability. The same patients were instructed to respond to six yes/no autobiographical questions by linking one imagery task to “yes” and the other to “no”. Activation patterns revealed that one patient of the 54 tested accurately responded to five of the six questions (no activation was observed in the regions of interest on the sixth question). A subsequent investigation instructed patients with varying levels of consciousness (i.e., VS, MCS, emerged from MCS, locked-in, healthy controls) to silently name a series of common objects presented visually [10]. Patients with normal or recovered consciousness (i.e., locked-in syndrome, emerged from MCS) activated the same network of language structures as healthy volunteers performing the same task. Of note, one patient in VS also activated the complete network. The remaining subjects, including patients in MCS who demonstrated clearly-discernible behavioral signs of conscious awareness on bedside examination, showed either partial or no activation of the regions of interest.

The common occurrence of negative neuroimaging findings in patients who demonstrate conscious awareness at the bedside is concerning and contributes to the poor statistical measures of performance for fMRI [11]. Here, we aim to identify the challenges inherent to this area of research and the threats to the validity of fMRI studies in patients with DoC. These issues will need to be overcome before fMRI becomes a reliable clinical tool for assessing conscious awareness.

INFLUENCE OF SUBJECT CHARACTERISTICS

Heterogeneous Injury Profiles

In view of the relatively small number of patients with DoC available for study, samples tend to be comprised of subjects with heterogeneous injury characteristics. Investigations often combine subjects with traumatic and non-traumatic injuries, markedly different lesion profiles, variable lengths of time from injury, and a variety of other factors into one cohort [5, 11]. The patient group is then routinely compared to a relatively homogenous cohort of healthy control subjects. Patient heterogeneity may lead to weak group-level results that reflect the variability of the sample rather than the presence or absence of conscious awareness [12]. Anatomical templates cannot be created or are imprecise due to the diverse nature of lesion profiles and lesions in brain regions hypothesized to be involved in performing the experimental task or structural abnormalities often prevent normalization of the brain to an anatomical template for group analysis. Although several groups are developing methods to improve spatial normalization in patients with significant structural anomalies [13-15], more work is needed to confirm the validity of these analytic procedures.

Another consequence of including participants with heterogeneous injury profiles is that study findings cannot be generalized to individuals or groups whose injury profiles differ from the experimental groups. Replication of results is also problematic due to the difficulty in obtaining a matched patient sample as is combining findings from multiple studies into a meta-analysis.

Cognitive Deficits

All active fMRI tasks aimed at eliciting covert command-following and communication require at least some coordination of multiple cognitive subsystems (e.g., auditory, language, working memory). Patients with DoC characteristically have associated cognitive and cortical sensory deficits, including slow processing speed, diminished capacity to sustain or shift attentional focus, language disturbance, rapid forgetting, hemi-spatial neglect, central deafness and cortical blindness, which may suppress or prevent network activation. These problems may disrupt performance on mental imagery tasks, in particular, which place high demands on mental control functions [16]. Visually-based paradigms may be less cognitively demanding, however, they require sustained eye-opening and sufficient apprehension of the stimulus. To the extent possible, stimuli and task demands should anticipate these areas of impairment to avoid misinterpretation of results [17]. Some of these challenges may be overcome by simplifying cognitive demands, avoiding over-reliance on a single sensory modality and properly calibrating the number and timing of stimuli presented.

Physical and Motor Impairments

fMRI studies may also be complicated by physical and motor impairments. Patients must be screened to ensure they are able to manage oral secretions due to risk of aspiration while lying supine. Improper positioning related to the presence of a cervical collar or splinting equipment, restlessness and oral reflexive movement (e.g., bruxism) may also compromise image acquisition. Another consideration that must be addressed is the possibility of incontinence during the scanning procedure. Although management of bowel and bladder routines are typically discussed in advance of data collection and tube feeding are typically withheld starting the night before the scan, incontinence may occur, necessitating at least temporary termination of data collection. While attending to personal care needs is necessary, this activity detracts from the amount of time allocated to acquire the imaging data and again jeopardizes interpretation of the results.

A number of other factors may lead to premature scan termination or uninterpretable results. These may include behaviors such as restlessness, eye-closure, and poor arousal as well as refusal to continue. The incidence of such behaviors should be systematically monitored and recorded within pre-specified time-frames during each scanning run (see Table for an example of a behavioral monitoring log). This information can then inform an overall Test Completion Code that provides a justification for incomplete or uninterpretable data sets. Table proposes a series of Test Completion Codes and how they may be used to better understand study findings. Although published data identifying the incidence and factors leading to scan termination is not currently available, we anticipate that systematic monitoring of behaviors during scanning sessions will allow investigators to examine questions related to the feasibility of fMRI studies and their potential for entering clinical practice.

Data Acquisition

A number of challenges may arise during data acquisition when conducting neuroimaging studies in subjects with DoC. A technical consideration that may affect the quality of the data acquired is the potential for metal implants on the skull, internal fixation of the jaw, and some non-injury related devices such as braces and permanent retainers to introduce image artifact. In patients who have had neurosurgical intervention to repair cranial defects, prior imaging must be examined to ensure that the metal, even if non-ferrous, does not cause image artifact. While these implants are typically non-ferrous and therefore safe for the MRI environment, they often cause artifact that may render the data uninterpretable [18]. If prior MRI studies are available, investigators may choose to exclude subjects based on the level of artifact visible on those scans. However, it is unlikely that prior imaging will utilize the same pulse sequences as the experimental scans, therefore, there remains some risk that artifacts will not be detected a priori.

To obtain fMRI data that are of an acceptable quality, subjects must lay supine in the scanner for an extended period of time while minimizing movement. The novel surroundings of the MRI scanner (e.g., dark and physically restrictive bore, close proximity of the radiofrequency coil to the face, loud noises, hard surface of the gurney and supine positioning , etc.) may contribute to patient discomfort and lead to behaviors that increase motion. Reflexive chewing movements, jaw clenching, posturing and restlessness are commonly-observed and may introduce motion artifact or prevent proper alignment of the head and scanner hardware. Unlike clinical scans, sedation is generally not an option as normalizing muscle tone and restlessness may come at the cost of compromising cognition. Methods of detecting and correcting for motion artifact are evolving but may not be adequate for the degree of motion often seen in this population [19].

Fluctuation in arousal level during image acquisition is frequently encountered in patients with DoC. While eye-closure does not always indicate loss of arousal, it can cause significant loss of data during administration of visually-based tasks. MRI compatible eye-tracking systems are helpful for monitoring eye-opening status while scanning, however, it is not possible to discern whether eye-closure signals a change in arousal without the aid of simultaneous EEG recording.

Clear exclusion criteria and extensive screening procedures in advance of the scan will prevent recruitment of some patients based on these complicating factors, however, given the frequent behavioral fluctuations that characterize patients with DoC, unanticipated events may lead to premature termination of data collection and occasionally lead to insufficient data for robust analysis. Loss of data related to any of these factors generally reduces statistical power and may threaten the validity of the results. Conditioning patients in advance of the scan by playing pre-recorded scanner noise, placing the patient in a mock MRI-scanner if one is available, simulating the positioning for extended periods of time, and introducing straps that may be used during the scan to control limb movement, may help determine how well the procedure may be tolerated and alleviate some of the issues on the day of data collection. Table shows complications that may occur during data acquisition and management strategies that may be considered to mitigate effects.

Data Analysis

There are no accepted standard procedures for fMRI data preprocessing and statistical analysis. Variability in scanner hardware, image acquisition parameters, data analysis approach (univariate versus mulitvariate; whole-brain versus region of interest), software, and processing pipelines complicates comparison of findings across studies and reduces the likelihood of replication. In addition, there is increased risk for obtaining disparate results using the same fMRI data set that ultimately reflect the selected analytic approach rather than underlying neurophysiology [20-25].

There is little consensus as to whether fMRI data should be analyzed in native anatomical space or an anatomical template. If the functional data are not spatially normalized to a template, it is difficult to determine the structural regions underlying the functional activation maps. This issue is exacerbated by injury-related structural abnormalities such as mass lesions and midline shift that distort and displace brain structures, frequently obscuring clear boundaries. Exclusion of patients with significant anatomic distortion may skew the study sample towards patients with relatively intact anatomy or increase the risk of interpretive error if patients with severe traumatic brain injury are included. When all subjects are mapped in the same anatomical space, determining the brain structures underlying functional maps is more straightforward and group-level activation maps can be generated. At the same time, spatial normalization introduces error as the process of warping individual anatomical images into a common space, despite marked differences in lesion volume and location, leads to registration errors and mislabeling of brain regions [13]. Most normalization algorithms will fail if presented with grossly abnormal anatomy while others rely on manual adjustments to the warping process. Thus, the extent of injury to the brain can introduce selection bias and reduce generalizability.

Data Interpretation

To the extent possible, measures should be taken to protect non-automated processes (e.g., drawing regions of interest on individual brains, determining whether activation patterns in patients sufficiently match those of healthy control subjects, defining what constitutes complete versus partial network activation) against potential sources of bias. In the absence of robust automated processes for identification of specific brain structures in patients with large structural abnormalities, investigators must rely on neuroradiologic review. To guard against subjectivity bias, results can be read by two raters, or the same rater can re-analyze the same results at a second time point. In fMRI studies designed to detect conscious awareness, it is particularly important to incorporate steps that ensure observer independence. Individuals reading the imaging results may be biased by knowledge of the behavioral profile of each patient and two individuals may arrive at separate conclusions examining the same set of images. Conversely, the radiologic data should not be accessible to those responsible for interpreting the behavioral findings. Unless adequate methodological safeguards are taken, there is a high likelihood that the conclusions reached will be tainted by unintentional bias. Unfortunately, much of the published literature in this area does not explicitly address the issue of observer independence, leaving open questions about the level of confidence associated with the results reported.

In many cases, however, the final judgment as to whether the pattern of activation sufficiently matches that of control subjects is made qualitatively [5, 9, 11, 26]. There are several reasons for the popularity of this approach. First, brain injury is inherently heterogeneous making spatial normalization difficult and necessitating individual “readings” of brain maps. Second, a standardized algorithm for determining whether activation patterns are within the range of “normal” has not been developed. Consequently, pre-established regions of interest (ROI) are interrogated during a specific task, after which the resulting activation maps are visually-inspected and a judgment made as to whether the ROI has been sufficiently activated to infer task performance. This approach is subject to examiner bias in that the same individual responsible for localizing brain structures (usually by comparing the anatomical scans to a brain atlas) determines whether the actual regions of activation sufficiently encompasses the predicted regions.

Dissemination of Results

There is little agreement as to how and when fMRI study results should be released. Nonetheless, precautions must be taken to avoid misuse and misinterpretation of investigational findings. In the United States, informed consent must be obtained from each subject prior to study enrollment. When decision-making capacity is impaired, as is the case with DoC patients, the surrogate must be approached. During the informed consent process, the surrogate should be made aware that participation will have no direct benefit and is primarily intended to advance knowledge rather than alter the course of clinical care. Opinions vary greatly as to whether investigational results should be released following completion of all data collection or immediately after individual results become available [27], if at all. There is also lack of agreement regarding who should receive the results- the surrogate, medical team or both. The Secretary’s Advisory Committee on Human Research Protections is developing recommendations regarding the appropriate mechanisms and methods for release of individual results and has published recommendations for release of general results (http://www.hhs.gov/ohrp/sachrp/commsec/sharing_study_data_and_results.html). Although research results are typically not released during the course of an investigation to prevent physical, psychological and emotional harm, there is increasing debate as to whether it is appropriate to release investigational functional neuroimaging results involving patients with DoC [28]. One argument for disclosing research results is that many clinical tools are not validated, have high rates of misdiagnosis, and may not be reproducible. This is especially pertinent in DoC because a gold standard for diagnosis does not exist. A related argument is that because diagnostic error is estimated to be between 30-40% in this population using routine clinical measures [2], surrogates should be furnished with all available information (accompanied by appropriate caveats regarding risks of use) and be permitted to make their own decision about how to weigh each element of information. Regardless of which course of action is chosen, the process and timeline for disclosure should be outlined in advance and stipulated when obtaining written consent.

CONCLUSION

The extensive list of exclusion criteria and narrow inclusion criteria, coupled with the relatively small number of patients with DoC, results in a restricted pool of eligible subjects. Study samples tend to be comprised of patients who are medically stable in the chronic phase of recovery with relatively intact brain anatomy but non-uniform lesion profiles. This unavoidably leads to selection bias, limits generalizability of the results, and undermines the validity of meta-analyses .

fMRI is a valuable tool that can provide information regarding potentially preserved mental processes following severe brain injury. For patients with DoC who may have associated motor, language, and cognitive impairments, functional neuroimaging may be the only modality that accurately reveals conscious awareness. The putative benefits of fMRI must be considered in the context of uncertainty and potential for harm. Functional imaging paradigms should be tailored to accommodate marked cognitive limitations and anticipate significant loss of data arising from behavioral or medical issues. Eligibility criteria should be clearly defined and protections incorporated into the protocol to protect against suspected sources of bias. The plan for data analysis and interpretation should be clearly spelled out in advance of opening enrollment and confidence limits communicated clearly.

In view of the lack of standard procedures for data acquisition and analysis, it is important to clearly describe all procedures used for data acquisition, analysis and interpretation. Furthermore, the small number of available DoC patients coupled with the extensive exclusion criteria for participation in fMRI research, leads to poor generalizability The clinical utility of the diagnostic use of fMRI in this population remains unclear as few studies have determined the sensitivity, specificity, and diagnostic accuracy of this method. The relationship between fMRI and behavioral findings requires further clarification to better understand the neurophysiologic substrate underlying MCS and the anticipated long-term outcome.

Table 1

In-Scanner behavior monitoring log sample.

Task/Run Time Post Scan Start	0:00	0:03		0:06		0:09		0:12...
Poor Arousal
Restlessness
Aggression
Verbal Refusal
Upper extremity flexion/ extension
Lower extremity flexion/ extension
Trunk rotation
Neck rotation
Spontaneous vocalization
Facial grimacing/chewing
Researcher/family talking
Other:

Table 2a

Proposed set of test completion codes that may be used to explain incomplete or uninterpretable data sets.

Test Completion Code	Description
1.0	Test completed without complication
2.0	Behavioral evidence of eye-closure/underarousal
3.0	Intermittent motion
4.0	Agitation/emotion lability
5.0	Medical complication
6.0	Equipment failure
7.0	Surrogate refusal
8.0	Patient refusal
9.0	Compromised by other factors (specify)

Table 2b

An example of the use of test completion codes to collect data on study feasibility.

	Test Completion Codes
Subject ID	Passive Paradigm	Active Paradigm	Resting State
S1	1.0	4.0	2.0
S2	2.0	1.0	9.0
% Scans without complication	50	50	0

Table 3

Complications that may occur during data acquisition and strategies to mitigate effects.

Complication	Mitigating Strategies
Improper positioning due to cervical collar or splinting	Review with the physician options for temporarily removing or modifying restraining equipment for scan duration
Excessive motion due to discomfort related to scanning environment (i.e., tight bore opening, excessive auditory stimulation, etc.)	Pre-condition patient to scanning setting in advance of study participation (i.e, simulate confining bore environment and RF coil, expose to pre-recorded scanner noise)
Restlessness	Introduce straps and/or sheets to reduce limb movement; this may be trialed in advance to assess tolerance
Eye-closure	Monitor eye-closure using MR compatible eye-tracking device or camera
Poor arousal	Assess ability to maintain eye-opening or demonstrate arousal prior to scan; facilitate arousal via frequent breaks and deep pressure stimulation during scanning
Poor management of oral secretions	Prior to scanning session, monitor ability to manage sections in supine position
Incontinence	Discuss with clinical staff holding intake of solids and liquids prior to scan; consider alteration of bowel/bladder schedule