A selective review of multimodal fusion methods in schizophrenia.

Schizophrenia (SZ) is one of the most cryptic and costly mental disorders in terms of human suffering and societal expenditure (van Os and Kapur, 2009). Though strong evidence for functional, structural, and genetic abnormalities associated with this disease exists, there is yet no replicable finding which has proven accurate enough to be useful in clinical decision making (Fornito et al., 2009), and its diagnosis relies primarily upon symptom assessment (Williams et al., 2010a). It is likely in part that the lack of consistent neuroimaging findings is because most models favor only one data type or do not combine data from different imaging modalities effectively, thus missing potentially important differences which are only partially detected by each modality (Calhoun et al., 2006a). It is becoming increasingly clear that multimodal fusion, a technique which takes advantage of the fact that each modality provides a limited view of the brain/gene and may uncover hidden relationships, is an important tool to help unravel the black box of schizophrenia. In this review paper, we survey a number of multimodal fusion applications which enable us to study the schizophrenia macro-connectome, including brain functional, structural, and genetic aspects and may help us understand the disorder in a more comprehensive and integrated manner. We also provide a table that characterizes these applications by the methods used and compare these methods in detail, especially for multivariate models, which may serve as a valuable reference that helps readers select an appropriate method based on a given research question.

Brief Introduction to Schizophrenia

Schizophrenia is a chronic, disabling mental disorder diagnosed on the basis of a constellation of clinical psychiatric symptoms and longitudinal course. The disease impairs multiple cognitive domains including memory, attention, and executive function (Heinrichs and Zakzanis, 1998). Although the causes and mechanisms of schizophrenia are still unclear, a hypothesis of neural network “disconnection” has been proposed (Friston and Frith, 1995). This hypothesis proposes that schizophrenia arises from dysfunctional integration of a distributed network of brain regions or a misconnection of neural circuitry leading to an impairment in the smooth coordination of mental processes, sometimes described as “cognitive dysmetria” (Andreasen et al., 1998). A number of studies have been published which have tried to delineate the underlying neural mechanisms of schizophrenia using functional magnetic resonance imaging (fMRI; Pearlson, 1997; Loeber et al., 1999; Curtis et al., 2001; McIntosh et al., 2008b; Yu et al., 2011a), structural MRI (sMRI; Giuliani et al., 2005; Strasser et al., 2005; Douaud et al., 2007), diffusion tensor imaging (DTI; McIntosh et al., 2008a; Kubicki et al., 2009; Sussmann et al., 2009), and genetics (Bahn, 2002; Williams et al., 2010b; Ripke et al., 2011). For example, in fMRI studies, deficits in dorsolateral prefrontal cortex (DLPFC; Hamilton et al., 2009) and the temporal lobe (Calhoun et al., 2008) have often been implicated in schizophrenia. In addition, DTI studies have found reduced integrity of the anterior limb of the internal capsule, uncinate fasciculus (UF), and anterior thalamic radiation (ATR) regions in schizophrenia (Kubicki et al., 2005b; Bellani et al., 2009; Sussmann et al., 2009). Finally, in genome wide association (GWAS) studies (Bahn, 2002; Shifman et al., 2004; Williams et al., 2009; Ripke et al., 2011), gene expression alterations have been reported in CACNA1C, ANK3, MIR137, and DISC1 recently. Given the breadth of findings across modalities, it is natural to evaluate the inter-relationship among them within a larger context.

Why Multimodal

Recently, collecting multiple types of brain data from the same individual using various non-invasive imaging techniques [MRI, DTI, electro-encephalography (EEG), MEG, etc.] has become common practice. Each imaging technique provides a different view of brain function or structure. For example, fMRI measures the hemodynamic response related to neural activity in the brain dynamically; sMRI provides information about the tissue type of the brain [gray matter (GM), white matter (WM), cerebrospinal fluid (CSF)]. DTI can additionally provide information on structural connectivity among brain networks. Another useful measure of brain function is EEG, which measures brain electrical activity with higher temporal resolution than fMRI (and lower spatial resolution). Typically these data are analyzed separately; however separate analyses do not enable the examination of the joint information between the modalities. In addition, focusing on a single modality in brain connectivity estimation can sometimes lead to contradicting conclusions (Plis et al., 2011).

Based on many previous findings in brain connectivity (Olesen et al., 2003; Rykhlevskaia et al., 2008; Camara et al., 2010; Yu et al., 2011b), it is plausible to assume covariation between brain function and structure. In addition, approximately 80% of total assayed genes display some cellular expression in brain (Lein et al., 2007). Therefore, it is reasonable to assume that these very diverse data may share certain underlying contributions to the etiopathology of brain disorders such as schizophrenia and jointly analyzing multimodal data may uncover previously hidden relationships that can only be partially detected in each modality alone. A motivating example is shown in Figure 1, in which a fiber bundle provides input to a distant location in varying degrees in two participants whose particular single nucleotide polymorphisms (SNPs) also differ, thus fMRI activity is different at the output location and possibly in functional connectivity (fMRI activity at the input is unaffected). The interactions between modalities (Figure 1B) would not be revealed by traditional separate analysis whereas a joint analysis (Figure 1C) would detect the underlying associations.

Figure 1

Motivating example for multimodal joint analysis. (A) Fiber bundle [fractional anisotropy (FA) from DTI] provides varied input to a distant region in two participants whose particular SNPs also differ, thus fMRI activity is different at the output location (orange circles) and possibly in functional connectivity, while fMRI activity at the input (red circles) is unaffected. Traditional separate analysis of each modality (B) would not reveal such modal interactions, whereas a joint analysis (C) may detect the underlying associations.

A key motivation for multimodal fusion is to take advantage of the cross-information provided by multiple imaging techniques, which in turn can be useful for identifying dysfunctional regions or potential biomarkers for many diseases. Basically, multimodal fusion refers to the use of a common symmetric model that explains different sorts of data (Friston, 2009). This is a complicated endeavor, and can generate results that are not obtainable using traditional approaches which focus upon a single data type or processing multiple datasets individually. However, in the real world, challenges often come from the fact that conclusions need to be drawn from high dimensional and noisy brain imaging data from only a limited number of subjects. Hence efficient and appropriate methods should be developed and chosen carefully.

In addition, multimodal brain imaging data has shown increasing utility in answering both scientifically interesting and clinically relevant questions. As well as providing the conceptual glue to bind together data from multiple types or levels of analysis, the related computational methods are also valuable for clinical research on the mechanisms of disease progression. For example, researchers often look for reliable relationships between few summary values from each modality, and certain behavioral dependent variables, e.g., the symptom scores of the patients with disorders, which may allow assessment of functional links between brain dynamics and human cognitive process. The incorporation of behavioral/cognitive data into multimodal fusion analysis which take the entire data into account and are not limited to a single region will not only improve our physiological understanding of brain diseases, but also provide insight into the neural bases of these human cognition or behavior (Makeig et al., 2009).

For schizophrenia, though strong evidences exist for functional, structural, and genetic abnormalities in each single-modality analysis, there is still lack of a diagnostic “gold standard” which is specific and sensitive enough to guide clinical decisions (Fornito et al., 2009). Even though GWAS studies have identified several risk genes that may influence brain functions affected in schizophrenia, they have not illuminated the etiology of the disease as they were perhaps anticipated to (Ripke et al., 2011; Williams et al., 2011). Therefore, it is likely at least in part, that a lack of consistent findings results from most studies favoring only one data type or not combining modalities in an integrated manner (Calhoun et al., 2006a).

Next, we will review several multimodal fusion applications in schizophrenia applied to different data types. Note that many fusion applications rely on studying correlations between highly distilled measures, e.g., from small regions of interest (Tregellas et al., 2007; Bates et al., 2009; Foucher et al., 2011). However the applications we reviewed mostly examine more complete relationships among the data types based on multivariate methods. Finally, we classify these applications based on the fusion models used, in order to provide a framework for selection of methods in future applications.

Multimodal Fusion Applications in Schizophrenia

Function–function

Functional magnetic resonance imaging–EEG (ERP)

The combination of fMRI and EEG is the most frequently used brain imaging fusion examples. Because it is believed that the spatial precision of fMRI can be complemented by the temporal precision of EEG (Friston, 2009) so that the neurovascular coupling mechanism can capture both neuronal and hemodynamic activity as two important components. One approach used for fMRI–EEG fusion is to constrain one modality with another, as reported in Dale and Halgren (2001), Eichele et al. (2005), Henson et al. (2010). While these are powerful techniques, a common limitation is that the potentially unrealistic assumptions of a fundamentally different nature than the known modality are imposed on the constrained data. For example, constraining EEG sources to lie within fMRI activated regions makes the implicit assumption that both signals have the same origin, or if present should be at least correlated, which is not always true. The fMRI signal is undoubtedly related to changes in blood follow, but it is by no means related in a simple ways to neuronal activity. Furthermore, transient EEG activity might not induce a detectable hemodynamic effect, and intense neuronal activity that is de-synchronized may lead to strong hemodynamic effects with on detectable average electrical activity (Jbabdi, 2009). By contrast, data-driven fusion methods are attractive for exploring data sets more fully, as they do not require prior hypotheses about the connection of interest.

Calhoun et al. (2006b, 2011) first proposed a joint independent component analysis (jICA) model (which assumes that two or more modalities share the same mixing matrix) to fuse together fMRI and EEG data, and aimed to identify specific differences in the neuronal chronometry of target detection for chronic schizophrenia patients compared to healthy controls (HC). One group-discriminative joint component was found via two-sample t-test (p < 0.0001), showing a clear difference in fMRI at bilateral fronto-temporal regions and in the event-related potentials (ERPs) during the N2–P3 latency range (ERPs are labeled based on their ordinal position following the stimulus onset, e.g., P3 for the third positive peak or N2 for the second negative peak). Note that both the hemodynamic and electro-physiologic phenomena were jointly expressed in this source, which have been previously implicated in schizophrenia, but no prior study had showed that changes in these two modalities were linked.

Correa et al. (2010b) also tried to differentiate schizophrenia from controls by applying multimodal canonical correlation analysis (mCCA) to fMRI–EEG data during performance of an auditory oddball (AOD) task (Kiehl and Liddle, 2001). Significant group differences were found in the bilateral temporal lobe/middle anterior cingulate region in fMRI, associated with the N2 and P3 peak in EEG. Multimodal CCA allows a different mixing matrix for each modality and is able to find a transformed coordinate system that maximizes inter-subject covariation across two or more data sets (Li et al., 2009). As shown in Figure 2, a set of linked components derived from a three-way fusion of sMRI, fMRI, and EEG were detected (Correa et al., 2010a), which significantly discriminate schizophrenia patients from controls. On examining the inter-subject modulation in conjunction with the spatial and temporal components, the results imply that patients with schizophrenia have less functional activity and less GM in the motor and temporal areas and also in part of the ERP N2/P3 complex. The comparison of t-tests for three modality (fMRI, sMRI, and EEG) versus two modality (fMRI and sMRI) analyses for this set of components is also listed, showing that the three-way analysis was more significant than a two- or one-way analysis, which further validates our motivation for multimodal fusion.

Figure 2

Functional magnetic resonance imaging/EEG/sMRI fusion by mCCA from Correa et al. (.

Structure–structure

Gray matter–white matter

Using joint ICA, Xu et al. also identified four joint sources that were significantly associated with schizophrenia. The linked GM–WM regions identified in each of the joint sources included: (1) temporal – corpus callosum, (2) occipital/frontal – inferior fronto-occipital fasciculus, (3) frontal/parietal/occipital/temporal –superior longitudinal fasciculus, and (4) parietal/frontal – thalamus (Xu et al., 2009), which include a large number of brain regional networks and reflecting the widespread nature of the disease. A complex pattern of regional increases and decreases in schizophrenia were reflected. For example, for joint source 1, linkage between less GM in temporal lobe and the less WM in corpus callosum is interesting and may be related to the posterior corpus callosum connections to temporal lobe (Woodruff et al., 1993; Downhill et al., 2001). While for joint source 4, larger thalamic WM concentrations were shown in schizophrenia, suggesting that GM difference in the parietal and frontal lobe are associated with WM difference in thalamus, consistent with the role of the thalamus as a relay station (Wolfarth et al., 1985). In summary, jICA provides a unified framework to identify joint GM–WM sources that show group differences, which can cover as much neurological ground in a single computational model as achieved by many traditional separate analyses (Shenton et al., 2001).

Moreover, the inter-relationship between GM and WM can be analyzed to identify the tissue distribution abnormalities in schizophrenia (Xu et al., 2011) by utilizing novel features called structural phase and magnitude images. Where the phase image indicates the relative contribution of GM and WM, and the magnitude image reflects the overall tissue concentration. Six networks were identified showing significantly lower WM-to-GM in schizophrenia, including thalamus, right precentral–postcentral, left pre-/post-central, parietal, right cuneus–frontal, and left cuneus–frontal sources. Such findings demonstrate that structural phase and magnitude images can naturally and efficiently summarize the associated relationship between GM and WM.

Function–structure

Cognitive dysfunction present in schizophrenia is often thought to be driven in part by disorganized connections between higher-order cortical fields, thus the combination of function and structure may provide more informative insights into altered brain connectivity (Rykhlevskaia et al., 2008).

Functional magnetic resonance imaging–GM

Figure 3 (Calhoun et al., 2006a) shows analyzed data collected from groups of schizophrenia patients and HCs using the jICA approach. The main finding was that group differences in bilateral parietal and frontal as well as posterior temporal regions in GM distinguished groups. A finding of less patient GM and less hemodynamic activity for target detection in these bilateral anterior temporal lobe regions was consistent with previous work. An unexpected corollary to this finding was that, in the regions showing the largest group differences, GM concentrations were larger in patients versus controls, suggesting that more GM may be related to less functional connectivity during performance of an auditory oddball task. Correa et al. (2008) also showed an interesting joint relationship between fMRI and GM by mCCA, with patients with schizophrenia showing more functional activity in motor areas and less activity in temporal areas associated with less GM as compared to HCs.

Figure 3

Auditory oddball/gray matter jICA analysis. Only one component demonstrated a significant difference between patients and controls. The joint source map for the auditory oddball fMRI data (A) and gray matter (B) data is presented along with the loading parameters for patients and controls (C).

Michael et al. (2010) introduced a method to identify inter-correlations among GM and fMRI voxels within the whole brain by reducing the cross-correlation matrix into histograms. Results show that the linkage between GM and functional activation in an auditory sensorimotor task (Schroder et al., 1999) is stronger in HCs than patients with schizophrenia. Specifically, GM regions in the cerebellum show more significant positive correlations with functional regions in HC. The cross-correlation can also be reduced to brain clusters (Michael et al., 2011) by fusing GM and fMRI contrast maps of a working memory task (Manoach et al., 1999). The maximum group difference occurred in medium difficulty working memory load. Particularly, the inter-cluster GM-Probe correlations for this load were positive in controls but negative in schizophrenia. While within one group, the inter-cluster correlation comparisons show no differences in controls but in patients with schizophrenia, indicating that the function-structure integrity during the recognition phrase is aberrant in schizophrenia.

Functional magnetic resonance imaging–DTI

Functional magnetic resonance, as a well-established neuroimaging technology, can act as a reference framework for validating conclusions derived from the relatively newer DTI method (Reinges et al., 2004), which can provide information regarding the integrity of WM tracts. Other recent work has shown that brain anatomical and functional connectivity are both altered in schizophrenia (Skudlarski et al., 2010).

Schlosser et al. (2007) observed a direct correlation in schizophrenia between frontal fractional anisotropy (FA) reduction and fMRI activation in regions in prefrontal and occipital cortices, which highlights a potential relationship between anatomical changes in a frontal-temporal anatomical circuit and functional alterations in the prefrontal cortex.

Sui et al. (2011) applied a blind data-driven model, mCCA + jICA, optimized for identifying correspondence across modalities, to real fMRI–DTI datasets from 164 subjects, including 62 HC, 54 SZ, and 48 bipolar disorder (BP) subjects. Only one joint group-discriminating component was detected between SZ and HC, including DLPFC and motor regions in fMRI of an auditory oddball detection task as well as parts of the ATR, SLF, and IFO WM tracts. The loading parameters of each modality also showed significant correlations with age. AOD_IC1 represents activations mainly in motor cortex, accompanied by a functional asymmetry with left dominance; see Figure 4A, consistent with the fact that the AOD task design required participants to push the button with fingers on their right hand. Controls had a very significant correlation r = 0.5, p = 4e − 5, while patient groups did not (p does not pass correction for multiple comparisons), implying that the motor regions of HC are normally more involved in the task with increasing age (Bennett et al., 2010), whereas schizophrenia patients have no such trend due to presumed motor system deficits (Rogowska et al., 2004).

Figure 4

Joint fMRI/FA component that is HC–SZ discriminative, from Sui et al. (. Spatial maps of the identified functional blobs (A) and WM regions (B) are displayed with the correlation plot between subjects’ loadings and ages. Specifically, HC in red line, SZ in blue line, BP in green line, and trend of all subjects in black line. (C) Shows a high-level brain interaction diaphragm according to the joint component. Functional region with a red solid line frame indicates a major portion activation and the dotted line frame indicates that only small part of it is activated. Abbreviations are defined below, SLF, superior longitudinal fasciculus; CST, corticospinal tract; IFO, inferior fronto-occipital fasciculus; ILF, inferior longitudinal fasciculus; ATR, anterior thalamic radiation; CGC, cingulum; FMAJ, forceps major; FMIN, forceps minor.

Fractional anisotropy_IC1, as shown in Figure 4B, as a joint component of AOD_IC1, also demonstrated a significant (p = 2e − 08), but anti-correlation with age. Note that all groups had low p values for this component, thus using one marker, suggesting all subjects have a general age-related decrease in WM integrity, in agreement with (Sullivan and Pfefferbaum, 2003; Grieve et al., 2007). We are also able to provide insights into the high-level brain function-structure network, which verified that the linked (joint) components do correspond to FA changes in known tracts and functional changes in distant regions connected to that tract (Figure 4C). Note that we are not directly performing fiber tractography; a strength of mCCA + jICA is that it can detect complicated FA/fMRI relationships without requiring a directly detected link.

Brain imaging–genotype

Understanding genetic influences on both healthy and disordered brain function/structure is a major focus of psychiatric neuroimaging (“imaging genomics”) and may provide important additional information. For example, using combined genetic and fMRI data can achieve better classification accuracy than using either alone (Yang et al., 2010), indicating that genetic and brain function represent different, but partially complementary aspects.

Functional magnetic resonance imaging–SNP

Liu et al. (2009) first proposed parallel ICA to investigate correlations between brain dysfunctional regions and putative disease susceptibility SNPs, as shown in Figure 5. A correlation of 0.38 between one fMRI component and one SNP component was found; both showed significant differences in loading parameters between the schizophrenia and control groups (p = 0.0006 for the fMRI; p = 0.001 for SNP). The fMRI component consisted of regions in parietal lobe, right temporal lobe, and bilateral frontal lobe. The relevant SNP component was contributed to significantly by 10 SNPs located in genes including those coding for the nicotinic alpha-7 cholinergic receptor (CHRNA7), aromatic amino acid decarboxylase (AADC), and disrupted in schizophrenia 1 (DISC1). The findings provide a proof-of-concept that genomic SNP factors can be investigated by using phenotypic imaging findings in a multivariate format.

Figure 5

Functional magnetic resonance imaging/SNP parallel ICA from Liu et al. (.

Meda et al. (2010) further extended the use of parallel ICA by simultaneously analyzing fMRI and 24 SNP markers that previously had been associated with schizophrenia. Three fMRI components (including PFC, ACC, STG, and MTG) correlated significantly with two distinct gene components including DAT and SLC6A4_PR, revealing specific interactions between schizophrenia risk genes on imaging phenotypes that represent brain function in attention/working memory/goal directed behavior, establishing a useful methodology to probe multivariate genotype–phenotype relationships.

Structural MRI–SNP

Jamadar et al. (2011) further adopted parallel ICA to examine the relationship between GM volumes and 16 SNPs spanning FOXP2 and four Reading Disability-related genes: DCDC2, DYX1C1, KIAA0319, and TTRAP. Five such GM–SNP relationships were identified. The superior prefrontal, temporal, and occipital networks were positively related to DCDC2 in the schizophrenia, but not control group. In addition, the identified networks closely correspond to the known distribution of language processes in the cortex. Thus, reading and language difficulties in schizophrenia may be related to distributed cortical structural abnormalities associated with Reading Disability-related genes.

Similarly, Jagannathan et al. (2010) linked sMRI and genetic (SNP) components using parallel ICA, and identified a sMRI component that significantly correlated with a genetic component (r = −0.54, p < 0.00005), which also distinguished SZ from HC. The GM deficits pointed to brain regions including frontal/temporal lobes and thalamus (p < 0.01), which are consistently implicated in previous reports. These deficits were related to SNPs from 16 genes associated with schizophrenia risk and/or involved in normal central nervous system development, such as AKT, PI3K, SLC6A4, and DRD2.

As to the neurodevelopmental and life span factors in schizophrenia, first episode and longitudinal computed tomography (CT), and MRI studies have shown that brain abnormalities in schizophrenia are present at onset of psychosis and are non-progressive (Nasrallah et al., 1986; Wood et al., 2009). These and other findings support the idea that schizophrenia is a developmental rather than a degenerative condition (e.g., Rund, 2009). Furthermore, the presence of ventriculomegaly and diminished hemispheric asymmetry in familial schizophrenics and in those of their relatives who appear to be transmitting the disorder, implies involvement of the genes controlling neurodevelopment (Frangou and Murray, 1996). All these evidence suggests that genetic vulnerability, environmental factors, and cerebral structural disturbances can act in combination to result in clinically manifest schizophrenia (Harrison and Owen, 2003). Fusion approaches can be used to characterize such changes by relating the fusion parameters with variables such as age (Sui et al., 2011).

Summary of Multivariate Methods

In summary, studies featuring multimodal combinations prove to be more informative in understanding brain activity and disorders and the complexities of schizophrenia make this disease an appropriate test bed for exploration using joint information derived from multimodal datasets. The existing multivariate fusion methods have different optimization priorities and limitations. Each method presents a different view in interpreting and connecting the multiple datasets based on their various hypotheses. In Table 1, we listed several multivariate fusion methods that have been applied in schizophrenia study. The following aspects are compared in detail, including feasible combinations, optimization assumptions, purpose of the analysis, requirement of priors, the number of the modalities that can be combined and the input data types, which may serve as a guideline on method selection based on a given research and data.

Table 1

Multivariate methods comparison and their applications to schizophrenia.

Methods	Combinations	Studies	Optimization assumptions	Goals and purposes	Need of priors	No. of modality	Input data
Joint ICA	fMRI–sMRI (GM), fMRI–EEG, fMRI–DTI (FA), GM–WM	Calhoun et al. (2006a), Calhoun et al. (2006b), Eichele et al. (2009), Calhoun et al. (2011), Franco et al. (2008), Xu et al. (2009)	X1 and X2 share the same mixing matrix, A1 = A2. The independence among joint components [S1, S2] is maximized	To examine a common mixing modulation across subjects among modalities and to find the linked source maps	No	2 is preferred, 3+ is possible	Features*
Multimodal CCA	fMRI–EEG, fMRI–sMRI (GM)	Correa et al. (2008), Correa et al. (2010b), Correa et al. (2010a)	Maximizes covariation of the mixing profiles across the two datasets, corr(A1, A2) = diag(r1, r2…rm)	To detect common as well as distinct level of connection between subject modulation	No	2 is classical, 3+ is possible	Features or raw data
mCCA + jICA	fMRI–DTI (FA)	Sui et al. (2011)	Assume the decomposed component from each modality have some degree of correlation between subject-mixing profiles. mCCA is first used in order to make the jICA job more reliable by providing a close initial match via correlation; jICA further separates the remaining mixtures in the joint maps	To achieve both flexible modal association (high or low correlation) and accurate source separation. It is easily to be applied to 3+ modalities	No	2+ is preferred	Features
Parallel ICA	fMRI–gene (SNP), GM–gene (SNP)	Liu et al. (2009), Meda et al. (2010), Jagannathan et al. (2010), Jamadar et al. (2011)	Maximizes the cost function based on both entropy (for all components) and the correlation term (for components whose the mixing profile correlations are above the threshold) by enhancing intrinsic interrelationships of the ICs	To identify both independent components and flexible connections between two modalities. It has been applied widely in fusion of imaging and genetic data	Yes	2	Features
CC-ICA	multitask fMRI (can also be applied to multimodal fusion)	Sui et al. (2009), Kim et al. (2010)	All modalities share a same subject-mixing matrix and a group difference criterion is incorporated into the traditional ICA cost function to be maximized. CC-ICA is optimized for detecting group differences	To improve the sensitivity of the components extraction to group differences as well as the decomposition accuracy	Yes	2 is preferred, 3+ is possible	Features
Other models	fMRI–EEG, fMRI–MEG, MRT–DTI	Eichele et al. (2005), Astolfi et al. (2004), Kubicki et al. (2003), Michael et al. (2011), Michael et al. (2010)

*Here the “feature” is defined as a distilled dataset representing an interesting part of each modality and it contributes as an input vector for each modality and each subject. The raw high dimensional data is preprocessed to generate a second-level output (that is “feature”), which can be a contrast map calculated from task-related fMRI by the general linear model, a component image from a first-level ICA, an FA map from DTI data, or channels from raw EEG signals.

In addition, for most of the above mentioned models, their inputs are from outputs of the general linear regression model. Since the statistical testing being performed is on the loading parameters within a regression framework, the statistical issues are similar to any regression analysis. Since small sample sizes may not generate enough statistical power, one major means to improve our results is to analyze data from more subjects, perhaps via multisite analyses. For example, in Sui et al. (2011), at least 48 SZ and 37 BP are needed to detect differences from HCs at a power of 0.8. More generally, when the effect size is 0.5, at least 64 subjects are needed per group (N = 2) to reach a statistical power of 0.8 in detecting group differences (Kenny, 1987).

Note that there are several other multivariate methods that have been successfully applied to brain imaging data fusion in other mental illnesses, such as Alzheimer’s disease, but have not yet been applied to schizophrenia. These methods include but not limited to partial least squares (Chen et al., 2009), linked ICA (Groves et al., 2011), and dynamic Bayesian networks (Plis et al., 2010). Hence, a future direction may lie in utilizing such approaches to explore schizophrenia.

In conclusion, the use of data fusion is a powerful technique. Selecting which fusion model to use should be done carefully but can be somewhat daunting, as there are many approaches available already. We have attempted to characterize and compare some of the available models in this paper. The use of data fusion approaches can help better elucidate the relationship among multiple modalities and facilitate new discoveries in brain disorders. The most promising avenues for future schizophrenia study may rest on developing better models that can complement and exploit the richness of the various data sets (Friston, 2009) and may enable a broader neuroscience perspective to be applied to neuroimaging so that key questions can be addressed in a theoretically grounded fashion.

Finally, certain caveats must also be considered in reviewing the evidence summarized in this paper. Schizophrenia is a clinically defined syndrome currently lacking specific biologic markers (biomarkers), though some robust deficits have been frequently identified in specific modalities, e.g., the DLPFC and superior temporal gyrus regions in MRI (Calhoun et al., 2004; Kim et al., 2009; Xu et al., 2009; Correa et al., 2010b), the ATR in WM tracts (Kubicki et al., 2005a; McIntosh et al., 2008a; Sui et al., 2011), and genetic differences in CACNA1C, DISC1, etc. (Liu et al., 2009; Williams et al., 2009). Thus, it is certainly possible that “schizophrenia” represents an agglomeration of biologically distinct diseases gathered by clinicians into a clinically convenient but biologically unrelated category of convenience, analogous to “dropsy.” To that extent this is true, no analyses will likely detect a uniform set of biomarkers to delineate a single disorder, because there are multiple disorders. The second caveat is that the overwhelming majority of schizophrenia subjects studied in the studies summarized are treated with powerful psychotropic medications that potentially alter both brain structure and function, so that commonalities detected in these imaging studies may in part result from such treatments for the disorder rather than being primary; this issue can only be resolved by studying first-episode drug-naïve subjects.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.