High resolution automated labeling of the hippocampus and amygdala using a 3D convolutional neural network trained on whole brain 700 μm isotropic 7T MP2RAGE MRI.

Image labeling using convolutional neural networks (CNNs) are a template-free alternative to traditional morphometric techniques. We trained a 3D deep CNN to label the hippocampus and amygdala on whole brain 700 μm isotropic 3D MP2RAGE MRI acquired at 7T. Manual labels of the hippocampus and amygdala were used to (i) train the predictive model and (ii) evaluate performance of the model when applied to new scans. Healthy controls and individuals with epilepsy were included in our analyses. Twenty-one healthy controls and sixteen individuals with epilepsy were included in the study. We utilized the recently developed DeepMedic software to train a CNN to label the hippocampus and amygdala based on manual labels. Performance was evaluated by measuring the dice similarity coefficient (DSC) between CNN-based and manual labels. A leave-one-out cross validation scheme was used. CNN-based and manual volume estimates were compared for the left and right hippocampus and amygdala in healthy controls and epilepsy cases. The CNN-based technique successfully labeled the hippocampus and amygdala in all cases. Mean DSC = 0.88 ± 0.03 for the hippocampus and 0.8 ± 0.06 for the amygdala. CNN-based labeling was independent of epilepsy diagnosis in our sample (p = .91). CNN-based volume estimates were highly correlated with manual volume estimates in epilepsy cases and controls. CNNs can label the hippocampus and amygdala on native sub-mm resolution MP2RAGE 7T MRI. Our findings suggest deep learning techniques can advance development of morphometric analysis techniques for high field strength, high spatial resolution brain MRI.

INTRODUCTION

Improvements in spatial resolution and contrast in structural MRI acquired at 7T or higher field strength holds promise for detecting disease‐related neuroanatomical abnormalities that may not be visible on clinical imaging obtained at 3T or less. Evidence for the clinical benefit of high field neuroimaging typically relies on radiological assessment of acquired images. A limitation of radiological assessment is that it is subjective and inter‐rater variability is a significant source of error. Computational morphometric image analysis techniques are alternatives to radiological assessment used extensively in lower field strength studies to model neuroanatomical structures and statistically identify tissue changes. Although there are a variety of publicly available computational tools available for analysis of neuroanatomical MRI acquired at lower strengths, these are generally (although not always) developed for analysis of T1‐weighted MRI acquired at 1 mm isotropic resolution. There are few software tools to analyze 7T or higher field strength imaging data acquired with sub‐mm isotropic resolution (Bazin et al., 2014; Seiger et al., 2015; Zaretskaya, Fischl, Reuter, Renvall, & Polimeni, 2018).

In this study, we used a recently developed 3D convolutional neural network (CNN) architecture “DeepMedic” (Kamnitsas et al., 2017), originally developed to label brain lesions, to train a model to label hippocampi and amygdala on whole brain 700 μm isotropic MP2RAGE MRI acquired at 7T in healthy controls and individuals with epilepsy. Successful application of this technique would support the utility of CNN‐based techniques for modeling subcortical neuroanatomical structures at high spatial resolution, and allow for analysis of images with novel contrast properties that differ from standard image acquisitions such as T1‐weighted MPRAGE.

While the increased field strength 7T allows imaging at higher spatial resolution and image contrast by virtue of the increased SNR and T1 values, the MP2RAGE acquisition is also of significant advantage. As discussed by Marques et al. (2010), the multiple GRE blocks in the acquisition allows for correction of proton density variation, T2* effects, and correction of B1 receive and transmit inhomogeneity. This calculated image approach which has comparatively low B1 requirements and provides nearly pure T1 weighting, has seen widespread use at 7T and may also be useful for quantitative 3T imaging in epilepsy (Kotikalapudi et al., 2019). With these properties and the large T1 differences in white, gray matter and CSF at 7T (~1.2, 2, and 4.3 s, [Rooney et al., 2007]), MP2RAGE is therefore an attractive candidate acquisition for tissue segmentation and may be useful for deep learning‐based computational image analyses.

Existing techniques to label neuroanatomical structures on brain MRIs differ from our CNN‐based approach in that CNNs and related “deep learning” techniques do not utilize predefined atlases or templates, and do not require co‐registration of these templates to an individual MRI. Instead, CNNs encode or “learn” imaging features that discriminate between the structure of interest and surrounding tissue. Several recent studies have utilized deep learning‐based approaches to segment hippocampus and other structures, but to the best of our knowledge none have been used to label high resolution 7T MP2RAGE (Chen, Dou, Yu, Qin, & Heng, 2018; Dolz, Desrosiers, & Ben Ayed, 2018; Fedorov et al., 2017; Goubran et al., 2020; Guha Roy, Conjeti, Navab, Wachinger, & Alzheimer's Disease Neuroimaging Initiative, 2019; Henschel et al., 2020; Ito, Nakae, Hata, Okano, & Ishii, 2019; Li, Yu, Gu, Liu, & Li, 2019; Mehta, Majumdar, & Sivaswamy, 2017; Milletari et al., 2017; Nogovitsyn et al., 2019; Sun et al., 2020; Thyreau, Sato, Fukuda, & Taki, 2018).

Our primary aim was to investigate whether CNNs can segment the hippocampus and amygdala. We evaluated the performance of the technique using the DSC metric to summarize the overlap between manual labels and CNN‐derived labels. We further investigated the clinical utility of the technique by applying the trained model to epilepsy patients imaged using the same MP2RAGE acquisition protocol.

The development of computational analysis techniques that can be applied to high resolution imaging is particularly important in epilepsy since one of the most effective potential treatments for individuals who do obtain adequate seizure control on medication is surgical resection of the seizure focus. In many patients the seizure focus is coincident with subtle neuroanatomical changes identified on postsurgical histopathology; however, many of these individuals have radiologically “normal” clinical imaging at 3T or lower. The clinical benefit of 7T imaging relative to 3T or lower field strength imaging to detect epilepsy‐related neuroanatomical changes is an active field of research in epilepsy neuroimaging (Canjels et al., 2020; Colon et al., 2018; De Ciantis et al., 2016; Feldman et al., 2019; Henry et al., 2011; Pittau et al., 2018; Santyr et al., 2017; Shah et al., 2019); we believe that the computational methods described in this study will be useful for addressing this question.

To facilitate further methodological developments by other research groups, we have made imaging data, manual and CNN‐based labels and scripts available at https://sites.google.com/site/hpardoe/hacl (Pardoe et al., 2020). All software used in this study was freely available to the public at the time of publication.

METHODS

Participants

We imaged 21 healthy controls (9 female, mean age 39 ± 15 years) and 16 epilepsy patients (8 female, mean age 36 ± 10 years) using the same acquisition protocol. Epilepsy participants were recruited from the University of Pittsburgh Medical Center Comprehensive Epilepsy Center. The primary inclusion criteria were a diagnosis of medically refractory focal epilepsy, epilepsy surgery candidates and tolerant of 7T MR imaging (nonclaustrophobic and ferromagnetic implant‐free). In the epilepsy participants, clinical imaging was radiologically classified as MRI negative in nine patients; for the MRI positive patients a single case of unilateral hippocampal sclerosis was identified based on clinical MRI, and potential hippocampal sclerosis was observed in another participant. For the remaining five MRI positive cases abnormalities included (i) left anterior temporal focal cortical dysplasia, (ii) right frontal focal cortical dysplasia, (iii) a left anterior lateral temporal lesion, (iv) bilateral nodular heterotopia, and (v) a right posterior fossa arachnoid cyst. The potential hippocampal sclerosis case had laser ablation surgery prior to 7T imaging. We excluded this subject from quantitative analyses, however, we applied our CNN‐based hippocampus & amygdala labeling model to imaging from this participant to qualitatively determine if postsurgical remnant hippocampal tissue could be labeled using our model, since morphometric analysis of postsurgical brain changes is a potential further application of the methods presented in this study. This study was approved by the University of Pittsburgh Human Research Protection Office and conforms to standards defined in the Declaration of Helsinki. All study participants gave their informed consent prior to inclusion in the study.

MRI acquisition

All studies were performed on a Siemens Magnetom whole body 7T 8 channel pTx system with an 8 × 2 transceiver and very high order shim insert. RF shimming and 1st–4th order shimming over the whole brain were performed as previously described (Hetherington, Avdievich, Kuznetsov, & Pan, 2010; Pan, Lo, & Hetherington, 2012). Images were acquired with acquisition parameters: 0.7 mm3 isotropic, TR/TI1/TI2 6s/800/2700, acquisition time 9 min.

Hippocampus and amygdala manual labeling

Hippocampi and amygdala structures were labeled using the ITK‐SNAP software package (http://www.itksnap.org/, [Yushkevich et al., 2006]). Structures were delineated in the coronal plane on consecutive slices in a posterior‐to‐anterior direction.

CNN training and testing

Facial features were removed from MRI scans using the “pydeface” software tool (https://github.com/poldracklab/pydeface). For each subject the 1 mm isotropic MNI152 template provided as part of the FSL software package was linearly coregistered to the target MP2RAGE scan using the FLIRT software tool (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FLIRT, Jenkinson & Smith, 2001). The estimated coregistration matrix was then applied to the MNI152 brain mask to generate an approximate binary mask in the space of each individual's scan. Voxel‐wise CNN sampling and prediction was limited to voxels within this brain mask.

For these analyses, the MP2RAGE acquisition was the single input channel used for model training. Five voxel‐level output classes were predicted, corresponding to left and right hippocampi, left and right amygdala and the image background. The DeepMedic approach utilizes a dual‐pathway approach, with one path designed to identify local discriminating features and an additional pathway that operates on down‐sampled input data to identify contextual features over a larger spatial scale. The default down‐sampling factor of 3 was used for our analyses. A final fully connected Conditional Random Field was used to derive “hard” binary voxel‐level labels from the probabilistic soft segmentations output by the preceding dual pathways. Image processing (CNN training and testing) was performed on a high performance compute cluster using NVIDIA Tesla V100 GPUs. We utilized a leave‐one‐out approach for evaluation of model performance. A model was trained for every subject in the study, with all subjects except the subject of interest used to train the model, and hippocampus and amygdala labels estimated in the subject of interest using this model.

Assessing performance of CNN‐based hippocampal and amygdala labeling

Performance of the CNN was assessed by calculating the DSC between manual and CNN‐based labels in the testing groups. The DSC assesses performance of image labeling tasks. If two labels perfectly overlap, the DSC = 1; if the two labels do not have any common voxels the DSC = 0. A multiple linear regression model was used to determine if (i) diagnosis, (ii) brain structure (hippocampus or amygdala), and (iii) left/right laterality were significantly correlated with the measured DSC; in this model the DSC was the dependent variable and the three variables listed above were explanatory variables.

Volume estimates were calculated from CNN‐based and manual labels by summing the number of labeled voxels for each structure (left and right hippocampus and amygdala) for each subject. The agreement between CNN‐based and manual volume estimates was assessed using a simple linear regression model for each structure, with the CNN‐based estimate as the dependent variable and the manual volume estimate as the explanatory variable. Healthy controls and epilepsy patients were assessed separately. The linear model was constrained to pass through the origin that is, the y‐axis intercept term was set at zero. For this analysis the p‐value for statistical significance was corrected for multiple comparisons using the Bonferroni method; an adjusted threshold 6.25 × 10−3 (0.05/8) was used.

For volumetric analysis of the epilepsy group we conducted both individual and group‐level comparisons. Individual z‐scores for each individual brain structure volume estimate were calculated from fractional structure volume estimates, with each hippocampal and amygdala volume divided by the head volume estimate for all cases. Group level comparisons of hippocampal and amygdala volume changes were assessed using multiple linear regression with hippocampus or amygdala volume as the dependent variable and diagnosis, age, sex, and head size as predictor variables. For these analyses the epilepsy patients were subclassified into cases with temporal lobe involvement based on all available clinical data (N = 6) and those with primarily extratemporal seizures (N = 10).

RESULTS

The trained CNN‐based predictive model successfully labeled all hippocampi and amygdala in healthy controls and epilepsy cases. Figure 1 shows an example of the automated hippocampal and amygdala labeling in a healthy control. The average DSC in our testing cohort = 0.88 ± 0.03 for the hippocampus and 0.8 ± 0.06 for the amygdala (Figure 2). There were some outliers with reduced DSC observed in the analyses; in the healthy control group these corresponded to subjects with reduced image quality presumably due to in‐scanner head motion and resulted in reduced DSCs across left and right hippocampal labels and the left amygdala (see Supporting Information). The outlier in the right hippocampus epilepsy group corresponded to the case with hippocampal sclerosis. The DSC for hippocampal labels was higher than for amygdala labeling (p <2 × 10−16). There was no evidence that DSC depended on diagnosis (p = .91). There was marginal evidence that right sided structures had greater DSC than left sided structures (p = .051). Summary volume statistics are provided in Table 1. The CNN‐based volume estimates were highly correlated with manual volume estimates for both left and right hippocampi and amygdala in epilepsy cases and controls (Figure 3, p <2 × 10−16 for all linear models, slope estimate range = [0.94, 0.99]), supporting the utility of the technique in clinical populations. In the single epilepsy case with clinical MRI indicating right hippocampal sclerosis, the CNN‐based method identified reduced volume in the affected hippocampus (Figure 4). In this case, the z‐score when comparing right hippocampal volume in the individual with the healthy controls = −3.64, which indicates significantly reduced volume. Applying our hippocampus and amygdala labeling model to the epilepsy case who had undergone laser ablation surgery in the left hippocampus shows that our predictive model successfully labeled remnant posterior hippocampus (Figure 5). Group level analyses did not identify any significant hippocampal volume changes in the epilepsy patients (p = .98) however amygdala volumes were increased in the cases with temporal lobe seizures (152 mm3 or 13% increase relative to control amygdala volumes, p = .017).

FIGURE 1

An example of automated CNN‐based labeling of the hippocampus and amygdala on whole brain 700 μm MP2RAGE acquired in a healthy control imaged at 7T. CNN‐based labels are shown on the right column, with manual labels shown in the middle column

FIGURE 2

Overlap between manual and automated CNN‐based labels for hippocampi and amygdala. A DSC = 1 indicates perfect overlap of the manual and automated labels. Overlap between manual and automated labels is greater for the hippocampus than the amygdala, which is likely due to the larger volume of the hippocampus and better definition of neuroanatomical boundaries between the hippocampus and surrounding brain structures compared with the amygdala

TABLE 1

Summary volume estimates for the hippocampus and amygdala derived using a convolutional neural network‐based labeling technique

	Right hippocampus volume (mm3, mean ± SD)	Left hippocampus volume (mm3, mean ± SD)	Right amygdala volume (mm3, mean ± SD)	Left amygdala volume (mm3, mean ± SD)
Healthy control	3,157 ± 294	3,046 ± 300	1,145 ± 217	1,170 ± 170
Epilepsy	2,926 ± 431	2,786 ± 735	1,215 ± 199	1,181 ± 259

FIGURE 3

Comparison of CNN‐based volume estimates (y‐axis) with manual volume estimates (x‐axis). The plots indicate high agreement between CNN‐based volume estimates and manual volume estimates in both epilepsy patients and healthy controls

FIGURE 4

CNN‐based labeling of the hippocampus and amygdala in an individual with hippocampal sclerosis in the right hippocampus (yellow arrows). The right column shows the CNN‐based labels and the middle column shows manual labels for comparison. The figure demonstrates that the CNN‐based technique is successfully able to label the atrophic hippocampus, supporting the clinical utility of the technique for mapping hippocampal changes in temporal lobe epilepsy

FIGURE 5

CNN‐based labeling in an epilepsy patient who has undergone laser ablation surgery. The ablated tissue region is indicated in the left column (yellow arrows). The right column indicates that the CNN‐based labeling technique is able to label remnant hippocampal tissue despite the presence of severe postsurgical neuroanatomical changes in the hippocampus and surrounding brain regions

DISCUSSION

Our study demonstrates that CNN‐based methods may be useful approach for labeling neuroanatomical structures on sub‐mm resolution 7T imaging. The DSC values estimated in our study are consistent with previous work; for example, Morey et al. (2009) compared automated hippocampal and amygdala volume estimates with manual segmentations using Freesurfer and FSL‐FIRST and found DSCs in the range [0.77, 0.82] for the hippocampus and [0.70, 0.75] for the amygdala, which are slightly lower than our observed DSCs of 0.88 for the hippocampus and 0.8 for the amygdala respectively. The reduced DSCs in the amygdala versus the hippocampus are likely due to the lower volume of the amygdala and relatively ambiguous neuroanatomical boundaries in the anterior end of the amygdala. Our analyses did not demonstrate that performance was reduced in epilepsy cases. We note however that our sample size is limited and medically refractory focal epilepsy is a broad category that may include a variety of distinct tissue pathologies that can vary in terms of tissue signal, contrast, and morphology. Application of the techniques described in this study to larger patient groups stratified by pathology or etiological factors will allow for more accurate assessment of CNN performance in the presence of lesional or dysmorphic tissue. Although the successful detection of reduced hippocampal volume in the patient with likely hippocampal sclerosis provides empirical support for the utility of this technique in lesional cases, our approach should be applied to a larger cohort of hippocampal sclerosis cases for a more rigorous evaluation. The inclusion of more cases with specific tissue pathology for CNN training will also likely improve performance. The finding of increased amygdala volume along with normal overall hippocampal volumes in cases with temporal lobe seizures was unanticipated, however, increased amygdala volume in temporal lobe epilepsy has been reported previously (Capizzano et al., 2019; Lv et al., 2014; Reyes et al., 2017).

Because of known problems with transmission homogeneity, a key aspect of structural imaging at 7T is the RF coil performance. In this study, the use of the 8 × 2 transceiver with RF shimming yielded sufficient B1+ homogeneity both experimentally and theoretically, so that additional methods such as the use of high permittivity pads was not necessary (Aussenhofer & Webb, 2013; Vaidya, Deniz, Collins, Sodickson, & Lattanzi, 2018). It should be noted however that it is unknown if our CNN model can be successfully applied to imaging data acquired with alternative hardware & acquisition protocols; this may require additional demonstration. Nonetheless, these datasets were acquired with routine performance, and provide target performance metrics which should be investigated in future studies. Performance is also likely to improve if a larger training dataset was used.

One of the more immediate potential applications of this technique is to segment hippocampal subfields. Arguably the canonical work in this field to date has been carried out by Wisse and colleagues using a 700 μm isotropic T2‐weighted acquisition at 7T (Wisse et al., 2012, 2014). Inspection of images provided in these publications suggests that the boundaries between hippocampal subfields are more radiologically evident than the imaging used in our study. It is possible that imaging that is predominantly T2‐weighted provides contrast that may be better suited to the task of hippocampal subfield segmentation. However it should be noted that with the minimal difference in T2 relaxation values between white and gray matter (Bartha et al., 2002; Michaeli et al., 2002), T1 lengthening that occurs at 7T, segmentation based on clarity of white‐gray matter interfaces is optimally acquired as a function of T1 weighting that is either directly included in the analysis pipeline or controlled for optimal effect (Visser, Zwanenburg, Hoogduin, & Luijten, 2010; Wisse et al., 2012).

In summary, we have demonstrated that CNNs can be trained to label the hippocampus and amygdala using high resolution MP2RAGE imaging acquired at 7T. We have provided preliminary evidence supporting the potential clinical utility of the technique in individuals with focal epilepsy. These findings suggest that CNN‐based methods may be a valuable tool for the next generation of morphometric analysis tools for high spatial resolution high‐field MRI neuroimaging.

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are openly available at https://sites.google.com/site/hpardoe/hacl, doi:10.25790/bml0cm.72.

Figure S1 CNN‐based hippocampal and amygdala labels estimated using scans shown in the top row had DSCs that were outliers (see Figure 2 main manuscript). Image quality is clearly reduced in these scans, presumably due to in‐scanner head motion. Images in the bottom row represent typical image quality scans with nonoutlier DSCs for hippocampal and amygdala labels.

Click here for additional data file.