Automated versus manual segmentation of brain region volumes in former football players.

Objectives: To determine whether or not automated FreeSurfer segmentation of brain regions considered important in repetitive head trauma can be analyzed accurately without manual correction. Materials and methods: 3 T MR neuroimaging was performed with automated FreeSurfer segmentation and manual correction of 11 brain regions in former National Football League (NFL) players with neurobehavioral symptoms and in control subjects. Automated segmentation and manually-corrected volumes were compared using an intraclass correlation coefficient (ICC). Linear mixed effects regression models were also used to estimate between-group mean volume comparisons and to correlate former NFL player brain volumes with neurobehavioral factors.
Results: Eighty-six former NFL players (55.2 ± 8.0 years) and 22 control subjects (57.0 ± 6.6 years) were evaluated. ICC was highly correlated between automated and manually-corrected corpus callosum volumes (0.911), lateral ventricular volumes (right 0.980, left 0.967), and amygdala-hippocampal complex volumes (right 0.713, left 0.731), but less correlated when amygdalae (right -0.170, left -0.090) and hippocampi (right 0.539, left 0.637) volumes were separately delineated and also less correlated for cingulate gyri volumes (right 0.639, left 0.351). Statistically significant differences between former NFL player and controls were identified in 8 of 11 regions with manual correction but in only 4 of 11 regions without such correction. Within NFL players, manually corrected brain volumes were significantly associated with 3 neurobehavioral factors, but a different set of 3 brain regions and neurobehavioral factor correlations was observed for brain region volumes segmented without manual correction. Conclusions: Automated FreeSurfer segmentation of the corpus callosum, lateral ventricles, and amygdala-hippocampus complex may be appropriate for analysis without manual correction. However, FreeSurfer segmentation of the amygdala, hippocampus, and cingulate gyrus need further manual correction prior to performing group comparisons and correlations with neurobehavioral measures.

Introduction

Analysis of brain regional volumes has yielded insight into the pathology and pathophysiology of a variety of neurological and psychiatric diseases including Alzheimer's disease (see reviews by Kantarci and Jack, 2003 and Busatto et al., 2008), schizophrenia (see metaanalysis by Olabi et al., 2011 and reviews by Hulshoff Pol and Kahn, 2008 and Shenton et al., 2010), post-traumatic stress disorder (see reviews by Ahmed-Leitao et al., 2016 and Milani et al., 2017), mild traumatic brain injury (see reviews by Shenton et al., 2012 and Mu et al., 2017) and repetitive head trauma (see reviews by Ng et al., 2014 and Koerte et al., 2015), to name just a few. Accurate and precise volumetric measurements are essential for both reliability and reproducibility. Given the time-consuming nature of manual segmentation, automated segmentation techniques are critical for studies involving large imaging datasets. Moreover, to be useful in the clinical setting, automated segmentation techniques are also critical given that time-consuming manual segmentation by a radiologist for interpretation is not feasible. However, in addition to segmenting the brain in a short period of time, automated segmentation must also provide levels of accuracy and precision that yield results similar to those obtained with manual segmentation, which is currently the gold standard.

Although some automated segmentation algorithms have shown potentially promising results (see review by Dill et al., 2015), many often provide suboptimal results (e.g. de Flores et al., 2015; González-Villà et al., 2016; Grimm et al., 2015; Haller et al., 2016; Næss-Schmidt et al., 2016; Schoemaker et al., 2016) and there is thus ongoing research to develop better algorithms (Akhondi-Asl et al., 2011; Inglese et al., 2015; Mendrik et al., 2015).

Neuroimaging volumetry studies routinely utilize freely-available automated segmentation tools such as FreeSurfer (http://surfer.nmr.mgh.harvard.edu; Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA, USA). Some studies of FreeSurfer have shown deficiencies in automated segmentation of the cerebral cortex (Makris et al., 2008), hippocampus (Cherbuin et al., 2009; de Flores et al., 2015; Grimm et al., 2015; Morey et al., 2009; Wenger et al., 2014), and amygdala (Grimm et al., 2015; Morey et al., 2009; Schoemaker et al., 2016), but data regarding the accuracy and precision of FreeSurfer is not readily available for other important and frequently studied regions including the cingulate gyrus, corpus callosum, and lateral ventricles, all areas important in the investigation of repetitive head trauma. Moreover, there are no published data that demonstrate whether study outcome measures are concordant or discordant when using automated segmentation as compared to manual segmentation.

Volumetric analysis of the brain is particularly important in individuals with exposure to repetitive head trauma as there is evidence that repetitive head impacts may result in regional brain atrophy (Bernick et al., 2015; Goddeyne et al., 2015; Laurent et al., 2010; McKee et al., 2009). Players of American football have a particularly high exposure to repetitive head impacts. For example, college American football players sustain a median of 420 head impacts per season and some players sustain over 2400 head impacts per season, as measured by accelerometers (Crisco et al., 2011).

The aim of this study was to determine whether or not FreeSurfer automated segmentation can be used reliably, without the need for manual brain volume editing, in studies of repetitive head impact that investigated the volumes of the cingulate cortex (left and right), corpus callosum, amygdala (left and right), hippocampus (left and right), amygdala-hippocampal complex (left and right), and lateral ventricles (left and right) in retired National Football Players (NFL) and same aged controls without history of contact sports or brain injury.

Methods

This study utilized data from the Diagnosing and Evaluating Traumatic Encephalopathy using Clinical Tests (DETECT) study, funded by the National Institutes of Health (NIH). The DETECT study details have been described in prior publications (Alosco et al., 2016, Alosco et al., 2017; Stamm et al., 2015; Stern et al., 2016). All study procedures were approved by the Boston University Medical Center Institutional Review Board and all neuroimaging procedures were approved by the Partners Institutional Review Board. All subjects provided written, informed consent.

Participants and procedure

There were two cohorts in the DETECT study: former NFL players with at least 12 years of organized football experience, at least 2 years of active participation in the NFL, and self-reported declines in cognition, mood, and behavior within 6 months of study commencement; and control subjects with no reported history of participation in organized contact sports or traumatic brain injury. All subjects were male, aged 40 to 69 years, spoke English as their first language, had no contraindication to MR imaging or lumbar puncture, and no history or diagnosis of central nervous system (CNS) disease.

Of the 96 enrolled former NFL player subjects, 10 were excluded due to inadequate or absent neuroimaging data, resulting in a final sample size of 86 former NFL players (age: 55.2 ± 8.0 years). Of these 86 subjects, complete neurobehavioral testing results were available for a total of 76 subjects. Neuroimaging data was available for all 28 control group subjects, 3 of whom were excluded due to image quality and 3 more were excluded due to subsequently identified CNS disease, contact sport participation, or history of mild traumatic brain injury, resulting in a final sample size of 22 control subjects (age: 57.0 ± 6.6 years).

All subjects were evaluated according to the DETECT neurobehavioral and neuroimaging protocol, including neuroimaging, structured psychiatric interview, and neuropsychological testing.

MRI data acquisition

DETECT neuroimaging was performed at Brigham and Women's Hospital on a 3-Tesla MRI system (Verio, Siemens Healthcare, Erlangen, Germany) with a 32-channel head array and the Syngo MR-B17 software suite. Only the T1-weighted magnetization prepared rapid gradient echo (TR = 1800 ms, TI = 1100 ms, TE = 3.36 ms, voxel size = 1 × 1 × 1 mm, acquisition matrix = 256 × 256, flip angle = 7°) sequence was used for this study.

Image processing

All T1-weighted images were visually inspected for quality. Brain masks of each subject were generated by FreeSurfer 5.3 (http://surfer.nmr.mgh.harvard.edu; Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA, USA) and corrected manually. Each brain was segmented using T1-weighted images and FreeSurfer 5.3. This process yielded label maps of deep gray matter, white matter, and CSF structures (including the hippocampus, amygdala, corpus callosum, and lateral ventricles). This process also yielded parcellation label maps of the cerebral cortex (including the cingulate gyrus) based on gyral and sulcal structures. The FreeSurfer option for utilizing T2 or FLAIR image contrast to improve pial surface estimations along CSF borders was not used for this study. Estimated total intracranial volumes were also calculated using the automated FreeSurfer method (Buckner et al., 2004).

FreeSurfer segmentation and parcellation maps were then loaded into the Editor module of Slicer 4.5.0 (http://www.slicer.org, Surgical Planning Laboratory, Brigham and Women's Hospital, Boston, Massachusetts, USA) (Fedorov et al., 2012) and overlayed on the aligned T1-weighted images with image interpolation turned off.

Following written directions based on the below-described approaches for each region, two trained raters manually corrected the image label maps of the amygdala, hippocampus, corpus callosum, and cingulate gyrus. One rater corrected approximately two-thirds of the cases while the second rater corrected the remainder. All cases were then reviewed by a single neuroanatomist for accuracy. A third trained rater manually corrected the image label maps of the lateral ventricles and a radiologist reviewed the corrected lateral ventricle label maps for accuracy. All investigators and raters were blind to group membership at the time of segmentation and review by a trained expert.

To evaluate inter-observer reliability, a fourth trained rater, following the same written directions, corrected the FreeSurfer image label maps of the amygdala, hippocampus, corpus callosum, and cingulate gyrus in 10 randomly chosen subjects, while a radiologist corrected the FreeSurfer image label maps of the lateral ventricles in 10 randomly chosen subjects. A neuroanatomist reviewed the label maps corrected by the trained raters. These individuals were blind to the previously corrected label maps and to group membership at the time of segmentation.

Amygdala and hippocampus

Amygdala and hippocampus volumes were manually corrected based on an approach described by Gurvits et al., 1996. Label maps were corrected on coronal slices, from anterior to posterior, using sagittal slices for verification. Close attention was given to the anterior portion of the amygdala at the height of the frontotemporal junction as these parts were variably included by FreeSurfer. The posterior boundary of the amygdala was defined as the last coronal slice before the appearance of the mammillary bodies. The anterior border of the hippocampus was defined as the image slice where the mammillary bodies first appeared. The posterior boundary of the hippocampus was defined as the coronal slice where the crus of the fornix was last seen. The amygdala and hippocampus were defined and strictly separate volume entities and therefore volume overlap between the structures was not allowed.

Although the hippocampus and amygdala are typically evaluated independently, the posterior aspect of the amygdala partially overlaps anatomically with the hippocampal head (Kiernan, 2012) and the boundary between the structures is generally not visible below the level of the temporal horn (Chera et al., 2009). As such, the amygdala-hippocampal complex was evaluated as an additional variable by adding the volumes of the two regions together for the purpose of analyzing hypothesized FreeSurfer imprecision in discriminating the boundary of these regions.

Cingulate gyrus

The cingulate gyrus was defined as the gyrus superior to the corpus callosum, identified primarily on sagittal images (Wible et al., 1997). The medial aspect of the cingulate gyrus was confirmed on sagittal images by identifying the callosomarginal fissure on a paramidsagittal slice. Then, working from medial to lateral on sagittal images, and using coronal images for verification, voxels extending to the corpus callosum, paracingulate gyrus, or beyond the rostrum of the corpus callosum into Brodmann's Area 25 were excluded.

Corpus callosum

Given high contrast between the high T1 signal of the corpus callosum relative to surrounding tissues, most borders of the corpus callosum were obvious. As drawn by FreeSurfer, the corpus callosum label maps were confined to the 5 most midline sagittal slices.

Lateral ventricles

Given high contrast between the low T1 signal of the ventricles and surrounding brain, most borders of the lateral ventricles were obvious. Working superiorly to inferiorly on the axial images, and using the coronal and sagittal images for confirmation, separate volumes were drawn for the right and left lateral ventricles, respectively. When the septum pellucidum was not clearly visible, a midline strip of 1 voxel in width was not assigned to either ventricle to ensure separation of the volumes. The volumes were terminated at the foramina of Monro and choroid plexus was excluded. Any voxels identified by FreeSurfer as lateral ventricle but extending into the foramina of Monro, third ventricle, or ambient cisterns were removed.

Neurobehavioral measures

As part of the DETECT protocol, data from the following neurobehavioral tests and self-report measures were converted to age-, gender-, and education-standardized scores and grouped into four factor scores, based on principal component analyses (Alosco et al., 2016). Mood and Behavior included the following tests: Apathy Evaluation Scale, Beck Depression Inventory II, Beck Hopelessness Scale, Barratt Impulsivity Scale, Behavior Rating Inventory of Executive Functioning - Adult Version, Center for Epidemiologic Studies - Depression Scale, Hamilton Depression Rating Scale, and the Brown-Goodwin Lifetime History of Aggression. Attention and Psychomotor Speed included: Controlled Oral Word Association Test, Delis-Kaplan Executive Function System Color Word Interference Test (inhibition/switching), Trail Making Test, and the Wechsler Adult Intelligence Scale – Revised Digit Symbol test. Verbal Memory included: Neuropsychological Assessment Battery (NAB) Story Learning (phrase unit immediate and delayed recall), and NAB List Learning (short and long delayed recall). Visual Memory included: Boston Qualitative Scoring System for the Rey-Osterrieth Complex Figure (immediate recall presence and accuracy and delayed recall presence and accuracy).

Statistical analyses

Statistical Analysis System (SAS) software (SAS version 9.4; SAS Institute Inc., North Carolina, USA) was used for all statistical analyses. Significance was set at a p-value below 0.05.

Part 1: inter-observer reliability of manually-corrected volumes

Intraclass correlation coefficients (ICC) were calculated to compare the two sets of manually-edited segmentation label map volumes for each region. Levels of correlation as classically described by Cicchetti (1994): low with a correlation coefficient below 0.40, fair from 0.40 to 0.59, good from 0.60 to 0.74, and excellent from 0.75 to 1.00.

Part 2: intraclass correlation of automated and manually-corrected segmentation volumes

Intraclass correlations (ICC) were calculated to compare the automated and manually-corrected segmentation label map volumes for each region. Scatter plots revealed a single outlier subject and that subject was excluded from correlation analysis of the lateral ventricles.

Part 3: brain volume differences between former NFL players and controls using automated segmentation with versus without manual correction

Linear mixed effects regression models were used to estimate between-group volume comparisons (Former NFL Players vs Controls) for the amygdalae, hippocampi, corpora callosa, cingulate gyri, and lateral ventricles. Two regression models were used: 1) a regression model with multivariate outcomes being the amygdalae, hippocampi, and cingulate gyri; and 2) a regression model with multivariate outcomes being the corpus callosum, amygdala-hippocampal complex, and lateral ventricles. All models were controlled for age, body-mass index (BMI), and estimated total intracranial volume (eTIV).

Part 4: association of former NFL player brain volumes with neurobehavioral factors using automated segmentation with versus without manual correction

A linear mixed-effects regression model was performed for all brain regions, excluding the lateral ventricles, to compare segmented brain volumes with grouped neurobehavioral factors. A false discovery rate adjustment was then calculated to account for multiple comparisons. A separate linear regression model was performed for the lateral ventricles due to the typical inverse relationship of lateral ventricle size with brain volumes where ventricle size gets larger as brain parenchyma gets smaller with atrophy (Barron et al., 1976). The neurobehavioral factor models were controlled for age, BMI, eTIV, and years of education.

Results

Part 1: inter-observer reliability of manually-corrected volumes

There was excellent inter-observer correlation for the corpus callosum (ICC = 0.956), left hippocampus (ICC = 0.764), left and right amygdala-hippocampal complex (ICC = 0.863, 0.919), left and right lateral ventricle (ICC = 0.990, 0.989), right amygdala (ICC = 0.887), and right cingulate cortex (ICC = 0.856). Good inter-observer correlation was present in all other regions except for the left cingulate cortex (ICC = 0.574) (Table 1), however there was an outlier identified with left cingulate cortex volumes measuring over 2 standard deviations from the mean and if that outlier was removed then the ICC improved to 0.723. Fair or poor inter-observer correlation was not present in any region.

Table 1

Intra-class correlation values of inter-observer reliability for manually-corrected volumes.

aA single outlier in the left cingulate cortex skews the results. Excluding the outlier results in an ICC of 0.724 (0.230–0.919), which is comparable to the ICC for the right cingulate gyrus.

CI = confidence interval.

Dark gray background = excellent correlation.

Light gray background = good correlation.

Levels of correlation as classically described by Cicchetti (1994): low with a correlation coefficient below 0.40, fair from 0.40 to 0.59, good from 0.60 to 0.74, and excellent from 0.75 to 1.00.

Part 2: intraclass correlation of automated and manually-corrected segmentation volumes

The correlation coefficients were high between automated and manually-corrected segmentation volumes for corpus callosum (ICC = 0.911) and right and left lateral ventricles (ICC = 0.977, 0.964). Good correlation was demonstrated with the left and right hippocampus (ICC = 0.637, 0.539), left and right amygdala-hippocampal complex (ICC = 0.731, 0.713) and right cingulate cortex (ICC = 0.639). Lower correlation was demonstrated for the left and right amygdala (ICC = −0.090, −0.170) and for the left cingulate cortex (ICC = 0.351) (Table 2). The correlation of automated and manually-corrected volumes was better for the amygdala-hippocampal complex than for either the amygdala or hippocampus alone, indicating that FreeSurfer better estimates the volume of the amygdala-hippocampal complex than either structure independently.

Table 2

Intra-class correlation values of manually-corrected versus automatically-generated volumes.

Dark gray background = excellent correlation ICC full cohort.

Light gray background = good correlation ICC full cohort.

Levels of correlation as classically described by Cicchetti (1994): low with a correlation coefficient below 0.40, fair from 0.40 to 0.59, good from 0.60 to 0.74, and excellent from 0.75 to 1.00.

Part 3: brain volume differences between former NFL players and controls using automated segmentation with versus without manual correction

There was generally a larger difference between the mean former NFL player and mean control subject brain region volumes when using manually-corrected volumes than when using uncorrected automated segmentation volumes (Table 3), indicating that there is greater variance in the manually-corrected brain volumes, a finding that is addressed in the discussion. There were also a larger number of statistically significant brain region volume differences between Former NFL Players and Controls when using manually-corrected volumes than when using uncorrected automated segmentation volumes (Table 3), indicating that comparisons performed without manual correction of brain volumes would not have identified all significant group differences in this study.

Table 3

Brain volume comparisons of former NFL players and controls using automated segmentation with versus without manual correction.

*p-Value adjusted for multiple comparisons except for corpus callosum and lateral ventricles as those correlations were calculated in separate regression models.

Dark gray background = significant at p < 0.05.

Part 4: association of former NFL player brain volumes with neurobehavioral factors using automated segmentation with versus without manual correction

No unedited or manually-corrected volumes were associated with the Mood and Behavior factor score. The manually-edited volumes of the cingulate cortex on the left and right were both associated with the Attention and Psychomotor Speed factor score (left: effect size = 31.053, p = 0.003; right: effect size = 25.730, p = 0.003), but no unedited volumes were associated with this neurobehavioral factor. No unedited or manually-corrected volumes were associated with the Verbal Memory factor score. The manually-corrected volume of the left ventricle was associated with the Visual Memory factor score (effect size = 3.870, p = 0.047), however this statistic would not likely hold under multiple comparisons, while the unedited volumes of the left amygdala, left hippocampus, and left amygdala-hippocampal complex were also associated with the Visual Memory score (amygdala: effect size = 0.798, p = 0.036; hippocampus: effect size = 2.617, p = 0.036; amygdala-hippocampus complex: effect size = 4.147, p = 0.014). There was thus no concordance between which manually-corrected volumes and which unedited volumes reached significance when correlating volumes with neurobehavioral factors (Table 4).

Table 4

Association of former NFL Player brain volumes with neurobehavioral factors using automated segmentation with versus without manual correction.

All p-values adjusted for multiple comparisons except for corpus callosum and lateral ventricles as those correlations were calculated in separate regression models.

Dark gray background = significant at p < 0.05.

Note: adjustments for multiple comparisons may lead to slightly different p-values in each study depending on the number of comparisons and the method of adjustment.

Discussion

The aim of this study was to determine whether or not FreeSurfer automated segmentation can be used accurately and reliably, without the need for manual brain volume editing, in studies of repetitive head impact that examine the cingulate cortex, corpus callosum, amygdala, hippocampus, and/or lateral ventricles.

Corpus callosum and lateral ventricles

In this study, automated segmentation volumes of the lateral ventricles and corpus callosum demonstrated excellent correlation with manually-corrected volumes. In addition, group comparison statistical inferences (i.e., whether or not volume differences between former NFL players and controls reached statistical significance) were the same whether using automated segmentation or manually-corrected lateral ventricle volumes and corpus callosum volumes. Similarly, statistical inferences when correlating former NFL player volumes with neurobehavioral scores were the same whether using automated segmentation or manually-corrected lateral ventricle volumes and corpus callosum volumes.

The between-group mean volume differences measured with and without manual correction are very similar for the corpus callosum. For the lateral ventricles, there is a slightly larger difference in the mean volumes when measured with and without manual correction. However, the ranges of values are quite large and the difference remains statistically insignificant. Moreover, the magnitude of the mean volume differences between former NFL players and controls is driven in part by the mixed effects model's correlation between the hemispheres and the model's incorporation of and control for the confounders of intracranial volume and age. For example, the absolute raw difference in size of the left lateral ventricle between former NFL players and controls is 1663 mL with manually corrected volumes and 736 mL (0.02%) with automatically segmented volumes, compared to 2300 mL and 1139 mL as calculated with the mixed effects model. The trend toward slightly higher differences measured with manual correction is described below in Section 4.3.

Post-mortem pathology studies have identified corpus callosum abnormalities in multiple diseases with neurodegenerative and psychiatric components including in schizophrenia (Bigelow et al., 1983; Rosenthal and Bigelow, 1972), multiple sclerosis (Evangelou et al., 2000), traumatic brain injury (Anderson and Bigler, 1994), and chronic traumatic encephalopathy (McKee et al., 2013; Mez et al., 2017). Neuroimaging studies have identified corpus callosum abnormalities in-vivo in both traumatic brain injury (see reviews by Shenton et al., 2012 and Mu et al., 2017) and in repetitive head impacts (see reviews by Ng et al., 2014 and Koerte et al., 2015), both of which are sustained by American football players. Similarly, abnormal enlargement of the lateral ventricles has been described in chronic traumatic encephalopathy (McKee et al., 2013; Mez et al., 2017) and in other diseases that have neurodegenerative features including multiple sclerosis (e.g., Turner et al., 2003), schizophrenia (e.g., Kempton et al., 2010), Alzheimer's disease (e.g., Nestor et al., 2008), and alcoholism (e.g., Fox et al., 1976). Both corpus callosum volume and lateral ventricle volume could therefore potentially serve as in-vivo biomarkers for neurodegenerative processes.

Given the excellent correlation of automated corpus callosum and lateral ventricle segmentation volumes generated by FreeSurfer with manually-corrected volumes, and the concordance of results when using either automated or manually-corrected volumes in this study, the corpus callosum and lateral ventricles can probably be reliably segmented by FreeSurfer without the need for manual editing, at least in large-scale studies.

Amygdala, hippocampus, and cingulate gyrus

In this study, automated segmentation volumes of the amygdala, hippocampus, and cingulate gyrus demonstrated poorer correlation with manually-corrected volumes than was the case for the corpus callosum and the lateral ventricles. Furthermore, statistical inferences differed when comparing volume differences between former NFL players and controls and also when evaluating former NFL player volume correlations with neurobehavioral scores, depending on whether automated segmentation or manually-corrected amygdala, hippocampus, and cingulate gyrus volumes were used. These findings are corroborated by markedly discrepant mean volume differences and estimated neurobehavioral score effect sizes in these brain regions, particularly in the cingulate cortices, when using automated segmentation as opposed to manually-corrected volumes.

The correlations of automated segmentation and manually-corrected volumes of the amygdala and hippocampus were however, somewhat improved when combined into a single amygdala-hippocampal complex volume. Moreover, the automated segmentation and manually-corrected amygdala-hippocampal complex volumes yielded the same statistical study results when comparing the volumes of former NFL players and control subjects and yielded 7 concordant results out of 8 when evaluating the relationships of former NFL player volumes with neurobehavioral scores. Similarly, the discrepancies between the mean volume differences and the estimated neurobehavioral score effect sizes are smaller than those seen when evaluating the amygdala and hippocampus as separate structures.

The amygdala, hippocampus, and cingulate gyrus are of great interest in neuroimaging studies of repetitive head impacts (see reviews by Ng et al., 2014 and Koerte et al., 2015), and mild traumatic brain injury (see reviews by Shenton et al., 2012 and Mu et al., 2017). In chronic traumatic encephalopathy, post-mortem studies have identified deposition of hyperphosphorylated tau in the limbic system, particularly in the amygdala and hippocampus but also to a lesser degree in the cingulate gyrus (McKee et al., 2013; Mez et al., 2017; Omalu et al., 2011). As with the corpus callosum and the lateral ventricle volumes, the volumes of the amygdala, hippocampus, and cingulate gyrus could thus potentially serve as in-vivo biomarkers for diseases with neurodegenerative features.

Mirroring the results of several other studies that demonstrated correlation coefficients often substantially <0.8, this study demonstrates suboptimal correlation of automated segmentation volumes with manually-corrected volumes of the amygdala and hippocampus (Cherbuin et al., 2009; de Flores et al., 2015; Grimm et al., 2015; Morey et al., 2009; Schoemaker et al., 2016; Wenger et al., 2014). This study also demonstrates that, at least in this cohort of former professional football players with history of exposure to repetitive head impacts, failing to manually edit the amygdala and hippocampus volumes generated by FreeSurfer led to very different brain volume group comparison and neurobehavioral correlation study results. Automated segmentation volumes of the amygdala, hippocampus, and cingulate gyrus are thus not adequate for meaningful study on their own without further manual editing.

Results are substantially improved, however, when the amygdala and hippocampus volumes are combined into an amygdala-hippocampal complex. The combined amygdala-hippocampal complex can thus possibly be segmented by FreeSurfer without the need for manual editing for large-scale studies, provided appropriate quality control is performed.

Technical considerations

There was greater variance in the sizes of the studied brain structures when the volumes were manually-edited, suggesting that the method of segmentation employed by FreeSurfer, which uses an atlas based approach on a training set to label the likely structural location of each voxel (Fischl et al., 2004), may artificially improve precision. This artificial precision may be related to the inherently normalizing process of stretching a fixed image atlas to brains that differ from the atlas in unique and variable ways that are better captured in the manual correction process. The results of a prior study, which showed that FreeSurfer relatively overestimates the size of smaller hippocampi but not larger hippocampi (Wenger et al., 2014), supports this hypothesis. Although artificial precision could be expected to spuriously increase statistical power, this study demonstrates more statistically significant results with manually-edited volumes, a finding presumably due to better accuracy achieved through the manual editing process because the manual editing process allows for careful evaluation of deep brain structure with low-contrast borders that may not be within the resolving power of FreeSurfer. In addition, as previously described by Wenger et al., 2014, manual segmentation and manual editing typically follow rules that define the often unclear borders between amygdala and the hippocampus and between the tail of the hippocampus and the lateral ventricle whereas FreeSurfer tends to be more inclusive in these areas (Fig. 1). Although Wenger et al. used a set of rules described by Pruessner et al. (2000) and this study used a set of rules described by Gurvits et al. (1996), this study replicates the qualitative results described by Wenger et al.

Fig. 1

(A) Axial 2D, (B) Coronal 2D, and (C) Sagittal 2D T1-weighted images of the brain. The manually-edited label map surfaces of the right and left amygdala are dark blue while the manually-edited label map surfaces of the right and left hippocampus are light blue. The outline of the label maps as drawn by FreeSurfer without manual correction are superimposed as orange outlines. The cingulate gyrus label map is presented in yellow. (D) Obliqued 3D reconstruction of the FreeSurfer generated label maps (orange) and manually-edited label maps (blue) created from a randomly selected DETECT study subject using the Model Maker module of Slicer 4.5.0. Note the manually-edited volume of the amygdala is substantially larger than the unedited volume as segmented by FreeSurfer. Also note that the unedited FreeSurfer volume of the hippocampus extends more posterior than the manually-edited volume. The label map was not modified for the 2D image creation but was smoothed for the 3D reconstruction using a sinc filter with a smoothing factor of 10. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Limitations

Just as there are significant differences between the results obtained with FreeSurfer 5 and prior versions of FreeSurfer (Gronenschild et al., 2012), the results of this paper may therefore not apply to other versions of FreeSurfer. Additionally, this study evaluated differences between measured volumes, but did not account for volume overlap or shape and the results may not be translatable to volume overlap or shape (Morey et al., 2009). The DETECT study also included only male subjects and the results of this study may therefore not be generalizable to female subjects. The DETECT study subjects were former professional football players exposed to repetitive head impacts and the conclusions drawn regarding the effect of using automated segmentation versus manually-corrected volumes on group comparison and neurobehavioral study results may not be translatable to studies of subjects with different pathologies. However, the conclusions drawn regarding automated segmentation and manually-corrected volume correlations are likely generalizable. Although the data set for this study appears too small to benefit from the multiple imputation statistical analysis method, which can reduce the number of label maps that require manual correction (Chua et al., 2015), studies with larger data sets may benefit from that approach. The image processing pipeline for this study did not utilize the -T2pial or -FLAIRpial options to optimize pial surface estimations. However, these options are typically useful when there is dura within the brain mask that is not adequately removed with skull stripping. In the utilized pipeline, each brain mask was reviewed and manually edited if necessary, thus likely yielding more precise results than the available automated FreeSurfer options.

Conclusions

Automated segmentation using FreeSurfer 5.3 yields excellent correlation with manually-edited volumes of the corpus callosum and lateral ventricles but suboptimal correlation for amygdala, hippocampus, and cingulate gyrus. In addition, using automated segmentation volumes leads to substantially different study results than using manually-corrected volumes when correlating brain volumes with neurobehavioral test scores in this cohort of former professional football players. Study result concordance is improved when the amygdala and hippocampus volumes are combined into an amygdala-hippocampal complex. Automated FreeSurfer-derived segmentation volumes of the corpus callosum and lateral ventricles, and amygdala-hippocampus complex may therefore be suitable for analysis without manual correction, provided appropriate quality control is performed. However, automated FreeSurfer-derived segmentation volumes of the amygdala, hippocampus, and cingulate gyrus should not be utilized for analysis without manual correction until further refinements are made to the FreeSurfer algorithm and appropriately tested.

Author disclosure statement

Dr. Robert Stern is a paid consultant to Avanir Pharmaceuticals, Inc. (Aliso Viejo, CA), Biogen (Cambridge, MA), and Eli Lilly (Indianapolis, IN). He receives royalties for published neuropsychological tests from Psychological Assessment Resources, Inc. (Lutz, FL, USA). He is on the Board of Directors of King-Devick Technologies, Inc. (Chicago, IL), and is a member of the NFL Players Association Mackey-White Health and Safety Committee. For all other authors, no competing financial interests exist.