Pitfalls in brain age analyses.

Over the past decade, there has been an abundance of research on the difference between age and age predicted using brain features, which is commonly referred to as the "brain age gap." Researchers have identified that the brain age gap, as a linear transformation of an out-of-sample residual, is dependent on age. As such, any group differences on the brain age gap could simply be due to group differences on age. To mitigate the brain age gap's dependence on age, it has been proposed that age be regressed out of the brain age gap. If this modified brain age gap is treated as a corrected deviation from age, model accuracy statistics such as R2 will be artificially inflated to the extent that it is highly improbable that an R2 value below .85 will be obtained no matter the true model accuracy. Given the limitations of proposed brain age analyses, further theoretical work is warranted to determine the best way to quantify deviation from normality.

INTRODUCTION

Over the past decade, there has been an explosion of research devoted to estimating individuals' ages using features derived from magnetic resonance images (MRIs) of the brain (Franke & Gaser, 2019). From studies using diffusion‐weighted features to complex functional connectivity metrics, the literature is extensive (Cole, 2020; Erus et al., 2015; Irimia, Torgerson, Goh, & van Horn, 2015; Li, Satterthwaite, & Fan, 2018; Lin et al., 2016). While age is easily measured through more conventional means, assessing the appearance of the brain with respect to the natural patterns of development and aging provides a framework for dimension reduction; from hundreds of thousands to millions of MRI measurements, these models aim to provide the age of the brain for each subject as a convenient summary measure. The predicted age from these models has been coined “brain age,” and the difference between age (sometimes referred to as “chronological age”) and brain age is typically referred to as the “brain age gap” (BAG). Predicted ages are calculated using the following fitted model:where is the predicted age of the ith subject, B is the jth brain feature for the ith subject, and f(⋅) is some function of the brain features.

BAG analysis was developed to address two major challenges in neuroscience and medicine: high‐dimensionality, and individual risk assessment. Neuroimaging data are high dimensional, with the average T1‐weighted scan containing approximately 1,200,000 voxels of brain tissue (Cosgrove, Mazure, & Staley, 2007). Importantly, different parts of the brain follow a variety of trajectories across the lifespan (Coupé, Catheline, Lanuza, & Manjón, 2017; Gennatas et al., 2017; Kennedy et al., 2015). Therefore, in order to better predict age, it is beneficial to use more brain features that complement each other (Varikuti et al., 2018). The main motivation, however, behind BAG analyses has been to develop a single number to represent an individual's deviation from some normal trajectory (de Lange & Cole, 2020). This is an admirable goal, since deviating from a normal trajectory may be indicative of or predictive of debilitating disorders (Marquand, Rezek, Buitelaar, & Beckmann, 2016).

Researchers often test whether members of a group tend to have their age overestimated compared to a control group, striving to assess whether the disorder is associated with the brain aging prematurely or lagging behind. For instance, Chung et al. (2018) asked if those at clinical high risk for psychosis had a larger BAG than healthy controls, and Liem et al. (2017) asked if the BAG differed across groups with varying degrees of objective cognitive impairment. Typically, these models are developed using regression or machine learning in one dataset, and are evaluated in a test set. The cross‐validation process involves dividing the training set into k folds, estimating the model parameters on k − 1 folds, applying the fitted model to the remaining fold, and repeating until every participant in the training set has a predicted age. This procedure helps avoid overfitting and reporting an inflated model accuracy statistic. Finally, the trained model is applied on a separate test set to predict age of each individual based on their brain features.

In this article, we further study the BAG with particular attention to its interpretation and its limitations. The BAG is a linear transformation of an out‐of‐sample residual (subsequently referred to as a “prediction error”). As such, it is dependent on the outcome variable (i.e., age) (Le et al., 2018). Therefore, differences in the BAG between groups may be due to differences in the brain, or due to differences in the age distributions across groups (Le et al., 2018; Smith, Vidaurre, Alfaro‐Almagro, Nichols, & Miller, 2019). A recently proposed solution to this problem involves regressing the BAG on age and taking the residuals from this model as a modified BAG (MBAG) that is orthogonal to age (Beheshti, Nugent, Potvin, & Duchesne, 2019; Chung et al., 2018; Liang, Zhang, & Niu, 2019; Smith et al., 2019). We show that this adjustment is problematic, because if this new prediction error is treated as a deviation from a subject's age, which it is not, metrics of model accuracy will be severely inflated.

KNOWN LIMITATIONS OF THE BRAIN AGE GAP

BAG analyses have historically failed to account for the fact that the difference between age and predicted age varies as a function of age (Beheshti, Maikusa, & Matsuda, 2018; Cole, Underwood, et al., 2017; Franke, Hagemann, Schleussner, & Gaser, 2015; Gaser et al., 2013; Liem et al., 2017; Nenadić, Dietzek, Langbein, Sauer, & Gaser, 2017; Steffener et al., 2016); however, recently several groups have pointed out that this assumption is false (Le et al., 2018; Liang et al., 2019; Smith et al., 2019). Smith et al. (2019) pointed out an extreme case of this error: when age has truly no relationship with brain features, the difference between age and predicted age (“brain age gap”) is a linear function of age, which implies that age explains 100% of the variance in the BAG. Smith et al. (2019) note that any subsequent analyses studying the relationship between this gap and other metrics is equivalent to relating a linear transformation of age to other metrics.

To flesh out the gravity of this observation, consider an example: If age does not vary as a function of any of the brain parameters, all coefficients, aside from the intercept, will be close to zero with high probability, and the intercept will be close to the mean age of the training sample. Let A be the age of the ith subject, B the jth brain feature for the ith subject, ε random error, and the mean age of the training sample. The brain age model is thus:

And the fitted values are:

For simplicity, let us assume that the coefficients are estimated to be exactly zero. Suppose the mean age of the training sample is 10 years old. Every person will have an estimated age of 10, so their BAG, , will be 10 − A . Thus, the BAGs are as follows: 15‐year‐olds have a BAG of −5, 10‐year‐olds have a BAG of 0, 5‐year‐olds have a BAG of 5, and so forth. Older participants are estimated as being younger than they are, and younger participants as older. The BAG is a linear transformation of a residual (i.e., ), which by definition varies as a function of the outcome variable, in this case age. This emphasizes the complexity of interpreting the BAG. Suppose two groups, A and B, are to be compared with respect to the BAG. If the brain features are linearly independent of age, then testing for differences in the BAG is equivalent to testing, “Is the mean age of group A different from the mean age of group B?” In cases where the brain features are linearly associated with age, the group comparison of BAGs may be viewed as similar to asking “Controlling for the brain features, is the mean age of group A different from the mean age of group B?” As regression of out‐of‐sample residuals (derived from a previous model) is not equivalent to multiple regression, this description is not quite correct (Chen, Hribar, & Melessa, 2018; Freckleton, 2002). Nonetheless, the interpretation of the BAG remains difficult.

Whether or not the brain parameters under study are associated with age, the predicted age for every subject will still be shrunk toward the mean age of the training sample. This is referred to as regression toward the mean, and was first documented by Sir Francis Galton in 1886 (Bland & Altman, 1994). As Liang et al. (2019) noted, this phenomenon is a common feature of many good models. Therefore, older subjects will have negative BAG estimates on average simply because they are older, while younger subjects will have positive estimates on average. Note that the example of brain features being unable to predict age is unrealistic, and is used merely for illustration purposes. Out of sample R 2s typically fall between .4 and .9, and vary by the brain features and age range utilized (Alam, Nakano, & Kobashi, 2016; Aycheh et al., 2018; Beheshti et al., 2018; Brown et al., 2012; Cole, Poudel, et al., 2017; Dean III et al., 2015; Dosenbach et al., 2010; Guggenmos et al., 2017; Kiehl et al., 2018; Kwak, Kim, Chey, & Youm, 2018; Lancaster, Lorenz, Leech, & Cole, 2018; Pardoe et al., 2017; Richard et al., 2018; Rudolph et al., 2017; Steffener et al., 2016).

It is important to note that regression toward the mean is not a failure, but a feature, of regression and related methods. If there is any randomness in a process, observations will tend toward the mean of the outcome variable rather than remain as extreme as they were upon initial sampling (Stigler, 1997). Regression toward the mean is a feature of regression that is actively useful for prediction. Since age is known with certainty, the notion of predicting it makes the construction of a residual awkward because residuals are an estimate of random error. Thus, as we continue to use age prediction as a means to reduce dimensionality, it is important to understand the limitations of using age as an outcome variable and subsequent associated analyses. Recognizing the dependence of the BAG on age, researchers have begun to develop methods to mitigate the age dependence of the BAG (Beheshti et al., 2019; Le et al., 2018; Smith et al., 2019). However, these methods require careful theoretical consideration as we demonstrate in the next section.

RISKS OF USING AN MODIFIED BRAIN AGE GAP

To mitigate the residuals' dependence on age, some researchers apply the following algorithm (Beheshti et al., 2019; Chung et al., 2018; Liang et al., 2019; Smith et al., 2019) (see the appendix for details on Beheshti et al.'s (2019) method). First, a training sample is used to estimate a mapping f(⋅) from brain features to age. Then, for a left out subject i with brain data B , B , …, B , the predicted age (“brain age”) is estimated as :Then the ith subject's BAG isRecognizing the BAG's dependence on age, the researcher poses a linear model of the BAG on age:where estimated parameters and are found from a regression using training data, and δ is random error. Thus, the effect of age is removed, producing the MBAG:which, as the prediction error from model (5), is approximately uncorrelated with age (only exactly uncorrelated if test data is used to estimate α and γ). Because the MBAG has been interpreted as a corrected residual, the MBAG is added to (or subtracted from; equivalent in correlation, see Supplement) age. This new variable is then referred to as the corrected predicted age:Because the researcher perceives this predicted age as corrected, they correlate it with age to assess their model's accuracy in predicting age. We will refer to as the “modified predicted age” and will show below why this age estimation is flawed.

The MBAG is by no means a more accurate measure of an out‐of‐sample residual, or prediction error (i.e., the “brain age gap”). The BAG itself is more dependent on age the less the brain features are associated with age. Again, consider the extreme case where age is independent of the brain features. Then, the BAG is completely determined by age, as explained in the previous section. If the MBAG is treated as an estimate of the deviation from age, the reported model accuracy (e.g., ) will always be inflated relative to the true model accuracy, and often drastically so (see Table 1 for details on papers that have reported inflated model accuracy statistics). When age has no true dependence on the brain features, the population covariance between age and predicted age, , is zero. However, when the MBAG is treated as the deviation from age, A  + MBAG, age and modified predicted age, , have an approximately perfect correlation of 1.

TABLE 1

Papers reporting inflated model accuracy statistics

Paper	Before modification	After modification
Beheshti et al. (2019)	CorrA,A^^2=.38	CorrA,A^^M^2=.88
Chung et al. (2018)	CorrA,A^^2=.66	CorrA,A^^M^2=.84
Liang et al. (2019)	MAE =1.57	MAE =1.32
Smith et al. (2019)	CorrA,A^=.06	CorrA,A^^M=.99

Abbreviation: MAE, mean absolute error.

In fact, the inflated correlation can be directly computed as a function of the sample estimates of the covariance between age and predicted age, the variance of age, and the variance of predicted age (see Supplement for derivations):If and are estimated in the test set, Equation (8) can be further simplified:The equation can be simplified even further if is a linear estimator:

To illustrate the inflated correlation effect and confirm that Equation (8) is correct, a series of simulations were run to compare the transformations that researchers describe performing to the above equation using R version 3.6.2 (R Core Team, 2019). Training and testing sets of 10,000 samples were simulated from each of a series of bivariate normal distributions, where the true correlation between age and the brain feature was varied between 0 and 1, with the correlation between age and the modified predicted age, , in the test set being the key outcome measure recorded. All model parameters were estimated in the training set. Since there is only one brain feature, the correlation between age and predicted age is the same as the correlation between age and the brain feature. Results using a single brain feature are detailed in Figure 1. A single brain feature was used so as to have easy control over the correlation between age and predicted age, but note that this result generalizes to any number of brain features. For a set of correlations between 0 and 1, the correlation between age and the modified predicted age, , was calculated using the theoretical formulation in Equation (8) (black line), and the inflated correlation was obtained using the previously described transformations (grey dots). The identity line is displayed to aid in visualizing that the inflated correlation is larger than the true correlation. The simulations confirmed that the theoretical formulation in Equation (8) is equivalent to the transformations researchers have described. In addition, Figure 1 illustrates that the degree of inflation is much greater for models that have lower values of Corr than for models that have higher values of Corr.

FIGURE 1

Inflated correlation, Corr , is a function of the true correlation, Corr. The inflated correlation is the correlation between age and the modified predicted age. The true correlation is the correlation between age and predicted age. To illustrate that the series of transformations that researchers perform is equivalent to Equation (8), correlations using both are plotted. r func is using Equation (8), and r trans is using the series of transformations. The identity line is displayed for ease of comparing the axes

Additional analyses were run using the Philadelphia Neurodevelopmental Cohort (PNC) to illustrate the findings in brain MRI data. Sample details, neuroimaging protocols, and processing can be found in Calkins et al. (2015), Gur et al. (2021), and Satterthwaite et al. (2014). Briefly, participants ages 8–22 were recruited through their primary care providers in the Philadelphia area. After complete description of the study, written informed consent was obtained from participants aged ≥18, and written assent and parental permission were obtained from children aged <18 and their parents/legal guardian. The University of Pennsylvania and Children's Hospital of Philadelphia Institutional Review Boards approved all procedures. Subjects were excluded for the purposes of these analyses if their cognitive assessment was conducted more than a year before or after their neuroimaging data was collected, or if their structural image did not pass stringent quality assurance measures. T1‐weighted images (voxel dimensions: 0.9375 × 0.9375 × 1) were obtained using a magnetization prepared, rapid‐acquisition gradient‐echo sequence. Receive coil shading was reduced by selecting the Siemens prescan normalize option, which is based on a body coil reference scan. Then, 132 regional volume values were extracted using the Advanced Normalization Tools software package (Tustison et al., 2013; Wang & Yushkevich, 2013), including cortical, subcortical, white matter, ventricle, and cerebellar volumes.

Elastic net models to predict age were built on youths ages 8–22 without a history of mental illness (“typically developing”). Hyperparameters were chosen using repeated five‐fold cross‐validation on the typically developing youth as implemented in the “caret” package, version 6.0‐86 (Kuhn, 2012). Then, a linear regression of the BAG on age was fit in the typically developing subjects (N = 317). Using the fitted values for the parameters from these models, the transformations previously described were applied to youth who met screening criteria for lifetime instance of a mental illness (N = 862). This real data example confirmed the theoretical and simulation findings (see Figure 2). Prior to any modification, the correlation between age (A) and predicted age () was .77. After applying the modifications, the correlation became .88. There were no differences between the typically developing youth and youth with a history of mental illness on age (t =  − 1.05, p = 0.29), the BAG (t = 0.72, p = 0.47), or the MBAG (t = 0.09, p = 0.39). Age and performance on the complex cognition tasks were highly associated (r = 0.54, p < .0001). After regressing the brain features out of age and multiplying by negative one—or constructing the BAG—this association weakened (r =  − .30, p < .0001). The MBAG and performance on the complex cognition tasks were not associated (r =  − .01, p = 0.71). These results indicate that the association between cognition and the BAG are driven by the association between age and cognitive performance. As such, it is critical that readers of past literature note whether or not age was controlled for when testing for effects on the BAG, as this has not always been common practice (e.g., Beheshti et al., 2018; Cole, Underwood, et al., 2017; Franke et al., 2015; Gaser et al., 2013; Liem et al., 2017; Nenadić et al., 2017; Steffener et al., 2016).

FIGURE 2

The inflated correlation finding was replicated in the Philadelphia Neurodevelopmental Cohort. Plotted are values for age (A), predicted age (), the brain age gap (BAG), the modified BAG (MBAG) and the modified predicted age () in the subset of participants who met screening criteria for an instance of mental illness in their lifetime. Panel (a) displayed the correlation between age and predicted age; Panel (b) the correlation between age and the brain age gap; Panel (c) the correlation between age and the modified brain age gap; and Panel (d) the correlation between age and the modified predicted age. The identity line is displayed in Panels (a) and (d)

Other common methods for addressing the fact that the BAG is correlated with age include scaling the predicted age by the slope and intercept from the regression of predicted age on age (de Lange & Cole, 2020), and including age as a covariate when testing for group differences on the BAG (Le et al., 2018). de Lange and Cole (2020) and Cole et al. (2018) suggest fitting the following regression:where θ is the coefficient for age, λ is the intercept, η is random error, and predicted age () is treated as a random variable. Then, they calculate a revised predicted age () as follows:As de Lange and Cole (2020) demonstrate, this results in a revised predicted age that is as exactly correlated with age as the original predicted age. This is the case because correlations are invariant to linear transformations. Cole et al. (2018) go on to calculate the revised BAG as follows (though this is not stated explicitly in the manuscripts, it can be inferred from how they define brain‐PAD, i.e., the BAG):

This results in a revised BAG that is uncorrelated with age. In the PNC, the correlation between age and both the predicted age () and the revised predicted age () in the group of youth with mental illness was 0.77. The correlation between the revised BAG and age was −0.08. Note that it is not exactly zero because the parameters in Equation (11) were estimated in the typically developing youth.

Le et al. (2018) suggest either testing for group differences on the MBAG, or preferably controlling for age when testing for group differences on the BAG. The latter approach is not susceptible to researchers misinterpreting the reduced variability of the MBAG as an improvement in prediction accuracy because the MBAG is never calculated. In the PNC, controlling for age results in a nonsignificant relationship between the BAG and performance on the complex cognition tasks (t = 1.066, p = 0.29), similar to the result obtained from testing for a correlation between the MBAG and cognition. While the transformations and analyses in both Cole et al. (2018) and Le et al. (2018) are not susceptible to being misinterpreted as producing an improvement in model accuracy, there is still much work to be done to determine whether either speaks to advanced or delayed development.

CONCLUSION

We have shown that a modified predicted age (“brain age”) based on a regression adjustment of the BAG results in a correlation between the modified predicted age and age never falling much below 0.85 regardless of the original predicted age and age correlation. Further, the interpretability of the modified brain age gap (MBAG) itself is limited by the fact that it is a prediction error from a regression to remove the effects of age from a residual obtained through a regression to predict age. By virtue of these limitations, we suggest that the modified version may not provide useful information about precocity or delay in brain development. In light of this, as well as the complexities associated with interpretations of the BAG and its dependence on age, we suggest that further methodological and theoretical work is warranted.

Many other transformations have been developed to mitigate the downstream effects of BAG's dependence on age (de Lange & Cole, 2020). Some are not susceptible to the inflated correlation issue described in this work. Methods include scaling the predicted age by the slope and intercept from the regression of predicted age on age (see equation (5) in de Lange and Cole (2020)), and including age as a covariate when testing for group differences in BAG (Le et al., 2018). The former results in a new BAG estimate that is uncorrelated with age, and the latter ensures that any group differences found on the BAG will be linearly independent of age. More fundamentally, though, the need for age‐correction in BAG comparisons highlights the question under study and the methodology employed. For example, controlling for age when testing for group differences on an in‐sample estimated BAG is the two‐step regression equivalent of including age as a covariate in a multiple regression with brain features predicting age. This inherent circularity is troublesome and emphasizes the need for methods that are designed to directly answer questions about similarity between brains of different age groups and disease states.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

AUTHOR CONTRIBUTIONS

Ellyn R. Butler: Conceptualized the work, wrote derivations, and wrote the manuscript. Andrew Chen, Thomas E. Nichols, and Russell T. Shinohara: Wrote derivations, and provided feedback on the framing of the work. Rabie Ramadan, Trang T. Le, Fengqing Zhang, and Haochang Shou: Provided statistical insights and edited the manuscript. Theodore D. Satterthwaite: Oversaw the processing of the neuroimaging data and edited the manuscript. Kosha Ruparel, Tyler M. Moore, and Ruben C. Gur: Provided inspiration for, and assisted in the framing of the work.