Nonlinear Z-score modeling for improved detection of cognitive abnormality.

INTRODUCTION: Conventional Z-scores are generated by subtracting the mean and dividing by the standard deviation. More recent methods linearly correct for age, sex, and education, so that these "adjusted" Z-scores better represent whether an individual's cognitive performance is abnormal. Extreme negative Z-scores for individuals relative to this normative distribution are considered indicative of cognitive deficiency.
METHODS: In this article, we consider nonlinear shape constrained additive models accounting for age, sex, and education (correcting for nonlinearity). Additional shape constrained additive models account for varying standard deviation of the cognitive scores with age (correcting for heterogeneity of variance).
RESULTS: Corrected Z-scores based on nonlinear shape constrained additive models provide improved adjustment for age, sex, and education, as indicated by higher adjusted-R2. DISCUSSION: Nonlinearly corrected Z-scores with respect to age, sex, and education with age-varying residual standard deviation allow for improved detection of non-normative extreme cognitive scores.

Introduction and background

The Advancing Research and Treatment for Frontotemporal Lobar Degeneration consortium and the Longitudinal Evaluation of Familial Frontotemporal Dementia Subjects are both generating data from families with either a strong family history of frontotemporal dementia (FTD) or mutations in genes known to be associated with FTD: microtubule associated protein tau (MAPT), progranulin (GRN), or chromosome 9 open reading frame 72 (C9orf72). Both of these studies have the common goal of identifying the most robust and reliable methods to track disease progression in familial dementia, so that clinical trials of disease-modifying therapies can be designed appropriately. To this end, it is important to identify indications of a clinical change as early and accurately as possible.

A common approach to identifying cognitive deficits based on neuropsychological test scores is to compare the scores to those of a large set of cognitively normal individuals. The scores are flagged as potentially cognitively deficient if they are significantly worse than the scores represented in the cognitively normal data set.

Naïve (uncorrected) Z-score approach

The simplest approach to determine whether an individual's score is outside of the normal range is to calculate Z-scores. The method is to find the difference between the individual's obtained test score and the expected score (which is the mean of the score for the normative sample) and then divide that difference by the standard deviation (SD) of the normative sample's scores for that particular measure. When calculating Z-scores, one should be mindful of the fact that for some tests, a high score indicates good performance, but for other tests (e.g., tests where time to complete a task is the outcome being measured), a high score indicates poor performance. Common practice is to reverse the +/− Z-score sign in these cases such that a positive Z-score indicates better-than-expected performance and a negative score indicates poorer-than-expected performance. If the individual's Z-score is significantly below expectation compared with the normative sample, then the individual is flagged as being potentially cognitively deficient.

A major limitation of using a simple Z-score approach, however, is that it does not account for an individual's age, sex, or education level. These factors are known to influence what a “normal” score would be for a particular individual. For example, in some tests, higher education gives the false appearance of potentially protecting against dementia (a.k.a. the “cognitive reserve hypothesis” [1]). That is, individuals with higher education are less likely to show their early indications of dementia because cognitively normal individuals with higher education levels generally do better on the tests and so they need to decline further to be detected relative to an uncorrected distribution of scores. Therefore, longitudinal studies using norms that do not account for education may fail to appropriately detect decline in those that are more highly educated or sometimes detect decline inappropriately among individuals who are less educated. (Note that although these norms are applied to longitudinal data, they are applied independently at each time point and not to measures of change.) By generating Z-score distributions specific to education levels (via modeling adjustments), we can determine whether an individual has an extreme Z-score relative to their own specific education level.

Z-score adjustment via linear regression

The Z-score approach to neurocognitive assessment has been extended by multiple groups to consider linear correction for age, sex, and education [2], [3]. However, the linear assumption represents a strong constraint on the correction approach because it imposes the assumption of a linear relationship between each of age and education with the neuropsychological outcome scores in the regression. Furthermore, the standard linear model takes the homogeneous variance estimated for the residuals of the fitted regression and uses this constant value in the denominator of the Z-score estimate for all individuals. This use of constant SD in the Z-score estimates leads to severe underestimation or overestimation of the magnitude of the Z-score when the SD associated with a particular age, sex, and education score differs drastically from that of the mean SD of the linear regression residual errors. Edland et al. [4] had previously examined nonlinear correction of cognitive scores for the Cognitive Abilities Screening Instrument with respect to age, sex, and education. The authors dealt with the nonlinearity through Box-Cox transformation of the outcomes.

Z-score adjustment via nonlinear regression

In this article, we propose a direct nonlinear modeling approach for the Z-scores with respect to age, sex, and education to allow for nonlinear relationships with the neuropsychological outcomes examined. In addition, our approach explicitly incorporates a secondary nonlinear modeling of residual variance such that it can vary according to the covariates. This leads to improved determination of whether scores are extreme relative to the individual's age, sex, and education level.

Methods

Normative data

We used a slightly extended version of the National Alzheimer's Coordinating Center (NACC) database Uniform Data Set (UDS) of normal controls that was used by Weintraub et al. [3] (extended by additional data acquired by the NACC after December 2016). Informed consent was obtained by the NACC for all subjects in the study. Permission from the NACC has been obtained by this study to perform and publish this research on this data. These cross-sectional data were acquired from 29 Alzheimer's Disease Centers and considered visits between March 2015 and May 2017. The data set contained a total of 4287 individuals who were each tested one time. We removed individuals with missing education data and those who were aged 92 or over (numbers were small and outcomes erratic beyond this age), leaving 4193 individuals. We further restricted our normative data set to the 3461 Caucasians because there were only relatively small numbers for other racial groups, which overly limits size for building normative tables for those racial groups. Finally, we removed subjects that did not perform testing in English leaving 3430 participants.

Preprocessing steps

The data were further restricted to an age range of 40 to 91 by considering all individuals less than age 40 as being equivalent to individuals who are 40 years of age. The reason for this age shift (translation) was that the number of individuals was very small below age 40 and therefore sometimes led to poor model fits below age 40. In addition, most neuropsychological scores showed an approximately constant trend across the younger ages of 40 to 50, and thus, a constant effect for before 40 was thought to be a reasonable assumption on our part. Similarly, education levels of less than 10 years were mapped to 11 years of education and education levels above 21 years were counted as equivalent to 21 years due to the small number of subjects in those ranges.

Shape constrained additive model

Our approach of extending to nonlinear models with heterogeneous residual variance modeling is based on additive models [5], [6], and more specifically, shape constrained additive models (SCAMs) [7]. Additive models enable the fitting of smoothly varying functions that relate the predictors to the outcome. SCAMs extend on additive models by constraining the shape of the fitted functions to be monotonically increasing or decreasing (they can also be used to enforce convexity/concavity, but we do not use that here). The monotonicity constraints are used to reflect our scientific knowledge that the neuropsychological scores that measure dementia generally decrease with age and increase with education level (assuming higher scores are good; the reverse is true for scores where higher scores imply poorer performance).

The general form for an additive model iswhere the term represents the outcome value for individual . The term is the intercept term for the model (i.e., the fitted value of the outcome when all predictors are set to zero). The functions, , are smooth functions (i.e., not containing sharp fluctuations) of the covariates, to be determined from the data. The smoothness is commonly imposed on the functions by defining them to be spline functions, that is, suitably constrained combinations of polynomial functions, and this is the approach taken here. The terms are assumed to be independently and identically distributed as normal with mean 0 and SD of .

Model fitting

SCAMs were fit separately to each of the following outcomes (variable names with capital letters are those used in National Alzheimer's Coordinating Center Uniform Data Set Data Dictionary found at https://www.alz.washington.edu/WEB/rdd_uds.pdf): Trail Making Test A (TRAILA), Trail Making Test B (TRAILB), Letter fluency F (UDSVERFC), Letter fluency L (UDSVERLC), Category fluency–animals (ANIMALS), Category fluency–vegetables (VEG), Multilingual Naming test–Test (or MINT) total (MINTTOTS), Number Span longest digit forward (DIGFORSL), Number Span longest digit backward (DIGBACLS), Craft memory–immediate (CRAFTVRS), Craft memory–delay (CRAFTDVR), Benson figure–copy (UDSBENTC), Benson figure–recall (UDSBENTD), Montreal Cognitive Assessment (or MoCA) total (MOCATOTS), Number Span forward total correct trials (DIGFORCT), and Number Span backward total correct trials (DIGBACCT) [3].

Models were originally fit allowing nonlinear corrections for age and education (each constrained to be monotonic), along with an additive term for sex; we do not have a nonlinear term for sex because it is a binary predictor (male/female). For all models, the nonlinear education effect was close to linear, so we refitted education in the SCAM as a simple linear term.

For the neuropsychological outcomes, the form of the SCAM we used was

The only term that has a nonlinear relationship with the outcome in this model is age; the sex and education variables both have linear multipliers and , respectively.

The residuals for each outcome were then extracted from the corresponding fitted model, and the SD of the residuals at each age was estimated using a sliding window of width 11 years centered at the age value being considered; the SD at an age was estimated by the sample SD of all residuals within the corresponding window. A second SCAM was then fitted to the estimated SD as a function of age. (The choice of an 11-year window was based on a compromise between having enough data in each window vs. the risk of smoothing out the change with age too much.) Note that the original nonlinear model for the outcome was fitted using either a homogeneous SD assumption or occasionally modeling a linear relationship of age with SD whenever it appeared to lead to a better visual fit. The point at which we really capture the nonlinear component of the SD is in the second nonlinear model for SD described below. (Note that we provide explicit details on whether the homogeneous variance assumption was used along with other model settings for each outcome in Supplementary Materials).

We observed changing variability of residuals with increasing age. However, for education versus SD, we did not see a clear relationship that warranted adjustment, and so we did not consider the potential for varying SD with education any further. For a few outcomes (Category fluency–animals and Number Span longest digit forward), the SD appeared to be clearly nonmonotonic with respect to age, in which case we fit unconstrained additive models rather than SCAMs.

The additive model for the SD therefore took the following form:where, , indexes the set of windowed SD estimates (one estimate for each age).

A lookup table for each outcome was generated based on these two SCAM/additive model fits. For each value of age, sex, and education level, an adjusted Z-score was generated, using the fitted mean and SD for that age, sex, and education level from the fitted SCAM/additive models.

The fitting of the nonlinear models often required some additional processing to the models or the data. For example, the direction of monotonic relationships (i.e., whether increasing or decreasing) or lack thereof needed to be specified; outliers were sometimes removed when they had a large effect on the model fit; SD additive model fits were sometimes weighted by the sample size (number of individuals) in each age window, and sometimes were not. The decisions as to the preferred modeling approach were made in discussions with clinical investigators. Full details of additional steps taken for each variable are given in Supplementary Material with this article.

For comparison purposes, we also fitted linear models corresponding to the models described in Weintraub et al. [3]. That is, we fitted linear models with age, sex, and education as additive predictors and with a constant residual variance.

Note that the results from these fitted linear models differ from those for the UDS, version 3, calculator in the study by Weintraub et al. because our snapshot of the data was taken later than theirs and therefore includes more data.

Models were fitted using the R statistical programming environment [8] along with the mgcv package for additive models [6] and the scam package for SCAMs [7]. All models were fitted with the default parameters, except that the SCAM package requires manual definition of the direction of monotonicity.

Results

We focus the results to illustrate the model fits for two outcomes: Trail Making Test B (TRAILS B), which is measured in seconds taken to complete (max. 300), and total number of animals (ANIMALS) named in 60 seconds (max. 77). Complete results for all remaining outcomes are provided between Supplementary Plots and Table 1.

Table 1

Comparisons of adjusted-R2 for linear versus additive models

Neuropsych measure	Adjusted-R2 linear model	Adjusted-R2 additive model	Adjusted-R2 SD additive model
Trail Making Test A	0.152	0.164	0.983
Trail Making Test B	0.161	0.183	0.989
Letter fluency F	0.050	0.055	0.302
Letter fluency L	0.061	0.066	0.000
Category fluency–animals	0.114	0.120	0.863
Category fluency–vegetables	0.137	0.145	0.000
Multilingual Naming Test (MINT) total	0.062	0.069	0.716
Number Span longest digit forward	0.021	0.021	0.911
Number Span longest digit backward	0.035	0.035	0.935
Craft Story memory–immediate	0.055	0.062	0.022
Craft Story memory–delay	0.062	0.068	0.177
Benson figure–copy	0.016	0.017	0.984
Benson figure–recall	0.084	0.086	0.959
MoCA total	0.140	0.149	0.996
Number Span forward total correct trials	0.027	0.027	0.732
Number Span backward total correct trials	0.040	0.040	0.360

NOTE. The adjusted-R2 linear and adjusted-R2 additive model columns indicate the adjusted-R2 for the respective models. Note that the results from the linear models differ from those for the UDS, version 3, calculator in Weintraub et al. because it was based on fitting to a later (larger) snapshot of the data. The final adjusted-R2 SD additive model column represents the adjusted-R2 for the second SCAM model fitting the SD across ages (i.e., comparing a model that allows the SD to vary with a constant estimated SD across all the ages). Note that there is no comparison for the SD model because we are really comparing against a constant SD model which would have an R2 of zero.

Abbreviations: MoCA, Montreal Cognitive Assessment; UDS, Uniform Data Set; SD, standard deviation.

TRAILS B

Fig. 1 displays plots of (A) TRAILS B versus age (years) and (B) TRAILS B versus education (years). SCAM-based fitted lines are added to Fig. 1A based on males with 10, 15, and 20 years of education and to Fig. 1B based on males of age 50, 60, 70, and 80. The (additive) sex effect was very small by comparison to the (nonlinear) age and (linear) education effects (i.e., the predicted TRAILS B scores were very close for the two sexes for a specific age and education level). Plots showing sex differences are given in Supplementary Materials for all outcomes and were actually relatively small (compared with age and education effects) for all outcomes. Note that, the small difference observed between age 50 and 60 in the education plot lines of Fig. 1B reflects the nonlinearity of the age effect where the lines can be seen in Fig. 1A to be relatively flat at younger ages.

Fig. 1

(A) shows a plot of Trail Making Test B scores versus age in years (based on males with 10, 15, and 20 years of education), and (B) shows Trail Making Test B scores versus education level measured in years (based on males of age 50, 60, 70, and 80). The small difference between ages 50 and 60 in the education plot lines reflects the nonlinearity in the age plot where the lines are relatively flat at younger ages. The sex effects were very small in comparison with age and education. Plots showing sex differences are given in Supplementary Materials.

There is clear nonlinearity in the relationship between age and TRAILS B, with an increasing trend that accelerates with increasing age. The SCAM model provided for an adjusted-R2 of 0.183 for the nonlinear model versus 0.161 for the corresponding linear correction model. (The adjusted-R2 is a version of R2 that is adjusted to account for the number of parameters in the model, the difference being to determine genuinely better fitting models from those that simply explain more variance by overfitting to noise). Note that although this observed increase in adjusted-R2 does not appear to be large, it does represent a 13% increase, and this improvement is going to have its largest effect at the age extremes where the linear fit of the mean function would depart the most from the true mean. In particular, notice that between age 40 and 55 (critical for FTLD assessment), the mean score is very flat. A linear fit would severely underestimate the mean score in individuals under age 45 which would lead to overcalling extreme Z-scores (based on high test scores) in younger subjects. Note also that in these data, the between-subject variability is actually very high and that places an upper ceiling on what (adjusted) R2 can be attained even with a perfect mean function. Indeed, it is this high variation in the scores of cognitively normal subjects that we are trying to capture in the normative scores distributions, but with removal of bias due to age, sex, and education.

Fig. 2 displays the window estimated SD versus age (blue line) and the corresponding fitted SCAM model (red line). Increasing SD was seen with age, and the SD increased by a factor of approximately 3 between age 40 and age 90. The nonlinear approach explicitly considers the age-specific value of the SD, in contrast to the linear correction approach, which assumes a constant SD. The large difference in variability across ages has major implications because the SD forms the denominator of the Z-score, thereby affecting up to a factor of 3 difference in the Z-score. The adjusted-R2 of the nonlinear fit to SD when compared with assuming a constant SD is 0.989. Note that the R2 for a constant fit to a varying SD is effectively 0 because none of the variation in the SD is explained by a constant value.

Fig. 2

Plot of SD of residuals for Trail Making Test B model versus age in years. Blue line shows raw SD curve based on sample SD estimates within the 11-year window centered on each point. The red line shows the corresponding SCAM model fit. Abbreviations: SCAM, shape constrained additive model; SD, standard deviation.

ANIMALS

Fig. 3 displays plots of (A) ANIMALS versus age (years) and (B) ANIMALS versus education (years). SCAM-based fitted lines are added to Fig. 3A based on males with 10, 15, and 20 years of education and to Fig. 3B based on males of age 50, 60, 70, and 80. The (additive) sex effect was small by comparison to the (nonlinear) age and (linear) education effects, and the interested reader can examine the (small) differences in the curves between the sexes via the plots in Supplementary Materials. No outliers were removed when fitting the SCAM models for ANIMALS.

Fig. 3

(A) shows a plot of Category fluency–animals versus age in years (based on males with 10, 15, and 20 years of education), and (B) shows Category fluency–animals versus education level measured in years (based on males of age 50, 60, 70, and 80). The sex effects were small in comparison with age and education. Plots showing sex differences are given in Supplementary Materials.

Nonlinearity in the relationship between age and ANIMALS is again clear, but this time with a decreasing trend that accelerates with increasing age. The SCAM model provided for an adjusted-R2 of 0.120 for the nonlinear model versus 0.114 for the corresponding linear correction model.

Fig. 4 displays the window estimated SD versus age (blue line) and the corresponding fitted SCAM model (red line). The SD was seen to vary with age, but for this outcome, the relationship did not appear to be monotonic, and so we modeled with an unconstrained additive model. The adjusted-R2 of the fit when compared with assuming a constant SD is 0.863.

Fig. 4

Plot of SD of residuals for Category fluency–animals versus age in years. Blue line shows raw SD curve based on sample SD estimates within the 11-year window centered on each point. The red line shows the corresponding SCAM model fit. Abbreviations: SCAM, shape constrained additive model; SD, standard deviation.

Plots of nonlinear fits to the complete set of neuropsychological outcomes are available in Supplementary Materials.

Overall results

Table 1 shows the improvements in adjusted-R2 obtained by using nonlinear fitted models for all of the outcomes considered. The improvements in fit for the nonlinear models can be seen by noting that the adjusted-R2 for the additive models are always equal to or higher than the corresponding adjusted-R2 for the linear models. The final column shows the adjusted-R2 in the second SCAM model for the outcome of SD compared with a constant SD model (i.e., by allowing it to vary with age rather than fixing to the estimated mean SD across all the data). Note that there is no comparison model for the SD model because we are really comparing against a constant SD model which would have an R2 of zero. It can be seen that in many cases, the gains obtained by nonlinearly fitting the SD are very high, indicating a clear pattern of variation in the SD with age across many outcomes. For some outcomes, the gains to fitting variable SD are not as strong; in cases where the adjusted-R2 for SD is essentially zero, the result is indicating that the data are providing no indication of a consistent pattern of variation in the SD with age and that fitting a constant SD across all ages is adequate.

Comparison study

For illustrative purposes, a comparison was undertaken of Animals and Trails B Z-scores based on the UDS calculator [3], published normative data of Heaton et al. and Tombaugh et al. [9], [10], and the adjusted calculator utilizing the nonlinear SCAM corrections. Z-scores were calculated for young, middle-aged, and older (ages 40, 55, and 75) males and females with varying degrees of education (10, 13, 16, and 20 years). Total words correct for Animals was randomly predetermined to be 18, and the time to complete Trails B was set at 65 seconds to directly compare methodologies. Consistent improvements (i.e., relative to published norms) in the interpretations of Z-scores were seen across these neurocognitive measures when allowing for nonlinear adjustment via SCAM fitting as compared with the NACC linear model for calculating Z-scores [3]. Table 2 depicts the Z-score comparisons between methods.

Table 2

Normative data for Category fluency–animals (ANIMALS) and Trail Making Test B (TRAILB) based on different normative calculators

	Age	Education	Sex	ANIMALS UDS Raw 18 Words/Z	ANIMALS Tombaugh Raw 18 Words/Z	ANIMALS Nonlinear Raw 18 Words/Z	TRAILB UDS Raw 65 Seconds/Z	TRAILB Heaton Raw 65 Seconds/Z	TRAILB Nonlinear Raw 65 Seconds/Z
1	40	10	M	−1.90	−0.43	−0.73	−1.10	+0.10	−0.27
2		10	F	−2.05	−0.43	−0.60	−0.85	0.00	−0.18
3		13	M	−2.64	−0.72	−1.04	−3.27	−0.50	−0.56
4		13	F	−2.79	−0.72	−0.90	−3.03	−0.60	−0.48
5		16	M	−3.39	−0.72	−1.35	−5.45	−0.80	−0.85
6		16	F	−3.54	−0.72	−1.21	−5.21	−0.90	−0.77
7		20	M	−4.38	−0.72	−1.76	−8.35	−1.00	−1.24
8		20	F	−4.53	−0.72	−1.62	−8.11	−1.10	−1.16
9	55	10	M	−0.93	−0.43	−0.67	+2.73	+0.60	−0.24
10		10	F	−1.08	−0.43	−0.55	+2.97	+0.50	−0.16
11		13	M	−1.67	−0.72	−0.96	+0.55	0.00	−0.52
12		13	F	−1.82	−0.72	−0.83	+0.80	−0.10	−0.44
13		16	M	−2.42	−0.72	−1.25	−1.63	−0.30	−0.81
14		16	F	−2.57	−0.72	−1.12	−1.38	−0.40	−0.73
15		20	M	−3.41	−0.72	−1.63	−4.53	−0.50	−1.19
16		20	F	−3.56	−0.72	−1.50	−4.28	−0.60	−1.11
17	75	10	M	+0.36	+0.37	−0.12	+7.83	+1.60	+0.67
18		10	F	+0.21	+0.37	+0.00	+8.08	+1.50	+0.71
19		13	M	−0.38	−0.05	−0.42	+5.65	+0.90	+0.52
20		13	F	−0.53	−0.05	−0.29	+5.90	+0.80	+0.56
21		16	M	−1.12	−0.05	−0.71	+3.48	+0.80	+0.37
22		16	F	−1.27	−0.05	−0.58	+3.72	+0.60	+0.42
23		20	M	−2.11	−0.05	−1.11	+0.57	+0.60	+0.18
24		20	F	−2.27	−0.05	−0.98	+0.82	+0.40	+0.22

NOTE. Z-scores in bold are impaired (≥ 1.5 SD below the mean); Z-scores in italics are ≥ 1.5 SD above the mean.

Abbreviations: UDS, UDS Calculator; Tombaugh, Published Norms of Tombaugh et al. (stratified by age/education); Nonlinear, Adjusted Calculator (Kornak et al.); Heaton, Published Norms of Heaton et al. (stratified by age/education/sex/Caucasian).

Case example

A 40-year-old male with 13 years of education presents for evaluation because he has a family history of FTD and has found out that he is a gene mutation carrier. He has no complaints, and his informant expresses no concern. He would like to establish a cognitive baseline. On testing, he generates 18 words on the animal fluency task and completes Trails B in 65 seconds. Utilizing the UDS norms, these scores are found to be markedly abnormal (see Table 2), Z = −2.64 and Z = −3.27, respectively. Similar scores are obtained on other measures of attention, language, and working memory, whereas the remainder of his cognitive profile is relatively normal. In the context of his family history and gene status, he is told that he has cognitive impairment consistent with FTD, that is, impairments in language and executive functioning. He is referred for further diagnostic testing, which is time-consuming, logistically and emotionally burdensome, and expensive. Ultimately, these diagnostics are negative, but he is told that an evolving process cannot be ruled out given his cognitive testing and gene status, although having the mutation does not determine with certainty that he will even develop the disease. He becomes anxious and despondent. Alternatively, with nonlinear adjustments from the proposed SCAM models, which are generally in agreement with well-established published norms, this individual would be found to be functioning in the low average range for animal fluency and within the average range for Trails B and other measures. He would be counseled on the fact that just because he has the mutation, it does not mean he will develop FTD, and his scores at present are not concerning. It would be recommended that he return for reevaluation in about a year, or sooner if he started to show symptoms.

Discussion

In this article, we have described a nonlinear extension to generating age-, sex-, and education-corrected cognitive Z-scores. This nonlinear approach does a good job of modeling potential nonlinearities between the predictors (age, sex, and education) and the cognitive outcomes. Specifically, age was seen to display clear nonlinearity with respect to multiple cognitive outcomes. Furthermore, the additional modeling of the nonlinear relationships between age and SD has a high impact on determining extreme scores; some individuals would be given Z-scores that are too extreme (i.e., individuals in age ranges that have high variance) and others would be given Z-scores that are too conservative (i.e., individuals in age ranges that have low variance).

As shown in Table 2, extreme Z-scores were much more likely to be obtained using the NACC calculator without nonlinear adjustments. Specifically, according to the published norms for Animals, none of the Z-scores calculated, regardless of age, sex, or education, were abnormally low or abnormally high (i.e., ≥ ±1.5 SD from the mean) for a set point of 18 words generated. The same was true for completing Trails B in 65 seconds. Nearly, all of the Z-scores obtained from the NACC calculator for both of these measures fell to one extreme or the other, most frequently toward impairment (−1.67 to −4.53 for animals and −1.63 to −8.35 for Trails B). In contrast, no Z-scores for Trails B using the nonlinear (SCAM) adjusted model were abnormally high or low. Only 4 of the 24 Z-scores calculated for Animals using the nonlinear (SCAM) adjusted model deviated from the norm, and for those that did, it was only to a mild degree (range of −1.50 to −1.76).

Limitations

There are some limitations with our nonlinear modeling approach.

First, we needed to occasionally manipulate data and variables, such as removing outliers or changing the form of the smoothing function (monotonic vs. nonmonotonic, etc.). Invariably, nonlinear modeling has increased difficulties over linear approaches with respect to linear approach and often require user manipulation. Clearly, a fully automated process for model estimation that provides completely accurate results every time is the ideal. However, although we acknowledge the need for manipulation in model estimation is as an issue, we argue that it is less problematic than when performing statistical inference. The goal here is optimal estimation of Z-scores, with optimal adjustment for variability, and not testing for statistically significant effects. The concern about potential manipulation (with implicit multiple comparison concerns) and possible overfitting are less central to estimation. Future research should carefully evaluate the generalizability of our models on Advancing Research and Treatment for Frontotemporal Lobar Degeneration/Longitudinal Evaluation of Familial Frontotemporal Dementia Subjects patient data over time.

Second, the SD windowing estimation approach is admittedly ad hoc. To the best of our knowledge, existing methodology and software is not readily available for the simultaneous estimation of nonlinear trend for both outcome and residual SD with respect to a continuous predictor. Future research could extend the methodology to simultaneous (joint) modeling for both the outcome and nonlinear heterogeneous SD.

Third, there are clearly non-normally distributed errors for many outcomes. Again, because statistical inference is not the primary concern, the effects of this limitation are mitigated. However, in our joint modeling of outcome and SD methodology development, we plan to incorporate the possibility of non-normal errors. Furthermore, there were a couple of outcomes (i.e., Number Span longest digit forward and longest digit backward) that were so heavily discretized (integer scale of 3 to 9, and 2 to 8, respectively) that we do not advocate performing any kind of corrected Z-score estimation for these outcomes.

Fourth, we have less data near the edges of our age and education ranges. Consequently, when considering individuals with age and/or education close to or beyond the limits of the data, it is important to be aware of the context of the model predictions. Indeed, when interpreting scores for individuals with covariates that are associated with few data points in the UDS, it would be important to report on the additional model uncertainty involved.

Fifth, our modeling approach treats the effects from each predictor as additive. Extension to nonadditive age and sex are available within the additive model framework; these extended models allow for arbitrary two-dimensional smooth functions of pairs of predictors (and can be extended to yet more dimensional predictors), allowing for complex interactions between these predictors. Although the extended modeling can provide improved fit in many cases (internal validity), such models are more difficult to interpret, and therefore there is a stronger need for external validation. This led to our decision to initially avoid such models for our Z-score correction process.

Sixth, a side effect of the nonlinear fitting approach with heterogeneous variance is that it is not always true that a Z-score will improve with age (i.e., become less extreme) for a constant value of a test score (which is what one might initially expect given that you expect decline with age). For example, if the SD is increasing with age at a faster rate than the mean score that is declining with age, then a nonchanging test score could become less extreme relative to the normative distribution as age increases. Whether this is viewed as a negative depends on your perspective, but when the objective is to determine whether or not a score is extreme given a specific age, it is the correct approach to consider the variance specific to that individual's age in the calculation. Note also that this effect of a constant score becoming less extreme with respect to the normative distribution with increasing age might be exacerbated for outcomes where the residuals are not normally distributed. In particular, scores which have a limiting value at certain ages (i.e., ceiling or floor effects) because the SD can change radically over a short age range. The fact that our approach accounts for this changing variance is positive relative to conventional approaches in that we are still able to detect which observations are extreme at a specific age. Further improvements might potentially be made on this front by explicitly modeling the non-normality in the data.

Seventh, our data was only based on Caucasians. Normative tables should be developed in the future for other racial groups as appropriate data become available.

Finally, we note that our nonlinear Z-score approach to flagging individuals with cognitive deficiency is based only on normative individuals, that is, we are examining what are unusual scores for normal controls, who are from different sites and in different settings. The conventional method for identifying abnormal cognitive performance is to compare an individual's obtained cognitive scores to scores from normative samples that are derived from studies examining large numbers (usually) of healthy individuals at one time point. These samples may or may not include individuals that at some point go on to demonstrate cognitive impairment. This potentially mixed sample actually decreases the ability to detect early cognitive changes, particularly if normative data are derived from small samples. Ideally, one would want a robust normative dataset that is derived from cognitively normal individuals who are subsequently tracked longitudinally and shown to be stable over time. Several studies have found superiority of robust norms over conventional norms in identifying cognitive impairment at baseline [11], [12], [13]. By utilizing robust norms, comparison of test scores in individuals with scores that decline over time to scores from those who remain cognitively stable has greater diagnostic utility both in terms of current status and predicting change over time. Unfortunately, the current normative sample has not been assessed longitudinally, so we do not have the capability to develop robust norms at this time.

The changing of environments is a concern when examining normal controls because the distribution of cognitive scores will differ according to whether data are collected in rural versus urban, multisite versus single site, hospital versus specialty clinic settings, etc. Examining such differences between environments needs to be a focus of future research as more data become available.

Conclusions

Allowing nonlinear model fitting via SCAM models provides improvement in fit for the relationship between the predictors (age, sex, and education level) and neurocognitive outcomes in control normal relative to linear models. The nonlinear modeling thereby leads to adjusted Z-scores that are more representative of the departures from cognitively normal levels relative to their specific age, sex, and education level.

Hence, adjusting Z-scores using nonlinear SCAM models provides improved adjustment for age, sex, and education level compared with linear adjustment.

The ultimate danger in basing judgments on inaccurate and unreliable scores is misdiagnosing an individual, which results not only in inadequate and/or inappropriate care but also limits our ability to develop disease-modifying treatments.

Systematic review: The Z-score approach to normalizing cognitive scores has had a long history using the simple approach of subtracting the mean and dividing by the standard deviation. More recently, linear methods corrected for age, sex, and education have been adopted, so that ensuing adjusted Z-scores better represent whether a score is outside the expected, or “normal,” range.

Interpretation: Cognitive scores versus each of age and education show clear nonlinear relationships and varying residual standard deviation with age. Adjusted Z-scores based on nonlinear shape constrained additive models (SCAMs) improve estimation of the degree of departure from the norm more precisely than linear or unadjusted Z-scores.

Future directions: Improved Z-score estimation allows for improved assessment of cognitive decline and treatment effects, leading to improved clinical diagnosis and treatment planning. We will post the nonlinear calculator on the National Alzheimer's Association Uniform Data Set website: https://www.alz.washington.edu/WEB/data_descript.html.