A Pragmatic, Data-Driven Method to Determine Cutoffs for CSF Biomarkers of Alzheimer Disease Based on Validation Against PET Imaging.

BACKGROUND AND OBJECTIVES: To elaborate a new algorithm to establish a standardized method to define cutoffs for CSF biomarkers of Alzheimer disease (AD) by validating the algorithm against CSF classification derived from PET imaging.
METHODS: Low and high levels of CSF phosphorylated tau were first identified to establish optimal cutoffs for CSF β-amyloid (Aβ) peptide biomarkers. These Aβ cutoffs were then used to determine cutoffs for CSF tau and phosphorylated tau markers. We compared this algorithm to a reference method, based on tau and amyloid PET imaging status (ADNI study), and then applied the algorithm to 10 large clinical cohorts of patients.
RESULTS: A total of 6,922 patients with CSF biomarker data were included (mean [SD] age: 70.6 [8.5] years, 51.0% women). In the ADNI study population (n = 497), the agreement between classification based on our algorithm and the one based on amyloid/tau PET imaging was high, with Cohen's kappa coefficient between 0.87 and 0.99. Applying the algorithm to 10 large cohorts of patients (n = 6,425), the proportion of persons with AD ranged from 25.9% to 43.5%. DISCUSSION: The proposed novel, pragmatic method to determine CSF biomarker cutoffs for AD does not require assessment of other biomarkers or assumptions concerning the clinical diagnosis of patients. Use of this standardized algorithm is likely to reduce heterogeneity in AD classification.

Alzheimer disease (AD) is the most common cause of dementia, and it currently affects more than 40 million people worldwide. The disease is neuropathologically characterized by extraneuronal accumulation of β-amyloid (Aβ) peptide in the brain (amyloid plaques), tau pathology in the form of intraneuronal deposits (neurofibrillary tangles) and dystrophic neurites surrounding plaques, massive synaptic loss, and neuronal death.[1] The clinical consequence of the disease entails progressive deterioration of cognitive function leading to dementia.

The diagnosis of AD in health care settings and population studies is primarily based on clinical criteria, undertaken at the stage of dementia[2] or mild cognitive impairment (MCI).[3] The clinical criteria have poor specificity[4] because of similarity in symptoms between many degenerative and nondegenerative disorders.[5] The discovery of specific biomarkers of AD neuropathologic lesions over the past 2 decades, consisting mainly of CSF biomarkers and PET imaging radioligands,[6,7] has improved the specificity of AD diagnosis and is likely to play a crucial role in the elaboration of therapeutic solutions in the future.[8] Tau and Aβ peptide biomarkers have been included in the new research diagnostic criteria of AD,[2] with the aim of increasing biological homogeneity of diagnosed cases.[9] The research criteria are based on the A/T/(N) classification with markers of Aβ deposition (A), pathologic tau (T), and neurodegeneration (N)[10]; each biomarker is categorized as positive or negative to yield AD diagnosis without the use of clinical diagnostic criteria.[11]

There exist CSF-based measures of Aβ peptide (CSF Aβ42, CSF Aβ42/40 ratio) and protein Tau (total tau: CSF Tau, phosphorylated tau: CSF p-Tau 181), which are amenable to the A/T/(N) classification.[12] Biomarkers are increasingly being used to diagnose AD, and a previous study showed that faced with discrepancies between the clinical presentation and biomarker profile the final diagnosis was based on the biomarker profile in up to 75% of cases.[13] The reliability and accuracy of biomarker-based diagnosis has implications for clinicians involved in AD diagnosis and their patients. A major concern is the considerable intersite variability in biomarker levels using standard ELISA methods,[14] leading to the recommendation that each biochemistry laboratory establishes its own cutoffs to determine positive status on these biomarkers.[2,15,16] Despite recent efforts from manufacturers to develop automated assays[17,18] and initiatives from research groups to standardize procedures,[16,19] universal cutoffs for CSF AD biomarkers remain to be established. In an international systematic review of 40 centers involved in AD diagnosis worldwide, only 16% reported using cutoffs provided by the manufacturer, 4% used cutoffs based on the literature, and the remaining used in-house cutoffs.[20] The methodology used to determine these cutoffs remains unclear because consensus on the gold-standard method to determine cutoffs to designate positive biomarker status does not yet exist.[21] Several parameters play a role in the observed variability of CSF biomarkers, including polypropylene tube used during the lumbar puncture.[22]

The most common method used to determine the threshold for CSF Aβ42 positivity is comparison with amyloid PET imaging.[23] Another method involves the use of rank-based thresholds (90th or 95th percentile) as is the case for CSF Aβ42 and tau.[24] Other methods include comparison between patients with AD and non-AD based on clinical criteria,[14] postmortem neuropathologic criteria,[25] or cutoffs based on the distribution of CSF Aβ42 across the total population.[26] All these methods have limitations, with some of them not being readily reproducible.

We propose a new method to standardize the procedure used to determine cutoffs for CSF biomarkers. The objective is to develop a simple algorithm that does not require biomarkers other than CSF biomarkers and can be used by others to homogenize the manner in which cutoffs are determined. Our strategy consists of using CSF p-Tau 181, a specific biomarker of AD,[27] to determine the cutoff values for beta-amyloid biomarkers to allow cross-validation between biomarkers. We first compared results of our algorithm with cutoffs based on amyloid and tau PET imaging using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI) study, and then, we applied our method to 10 patient cohorts drawn from memory centers.

Methods

Study Population

The ADNI study, launched in 2003, is a global research study involving 63 sites in the United States and Canada that aims to characterize the progression of AD in the human brain with clinical, imaging, genetic, and biospecimen biomarkers through the process of normal aging, MCI to dementia, or AD.[28]

Memory center patients were drawn from several research centers in Europe (France [Paris, Lille, and Montpellier], Sweden [Gothenburg], Spain [Barcelona], Belgium [Brussels], and the Netherlands [Amsterdam]). The technique used for CSF biomarker dosage was the same within each center. All patients had CSF biomarker assessment as part of their investigation for a cognitive disorder.

Standard Protocol Approvals, Registrations, and Patient Consents

Ethical clearance was obtained by the institutional review boards of all participating sites. All participants provided written informed consent.

Assessment of CSF Biomarkers

CSF concentrations of Aβ42, Aβ40, total Tau, and p-Tau 181 were measured with commercially available immunoassays, using the manufacturer's procedures. Four different methods were used: (1) The Elecsys immunoassays using the cobas e601 analyzer (Roche Diagnostics GmbH). (2) The INNOTEST immunoassays (Fujirebio Europe, Gent, Belgium). (3) The Lumipulse G1200 (Fujirebio Europe, Gent, Belgium). (4) The Euroimmun analyzer I-2P (Euroimmun AG, Luebeck, Germany).

Some centers (Paris, Montpellier, and Lille) contributed 2 patient cohorts because they used 2 different methods over time for the dosage of biomarkers. CSF samples in the ADNI study were analyzed using Elecsys immunoassays. We decided not to include older CSF ADNI data from the Luminex platform because of the long delay, approximately 5 years, between the CSF and tau PET measures. More complete information regarding CSF data in the ADNI is available online.[29]

Amyloid and Tau PET Imaging (ADNI)

We used data from the ADNI study on participants with data on CSF biomarkers and at least 1 PET imaging of beta-amyloid or tau radiotracer; further information on acquisition of PET data in the ADNI is provided on the ADNI website.[29] Amyloid PET imaging was performed using florbetapir (AV-45) radioligand,[30] and we used the following data: UCBERKELEYAV45_05_12_20-2.csv. Positivity for florbetapir PET imaging was defined by a global standard uptake value ratio (SUVR) higher than 1.11 using the whole cerebellum as a reference region; this cutoff was defined as the upper 95% CI above the mean in a group of young, cognitively normal controls in cross-sectional analyses.[31]

Positivity for tau PET imaging was determined using flortaucipir (AV-1451) imaging,[32] and we used the following data: UCBERKELEYAV1451_05_12_20.csv. Flortaucipir SUVR maps were generated using the inferior cerebellar gray matter as a reference region.[32] Positivity of flortaucipir was defined as an SUVR of the Braak 1 and 2 composite region higher than 1.32, which has been found to be the optimal cut points to separate Aβ+ AD patients from Aβ− elderly controls in cross-sectional analyses.[33]

Algorithm for CSF Cutoff Determination

The algorithm was defined before data analyses, based on consensus between the authors of the manuscript; this group includes clinicians and biologists with extensive experience in the field of AD biomarkers. The steps of the algorithm are shown in Table 1.

Table 1

Algorithm Used to Determine Cutoffs for CSF Biomarkers

The first step consisted of identifying participants with “low CSF p-Tau 181” (between the 10th and 30th percentile of the CSF p-Tau 181 distribution) and “high CSF p-Tau 181” (between 80th and 100th percentile), separately in each cohort. Participants with values between 0 to 10th percentile were removed from the analyses to avoid abnormally low values that reflect either measurement error or normal pressure hydrocephalus.[34] We then determined the ability of CSF Aβ42/40 ratio and/or CSF Aβ42 to discriminate between participants with “high CSF p-Tau 181” and “low CSF p-Tau 181” using the area under the receiver operating characteristic (ROC) curve (AUC). Optimal cutoffs for CSF Aβ42/40 ratio and CSF Aβ42 were defined as the lowest distance to the top left corner of the ROC curve. The known analytical variability in the CSF Aβ42 assays implies that values near the cutoff are difficult to classify as normal or abnormal, leading several teams to use the term “gray zone” to describe values 10% around the threshold.[12,35] We used values ≤90% to identify participants with “low CSF Aβ42/40 ratio” and ≥110% for “high CSF Aβ42/40 ratio” and then performed ROC curve analysis to determine the AUC and optimum cutoffs for CSF Tau and CSF p-Tau 181 to discriminate between these 2 groups (high vs low CSF Aβ42/40 ratio). In the absence of data on the CSF Aβ42/40 ratio, we used CSF Aβ42.

Stata code used to derive the algorithm has been uploaded to a GitHub repository[36]: the “Sensspec” Stata module was used to compute sensitivity and specificity.[37]

Statistical Analysis

The characteristics of participants were examined in each cohort; proportions were calculated for categorical variables and mean and SD for continuous variables.

Figure 1 illustrates the overall design of the study. The first step consisted of validation of the proposed algorithm using data from the ADNI study by comparing CSF biomarker positivity determined using our proposed algorithm with that based on tau and amyloid PET imaging. Amyloid positivity in the ADNI was defined using florbetapir amyloid PET imaging (cutoff for SUVR = 1.11), and then, the AUC for CSF Aβ42/40 ratio and CSF Aβ42 was used to discriminate between positive and negative cases. The optimal cutoffs for CSF Aβ42/40 and CSF Aβ42 were established as the lowest distance to the top left corner in the ROC curve. We used the same method to determine cutoffs for CSF Tau and CSF p-Tau 181 using flortaucipir tau PET imaging (tau positive if SUVR ≥1.32). The agreement between the algorithm and the PET method to determine cutoffs was examined using Cohen's kappa coefficient[38] and the overall percent agreement, defined as the number of true positive and true negative divided by the total number of participants.

Figure 1

Procedures Used in the Application of the Algorithm

Aβ = β-amyloid; ADNI = Alzheimer's Disease Neuroimaging Initiative; AUC = area under the ROC curve; p-Tau 181 = tau phosphorylated at threonine 181; ROC = receiver operating characteristic.

In a second step, we applied our algorithm (Table 1) to 10 patient cohorts drawn from memory clinics. We compared the AUC of ROC curves between CSF Aβ42/40 ratio and CSF Aβ42 for discriminating between high and low levels of CSF p-Tau 181 using a nonparametric approach based on an estimated covariance matrix (“roccomp” command in Stata).[39] We estimated the reliability of the cutoffs using the following rule: strong reliability if the 3 AUCs used for the cutoff determination were higher than 0.85, medium reliability if at least 1 AUC used was between 0.75 and 0.85, and low reliability if at least 1 AUC was lower to 0.75.

We then applied the cutoffs established to determine the proportion of CSF biomarker profiles in each cohort of patients using the AT(N) classification: A+ (CSF Aβ42/40 ratio or CSF Aβ42 lower than the cutoff) and T+ (CSF p-Tau 181 higher than the cutoff).

As the thresholds used to determine low and high levels of p-tau 181 (step 2, Table 1) and Aβ markers (step 5, Table 1) are somewhat arbitrary, in sensitivity analysis we examined other thresholds to test the robustness of the algorithm. These analyses were undertaken on the ADNI to compare results with tau and amyloid PET criteria.

All resulting p values were 2-tailed, and p < 0.05 was considered statistically significant. Statistical analyses were performed using Stata Statistical Software: Release 14 (StataCorp LP, College Station, TX).

Data Availability

Data are available for the purposes of replicating procedures and results from the corresponding author on request.

Results

Characteristics of the Participants

A total of 6,922 patients from 11 cohorts with data on CSF biomarkers were included in this study; their characteristics are shown in eTable 1 (links.lww.com/WNL/C64). The mean (SD) age of patients ranged from 62.8 (7.1) to 72.7 (8.0) years, the mean (SD) Mini-Mental State Examination score was between 20.0 (5.7) and 27.2 (2.0), and the proportion of women was from 43.3% to 56.2%. The percentage of patients with dementia in the various cohorts ranged from 13.3% to 54.6%. Fujirebio Lumipulse was used in 4 cohorts, Fujirebio INNOTEST and Roche Elecsys in 3 cohorts, and Euroimmun in 1 cohort. The distribution of CSF biomarkers in all cohorts is shown in eFigure 1.

Validation of the Algorithm in the ADNI Study

In the ADNI, the CSF biomarkers were assessed using Elecsys immunoassays; the mean (SD) delay between CSF biomarker assessment and tau PET imaging was 0.77 (1.9) years and 2.9 (2.8) years for amyloid PET imaging. Table 2 shows the AUC and corresponding optimal cutoffs for CSF biomarkers in the ADNI study using 2 methods: 1 based on amyloid and tau PET imaging and 1 based on our algorithm. The agreement between these 2 methods was high, with Cohen's kappa coefficient greater than 0.85 (range 0.87–0.99) and overall percent agreement greater than 0.90 (range 0.93–0.99) for all biomarkers. The confusion matrix of the classification of ADNI participants using the 2 methods is shown in eTable 2 (links.lww.com/WNL/C64).

Table 2

CSF Biomarker Cutoffs in the ADNI Study Based on Amyloid and Tau PET Imaging and Our Algorithm

CSF Aβ Markers to Discriminate Between High and Low CSF p-Tau Levels

The ability of CSF Aβ42 and CSF Aβ42/40 ratio to discriminate “high CSF p-Tau 181” from “low CSF p-Tau 181” in the 10 patient cohorts is presented in eTable 3 (links.lww.com/WNL/C64). The AUC associated with CSF Aβ42/40 ratio ranged from 0.86 to 0.99, whereas the AUC associated with CSF Aβ42 ranged from 0.55 to 0.87. In all centers, CSF Aβ42/40 ratio outperformed CSF Aβ42 in discriminating high from low CSF p-Tau 181 (p < 0.001); Figure 2 illustrates the comparison of ROC curves for these 2 markers in 4 cohorts.

Figure 2

Ability of CSF Aβ42/40 Ratio (Red) and CSF Aβ42 (Blue) to Discriminate Between High and Low Levels of CSF p-Tau 181

ROC curve analysis. Aβ = β-amyloid; ADNI = Alzheimer's Disease Neuroimaging Initiative; AUC = area under the ROC curve; p-Tau 181 = tau phosphorylated at threonine 181; ROC = receiver operating characteristic.

CSF Tau and p-Tau to Discriminate Between High and Low CSF Aβ Levels

eTable 4 (links.lww.com/WNL/C64) shows the AUC corresponding to CSF Tau and CSF p-Tau 181 to discriminate between “low Aβ amyloid” and “high Aβ amyloid.” Overall, CSF p-Tau 181 was associated with high AUC values to discriminate low from high CSF Aβ42/40 ratio (range from 0.84 to 0.97), whereas slightly lower AUCs were observed for CSF Tau (range from 0.79 to 0.90).

Application of the Algorithm in the Patient Cohorts From Memory Centers

CSF biomarker cutoffs identified by the proposed algorithm are presented in Table 3. For CSF Aβ42, the cutoffs ranged from 505 to 978 pg/mL, depending on the center and the technique used. The reliability of the cutoffs was strong for 7 of the 10 cohorts, medium in 2 cohorts, and low for 1 of them.

Table 3

Optimal CSF Biomarker Cutoffs in Each Center, Sorted According to the Technique Used in the Analyses

The proportion of CSF AD profiles (A+/T+) in each center is shown in eFigure 2 (links.lww.com/WNL/C64) and ranged from 25.9% to 43.5% of persons seen in these centers.

Sensitivity Analyses

We reran the analyses using other thresholds for defining high/low levels of phosphorylated tau markers (step 2, Table 1) and beta-amyloid peptide markers (step 5, Table 1) in the algorithm; results are shown in eTables 5 and 6 (links.lww.com/WNL/C64). Overall, these analyses did not show improvement in the thresholds chosen in our algorithm.

Discussion

Using a large, multicenter study of around 6,000 participants, we propose a new method to determine cutoffs for CSF biomarkers in clinical settings, which was validated against amyloid and tau PET imaging. Our method has the advantage of being applicable in other research settings because it is based on simple statistical analysis and does not require clinical or biomarker data other than CSF biomarkers. Our method, which consists of proposing a method to homogenize the determination of cutoffs, will allow greater transparency in the use of biomarkers for the diagnosis of AD. Two main lessons can also be learnt from our results. One, despite the recent development of automated assays, there remains a significant variation in absolute biomarker thresholds between sites, and further efforts to standardize procedures should be pursued, particularly for preanalytic parameters. Therefore, our algorithm did not aim to provide universal cutoffs for CSF biomarkers. Two, our results plead for the use of CSF Aβ42/40 ratio instead of CSF Aβ42 alone, at least for the identification of patients with fibrillar tau pathology. The amyloid ratio was excellent at discriminating between individuals with high and low phosphorylated tau levels, the AUC was higher than 0.95 in most of the centers, whereas CSF Aβ42 alone had lower discrimination. The extent to which this translates to diagnostic superiority of CSF Aβ42/40 ratio against CSF Aβ42 alone in clinical settings remains to be demonstrated.

Defining biomarker thresholds is a common challenge in medicine, but it is particularly challenging for AD diagnosis because the difference between normal and pathologic conditions is not always clear, and there is great variability in measured biomarkers. The current gold standard uses amyloid PET imaging to determine CSF Aβ cutoffs. However, this method requires identification of positive and negative cases based on PET results, which raises questions on how to define cutoffs for PET and the accuracy of such definitions. A further concern is that several studies show discrepancies between amyloid PET imaging and CSF Aβ assessment[40] because the latter can show abnormalities earlier in the disease process reflected in the low value for CSF Aβ42 and normal amyloid PET imaging.[41]

Cutoffs based on clinical diagnosis (AD vs non-AD categorization) have also been proposed, but it has limitations because of the lack of specificity of diagnostic criteria, with approximately 30% of false positives compared with neuropathologic findings.[4] Phosphorylated tau appears to be the most specific marker of AD, despite elevated levels in rare conditions such as chronic traumatic encephalopathy.[42] Low levels of CSF Aβ42 have been reported in other frequent causes of dementia, including Lewy body disease[43] and vascular dementia.[44] Increasingly, attempts are being made to identify blood-based biomarkers of AD,[45] particularly phosphorylated tau isoforms in plasma.[46] Whether blood-based biomarkers are useful in clinical settings for the diagnosis of patients remains unclear,[47] particularly for determining cutoffs. Our approach based on cross-validation between biomarkers could be useful in this context.

Our approach was based on cross-validation of biomarkers by first determining the ability of Aβ markers to discriminate between high vs low levels of phosphorylated tau. The cross-validation of biomarkers has been used previously for defining imaging biomarker cutoffs, using the results of amyloid PET imaging to define tau PET, FDG-PET, and structural MRI biomarker cutoffs.[33] A disadvantage of this approach is that individuals with AD can have variable degrees of tau and Aβ pathology, and the approach we used may misclassify some individuals. The existence of multiple pathologies, involving proteins such as TDP-43 or alpha-synuclein, may contribute to the clinical expression of disease but is unlikely to affect AD classification.[48,49] Our aim was not to compare CSF biomarkers with PET imaging for AD diagnosis. Both techniques have their advantages and disadvantages, and both can be used to define the A/T/N status.[10] Although PET imaging is informative on localization of neuropathologic lesions as well as change therein over time, CSF biomarkers are more readily available in many diagnostic centers because of cost and feasibility issues.

The main strength of this study is elaboration of a pragmatic method to determine cutoffs for CSF biomarkers of AD so that it can be readily replicated in other centers. The validity of the algorithm was established by comparing findings with amyloid and tau PET imaging. There are also a number of limitations. One, CSF biomarker assessment and PET imaging were not undertaken at the same time in ADNI and whether this affects determination of cutoffs is unclear. It is worth noting that few studies have longitudinal data, and they show slow change in CSF AD biomarkers.[50] Two, we used PET biomarkers to validate the algorithm, but tau and amyloid imaging positivity remains somewhat arbitrary, and a true gold standard for identifying AD and non-AD is lacking. Premortem CSF assessment and neuropathologic confirmation would be useful in future studies. Three, the data in our analyses did not come from a centralized assessment of CSF biomarkers. However, our objective was not to propose a universal cutoff but a standardized method that can be used to determine cutoffs in each study. Four, we assumed that the distribution of CSF p-Tau among patients offers sufficient variability to establish reliable cutoffs for CSF Aβ markers. Five, the algorithm is best suited for use in memory clinics with sufficient proportion of patients with AD and non-AD, but whether this method is suited for other settings, for example, a population of at-risk older adults, remains to be determined. Finally, many parameters such as age, APOE e4 status, or the stage of disease are likely to affect biomarker levels, and how these parameters affect the diagnosis of AD needs to be investigated in future studies.

To conclude, we propose a novel, pragmatic method to determine CSF AD biomarker cutoffs in clinical settings, which does not require assessment of other biomarkers or assumptions concerning the clinical profile of patients. The underlying reasoning behind our approach is that a common method for determining cutoffs will be useful in reducing heterogeneity in research and clinical settings that undertake research on AD. Our results suggest that use of CSF Aβ42/40 ratio instead of or in addition to CSF Aβ42 alone should be promoted to determine the Aβ status based on CSF biomarkers.