Interlaboratory validation of cerebrospinal fluid α-synuclein quantification in the diagnosis of sporadic Creutzfeldt-Jakob disease.

INTRODUCTION: Cerebrospinal fluid α-synuclein level is increased in sporadic Creutzfeldt-Jakob disease cases. However, the clinical value of this biomarker remains to be established. In this study, we have addressed the clinical validation parameters and the interlaboratory reproducibility by using an electrochemiluminescent assay.
METHODS: Cerebrospinal fluid α-synuclein was quantified in a total of 188 sporadic Creutzfeldt-Jakob disease and non-Creutzfeldt-Jakob-disease cases to determine sensitivity and specificity values and lot-to-lot variability. Two round robin tests with 70 additional cases were performed in six independent laboratories.
RESULTS: A sensitivity of 93% and a specificity of 96% were achieved in discriminating sporadic Creutzfeldt-Jakob disease. No differences were detected between lots. The mean interlaboratory coefficient of variation was 23%, and the intralaboratory coefficient of variations ranged 2.70%-11.39%. Overall, 97% of samples were correctly diagnosed. DISCUSSION: The herein validated α-synuclein assay is robust, accurate, and reproducible in identifying Creutzfeldt-Jakob disease cases. Thus, it is ready for implementation in the clinical practice to support the diagnosis of Creutzfeldt-Jakob disease.

Background

α-Synuclein (aSyn) is a highly abundant presynaptic neuronal protein associated with the etiology of aSyn aggregation disorders such as Parkinson's disease and dementia with Lewy bodies [1]. The study of aSyn as a potential diagnostic marker in biological fluids has been mainly focused on these disorders, in which cerebrospinal fluid (CSF) aSyn shows a minor reduction. In this case, the diagnostic value of aSyn quantification remains from poor to modest depending on the cohort and methodological approach used [2], [3].

aSyn concentrations in biological fluids have been also scrutinized in neurological and neurodegenerative disorders with non-aSyn etiology. In this regard, quantification of CSF aSyn by new high-sensitive approaches such as chemiluminescent-based platforms or mass spectrometry allows the discrimination of sporadic Creutzfeldt-Jakob disease (sCJD), the most prevalent form of human prion disease, from other neurological and neurodegenerative conditions with high diagnostic accuracy [4], [5], [6], [7]. Moreover, a prognostic value for CSF aSyn quantification in sCJD cases has been recently suggested [5]. Although the precise reason for elevated CSF aSyn levels in sCJD is unknown, it is speculated that this phenomenon may be related to the massive synaptic damage occurring in prion diseases [8], [9].

Although the presence of elevated CSF aSyn levels in sCJD cases has been replicated in several cohorts and by different quantification methods [4], [7], [8], [10], [11], the implementation of diagnostic tests for clinical routine requires a standardization process to thoughtfully scrutinize interlaboratory reproducibility, assay robustness, and precision, as well as reference limits or diagnostically optimal cutoff values. Indeed, laboratory-to-laboratory differences are associated not only with different laboratory performance but also with variability between lots or to assay parameters such as robustness and stability [12].

In the present study, we tested the diagnostic accuracy of CSF aSyn quantification using a new electrochemiluminescence-based human aSyn assay from Meso Scale Discovery (MSD)TM (Gaithersburg, MD) in the discrimination of sCJD from non-CJD cases. Furthermore, lot-to-lot variability was assessed and interlaboratory reproducibility was determined through two round robin tests involving six laboratories from five European countries.

Methods and materials

Ethics

The study was conducted according to the revised Declaration of Helsinki and Good Clinical Practice guidelines and approved by local ethics committees.

Samples

All CSF samples used in this study were collected at the National Reference Center for Transmissible Spongiform Encephalopathies (University Medical Center Göttingen, Germany). Blood contamination in the samples was tested using the Hemastix strips (Siemens), and specimens containing more than 25 erythrocytes/mm3 and/or hemoglobin contamination were excluded from this study. Two sets of samples were evaluated. Initially, a total of 188 samples (83 non-CJD and 105 sCJD cases) were used for the establishment of the diagnostic parameters of CSF aSyn quantification in the discrimination of sCJD from non-CJD cases. sCJD cases were probable or definite according to established criteria [13], [14] and indicated in Fig. 1. Non-CJD cases used for this purpose were patients with neurological or neurodegenerative diseases other than prion disease. General neurological diseases (n = 68) included the following diagnoses: (1) psychosis; (2) bipolar disorder; (3) schizophrenia; (4) depression; (5) ischemia; (6) multiple infarct; (7) cerebral vasculitis; (8) epilepsy; (9) meningitis; (10) alcohol abuse; (11) vertigo; (12) acute or chronic headache; (13) pain syndromes; (14) acute hypoxia; (15) vascular encephalopathy; (16) cerebral lymphoma; (17) astrocytoma; and (18) paraneoplasia. Neurodegenerative diseases (n = 15) included the following diagnoses: (1) Alzheimer's disease (AD); (2) Parkinson's disease; (3) Parkinson's disease dementia; (4) dementia with Lewy bodies; (5) corticobasal degeneration; (6) frontotemporal dementia; and (7) vascular dementia.

Fig. 1

Interlaboratory validation of CSF aSyn quantification in the diagnostic context of sCJD. (A) Round robin tests in 40 CSF cases (n = 20 non-CJD and n = 20 sCJD) and (B) in 30 CSF cases (n = 15 non-CJD and n = 15 sCJD cases). Diagnosis, demographics (age and gender), aSyn concentration (mean value ± standard deviation), and coefficient of variability (CV%) for each case as well as percentage of laboratories reaching a correct diagnosis are indicated. Red numbers indicated either mean CSF aSyn values below cutoff or cases in which correct diagnosis was not achieved in all the laboratories. aSyn concentrations for each case were plotted. Red dashed line indicates cutoff value (1000 pg/mL aSyn). Abbreviations: CSF, cerebrospinal fluid; aSyn, α-synuclein; sCJD, sporadic Creutzfeldt-Jakob disease.

In addition, a total of 70 samples (35 non-CJD and 35 sCJD cases), for which aSyn levels were not previously evaluated, were used for the round robin tests. Diagnoses for non-CJD cases (neurological and neurodegenerative diseases) are stated in Fig. 1.

In all cases, neurological diseases were diagnosed according to International Classification of Diseases, 10th Revision, definitions and neurodegenerative diseases according to established diagnostic criteria [13], [15], [16], [17], [18], [19], [20].

Round robin tests

Aliquots of CSF samples (25 μL) were centrally collected and shipped under equal conditions (tubes, volumes, number of freezing/thawing cycles, and identical dry ice carrier overnight) to the participant laboratories. All laboratories were blinded to the diagnosis of the samples. Non-CJD and sCJD samples were randomly distributed over the assay plates to prevent any potential within-plate position bias.

CSF tests

CSF aSyn was quantified using two commercially available MSD aSyn kits: (1) K151TGD and (2) the newly developed U-plex aSyn assay (K151WKK). Assays were performed according to manufacturer's instructions using a 1:8 CSF dilution. Laboratory technicians from each laboratory were trained by MSD personnel before the performance of the round robin tests. CSF Tau concentrations and presence or absence of 14-3-3 protein were available for all cases and analyzed according to established protocols [21].

Statistical analysis

Mann-Whitney U tests were used to compare two groups of samples after testing for parametric distribution. To assess the diagnostic accuracy of CSF aSyn in the discrimination of sCJD from non-CJD cases, receiver operating characteristic curve analyses were carried out, and areas under the curve with 95% confidence intervals were calculated using GraphPad Prism 6.01. The best cutoff value was then estimated based on the Youden index (sensitivity + specificity − 1). Spearman rank correlation coefficients were used to assess associations between continuous biomarker levels. Agreement between MSD assays and between two different lots was investigated through a Passing-Bablok regression [22], using the MethComp package in R [23].

Results

Bridging assay and establishment of diagnostic parameters

We initially performed a bridging experiment between the previously commercially available MSD aSyn kit (K151TGD) and the newly developed MSD U-plex aSyn kit (K151WKK). The aim of this experiment was to assess differences in assay sensitivity among both the tests, as available cutoff values were previously determined using the K151TGD kit [5].

A total of 188 samples (83 non-CJD and 105 sCJD cases) were tested using both the assays. A high correlation was observed between the values obtained by both the methods (ρ = 0.99, P < .0001). However, mean values using U-plex assay in the 188 cases were 24% higher (5821 pg/mL aSyn) than those detected for the K151TGD assay (4713 pg/mL aSyn) (P < .001) (21% for non-CJD and 25% for sCJD). To compare the performance of both the assays, we conducted a Passing-Bablok regression analysis, which revealed a proportional bias between both the methods because the 95% confidence interval for the slope does not include 1 (Fig. 2A). Thus, although the commercial source stated that the sensitivities for both the kits were comparable, our results clearly indicated that the U-plex assay was more sensitive than the previous assay. This observation compelled us to establish new cutoff values for the discrimination of non-CJD from sCJD cases for the U-plex assay. aSyn values were significantly higher in sCJD (10,030 ± 9602 pg/mL aSyn) than in non-CJD cases (433 ± 252 pg/mL aSyn) (P < .001) (Fig. 2B), in agreement with previous reports [4], [5]. The area under the curve from receiver operating characteristic curves was 0.9935 (95% confidence intervals: 0.98–0.99). A cutoff value of 1000 pg/mL aSyn allowed discrimination of sCJD from non-CJD cases with a sensitivity of 93% and a specificity of 96% (Fig. 2C). The overall discrimination power of CSF aSyn was superior to that offered by CSF tau and 14-3-3 for the same set of samples (sensitivities of 93% and 92% and specificities of 92% and 94% for tau and 14-3-3, respectively).

Fig. 2

Establishment of diagnostic parameters for CSF αSyn quantification in the diagnosis of sCJD cases. (A) Passing-Bablok regression of the CSF aSyn quantification using two Meso Scale DiscoveryTM assays: MSD aSyn (K151TGD) and the MSD U-Plex aSyn (K151WKK). The 95% CI for the intercept and the slope are indicated. (B) CSF aSyn concentrations in non-sCJD and sCJD cases. Statistically significant differences were detected between non-CJD and sCJD cases (P < .001). Numbers of cases analyzed, mean, and standard deviation values as well as 95% coefficient interval values are indicated. (C) ROC curve for aSyn in the comparative analysis between non-sCJD cases and sCJD cases. Sensitivity and specificity, receiver operating characteristic (ROC) curves, and derived area under the curve (AUC) with 95% coefficient interval were calculated. Based on Youden Index, with an optimal cutoff of 1000 pg/mL aSyn, 93% sensitivity and 96% sensitivity was achieved in the discrimination of sCJD from non-CJD cases. Abbreviations: CSF, cerebrospinal fluid; aSyn, α-synuclein; sCJD, sporadic Creutzfeldt-Jakob disease; 95% CI, 95% confidence interval.

Lot-to-lot consistency

To validate lot-to-lot consistency for the U-plex assay, a total of 20 CSF cases (10 non-CJD and 10 sCJD) were tested using two different assay lots. As expected, sCJD samples showed increased aSyn values compared with non-CJD cases (P < .001) (Fig. 3A). No significant differences were observed when aSyn levels and coefficient of variation (CV) values from lot A to lot A analysis were compared to those obtained from lot A to lot B (P = .98 and P = .82, respectively) (Fig. 3A). A Passing-Bablok regression analysis indicated no significant bias between lot A and lot B (Fig. 3B).

Fig. 3

Assessment of lot-to-lot variability. (A) CSF aSyn concentrations analyzed in 20 CSF samples (10 non-CJD and 10 sCJD cases) using the same or different assay lots. sCJD samples showed increased aSyn values compared with non-CJD cases (P < .001). Mean aSyn values and mean interrun CV values derived from the analysis of the same set of samples in lot A and lot B are shown. P value is indicated. (B) Passing-Bablok regression of the CSF aSyn concentrations analyzed using two different lots. The 95% CI for the intercept and the slope are indicated. Abbreviations: CSF, cerebrospinal fluid; aSyn, α-synuclein; sCJD, sporadic Creutzfeldt-Jakob disease; CV, coefficient of variation; 95% CI, 95% confidence interval.

Round robin test

To assess interlaboratory reproducibility, two sets of CSF samples were tested in six laboratories with the same lot of the MSD U-plex aSyn kit. The samples delivered to the participants were not previously tested for aSyn and were selected exclusively according to their clinical diagnosis. The first round robin test included 40 cases (20 non-CJD and 20 sCJD cases) (Fig. 1A), whereas 30 cases (15 non-CJD and 15 sCJD) were used in the second test (Fig. 1B). Mean CSF aSyn concentrations derived from the measurements of the six participant laboratories were higher in sCJD than in non-CJD cases (P < .001) (Fig. 1A and B).

In the first test, coincidence on differential diagnosis based on the previously established cutoff values (1000 pg/mL aSyn) was reached in all but 3 cases. A non-CJD case diagnosed as AD (ID 18) tested positive in one of the laboratories with aSyn values slightly above cutoff (1073 pg/mL). This case was positive for 14-3-3 protein in the CSF, indicative of prion disease, but had a tau value below the sCJD cutoff (1300 pg/mL). In addition, two sCJD cases (ID 24 and 25) were not correctly identified by 3 and 4 laboratories, respectively. Both the cases presented border-line levels for sCJD tau cutoff, but elevated 14-3-3 was detected in the CSF of both the cases.

In the second test, full agreement except in one sCJD sample (ID 22) was reached (Fig. 1B). For this case, 4 out of 6 laboratories missed the diagnosis of sCJD based on CSF aSyn concentrations. Interestingly, although sCJD diagnosis for this case had a neuropathological confirmation, CSF 14-3-3 and total tau tested negative.

The overall percentage of samples correctly diagnosed as non-CJD or sCJD cases among all laboratories was 97%. When stratified by laboratories, participant percentages were 97% (laboratory 1), 96% (laboratory 2), 98% (laboratory 3), 98% (laboratory 4), 96% (laboratory 5), and 97% (laboratory 6). Mean interlaboratory CV values were 25% and 20% for the first and second round robin tests, respectively. The mean intralaboratory CV ranged from 2.70% to 11.39% (mean value = 5%) (Fig. 4), and these differences were not associated with a differential percentage of correctly diagnosed cases.

Fig. 4

Intralaboratory coefficient of variability achieved by different laboratories in the quantification of CSF aSyn. Graphical visualization and mean ± standard deviation values on the intralaboratory coefficient of variability (CV) achieved by the six participant laboratories. Abbreviations: CSF, cerebrospinal fluid; aSyn, α-synuclein.

Discussion

The assessment of interlaboratory performance is a key step in the validation process before the introduction of a new biomarker in clinical practice and/or its incorporation in diagnostic criteria. In the field of neurodegenerative disease, several biological fluid biomarkers, especially those derived from CSF, are currently used as supportive tools in the clinical diagnosis of several diseases. In AD, CSF total tau, phospho tau, and amyloid β 42 quantification support AD diagnosis and are part of the most updated diagnostic criteria [19], [24]. In sCJD, CSF tau, 14-3-3, and the real-time quaking-induced conversion (RT-QuIC) are established tests in clinical practice in prion surveillance units [13], [25], [26], although only 14-3-3 is present in the World Health Organization criteria for probable sCJD [27]. While tau and 14-3-3 are surrogate markers of neuronal damage, the RT-QuIC assay detects the presence of abnormal prion protein, and therefore, it is a test associated to the primary causative agent of the prion pathology.

Surprisingly, although huge efforts have been carried out in the study of preanalytical and analytical conditions affecting biomarker outcomes [28], [29], [30], [31] and consensus guidelines have been reported [12], [32], just a few studies have reported the performance of a given set of samples in different laboratories, particularly in the case of aSyn [25], [33], [34], [35], [36]. Furthermore, among these studies, some of them are limited by using low numbers of cases, whereas others did not study the performance of the assays on their diagnostic context. The later point is of special importance as interlaboratory assessment would gain benefit if it comes along with the study of their clinical applicability, namely, on the degree of agreement of differential laboratories in reaching a correct diagnosis.

Several indications suggested that the U-plex human aSyn kit test would be a good candidate to validate its potential interlaboratory performance in the diagnosis of sCJD as the final step before its introduction in clinical practice. On one hand, we previously demonstrated the increased sensitivity of electrochemiluminescence platforms over classical colorimetric assays in the quantification of CSF aSyn [6], leading to a better discriminatory power between sCJD and non-CJD cases. On the other hand, according to the manufacturer, the U-plex human aSyn kit was developed following “fit for purpose” principles [37] and is consistent with guidance from the Clinical and Laboratory Standards Institute (www.clsi.org). The certificate of analysis provided in the kit indicates specifications for sensitivity, specificity, accuracy, and precision. In addition, the assay is validated for robustness, stability, matrix affects, and samples.

First, we demonstrated that the U-Plex assay was more sensitive in the detection of aSyn levels than the predecessor test from the same manufacturer. Therefore, it was mandatory to determine the diagnostic parameters for the discrimination of non-CJD from sCJD cases for the U-Plex kit. With a cutoff value of 1000 pg/mL aSyn, the U-Plex assay was able to discriminate non-CJD from sCJD cases with 93% sensitivity and 96% specificity, which is better than the discrimination performed based on tau and 14-3-3. The sensitivity and specificity values herein presented for CSF aSyn are in range with those previously reported for the K151TGD assay [5], for an aSyn in-house assay [4], and of the prion biomarkers showing the higher diagnostic accuracy such as RT-QuIC and p-tau/tau ratio [26], [38], [39].

According to manufacturer's specifications, the setup of the assay provides a lot-to-lot consistency, potentially solving a current problem in diagnostic centers where periodic re-evaluation and validation of diagnostic parameters need to be performed after a new lot is supplied by commercial supplier [40]. We were able to validate the manufacturer's statement, as we found a substantial agreement between the performances of two lots. In addition, CVs were not statistically different when the same samples were analyzed in the same lots or in different lots.

A salient finding from our study is the high agreement achieved between laboratories in reaching a correct diagnosis based on CSF aSyn levels. It is worth to mention that those cases not correctly classified were in all the cases misdiagnosed by more than one laboratory and presented, in most of the cases, a nonclassical (atypical) CSF profile regarding tau and/or 14-3-3, even those with neuropathological confirmation. This indicates that these cases, blindly selected for CSF biomarker profile, could also have missed the diagnosis based on currently implemented CSF tests. Nevertheless, the overall agreement among laboratories for all measurements and cases was higher, being 97% with CV of 25% and 20% in a total of six independent laboratories.

A worldwide multicentre comparison of assays for CSF biomarkers in AD reported inter-CV of 31%, 21%, and 13% for amyloid β 42, tau, and P-tau, respectively, with relatively high intra-CV values (7%–25%). Another study on CSF AD biomarkers shows large interlaboratory variability, likely caused by factors related to analytical procedures and the analytical kits, with inter-CV ranging from 13% to 36% [41]. Regarding aSyn, a recent worldwide multicenter comparison was performed in 17 laboratories. This study reported comparable results with acceptable variation of about 20% CV relative to the results from a reference laboratory among most of the participating laboratories. However, there was high variation in absolute values of CSF aSyn when the same samples and same lots of assays are applied [35].

The limited amount of studies in the field of aSyn biomarkers with a similar setup as the present study (high number of samples and laboratories) impedes a precise comparison between studies. However, several observations indicate that the aSyn U-Plex kit is suitable for (research) application in the diagnostic context of sCJD. First, the degree of agreement in achieving a correct diagnosis based on a cutoff value established in one of the laboratories (laboratory 1) was high. Second, the agreement in reaching an accurate diagnosis was independent of the intralaboratory CV, as laboratories with high intralaboratory CV showed the same accuracy to those with low CV values. Third, interlaboratory CVs (≈20%) were similar to those reported in other studies [33], [36], whereas mean intralaboratory values were low (5%). At this point, the lack of MSD platform in some clinical routine laboratories can be considered the main impediment preventing the widespread application of this test. Another limitation of this study is that not all CJD cases had a definite diagnosis by means of neuropathological assessment. However, this was not a source of bias in our study because all probable sCJD cases were correctly identified in all the laboratories.

In total, we have validated a robust assay to measure CSF aSyn, characterizing its consistency and accuracy in identifying CJD cases as well as acceptable precision values in the evaluation of interlaboratory and intralaboratory comparison. The overall superior discrimination potential of this assay in the differential diagnosis over other methods (14-3-3 and tau quantification or RT-QuIC) currently used in the clinical routine is a prominent hallmark of our studies, which together with its easy, rapid, and cost-effective performance supports the immediate implementation in the clinical practice. Hence, we expect that our findings will provide the opportunity in the short term to exploit this ready-to-use assay as a valuable tool in the diagnosis of CJD.

Systematic review: The quantification of cerebrospinal fluid (CSF) α-synuclein (aSyn) by new high-sensitive approaches such as chemiluminescence-based platforms allows the discrimination of sporadic Creutzfeldt-Jakob disease (sCJD) from other neurological and neurodegenerative conditions with high diagnostic accuracy.

Interpretation: We evaluated the diagnostic accuracy of CSF aSyn quantification by a new chemiluminescent human aSyn assay in the discrimination of sCJD from non-CJD cases. Lot-to-lot variability was assessed and interlaboratory reproducibility determined through two round robin tests involving six laboratories from five European countries. The high degree of agreement between laboratories reaching a correct diagnostic, lot-to-lot bridging and high diagnostic accuracy of the assay in discriminating sCJD cases supports the implementation of the hereby evaluated test into clinical routine in prion disease diagnostic centers.

Future directions: Further studies using independent large study populations based on the proposed cutoff value will help to confirm the accuracy of this test in clinical practice.