OCT Variability Prevents Their Use as Robust Biomarkers in Multiple Sclerosis.

OBJECTIVE: To evaluate the agreement between the peripapillary retinal nerve fiber layer (pRNFL) and foveal thickness (FT) measurements among three different spectral domain-optical coherence tomography (SD-OCT) instruments in a sample of multiple sclerosis (MS) patients and a healthy age-matched control group.
METHODS: An observational cross-sectional study with three groups: healthy subjects and MS patients w/w a previous clinical diagnosis of optic neuritis (ON) was conducted. The pRNFL and FT were measured using three different SD-OCT instruments (OCT PRIMUS 200 and OCT CIRRUS 500 SD-OCT [Carl Zeiss Meditec] and OCT 3D 2000 [Topcon]).
RESULTS: Twenty eyes from 10 healthy subjects matched in age with MS patients without a previous history of eye disease and 62 MS eyes from 31 MS patients (29 eyes without history of ON and 33 eyes with history of ON) were enrolled. Healthy subjects and MS patients without ON did not show differences between the pRNFL and FT thickness (P>0.99) with any of the instruments. However, MS eyes with a previous episode of ON showed thinner pRNFL and FT (P<0.01). PRIMUS and CIRRUS OCT showed better agreement of the pRNLF and FT in both healthy and MS eyes. However, 3D OCT showed less agreement in the pRNFL measurement with CIRRUS in both healthy and MS eyes.
INTERPRETATION: Although OCT is a valuable technology to improve MS patient assessment, differences between devices must be taken into account. It is necessary to create an international group that standardizes the measurement conditions and above all that provides reference bases for normal subjects.

Introduction

Optical coherence tomography has demonstrated that many multiple sclerosis (MS) patients show significant thinning of the inner retinal layers,1–6 even without a past history of clinical optic neuritis (ON).7–10 Even more, analysis of a large-scale cross-sectional data set of MS patients by spectral-domain (SD)-OCT suggested that findings may help to identify distinct patterns in different MS subtypes.4 A recent meta-analysis has proposed that OCT scans from two different ocular regions (optic disc and macular area) should be routinely included in MS clinical practice, research, and trials,2 because OCT could be a predictive biomarker of disease duration, clinical assessment and even for assessing drug efficacy (in patients without ON).10 Furthermore, detectable retinal axonal loss is assumed to occur early in the course of the MS disease.11

Currently SD-OCT is considered to be the most suitable technique and outperforms Visual Evoked Potential (VEP) and B-mode transorbital sonography (TOS) to detect subclinical damage discriminating MS patients from healthy controls. It also shows a progressive decline in optic nerve thickness over time in MS patients.11

Therefore OCT has been accepted as a non-invasive method able to quantify intraretinal layer volumes and it is widely used to quantify the peripapillary retinal nerve fiber layer (pRNFL) and foveal and macular thickness (FT/MT), providing a clinical objective method for monitoring axonal injury in MS and other neurodegenerative diseases.11,12 However, it does not seem that at the clinical level this ancillary test has been incorporated in a routine way in most hospitals.

Different OCT technologies are currently available, namely time domain (TD-OCT) technology, SD-OCT technology, swept source (SS-OCT) technology and others which are in development.13–15 Technological evolution has been so accelerated in recent years, significantly improving the sensitivity of the equipment, and some reference values that exist in the literature have been questioned preventing the ability to compare new studies.16

As already mentioned, the critical parameters in the diagnosis and management of patients with MS are pRNFL and FT, but these measures have shown significant differences from one instrument to another17,18 providing non-interchangeable values in healthy eyes and in MS patients,19 even when SD-OCT devices have been compared to TD-OCT or even between SD-OCT devices.17,18

The magnitude of this variability has not received enough attention and could represent a serious limitation when making comparative analysis with previously published results or when designing multi-center studies with different equipment, even if they are based on SD-OCT.

Thus, the purpose of this study was to evaluate the agreement and the coefficients of variation between pRNFL and FT measurements collected by three commercially-available SD-OCT instruments in a sample of MS patients w/w clinical history of ON and a control group of age-matched healthy subjects.

Methods

Design

This study was an observational cross-sectional study.

Participants

MS patients, diagnosed following internationally accepted standards,20 were recruited from the Neuro-ophthalmology (Ophthalmology) and Demyelinating Diseases (Neurology) Units of the University Hospital Clinico of Valladolid (Valladolid, Spain). Data from both eyes of patients in any MS stage (progressive and relapsing-remitting stage), between 20 and 60 years of age with no concurrent disease other than MS, were collected. Patients with retinal diseases, glaucoma and high refractive errors (myopia higher than 4 dioptres) were excluded after extensive ophthalmological examination. Also, MS patients with suspected ON episode within the previous 3 months were also excluded. Healthy subjects were recruited to get a sample age and sex matched with MS patients. Eye examinations of that group should be fully normal, except for the existence of a refractive error less than 4 diopters.

Eyes were divided into the following three study groups: Group 1: healthy eyes; Group 2: MS eyes with a previous clinical history and diagnosis of ON with more than three months since the acute event; and Group 3: MS eyes without a previous clinical history of ON. ON was defined by the presence of retro-ocular/orbital pain and a sudden decrease in visual acuity accompanied by altered contrast sensitivity and color tests, and the presence of relative afferent pupillary defect at the time of the episode.

Approval of the Valladolid-East Health Area Institutional Review Board (IRB)/Ethics Committee was obtained in June 2017. Written informed consent was obtained previously from all participants enrolled in the study. All subjects were treated in accordance with the Declaration of Helsinki.

Instrumentation

The following data were collected from all eyes included in the study: best-corrected Snellen visual acuity (6 meters), color vision (Roth 28-hue test, Richmond Product, Albuquerque, USA), contrast sensitivity (CSV-1000E test, Vectorvision, Ohio, USA), standard automated perimetry SITA 30–2 using the Humphrey visual field analyzer (Carl Zeiss Meditec, Jena, Germany) and the pRNFL and FT measured with three different OCTs (OCT PRIMUS 200 [Carl Zeiss Meditec, Jena, Germany]; OCT CIRRUS 500 SD-OCT [Carl Zeiss Meditec, Jena, Germany] and OCT 3D 2000 [Topcon Corporation, Tokyo, Japan]).

The OCT PRIMUS 200 and the CIRRUS 500 SD-OCT use a super-luminescent diode (SLD) laser with a center wavelength of 840 nm (27,000 to 68,000 A-scan/sec). TOPCON 3 D 2000 is also an SD-OCT that uses 840 nm SLD and provides structural tomography (27,000 A-scan/sec) and high resolution (12.3 MP) color fundus photography images (45 degrees field of view) in a single capture.

Measurement Procedures

All OCT measurements were conducted following APOSTEL21 recommendations, and the manufacturer’s guidelines in a darkened room, after pupil dilatation (by tropicamide 1%) by three experienced operators. All devices were calibrated before the study. Measurements of different OCTs were collected at same day and daytime in each patient.

The pRNFL was measured in a circle with a diameter of 3.46 mm around the head of the optic nerve in a 255-A individual scan using OCT PRIMUS and 3D OCT TOPCON. Additionally, a three-dimensional scan of a 6 × 6-mm2 area was centered on the optic disc using the information from a 1024 (depth) × 200 × 200-point parallelepiped (predetermined by the manufacturer) with CIRRUS OCT. The mean pRNFL thickness was used as a final value.

The FT was measured in an image that was taken using the nine sectors grid based on the ETDRS with three concentric circles of 1, 3 and 6 mm in diameter. The ETDRS grid was centered on the fovea using the software of each OCT. The retinal thickness provided by each equipment of the central circle (fovea) was used as the final value.

Any scans were excluded if the image signal was less than 7 over 10 or if the images showed artefacts that were inaccurate to guarantee the quality of the measurements (such as, decentered scans, algorithm failure, poor illumination and non-homogeneous reflectivity) according with OSCAR-IB22 quality criteria.

Statistical Analysis

Statistical analysis was performed using the R version 3.5.1. (02/07/2018) [R Core Team, 2018]. A normal distribution of variables was assessed using the Shapiro–Wilk test (P-values >0.05 indicated that data were normally distributed). The mean values of the pRNFL and FT measurements between healthy and MS eyes (w/w history of ON) were calculated. Differences between the pRNFL and MS in healthy and MS patients (w/w history of ON) were assessed using analysis of variance (ANOVA) and posterior pairwise (post hoc) analysis between groups using the Student’s t-test and the Bonferroni correction (P-values <0.05 were considered statistically significant).

Agreement between the pRNFL and FT in healthy and MS eyes was evaluated using Bland-Altman analysis. The differences between the measurements of both devices were plotted against the means of both techniques, and 95% limits of agreement (LoA) were calculated (mean difference ± 1.96 standard deviation [SD]). The intra-class correlation coefficient (based on analysis of variance [ANOVA]) was also calculated and a 95% interval of confidence for ICC was also calculated.

Results

Twenty healthy eyes from 10 healthy subjects (mean age of 36.80±11.72 years, range 22–53 years; 60% women and 40% men) without a previous history of eye disease and 62 MS eyes from 31 MS patients (mean age of 34.45±10.31 years, range 28–51 years; 56% women and 44% men; 29 eyes without a history of ON mean age of 37.96±10.83 years 60% women and 40% men and 33 eyes with ON history mean age of 34.80±7.25 69% women and 31% men) were enrolled in this study. Study groups showed non-statistically differences in age (P=0.954) and sex distribution (P=0.835). Just Caucasian patients were enrolled. All MS patients were explored by CIRRUS and PRIMUS but only data of 26 MS eyes (9 without a history of ON and 17 with a history of ON) could be collected by Topcon 3D OCT due to the impossibility of some patients to repeat the three tests on the same day and at similar times.

Statistical differences were found between healthy and MS eyes in pRNFL (CIRRUS F=9.23, P<0.01; PRIMUS F=13.65, P<0.01 and 3D Topcon OCT F=8.73, P<0.01) and in FT (CIRRUS F=5.07, P=0.01; PRIMUS F=4.36, P=0.01) except with 3D Topcon OCT (F=2.15, P=0.13). Healthy eyes and MS eyes without a previous ON history showed similar pRNFL and FT results (P>0.99) using the three OCT devices. MS eyes with a previous episode of ON showed pRNFL and FT thinner than healthy eyes and MS eyes without ON history. Table 1 summarizes the OCT measurements in all healthy and MS eyes.

Table 1

Summary of pRNFL and FT Measurements (Mean ± Standard Deviation and 95% Interval Confidence)

	CIRRUS	PRIMUS	3D OCT*
Retinal nerve fiber layer (pRNFL), μm
Healthy	91.91±8.66 (87.86 to 95.96)	93.75±9.92 (89.11 to 93.39)	96.64±7.48 (92.92 to 100.36)
MS without ON	91.34±8.55 (88.09 to 94.59)	91.96±9.07 (88.51 to 95.41)	96.58±7.03 (91.18 to 101.99)
MS with ON	79.41±9.66 (75.98 to 82.83)	78.08±10.76 (74.90 to 81.27)	80.10±14.90 (72.44 to 87.76)
P (Healthy – MS without ON)	>0.99 (T27=−0.78)	>0.99 (T27=−0.18)	>0.99 (T16=−0.05)
P (Healthy – MS with ON)	<0.01 (T35=3.59)	<0.01 (T35=4.71)	<0.01 (T22=4.10)
P (MS without - with ON)	<0.01 (T24=3.66)	<0.01 (T24=4.08)	<0.01 (T24=3.82)
Foveal Thickness (FT), μm
Healthy	290.71±21.72 (280.55 to 300.88)	284.37±19.62 (275.19 to 293.55)	267.91±16.48 (260.20 to 275.63)
MS without ON	288.34±15.74 (282.35 to 294.32)	287.67±16.19 (281.51 to 293.82)	272.92±15.32 (261.15 to 284.72)
MS with ON	274.54±18.75 (266.96 to 282.11)	275.26±15.39 (269.80 to 280.71)	260.29±15.05 (270.65 to 283.42)
P (Healthy – MS without ON)	>0.99 (T27=0.55)	>0.99 (T27=−0.22)	>0.99 (T27=−0.77)
P (Healthy – MS with ON)	0.01 (T35=2.26)	0.06 (T35=2.52)	0.44 (T35=1.45)
P (MS Without - with ON)	0.01 (T24=2.38)	<0.01 (T24=2.49)	0.17 (T24=2.02)

Note: *3D OCT values from 26 MS eyes (9 without a history of ON and 17 with a history of ON).

Abbreviations: MS, multiple sclerosis; ON, optic neuritis.

PRIMUS and CIRRUS OCT showed better agreement for the pRNFL (Figure 1 and Table 2) than for the FT (Figure 2 and Table 2) in healthy and MS eyes. However, 3D OCT showed worse agreement for the pRNFL measurement with CIRRUS and the FT in healthy and MS eyes (Figure 3 and Table 2). The FT measurements of 3D OCT showed worse agreement in both healthy and MS eyes (Figure 4). Table 2 summarizes the agreement of the pRNFL and FT measurements. Similar trend was found between eyes with and without ON (Table 3).

Figure 1

Bland-Altman plots depict the agreement between PRIMUS and CIRRUS for RFNL measurement in healthy (top) and MS patients’ eyes (bottom).

Table 2

Summary of OCT Agreement for pRNFL and FT Measurements (Mean Difference ± Standard Deviation)

	Healthy	P	MS	P
Retinal nerve fiber layer (pRNFL), μm
PRIMUS versus CIRRUS	1.69±4.32LoA −6.94 to 10.33r2=0.20 P= 0.48ICC: 0.82 (0.59–0.92)	0.11 (T17=1.67)	0.03±4.48LoA −8.93 to 9.00 r2=0.06 P= 0.65ICC: 0.94 (0.91–0.97)	0.96 (T59=0.06)
3D OCT* versus CIRRUS	3.38±5.36LoA −7.35 to 14.10r2=0.08 P= 0.75ICC: 0.66 (0.33–0.85)	0.02 (T17=2.67)	−0.28±8.9LoA −18.09 to 17.53r2=0.51 P< 0.01ICC: 0.70 (0.29–0.88)	0.87 (T25=−0.16)
Foveal Thickness (FT), μm
PRIMUS versus CIRRUS	−6.34±10.66LoA −27.66 to 14.97r2=0.20 P= 0.39ICC: 0.83 (0.63–0.93)	0.02 (T19=−2.66)	−1.74±3.34LoA −8.42 to 4.95r2=0.13 P= 0.33ICC: 0.97 (0.96–0.98)	<0.01 (T56=−3.92)
3D OCT* versus CIRRUS	−22.8±12.38LoA –47.56 to 1.96r2=−0.34 P= 0.05ICC: 0.33 (−0.11–0.67)	< 0.01 (T19=−8.24)	−12.37±6.27LoA −24.9 to 0.16r2=0.04 P= 0.85ICC: 0.67 (0.40–0.82)	<0.01 (T25=−10.1)

Note: *3D OCT values from 26 MS eyes (9 without a history of ON and 17 with a history of ON).

Abbreviations: LoA, limits of agreement; ICC, Intraclass coefficient of correlation (95% interval of confidence for ICC).

Figure 2

Bland-Altman plots depict the agreement between PRIMUS and CIRRUS for FT measurement in healthy (top) and MS patients’ eyes (bottom).

Figure 3

Bland-Altman plots depict the agreement between Topcon and CIRRUS for RFNL measurement in healthy (top) and MS patients’ eyes (bottom).

Figure 4

Bland-Altman plots depict the agreement between Topcon and CIRRUS for FT measurement in healthy (top) and MS patients’ eyes (bottomp).

Table 3

Summary of OCT Agreement for pRNFL and FT Measurements in MS Patients with and without on (Mean Difference ± Standard Deviation)

	Without ON	P	With ON	P
Retinal nerve fiber layer (pRNFL), μm
PRIMUS versus CIRRUS	0.97±4.09LoA −7.04 to 8.98r2=0.94 P= 0.01ICC: 0.91 (0.82–0.96)	0.21 (T28=1.28)	−1.34±5.21LoA −11.55 to 8.87r2=0.84 P< 0.01ICC: 0.89 (0.88–0.97)	0.16 (T30=−1.43)
3D OCT versus CIRRUS	3.40±11.57LoA −18.28 to 26.06r2=0.39 P= 0.26ICC: 0.55 (−2.34–0.95)	0.58 (T4=−0.68)	3.13±7.94LoA −12.43 to 18.68r2=0.66 P= 0.01ICC: 0.80 (0.35–0.96)	0.30 (T7=−1.11)
Foveal Thickness (FT), μm
PRIMUS versus CIRRUS	−1.39±4.27LoA −9.76 to 6.99r2=0.88 P< 0.01ICC: 0.96 (0.92–0.98)	0.09 (T28=−1.74)	−2.13±5.99LoA −13.87 to 9.61r2=0.86 P< 0.01ICC: 0.96 (0.92–0.98)	<0.06 (T30=−1.98)
3D OCT versus CIRRUS	−13.54±5.62LoA −24.56 to −2.52r2=−0.85 P< 0.01ICC: 0.53 (−0.17–0.88)	< 0.01 (T7=6.81)	−12.73±6.49LoA −25.44 to 0.01r2=0.85 P< 0.01ICC: 0.67 (0.28–0.87)	<0.01 (T15=7.85)

Abbreviations: LoA, limits of agreement; ICC, Intraclass coefficient of correlation (95% interval of confidence for ICC).

Discussion

OCT technology has not only revolutionized eye-care, but is now widely accepted as an important ancillary test in neurology and neuro-ophthalmic practice16,23,24 given the fact that retina is part of the central nervous system.1,25 SD-OCT is a well-known, reproducible and non-invasive imaging technique that uses near-infrared light to generate cross-sectional or three-dimensional images of the retina12,13,26 providing automated segmentation that generates objective and valid measurements of retinal layer volume and thickness.26 In fact, the use of SD-OCT (mainly pRNFL and FT or volume) has been suggested as potential biomarker for diagnosis, follow-up, and prognosis not only in MS,2–5,8,11,12 but also in other neurological diseases26 including Parkinson's27 and Alzheimer's28 with controversial results.29,30 Although current diagnosis exclusively based on OCT is not realistic approach, in some diseases there are highly specific findings that can help to establish a correct diagnosis and follow up.31

Recently, the entry into the clinic of OCT-angiography (OCT-A) has opened new expectations, not only in the better knowledge of retinal diseases, but also in neurodegenerative diseases.32 And some papers have reported a vessel density reduction in retinas of MS patients and a possible clinical correlation between vessel density and Expanded Disability Status Scale (EDSS), suggesting that angio-OCT could be a good marker of disease and of disability in MS. And that a peripapillary region vessel density reduction, revealed through OCT-A, could be an early event in MS.33

This technological acceleration presents certain problems and the real fact is that despite the scientific evidence, systematic examinations of MS patients with OCT have not become popular in hospitals in the way that may be presumed. In our opinion the lack of key elements, such as reference values are some of the limitations. This work exclusively intends to show that OCT equipment significantly conditions the results and prevents the use of the bibliography values as references.

For pRNFL values there are few published papers.11,34,35 One of them34 is made on Chinese population and values could not been extrapolated to the Caucasian population. One35 is the result of a total of a meta-analysis of 16,084 European adults from 8 cohort studies (mean age range 56.9±12.3 to 82.1±4.2 years) of the European Eye Epidemiology (E3) consortium, all of them made by SD-OCT. They described determinants of the pRNFL thickness such as age, refraction, and presence of systemic vascular and neurovascular diseases. In fact, they obtain normal mean values of 86.8±21.4 µm to 104.7±12.5 µm while our values are lower even though our population of normal subjects is significantly younger (Table 1).

The Cracelen-Gadea et al report11 is of particular interest because it used the Spanish population of similar ages (35.0±5.84), with MS patients and because it is carried out with same equipment (Topcon SD 3D OCT). The pRNFL mean value in normal controls was 104.28±8.5 µm, in MS patients without ON it was 97.31±13.0 µm and in MS patients with past history of ON it was 93.5±11.8 µm. All values differing from those obtained in our study with the same equipment, being significantly lower in the three studied groups (Table 1).

Some reports have even proposed a cut-off value of 75 µm for pRNFL for predict the visual recovery after ON.36,37 This value is close to our results with both Zeiss SD-OCTs in MS patients with established ON, but lower than the value obtained with 3D OTC (Topcon) (Table 1). Recently, Coric et al38 have proposed that a value lower than 85 µm for pRNFL measured with SD OCT in MS patients with ON increases odds of cognitive impairment. Nevertheless, our and others' results11 make this simplification of questionable clinical value.

Additionally, inter-eye percentage difference with SD-OCT have been proposed to improve MS-associated NO diagnosis.39 But our results showed a high inter-subject variability in pRNFL and FT in healthy and MS eyes prevented this possibility. For example, depending of the OCT, the coefficient of variation (SD/mean value) of pRNFL varies from 7.7% to 10.6% in control eyes and from 7.3% to 9.9% MS eyes without ON (Table 1), despite a careful and refined technique of obtaining the images with repeatable OCT devices (see methodology section). Moreover, the variability of the FT is slightly lower (approximately 7% in healthy eyes and close to 5.5% in MS eyes without ON). These values suggest that inter-subject variability could affect the accuracy of proposed cut-off values.

Although the aetiology of axonal atrophy and progressive ganglion cell loss in MS (in the absence of ON) is still unclear, several studies have reported statistically significant thinning of the pRNFL and GCIPL in patients with MS with and without a history of ON compared with that of healthy control subjects.2,40

Our results agree with previous reports showing thinner pRFNL and FT in MS patients w/w ON history.2,3,11 However, we did not find differences in the pRFNL between healthy and MS eyes without ON history (thinning lower than 2 μm), although it has been previously reported.3,40–42

The small difference between pRFNL of healthy versus MS patients without ON history is close to previously reported pRFNL repeatability achieved with SD-OCT, ranged from 3.9 μm to 5.3 μm43 or between 0.3% to 1%.44 Moreover an intra-observer variability less than 2% has been also reported with pRFNL measurements of SD-OCT. Thus, effects of repeatability and intra-observer variability could affect the ability to detect the small difference in pRFNL between healthy and MS patients without ON. So, pRFNL values must be used with caution in MS patients without a previous history of ON.45

The involvement of macular preganglionic elements’ function, during the neurodegenerative process of MS, is controversial and there are few references on this topic.46

Regarding to FT results, our results show slightly lower FT values (between –7.62 μm to –16.17 μm depending of the OCT) in MS patients with a history of ON compared healthy eyes, than previously reported ones (–6.18 μm 95% CI between –8.07 μm to –4.28 μm).2 However, FT thinning in MS patients without ON history (between –2.37 μm to –5.01 μm depending of the OCT) is closer to previous described thinning –2.15 μm 95% CI between –3.15 μm to –1.15 μm.2

In this current study, CIRRUS OCT showed better agreement (lower LoA) with PRIMUS than with Topcon 3D OCT in healthy and MS eyes (with and without ON) for both pRNFL and FT measurements (Tables 2 and 3). This could be explained because CIRRUS and PRIMUS are both from the same manufacturer and share technology and features (similar algorithms, scan protocols, etc.). Differences between devices suggest that CIRRUS and PRIMUS OCT could provide interchangeable measurements; however, it cannot be assumed that different OCTs provide equivalent values, which must be considered in future research and major reviews or meta-analysis. These differences could be of relevance in MS patients follow up, because a pRNFL atrophy rate of –0.36 to –1.4 μm/year have been described in longitudinal reports,2 that suppose a close value to agreement found in this study in MS patients w/w ON. Thus, agreement studies assessing differences of different SD-OCT in MS patients are necessary to facilitate accurate information to manage those patients.

In summary, OCT is a valuable technology to improve MS patient assessment and follow up, although differences between devices make comparisons difficult because SD-OCT outcomes are not interchangeable. Moreover, the variability of the different measures, which are currently high, must be considered. These issues are of paramount relevance in further research in MS patients involving multicenter studies with a large sample size and follow-up, which are necessary to provide cut-off values to improve early detection of MS and follow-up.