Multicenter reliability of semiautomatic retinal layer segmentation using OCT.

OBJECTIVE: To evaluate the inter-rater reliability of semiautomated segmentation of spectral domain optical coherence tomography (OCT) macular volume scans.
METHODS: Macular OCT volume scans of left eyes from 17 subjects (8 patients with MS and 9 healthy controls) were automatically segmented by Heidelberg Eye Explorer (v1.9.3.0) beta-software (Spectralis Viewing Module v6.0.0.7), followed by manual correction by 5 experienced operators from 5 different academic centers. The mean thicknesses within a 6-mm area around the fovea were computed for the retinal nerve fiber layer, ganglion cell layer (GCL), inner plexiform layer (IPL), inner nuclear layer, outer plexiform layer (OPL), and outer nuclear layer (ONL). Intraclass correlation coefficients (ICCs) were calculated for mean layer thickness values. Spatial distribution of ICC values for the segmented volume scans was investigated using heat maps.
RESULTS: Agreement between raters was good (ICC > 0.84) for all retinal layers, particularly inner retinal layers showed excellent agreement across raters (ICC > 0.96). Spatial distribution of ICC showed highest values in the perimacular area, whereas the ICCs were poorer for the foveola and the more peripheral macular area. The automated segmentation of the OPL and ONL required the most correction and showed the least agreement, whereas differences were less prominent for the remaining layers.
CONCLUSIONS: Automated segmentation with manual correction of macular OCT scans is highly reliable when performed by experienced raters and can thus be applied in multicenter settings. Reliability can be improved by restricting analysis to the perimacular area and compound segmentation of GCL and IPL.

Optical coherence tomography (OCT) can be used to demonstrate retinal atrophy in patients with MS.[1-3] The technique has become a valuable outcome measure in neuroprotective clinical trials.[4,5] There is pathologic evidence of significant retinal injury in MS,[6] which was confirmed by OCT[7,8] including demonstration of progressive retinal neurodegeneration.[3,9] Macular volume scans allow quantification of retinal layers such as the ganglion cell layer (GCL) and inner nuclear layer (INL). These may be predictive of functional or clinical outcomes, such as visual acuity in the case of GCL,[10,11] or relapse rate and overall disability.[12,13]

Although further longitudinal investigations are needed, OCT is already used in clinical trials. The capacity to assess the effects of investigational compounds on respective retinal layers could enhance their diagnostic and prognostic value.[4] This underscores the necessity for validated segmentation algorithms, allowing reliable postprocessing in international, multicenter clinical trial settings. Currently, fully manual, semiautomatic, and fully automatic intraretinal segmentation are used in OCT studies. Fully automated and manual segmentation studies have been performed on spectral domain (SD) OCT platforms such as the Heidelberg Engineering Spectralis and Carl Zeiss Meditec Cirrus machines, but all with a limited number of raters and in single centers.[14] Although these studies have suggested reasonable within-patient reproducibility and general agreement between devices, reliability across centers and raters remained unclear.

Here, we tested the multicenter inter-rater reliability of semiautomatic intraretinal segmentation comparing 5 independent raters from 5 academic OCT centers in Europe and the United States.

Methods

Participants

The centers in Barcelona, Berlin, and San Francisco each provided OCT data sets from 3 patients with MS and 3 age- (maximal difference of 3 years) and sex-matched healthy controls from their center's research database in March 2013. Data were anonymously shared between the centers using the devices built-in export function. Demographic data of participants were not retained in this process.

Standard protocol approvals, registrations, and patient consents

All data were originally obtained from studies approved by the respective local ethics committees, and all participants provided written informed consent according to the Declaration of Helsinki. Inclusion of fully anonymized OCT data in this study did not need additional permission.

Optical coherence tomography

All OCTs were recorded with Spectralis SD-OCT (Heidelberg Engineering, Inc, Heidelberg, Germany). All participants were examined with the OCT's built-in macular volume scan (25 vertical B-scans and 1024 A-scans, scan angle = 20° × 20°, automatic real-time function [ART] = 49). For quality control (QC) of all OCT scans, the quality criteria as defined by the OSCAR-IB study were used.[15] OCT scans of the left eyes were arbitrarily selected to enter the analysis. One scan had to be excluded because of file corruption (the file could not be opened). All other scans passed QC, leading to a total analysis of 17 scans.

Semiautomatic intraretinal segmentation

All B-scans were automatically presegmented using segmentation beta-software (Spectralis Viewing Module version 6.0.0.7) of the Heidelberg Eye Explorer (version 1.9.3.0) provided by the manufacturer, which is almost identical to the final release version of the segmentation algorithm. Five operators from different medical centers (Hospital Clínic de Barcelona; NeuroCure Clinical Research Center Berlin; University of California, San Francisco; University Hospital Zürich; and VU University Medical Center, Amsterdam) manually corrected the 25 segmentation lines of each of the 17 presegmented B-scans. The segmentation lines indicate the following retinal layers: retinal nerve fiber layer, GCL, inner plexiform layer (IPL), INL, outer plexiform layer (OPL), and outer nuclear layer (ONL). The ONL includes the inner segments of the photoreceptors (figure 1A). As an additional parameter, the compound thickness of the GCL and IPL was calculated (ganglion cell inner plexiform layer [GCIP]). The mean thickness of the individual retinal layers was computed in an area of 6 mm around the fovea as defined by the Early Treatment Diabetic Retinopathy Study (ETDRS) (figure 1B).[16]

Figure 1

Example OCT scan

(A) Optical coherence tomography B-scan with segmentation lines generated from the mean of all 5 raters with color-coded SD between raters (1 pixel ≙ 3.78 μm). Shown are the retinal nerve fiber layer (RNFL), ganglion cell layer (GCL), inner plexiform layer (IPL), inner nuclear layer (INL), outer plexiform layer (OPL), and outer nuclear layer (ONL). For analysis, the GCL and IPL were also segmented as a compound layer (GCIP). In the example B-scan, the segmentation lines with higher levels of disagreement between raters (indicated by higher SDs) are the IPL, the area of the fovea, and segmentation lines crossing blood vessel artifacts. (B) Example thickness map of the GCIP on the fundus image. The mean of all thickness values within the 6-mm ETDRS circle (white circle) was used for analysis. GCIP = ganglion cell inner plexiform layer.

Statistical analysis

Intraclass correlation coefficients (ICCs) with the degree of consistency among measurements (norm-referenced reliability) were used to analyze the level of agreement between individual grader-corrected segmentation results. Local differences across the volume scans were analyzed by calculating the ICC for each A-scan of all layers and visualizing the results as color-coded ICC heat maps. Correlations between the mean layer thicknesses and SD of corrected scans from different centers were analyzed using the Spearman rho. Quantitative OCT analysis and statistical analysis were performed using MATLAB 2012b (Mathworks, Ismaning, Germany) and R (version 3.3.3).

Results

First, we investigated whether intraretinal segmentation produced similar results on a group level between the different centers (figure 2). This was the case for RNFL, GCL, IPL, and GCIP, but not for INL, OPL, and ONL.

Figure 2

Intraretinal layer thickness measurements

Boxplots for different retinal layer thicknesses of OCT scans from all 17 subjects corrected by all 5 raters (Hospital Clínic de Barcelona [HCB]; NeuroCure Clinical Research Center [NCRC], Berlin; University of California, San Francisco [UCSF]; University Hospital Zürich [USZ]; and VU University Medical Center [VUMC], Amsterdam). The dots represent outliers. GCIP = ganglion cell inner plexiform layer; GCL = ganglion cell layer; INL = inner nuclear layer; IPL = inner nuclear layer; ONL = outer nuclear layer; OPL = outer plexiform layer; RNFL = retinal nerve fiber layer.

On an individual measurement level, overall reliability between raters was excellent for RNFL, GCL, IPL, GCIP, and ONL (all ICCs > 0.96) and good for INL and OPL (all ICCs > 0.85). Reliability was similar when patients with MS and controls were analyzed separately (table).

Table

Intraretinal segmentation inter-rater reliability

To identify the image areas of disagreement between the centers, we calculated intercenter reliability as an ICC for every A-scan. Here, the perimacular area showed the highest reliability, while the scan's borders and the fovea were subject to higher disagreement between graders (figure 3). RNFL, GCL, GCIP, and ONL showed high local agreement. Of note, the temporal RNFL (figure 3A) and the peri-foveal area in the ONL (figure 3G) showed weaker agreement. The latter most likely reflects Henle fibers.[16,17] IPL, INL, and OPL showed weak local agreement (figure 3). In addition, on single B-scans, areas with blood vessels showed higher disagreement between centers (figure 1A). The intercenter variabilities for the different retinal layers were not correlated with the respective mean layer thicknesses (all p > 0.05).

Figure 3

Local intraclass correlation coefficients

Color-coded interclass correlation coefficients (ICCs) for the manually corrected intraretinal layer segmentation across all positions in macular volume scans (1024 A-scans and 25 B-scans). Seven different intraretinal layers were assessed (A-G): retinal nerve fiber layer (RNFL), ganglion cell layer (GCL), inner plexiform layer (IPL), combined GCL and IPL (GCIP), inner nuclear layer (INL), outer plexiform layer (OPL), and outer nuclear layer (ONL). The central foveal scan and peripheral areas outside the ETDRS macular volume are predilection sites of disagreement. GCIP = ganglion cell inner plexiform layer.

Finally, we analyzed whether segmentation results differed after manual correction in comparison with uncorrected segmentation directly derived from the software (figure 4). After manual correction, the outer retinal layers showed the most pronounced differences in comparison with purely automated segmentation and the broadest distribution of differences (range OPL = 8.8 μm; range ONL = 9.4 μm). The inner retinal layer thicknesses differed less profoundly (range RNFL = 1.9 μm, range GCL = 3.4 μm, range IPL = 3.8 μm, and range GCIP = 2.4 μm). The differences between manual and automatic segmentation were centered on zero for the inner layers. By contrast, the manual corrected segmentation volume tended to be lower in outer layers (OPL and ONL).

Figure 4

Differences between semiautomatic and fully automatic segmentation

Boxplots show the differences between automated and manually corrected segmentation for the retinal nerve fiber layer (RNFL), ganglion cell layer (GCL), inner plexiform layer (IPL), combined ganglion cell and inner plexiform layer (GCIP), inner nuclear layer (INL), outer plexiform layer (OPL), and outer nuclear layer (ONL). Manual correction had the biggest influence on the outer layers (difference of up to ca. −10 μm) and less pronounced influence on the inner layers. The stars represent outliers.

Discussion

We analyzed the inter-rater reliability of semiautomatic intraretinal segmentation of macular volume B-scans from OCT. Color-coded heat maps illustrating ICCs between segmentation results from 5 different experienced, academic OCT readers provide a fast overview of the agreement between raters and help to rapidly identify areas of disagreement. Furthermore, analysis on a single B-scan level as illustrated in figure 1A visualizes local deviations between segmentation lines of different raters revealing areas of higher inter-rater variability. An automated segmentation algorithm for Spectralis OCT macular volume scans was recently made available, and a version almost identical to it has been applied in our study. We aimed to determine the inter-rater reliability of manually corrected segmentation of retinal layers compared with results from automated segmentation.

Mean layer thickness when corrected manually showed only little variation between graders in the case of the innermost layers RNFL, GCL and IPL, and ONL. Consistency of results could even be improved when GCL and IPL were segmented as a compound layer (GCIP; figures 2 and 3). The heat maps again emphasize the much better inter-rater reliability of RNFL, GCIP, and ONL compared with INL and OPL. Spatial reliability analysis using heat maps revealed the fovea and the scan periphery as sites of greatest disagreement. Segmentation variation at the fovea is most probably due to thinning of the innermost retinal layers at the foveal pit. Whereas in the scan periphery (outreaching the ETDRS macular volume), major sources of disagreement were induced by retinal vessel artifacts (figure 1A). Comparing manually corrected with automated segmentation results revealed that inner retinal layers (RNFL, GCL, IPL, and GCIP) were accurately identified by automatic segmentation with little spread of thickness differences, whereas the outer retinal layers (INL, OPL, and ONL) showed stronger disagreement and improved with manual correction.

Accurate quantification of retinal layer integrity is relevant in the context of degenerative diseases involving the retina such as MS, especially for understanding the extent and timing of injury to neurons beyond the site of the presumed initial injury. For neurodegenerative diseases, OCT allows evaluation of injury to partially segregated neuronal and axonal compartments and is at a resolution 2–3 orders of magnitude better than what can be achieved via MRI. OCT's quantification also has a tremendous advantage, given its ease and immediacy. However, for multicenter clinical trials, quality criteria and standardized correction methods are required, and a standardized training of expert OCT readers needs to be established. The methods used here are far less labor intensive than those required for brain MRI. In addition, the afferent visual pathway is an increasingly recognized model that allows quantifying MS and optic neuritis (ON) related neuro-axonal damage through solid structure-function investigations.[18-22] Therefore, measures of retinal atrophy assessed by OCT have gained significant interest as a powerful outcome tool to monitor neurodegeneration and protective therapies.[23] Recently, RNFL thickness quantified by OCT proved its value as a primary outcome measure in patients with acute ON treated with erythropoietin vs placebo in addition to steroids,[4] while a number of other trials are ongoing. However, there has been increasing interest in measuring deeper retinal layers such as the GCIP that do not exhibit the acute swelling seen in the RNFL in ON.[24] Results from OCT macular B-scan segmentation (GCIP thickness at month 1) proved to be predictive of visual outcomes at month 6 following MS associated ON.[25] Given the increasing use of OCT in multicenter clinical trials, the inter-rater reliability of macular multilayer segmentation along with OCT quality reading[15,26] will be key to reduce noise and subsequently the sample size required for therapeutic clinical trials. Quality criteria have not yet been proposed for macular volume scans, as it remains unclear how the built-in algorithms perform. With respect to segmentation of macular volume B-scans, the results of our study suggest that restriction to the inner sections of the ETDRS grid and inner retinal layers (including some layers improved with manual correction such as INL) should reduce noise. The inner ETDRS grid has also been shown to have good retest reliability.[16] Fortuitously, these areas and layers also have been suggested to demonstrate the most pronounced changes over the course of the disease. In addition to its potential use as an end point in clinical trials, recent OCT studies on disease mechanisms provided evidence of retinal pathology and neuro-axonal damage appearing as early as in the stage of non-ON clinically isolated syndromes.[6,27]

For the Cirrus HD-OCT (Carl Zeiss Meditec, Dublin, CA, USA), single-center inter-rater correlation coefficients of 0.99 for GCL + IPL and RNFL + GCL + IPL, 0.91–0.94 for INL + OPL, and 0.92–0.93 for ONL + photoreceptor segments have been reported in patients with MS using prototype segmentation software.[7]

Regarding Spectralis SD-OCT scans, Hood et al.[28] reported single-center inter-rater correlations between 2 graders using MATLAB-derived segmentation software for one horizontal B-scan in 30 patients with retinitis pigmentosa and 23 control subjects. Correlation coefficients were 0.95–0.96 for RNFL and GCL + IPL. Correlation coefficients were not reported for other retinal layers. Seigo et al.[14] assessed inter-rater reliability for fully manual MATLAB-derived segmentation of the central horizontal macular B-scan traversing the fovea acquired with Spectralis SD-OCT in 52 patients with MS and 30 control subjects. GCIP, ONL, and ONL + photoreceptor segment thickness measures demonstrated excellent inter-rater reproducibility with correlation coefficients of 0.94–0.95, whereas the correlation coefficient for the RNFL was only 0.64 and for the INL 0.51.

A limitation of our study is the rather small sample size. However, given that 25 B-scans per subject were segmented, the analysis is based on 425 B-Scans from 17 subjects.

To conclude, automated segmentation with manual correction by trained raters of Spectralis macular OCT scans is highly reliable and can thus be applied in multicenter settings. Reliability can be improved by restricting analysis to the perimacular area and by compound segmentation of GCL and IPL. The foveal pit and the scan periphery with retinal vessel artifacts were the sites of greatest disagreement.

Intraretinal layer thicknesses might serve as important outcomes for trials in MS or MS ON in the future, and established methods for optimizing the inter-rater reliability of retinal layer segmentation will be crucial. When fully automated segmentation becomes broadly available, the approaches presented in this study could easily be adapted to longitudinal OCT data to compare segmentation results across follow-up visits. This may help to reliably identify segmentation errors and discriminate them from true degenerative changes.