Automated Segmentability Index for Layer Segmentation of Macular SD-OCT Images.

PURPOSE: To automatically identify which spectral-domain optical coherence tomography (SD-OCT) scans will provide reliable automated layer segmentations for more accurate layer thickness analyses in population studies.
METHODS: Six hundred ninety macular SD-OCT image volumes (6.0 × 6.0 × 2.3 mm3) were obtained from one eyes of 690 subjects (74.6 ± 9.7 [mean ± SD] years, 37.8% of males) randomly selected from the population-based Rotterdam Study. The dataset consisted of 420 OCT volumes with successful automated retinal nerve fiber layer (RNFL) segmentations obtained from our previously reported graph-based segmentation method and 270 volumes with failed segmentations. To evaluate the reliability of the layer segmentations, we have developed a new metric, segmentability index SI, which is obtained from a random forest regressor based on 12 features using OCT voxel intensities, edge-based costs, and on-surface costs. The SI was compared with well-known quality indices, quality index (QI), and maximum tissue contrast index (mTCI), using receiver operating characteristic (ROC) analysis.
RESULTS: The 95% confidence interval (CI) and the area under the curve (AUC) for the QI are 0.621 to 0.805 with AUC 0.713, for the mTCI 0.673 to 0.838 with AUC 0.756, and for the SI 0.784 to 0.920 with AUC 0.852. The SI AUC is significantly larger than either the QI or mTCI AUC (P < 0.01).
CONCLUSIONS: The segmentability index SI is well suited to identify SD-OCT scans for which successful automated intraretinal layer segmentations can be expected. TRANSLATIONAL RELEVANCE: Interpreting the quantification of SD-OCT images requires the underlying segmentation to be reliable, but standard SD-OCT quality metrics do not predict which segmentations are reliable and which are not. The segmentability index SI presented in this study does allow reliable segmentations to be identified, which is important for more accurate layer thickness analyses in research and population studies.

Introduction

Optical coherence tomography (OCT) is a noninvasive, noncontact imaging modality designed to provide cross-sectional images of the retinal structure with high axial resolution.[1] Spectral-domain OCT (SD-OCT) with faster scanning speed and higher spatial resolution has become essential clinical tools to diagnose and monitor retinal diseases.[2,3] Reliable quantification of the retinal structures such as retinal nerve fiber layer (RNFL), ganglion cell layer (GCL), outer segment, and choroidal thickness is crucial.[4-9]

OCT scans with insufficient image quality, either because of media opacities or aberrations, or because of improper patient alignment, negatively affect intraretinal layer segmentation and cause unreliable measurements.[10,11] A metric that quantifies this quality would ideally be a continuous variable because different uses of the retinal layer segmentations may require different levels of reliability; thus, our preference for a continuous variable that can be thresholded. For example, population studies where the segmentation results may be used without human review of the segmentation quality may require a different threshold than segmentation in a clinical context where the segmentation is also reviewed by the clinician. Figure 1 shows examples of successful and failed intraretinal layer segmentations in macular SD-OCT volumes with good, moderate, and bad image quality. Figure 2 shows that including all layer analyses (420 successful RNFL segmentations as well as 270 failed segmentations) obtained from a population study into average RNFL thickness yields suboptimal results. The standard deviation (SD) RNFL thickness map has large regional variations because of the failed segmentations caused by the image quality issues (Figs. 1C, 1F).

Figure 1

Automated intraretinal layer segmentations of macular SD-OCT volumes with good (A), moderate (B), and bad (C) image quality and corresponding segmentability index SI, which is explained in Methods. (D–F) B-scans in (A–C) overlaid with automatically detected layers (red line: ILM, orange line: boundary between RNFL and GCL, green line: outer boundary of retinal pigment epithelium [RPE]).

Figure 2

Mean and SD RNFL thickness maps of 690 macular OCT volumes including 420 volumes with successful RNFL segmentations and 270 volumes with failed segmentations. The number at the bottom of the thickness map is mean ± SD thickness of the RNFL. The thickness maps were created after horizontally flipping left eye OCT volumes and aligning the OCT volumes based on the fovea.

Several OCT manufacturers supply signal quality metrics, and at least two researchers have proposed alternative signal metrics.[12,13] Stein et al.[12] have introduced the quality index (QI), based on the image histogram information, which is expressed as where Low, Noise, Saturation are the pixel intensity values corresponding to 1%, 75%, 99% of all pixels, and Middle is the mean value of Noise and Saturation pixel intensities.[12] Higher QI values represent better OCT image quality. In this paper, an entire OCT volume consisting of multiple B-scans was treated as a single image to obtain a QI value.

Huang et al.[13] has proposed the maximum tissue contrast index (mTCI), based on the intensity histogram decomposition model, which is calculated as where N, N, and N denote the voxel intensity values of the mode point (the highest peak in the voxel intensity histogram), the separation point (99% of the accumulative density of the background voxels) between the vitreous having low reflectance and the foreground corresponding to various retinal tissues having higher reflectance, and the saturation point (99.9% of all voxels), respectively. Higher mTCI values represent better image quality, and the lowest value is 1.

While some prior studies showed a positive relationship between OCT manufacturers' image quality indices and RNFL thickness measurements,[14-16] other studies failed to show any such relationship.[17,18] Because the QI, mTCI, and similar quality metrics correspond to visual qualitative assessment of image quality, they necessarily identify those OCT scans that are expected to lead to correct intraretinal layer segmentations. Therefore, we have developed a new segmentability index SI that quantifies the expected RNFL segmentability of the OCT scans.

The purpose of the present study is to introduce the segmentability index and verify that it can be used to differentiate SD-OCT scans that will result in reliable RNFL segmentations.

Methods

Human Subjects and Data Acquisition

The Rotterdam Study is a prospective population-based cohort study organized in the city of Rotterdam in the Netherlands and ongoing since 1990.[19] The Rotterdam Study has been approved by the Medical Ethics Committee of the Erasmus MC and by the Ministry of Health, Welfare, and Sport of the Netherlands, implementing the “Wet Bevolkingsonderzoek: ERGO (Population Studies Act: Rotterdam Study).” All participants provided written informed consent prior to participation in the study and gave permission to obtain information from their treating physicians.

Macular SD-OCT image volumes were obtained from one eyes of all subjects using Topcon OCT devices (3D OCT-1000, software version: 3.44; Topcon Europe, Capelle aan den Ijssel, the Netherlands). The Topcon software is a custom version for the Rotterdam Study. It provides an image quality score. Each volume is composed of 512 × 128 × 650 voxels, corresponding to physical dimensions of 6.0 × 6.0 × 2.3 mm3, and the voxel image intensity depth is 2 bytes.

Six hundred ninety macular SD-OCT image volumes were obtained from one eye (672 right eyes, 18 left eyes) of 690 subjects (74.6 ± 9.7 [mean ± SD] years, 37.8% of males) randomly selected from the population-based Rotterdam Study. The dataset consisted of 420 OCT volumes with successful automated RNFL segmentations and 270 volumes with failed segmentations. The dataset was divided into a training set (80%) including 336 OCT volumes with successful RNFL segmentations, 216 volumes with failed segmentations, and a testing set (20%) that included 84 volumes with successful segmentations, as well as 54 volumes with failed segmentations.

To validate our automated identification method of the OCT volumes providing successful automated RNFL segmentations, an expert created ground truth by manually categorizing the 690 macular OCT volumes into two groups: 420 OCT volumes with successful RNFL segmentations and 270 with globally/locally failed segmentations. The OCT volumes having small RNFL segmentation errors in small regions were regarded as having failed segmentations. For the ground truth determination, only RNFL boundary accuracy was taken into account, and any other information like scan centering was not considered.

Automated Intraretinal Layer Segmentation

Ten intraretinal layers (11 surfaces) from the macular SD-OCT volumes were automatically detected by applying our previously reported automated graph-based segmentation method, the Iowa Reference Algorithms (available in the public domain from https://www.iibi.uiowa.edu/content/shared-software-download/).[20-22] Out of the 11 surfaces, the internal limiting membrane (ILM), the boundary between RNFL and GCL, and the outer boundary of RPE were used for determining our segmentability index SI. The ILM and the boundary between RNFL and GCL define the RNFL, and the outer boundary of RPE was used because segmentation of the boundary between RNFL and GCL was constrained by that of the ILM and the outer boundary of RPE. The layer segmentation method used three-dimensional (3D) edge-based cost functions that are inverted, Gaussian-smoothed gradient magnitudes of the OCT voxel intensities: the dark-to-bright transition from top to bottom of the OCT volume for segmentation of the ILM and the bright-to-dark transition for segmentation of the boundary between RNFL and GCL, and the outer boundary of RPE. Using these cost functions, optimal surfaces are detected as having minimal aggregate costs from all feasible surfaces. Figure 3 shows automated intraretinal layer segmentation results of a macular OCT volume.

Figure 3

Automated intraretinal layer segmentation results for a macular SD-OCT volume. (A) B-scan of the OCT volume. (B) Cost function image for segmentation of the ILM. (C) Cost function image for segmentation of the boundary between RNFL and GCL, and the outer boundary of RPE. (D) B-scan in (A) overlaid with three detected surfaces. (E) 3D rendering of the three detected surfaces.

Segmentability Index

Our new metric, the segmentability index SI, expresses the expected reliability of automated layer segmentations, especially, RNFL segmentations for this study. The SI can be obtained from a random forest regressor[23] based on the 12 following features:

Features 1 and 2: mean and SD of whole OCT voxel intensities (Fig. 4A).

Figure 4

Feature images for calculation of segmentability index SI. (A) The mean and SD of OCT voxel intensities are calculated from the entire 3D OCT volume. (B) The mean and SD of edge-based costs of the top-to-bottom dark-to-bright transitions are calculated from the entire 3D OCT volume. (C) The mean and SD of edge-based costs of the bright-to-dark transitions are calculated from the entire 3D OCT volume. (D–F) The means and SDs of on-surface costs of three detected surfaces are calculated along the respective surfaces detected in 3D.

Features 3 and 4: mean and SD of whole edge-based costs of the dark-to-bright transition from top to bottom of the OCT volume that are inverted, Gaussian-smoothed gradient magnitudes of the OCT voxel intensities (Fig. 4B).

Features 5 and 6: mean and SD of whole edge-based costs of the bright-to-dark transition that are inverted, Gaussian-smoothed gradient magnitudes of the OCT voxel intensities (Fig. 4C).

Features 7 and 8: mean and SD of the on-surface costs consisting of the edge-based costs of the dark-to-bright transition along the ILM (Fig. 4D).

Features 9 and 10: mean and SD of the on-surface costs consisting of the edge-based costs of the bright-to-dark transition along the boundary between RNFL and GCL (Fig. 4E).

Features 11 and 12: Mean and SD of the on-surface costs consisting of the edge-based costs of the bright-to-dark transition along the outer boundary of RPE (Fig. 4F).

Features 1 through 6 were included because automated segmentation of the RNFL is globally affected by OCT voxel intensities and edge-based costs. Features 7 through 10 are directly related to RNFL segmentation. If the RNFL is successfully detected, its aggregate on-surface costs are small. Otherwise, they are large. Features 11 and 12 were also used because segmentation of the outer boundary of RPE affects the boundary between RNFL and GCL. Important features were identified with respect to increase in mean square error using a training set. A feature selection step was performed by 3-fold cross-validation using the training set in terms of area under the curve (AUC). The SI ranges from 0 to 1. Larger SI values correspond to an OCT volume allowing a more reliable RNFL segmentation. Statistical computing software (R version 2.15.2, R Development Core Team; provided in the public domain by R Foundation for Statistical Computing, Vienna, Austria, available at http://www.r-project.org/, ‘randomForest' package) was used for the random forest regression. Appendix shows the R source code for random forest regression using a four training dataset and a two testing dataset.

Statistical Analysis

The SI approach was compared with the QI and mTCI methods using receiver operating characteristic (ROC) analysis. The accuracy is the ratio of the number of RNFL segmentations correctly identified by, respectively, the QI, mTCI, or SI to that of all RNFL segmentations. The true positive rate (sensitivity) is the ratio of the number of successful RNFL segmentations passed by respectively the QI, mTCI, or SI and the ground truth to that of all successful RNFL segmentations, and the false positive rate (1 – specificity) is the fraction of the number of successful RNFL segmentations passed by respectively the QI, mTCI, or SI and rejected by the ground truth to that of all unsuccessful RNFL segmentations. The AUC was used to measure aggregated regression performance. The AUC difference between the QI or mTCI and SI ROC curves was statistically assessed using the 95% confidence interval (CI) by the method of DeLong et al.,[24] and the R statistical computing software package was used. The operating point of the QI, mTCI, or SI curves were chosen so as to result in a set with the same number of OCT volumes with RNFL segmentations expected to be successful as the ground truth, though obviously the specific OCT volumes in the sets may be different. The threshold, accuracy, sensitivity, and specificity for each ROC curve were measured at the operating point. An unpaired t-test (95% CI) was used to compare the average of RNFL thicknesses in good quality as determined by either the ground truth or the QI, mTCI, or SI, respectively, using Excel (Microsoft Corp., Redmond, WA).

Results

The relative importance of all 12 features was tested with respect to increase in mean square error using the training set (Fig. 5). Feature 12 (SD of the on-surface costs consisting of the edge-based costs of the bright-to-dark transition along the outer boundary of RPE) is the most important feature, and Feature 6 (SD of whole edge-based costs of the bright-to-dark transition) is the least important feature of the employed feature set. Based on the importance, a feature selection step was performed on the training set in terms of AUC, and the AUC obtained by the SI approach using all 12 features represented the highest value (0.868).

Figure 5

Table representing a list of the importance of 12 features in the training set including 336 OCT volumes with successful RNFL segmentations and 216 volumes with failed segmentations.

The ROC graph in Figure 6 shows the performance of the QI, mTCI approaches and the SI methods using Features 1 through 6, Features 1 through 12 for automated identification of the OCT volumes expected to provide reliable automated RNFL segmentations. The 95% CIs and AUCs of the QI, mTCI, SI (Features 1–6), and SI (Features 1–12) ROC curves are summarized in the Table. The AUC of the SI (Features 1–12) ROC curve is significantly larger than that of the QI, mTCI, or SI (Features 1–6) curve (P < 0.039). In addition, it shows the range, threshold, accuracy, sensitivity, and specificity at the operating point for the four approaches.

Figure 6

ROC graph of the QI, mTCI approaches and the SI methods using Features 1–6 and Features 1–12. The markers show the operating points which are chosen so as to result in a set with the same number (84) of OCT volumes with RNFL segmentations expected to be successful as the ground truth.

Table

Range, 95% CI, AUC, Threshold, Accuracy, Sensitivity, and Specificity at the Operating Point for QI, mTCI, SI (Features 1–6), and SI (Features 1–12) ROC Curves

To better understand the OCT scans flagged as insufficient by the different indices, Figure 7 shows the mean and SD RNFL thickness maps for 138 macular OCT volumes with both successful and unsuccessful segmentations, as well as a subset of 84 volumes from the 138 that were deemed successful by either the independent standard or by the QI, mTCI, SI (Features 1–6), and SI (Features 1–12) approaches. The mean RNFL thickness of the 84 volumes with successful segmentations determined from the independent standard is 38.12 ± 5.62 μm, and that with success identified by the QI, mTCI, SI (Features 1–6), or SI (Features 1–12) approach is 39.46 ± 6.32, 39.16 ± 6.10, 38.98 ± 6.57, or 38.16 ± 5.49 μm, respectively. Though Figure 7 shows regional differences, the RNFL thickness determined from the independent standard is not significantly different from that identified by the QI, mTCI, SI (Features 1–6), or SI (Features 1–12) approaches (P > 0.148).

Figure 7

Mean (first, third rows) and SD (second, fourth rows) RNFL thickness maps of 138 testing macular OCT volumes with all segmentations and 84 testing volumes with successful segmentations determined from independent standard and with segmentation success identified by the QI, mTCI approaches, and the SI methods using Features 1 through 6 and Features 1 through 12. The number at the top of the mean thickness map is a mean ± SD QI, mTCI, or SI value of the OCT volumes, and that at the bottom of the thickness map is mean ± SD thickness of the RNFL. The thickness maps were created after horizontally flipping left eye OCT volumes and aligning the OCT volumes based on the fovea.

Discussion

For the accurate thickness analyses of population studies, it is necessary to automatically identify the macular OCT scans that have resulted in reliable automated intraretinal layer segmentations. Figure 7 shows the importance of identifying the OCT scans with accurate automated RNFL segmentations. The mean RNFL thickness map created using 84 good quality OCT scans as determined from the independent standard shows clearer arcuate nerve fiber bundle regions than that derived from 138 OCT scans that include both good quality and bad quality scans. The SD RNFL thickness map obtained from the 138 OCT scans has large variations in any region due to 54 mis-segmented RNFLs.

To automatically identify those OCT scans that can be expected to provide reliable automated RNFL segmentations, we have introduced a segmentability index SI obtained from a random forest regressor trained by the features using OCT voxel intensities, edge-based costs, and on-surface costs. The SD of the on-surface costs consisting of the edge-based costs of the bright-to-dark transition along the outer boundary of RPE (Feature 12) is the most important feature for the identification (Fig. 5), which means that the appearance of the outer boundary of RPE has a large effect on the expected reliability on automated segmentation. The second most important feature is the mean of the on-surface costs consisting of the edge-based costs of the bright-to-dark transition along the boundary between RNFL and GCL (Feature 9), which is directly related to the RNFL segmentation. The feature selection step did not show any performance improvement in terms of AUC, so all 12 Features were used for the SI approach.

Based on the ROC analysis, the SI using all 12 Features showed significantly better performance than the QI or mTCI in terms of AUC (P < 0.01; Fig. 6). Additionally, it showed significantly better performance than the SI using Features 1 through 6 (P = 0.038), which means that the means and SDs of the on-surface costs (Features 7–12) are useful for the identification.

The mean and SD RNFL thickness maps of the macular OCT scans with segmentation success identified by the SI using all 12 features are more similar to those of the OCT scans with successful segmentations determined from the independent standard than those of the OCT scans with segmentation success identified by the QI, the mTCI, or the SI using Features 1 through 6 (Fig. 7). The QI and mTCI thickness maps may be thicker on average because OCT scans with thick RNFLs, and thus are bright in the OCT image, may correlate with high QI and mTCI values, as further discussed below.

Figure 8 shows an example of a macular OCT volume in which the SI using all 12 features (successfully) identifies a successful RNFL segmentation but the QI and mTCI fail, based on the thresholds mentioned in Table. The means and SDs of the on-surface costs (Features 7–12) calculated using three reliable surface segmentations obtained from the method helped the SI approach to identify the OCT scans with successful RNFL segmentations compared to the QI or mTCI approaches.

Figure 8

A macular OCT volume in which the SI approach using all 12 features succeeds to identify a successful RNFL segmentation but the QI and mTCI methods fail, based on the thresholds in the Table. (A) B-scan of the OCT volume (QI = 12.7, mTCI = 2.08, SI [Features 1–6] = 0.605, and SI [Features 1–12] = 0.765). (B) B-scan in (A) overlaid with the successful RNFL segmentation.

Previous studies reported the positive relationship between OCT manufacturer's image quality index and RNFL thickness. Darma et al.[14] assessed the effect of media opacities on intraretinal layer thickness measurements of macular Topcon OCT scans (3D OCT-1000) using a set of filters with known optical density, and the layer thickness measurements were influenced by image degradation caused by optical density filters. Cheung et al.[15] examined the relationship between signal strength and RNFL measured by TD-OCT (Stratus; Carl Zeiss Meditec, Oberkochen, Germany), and the RNFL thickness in general increased with the signal strength of the image. Wu et al.[16] identified factors associated with variability in RNFL thickness measurements obtained by repeat TD-OCT (Stratus), and signal strength variability was associated with variability in RNFL thickness measurements. Other studies did not show any relationship between them. Rao et al.[17] evaluated the effects of signal strength, age, sex, optic disc size, and axial length on the normal ONH, RNFL, macular measurements with SD-OCT (RTVue; Optovue Inc., Fremont, CA), and any identifier including the signal strength did not influence the RNFL measurements. Samarawickrama et al.[18] examined the influence of different signal strengths on measurements of macular, ONH, and RNFL parameters using TD-OCT (Stratus), and significant differences in RNFL parameters with increasing signal strength were not observed.

There are some limitations to the proposed identification approach. First, in the present study, the method was tested using a subset of the population-based dataset, most of which are normal OCT scans. The results need to be replicated with OCT scans showing pathology for more general use. Second, the method was biased toward the opinion of the expert who created the reference standard. For more objective validation of the method, a reference standard created by multiple experts is required.

The method used the RNFL segmentations as ground truth. For the ground truth, other intraretinal layer segmentations such as GCL or total retina, or the combination of all layer segmentations can be used. Incorporation of layer-specific textural features may further improve the accuracy of the method. This identification method can be easily extended for validation of general automated image segmentation.

In conclusion, the segmentability index SI is well suited to identify the OCT scans expected to provide reliable automated intraretinal layer segmentations, which allows more accurate and more reliable automated analyses of intraretinal layers in population studies.

Table

Table