Qualitative comparison of choroidal vascularity measurement algorithms.

PURPOSE: To compare the accuracy of manual and automated binarization technique for the analysis of choroidal vasculature.
METHODS: This retrospective study was performed on a total of 98 eyes of 60 healthy subjects. Fovea-centered swept source optical coherence tomography (SS-OCT) scans were obtained and choroidal area was binarized using manual and automated image binarization technique separately. Choroidal vessel visualization in the binarized scans were subjectively graded (grades 0-100) by comparing them with the original OCT scan images by two masked graders. The subjective variability and repeatability was compared between two binarization method groups. Intergrader and intragrader variability was estimated using paired t-test. The degree of agreement between the grades for each observer and between the observers was evaluated using Bland-Altman plot.
RESULTS: The mean accuracy grades of the automatically binarized images were significantly (P < 0.001) higher (93.38% ± 1.70%) than that of manually binarized images (78.06% ± 2.92%). There was a statistically significant variability and poor agreement between the mean interobserver grades in the manual binarization arm.
CONCLUSION: Automated image binarization technique is faster and appears to be more accurate in comparison to the manual method.

Choroidal vascularity index (CVI) is a novel in vivo approach to measure the vascular status of the choroid using optical coherence tomography (OCT). Various studies have demonstrated alterations in the CVI value in diseases, such as central serous chorioretinopathy,[1] age-related macular degeneration,[2] and diabetic retinopathy.[3] It has also been observed to be very helpful in disease monitoring and assessing treatment response.[14]

The measurement of CVI requires the assessment of choroidal vessel luminal area and the total choroidal area by binarizing technique. Binarization is an essential step in the assessment of CVI. As CVI could provide a quantitative assessment of the choroidal vasculature at the scanned area, generalized changes can be more easily detected, especially in the absence of any detectable focal change. This could be useful in prognosticating and monitoring diseases in which choroidal vasculature is implicated in the pathogenesis, such as Central serous chorioretinopathy (CSCR), Vogt Koyanagi Harada syndrome (VKH), choroidal neovascular membrane (CNVM). Since the choroidal vascular structures are not easily discernible in OCT imaging, various attempts have been made to quantify choroidal vascularity. Manual techniques involve manual demarcation of choroidal boundaries and binarization techniques using Image J editing software (provided in the public domain by the National Institutes of Health, Bethesda, MD, USA; https://imagej.nih.gov/ij/).[56789101112131415] This technique involves multiple steps, which makes the process complex, tedious, and prone to subjective errors. Whereas automated algorithms that automatically segments the choroidal boundary and binarizes the image to provide the choroidal vascularity.[16171819] Therefore, automated algorithm is less time consuming, more reproducible, and, moreover, can be applied on volumetric data to obtain an overall choroidal vascularity at macular region. However, due to lack of any gold standard protocol, it remains unclear as to which of the binarization technique is the most accurate.

This paper aims at comparing the conventional manual binarization technique with the recently proposed automated binarization technique to check for the binarization accuracy and consistency.

Methods

This was a retrospective study on a total of 98 eyes from 60 healthy subjects seen at a tertiary eye care center, Hyderabad, India. Available data of healthy subjects from 2016 to 2018 were included. Subjects with any ocular or systemic comorbidities were excluded. Baseline information, including demographic details, age, and gender, was collected retrospectively. Swept source device (Triton®, Topcon Corporation, Tokyo, Japan) was used for acquisition of OCT images passing through the central fovea.

Binarization

Binarization was done on each scan separately using manual and automated methods. Binarized scans were compared against original OCT scans by two masked graders, for accuracy and repeatability (inter- and intragrader variability).

Manual method

In this method, we adopted a previously reported semi-binarization technique using public domain ImageJ software (version 1.47; provided in the public domain by the National Institutes of Health, Bethesda, MD, USA; https://imagej.nih.gov/ij/).[20] This involved manual binarization using Niblack auto local thresholding followed by segmentation of the choroid using polygon tool over the binarized image.

Automated method

This method involved automated binarization of an OCT scan and automated segmentation of the binarized image using a previously validated algorithm.[2122]

The automated methodology consists of various steps including (1) denoising, (2) localization of choroidal inner boundary (CIB), and (3) choroid outer boundary detection.

Appropriate denoising removes the noise from OCT images and enhances the algorithmic accuracy. This is followed by localization of retinal pigment epithelium (RPE) for determination of the CIB and specifying a region of interest between CIB and sclera. Following this, the algorithm demarcates the choroid outer boundary.

Subjective grading of binarized scans: Subjective grading was performed by two qualified observers (Graders A and B), experienced in retinal imaging and OCT interpretation. Binarized scans obtained from both the methods were graded separately against respective original OCT scans. Binarized scan and original scan were compared side-by-side with maximum screen brightness to enhance the contrast while grading to avoid technical bias and intraobserver variability. It was graded on a scale of 0–100 percentage (%) where 0 refers least vascular area binarized with respective to original scan and 100 refers maximum vascular area binarized. Each scan was graded twice by each grader with a minimum of 1 week time interval in between, to avoid intragrader bias.

Password protected device was used to run both the methods and the data confidentiality was maintained throughout the study.

Statistical analysis

Statistical analysis was performed using Excel (Microsoft Excel 2016, Microsoft, Redmond, WA, USA). Intergrader and intragrader variability was estimated using paired t-test. P value of < 0.05 was considered statistically significant for all the variables in this study. The degree of agreement between the grades for each observer and between the observers was evaluated using Bland–Altman plots with the help of MedCalc statistical software, version 18.5.

Results

A total of 98 eyes of 60 healthy subjects were analyzed. The mean age of the subjects was 47.90 ± 18.84 years (21–83 years), with 43 males (44%) and 55 females (56%).

Accuracy report

Mean binarization percentages of both the methods were compared to estimate the accuracy levels. The mean grade of manual method by both the graders was 78.06 ± 2.92%, whereas the mean grade of automated method was 93.38 ± 1.70%. The difference in the two grades was statistically significant (P < 0.00). An example of the grade given by a single observer for manual and automated binarized images of a particular image is shown in [Fig. 1].

Figure 1

Comparison of the two binarization techniques. The manual binarized image was graded as 85%, whereas automated binarized was graded as 95% for this particular image by a single observer

Binarization results

Intraobserver grading variability

There was no statistically significant difference in the grades in the manual or automated binarization arm among the graders. The results are summarized in Table 1.

Table 1

Intraobserver variation of grades

Interobserver grading variability

As shown in Table 2, there was a statistically significant difference between both the graders in manual binarization arm (P < 0.00). However, the difference was not statistically significant (P > 0.05) in the automated binarization arm.

Table 2

Interobserver variation of grades

The depiction of agreement in regard to grading between two observers was obtained using Bland–Altman plot [Fig. 2]. The level of agreement was seen to be the least with manual grades between the observers (interobserver variation) showing the limits of agreement to be very wide (5.5–28.3%) with a high bias (16.9%).

Figure 2

Bland–Altman plot showing levels of agreement between manual and automated binarization grades between and among the graders. Interobserver agreement in the manual binarized grades appeared to be the lowest (Graph 5)

Discussion

Automated method of segmentation and binarization was found to be more accurate than manual method in the current series. There are various differences in the mode of operation of the two methods. The process of manual binarization involves the steps of converting the gray scale image into an 8-bit image using ImageJ. This is followed by using Niblack auto local thresholding to get a binarized image. The process of segmentation involves manual identification of the inner and outer choroidal boundaries.[23] On the other hand, the automated binarization algorithm involves automatically driven preprocessing, exponential and nonlinear enhancement, and thresholding,[22] thus reducing the output time.

Databases are flooded with studies evaluating choroidal vascular parameters using manual or automated binarization and segmentation techniques. However, due to lack of any standard guidelines for estimation of these parameters, the validity of these methods is questionable. In order to address this problem, the processed images derived from these two methods were compared with the original OCT image. We found a similarity of only 78.06 ± 2.92% between the original OCT image and the image obtained from manual binarization, whereas there was 93.88 ± 1.70% similarity between the automated image and the original OCT image (P < 0.00). This indicates a high level of accuracy for the automated image. It was seen that the smaller vessels of the Sattler and chorio-capillary layers were better captured, with an increased coverage of the intervening stroma, by the automated system. In contrast, the manual process included these areas as dark pixels [Fig. 1]. This could lead to an overestimation of the choroidal luminal area and thus give a false high value during measurement of CVI when manual binarization is used.

In order to check the validity of the grades, two separate observers were involved in the grading process and each image was graded twice by each grader at an interval of 1 week to reduce bias. These observers were masked to other observer's results as well as to their own responses. Both intra- and intergrader variability was low in the automated binarized images and the difference was not statistically significant. The significant interobserver variation of grades (P = 0.00) and the low agreement in Bland–Altman plot seen with the manual binarization arm can be justified by the subjective nature of the grading system. Nevertheless, the grades given by the two observers in the manual arm were still far less than the grades of the automated images to have affected the significance of difference in the final accuracy grades.

The overall process of manual binarization can be cumbersome and time consuming. On the other hand, the faster output time of the automated process can be utilized to analyze out high volume data. This can particularly be beneficial if volume analysis of the choroidal vasculature is to be performed in the future. The process may involve the analysis of multiple B-scan images and compiling it into a single volumetric data. This can drastically reduce the output time and facilitate a larger data set. Furthermore, the high level of accuracy of the automated process, as described in the study, can help to provide a better estimate of the choroidal vascularity status. In addition, as Niblack thresholding fails to normalize each image before binarization, unlike our automated algorithm, may lead to inconsistent results.

A major drawback of the study is its subjective nature and liability to human bias. Although the inclusion of two independent graders would have reduced the bias to some extent, higher grades could have been assigned by the graders in favor of the automated image owing to the fact that both the graders had considerable knowledge about the images. Nevertheless, the significant difference in the grades between the manual and automated process cannot be ignored. Second, the images taken in the study were from only normal patients. Thus, the validity of the model cannot be representative of diseased eyes. At last, none of the binarization technique could be true a representative of choroidal vascularity unless compared with histological specimens.

Conclusion

Automated binarization and segmentation is faster and appears to be a more accurate alternative to the manual method for the quantification of CVI. It can be helpful especially when dealing with large data sets and may prove to be a convenient tool in the future for rapid assessment and monitoring of patients with choroidal disorders.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.