Generative Adversarial Network Based Automatic Segmentation of Corneal Subbasal Nerves on In Vivo Confocal Microscopy Images.

Purpose: In vivo confocal microscopy (IVCM) is a noninvasive, reproducible, and inexpensive diagnostic tool for corneal diseases. However, widespread and effortless image acquisition in IVCM creates serious image analysis workloads on ophthalmologists, and neural networks could solve this problem quickly. We have produced a novel deep learning algorithm based on generative adversarial networks (GANs), and we compare its accuracy for automatic segmentation of subbasal nerves in IVCM images with a fully convolutional neural network (U-Net) based method.
Methods: We have collected IVCM images from 85 subjects. U-Net and GAN-based image segmentation methods were trained and tested under the supervision of three clinicians for the segmentation of corneal subbasal nerves. Nerve segmentation results for GAN and U-Net-based methods were compared with the clinicians by using Pearson's R correlation, Bland-Altman analysis, and receiver operating characteristics (ROC) statistics. Additionally, different noises were applied on IVCM images to evaluate the performances of the algorithms with noises of biomedical imaging.
Results: The GAN-based algorithm demonstrated similar correlation and Bland-Altman analysis results with U-Net. The GAN-based method showed significantly higher accuracy compared to U-Net in ROC curves. Additionally, the performance of the U-Net deteriorated significantly with different noises, especially in speckle noise, compared to GAN. Conclusions: This study is the first application of GAN-based algorithms on IVCM images. The GAN-based algorithms demonstrated higher accuracy than U-Net for automatic corneal nerve segmentation in IVCM images, in patient-acquired images and noise applied images. This GAN-based segmentation method can be used as a facilitating diagnostic tool in ophthalmology clinics. Translational Relevance: Generative adversarial networks are emerging deep learning models for medical image processing, which could be important clinical tools for rapid segmentation and analysis of corneal subbasal nerves in IVCM images.

Introduction

When Marvin L. Minsky designed the first confocal microscopy in 1957, he had no idea how his invention revolutionized his father's profession, ophthalmology. The three-dimensional nature of biological structures is always a major challenge in medical imaging and laser scanning confocal microscopy (LSCM), also known as in vivo confocal microscopy (IVCM). IVCM is one of the most successful imaging techniques for three-dimensional biological structures in clinical settings. After the late 1980s when the first LSCM was introduced, special attention has grown toward LSCM in developing new methods and applications, especially in medicine. After the clinical approval of the US Food and Drug Administration, IVCM has started to be used in ophthalmology clinics for diagnosis of corneal diseases. The main advantage of LSCM is its noninvasive, reproducible, and easily applicable nature, which utilizes light rather than a physical technique in sectioning the specimen. Because of this structural advantage, IVCM enables detailed assessment of sensory nerve anatomy in the dermis and cornea.,

The cornea is the most innervated tissue of the human body. Innervation of the cornea is made by subbasal corneal nerves, which penetrate from the periphery to the center of the cornea and create a plexus between Bowman's layer and the basal epithelium. They have significant roles in the homeostasis of the cornea, like temperature, tactile sensation, and nociception. Besides corneal diseases, various metabolic and neurological diseases affect corneal subbasal nerve plexus negatively. IVCM is an excellent diagnostic tool to observe corneal structure without any invasive procedure. Especially, morphological assessment of the subbasal nerve plexus of the cornea with IVCM guides clinicians about inflammatory and neurodegenerative processes which affect the cornea. IVCM imaging of the corneal subbasal nerve changes is essential for all corneal diseases, for instance, keratitis, keratoconus, corneal dystrophies, and dry eye disease. Moreover, IVCM imaging of corneal nerves helps to assess peripheral neuropathies caused by various metabolic conditions, such as type-1 diabetes or chemotherapy-induced peripheral neuropathies., The corneal subbasal nerve plexus is also an important indirect indicator of central neurodegeneration in multiple sclerosis (MS), Parkinson's disease, and various dementias.–

The ground-breaking properties of IVCM imaging also bring along its limitations. Widespread and frequent use of IVCM leads to a vast quantity of images to analyze, extensive time periods for manual segmentation and labeling of images, and increased subjective evaluation between ophthalmologists.– Automated segmentation and analysis of IVCM images can easily solve these significant problems in the clinical decision making process.– Even with novel image acquisition methods and image processing programs, inter-rater variability in manual image segmentation and analysis of IVCM images persist as a significant problem that can be solved with automated segmentation methods. However, conventional segmentation algorithms are not feasible for clinical applications because of the uneven illumination and noisy nature of IVCM images of the cornea and their high error rates compared to other clinical diagnostic tests.

Deep learning-based methods have shown to be successful in various tasks in medical imaging analysis.– A fully convolutional neural network (U-Net) architecture is one of the most commonly used deep learning architectures used for medical image segmentation.– U-Net has also been shown to improve the performance of medical image segmentation by concatenating feature maps in the upsampling path with the corresponding cropped feature map in the downsampling path via skip connection. Convolutional neural networks (CNNs) are the backbones of the U-net architectures, and they have been shown to achieve performances close to human experts in the analysis of medical images. CNNs recently have become effective in various medical imaging tasks, including classification, segmentation, and registration and image reconstruction.,– Generative adversarial networks and their extensions have also provided solutions to many medical image analysis problems, such as image reconstruction, segmentation, detection, or classification.– Furthermore, generative adversarial networks (GANs) have been shown to segment the images well and resolve the scarcity of the labeled data in the medical field with the help of generative models.

In this study, we concentrate on the segmentation of the subbasal nerves in confocal microscopy images using deep learning methods. A corneal nerve segmentation network (CNS-Net) has previously been established with CNN for corneal subbasal nerve segmentation in IVCM. Supervised learning has been successfully applied to trace corneal nerves in IVCM using clinical data automatically., Deep convolutional neural network architectures for the automatic analysis of subbasal nerves have been used.

The purpose of this study is to establish a generative adversarial network for automated corneal subbasal nerve fiber (CNF) segmentation and evaluation within IVCM. GANs are specific deep learning structures that have recently attracted extensive attention. GANs consist of two separate networks, which are generators and discriminators, and they have proved to be able to produce images which are similar to original ones. The original GANs use random noise data as input, whereas conditional GANs (CGANs) integrate input images as a conditioning variable. Thus, this allows the network to act as a picture-to-picture translator., We compare this GAN-based method with the U-net network, which is a widely applied network structure for image segmentation. Last, we have applied different types of noise on IVCM images to simulate realistic noise often encountered in the ophthalmology clinics and demonstrate performance changes in different neural network-based methods.

Methods

In Vivo Confocal Microscopy Image Acquisition

This study was approved by the local medical ethics committee and adhered to the tenets of the World Medical Association Declaration of Helsinki. After written informed consent was obtained from all subjects, a complete ophthalmological examination was performed and both corneas were photographed with an in vivo confocal microscopy (Heidelberg Retinal Tomography 3 Rostock Cornea Module, Heidelberg Engineering GmbH, Heidelberg, Germany).

In this study, the main purpose was the comparison of U-Net and GAN-based algorithms on automatic segmentation of corneal subbasal nerves in the IVCM images. Because of this, the healthy subjects and the patients with chronic ocular surface problems (peripheral neuropathy, meibomian gland dysfunction, etc.) were randomly included to simulate clinical settings. All subjects were surveyed regarding eye dryness, burning, aching, epiphora, and contact lens to remove the effects of any other factor on the subbasal nerve plexus. The subjects who did not meet with specified conditions were excluded from the study.

All subjects underwent corneal IVCM to analyze the total fiber length of the corneal subbasal nerve plexus (CNFL). Before the examination, two drops of local anesthetic (Oxybuprocaine hydrochloride 0.4%) and a drop of lubricant (Carbomer 2 mg/g liquid gel) were applied to both eyes. The IVCM images were taken in Heidelberg Retinal Tomography 3 Rostock Cornea Module. Each image size was 384 × 384 pixels, which covers 400 µm × 400 µm area on the cornea. The full thickness of the central cornea was scanned using the section mode. The duration of the examination was approximately 5 minutes for each eye. Three images per eye were selected, and a total of 510 images from 85 subjects were segmented by 2 blinded experienced graders (authors E.Y. and A.Y.T.). The segmented images were selected for deep learning methods by one blinded experienced IVCM expert (author A.S.) to validate corneal subbasal nerve segmentation. Graders made all image segmentation using ImageJ software (NIH, Bethesda, MD, USA; http://imagej.nih.gov/ij). NeuronJ, which is the most commonly used semi-automated nerve analysis plug-in of ImageJ, was used to quantify nerve fibers. All visible nerve fibers were traced in the image, and their total length was calculated as µm. After manual segmentation of IVCM images by three blinded graders, raw IVCM images and their nerve tracing masks were uploaded to the IVCM image database without any patient information (Fig. 1). Five hundred five (505) of 510 IVCM images had been uploaded with segmentation masks for deep learning. Five (5) of 510 IVCM images were excluded from the study due to several artifacts.

Figure 1.

Example IVCM images and masks for U-Net and GAN-based segmentation methods.

Design of Neural Networks for IVCM Image Analysis

The downsampling and upsampling blocks consist of 2 convolutional layers with a kernel size of 3 × 3 and ReLU activations for U-Net (Fig. 2). The input IVCM image sizes were 384 × 384 in this study. At the end of the pipeline, the final convolutional layer was activated with the logistic (sigmoid) function. Two networks’ gradual improvement at each epoch was conducted at the end of the training process for GAN. The generator was trained to produce the real images (segmented neurite maps), and the discriminator was trained to differentiate the real ones (the ground-truth maps) from those generated by the generator. The generator was expected to become a predictor within an error rate when the IVCM image was given as input. The CGAN Network structure is shown in Figure 3.

Figure 2.

U-Net structure.

Figure 3.

General Conditional Generative Adversarial Network structure used in the study. Blue mask generated by the generator is concatenated with the input image and labeled as fake, while the green mask is obtained by experts and labelled as real.

A similar approach was previously applied to retinal fundus images to predict vessel maps, where the network structure was named V-GAN. Various types of a discriminator for generative adversarial networks, such as whole image (an ImageGAN), a single-pixel (a PixelGAN), or an NxN patch size (a PatchGAN), could be selected. The discriminator tries to evaluate the truthfulness of the generated region of the image in all these types. A patch size of 3 × 3 was evaluated in this study. A modified version of U-Net was constructed for the generator part of GAN. The generator and the discriminator (PatchGAN) network structures are presented in Figure 4 and Figure 5, respectively.

Figure 4.

Generator part of GAN structure.

Figure 5.

PatchGAN structure used for the discriminator part for GAN.

The Dataset and The Training Process

The dataset was constructed with 505 image pairs taken from 85 patients. Each image pair consisted of an input IVCM image and a corresponding neurite binary map image. The latter was created from .ndf files annotated by experts in the laboratory using the NeuronJ plugin of the ImageJ suite. The dataset was further divided into two subsets: training and testing. The distribution was randomly conducted, and the test group had 102 image pairs whereas the training group had 403. The selection in dividing the images into groups was based on individuals and not images to ensure that the network was tested on images not similar to the ones used in the training process. Training dataset input images were augmented with several image operations, including flipping horizontally, contrast, gamma, and brightness adjustments to increase the variance in the input space. Any augmentation function was randomly applied or not applied at each update step (i.e. with each batch in every epoch), with a probability of 0.5 from a uniform distribution.

The networks and training processes were utilized with the TensorFlow framework. The Adam optimizer function with an initial learning rate of 0.0002 and a beta value of 0.5 is run while the learning rate was decreased over time with a scheduler. In GAN training, both the generator and the discriminator were updated at each iteration step. The training was conducted on a single NVIDIA Tesla V100 GPU card (approximately 0.2 image per second) and loss curves for training and validation loss over epochs were generated to examine over-fitting issues (Supplementary Fig. S1).

In order to assess the robustness of the trained networks under noisy inputs, three separate noisy images were artificially generated and evaluated. These were (1) Gaussian distributed noise with the standard deviation of 0.1, (2) salt and pepper noise with the proportion value of 0.034, and (3) speckle noise with the standard deviation of 0.25. All noises were randomly distributed, and the parameters were chosen such that all of the noisy inputs had approximately the same peak signal-to-noise ratios (PSNRs) at 20.0 according to previous literature and international standards., The mean signal-to-noise ratio was calculated approximately as 12.0 (Supplementary Fig. S2).

Statistical Analysis

Intraclass correlation coefficient (ICC) was calculated from CNFL by an experienced third grader (author A.S.) to confirm inter-rater reliability between experts and to validate manual segmentation results of two graders (authors E.Y. and A.Y.T.). ICC between two graders had been found 0.97 (0.96–0.98) with a 95% confidence interval. Segmentation procedure repeated by experienced IVCM expert (author A.S.) in disputed images to increase the reliability of the manual segmentation.

Corneal nerve fiber segmentation results of U-Net and GAN-based algorithms were compared with the manual segmentation results as the ground truth. Image to image comparison results of different algorithms analyzed with different statistical tests for all diagnostic parameters. Corneal subbasal nerve segmentation results of U-Net and GAN-based methods were statistically analyzed with Pearson's R correlation for correlation coefficient and Bland-Altman analysis for bias calculation. Accuracy, sensitivity, and specificity of U-Net and GAN-based methods were also calculated with receiver operating characteristics (ROC) from the confusion matrix. Two-sided p value < 0.05, which was also equal to 95% confidence interval and ±1.96 standard deviation (SD) interval, was considered statistically significant. For Pearson's R correlation, r > 0.80 was considered as a solid positive correlation. Statistical analysis results were indicated as (mean ± SD) or the coefficient of repeatability CR (−1.96 SD +1.96 SD), which is calculated according to the original article of Bland and Altman published in 1986.

Results

U-Net and GAN-Based Segmentation Results

The U-Net based segmentation method demonstrated high levels of correlation (r = 0.883, R2 = 0.779) and low levels of bias (1.189 ± 1.929) compared to the segmentation results of the experts (Fig. 6). The coefficient of repeatability was also calculated as 4.425 (3.892–5.128). These results showed resemblance with U-Net based DeepNerve algorithm, which was used for corneal subbasal nerve segmentation of macaques.

Figure 6.

Correlation (a) and Bland Altman (b) plots for U-Net structure.

The GAN algorithm also performed similarly to U-Net in correlation and Bland Altman analysis for subbasal nerve segmentation from IVCM images (Fig. 7). GAN-based segmentation of corneal subbasal nerves in IVCM images showed high correlation (r = 0.847, R2 = 0.717) and low levels of bias (3.279 ± 2.141) compared to the results of the experts’ segmentation. In Bland-Altman's analysis, the coefficient of repeatability was also calculated as 7.664 (6.741–8.882). It could be noticed that GAN displayed similar performance with U-Net based algorithm for segmentation of corneal subbasal nerves in IVCM images.

Figure 7.

Correlation (a) and Bland Altman (b) plots for GAN structure.

We also observed that GAN showed significantly higher accuracy for the segmentation of IVCM images when ROC curves were compared for both methods. Whereas the area under the curve for U-Net was 0.8934, the area under the curve for GAN was 0.9439 (Fig. 8). When IVCM images were compared one by one, U-Net and GAN-based algorithms masked similar areas in high-quality IVCM images. On the other hand, the U-Net based segmentation method showed a higher number of false-positive pixels for images with artifacts (e.g. uneven illumination and device-related noise) and distractor objects (e.g. dendritic cells, damaged nerves, and corneal foldings) compared to the GAN algorithm (see Fig. 1). Therefore, we had included three of the most common noise types in medical imaging to the IVCM images and then applied automatic segmentation with U-Net and GAN-based algorithms.

Figure 8.

ROC curves for GAN and U-Net structures.

Noise Simulation Results

Three different types of noises were applied on IVCM images to simulate daily challenges in ophthalmology clinics for neural networks-based segmentation of IVCM images. Gaussian, salt and pepper (S&P), and speckle type noises with 12 signal-to-noise ratio (SNR), and 20 peak SNR levels were applied on IVCM images. The accuracy of the U-Net significantly decreased with speckle type noise (0.818 ± 0.029) compared to the original images (0.893 ± 0.022) and the images with other noises (0.883 ± 0.023 for Gaussian, 0.879 ± 0.024 for S&P). For the GAN-based method, Gaussian and S&P noises (0.915 ± 0.020 for Gaussian and 0.914 ± 0.020 for S&P) did not decrease the accuracy of corneal subbasal nerve segmentation significantly. However, speckle type noise (0.878 ± 0.024) significantly decreased the accuracy of the automatic segmentation of GAN compared to the results without noise (0.944 ± 0.015; Fig. 9).

Figure 9.

ROC curves for U-Net (a) and GAN (b) structures with different noises.

Discussion

In our study, we have compared U-Net and GAN-based corneal subbasal nerve segmentation methods in IVCM images. Although U-Net and GAN demonstrated equal performance in correlation and Bland Altman analyses, GAN showed significantly higher accuracy in ROC analysis. Additionally, we have simulated noise types, which clinicians often encounter during medical image acquisition, by applying Gaussian, S&P, and speckle type noises with equal SNR levels. Both U-Net and GAN-based segmentation methods were affected at most by speckle type noise. The decrease in U-Net based segmentation algorithm accuracy was significantly higher than the GAN-based segmentation algorithm in all types of noises.

Our study is the first application of GAN-based algorithms on IVCM images, to the best of our knowledge. GAN-based algorithms have been studied to the segmentation of medical images from various imaging modalities, such as computed tomography, magnetic resonance, x-radiation, and ultrasound imaging. In ophthalmology, GANs are used for the segmentation of retinal vessels in fundus images., Traditional segmentation approaches, such as graph-cut methods, have utilized pixel-wise correspondence over the decades with some significant caveats, including artifacts and leakages. GANs have a potential to bring out the best of the approaches, where the discriminator unit is acting as a shape regulator. Although the effect of regularization is reported to be more eminent with compact shapes, unlike vessels or neurites, the discriminator's perception size might be set up from the whole image to a single pixel, as in the ImageGAN, PatchGAN, and PixelGAN examples. To regularize the network and overcome the collapses in the training process, Li and Shen proposed a method that combines CGAN with AC-GAN and introduces a classifier loss term to their structure in the cell segmentation task. Although a pre-processing step is conducted in a study compared to ours, the classification loss is shown to improve the resulting segmentations. Unannotated images and the annotated ones may also be fed into the segmentation workflow where the former ones will help the training, leading to a more robust discrimination process and more accurate generated segmentation maps. The main disadvantage of GAN-based algorithms is training separate networks together. During the training of the networks, GAN creates a zero-sum game to reach an equilibrium point between these networks, which is intuitively in contrast to the conventional algorithms, where the objective function is to minimize the loss of function. Therefore, several issues are observed in training of GANs, as experienced in this study as well, including oscillations, mode collapses, diminishing, or exploding gradients. In order to overcome and minimize these effects, a tedious fine-tuning approach must be followed. On the other hand, GANs have two significant advantages over U-Net or similar structures. Firstly, GANs let the generator networks produce near-realistic images. In medical imaging literature, this is efficiently utilized in image synthesis where the number of the images of a dataset is insufficient, as in many situations in medical image acquisition, to expand the datasets. Second, GANs are more robust to artifacts and digital image noise sources in general with the help of the discriminator enforcing the generator to produce better outputs (e.g. segmentation maps in this case).

Segmentation of corneal subbasal nerves in IVCM images is still made manually by the experts in ophthalmology clinics. Existing automated corneal subbasal nerve analysis software still has lower accuracy compared to the experts in clinical applications. Neural networks-based algorithms are significantly superior to the other automated corneal subbasal nerve segmentation software, and preliminary studies hope about clinical applications., Different types of neural network-based methods have been used for corneal subbasal nerve segmentation, but most of the studies concentrated on U-Net based algorithms., Oakley et al. produced an auto-encoder and U-Net based neural network algorithm with high accuracy for corneal subbasal nerve segmentation in IVCM images of macaques. They have implemented pre-processing and post-processing algorithms to increase the accuracy of U-Net based segmentation methods. Wei et al. built a CNN-based structure for corneal subbasal nerve segmentation in IVCM images and tested it on patients acquired IVCM images in an ophthalmology clinic. Unfortunately, this study did not make inter-rater reliability measurement, subject separation, and correlation analysis to increase the reliability of the segmentation method. Our study compared the GAN-based corneal subbasal nerve segmentation algorithm with a similar CNN-based algorithm of Wei et al., and we showed significantly higher accuracy for corneal subbasal nerve segmentation compared to the CNN-based method. We applied a three-grader system and inter-rater reliability test to increase the reliability of manual segmentation of corneal subbasal nerves. We did not use any pre-processing or post-processing algorithms to show the actual capacity of the GAN-based segmentation algorithm. In addition to them, we made subject separation for test and training image sets and applied common types of medical imaging noises in the experiments.

In our study, the pre-processing methods for image analysis, such as background subtraction or edge detection algorithms, were not applied to the IVCM images to assess the actual capacities of U-Net and GAN-based algorithms. As with any deep learning-based technique, such a task requires numerous training images to generalize the underlying structures to be predicted. The more IVCM images collected and annotated by experts, the more robust and efficient the networks become. The model selection and adapting, hyperparameter optimization, and convergence of the networks are only some of the significant difficulties in most deep learning-based studies. Particularly, GANs are shown to be prone to some limitations in training due to several reasons, including nonconvergence issues where the model oscillates or never converges, vanishing or exploding gradients where the discriminator overwhelms over the generator, and mode collapses in which the generator does not improve. Even though the cohort size of the study was small compared to other studies, GANs showed significantly better performance than CNN based algorithms. Good performance of the GAN based algorithms in augmented and limited data sets has been shown in previous literature. Because of privacy concerns, access to biomedical images for deep learning applications decreased in recent years. We also applied image augmentation, and it helped the training process with small-sized data sets by increasing the variance within the input space. These augmentation operators were selected randomly whether to apply or not at each batch step of the training process. GANs with augmentation operators could be helpful for biomedical image analysis with a minimal number of images for various modalities.

GAN-based algorithms promise high hopes for biomedical imaging applications. In this first GAN study on IVCM images, we have focused on corneal subbasal nerve segmentation. However, applications of GAN-based algorithms in IVCM images could not be limited to the segmentation of the corneal nerve plexus. GAN-based algorithms also could be used in IVCM images of different layers of the cornea. GANs are currently used for image reconstruction, denoising, super-resolution, classification, object detection, and cross-modality image synthesis in various biomedical imaging modalities, and they also could be used in IVCM images for similar purposes. Because of their generator-discriminator reciprocal structure, they have significantly higher success rates than the other deep learning algorithms. Because of their structures, GANs could be precious diagnostic tools to discriminate lesions in specific conditions with a very limited number of images. In this study, we mainly focused on its capability of segmentation for corneal subbasal nerve plexus of the patients with ocular surface problems and healthy subjects. In further studies, we will investigate its capability of diagnostic classification for various corneal nerve-related lesions and diagnostic precision of various corneal pathologies in IVCM images.

Conclusions

The conjunction of Minsky's two important works, IVCM and artificial intelligence, is opening new horizons in medical imaging. Generative adversarial networks are still in their infancy, but their applications on various medical imaging modalities indicate great potentials. In our study, we compared GAN with U-Net, which is a commonly used deep learning method for image segmentation, for segmentation of IVCM images. Our study demonstrated the superiority of the GAN-based algorithm for accurate segmentation of corneal subbasal nerves compared to the U-Net based algorithm.