A multi-center study of prediction of macular hole status after vitrectomy and internal limiting membrane peeling by a deep learning model.

BACKGROUND: To develop a deep learning (DL) model for prediction of idiopathic macular hole (MH) status after vitrectomy and internal limiting membrane peeling (VILMP) based on optical coherence tomography (OCT) images from four ophthalmic centers.
METHODS: Eyes followed up at 1 month after VILMP for full-thickness MH were included. In the internal training set, 920 preoperative macular OCT images (as the input) and post-operative status of MH (closed or open, as the output) of 256 eyes from two ophthalmic centers were used to train the DL model using VGG16 algorithm. In the external validation set, 72 preoperative macular OCT images of 36 MH eyes treated by VILMP from another two ophthalmic centers were used to validate the prediction accuracy of the DL model.
RESULTS: In internal training, the mean of overall accuracy for prediction of MH status after VILMP was 84.6% with a mean area under the receiver operating characteristic (ROC) curve (AUC) of 91.04% (sensitivity 85.37% and specificity 81.99%). In external validation, the overall accuracy of predicting MH status after VILMP was 84.7% with an AUC of 89.32% (sensitivity 83.33% and specificity 87.50%). The heatmaps showed that the area critical for prediction was at the central macula, mainly at the MH and its adjacent retina.
CONCLUSIONS: The DL model trained by preoperative macular OCT images can be used to predict postoperative MH status after VILMP. The prediction accuracy of our DL model has been validated by multiple ophthalmic centers. 2021 Annals of Translational Medicine. All rights reserved.

Introduction

Idiopathic macular hole (MH) is a discontinuation of the neurosensory retina at the center of the macula (1,2). The mechanism of MH development is believed to be caused by pathological vitreoretinal traction at the macula (3,4). Patients with MH typically experience progressive visual impairment and metamorphopsia (5). The prevalence of MH ranges from 0.1% to 0.8% in adults over 44 years old (6-9), while the age- and sex-adjusted incidence has been reported to be 7.8 persons or 8.69 eyes per 100,000 population (10). According to the Beijing Eye Study, the prevalence of MH is 0.09%±3.04% in China (7). Older age and female gender are possible risk factors for MH development (5,10,11). Patients with full-thickness MH in one eye are also at risk of MH development in the other eye (5,12). Vitrectomy and internal limiting membrane peeling (VILMP) has been commonly used to treat MH, achieving a success rate of within 80–95% (13-17). A standard VILMP surgery includes a three-port vitrectomy (23-gauge or 25-gauge), internal limiting membrane (ILM) peeling with or without staining, and air tamponade.

Despite high success rate of MH surgery, in some cases the MH remains open after routine VILMP (18). In patients with an open MH after initial surgery, a second surgery is often required. However, the second surgery is mainly associated with high medical costs and less favorable visual outcomes compared with primary closure (19). Therefore, it is of clinical importance to identify MH in the risk of surgical failure after standard VILMP. Risk factors of unclosed MH include older age, larger base diameter, longer minimum linear dimension, longer hole duration, etc. (14,20-22). Among these factors, parameters related to MH dimensions are of significant importance (14,23).

With the development of optical coherence tomography (OCT) technology, it has been widely used for diagnosis and follow-up of MH. To better predict MH status after surgery, several OCT parameters of MH have been proposed (14,23-25). For instance, some researchers have demonstrated that preoperative MH diameter determined by OCT is related to post-operative closure of the MH (25). In another study, it has been shown that the hole form factor (HFF) of MH is associated with anatomical success rate following the initial surgical procedure (23). However, the measurements of these parameters vary between technicians and the prediction accuracy of them is not satisfactory (14).

Deep learning (DL) is state-of-the-art technology in artificial intelligence, which has sparked tremendous global interest over the last few years (26). DL systems have expert-level performance in detecting various ocular diseases including diabetic retinopathy (DR) and age-related macular degeneration (AMD) using clinical ocular images (27,28). On the other hand, DL systems also have been used for predicting treatment outcomes of ocular diseases based on pretreatment clinical images. Recently, a DL-based model developed by Gupta et al. has been applied to predict and monitor retinopathy of prematurity (ROP) regression after treatment (29). However, to the best of our knowledge, no DL model has been developed to predict MH status based on the preoperative macular OCT images. If available, such a DL model could help vitreoretinal surgeons to accurately select patients, who are the most likely to have unclosed MH after routine VILMP, and a new surgical technique such as inverted ILM flap can be suggested to those patients (30).

This study aimed to develop a DL model to automatically predict MH status after routine VILMP based on preoperative macular OCT images from a multi-center population.

Methods

Patients and preparation of datasets

Eyes followed up at 1 month after VILMP for MH were retrospectively included in this study. Eyes with a macular hole caused by known etiologies such as trauma, macular edema, epiretinal membrane, high myopia, retinal detachment or retinoschisis were excluded. All the eyes received comprehensive ophthalmologic examinations including best-corrected visual acuity (BCVA), slit-lamp biomicroscope anterior segment and fundus examination, intraocular pressure (IOP) measurement, and SD-OCT scanning (Spectralis; Heidelberg Engineering, Heidelberg, Germany) before and after VILMP. The study was conducted according to the Declaration of Helsinki (as revised in 2013) and was approved by the Institutional Review Board of GPPH (No. GDREC2020067H). Informed consent was taken from all the patients. A total of 920 preoperative macular OCT images of 256 eyes from the Department of Ophthalmology, Guangdong Provincial People’s Hospital (GDPH, 213 images of 54 eyes) and the Zhongshan Ophthalmic Center (707 images of 202 eyes) were collected for training and internal validation of the proposed DL model. A total of 72 preoperative macular OCT images of 36 eyes from the Department of Ophthalmology, Zhujiang Hospital of Southern Medical University (ZHSMU, 44 images of 27 eyes) and the Department of Ophthalmology, the First Affiliated Hospital of Kunming Medical University (FAHKMU, 28 images of 9 eyes) were included in the external validation dataset.

OCT examination and IMH status labeling

OCT examinations were performed by experienced technicians. A custom of 20°×20° volume acquisition protocol was used to obtain a set of high-speed scans from each eye. With this protocol, 25 horizontal and central vertical cross-sectional B-scan images were obtained, which consisted of 512 A-scans each. The image through the fovea was determined by simultaneous evaluation of the red-free image on the computer monitor of the OCT scanner. To establish a standardized image format of the dataset for subsequent training and validation, all scans were saved in the TIFF format. Before training, all OCT images had undergone a layered labeling system consisting of multiple layers of trained graders with increasing expertise for verification and correction of image labels. The first layer of graders conducted quality control and excluded images with low quality. The images taken from improper positioning, with low signals, or with strong motion artifacts causing misalignment and blurring of sections were excluded. The second layer of graders consisted of two Chinese board-certified ophthalmologists (YH, YX) who independently labeled the MH status (closed or open) according to the OCT images that had passed the first layer. MH closure was defined as restoration of continuity of neurosensory retina at the central fovea in all the post-operative OCT scans. Finally, the third layer grader who was a senior retinal specialist (YH) verified the true labels of MH status and generated the final decision if there was any discordance between the 2 ophthalmologists. The MH status verified by the third layer grader was used as the true label of each OCT image.

Training and validation of DL-based model

The raw OCT images were first preprocessed to normalize the input data. The saturated pixels with an intensity value of 255 were removed from the raw OCT images and the retinal layers based on smooth pixel intensity were cropped. The OCT images were then resized into 224*224 pixels based on the requirements of Visual Geometry Group (VGG) 16 (Department of Engineering Science, the University of Oxford, Oxford, UK) (31). The VGG16 with 16 convolutional layers was used as the benchmark DL-based model in the whole experiments, wherein the OCT images (B-scans containing the macular center) were the input, and the status of MH after VILMP were the output ().

Figure 1

Demonstration of construction of the deep learning model. A deep learning model was trained to predict MH status (closed or open) after surgery using VGG16 network. MH, macular hole; OCT, optical coherence tomography; VGG16, Visual Geometry Group 16 Layers.

The training started with multiple iterations with a batch size of 16 images, with the initial learning rate of 0.001 and stopped at 15 epochs. For each training iteration, a cross-entropy loss function was used as the objective loss function to update the optimization parameters, and a stochastic gradient descent (SGD) algorithm with Nesterov momentum term was used to optimize a pre-defined loss of function to train neuron weights via back propagation. At every epoch, the performance of the convolutional neural network (CNN) was assessed using the validation dataset. The VGG16 model was a binary classification model output by the Softmax classifier, and 2 nodes were used in the last layer to generate predictions.

In the training dataset, there were 96.88% of patients who had more than one OCT image per eye, with an average of 3.6 OCT images per eye. To determine the most appropriate hyperparameters for our final model, the popular ten-fold cross-validation scheme was used on the training dataset, which was divided into ten portions randomly. To obtain fully independent folds, python code was used to read the labels and the corresponding index positions of all images. The code was then used to remove the repeated names of different index position images of the same patient (in other words, only one patient name was kept in several images of the same patient). Therefore, the partition was carried out at the patient-level, and the original index position images belonging to the same patient were divided into the same partition to ensure the completely independent folds. In each run, nine portions of the dataset were employed to train the DL model, and the other one was used for model testing to facilitate parameter selection and tuning. The experiments were conducted until each portion was tested. Using the above-chosen hyperparameters, the DL system was re-trained using the entire training dataset, and its predictive performance was evaluated based on our independent external validation dataset, in which the data were not involved in the development of DL-based method.

Statistical analysis

After training, validating and testing our DL model, we could obtain the predictive output of each OCT image. The exact breakdown of performance regarding the correlation of predicted labels obtained from our DL-based model with true labels was depicted as confusion matrices. The confusion matrices were used to calculate the overall accuracy of prediction of MH status after VILMP.

The area under the receiver operating characteristic (ROC) curve (AUC) was used to evaluate the accuracy of DL model in predicting the MH status after surgery. A series of true positive rates (TPRs) and false positive rates (FPRs) were obtained to form ROC curves. The TPR was also known as sensitivity, and the FPR results were obtained by subtracting the specificity value from 1. The optimal cut-off point was obtained by using the highest Youden’s index (sensitivity+specificity−1), and the corresponding optimal sensitivity and specificity values were recorded. The overall accuracy and the AUC in internal validation were presented as mean and 95% confidence interval (CI).

Visualization method for the proposed DL-based model

To visualize the critical area in OCT images that was highly correlated with MH status after VILMP, Gradient-weighted Class Activation Mapping (Grad-CAM), an approach proposed by Ramprasaath R. Selvaraju et al., was used to aid interpretation of the results and increase the model transparency (32). The gradient indicated the contribution of each spot of the feature map to the output probability value. The region with the largest gradient had the greatest impact on the output probability value. The highlighted region in the heatmap represented the part of the OCT image most critical for accurately predicting the MH status after VILMP.

Results

Demographics of the eyes included were shown in . There were 208 eyes with closed MH and 84 eyes with open MH at 1-month visit (to balance the number of closed and open MH, more eyes with an open MH were deliberately included). In internal validation, the mean overall accuracy for predicting the MH status after VILMP was 84.6% (95% CI: 82.89–86.31%), with a mean area under the receiver operating characteristic (ROC) curve (AUC) of 91.04% (95% CI: 90.09–91.99%, sensitivity 85.37% and specificity 81.99%). In external validation, the overall accuracy for predicting the MH status after VILMP was 84.7%, with an AUC of 89.32% (sensitivity 83.33% and specificity 87.50%). The prediction accuracy of MH closure and opening was 85.4% and 83.3%, respectively (). Heatmaps showed that the area critical for prediction was at the central macula, mainly at the MH and its adjacent retina, suggesting that the DL-based model could assist in successfully identifying the pathologic region that was the most critical for accurate prediction of MH status after VILMP ().

Table 1

Patient demographics

Characteristics	All eyes	Internal validation	External validation
Number of eyes	292	256	36
Age, mean (SD), years	60.36 (10.83)	60.35 (10.31)	60.44 (14.58)
Sex, females, n (%)	185 (63.37)	165 (64.45)	20 (55.56)
Duration of symptoms, mean (SD), months	7.20 (12.35)	7.11 (11.98)	7.89 (15.25)
Preoperative BCVA, mean (SD), logMAR	1.04 (0.43)	1.04 (0.44)	1.01 (0.32)
Number of images	992	920	72
Images with a closed MH, n (%)	681 (68.65)	633 (68.80)	48 (66.67)

MH, macular hole; SD, standard deviation; BCVA, best-corrected visual acuity; logMAR, the logarithm of minimal angle resolution.

Figure 2

The receiver operating characteristic (ROC) curves of the deep learning model. The ROC curves for the prediction task in the internal validation set (red line) and external validation set (blue line). The area under ROC curves (AUC) for the internal validation set and the external validation set was 91.04% and 89.32%, respectively.

Figure 3

Confusion matrix for binary classifications using the deep learning model. Ground true labels are on the vertical axis and predicted labels are on the horizontal axis. (A) Confusion matrix for the internal validation set. The overall accuracy was 84.6%, while the accuracy for predicting macular hole closure or opening was 89.1% and 74.6%, respectively. (B) Confusion matrix for the external validation set. The overall accuracy was 84.7%, while the accuracy for predicting macular hole closure or opening was 85.4% and 83.3%, respectively.

Figure 4

Heatmaps highlighting the pathological area highly correlated with macular hole status after surgery. The heatmaps were generated by Gradient-weighted Class Activation Mapping (Grad-CAM). The heatmaps demonstrate the critical area in optical coherence tomography images that were highly correlated with an accurate prediction of macular hole status after surgery.

Discussion

To the best of our knowledge, the present study is the first to predict the anatomical outcome of standard MH surgery based on preoperative macular OCT images from a multicenter dataset. The results show that the accuracy of our DL model in predicting MH status (i.e., closed or open) is promising. The performance of DL model is as follows: a mean overall accuracy of 84.6% and a mean AUC of 91.04% in internal validation, and an overall accuracy of 84.7% and an AUC of 89.32% in external validation. Moreover, heatmaps of the DL model show that the area highlighted is the most predominant pathologic region of the MH. These findings indicate that our DL model is capable of successfully predicting the MH status by accurately recognizing the critical pathologic region in preoperative OCT images.

The mechanisms of MH development mainly involve anterior and tangential vitreoretinal traction to the fovea (3). Accordingly, release of vitreoretinal traction by VILMP has become a standard treatment for FTHM with success rates of VILMP vary from 80% to 95% (13-17). Despite high success rates of VILMP, the MH remains to be open after uneventful surgery in a small number of cases (18,30). Preoperative OCT parameters of MH are important predictive factors associated with MH status after VILMP surgery (14,20). Therefore, some researchers have suggested that adjusting surgical planning according to OCT parameters of the MH to increase success rates (33). These previous findings constitute the basis of the possibility and necessity for the prediction of MH status after VILMP surgery. However, current measurements of the MH OCT parameters used for predicting are manual, expertise-requiring and time-consuming. The measurements also vary between different technicians. It is particularly beneficial and meaningful to develop an automated system to select MH at a high risk of surgery failure after standard VILMP.

Using the DL technology, information in the preoperative OCT images associated with postoperative MH status can be automatically extracted and processed. The information is unique for each OCT image, and contains parameters that have been reported before and parameters that are unknown so far. For each OCT image, the information is unique and is integrated by DL model. By corresponding the preoperative OCT information to the postoperative MH status after repeated training, the ML model can predict the MH status based on the OCT images. The DL method we have adopted in this study (VGGNet) consists of 16 layers, mainly composing of small 3×3 convolution operations and 2×2 pooling operations (31). Therefore, stacking multiple small convolution kernels can increase the depth of the network to acquire richer image characteristics, while limiting the number of parameters. For example, by stacking three 3×3 convolutional layers instead of using a single 7×7 convolutional layer (i.e., AlexNet), some limitations can be overcome. Firstly, it combines three nonlinear functions rather than one, making the decision function more discriminative and representative. Secondly, the number of parameters is reduced, while the receptive field remains unchanged. In addition, the use of small convolution kernels also plays a role in regulating and improving the effectiveness of different convolution kernels.

DL studies based on OCT images mostly concentrate on image segmentation, involving complex feature selection and extraction. In addition, introducing a minor error in the segmentation can lead to classification error (34). However, this problem can be avoided in the present study by training the DL model to learn the prediction features directly from the OCT images. Therefore, compared to CNN with shallower architectures, our DL-based model can automatically learn richer and more discriminative OCT image features to present an accurate prediction.

Our deep neural network also contains the batch normalization (BN) layer and the dropout layer. The BN layer can pull the distribution of neuron input value of each neural network layer back to the standard normal distribution with a mean of 0 and a variance of 1. Therefore, the BN layer not only improves the training speed, but also prevents gradient exploding and gradient vanishing. Moreover, the BN layer can avoid overfitting to a certain extent. In addition, the dropout layer, placed after the fully connected layer, is used as a trick for training deep neural networks. In each training batch, randomly dropping out half of the feature detectors (assigning half of the hidden layer nodes to 0) can reduce the interaction between the feature detectors (hidden layer nodes), and reduces the complex co-adaptation relationship between the neurons. This approach forces the network more robustly and significantly avoided overfitting.

A major strength of our study is the heterogeneity of the datasets. The DL-based model is trained by OCT images from two different ophthalmic centers, and it has been validated by an independent dataset from another two ophthalmic centers. The heterogeneity of the datasets is very important for the development and validation of DL model (35). Data from patients with different backgrounds and demographics can help train the DL-based model to learn the “true features” critical for making an accurate prediction. Successful validation of the DL models by different external datasets would be more convincing for predicting the ability and popularity of the models. While we increase the heterogeneity of our datasets, we have only included patients with idiopathic MH but not MH secondary to known etiologies such as trauma. This is because MH caused by known etiologies might have different pathogenesis and prognostic factors from idiopathic MH. The inclusion of these cases may reduce the efficacy of training, leading to poor performance of the DL-based model. Thus, to ensure the prediction accuracy of the DL-based model, MH secondary to known etiologies is excluded for the current study.

Another artificial intelligence technology called machine learning (ML) also has been proposed to predict treatment outcomes of some ocular diseases, such as AMD and macular edema. Unlike DL models, ML models use text information instead of images as input. In a recent study, information extracted from OCT images taken during the upload phase of 3 anti-VEGF injections was incorporated into an ML-based model to predict recurrence of macular edema associated with retinal vein occlusion. The results were encouraging with an AUC of 0.76–0.83 (36). However, unlike the severity of macular edema, the parameters of MH morphology are often manually measured, restricting the use of ML model in predicting the MH status after surgery. In the present study, macular OCT image is used as a whole to train the DL-based model. The DL-based model can automatically process and analyze anatomical parameters of the MH. Hence, the problems associated with manual measurement are avoided.

However, there are several limitations in this study. One limitation is the small sample size. One may argue that an external validation with 36 eyes is not convincing enough. Although the sample is not large considering this is a DL-based study, we have shown excellent accuracy of our DL-based model in MH status prediction after VILMP. Moreover, the eyes in the external validation set are obtained from two ophthalmic centers different from the internal validation set, suggesting satisfactory adaptability of the DL model. On the other hand, this is a preliminary study conducted to evaluate the possibility of predicting postoperative MH status using DL model. Further studies with a larger sample size from multiple centers are warranted to validate the results of the present study.

In conclusion, the proposed DL-based model has demonstrated high accuracy and transparency in automated prediction of MH status after VILMP based on preoperative macular OCT images. Our results indicate the feasibility of automatic prediction of MH status after routine vitreoretinal surgery. In case of further improving and fine-tuning, the proposed DL-based model can be used to select patients with a MH that is very unlikely to be closed using routine vitreoretinal surgery and more progressive surgical methods can be suggested to these patients.

The article’s supplementary files as