Optical coherence tomography angiography image quality assessment at varying retinal expertise levels.

PURPOSE: To compare optical coherence tomography angiography (OCT-A) image quality gradings performed by readers of varying retinal expertise levels in different retinal diseases.
METHODS: Central 3 × 3 mm2 OCT-A images (AngioVue, Optovue) of 57 healthy controls (50.9 ± 22.4 years) and 148 patients (66.5 ± 14.1 years) affected by various chorioretinal diseases were retrospectively analyzed including early age-related macular degeneration (AMD, n = 26), neovascular AMD (nAMD, n = 22), and geographic atrophy due to AMD (GA, n = 6), glaucoma (n = 28), central serous chorioretinopathy (CSC, n = 14), epiretinal membrane (EM, n = 26), retinitis pigmentosa (RP, n = 16), and retinal venous occlusion (RVO, n = 10). A senior expert in medical retina (SE), an ophthalmology resident (OR), and a non-ophthalmologic medical doctor (MD) independently assessed OCT-A image quality using the motion artifact score (MAS) and the segmentation accuracy score (SAS).
RESULTS: Regarding MAS, inter-reader agreement between SE and OR was 93.7% (Cohen's kappa = 0.907) and 85.4% (Cohen's kappa = 0.786) between SE and MD. Regarding SAS, inter-reader agreement between SE and OR was 95.1% (Cohen's kappa = 0.92) and 92.2% (Cohen's kappa = 0.874) between SE and MD. In the SAS analysis, signal strength index (SSI) and presence of retinal pathology had a significant influence on the overall agreement (P = 0.046; P < 0.001).
CONCLUSIONS: OCT-A image quality assessment can be performed most reliably by an ophthalmologist with knowledge in retinal image analysis. Yet, well-instructed non-ophthalmologic assessors show only slightly inferior results and, thus, may be integrated in routine OCT-A image quality assessment as well.

Introduction

The introduction of optical coherence tomography angiography (OCT-A) marks the beginning of a new era in retinal imaging.1, 2 This recent modality has already improved the visualization of different pathologies, and it also seems to provide new perspectives on the pathophysiology of chorioretinal diseases.3, 4 Compared to two-dimensional, en-face modalities like fluorescein angiography or single structural OCT B-scans, OCT-A imaging is based on a more comprehensive data set. Therefore, OCT-A image quality is also affected by more variables, and its evaluation becomes automatically more elaborate.

Presently, image quality represents one major limitation of this new imaging modality mainly determined by the presence of imaging artifacts. Many such artifacts have been identified in recent studies, and an increasing number of authors address the issue of how to systematically assess OCT-A image quality.5, 6, 7, 8 Besides, studies have shown that if such image artifacts and errors are not identified and corrected, qualitative and quantitative image data become inevitably flawed and both intra-individually and inter-individually incomparable.7, 9, 10 Two important factors determining OCT-A image quality are motion artifacts and segmentation errors. Both must be identified and ruled out prior to a detailed qualitative or quantitative image analysis.

Recently, we introduced a motion artifact score (MAS) to systematically assess motion related artifacts in OCT-A imaging and a segmentation accuracy score (SAS) to evaluate the success of automatic segmentation in different pathologies.8, 11 The aim of this study was to compare OCT-A image quality assessments with regards to motion artifacts and segmentation accuracy performed by graders holding varying degrees of expertise in retinal diseases.

Methods

In a retrospective analysis, 205 subjects were included. All participants were identified from the database of the medical retina clinic of the Department of the University of Muenster Medical Center between August 2016 and December 2017 including 57 eyes of 57 healthy controls (aged 50.9 ± 22.4 years) and 148 eyes of 148 patients (aged 66.5 ± 14.1 years). Parts of this patient group had been analyzed with regards to image artifacts before. Included patients were affected by early/intermediate age-related macular degeneration (AMD, n = 26), neovascular AMD (nAMD, n = 22), geographic atrophy due to AMD (GA, n = 6), glaucoma (n = 28), central serous chorioretinopathy (CSC, n = 14), epiretinal membrane (EM, n = 26), retinitis pigmentosa (RP, n = 16), and retinal venous occlusion (RVO, n = 10). If image data from multiple examinations were available, the image with the highest signal strength index (SSI) was chosen. SSI and best corrected visual acuity (BCVA) were documented. Eyes with media opacities or a history of refractive surgery or intraocular inflammation were not included. An additional exclusion criterion was dry eye disease which often causes an inferior imaging quality. Informed consent was obtained from each included patient. Study procedures adhered to the tenets of the Declaration of Helsinki.

Optical coherence tomography angiography

OCT-A imaging was conducted after pupillary dilation with a commercial spectral domain OCT-system (AngioVue, RTVue XR Avanti SD-OCT, Optovue, Fremont, CA, USA) using the 3 × 3 mm2 field as described elsewhere.8, 11 Imaging was performed using eye tracking function (DualTrac™) of the device as well as an artifact removal function. Images with inferior SSI values (<45) were excluded. The proprietary software performs an automatic segmentation according to a set of reference planes.

Parameters for optical coherence tomography angiography image quality evaluation and comparison between graders were

OCT-A motion artifact score (MAS)

OCT-A segmentation accuracy score (SAS)

OCT-A MAS includes motion artifacts caused by eye movement and motion artifacts due to software correction of eye movement. The presence/absence or degree of these five artifacts in the en-face OCT-A image of the superficial capillary plexus defines the attribution to OCT-A MAS 1 through 4 (Fig. 1, Table 1). The application of MAS generally requires an image centered on the fovea as well as the absence of vitreous opacities and clear media.

Fig. 1

Examples of reduced optical coherence tomography angiography (OCT-A) image quality due to motion artifacts and segmentation errors. All 3 × 3 mm2 en-face images of the superficial capillary plexus are shown with corresponding B-scan of the marked position below (white line). (a) exemplarily shows a healthy subject without any motion artifact. (b) shows a healthy subject with distinct vessel doubling. Note the duplication of the vessels along the course, The OCT scan below elucidates the origin of vessel doubling in the en-face image due to motion in between scanning sequences. As the image shows vessel doubling in more than two quadrants, it serves as an example for motion artifact score (MAS) 4. (c) illustrates the motion artifact type of a black line (white arrow) in a patient with intermediate age-related macular degeneration (AMD) (d) shows segmentation errors in a patient with a retinal vein occlusion. Thin white line marks the localization of the automatic segmentation of the inner plexiform layer (IPL). Due to pathologic changes in this patient such as inner retinal cysts, subneurosensory fluid and edematous swelling of the inner retinal layers, the automatic segmentation algorithm misinterprets the inner retinal bands and erroneously sets the IPL segmentation too far posteriorly resulting in a false en-face image.

Table 1

Motion artifact score (MAS).

Motion artifact score (MAS)
1	No or slight quilting, absence of all other artifacts due to motion or software correction
2	Slight or moderate quilting, non-significant black line
3	Moderate quilting or significant quilting in one or two quadrants, displacement in one or two quadrants, vessel doubling in one or two quadrants, stretch artifacts in one or two quadrants, non-significant black line
4	Significant quilting in more than two quadrants, displacement in more than two quadrants, vessel doubling in more than two quadrants, stretch artifacts in more than two quadrants, significant black line

The OCT-A SAS is evaluated for all reference planes in all OCT B-scans. Segmentation was regarded as inaccurate if either segmentation deviated from the correct plane by more than 50 μm.8, 13, 14 Deviation of correct segmentation was quantified using the caliper function of the integrated software tool. SAS I was defined as ≤ 5% of all scans with inaccurate segmentation in either reference plane and SAS II as > 5% of all scans with inaccurate segmentation. In SAS IIA, segmentation errors occurred only in one reference plane, while in SAS IIB, errors were present in more than one reference plane (Fig. 1, Table 2).

Table 2

Segmentation accuracy score (SAS).

Segmentation accuracy score (SAS)
1	Incorrect segmentation in ≤ 5% of all scans
2	Incorrect segmentation in >5% of all scans	A	Segmentation error in only one segmentation boundary [Inner limiting membrane, Inner plexiform layer (IPL) or retinal pigment epithelium reference]
		B	Segmentation error in two or more segmentation boundaries

All images were independently graded by a senior expert in the field of retinal imaging (SE), an ophthalmology resident (OR), and a non-ophthalmologic medical doctor (MD) for assessing MAS and SAS. The SE reader has many years of experience in retinal imaging in clinical routine and clinical research/reading center and served as the reference reader. The OR reader has a three-year experience in retinal imaging in clinical routine and in clinical research particularly OCT and OCT-A. Both are familiar with MAS and SAS. The MD reader had neither experience in practical ophthalmology nor in retinal imaging. Prior to the start of the study, the MD reader was given a detailed tutorial on retinal anatomy, basic principles of OCT and OCT-A as well as the software of the device and the use of MAS and SAS. Additionally, a supervised test-grading was performed in 50 OCT-A images that were not part of the study population.

Statistical methods

The goal was to measure the agreement between raters regarding both the MAS and the SAS to show the inter-rater reliability of both scores. The analysis strategy is only explained for the MAS. The same methods were applied to the SAS.

The SE rating was taken as reference, and the OR rating and the MD rating were both compared to this reference. Agreement was measured as percentage of concordance, i.e. as proportion of cases (eyes) in which the MAS rating of OR (MD) was identical to the MAS rating of SE. The two percentages were compared via Fisher's exact test. Additionally, Cohen's kappa coefficient was calculated to compare the MAS between OR (MD) and SE. The overall agreement was measured as percentage of cases in which the MAS was identical for all three raters, and Fleiss' kappa coefficient was calculated. Additionally, for all given percentages, 95% confidence intervals (CI) were calculated.

Different tests were performed to find out whether there was an effect of patients' characteristics on the overall agreement rate. For binary variables, Fisher's exact test was used. For categorical variables with more than two categories, Chi-squared tests were used. For continuous variables, non-parametric methods were chosen (e.g. Mann-Whitney-U test). Distribution of continuous variables was described by mean ± standard deviation, categorical variables by absolute and relative frequencies.

All analyses are exploratory and P-values and confidence intervals have to be interpreted accordingly. P-values were considered statistically noticeable if P < 0.05 and highly noticeable if P < 0.01. No adjustment for multiple testing was performed.

All analyses were performed using IBM SPSS Statistics 24 for Windows (IBM, Armonk, NY, United States) and SAS software, version 9.4, for Windows (SAS Institute, Cary, NC, USA).

Results

205 [116 (56.6%) females, 89 (43.3%) males] patients and controls were included. Mean age was 62.2 ± 18.2 years (range, 17–94 years) (healthy cohort: 51.0 ± 22.4 years; retinal pathologies: 66.5 ± 14.1 years; P < 0.001). BCVA was 0.61 ± 0.3 (healthy cohort: 0.87 ± 0.22; retinal pathologies: 0.51 ± 0.28; P < 0.001). Mean SSI was 63.7 ± 9.2 (healthy cohort: 69.3 ± 9.8; retinal pathologies: 61.5 ± 4.9; P < 0.001). Mean MAS results of the reference examiner (SE) were 1.92 ± 0.922. The SE attributed 52.7% of the images to SAS I, 27.8% to SAS 2A and 19.5% to SAS 2B.

Inter-reader agreement - motion artifact score

The inter-reader agreement concerning the MAS showed a high concordance, both between SE and OR [93.7% agreement, 95% CI = (89.4%, 96.6%), Cohen's kappa = 0.907] and between SE and MD [85.4% agreement, 95% CI = (79.8%, 89.9%), Cohen's kappa = 0.786] (Fig. 2). The agreement rate between SE and OR was noticeably higher than between SE and MD (P = 0.009, exact fisher test). In all cases of disagreement, there was a difference of one score point between OR/MD and SE. The overall agreement rate was also high with 171/205 cases in which all readers gave the same rating [83.4%, 95% CI = (77.6%, 88.2%), Fleiss Kappa = 0.838].

Fig. 2

Motion artifact score (MAS) agreement rate (with 95% Clopper-Pearson confidence interval) differentiated for all patients (n = 205), the healthy cohort (n = 57), and the cohort with retinal pathologies (n = 148), regarding the overall agreement rate among all three readers, the inter-rater concordance of senior reader and ophthalmology resident (OR) and the inter-rater concordance of senior reader and non-ophthalmologic medical doctor (MD).

Overall inter-reader agreement was neither influenced by age (P = 0.113, Mann-Whitney-U test), gender (P = 0.257, exact fisher test), BCVA (P = 0.276, Mann-Whitney-U test), SSI (P = 0.572 Mann-Whitney-U test), nor presence of retinal pathology (P = 0.403 exact fisher test).

Inter-reader agreement - segmentation accuracy score

The inter-reader agreement concerning SAS showed also a high concordance between SE and OR [95.1% agreement, 95% CI = (91.2%, 97.6%), kappa = 0.920] likewise for SE and MD [92.2% agreement, 95% CI = (87.6%, 95.5%), kappa = 0.874] (Fig. 3). There was no noticeable difference in the agreement rate between SE and OR in comparison with the agreement rate of SE and MD (P = 0.311, exact fisher test). The overall agreement rate was also high with 183/205 cases in which all readers gave the same rating [89.3%, 95% CI = (84.2%, 93.1%), Fleiss Kappa = 0.881].

Fig. 3

Segmentation accuracy score (SAS) agreement rate (with 95% Clopper-Pearson confidence interval) differentiated for all patients (n = 205), the healthy cohort (n = 57), and the cohort with retinal pathologies (n = 148), regarding the overall agreement rate among all three readers, the inter-rater concordance of senior reader and ophthalmology resident (OR) and the inter-rater concordance of senior reader and non-ophthalmologic medical doctor (MD).

Inter-reader agreement was not affected by age (P = 0.261), gender (P = 0.117), or BCVA (P = 0.965), whereas presence of retinal pathology shows a highly noticeable influence on the overall agreement. All 22 cases of disagreement were patients with retinal pathologies. There was no disagreement in the healthy cohort (P < 0.001, exact fisher test).

Most disagreements were found in patients with early/intermediate AMD (disagreement in 8/26 cases, agreement rate = 69.2%), RP (disagreement in 5/16 cases, agreement rate = 68.8%), and patients with GA (disagreement in 2/6 cases, agreement rate = 66.7%). There was also a noticeable difference regarding a favorable influence of a high SSI on the overall reader agreement (P = 0.046, Mann-Whitney-U test).

Discussion

Currently, the clinical applicability of OCT-A is a subject of debate within the retinal specialist community. Partly, the appropriate use for diagnosis and treatment is being discussed controversially. In this context, the importance of OCT-A image quality must be emphasized. Recent studies showed that OCT-A image quality achieved in healthy subjects cannot be maintained in patients with retinal diseases.8, 17 Yet, few studies have been published on OCT-A image quality so far, an absolute requirement for reliably evaluating OCT-A data.

Errors in automatic segmentation, motion artifacts, and projection artifacts represent the major challenges in the field of OCT-A image artifacts. While many studies proved a correct automatic segmentation in normal maculae, algorithms often perform poorly in the presence of pathologic alterations. From a technical point of view, automatic segmentation appears to be the most difficult challenge to address at present whereas OCT-A devices have significantly improved regarding the suppression of motion artifacts. Post-acquisition software as well as eye-tracking systems have contributed to reduce motion artifacts and have been implemented in all OCT-A machines by now.11, 18 In patients with severe macular pathologies and very poor visual acuity, motion artifacts still represent a significant limitation in OCT-A image quality. Projection artifacts are an inherent phenomenon of OCT-A technology. Novel algorithms aim at removing projection artifacts by resolving the ambiguity between in situ and projected flow signals. Such algorithms appear very promising and presumably they will be integrated in commercial OCT-A devices shortly.

Image quality indices are provided by the manufacturers, like the SSI of the device used in this study. As evident in our data, high SSI values may have a positive influence on inter-reader agreement. However, SSI values do not reflect the presence of imaging artifacts. What exactly represents the basis for calculating such indices remains unclear. Notably, as previous studies showed, high manufacturer indices do not preclude motion artifacts or segmentation errors. So far, there is no software available to automatically quantify the extent of segmentation failure and motion artifacts. Therefore, a structured analysis of image quality parameters by a trained reader is mandatory. Otherwise, a valid quantitative OCT-A data analysis is not possible. For this purpose, MAS and SAS proved to be valuable tools.8, 11

Regarding a successful image analysis in ophthalmology, two prerequisites appear fundamental. Firstly, the grader must possess basic knowledge on the technical functioning of the imaging modality being used and the appearance of different types of image artifacts. And he should adopt and keep to a structured image evaluation approach. Secondly, the grader must have a profound knowledge on the retinal anatomy and pathologic changes that may occur in the course of different chorioretinal diseases and disease stages. Currently, knowledge on the influence of reader experience on OCT-A image evaluation in the literature is scarce. Souedan and co-workers compared sensitivity and specificity of choroidal neovascularization (CNV) detection in OCT-A images judged by graders holding varying degrees of expertise in retinal diseases. Among other things, the authors described a rapid learning curve in OCT-A interpretation in the least experienced group and in general a good sensitivity and specificity in CNV detection with comparable success among different expertise levels. However, non-ophthalmologic readers were not included.

As evident in our data, agreement between both ophthalmologic readers in our study was better compared to the agreement between the senior ophthalmologic reader and the non-ophthalmologic reader. The non-ophthalmologic reader in our study is a resident in radiology, not familiar with retinal imaging or retinal pathologies, but experienced in image analysis, particularly in analyzing a large amount of image data in a short period of time. Presumably, a reader without any experience in image analysis would have shown even less concordance with the results of the senior expert. Apparently, a certain level of ophthalmologic knowledge is required to warrant the highest possible grading quality. Similar results were presented by Guagliano et al. who reported a high inter-rater agreement between trainee ophthalmologists and expert ophthalmologists in interpreting fundus photography images and fluorescein angiography images of retinopathy of prematurity patients. Interpretation of image data is ubiquitous in today's clinical ophthalmologic practice, and ophthalmology residents become familiar with the task of evaluating findings in ophthalmologic imaging quickly. The basic knowledge on retinal diseases and the familiarity with image analysis presumably explains the good agreement between the senior ophthalmologist and the OR, consistent with the results of Souedan and Guagliano.20, 21 The results further suggest that an OCT-A image quality assessment should be performed by ophthalmologists. In many chorioretinal diseases, findings in retinal imaging play an important role in therapeutic decisions. Therefore, a high image quality followed by an accurate image evaluation is compulsory. For example, in the Comparison of Age-related Macular Degeneration Treatments Trials, a comparison between the treatment decisions by ophthalmologists and the identification of fluid on OCT scans by the reading center showed an agreement in only 69% of all examinations. Inconsistencies were instances of missed anti-VEGF treatments when the patient was not treated by the ophthalmologist although the reading center detected fluid.

Delegating image quality gradings to completely untrained non-ophthalmologic staff without a certain knowledge in retinal diseases does not appear as a suitable solution to facilitate the work load of image evaluation in OCT-A without reducing the quality of image evaluation at the same time. However, as our data show, with prior instructions regarding image quality analysis and basic medical knowledge, a non-ophthalmologic grader can reach agreement rates only slightly inferior to those of an OR. Possibly, the integration of deep learning algorithms represents a future chance to optimize processes and working speed particularly in the context of growing image data volumes. Recent studies have shown that software algorithms are capable of differentiating OCT scans of normal maculae from OCT scans with alterations due to AMD. Furthermore, algorithms can be taught to recognize typical image patterns of various retinal diseases with a high reliability. In the future, such deep learning algorithms should be used to recognize MAS- and SAS-relevant artifacts. Software assistance such as warnings in case of inferior image quality appears reasonable.

There are several limitations to our study. The analysis was restricted to motion-related artifacts and segmentation errors. The results cannot be transferred to other OCT-A devices or other software algorithms. The disease stage affects the prevalence of both artifact types. Yet, in all pathology groups, mild and severe forms of the disease were present. Readers cannot be blinded to the diagnosis of the eye being graded. They were, however, blinded to the gradings of fellow readers.

In conclusion, OCT-A image quality assessment can be performed most reliably by an ophthalmologist with knowledge in retinal image analysis. Yet, well-instructed non-ophthalmologic assessors show only slightly inferior results and, thus, may be integrated in routine OCT-A image quality assessment as well.