Clinical validation of RIA-G, an automated optic nerve head analysis software.

Purpose: To clinically validate a new automated glaucoma diagnosis software RIA-G.
Methods: A double-blinded study was conducted where 229 valid random fundus images were evaluated independently by RIA-G and three expert ophthalmologists. Optic nerve head parameters [vertical and horizontal cup-disc ratio (CDR) and neuroretinal rim (NRR) changes] were quantified. Disc damage likelihood scale (DDLS) staging and presence of glaucoma were noted. The software output was compared with consensus values of ophthalmologists.
Results: Mean difference between the vertical CDR output by RIA-G and the ophthalmologists was - 0.004 ± 0.1. Good agreement and strong correlation existed between the two [interclass correlation coefficient (ICC) 0.79; r = 0.77, P < 0.005]. Mean difference for horizontal CDR was - 0.07 ± 0.13 with a moderate to strong agreement and correlation (ICC 0.48; r = 0.61, P < 0.05). Experts and RIA-G found a violation of the inferior-superior NRR in 47 and 54 images, respectively (Cohen's kappa = 0.56 ± 0.07). RIA-G accurately detected DDLS in 66.2% cases, while in 93.8% cases, output was within ± 1 stage (ICC 0.51). Sensitivity and specificity of RIA-G to diagnose glaucomatous neuropathy were 82.3% and 91.8%, respectively. Overall agreement between RIA-G and experts for glaucoma diagnosis was good (Cohen's kappa = 0.62 ± 0.07). Overall accuracy of RIA-G to detect glaucomatous neuropathy was 90.3%. A detection error rate of 5% was noted.
Conclusion: RIA-G showed good agreement with the experts and proved to be a reliable software for detecting glaucomatous optic neuropathy. The ability to quantify optic nerve head parameters from simple fundus photographs will prove particularly useful in glaucoma screening, where no direct patient-doctor contact is established.

RCT Entities:   Population Interventions Outcomes

Optic nerve head changes form the mainstay of glaucoma diagnosis.[12] The gold standard for evaluating the optic nerve head for glaucoma is clinical stereoscopic slit-lamp biomicroscopy aided by retinal nerve fiber layer analysis which requires an expert ophthalmologist and expensive equipment. This method of screening for glaucoma is therefore not suitable in a community seating and telemedicine-based evaluation of fundus images is used. This too requires an ophthalmologist to review the images at the backend. To circumvent this, there have been multiple attempts by biomedical engineers to develop software for automated detection of nerve head changes from relatively inexpensive fundus photographs.[34567891011121314151617181920212223242526]

Unfortunately, over time it has been realized that this is a formidable task given the structural variability of the optic nerve head. Accurate detection of optic nerve head changes is a challenge even for expert ophthalmologists and there is considerable intra- and interobserver variability.[27] If a machine has to detect glaucomatous optic nerve head changes, it would require to detect potential aberrations in the disc, analyze them with respect to probability of these being glaucomatous, and learn from its outputs to continue improving. In essence, it has to use artificial intelligence.

Currently, most experimental attempts at automated image analysis software have been aimed at detecting the cup–disc ratio (CDR) from fundus photographs and this amounts to detecting something best seen in a three-dimensional depth from a two-dimensional image, therefore causing errors. In addition, as we know, this in itself is an incomplete parameter for diagnosing glaucomatous neuropathy. The current software algorithms for CDR detection use various features including color (color cup), blood vessel anatomy (contour cup), best fit shapes (arbitrary cup), edge detection, or manual marking for identification.[25262728] A few software have gone a step further to try and detect the minimum rim width of the neuroretinal rim (NRR).[82026] This gives a distinct advantage over the CDR alone as it may be used to derive the disc damage likelihood scale (DDLS).[2930] However, owing to its complexity, very few automated software have been able to output a DDLS stage and their accuracy levels are relatively low.[3132] One such software recently available for medical use is RIA-G (Kalpah Innovations, Vishakhapatnam, India) which analyzes the optic nerve head to quantify CDR, NRR changes, and DDLS to output the probability of these changes being glaucomatous.

The aim of this study is to validate the new software and establish its sensitivity and specificity for identifying glaucomatous optic neuropathy.

Methods

A double-blinded study was conducted at a tertiary level institution after prior approval from the institutional review board. A total of 275 monoscopic fundus images formed the sampling frame. These were randomly selected (using a random number generator) from a larger database of images which had been clicked by trained ophthalmic technicians at the institute. The selection criteria for an image to be included in the larger database were (1) 30° field of view, (2) macula centered, and (3) entire optic disc visible (without obvious disc abnormalities, e.g. myelinated nerve fibers or disc coloboma). The selected images were then screened for quality by the ophthalmologists who used subjective criteria (ability to discern disc and vasculature details and appropriate exposure) to determine whether they could confidently define the disc parameters. Images that were of unsatisfactory quality were excluded from the analysis.

The selected images were evaluated by RIA-G and outputs relayed directly into an Excel sheet. RIA-G is available as a cloud-based or standalone system where fundus photographs can be uploaded and evaluated by the user for quantifying parameters related to the optic nerve head. The software marks out the disc and cup and provides a quantitative output including the vertical CDR, horizontal CDR, NRR thickness, (Inferior-Superior-Nasal-Temporal) ISNT rule violation, and DDLS stage. It further analyzes these and indicates presence or absence of glaucomatous optic neuropathy. The outputs are color coded into red, yellow, and green depending on the probability of a particular finding being beyond acceptable norms.

The same selected images were also independently evaluated by three ophthalmologists (experts), each at least having 5 years of experience in fundus examination. The parameters reported by them were vertical CDR, horizontal CDR, ISNT rule violation, and DDLS staging. A final decision of whether the image showed a glaucomatous or a nonglaucomatous disc was also recorded by the experts. The final accepted values of the parameters were those which were reported identical by at least two of the three ophthalmologists. If the values reported by all three ophthalmologists for the same parameter differed, that particular parameter was excluded from the analysis. If a particular parameter was not reported by RIA-G, then that parameter was excluded from the analysis. An independent observer compared the outputs of RIA-G with those of the expert ophthalmologists.

Agreement between the raters was defined independently for each parameter. For vertical and horizontal CDRs, a variation within ±0.09 was considered acceptable since the ophthalmologists reported ratio in 1 decimal and the software reported the ratio in 2 decimals. For ISNT rule, a reversal of the inferior–superior (I-S) NRR thickness or the nasal and temporal NRR thickness was reported as violations. The first ISNT analysis was a comparison of the I-S rim to recognize if there was a change in the rim anatomy, and if the inferior rim thickness was noted to be less than the superior rim, a violation was recorded. In the second ISNT analysis, a comparison of the nasal and temporal rim was done, and if there was a change in the rim anatomy and the nasal rim thickness was noted to be less than the temporal rim thickness, it was recorded as a violation. The ophthalmologists reported the DDLS stage and an exact match was needed in the RIA-G report for it to be labeled as concurrent. Finally, the ophthalmologists reported whether glaucomatous optic neuropathy was present or absent (suspects were considered as glaucomatous neuropathy present) and the same was compared with the result from the RIA-G output. Statistical analysis was conducted using SPSS V24.0 (IBM Corp., Armonk, NY, USA). Interrater agreement was evaluated using interclass correlation coefficient (ICC) and Cohen's kappa as appropriate. Sensitivity, specificity, and predictive value were calculated using 2 × 2 tables and appropriate formulae.

Results

A database of 550 unique fundus photographs which met the inclusion criteria were used for the study. A total of 275 fundus photographs were randomly selected and evaluated by the ophthalmologists and RIA-G. Six photographs were rejected by the ophthalmologists in view of poor quality and obscured disc details. Of the remaining 269 images, 15 had a detection error on the RIA-G and were excluded from the analysis. A further 25 images were excluded as there was disagreement between all three experts. For the final analysis, 229 images were considered valid. The clinical and demographic profile of the patients whose photographs were selected is depicted in Table 1.

Table 1

Clinical and demographic profile of patients whose fundus photographs comprised the sampling frame

Demographic and clinical profile
Parameter	Mean/number [mean±SD (median (range)]
Age	58.9±14 [median 60 (6-93)]
Ethnicity	Indian race (n=229)
Ref error	−0.3±2.17 [median 0 (+4.5-−11)] (n=198)
Cup-disc ratio (vertical)	0.54±0.14 [median 0.5 (0.2-0.9)]
Cup-disc ratio (horizontal)	0.48±0.15 [median 0.5 (0.1-0.9)]
Disc size	30 small, 156 average, 44 large
DDLS	3 (1-7)

SD: Standard deviation; DDLS: Disc damage likelihood scale

The mean difference between the vertical CDR defined by RIA-G and the ophthalmologists was − 0.004 ± 0.1 with a slight negative skew of −0.3 indicating that generally the software slightly overreported the vertical CDR [Fig. 1]. There was a strong correlation between the vertical CDR reported by the ophthalmologists and the software. (r = 0.77, P < 0.005). Fig. 2 depicts the Bland–Altman plot for vertical CDR showing that there is no consistent bias of RIA-G outputs versus the experts. However, there is an element of proportional bias of 7.4% and −7.1% in eyes with vertical CDR of ≤0.3 and ≥0.7, respectively. ICC for evaluating agreement between RIA-G and the ophthalmologists was 0.79 (single-rating, absolute-agreement, two-way random-effects model with two raters across 229 subjects) which implies a good degree of agreement.

Figure 1

Vertical cup–disc ratio difference between RIA-G and ophthalmologists. Graphical description of the difference in vertical cup–disc ratio as detected by RIA-G and the ophthalmologists. Note the bell-shaped curve with a slight negative skew indicating that the software slightly overreported the vertical cup–disc ratio

Figure 2

Bland–Altman plot depicting agreement between RIA-G and ophthalmologists with regard to vertical cup–disc ratio. Bland–Altman plot between the mean of vertical cup–disc ratio of RIA-G and ophthalmologists (x-axis) and the difference in vertical cup–disc ratio between RIA-G and ophthalmologists (y-axis). As seen in the plot, there is no consistent bias of RIA-G versus the experts as the data points are nearly equally represented on both sides of the x-axis

The mean difference between the horizontal CDR defined by RIA-G and the ophthalmologists was − 0.07 ± 0.13 with a slight negative skew of − 0.4 indicating that generally the software slightly overreported the horizontal CDR [Fig. 3]. There was a moderate to strong correlation between the horizontal CDR reported by the ophthalmologists and the software. (r = 0.61, P < 0.05). Fig. 4 depicts the Bland–Altman plot for horizontal CDR showing that there is no consistent bias of RIA-G outputs versus the experts. However, there is an element of proportional bias of 35.1% and −25.2% in eyes with horizontal CDR of ≤0.3 and ≥0.7, respectively. ICC for evaluating agreement between RIA-G and the ophthalmologists was 0.48 (single-rating, absolute-agreement, two-way random-effects model with two raters across 229 subjects) which implies an average degree of agreement.

Figure 3

Horizontal cup–disc ratio difference between RIA-G and ophthalmologists. Graphical description of the difference in horizontal cup–disc ratio as detected by RIA-G and ophthalmologists. Note the bell-shaped curve with a slight negative skew indicating that the software slightly overreported the horizontal cup–disc ratio

Figure 4

Bland–Altman plot depicting agreement between RIA-G and ophthalmologists with regard to horizontal cup–disc ratio. Bland–Altman plot between the mean of horizontal cup–disc ratio of RIA-G and the ophthalmologists (x-axis) and the difference in horizontal cup–disc ratio between RIA-G and ophthalmologists (y-axis). As seen in the plot, there is no consistent bias of RIA-G versus the experts as the data points are nearly equally represented on both sides of the x-axis

NRR changes were evaluated as a violation of the ISNT rule. Experts found a violation of the I-S rim in 47 images while RIA-G reported a violation in 54 images. An agreement between the experts and RIA-G was seen in 198 images (85.6%), while in 20 images RIA-G had overreported the I-S violation and had missed it in 11 images. With regard to NRR changes, the interrater agreement between RIA-G and the experts was good with a kappa coefficient of 0.56 ± 0.07 [95% confidence interval (CI) 0.42–0.70]. The sensitivity of RIA-G for detecting I-S violation was 78.3% (95% CI 62.0–87.7), while the specificity was 90% (95% CI 83.7–93.2). The positive predictive value of a RIA-G decision for I-S violation was 70% (95% CI 53.6–73.7). None of the 229 cases had a nasal–temporal violation. RIA-G and the experts had a 100% agreement with regard to this parameter.

In 152 of the 229 cases (66.2%), the DDLS stage of the experts and software had a perfect match. Of the remaining images, RIA-G overreported the stage by 3, 2, and 1 in 3, 7, and 31 cases, respectively. It underreported the stage by 1, 2, and 3 in 32, 3, and 1 cases, respectively. If a variation of ±1 stage from the experts’ consensus is considered acceptable, then RIA-G had a 93.8% agreement with the ophthalmologists. The ICC for evaluating agreement between RIA-G and the ophthalmologists was 0.51 (single-rating, absolute-agreement, two-way random-effects model with two raters across 229 subjects) which implies an average degree of agreement.

The final aspect of the results involves the comprehensive diagnosis of glaucomatous optic neuropathy keeping in account all the parameters defined above. Experts found glaucomatous optic neuropathy in 34 of the 229 cases, while RIA-G reported it in 28 of the 229 cases. The interrater agreement was substantial with a kappa coefficient of 0.62 ± 0.07 (95% CI 0.48–0.77). In 90% of cases, diagnosis of glaucomatous or no glaucomatous optic neuropathy by RIA-G matched that of the experts [Fig. 5]. The sensitivity of RIA-G to diagnose glaucomatous optic neuropathy was found to be 82.3% (95% CI 65.5–93.2), while the specificity was 91.8% (95% CI 86.4–94.8). The positive predictive value of RIA-G for diagnosing glaucomatous optic neuropathy was 63.6% (95% CI 50.5–72.7), while the negative predictive value for ruling it out was 96.7% (95% CI 93.5–98.4). Overall agreement between RIA-G and the experts for identification of glaucomatous neuropathy was 90.4%. Overall accuracy of RIA-G to detect glaucomatous optic neuropathy was 90.3% (95% CI 85.3–93.5).

Figure 5

Venn diagram depicting agreement between RIA-G and ophthalmologists for diagnosis of glaucomatous optic neuropathy. Venn diagram depicting the agreement between RIA-G and experts for final diagnosis of glaucomatous optic neuropathy. Note that in 206/229 cases, the decision of RIA-G matched that of the experts

Sample outputs provided by the RIA-G software demonstrating a case of agreement and disagreement with the experts is shown in Fig. 6. This shows how a real world report is presented to the user after the image analysis is completed [Fig. 6].

Figure 6

Examples of RIA-G output. Output of RIA-G demonstrating the various optic nerve head parameters in (a) A case where there was full agreement between the experts and the software. (b) A case where RIA-G overreported the vertical and horizontal cup–disc ratios and DDLS and wrongly labeled a disc to have a high risk of being glaucomatous. The experts likened the vertical cup–disc and horizontal cup–disc ratio to be 0.6 each and the DDLS to be 2 with no ISNT violation and a low risk of glaucoma

Discussion

Current optic nerve head imaging modalities include Heidelberg retinal tomography (HRT), GDx and optical coherence tomography (OCT).[30] They aid in the diagnostic process by providing quantitative information regarding the clinically visible (optic nerve head) and clinically invisible (retinal nerve fiber layer thickness) parameters. On their own, they have varying sensitivities and specificities for glaucoma diagnosis ranging from 87% and 63.9%, respectively, for HRT to 35.1% and 97.2% for GDx and 76.9% and 78.5%, respectively, for OCT.[30] RIA-G, though not evaluating the retinal nerve fiber layer thickness, supplements clinical decision-making through quantifying the optic nerve head parameters and reducing subjectivity in glaucoma diagnosis. This study found a sensitivity of 82.3% and specificity of 91.8% which is comparable to that achieved by the more expensive technologies above. However, a comparative study is needed to evaluate the different technologies on one stage.

Automated fundus image analysis software are designed to identify and quantify optic nerve head changes. An increase in the vertical CDR is a marker for possible glaucoma and most software rely on this parameter alone. Since there is no other commercially available software which is directly comparable to RIA-G, no real-world studies exist, and hence comparisons with previous experimental studies are discussed herein. Previous laboratory experiments with automated software have found an accuracy of CDR detection ranging between 62.5% and 87%.[610] RIA-G has shown an accuracy of 64.1% which is within the range albeit on the lower side. Previous literature has demonstrated a CDR error varying between 0.064 and 0.09, and this is much higher than that by RIA-G which is 0.004.[91418] This implies that RIA-G makes a very close estimation to that of the expert ophthalmologists.

The NRR is a very important parameter and plays a bigger role in glaucomatous optic neuropathy evaluation than the CDR. Very few software exist which evaluate the NRR in the way that RIA-G does.[1718192023] In one such software, the developers have evaluated the accuracy of rim–disc ratio measurement and found the sensitivity and specificity to be 78.1% and 98.1%, respectively.[10] However, they have not used this for any clinical significance in the article. Rim–disc ratio measurement is important for DDLS scoring. Khan et al. found a sensitivity of 62.5% and specificity of 87.5% for ISNT violation in a dataset of 15 cases which is lower than RIA-G.[17]

The sensitivity of RIA-G to detect glaucoma in the form of glaucomatous optic neuropathy was found to be 82.3%. Chakrabarty et al. and Khan et al. found lower values of sensitivity for glaucoma detection through their software with values of 71.6% and 73%, respectively.[517] However, a few other software relying predominantly on CDR have found better sensitivity of up to 100%.[456]

The positive predictive value of detection of glaucomatous optic neuropathy is moderate, but the negative predictive value is very high at over 96%. This implies that the software will more easily be able to rule out glaucoma than predict its presence. While for most screening tests, the higher positive predictive value is more important, in a field setting, the ability to rule out a potentially sight threatening illness may come handy and prevent unnecessary referrals to a higher center. There is a risk of missing certain cases of glaucoma and the software would need to be tuned for a better positive predictive value.

The DDLS stage was accurately detected in 94% of cases within ±1 stage of the exact stage detected by the ophthalmologists. This holds relevance in a screening scenario where a detection ±1 stage can give a very good idea about the severity of disc damage and the possible need for urgent referral of the patient to a specialist.

The specificity of RIA-G for detection of glaucomatous optic neuropathy was 91.8% which is better than that previously reported in literature. Previous software for automated glaucoma detection have specificity in the range of 71.7%–87% and thus have a lesser ability to confidently diagnose glaucoma when compared with RIA-G.[56715]

The diagnostic accuracy of RIA-G was found to be 90.3% which is at par with that reported by Acharya et al. at 91%, significantly better that that reported by Khan et al. at 82% and slightly lower than the 94% reported by Issac et al.[81417]

The enhanced accuracy, sensitivity, and specificity of RIA-G emanate from its diagnostic algorithm.[22] For detection of glaucomatous optic neuropathy, RIA-G uses the information encoded in the spatial domain and the phase component of the frequency domain (using Fourier transformation) to detect the optic cup and the optic disc. The increasing frequency in images is associated with the abrupt transitions in brightness or pixel value. Furthermore, noise is usually embedded in the high end of the spectrum, so low-pass filtering is used for noise reduction. The optic disc is detected followed by the optic cup. This gives an advantage of decrease in processing time, as the segmentation for optic cup is done only within the segmented disc region. For the segmentation of the optic cup, logical operators are used to limit the processing to within the disc region. The notches and the kinks of the vessels are in-painted using the neighborhood pixel to blend it into the green channel pixel values of the cup. Thus, the cup is detected by taking the contour and kinks into consideration and not the color.

RIA-G evaluates the optic nerve in a holistic manner analyzing more than one parameter to establish the diagnosis of glaucomatous optic neuropathy. The final diagnosis is achieved by comparing the final output parameter combinations with various permutations pre-fed into the algorithm. In addition, the algorithm keeps enhancing its diagnostic accuracy through semi-supervised learning. In view of this, it is seen to have a good specificity though the sensitivity is slightly low. However, since the software is meant to aid a clinician in glaucoma diagnosis, this works to its favor. When clubbed with the DDLS score, RIA-G provides an excellent accuracy. There are, however, certain limitations which need to be addressed. These include the detection error rate of about 5.5% which though relatively low is still significant (causes included tilted disc, severe peripapillary atrophy, vasculature anomalies, etc.). This can be overcome by the manual cup and disc marking option. Another limitation is the absence of absolute quantitative values including disc area and rim area, which though reported in ratio are not reported in cubic millimeter. This is insignificant to routine cases but helpful when evaluating very small or large discs. A limitation specific to this study is the relatively few number of cases with advanced glaucomatous cupping, where it is likely that the software may show errors. Next, the experts evaluated monoscopic fundus photographs and may not have accurately diagnosed glaucomatous neuropathy since that is best diagnosed on stereoscopic examination. However, as the software only evaluates two-dimensional fundus photographs, the experts too were made to evaluate the same in the study protocol. A limitation of the software wherein disc hemorrhages and retinal nerve fiber changes are not taken into account for diagnosing glaucomatous optic neuropathy would mean that the experts may have been able to estimate the presence of glaucomatous optic neuropathy slightly better.

Conclusion

RIA-G is the only commercially available, clinically usable software for automated glaucoma detection. It is likely to play a significant role in glaucoma screening and aiding diagnosis, particularly where a glaucoma expert is not available and limited doctor–patient contact exists.

Financial support and sponsorship

Nil.

Conflicts of interest

Kalpah Innovations had provided free access to RIA-G during the period of the study. The first author has been a clinical advisor to Kalpah Innovations.