Use of offline artificial intelligence in a smartphone-based fundus camera for community screening of diabetic retinopathy.

PURPOSE: The aim of the study was to analyse the reliability of an offline artificial intelligence (AI) algorithm for community screening of diabetic retinopathy.
METHODS: A total of 1378 patients with diabetes visiting public dispensaries under the administration of the Municipal Corporation of Greater Mumbai between August 2018 and September 2019 were enrolled for the study. Fundus images were captured by non-specialist operators using a smartphone-based camera covering the posterior pole, including the disc and macula, and the nasal and temporal fields. The offline AI algorithm on the smartphone marked the images as referable diabetic retinopathy (RDR) or non-RDR, which were then compared against the grading by two vitreoretinal surgeons to derive upon the sensitivity and specificity of the algorithm.
RESULTS: Out of 1378 patients, gradable fundus images were obtained and analysed for 1294 patients. The sensitivity and specificity of diagnosing RDR were 100% (95% CI: 94.72-100.00%) and 89.55% (95% CI: 87.76-91.16%), respectively; the same values for any diabetic retinopathy (DR) were 89.13% (95% CI: 82.71-93.79%) and 94.43% (95% CI: 91.89-94.74%), respectively, with no false-negative results.
CONCLUSION: The robustness of the offline AI algorithm was established in this study making it a reliable tool for community-based DR screening.

Diabetic retinopathy (DR) is one of the primary causes of vision loss among the middle aged and the elderly. The prevalence of DR is reported to be 18% in India and 9.3% globally.[1234] Globally, the numbers are projected to rise to 10.2% (578 million) by 2030 and 10.9% (700 million) by 2045.[3] The prevalence with age increases from 37.41% to 68.52% and 78.34% among people aged between 40 and 50 years, 51 and 60 years and 61 and 70 years, respectively.[5] In recent times, there has been a rapid increase in the prevalence of diabetes in the middle- and low-income countries (LMIC), from 4.7% in 1980 to 8.5% in 2014,[6] with latest data showing that LMIC now bear the highest burden of diabetes mellitus.[7] India's population is one-sixth of that of the world and it is home to around 73 million patients with diabetes.[89] Various population-based surveys have shown that almost 10–30% of the patients with diabetes are prone to developing DR.[1011]

In most cases of DR, the condition remains asymptomatic till the advanced stages of the disease and hence it is only detected when the damage is already done.[12] Routine screening holds the key to early detection and timely treatment to prevent cases of vision loss from DR. Advances in medical devices and awareness regarding screening for DR over the past few decades have led to development of standardised screening protocols in developed nations like the United States, the United Kingdom, Australia and other countries in the EU,[412] but most of them follow protocols that are more sophisticated, require specialised training for technicians and conducted in clinic-based set-up using expensive, desktop-based cameras.[12] Such models are not feasible in developing countries like India where there is a large population to screen and the resources are limited.

Smartphone-based retinal imaging is a convenient and cost-effective alternative to the conventional retinal imaging methods and has been scientifically validated for screening for DR.[121314] Similarly, automated analysis of retinal images captured using low-cost, smartphone-based devices has also been validated for community screening purposes.[15] Use of offline artificial intelligence (AI) algorithm to provide real-time analysis of the images captured on a smartphone was validated in a pilot study of 231 patients.[12] Our current study aims to establish the reliability of an offline AI in a smartphone-based fundus camera for community screening of DR by minimally trained non-specialist health worker.

Research done so far

Prior to this study, we conducted a comprehensive literature review across research databases like PubMed, Google Scholar, Web of Science and Cochrane, with the keywords ‘Artificial Intelligence’, ‘deep learning’, ‘machine learning’, ‘population screening’, ‘community screening’, ‘prevalence and severity of diabetic retinopathy’ to obtain a range of studies conducted previously that dealt with application of offline AI in population or community-based screening of DR. Our previous pilot study first reported the use of an offline AI algorithm for screening of DR. However, it had a small sample size of 213 patients. The current study shows the reliability of the offline AI in a large sample size of 1378 patients.

Implications of all available evidence

Our findings from the pilot study had shown that the offline AI algorithm could be highly effective in screening programs for DR as it recorded a high sensitivity (100%) and specificity (88.5%). However, its applicability in a community-based screening programme could only be assessed if these results could be reproduced on a larger scale. We calculated a suitable sample size and conducted this study. The results showed that this algorithm could be successfully used to screen for DR even on a large scale.

Methods

An institutional review board approval was obtained and informed consent was taken from all participants. The study was conducted adhering to the tenets of the Declaration of Helsinki. Both the offline automated analysis and the smartphone-based, non-mydriatic retinal imaging system are based on proprietary technologies, with none of the authors having any financial interest herein.

It was a cross-sectional study conducted from August 2018 to September 2019 on patients visiting the various dispensaries administered by the Municipal Corporation of Greater, Mumbai, on designated days. Forty-seven municipal dispensaries were covered in the study. Written informed consent was taken from all the patients. Preliminary data, such as age, sex, duration since diabetes onset, and postprandial blood glucose level, were collected.

Capture and grading of fundus images

Patient's eyes were dilated using single drop of 1% tropicamide eye drops, which has previously been found to cause minimal risk of angle-closure glaucoma.[1216] Fundus imaging was then done by healthcare workers with minimal experience in fundus imaging. The health workers were trained for a period of 2 weeks by optometrists to take both dilated and undilated fundus images. Training was also done on how to run the AI and document the results. Images were captured using the Remidio Non-Mydriatic Fundus on Phone (Remidio Innovative Solutions Pvt. Ltd.). An anterior segment photograph was first captured, followed by three fields of the fundus (namely, the posterior pole, including the disc and macula, and the nasal and temporal fields), in accordance with the Early Treatment Diabetic Retinopathy Study protocol. The offline AI algorithm on the smartphone marked images of poor quality, hence prompting the operator to take additional pictures of the same retinal view until the images were deemed acceptable by the AI system or the operator decided otherwise.

The images so captured were subjected to automated analysis by the Medios AI (Remidio), a proprietary offline automated analysis of retinal images on a smartphone to detect RDR taken as grades of moderate or higher severity of Non Proliferative Diabetic Retinopathy (NPDR) and Proliferative Diabetic Retinopathy (PDR).

Grading by human graders

The images were stored on a Health Insurance Portability and Accountability Act compliant cloud server (Amazon Web Services) and graded by two vitreoretinal surgeons who were masked to the AI-grading results. In case of a discrepancy between the grading of the two surgeons, an adjudication was done by a third vitreoretinal surgeon. The grading of retinopathy was done according to the International Clinical DR severity scale.[12] The final diagnosis for each patient was determined by the stage of DR of the more affected eye per International Clinical DR severity scale.

Grading by the AI

The AI algorithm was run offline on the smartphone by the operator immediately after image acquisition. The offline automated analysis graded these images as RDR and non-RDR. Class activation mapping shows the lesions which are detected by the AI. Severity of retinopathy, that is, demarcating between mild, moderate and severe non-proliferative and proliferative DR, was not indicated by AI.

The offline AI application is integrated into the smartphone-based, nonmydriatic retinal imaging system. It is a component of the camera control application and thus seamlessly integrates into the image acquisition workflow. It is designed to binary-type-only RDR and non-RDR and does not grade the severity of retinopathy. It is a robust algorithm which has been trained using both high-quality images taken from datasets like EyePACS as well as using a mix of mydriatic (Kowa VX-10a mydriatic camera) and non-mydriatic (Remidio FOP NM-10) images captured at hospitals and screening camps having an equal proportion of healthy as well as diseased eye images. The algorithm's functional efficacy has been previously validated using three different datasets, generated using the mydriatic version of the same product (a separate device) as well as the mydriatic mode of the same product (an additional feature on the same device). The functioning mechanism of the algorithm is similar to that discussed in our previous study.[12]

Statistical analysis

Sensitivity and specificity for both any DR and RDR were calculated comparing the grading given by the AI algorithm to that of the graders (referred to as ground truth) for both patient-wise and eye-wise analysis. The minimum number of patients needed to be screened was determined using a standard protocol,[17] and then calculated, assuming a margin of error of 2% for this study on either side of the mean (given a previously published mandate from the US Food and Drug Administration of an end point of at least 86% diagnostic sensitivity of RDR).[12] At the 95% CI, and a population of 18.4 million in Mumbai (Census of 2011), with a maximum DR prevalence of 15.37%,[18] this resulted in a minimum sample size of 1250 patients.

Results

A total of 1378 patients were enrolled in the study, of which 732 (53.12%) were male and 646 (46.87%) were female with an average (SD) age of 54.9 (10.43) years. Their mean (SD) duration of diabetes was recorded as 5.89 (5·73) years with an average (SD) post-prandial blood sugar level of 208.46 (80.20) mg/dL. Out of these 1378 enrolled patients, images captured of one or both eyes of 84 patients were deemed ungradable by the human grader and hence were excluded from the final analysis.

There was excellent intergrader agreement between the vitreoretinal surgeons (K = 89). As per the grading by the vitreoretinal surgeons [Table 1], 1151 (88.9%) patients had no evidence of DR, 70 (5.4%) had mild NPDR, 48 (3.7%) had moderate NPDR, 14 (1.08%) had severe NPDR, and 6 (0.46%) had PDR, making a total of 68 cases of RDR. Eighty-one patients (6.2%) labelled by the ophthalmologists as non-RDR were incorrectly diagnosed by the AI as RDR. All patients diagnosed as RDR by specialist grading were correctly diagnosed by the AI. Among 70 (5%) individuals who were diagnosed by the ophthalmologists as mild NPDR, 55 patients (78%) were diagnosed as having RDR by the AI, while 15 (21.4%) were diagnosed as no RDR. This gave a sensitivity and specificity of diagnosing RDR as 100% (95% CI: 94.72–100.00%) and 89.55% (95% CI: 87.76–91.16%), respectively; the same values for any DR were 89.13% (95% CI: 82.71–93.79%) and 94.43% (95% CI: 91.89–94.74%), respectively [Table 2].

Table 1

Comparison of Agreement Between Medios AI and Ground Truth

Medios AI	No DR	RDR
Ground Truth (Per Patient)
No DR	1151	81
Mild NPDR	15	55
Moderate NPDR	0	48
Severe NPDR	0	14
PDR	0	6
Per Eye
No DR	2290	125
Mild NPDR	24	80
Moderate NPDR	0	77
Severe NPDR	0	24
PDR	0	6

Table 2

Performance Metrics of Medios AI

Pathology	Sensitivity	Specificity
Per-patient
RDR	100% (95% CI 94.72% - 100.00%)	89·55% (95%CI 87.76% - 91.16%)
Any DR	89.13% (95% CI 82.71% to 93.79%)	94.43% (95% CI 91.89% to 94.74%)
Per Eye
RDR	100.0% (95% CI 96.61% - 100.00%)	91.86% (95% CI 90.72% - 92.90%)
Any DR	88.63% (95% CI 83.55% - 92.57%)	94.82% (95% CI 93.86% - 95.67%)

An eye-wise analysis was performed for the entire set, a total of 2756 eyes, to further evaluate the diagnostic accuracy of the AI. The sensitivity for detection of RDR continued to remain 100% (95% CI: 96.61–100%) while the specificity became 91.86% (95% CI: 90.72–92.90%). The sensitivity and specificity for any DR were 88.63% (95% CI: 83.55–92.57%) and 94.82% (95% CI: 93.86–95.67%) in this analysis [Table 2].

The positive predictive value (PPV) for this analysis was 33.33% and the negative predictive value (NPV) was 100% with 136 (10.5%) false-positive cases and no false-negative cases.

Discussion

The current study takes off from the previous pilot study that evaluated the diagnostic efficacy of an offline AI algorithm to diagnose RDR.[12] The current study included 1378 patients who had enrolled at various public dispensaries under the administration of the Municipal Corporation of Greater, Mumbai. The main purpose of this study was to establish the reliability of an offline AI in a smartphone-based fundus camera for community screening of DR by healthcare workers with minimal experience in fundus imaging.

India has a population of over 1.3 billion, of which around 73 million are diabetics.[11219] The prevalence of DR in India is around 18%[1] and there is only one ophthalmologist for every 100,000 people.[12] This smartphone-based offline AI model can be used to bridge the gap in the doctor to population ratio, more so in the rural and slum areas. The AI also provides heat maps of the lesions which can be used to educate the patient. Several studies have used the AI-based screening model for DR, the previous versions having used either internally generated or standard datasets to show high sensitivities and specificities of autonomous AI algorithms. For Abràmoff et al.,[2] there were 900 patients with no prior history of DR, where the autonomous AI detected RDR for patients with a sensitivity of 87.2% (95% CI: 81.8–91.2%) and a specificity of 90.7% (95% CI: 88.3–92.7%). Similarly Gulshan et al.[1] conducted a retrospective study with 128,175 retinal images from EYEPACS-1 and MESSIDOR-2 datasets where the sensitivity was 90.3% (95% CI: 87.5–92.7%) and specificity was 98.1% (95% CI: 97.8–98.5%) for EYEPACS-1, and sensitivity was 87.0% (95% CI: 81.1–91.0%) and the specificity was 98.5% (95% CI: 97.7–99.1%) for Messidor-2 dataset, respectively. Ting et al.[20] conducted a multi-ethnic study with 71,896 images from 14,880 patients; the Deep Learning System, used herein, had a sensitivity of 90.5% and specificity of 91.6% for detecting RDR. Hence, it has already been validated that deploying an AI-based model has an increasingly high accuracy of detecting RDR. However, all these studies have been conducted under in-clinic settings with images captured by trained professionals using conventional, desktop-based imaging systems. This study was conducted on images captured on a smartphone by healthcare workers with minimal experience in fundus imaging. There were several images which were deemed below quality standards as per the AI. However, these images were still used in the analysis to assess the reliability of the offline AI algorithm in such situations which is expected to occur during community screening.

The robustness of an automated AI algorithm in detecting cases of DR can be evaluated best in diverse clinical or population-based settings, wherein there are people of different origins, with variations in retinal pigmentation, differences in focusing, pupillary dilation, media opacities and light contrast, among other factors, leading to a heterogeneous mixture of traits.[16] Mumbai has a population of around 2 crores with people of various ethnicities living here.[21] Hence, the current scenario was an ideal situation to assess the robustness of the AI.

The sensitivity and specificity of detecting RDR were 100% and 89.55%. This was above the superiority end point of sensitivity of 85% and specificity of 82.5% deemed by the FDA.[12] The number of false-positive cases was 136 (10.5%), of which 55 cases were that of mild NPDR, 22 cases of non-DR pathologies like glaucoma, retinitis pigmentosa, age-related macular degeneration, gliosis, macular scars and asteroid hyalosis while the remaining were cases of normal eyes being diagnosed as cases of RDR [Table 3] [Fig. 1]. This can be attributed to a combination of extremely high sensitivity and minimally experienced operators capturing images which were below quality standards. This is also highlighted in the PPV for this study being 33.33%. False-positive readings might lead to an increase in the number of referrals and might also cause undue stress and anxiety among patients. However, the false positives also help in picking pathologies other than DR that would have required a referral. There were no false negatives in the study with the NPV being 100%. False-negative cases lead to increased risk of disease progression and vision loss. No false negative would mean absolutely no chances of missing a referable condition making this algorithm good for community screening.

Table 3

Non-DR pathologies detected by the AI

Pathology	Count
BRVO	4
Asteroid hyalosis	4
Macular scar	1
Glaucoma	2
Gliosis	1
AMD/Drusens/Cotton wool spots	8
Macular hole	1
Retinitis pigmentosa	1

Figure 1

Activation maps of crucial non-DR pathologies detected as DR by the offline AI algorithm. (a and b) BRVO; (c and d) asteroid hyalosis; (e and f) age-related macular degeneration and (g, h) multiple hard drusens

Conclusion

This study was a community-based study for screening of RDR. The images were captured on a smartphone-based fundus camera by healthcare workers with minimal experience in fundus imaging. Thus, not all images were of excellent quality. This would be expected while conducting large-scale community screening. Despite including poor quality images, the sensitivity and specificity of detecting RDR by the AI were 100% and 89.55%. Being an offline AI algorithm, the real-time analysis of images brings another novel edge to this study, emphasising its applicability in remote areas where despite limited access to internet, results can be given immediately to the patient. The lesion detection map on the images would also help the health worker in patient education.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.