Agreement on Grading Retinal Findings of Patients with Diabetes Using Fundus Photographs by Allied Medical Personnel when Compared to an Ophthalmologist at a Diabetic Retinopathy Screening Program in Nepal.

INTRODUCTION: Diabetic retinopathy (DR) is the emerging cause of blindness in developing countries. This study aimed to assess the accuracy of grading retinal findings of patients with diabetes using fundus photographs by allied medical personnel (AMP) when compared to an ophthalmologist.
MATERIALS AND METHODS: Six AMPs were enrolled for grading fundus photographs of patients with diabetes after three and six months of training twice at two weeks interval. The total number of fundus photographs graded were 1,344. Grading by a retina specialist was used as the gold standard. Intra-rater and inter-rater agreement was assessed using the kappa coefficient (k).
RESULTS: The intra-rater agreement of half of the AMPs in both three months and six months was almost perfect for grading of any retinal abnormalities, and substantial for retinal hemorrhages. In three months, the overall inter-rater agreement at the second rating was moderate for any retinal abnormalities (k=0.60) and retinal hemorrhage (k=0.48) and was fair for macular exudates (k=0.35). The overall inter-rater agreement of AMPs in six months at the second rating was substantial for any retinal abnormalities (k=0.61), fair for retinal hemorrhage (k=0.30), and moderate for macular exudates (k=0.49). The overall inter-rater agreement at first rating was almost similar to the second rating in both three months and six months.
CONCLUSION: Intra-rater agreement of AMPs was almost perfect for any retinal abnormalities and substantial for retinal hemorrhage. The inter-rater agreement of AMP was substantial for any retinal abnormalities and moderate for retinal hemorrhages and macular exudates. The agreement results were almost similar at three months and six months. AMPs could be utilized in screening of DR and other retinal pathologies for timely referral to reduce the blindness in low-resource settings.

Introduction

Diabetes mellitus (DM) is one of the four major non-communicable diseases leading to premature death globally by affecting vital organs of the body.1–4 Ocular problems are more common among patients living with diabetes when compared to patients without diabetes. DM affects both the anterior and posterior segment of the eye.5–7 Diabetic retinopathy (DR) is the most common micro-vascular complication of diabetes. DR can lead to irreversible visual impairment and blindness if treatment is delayed. DR is the fifth leading cause of blindness worldwide, despite being a preventable condition.8 DR is the emerging cause of blindness in developing countries due to an increase in diabetes in recent years.1–4 DR progresses from various stages of non-proliferative DR (NPDR) to proliferative DR (PDR), therefore the stages of DR is characterized by various retinal findings such as microaneurysms, retinal hemorrhages, exudates, neovascularization, vitreous hemorrhages, tractional retinal detachment, macular edema, etc.8 Besides DR, other retinal problems among patients with DM and those associated with a concurrent diagnosis of hypertension include branch retinal vein occlusion (BRVO) and central retinal vein occlusion (CRVO). Both BRVO and CRVO are characterized by retinal hemorrhages, exudates and macular edema.9–12 Several patients with DM have concurrent hypertension and hyperlipidemia. Hypertensive retinopathy is a common retinal condition among those diagnosed to have uncontrolled hypertension for prolonged periods.13

DM is a public health problem and assumes epidemic proportions in Nepal.14 With the rise of diabetes, DR leading to blindness is one of the emerging problems like in other developing countries.15–18 However, routine screening for DR and other retinal condition is limited. There is low awareness of DR in the community.16,19,20 DR screening using fundus photography is effective and growing in popularity in recent years.21–24

In Nepal, the major part of eye health care service is provided by non-governmental organizations. Teaching institutions and limited public hospitals have separate eye departments. Integration of diabetic eye care services for comprehensive diabetes care is limited. There is also limited DR screening at primary eye care centers by mid-level ophthalmic personnel (MLOP). DR screening is conducted mainly by ophthalmologists at tertiary eye hospitals. Tele-ophthalmology service is still under development.

Allied medical personnel (AMP) are the mid-level health personnel, supporting patient management under the physicians. Previous studies conducted among MLOP revealed good agreement on grading DR and other retinal disease using fundus photographs when compared to the ophthalmologist. AMPs, working in public hospitals, are more numerous than the primary care physicians and endocrinologists in Nepal. They could help with fundus photography, grading of fundus photographs and counselling under the physicians for referral of vision-threatening DR and other retinal conditions. There is a gap in the specialized workforce and this can be fulfilled by the training of more readily available AMPs. Less resources are also required for the training of AMPs. The cost benefit ratio and efficiency of use of AMPs is improved. This will help to reduce the resulting irreversible visual impairment and blindness with timely treatment.

This study aims to assess the accuracy of detection of retinal findings on fundus photography by AMP when compared to the ophthalmologist at a DR screening program in Nepal. The study findings can provide guidance in integrating DR screening in comprehensive diabetes management. This could also help in opportunistic screening of other blinding retinal diseases among patients living with diabetes at a public health system in Nepal and other similar resource-limited countries.

Materials and Methods

This is an observational agreement study on grading retinal findings among patients with diabetes by AMP when compared to the ophthalmologist. Taking the proportion of positive rating as 0.9 and Kappa value to detect the difference =0.7 with 80% power of the test, the required number of eyes screened with fundus photographs of patient with diabetes was 1,340.25,26

Training of AMPs: AMPs completed the government certified certificate level in either nursing or general medicine. Six AMPs working in public hospitals were given six months of training. Out of six AMPs, three completed a certificate in nursing and three completed a certificate in general medicine.

The training included three weeks of theory comprising of anatomy and physiology of the eye as well as education on the findings of major ocular and retinal problems. The training was conducted over a period of seven hours each day for six months. Faculties mostly involved in training included fellowship trained retina specialist, trained photographers, endocrinologist, physicians, dietician, and public health experts. The major section of training incorporated was on DM, DR classification, and management. The rest of the period was provided for practical sessions on history taking, visual acuity assessment, detailed anterior segment evaluation of eye using torch light. Training was provided for taking fundus photographs of patients visited for diabetes management at the physician’s clinic and grading the retinal findings.

Sampling technique: 1,344 consecutive non-mydriatic fundus photographs of persons with diabetes screened for DR at the DR screening program were included in the study. DR screening was conducted at collaborating public hospitals (Bir Hospital, Gangalal National Heart Center, and Patan Hospital) as a part of DR project under Tilganga Institute of Ophthalmology (TIO) in 2016. DR Screening was carried out using Nidek-10 non-mydriatic fundus camera (Versacam & Trade & Alpha, France), Zeiss portable fundus camera (Zeiss Visucout 100). Fundus photographs were captured in screen monitor of computer. All fundus photographs were graded using the same 48 inch computer monitor by all the graders.

The fundus photographs were disc and macular centered non-mydriatic fundus photographs of both eyes of patients living with diabetes. Fundus photographs were graded by AMPs at three and six months of training twice at an interval of two weeks. The ophthalmologist, fellowship trained retina specialist, graded the fundus photographs in one time. The grading of the fundus photographs by the ophthalmologist was considered the gold standard for comparison with the AMP grading.

A proforma was developed to recording details of retinal findings in the fundus photographs of patient with diabetes. All graders (AMPs on training and a fellowship trained retina specialist) were asked to assess fundus photographs for any retinal abnormalities. If abnormal, graders were asked to note for the presence of microaneurysms, retinal hemorrhages, macular exudates, exudates elsewhere, other retinal abnormalities, new vessels at the disc, new vessels elsewhere, and any optic disc abnormality. They were also asked to state if the fundus pictures were non-gradable. Pre-testing was done with over 50 fundus photographs in three months. Both the intra-rater (the agreement of the same grader when grading twice) and inter-rater (the agreement between AMP and ophthalmologist) agreement were assessed on retinal findings.

An abnormal optic disc was defined as the presence of new vessels, large cup disc ratio, disc edema or pale optic disc.

Agreement was assessed using kappa coefficient (k).25,26 The agreement was less than chance if kappa was <0, slight agreement if kappa was 0.01–0.20, fair agreement if kappa was 0.21–0.40, moderate agreement if kappa was 0.41–0.60, substantial agreement if kappa was 0.61–0.80 and almost perfect agreement if kappa was 0.81–0.99.

The study was approved by the Tilganga Institute of Ophthalmology, Kathmandu, Nepal Institutional Review Committee (TIO-IRC) on 2016 (TIO-IRC approval number; 04/2016). This study was conducted in accordance with the Declaration of Helsinki.

Statistical Analysis

Data analysis was done in SPSS version 20. Intra-rater and inter-rater agreement were assessed using kappa coefficient (k). Sensitivity, and specificity were assessed for the inter-rater agreement.

Results

Thirteen hundred forty-four fundus pictures of each eye of 672 patients living with diabetes were graded by six AMP and an ophthalmologist. The mean age of patients was 58.4 years. Male and females comprised of 49% and 51% respectively. Retinal findings were present in 25.8% of patients.

Intra-Rater Agreement on Retinal Findings in Three Months and Six Months

In three months, among the six AMPs, half had almost perfect intra-rater agreement for grading any retinal abnormalities (k=0.84), substantial for grading retinal haemorrhage (k=0.76), and the majority had substantial agreements (k=0.76) for grading macular exudates.

In six months grading, the intra-rater agreement for any retinal abnormalities (k=0.86) was almost perfect for two AMPs, half had substantial intra-rater agreements for retinal hemorrhage (k=0.77) and macular exudates (0.86). Table 1.

Table 1

Intra-Rater Agreement of AMP on Retinal Findings in 3 Month and 6 Months

Retinal Findings	Time(Months)	AMP 1K (95% CI)	AMP 2K (95% CI)	AMP 3K (95% CI)	AMP 4K (95% CI)	AMP 5K (95% CI)	AMP 6K (95% CI)
Any retinal pathologies	3	0.84 (0.74–0.93)	0.81 (0.68–0.89)	0.84 (0.71–0.92)	0.65 (0.53–0.75)	0.67 (o.54–0.78)	0.76 (0.63–0.89)
	6	0.86 (0.74–0.94)	0.80 (0.68–0.90)	0.85 (0.73–0.96)	0.79 (0.69–0.87)	0.59 (0.47–0.69)	0.71 (0.57–0.84)
Retinal hemorrhage	3	0.08 (0–0.46)	0.76 (0.43–0.90)	0.41 (0.13–0.71)	0.73 (0.46–0.86)	0.58 (0.36–0.86)	0.76 (0.56–1.0)
	6	0.75 (0.50–1.00)	1.00	0.24 (0.22–0.64)	0.77 (0.55–0.99)	0.36 (0–0.63)	0.70 (0.30–1.00)
Macular Exudates	3	0.70 (0.45–0.98)	0.75 (0.47–0.95)	0.76 (0.44–0.97)	0.62 (0.29–0.81)	0.54 (0.28–0.80)	0.70 (0.34–0.91)
	6	0.57 (0.12–3.88)	0.66 (0.13–4.07)	0.86 (0.14–4.07)	0.84 (0.09–5.19)	0.82 (0.09–5.07)	0.73 (0.14–3.54)
Abnormal optic disc	3	0.39 (0.00–0.75)	0.51 (0.19–0.79)	0.46 (0.18–0.71)	0.48 (0.15–0.70)	−.02 (−.03–0.00)	0.35 (−.02–0.66)
	6	0.66 (0.45–0.77)	0.84 (0.76–0.89)	0.74 (0.60–0.80)	0.56 (0.10–0.80)	0.43 (0.32–0.52)	0.29 (0.02–0.57)

Note: * Means 95% CI could not be calculated in SPSS version 20.

Abbreviations: AMP, allied medical personnel, CI, confidence interval, k, kappa.

Inter-Rater Agreement on Retinal Findings in Three Months and Six Months at First Rating

In three months, the inter-rater agreement at the first rating was substantial (k=0.68) for any retinal abnormalities, moderate (k=0.52) for retinal haemorrhage, fair (k=0.40) for macular exudates. The sensitivity for any retinal abnormalities was 90.48% and specificity was 83.67%. The sensitivity for retinal haemorrhage was 81.08% and specificity was 83.24%.

In six months, the inter-rater agreement of AMPs was almost perfect (k=0.68) for any retinal abnormalities. The inter-rater agreement was fair (k=0.37) for retinal haemorrhage, moderate (k=0.48) for macular exudates. The sensitivity for any retinal abnormalities was 65.5% and specificity was 97.02%. The sensitivity for retinal haemorrhage was 66.67% and specificity was 82.97%. Table 2.

Table 2

Inter-Rater Agreement of AMP on Retinal Findings in 3 Month and 6 Months at the First Rating

Retinal Findings	Time(Months)	Kappa(95% CI)	Sensitivity (%)	Specificity (%)
Any retinal pathologies	3	0.68 (0.63–0.73)	90.48	83.67
	6	0.68 (0.63–0.73)	65.52	97.02
Retinal hemorrhage	3	0.52 (0.36–0.64)	81.08	83.24
	6	0.68 (0.63–0.73)	66.67	82.97
Macular Exudates	3	0.40 (0.29–0.51)	83.72	58.87
	6	0.48 (0.35–0.58)	82.50	68.22
Other retinal pathologies	3	0.40 (0.26–0.55)	94.04	40.68
	6	0.38 (0.22–0.54)	93.17	39.58
Abnormal optic disc	3	0.20 (0.15–0.26)	85.77	98.45
	6	0.32 (0–0.80)	99.03	33.33

Abbreviations: AMP, allied medical personnel, CI, confidence interval.

Inter-Rater Agreement on Retinal Findings in Three Months and Six Months at the Second Rating

In three months, the inter-rater agreement at the second rating was moderate (k=0.60) for any retinal abnormalities, moderate (k-0.48) for retinal hemorrhage, and fair (k=0.35) for macular exudates. The sensitivity for any retinal abnormalities was 87.45%, specificity was 76.45%. The sensitivity for retinal hemorrhage was 81.08% and specificity was 82.97%.

In six months, the inter-rater agreement of AMP was substantial (k=0.61), for the detection of any retinal abnormalities, fair (k=0.30) for retinal haemorrhages, and moderate (k=0.49) for macular exudates. The sensitivity for any retinal abnormalities was 59.54% and specificity was 95.79%. The sensitivity for retinal haemorrhage was 62.5% and specificity was 80.54%. (Table 3).

Table 3

Inter-Rater Agreement of AMP on Retinal Findings in 3 Months and 6 Months at the Second Rating

Retinal Findings	Time (Months)	Kappa (95% CI)	Sensitivity (%)	Specificity (%)
Any retinal pathologies	3	0.60 (0.55–0.64)	87.45	76.45
	6	0.61 (0.56–0.66)	59.54	95.79
Retinal hemorrhage	3	0.48 (0.36–0.60)	84.09	77.78
	6	0.30 (0.13–0.43)	62.5	80.54
Macular Exudates	3	0.35 (0.25–0.46)	83.72	58.87
	6	0.49 (0.36–0.61)	90.28	62.96
Other retinal pathologies	3	0.42 (0.28–0.55)	92.90	44.93
	6	0.33 (0.18–0.47)	91.67	37.25
Abnormal optic disc	3	0.04 (0.03–0.06)	67.84	98.74
	6	0.32 (0-0.79)	99.03	33.33

Abbreviations: AMP, allied medical personnel, CI, confidence interval.

Discussion

In our study, the intra-rater agreement of half of the AMPs in both three months and six months was almost perfect for grading of any retinal abnormalities, and substantial for retinal haemorrhage. Half of the AMP had moderate agreement for abnormal optic disc in three months, but substantial in six months. These findings show that the intra-rater agreement in both three months and six months were almost similar with a slight improvement in six months grading for abnormal optic disc. The intra-rater agreement indicates the consistency in results. In our study, there is variability in intra-rater agreement of AMP in both three months and six months. These findings showed that emphasis should be given in monitoring and refresher training of each AMP for grading the retinal findings. A study conducted by Thapa et al to detect retinal diseases using fundus photographs by involving two MLOPs at the primary eye care center in Nepal also reported the variability in intra-rater agreement, despite receiving the same training.27 The AMPs work at public hospitals and responsible for counselling and management of diabetes under the physicians. MLOPs work in the primary eye care center and responsible for management of simple ocular problems, and timely referral of complex ocular disorders. The intra-rater agreement on any retinal abnormalities and retinal haemorrhages by two MLOPs ranged from moderate to substantial similar to our study.27

In our study, at the first rating in three months and six months, the inter-rater agreement of AMP and ophthalmologist was substantial for any retinal abnormalities, moderate for macular exudates, and fair for abnormal optic disc. The agreement results are similar at first rating in both three months and six months. In the second rating, the inter-rater agreement was moderate for grading any retinal abnormalities in three months, while substantial in six months. For macular exudates, the inter-rater agreement was fair in three months while moderate for six months. For abnormal optic disc, there was slight agreement in three months and was fair at six month. Overall, there was slightly better results at second rating when compared to first rating and also in six months when compared to three months. This could be due to more exposure on grading of same fundus photographs. In another study conducted in Nepal, the inter-rater agreement of the first MLOP and second MLOP when compared to the retina specialist was slightly improved at second rating when compared to the first rating like in our study.27

The inter-rater agreement on detection of any DR by MLOP when compared to retina specialists was substantial in a previous study conducted at a primary eye care center in Nepal.28 The agreement results were comparable to our study. Study reported from Singapore showed the inter-rater agreement for DR grading between non-physician graders (NPGs) and the retinal specialist was substantial, while the agreement between family physicians (FPs) and the retinal specialist was only fair.29 In our study, AMP are the non-physician graders, and our results are similar to this study.29 In a study conducted in Sri Lanka, the sensitivity and specificity for grading referable DR (moderate NPDR and above category) for grader 1 (family physician) was 88.7% and 94.9%. For grader 2 (family physician) sensitivity was 92.5% and specificity was 96.4%.30 This study findings showed slightly better results when compared to our study. This difference could be because of inclusion of only referable DR cases and mydriatic fundus photographs in their study. Gill et al reported the overall moderate agreement for DR grading using pan optic ophthalmoscope by family physicians.31 Better results in our study when compared to this study could be because of grading in fundus photographs.

Griffith et al reported grading on referable DR, 100% sensitivity using ophthalmoscopy by primary care physicians when compared to 94% sensitivity for general ophthalmologist and 100% for retina specialist using fundus photographs.32 This study showed better results when compared to our study. The difference could be due to enrolment of referable cases only when compared to our series that included all cases.

Our study revealed that training of AMP could support the detection of vision-threatening retinal problems among patients living with diabetes for timely referral and treatment to avoid irreversible visual impairments. Screening by AMP could also be a cost effective method for DR screening and opportunistic screening of other vision-threatening retinal conditions among patients with diabetes under the physician in public hospitals, especially from remote parts of the country.

The major strength of the study is the large number of fundus photographs used for grading of retinal findings. Both the intra-rater and inter-rater agreement of AMPs were assessed along with sensitivity and specificity of the retinal findings. The other strength of the study is the involvement of more AMPs as in a real practice scenario.

The task shifting approach by engaging AMP and MLOP will make the service more efficient and in-cooperation of telemedicine may improve eye care and specifically retina care access to the underserved communities.

Conclusion

The intra-rater agreement for the majority of AMP was substantial for any retinal abnormalities, retinal hemorrhage and macular exudates. The inter-rater agreement between the AMP and ophthalmologists was substantial for any retinal abnormalities, moderate for retinal hemorrhages and macular exudates, and fair for optic disc abnormality. Both intra-rater and inter-rater agreement results were almost similar at three and six months. These findings showed that the training of grading retinal findings to AMP can be shortened to three months to reduce the resources for training. AMP can serve as first level graders in screening for DR and other retinal pathologies. This can help in referral and timely treatment to reduce the resulting blindness from these vision threatening retinal disorders among patients living with diabetes especially residing in remote parts of the country.