Evaluation of a New Model of Care for People with Complications of Diabetic Retinopathy: The EMERALD Study.

PURPOSE: The increasing diabetes prevalence and advent of new treatments for its major visual-threatening complications (diabetic macular edema [DME] and proliferative diabetic retinopathy [PDR]), which require frequent life-long follow-up, have increased hospital demands markedly. Subsequent delays in patient's evaluation and treatment are causing sight loss. Strategies to increase capacity are needed urgently. The retinopathy (EMERALD) study tested diagnostic accuracy, acceptability, and costs of a new health care pathway for people with previously treated DME or PDR.
DESIGN: Prospective, multicenter, case-referent, cross-sectional, diagnostic accuracy study undertaken in 13 hospitals in the United Kingdom. PARTICIPANTS: Adults with type 1 or 2 diabetes previously successfully treated DME or PDR who, at the time of enrollment, had active or inactive disease.
METHODS: A new health care pathway entailing multimodal imaging (spectral-domain OCT for DME, and 7-field Early Treatment Diabetic Retinopathy Study [ETDRS] and ultra-widefield [UWF] fundus images for PDR) interpreted by trained nonmedical staff (ophthalmic graders) to detect reactivation of disease was compared with the current standard care (face-to-face examination by ophthalmologists). MAIN OUTCOME MEASURES: Primary outcome: sensitivity of the new pathway. SECONDARY OUTCOMES: specificity; agreement between pathways; costs; acceptability; proportions requiring subsequent ophthalmologist assessment, unable to undergo imaging, and with inadequate images or indeterminate findings.
RESULTS: The new pathway showed sensitivity of 97% (95% confidence interval [CI], 92%-99%) and specificity of 31% (95% CI, 23%-40%) to detect DME. For PDR, sensitivity and specificity using 7-field ETDRS images (85% [95% CI, 77%-91%] and 48% [95% CI, 41%-56%], respectively) or UWF images (83% [95% CI, 75%-89%] and 54% [95% CI, 46%-61%], respectively) were comparable. For detection of high-risk PDR, sensitivity and specificity were higher when using UWF images (87% [95% CI, 78%-93%] and 49% [95% CI, 42%-56%], respectively, for UWF versus 80% [95% CI, 69-88%] and 40% [95% CI, 34%-47%], respectively, for 7-field ETDRS images). Participants preferred ophthalmologists' assessments; in their absence, they preferred immediate feedback by graders, maintaining periodic ophthalmologist evaluations. When compared with the current standard of care, the new pathway could save £1390 per 100 DME visits and between £461 and £1189 per 100 PDR visits.
CONCLUSIONS: The new pathway has acceptable sensitivity and would release resources. Users' suggestions should guide implementation.

Diabetic macular edema (DME) and proliferative diabetic retinopathy (PDR) are the major sight-threatening complications of diabetic retinopathy that, in its turn, is the most common microvascular complication of diabetes. Diabetic macular edema and PDR are leading causes of sight impairment and blindness worldwide.2, 3, 4

Treatment for DME includes macular laser therapy, intravitreal anti–vascular endothelial growth factor (VEGF) therapies, and intravitreal steroids.5, 6, 7, 8, 9, 10, 11, 12, 13, 14 Macular laser treatment is delivered in a single session; retreatments may be required and, if so, usually are administered at 3- to 4-month intervals. Anti–vascular endothelial growth factor agents are administered monthly until the macula is dry; for the great majority of patients, this is not achieved during the first year of treatment. As soon as DME has resolved, patients are followed up every 3 to 4 months after macular laser therapy; after anti-VEGF therapy, patients initially are followed up monthly and every 1 to 4 months thereafter., Intravitreal steroids are administered at less frequent intervals than anti-VEGF agents, but patients receiving them still require close follow-up because they can lead to an increase in intraocular pressure. Independently of the treatment received, follow-up continues for the rest of the patient’s life because DME may recur and further treatment may be required to prevent sight loss.

Laser panretinal photocoagulation (PRP) remains the mainstay therapy for PDR. Laser PRP most often is completed in 2 sessions. Recent trials have shown anti-VEGF agents to be noninferior to PRP for the treatment of PDR., However, anti-VEGF agents do not seem to be cost effective when compared with laser therapy, except in patients with concomitant DME. As soon as regression of PDR is noted, patients are followed up every 6 to 12 months for life because PDR also may recur.

At follow-up appointments, ophthalmologists with expertise in retinal diseases examine the retina by slit-lamp biomicroscopy and determine whether recurrence of DME, PDR, or both is present. Spectral-domain (SD) OCT is used routinely to aid the diagnosis of DME. Although the prevalence of DME and PDR is not very high (approximately 7% of all people with diabetes),, given the very high prevalence of diabetes in the population,, with approximately 463 million adults worldwide living with diabetes, and the requirement for patients to be reviewed frequently and for life, as underlined above, diabetic eye disease is posing major problems of capacity in ophthalmic clinics in many countries, especially because of the shortage of ophthalmologists. As a result, patients’ appointments often are delayed, and treatments are not administered in a timely fashion. Delays in follow-up appointments in secondary care have been shown to lead to sight loss and even blindness in people with diabetic retinopathy. The challenge that diabetes poses to health care systems in developed and especially developing countries was highlighted recently. Retinal clinics are stretched further because anti-VEGF agents are used also to treat other diseases, including age-related macular degeneration and retinal vein occlusion. Cancellations of all routine appointments worldwide during the ongoing coronavirus disease 2019 pandemic have exacerbated this problem to unprecedented levels. Thus, it is imperative that new ways to increase efficiency and capacity of ophthalmic clinics are identified and, if safe and acceptable, are implemented.

The Effectiveness of Multimodal Imaging for the Evaluation of Retinal Oedema and New Vessels in Diabetic Retinopathy (EMERALD) study was conceived with the above purpose. It tested whether patients with DME, PDR, or both previously treated successfully (i.e., DME cleared and PDR became inactive) could be followed up through a new care pathway involving multimodal retinal imaging assessed by trained nonmedical staff (ophthalmic graders). Diagnostic accuracy, cost consequences, and acceptability of this new pathway to patients and health care professionals were evaluated against the current standard of care (face-to-face evaluation of patients by ophthalmologists).

Methods

Ethical approval was obtained from the Office for Research Ethics Committees Northern Ireland (ORECNI). The Belfast Health and Social Care Trust acted as sponsor and approved the study, and the study was conducted in accordance with the ethical principles of the Declaration of Helsinki. The EMERALD study was funded by the Health Technology Assessment of the National Institute for Health Research in the United Kingdom (identifier, 13/142/04).

Patient and Public Involvement

At study conception, a patient and public involvement (PPI) group was established, with the help of Diabetes UK, Northern Ireland. Meetings and discussions between EMERALD study researchers and the EMERALD PPI group took place early on, at the planning stages of the project, to confirm that the research question was important and that the tests proposed were adequate and feasible to patients. In addition, the PPI group provided help and input to the elaboration of participant-related materials for the study and will provide support with the dissemination of findings.

Study Design, Setting, Participants, and Recruitment Period

The EMERALD study was designed as a case-referent, cross-sectional, multicenter, diagnostic study with sampling of patients and data collection carried out prospectively, providing a cost-efficient design with low risk of bias in terms of diagnostic accuracy. The study was conducted in ophthalmic clinics of 13 National Health Service (NHS) hospitals across the United Kingdom, with sites in England (n = 11), Scotland (n = 1), and Northern Ireland (n = 1). Eligible participants were adults with diabetes (type 1 or 2) with previously successfully treated DME or PDR in one or both eyes. Participants were considered to have been treated successfully if, at the last visit in clinic, no further treatment had been indicated by the treating ophthalmologists because of lack of activity of PDR or DME. Only participants unable to speak or understand English and those unable to provide informed consent were excluded. Participants were identified through clinical records, electronic databases, or while in the clinic. At the time of enrollment, DME and PDR could be active or inactive. An ophthalmologist confirmed eligibility; for those willing to participate, informed consent was obtained before enrollment. Participants were recruited between October 26, 2017, and June 7, 2019.

Clinical Pathways Assessed and Training of Ophthalmic Graders

New Pathway: Ophthalmic Grader Pathway

The new pathway tested consisted of the review of SD OCT scans to detect DME and of 7-field Early Treatment Diabetic Retinopathy Study (ETDRS) and ultra-widefield (UWF) images to detect PDR by trained and tested ophthalmic graders (see below). Ophthalmic graders determined whether there was active or inactive DME or PDR, or whether they were unsure or unable to grade images, in which case patients would be referred for an ophthalmologist assessment. If no DME or no active PDR was present, the grader would arrange a review appointment for the patient in the ophthalmic grader pathway at a predetermined interval.

Standard-of-Care Pathway (Reference Standard)

The standard-of-care pathway for DME and PDR was the current standard of care: face-to-face evaluation of patients by ophthalmologists using slit-lamp biomicroscopy and SD OCT scans. Active or inactive DME or PDR were judged by ophthalmologists based on clinical examination and, for DME, findings on SD OCT.

Enhanced Reference Standard for Proliferative Diabetic Retinopathy

Because ophthalmologists may miss new vessels when evaluating patients by slit-lamp biomicroscopy, the EMERALD study included an enhanced reference standard for PDR. This consisted of the reference standard, as above, supplemented by the evaluation of 7-field ETDRS and UWF images, both reviewed by an ophthalmologist expert in diabetic retinopathy. If active PDR was detected in 1 of these 3 evaluations (slit-lamp biomicroscopy, 7-field ETDRS images, or UWF fundus images), the patient was considered to have active PDR based on the enhanced reference standard.

Images were obtained by trained ophthalmic photographers and imaging technicians at participating sites. Seven-field ETDRS images were obtained using standard fundus cameras available at each participating site. The Optos system (Optos, Inc) was used to obtain UWF images.

In the EMERALD study, all participants went through the standard-of-care pathway (i.e., they were reviewed by an ophthalmologist who set the reference standard). Spectra-domain OCT scans were obtained according to the standard of care. For the purpose of the study, 7-field ETDRS and UWF images were obtained to detect PDR in the ophthalmic grader pathway and for the enhanced reference standard.

Anonymized images were transferred from participating sites to a central facility, then were assigned randomly to graders and ophthalmologists in the clinical sites. The EMERALD study used a commercially available platform (Ophthalsuite; BlueWorks, Coimbra, Portugal) for graders to see all images on computer screens.

Selection and training of ophthalmic graders was as follows. First, local principal investigators suggested names of individuals at their sites with experience obtaining or grading images of patients with DME or PDR. These individuals were approached to confirm their interest and willingness to participate in the EMERALD study. They were asked to fill out a questionnaire detailing their experience recognizing features of DME and PDR; those who stated they did not have experience and those unwilling to be part of the study were not invited to participate in the EMERALD study.

Candidates to be ophthalmic graders then received formal training. During training, which included a 2-day face-to-face meeting and 2 additional half-day webinar sessions, features of active and inactive DME and PDR were reviewed and discussed, and extensive clinical examples were presented. A web-based teaching module with examples of DME and PDR was provided also, so that graders could consolidate their knowledge. Graders received clear guidelines on when patients needed referral to ophthalmologists. The following definitions for active and inactive DME and PDR were given:

Active DME was defined as DME with central retinal thickness of more than 300 μm on SD OCT, and/or the presence of intraretinal or subretinal fluid, or both, on SD OCT resulting from DME. Isolated or sparse small intraretinal cysts were not considered DME.

Inactive DME was defined as no intraretinal or subretinal fluid.

Active PDR was defined by the presence of subhyaloid or vitreous hemorrhage, active new vessels (new vessels with lack of fibrosis on them), or both.

Inactive PDR was defined by a lack of subhyaloid or vitreous hemorrhage and lack of active new vessels.

After training, ophthalmic graders were required to take a test involving the reading of SD OCT, 7-field ETDRS, and UWF images with and without DME and with and without active PDR. Those reaching a minimum of 80% of correct answers were invited to take part in the EMERALD study. If they failed the first test, graders could undergo further training and take a new test, but if the 80% minimum was not attained, they were unable to be graders for the EMERALD study.

Masking

Ophthalmic graders were masked to the reference standard. To ensure this, they did not interpret images from patients recruited at their own center and had no access to results of the reference standard. They did not read 7-field ETDRS, UWF, and SD OCT images of the same eye to ensure reading of one imaging technology would not influence the reading of the other. Ophthalmologists performing the standard-of-care evaluation (i.e., setting the reference standard) were masked to the findings and decisions made by ophthalmic graders (who reviewed images at a later date) and to the enhanced reference standard.

Outcome Measures

The primary outcome measure was sensitivity of the new pathway to detect DME and active PDR. The secondary outcome measures included specificity, concordance, costs, acceptability of the new pathway to patients and health care professionals, the proportion of patients requiring subsequent assessment by ophthalmologists, the proportion of patients unable to undergo imaging, and the proportion of patients with images of inadequate quality for interpretation.

Acceptability of the New Pathway to Patients and Health Care Professionals

Focus group discussions were undertaken. Participants were approached and consent was obtained from those willing to participate in focus group discussions at the same time that they were approached to participate in the main diagnostic accuracy study. Ophthalmologists and ophthalmic photographers and graders also were invited to participate in separate focus group discussions. Detailed methodology and results of this qualitative part of the EMERALD study will be published separately.

Sample Size and Statistical Analysis

The sample size was determined based on setting a target of the number of people with reactivated (active) DME and PDR required to enable sensitivity to be tested against a prespecified target level of 80%. The required sample size was calculated using formula T1 from Obuchowski in Microsoft Excel (Microsoft); it was a Wald test-based calculation. This level was considered the minimum acceptable for the new pathway to be clinically viable. A lower specificity was considered acceptable; a target of 65% was used to confirm sufficiency of the sample size to assess specificity. Eighty-nine participants with DME or PDR that had reactivated (active DME or PDR) was sufficient to detect if the sensitivity of the new pathway was 10% and 12% higher than the 80% minimal target set with 80% and 90% power, respectively, at the 2-sided 5% significance level. Ninety-three participants whose disease did not reactivate would enable detection of a specificity 15% higher than the 65% target with 90% power. A 95% confidence interval (CI) for the ophthalmic grader pathway sensitivity and specificity would have a confidence interval (Wilson method) with a width of 10% to 20%, depending on the observed level. Allowing for 10% missing or indeterminate results, 104 individuals whose disease had reactivated and 104 whose disease had not were required (208 each for DME and PDR), leading to a maximum of 416 participants in the study overall. Because participants could have both DME and PDR and could contribute to both targets, the number of participants required could be fewer than 416.

Separate analyses were planned for DME and PDR. Participants were categorized as having active or inactive DME or PDR according to the reference standard at the person level. Those with previously successfully treated DME or PDR constituted eligible participants for each analysis (DME and PDR) for the new pathway. This person-based assessment reflects the consequences of the clinical decision in clinical practice. The diagnostic performance of the new pathway was quantified against the reference standard. Reflecting how the new pathway would function in practice, unsure, ungradable, and active classifications required referral and examination by an ophthalmologist under the main analyses. The impact of using 7-field ETDRS images versus UWF images on the diagnostic performance of the new pathway was assessed under the principal analyses for PDR using both the reference standard and enhanced reference standard. Agreement between PDR assessment methods was quantified.

Planned sensitivity analyses included (1) assessment of the impact of unsure and ungradable classifications on the diagnostic performance of the ophthalmic grader; (2) using the ophthalmologist’s decision that further treatment was required, rather than presence of active disease; (3) detection of severe disease (central-involving in DME or subhyaloid or vitreous hemorrhage in PDR); (4) diagnostic performance within routine NHS clinics versus research clinics; and (5) for PDR only, diagnostic performance of the ophthalmic grader against the enhanced reference standard (Table S1, available at www.aaojournal.org).

Secondary analyses included evaluation of eye-level data: analysis of all patients (with or without DME or PDR), assessment of the overall referral (for DME and PDR), and use of visual acuity as a proxy to detect active disease. Additional post hoc analyses were carried out in the PDR group only to aid the understanding of findings of preplanned analyses (Table S1).

The main analysis and sensitivity analysis included only eligible participants for the particular pathway (for the new DME pathway, patients with at least 1 eye with previously successfully treated DME; for the new PDR pathway, patients with at least 1 eye with previously successfully treated PDR). These participants might have had an ineligible eye, but because these analyses were based on the person level (because this is what will happen in real life if the pathway is introduced), each of the two eyes would have been taken into consideration for the analysis. For example, if a participant had a right eye with previously treated and inactive DME, this participant would have entered the DME pathway. If a recurrence of DME were present in the right eye at the time of the EMERALD evaluation, the patient would have been considered to have active DME. Equally, if this same participant showed persistence (i.e., had never been treated successfully before the EMERALD evaluation, but showed active disease at the time of the visit) or de novo disease (active disease at the time of the EMERALD evaluation but never present before) in the left eye, the participant also would have been considered to have active DME. If this same participant did not have PDR in the right eye or left eye before (i.e., not eligible for the PDR pathway) but, at the time of the EMERALD study, showed de novo PDR in 1 eye, this participant would not have been included in main or sensitivity analyses for PDR but would have been included in the secondary analysis. The converse also was true for the DME main and sensitivity analyses, and correspondingly the inclusion of de novo DME in the secondary analysis.

For all diagnostic accuracy analyses, the sensitivity, specificity, and positive and negative likelihood ratios were calculated (with appropriate 95% CIs using Wilson’s method and the diagt command in Stata software, respectively). The difference in sensitivity and specificity between 7-field ETDRS and UWF images assessed by the ophthalmic graders was compared with corresponding 95% CIs produced using Newcombe’s method for paired data and McNemar’s test for the main analysis and sensitivity analysis 1. All analyses were carried out using Stata software version 15 (Stata Corp) and without imputation of missing data.

Health Economic Evaluation

Resource use was captured on EMERALD case report forms at each participant’s EMERALD clinic visit to compare costs of delivering the standard-of-care pathway, the ophthalmic grader pathway, and the enhanced reference standard. The cost analysis took the perspective of the NHS and personal social services and was estimated in United Kingdom pounds sterling using 2019 through 2020 prices. Costs included staff costs, based on the time and staff (including grade) required to obtain best-corrected visual acuity, SD OCT images, 7-field ETDRS images, and UWF fundus images. Costs included time and grade of the ophthalmologist evaluating the patient in the clinic, including undertaking slit-lamp biomicroscopy, review of the SD OCT images to assess DME, as well as the time invested counseling the patient. Times taken by graders to grade SD OCT images and by graders and ophthalmologists (for the purpose of the enhanced reference standard) to grade 7-field ETDRS and UWF fundus images also were obtained and costed. Hourly wage rates for staff costs were obtained from the Unit Costs of Health and Social Care 2019. Other costs included the equipment required, overhead, and consumables. The equipment costs included acquisition and maintenance costs, considering the lifetime of the equipment and estimated throughput per year. Data were not collected on costs to patients.

It was hypothesized that the new pathway would show similar sensitivity as the standard-of-care pathway but at lower cost, making the analysis a cost-consequence one, including assessment of ophthalmologist time released by the new pathway. Diabetic macular edema and PDR were assessed separately. Detailed methodology and results of the health economic evaluation will be published separately.

Statistical analysis and health economic plans were agreed upon and made accessible on the EMERALD website (http://www.nictu.hscni.net/emerald-trial/#) before commencement of data analysis. Further methodologic details of the EMERALD study can be found in the published protocol (https://bmjopen.bmj.com/content/9/6/e027795). The EMERALD study was executed and reported following the Standards for Reporting of Diagnostic Accuracy Studies (STARD) guidelines and was registered prospectively (Clinicaltrials.gov identifier, NCT03490318; ISRCTN identifier, ISRCTN-10856638).

Results

Diagnostic Accuracy

We recruited 397 participants, of whom 272 were eligible with DME and 281 were eligible with PDR (Fig 1; Table S2 and S3, available at www.aaojournal.org). Participants were recruited consecutively, whether they had active or inactive DME or PDR at the time of the EMERALD visit, with no case selection. We had planned to continue recruitment until we had achieved the minimum number of eligible participants for each group (104 individuals each for the active and inactive DME and PDR groups). Because participants could contribute to both the DME and PDR pathways, by February 8, 2019, we had recruited enough participants for 3 groups (active DME, inactive DME, and inactive PDR). People with previously successfully treated and active PDR seemed to be seen in clinics less frequently, and thus, numbers recruited in this group had not reached the required minimum. Thus, from February 8, 2019, when we had 67 participants with active PDR (167 with inactive PDR, 141 active DME, and 107 with inactive DME, respectively), we decided actively to recruit patients who had active PDR only and to ask sites actively to pursue eligible participants for this group (e.g., recruiting from casualty, where these patients initially could seek treatment). Consecutive potentially eligible participants with active PDR then were approached until recruitment for this group was completed also (and surpassed, because recruitment was not halted until all potentially eligible participants identified and approached for the active PDR group had been assessed). Because participants could contribute to all other groups, as mentioned above, the number of eligible participants in all groups increased and was higher by the end of the study than that required based on sample size calculations.

Figure 1

Effectiveness of Multimodal Imaging for the Evaluation of Retinal Oedema and New Vessels in Diabetic Retinopathy flow diagram. DME = diabetic macular edema; O-FTF = Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy; PDR = proliferative diabetic retinopathy; SD = spectral-domain.

Demographics of participants are shown in Table 1. In total, 157 of 397 patients (40%) demonstrated severe disease (central-involving DME) in the DME group, and 132 patients were eligible for the new pathway. In the PDR group, severe disease (PDR with preretinal or vitreous hemorrhage) was present in 77 of 397 participants (19%), and 75 patients were eligible for the new pathway.

Table 1

Demographic Characteristics of Effectiveness of Multimodal Imaging for the Evaluation of Retinal Oedema and New Vessels in Diabetic Retinopathy Study Participants

	Patients with Diabetic Macular Edema (n = 317)	Eligible for Diabetic Macular Edema in the New Pathway (n = 272)	Patients with Proliferative Diabetic Retinopathy (n = 287)	Eligible for Proliferative Diabetic Retinopathy in the New Pathway (n = 281)	Total (n = 397)
Gender
Male	205 (65)	175 (64)	187 (65)	185 (66)	257 (65)
Female	112 (35)	97 (36)	100 (35)	96 (34)	140 (35)
Age (yrs)
18–59	135 (43)	113 (42)	151 (53)	148 (53)	188 (47)
60 and older	182 (57)	159 (58)	136 (47)	133 (47)	209 (53)
Ethnic origin
White	274 (86)	240 (88)	240 (84)	234 (83)	340 (86)
Black	20 (6)	17 (6)	19 (7)	19 (7)	26 (7)
Asian	16 (5)	11 (4)	20 (7)	20 (7)	22 (7)
Middle Eastern	3 (1)	1 (<1)	5 (2)	5 (2)	5 (1)
Other	4 (1)	3 (1)	3 (1)	3 (1)	4 (1)

Data are no. (%).

All participants except 34 (9%) had all images (i.e., SD OCT, 7-field ETDRS, and UWF images) obtained for testing the ophthalmic grader pathway on the same day as the reference standard. The great majority of eyes (92%–97% of eyes, depending on the imaging technology used) could be imaged, and few images were ungradable (1% of SD OCT images, 6% of 7-field ETDRS images, and 5% of UWF images). Details for missing images are summarized in Table S4 (available at www.aaojournal.org).

In the main analysis, ophthalmic graders showed sensitivity of 97% (142/147; 95% CI, 92%–99%) and specificity of 31% (35/113; 95% CI, 23%–40%) when compared with the reference standard to detect DME (Table 2). Similar results were found when evaluating people with DME requiring further treatment, with central-involving DME, and when only referral for active DME was considered (i.e., excluding those with unsure and ungradable results) and when patients were assessed in the NHS versus research clinics (Table 2; Table S5, available at www.aaojournal.org).

Table 2

Diagnostic Performance of the Ophthalmic Grader Pathway for the Diagnosis of Diabetic Macular Edema

	Positive Test Results	Reference Standard	Diagnostic Parameter	No./ Total No.	Estimate (95% Confidence Interval)
Main	Ophthalmic graders referral∗ for DME based on SD OCT images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy with the addition of SD OCT scans to assess active DME in either eye	Sensitivity (%)	142/147	97% (92%–99%)
			Specificity (%)	35/113	31% (23%–40%)
			Positive likelihood ratio	—	1.40 (1.23–1.59)
			Negative likelihood ratio	—	0.11 (0.04–0.27)
SENA1	Ophthalmic graders identified active DME based on SD OCT images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy with the addition of SD OCT scans to assess active DME in either eye	Sensitivity (%)	139/146	95% (90%–98%)
			Specificity (%)	43/113	38% (30%–47%)
			Positive likelihood ratio	—	1.54 (1.32–1·78)
			Negative likelihood ratio	—	0.13 (0.06–0.27)
SENA2	Ophthalmic graders referral for DME based on SD OCT images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy with the addition of SD OCT scans to assess active DME in either eye requiring treatment	Sensitivity (%)	81/85	95% (89%–98%)
			Specificity (%)	36/175	21% (15%–27%)
			Positive likelihood ratio	—	1.20 (1.10–1.31)
			Negative likelihood ratio	—	0.23 (0.08–0.62)
SENA3	Ophthalmic graders identified central involving active DME based on SD OCT images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy with the addition of SD OCT scans to assess central involving active DME in either eye	Sensitivity (%)	121/129	94% (88%–97%)
			Specificity (%)	72/128	56% (48%–65%)
			Positive likelihood ratio	—	2.14 (1.75–2.62)
			Negative likelihood ratio	—	0.11 (0.06–0.22)
SENA6	Ophthalmic graders referral for DME based on SD OCT images in routine clinic	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy with the addition of SD OCT scans to assess active DME in either eye in routine clinic	Sensitivity (%)	81/85	95% (89%–98%)
			Specificity (%)	26/65	40% (29%–52%)
			Positive likelihood ratio	—	1.59 (1.30–1.95)
			Negative likelihood ratio	—	0.12 (0.04–0.32)

DME = diabetic macular edema; SD = spectral-domain; SENA = sensitivity analysis; — = not available.

Grader referral for DME: active + unsure + ungradable.

In the main analysis, ophthalmic graders showed lower sensitivity but higher specificity to detect PDR; both were similar (paired differences in sensitivity, –3% [95% CI, –14% to 8%; P = 0.55]; and specificity, 5% [95% CI, –5% to 16%; P = 0.31]) whether they used 7-field ETDRS images (sensitivity, 85% [87/102; 95% CI, 77%–91%]; specificity, 48% [77/160; 95% CI, 41%–56%]) or UWF images (sensitivity, 83% [87/105; 95% CI, 75%–89%]; specificity, 54% [86/160; 95% CI, 46%–61%; Table 3). Results against the enhanced reference standard were similar to those against the reference standard (for 7-field ETDRS images: sensitivity, 82% [111/135; 95% CI, 75%–88%]; specificity, 54% [68/127; 95% CI, 45%–62%]; for UWF images: sensitivity, 80% [110/138; 95% CI, 72%–86%]; specificity, 60% [76/127; 95% CI, 51%–68%]). Diagnostic accuracy results were similar to those of the main analysis when grading patients requiring further treatment (Table 3; Table S6, S7, and S8, available at www.aaojournal.org). Sensitivity and specificity to detect more severe disease (PDR with subhyaloid or vitreous hemorrhage, or both) seemed to be slightly higher (not formally compared) when using UWF imaging (sensitivity, 87% [62/71; 95% CI, 78%–93%]; specificity, 49% [95/193; 95% CI, 42%–56%]) instead of 7-field ETDRS imaging (sensitivity, 80% [53/66; 95% CI, 69%–88%]; specificity, 40% [79/196; 95% CI, 34%–47%]). Findings were similar whether patients were assessed in the NHS or research clinics. Sensitivity and specificity were lower when considering only referrals for active PDR (i.e., excluding those with unsure and ungradable results; Table 3).

Table 3

Diagnostic Performance of the Ophthalmic Grader Pathway for the Diagnosis of Proliferative Diabetic Retinopathy

	Results	Reference Standard	Diagnostic Parameter	No./Total No.	Estimate (95% Confidence Interval)
Main	Ophthalmic graders referral∗ for PDR based on ultra-widefield fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye	Sensitivity (%)	87/105	83% (75%–89%)
			Specificity (%)	86/160	54% (46%–61%)
			Positive likelihood ratio	—	1.79 (1.48–2.16)
			Negative likelihood ratio	—	0.32 (0.20–0.50)
	Ophthalmic graders referral for PDR based on 7-field ETDRS fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye	Sensitivity (%)	87/102	85% (77%–91%)
			Specificity (%)	77/160	48% (41%–56%)
			Positive likelihood ratio	—	1.64 (1.39–1.95)
			Negative likelihood ratio	—	0.31 (0.19–0.50)
SENA1	Ophthalmic graders identified active PDR based on ultra-widefield fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye	Sensitivity (%)	66/105	63% (53%–71%)
			Specificity (%)	116/159	73% (66%–79%)
			Positive likelihood ratio	—	2.32 (1.73–3.12)
			Negative likelihood ratio	—	0.51 (0.39–0.66)
	Ophthalmic graders identified active PDR based on 7-field ETDRS fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye	Sensitivity (%)	70/99	71% (61%–79%)
			Specificity (%)	110/158	70% (62%–76%)
			Positive likelihood ratio	—	2.33 (1.78–3.04)
			Negative likelihood ratio	—	0.42 (0.30–0.58)
Additional 1	Ophthalmologist assessment identified active PDR based on ultra-widefield fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye	Sensitivity (%)	74/103	72% (62%–80%)
			Specificity (%)	137/159	86% (80%–91%)
			Positive likelihood ratio	—	5.19 (3.46–7.80)
			Negative likelihood ratio	—	0.33 (0.24–0.45)
	Ophthalmologist assessment identified active PDR based on 7-field ETDRS fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye	Sensitivity (%)	65/98	66% (57%–75%)
			Specificity (%)	134/154	87% (81%–91%)
			Positive likelihood ratio	—	5.11 (3.31–7.87)
			Negative likelihood ratio	—	0.39 (0.29–0.51)
SENA2	Ophthalmic graders referral for PDR based on ultra-widefield fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye requiring treatment	Sensitivity (%)	77/90	86% (77%–91%)
			Specificity (%)	91/175	52% (45%–59%)
			Positive likelihood ratio	—	1.78 (1.49–2.13)
			Negative likelihood ratio	—	0.28 (0.16–0.47)
	Ophthalmic graders referral for PDR based on 7-field ETDRS fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye requiring treatment	Sensitivity (%)	74/84	88% (79%–93%)
			Specificity (%)	82/178	46% (39%–53%)
			Positive likelihood ratio	—	1.63 (1.40–1.91)
			Negative likelihood ratio	—	0.26 (0.14–0.47)
SENA4	Ophthalmic graders referral for PDR based on ultra-widefield fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR with preretinal or vitreous hemorrhage in either eye	Sensitivity (%)	62/71	87% (78%–93%)
			Specificity (%)	95/193	49% (42%–56%)
			Positive likelihood ratio	—	1.71 (1.45–2.02)
			Negative likelihood ratio	—	0.26 (0.14–0.48)
	Ophthalmic graders referral for PDR based on 7-field ETDRS fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR with preretinal or vitreous hemorrhage in either eye	Sensitivity (%)	53/66	80% (69%–88%)
			Specificity (%)	79/196	40% (34%–47%)
			Positive likelihood ratio	—	1.35 (1.14–1.59)
			Negative likelihood ratio	—	0.49 (0.29–0.82)
Additional 2	Ophthalmologist assessment identified active PDR based on ultra-widefield fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR with preretinal or vitreous hemorrhage in either eye	Sensitivity (%)	57/70	81% (71%–89%)
			Specificity (%)	153/192	80% (73%–85%)
			Positive likelihood ratio	—	4.01 (2.96–5.42)
			Negative likelihood ratio	—	0.23 (0.14–0.38)
	Ophthalmologist assessment identified active PDR based on 7-field ETDRS fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR with preretinal or vitreous hemorrhage in either eye	Sensitivity (%)	42/64	66% (53%–76%)
			Specificity (%)	145/188	77% (71%–83%)
			Positive likelihood ratio	—	2.87 (2.09–3.94)
			Negative likelihood ratio	—	0.45 (0.31–0.63)
SENA5	Ophthalmic graders referral for PDR based on ultra-widefield fundus images	Enhanced reference standard	Sensitivity (%)	110/138	80% (72%–86%)
			Specificity (%)	76/127	60% (51%–68%)
			Positive likelihood ratio	—	1.98 (1.58–2.49)
			Negative likelihood ratio	—	0.34 (0.24–0.49)
	Ophthalmic graders referral for PDR based on 7-field ETDRS fundus images	Enhanced reference standard	Sensitivity (%)	111/135	82% (75%–88%)
			Specificity (%)	68/127	54% (45%–62%)
			Positive likelihood ratio	—	1.77 (1.45–2.17)
			Negative likelihood ratio	—	0.33 (0.22–0.49)
Additional 3	Ophthalmic graders referral for PDR based on ultra-widefield fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye plus ophthalmologist assessment identified active PDR in either eye based on ultra-widefield fundus images	Sensitivity (%)	101/125	81% (73%–87%)
			Specificity (%)	80/140	57% (49%–65%)
			Positive likelihood ratio	—	1.89 (1.53–2.32)
			Negative likelihood ratio	—	0.34 (0.23–0.49)
	Ophthalmic graders referral for PDR based on 7-field ETDRS fundus images	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye plus ophthalmologist assessment identified active PDR in either eye based on 7-field ETDRS fundus images	Sensitivity (%)	103/122	84% (77%–90%)
			Specificity (%)	73/140	52% (44%–60%)
			Positive likelihood ratio	—	1.76 (1.46–2.13)
			Negative likelihood ratio	—	0.30 (0.19–0.46)
SENA6	Ophthalmic graders referral for PDR based on ultra-widefield fundus images in routine clinic	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye in routine clinic	Sensitivity (%)	63/77	82% (72%–89%)
			Specificity (%)	47/92	51% (41%–61%)
			Positive likelihood ratio	—	1.67 (1.32–2.11)
			Negative likelihood ratio	—	0.36 (0.21–0.60)
	Ophthalmic graders referral for PDR based on 7-field ETDRS fundus images in routine clinic	Ophthalmologist face-to-face clinical evaluation using slit-lamp biomicroscopy to assess active PDR in either eye in routine clinic	Sensitivity (%)	60/74	81% (71%–88%)
			Specificity (%)	41/91	45% (35%–55%)
			Positive likelihood ratio	—	1.48 (1.19–1.83)
			Negative likelihood ratio	—	0.42 (0.25–0.71)

ETDRS = Early treatment diabetic retinopathy study; PDR = proliferative diabetic retinopathy; SENA = sensitivity analysis; — = not available.

Grader referral for PDR: active + unsure + ungradable.

Results of post hoc additional analyses for PDR, and the secondary analyses, are shown in Table 3 and Table S9 (available at www.aaojournal.org). The additional analyses for PDR tended to show similar results or increased specificity with reduced sensitivity. Secondary analyses showed very high sensitivity with low or very low specificity. No adverse events were experienced by participants in either pathway.

Acceptability

Thirty-six participants attended focus groups organized in Northern Ireland (n = 4), Scotland (n = 2), and England (n = 4). Participants voiced preference for face-to-face examinations by ophthalmologists, where information about their eye condition could be received and discussed and where they would have the opportunity to ask questions and have anxieties assuaged. In their absence, they wished immediate results from the grader’s assessment and maintaining periodic evaluations by ophthalmologists, even if at longer intervals. Participants were uncertain of professional identity, training, and performance of photographers and graders. Graders and ophthalmologists supported the new pathway, but graders expressed caution about their ability to answer questions from patients unrelated to the activity of their disease.

Cost-Consequence Analysis

For DME, the cost difference (savings) for the grader’s pathway is £1390 per 100 follow-up visits. For PDR, the cost difference (savings) for the grader’s pathway is £461 for 7-field ETDRS images and £1889 for UWF images per 100 follow-up visits. The main driver of the difference in costs of imaging methods for PDR was the time to obtain and read images (Table S10, available at www.aaojournal.org). Costs for the grader pathway took into account the specificity of the pathway (i.e., in each 100 patients, a proportion of false-positive results still need to be referred to an ophthalmologist, with the reference standard cost for ophthalmologist follow-up applied).

Discussion

The new ophthalmic grader pathway showed high sensitivity to detect DME of more than 90% in all analyses, suggesting that it would be safe to implement in clinical practice. The pathway showed lower sensitivity to detect PDR, albeit more than the 80% level set. Importantly, the sensitivity of the ophthalmic grader pathway to detect high-risk PDR, with preretinal or vitreous hemorrhage, or both, was higher (87%) when using UWF fundus images. It should be highlighted that the risk and consequences of a recurrence of PDR in eyes previously treated with PRP would not be expected to be as high or as severe as if active disease were to occur in treatment-naive eyes. If a vitreous hemorrhage were to develop, patients would experience floaters and could be instructed to contact ophthalmic clinics immediately for timely evaluation. In most instances, the course of action is observation until the hemorrhage clears, and then further PRP treatment if required. With this in mind, the ophthalmic grader pathway for PDR is considered adequate and justifiable, especially in areas and at times of high demand for services that prevent people with severe eye diseases from accessing timely care. Given that UWF images showed higher sensitivity to detect high-risk PDR and were less costly than 7-field ETDRS images, they may be preferred.

The specificity of the new pathway to detect DME (31%) and PDR (54%–60%) was not high. The lower the specificity, the more patients with false-positive results who have to be seen by the ophthalmologist. However, even a poor specificity could provide useful savings in ophthalmologist time. In the EMERALD study, images were evaluated without any information about patients (i.e., masked to any clinical data, including previous images). Although this was a strength in scientific design, it is likely that if clinical information (e.g., location of previously identified new vessels) and previous images (e.g., SD OCT scans of previously treated DME or images of new vessels after PRP treatment) were to be available, the sensitivity and specificity of the new pathway would have been higher. Indeed, if the new pathway is implemented in clinical practice, previous clinical information and images could be available to ophthalmic graders.

The new ophthalmic grader pathway, if implemented appropriately, would help health services to increase capacity, would reduce waiting times for patients to be seen in clinics, and subsequently, would save sight. For example, the pathway could be implemented as a 1-stop clinic, with images and image review being carried out at the same session and ophthalmic graders providing the results to patients immediately. If ophthalmologists were to be running parallel clinics, they could provide advice to graders, if needed, in questionable cases, increasing the efficiency of the service and reducing the number of patients who would need to return for a further ophthalmologist assessment. If planned adequately, it may even be possible to administer treatments to active patients requiring them at the same visit (e.g., after these clinics have been running for some time, it would be possible to determine the average number of patients requiring input from the ophthalmologist as well as those requiring treatment and to plan accordingly). Patients with previously successfully treated and stable disease (DME, PDR, or both) could be preselected by ophthalmologists to go into the ophthalmic grader pathway. Based on the EMERALD study, patients could be moved to the grader’s pathway as soon as further treatment for DME or PDR is not indicated. Alternatively, ophthalmologists may decide, for example, to refer to the grader pathway those patients with PDR with adequate laser PRP whose disease has remained stable for a number of months already (e.g., 3–4 months), patients with DME who received focal laser treatment and in whom DME has resolved, and patients with DME who received anti-VEGF therapy and who remained free of fluid for a certain period (e.g., 2–3 consecutive visits). Based on the focus group work conducted in the EMERALD study and to ensure acceptability by patients of the new pathway, it would be important that, from time to time, patients whose disease remains inactive are still seen by ophthalmologists.

No clear view exists on what should be the minimal sensitivity and specificity acceptable for diagnostic or surveillance pathways. Figures of 80% for sensitivity and 95% for specificity have been quoted in many articles on screening for diabetic retinopathy. These figures seem to have originated from a 1997 British Diabetic Association document, based on a consensus conference in 1995 (however, this document is no longer accessible). Surveillance of previously treated patients, in any case, is a rather different scenario and would pose different, known, risks than DR screening, in which those naïve to treatment are followed up at less frequent intervals.

In the future, it may be possible to use automatic image analysis, including artificial intelligence, to determine presence of active DME or PDR on fundus and SD OCT images. Recent studies demonstrated excellent sensitivity and specificity of artificial intelligence methods to determine presence of referable diabetic retinopathy (defined as presence of moderate and higher stages of nonproliferative diabetic retinopathy, PDR, or DME) in fundus images when compared with evaluation by retinal specialists., Indeed, an artificial intelligence system (IDx-DR; IDx Technologies, Inc., Coralville, IA) has been developed and approved by the Food and Drug Administration for the automated diagnosis of diabetic retinopathy. However, studies on which this program was developed included mostly treatment-naive patients, and thus, it remains to be elucidated if its diagnostic performance would be the same in the more complex group of previously treated patients who will have demonstrated alterations in retinal structure even when active disease is not present.

The concept of what has been widely called virtual clinics (evaluation of patients by looking at their images rather than through a face-to-face consultation in clinic) is not new. Published studies presented the experience of several groups using this form of evaluation for people with age-related macular degeneration and other medical retinal diseases, including diabetic retinopathy41, 42, 43, 44 and glaucoma. These studies showed that implementation of virtual clinics was feasible and reduced patients’ time in clinic, improving patients’ journeys, and seemed to increase the efficiency of the service. However, most studies were based on the assessment of images by ophthalmologists, rather than allied nonmedical staff, included newly referred patients, rather than previously treated ones, and had a selected population, rather than all-comers. Very few studies evaluated the acceptability of virtual clinics to patients and health care professionals; these used questionnaires, and had low ascertainment (46%–61%).

The EMERALD findings may be of greatest relevance to countries with tax-funded health care systems, those having difficulties coping with health care demands, especially because of the shortage of ophthalmologists, and in particular low- and middle-income countries and rural and underserved populations, interested in identifying more efficient and less costly health care strategies. The EMERALD study also could serve as an example of using allied health care professionals in other areas of health care in ophthalmology and even outside this specialty.

Strengths of the EMERALD study include its multicenter nature, strong methodology, adequate power and recruitment, and lack of patient selection, making results more generalizable and applicable to routine care. Caveats include the fact that images of the iris and anterior chamber angle were not obtained for the evaluation of people with PDR. Although it would be very rare that new vessels would develop in these structures in eyes previously treated with laser PRP with no concomitant active new vessels elsewhere (NVE) or new vessels in the disc (NVD), if present, they would be missed. Additionally, fluorescein angiography was not undertaken as part of the study to determine activity of PDR. It would be essential, if the new pathway is implemented, that recommendations from the focus group discussions were to be followed to ensure its acceptability to users.