The design and implementation of a study to investigate the effectiveness of community vs hospital eye service follow-up for patients with neovascular age-related macular degeneration with quiescent disease.

IntroductionStandard treatment for neovascular age-related macular degeneration (nAMD) is intravitreal injections of anti-VEGF drugs. Following multiple injections, nAMD lesions often become quiescent but there is a high risk of reactivation, and regular review by hospital ophthalmologists is the norm. The present trial examines the feasibility of community optometrists making lesion reactivation decisions.MethodsThe Effectiveness of Community vs Hospital Eye Service (ECHoES) trial is a virtual trial; lesion reactivation decisions were made about vignettes that comprised clinical data, colour fundus photographs, and optical coherence tomograms displayed on a web-based platform. Participants were either hospital ophthalmologists or community optometrists. All participants were provided with webinar training on the disease, its management, and assessment of the retinal imaging outputs. In a balanced design, 96 participants each assessed 42 vignettes; a total of 288 vignettes were assessed seven times by each professional group.The primary outcome is a participant's judgement of lesion reactivation compared with a reference standard. Secondary outcomes are the frequency of sight threatening errors; judgements about specific lesion components; participant-rated confidence in their decisions about the primary outcome; cost effectiveness of follow-up by optometrists rather than ophthalmologists.DiscussionThis trial addresses an important question for the NHS, namely whether, with appropriate training, community optometrists can make retreatment decisions for patients with nAMD to the same standard as hospital ophthalmologists. The trial employed a novel approach as participation was entirely through a web-based application; the trial required very few resources compared with those that would have been needed for a conventional randomised controlled clinical trial.

RCT Entities:   Population Interventions Outcomes

Introduction

Neovascular age-related macular degeneration (nAMD) is common and can cause severe sight loss and blindness. Currently, patients with nAMD are treated with intravitreal injections of drugs that inhibit vascular endothelial growth factor (anti-VEGF).[1] These drugs ameliorate the exudative manifestations of the posterior fundus and improve the morphological appearance of the retina, leading to stabilisation or improvement of visual acuity in most patients.[2, 3] The nAMD lesion can be rendered quiescent but re-activation of the lesion is common.

One of two review strategies are typically used: (a) review monthly until active disease recurs, or vision drops or (b) treat even if there is no fluid at the macula (usual criterion for retreatment) but extend the interval between review visits. The former is burdensome for patients and for the National Health Service (NHS), and the latter leads to overtreatment with its additional risks and expense. Even without patients receiving treatment, regular monthly review requires ophthalmologists' time and other health service resources.

While there is no evidence on the effectiveness of community follow-up by optometrists for nAMD, there is considerable data supporting their role in the provision of ‘shared care' with the United Kingdom Hospital Eye Service (HES) for other eye diseases such as glaucoma, diabetes, and emergency eye care.[4, 5, 6, 7, 8]

A review outlined different approaches to increase the capacity in nAMD services across the United Kingdom.[9] The case studies in the review show a variety of scenarios. Many involve extended roles for optometrists and nurse practitioners but these occur in the HES. The effectiveness of these management pathways has not yet been formally evaluated. Some studies have investigated the potential of remote care, which involves assessments by a retinal specialist of optical coherence tomograms (OCT) captured in outreach services.[10, 11]

There is the opportunity in the United Kingdom, and other countries with a widely available optometric primary care service, for a shared care scheme for patients with quiescent nAMD, with community optometrists taking responsibility for regular review and referring patients with reactivated nAMD back to eye clinics for retreatment. Community optometrists already have the necessary training to recognise nAMD (they are responsible for the majority of referrals to the HES), and some United Kingdom community optometric practices have already invested in the technology for performing digital colour fundus (CF) and OCT photography and use these technologies for decisions about diagnosis and referral. Identifying a reactivated lesion is more difficult as this requires differentiation of quiescent from active disease rather than detection of disease in a previously normal eye. The skill and ability of optometrists to differentiate quiescent from active nAMD has not been formally evaluated nor, as far as we are aware, has a shared care management scheme for nAMD.

Long-term studies[12, 13, 14, 15, 16, 17] indicate that quiescent neovascular lesions frequently reactivate. As the workload associated with reviewing and treating nAMD continues to rise, many NHS hospitals are struggling to provide regular monthly reviews with around 25% of patients having fewer than 7 visits per year.[18] Therefore, we sought to evaluate whether community optometrists can be trained to make decisions about the need for retreatment in patients with quiescent nAMD with the same accuracy as ophthalmologists, as a necessary step in establishing the feasibility of a shared care scheme.

A conventional, parallel-group trial that randomised patients to retreatment decision making by either ophthalmologists or optometrists, comparing ensuing outcomes in the two groups, was not considered feasible for two reasons. First, patients might be unwilling to consent to randomisation to decision-making by optometrists, perceiving it to be potentially risky. Second, a conventional trial would be expensive and take a long time. Therefore, the Effectiveness of Community vs Hospital Eye Service (ECHoES) trial was designed as a virtual trial. We hypothesised that the ability of suitably trained optometrists to make correct lesion classification decisions from clinical vignettes would not be inferior to that of ophthalmologists. The trial design allows lesion classification decisions made by participants in the two professional groups for the same vignettes to be compared against each other and against a reference standard.

Objectives

The objectives of the ECHoES trial are to:

Compare the proportion of lesion classification decisions made by optometrists and ophthalmologists scored as ‘correct' (against a reference standard).

Estimate the agreement, and nature of disagreements, between lesion classification decisions made by optometrists and ophthalmologists.

Estimate the influence of vignette clinical and demographic information on lesion classification decisions.

Estimate the cost-effectiveness of follow-up in the community by optometrists compared with follow-up by ophthalmologists in the HES.

Ascertain the views of patient representatives, optometrists, ophthalmologists, and clinical commissioners on the proposed shared care model.

Materials and methods

Trial design

The ECHoES trial was designed to emulate the hypothetical parallel-group trial described in Table 1. However, the ECHoES trial is more analogous to a cross-over trial than a parallel group trial, since all vignettes were reviewed by both optometrists and ophthalmologists, in a randomised balanced incomplete block design.[19] This well-established design is only possible with limited permutations of the total number of vignettes, participants, and vignettes per participant. We used a total of 288 vignettes. Forty-eight ophthalmologists and forty-eight optometrists were assigned forty-two vignettes. The sample of vignettes differed from participant to participant within each professional group but each vignette was assigned to seven participants ((48 × 42)/7=288). Pairs of participants from each profession were assigned exactly the same selection of vignettes.

Table 1

Research question components tested in the ECHoES trial compared with a hypothetical parallel group trial

Research question component	ECHoES (cross over) trial	Conventional (parallel group) trial
Population	Vignettes representing patients with quiescent nAMD being monitored for reactivation	Patients with quiescent nAMD being monitored for nAMD reactivation
Intervention	Assessment of vignettes for nAMD reactivation by a traineda optometrist	Monthly review by a community optometrist, after training,b to detect nAMD reactivation
Comparator	Assessment of vignettes for nAMD reactivation by a traineda ophthalmologist	Monthly review by an ophthalmologist in the HESb to detect nAMD reactivation
Outcome	Decision about nAMD reactivation (presumed to lead to appropriate treatment to persevere visual acuity)	Visual acuity

In the ECHoES trial, both professional groups are required to undergo identical training.

In the parallel group trial, optometrists would be required to undergo training (not defined here but which could be similar to the training provided in the ECHoES trial). In a pragmatic design, ophthalmologists in the HES might be assumed to be adequately trained already, since both trials will/would recruit from ophthalmologists who already make such decisions in the HES.

The trial has been approved by a NHS Research Ethics Committee (ref 11/129/195) and is registered (ISRCTN 07479761). All professional and qualitative research participants gave written informed consent.

Vignettes (cf. ‘population')

A database of vignettes for the ECHoES trial was created from information collected for the IVAN (alternative treatments to inhibit VEGF in age-related choroidal neovascularisation) trial, which evaluated alternative anti-VEGF treatment for nAMD.[12, 20, 21] The trial collected 3-monthly CF and OCT images (held by the Network of Ophthalmic Reading Centres for the United Kingdom, NetwORC), characterising lesions with varying levels of activity including complete quiescence. The vignettes in the ECHoES trial were populated only with OCTs captured on spectral domain systems (≈25% of all available OCT images).

Each vignette consisted of sets of CF and OCT images from the study eye at two time points (‘baseline' and ‘index'), with accompanying clinical information and visual acuity measurements (Figure 1). The baseline set were images from a study visit when the lesion was quiescent (ie, all macular tissue compartments were fluid free). The index set consisted of images from another study visit at which the lesion could be active or quiescent. Participants reviewed the images and clinical information and classified the lesion as reactivated, suspicious or quiescent at the index visit. This task is similar to ‘2-stop' nAMD service provision in some hospitals, whereby patients have retinal imaging at one appointment (without a clinical assessment), followed by review of the images and recall for a second appointment only if the lesion is suspicious or has reactivated.

Figure 1

Example vignette.

Images for vignettes were selected as follows. First, the database was scrutinised to identify OCTs captured on a spectral domain system. Second, visits were identified for which the OCT grading data were consistent with a quiescent lesion. Finally, a senior grader from the NetwORC and a retinal specialist reviewed the proposed ‘baseline' sets of images; for each confirmed quiescent lesion, CFs and spectral domain OCTs for other visits for the same patient were inspected to identify another study visit for which eligible images were available. Images had to be of sufficient clarity and quality to allow interpretation of disease activity by clinicians.

The clinical information presented in the vignettes can been seen in Figure 1.

Ophthalmologist and optometrist assessors (cf. ‘comparator' and ‘intervention')

Ophthalmologists were required to have ≥3 years post-registration experience in ophthalmology, have passed part 1 of the Royal College of Ophthalmologists or the Diploma in Ophthalmology or equivalent, and have experience within the AMD service. Optometrists were required to be fully qualified, registered with the General Optical Council (GOC) for ≥3 years, and not be participating in any AMD shared care schemes.

Training of participants

Both ophthalmologists and optometrists are qualified to detect retinal pathology but optometrists may not have the skills to detect lesion reactivation. Eligible ophthalmologists may also not have been fully trained in these skills (despite the inclusion criterion of ‘have experience within an AMD service') since clinical trainees without specialist skills often staff retina clinics in the HES. Therefore, all participants received the same training, consisting of two elements: first, participants had to attend two lecture presentations, with the opportunity to ask questions; second, each participant had to assess a training set of vignettes and achieve a criterion level of performance.

Lectures (presented as webinars) with standard ‘voice-over' commentaries were developed to ensure all participants had the same background knowledge at the point of starting their assessments. Attendees could ask questions, which were answered by an expert retinal specialist hosting the webinar. We intended that the training package should be well defined and reproducible so that it could be made widely available for use in the community. (If the trial hypothesis is refuted, we will need to examine whether this finding arose due to inadequate training and consider how training could be improved.)

The lectures consisted of short explanatory material on: (a) the background to the ECHoES trial, (b) diagnosis, management, and current treatment pathways in nAMD, (c) the response to treatment and functional and morphological outcomes to be expected with current treatments, (d) monitoring of outcomes and retreatment decision-making criteria, (e) a range of clinical presentations highlighting different lesion features and their importance for retreatment decision-making, and (f) how to use the web-based application. This material was split over two sessions, in total lasting 85 min (Supplementary Table A1).

With regard to the assessment of the vignettes, the lectures explained that in order for a lesion to be classified as reactivated, at least one of the features (sub-retinal fluid (SRF), intra-retinal cysts (IRC), diffuse retinal thickening (DRT), blood, and exudates) must be present. For a lesion to be classified as quiescent, all of the features must be absent or must not have increased since baseline. Figure 2 and Supplementary Table A2 show the order in which participants were asked to make judgements about the presence or absence of features indicating lesion activity.

Figure 2

Example responses, feedback summary, and query.

After participants had attended both webinars, they received a set of 24 training vignettes selected at random from the vignette database excluding any vignette pre-assigned for their main trial assessments. Participants were required to correctly classify the lesion status for 18 of these training vignettes, which we considered a credible criterion for proceeding given the difficulty of classifying some vignettes. If a participant failed to attain this criterion, then a second attempt was permitted with a different training set of vignettes. Participants who failed to meet the criterion twice were withdrawn from the trial. Participants who attained the pass score could then access the 42 vignettes assigned to them for their main trial assessments. A consolidated standards of reporting trials (CONSORT)-style diagram shows participant enrolment and flow through the ECHoES trial (Figure 3).

Figure 3

Participant flow. At the start of the trial, we were unsure of how many participants we would need to recruit in order to meet our target of 48 participants in each group. Therefore, we over recruited at the consent stage and asked a number of participants to complete the webinar and then ‘wait and see' whether we needed them to participate in the main trial. We also slightly over recruited at each stage of the trial to account for drop outs. This resulted in a small number of participants being withdrawn at various stages of the trial because they were no longer required.

The reference standard

The reference standard was established from the judgements of three medical retina expert investigators (UC, SPH, and TP). The experts independently assessed the features of index images in exactly the same way as participants (see above). All three experts assessed all 288 vignettes.

Judgements of experts did not always agree and a consensus meeting was held to review the 24% of vignettes for which experts' classifications of lesion status disagreed (over half of disagreements arose from one expert classifying the lesion as suspicious). At this meeting, experts reviewed the vignettes together, without reference to their previous assessments, and reached a consensus agreement. The reference standard classification was the classification agreed by all three experts: reactivated (142 vignettes, 49.3%), quiescent (141 vignettes, 49.0%), or suspicious (5 vignettes, 1.7%). Allowing classifications of suspicious was unplanned but required either because of unusual features of the images or because the experts could not reach consensus.

Outcomes

Primary outcome

The primary outcome is a participant's judgement of the reactivation status of a lesion from the index images in a vignette, ie, ‘reactivated', ‘suspicious', or ‘quiescent', compared with the reference standard. The number of ‘correct' assessments (defined as both participant and reference standard classifying a vignette as reactivated, or both parties classifying a vignette as suspicious or quiescent) will be compared between the two groups.

Secondary outcomes

The frequency of potentially sight threatening errors (where the reference standard classification is reactivated and the participant classification is quiescent);

Participants' judgements about the presence or absence, and increase from baseline, of lesion components (SRF, IRC, DRT, pigment epithelial detachment (PED), blood, and exudates);

Participant-rated confidence in their decisions about the primary outcome on a 5-point scale.

Economic evaluation

An economic evaluation is being undertaken alongside the main trial, in accordance with recognised economic evaluation guidelines,[22, 23, 24] to estimate the incremental cost and effectiveness of optometrists compared with ophthalmologists performing retreatment assessments for patients with quiescent nAMD. This will help to determine which professional group represents the best use of scarce NHS resources in this context. The main outcome measure is the cost per accurate lesion classification decision.

Data on resource use and costs are being collected using bespoke costing questionnaires for optometrists, including sections on staff costs, any equipment and consumable costs and overheads for optometrist practices. As some optometrist practices are likely to incur set up costs associated with assessing the need for retreatment, the questionnaire specifically addresses, for example, what new equipment items would need to be purchased to perform this additional activity. The costs associated with training optometrists to perform the assessments will also be estimated, as this is important information for policy makers to decide whether or not to implement shared care for nAMD across the United Kingdom. For information on the costs of ophthalmologists performing the assessments, cost data from the IVAN trial will be used.

The baseline analysis will calculate the average cost and outcome for each monitoring review of a ‘patient' by a participant (ie, lesion classification for a vignette and the consequences of the classification). Based on this information, the incremental cost-effectiveness ratios for the different assessment options will be derived, producing an incremental cost per accurate retreatment decision. Sensitivity analysis will be used to demonstrate the impact of variation around the key parameters in the analysis on the baseline cost-effectiveness results. Results will be expressed in terms of a cost-effectiveness acceptability curve, showing the likelihood that the results fall below a given cost-effectiveness threshold.

Qualitative research

Qualitative research is also being undertaken alongside the main trial to explore the feasibility and acceptability of the proposed shared care model for patients with quiescent nAMD to different stakeholders (including ophthalmologists, optometrists, services users, and commissioners). The aim of the qualitative research is to identify the different stake holders' perspectives on facilitators and barriers to the proposed shared care model.

Focus groups and interviews were conducted using a topic guide to ensure that discussions cover the same issues but with the flexibility to allow new issues of importance to the informants to emerge. Discussions were audio-recorded and transcribed verbatim, and will be analysed thematically using constant comparative techniques.

Implementation and trial management

The web application

A secure web-based application was developed to allow participants to take part in the trial remotely. Participation in the trial was managed entirely through the web application, with the application secured by password protection.

The database underpinning the web application managed recruitment, consent, and the presentation of vignettes allocated to each participant including the training sets of vignettes. As participants assessed vignettes, the application recorded their responses to questions about the individual features of index images as well as the overall lesion classification status. Pop-up prompts prevented data entry or key stroke errors. The systems presented a summary of the participant's responses, together with any query, when the lesion status classification was entered (Figure 2). The participant either confirmed that the responses were correct or made changes and, finally, assigned a score from 1 to 5 to describe their confidence in the assessment (1=not confident at all and 5=very confident). Once a participant had confirmed their responses, none of the data could be changed.

The web application also gave the trial management team tools for managing and monitoring the conduct of the trial, including both an individual's progress and the progress of the participants collectively. Additionally, the application provided an electronic delegation log and electronic sign-off of any study amendments for research governance purposes.

Selection and recruitment of participants

Information about the trial was publicised in optometry journals and forums to attract optometrists and circulated to ophthalmologists who are members of the United Kingdom and Welsh medical retina groups. Potential participants could read the participant information sheet and register an interest in joining the trial through the ECHoES trial public website.[25]

Once a participant had registered an interest, the study team checked their eligibility against the General Medical Council and General Optical Council registries. Eligible participants were then sent a study consent form by e-mail.

Sample size

With respect to the primary outcome, the trial was designed to answer the non-inferiority question ‘Is the performance of optometrists as good as that of ophthalmologists?' A sample of 288 vignettes was chosen to have at least 90% power to test the hypothesis that the proportion of vignettes for which lesion status was correctly classified by the optometrist group was no more than 10% lower than the proportion correctly classified identified by the ophthalmologist group, if the proportion of vignettes for which lesion status was correctly classified by the ophthalmologist group was at least 95%. This sample size calculation assumed that each vignette would be assessed by only one ophthalmologist and one optometrist. However, as each vignette was assessed seven times by each group, the trial in fact has 90% power to detect non-inferiority for lower proportions of vignettes correctly classified by the ophthalmologist group.

Plan for statistical analyses

All primary and secondary outcomes will be analysed using mixed-effect regression models, adjusting for the order the vignettes were viewed as a fixed effect (tertiles: 1–14, 15–28, and 29–42), and participant and vignette as random effects. All outcomes are binary and as such will be analysed using logistic regression with group estimates presented as odds ratios with 95% confidence intervals. The sensitivity and specificity of the primary outcome will also be presented. In addition to the group comparisons, the influence of key vignette information on participants' ability to assess the primary outcome correctly will be investigated using Poisson regression. Further details will be included in the statistical analysis plan.

Trial status

Recruitment has closed and data are being analysed. A total of 155 health-care professionals, 72 ophthalmologists, and 83 optometrists, registered an interest, of whom 62 ophthalmologists and 67 optometrists consented, and 125 started the training phase (Figure 3). As planned, 96 participants (48 from each profession) completed the trial and constitute the analysis population. There are no missing data.

Discussion

The ECHoES trial is funded through the rapid trials funding call advertised by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme in October 2011. The intention was to commission projects that would influence clinical practice in the NHS within 3–4 years. In addition to the novel design of the ECHoES trial, one of the main challenges has been the requirement to carry out the research in a short time period.

The ECHoES trial design has several advantages over a conventional randomised trial in which recruitment of patient participants would have had considerable resource implications, been costly, time consuming and potentially unacceptable to patients with nAMD. The data collection element was small; it consisted mainly of participants' responses to questions about image features, lesion status classifications, referral decisions, resource use and costs. It used very little paper because data were entered directly into the web application and most of the trial related administration was also carried out electronically. Very few face-to-face meetings were required to develop the vignettes and the reference standard was created mainly by the experts using the web application. Trial administration and participant correspondence/progression through the trial was managed electronically so there are negligible archiving costs.

Table 2 shows some of the strengths and limitations to the ECHoES study design. A key challenge was the paucity of suitable baseline and index image pairs, requiring the same vignettes to be used for training purposes as well as for the main trial assessments. We avoided any bias by excluding a participant's main trial vignette assessments from their training vignettes. However, this restriction meant that standby participants could not assess any training vignettes until the study team knew which existing participant they were replacing. Since no similar study has been previously attempted, we had to make assumptions about the effectiveness of training and the number of participants required to ensure that 48 from each profession would complete their main assessments.

Table 2

Strengths and limitations of the ECHoES study design

	Strengths	Limitations
Real life data (IVAN data set)	Data used are from many centres, wide range of participants so likely to representative	OCT technology was not state of the art when IVAN was carried out. Newer OCT equipment that produces images with better resolution would now be available to Optometrists. Uncertain whether participants' risk taking for vignettes is similar to real life
Traininga	Delivered identically to all participants in a format that could easily be accessed by large numbers of optometrists multiple times	Developed pragmatically, on the basis of knowledge/skills considered important for the task but also according to what we thought the NHS would ‘pay' for training if implemented
Health-care professionals	Real life participants, although volunteers (remunerated for their time but unlikely to cover full cost). Optometrists likely to be enthusiasts, interested in providing shared care—but also likely to be true if shared care were offered/commissioned
Health economics		Difficult to quantify investment required by an optometrists practice to set up doing shared care

If optometrists are found to be inferior to ophthalmologist one could almost certainly argue that this was because the training provided in the trial was inadequate. This possibility will also be investigated through the qualitative research.

This trial is of significant interest to the NHS as it addresses the question of whether community optometrists can make retreatment decisions for patients with quiescent nAMD to the same standard as hospital ophthalmologists.