Core outcome set for three ophthalmic conditions: a healthcare professional and patient consensus on core outcome sets for amblyopia, ocular motility and strabismus (COSAMS Study).

OBJECTIVES: Amblyopia, strabismus and ocular motility disorders are common conditions with significant impact on visual function, appearance and quality of life. We aimed to establish a core set of outcomes for each of the three conditions for use in clinical trials and routine clinical practice.
DESIGN: A comprehensive databank of outcomes was developed from a systematic review of the literature and a series of focus groups with healthcare professionals, researchers, patients and carers. The databank of outcomes was scored in a two-round Delphi Survey completed by two stakeholder groups: healthcare professionals/researchers and patients/carers. Results of the online Delphi were discussed at a face-to-face consensus meeting where the core outcome sets were finalised.
SETTING: UK-wide consultation. PARTICIPANTS: Researchers, clinicians, patients and carers. OUTCOME MEASURES: Core outcome sets.
RESULTS: For amblyopia, strabismus and ocular motility, 40/42/33 participants contributed to both rounds of the Delphi; six/nine/seven members attended consensus meetings, respectively. Consensus was reached on ten core outcomes for both amblyopia and ocular motility and nine for strabismus. All three conditions shared the core outcomes: adverse events, cost, vision-related quality of life and ocular alignment. The strabismus and ocular motility disorder core sets included, in addition, measuring the deviation, binocular vision, ocular movement, patient satisfaction and symptoms. The amblyopia set, distinct from the sets for the other two conditions, included best corrected distance and near visual acuity, spherical and cylindrical refraction, compliance and treatment-related and functionality/long-term impacts.
CONCLUSIONS: The study used robust consensus methods to develop a core outcome set for three ophthalmic conditions. Implementation of these core outcome sets in clinical trials and routine clinical practice will ensure that the outcomes being measured and reported are relevant to all stakeholders. This will enhance the relevance of study findings and enable comparison of results from different studies. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study followed robust methodology as guided by the Core Outcome Measures in Effectiveness Trials Initiative.

We targeted amblyopia, strabismus and ocular motility disorders, which are common ophthalmic conditions.

The study included key stakeholders including researchers, clinicians, patients and carers.

Attrition rates in the Delphi process were moderate but similar to other Core Outcome Set Studies.

Larger response numbers, including international participants, would be preferable for wider generalisability.

Introduction

Amblyopia (lazy eye) and strabismus (squint) occur in up to 5% of the general population.1 2 It is unknown how prevalent ocular motility disorders (abnormal eye movements) are in the general population. These conditions often present in children and can lead to long-term problems for children and young adults such as blurred vision, double vision, low esteem and even blindness if not treated.3 There are several approaches to the management of these conditions including occlusion, penalisation, spectacles, prisms, drugs, surgery, botulinum toxin, exercises, watchful waiting or a combination of two or more of the above.4–20

Interventional systematic reviews in this field of research have identified that there is considerable variation in the outcomes being measured and reported in primary research studies, which impacts on the ability to compare and synthesise outcome results across studies. Moreover, it was noted that there is a paucity of outcome data available on important patient outcomes such as quality of life, long-term outcome and the cost of treatment.4–20 To mitigate these issues and to increase the relevance of research, a core outcome set (COS) can be developed, which represents an agreed standardised set of outcomes that should be measured and reported in all studies for a specific area of health or healthcare. A search of the Core Outcome Measures in Effectiveness Trials (COMET) database revealed that there are several studies that have investigated important outcomes for the eyes and vision disease; examples include cataract,21 22 glaucoma23 and age-related macular degeneration,24 but none have specifically looked at amblyopia, strabismus or ocular motility disorders.25

The aim of this study was to develop COSs for use in clinical trials and routine practice for all intervention types for the treatment of amblyopia, strabismus and ocular motility disorders in children and adults that includes input from all stakeholders. While we aim to develop three separate COSs for each of the ophthalmic conditions, we anticipate that there could be considerable overlap in the importance of certain outcomes across these conditions. This is due to the fact that the three conditions often overlap and coexist in patients, are frequently targeted within the same research studies, and are usually managed by the same group of healthcare professionals.

Methods

The development of the COS Study involved three stages (figure 1): (1) the generation of a long list of outcomes, (2) a two- round online Delphi Survey and (3) face-to-face consensus meetings to discuss the results of the Delphi Survey and agree on the COS. The process considered the minimum standards for the design of a COS Study (Core Outcome Set—Standards for Development (COS-STAD)), which included careful consideration of the scope, stakeholders and consensus process.26

Figure 1

Study flowchart. (A amblyopia, S strabismus, OM ocular motility, HCP health care professional)

Outcome list generation

A databank of outcomes was generated from two sources: first, a systematic review of outcomes reported by researchers and clinicians in studies for the treatment of the conditions under evaluation and, second, using three separate focus groups (one for each condition) containing a mix of healthcare professionals, researchers, patients and carers. The detailed search strategy, methods and results for the systematic review have been published elsewhere.27 Outcomes from the systematic review and suggested outcomes from the recorded focus group meetings were extracted verbatim and grouped into suitable domains to facilitate easy classification. The final list was checked by experts in all three clinical conditions (SA-J and FJR), who also had the opportunity to use their clinical expertise to add additional outcomes to the list. In preparation for the Delphi Survey, clinical assessment outcomes used only by healthcare professionals were either separated out (not to be scored by patients) or combined into a simplified outcome for patients to score. Each outcome was written using plain language and feedback sought from four researchers from the health service research department, University of Liverpool, and a clinician from a local hospital on the acceptability and their understanding of the wording used. The databank of outcomes can be found in online supplemental table 1.

Online Delphi Survey

The databank of outcomes was used to populate an online Delphi Survey, which was administered using DelphiManager.28 Participants were invited from two key stakeholder groups. The first group consisted of healthcare professionals involved in the care for people with one of the three conditions or researchers working within this field. Invitations to participate were sent by email flyers to national professional organisations including the British and Irish Orthoptic Society, paediatric ophthalmology networks, and local groups linked with the University of Liverpool. The second group included patients or carers of patients affected by at least one of the three conditions of interest. Patients and carers were invited to participate into the survey using flyers distributed on the University of Liverpool noticeboards, newsletters (via the professional society), social media (Twitter) and ophthalmology departments in local hospitals including Aintree University Hospital, the Royal Liverpool University Hospital and Southport and Ormskirk hospitals. Through routine clinical practice, the study authors (SA-J and FJR) and healthcare professionals were also encouraged to distribute the patient survey links to their relevant patients if they showed an interest in the study.

Four surveys were set up, one for the healthcare professionals and researchers that contained the outcomes to be scored for all three conditions and three separate surveys containing only the outcomes relevant to patients and carers associated with each individual condition. The Delphi process was completed using two rounds (hereafter referred to R1 and R2). In each round, participants were presented with the list of outcomes and asked to score each outcome on how important it was to include in the COS, using a 9-point Likert Scale, with 1–3 labelled ‘not important’, 4–6 labelled ‘important but not critical’ and 7–9 labelled as ‘critically important’.29 Participants had the option to indicate ‘unable to score’ on any outcome they felt unable to score, and at the end of R1, participants were invited to submit additional outcomes they thought were missing from the list. These outcomes were reviewed by the study authors (SA-J and FJR), and any outcomes that represented a new relevant outcome were added to the list to be scored in R2. Irrespective of participant scoring, no outcomes were removed from the list between R1 and R2. During R2, participants were shown the distribution of scores for both stakeholder groups for each outcome along with their own score from R1 and asked to score the outcome again, using the same scale, taking this extra information into account.

Consensus meeting

Separate face-to-face consensus meetings were held at the University of Liverpool, in the UK, for each of the three conditions. Participants who either had an active role in the focus groups or completed both rounds of the Delphi Survey were invited to attend, although others with an interest in the project were invited to ensure each meeting had a balanced mix of participants from both stakeholder groups. In advance of the meeting, participants received a copy of their scores from the online survey (if appropriate) and a consensus matrix (online supplemental table 1) detailing the results of R1 and R2 by stakeholder group and which outcomes had reached a priori definition of consensus in, consensus out or no consensus (table 1). The consensus definition is similar to that used in other COS development studies.

Table 1

Definition of consensus

Consensus classification	Description	Definition of consensus
Consensus in	Consensus that the outcome should be included in the core set	≥70% of participants scoring the outcome as ‘7–9’ (critically important)
Consensus out	Consensus that the outcome should not be included in the core set	≥70% of participants scoring the outcome as ‘1–3’ (not important)
No consensus	Uncertainty about the importance of the outcomes	Anything else

The meeting for amblyopia was chaired by a non-clinical researcher with expertise in COS development methodology (JJK), while the meeting for strabismus and ocular motility was chaired by an investigator with a clinical background (SA-J).

In order to facilitate the discussion, all outcomes that had reached consensus ‘in’ after R2 for both stakeholder groups were presented first, followed by outcomes that reached consensus ‘in’ for only one stakeholder group. All outcomes that scored critical for inclusion for 50%–69% of the participants for either both or one of the stakeholder groups in R2 were presented next followed by all other outcomes that were scored by both stakeholder groups. Outcomes that were only scored by healthcare professionals and researchers were discussed last. Results for each outcome from the Delphi were shown to the participants with more time allocated to discussing outcomes where there was more uncertainty on whether the outcome should be included in the COS or not. Views for and against inclusion in the COS were sought by the meeting chair, who also ensured that participants had equal opportunity to comment prior to voting. Voting was undertaken anonymously using Poll Everywhere30 software, which was linked to mobile and tablet devices. The definition of consensus used in the Delphi Survey (table 1) was applied to the consensus meeting. The final COS was presented at the end of the meetings.

Study registration, ethics and reporting guidance

The study was prospectively registered with the COMET Initiative.31 Ethical approval was obtained from the University of Liverpool institutional research ethics committee for the focus groups, online survey and consensus meetings to be undertaken with healthcare professionals and patients (ref. numbers: 2063 and 2260). Informed consent was obtained from participants. The study is reported in line with the Core Outcome Set—Standards for Reporting guidance32 (online supplemental table 2).

Patient and public involvement

The study was supported by a patient advisory group, which provided input to this research study. The patient advisory group met on a regular basis for the duration of the study. Patients contributed to the design of the study and were involved at all stages of the survey and consensus meetings.

Results

A summary of the COS development process is shown in figure 1. The final COS contains ten, nine and ten outcomes across seven, six and seven domains for amblyopia, strabismus and ocular motility, respectively (tables 2–4). Ocular alignment, vision-related quality of life, adverse events and cost were common to all three conditions.

Development of the databank of outcomes

The systematic review and focus groups of healthcare professionals, researchers, patients and carers identified 31, 61 and 78 individual outcomes for amblyopia, strabismus and ocular motility, respectively. These were combined with a list of outcomes suggested by professional experts (SA-J and FJR) resulting in a total of 40, 70 and 106 outcomes for amblyopia, strabismus and ocular motility, respectively. The outcomes were classified into 12 domains (symptoms, visual function, refraction, oculomotor function, quality of life, treatment dependency, signs, investigations, long-term outcome, compliance, adverse events and cost), and outcomes that were not considered to be patient relevant were separated out or combined. As an example, ‘refractive status’, ‘spherical and cylindrical refraction’ and ‘median spherical equivalence’ were combined into a single outcome ‘refractive status’ for patients as they all have a similar meaning but are often referred to separately by healthcare professionals. Details of all outcomes including domain classification, combined outcomes and plain language descriptions of outcomes are provided in online supplemental table 1.

Online Delphi

Thirty-three healthcare professionals/researchers scored all outcomes for both R1 and R2 of the amblyopia component of the online survey, while 29 completed for strabismus and ocular motility. Three patients/carers completed both rounds for amblyopia, while nine completed both rounds for strabismus and five for ocular motility (figure 1). At the end of R1, five outcomes for amblyopia, 12 for strabismus and 23 for ocular motility reached consensus ‘in’ for both stakeholder groups. After a review of all additional outcomes suggested by participants in R1, three new outcomes were added to the strabismus survey in R2 (improvement in angle by a set amount (suggested by a healthcare professional) and immediate result post surgery and long-term discomfort from scar tissue (both suggested by a patient)).

On completion of R2, ten outcomes reached consensus ‘in’ for amblyopia across both stakeholder groups, while 17 and 32 outcomes reached the same criteria for strabismus and ocular motility, respectively.

Six, nine and seven voting participants attended the consensus meeting for amblyopia, strabismus and ocular motility, respectively, with an even balance of healthcare professional/researchers and patients present (figure 1).

Amblyopia

For amblyopia, future functionality/long-term impact and adverse events reached the consensus ‘in’ criteria for both stakeholder groups in both rounds of the Delphi and remained in the COS. Despite reaching consensus ‘in’ for both rounds of the Delphi for both stakeholder groups, intolerable diplopia and occlusion amblyopia (both adverse events) were not included in the final COS as it was felt that these could be captured under ‘adverse events’ and therefore were not critical for separate inclusion in the COS. Long-term outcome was also excluded following discussion as the group felt that there was currently no agreed set time for measuring long-term objective outcomes. Best corrected visual acuity (BCVA) and compliance marginally did not reach consensus ‘in’ during R2 of the Delphi but made the final COS after discussion. Following a discussion on the other visual function outcomes, near visual acuity was also added because it was noted that it was a good marker of early improvement for the treatment of amblyopia and important for children as it is important to their education. Refractive status reached consensus for both groups in R2, but following discussion, this was replaced by spherical and cylindrical refraction (scored only by healthcare professionals in the Delphi) because it was successfully argued that this was a more precise measurement of refractive status. The list of outcomes within the quality of life domain was discussed simultaneously. While this was not listed specifically as an outcome in the Delphi, participants agreed to include visual-related quality of life in the core set as it was felt that a generic health-related quality of life outcome was not sensitive enough. Psychological impact of treatment was scored only by healthcare professionals in the Delphi but reached consensus ‘in’ during R2. Following discussion led by a parent participant, the panel derived a new outcome to include treatment-related impact into the final COS in order to capture the effect of treatment, such as patching on children, which could be long-lasting. For both Delphi rounds, cost outcomes did not reach consensus ‘in’ by either stakeholder groups; however, the consensus panellists successfully advocated for its inclusion as a core outcome as cost outcome data are vital information for contemporary health systems.

Strabismus

For strabismus, symptoms and patient satisfaction reached the consensus ‘in’ criteria for both stakeholder groups in both rounds of the Delphi and remained in the COS. BCVA also reached consensus ‘in’ for both rounds and groups in the Delphi although the consensus panel argued that any change in vision and/or loss of vision as an adverse event would be very significant and reportable as per standard healthcare safety procedures.33 At the consensus meeting, participants noted that strabismus interventions aim to change the strabismus angle and visual acuity (VA) should not be affected by the intervention unless an adverse event occurred. Thus, a change in VA would be captured within adverse events. On this basis, a decision was taken to exclude VA from the core set. All remaining visual function outcomes were discussed simultaneously, and while the postop diplopia test reached consensus ‘in’ during the Delphi exercise, the consensus panel voted in favour of including binocular vision as core, as it was more representative of a group of visual-function-related outcomes. Oculomotor function outcomes were discussed simultaneously, and it was highlighted that ocular movement was critical to be reported in all strabismus types as a change caused by the intervention would be significant. Quantifying both the ocular alignment and deviation was also seen to be critical in the context of any strabismus type, and these were included as core outcomes. Visual-related quality of life, adverse events and cost were also included in the COS for reasons discussed for amblyopia.

Ocular motility

The discussions for ocular motility closely followed those of strabismus with the addition of clinical signs being added as an extra core outcome. Similar to adverse events, this outcome was a catch all for all clinical signs, which were scored individually in the Delphi exercise. This strategy was seen favourably by the meeting participants as many subconditions of ocular motility have specific signs associated with them. One example for this is corneal exposure in the ocular motility condition of thyroid eye disease but which is not relevant in other ocular motility disorders.

Discussion

This study has developed a set of core outcomes for the treatment of three ophthalmic conditions using a robust consensus process involving healthcare professionals, researchers, patients and carers. Consensus was reached on what should be measured in each of the three COS. They consisted of nine to ten outcomes distributed across six to seven domains to cover all important aspects related to treatment (objective clinical signs, adverse events, subjective or patient-reported outcomes and health economics). While these three COSs were developed independently, there are some parallels, and as a consequence, four outcomes (ocular alignment, vision-related quality of life, adverse events and cost) were common to all three conditions. The amblyopia COS captures the condition’s unique features by reporting additionally on ‘best corrected visual acuity’, ‘near visual acuity’, ‘compliance’, ‘spherical and cylindrical refraction’, ‘treatment-related impact’ and ‘future functionality/long-term impact’, keeping in mind that children are the predominantly affected population. The COS for strabismus and ocular motility disorders, on the other hand, include ‘binocular vision’, ‘ocular movement’, ‘measuring the deviation’, ‘symptoms’ and ‘patient satisfaction’. The ocular motility disorder COS was unique in additionally reporting ‘clinical signs’ related to the relevant conditions.

We recommend that, as a minimum, these core outcomes are used in future trials of interventions to treat amblyopia, strabismus and ocular motility disorders. We also advocate that these outcomes are recorded in routine clinical practice to ensure that the outcome data collected re meaningful and important.

A strength of this study is that it was prospectively registered with the COMET Initiative and it was developed using the COS-STAD recommendations.26 Engagement with patient participants was particularly challenging, and we sought to improve patient input by offering paper copies of the Delphi Survey with prepaid return envelopes in orthoptic clinics, although this was later abandoned after a number of sessions when there was no uptake. As a consequence of a relatively low number of patients responding to the Delphi and attrition between the two rounds, there was concern that consensus was not being achieved at the end of the final round given that the number of outcomes reaching consensus for both stakeholder groups had increased dramatically from R1. While measures were taken to ensure survey participation and retention were maximised (including sending reminders and extending deadlines for completion), it was felt that after several months of keeping the survey open, our efforts became futile. In order to compensate for this, we ensured that the consensus meetings where the final COS were ratified contained a good balance of healthcare professionals and patients. The main limitation of this study was that the consensus process was based using only participants in the UK. However, as a starting point, we have reason to believe that this COS could also be useful in other countries and settings.

Further consensus work is needed to refine and establish the best measurement instruments and time points for when to measure these core outcomes. To assist this process, the systematic review for generating the databank of outcomes also recorded the measurement instruments and timings associated with each outcome.27 Moreover, for some outcomes, the metric (eg, change from baseline or interocular difference (IOD) of BCVA) and method of aggregation (eg, mean or median)22 would need to be determined. Defining success criteria (eg, 8 or 10 dioptres from orthophoria for alignment, for distance and/or near) is another aspect of outcome refining and definition to be done by further work. The generalisability of the COS also needs to be reviewed in healthcare settings outside the UK. While the review of outcomes identified studies from around the world (with prominence from the USA, the UK, China and various European countries), the formal consensus process was undertaken using only participants from the UK, and those attending the consensus meeting were mostly localised to the North West of England.

There are few reported COSs in the literature that relate to the three conditions in this study. Chiu et al recommended four outcomes for reporting results of surgery for intermittent exotropia.34 Their study aimed to explore the extent of standardisation of outcomes reported in surgical studies for the condition. However, the study was limited by the extent of literature review for this specific condition (10-year literature search period) and lack of external consensus. A short narrative review of outcome measurements for size of deviation showed considerable variability across the tests available and the recommendations for their use. They suggested four core outcomes for all future studies: alignment, near stereoacuity, control score and quality of life score. If assigning near stereoacuity and control score to ‘binocular vision’, their outcomes map to those reported in our COS for strabismus.

Moreover, two recently published studies attempted to define criteria for success in treatment, one for amblyopia and the other for strabismus surgery, which could be considered complementary to the COS and not alternatives because they essentially give more definitions of primary outcomes rather than suggesting a set of specific outcomes to be measured in research.

A report was published by Shoshany et al35 stated that the IRIS measures for amblyopia developed by the American Academy of Ophthalmology (IRIS7,36 modified in 2019 to IRIS5035) provide uniform criteria for defining amblyopia treatment success. Treatment was defined as ‘successful’ if corrected IOD was less than 0.23 logMAR 12–18 months after first diagnosis. IRIS50 considers improvement in VA, which may be relevant to patients who had dense amblyopia at baseline but nevertheless improved. Thus, IRIS50 may be a more practical reporting measure than IRIS7. In general, Shoshany et al propose that these measures will allow more efficient reporting of quality metrics and rapid and objective assessment of new amblyopia treatments.35

In addition, a study aiming to define successful outcomes for strabismus surgery was published by Serafino et al.37 Although this study did not state an intention to develop a COS, there are a lot of similarities and overlap in the objectives and methodology used. A Delphi process was used to identify areas of consensus and disagreement among experts for the definition of success post strabismus surgery. The panel of experts in their study represented wide international geographical areas and included experts who were chosen based on their peer-reviewed publications, participation at international meetings and surgical experience. The study concluded the following: they achieved consensus on which strabismus types need their separate set of outcome criteria. They also identified the importance of ‘stereopsis’ and ‘the range of single vision’ for inclusion of success definition in some strabismus types, which interestingly could be mapped to ‘binocular vision’ in our strabismus COS. The study also found that there was no consensus on the length of time after surgery for determination of success, magnitude of deviation consistent with success and whether manifest or latent deviation should be considered to define success, which the review of our study27 has also found and which we are advocating to define, by future work. Differences from our study are that their survey did not involve scoring of outcomes, there was no systematic search of literature of reported outcomes prior to survey construction and patients or service users were not consulted in the process.

A search in the COMET Initiative database in April 2020 did not reveal registration of any further additions of similar studies in the database. It is advantageous to register COS studies in the database to facilitate collaborative work of similar scope and to avoid duplication of efforts and waste of research.

Conclusion

The three COSs developed from this study can be applied to future trials and routine data collection for all intervention types to treat the three ophthalmic conditions considered. Their use will allow the comparison of outcome data to be made across studies and to better inform treatment decisions. Future work will include seeking consensus on how these outcomes should be measured and to evaluate the acceptability of the current COS to patients and professionals in other countries, particularly where healthcare systems differ from the UK.

Table 2

Final COS for amblyopia

Domain	Outcome
Visual function	1. Best corrected visual acuity
	2. Near visual acuity
Refractive status	3. Spherical and cylindrical refraction
Oculomotor function	4. Ocular alignment (is there an ocular deviation?)
Quality of life	5. Vision-related quality of life (for example, activities of daily living)
	6. Treatment-related impact (for example, negative effects of patching on children during treatment)
	7. Future functionality / long-term impact
Compliance	8. Compliance
Adverse events	9. Any adverse events (for example, intolerable diplopia, occlusion amblyopia)
Cost	10. Cost (for example, cost to services, families, and individuals)

Table 3

Final core outcome set for strabismus

Domain	Outcome
Symptoms	1. Symptoms (eg, diplopia and appearance of the strabismus)
Visual function	2. Binocular vision (eg, stereoacuity and binocular single vision)
Oculomotor function	3. Ocular alignment (are the eyes straight?)
	4. Measurement of deviation (what is the amount of deviation?)
	5. Ocular movement (specifically incomitance, latent nystagmus, DVD and A&V pattern)
Quality of life	6. Vision-related quality of life: psychosocial aspects (such as self-esteem, confidence, behaviour and social interaction) and functional aspects (such as activities of daily living)
	7. Patient satisfaction
Adverse events	8. Any adverse events (eg, intolerable diplopia, recurrence of the deviation and overcorrection or undercorrection of the deviation)
Cost	9. Cost (eg, cost to services, families, and individuals)

DVD - dissociated vertical deviation, A&V - A and V pattern

Table 4

Final core outcome set for ocular motility disorders

Domain	Outcome
Symptoms	1. Symptoms (eg, diplopia and appearance of the eye deviation)
Visual function	2. Binocular vision (eg, stereoacuity, field of binocular single vision and postop diplopia test)
Oculomotor function	3. Ocular alignment (are the eyes straight?)
	4. Measurement of deviation (what is the amount of deviation?)
	5. Ocular movement (specifically incomitance, latent nystagmus, DVD and A&V pattern)
Quality of life	6. Vision-related quality of life: psychosocial aspects (such as self-esteem, confidence, behaviour and social interaction) and functional aspects (such as activities of daily living)
	7. Patient satisfaction
Adverse events	8. Any adverse events (eg, intolerable diplopia, recurrence of the deviation and adverse effects from patches or prisms)
Cost	9. Cost (eg, cost to services, families and individuals)
Clinical signs	10. Clinical signs (eg, corneal exposure, proptosis/exophthalmos and enophthalmos)

DVD - dissociated vertical deviation, A&V - A and V pattern