Factors influencing accuracy of referral and the likelihood of false positive referral by optometrists in Bradford, United Kingdom.

AIMS: Levels of false positive referral to ophthalmology departments can be high. This study aimed to evaluate commonality between false positive referrals in order to find the factors which may influence referral accuracy.
METHODS: In 2007/08, a sample of 431 new Ophthalmology referrals from the catchment area of Bradford Royal Infirmary were retrospectively analysed.
RESULTS: The proportion of false positive referrals generated by optometrists decreases with experience at a rate of 6.2% per year since registration (p<0.0001). Community services which involved further investigation done by the optometrist before directly referring to the hospital were 2.7 times less likely to refer false positively than other referral formats (p=0.007). Male optometrists were about half as likely to generate a false positive referral than females (OR=0.51, p=0.008) and as multiple/corporate practices in the Bradford area employ less experienced and more female staff, independent practices generate about half the number of false positive referrals (OR=0.52, p=0.005).
CONCLUSIONS: Clinician experience has the greatest effect on referral accuracy although there is also a significant effect of gender with women tending to refer more false positives. This may be due to a different approach to patient care and possibly a greater sensitivity to litigation. The improved accuracy of community services (which often refer directly after further investigation) supports further growth of these schemes.

Introduction

Optometrists in the United Kingdom work in both Primary Care (High Street optical practice) and in Secondary Care. There is a large diversity of roles in Secondary Care, which mainly exist in National Health Service (NHS) hospitals, but also increasingly in private provider clinics. Only two studies have reported what proportion of referrals from optometrists in Primary Care for all pathologies were correct, and what proportion were false positive.1, 2 There is, however, a large body of research covering false positive referrals for glaucoma patients, which has a low incidence and thus relatively high false positive referral rates, with levels ranging from 29 to 68%.4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 The literature is in agreement that a significant number of false positive referrals are being generated by optometrists, and some authors have developed strategies to improve referral accuracy for possible glaucoma.16, 17, 18, 19 However, no study has so far investigated commonality between false positive referrals.

The aims of this study were to determine:

The levels of false positive referrals by optometrists and general practitioners (GPs) to the Hospital Eye Service (HES).

The factors that influence false positive referrals and the accuracy of referral. Factors considered for inclusion were patient age, gender and ethnicity, pathology referred for, referral format, final diagnosis, legibility, type of referring clinician, type of referring practice, referrer gender and years the referrer has been registered.

Fear of litigation, and an increase in modern diagnostic equipment in practice may increase the likelihood of Optometrists screening their patients for as many pathologies as possible whereas the decision to screen should take into account risk factors and the social cost.21, 22 Myint and colleagues found that lack of time to repeat measurements, or remuneration for doing such, as the most commonly reported barriers to effective glaucoma detection in the UK. Fewer optometrists in Scotland reported this, which is coincident with a study by Ang and colleagues investigating the effect on glaucoma referrals of the 2006 General Ophthalmic Services (GOS) contract in Scotland. The new contract replaced the refraction centred NHS sight test with a more comprehensive eye examination that does not necessarily include refraction and allowed funding for repeat appointments when necessary. The study found that after the introduction of the new contract there was a significant reduction in false positive referrals and a significant increase in true positive referrals.

Methods

The hospital records of a random sample of 431 (25% out of a total of approximately 1750) new referrals to Bradford Royal Infirmary (BRI) ophthalmology department, Bradford, England were retrospectively analysed. The referrals were identified from the hospital booking system by selecting the first 30% of new outpatient appointments booked each month between December 2007 and December 2008. The final sample was 5% less due to wrongly categorised patients (mostly ear/nose/throat patients from adjoining department), missing/in-use notes, missing/illegible referrals and other reasons. The presence of the following information was recorded from the referral: date, patient gender, patient age, patient ethnicity, referrer name, referral format, referrer address, type of referring clinician, legibility, any diagnosis given or alluded to and final diagnosis at the hospital (also classified into one of 18 groups based on the International Classification of Diseases-10 (ICD-10), World Health Organisation). The General Medical Council and General Optical Council publicly accessible registers were used to obtain the gender of the referrer and number of years since last registration. The legibility of handwritten referrals was graded by one person (CD) as; fully legible, illegible in part but understandable overall, or not legible enough to understand the reason for referral.

Definition of a false positive referral

In advance of data collection the authors attempted to fairly define what a false positive referral would be, although any single definition will have problems accurately representing the data. Previous studies have used many differing methods of classifying the accuracy of referrals.4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15

A false positive referral was identified by either of the following:

The ophthalmologist examined the patient, and subsequently discharged the patient due to the absence of significant ocular pathology. The ophthalmologist's decision to discharge must not have been solely influenced by clinical techniques that were not currently commonly available to the referring practitioner.

The examining ophthalmologist diagnosed the patient with, or was suspicious of, pathology that was unrelated to the diagnosis given or implied by the referring practitioner. The ophthalmologist was happy that the pathology for which the patient was referred for was not present, with this decision not being influenced solely by clinical techniques that were not currently commonly available to the referring practitioner.

Fundoscopy, either direct or indirect using a non-contact lens, tonometry and central visual field screening are examples of techniques that should all be available in UK optometric practices according to College of Optometrists 2007 guidelines and previous literature.20, 25 Examples of techniques not currently widespread in UK optometric practices are pachymetry, gonioscopy, optical coherence tomography and fluorescein angiography. General Medical Practitioners, unless they have a special interest in Ophthalmology (none did in the present study), tended to only have case history, direct observation, fundoscopy, pupil assessment and visual acuity measurements at their disposal.

Data analysis

Data were analysed with a logistic regression model using Stata version 9.0 (Stat Corp., College Station, USA). Variables of interest were incorporated sequentially and their statistical significance was assessed. The predictor variables were; type of referring clinician, referrer gender, years the referrer has been registered, type of practice (i.e. independently owned local practice(s) or a nationwide company with multiple practices), pathology classification, format of referral, legibility, age of patient, gender of patient, and ethnicity of patient. The outcome variable was whether the referral had been defined as false positive. Significance of the two-level factors was determined by the ‘Z’-statistic, while the significance of a higher number of factors was tested using a likelihood ratio (χ2) test after dropping individual factors from the model. Factors with a p-value less than 0.1 were provisionally retained, whereas those above 0.1 were dropped. The final model adopted was the most parsimonious one that was felt to adequately explain the data, with the final level of significance set at p < 0.05. Factors were first considered in a multiple logistic regression model. When collinearity or missing data were a problem, univariate logistic regression analyses were used. The results have been described using odds ratios (OR).

Statement of ethics

The study complied with the tenets of the Declaration of Helsinki and ethical approval was given by the Bradford NHS Research Ethics Committee (Reference 07/Q1202/41). We certify that all applicable institutional and governmental regulations concerning the ethical use of human volunteers were followed during this research.

Results

The number (and percentage) of false positive referrals from all primary care clinicians are shown in Table 1. It was not possible to ascertain the practice type in 16 referrals (Diabetic Retinal Screening technician referrals were not included in practice-type analyses as it was not possible to ascertain any information about the screener, but pre-registration student referrals were). It was not possible to ascertain the optometrist gender in 20% (n = 85) of referrals (pre-registration student referrals were not included in the gender analyses as we were unaware of the gender of the supervisor who was legally responsible for the referral). Referrals were included from 112 optometrists and the maximum number of referrals from a single optometrist was 19 (median 2).

Table 1

The proportion of referrals defined as false positive from all sources. Numbers are lower for the subset with both gender and practice type as it was not possible to ascertain both metrics from all referrals.

Source of referral	False positive
General Medical Practitioner (n = 131)	4 (3%)
Optometrist (n = 366)	105 (29%)
Pre-registration optometrist (n = 26)	11 (42%)
Diabetic retinopathy screening service (n = 9)	0
Female optometrists (n = 122)	47 (39%)
Females in multiple practice (n = 82)	36 (44%)
Females in independent practice (n = 40)	11 (28%)
Male optometrists (n = 159)	36 (23%)
Males in multiple practice (n = 68)	21 (31%)
Males in independent practice (n = 91)	15 (16%)
Multiple optical practice (n = 206)	74 (36%)
Independent optical practice (n = 169)	38 (22%)

The relationship between optometrist false positive referrals and eye disease category is shown in Table 2. A multiple logistic regression was performed to detect the differences in false positive levels for all variables and associated significance values. As shown in Table 1, almost all GP referrals were not false positive and therefore these data were removed from the following multiple regression due to this almost perfect prediction and the resulting distortion of the remaining analyses. No significant effects were found for patient gender, patient age, patient ethnicity, legibility of referral or type of referring clinician (all p > 0.10). The proportion of false positive referrals generated by optometrists decreases with experience at a rate of 6.2% per year since registration (p < 0.0001). Direct referrals using community services (e.g. In this instance, the ‘cataract choice service’) were 2.7 times less likely to be false positive than other referral formats (p = 0.007). When compared individually, community service referral methods were 3.0 times less likely to result in a false positive referral than GOS18 referrals and 3.5 times less likely than a letter.

Table 2

Number and percentage of referrals from optometrists defined as false positive in each referral diagnosis category.

ICD-10 code	Diagnosis category	False positive referrals
H30-H32	Disorders of choroid (n = 4)	3 (75%)
H46-H48	Disorders of optic nerve and visual pathway (n = 7)	5 (71%)
H43-H45	Disorders of vitreous body and globe (n = 18)	12 (67%)
H35-H36.8	Other disorders of retina (n = 23)	11 (48%)
H53-H59	Visual disturbances and other disorders of eye and adnexa (n = 25)	11 (44%)
H33	Retinal detachments and breaks (n = 7)	3 (43%)
H10-H13	Disorders of conjunctiva (n = 3)	1 (33%)
H49-H53	Disorders of ocular muscles, binocular movement, binocular vision, amblyopia, accommodation and refraction (n = 7)	2 (29%)
H40.0	Glaucoma suspect (n = 79)	22 (28%)
H35.3.1	Age related macular degeneration (n = 25)	7 (28%)
H34	Retinal vascular occlusions (n = 14)	3 (21%)
H36.0	Diabetic retinopathy (n = 30)	6 (20%)
H25-H28	Disorders of lens (n = 93)	15 (16%)
H00-H06	Disorders of eyelid, lacrimal system and orbit (n = 13)	2 (15%)
H16-H19	Disorders of the cornea (n = 15)	2 (13%)
	None (n = 3)	0

Univariate analyses showed there was a significant difference between proportions of false positive referrals generated by optometrists in independent and multiple practice, with independent practices generating about half the number of false positives as multiple/corporate practices (OR = 0.52, p = 0.005, N = 376). When controlling for optometrist experience (years since registration) the sample size reduced to 294 as registration date was not ascertained for 82 referrals. This caused the effect of practice type on false positives to be reduced, with independents only generating 30% fewer false positive referrals and the difference became not significant (OR = 0.7, Z = −1.28, p = 0.20, N = 294).

Male optometrists were about half as likely to generate a false positive referral than females (OR = 0.51, Z = −2.64, p = 0.008, N = 305). Female optometrists were younger and a greater proportion worked in multiple practices. The effect still remained and was still significant when controlling for years since registration (OR = 0.57, Z = −2.18, p = 0.029, N = 298). However when including practice type and years registered as confounders the effect was not significant (OR = 0.62, Z = −1.79, p = 0.073, N = 294).

To allow for statistical analysis the diagnosis categories were further condensed into the five biggest groups, which were; disorders of lids/lashes, disorders of lens, glaucoma, visual disturbance/other and the remainder were grouped together. A just statistically significant link between false positives and diagnosis category (Likelihood Ratio, LR χ2 = 9.7, p = 0.046) was found. The rank order from lowest to highest false positive proportion was; (1) lens, (2) lid/lashes, (3) glaucoma, (4) everything else, and finally (5) visual disturbance/other, which had the most false positives.

Optometrist gender, practice type and years since registration

The combination of gender of optometrist, years since registration and type of practice was significantly linked with levels of false positive referrals (χ2 = 24.9, p < 0.0001) but further analysis was required to find the variable(s) driving this link. When using gender and years since registration as confounders the effect of practice type was not significant (p = 0.38) and could therefore be dropped from the analysis which still left significant effects for both years registered (Z = −3.9, p < 0.0001) and gender (Z = −2.02, p = 0.043). There was no significant interaction effect for years since registration and gender (p = 0.63) therefore the gender effect does not appear to be related to years since registration. In summary, years since registration is the most important variable that drives an increase in false positive referrals but gender is also significant.

Discussion

Diagnosis category

The most frequently referred diagnosis categories “Disorders of lens” and “Glaucoma suspect” have relatively low false positive rates of 16% and 28% respectively (Table 2). The fact they are encountered so frequently means Optometrists may be familiar and confident with these pathologies which is a likely explanation for the majority of referrals being appropriate. Referrals for disorders of the choroid, optic nerve, visual pathway and “other visual disturbances” are most frequently found to be false positive. These referrals are not encountered frequently in primary care, and therefore a lack of familiarity and confidence may contribute to the higher false positive rate. Alternatively, for example in the case of “Vitreous body and globe” the referrals may be potentially tentative in nature.

Referring clinician: GPs and optometrists

According to our ‘false positive’ definition, there were very few false positive referrals from GPs. This means that nearly all of these referrals were appropriate within the remit of that clinician's speciality, but this may say more about the limited ophthalmic clinical techniques available to GPs. If these patients had attended their optometrist it is possible that they may not have been referred, but investigated and managed appropriately. As shown in Table 3, 73 of 131 (56%) of the GP referrals agreed with the eventual diagnosis at the hospital, lower than the figure for optometrists (244/366, 67%), plus the majority of GP referrals are for lids/lashes/lacrimal disorders, which may be easier to diagnose. Similar relative figures have been found in earlier studies.2, 26 It should be noted that over the last 25 years, it would appear that the overall proportion of referrals from optometrists to secondary eye care has increased by 33% relative to GPs.

Table 3

The proportion of referrals from General Medical Practitioners and optometrists that were deemed “False Positive” according to the definition used in the study, compared to the proportion of referrals where the referral diagnosis agreed with the final diagnosis in the hospital.

Referrer	False positive	Diagnosis agrees with hospital
General Medical Practitioner (n = 131)	4 (3%)	73 (56%)
Optometrist (n = 366)	105 (29%)	244 (67%)

Referral format

The community service referral schemes, for example the cataract choice service which refers directly to the hospital, were significantly less likely to result in false positive referrals. It has been shown previously that direct cataract referral methods result in better quality referrals containing more relevant clinical information. This is further validation of community service referral schemes which tend to require additional diagnostic techniques to be performed or more protracted discussion with the patient and therefore usually require additional funding. Fewer false positive referrals reaching the hospital can result in an overall saving for the NHS dependent on the fee paid for community services, at least for glaucoma referrals.16, 17

Years since registration

Univariate analysis showed that a more experienced clinician was significantly less likely to generate a false positive referral. This seems logical, and improvement in diagnostic proficiency with increasing experience has been shown before across various medical disciplines,28, 29, 30 but it is a novel finding for optometry. If an inexperienced optometrist is unsure of a diagnosis, it would be unfair and potentially dangerous to criticise or discourage referral as there is a natural learning curve with experience in any profession. Of course to fully understand the situation it is desirable to quantify the numbers of patients that are not referred but should have been (false negative) i.e. we would hope that the decrease in false positive referrals with experience is not accompanied by an increase in false negatives.

Gender

There was a difference between referrals made by optometrists of different genders, with female optometrists generating significantly more referrals defined as false positive. The analysis also showed no interaction between years since registration and referrer gender, which means the gender effect is not significantly affected by experience however when including both practice type and years registered as confounders the effect was not significant (p = 0.073). It is possible that ‘years registered’ may be a less accurate reflection of experience for females as it does not account for career breaks or flexible working practices for maternity leave or childcare.

It has been previously documented that behaviour and decision making is different between male and female medical physicians, with the rates of screening,31, 32, 33, 34, 35 referral and the likelihood of initiating or intensifying treatment37, 38 being higher for female doctors in the majority of studies. This appears to hold true in primary care optometry although all the previous literature is on doctors with different study populations. Lurie and colleagues attempted to find a reason for these differences and discovered that female physicians felt more personal responsibility for ensuring that their patients received screening, and reported more comfort in performing Pap smears and breast examinations. Similarly, female physicians were seen as more caring by patients and wrote longer referral letters. Similar investigation is required in order to ascertain why this difference exists for optometrists, although a recent but not peer reviewed survey of 808 eyecare practitioners, of which 54% were women, found that women were more likely to agree with the statement that ‘I feel vulnerable to the possibility of litigation in relation to my work’ (31% of women agree vs 24% of men).

Type of optometric practice: multiple and independent

Referrals from independent practice result in fewer false positive referrals than those from multiples although further analysis suggested that this was because multiples tend to employ less experienced staff and, to a lesser extent, more female staff (Table 1). There has been some investigation into differences between high and low volume medical practices with mixed results. Curran and colleagues found higher-volume medical practices to be more likely to screen for prostate cancer, whereas Zyzanski et al. found high-volume practices less likely to schedule well (preventative) care. Zyzanski and colleagues also found high-volume physicians to naturally have 30% shorter visits and lower up-to-date rates of preventative services. The contradictions within the literature indicate that this inequality requires further investigation. The business structures of both modes of practice also differ and therefore the commercial pressures on clinicians may be different. There is no financial disincentive to generating an inaccurate referral. Indeed in an increasingly litigious modern society there is a potential financial incentive to refer whenever there is any element of doubt.

Limitations of the study

The definition of a false positive referral used by this study is limited in that it depends on what clinical techniques are commonly available to the referring practitioner. It is therefore naturally weighted in favour of those offering few ophthalmic techniques and may conceal some poor referrals. For example, any glaucoma referral by a GP would not have been deemed as false positive regardless of the outcome, because GPs do not have the equipment required to accurately make a diagnosis of glaucoma.

Most research into referral accuracy is performed retrospectively as an audit with associated limitations, but these have to be balanced against a prospective study potentially biasing referrer behaviour. For this type of study a retrospective audit with known limitations is probably of more value than a prospective study with unknown levels of bias. A limitation of the study was the lack of investigation of false negative decisions, i.e. patients who were not referred but should have been. False positive referrals result in wasted resources, wasted patient time and unnecessary psychological harm but lower levels must only be strived for without generating an increase in false negative decisions. This omission was due to logistical and financial difficulties of sourcing a significant sample of patients not referred, obtaining agreement for their re-assessment and arranging for an ophthalmological examination (the gold standard). The presence of local optometric community services (at the time of data collection; Cataract direct referral, cataract post-op service and glaucoma monitoring scheme) may affect referral behaviour of the clinicians in the catchment area and this will likely make the results not representative of all areas in the UK. Of course, this is a problem for many areas given the frequent changes and variability of provision of community ophthalmic services across England therefore these results will be more relevant to the areas with community services matching those present during data collection. Other limitations include that ‘years since registration’ was used as a measure of experience, yet this does not consider working practices (full time or part-time) and career breaks. It may have been preferable to have documented ‘full-time equivalent years of practice since qualification’ or similar instead of, or in addition to ‘Years since qualified’. Gender was not ascertainable in 85 cases as the name was illegible or not provided, but lack of referrer on optometric referral forms has unfortunately been found to be reasonably common at 19–31%.27, 45, 46

Overall there were 366 referrals from 112 optometrists. The median number of referrals from each optometrist was 2, therefore there was a small number of higher volume referrers, with the highest number of referrals from a single optometrist being 19. Although the higher volume referrers were of mixed gender, they tended to work in multiple/corporate practices, which is to be expected as these see a higher volume of patients. Having a small group of higher volume referrers gives potential for bias, therefore further work in other locations or on a nationwide basis is required to confirm whether the findings in the present study are relevant nationally.

Recommendations to improve false positive referral rates from optometrists

It may be possible to improve false positive rates from inexperienced or less confident optometrists by highlighting this feature to mentors in their early clinical careers and ensuring feedback is given from hospital clinicians. Targeted continuing education could be developed for these groups, or for optometrists who may have taken a career break (e.g. maternity leave). However, the best approach to reducing false positive referrals in the short term appears to be by using community service referral protocols and perhaps more simply by replacing the overarching GOS-18 referral form with more specific forms for the most common referrals of cataract, glaucoma and “other”.

Conclusions

False positive referrals were not found to be affected by patient gender, ethnicity, age, or legibility of referral (all p > 0.10). Clinician experience has the greatest effect on referral accuracy and this seems logical. As practitioners become more experienced, they appear to become more confident about their ability to monitor or manage patients rather than refer them. We assume that more experienced clinicians will not make more false negative decisions, but this needs to be determined. There is also a significant effect of gender on referral accuracy with women tending to refer more false positives and this may be due to a different approach to patient care and possibly a greater sensitivity to litigation. Finally, referrals using community (enhanced) services were over 3 times less likely to be false positive than GOS18 and letter referrals and this may be open to improvement by the production of referral forms designed specifically for glaucoma, cataract and “other” conditions.

Funding

This study was funded by the University of Bradford.

Conflicts of interest

The authors have no conflicts of interest to declare