Using Electronic Clinical Decision Support to Examine Vision Rehabilitation Referrals and Practice Guidelines in Ophthalmology.

Purpose: To examine ophthalmologist use of an electronic health record (EHR)-based clinical decision support system (CDSS) to facilitate low vision rehabilitation (LVR) care referral.
Methods: The CDSS alert was designed to appear when best documented visual acuity was <20/40 or hemianopia or quadrantanopia diagnosis was identified during an ophthalmology encounter from November 6, 2017, to April 5, 2019. Fifteen ophthalmologists representing eight subspecialties from an academic medical center were required to respond to the referral recommendation (order, don't order). LVR referral rates and ophthalmologist user experience were assessed. Encounter characteristics associated with LVR referrals were explored using multilevel logistic regression analysis.
Results: The alert appeared for 3625 (8.9%) of 40,931 eligible encounters. The referral rate was 14.8% (535/3625). Of the 3413 encounters that met the visual acuity criterion only, patients who were worse than 20/60 were more likely to be referred, and 32.4% of referred patients were between 20/40 and 20/60. Primary reasons for deferring referrals included active medical or surgical treatment, refractive-related issues, and previous connection to LVR services. Eleven of the 13 ophthalmologists agreed that the alert was useful in identifying candidates for LVR services. Conclusions: A CDSS for patient identification and referral offers an acceptable mechanism to apply practice guidelines and prompt ophthalmologists to facilitate LVR care. Further study is warranted to optimize ophthalmologist user experience while refining alert criteria beyond visual acuity. Translational Relevance: The CDSS provides the framework for multi-center research to assess the development of pragmatic algorithms and standards for facilitating LVR care.

Introduction

When vision loss affects the performance of everyday activities, the Preferred Practice Pattern from the American Academy of Ophthalmology recommends consideration of low vision rehabilitation (LVR) services in addition to ongoing medical or surgical management. Although not restorative, LVR has been shown to be effective in nearly half of patients accessing outpatient services and lowers risk of depression., Despite LVR effectiveness and the existence of practice guidelines, under-referral by ophthalmologists and under-utilization by patients remain common.– As most patients are unaware of the existence of LVR services, connection to care relies heavily on the recommendation and referral from the ophthalmologist. The purpose of the study was to examine ophthalmologist use and response to an electronic health record (EHR)-based clinical decision support system (CDSS) with the aims of identifying patients potentially in need of LVR and facilitating referral for care.

One contributing factor to under-referral may be the lack of systematic practices in identifying patients who may benefit from LVR. Eye care differs from other medical disciplines such as orthopedics or neurosurgery where more commonly an acute event (e.g., fall with hip fracture, stroke) may prompt referral and integration of rehabilitation care (e.g., physical, speech language therapy). Comparatively, vision loss is often gradual and ophthalmic treatment chronic; thus, referral may be easily overlooked. Few studies have examined referrals to LVR, as the process is not consistently documented in the encounter notes. The constructs of referral and utilization have also been mischaracterized enough to be misunderstood; identification of the former is essential to positively modify the latter. Ultimately, any successful initiative should result in improved utilization of LVR; however, systematic approaches have yet to be applied to improve compliance with referral guidelines.,,,

The definition of low vision is multifaceted and may also contribute to the variability in determining whether someone has low vision and, subsequently, under-referral. The definition of low vision includes (1) measures of impairment, such as visual acuity and visual field; and (2) a loss in visual ability, defined as the ability to perform everyday activities that depend on vision (e.g., reading, driving, recognizing faces). As visual ability varies among people and may depend, for example, on contrast sensitivity loss, comorbidities, age, and social support, it is difficult to predict when someone should be referred for LVR, let alone who is most likely to benefit from treatment. In the absence of a single metric to characterize LV and predict those who would benefit, surrogate measures such as visual acuity and visual field are often used to define criteria and calculate incidence and prevalence statistics.–

CDSSs such as alerts and order sets can integrate algorithms into the EHR to aid in clinical decision making and optimize healthcare delivery, including LVR referral.– Deployment of EHR-based CDSS has been touted as a cost-effective and sustainable solution for improving quality of care in other specialties, including respiratory medicine, diabetes, and chronic kidney disease. Given the high adoption rate of EHRs, the use of structured data fields in ophthalmology, and an interest in CDSSs in glaucoma care, CDSSs may hold promise in standardizing practices in eye care. We therefore set out to develop and test a CDSS for ophthalmologists to identify patients with low vision and refer them for care. Previously published details on the development and testing of CDSSs have revealed reliability in identifying patients with low vision. We report here on ophthalmologist use and response to the CDSS.

Methods

The Johns Hopkins School of Medicine Institutional Review Board determined the project was exempt from review.

Project Participants and Setting

Participation was requested from each subspecialty division at the Johns Hopkins Wilmer Eye Institute. A total of 15 ophthalmologist users participated, representing six practice locations (main hospital and satellite clinics) and eight subspecialties (comprehensive ophthalmology, cornea, glaucoma, neuro-ophthalmology, oculoplastics, pediatrics and adult strabismus, retina, and uveitis). Following a user-centered design approach, group and individual meetings were held during CDSS development, prior to project launching, during implementation, and upon completion to understand user needs, communicate project details, and obtain user feedback. Quarterly reports presenting individual and summary data on alert firing and referral ordering frequencies were provided to ophthalmologists.

Electronic Alert Design for Patient Identification and Low Vision Referral

The electronic alert was created on the institutional EHR (Epic Systems, Verona, WI) and was designed to identify patients meeting LVR referral consideration based on the American Academy of Ophthalmology LVR Preferred Practice Pattern, generate an alert notifying the ophthalmologists when referral criteria were met, and document responses. Response to the alert was mandatory, and the EHR encounter could not be signed until a response to the alert was selected. Usual care practices were followed by ophthalmologists regarding discussion with patients, notifying the patient care team to schedule LVR clinic visits, etc. The CDSS was active from November 6, 2017, to April 5, 2019.

The consensus for the final alert design featured a set of firing criteria, suppression criteria, and a dismissible alert at the opening of the EHR encounter; when a mandatory alert appeared, one of three response options required selection (Fig. 1) prior to closing the encounter. The firing criteria included best documented visual acuity worse than 20/40 in the better eye or a diagnosis related to hemianopia or quadrantanopia (International Classification of Disease, Tenth Revision [ICD-10] codes of H53.47 or H63.46). Suppression criteria included (1) patient age younger than 5 years old; (2) ophthalmic surgery ordered in the next 3 months or performed in the past 3 months; (3) prior LVR clinic visit(s) within the same institute in the past 12 months; and (4) prior alert actions that suppress alert firing for the current encounter. Alert response options and the subsequent suppression conditions included order (no suppression); don't order–under low vision care (365 days alert suppression); and don't order–other reasons (no suppression) with a free text field for comments.

Figure 1.

Low vision rehabilitation referral alert. The user interface featured a dismissible notification at chart opening as a reminder that the low vision best practice advisory was active for the current encounter. It also included a mandatory alert prior to signing the encounter that displayed three physician response options: “order,” “don't order–under low vision care,” or “don't order–other reasons.” A free-text comment field was available when physicians selected “don't order–other reasons.”

Alert Firing and Response Assessment

Eligible encounters were defined as ophthalmology office encounters that did not meet any of the suppression criteria and were not associated with false-positive or false-negative alert firing. The alert firing rate was calculated as the proportion of encounters with the alert firing compared to all eligible encounters. The suppression rate was calculated as the proportion of encounters meeting any of the suppression criteria compared to all ophthalmology office encounters. We report on the proportions of ophthalmologist responses with “order” and “don't order” (including response options of “defer–prior VR,” “consider at next visit,” “consider in 3 months,” “consider in 1 year,” “patient refuses,” and “not recommended” from Phase I, and response options of “don't order–under low vision care” and “don't order–other reasons” from Phases II and III).

Ophthalmologist User Experience Survey

Upon project completion, ophthalmologists were surveyed regarding their experience with the system. Users were asked to choose from strongly disagree, disagree, neutral, agree, or strongly agree as a response to the following statements: (1) The alert is useful in identifying candidates for low vision rehabilitation services. (2) The alert changed my management of referral recommendations for patients with vision impairment. Ophthalmologist perspectives on the visual acuity threshold for alert firing and other CDSS criteria that should be considered were also assessed.

Statistical Analysis

Descriptive statistics were calculated for proportions of eligible encounters, alert firing rate, suppression rate, distribution of response options to the alert, and ophthalmologist survey results. Ophthalmologist referral order rate by month was assessed using the χ2 test. The overall referral order patterns were compared by ophthalmologist gender. Patient and encounter characteristics, including alert firing criteria (visual acuity, ICD-10, or both), age, gender (female, male), race (white, black, Asian, other), ethnicity (Hispanic, non-Hispanic), clinic location (main hospital, satellite clinics), and visual acuity categories (≥20/40, <20/40 and ≥20/60, <20/60 and >20/200, ≤20/200 and >20/500, and ≤20/500), were extracted from the EHR and compared between encounters where the ophthalmologists did and did not order LVR referral. Patient encounter characteristics associated with ophthalmologist referral orders were explored using multilevel logistic regression models among those that met visual acuity criteria. Because the encounter-level referral response may not be independent within the same patient or the same ophthalmologist due to the patient-level characteristics and ophthalmologist referral practices, ophthalmologist and patient-level clustering effects on encounter-level referral responses were accounted for using mixed-effects logistic regression modeling. All analyses were conducted using STATA 15 (Stata Corp., College Station, TX).

Results

A total of 60,860 ophthalmology office visit encounters representing 27,339 patients occurred across the 15 participating physicians during the 17-month project period. Five of 15 ophthalmologists (33%) were female, and seven of 13 ophthalmologists (54%) had been in practice for more than 10 years, ranging from 3 to 37 years (Supplementary Table S1). There were 40,931 (67.3%) eligible encounters, 293 (0.5%) were associated with false-positive or false-negative firing status, and the remaining 19,636 (32.3%) were suppressed (Supplementary Fig. S1).

Alert Firing Rates and Physician Responses

The alert appeared for 3625 (8.9%) of the 40,931 eligible encounters. Among them, 3413 (94.2%) met the visual acuity criterion, 169 (4.7%) met the ICD-10 firing criterion, and 43 (1.2%) met both visual acuity and ICD-10 firing criteria. Individual ophthalmologist alert firing rates ranged from 4.5% to 18.4% (Table 1).

Table 1.

Individual Ophthalmologist and Overall Group Alert Firing Rate and Response Option Distributions

			Response Options,b n (%)
	Eligible	Alert		Don't
	Encounters, n	Fired,a n (%)	Order	Order
Overall	40,931	3625 (8.9)	535 (14.8)	3090 (85.2)
Individual ophthalmologist user
Comprehensive ophthalmology physician 1	4010	207 (5.2)	41 (19.8)	166 (80.2)
Comprehensive ophthalmology physician 2	2183	126 (5.8)	7 (5.6)	119 (94.4)
Cornea physician 1	2058	230 (11.2)	21 (9.1)	209 (90.9)
Cornea physician 2	2466	165 (6.7)	17 (10.3)	148 (89.7)
Glaucoma physician 1	1874	102 (5.4)	8 (7.8)	94 (92.2)
Glaucoma physician 2	3239	255 (7.9)	60 (23.5)	195 (76.5)
Glaucoma physician 3	3082	450 (14.6)	117 (26.0)	333 (74.0)
Neuro-ophthalmology physician 1	3604	317 (8.8)	62 (19.6)	255 (80.4)
Neuro-ophthalmology physician 2	3042	287 (9.4)	75 (26.1)	212 (73.9)
Oculoplastics physician 1	2551	115 (4.5)	16 (13.9)	99 (86.1)
Pediatrics and adult strabismus physician 1	1279	57 (4.7)	10 (17.5)	47 (82.5)
Pediatrics and adult strabismus physician 2	3298	294 (8.9)	16 (5.4)	278 (94.6)
Retina physician 1	1594	293 (18.4)	3 (1.0)	290 (99.0)
Retina physician 2	3703	567 (15.3)	69 (12.2)	498 (87.8)
Uveitis physician 1	2948	160 (5.4)	13 (8.1)	147 (91.9)
Ophthalmologist gender
Female	13,438	1236 (9.2)	184 (14.9)	1052 (85.1)
Male	27,493	2389 (8.7)	351 (14.7)	2038 (85.3)

Alert firing rate was calculated as the number of encounters with alert firing divided by number of eligible encounters.

Response option rates were calculated as the number of encounters with user responses in categories of “order” and “don't order” (including “don't order–under low vision care” and “don't order–other reasons”) divided by number of encounters with alert firing.

Overall, ophthalmologists responded “order” to the alert in 535 (14.8%) encounters and “don't order” in 3090 (85.2%) encounters. Among the 3090 encounters, deferred referral comments were noted in 905 (29.3%). Ophthalmologists from glaucoma, neuro-ophthalmology, pediatrics and adult strabismus, and uveitis subspecialties were more likely to document comments, whereas those from comprehensive ophthalmology, oculoplastics, and retina subspecialties were less likely to document comments. The most common reasons for deferring referral included under-active medical or surgical treatment (42%), refractive-related issues (16%), and previous connection to low vision rehabilitation services (16%). Other reasons included adequate functionality, comorbidities, patient refuses or defers, geographical inaccessibility, services provided through other facilities, and acute condition with potentially reversible vision. Ophthalmologists’ referral order rates ranged from 1.0% to 26.1% (Table 1). No difference in referral rate was observed between female and male ophthalmologists (14.9% vs. 14.7%; P = 0.88). Over the 17-month period, little fluctuation was observed with the distributions of referral order patterns (P = 0.13, χ2 test) (Fig. 2).

Figure 2.

Distributions of ophthalmologist referral order rates over time.

Patient Encounter Characteristics by Referral Order Status

Overall, LVR referrals were most likely to be ordered in encounters meeting both visual acuity and ICD-10 criteria (21/43, 48.8%), followed by those only meeting ICD-10 criterion (41/169, 24.3%). Distributions of patient encounter characteristics including age, race, and visual acuity differed by LVR referral order status, with LVR referral less likely in encounters with patients between 5 and 20 years of age, of white race, or with visual acuity at least 20/60 (Table 2). No statistical relationship was found between referral and patient gender, ethnicity, or clinic location on an encounter level.

Table 2.

Patient and Encounter Characteristics by Alert Response

	Encounters with Alerts (N = 3625)
	Referral Ordered (N = 535), n (%)	Referral Not Ordereda (N = 3090), n (%)	P
Alert reason			< 0 .001
Only visual acuity criterion met	473 (88.4)	2940 (95.2)
Only ICD-10 criterion met	41 (7.7)	128 (4.1)
Both visual acuity and ICD-10 criteria met	21 (3.9)	22 (0.7)
Age (y), mean ± SD	65 ± 22	63 ± 25	0 .04
Age groups (y)			0 .001
≥5, <20	31 (5.8)	318 (10.3)
≥20, <40	45 (8.4)	275 (8.9)
≥40, <65	136 (25.4)	613 (19.8)
≥65, <80	155 (29.0)	984 (31.8)
≥80	168 (31.4)	900 (29.1)
Gender			0.23
Female	281 (52.5)	1709 (55.3)
Male	254 (47.5)	1381 (44.7)
Raceb			0 .002
White	288 (54.4)	1899 (62.4)
Black	178 (33.7)	788 (25.9)
Asian	29 (5.5)	162 (5.3)
Other	34 (6.4)	193 (6.3)
Ethnicityb			0.47
Non-Hispanic	500 (96.7)	2885 (97.3)
Hispanic	17 (3.3)	81 (2.7)
Clinic location			0.16
Satellite clinic	308 (57.6)	1677 (54.3)
Main hospital	227 (42.4)	1413 (45.7)
Visual acuity categoryb			< 0 .001
≥20/40	40 (7.5)	127 (4.1)
<20/40, ≥20/60	161 (30.2)	1388 (44.9)
<20/60, >20/200	195 (36.2)	994 (32.2)
≤20/200, >20/500	65 (12.2)	265 (8.6)
≤20/500	73 (13.7)	315 (10.2)

Bold font indicates statistically significant at P < 0.05 level.

Referral not ordered included alert responses of “don't order–under low vision care” (including response option of “defer–prior VR” from Phase I) and “don't order–other reasons” (including response options of “consider at next visit,” “consider in 3 months,” “consider in 1 year,” “patient refuses,” and “not recommended” from Phase I).

Race, ethnicity, and visual acuity information was missing for 54 (1.5%), 142 (3.9%), and 2 (0.06%) encounters, respectively.

Of the 3413 encounters that met the visual acuity alert criterion only, LVR referral was ordered for 473 (13.9%). Distributions of age group, race, and visual acuity categories differed significantly between encounters with LVR referral ordered and not ordered (Table 3). Notably, 153 (32.4%) encounters with LVR referral had better-eye visual acuity between 20/40 and 20/60. In the regression model accounting for the patient and ophthalmologist clustering effects and adjusted for patient encounter demographics, clinic location, and visual acuity categories, no associations were found between referral response and patient encounter demographics or clinic location. Encounters with visual acuity categories worse than 20/60 were more likely to be referred compared to those with <20/40 and ≥20/60 visual acuity (odds ratios ranged between 1.88 and 2.75).

Table 3.

Patient and Encounter Characteristics by Alert Response: Encounters Meeting Visual Acuity Criterion Only

	Descriptive Statistics (N = 3413)			Regression Analysisa Outcome: Referral Ordered
	Referral Ordered (n = 473)	Referral Not Ordered b (n = 2940)	P	Odds Ratio	95% CI
Age (y), mean ± SD	66 ± 22	63 ± 25	0 .048	—	—
Age group (y)			0 .002
≥5, <20	28 (5.9)	302 (10.3)		0.57	0.28–1.14
≥20, <40	42 (8.9)	262 (8.9)		Reference	—
≥40, <65	112 (23.7)	557 (19.0)		1.07	0.66–1.72
≥65, <80	131 (27.7)	935 (31.8)		0.83	0.52–1.33
≥80	160 (33.8)	884 (30.1)		1.37	0.85–2.21
Gender, n (%)			0.32
Female	254 (53.7)	1653 (56.2)		Reference	—
Male	219 (46.3)	1289 (43.8)		1.03	0.80–1.31
Race, n (%)c			0 .001
White	250 (53.5)	1806 (62.4)		Reference	—
Black	160 (34.3)	752 (26.0)		1.27	0.95–1.69
Asian	27 (5.8)	153 (5.3)		1.08	0.62–1.86
Other	30 (6.4)	182 (6.3)		1.08	0.61–1.88
Ethnicity, n (%)c			0.54
Non-Hispanic	441 (96.7)	2741 (97.3)		Reference	—
Hispanic	15 (3.3)	77 (2.7)		0.94	0.43–2.02
Clinic location, n (%)			0.19
Satellite clinic	199 (42.1)	1331 (45.3)		Reference	—
Main hospital	274 (57.9)	1609 (54.7)		1.20	0.88–1.62
Visual acuity category, n (%)			< 0 .001
<20/40, ≥20/60	153 (32.4)	1370 (46.6)		Reference	—
<20/60, >20/200	185 (39.1)	992 (33.7)		1.99	1.50–2.63
≤20/200, >20/500	65 (13.7)	264 (9.0)		2.75	1.83–4.13
≤20/500	70 (14.8)	314 (10.7)		1.88	1.27–2.76

Bold font indicates statistically significant at P < 0.05 level.

Multilevel logistic regression model was adjusted for age categories, sex, race, ethnicity, service location, and visual acuity categories, accounting for patient-level and physician-level clustering effect.

Alert not ordered included alert responses of “don't order–under low vision care” (including response option of “defer–prior VR” from Phase I) and “don't order–other reasons” (including response options of “consider at next visit,” “consider in 3 months,” “consider in 1 year,” “patient refuses,” and “not recommended” from Phase I).

Race and ethnicity information was missing for 53 (1.6%) and 139 (4.1%) encounters, respectively.

Ophthalmologist Survey Findings

Thirteen ophthalmologists completed the survey. For the statement “the alert is useful in identifying candidates for LVR services,” 11 (85%) agreed or strongly agreed and two (15%) were neutral. For the statement “the alert changed my management of referral recommendations for patients with vision impairment,” eight (62%) agreed or strongly agreed, two (15%) selected neutral, and three (23%) disagreed. Additionally, five (38%) physicians regarded best documented visual acuity worse than 20/40 as a reasonable alert firing criterion, whereas eight (62%) regarded it as unreasonable. When asked to suggest a preferred visual acuity criterion, four suggested visual acuity worse than 20/60, two suggested worse than 20/70, and another two suggested worse than 20/80. When asked about any other alert firing criteria that should be considered, physicians suggested visual field constriction, lost to LVR follow-up (greater than 2–3 years ago), and patients with monocular vision status. Referral rates were higher among physicians who agreed that the alert was useful in identifying candidates for LVR services compared with those who responded neutral (17% vs. 10%; P < 0.001).

Discussion

CDSSs in ophthalmology, specifically LVR, can employ clinical practice guidelines and augment LVR service referral. Using criteria paralleling the American Academy of Ophthalmology Vision Rehabilitation Preferred Practice Pattern, specifically visual acuity worse than 20/40 and diagnoses related to neurologic visual field loss (hemianopia and quadrantanopia), ophthalmologists recommended a LVR referral for 15% of eligible encounters (n = 40,931). Although it is not possible to retrospectively examine the effects of this CDSS on referral practices given EHR data and documentation practices, and because rates will vary based on CDSS criteria employed, this approach offers a pragmatic strategy to apply clinical guidelines in real time and prospectively measure interventions to maximize referral and ultimately utilization of LVR care.

Few prior comparative results are available. However, in a small study (n = 143) using the criterion of 20/60 visual acuity or worse in at least one eye (as compared to this study, which used best documented visual acuity worse than 20/40 in the better eye), the authors observed that 11% of patients had a notation in the EHR of a referral to LVR. As “low vision” is a multifaceted diagnosis, adaptation to vision loss is variable and LVR may be considered elective, it is not readily apparent what proportion of patients should be referred for, let alone utilize, LVR services. Free-text comments provided by ophthalmologists as to why referral was not ordered referenced reasonable considerations for deferral, including being under active medical or surgical treatment, refractive-related issues, previous or current connection to LVR, patient functioning well, and comorbidities (e.g., dementia, developmental delay).

Rates of LVR referral largely remained stable during the 17-month period, despite several alert modifications and version updates. On average, alerts appeared in 9% of encounters, which may have minimized potential “alert fatigue.” Overall, ophthalmologists found the alert useful in identifying patients with low vision and found that it changed their management of patient referral recommendations to LVR services. There was, however, less agreement regarding the visual acuity criteria that should be used to trigger the alert for referral consideration. The guidance by the current American Academy of Ophthalmology Preferred Practice Pattern is to “recognize” and “respond” by advising the patient that vision rehabilitation is an option when best-corrected visual acuity is worse than 20/40 in the better eye, when there is loss in contrast sensitivity or peripheral field, or when a scotoma or vision is interfering with the performance of everyday activities. Given that best-corrected visual acuity is often not assessed, raising the visual acuity alert threshold to 20/60 (best documented), as suggested by a narrow majority of physicians, may overlook nearly one-third of patients for whom a referral was ordered and were between 20/40 and 20/60. As evident in stroke or glaucoma-related vision loss, some visual tasks are affected with visual acuity better than 20/30. However, lowering the alert threshold may burden ophthalmologist users and further impede workflow, such as in cases of uncorrected refractive error (e.g., pediatric ophthalmology) or when best-corrected visual acuity is not available (e.g., retina, oculoplastics).,

Including additional metrics such as other visual field loss diagnoses and difficulty with performing everyday activities in the alert criteria may improve the CDSS sensitivity and specificity and better align with the Preferred Practice Pattern without increasing respondent burden. However, as visual field and ICD-10 diagnoses other than hemianopia and quadrantanopia were not used by participating ophthalmologists, including glaucoma providers, implementing such an approach would require changes in documentation practices. Regarding inclusion of patient-reported functional difficulties, a primary consideration for recommending LVR among glaucoma specialists and arguably the gold standard for referral, one report revealed that visual function (e.g., documentation of visual problems or ability) is the EHR element least likely to be documented by glaucoma specialists. As visual function is commonly documented in the EHR chief complaint field using free text and may not be specific to functional concerns (e.g., follow up for pressure check, injection visit), automating this process would require natural language processing or changes in documentation practices using standard data elements. Customizing the alert criteria to the subspecialty and even the ophthalmologist may offer improvements to the user experiences, although modifications to any of the parameters (e.g., visual acuity, diagnosis, suppression criteria) will likely affect comparative referral and subsequent utilization rates. Thus, with any CDSS, finding the optimal balance with consideration of the physician, the patient, and clinical guidelines is essential.

Referral recommendations were not associated with ophthalmologist gender or patient demographics or clinic location, which highlights that ophthalmologist referral preferences were not affected by patient age, gender, race, ethnicity, or site of service. However, encounters with worse visual acuity were more likely to be referred for LVR services. Given that all ophthalmologists were familiar with the American Academy of Ophthalmology Preferred Practice Pattern and participated in the development of the CDSS and alert criteria, variability in referral recommendations was more likely related to attitudes and beliefs about LVR rather than knowledge and awareness. Physicians who reported that the CDSS was useful were more likely to refer, which may indicate that the alert serves as a reminder as LVR services may easily be forgotten given time constraints and information overload during busy clinics.,

We have developed and tested a CDSS to begin to address the longstanding challenges of assessing and improving adherence to low vision clinical guidelines. The approach as designed and implemented meets many of the key indicators when defining a successful CDSS. Strengths of this work include early and sustained involvement from ophthalmologists from every subspecialty in both hospital and satellite clinical settings and an adaptive–integrative approach that uses the EHR and American Academy of Ophthalmology guidelines as a foundation for the CDSS criteria. Additionally, as this is one of the first applications of CDSSs in ophthalmology, testing in a single closed system allowed for a more careful audit and measurement of the effectiveness of a CDSS, providing a foundation to apply this approach and determine generalizability to LVR referral practices elsewhere. This CDSS can be used in future LVR studies to assess changes in metrics (e.g., referral, utilization, standardization of provider practices) as interventions are applied at the encounter, patient, and physician level. Limitations of this work include the inability to evaluate whether the alert changed physician referral behavior as there were no historic data. Also, the alert did not reference patient-reported visual ability concerns or diagnoses other than hemianopia and quadrantanopia, as the participating ophthalmologist users deemed that these changes would increase the project scope to include involvement of technicians or represent significant modification to usual care practices. Future work may include use of visual ability in the CDSS criteria as part of a multicenter collaborative effort to assess both the refinement of pragmatic physician- and patient-centered criteria and the generalizability of the CDSS in connecting patients to LVR care.