Validation of the New York University Langone Eye Test Application, a Smartphone-Based Visual Acuity Test.

Purpose: To validate and assess user satisfaction and usability of the New York University (NYU) Langone Eye Test application, a smartphone-based visual acuity (VA) test. Design: Mixed-methods cross-sectional cohort study. Participants: Two hundred forty-four eyes of 125 participants were included. All participants were adults 18 years of age or older. Participants' eyes with a VA of 20/400 (1.3 logarithm of the minimum angle of resolution [logMAR]) or worse were excluded.
Methods: Patients were tested using the clinical standard Rosenbaum near card and the NYU Langone Eye Test application on an iPhone and Android device. Each test was performed twice to measure reliability. Ten patients were selected randomly for subsequent semistructured qualitative interviews with thematic analysis. Main Outcome Measures: Visual acuity was the parameter measured. Bland-Altman analysis was used to measure agreement between the results of the NYU Langone Eye Test application and Rosenbaum card, as well as test-retest reliability of each VA. The correlation between results was calculated using the intraclass correlation coefficient. Satisfaction survey and semistructured interview questions were developed to measure usability and acceptability.
Results: Bland-Altman analysis revealed an agreement between the application and the Rosenbaum near card of 0.017 ± 0.28 logMAR (iPhone) and 0.009 ± 0.29 logMAR (Android). The correlation between the application and the Rosenbaum near card was 0.74 for both the iPhone and Android. Test-retest reliability was 0.003 ± 0.22 logMAR (iPhone), 0.01 ± 0.25 logMAR (Android), and 0.01 ± 0.23 logMAR (Rosenbaum card). Of the 125 participants, 97.6% found the application easy to use, and 94.3% were overall satisfied with the application. Thematic analysis yielded 6 key themes: (1) weaknesses of application, (2) benefits of the application, (3) tips for application improvement, (4) difficulties faced while using the application, (5) ideal patient for application, and (6) comparing application with traditional VA testing. Conclusions: The NYU Langone Eye Test application is a user-friendly, accurate, and reliable measure of near VA. The application's integration with the electronic health record, accessibility, and easy interpretation of results, among other features, make it ideal for telemedicine use.

Visual acuity (VA) is the measurement of the ability to discriminate 2 stimuli separated in space at high contrast relative to the background and is the most commonly performed ophthalmic examination in clinical practice., Visual acuity is one of the most important analytical steps for ophthalmologic diagnosis, measurement of treatment effectiveness, and determinant of prognosis of disease. Various methods exist to measure VA, the gold standard being the ETDRS chart. However, other methods of testing VA are used commonly by eye care practitioners, including the Snellen chart and Rosenbaum near vision card. Although VA can be tested accurately and reliably in general ophthalmic offices, the lack of suitable remote home monitoring of VA needs to be addressed. The current coronavirus disease 2019 pandemic also has highlighted the importance of home monitoring and telemedicine. Telemedicine has emerged as a critical technology to bring medical care to patients while attempting to reduce the transmission of severe acute respiratory syndrome coronavirus 2 among patients, families, and clinicians. Similarly, technological tools such as digital images, video, web-based patient portals, tablets, and smartphones are growing in popularity as a means of health information exchange between patients and health care providers. The incorporation of smartphone technology in particular into daily modern medical practice has grown rapidly. The use of smartphone applications also has been shown to minimize costs to society associated with frequent clinic visits, lost patients’ or family members’ work hours, and transportation expenses.

The growing demand for fast and reliable measures of VA using technology is underscored by the variety of VA smartphone applications developed. However, VA applications can be released without validation or reliability testing and yet be used widely in clinical practice. Furthermore, on a review of applications that test VA found in the United States Apple App Store, Steren et al concluded that none of the applications were suitable for telemedicine use, citing lack of accuracy, ability to zoom on the letters, and the inability of the physician to interpret the results virtually as some of the main reasons for this.

The New York University (NYU) Langone Eye Test application was developed by one of the authors (L.A.A.) to address these deficiencies and be used effectively for telemedicine visits. In this study, we validated the NYU Langone Eye Test smartphone application, an application that replicates the use of the near VA screening cards used in clinical practice. We compare the results of the application with those of a clinical standard near card (Rosenbaum near card) and test the application’s reliability in comparison with that of the near card. We also included a qualitative arm in the study to gauge the usability and satisfaction of the application among patients.

Methods

Study Design

This was a mixed-methods cross-sectional study aimed at evaluating the accuracy and usability of the smartphone-based NYU Langone Eye Test application. This study was conducted at the NYU Langone Eye Center, New York, New York, from April 2021 through November 2021. Patients and their family members 18 years of age and older who sought treatment at the ophthalmology clinic were invited to participate in the study. Employees and students also were eligible for recruitment.

Ethical Approval

This study was approved by the NYU Langone Health Institutional Review Board and was conducted in accordance with the tenets of the Declaration of Helsinki. All participants provided informed consent before participating in the study, and their privacy and confidentiality were protected throughout the analysis.

Application Development

The NYU Langone Eye Test application was developed by one of the authors (L.A.A.) and her team in the Innovations Lab at the NYU Langone Department of Ophthalmology. The application was designed by clinical ophthalmologists working in conjunction with a team of software application developers. Within the application are 2 methods of measuring VA: (1) the eye test can be accessed through patients’ MyChart, and the subsequent results of the test are uploaded directly to the patients’ Epic electronic health record; and (2) the application can be used in a standalone fashion, wherein the visual test can be accessed directly within the application and the results of the test are displayed on the screen without uploading to the electronic health record. The NYU Langone Eye Test application can be downloaded for free on the Apple App Store (https://apps.apple.com/us/app/nyu-langone-eye-test/id1520661282#?platform=iphone) and Google Play Store (https://play.google.com/store/apps/details?id=org.nyumc.rpm.snapeye).

Visual Acuity Measurements

Patients underwent VA testing using the Rosenbaum near card (Supplemental Fig 1) and the smartphone-based NYU Langone Eye Test application (Supplemental Fig 2). The application was downloaded from the Apple Store and Google Play store onto an iPhone 8 Plus running iOS version 14.3 and a Motorola Moto x4 running Android version 9, respectively. Each participant was tested on both devices. Antiglare screen protectors were placed on both devices to prevent glare from light sources in the room, which can affect VA measurements.

Participants with VA of 20/400 or worse, as measured on the Rosenbaum card, were excluded from the study. This is because the application does not measure VA of 20/400 or worse because of screen size restrictions. Participants were tested in a private examination room. The first 50 participants performed the VA testing in the same room under the same lighting conditions under standardized conditions. The second 75 participants were tested in different rooms with different lighting conditions to emulate a clinical setting. One examiner tested VA of participants using the Rosenbaum card, and another examiner tested the application on the smartphone devices. We followed a masked protocol in which examiners were unaware of the previously measured VA.

When assessing VA using the Rosenbaum near card, participants were asked to hold the card at a distance of 14 inches from the eyes while wearing previously obtained corrective lenses for near vision (if any). The distance between the card and the participant’s eye was measured using a tape attached to the back of the card, and the examiner monitored this distance to avoid fluctuation during the study. Right eye VA was measured first by asking the participant to cover the left eye (monitored by the examiner to ensure the eye is covered for the duration of the study). The participants were encouraged to read the smallest optotypes possible and to make their best guess. Visual acuity was recorded for the right eye, and the process was repeated for the left eye.

Visual acuity then was assessed using the NYU Langone Eye Test application by a different examiner. The application downloaded on the iPhone was tested first. Participants were asked to read the instructions presented on the application and to hold the phone at a distance of 14 inches from their eyes. This distance again was measured using a tape attached to the phone and was monitored by the examiner throughout. Similarly, VA of the right eye was recorded first, and the process was repeated for the left eye. The VA testing using the application was repeated using the Android device. All the VA tests then were performed again on the Rosenbaum cards and smartphone devices. A visual representation of the workflow is illustrated in Figure 3.

Figure 3

Diagram showing visual acuity testing workflow.

Sample Size and Statistical Analysis

A sample size of 125 participants was chosen based on the methodology of a previous investigation, which compared a VA smartphone application and a near card. In this study, the authors were able to find a significant difference using a sample of 100 participants. Thus, we elected to use a slightly larger sample size. Bland–Altman analysis was used to show the agreement between VA measurements using the Rosenbaum cards and the application. Mean ± 2 standard deviations (1.96 × standard deviation) was used to represent the 95% limits of agreement for the Bland-Altman scatterplots. Test–retest variability similarly was examined by calculating the agreement between test and retest variability using Bland–Altman analysis.

Application Usability and Acceptability

We developed an 8-item Likert scale usability and satisfaction survey completed by all participants in the study on completion of VA tests. This survey was modified from the mHealth App Usability Questionnaire, a validated usability questionnaire for mobile health applications.

We reached out to 10 random participants for further participation in the qualitative arm of this study. After additional consent, semistructured qualitative interviews were conducted by 2 members of the research team (D.C. and G.H). Interviews were conducted and recorded using a Health Insurance Portability and Accountability Act–compliant virtual conferencing platform. Interview duration ranged from 3 to 10 minutes, with a mean of 6 minutes. Our interview questions (Table 1) were written based on the Capability, Opportunity, and Motivation for Behavior framework. These questions were designed to assess the following aspects that may impact behavior regarding our novel application: psychological capability, physical capability, social opportunity, physical opportunity, automatic motivation, and reflective motivation.

Table 1

Interview Questions

Model	Theme	Question(s)
Capability	Psychological	Did you feel confident using the application? How easy was it to use the application?
	Physical	How do you think you performed on the testing?
Opportunity	Social	When using the application, were you alone or were others around? Did you require assistance?
	Physical	What were the challenges of using the application? What were your favorite features of the application? What features did you dislike? How does this compare with in-office testing?
Motivation	Automatic	What would encourage you to use this application more often? What would prevent you from using this application? Did you have any safety or security concerns?
	Reflective	What would you say are the benefits of using this application? What do you think the value of the test at home is? What do you think the value of the test at home is? For whom do you think it is most appropriate to use this application? If you had to use this application repeatedly, is there anything you would do differently? Do you have any additional comments?

Audio recordings were transcribed verbatim. Thematic analysis was completed by 2 of the authors (G.H. and D.C.) according to Braun and Clarke’s established 6-phase protocol. Transcription texts were read and reread to ensure familiarity with interview content. Meaningful categories were documented as codes using an inductive approach. Our coding structure was refined iteratively until the 2 authors reached a consensus on a stable coding structure. Transcript texts then were uploaded onto NVivo version 12 (QSR International). The 2 authors independently coded all transcript texts. To assess the interrater reliability of the coding, we calculated a Cohen’s κ coefficient (κ = 0.80). We then examined the coded texts, using a deductive approach to organize salient codes into subthemes, which were categorized under key themes. These preliminary themes then were refined and finalized by the research team.

Results

Demographics

A total of 244 eyes from 125 participants were included in the study. Demographic data for the participants are included in Table 2.

Table 2

Demographic Characteristics

Characteristic	Data
No. of participants	125
No. of eyes	244
Visual acuity (logMAR)	0.143 ± 0.213
iPhone users	91 (79.8)
Android users	23 (20.2)
Age (yrs)
Mean ± standard deviation	47.89 ± 21.08
Range	19–100
Sex
Male	47 (37.6)
Female	78 (62.4)

logMAR = logarithm of the minimum angle of resolution.

Data are presented as no. (%), unless otherwise indicated.

NYU Langone Eye Test Application for iPhone versus Rosenbaum Near Card

The median VAs measured using the application on the iPhone and Rosenbaum near card were 0.065 logarithm of the minimum angle of resolution (logMAR) and 0.032 logMAR, respectively. Bland–Altman analysis showed an agreement of 0.017 ± 0.28 logMAR between the results of both VA measurements. Also, a positive correlation was found between the VAs with an intraclass correlation coefficient of 0.74. The Bland–Altman plots and correlation graphs for these measures are shown in Figure 4.

Figure 4

A, Bland–Altman plot showing agreement between the New York University Langone Eye Test application on iPhone and the Rosenbaum card of 0.017 ± 0.28 logarithm of the minimum angle of resolution (logMAR). B, Correlation graph between iPhone and Rosenbaum visual acuity measurements. Intraclass correlation coefficient was 0.74.

NYU Langone Eye Test Application for Android versus Rosenbaum Near Card

We also compared VAs measured between the NYU Langone Eye Test application downloaded on an Android device and the Rosenbaum near card. The median VAs measured using the application on the Android device and Rosenbaum near card were 0.065 logMAR and 0.032 logMAR, respectively. The Bland-Altman plot shows an agreement of 0.009 ± 0.29 logMAR between the results of both VA measures. The intraclass correlation coefficient between the application on Android and the Rosenbaum card was found to be 0.74. The Bland–Altman plots and correlations graphs for these measurements are shown in Figure 5.

Figure 5

A, Bland–Altman plot showing agreement between the New York University Langone Eye Test application on Android and the Rosenbaum card of 0.009 ± 0.29 logarithm of the minimum angle of resolution (logMAR). B, Correlation graph between Android and Rosenbaum visual acuity measurements. Intraclass correlation coefficient was 0.74.

Test–Retest Variability

We compared the VAs measured on the Rosenbaum card with a retest of VA measured on the same Rosenbaum card. This was used as a reference for how well the NYU Langone Eye Test application agreed and correlated with the Rosenbaum card. The results of this also were used as a reference for the test–retest reliability of the application on both iPhone and Android devices. The intraclass correlation coefficient between test and retest of the Rosenbaum card was 0.85. The correlation graph can be seen in Figure 6.

Figure 6

A, Bland–Altman plot showing agreement between the Rosenbaum card and its retest of 0.01 ± 0.23 logarithm of the minimum angle of resolution (logMAR). B, Correlation graph between Rosenbaum card and retest visual acuity measurements. Intraclass correlation coefficient was 0.85.

Test–retest variability was calculated using Bland–Altman analysis to measure the agreement in VA measurements after repeating the test with the same measurement tool. We calculated this for the NYU Langone Eye Test application on both devices and for the Rosenbaum card. The test–retest variabilities were as follows: iPhone, 0.003 ± 0.22 logMAR; Android, 0.01 ± 0.25 logMAR; and Rosenbaum card, 0.01 ± 0.23 logMAR. A visual representation of the Bland–Altman plots can be seen in Figure 7.

Figure 7

A, Bland-Altman plot showing test–retest variability of the New York University (NYU) Langone Eye Test on iPhone of 0.003 ± 0.22 logarithm of the minimum angle of resolution (logMAR). B, Bland–Altman plot showing test–retest variability of the NYU Langone Eye Test on Android of 0.01 ± 0.25 logMAR. C, Bland–Altman plot showing test–retest variability of the Rosenbaum card of 0.01 ± 0.22 logMAR.

Subgroup Analysis

A subgroup analysis was performed to determine whether testing under experimental conditions versus a clinical setting influenced the agreement, correlation, or both between the NYU Langone Eye Test application and the Rosenbaum card. The 50 participants tested under experimental conditions were found to have a higher correlation and agreement between the application and the near card compared with the 75 participants tested under conditions that reflect a clinical setting. These results can be seen in Table 3.

Table 3

Subgroup Analysis of Experimental Conditions versus Real-World Conditions in Agreement and Correlation between the NYU Langone Eye Test Application and the Rosenbaum Card

Device by Conditions	Agreement with Rosenbaum Card (Mean ± 95% Limits of Agreement, logMAR)	Correlation with Rosenbaum Card (Intraclass Correlation Coefficient)
Experimentalconditions (50 participants)
iPhone	0.008 ± 0.16	0.90
Android	0.001 ± 0.13	0.92
Real-worldconditions (75 participants)
iPhone	0.024 ± 0.35	0.66
Android	0.016 ± 0.36	0.63

logMAR = logarithm of the minimum angle of resolution; NYU = New York University.

Usability and Acceptability

A total of 97.6% of participants reported that the application was easy to use, and 97.5% reported that the instructions were easy to understand. Furthermore, 89.4% liked the design of the application, and 96.8% thought that the amount of time required to use the application was appropriate. Moreover, 91.8% reported that they would use the application as part of a future telehealth visit, and 74.6% reported that they feel more confident in seeking out a telehealth appointment with the application. Regarding satisfaction, 94.3% were overall satisfied with the application, and 67.5% preferred using the application instead of the Rosenbaum card. All results can be found in Supplemental Table 4.

Thematic Analysis

Interview data were coded into 6 overarching key themes: (1) weaknesses of the application, (2) benefits of the application, (3) tips for application improvement, (4) difficulties faced while using the application, (5) ideal patient for the application, and (6) comparing the application with traditional VA testing. These 6 key themes yielded 20 subthemes. Themes and subthemes are summarized in Supplemental Figure 8. The unabridged thematic analysis with example quotations is included in Supplemental Table 5.

Discussion

In this study we found that the NYU Langone Eye Test application corresponds well to the Rosenbaum near card and is effective at measuring near VA. Bland–Altman analysis demonstrated that the average difference between VA measured using the application on both the iPhone and Android device was within 2.9 ETDRS lines (0.029 logMAR) of the Rosenbaum card’s measured VA. Furthermore, our subgroup analysis showed that the agreement was within 1.5 ETDRS lines (0.155 logMAR) in those participants tested under experimental conditions. This is comparable with the clinical standard measurements that cannot measure a 1-line change in VA reliably. It is important to note that in this study, we compared the application with the Rosenbaum near card to demonstrate how the application performs relative to a commonly used form of testing in clinical practice. Furthermore, some studies have shown that near card VA tests such as the Rosenbaum and Runge near card have good agreement with the ETDRS chart., Therefore, we expect the NYU Langone Eye test application to function at least at a level achieved by various near cards used in current practice.

Additionally, the NYU Langone Eye Test application demonstrated high test–retest reliability, which is arguably more valuable than a one-to-one agreement with the Rosenbaum card, Snellen chart, or ETDRS wall chart. A 0.22-logMAR and 0.25-logMAR variability was found in the test–retest VAs measured by the application on the iPhone and Android devices, respectively. This was comparable with the reliability of the Rosenbaum card demonstrated in this study (difference, 0.23 logMAR between first and second test). The reliability of the application only slightly underperformed that of the clinical standard eye charts measured in previous studies. Lim et al demonstrated that in routine clinical practice, a ± 0.14 and ± 0.18 logMAR variability was found in the test–retest VAs measured by the ETDRS and Snellen charts, respectively. This is consistent with other studies that have reported a variability of approximately 2 ETDRS lines (0.18 logMAR) for clinically used VA charts.,16, 17, 18 The high reproducibility of the NYU Langone Eye Test application implies that a change in results on the application can be met with a high suspicion of VA change, which makes it an ideal method for home monitoring of VA. Similarly, this application addresses several limitations of other smartphone VA applications, as discussed in a review by Steren et al, that make them unsuitable for virtual consultation telemedicine. The inability to zoom in on the letters, free price on the application, and integration of VA results with the Epic electronic health record are some of the features that make the NYU Langone Eye test application ideal for teleophthalmology use.

In general, this application was well accepted by the participants interviewed. Weaknesses of the VA application identified by patients primarily dealt with the WiFi-dependent nature of the application, potential accuracy concerns, losing face-to-face interactions, and that this application may not be applicable or useful for every patient. The primary benefit of this application is that it allows the patient to collect more data about their vision and more flexibility in the collection of these data. Potential applications of this application suggested by our patients include postoperative care, posttraumatic eye injury monitoring, and as a screening tool that can be used to determine the need for an in-person versus teleophthalmology visit. Patients also did not express security concerns, and they enjoyed the convenience and accessibility benefits that a mobile VA application allowed. Areas of future improvement include the incorporation of parallel patient education materials and several user-interface or design suggestions. Multiple patients specifically inquired about an automatic phone–eye distancing mechanism and the potential for an application like this to perform refraction. Patients generally found this application comparable with traditional VA testing, if not better. Those patients who expressed a preference for traditional VA monitoring primarily were concerned about the accuracy of this application.

Study Limitations

This study has limitations that need to be considered. Evaluation of the agreement between the application and Rosenbaum near card was based on actual patient responses; however, optotype sizes were not measured and compared between tests. Two smartphone devices were tested in the study; however, it is possible that different devices can have variation in VA results because of different contrast ratios and pixel densities. For example, even among smartphones of the same company, the pixel densities may vary, the iPhone 8 has an approximate pixel per inch density of 326 compared with the iPhone 8 Plus, which has a pixel per inch density of 401. More research is needed among phones with the same operating system from the same companies to evaluate the impact of pixel density on the results of smartphone VA tests. It is important to test more smartphones to evaluate whether notable differences exist between devices of the same company and other companies that run on different operating systems. In our study, we also elected to use antiglare screen protectors, given that previous published studies have shown significantly improved accuracy in VA readings on devices with antiglare protectors versus those without them. Given that some patients may have limited access to antiglare protectors, further research into the impact of anti-glare on this specific application is warranted. Given the discrepancies in device operating system, pixel densities, and use of glare protectors among others, our results may not be generalizable to all devices. Furthermore, the subgroup analysis demonstrated that participants tested under standardized conditions demonstrated an increased agreement between the application and the Rosenbaum near card than those tested under varying conditions. A standardized testing environment may not be feasible when the application is used by patients at home. With this in mind, we suggest that patients use the application with the recommendations set forth in a systematic review of VA applications performed by Samanta et al: (1) the patient be oriented to the device in a clinical setting, (2) the device settings stay the same between testing dates, and (3) the patients use the same room, refractive correction, and distance for each test.