Teaming-up nurses with ophthalmologists to expand the reach of eye care in a middle-income country: Validation of health data acquisition by nursing staff in a telemedicine strategy.

Telemedicine can be used to conduct ophthalmological assessment of patients, facilitating patient access to specialist care. Since the teleophthalmology models require data collection support from other health professionals, the purpose of our study was to assess agreement between the nursing technician and the ophthalmologist in acquisition of health parameters that can be used for remote analysis as part of a telemedicine strategy. A cross-sectional study was conducted with 140 patients referred to an ophthalmological telediagnosis center by primary healthcare doctors. The health parameters evaluated were visual acuity (VA), objective ophthalmic measures acquired by autorefraction, keratometry, and intraocular pressure (IOP). Bland-Altman plots were used to analyze agreement between the nursing technician and the ophthalmologist. The Bland-Altman analysis showed a mean bias equal to zero for the VA measurements [95%-LoA: -0.25-0.25], 0.01 [95%-LoA: -0.86-0.88] for spherical equivalent (M), -0.08 [95%-LoA: -1.1-0.95] for keratometry (K) and -0.23 [95%-LoA: -4.4-4.00] for IOP. The measures had a high linear correlation (R [95%CI]: 0.87 [0.82-0.91]; 0.97 [0.96-0.98]; 0.96 [0.95-0.97] and 0.88 [0.84-0.91] respectively). The results observed demonstrate that remote ophthalmological data collection by adequately trained health professionals is viable. This confirms the utility and safety of these solutions for scenarios in which access to ophthalmologists is limited.

Introduction

Visual impairment is a serious problem in many different populations [1]. According to the World Health Organization (WHO), it was estimated that in 2015 there would be 36 million blind people in the world and millions more people with visual impairment lacking adequate treatment [2]. In Tropical Latin America, which includes Brazil, 1.64% of the population have visual problems considered severe or moderate [3]. The WHO developed a global action plan to reduce avoidable visual impairment and offer treatment to people with visual impairment. One of the action plan’s goals was to increase investment in scientific research, with the objective of developing cost-effective, evidence-based diagnostic techniques and interventions with adequate strategies for eye health care, particularly for mid and low-income populations [2].

Employing telemedicine in ophthalmology is one way of facilitating patient access to specialist care, particularly for those in localities where there is a dearth of ophthalmologists, reducing travel costs, time off work, and waiting times for consultations [4]. This technology offers real-time interaction between doctors and patients and enables examinations to be conducted at a distance [5]. Telemedicine can be used to conduct ophthalmological assessment of patients, facilitating screening for diseases such as diabetic retinopathy and glaucoma [6] and enabling correction of refractive errors [7].

Teleophthalmology models currently in use require data collection support from other health professionals, such as technology professionals, nurses, or nursing technicians. These trained professionals provide in-person support for collection of parameters, which may be conducted under ophthalmologist supervision, or unsupervised, and may be synchronous or asynchronous [7-9].

An important challenge to be overcome in implementing teleophthalmology is to guarantee the quality of the data collected by these health professionals, which is generally based on training of human resources, standardization of routines, and regular assessments of the performance required to provide the reliable data needed for distance diagnosis [8, 10]. In this scenario, assessment of the fundamental parameters measured by the professionals involved in telediagnosis is extremely relevant to achieving development and ensuring the reliability of telemedicine.

The coronavirus disease 2019 pandemic illustrates the increasingly important role of telemedicine as a method of clinician-patient interaction. In this scenario, investigating interobserver agreement between specialist doctors and support professionals when acquiring health data for transmission and remote interpretation is fundamental for the validity of remote diagnosis. The objective of this study was to assess agreement between the nursing technician and the ophthalmologist in acquisition of health parameters that can be used for remote analysis as part of a telemedicine strategy. The health parameters evaluated in this study were visual acuity and objective ophthalmic measures acquired by autorefraction, keratometry, and air-jet intraocular pressure measurement.

Methods

A cross-sectional study was conducted during the months from October to December of 2017 at a teleophthalmology center in the city of Porto Alegre, Brazil. A total of 140 patients with ophthalmological complaints who had been referred to the ophthalmological telediagnosis center by primary care (PC) doctors were enrolled consecutively on the study. The teleophthalmology center (TeleOftalmo Project) consists of two remote examination rooms where visual acuity tests and automated refractometry and keratometry acquisition are conducted, photos are taken of the anterior segment of the eye under a slit-lamp, and retinography and non-contact air-jet tonometry are performed by a nursing team (nursing technicians and nurses) with support from ophthalmologist doctors who are responsible for overseeing and supervising the examinations via video links. The off-site ophthalmologist remotely evaluates the images and data simultaneously or immediately after its collection by the nurse technicians.

The objective of the ophthalmological telediagnosis model adopted is to diagnose eye conditions that are prevalent among patients cared for on the Brazilian National Health Service (SUS—Sistema Único de Saúde). Doctors working in PC refer patients with nonspecific low visual acuity, refractive errors, strabismus, eyelid lesions, conjunctival lesions, and cataracts without prior indication for surgery and also diabetic patients for diabetic retinopathy screening. Patients in need of emergency treatment or with indications for medical or surgical treatment (e.g. laser, intraocular corticoids) were not eligible for the study [11].

Data collection was conducted independently, in two remote examination rooms equally equipped with an ophthalmology specialist experienced in ambulatory consultations and by nursing technicians (NT) who had been trained in advance to collect the data involved. The NT had undertaken a training program with ophthalmological doctors, 10 hours of which were dedicated to theoretical-practical training on acquisition of examinations conducted at the TeleOftalmo Project [11] and another 14 hours of which were dedicated to practical training with the equipment used. At the time of data collection, the NT had been working at the project for 4 months and had had no previous experience with teleophthalmology.

Patients were randomized to be examined first by the NT or by an ophthalmologist. The randomization list was generated electronically and a nurse supervisor who did not take part in the examinations was responsible for patient allocation.

All examiners followed a standardized assessment protocol used for consultations at the ophthalmological telediagnosis center and data were collected using standardized and pretested forms. After each patient had undergone all of the examinations, the NT recorded the results on an on-line platform, used to store patients’ registration data, in accordance with the normal flow of care. The ophthalmology specialist recorded the same information for each patient on an identical electronic form. Both sets of data collection were performed on the same day, in random order, and in sequential time. On average, patients were seen by the nurse technicians and by the ophthalmologist between 30 minutes and 1 hour apart.

The following data were collected: visual acuity (VA) without correction, using the modified Bailey-Lovie chart, refractomety and automated keratometry (Visuref 100®, Zeiss, Germany), and intraocular pressure (IOP), measured by non-contact air jet tonometry (Visuplan 500®, Zeiss, Germany).

Visual acuity was measured one eye at a time, covering the left eye (LE) when testing the right eye (RE) and the opposite when testing the LE, with the VA chart placed 4 meters away from the patient. During each test, the patient read the letters shown on each line out loud, or indicated the position of the tumbling E in the case of illiterate patients, starting with the line of the largest optotypes. For each measurement, the smallest line on which at least 3 of the 5 optotypes were correctly identified was noted. The measure was recorded as a Snellen fraction together with the number of optotypes not identified on the same line (for example, 20/25–2) or identified on the following line (for example, 20/25 +1). The autorefraction, keratometry, and IOP measurements were recorded as the mean of three automated measurements performed by the machines. Examinations were conducted one eye at a time, starting with the RE.

For the purposes of analysis and interpretation of results, the following cutoffs were predefined, as specified in Table 1:

Table 1

Limits of normality and expected variability for the ophthalmological parameters assessed.

Ophthalmological parameters	Limit of normality	Variability expected
Visual acuity	20/40 or 0.3 logMAR	1 line or 0.1 logMAR
Autorefraction	< 0.5 diopters	0.5 diopters
Keratometry	< 48 diopters	0.32 diopters
Intraocular pressure	≤ 21mmHg	4 mmHg

Normative values for parameters: The usual cutoff points employed in clinical practice were adopted, for analysis of the capacity of each test to detect suspected or abnormal cases. The use of cutoff points in this study was based on a theoretical scenario in which each test, in isolation, would be indicative of a given clinical condition or suspicion of that condition (for example, IOP exceeding 21mmHg as indicative of ocular hypertension, glaucoma, or suspected glaucoma). For visual acuity, a cutoff point of 20/40 was adopted, in line with the new WHO classification of visual impairment (ICD-11).

Expected variability: For the VA measurement, variability between repeated measures was expected to be in line with a previous test-retest study in the literature [12]. For autorefraction, automated keratometry and IOP, the assumed variability was that indicated by the manufacturer of the equipment, if available, or if not available, data from previous studies in the literature using similar equipment were used [13-15].

Analysis of the data

Since we have four different measures, with same importance for study objectives, with uncertainty for dispersion parameters, the sample size estimate was based on effect size. Assuming a design effect (DEFF) of 1.7 [16], the study would need a sample size of 262 pairs (131 patients) to achieve 80% power and a 5% significance level (two tailed) to detect a standardized mean difference of 0.23 (Cohen’s d) between pairs, that is considered a small to non-important effect size in health sciences [17]. Allowing for possible losses, a sample of 140 patients would be sufficient.

Continuous variables were expressed as means and standard deviations, or medians with interquartile ranges, while categorical variables were expressed as absolute and relative frequencies, with cumulative frequencies when appropriate. To perform the statistical analysis comparing the two assessments, the VA measurements were converted into the logarithm of the minimum resolution angle (logMAR), which corresponds to a continuous measure of VA with a linear scale, suitable for quantitative comparison of the mean VA. Visual acuities that were not covered by the values contained in the table employed (for example, finger counting or light/dark perception) were excluded from the analysis. The components of refraction measurement (sphere, cylinder, and axis) were converted to power vectors using the method described by Thibos and Horner [18], to obtain the independent parameters M, J0, and J45, in diopters (D). Cohen’s Kappa was calculated to measure the inter-rater reliability between the nursing technician and the ophthalmologist measurements in categorical variables. Bland-Altman plots were used to analyze agreement between the nursing technician and the ophthalmologist, in addition to scatter plots of comparison of assessments. The differences between measures taken by the nursing technicians and the ophthalmologists were assessed using mixed generalized linear models, to consider the dependence between the measurements and eyes. Sensitivity, specificity, Cohen’s Kappa and 95% confidence intervals [95%CI] were calculated for VA and IOP, using the ophthalmologists’ results as the reference standard. For visual acuity, a cutoff point of 20/40 was adopted, in line with the new WHO classification of visual impairment (ICD-11) and a cutoff point of ≥ 21mmHg as indicative of ocular hypertension, glaucoma, or suspected glaucoma.

Statistical analyses were conducted using R Core Team statistical software [19]. Informed consent was obtained from participants after explanation of the nature of the study and its possible consequences. The study was approved by the Research Ethics Committee at the Hospital de Clínicas de Porto Alegre (CAAE 64499316.1.0000.5327, consolidated opinion number 2.757.077). A printed consent form was applied and the participants signed in 2 copies, one kept by the researchers and the other delivered to the participants. The study did not include underage participants.

Results

A total of 140 patients were enrolled on the study, with a median age of 53 years [IQR: 39.0–65.0], 67.1% (94/140) of whom were female and 69.3% (97/140) of whom were white. A total of 254 visual acuity measurements were taken, counting those done by the doctors and the technicians, with 270 keratometry measurements, and 278 intraocular pressure measurements.

Twenty-six of the visual acuity measurements were off the acuity chart, because vision was worse than the lower limit (visual acuity measurements using finger count or light perception), and were therefore excluded from the analysis. Twenty-six, 10, and 2 autorefraction, keratometry, and intraocular pressure measurements, respectively, could not be collected because of equipment limitations (for example, keratometry “out of range”) or because of anatomic ocular abnormalities that prevented measurement (for example, an opaque cornea).

Agreement between visual acuity assessments

Mean VA measured by the doctor was 0.22 logMAR [95%CI: 0.00–0.50]. The difference between the measurements made by technicians and the doctors (0.00 [95%CI]: 0.06–0.08) was not statistically significant (p = 0.495).

The Bland-Altman plot (Fig 1A) showed a mean bias equal to zero for the VA measurements taken by the doctors and the nursing technicians, and the limits of agreement showed that the differences between the two measures were less than -0.25 and 0.25 logMAR. The measures had a high linear correlation (R [95%CI]: 0.91 [0.88–0.93]; p<0.001). A total of 254 eyes were assessed, of which the VA measured for 116 pairs (45.7%) were perfectly in agreement, while disagreement a maximum of 0.10 logMAR for 94 pairs (37.0%) (Fig 2A). Therefore, 210 pairs (82.7%) exhibited agreement within the expected inter-test variability (up to 0.10 logMAR). Considering the preestablished criterion for low VA (>0.3 logMAR), 108 eyes (42.5%) were detected with low VA according to the doctors’ measurements and there was disagreement for 20 eyes (7.87%). Agreement for detection of low acuity was 92.0% ([95%CI]: 88.0–95.0), with a kappa of 0.86 ([95%CI]: 0.77–0.95). The diagnostic properties for VA assessment are shown in Table 2.

Fig 1

Bland-Altman plots comparing assessments made by doctors and nursing technicians for: A) Visual acuity, in logMAR; B) Autorefraction spherical equivalent (M), in diopters; C) Keratometry, in diopters; and D) Intraocular pressure, in mmHg.

Fig 2

Percentage disagreement between doctor and nursing technicians for: A) Visual acuity, in logMAR; B) Autorefraction spherical equivalent (M), in diopters; C) Keratometry, in diopters; and D) Intraocular pressure, in mmHg.

Table 2

Diagnostic properties of assessment of visual acuity by doctors (gold standard) and nursing technicians.

	Ophthalmology specialists
Technicians	Visual impairment	VA normal	Total
Visual impairment	98	10	108
VA normal	10	136	146
Total	108	146	254

Sensitivity: 0.91 ([95%CI]: 0.84–0.95); Specificity: 0.93 ([95%CI]: 0.88–0.97); Positive predictive value: 0.91 ([95%CI]:0.84–0.95).

Negative predictive value: 0.93 ([95%CI]: 0.88–0.97).

Visual impairment defined as acuity exceeding 0.3 LogMAR.

VA, visual acuity.

Considering the order of tests, patients’ performance was better in the second VA, irrespective of whether the measurement was taken by the technicians or the doctors (data shown in Supporting Information). No differences were observed in agreement related to laterality (RE or LE).

Agreement in autorefraction assessments

The difference between means for technicians and doctors for the autorefraction spherical equivalent (0.00 [95%CI]: -0.13–0.13) was not statistically significant (p = 0.690).

The Bland-Altman plot (Fig 1B) illustrates the data for spherical equivalent (M) results. Bias was 0.01D between doctors and nursing technicians and the limits of agreement indicate that the differences between the two measures fall within the range of -0.86 to 0.88D. The correlation between measurements was 0.97 ([95%CI]: 0.96–0.98; p<0.001).

The spherical equivalent M results were in agreement for 77 assessments (30.3%), while 166 assessments (65.4%) had up to 0.5 D disagreement (Fig 2B). No differences in the agreement assessment were observed in relation to laterality or order of assessment (doctor first or technician first).

Agreement in surface curvature of the cornea (keratometry) assessments

The mean doctor-assessed keratometry result (K) was 43.8D ([95%CI]: 42.8–45.0). The difference between means for technicians and doctors (-0.13 [95%CI]: -0.25–0.00) was significant (p<0.001), but was not clinically relevant.

The Bland-Altman plot (Fig 1C) illustrates the data for keratometry (K) results in diopters (D). Bias was -0.08D between doctors and nursing technicians and the limits of agreement indicate that the differences between the two measures fall within the range of -1.11 to 0.95D. The correlation between measurements was 0.96 ([95%CI]: 0.95–0.97; p<0.001).

For the keratometry assessments, results for 59 pairs of eyes (21.9%) were in agreement and results for 203 pairs of eyes (75.2%) disagreed by less than 1D (Fig 2C). The test’s diagnostic properties could not be evaluated because just 1 participant had a value above the limit of normality (>48D) according to measurements taken by the doctor and none were outside of the limits of normality according to the nursing technician. No differences in the agreement assessment were observed in relation to left or right eyes or order of assessment.

Agreement in intraocular pressure measurements

The mean ophthalmologist-assessed IOP of patients was 15.0 mmHg ([95%CI]: 13.0–18.0). The Bland-Altman plot (Fig 1D) illustrates the data for IOP in mmHg. Bias between doctors and nursing technicians was -0.23 mmHg and the limits of agreement indicate that the differences between the two measures fall within the range of -4.44 to 4.00 mmHg. The correlation between measurements was 0.88 ([95%CI]: 0.84–0.91; p<0.001).

The difference between means for technicians and doctors (0.00 [95%CI]: -1.00–1.00) was not statistically significant (p = 0.085). A total of 72 IOP measurements (25.9%) were in agreement and 163 (58.6%) disagreed by up to 2 mmHg (Fig 2D).

Considering the outcome suspected glaucoma (IOP ≥ 21 mmHg), 20 patients had IOP ≥ 21 mmHg according to the doctors’ assessment, compared to 17 patients according to the nursing technicians’ results, with diagnostic agreement of 98.0% ([95%CI]: 96.0–99.0) and kappa of 0.86 ([95%CI]:0.73–0.98). The diagnostic properties for assessment of suspected glaucoma are shown in Table 3.

Table 3

Diagnostic properties of assessment of intraocular pressure measured by doctor (gold standard) and nursing technicians.

	Ophthalmology specialists
Nursing technicians	IOP > 21mmHg	IOP ≤ 21mmHg	Total
IOP > 21mmHg	16	1	17
IOP ≤ 21mmHg	4	257	261
Total	20	258	278

Sensitivity: 0.80 ([95%CI]:0.56–0.94); Specificity: 1.00 ([95%CI]: 0.98–1.00); Positive predictive value: 0.94 ([95%CI]:0.71–1.00)

Negative predictive value: 0.98 ([95%CI]: 0.96–1.00).

No differences in the agreement assessment were observed in relation to left or right eyes or order of assessment (S1 Fig).

Correlation plots for nursing technicians against doctors’ assessments for all measures analyzed (S2 Fig) and Bland-Altman plots for LE against RE (S3 Fig) are provided in the Supporting information.

Discussion

Our results show that there were small discrepancies between the data collected by the doctors and by technicians. However, these discrepancies did not have major clinical relevance that could impact the ophthalmologist’s decision on the patient’s diagnosis. These findings provide evidence of the safety of teleophthalmology strategies with in-person support from nursing technicians.

Some methodological aspects of our study merit more in-depth discussion. Part of the variability in visual acuity assessments can be explained by the inherent effects of repeated measures in patients and would also be observed in repeated assessments conducted by a single professional.

In relation to visual acuity, patients may memorize the letters shown on the Snellen chart because of their repeated use during ophthalmological examinations [20]. This explains the fact that patients performed better during the second administration and, as a result, analysis of agreement between measures revealed greater variability, albeit still acceptable. The acuity charts used for data collection displayed the same optotypes, which is a limitation of the study that could have facilitated memorization by the patients. Additionally, measurement of acuity requires adequate training, experience with data collection, and support from qualified professionals to be conducted satisfactorily [21]. The nursing technicians who took measurements at the project only had 4 months’ experience and this may also have been an important factor in some of the measurement discrepancies.

The autorefraction measure, expressed as the spherical equivalent M, exhibited satisfactory agreement in our study. The eye’s refraction is subject to physical accommodation fluctuations, with frequency of 1 to 5 Hz and amplitude of approximately ± 0.1 to ± 0.25 D of ideal accommodation [22, 23] Subjective refraction is considered the gold-standard method for determination of the refractive status of the eyes, because it takes into consideration important factors not accounted for by autorefractors, such as neural processes and binocular equilibrium, although it too exhibits interobserver variability [24]. Because of this variability, refraction errors are not normally assessed to precision exceeding ± 0.25 D for smaller refraction errors or ± 0.50 D for larger refraction errors [25].

The keratometry measurements exhibited good agreement in the data from our study, but the spectrum of patients was restricted to healthy patients, so the data do not reveal how good performance would be in examinations of patients with keratoconus or other diseases of the cornea. Since it is a disease that does not reveal symptoms or clinical signs, it is extremely important to diagnose keratoconus during the initial stages of the disease [26]. Measurements of refraction and visual acuity are less sensitive for keratoconus than topography, but changes in these measurements are nevertheless important signs of the progression of this disease [27]. Our results cannot therefore be generalized to patients with suspected keratoconus.

The variability of examinations such as intraocular pressure measurement is dependent on minor variations such as patient positioning and breathing. In our study, agreement between tonometry measurements taken by nursing technicians and ophthalmology specialists was acceptable, but discrepancies did affect the classification of diagnostic suspicion for some patients. A number of important factors can interfere with this measure. In this study, patients were examined using Zeiss non-contact tonometers, with an expected variability of up to 4 mmHg, according to the manufacturer [28] and since the variability between professionals was within this limit, we can infer that the nursing technicians’ assessment did not exhibit variability, considering expected variability. Non-contact tonometers (NCTs) have the advantage of pushing back the cornea with an air jet, reducing the possible risk of cross-infection, but are not always precise [29, 30]. Repeated readings with contact tonometers are more reproducible than other automated measurements [29] and are the gold standard for IOP measurement. Kotecha et al. [31] published studies showing that ophthalmologist doctors achieve better agreement in IOP measurements using contact tonometers and nursing technicians have better agreement with non-contact tonometers. Other variables can also interfere with IOP measures, including astigmatism, corneal thickness, biomechanics, accommodation, breathing, heart rate, and daily variation, resulting in a maximum clinically acceptable variation of ± 3.0 mmHg [32].

Although the study has some limitations, such as the nursing technicians’ length of experience with assessment of ophthalmological parameters (4 months), use of air-jet tonometry measurements only, and use of acuity charts showing the same optotypes, the results observed show that assessment by nursing technicians exhibits acceptable agreement.

One relevant aspect of the setting in which this study was conducted is that there is no specific professional category for ophthalmological assistants in Brazil and so very often nursing technicians are trained to perform this role, as is the case at the service where the study was conducted. Similar to Brazil, other countries in Latin America don’t have standard professionals who can assist ophthalmologists. In Peru, for example, there are few certified ophthalmological nurses, the majority of them being trained in their work center only. In Honduras, healthcare professionals who have a role assisting in ophthalmologic procedures do not have formal training. The optometrist is only formally recognized in Peru, Honduras and Mexico. Meanwhile, countries such as Chile and Uruguay have ophthalmological technicians who collect refraction data [33].

It is known that continuous training and continuing education lead to establishment of standards for the quality of the assessments conducted by these professionals. In other countries, ophthalmology assistant professionals have helped in screening and follow-on ophthalmological diseases such as glaucoma and diabetic retinopathy in primary care [34-36].

Even though some studies have shown good accuracy in telemedicine diagnosis in the ophthalmology field [37-39]. The evaluation of the data necessary to perform a synchronous remote diagnosis, collected by technicians who are not specialists in ophthalmology, is important in order to validate the care provided by a non-medical professional for ophthalmological diagnosis by telemedicine.

The results found demonstrate that actions employing distance ophthalmological data collection by adequately trained health professionals with secondary or technical education are feasible options for providing care to populations that do not have access to specialist services. With support from adequately trained health professionals, ophthalmological healthcare provided via telemedicine enables access to eye health to be extended to populations with limited access to traditional services.

Graphs illustrating correlation between assessments by nursing technicians and doctors: A. Visual acuity, in logMAR; B. Spherical equivalent M, for refraction in diopters; C. Keratometry, in diopters; D. Intraocular pressure, in mmHg.

(PDF)

Click here for additional data file.

Graph illustrating correlation and discrepancies between measures by order of examination: A) Visual acuity; B) Spherical equivalent M; C) Keratometry; D) Intraocular pressure.

(PDF)

Click here for additional data file.

Assessment correlation graphs and Bland-Altman plots comparing assessments by doctors and nursing technicians for: A) Visual acuity for right eye; B) Visual acuity for left eye; C) Spherical equivalent M for right eye; D) Spherical equivalent M for left eye; E) Keratometry for right eye; F) Keratometry for left eye. G) Intraocular pressure for right eye; H) Intraocular pressure for left eye.

(PDF)

Click here for additional data file.

Database acquisition by nursing staff.

(XLSX)

Click here for additional data file.