Comparison of wide field imaging by nurses with indirect ophthalmoscopy by ophthalmologists for retinopathy of prematurity: a diagnostic accuracy study.

OBJECTIVES: Retinopathy of prematurity (ROP) is a vasoproliferative disease of the preterm retina with the potential to cause irreversible blindness. Timely screening and treatment of ROP are critical. Neonatal nurses trained in wide field digital retinal photography (WFDRP) for screening may provide a safe and effective strategy to reduce the burden of ophthalmologists in performing binocular indirect ophthalmoscopy (BIO). The objective of the study was to determine the diagnostic accuracy of WFDRP in the diagnosis of referral warranting ROP (RWROP).
DESIGN: Prospective diagnostic accuracy study.
SETTING: A tertiary neonatal intensive care unit in Perth, Western Australia. PARTICIPANTS: Preterm infants who fulfilled the Australian ROP screening criteria (gestational age (GA) <31 weeks, birth weight (BW) <1250 g). INTERVENTION: Sets of 5-6 images per eye (index test) were obtained within 24-48 hours prior to or after the BIO (reference standard), and uploaded onto a secured server. A wide field digital camera (RetCam, Natus, Pleasanton, California, USA) was used for imaging. A paediatric ophthalmologist performed the BIO. The ophthalmologists performing BIO versus reporting the images were masked to each other's findings. PRIMARY OUTCOME: The area under the receiver operating characteristic (ROC) curve was used as a measure of accuracy of WFDRP to diagnose RWROP.
RESULTS: A total of 85 infants (mean BW; 973.43 g, mean GA; 29 weeks) underwent a median of two sessions of WFDRP. There were 188 episodes of screening with an average of five images per eye. WFDRP identified RWROP in 7.4% (14/188 sessions) of examinations. In one infant, BIO showed bilateral plus disease and WFDRP did not pick up the plus disease. WFDRP image interpretation had a sensitivity of 80%, specificity of 94.5% for the detection of RWROP. The 'area under the ROC curve' was 88% when adjusted for covariates.
CONCLUSIONS: WFDRP by neonatal nurses was feasible and effective for diagnosing RWROP in our set up. TRIAL REGISTRATION NUMBER: ACTRN12616001386426. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The expertise of neonatal nurses was used for successful implementation of nurse-led wide field imaging for retinopathy of prematurity (ROP) screening in the sole tertiary neonatal unit in Western Australia.

The validity of the results of this prospective diagnostic accuracy study is enhanced by masking of the ophthalmologists.

Limitations include the low prevalence of referral warranting ROP and lack of image interpretation by the nurses.

Introduction

Retinopathy of prematurity (ROP) is a vasoproliferative disease of the preterm retina with the potential to cause irreversible blindness.1 Hence, the significance of timely screening and treatment of ROP cannot be understated. The demand for ROP screening has increased globally, following increased survival of extremely preterm infants.2 Currently, Australia and New Zealand guidelines recommend that all infants with a birth weight under 1250 g or gestation under 31 weeks should be screened for ROP.3

The standard screening method for ROP involves binocular indirect ophthalmoscopy (BIO) performed by a qualified ophthalmologist.4 However, considering the shortage of such specialised workforce, it may be difficult to meet the increased demand for ROP screening. Considering this imbalance in demand versus supply, wide field digital retinal photography (WFDRP) has been suggested as an alternative for ROP screening.

We have systematically reviewed studies on WFDRP by non-ophthalmologists as a potential strategy to address the shortage of ophthalmologists for ROP screening. Six studies were included in the review (three prospective; N=120, three retrospective; N=579) where the non-ophthalmologists acquired the images and the images were interpreted by the ophthalmologists.5–10 All had methodological limitations based on the assessment by the Quality Assessment of Diagnostic Accuracy Studies-2 tool. Meta-analysis could not be performed to derive pooled estimates for sensitivity and specificity. Overall, the six included studies reported sensitivity of 45.5%–100% with the majority >90%; specificity 61.7%–99.8% with the majority >90%, positive predictive value 61.5%–96.6% and negative predictive value of 76.9%–100% for diagnosing clinically significant ROP.11 Subsequent to our systematic review, the e-ROP investigators emphasised the importance of validation of imaging, and assessment of logistical issues (eg, availability of ROP specialists, workload and prevalence) before adopting WFDRP in neonatal units.12

Some of the level III neonatal units in Australia and New Zealand have adopted WFDRP to reduce the workload of the ophthalmologists.13 14 Our unit is the sole neonatal tertiary referral centre for the state of Western Australia. It annually admits ~300 preterm infants <32 weeks gestation, including ~120 born before 28 weeks. Until 2016, we had only two ophthalmologists available for ROP service. Gilbert et al15 have suggested that neonatal nurses and doctors should take ownership of screening for effective management of ROP. The success of such an approach on a large scale in a resource-limited set up is documented by the success of the KIDROP programme in India.16 Considering the expertise of neonatal nursing staff in neonatal intensive care unit (NICU), their understanding of neonatal pathophysiology, and availability on a shift basis, we aimed to assess the feasibility and validity of a neonatal nurse-led WFDRP programme for identifying infants needing referral for ROP.

Our hypothesis was that images taken by the trained nurses using WFDRP and interpreted by an off-site ophthalmologist by telehealth technology will be able to identify all infants with referral warranting ROP (RWROP).17

Methods and participants

Study design and setting

This was a single centre prospective diagnostic accuracy study conducted in our level III neonatal unit at KEM Hospital, Perth, Western Australia. Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research.

Eligibility

Parents of eligible infants who fulfilled the Australian and New Zealand Neonatal Network ROP screening criteria3 were approached for consent before enrolling their infant in the study.

Screening protocol

The first examination was based on the recommendations of American Academy of Pediatrics, American Academy of Ophthalmology and American Association for Pediatric Ophthalmology and Strabismus.18 The follow-up examinations were determined by the ophthalmologists performing the BIO examination.

Standard examination

BIO was performed by a screening ophthalmologist using a 28D condensing lens and scleral depression. Pupillary dilatation was achieved with 2.5% phenylephrine and 0.5% tropicamide and the BIO was performed after the pupils were dilated.

Index test

WFDRP was performed by trained neonatal nurses within 24–48 hours of the routine BIO, using a standard imaging protocol. The images were obtained by the wide field digital camera (RetCam, Natus, Pleasanton, California, USA), which give 130° field of view after pupillary dilatation. For each eye, a set of five images (sisc positioned in the centre, and then to the extreme nasal, temporal, superior and inferior to visualise as much retina as possible) was obtained and uploaded on to the secure server. The images were read using telehealth facilities off-site by an ophthalmologist (figure 1)

Figure 1

The workflow of two screening models. NICU, neonatal intensive care unit.

Training of neonatal nurses

Imaging staff were recruited from the pool of neonatal trained nurses. They completed the online learning package (Natus) before at least ten supervised practical sessions using a model eye. They also had regular feedback on their imaging sessions from the ophthalmologist and ongoing education in ROP.

The details of ROP (stage, zone, plus disease) were recorded as per the international guidelines. The examinations continued until the infant was considered no longer at risk of developing sight-threatening ROP (usually at 37 weeks postmenstrual age), and the retinal vessels had extended beyond zone II.2 18–21 RWROP was defined as (1) any zone I disease or (2) any stage 3 or more diseases OR (3) presence of plus disease. Treatment decisions were based on the findings of the gold standard BIO.17

Strategy for masking and ensuring best clinical care

During the study, ophthalmologist SR sent his BIO reports to investigator SEA. The WFDRP images obtained by the nurses were interpreted by ophthalmologist GCL who reported his findings to investigator SEA. After receiving reports from BIO (SR) and WFDRP (GCL), investigator SEA notified both ophthalmologists if clinically significant ROP was reported by either one or both of them, to ensure best clinical care. At no time did SCR and GCL communicated about their findings to ensure optimal masking.

Data collection

The data included birth weight and gestation, gender, age at ROP screening and the findings of consecutive BIO and WFDRP.

Sample size

In order to estimate sensitivity and specificity of 95% (comparing the wide field imaging to the BIO) and assuming a prevalence of RWROP as 10%, a minimum of 183 subjects was required to estimate sensitivity and specificity within ±10% (clinically accepted precision) for the 95% CI.

Approach to analysis

Diagnostic test evaluation uses sensitivity and specificity as measures of accuracy comparing with the standard examination. In conditions like ROP where the results are recorded in the ordinal scale, the sensitivity and specificity can vary across different thresholds. In this situation, plotting of sensitivity versus 1−specificity can be an effective measure of accuracy and can give a meaningful interpretation. The plotting is called receiver operating characteristic (ROC) curve and definite integral below the two points are called the area under the curve (AUC).22 Statistical analysis was performed by using IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY: IBM Corp.

Patient and public involvement

Patients and the public were not involved in the design, conduct, reporting or dissemination of our research.

Results

A total of 85 infants were recruited during the study period. Their median birth weight and gestation were 973 g and 29 weeks, respectively. They were screened at 4 weeks of age but not before 31 weeks’ postconception age. There were 188 episodes of paired eye examinations resulting in images from 366 eyes. On an average, five images were obtained for each eye. The number of examinations per infant and the corresponding number of infants are shown in table 1.

Table 1

Number of examinations per infant and the corresponding number of infants

Examinations (n)	Infants (n)
1	35
2	24
3	11
4	10
5	5
7	1

The flow of participants and the results are depicted in figure 2.

Figure 2

STARD flow diagram of the study comparing the WFDRP and BIO. BIO, binocular indirect ophthalmoscopy; STARD, Standards for Reporting of Diagnostic Accuracy Studies; WFDRP, wide field digital retinal photography.

Despite the slightly hazy images in eight examinations, interpretation was possible from all acquired images. All WFDRP reports were returned by the ophthalmologist to the NICU staff within 12 hours of imaging. WFDRP identified RWROP in 7.4% of examinations (14/188 sessions). The comparison of wide field imaging findings with clinical examination findings for RWROP is given in table 2.

Table 2

The comparison of wide field imaging findings with BIO examination for RWROP

Imaging diagnosis of RWROP	BIO diagnosis of RWROP		Total
	Positive	Negative
Positive	4	10	14
Negative	1	173	174
Total	5	183	188

BIO, binocular indirect ophthalmoscopy; RWROP, referral warranting retinopathy of prematurity.

The results of WFDRP examinations were subjected to ROC curve analysis and other measures of accuracy were calculated with 95% CIs to determine their diagnostic values. Compared with BIO by the ophthalmologist, remote interpretation of WFDRP images had a sensitivity of 80%, specificity of 94.5%, positive predicative value of 28.57% and negative predicative value of 99.43% for the detection of RWROP. The AUC for diagnosing RWROP was 0.883 (figure 3) and the summary values are given in table 3.

Figure 3

Covariate control adjustment model area under the ROC curve=0.88 for diagnosing referral warranting ROP. ROC, receiver operating characteristic; ROP, retinopathy of prematurity.

Table 3

Area under the ROC curve and other measures of accuracy with 95% CIs

Observations (n)	376
Area under the ROC curve (95% CI)	0.883 (0.66–1.10)
Sensitivity, % (95% CI)	80 (28.4–99.5)
Specificity, % (95% CI)	94.5 (90.2–97.3)
Positive predictive value, % (95% CI)	28.6 (8.4–58.1)
Negative predictive value, % (95% CI)	99.4 (96.8–100)

ROC, receiver operating characteristic.

In one infant, BIO showed bilateral plus disease whereas the simultaneous WFDRP was interpreted as ‘on the verge of becoming pre-plus’ and an earlier review within 48 hours was sought by the WFDRP interpreting ophthalmologist (GCL). Eventually, the eye findings were deemed significant to warrant treatment by both ophthalmologists. In another infant, WFDRP showed bilateral plus disease that was not detected by BIO. This infant was ultimately treated with laser therapy.

In six examinations, there were disagreements on the presence of RWROP between WFDRP versus BIO on earlier examinations. The findings by both BIO and WFDRP were in total agreement on subsequent examinations. The timing of diagnosis of RWROP was compared between WFDRP and BIO and the difference was not statistically significant (p=0.3173). The WFDRP detected RWROP earlier than BIO in six infants. In four of these six infants, the disagreement about the presence of RWROP involved both eyes, whereas it involved only one eye in the remaining two infants.

The BIO failed to detect RWROP for 2 weeks in two infants (both eyes: one infant, one eye: one infant). In one infant, WFDRP detected RWROP, but not by BIO. Subsequent WFDRP examination in the same infant showed no RWROP.

On the other hand, in three infants RWROP was detected earlier by BIO. Two of these infants had stage 3 disease detected on BIO but not on WFDRP. Subsequent BIO examinations showed stage 2 disease in these two infants. The other infant was thought to be developing the pre-plus disease by WFDRP, but deemed to be at ‘plus’ disease by BIO. There were no adverse events related to the eye examinations during the study period.

Discussion

Our results indicate the feasibility of implementing neonatal nurse-led ROP screening and validity of WFDRP in identifying RWROP versus the standard BIO in our set up. WFDRP images had a sensitivity of 80%, specificity of 94.5%, positive predicative value of 28.57%, and negative predicative value of 99.43% for the detection of RWROP. The AUC of 88% indicates that WFDRP is a reliable tool for detecting RWROP. Our results are supported by other studies which did not involve neonatal nurses for retinal imaging.12 23

The photographic screening for ROP (photo-ROP 2008) study is one of the earlier prospective multicentre studies that compared digital imaging (RetCam-120 camera) versus BIO by an ophthalmologist.24 The primary outcome was clinically significant ROP (CSROP) defined as findings on digital images severe enough to warrant an on-site examination. The results showed a sensitivity of 92% and specificity of 37.21% for CSROP and sensitivity of 92% and specificity of 67.39% for early treatment for ROP (ETROP) prethreshold type I. There was no significant difference in timing of ROP diagnosis by WFDRP or BIO provided that the images were readable. It is important to note that an ophthalmologist obtained the images using an older generation camera. In the largest multicenter study until where imaging technicians obtained retinal images using the new generation camera, the sensitivity for detection of RWROP increased from 81.9% to 90% when both eyes were considered for analysis.12

The inability to reach the sensitivity of 100% and the difficulty in appreciating the subjectivity in diagnosing retinal vascular changes needs to be discussed. We missed a case of RWOP which was picked up by BIO, considered as the standard test. However, the limitations of BIO are clear in two of the largest studies (CRYOROP, 1988 and ETROP, 2003) that reported 12% and 15% disagreement between the first and second (‘confirmatory’) BIO, respectively.25 26 It is important to note that the second examiner was assumed to be correct in these two studies.

In our study, plus disease was noted on BIO in one infant but the ophthalmologist reading the WFDRP disagreed with this. Slidsborg et al27 reported poor inter-reader agreement between four international ROP experts based in different countries, for diagnosis of aggressive posterior ROP in any of the 243 images. Wallace et al28 reported poor agreement on the plus and pre-plus disease in 10% and 27% of infants, respectively, when three experienced ROP experts assessed cropped retinal images from 181 infants. Chiang et al29 reported on a set of 34 retinal images interpreted by 22 ROP experts using a ‘3-level’ (plus, pre-plus or neither) and ‘2-level’ (plus or not plus) categorisation. There was an agreement only on 4/34 images in 3-level and 7/34 images in 2-level categorisation.29

Larger studies (eg, e-ROP: 188 image sets) have reported statistically significant discrepancy between WFDRP and bedside BIO for diagnosing RWROP.30 The trend towards earlier diagnosis of RWROP by WFDRP did not reach statistical significance in our study, probably due to the small sample (p=0.3173). The e-ROP study reported minor (4.9%) adverse events (eg, desaturation, bradycardia) during imaging in 1257 infants.31 There were no adverse events related to ROP screening in our study, except for minor oxygen desaturations that did not need intervention. The expertise of neonatal nurses in handling infants during ROP screening may explain these findings.

The expertise of neonatal nurses was used for the successful implementation of nurse-led wide field imaging for ROP screening in the sole tertiary neonatal unit in Western Australia. Except for a few, the quality of images was satisfactory for interpretation. The validity of our results is enhanced by masking of the ophthalmologists. Our approach also ensured the best clinical care for the infants.

The limitations of our study include its small sample size, low incidence of RWROP and lack of image interpretation by nurses. Furthermore, we did not assess the time taken for each retinal screening. Given the relatively small population size of Western Australia, there were inherent difficulties in achieving a larger sample size. The lack of image interpretation by nurses reflects our logistical difficulties in staffing the unit, which is one of the largest and busiest tertiary NICU in the Southern hemisphere. However, we plan to focus on credentialing neonatal nurses in the interpretation of retinal images. Assessing the reproducibility of our results in other set ups is important for guiding clinical practice.

In conclusion, we report the successful implementation of neonatal nurse-led ROP screening in our set up and the validity of WFDRP in identifying RWROP using our approach. Large studies are required to confirm whether the trend favouring an earlier diagnosis of RWROP by WFDRP reaches statistical significance.